Setup Drone cache

[allanrenucci]

Here is my take at setting up a cache for Drone: dotty-staging@345cf90

The issue is that we have four instances of sbt running in parallel sharing the same cache. When there is nothing to update everything works fine but if sbt tries to update the ivy2 cache, it fails with the following exception:

Stacktrace

Waiting for lock on /drone/cache/ivy2/.sbt.ivy.lock to be available...
java.io.IOException: Resource deadlock would occur
	at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
	at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90)
	at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1073)
	at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
	at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:89)
	at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
	at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
	at xsbt.boot.Using$.withResource(Using.scala:10)
	at xsbt.boot.Using$.apply(Using.scala:9)
	at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
	at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
	at xsbt.boot.Locks$.apply0(Locks.scala:31)
	at xsbt.boot.Locks$.apply(Locks.scala:28)
	at sbt.IvySbt.withDefaultLogger(Ivy.scala:65)
	at sbt.IvySbt.withIvy(Ivy.scala:128)
	at sbt.IvySbt.withIvy(Ivy.scala:125)
	at sbt.IvySbt$Module.withModule(Ivy.scala:156)
	at sbt.IvyActions$.updateEither(IvyActions.scala:168)
	at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1488)
	at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1484)
	at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$121.apply(Defaults.scala:1519)
	at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$121.apply(Defaults.scala:1517)
	at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:37)
	at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1522)
	at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1516)
	at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:60)
	at sbt.Classpaths$.cachedUpdate(Defaults.scala:1539)
	at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1466)
	at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1418)
	at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
	at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
	at sbt.std.Transform$$anon$4.work(System.scala:63)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
	at sbt.Execute.work(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
  

So for now, we don't have a better solution than updating the docker image every time a dependency is updated.

[allanrenucci]

cc/ @liufengyun

[OlivierBlanvillain]

We could maybe work around the issue by lunching the first sbt instance "as is", and for the 3 other do

$ cp -r path/to/.ivy2 path/to/ivyN # for N = 1 2 3
$ sbt -ivy path/to/ivyN

[allanrenucci]

We could maybe work around the issue by lunching the first sbt instance "as is", and for the 3 other do...

Drone will update the cache with what's in path/to/.ivy2. The downside with this solution is that only the fist instance of sbt will be able to update the cache. The instances of sbt don't run the same tests and don't pull the same dependencies.

[OlivierBlanvillain]

Maybe we could add another sbt instance just for that, ie N = 0 does sbt */update on path/to/.ivy2 and nothing else.

[anatoliykmetyuk]

I believe the cache is already implemented. When the Docker image is created, it builds the cache by calling updateCommunityBuild:

https://github.com/lampepfl/dotty-drone/blob/2c350e6ff3bdac2ffcfc0bbe20421469bc59e5aa/dotty-docker/Dockerfile#L29

This command will run the test suite CommunityBuildTest. This suite is identical to the one that runs the community build tests – except instead of running these tests, an update command is executed to fetch all the dependencies in the Ivy cache:

https://github.com/lampepfl/dotty/blob/1246b4bb3db980ab717d2635555baca49ea2c88b/community-build/test/scala/dotty/communitybuild/CommunityBuildTest.scala#L64

/cc @smarter

[allanrenucci]

Drone caching is not implemented. The point of setting up Drone caching is to not have to update the Docker image every time dependencies change.

[anatoliykmetyuk]

So the idea is to not store the deps in the image itself but have a separate cache independent from the image, which the image accesses?

[allanrenucci]

So the idea is to not store the deps in the image itself but have a separate cache independent from the image, which the image accesses?

Exactly. For example, drone-volume-cache is a plugin that uses a volume on the host machine for the cache.

[smarter]

Closing as we're migrating to github actions which has a caching mechanism: #8353