Cache operations can take up a long time. For our builds, the do in fact take up the majority of the time. Right now that works roughly as follows:

• On restore
  • A compressed archive is downloaded to a location
  • After that is complete, the tarball is unpacked into position
• On store
  • A compressed archive is created and written to a temporary location
  • That tempfile is then uploaded to the cache

For example, this is the code that actually does the restore:

The archives are a tar file, which supports streaming (de)compression. Thus, it should be possible to interleave the download and unpacking (as well as packing and upload) resulting in less overall latency.

In the bash version this would've been as simple as piping the sftp output to tar -x and vice-versa for uploads; in Go this might be slightly more tricky but in general possible and an easy win for faster builds.