Kvamme62/zgycopyc misc

Merged Paal Kvamme requested to merge kvamme62/zgycopyc-misc into master

The main purpose of the changes is to prevent a ZGY copy aborting because the token that is currently used for ad-hoc testing times out after 60 minutes.

Tokens can now be read from file, with the file being polled every 5 minutes for changes. The refresh is not automatic. The user has to invoke some command to get a valid token every 55 minutes or less. But at least it is possible to get the job done.

The PR also implements the --noise option in the C++ version of the copy tool. This is used for testing when it is desirable to have no dead traces.

Copying is now allowed between two different repositories, allowing sdurl to be specified separately for the source and the target. This means you don't need a local file for intermediate storage and you don't need to build the low resolution data twice. The data being copied will still be copied into and out of memory buffers in the machine doing the copy. So it is still a good idea to do large copies from a cloud VM.

Some scripts to help set up a cloud VM for this kind of copying have been updated.

Also add --update and --finalize command line options to the copy tool. These are mainly intended for testing. If or when update gets supported on the cloud these options can also be used as an alternative way of preventing token expiration in long running copies.

Merge request reports