Draft: Adding CI logic for continuous release packages.

These can be used for any project that wants a new release version every time an MR merges to the default branch. However, the intention is to use these on libraries, rather than services.

How does it work

Each time a pipeline runs on the default branch, it will check to see if any MRs have been merged into the default branch that were not part of the most recent tag. If it finds any, it computes the next version number by looking at the impact level (major/minor/patch) of the new MRs, then automatically creates the tags.

Impact Level is determined by scanning for labels on the MRs: ImpactMajor , ImpactMinor, or ImpactPatch. These correspond as you'd expect -- Major implies breaking changes, Minor is feature extensions with backwards compatibility, and Patches are fixes with no feature changes.

Latest Release is derived from Tags

To determine the latest release, the logic will scan for all Git tags and parse them into version numbers. It looks for the largest number, and uses that as the basis for computing what the next version should be.

This technically means you can "unrelease" by deleting the tag. That can be used to trick the logic into re-using a version number. This is probably a bad idea -- deleting and reassigning tags in Git can cause confusion -- and you should be careful before doing something like this.

Special Case: Major Version 0

If the latest release is a 0.x version, then the code is considered to be in a young, unstable state. Backwards compatibility is never presumed, and both major and minor MRs will increment the second version number (the "minor" version).

Whenever a service is ready to move to the big 1.0.0 release, run a pipeline with the STABLE_RELEASE environment variable set to true[1]. This will force the version selection to move up to 1.0.0, creating the tag for it.

Once a version has been release with a major version > 0, the special case no longer applies (regardless of whether the STABLE_RELEASE flag is set or not). From then on, MRs with major impact update the first number, and MRs with minor impact update the second.

[1] -- Technically, any non-zero integer or matching any of these regexs works: /^true$/i, /^t$/i, /^yes$/i, /^y$/i

Missing Impact Statements

If an MR is recently merged into the default branch, but doesn't have an impact statement, the CI pipeline will fail. It will output an error message that the MR is missing an Impact statement, and refused to create a release because of that.

To fix this, edit the MR and add the label. This can be done after the merge, and doesn't require re-opening, reverting, or otherwise messing with the Git history. Then, simply re-run the job on the default branch's pipeline (or launch a new one) to get it to create the appropriate tag.

Multiple MRs

Most of the time, each version will be created based on the impact of a single MR that just merged. Since the tag is made every time a pipeline runs on the default branch, the normal routine is to merge one MR and get a tagged version from it.

However, there are some edge cases that lead to multiple MRs merging "at once". One case is if the impact label is omitted from the MR. The pipeline will refuse to create a release until it is added. If another MR merges before the label is added, then once the labels are in place, there will be multiple MRs to release at once. Another case would be if MRs are merged into other feature branches, which are then merged into the default branch. This single action brings multiple MRs in at once.

In these cases, it uses the highest impact level to determine the result. This should feel fairly intuitive to users.

Squash Merges

When squash merging an MR, you lose the commit history. Since Git is using the commit graph to determine reachability, squashing can impact it in some rare circumstances.

In a squash merge, GitLab will create a new commit representing the squashed content, then merge that new commit into the default branch. The reported merge_commit_sha will the merge commit between this new squashed content and the default branch, so it will still be detected as a reachable MR.

If an MR is squash merged into a feature branch, and then that feature branch is squashed merged into the default branch, you'll lose traceability of the original MR, and it will show as unreachable despite being merged in. The second squash merge erased the merge_commit_sha of the first MR, so the lineage is lost.

Deeper analysis could be written to look for these cases, but they are considered to be rare enough to omit for brevity.

Build System Specifics

Each library is intended to include continuous-release-${SYSTEM}.yml into their main pipelines, where ${SYSTEM} refers to their build system. Those all in turn include continuous-release-general.yml, which has some common functionality.

Maven

The Maven build system will invoke an mvn versions:set to apply the new version to the pom.xml files, in addition to creating the version tag. These pom.xml changes are committed to a detached HEAD commit, which becomes the tag. The main branch is not modified.

Python

The Python build system will output the version to a special file named VERSION. This is also committed to a detached HEAD, which becomes the tag, and leaves the main branch unmodified.

In the case of Python, it is likely that the VERSION file will already be set to the correct value. This means the git commit will have no effect, and the tag will be applied to the same commit as the branch.

Simple

The simple build system is for situations where no changes are required. It applies the tag directly to the default branch and pushes it.

Backports

In addition to creating tags for releases, the CI logic will also drop behind release branches in two different flavors: major release branches and minor release branches. The major release branches track all the changes that are part of the latest release in that major version family. The minor release branches similarly track changes part of the latest release in that minor version family (which is a dotted pair).

For example, if the latest release on a library is 2.1.0, the you would have the following:

• Major Release, release/2 -- effectively the same as main, it contains all MRs
• Major Release, release/1 -- contains all MRs up to the latest 1.*.* version
• Major Release, release/0 -- contains all the unstable MRs, up to the latest 0.*.* version
• Minor Release, release/2.1 -- effectively the same as main, it contains all MRs
• Minor Release, release/2.0 -- contains all the patch MRs up to the latest 2.0.* version
• Minor Release, release/1.0, release/1.1, etc. -- All patches up to the corresponding minor version
• Minor Release, release/0.1, release/0.2, etc. -- All patches to the unstable versions

Normal Workflow

Normally, new MRs would merge into main with an appropriate Impact label. The CI pipeline would automatically make a new tag for the release, with the appropriate version number. Then,

If the MR is of type ImpactPatch, the latest major & minor release branches are updated by merging them together with main. Since these always contain the latest code, this merge will not have any conflicts. After everything settles, the latest major & minor branches will still be up to date with main, containing all MRs that main does. Developers do not need to cherry-pick MRs into the two release branches.

If the MR is of type ImpactMinor, the latest major release branch is updated through merging, like above. However, a new minor release branch is created for the new version family. The previous minor release branch is left behind, and is now an available target for cherry-picking and direct patching.

If the MR is of type ImpactMajor, no branches are merged into main. Instead, new major & minor release branches are created, leaving the others behind to be targets for cherry-picking and direct patching.

Cherry-picking into a Past Release

Sometimes a patch (or possibly a minor update) will be cherry-picked into past release branches. To do this, first merge the MR into main as usual, then cherry-pick the MR into either the release/N or release/N.M branch, as appropriate. Note that ImpactMinor cherry-picks can only be merged into major release branches. ImpactPatch cherry-picks can be applied to either.

If the cherry-pick goes into a major release branch, then the latest minor release branch of that family is automatically updated via merging, like the normal workflow above. For example, if the latest version is 2.0.0 and the latest version 1 code is 1.2.0, cherry-picking a patch into release/1 will create a new tag and update the release/1.2. The result will be a tag named 1.2.1, and both the release/1 and release/1.2 branches will point to this.

Direct Patches

Sometimes a patch only reasonably applies to a previous release, and it doesn't make sense to apply it to main first. In these cases, simply open a direct MR targeting the appropriate release branch. The CI logic treats the tagging process identically for these kinds of MRs as it does for cherry-picks.

Choosing the right Release Branch

Though the CI logic does not currently throw errors or prevent the case, you should never try to merge into a release branch that represents the latest version of its type. If you want to update that version, merge to its parent type. For example, if the latest major release branch is release/2, and you want to apply a patch to it, instead merge the patch to main. The logic there will automatically update the release/2 and the release/2.X branches.

Similarly, when to apply a patch to the latest of a minor release family, apply it to the corresponding major release branch instead. It will merge into the one you want.

Prep Commits

After creating a release, the release scripts will create Prep commits for the default branch, major release branch, and minor release branch. The prep commit sets the version to a SNAPSHOT version (or equivalent), with the main version bumped by one minor number (for default / major) or one patch number (for minor). This is important to do, so that any new commits being applied to these branches (like merged MRs) do not overwrite the released package, but instead update a SNAPSHOT.

To Do

These items/features are still needed before the MR can be merged.

Avoid Maven Prep Commits?

For maven systems, it may be beneficial to change the artifact version to ${env.CI_COMMIT_REF_NAME}-SNAPSHOT for all branches. This way, the version never needs to be updated, and it automatically codifies the "work in progress" version for a branch -- that is, services can test a library before it merges by linking (temporarily) to ${BRANCH_NAME}-SNAPSHOT. That pattern is already is use, but requires manual intervention. Using ${env.CI_COMMIT_REF_NAME}-SNAPSHOT as the version for all branches seems to make things easier for all parties.

This does not have to be decided / implemented at the same time as this MR. But if it is, the continuous-release-maven.yml file will need to be updated accordingly.

Incremental Docker Images

This MR is using the incremental release scripts image. That's a bleeding edge container, analogous to using a SNAPSHOT dependency. Before we can consider merging this, we need to finalize osdu/platform/deployment-and-operations/release-scripts!13 and make a stable tag from the result. Then, adjust the image used in this MR.