Skip to content

ADR - Update Info API and Git tags to support core and CSP specific implementation version

Decision Title

Update Info API to support core and CSP specific implementation version

Summary

This proposes to directly support independent provider releases of services will reduce the amount of effort spent re-deploying logic to provider implementations that are unaffected by the fixes / patches.

This requires changing procedures around build configuration, release management, and reporting via the /info endpoint.

Status

Note that this ADR was originally proposed as #98 (closed), but was re-written to include details that emerged during the discussion and approval process.

  • Proposed
  • Approved
  • Implementing (incl. documenting)
  • Testing
  • Released

Context

Most of the services in the OSDU Data Platform use a Service Provider Interface abstraction. This abstraction allows for the bulk of the service logic and implementation of the external API to be done in a common core library, yet allow for individual service providers to implement the backend logic differently.

To this point, these various libraries (the core and each of the provider implementations) have been kept version synchronized -- exactly identical versions. During the release process, the Git tag has reflected that common version to enable easy lookup of the code used. And, for runtime inspection, a special endpoint (/info) reports back the version, commit SHA, and several other pieces of information.

Problem statement

Sometimes new patches -- typically bug fixes / security upgrades -- apply only to one particular provider implementation. When coupled with the common synchronized version, this causes the core library version (as well as all other provider implementations) to change versions to match -- even though no changes occurred in these libraries.

Moreover, when the tags are made for the patch, other providers appear to be out of date. This puts those providers in the position to choose between redeploying code that is functionally equivalent or leaving existing versions and trying to document that the upgrade had no effect on their logic.

Proposed solution

Scope Limitations

Patches Only

This proposal limits provider-specific releases to patches only -- no support is provided for a minor (feature additive) or major (breaking) change. This limitation reduces the complexity of the release strategy immensely, and covers most use cases well.

Maven Focused

The proposal was designed with core services in mind, which are predominantly build using Java code and a Maven build system. The strategies implement well in these cases, but may be extensible to other build systems in different ways.

To maintain simplicity, this ADR will only refer to projects that use Maven, and have an SPI abstraction layer.

Services Only

This proposal does not apply to libraries or utilities.

Side Effect

No Longer Proper Semantic Version Style

Tags will now have extra information that doesn't exactly match the semantic versioning style. In cases where tags have 4 numeric parts, though, they can be directly compared left to right as you would expect.

For example:

  • 1.10.2-ABC.4 < 1.10.2-ABC.5
  • 1.10.2-ABC.4 < 1.10.3
  • 1.10.2-ABC.4 < 1.11.0
  • 1.10.2-ABC.4 < 1.11.0-ABC.2

The addition of the text element identifying the provider implementation is the biggest variation from a standard semantic version. Also, the provider patch number will always be greater than the core patch version (there cannot be a 1.10.3-ABC.2, for instance).

The /info endpoint will consist of two separate version numbers, each of which is semantic style in itself. However, the two versions are related and constrained by each other. If the core version is X.Y.N and the provider is X.Y.M, then you know the service is for the service group version X.Y, there have been N patches made to the core code, and M - N patches made to the provider logic.

Converting between the core + provider version pair and the Git tag will not be possible in general, but will be computable in specific circumstances (such as the matching pair case -- [X.Y.N, X.Y.N]), and can also be converted using a list of all tags or knowledge of the order in which patches were applied. As an example, the pair [X.Y.1, X.Y.2] is ambiguous. If the core patch came first, then the final tag would be vX.Y.1-ABC.2. But, if the provider patch came first, then the final tag would be vX.Y.1.

Build Configuration

Summary

  • Provider-based POMs get a separate provider version, and the rest use a core version.
  • Major + Minor will match between them
  • Changes to a provider increments only the provider version
  • Changes to core code increments ALL versions (core + all providers)

Details

The services are designed to have two distinct POM "trees", i.e. parent POMs plus a set of child modules. One parent POM is stored in the repository root (for the main code), and one is in the testing/ folder (for integration tests). Each tree has a set of common code, which are organized as a code library and stored in service-code and testing/service-test-core (where service is replaced with the specific service name). Then, specific providers that have implementations show up in providers/service-abc and testing/service-test-abc (where, again, service is replace with the service name and abc is replace with the provider name).

Both of the parent POMs and both of the core libraries should be synchronized to have identical version numbers, matching the overall version of the service. Then, for each provider implementation, the provider implementation and the provider testing library will be synchronized among themselves. This leads to a layout something like this:

+ pom.xml (CORE)
  + service-core/pom.xml (CORE)
  + providers
    + service-abc/pom.xml (ABC)
    + service-xyz/pom.xml (XYZ)
  + testing
    + pom.xml (CORE)
    + service-test-core/pom.xml (CORE)
    + service-test-abc/pom.xml (ABC)
    + service-test-xyz/pom.xml (XYZ)

Each file here is marked with the artifact version that it would use (by "artifact version", I mean project.version in the POM file). Notice that the parent poms and the core libraries are all synchronized to use the same, base version of the service. Each provider gets their own version, but will be synchronized between their main library and testing library.

Since only patches are permitted for provider specific versions, the first two version components will always match.

Versioning Responsibility

The Release Coordinators will manage all version changes as part of the release process. MRs should not modify the artifact versions of the various POM files, nor the dependency versions on the self-built core libraries -- that is, the core library for the same service. They can, however, modify first-party or third-party library versions that come from outside sources, including os-core-common or a provider's core common library.

SNAPSHOT library versions will still be used, so all artifact versions in all POM files will have a version ending in -SNAPSHOT. On the default branch (master / main), the both the core versions and all provider versions will be set to the same value, since all development here is occurring within the milestone. The versions won't start to differentiate until a release is made -- and then will only differ on the release branch and tags.

Every provider must use the latest core

When a patch is made to the core library, it will be automatically applied to all providers. We will not have any mechanism to skip a core patch for a particular provider or have a change made in the core library only apply to a single provider. Any change to the service-core code, everybody's version bumps and everybody redeploys.

This is not expected to be an onerous requirement -- patches are by their nature unlikely to cause compatibility issues. If a particular patch would cause a lot of effort for a provider, that's a good indication that it isn't really a patch, but instead a new feature that should wait for the next full milestone.

Info Endpoint

Summary

  • Each service's /info endpoint will need to report two different values -- the provider version and the core version

Details

All the core services have a special endpoint that returns information about the service, including the artifact version, the Git commit ID, and more. This endpoint will need to be modified to report two different fields -- coreVersion and providerVersion instead of just version.

For example, you may see a return such as this:

{
  "groupId": "org.opengroup.osdu.indexer",
  "artifactId": "indexer-azure",
  "coreVersion": "1.10.0",
  "providerVersion": "1.10.2",
  "buildTime": "2023-01-17T12:30:00.500Z",
  ...
}

This work can be done independently from the release script / tagging changes. Until complete, the reported version will be the core version only, which is acceptable during the transition.

MR Procedure Changes

Summary

  • Labels for providers (such as AWS, Azure, ~GCP, IBM) and for core code ( Common Code) must be used in every MR

Details

These labels are already in widespread use to help teams determine proper reviewers for code. This proposal extends this to be used automatically by release scripts to determine whether a particular MR cherry-pick should increment the core version or a particular provider version.

Release Process Changes

Summary

  • Changes to provider code increments the provider patch number
  • Changes to core code increments all patch numbers
  • Tags with only provider changes have the core version and the provider name / patch number -- vX.Y.N-ABC.M
  • Tags with core changes have only the core version -- vX.Y.N

Details

The default branches will have all versions (core and provider) set to be the version being built for the upcoming milestone, similar to how it is done currently. Release branches will begin with all versions the same, but can evolve over time based on the kinds of patches that come in. Here's an example flow that shows a couple of different kinds of patches.

For purposes of example, this assumes that the release is being prepared for version 1.10 of the Core Service Group.

Time Event Branch / Tag Core Version Provider ABC Version Provider XYZ Version
T1 Create Release Branch Branch, release/1.10 1.10.0-SNAPSHOT 1.10.0-SNAPSHOT 1.10.0-SNAPSHOT
T2 Create Tag Tag, v1.10.0 1.10.0 1.10.0 1.10.0
T3 Prepare for Next Patch Branch, release/1.10 1.10.1-SNAPSHOT 1.10.1-SNAPSHOT 1.10.1-SNAPSHOT

T1 -- Create Release Branch: This is the very first step, where the release/1.10 branch is created for the first time. The core and provider versions will match, and will be inherited from the default branch. Immediately after this step, the default branch would be prepared for the next release, setting versions to 1.11.0-SNAPSHOT.

T2 -- Create Tag: At this point, the release branch has been tested, and any lingering work has merged in. Now, we create a "release commit" -- a commit based on the release branch that alters the artifact versions, removing the SNAPSHOT part. That commit is tagged, and since it is the initial release, it gets the simple v1.10.0 tag.

T3 -- Prepare for Next Patch: Once the tag is made, the release branch immediately prepares for the next patch. Versions of core and provider POMs are incremented to the next patch numberu (but kept as a SNAPSHOT) -- we are "building towards" version 1.10.1, therefore it is 1.10.1-SNAPSHOT.

Time Event Branch / Tag Core Version Provider ABC Version Provider XYZ Version
T3 Prepare for Next Patch Branch, release/1.10 1.10.1-SNAPSHOT 1.10.1-SNAPSHOT 1.10.1-SNAPSHOT
T4 Cherry-pick Provider ABC Patch Branch, release/1.10 1.10.1-SNAPSHOT 1.10.1-SNAPSHOT 1.10.1-SNAPSHOT
T5 Create Tag Tag, v1.10.0-ABC.1 1.10.0 1.10.1 1.10.0
T6 Prepare for Next Patch Branch, release/1.10 1.10.1-SNAPSHOT 1.10.2-SNAPSHOT 1.10.1-SNAPSHOT

T4 -- Cherry-pick Provider ABC Patch: In this step, we imagine a cherry-pick MR that patches only provider "ABC". We approve and merge that MR into the release branch, but version numbers don't immediately change as part of that operation.

T5 -- Create Tag: Once we're happy with the integration test results for the branch, we repeat the step to create the tag. A new commit is made, dropping all the SNAPSHOT suffixes from the versions. But crucially, if the particular library has not had any code changes then the version is set to match the previous release tag. In this case, we see that "Core" and "XYZ" are set to 1.10.0, even though the release branch the tag was created from was 1.10.1-SNAPSHOT. This is a counterintuitive consequence of using a single release branch to work on several different libraries at the same time.

Then, the tag is created using the core version first, then the provider name, then the patch number of the provider version. We don't need the full provider version, because we know that the major and minor numbers will match. In this case, v1.10.0-ABC.1 because it is an "ABC" patch, and ABC's version is 1.10.1.

T6 -- Prepare for Next Patch: Again, we prepare the release branch, incrementing the version for provider ABC to 1.10.2-SNAPSHOT. Other versions can remain at 1.10.1-SNAPSHOT -- they are still "building towards" their first patch.

Time Event Branch / Tag Core Version Provider ABC Version Provider XYZ Version
T6 Prepare for Next Patch Branch, release/1.10 1.10.1-SNAPSHOT 1.10.2-SNAPSHOT 1.10.1-SNAPSHOT
T7 Cherry-pick Core Patch Branch, release/1.10 1.10.1-SNAPSHOT 1.10.2-SNAPSHOT 1.10.1-SNAPSHOT
T8 Create Tag Tag, v1.10.1 1.10.1 1.10.2 1.10.1
T9 Prepare for Next Patch Branch, release/1.10 1.10.2-SNAPSHOT 1.10.3-SNAPSHOT 1.10.2-SNAPSHOT

T7 -- Cherry-pick Core Patch: In this step, we imagine a cherry-pick MR that patches the core code. We similarly approve and merge that MR into the release branch, and version numbers remain as they are.

T8 -- Create Tag: Now that we're happy with the code, we create a tag commit. In this commit, every version is incremented from the last release (effectively just remove the SNAPSHOT part). With this commit, we tag it with the core version only -- in this case v1.10.1.

The ABC provider library is at version 1.10.2 here -- it's been patched twice, once by itself and once as a consequence of patching the common code. But, that information doesn't show up directly in the tag spelling, instead it will be in the /info endpoint, the code, and can also be deduced from the list of previous tags.

We do it this way to avoid needing to make multiple tags per release, explicitly named each pairwise combination. It is hoped that provider patches are rare enough that it is easier / better to use a simple single tag for all providers.

T6 -- Prepare for Next Patch: One more time, we prepare the release branch, incrementing the version for all components to their next SNAPSHOT version.

Final Results

After all this, the ABC provider will see three tags: v1.10.0, v1.10.0-ABC.1, and v1.10.1. All other providers will see two: v1.10.0 and v1.10.1.

Edited by David Diederich
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information