Skip to content

ADR: Gradual transition to One CI CD Pipeline - New Core Release Cycle

Gradual transition to One CI-CD Pipeline

Status

  • Proposed
  • Trialing
  • Under review
  • Approved
  • Retired

Context & Scope

During the transition to Venus, we need to decouple Community Implementation pipelines from CSP pipelines. A gradual approach that won't break existing configurations is preferable, as suggested by CSP representatives.

Current limitations blocking the transition:

All CSP implementations are tightly coupled with the project.

CSP OSDU implementations are considered sub-modules of projects, making them a mandatory part of the build. Without additional configurations, building them separately is impossible.

Example: https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/blob/master/pom.xml#L37

    <modules>
        <module>legal-core</module>
        <module>legal-core-plus</module>
        <module>provider/legal-gc</module>
        <module>provider/legal-azure</module>
        <module>provider/legal-aws</module>
        <module>provider/legal-ibm</module>
    </modules>

Core modules are built in the same manner as deployable services:

Package registry: https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/packages

We build only SNAPSHOT versions:

<dependency>
  <groupId>org.opengroup.osdu.legal</groupId>
  <artifactId>legal-core</artifactId>
  <version>0.28.0-SNAPSHOT</version>
</dependency>

Or release versions:

<dependency>
  <groupId>org.opengroup.osdu.legal</groupId>
  <artifactId>legal-core</artifactId>
  <version>0.27.0</version>
</dependency>

All CSP implementations depend on the current version of the Core module.

For example https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/blob/master/provider/legal-ibm/pom.xml#L21

	<artifactId>legal-ibm</artifactId>
	<packaging>jar</packaging>
	<properties>
		<os-core-lib-ibm.version>0.26.0-rc8</os-core-lib-ibm.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.opengroup.osdu.legal</groupId>
			<artifactId>legal-core</artifactId>
			<version>0.28.0-SNAPSHOT</version>
		</dependency>

All build stages are inseparable, any change triggers build for everyone:

image

Decision

Update the release cycle for Core modules.

Make it similar to the libraries release cycle, and produce a stable version after each code merge. (OSDU libraries have moved to a continuous release cycle. I'm not sure if this applies to Core modules, but the main point remains: have a stable version after each merge.)

<dependency>
  <groupId>org.opengroup.osdu.legal</groupId>
  <artifactId>legal-core</artifactId>
  <version>0.28.0-rc1</version>
</dependency>

Switch CSP modules to use stable (non-SNAPSHOT) versions of core modules.

Instead of relying on the SNAPSHOT version of the Core module, switch to the stable version. This will reduce the need to rebuild a CSP module with each change in the Core module.

	<artifactId>legal-ibm</artifactId>
	<packaging>jar</packaging>
	<properties>
		<os-core-lib-ibm.version>0.26.0-rc8</os-core-lib-ibm.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.opengroup.osdu.legal</groupId>
			<artifactId>legal-core</artifactId>
			<version>0.28.0-rc1</version>
		</dependency>

If a CSP team needs to update a module simultaneously, for example, when developing a feature and wanting it for their module immediately, they could switch to a SNAPSHOT version. After the merge, scripts should switch the module back to the stable version produced after the merge.

Community Implementation modules (Core-Plus) should stay with the SNAPSHOT version.

Decouple CSP modules from the main project, but keep them where they are.

It could be done in 2 ways, either remove CSP services from the root POM module section:

    <modules>
        <module>legal-core</module>
        <module>legal-core-plus</module>
    </modules>

Or update the Maven build script: https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/blob/master/build/maven.yml?ref_type=heads

Example: mvn --projects legal-core,legal-core-plus clean install

Enhance CI/CD configurations to divide trusted pipelines per CSP.

We could use GitLab labels as a condition to run the CSP trusted pipeline. We are already using them, but currently only as markers:

image

If converted into a condition for particular jobs to run, we can have some level of decoupling without introducing breaking changes.

Rationale

Update the release cycle for Core modules.

Without transitioning to stable Core module versions, it is difficult to achieve independence and guarantee stability for CSP modules.

Even if the proposed ADR is not accepted, switching to a separate release cycle for Core modules will be needed later, after Mercury is finished. In Venus, every consumer of the Core module will start using it as a library, without the direct connection present in the current repository.

Switch CSP modules to use stable (non-SNAPSHOT) versions of core modules.

The rationale is the same as for updating the release cycle.

Decouple CSP modules from the main project, but keep them where they are.

Currently, we can't compile only the Core module because CSP modules and Core modules are connected in the root POMs. Decoupling them will help establish some independence, but keeping CSP modules in the same repository will help avoid breaking changes, maintain CI/CD configurations, and more.

Enhance CI/CD configurations to divide trusted pipelines per CSP.

CI-CD pipelines are already quite independent for each CSP, we have custom conditions to trigger some of them, separate configurations and jobs.

Example: https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/blob/master/devops/gc/pipeline/override-stages.yml#L82

  only:
    variables:
      - $GC == '1'

Making 1 more condition to run can give us needed decoupling while keeping all that was built before by CSPs to run their processes.

Consequences

Pros:

  • No really breaking changes.
  • No need to rewrite CI-CD configurations a lot
  • Backward compatible, CSP will always be able to use the latest Core version, run their pipelines if needed, etc.
  • We will have a new ability, to introduce new features without the need to do mock implementation in CSP modules. (Usually when changes are required at the CSP level some kind of stubs or mocks are introduced there if the CSP team does not have the capacity to implement them right now)

Cons:

  • A separate release cycle for Core modules should be developed.
  • CI-CD should be updated to utilize labels as conditions to run CSP pipelines.
  • Root POMs could contain shared dependencies or configurations, so if we remove CSP modules from there, we should replicate those configurations in their POM files.
  • Using Maven arguments to decouple mvn --projects legal-core,legal-core-plus clean install could be tricky if we want to use labels. If for example Azure label is applied, the command should change to mvn --projects legal-core,legal-core-plus,legal-azure clean install

When to revisit

Tradeoff Analysis - Input to decision

Alternatives and implications

Move Community Implementation into a separate project.

To ensure CI is well maintained and new features are implemented, we need to establish policies that everyone must follow.

Dev workflow could become less productive, it will be required to update Core in an original repo, then go to the Community Implementation repo, back and forth.

A release cycle change for Core modules will still be needed, as Community Implementation will require a stable version of Core modules.

Drop CSP modules.

While this is the goal of Venus and preferable for the Community, it may not be accepted by CSP representatives at the moment.

Simplified transition without new release cycle

#50

Decision criteria and tradeoffs

Decision timeline

Edited by Rustam Lotsmanenko (EPAM)