ADR: Simplified Gradual Transition to One CI/CD Pipeline

Gradual transition to One CI-CD Pipeline

Status

Context & Scope

During the transition to Venus, we need to decouple Community Implementation pipelines from CSP pipelines. A gradual approach that won't break existing configurations is preferable, as suggested by CSP representatives.

Current limitations blocking the transition:

All CSP implementations are tightly coupled with the project.

CSP OSDU implementations are considered sub-modules of projects, making them a mandatory part of the build. Without additional configurations, building them separately is impossible.

Example: https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/blob/master/pom.xml#L37

    <modules>
        <module>legal-core</module>
        <module>legal-core-plus</module>
        <module>provider/legal-gc</module>
        <module>provider/legal-azure</module>
        <module>provider/legal-aws</module>
        <module>provider/legal-ibm</module>
    </modules>

All build stages are inseparable, any change triggers build for everyone:

Decision

Decouple CSP Modules from the Main Project by Moving Them into Separate Maven Profiles

Maven Profiles offers a powerful way to create flexible build configurations without drastic changes. They can be activated by environment variables, files, and various other conditions.

We can place core and core-plus modules in the default profile and assign CSP modules to their dedicated profiles. This setup allows us to separate compile jobs per CSP using these profiles later.

For more details, see: Maven Profiles Guide

Proposed usage:

        <profile>
            <id>core-build</id>
            <activation>
                <activeByDefault>true</activeByDefault>
            </activation>
            <modules>
                <module>legal-core</module>
                <module>legal-core-plus</module>
            </modules>
        </profile>
        <profile>
            <id>aws-build</id>
            <modules>
                <module>provider/legal-aws</module>
            </modules>
        </profile>
        <profile>
            <id>azure-build</id>
            <modules>
                <module>provider/legal-azure</module>
            </modules>
        </profile>
        <profile>
            <id>ibm-build</id>
            <modules>
                <module>provider/legal-ibm</module>
            </modules>
        </profile>
        <profile>
            <id>gc-build</id>
            <modules>
                <module>provider/legal-gc</module>
            </modules>
        </profile>

Update CI/CD configurations, by adding CSP dedicated build.

Using Maven profiles, we can select what to compile by setting -P in the command line.

Proposed usage:

gc-compile-and-unit-test:
  extends:
    - .maven
    - .skipForTriggeringMergeRequests
  stage: csp-build
  allow_failure: true
  needs: ['compile-and-unit-test']
  script:
    - mvn clean package -f pom.xml -P gc-build
  after_script:
    - find . -path '*/target/*.jar' -o -name 'maven-*-output.txt' | xargs du -hsc

And result:

CSP-dedicated jobs should be reconfigured to run the new jobs, ensuring they execute only after the CSP module compiles successfully.

gc-containerize-gitlab:
  needs: ["gc-compile-and-unit-test", "download_gc_plugins"]

CSP compilation and all subsequent jobs should be set to not voting or allowed to fail.

allow_failure: true

Rationale

By keeping the CSP module related to the root project we can keep configurations, dependency management, and everything that is common in place. But Maven profiles can give us needed flexibility to choose what to build.
Dedicated CSP compile jobs could give us the required decoupling.

Consequences

Pros:

No need to duplicate root POM configurations across provider POMs.
Flexible approach: we can build all CSPs in one job or separate them into distinct jobs as needed.
Convenient for local development, with native support for Maven profiles in most Java-oriented IDEs.
Compatible with existing build jobs—compiled results are available after adding profiles without modifying job definitions.
The transition can be achieved with less effort.
We would be able to build CI OSDU even if other CSPs are failing.

Cons:

Slightly more complex Maven build configuration.
CSP modules remain child modules of the root project, meaning full decoupling in the future would require starting from scratch, replicating configurations, etc.
CSPs will be using the latest SNAPSHOT core module, but their pipeline becomes non-mandatory to pass, meaning the CSP module could often be not compiling or with runtime errors between releases.

When to revisit

Tradeoff Analysis - Input to decision

Alternatives and implications

Transition with new release cycle for core modules - #47

Decision criteria and tradeoffs

Decision timeline

Next Steps

Define Tasks:
- Create Maven profiles for each CSP.
- Update the CI/CD configurations to incorporate the new profiles.
- Reconfigure CSP-dedicated jobs to execute after successful CSP module compilation.
- Set CSP compilation jobs to "not voting" or allow them to fail.
Communicate Changes:
- Inform the development teams about the new CI/CD structure and the changes in how CSPs will be built and tested.
Monitor Transition:
- Track the progress of the transition and gather feedback from the teams on the new setup.

Open Questions

Q: How should scan jobs be managed within the new CI/CD pipeline? For example with updated Maven FOSSA scans only core modules: https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/jobs/3202649

A: By enabling all profiles we can scan all providers as we used to, but I'd leave that for everyone to decide.

Q: Should we consider the release phase? What can it look like?

A: Enabling all profiles will allow us to release as we used to.

Q: Should we include integration test modules?

A: As we've started acceptance test adoption, maybe it's not worth applying the proposed approach to integration tests. #49

Edited Oct 30, 2024 by Rustam Lotsmanenko (EPAM)