ADR: Acceptance Testing for the One CI/CD Pipeline and the Foundation for Certification Tests.

Acceptance Testing for the One CI/CD Pipeline and the Foundation for Certification Tests.

Status

Context & Scope

Core services implement integration tests in Java using HTTP Client as a library to validate the API behavior. These integration tests reside in the testing folder of the respective core services. For example, the integration tests for the Partition core service are structured and hosted in the System/Partition/testing folder which contains integration tests for all providers as shown below.

Those tests are usually the most up-to-date, due to being closer to developers who introduce new features, fix bugs, etc.
It is safe to claim that they are thorough, usually service testing module has at least ~ 100 tests implemented, sometimes more.
To run tests each OSDU implementation requires a respective testing module for each service.
Tests can be overridden, often provider modules contain custom logic for authentication issuing and sometimes adjust test cases, which discourages standardization.
For transition to one CI-CD Pipeline, we need a single point of truth, reusable tests that can seamlessly verify any OSDU platform. Otherwise, Cloud Providers won't have a proper way to validate their platforms.
We do not have an agreement on certification tests, on what can be used as a certification framework.

Decision

Acceptance Testing for the One CI/CD Pipeline based on existing integration tests.

We could transform our existing integration tests into acceptance tests for a single CI/CD pipeline that could be reused by all CSPs to verify their platforms. These changes will help us to move to the One CI/CD Pipeline. The purpose of the proposed changes is to have a single testing module based on existing core-test module that is reusable by any OSDU implementation, allowing all CSPs to validate their platforms using a single module.

Step N1. Move initializers to the core module.

The purpose of provider implementations, by design, is to launch tests and provide configuration, which happens through inheritance (for Java-based tests). This way of running tests has created fertile ground for adjusting test cases when needed, introducing discrepancies, etc.

Solution:

Move those launchers back to the core, making the core module self-sufficient.
Make core test classes final forbidding overrides.
Decouple provider modules from a core-test module.

Example of provider-specific test launchers:

Q: How are we going to decouple the provider modules from the core test modules?

It's an open question:

We could either move the refactored core test module to a new folder named acceptance-test, or
Unlink the provider modules from the core module right where they are.

Moving the core module out will cause code duplication and require simultaneous support for both modules, while unlinking the provider modules will require effort to update the CI/CD configurations to use the common module.

Step N2: Introduce a common, standards-based configuration.

Each provider has its own configurations, at least for token issuing. In some cases, there may also be configurations to access low-level resources, such as Blob storage. These should be replaced by standards whenever possible or removed if there is no generic approach.

Solution for token issuing, two ways of configuring are proposed:

OIDC client credentials flow, which can be implemented using the Nimbus Java library.
Direct token injection into the testing runner environment.

Solution for low-level resource configuration:

Delete them. If possible, refactor any test cases that depend on them.

Example of provider-specific token issuing configuration:

Step N3, resolve discrepancies.

Some tests may have overrides in CSP test modules, making overall verification inconsistent across providers.

Solution:

The core version of a test should be prioritized.
Inform CSPs in the MR description, about those discrepancies.

Example of test override:

Later on during the MR review, we could reach an agreement on how to handle those differences: either update the CSP OSDU service, implement flexible assertions for testing, or potentially delete the test case if deemed invalid.

Step N4, CI\CD Updates.

TBD, the solution depends on what path to have acceptance tests chosen, whether it's a forked module or a replacement for existing CSP modules.

Integration tests as a foundation for Certification Tests.

These changes are not strictly necessary for the transition to a single CI/CD Pipeline, but they could be included as it presents a good opportunity to introduce not only a common testing framework but also to lay the foundation for future OSDU certification frameworks.

Step N1. Executable tests.

Currently, it's not possible to use integration tests outside of the build stages. This means that in a test environment, Git must be installed, Community GitLab should be accessible to download the repository, and Maven must be installed to launch the testing phase. These limitations reduce the use cases and convenience of integration tests.

Solution :

Add the ability to launch tests as a standalone JAR file. For this, the Maven Assembly Plugin can be used, to pack all dependencies into a single JAR file and the tests need a main method.
Use at least the JUnit 5 testing framework or any other framework that is convenient for console execution. The JUnit 4 framework is undesirable for this purpose, as it does not provide a proper API for console launching.

This will allow us to run tests like any other Java app.

java -jar storage-test-core-0.28.0.jar

Step N2. Reporting.

If tests are used outside of CI/CD pipelines, checking the results as they are currently represented might be inconvenient. One would need access to the environment output or to a console.

Solution :

Add a reporting tool. Allure reporting is proposed as it is compatible with multiple testing frameworks and works out of the box. It also works with Python, JavaScript, etc.

Rationale

Acceptance Testing for the One CI/CD Pipeline based on existing integration tests.

Reusing existing integration tests will reduce the overall required effort compared to building them from scratch.
Reusing existing CI/CD configurations is possible for the Community implementation and, to some extent, for CSP providers.
We already have decent tests developed over time by contributors.
We can assign tasks to developers without wide expertise in OSDU API, unlike developing acceptance tests from scratch.
These tests are usually the most up-to-date, and their maintenance does not require different qualifications, as they typically use the same language as the service they were written for.
We need to unlink the provider test modules as we are going to harden the test cases defined in the core module, thereby forbidding overrides and deviations.
It is possible to test any OSDU platform by providing only URLs and authentication, so there is no actual need for custom configuration implementations.

Integration tests as a foundation for Certification Tests.

If we agree to use existing integration tests as a common testing framework for all CSPs, then their use as part of a certification process is within reach, as they would be universally applicable to any OSDU platform.
Proper reporting could streamline the overall certification and verification process.
It’s reasonable to implement this alongside the development of acceptance tests for the One CI/CD pipeline, as this would avoid the need to verify changes twice across all repositories.

Consequences

Acceptance Testing for the One CI/CD Pipeline based on existing integration tests.

Decoupling provider tests from core tests may require multiple updates to the CI/CD configurations.
Moving the updated core test module to a new location could result in code duplication and the need to support both versions temporarily.
Some test cases may not be suitable for the new approach if the results can only be verified through low-level access.

Integration tests as a foundation for Certification Tests.

Maven build configuration would inevitably become more complex.
Additional time would be required for tasks to complete.
It's recommended to migrate from Junit4 if we want to use integration tests as part certification framework, to be able to execute them outside of the CI\CD build scope.

When to revisit

Tradeoff Analysis - Input to decision

Alternatives and implications

For acceptance tests:

Develop acceptance tests from scratch

It would require expertise and extensive communication to agree on what to test and how.

But tests could be cleaner, follow the same pattern, use the same technologies, etc.

For certification tests:

Use ONLY postman collections that are used to verify pre-ship environments.

They could be a little bit behind the currently implemented functionality and might require coordination between different teams.

But they are cloud agnostic from the beginning, thus easily to reuse.

Decision criteria and tradeoffs

Decision timeline

Edited Oct 17, 2024 by Chad Leong