Add Dependency Analysis to Maven and Python CI Pipelines

David Diederich requested to merge dependency-generation into master

This MR adds an extra step in the scan stage of the maven and python standard pipelines, which generates a JSON output listing the project's dependencies. To support the new analyze-*-dependencies, some changes were necessary to the compile-and-unit-test jobs, which now output additional information during the build to assist in the dependency generation.


1. Tagging Notes Generation

My primary motivation is speeding up the tagging notes generation. Prior to this MR, the dependencies tables are all created by building every service locally on my own machine, and publishing the results as a wiki. This takes quite a bit of time and slows down the tagging week. By having the dependency table generated in the pipeline, the notes can be made by downloading the table rather than computing it.

Longer term, I hope to remove all required computation from tagging notes -- then it can be a single page display that live updates.

2. Easier to Review Dependency Changes

For any MRs that propose to update dependencies, we can now see the resulting table in the pipeline itself. This helps confirm that all places that specified the old dependency were caught, and that it is now what it should be.

3. Future Use for Automatic Updates / Linting

I have plans to introduce more pipeline stages to automatically open MRs that upgrade the first-party dependencies. They'll use this generation to determine what needs to change (if anything), and to confirm that the change was impactful.

Along the same lines, some stages can be created to verify that the artifact versions are correct.

Multi Pipeline

Here's a multi pipeline demonstrating this in use across the services.

Other References

The osdu/platform/deployment-and-operations/release-scripts> project builds the container images used in these stages.

osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services!508 (merged) is needed to make this work for wellbore domain services.

Edited by David Diederich

Merge request reports