Generate the dependency tree, rather than the list, for use with the release scripts
This will allow the formatting output to show a tree view of any flagged dependencies. The goal of this is to make it easier to determine why vulnerable dependencies are included, and streamline the process of identifying and fixing them.
The Changes
The main change comes from osdu/platform/deployment-and-operations/release-scripts!1 (merged), which is embedded in the release script containers (tagged as v2.0).
This change uses mvn dependency:tree rather than mvn dependency:list for analyzing the maven dependencies.
The first effect -- each root pom file now generates a different maven output file -- mvn-dependency-list-output.txt no longer exists, and now mvn-dependency-tree-output.txt does.
This new file is picked up by the analyze-maven-dependencies job as part of running the analyze-maven.js script, which is part of the release scripts container used as the image for that job.
That causes the resulting compiled data object, maven-dependencies.json to have an extra field -- dependencyTree.
The new dependency tree data is additional -- other information is still available as before and in the same format.
However, the format-dependencies.js is now smart enough to see that and provide some extra context for error outputs.
Seeing the Tree
The output is printed in the job log of every analyze-maven-dependencies job, which is included by default in every pipeline that uses the standard build/maven.yml CI include.
Here's an example output:
Branch: HEAD
SHA: 8518b54ba7104d92723445510306eaf063856455
Maven: 0.17.0-SNAPSHOT
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ---------------- | ---------------- |
| core-lib-azure | 0.14.0-rc2 | 0.6.1 |
| core-lib-gcp | 0.15.0 | |
| os-core-lib-aws | 0.16.0-SNAPSHOT | 0.13.0, 0.3.16 |
| obm | 0.15.0 | |
| oqm | 0.15.0 | |
| os-core-common | 0.13.0 | 0.13.0 |
| os-core-lib-ibm | 0.16.0-rc1 | 0.15.2, 0.7.0 |
| os-schema-core | 0.17.0-SNAPSHOT | 0.16.0-SNAPSHOT |
| os-schema-test-anthos | | 0.17.0-SNAPSHOT |
| os-schema-test-gcp | | 0.17.0-SNAPSHOT |
| osm | 0.15.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.2.2, 2.11.4 | 2.13.2.2, 2.11.3 |
| (3rd Party) net.minidev.json-smart | 2.4.7 | 2.3 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-slf4j-impl | 2.17.1 | 2.13.3 |
| (3rd Party) org.springframework.spring-webflux | 5.3.12 | |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22 | 5.3.22 |
Critical: Found Vulnerable Jackson Databind dependency (<2.12.6.1 || >=2.13.0 <2.13.2.1)
├─ _Root_
│ └─ org.opengroup.osdu.os-schema-gcp == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-schema-core == 0.17.0-SNAPSHOT
│ └─ com.fasterxml.jackson.core.jackson-databind == 2.11.4
└─ testing/
└─ org.opengroup.osdu.schema-test-azure == 0.17.0-SNAPSHOT
└─ org.opengroup.osdu.os-schema-core == 0.16.0-SNAPSHOT
└─ com.fasterxml.jackson.core.jackson-databind == 2.11.3
Critical: Found Vulnerable Spring WebFlux dependency (<5.2.20 || >=5.3.0 <5.3.18)
└─ _Root_
└─ org.opengroup.osdu.os-schema-azure == 0.17.0-SNAPSHOT
└─ com.azure.spring.azure-spring-boot-starter-active-directory == 3.4.0
└─ org.springframework.boot.spring-boot-starter-webflux == 2.4.12
└─ org.springframework.spring-webflux == 5.3.12(Note: GitLab removes blank lines when displaying job logs online, which makes it slightly harder to read.)
The dependency table is the same as before, listing the version numbers for each POM root. But if there are any critical package warnings, the truncated dependency tree is shown to see how it is included. The truncation works by showing the vulnerable package(s) and its ancestors only.
Multi Project Test
Here's a multi-project test that shows this being used in several projects. The ones with compilation failures are failing for normal compile reasons (missing dependencies, for example), and to the best of my knowledge not because of this scanning.
Full disclosure, that test was made using an "incremental" image of the release-scripts -- basically it was tagged to a SHA (34b45d6ee4283fe5573d94ec27ae481cf35d1c4b. This SHA was the one tagged as v2.0, so I'm fairly confident it's a good test.
Known Shortcomings
Hard-coded Vulnerability List
The vulnerability list (what to show as "critical" / what to include in the table) is configured with a JSON object that's embedded in the formatting scripts. Therefore, changes to this require new tags on release-scripts and updates here to alter.
I'd like to make this some sort of easily configurable input, so we can change our focus from time to time and see the changes reflected in the analysis outputs without having to link to new CI logic.
Eventually.
Tagging Notes Usage
I will be updating my Tagging Notes generator to parse and render this data in the wiki pages as well, so we can see it in one place. I still have to figure out the right way to format this display so as not to overwhelm the page.
And, this uses a different hard-coded configuration for its tables. Which is to say, it uses the same source code pulled off the machine that is running the Tagging Notes (i.e., my local laptop), so it picks up alterations without new images (even without committing it).