Utilizing Standard Pipelines
I'd like this project to consider merging your CI pipeline work with the osdu/platform/ci-cd-pipelines> project, and utilize more jobs by includes than using local CI config.
Some Reasons to Consider
Copy/paste code is hard to keep maintained
Most of your CI logic appears to have started as a copy/paste from the main repository, anyway. But keeping it local means that developers need to update changes in multiple places, and when they're working on the improvements they don't have your use case in mind.
This included some recent developments to get the dev2 environment going, but it also includes the changes to the FOSSA scanning -- you're still using an older, unmaintained image for the scanning. And, when I did the changes, I worked test examples for maven and pip, the two supported build systems. If npm had been there, I would have had it in mind.
You miss new pipeline developments
I'm moving pieces of the release management scripts into the pipeline to make more aspects of the tagging process happen automatically from branch creation. For now, it's only dependency scanning data, but upgrades are planned to do more stages from there.
The GitLab Ultimate scanners check for security vulnerabilities, and the InfoSec team utilizes these results to plan their work. These scanners aren't running on your project, but would be if included the appropriate CI configuration -- or at least, we'd see what needs to be improved on those scanners to function if they don't work out of the box.
Your improvements aren't available to others
Any improvements you make to the CI process after you've copied it remains in your local repository. Others could benefit from having this available in a common location. Supporting another language gives future OSDU projects more capabilities right at the start. You'd even get to define the basic processes for these.
Open to Discussion
I'd like to hear more about how the custom pipelines came to be, and if they are serving a need that can't be generalized. For steps that are truly custom and unique to your project, it makes sense to have them as local CI config files.
If we do decide to start using more of the standard pipeline logic, I think we'll need to implement it slowly, a piece at a time. Of course, if you think a big bang MR is better, I'd consider that, too.
Thank you in advance for your thoughts.