CI-CD Pipelines issueshttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues2024-02-08T13:08:50Zhttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/19Code coverage Report2024-02-08T13:08:50Zethiraj krishnamanaiduCode coverage ReportCurrently there is no way to report code coverage for Unit Tests. Can we please update CI-CD pipeline to include scanning and code coverage report generation?Currently there is no way to report code coverage for Unit Tests. Can we please update CI-CD pipeline to include scanning and code coverage report generation?M1 - Release 0.1David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/1Check Dependencies / Libraries during Container Build2022-12-27T09:28:27ZPaco Hope (AWS)Check Dependencies / Libraries during Container BuildWhen containers are built, the version numbers of all libraries and dependencies must be checked against an inventory of known vulnerabilities. This includes:
- Language runtimes (Java, Python, NodeJS, etc.)
- Web server software (e.g., ...When containers are built, the version numbers of all libraries and dependencies must be checked against an inventory of known vulnerabilities. This includes:
- Language runtimes (Java, Python, NodeJS, etc.)
- Web server software (e.g., Apache, nginx, JBoss, Tomcat)
- Operating systems (RHEL, Windows, etc.)
- Component libraries
### Operator Input
- This is a requirement raised by **Total**
- **ConocoPhillips** has raised OWASP Top 10 and STRIDE as recommendations.https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/29AWS pipelines are triggered on each commit2022-06-17T19:04:58ZMikhail Piatliou (EPAM)AWS pipelines are triggered on each commitAWS pipelines are started on every commit, it seems all the service projects are affected and the issue is in the common pipelines.
For instance,
https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/pipelines/1...AWS pipelines are started on every commit, it seems all the service projects are affected and the issue is in the common pipelines.
For instance,
https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/pipelines/116060
https://community.opengroup.org/osdu/platform/system/indexer-service/-/pipelines/116008
https://community.opengroup.org/osdu/platform/system/indexer-queue/-/pipelines/116160
Cc: @divido @Oleksandr_Kosse @Kateryna_KurachM13 - Release 0.16Marc Burnie [AWS]Marc Burnie [AWS]https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/27FOSSA is missing license information on many licenses2022-03-07T13:27:28ZDavid Diederichd.diederich@opengroup.orgFOSSA is missing license information on many licensesThere is a major bug in FOSSA that drops licenses for packages that it used to find them for. This is scheduled to be fixed by FOSSA by the end of April.
### Original Guidance
When we first discovered this, guidance was to ignore the `...There is a major bug in FOSSA that drops licenses for packages that it used to find them for. This is scheduled to be fixed by FOSSA by the end of April.
### Original Guidance
When we first discovered this, guidance was to ignore the `fossa-check-notice` output. That checker was suggesting that many attributions (often hundreds) be removed from the `NOTICE` file. Obviously, we didn't remove these libraries from the projects, so removing the attributions was an error. We decided it was better to have a stale / non-updating NOTICE file that was right at some point in the recent past; than to have an "up-to-date" known-wrong NOTICE.
### One Way Diffs
Given the long time estimate for a fix, we need a better approach in the meantime. We need to do as much as we can to keep these up to date with changing package dependencies to make sure that all the projects we use are getting proper attribution, but with limited tooling support.
The best idea suggested is to effectively perform a "one way diff". If the generated NOTICE has a new entry, we flag it to be included in the commited version. But, if it suggests removing one, we ignore that suggestion. This will lead to over-attributing, but this is better than under-attributing. If a dependency was known to be removed, it can be manually deleted from the NOTICE file.
Eventually, after FOSSA fixes their scanners, we can reset the NOTICE to match the current result.
### New Guidance
After this issue is implemented, we should begin applying FOSSA NOTICE files that are suggested by the `fossa-check-notice` stage once again. Because of the specifics of the implementation, caching is turned off for a while. Which could lead to some back-to-back NOTICE failures -- sorry -- but with the NOTICE in grow-only mode, it should stabilize before too long.M11 - Release 0.14David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/24Spotbugs fails to build search-byoc2022-02-22T13:16:57ZDavid Diederichd.diederich@opengroup.orgSpotbugs fails to build search-byocDuring the spotbugs-sast step, the analyzer begins by compiling (`mvn install`) the various projects it finds. This is required for the spotbugs tool, which operates on the built java code.
For search, this is [failing](https://communit...During the spotbugs-sast step, the analyzer begins by compiling (`mvn install`) the various projects it finds. This is required for the spotbugs tool, which operates on the built java code.
For search, this is [failing](https://community.opengroup.org/osdu/platform/system/search-service/-/jobs/407445) for the search-byoc project. Because of the compilation error, none of the spotbugs findings are reported.
### Parent-Child POMs
The OSDU Data Platform utilizes a parent-child pom structure for most services. Normally, you would only run the `mvn install` on the parent -- doing so on the children is unnecessary -- and in this case, there are dependencies specified in the parent that are needed in the child. GitLab is [already aware](https://gitlab.com/gitlab-org/gitlab/-/issues/24076) of this limitation to the spotbugs scanner.
### Possible Workaround
The compilation step can be turned off, which is something GitLab recommends for complicated or custom build processes. We can then build the projects using only the parents (like we do in `compile-and-unit-test`).
The build logic needs to be added to the `spotbugs-sast` job -- even though we built it previously in the pipeline (and passed the `target/` folders along as artifacts), the final install location is the `.m2` cache, which is stored as runner _cache_ (not artifacts). Therefore, we can't know reliably that the runner that happened to pick up the `spotbugs-sast` job is the same one that executed the `compile-and-unit-test` job. That said, we can speed things up by running `mvn install` without doing a clean rebuild -- the `target/` folders will be current.David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/26Automatic Retry for Failed Deployments / Integration Tests2022-02-15T13:47:08ZDavid Diederichd.diederich@opengroup.orgAutomatic Retry for Failed Deployments / Integration TestsStandard procedure when a pipeline fails in a CSP deployment or integration test step is to retry the job.
The purpose of this is to determine if the failure is a temporary failure in the infrastructure, or a more lasting failure (possib...Standard procedure when a pipeline fails in a CSP deployment or integration test step is to retry the job.
The purpose of this is to determine if the failure is a temporary failure in the infrastructure, or a more lasting failure (possibly related to the code change itself).
These retry operations can be automated by GitLab itself, reducing the need for the submitter to notice and act on them.
Also, during the tagging week, the script that generates the Tagging Notes retries any job on the latest pipeline that has a failure.
This logic is more precise -- it only retries the deployment and integration test steps, avoiding wasting time re-invoking compile or scanning operations.M11 - Release 0.14David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/25Cache FOSSA NOTICE files2022-01-18T16:36:57ZDavid Diederichd.diederich@opengroup.orgCache FOSSA NOTICE filesFOSSA's generated attribution files (NOTICE) are sometimes erratic, and change in ways that are not legally significant. I've seen many cases of changes like:
1. Different ordering of packages
2. Duplicate packages with different counts...FOSSA's generated attribution files (NOTICE) are sometimes erratic, and change in ways that are not legally significant. I've seen many cases of changes like:
1. Different ordering of packages
2. Duplicate packages with different counts (package listed 2 times vs 3 times)
3. Changing project names, usually in case (Package Name vs package-name)
4. Changing project URLs, where both are valid (github.com/project vs mvnrepository.com/project)
To address this, the [`fossa-with-cache`](https://community.opengroup.org/divido/fossa-with-cache) attempts to serve as an intermediary between the GitLab CI Pipelines and the FOSSA servers.
# Purpose of this Issue
This issue is tracking the evolution of the `fossa-with-cache` tool until it gets enough stability to be included in the `scanners/fossa-*.yml` files. To begin, the logic has been applied to pilot services, where it will be adjusted based on real development experience. Unfortunately, in order to really test it we must merge it to the default branch first. So, this issue will remain as a common source for feedback, rather than an issue or MR on the pilot services. Then, this will be closed out once the logic has been moved into this project for general use.
# Storage Service Pilot
The first pilot service is storage. Initial logic was included in osdu/platform/system/storage!308.
# Wellbore Domain Services Pilot
The second pilot service is wellbore domain services. Initial logic was included in osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services!336.M10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/21[Feature ask][Azure] CI-CD pipeline should allow services to run Integration ...2021-08-10T07:39:13ZAman Verma[Feature ask][Azure] CI-CD pipeline should allow services to run Integration tests from multiple folders_Current bahavior_
===
Each service has an entry named `AZURE_TEST_SUBDIR` in `gitlab-ci.yaml` which indicates the location of pom.xml for test modules of that service for any given CSP.
This model works well when all the integration te..._Current bahavior_
===
Each service has an entry named `AZURE_TEST_SUBDIR` in `gitlab-ci.yaml` which indicates the location of pom.xml for test modules of that service for any given CSP.
This model works well when all the integration tests are located in one folder only. Till date, that is indeed the case. There are two types of Integration test frameworks being used in OSDU services as of now-
1. JUNIT- Most of the ITs are located in `<service>`-test-core. CSPs can override existing tests/ add new tests in `<service>`-test-`<CSP>` folder. The tests are nothing but methods in test classes which can be overridden in child classes.
2. Cucumber- For R3 services, cucumber seems to be the preferred method. Here there's just one test folder- `<service>`-test-core and all the tests are located there. Examples are WKS, schema etc. Cucumber tests *DO NOT OFFER* overriding mechanism however and it causes problems while writing CSP specific tests.
_Shortcomings in current behaviour_
===
Since the pipeline allows only one test directory as a parameter, it mandates that all the integration tests should be written under one folder only. However, for cucumber framework, it does not cater to the need of having two folders from where tests need to run. For example, `<service>`-test-core might have tests for all the APIs in `<service>`-core while `<service>`-test-azure might have all the tests for azure specific APIs. Now we need to run tests from both these folders.
_Use case_
===
In schema service, we are introducing an azure only API for handling system schema. Similar azure only APIs exist in workflow service. MR https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/112/diffs
_Proposed changes_
===
`AZURE_TEST_SUBDIR` should accept a list of directories instead of just one directory. The list should be handled accordingly in ci-cd scripts and tests should be run for each directory in the list.
cc: @polavishnu , @manishk , @pbehedeAman VermaAman Vermahttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/23Spotbugs Failing in some Services with Out of Memory Exception2021-07-06T15:38:33ZDavid Diederichd.diederich@opengroup.orgSpotbugs Failing in some Services with Out of Memory ExceptionSome services, such as [Partition](https://community.opengroup.org/osdu/platform/system/partition/-/jobs/404507) fail in the spotbugs step. If re-run with `SECURE_LOG_LEVEL` set to `"debug"`, we see that the [failure](https://community.o...Some services, such as [Partition](https://community.opengroup.org/osdu/platform/system/partition/-/jobs/404507) fail in the spotbugs step. If re-run with `SECURE_LOG_LEVEL` set to `"debug"`, we see that the [failure](https://community.opengroup.org/osdu/platform/system/partition/-/jobs/404635#L878) is `java.lang.OutOfMemoryError`
From that [same debug output](https://community.opengroup.org/osdu/platform/system/partition/-/jobs/404635#L868), spotbugs is run with `java -Xmx1900M`.M7 - Release 0.10David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/2Integrate application source code scan (SAST) during build2021-06-16T22:19:28ZRon GreevesIntegrate application source code scan (SAST) during buildUse a tool like Veracode, HCL ASoC, or one of the [free alternatives](https://owasp.org/www-community/Source_Code_Analysis_Tools).
- tool should be fast enough that it can run during developer build/test workflow.
- should also run durin...Use a tool like Veracode, HCL ASoC, or one of the [free alternatives](https://owasp.org/www-community/Source_Code_Analysis_Tools).
- tool should be fast enough that it can run during developer build/test workflow.
- should also run during QA- and Prod-stage builds.https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/3Integrate dynamic analysis security penetration test (DAST) in build process2021-06-16T22:19:27ZRon GreevesIntegrate dynamic analysis security penetration test (DAST) in build processUse a tool like [OWASP ZAP](https://owasp.org/www-project-zap/) (or Veracode or HCL ASoC if you're feeling flush)
- tool should run automatically in QA- and Prod-stage builds.
### Operator Inputs
- **Shell** has listed this as a require...Use a tool like [OWASP ZAP](https://owasp.org/www-project-zap/) (or Veracode or HCL ASoC if you're feeling flush)
- tool should run automatically in QA- and Prod-stage builds.
### Operator Inputs
- **Shell** has listed this as a requirementhttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/4Integrate SSL configuration testing into build process2021-06-16T22:19:26ZRon GreevesIntegrate SSL configuration testing into build processUse a tool such as [Qualys SSL Server test](https://www.ssllabs.com/ssltest/).Use a tool such as [Qualys SSL Server test](https://www.ssllabs.com/ssltest/).https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/5Integrate web header testing in build process2021-06-16T22:19:25ZRon GreevesIntegrate web header testing in build processThe goal is to add missing compliant headers and remove any non-compliant headers from all web requests.The goal is to add missing compliant headers and remove any non-compliant headers from all web requests.https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/6Integrate web root analysis (for any exposed web urls) during deploy process2021-06-16T22:19:24ZRon GreevesIntegrate web root analysis (for any exposed web urls) during deploy processHere are some tools that might help:
| Name | URL | Note |
| ------ | ------ | ------ |
| IBM X-Force | https://exchange.xforce.ibmcloud.com/ |Login as guest. Gives a rating of 1 (good reputation) to 5 (worst). No captcha required. |
| ...Here are some tools that might help:
| Name | URL | Note |
| ------ | ------ | ------ |
| IBM X-Force | https://exchange.xforce.ibmcloud.com/ |Login as guest. Gives a rating of 1 (good reputation) to 5 (worst). No captcha required. |
| Is It Hacked | http://www.isithacked.com/ | No score/rating, but give checkmarks for whether your site is compromised (no historical info). No captcha required. |
| Norton | https://safeweb.norton.com/ | No login required, gives two reviews, one from Norton and another from community. No captcha required. |
| Sucuri | https://sitecheck.sucuri.net/ | No login required, check for malware and against blacklisted sites. No captcha required. |
| TrendMicro | https://global.sitesafety.trendmicro.com/ | Gives OK mark if the site is safe. No captcha required. |
| ZScaler | https://zulu.zscaler.com/ | Rating of 0-100, 0 means good. Seems very comprehensive. No captcha required. |https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/12Setup GCP CICD pipelines for R2 services [GCP]2021-06-16T22:19:23ZElizaveta Zeldina (EPAM)Setup GCP CICD pipelines for R2 services [GCP]Set-up CICD pipeline in Community GitLab with `GCP Community Integration` as a target environment for R2 services:
- [x] [Delivery](https://community.opengroup.org/osdu/platform/system/delivery)
- [x] [Storage](https://community.opengro...Set-up CICD pipeline in Community GitLab with `GCP Community Integration` as a target environment for R2 services:
- [x] [Delivery](https://community.opengroup.org/osdu/platform/system/delivery)
- [x] [Storage](https://community.opengroup.org/osdu/platform/system/storage)
- [x] [Indexer](https://community.opengroup.org/osdu/platform/system/indexer-service)
- [x] [Search](https://community.opengroup.org/osdu/platform/system/search-service)
- [x] [Entitlements](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-gcp)
- [x] [Legal](https://community.opengroup.org/osdu/platform/security-and-compliance/legal)
- [x] [Indexer Queue](https://community.opengroup.org/osdu/platform/system/indexer-queue)Release 2.0Dmitriy RudkoDmitriy Rudko2020-07-31https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/13Core Common Lib2021-06-16T22:19:22Zethiraj krishnamanaiduCore Common LibWe have two cases where a developer could accidentally override the generated core common lib jar file
1) When a developer creates a new branch in core common lib, GitLab automatically builds and push's jar file to artifacts repo.
2) Wh...We have two cases where a developer could accidentally override the generated core common lib jar file
1) When a developer creates a new branch in core common lib, GitLab automatically builds and push's jar file to artifacts repo.
2) When a developer creates MR, they could use the same version in the POM file so technically they can add new classes and override the already generated jar file which could break all existing services.M1 - Release 0.1ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/14Investigate GitLab Scanners for Performance2021-06-16T22:19:21ZDavid Diederichd.diederich@opengroup.orgInvestigate GitLab Scanners for PerformanceThe gemnasium dependency scanner and the spotbugs SAST scanner are some of the longest parts of the CI/CD pipeline at the moment. It would be helpful to know if there are configuration parameters that can increase performance, and know w...The gemnasium dependency scanner and the spotbugs SAST scanner are some of the longest parts of the CI/CD pipeline at the moment. It would be helpful to know if there are configuration parameters that can increase performance, and know what the effect will be on the quality of the results.
Some initial areas to investigate:
- Make sure the artifacts and maven cache are being populated down to the scanners, so that these elements are not being regenerated.
- GitLab scanners seem to have a configuration for the most intense scanning built into their analyzers. Perhaps a fork of their projects to change these settings?David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/16Fork-based MRs can't trigger trusted tests2021-06-16T22:19:20ZDavid Diederichd.diederich@opengroup.orgFork-based MRs can't trigger trusted testsMRs coming from forks execute their pipelines in the context of the namespace the merge is coming from -- not the context it is going into. That is typically going to be someone's personal namespace.
In theory, this would be considered...MRs coming from forks execute their pipelines in the context of the namespace the merge is coming from -- not the context it is going into. That is typically going to be someone's personal namespace.
In theory, this would be considered a non-trusted source and try to launch a trusted version of the pipeline to execute the tests. We need to figure out process for two points:
1. The trigger-trusted-tests step is tagged as 'osdu-small'. That runner only executes in the OSDU namespace, so the job ends up stuck. We need a way to fall back to the general runners in this case.
2. The target repository wouldn't have the commits in it, so the process of marking the commit as trusted would be trickier. Need to explore possible ways of doing this, and document at least one method for trusted committers.https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/17Create a PMC project to manage CI/CD2021-06-16T22:19:20ZStephen Whitley (Invited Expert)Create a PMC project to manage CI/CDWe should start managing the CI/CD environment as a project with requirements and active management of issues.We should start managing the CI/CD environment as a project with requirements and active management of issues.David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/20Add Changes Needed for Catalog Upload2021-02-17T18:31:51ZNicholas KarskyAdd Changes Needed for Catalog Uploadadd changes to azure.yml needed to check for catalog changesadd changes to azure.yml needed to check for catalog changesNicholas KarskyNicholas Karsky