Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2023-08-18T22:22:24Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/312Patching log4j vulnerability2023-08-18T22:22:24ZSpencer Suttonsuttonsp@amazon.comPatching log4j vulnerabilityPart of the #102 seriesPart of the #102 seriesM10 - Release 0.13Spencer Suttonsuttonsp@amazon.comSpencer Suttonsuttonsp@amazon.comhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/311log4j-vuln-fix by upgrading to log4j 2.16.0 version2023-08-18T22:22:25ZAshwani Pandeylog4j-vuln-fix by upgrading to log4j 2.16.0 version| module pom changes | Ref Issue |
| ------ | ------ |
| IBM | osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm#2
| core | osdu/platform/system/lib/core/os-core-common#54
Part of the #102 series| module pom changes | Ref Issue |
| ------ | ------ |
| IBM | osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm#2
| core | osdu/platform/system/lib/core/os-core-common#54
Part of the #102 seriesM10 - Release 0.13David Diederichd.diederich@opengroup.orgShrikant GargDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/310GCP osm-obm-oqm2021-12-13T09:51:00ZRostislav Dublin (EPAM)GCP osm-obm-oqm## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
---
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud p...## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
---
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Storage works with KV, Blobs and messaging directly.
## What is the new/expected behavior?
Storage service will use EPAM OSM, OBM, OQM mappers for data management flexibility
## Have you added/updated Unit Tests and Integration Tests?
yes
## Any other useful information
### Features of implementation
This is a universal solution created using EPAM OSM, OBM and OQM mappers technology. It allows you to work with various implementations of data stores and message brokers.
#### Limitations of the current version
In the current version, the mappers are equipped with several drivers to the stores and the message broker:
* OSM (mapper for KV-data): Google Datastore; Postgres
* OBM (mapper to Blob stores): Google Cloud Storage (GCS); MinIO
* OQM (mapper to message brokers): Google PubSub; RabbitMQ
#### Extensibility
To use any other store or message broker, implement a driver for it. With an extensible set of drivers, the solution is unrestrictedly universal and portable without modification to the main code.
#### Mapper tuning mechanisms
This service uses specific implementations of DestinationResolvers based on the tenant information provided by the OSDU Partition service. A total of 6 resolvers are implemented, which are divided into two groups:
##### for universal technologies:
* for Postgres: mappers/osm/PgTenantOsmDestinationResolver.java
* for MinIO: mappers/oqm/MioTenantOqmDestinationResolver.java
* for RabbitMQ: mappers/oqm/MqTenantOqmDestinationResolver.java
###### Their algorithms are as follows:
* incoming Destination carries data-partition-id
* resolver accesses the Partition service and gets PartitionInfo
* from PartitionInfo resolver retrieves properties for the connection: URL, username, password etc.
* resolver creates a data source, connects to the resource, remembers the datasource
* resolver gives the datasource to the mapper in the Resolution object
##### for native Google Cloud technologies:
* for Datastore: mappers/osm/DsTenantOsmDestinationResolver.java
* for GCS: mappers/oqm/CsTenantOqmDestinationResolver.java
* for PubSub: mappers/oqm/PsTenantOqmDestinationResolver.java
###### Their algorithms are similar,
Except that they do not receive special properties from the Partition service for connection, because the location of the resources is unambiguously known - they are in the GCP project. And credentials are also not needed - access to data is made on behalf of the Google Identity SA under which the service itself is launched. Therefore, resolver takes only the value of the **projectId** property from PartitionInfo and uses it to connect to a resource in the corresponding GCP project.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/308Experimental FOSSA caching2023-08-18T22:22:27ZDavid Diederichd.diederich@opengroup.orgExperimental FOSSA cachingThis MR adds in some experimental FOSSA logic, which is intended to ease the pain of using FOSSA. I'd like to add it to one service to get some "real-world" testing, then apply it globally once we see it works reasonably. I'm very intere...This MR adds in some experimental FOSSA logic, which is intended to ease the pain of using FOSSA. I'd like to add it to one service to get some "real-world" testing, then apply it globally once we see it works reasonably. I'm very interested in feedback on how this is working and ways that it could be improved before we make it standard.
Note that this uses the image `fossa-with-cache/incremental:latest` -- meaning it grabs the latest build rather than a stable release one. This is to make it easier to iterate and refine during this testing phase. Before it is moved to the ci-cd-pipelines project, it will be changed to follow a release tag.
### Caching of the NOTICE Files
The primary feature of the [`fossa-with-cache`](https://community.opengroup.org/divido/fossa-with-cache) tool is that it caches the NOTICE that FOSSA generates and associates it to a list of project dependencies. If the dependencies do not change, the tool will re-use the previously generated NOTICE rather than asking FOSSA to create a new one. This should cut down on instances of small, jittery changes in the NOTICE from one generation to the next.
The cache ages out after a period of time (default = 1 week), so the first pipeline run after that age will regenerate even if there's a cached version available. This is to avoid going too long before reconsulting FOSSA -- FOSSA's database is constantly evolving, and it may have new information that the previous run didn't.
### Single package names, list of URLs
When multiple different package names refer to the same package (like spring-security & Spring Security), a single canonical name is used instead. When multiple URLs are used, they are concatenated into a comma separated list. The configuration for this is hard-coded, and will need to be updated over time. But, at least for the known cases, this should cut down on NOTICE differences that are only these typographical changes.
This does mean that every NOTICE file will change the first time the pipeline uses this logic. This is unavoidable, but should only happen once on each project.
### NOTICE files can be updated without re-running the pipeline
If there are new commits to a branch, and the only changes are the NOTICE file, the `fossa-check-notice` stage will compare the generated (or cached) NOTICE against the latest committed version, rather than the one associated with the pipeline. This allows developers to fix a NOTICE file without having to re-run the entire pipeline.
This is a bit counterintuitive from a CI perspective. Normally, you expect that pipelines only operate on the specific commit that generated them. However, in this case, I think the non-standard technique is useful to reduce the cost associated with a NOTICE difference.
The skipped pipeline also shows up in the list of pipelines, including the MR list. We'll need to know to look for previous pipelines if we see a skipped pipeline with commit message "Update NOTICE" or similar.
In the case of trusted branches, the NOTICE should be updated on the regular development branch first. Then, move the trusted branch to match (`git branch -f trusted-devBranch devBranch` or similar) and push it up. Trusted branches already suppress pipelines, so there's no need to push with `-o ci.skip` on that one. Finally, re-run the `fossa-check-notice` on the protected pipeline. If you only update the trusted branch, the pipeline will pass but the NOTICE files won't get merged in. If you only update the development branch, the pipeline won't see them and will continue to fail. I may add some additional commentary to the pipeline failure message to help with this, or perhaps add special logic around the trusted branches to make this easier. Interested in feedback / ideas on this.
**Example: This MR**
For [pipeline 79827](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/79827), the [original `fossa-check-notice` job (747962)](https://community.opengroup.org/osdu/platform/system/storage/-/jobs/747962) failed with differences in the NOTICE. I followed the steps to download the cached version, and [re-ran the job (748097)](https://community.opengroup.org/osdu/platform/system/storage/-/jobs/748097). The [pipeline (79837)](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/79837) from the changed NOTICE was skipped; and that shows up in this MR as the latest pipeline.M10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/307tomcat embed core vuln fix2022-09-16T08:06:07ZGokul Nagaretomcat embed core vuln fixfixes issue - https://community.opengroup.org/osdu/platform/system/storage/-/issues/101fixes issue - https://community.opengroup.org/osdu/platform/system/storage/-/issues/101M10 - Release 0.13Anuj GuptaShrikant GargAnuj Guptahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/305disabled spring web security2023-08-18T22:22:30ZBhushan Radedisabled spring web security# Merge request template# Merge request templateM10 - Release 0.13Anuj GuptaBhushan RadeAnuj Guptahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/303apply date and unti conversion for records with AsIngestedCoordinates2022-09-16T08:15:48ZNeelesh Thakurapply date and unti conversion for records with AsIngestedCoordinatesRelated Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/99Related Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/99M10 - Release 0.13https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/297Catch and throw datastore app exception with non-500 error code2023-08-18T22:22:32ZAlok JoshiCatch and throw datastore app exception with non-500 error code# Merge request template
Refer to this issue https://community.opengroup.org/osdu/platform/system/storage/-/issues/97# Merge request template
Refer to this issue https://community.opengroup.org/osdu/platform/system/storage/-/issues/97M10 - Release 0.13Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/293Properties change for Reference module (GONRG-3574)2021-11-08T14:16:06ZArtem Dobrynin (EPAM)Properties change for Reference module (GONRG-3574)## Type of change
- [X] Bug Fix
- [ ] Feature
Modified properties to comply with env variables in GCP
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if ...## Type of change
- [X] Bug Fix
- [ ] Feature
Modified properties to comply with env variables in GCP
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
- [X] Reference
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
## What is the new/expected behavior?
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationM10 - Release 0.13Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/291(GONRG-3326) update hosts2023-08-18T22:22:33ZVladislav Shishko (EPAM)(GONRG-3326) update hosts# Merge request template# Merge request templateM10 - Release 0.13Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/290Use version 0.12.0-SNAPSHOT of core lib aws2023-08-18T22:22:35ZBill WangUse version 0.12.0-SNAPSHOT of core lib aws# Merge request template# Merge request templateM10 - Release 0.13Bill WangBill Wanghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/287remove record versions support for patch api2022-09-16T08:13:53ZYauheni Lesnikauremove record versions support for patch apiIssue : https://community.opengroup.org/osdu/platform/system/storage/-/issues/95Issue : https://community.opengroup.org/osdu/platform/system/storage/-/issues/95M10 - Release 0.13Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/257integration tests fix for schema api feature flag2023-08-18T22:23:04ZLarissa Pereiraintegration tests fix for schema api feature flagRelated ADR: [62](https://community.opengroup.org/osdu/platform/system/storage/-/issues/62)
**Background** Instead of complete irreversible removal of the Storage Schema endpoints, a fail-safe option to restore application functionality...Related ADR: [62](https://community.opengroup.org/osdu/platform/system/storage/-/issues/62)
**Background** Instead of complete irreversible removal of the Storage Schema endpoints, a fail-safe option to restore application functionality is needed in cases where missed dependencies on Storage Schemas APIs have gone undetected.
This MR fixes any integration test failures due to the feature flag by skipping the tests if the feature flag has been disabled (viz. schema endpoints are disabled)M10 - Release 0.13https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/397Cherry-pick "Limit exception remove" into release/0.142022-04-04T20:41:27ZDavid Diederichd.diederich@opengroup.orgCherry-pick "Limit exception remove" into release/0.14Original MR: !377Original MR: !377M11 - Release 0.14David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/396Updating NOTICE2022-04-04T18:49:58ZDavid Diederichd.diederich@opengroup.orgUpdating NOTICEM11 - Release 0.14David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/392Removed StorageConfigProperties. Fixed typos in README files (GONRG-4534)2022-03-29T12:57:31ZRiabokon Stanislav(EPAM)[GCP]Removed StorageConfigProperties. Fixed typos in README files (GONRG-4534)## Type of change
- [x] Bug Fix
- [ ] Feature
Removed StorageConfigProperties. Fixed typos in README files.
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementatio...## Type of change
- [x] Bug Fix
- [ ] Feature
Removed StorageConfigProperties. Fixed typos in README files.
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBMM11 - Release 0.14Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/390Updated storage service to use RedisAzureCache2023-08-18T22:21:39ZShiv SinghUpdated storage service to use RedisAzureCache# Merge request template# Merge request templateM11 - Release 0.14Shiv SinghShiv Singhhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/389Added user email attribute to OqmMessage (EPAM - GONRG-4620)2023-08-18T22:21:41ZDmitrii Novikov (EPAM)Added user email attribute to OqmMessage (EPAM - GONRG-4620)## Type of change
- [ ] Bug Fix
- [x] Feature
https://jiraeu.epam.com/browse/GONRG-4620
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud...## Type of change
- [ ] Bug Fix
- [x] Feature
https://jiraeu.epam.com/browse/GONRG-4620
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Added user email attibute to OqmMessage for AuditLogger in IndexerService.M11 - Release 0.14Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/387update DatesConversion for the nested properties (increase os-core-common ver...2022-09-16T07:41:25ZYauheni Lesnikauupdate DatesConversion for the nested properties (increase os-core-common version)Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/115Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/115M11 - Release 0.14Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/386Remove the definition of os-core-common version in storage-aws2022-03-21T17:07:37ZYifei XuRemove the definition of os-core-common version in storage-aws## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provi...## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
## What is the new/expected behavior?
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationM11 - Release 0.14JoeYifei XuOkoun-Ola Fabien HouetoJoe