-[`External Data Services (Experimental/Early Release)`](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources) - `Azure`, `GCP`
-[`Geospatial Consumption Zone (Experimental/Early Release)`](https://community.opengroup.org/osdu/platform/consumption/geospatial) - `IBM`, `AWS (Developer/Local only)`
...
...
@@ -60,30 +61,31 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
* The content of TubularAssemblyType was by accident replicated in TubularAssenblyStatusType. The content has been corrected for [TubularAssemblyStatusType:1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/TubularAssemblyStatusType.1.0.0.md).
### Core and Helper Services
-`Indexer`
- Info api implementation for adding outerservice version in response. See MR [!424](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/424)
- Info api implementation for adding outerservice version in response. See MR [!424](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/424"info api implementation for adding outerservice version in response")
-`Storage`
- Include collaboration context when pushing to new record changed topic. See MR [!553](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/553)
- Update Storage to be Collaboration Context Aware. See MR [!546](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/546)
- Changed CRC32C to MD5 to calculate a checksum (GONRG-5884). See MR [!542](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/542)
- Added ability to config redis password and ssl (GONRG-5301). See MR [!540](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/540)
- ADR 92: Populate recordBlocks in record modification events. See MR [!539](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/539)
- Include collaboration context when pushing to new record changed topic. See MR [!553](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/553"Include collaboration context when pushing to new record changed topic")
- Update Storage to be Collaboration Context Aware. See MR [!546](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/546"Update Storage to be Collaboration Context Aware")
- Changed CRC32C to MD5 to calculate a checksum (GONRG-5884). See MR [!542](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/542"Changed CRC32C to MD5 to calculate a checksum (GONRG-5884)")
- Added ability to config redis password and ssl (GONRG-5301). See MR [!540](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/540"added ability to config redis password and ssl (GONRG-5301)")
- ADR 92: Populate recordBlocks in record modification events. See MR [!539](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/539"ADR 92: Populate recordBlocks in record modification events")
-`File Service`
-`AWS` - Now the file checksum is added to the metadata. See [ADR](https://community.opengroup.org/osdu/platform/system/file/-/issues/64"ADR: Calculate Checksum before saving metadata") and MR [!383](https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/383)
-`AWS` - Now the file checksum is added to the metadata. See [ADR](https://community.opengroup.org/osdu/platform/system/file/-/issues/64"ADR: Calculate Checksum before saving metadata") and MR [!383](https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/383"Implement the file checksum in AWS")
-`Schema`
- OSDU-DD-Delivery-M15 (v0.18.0). See MR [!407](https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/407)
- OSDU-DD-Delivery-M15 (v0.18.0). See MR [!407](https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/407"OSDU-DD-Delivery-M15 (v0.18.0)")
-`CRS Conversion v3`
- "Add convertBinGrid endpoint". See MR [!235](https://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service/-/merge_requests/235)
- "Add convertBinGrid endpoint". See MR [!235](https://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service/-/merge_requests/235'Resolve "Add convertBinGrid endpoint"')
-`Policy`
- Translate, Logging and Correlation ID update. See MR [!256](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/merge_requests/256)
- Translate, Logging and Correlation ID update. See MR [!256](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/merge_requests/256"Translate, Logging and Correlation ID update")
-`Lib`
- Expose Collaboration Context. See MR [!183](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/183)
- Add validator to record ancestry to make sure parent record id is valid. See MR [!174](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/174)
- Expose Collaboration Context. See MR [!183](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/183"Expose Collaboration Context")
- Add validator to record ancestry to make sure parent record id is valid. See MR [!174](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/174"add validator to record ancestry to make sure parent record id is valid")
### Ingestion services
- Manifest-based Ingestion - Major performance improvement.
- osdu-ingestion-lib - writing records in parallel/concurrent manner into Storage service. See [MR](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/30)
- osdu-ingestion-lib - writing records in parallel/concurrent manner into Storage service. See [MR](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/30"GONRG-5877: Parallelize processing manifest")
- osdu-airflow-lib - Added new Airflow variable core__ingestion__thread_save_number to tune a number of concurrent writings into Storage service for ProcessManifestOperatorR3 operator. This variable is optional (default: 1). See [MR](https:/community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/39)
- Manifest-based Ingestion - Manifest by reference.
-`AWS` now support ingestion of manifest by reference.
...
...
@@ -190,54 +192,53 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
## Bug Fixes
### Core and Helper Services
-`Storage`
- Bug Fix for Azure: update the message to not be sent to recordstopic v2 if feature flag is off. See MR [!582](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/582)
- Added spring-boot-maven plugin version. See MR [!560](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/560)
- Removing the azure message bus message id. See MR [!550](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/550)
- Storage Performance (GONRG-4754). See MR [!535](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/535)
- Delete all versions of the blob when purging a record. See MR [!534](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/534)
- Consume latest version of core common library and update the way of getting parent records. See MR [!532](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/532)
- Bug Fix for Azure: update the message to not be sent to recordstopic v2 if feature flag is off. See MR [!582](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/582"Bug Fix for Azure: update the message to not be sent to recordstopic v2 if feature flag is off")
- Removing the azure message bus message id. See MR [!550](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/550"Removing the azure message bus message id")
- Storage Performance (GONRG-4754). See MR [!535](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/535"Storage Performance (GONRG-4754)")
- Delete all versions of the blob when purging a record. See MR [!534](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/534"Delete all versions of the blob when purging a record")
- Consume latest version of core common library and update the way of getting parent records. See MR [!532](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/532"consume latest version of core common library and update the way of getting parent records")
-`CRS Catalog`
- Jar type vulnerability fix for catalog service. See MR [!220](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/merge_requests/220)
-`Unit`
- Unit Service vulnerability fix. See MR [!222](https://community.opengroup.org/osdu/platform/system/reference/unit-service/-/merge_requests/222)
- Squashed commit of the following. See MR [!216](https://community.opengroup.org/osdu/platform/system/reference/unit-service/-/merge_requests/216)
- Jar type vulnerability fix for catalog service. See MR [!220](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/merge_requests/220"jar type vulnerability fix for catalog service")
-`Unit`
- Unit Service vulnerability fix. See MR [!222](https://community.opengroup.org/osdu/platform/system/reference/unit-service/-/merge_requests/222"Unit Service vulnerability fix")
- Squashed commit of the following. See MR [!216](https://community.opengroup.org/osdu/platform/system/reference/unit-service/-/merge_requests/216"Squashed commit of the following")
-`Dataset`
- Aws integration merge. See MR [!276](https://community.opengroup.org/osdu/platform/system/dataset/-/merge_requests/276)
- Aws integration merge. See MR [!276](https://community.opengroup.org/osdu/platform/system/dataset/-/merge_requests/276"Aws integration merge")
-`Register`
- White source updates. See MR [!262](https://community.opengroup.org/osdu/platform/system/register/-/merge_requests/262)
- White source updates. See MR [!262](https://community.opengroup.org/osdu/platform/system/register/-/merge_requests/262"white source updates")
-`Schema`
- Fix spring-boot-maven plugin version. See MR [!412](https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/412)
- Fix spring-boot-maven-plugin. See MR [!421](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/421)
- Whitesource-update. See MR [!412](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/412)
- Removed retry attempts and call to storage service. See MR [!384](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/384)
- Fix spring-boot-maven-plugin. See MR [!421](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/421"Fix spring-boot-maven-plugin")
- Whitesource-update. See MR [!412](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/412"whitesource-update")
- Removed retry attempts and call to storage service. See MR [!384](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/384"Removed retry attempts and call to storage service")
-`Search`
- Added spring-boot-maven plugin version. See MR [!375](https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/375)
- Parse-query changes. See MR [!372](https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/372)
- Parse-query changes. See MR [!372](https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/372"parse-query changes")
-`Partition`
- Update vulnerable dependencies. See MR [!285](https://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/285)
- Update vulnerable dependencies. See MR [!285](https://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/285"update vulnerable dependencies")
-`File`
- Bypass checksum generation for file size more than 5 gbs. See MR [!368](https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/368)
- Bypass checksum generation for file size more than 5 gbs. See MR [!368](https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/368"Bypass checksum generation for file size more than 5 gbs")
-`Entitlements`
- Added spring-boot-maven plugin version. See MR [!382](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/merge_requests/382)
- Update vulnerable dependencies. See MR [!315](https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/merge_requests/315"update vulnerable dependencies")
-`Policy`
- Test ibm bucket. See MR [!253](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/merge_requests/253)
- Test ibm bucket. See MR [!253](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/merge_requests/253"Test ibm bucket")
### Ingestion services
-`Lib`
- Upgrade lombok version to fix ingestion request deserialization. See MR [!329](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/329)
- Trusted development. See MR [!147](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/merge_requests/147)
- Trusted development. See MR [!143](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/merge_requests/143)
- Fix Vulnerabilities. See MR [!12](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/eds-dms/-/merge_requests/12)
- Upgrade lombok version to fix ingestion request deserialization. See MR [!329](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/329"upgrade lombok version to fix ingestion request deserialization")
- Trusted development. See MR [!147](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/merge_requests/147"Trusted development")
- Trusted development. See MR [!143](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/merge_requests/143"Trusted development")
- Fix Vulnerabilities. See MR [!12](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/eds-dms/-/merge_requests/12"Fix Vulnerabilities")
-`Segy to Zgy Conversion`
- Fix: segy_configure_remote_access to set http headers. See MR [!163](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/163)
- Fix: segy_configure_remote_access to set http headers. See MR [!163](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/163"fix: segy_configure_remote_access to set http headers")
### DDMS
...
...
@@ -276,7 +277,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
* Ignite Cache Name
* We now require a separate identifying string for the name of a cache, as configured in either `application-local.yml`, `application-remote.yml`, or `application-k8s-remote.yml`
* Each kind previously needed to contain the `name` and `chronExpr` properties, but now must contain a `cacheName` property
* The reason for this change is due to our new support of Persistence-Enabled Ignite Clusters, which strictly impose that cache names do not contain special characters such as `:` due to conflicts with file storage. Please note that taking the kind name and simply replacing all `:` characters with an underscore `\\\\\\\_` should work fine, but you are welcome to name your caches however you like, avoiding special characters.
* The reason for this change is due to our new support of Persistence-Enabled Ignite Clusters, which strictly impose that cache names do not contain special characters such as `:` due to conflicts with file storage. Please note that taking the kind name and simply replacing all `:` characters with an underscore `\\\\\\\\\\\\\\\_` should work fine, but you are welcome to name your caches however you like, avoiding special characters.
* Additionally, you will need to configure `koop-config.json` to include the `cacheName` property or else the Provider will be unable to find the cache by `kind` entity alone.
## Known Issues/Gaps
...
...
@@ -289,8 +290,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
EDS DMS - In the current version EDS DMS can manage bulk data files from one Data Provider only. It will not support a combination of dataset IDs from different data providers.Also, testing can be done using eds-DMS API and not from data set API.
