Update M15 Release Notes authored by Okoun-Ola Fabien Houeto's avatar Okoun-Ola Fabien Houeto
...@@ -47,10 +47,12 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha ...@@ -47,10 +47,12 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
* The content of TubularAssemblyType was by accident replicated in TubularAssenblyStatusType. The content has been corrected for [TubularAssemblyStatusType:1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/TubularAssemblyStatusType.1.0.0.md). * The content of TubularAssemblyType was by accident replicated in TubularAssenblyStatusType. The content has been corrected for [TubularAssemblyStatusType:1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/TubularAssemblyStatusType.1.0.0.md).
### Core and Helper Services ### Core and Helper Services
- `File Service` - `File Service`
- `AWS` - Now the file checksum is added to the metadata. See [ADR](https://community.opengroup.org/osdu/platform/system/file/-/issues/64) - `AWS` - Now the file checksum is added to the metadata. See [ADR](https://community.opengroup.org/osdu/platform/system/file/-/issues/64 "ADR: Calculate Checksum before saving metadata")
### Ingestion services ### Ingestion services
- Manifest-based Ingestion - Manifest by reference. - Manifest-based Ingestion - Manifest by reference.
- `AWS` now support ingestion of manifest by reference. - `AWS` now support ingestion of manifest by reference.
...@@ -71,6 +73,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha ...@@ -71,6 +73,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
### GCZ (Geospatial Consumption Zone) ### GCZ (Geospatial Consumption Zone)
* `AWS` - Experimental - As a developer, you can run the GCZ locally and connect to an OSDU on AWS instance. See [GCZ readme](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/tree/master/osdu_gcz_wiki).
* **Ingestion** * **Ingestion**
* Transformer now supports limited Ingestion of `WellLog` records via ID-to-ID spatialization * Transformer now supports limited Ingestion of `WellLog` records via ID-to-ID spatialization
* Example: Store Well Logs to Ignite **as Well Surface Locations only** with Well Log attributes - then view on a map * Example: Store Well Logs to Ignite **as Well Surface Locations only** with Well Log attributes - then view on a map
...@@ -153,9 +156,11 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha ...@@ -153,9 +156,11 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
* New class names: `FeatureCacheLocalSynchronizer`, `FeatureCacheRemoteSynchronizer` * New class names: `FeatureCacheLocalSynchronizer`, `FeatureCacheRemoteSynchronizer`
## Bug Fixes ## Bug Fixes
### Core and Helper Services ### Core and Helper Services
### Ingestion services ### Ingestion services
- Manifest-based Ingestion - Major performance improvement. - Manifest-based Ingestion - Major performance improvement.
- osdu-ingestion-lib - writing records in parallel/concurrent manner into Storage service. See [MR](https://urldefense.com/v3/\__https:/community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/30_\_;!!GF_29dbcQIUBPA!3PiTTmCUIwwQ6UToa6PfFOgrrAkdvsHADXCXk_QEpa6sG5oPUV6xrdRcoDu2jKjk-\_ar4V9ssumqtnn9kSfqrMapqwDrFqIJlts%24) - osdu-ingestion-lib - writing records in parallel/concurrent manner into Storage service. See [MR](https://urldefense.com/v3/\__https:/community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/30_\_;!!GF_29dbcQIUBPA!3PiTTmCUIwwQ6UToa6PfFOgrrAkdvsHADXCXk_QEpa6sG5oPUV6xrdRcoDu2jKjk-\_ar4V9ssumqtnn9kSfqrMapqwDrFqIJlts%24)
- osdu-airflow-lib - Added new Airflow variable core__ingestion__thread_save_number to tune a number of concurrent writings into Storage service for ProcessManifestOperatorR3 operator. This variable is optional (default: 1). See [MR](https://urldefense.com/v3/\__https:/community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/39_\_;!!GF_29dbcQIUBPA!3PiTTmCUIwwQ6UToa6PfFOgrrAkdvsHADXCXk_QEpa6sG5oPUV6xrdRcoDu2jKjk-\_ar4V9ssumqtnn9kSfqrMapqwDr98gQipI%24) - osdu-airflow-lib - Added new Airflow variable core__ingestion__thread_save_number to tune a number of concurrent writings into Storage service for ProcessManifestOperatorR3 operator. This variable is optional (default: 1). See [MR](https://urldefense.com/v3/\__https:/community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/39_\_;!!GF_29dbcQIUBPA!3PiTTmCUIwwQ6UToa6PfFOgrrAkdvsHADXCXk_QEpa6sG5oPUV6xrdRcoDu2jKjk-\_ar4V9ssumqtnn9kSfqrMapqwDr98gQipI%24)
...@@ -163,7 +168,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha ...@@ -163,7 +168,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
### DDMS ### DDMS
- `Wellbore DDMS` - `Wellbore DDMS`
- `AWS` - Fixed performance issues for reading wellbore ddms data. See [Issue](https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/298) - `AWS` - Fixed performance issues for reading wellbore ddms data. See [Issue](https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/298 "Slow performance for reading wellbore ddms data")
- lib updates: [fastAPI 0.70](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/659 "upgrade fastapi 0.70"), [Dask 2022.10.2](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/612 "Upgrade Fsspec and its dependencies: aws, azure, gcp + Dask to LATEST version"), [ffspec and implementations (adfs, s3fs, gcfs ...)](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/612 "Upgrade Fsspec and its dependencies: aws, azure, gcp + Dask to LATEST version"), [PyArrow 9.0.0](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/649 "Enh: upgrade PyArrow library to version 9.0.0 due to lots of improvement on memory consumption side"). - lib updates: [fastAPI 0.70](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/659 "upgrade fastapi 0.70"), [Dask 2022.10.2](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/612 "Upgrade Fsspec and its dependencies: aws, azure, gcp + Dask to LATEST version"), [ffspec and implementations (adfs, s3fs, gcfs ...)](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/612 "Upgrade Fsspec and its dependencies: aws, azure, gcp + Dask to LATEST version"), [PyArrow 9.0.0](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/649 "Enh: upgrade PyArrow library to version 9.0.0 due to lots of improvement on memory consumption side").
- Azure lib: retry mechanism to work around ClientAuthentication issues to blob storage (see [changes](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/lib/wellbore-cloud/wellbore-azure-lib/-/merge_requests/36 "Add new RetryableDefaultAzureIdentity class to authenticate to Azure services"), imported by [!659](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/626 "Change version of OSDU core lib Azure to change the way credentials are...")). - Azure lib: retry mechanism to work around ClientAuthentication issues to blob storage (see [changes](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/lib/wellbore-cloud/wellbore-azure-lib/-/merge_requests/36 "Add new RetryableDefaultAzureIdentity class to authenticate to Azure services"), imported by [!659](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/626 "Change version of OSDU core lib Azure to change the way credentials are...")).
- `Reservoir DMS` - `Reservoir DMS`
...@@ -197,7 +202,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha ...@@ -197,7 +202,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
* Ignite Cache Name * Ignite Cache Name
* We now require a separate identifying string for the name of a cache, as configured in either `application-local.yml`, `application-remote.yml`, or `application-k8s-remote.yml` * We now require a separate identifying string for the name of a cache, as configured in either `application-local.yml`, `application-remote.yml`, or `application-k8s-remote.yml`
* Each kind previously needed to contain the `name` and `chronExpr` properties, but now must contain a `cacheName` property * Each kind previously needed to contain the `name` and `chronExpr` properties, but now must contain a `cacheName` property
* The reason for this change is due to our new support of Persistence-Enabled Ignite Clusters, which strictly impose that cache names do not contain special characters such as `:` due to conflicts with file storage. Please note that taking the kind name and simply replacing all `:` characters with an underscore `\_` should work fine, but you are welcome to name your caches however you like, avoiding special characters. * The reason for this change is due to our new support of Persistence-Enabled Ignite Clusters, which strictly impose that cache names do not contain special characters such as `:` due to conflicts with file storage. Please note that taking the kind name and simply replacing all `:` characters with an underscore `\\\\\\\_` should work fine, but you are welcome to name your caches however you like, avoiding special characters.
* Additionally, you will need to configure `koop-config.json` to include the `cacheName` property or else the Provider will be unable to find the cache by `kind` entity alone. * Additionally, you will need to configure `koop-config.json` to include the `cacheName` property or else the Provider will be unable to find the cache by `kind` entity alone.
## Known Issues/Gaps ## Known Issues/Gaps
...@@ -210,7 +215,6 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha ...@@ -210,7 +215,6 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
EDS DMS - In the current version EDS DMS can manage bulk data files from one Data Provider only. It will not support a combination of dataset IDs from different data providers.Also, testing can be done using eds-DMS API and not from data set API. EDS DMS - In the current version EDS DMS can manage bulk data files from one Data Provider only. It will not support a combination of dataset IDs from different data providers.Also, testing can be done using eds-DMS API and not from data set API.
## Supporting Information ## Supporting Information
## OSDU Community utilities ## OSDU Community utilities
\ No newline at end of file