@@ -47,10 +47,12 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
* The content of TubularAssemblyType was by accident replicated in TubularAssenblyStatusType. The content has been corrected for [TubularAssemblyStatusType:1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/TubularAssemblyStatusType.1.0.0.md).
### Core and Helper Services
-`File Service`
-`AWS` - Now the file checksum is added to the metadata. See [ADR](https://community.opengroup.org/osdu/platform/system/file/-/issues/64)
-`AWS` - Now the file checksum is added to the metadata. See [ADR](https://community.opengroup.org/osdu/platform/system/file/-/issues/64"ADR: Calculate Checksum before saving metadata")
### Ingestion services
- Manifest-based Ingestion - Manifest by reference.
-`AWS` now support ingestion of manifest by reference.
...
...
@@ -71,6 +73,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
### GCZ (Geospatial Consumption Zone)
*`AWS` - Experimental - As a developer, you can run the GCZ locally and connect to an OSDU on AWS instance. See [GCZ readme](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/tree/master/osdu_gcz_wiki).
***Ingestion**
* Transformer now supports limited Ingestion of `WellLog` records via ID-to-ID spatialization
* Example: Store Well Logs to Ignite **as Well Surface Locations only** with Well Log attributes - then view on a map
...
...
@@ -153,9 +156,11 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
* New class names: `FeatureCacheLocalSynchronizer`, `FeatureCacheRemoteSynchronizer`
## Bug Fixes
### Core and Helper Services
### Ingestion services
- Manifest-based Ingestion - Major performance improvement.
- osdu-ingestion-lib - writing records in parallel/concurrent manner into Storage service. See [MR](https://urldefense.com/v3/\__https:/community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/30_\_;!!GF_29dbcQIUBPA!3PiTTmCUIwwQ6UToa6PfFOgrrAkdvsHADXCXk_QEpa6sG5oPUV6xrdRcoDu2jKjk-\_ar4V9ssumqtnn9kSfqrMapqwDrFqIJlts%24)
- osdu-airflow-lib - Added new Airflow variable core__ingestion__thread_save_number to tune a number of concurrent writings into Storage service for ProcessManifestOperatorR3 operator. This variable is optional (default: 1). See [MR](https://urldefense.com/v3/\__https:/community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/39_\_;!!GF_29dbcQIUBPA!3PiTTmCUIwwQ6UToa6PfFOgrrAkdvsHADXCXk_QEpa6sG5oPUV6xrdRcoDu2jKjk-\_ar4V9ssumqtnn9kSfqrMapqwDr98gQipI%24)
...
...
@@ -163,7 +168,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
### DDMS
-`Wellbore DDMS`
-`AWS` - Fixed performance issues for reading wellbore ddms data. See [Issue](https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/298)
-`AWS` - Fixed performance issues for reading wellbore ddms data. See [Issue](https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/298"Slow performance for reading wellbore ddms data")
- lib updates: [fastAPI 0.70](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/659"upgrade fastapi 0.70"), [Dask 2022.10.2](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/612"Upgrade Fsspec and its dependencies: aws, azure, gcp + Dask to LATEST version"), [ffspec and implementations (adfs, s3fs, gcfs ...)](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/612"Upgrade Fsspec and its dependencies: aws, azure, gcp + Dask to LATEST version"), [PyArrow 9.0.0](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/649"Enh: upgrade PyArrow library to version 9.0.0 due to lots of improvement on memory consumption side").
- Azure lib: retry mechanism to work around ClientAuthentication issues to blob storage (see [changes](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/lib/wellbore-cloud/wellbore-azure-lib/-/merge_requests/36"Add new RetryableDefaultAzureIdentity class to authenticate to Azure services"), imported by [!659](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/626"Change version of OSDU core lib Azure to change the way credentials are...")).
-`Reservoir DMS`
...
...
@@ -197,7 +202,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
* Ignite Cache Name
* We now require a separate identifying string for the name of a cache, as configured in either `application-local.yml`, `application-remote.yml`, or `application-k8s-remote.yml`
* Each kind previously needed to contain the `name` and `chronExpr` properties, but now must contain a `cacheName` property
* The reason for this change is due to our new support of Persistence-Enabled Ignite Clusters, which strictly impose that cache names do not contain special characters such as `:` due to conflicts with file storage. Please note that taking the kind name and simply replacing all `:` characters with an underscore `\_` should work fine, but you are welcome to name your caches however you like, avoiding special characters.
* The reason for this change is due to our new support of Persistence-Enabled Ignite Clusters, which strictly impose that cache names do not contain special characters such as `:` due to conflicts with file storage. Please note that taking the kind name and simply replacing all `:` characters with an underscore `\\\\\\\_` should work fine, but you are welcome to name your caches however you like, avoiding special characters.
* Additionally, you will need to configure `koop-config.json` to include the `cacheName` property or else the Provider will be unable to find the cache by `kind` entity alone.
## Known Issues/Gaps
...
...
@@ -210,7 +215,6 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha
EDS DMS - In the current version EDS DMS can manage bulk data files from one Data Provider only. It will not support a combination of dataset IDs from different data providers.Also, testing can be done using eds-DMS API and not from data set API.