... | ... | @@ -47,10 +47,12 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha |
|
|
* The content of TubularAssemblyType was by accident replicated in TubularAssenblyStatusType. The content has been corrected for [TubularAssemblyStatusType:1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/TubularAssemblyStatusType.1.0.0.md).
|
|
|
|
|
|
### Core and Helper Services
|
|
|
|
|
|
- `File Service`
|
|
|
- `AWS` - Now the file checksum is added to the metadata. See [ADR](https://community.opengroup.org/osdu/platform/system/file/-/issues/64)
|
|
|
- `AWS` - Now the file checksum is added to the metadata. See [ADR](https://community.opengroup.org/osdu/platform/system/file/-/issues/64 "ADR: Calculate Checksum before saving metadata")
|
|
|
|
|
|
### Ingestion services
|
|
|
|
|
|
- Manifest-based Ingestion - Manifest by reference.
|
|
|
- `AWS` now support ingestion of manifest by reference.
|
|
|
|
... | ... | @@ -68,9 +70,10 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha |
|
|
- Handle EDS [incremental fetch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/external-data-framework/-/issues/252 "Handle EDS incremental fetch scenario") scenario
|
|
|
- Provide [default date and time](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/external-data-framework/-/issues/253 "Provide default date and time to LastSuccessfulRunDateUTC and CreateTimeMax in ConnectedSourceDataJob (CSDJ))") to LastSuccessfulRunDateUTC in ConnectedSourceDataRegistry
|
|
|
- [EDS DMS](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/external-data-framework/-/issues/52 "EDS DMS - Options for Consumer to manage bulk data files (ex: LAS, DLIS, SegY) from Data Provider") - Options for Consumer to manage bulk data files (ex: LAS, DLIS, SegY) from Data Provider
|
|
|
|
|
|
|
|
|
### GCZ (Geospatial Consumption Zone)
|
|
|
|
|
|
* `AWS` - Experimental - As a developer, you can run the GCZ locally and connect to an OSDU on AWS instance. See [GCZ readme](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/tree/master/osdu_gcz_wiki).
|
|
|
* **Ingestion**
|
|
|
* Transformer now supports limited Ingestion of `WellLog` records via ID-to-ID spatialization
|
|
|
* Example: Store Well Logs to Ignite **as Well Surface Locations only** with Well Log attributes - then view on a map
|
... | ... | @@ -153,9 +156,11 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha |
|
|
* New class names: `FeatureCacheLocalSynchronizer`, `FeatureCacheRemoteSynchronizer`
|
|
|
|
|
|
## Bug Fixes
|
|
|
|
|
|
### Core and Helper Services
|
|
|
|
|
|
### Ingestion services
|
|
|
|
|
|
- Manifest-based Ingestion - Major performance improvement.
|
|
|
- osdu-ingestion-lib - writing records in parallel/concurrent manner into Storage service. See [MR](https://urldefense.com/v3/\__https:/community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/30_\_;!!GF_29dbcQIUBPA!3PiTTmCUIwwQ6UToa6PfFOgrrAkdvsHADXCXk_QEpa6sG5oPUV6xrdRcoDu2jKjk-\_ar4V9ssumqtnn9kSfqrMapqwDrFqIJlts%24)
|
|
|
- osdu-airflow-lib - Added new Airflow variable core__ingestion__thread_save_number to tune a number of concurrent writings into Storage service for ProcessManifestOperatorR3 operator. This variable is optional (default: 1). See [MR](https://urldefense.com/v3/\__https:/community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/39_\_;!!GF_29dbcQIUBPA!3PiTTmCUIwwQ6UToa6PfFOgrrAkdvsHADXCXk_QEpa6sG5oPUV6xrdRcoDu2jKjk-\_ar4V9ssumqtnn9kSfqrMapqwDr98gQipI%24)
|
... | ... | @@ -163,7 +168,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha |
|
|
### DDMS
|
|
|
|
|
|
- `Wellbore DDMS`
|
|
|
- `AWS` - Fixed performance issues for reading wellbore ddms data. See [Issue](https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/298)
|
|
|
- `AWS` - Fixed performance issues for reading wellbore ddms data. See [Issue](https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/298 "Slow performance for reading wellbore ddms data")
|
|
|
- lib updates: [fastAPI 0.70](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/659 "upgrade fastapi 0.70"), [Dask 2022.10.2](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/612 "Upgrade Fsspec and its dependencies: aws, azure, gcp + Dask to LATEST version"), [ffspec and implementations (adfs, s3fs, gcfs ...)](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/612 "Upgrade Fsspec and its dependencies: aws, azure, gcp + Dask to LATEST version"), [PyArrow 9.0.0](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/649 "Enh: upgrade PyArrow library to version 9.0.0 due to lots of improvement on memory consumption side").
|
|
|
- Azure lib: retry mechanism to work around ClientAuthentication issues to blob storage (see [changes](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/lib/wellbore-cloud/wellbore-azure-lib/-/merge_requests/36 "Add new RetryableDefaultAzureIdentity class to authenticate to Azure services"), imported by [!659](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/626 "Change version of OSDU core lib Azure to change the way credentials are...")).
|
|
|
- `Reservoir DMS`
|
... | ... | @@ -197,7 +202,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha |
|
|
* Ignite Cache Name
|
|
|
* We now require a separate identifying string for the name of a cache, as configured in either `application-local.yml`, `application-remote.yml`, or `application-k8s-remote.yml`
|
|
|
* Each kind previously needed to contain the `name` and `chronExpr` properties, but now must contain a `cacheName` property
|
|
|
* The reason for this change is due to our new support of Persistence-Enabled Ignite Clusters, which strictly impose that cache names do not contain special characters such as `:` due to conflicts with file storage. Please note that taking the kind name and simply replacing all `:` characters with an underscore `\_` should work fine, but you are welcome to name your caches however you like, avoiding special characters.
|
|
|
* The reason for this change is due to our new support of Persistence-Enabled Ignite Clusters, which strictly impose that cache names do not contain special characters such as `:` due to conflicts with file storage. Please note that taking the kind name and simply replacing all `:` characters with an underscore `\\\\\\\_` should work fine, but you are welcome to name your caches however you like, avoiding special characters.
|
|
|
* Additionally, you will need to configure `koop-config.json` to include the `cacheName` property or else the Provider will be unable to find the cache by `kind` entity alone.
|
|
|
|
|
|
## Known Issues/Gaps
|
... | ... | @@ -210,7 +215,6 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha |
|
|
|
|
|
EDS DMS - In the current version EDS DMS can manage bulk data files from one Data Provider only. It will not support a combination of dataset IDs from different data providers.Also, testing can be done using eds-DMS API and not from data set API.
|
|
|
|
|
|
|
|
|
## Supporting Information
|
|
|
|
|
|
## OSDU Community utilities |
|
|
\ No newline at end of file |