... | ... | @@ -10,22 +10,22 @@ For detailed services that are included in this release, please refer to [M15 ta |
|
|
|
|
|
## New Features
|
|
|
|
|
|
### Data Definition
|
|
|
### Data Definition
|
|
|
|
|
|
**_Attention:_** Please use [tag v0.18.1](https://community.opengroup.org/osdu/data/data-definitions/-/tags/v0.18.1) or [release/0.18](https://community.opengroup.org/osdu/data/data-definitions/-/tree/release/0.18) because tag v0.18.0 contains one OperationsReport.1.2.0 schema resource, which fails to register. The patch v0.18.1 corrects this mistake.
|
|
|
**_Attention:_** Please use [tag v0.18.1](https://community.opengroup.org/osdu/data/data-definitions/-/tags/v0.18.1) or [release/0.18](https://community.opengroup.org/osdu/data/data-definitions/-/tree/release/0.18) because tag v0.18.0 contains one OperationsReport.1.2.0 schema resource, which fails to register. The patch v0.18.1 corrects this mistake.
|
|
|
|
|
|
No in-place schema changes are required in M15 (no M14 schema resources were changes). Refer here for [full changes "OSDU schemas"](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/ChangeReport.md#snapshot-2022-09-23-towards-m15) and [full changes "OSDU reference-values"](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/ReferenceValues/ChangeLogs/M15/README.md)
|
|
|
|
|
|
* **New Entity Types**
|
|
|
* **New Entity Types**
|
|
|
* **_Seismic Domain_**: New template reference data type [SEGY-HeaderMappingTemplate:1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/SEGY-HeaderMappingTemplate.1.0.0.md) to support SEG-Y parsing assistance in particular for Revision 0 files. [Worked examples here](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/Examples/WorkedExamples/SeismicLoadingManifests/README.md#seg-y-header-template).
|
|
|
* **_Schema Upgrade Support_**: In preparation for the Schema Upgrade Service, Data Definitions provided the resources in form of reference data records (as manifest) for
|
|
|
* **_Schema Upgrade Support_**: In preparation for the Schema Upgrade Service, Data Definitions provided the resources in form of reference data records (as manifest) for
|
|
|
* [SchemaUpgradeSpecification:1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/SchemaUpgradeSpecification.1.0.0.md) containing JOLT specifications, which can be augmented by platform operators. [Worked examples here](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/Examples/WorkedExamples/Upgrades/README.md#schema-upgrades).
|
|
|
* [SchemaUpgradeChain:1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/SchemaUpgradeChain.1.0.0.md) declaring upgrade candidates for target schema versions. [Worked examples here](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/Examples/WorkedExamples/Upgrades/README.md#example-chain-for-wellboremarkerset120).
|
|
|
* [ReferenceValueUpgradeLookUp:1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/ReferenceValueUpgradeLookUp.1.0.0.md) - a similar reference data structure for to supersede deprecated reference values. Also here, platform owners can control **_what_** reference value upgrades are executed (provided the Schema Upgrade Service delivers the capability). [Worked examples here](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/Examples/WorkedExamples/Upgrades/README.md#reference-value-upgrades).
|
|
|
* **Incremental Entity Versions**
|
|
|
* **_Connected Data Sources_**: [ConnectedSourceDataJob:1.2.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/master-data/ConnectedSourceDataJob.1.2.0.md) with [additional properties](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/Guides/MigrationGuides/M15/ConnectedSourceDataJob.1.1.0.md).
|
|
|
* "CreateTimeMax" a new attribute - to control the incremental fetch records from the data provider
|
|
|
* "FailedRecords" existing attribute, but a change in the format
|
|
|
* "FailedRecords" existing attribute, but a change in the format
|
|
|
* “OnIngestionSchemaAuthority” to control "kind" of created record from fetch-and-ingest at target (Operator)
|
|
|
* **_Well Delivery_**
|
|
|
* [OperationsReport:1.2.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/master-data/OperationsReport.1.2.0.md) correcting some property types, [see details](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/Guides/MigrationGuides/M15/OperationsReport.1.1.0.md).
|
... | ... | @@ -33,7 +33,7 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha |
|
|
* [Wellbore:1.2.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/master-data/Wellbore.1.2.0.md) now carries a `LaheeClassID` property superseding the `DrillingReasonTypeID`. PPDM delivered the [reference values](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/LaheeClass.1.0.0.md#reference-values-for-laheeclass100)
|
|
|
* Facilities now offer `Remark` strings for facility events, operators and states. This causes minor, incremental versions for Rig:1.1.0, StorageFacility:1.1.0, Well:1.2.0 and Wellbore:1.2.0.
|
|
|
* **_Seismic Domain_**:
|
|
|
* AbstractBinGrid has been revised to disambiguate the usage. It is now recommended to associate the inline/crossline numbers and ABCD labels with the ABCDBinGridSpatialLocation (using the [AbstractAnyCrsFeatureCollection:1.1.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/abstract/AbstractAnyCrsFeatureCollection.1.1.0.md) structure with a declared schema fragment for the `properties` in [AbstractGeoJson.PropertiesBinGridCorners](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/abstract/AbstractGeoJson.PropertiesBinGridCorners.1.0.0.md)). This is linked to the [CRS Converter convertBinGrid](https://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service/-/issues/63) enhancement.
|
|
|
* AbstractBinGrid has been revised to disambiguate the usage. It is now recommended to associate the inline/crossline numbers and ABCD labels with the ABCDBinGridSpatialLocation (using the [AbstractAnyCrsFeatureCollection:1.1.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/abstract/AbstractAnyCrsFeatureCollection.1.1.0.md) structure with a declared schema fragment for the `properties` in [AbstractGeoJson.PropertiesBinGridCorners](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/abstract/AbstractGeoJson.PropertiesBinGridCorners.1.0.0.md)). This is linked to the [CRS Converter convertBinGrid](https://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service/-/issues/63 "Add convertBinGrid endpoint") enhancement.
|
|
|
* **PPDM Reference Value Updates**
|
|
|
* [BitType:1.0.1](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/BitType.1.0.1.md) one more superseded definition.
|
|
|
* [SeismicEnergySourceType:1.0.1](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/SeismicEnergySourceType.1.0.1.md) now with PPDM governance.
|
... | ... | @@ -45,50 +45,140 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha |
|
|
* [LaheeClass:1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/LaheeClass.1.0.0.md)
|
|
|
* **Corrections**
|
|
|
* The content of TubularAssemblyType was by accident replicated in TubularAssenblyStatusType. The content has been corrected for [TubularAssemblyStatusType:1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/reference-data/TubularAssemblyStatusType.1.0.0.md).
|
|
|
|
|
|
|
|
|
### Core and Helper Services
|
|
|
|
|
|
### DDMS
|
|
|
- `Wellbore DDMS`
|
|
|
- Support of Well 1.2.0 and [Wellbore 1.2.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/master-data/Wellbore.1.2.0.md), see [!#656](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/656)
|
|
|
|
|
|
- `Wellbore DDMS`
|
|
|
- Support of Well 1.2.0 and [Wellbore 1.2.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/v0.18.1/E-R/master-data/Wellbore.1.2.0.md), see [!#656](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/656 "Add support for Well 1.2.0 and Wellbore 1.2.0")
|
|
|
- `Reservoir DMS`
|
|
|
- REST API capability to create manifest information from RDMS content at the end of collaboration workflow, allows to provide outside information such as ACL, legal tags and existing dataset reference
|
|
|
- Server capability to clone dataspace
|
|
|
|
|
|
### External Data Services (EDS)
|
|
|
- Enable the end-user to configure [schema authority variable](Enable the end-user to configure schema authority variable at the Data Operator side) at the Operator end
|
|
|
- Handle EDS [incremental fetch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/external-data-framework/-/issues/252) scenario
|
|
|
- Provide [default date and time](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/external-data-framework/-/issues/253) to LastSuccessfulRunDateUTC in ConnectedSourceDataRegistry
|
|
|
- [EDS DMS](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/external-data-framework/-/issues/52) - Options for Consumer to manage bulk data files (ex: LAS, DLIS, SegY) from Data Provider
|
|
|
|
|
|
- Enable the end-user to configure [schema authority variable](Enable%20the%20end-user%20to%20configure%20schema%20authority%20variable%20at%20the%20Data%20Operator%20side) at the Operator end
|
|
|
- Handle EDS [incremental fetch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/external-data-framework/-/issues/252 "Handle EDS incremental fetch scenario") scenario
|
|
|
- Provide [default date and time](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/external-data-framework/-/issues/253 "Provide default date and time to LastSuccessfulRunDateUTC and CreateTimeMax in ConnectedSourceDataJob (CSDJ))") to LastSuccessfulRunDateUTC in ConnectedSourceDataRegistry
|
|
|
- [EDS DMS](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/external-data-framework/-/issues/52 "EDS DMS - Options for Consumer to manage bulk data files (ex: LAS, DLIS, SegY) from Data Provider") - Options for Consumer to manage bulk data files (ex: LAS, DLIS, SegY) from Data Provider
|
|
|
|
|
|
### GCZ (Geospatial Consumption Zone)
|
|
|
|
|
|
#### New Features
|
|
|
|
|
|
* **Ingestion**
|
|
|
* Transformer now supports limited Ingestion of `WellLog` records via ID-to-ID spatialization
|
|
|
* Example: Store Well Logs to Ignite **as Well Surface Locations only** with Well Log attributes - then view on a map
|
|
|
* Supports any `kind` where Search results lack geometry but contain clear references to another geometry-containing `kind`
|
|
|
* See our [Kind Spatial Source](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/blob/master/osdu_gcz_wiki/deployment/pre_deployment_config.md#kind-spatial-source) docs for more info
|
|
|
* Now supports geometry ingestion from the `SpatialArea.Wgs84Coordinates` property in addition to the already-supported `SpatialLocation.Wgs84Coordinates` property.
|
|
|
* **Persistence**
|
|
|
* Remote Profile is now compatible with Persistence-Enabled Ignite Clusters
|
|
|
* With [Persistence Enabled](https://ignite.apache.org/docs/latest/persistence/native-persistence) on your Ignite Cluster and your [Transformer configured](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/blob/master/osdu_gcz_wiki/deployment/pre_deployment_config.md#enabling-persistence-within-ignite-remote-profile-only) appropriately, utilize Ignite's on-disk storage capabilities.
|
|
|
* **Testing**
|
|
|
* Three demo [web apps](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/tree/master/osdu_gcz_test_assets/webapps) have been added to the new [osdu_gcz_test_assets](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/tree/master/osdu_gcz_test_assets) directory for testing purposes
|
|
|
* Easily observe data served from the Provider on a map using three different workflows:
|
|
|
* `esri_jsapi_test.html` - ArcGIS JavaScript API + Provider Endpoint
|
|
|
* `leaflet_esri_test.html` - Open Source Leaflet JS Library + Esri Plugin + Provider Endpoint
|
|
|
* `leaflet_geojson_test.html` - Open Source Leaflet JS Library + Provider Endpoint using GeoJSON spec output
|
|
|
* Just plugin your Provider Feature Layer Endpoints where indicated in each file before opening in the browser
|
|
|
* Unit Tests refactored to reduce redundant code duplication
|
|
|
* Special thanks to Xiangliang Meng from AWS for authoring those refactors
|
|
|
* New Provider endpoint created to expose `koop-config.json` to help users understand how the GCZ has configured its layer definitions - try it out in our updated Postman Collection in the Provider Metadata folder - "Get Layer Definitions" request
|
|
|
* `{{PROVIDER_URL}}/FeatureServer/layers/info`
|
|
|
* [Postman Collection](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/blob/master/osdu_gcz_test_assets/README.md#postman) has been reworked
|
|
|
* Uses collection variables for easy plugin-and-test workflow
|
|
|
* Simply plugin `PROVIDER_URL` and `AMBASSADOR_URL`
|
|
|
* Now covers more test cases and more request parameters
|
|
|
* Includes examples on Ambassador endpoints and the new layer definitions endpoint
|
|
|
* **Environment Support**
|
|
|
* Now supports a Kubernetes Profile for deploying/running GCZ components within a Kubernetes Cluster
|
|
|
* See updated documentation for the breakdown on [configuring](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/blob/master/osdu_gcz_wiki/deployment/pre_deployment_config.md#k8s-remote-profile) and [running](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/blob/master/osdu_gcz_wiki/deployment/deployment_procedure.md#k8s-remote-profile) the K8S Remote Profile.
|
|
|
* Special thanks to the folks at IBM for building out this solution: Bhushan Rade, Shrikant Garg, and Anuj Gupta
|
|
|
* New deployment documentation added specifically for AWS integration. [Find here](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/tree/master/osdu_gcz_wiki/deployment/aws).
|
|
|
* Special thanks to Morris Estepa from AWS for authoring the AWS Deployment Docs
|
|
|
* **Ambassador**
|
|
|
* Admin endpoint now supports manual cache updates via the `/gcz/transformer/admin/updateCache` endpoint
|
|
|
* Before, cache updates could only occur on the predefined schedule. Now, cache updates can be initiated at any time via the endpoint.
|
|
|
* _Auth support for admin endpoint is in-development_
|
|
|
* **Build**
|
|
|
* `mvn clean install` now skips unit tests by default. No need to run `mvn clean install -dSkipTests=true` for a typical build.
|
|
|
* Build docs have been updated to reflect this change
|
|
|
* **Auth**
|
|
|
* `clientID` and `clientSecret` can now be referenced via environmental variables in the `application` YAML files. No additional configuration required, simply input the name of the environmental variable for each respective property, and the Transformer will automatically scan the environment for that variable. Otherwise, the Transformer will read the values explicitly:
|
|
|
|
|
|
```yaml
|
|
|
clientID: GCZ_CLIENT_ID_ENV_VARIABLE
|
|
|
clientSecret: GCZ_CLIENT_SECRET_ENV_VARIABLE
|
|
|
```
|
|
|
* **Docs**
|
|
|
* Updated with new info on:
|
|
|
* Kubernetes Deployment
|
|
|
* Ambassador Endpoints (`cacheSize` and `updateCache`)
|
|
|
* Provider layer query parameters (`f` for specifying `geojson` spec output)
|
|
|
* How to leverage YAML configuration files that live outside of the project directory
|
|
|
* Webapp demos
|
|
|
* And more...
|
|
|
* `koop-config.json` - The default fields for our Seismic 3D layer have been expanded from 1 field to 9 fields
|
|
|
* **Logs**
|
|
|
* Ingestion stats now summarized per-`kind` / per-day, as `dataIngestionError.log` files within the target/logs directory. Example:
|
|
|
|
|
|
```log
|
|
|
2022-12-13 15:52:54.537 - Cannot create feature with id: `osdu:master-data--SeismicAcquisitionSurvey:ZMTest' from kind: 'osdu:wks:master-data--SeismicAcquisitionSurvey:1.0.0'
|
|
|
2022-12-13 15:52:54.540 - Cannot create feature with id: `osdu:master-data--SeismicAcquisitionSurvey:ST0202R08' from kind: 'osdu:wks:master-data--SeismicAcquisitionSurvey:1.0.0'
|
|
|
2022-12-13 15:52:54.541 - Could not process 2 features from kind: 'osdu:wks:master-data--SeismicAcquisitionSurvey:1.0.0' due to invalid geometries
|
|
|
2022-12-13 15:52:54.778 -
|
|
|
|
|
|
######## Local profile - Cache Load Summary for kind osdu:wks:master-data--SeismicAcquisitionSurvey:1.0.0 ########
|
|
|
5 records for kind osdu:wks:master-data--SeismicAcquisitionSurvey:1.0.0 found. 3 succeeded. 2 had errors.
|
|
|
########################################################################################
|
|
|
|
|
|
2022-12-13 15:52:55.708 -
|
|
|
|
|
|
######## Local profile - Cache Load Summary for kind osdu:wks:work-product-component--SeismicLineGeometry:1.0.0 ########
|
|
|
4 records for kind osdu:wks:work-product-component--SeismicLineGeometry:1.0.0 found. 4 succeeded. 0 had errors.
|
|
|
########################################################################################
|
|
|
```
|
|
|
* App logs generally improved for clarity
|
|
|
* **Miscellaneous**
|
|
|
* Provider
|
|
|
* The configuration property `igniteTable` has been omitted from both the documentation and our example config file as it is no longer necessary
|
|
|
* Transformer
|
|
|
* Class names have been updated in the code to give more clarity to applicable profiles:
|
|
|
* New class names: `FeatureCacheLocalSynchronizer`, `FeatureCacheRemoteSynchronizer`
|
|
|
|
|
|
## Bug Fixes
|
|
|
|
|
|
### Core and Helper Services
|
|
|
|
|
|
### Ingestion services
|
|
|
- Manifest-based Ingestion - Major performance improvement.
|
|
|
- osdu-ingestion-lib - writing records in parallel/concurrent manner into Storage service. See [MR](https://urldefense.com/v3/__https:/community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/30__;!!GF_29dbcQIUBPA!3PiTTmCUIwwQ6UToa6PfFOgrrAkdvsHADXCXk_QEpa6sG5oPUV6xrdRcoDu2jKjk-_ar4V9ssumqtnn9kSfqrMapqwDrFqIJlts$)
|
|
|
- osdu-airflow-lib - Added new Airflow variable core__ingestion__thread_save_number to tune a number of concurrent writings into Storage service for ProcessManifestOperatorR3 operator. This variable is optional (default: 1). See [MR](https://urldefense.com/v3/__https:/community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/39__;!!GF_29dbcQIUBPA!3PiTTmCUIwwQ6UToa6PfFOgrrAkdvsHADXCXk_QEpa6sG5oPUV6xrdRcoDu2jKjk-_ar4V9ssumqtnn9kSfqrMapqwDr98gQipI$)
|
|
|
|
|
|
- Manifest-based Ingestion - Major performance improvement.
|
|
|
- osdu-ingestion-lib - writing records in parallel/concurrent manner into Storage service. See [MR](https://urldefense.com/v3/\__https:/community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/30_\_;!!GF_29dbcQIUBPA!3PiTTmCUIwwQ6UToa6PfFOgrrAkdvsHADXCXk_QEpa6sG5oPUV6xrdRcoDu2jKjk-\_ar4V9ssumqtnn9kSfqrMapqwDrFqIJlts%24)
|
|
|
- osdu-airflow-lib - Added new Airflow variable core__ingestion__thread_save_number to tune a number of concurrent writings into Storage service for ProcessManifestOperatorR3 operator. This variable is optional (default: 1). See [MR](https://urldefense.com/v3/\__https:/community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/39_\_;!!GF_29dbcQIUBPA!3PiTTmCUIwwQ6UToa6PfFOgrrAkdvsHADXCXk_QEpa6sG5oPUV6xrdRcoDu2jKjk-\_ar4V9ssumqtnn9kSfqrMapqwDr98gQipI%24)
|
|
|
|
|
|
### DDMS
|
|
|
|
|
|
- `Wellbore DDMS`
|
|
|
- lib updates: [fastAPI 0.70](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/659), [Dask 2022.10.2](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/612), [ffspec and implementations (adfs, s3fs, gcfs ...)](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/612), [PyArrow 9.0.0](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/649).
|
|
|
- Azure lib: retry mechanism to work around ClientAuthentication issues to blob storage (see [changes](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/lib/wellbore-cloud/wellbore-azure-lib/-/merge_requests/36), imported by [!659](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/626)).
|
|
|
|
|
|
- lib updates: [fastAPI 0.70](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/659 "upgrade fastapi 0.70"), [Dask 2022.10.2](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/612 "Upgrade Fsspec and its dependencies: aws, azure, gcp + Dask to LATEST version"), [ffspec and implementations (adfs, s3fs, gcfs ...)](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/612 "Upgrade Fsspec and its dependencies: aws, azure, gcp + Dask to LATEST version"), [PyArrow 9.0.0](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/649 "Enh: upgrade PyArrow library to version 9.0.0 due to lots of improvement on memory consumption side").
|
|
|
- Azure lib: retry mechanism to work around ClientAuthentication issues to blob storage (see [changes](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/lib/wellbore-cloud/wellbore-azure-lib/-/merge_requests/36 "Add new RetryableDefaultAzureIdentity class to authenticate to Azure services"), imported by [!659](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/626 "Change version of OSDU core lib Azure to change the way credentials are...")).
|
|
|
- `Reservoir DMS`
|
|
|
- Fixes around entitlement and group management
|
|
|
|
|
|
### External Data Services (EDS)
|
|
|
- [EDS-Ingest](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/3) - show in the Airflow log the exact body of the query being executed to bring data from the source
|
|
|
- Fetch query does not respect [wildcard syntax](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/2)
|
|
|
- Corrected Airflow [Xcom summary](https://urldefense.proofpoint.com/v2/url?u=https-3A__community.opengroup.org_osdu_platform_data-2Dflow_ingestion_external-2Ddata-2Dsources_core-2Dexternal-2Ddata-2Dworkflow_-2D_issues_7&d=DwMFAg&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=xyT-FeJSWBkH8qkgDyqzDmlFQPgSEWMh95lIMNoshfd14TcRm-PyeYRL64uc41GW&m=KvoovV3EbgucLHzgeCsGkR-0DkpYfDtyYqSLoXcqJw9_R7N1URARutOp7ecB31UZ&s=s_NGtT1cUISSbui_vSvYbjCZtyE43_j61oA-MKLqX7g&e=) from Ingest task
|
|
|
|
|
|
- [EDS-Ingest](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/3 "EDS-Ingest - show in Airflow log exact body of query being executed to bring data from source") - show in the Airflow log the exact body of the query being executed to bring data from the source
|
|
|
- Fetch query does not respect [wildcard syntax](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/2 "EDS Preship test - fetch query does not respect wildcard syntax")
|
|
|
- Corrected Airflow [Xcom summary](https://urldefense.proofpoint.com/v2/url?u=https-3A__community.opengroup.org_osdu_platform_data-2Dflow_ingestion_external-2Ddata-2Dsources_core-2Dexternal-2Ddata-2Dworkflow\_-2D_issues_7&d=DwMFAg&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=xyT-FeJSWBkH8qkgDyqzDmlFQPgSEWMh95lIMNoshfd14TcRm-PyeYRL64uc41GW&m=KvoovV3EbgucLHzgeCsGkR-0DkpYfDtyYqSLoXcqJw9_R7N1URARutOp7ecB31UZ&s=s_NGtT1cUISSbui_vSvYbjCZtyE43_j61oA-MKLqX7g&e=) from Ingest task
|
|
|
|
|
|
#### GCZ (Geospatial Consumption Zone)
|
|
|
|
|
|
* Transformer
|
|
|
* Fix `batchSize` config option not respected when below `1000`
|
|
|
* Provider
|
|
|
* Fix Provider crashing that occurred when spatial intersections were attempted on a Feature Layer using an `esriGeometryPoint` or invalid geometry
|
|
|
* Fix Provider not responding to manual spatial queries when using WKT representation of geometry
|
|
|
|
|
|
## :exclamation: Breaking Changes
|
|
|
|
... | ... | @@ -97,8 +187,27 @@ No in-place schema changes are required in M15 (no M14 schema resources were cha |
|
|
- `Wellbore DDMS`
|
|
|
- API `POST /ddms/v3/records/delete` is deprecated, see [OpenAPI spec](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/release/0.18/spec/generated/openapi.json). Consumers should use either the typed delete APIs from WDMS (e.g. `DELETE /ddms/v3/wells/{wellid}`) or the core storage service delete API.
|
|
|
|
|
|
### Known Issues/Gaps
|
|
|
#### GCZ (Geospatial Consumption Zone)
|
|
|
|
|
|
* Configuration file name change
|
|
|
* `application.yml` has been renamed to `application-local.yml`
|
|
|
* Ignite Cache Name
|
|
|
* We now require a separate identifying string for the name of a cache, as configured in either `application-local.yml`, `application-remote.yml`, or `application-k8s-remote.yml`
|
|
|
* Each kind previously needed to contain the `name` and `chronExpr` properties, but now must contain a `cacheName` property:
|
|
|
|
|
|
<pre>
|
|
|
|
|
|
<span dir="">`- name: osdu:wks:master-data--Well:1.0.0 cacheName: osdu_wks_master-data--Well_1.0.0 chronExpr: 0 0 8 ?`</span>
|
|
|
|
|
|
</pre>* The reason for this change is due to our new support of Persistence-Enabled Ignite Clusters, which strictly impose that cache names do not contain special characters such as `:` due to conflicts with file storage. Please note that taking the kind name and simply replacing all `:` characters with an underscore `\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\_` should work fine, but you are welcome to name your caches however you like, avoiding special characters.
|
|
|
* Additionally, you will need to configure `koop-config.json` to include the `cacheName` property or else the Provider will be unable to find the cache by `kind` entity alone.
|
|
|
|
|
|
## Known Issues/Gaps
|
|
|
|
|
|
#### GCZ (Geospatial Consumption Zone)
|
|
|
|
|
|
* WellLog support / Spatialization will only pull geometries from the `SpatialArea.Wgs84Coordinates` or `SpatialLocation.Wgs84Coordinates` properties from a record gathered via the Search API. By default, this is typically where Well records store their surface locations, and as such, Well Logs can only currently be displayed via the Surface Location of a Well, and **not** via the bottomhole or midpoint location.
|
|
|
|
|
|
## Supporting Information
|
|
|
|
|
|
## OSDU Community utilities |
|
|
## OSDU Community utilities |
|
|
\ No newline at end of file |