Milestone 8 - September 30, 2021

For detailed services that are included in this release, please refer to M8 tagging notes:

• R3 Milestone 8 deliverables - 0.11.0

New Features

• Data Definition
  • None major for this milestone. See some information in "Deprecated" section below about "Collection" entity.
• Core Services
  • Storage
    • SpatialLocation supports WGS84 now. More information
  • Search
    • Audit Trail attributes (ex: createUser, createTime, modifyUser, modifyTime) are now available from Search by enabling these for Index. More information
  • Version info. endpoint is now available for all the core services. More information
• Python SDK
  • Performance improvement to send requests to Search services by batches
  • Base Client is refactored based on this ADR which has the implementation here
  • SDK Configuration can be done now via python-dotenv without using open_api.ini. More information
• Workflow Services
  • Airflow 2.0 backward compatibility introduces possibility to use the same Manifest Based Ingestion code base both for Airflow >=1.10.10 and Airflow 2.0. More information
  • Started movement of OSDU airflow common code to https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib project.
• Data Ingestion
  • Manifest Based Ingestion
    • An implementation for splitting Large Manifest into smaller chunks has taken in place which allows parallel ingestion. More information
    • External Data Framework had a refactoring in validate_referential_integrity.py for accepting a whitelist of non-validated properties which allows a configuration-driven approach for excluding validation of those properties that match OSDU ID patterns. This is tested on GCP Community Env.
  • CSV Parser Enhancements
    • CSV ingestor is able to generate OSDU compliance records including Arrays and nested Arrays.
    • The support of OSDU Governance and Schema constructs plus Airflow 2.0 support is finalized
  • WITSML Parser
    • No new changes
  • Core Major improvement in Integration Testing coverage has been done to ensure the consistency of behavior across all the CSPs. More details can be found here and here. Tests are all pass for Azure,GCP and IBM.
• DDMS
  • Wellbore
    • Data Access API’s
      • Support OSDU entities: Well 1.0.0, Wellbore 1.0.0 , WellLog 1.1.0, WellboreTrajectory 1.1.0, WellboreMarkerSet 1.0.0
      • API’s to access bulk data (some CSPs only)
    • Log recognition service
  • Seismic
    • OpenVDS and OpenZGY format conversion features are now available on all CSPs.
    • DAGs There is an update for SEGY -> OpenVDS Parser to support Airflow 2+. This is a core logic update which impacted on CSPs implementations Read more
    • DAGs There is an update for SEGY -> OpenZgy Converter to support Airflow 2+. Added • support for AWS (fixed region env var)
    • CSP agnostic impersonation token
    • updated dataset list endpoint to return paginated results if limit and cursor query parameters are present
    • set the subproject and dataset creator using the subid/sub value from the payload jwt

Bug Fixes

• E&O
  • Wrong CURL command in Swagger of DELETE API is fixed. see the issue here
• Legal Service
  • Problem with Partition Service is fixed and you may see more details here
• Reference & Helper Service
  • CRS Conversions 5XX error is fixed upon receiving a bad request by adding a new logger. Read more here
  • Unit Logger is fixed which was causing 500 error in unit-service. See more in Unit-Service-Availability Fast Burn Alert issue.
• Seismic DDMS
  • misc. security vulnerabilities fixes
  • misc. performance / scalability improvements
  • misc. observability improvements
• Manifest Based Ingestion
  • The “frame of reference” handling issue - (Unit of Measure) by Manifest based Ingestion is fixed.

Breaking Changes

Deprecation Notice

• Storage Service Referring to Deprecation Notice in M7 Schema Management has been deprecated in this release.
• master-data--Collection will be deprecated (removed from the deployment, but no longer registered); the next snapshot will come with work-product-component--PersistedCollection instead. The next snapshot change report can be previewed here.

Known Issues

• E&O is an experimental feature, meant to be deployed in a dedicated test partition for evaluation and community feedback. It is not recommended to be enabled in production environments as there is a known impact in manifest-based Ingestion (integrity check)
• CSV Ingestion is not available in AWS.
• WITSML Parser is not available in Azure.
• WITSML Parser - works for Well, Wellbore Master data types. But has issues for work-product component types Log, Trajectory, Marker, Tubular.
• Wellbore DDMS Bulk Access API is not available in AWS.