ADR: Creation of Rock and Fluid Sample DDMS
ADR: Creation of Rock and Fluid Sample DDMS
Creation of Rock and Fluid Sample DDMS to support storage and analysis of rock and fluid sample data.
Status
-
Proposed -
Trialing -
Under review -
Approved -
Retired
Context & Scope
Rock and fluid samples are physical specimens extracted either directly from the subsurface or from collection points at the surface. These samples can be associated with a wellbore, but sometimes are not tied to a specific wellbore. Information and analysis of these samples are used in static reservoir model, dynamic reservoir modeling, facility design, flowline design, drilling design, and more.
Personas that typically use rock and fluid samples can include:
- Data Managers/Stewards
- Petrophysicists
- Petrologists
- Petrographers
- Geochemists
- Geologists
- Reservoir Modelers
- Reservoir Engineers
- Flow Assurance Engineers
- Facilities Engineers
- Lab Inventory Professionals
- Lab Analysis Professionals
Typically, the lifecycle of sample data follows this general flow: a) putting together a plan to collect samples, b) working with vendors to collect samples, c) samples are shipped to labs, d) samples can be analyzed at one or more labs, and e) placed in long term storage, as shown in Figure 1 below.
Figure 1. Example of Rock and Fluid Sample Business Process Flow
Like most subsurface data, analyze of the samples is a highly iterative process and multiple data sets, and versions of, are generated. Storage of this data is highly variable as there is not an industry standard storage format. Data can be stored in a multitude of file formats with internal structures varying from vendor to vendor.
In order to analyze and best govern fluid sample data across this diverse data storage, there is a need for consistent data definitions, data structure, and a common abstractions (e.g., APIs). Otherwise, analyzing this dataset is a highly burdensome process requiring the data consumers to exert a high degree of manual effort in formatting and prepping the data for analysis.
Decision
Enhance the OSDU Platform to provide a standard optimize method to store and consume Rock and Fluid Sample data and in particular support RCA and PVT.
And in doing so, creating a new DDMS in lieu of enhancing an existing DDMS.
The initial release of this DDMS aims to satisfy these use cases:
- Provide optimized bulk content storage:
- Enable data consumers to store bulk data extracted from reports in a way that is consistent and easy to access later
- DDMS supporting this by have endpoints for bulk data ingestion/storage to well-defined data DDMS data models
- Support query, filter, and deliver bulk data within one dataset:
- Enable data consumers to find wells that have a specific rock and fluid attribute (E.g., permeability values > 18 mD in RCA) for a specific country
- DDMS supporting this by:
- Supporting query by any catalog data/metadata (e.g. geocontexts, lab vendor, dates, experimental method, technical assurance, etc.) or DDMS bulk data
- Supporting filtering for value ranges and other criteria (e.g., depths, pressures, temperatures, etc.)
- Enable trace bulk data back to OSDU Catalog records:
- Enable data consumers to find the original RCA report after working with bulk data
- DDMS supporting this by having endpoints that allow search against the Core Search service, for WKSs relevant to Rock and Fluid data, linked to DDMS Datasets
Rationale
A DDMS does not exist today that provides optimize storage and consumption of rock and fluid sample data. This particular data set does not neatly fit into an existing DDMS (e.g., wellbore, reservoir, seismic, etc. ). It has a close affinity to the Wellbore Data Domain and therefore Wellbore DDMS, but the data is not always associated with and can exist without a wellbore. For that reason and using best practices of domain driven design, it was decided to create a new DDMS that can support an array of optimized storage and consumption specific to the Rock and Fluid Sample data domain.
Note, the Data Definition Data Domain teams are currently aligning OSDU Data Definitions by Data Domains. The decision to leverage domain drive design is in line with this effort. Current proposal can be found here.
Consequences
Rock and Fluid Sample DDMS would be added as an experimental feature of the platform. The initial MVP would provide capabilities to store and consume RCA and PVT data. This would be both the metadata records associated with those data types (e.g. Coring, RockSample, RockSampleAnalysis, etc.) and the content. For the metadata records, we will utilize the core Storage Service, and not store separate instances. However, validation will occur before storing the JSON to the core storage service. Content will initially be stored in a single parquet file per analysis and will utilize the core file service for storage/management of the file.
The initial donation will only include an Azure SPI implementation. The consequence is that we'll need to work with the other CSPs post-donation on the creation of the other SPI layers.
No content parsers will be provided as part of the donation. This is due to both lack of an industry standard of sample analysis report content and the commercial nature of the products in this space (likely ML will need to be applied to extract the data from PDFs and other documents).
When to revisit
Future considerations to revisit this DDMS when:
- DDMS architecture fundamentally shifts approach away from being domain-driven
- OSDU community aligns to refactor scope of this DDMS with another yet to be built DDMS
Tradeoff Analysis - Input to decision
Alternatives and implications
The main alternative considered was to add rock and fluid sample endpoints to the Wellbore DDMS, but as has been stated above, not all samples are tied directly to wellbores (e.g. sea surface captured fluid samples). The implication of having this be a separate service is there is potentially some duplication in how the service handles storage of the data (in the case of the initial MVP that is parquet).
Decision criteria and tradeoffs
The team evaluated ties to other domains covered by existing DDMSs, and we did not find a good match. Following the domain driven design approach by creating this as a separate DDMS allows it to evolve to meet the needs of the consumers of rock and fluid samples and the unique needs that domain would have. We expect this will evolve quite a bit as we expand the service to accommodate more analytics use cases that span multiple sample reports. The benefit of allowing it to independently evolve would outweigh the potential code duplication elimination that would come from shoehorning this domain into the Wellbore DDMS.
Architecturally, the DDMS has been designed to conform with the 11 OSDU DDMS Key Principles and design best practices:
- https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-(C)/Design-and-Implementation/Domain-&-Data-Management-Services
- https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-(C)/Reference-Architecture/Functional-Architecture/Data
- https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/home
Specific to this DDMS, these considerations were factored into the design and architecture:
- Domain-driven API
- Contextual boundaries
- Microservices-based
- Modular
- Vendor & technology agnostic
- Balanced between analytic performance and cloud resource cost
- Flexible component composition for specific use cases and requirements
Figure 2. Target DDMS Architecture
Decision timeline
A prototype was developed for the Rock and Fluid Sample DDMS in Q1 2023 by ExxonMobil and EPAM. We are trying to target the inclusion of an MVP of the RAFS DDMS in either M18 or M19.