ADR: Worker Service for Wellbore Bulk Data Access

Status

• Proposed
• Trialing
• Under review
• Approved
• Retired

Context & Scope

Currently, as of M16, Wellbore DDMS is experiencing performance challenges involving WellLogs operations with large bulk data (>1 Gb), especially on data reading. It was also observed that Wellbore DDMS requires a significant amount of memory in comparison to the amount of data manipulated to serve incoming requests. See issues #21 and #27.

Wellbore DDMS is composed of a general main service, which is responsible for handling both client facing API requests, and data access operations to underlying bulk data store. In turn, the bulk data management implementation in WDDMS is highly based on Dask.

For instance, for a large WellLog dataset stored in Wellbore DDMS, the associated data will not be be located in a individual parquet file, but rather distributed in several distinct parquet files. When a request to retrieve the bulk data associated to a specific subset of WellLog curves, including or not the optional reference range, is received, Dask is used to process all required parquet files, across which the queried data is stored, and extract the cropped data corresponding to the selected curves and range from the WellLog dataset. All operations in the described workflow are executed end to end in the same container for a given request.

Though the main service approach and Dask capabilities provide a simple and straighforward deployment, it was identified, from previous analysis, that such pairing poses considerable limitations on Wellbore DDMS performance and scalability capacity.

Trade-off Analysis

Standard Python framework already offers a good support for I/O bound operations (see asyncio), however, when it becomes more complex to deal with CPU bound operations and data transformation operations, Dask brings a first answer to that. For instance, when reading and writing large WellLog datasets, Dask provides a concise and straighforward implementation to reconciliate data from multiple parquet files.

Nevertheless, if Dask appears to be a good solution for heavy computation, in most WDDMS' supported scenarios of data queries/filters, Wellbore DDMS is primarily constraint by I/O operations rather than by data transformation operations. Additionally, Dask showed not to be efficient when handling several queries involving smaller amounts of data, as its minimum required memory footprint does not scale down based on the smaller volumes of data.

Dask cluster is implemented as a process based local cluster, which also brings several issues:

• Dask workers are internal to the pods and therefore cannot be shared with other WDDMS service instances.
• The scaling/resources request are indirectly done through WDDMS, not the Dask workers.
• Dask workers are actually process forks of WDDMS which leads to unnecessary memory usage even at startup or when idle.

Finally, we spotted several memory leaks within Dask and there are several memory managment related issues open in Dask's GitHub.

Decision

Dask remains a great tools but it does not fit the needs of WDDMS. Therefore Dask will be removed and replaced by a new dedicated service responsible for bulk data access only called wddms bulk data worker service.

wddms bulk data worker service will be specialized in bulk I/O and bulk data manipulation (transformation, filtering), while WDDMS main service will keep all domain knowledge/responsibility such as meta data manipulation or consistency rules for instance, but will delegate bulk data operations to the wddms bulk worker service.

wddms bulk worker service will not use Dask at all. This means the current bulk data acces layer in WDDMS will not be moved as-is into the new dedicated service but reworked and tailored to WDDMS specific needs.

The image below illustrates side by side how scaling and workload distribution occur in the current and the target designs. In the current implementation, an incoming request to retrieve a large amount of data will be limited to the Dask workers resources of a single WDDMS pod though Dask workers from other WDDMS instances might be available. In the target design, unlike the current architecture, all processing capacity of the wddms bulk workers instances is available to be used by any WDDMS instances. That arrangement unlocks a better scaling capability as it is done directly on bulk data workers upon needs.

Security Implications

In the current design, the authorization (ACL/policy) checks and the bulk data access operations in WDDMS are performed in the same service instance. Bulk data will only be served to valid users entitled to access the associated work product component record.

The changes proposed in this ADR separate the data access control layer, located in the main WDDMS service, from the bulk data access itself, located in the new wddm bulk worker service. See below, the changes in the communication patterns in the current vs target design diagrams.

Allowing users or other services to directly access wddms bulk woker service endpoint would permit bypassing the data access control checks in the main WDDMS service.

Therefore, with the new topology, additional deployment configuration settings will be required to preserve a compliant and secure data access control in WDDMS,

• wddms bulk woker service must not be accessible from the external network
• wddms bulk woker service will only accept requests from WDDMS main service instances

Current

Target