ADR - DELETE API in Dataset Service

We are often left to address the gaps from architectural principles (which stay at a pretty high and abstract level) to the actual implementation detail. Here is an attempt to bridge that gap by providing a set of Lightweight Architecture Decision Records (LADRs) which are simple to follow and can be implemented in a given team/project by the developers

Decision Title

Status

Context & Scope

Today, Dataset Service containes a set of APIs to allow for registering, storing, and retrieving datasets. However, the service does not include an API supporting DELETE operation.

SLB (as MS partner) has sponsored feature request to donate DELETE endpoint on behalf of their end customers.

This ADR is focused on the followings:

Operation is Delete
Service is Dataset Service
Schema type should be dataset--File.*
Using Community version

Proposed Requirements

This API should be supporting soft delete with purge.

Currently there is no API supporting the delete request from the user if the dataset:

is no longer needed
no longer owned by the user
taking too much space
has been added in error

Scenarios from ARM CSS workflows as shared by SLB:

As a consulting RE for a CCS project, at the end of each consulting project, I need to send the outcome of the projects (reports and data) to the client and delete the clients' data used for the project from my system. This would comprise the Petrel projects and the liberated data stored in OSDU during the project.
As the asset manager in company A for field X that has been sold to company B, I need to permanently delete all our date related to that field from our systems including cloud storage
As the data manager for company A I need to control my data storage costs. I regularly review the simulation data held in our cloud system of record against company data retention policies and I archive data that needs to be kept but is not in current use. I destroy data that is not in compliance with company data retention policies.
As the asset manager for field X I need to control my departmental storage costs to help keep within my cloud IT budget. At the end of each project, I review the data generated during the project studies. I delete data (simulation cases and ensembles) that are not needed for the next phase of the project and that can be regenerated if necessary later from the input data and Petrel projects that I retain.

Decision

Enabling users to leverage Delete API in the Dataset Service to remove unwanted Data like:

is no longer needed
no longer owned by the user
taking too much space
has been added in error

Rationale

Customers have been using the DELETE operation in the File Service. File service is set to deprecate in the near future. Customers need to migrate to the Dataset Service which does not have the DELETE operation.

From M21 release notes, we are now actively engaged in the process of integrating the functionalities of the File and Dataset services. It is important to note that, going forward, our primary focus will be on enhancing and prioritizing the capabilities of the Dataset service in the domain of file/dataset delivery. As of the present, both services remain accessible and operational.

Consequences

To ensure:

The integrity of the data, it is recommended that the customer restrict access to the DELETE API to prevent any inadvertent deletion.
The consistency of the data, it is imperative that the customer properly manage data deletion at the application layer to prevent any orphan references.

Decision timeline

End of Q2 CY24.

Edited Mar 04, 2024 by Mahsa Hanifi