ADR: API to purge a batch of storage records
New API in Storage service to purge a batch of records
Status
-
Proposed -
Trialing -
Under review -
Approved -
Retired
Context & Scope
The OSDU Storage service provides 2 ways to delete a record. One way is to logically delete the record in which the record with same id can be revived later because its version history is maintained.The other way is to permanently delete the record (called as purging) in which case, the record's version history is deleted too. This operation cannot be undone meaning records purged cannot be revived.
In both types of deletions, the record content cannot be accessed using storage or search service.
The storage service provides separate APIs for logical deletion (POST /records/{id}:delete
) and purging of records (DELETE /records/{id}
).
The storage service provides API for logical deletion of batch of records (POST /records/delete
), but such an API is not available for purging of records.
The proposal is to provide an API on storage service to support purging of batch of records, where the maximum batch of size 500 will be supported. Only the record IDs passed in the request body will be deleted not including any linked records or files if they exist. Cleaning up of all the linked records, such as child records, records in relationship block, and actual data (files ingested via workflow service), would not be in the scope of this API. It would be the user's responsibility.
The new bulk API will work on active as well as non-active (soft-deleted) records, similar to the existing purge API.
Purging of records can be performed by the owner of the records and the owner should be part of users.datalake.admins group.
The API response would be similar to the response of the logical deletion API that is POST /records/{id}:delete
In case of partial success, the response code would be 207 and the not-deleted-record-IDs would be listed in the response.
Tradeoff Analysis
In the absence of an API to purge a batch of records, users would have to call the DELETE API once for every record and it would increase the number of calls to the storage service.
Decision
Provide an admin-only API to purge a batch of records, with maximum batch size of 500 records. The Open API specs for storage service with new API is here:
storage_openapi_batchpurge.yaml
Consequences
- New API on Storage service would be available.
- Documentation of Storage service should be modified with details for the new API.