ADR - Project & Workflow Services - Core Services Integration - Copy Record references between namespaces

This ADR focuses on copying record references between namespaces

Status

• Proposed
• Trialing
• Under review
• Approved
• Retired

Background

In our previous ADR, we looked at creating namespaces for storage records. This allowed us to separate out record changes in the system of record and system of engagement. However we do need to be able to move changes between the system of record and engagement.

This ADR builds on the previous one by proposing to add a new API into Storage service to enable this.

As outlined in the previous ADR each namespace holds the reference to the Record versions it holds for a that Record ID. The API will move the Record references held in a specific namespace into a provided target namespace.

Out of scope

For this ADR we are only looking at how we can copy references between namespaces.

We are not deciding on

• How conflicts will be handled when the destination namespace has a newer version that already exists.
• How collaborations will act on this or control this behavior or even what a collaboration entity looks like

Solution

The below diagram shows how the reference system works for a specific Record ID that exists both in a collaboration and the system of record.

The metadata object of the Record holds the reference to the specific versions that exist to that context. So even though other versions may exist in different contexts they are not accessible outside the context(s) referenced by it.

This new API will take a source context, target context, Record ID and Record version. It will attempt to copy the given reference from the source into the target.

For example in the above diagram, I could request to copy the V4 data object from collaboration 1 into the system of Record metadata (promoted) using the following API call

curl -X 'PUT' \
  '<osdu>/api/storage/v2/records/copy' \
  -header  'data-partition-id: opendes' \
  --header 'authorization: Bearer <JWT>' \
  --header  'Content-Type: application/json' \
  --header  'x-collaboration: id=<source-collaboration-id>,application=<app-name>;' \
  --data '{ 
             "target": ""
             "records": [{ 
                 "id": "<record-id>"
                 "version": "<record-version>",
             }
         ]}'\
  -- data-raw

Finally you can copy data from the system of record to a specific collaboration by not providing an x-collaboration id directive value and supplying a target.

curl -X 'PUT' \
  '<osdu>/api/storage/v2/records/copy' \
  -header  'data-partition-id: opendes' \
  --header 'authorization: Bearer <JWT>' \
  --header  'Content-Type: application/json' \
  --header  'x-collaboration: application=<app-name>;' \
  --data '{ 
             "target": "<target-collaboration-id>"
             "records": [{ 
                 "id": "<record-id>"
                 "version": "<record-version>",
             }
         ]}'\
  -- data-raw

The new API should require the services.storage.admin permission to run as it is a privileged operation to move data between contexts.

Here is the full API specification of the new API.

  /records/copy:
    put:
      tags:
        - Records
      summary: Copy a Record reference form one namespace to another
      description: This API attempts to copy all the Record references it is provided from the given source namespace to the target namespace. All refences will be copied or all will fail as a transaction. IF the target namesapce does not et exist it will be created. It requires 'services.storage.admin' permission to call
      operationId: copyReference
      requestBody:
        description: The references to copy
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/copyReferences'
        required: true
      responses:
        '200':
          description: Successful operation
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/copyReferences'          
        '400':
          description: Invalid Record IDs provided
        '404':
          description: Records not found
        '409':
          description: One or more references already exist in the target namespace

Context & Scope

Decision

Rationale

Consequences

We will have a new API where the data access layer needs to be implemented by every CSP.

The hard delete API in storage service needs to add extra validation that the data blob being deleted from storage is not referenced in a different context. With this API the same version of data can be referenced in multiple contexts meaning a purge could delete a blob referenced in a different context and leave a dangling reference without this extra check.

When to revisit

---

Tradeoff Analysis - Input to decision

Alternatives and implications

Decision criteria and tradeoffs

Decision timeline