Skip to content

ADR: Additional attributes in Storage record create and update notification events

Additional attributes in Storage record create and update notification events

Status

  • Proposed
  • Trialing
  • Under review
  • Approved
  • Retired

Context & Scope

Information received from notification service on record create/update after recent ADR has:

3 fields like below in case of record create event

{
  "id":"opendes:Wellbore:id1",
  "kind":"opendes:wks:master-data--Wellbore:1.0.0",
  "op":"create"
}

and 4 fields like below in case of record update event whenever there is a kind change:

{
  "id":"opendes:Wellbore:id1",
  "kind":"opendes:wks:master-data--Wellbore:1.1.0",
  "op":"update",
  "previousVersionsKind": "opendes:wks:master-data--Wellbore:1.0.0"
}

However, storage service when creating or updating the records has more information than this and publishing it can be useful for applications who are consumers of these notification events. For example, specific blocks from a record like ACLs, legal tags and record tags can only be updated using storage service's PATCH API and in such cases record version is not changed. This information is known to storage service when processing the PATCH request and; if published; can be used by consumer applications to decide whether to process the record or not. However these details are not yet published in the current notification events.

Tradeoff Analysis

Consumer applications (like WKS service) can keep more information about the previously processed records, their ids, versions etc. and decide whether to process the record received in new event or not, but it will require a lot of book-keeping from the consumers side and instead the originator of the event i.e. storage service can provide this information.

Consumer applications (like WKS service) can read the latest version of the record and compare it with previously processed records and decide whether to process the record received in new event or not, but it will require reading every record from the event which will increase load on storage service. Instead if the storage service can provide this information as a part of record change events, a lot of processing at consumers side can be saved.

Decision

We can enhance current behavior of storage service and provide more information in the record change event. Downstream services must be updated to consume the updated events structure. Currently the record changed event is structured as below

{
  "id":"opendes:Wellbore:id1",
  "kind":"opendes:wks:master-data--Wellbore:1.0.0",
  "op":"create"
}

or

{
  "id":"opendes:Wellbore:id1",
  "kind":"opendes:wks:master-data--Wellbore:1.1.0",
  "op":"update",
  "previousVersionsKind": "opendes:wks:master-data--Wellbore:1.0.0"
}

We would like to publish below additional information when the record is updated i.e. when op is update.

  • New attribute recordBlocks: to show the names of the updated/added/deleted blocks from the record i.e. data, metadata

    acl, legal, tags and meta blocks in the record are considered as metadata blocks.

    The block name will be followed with + sign in case a block is present only in the updated version of the record.

    The block name will be followed with - sign in case a block is absent in the updated version of the record.

    The blocks which are updated in the record update event won't have any sign.

update event with new attributes in case of no version change in record.

In the below example, only metadata blocks like tags and legal in the record are updated.

{
    "id":"opendes:Wellbore:id1",
    "kind":"opendes:wks:master-data--Wellbore:1.0.0",
    "op":"update",
    "recordBlocks":"metadata"
}

update event with new attributes in case of version change in record without any schema change:

In the below example, data block is only updated in the record.

{
    "id":"opendes:Wellbore:id1",
    "kind":"opendes:wks:master-data--Wellbore:1.0.0",
    "op":"update",
    "recordBlocks":"data"
}

update event with new attributes in case of schema version minor upgrade with addition of block:

In the below example, data block is updated and meta block (which is one of the metadata blocks) is added in the new version of the record.

{
    "id":"opendes:Wellbore:id1",
    "kind":"opendes:wks:master-data--Wellbore:1.1.0",
    "op":"update",
    "recordBlocks":"data metadata+",
    "previousVersionsKind": "opendes:wks:master-data--Wellbore:1.0.0"
}

update event with new attributes in case of schema version major upgrade with deletion of block:

In the below example, data block is updated and meta block (which is one of the metadata blocks) is removed in the new version of the record.

{
    "id":"opendes:Wellbore:id1",
    "kind":"opendes:wks:master-data--Wellbore:2.0.0",
    "op":"update",
    "recordBlocks":"data metadata-",
    "previousVersionsKind": "opendes:wks:master-data--Wellbore:1.0.0"
}

PUT API behavioral changes

  • When user calls storage PUT API to update only tags, acl and/or legal blocks of an existing record, then the version of the record won't be incremented. This will make the PUT API behavior consistent with the PATCH API.

The consumer services of this event notification can efficiently process the records or even ignore the records based on the recordBlocks. For example if only metadata blocks like (ACLs or legal tags) for a record are changed, then a consumer service which is doing quality assessment on the record can choose to ignore the record completely or process it differently by just PATCHing the derived records instead of complete re-processing of the record. Another example could be because only metadata blocks are modified for a record, then the WKS service can decide to only PATCH the metadata blocks of the corresponding WKS records instead of recreating the complete WKS record.

Below table lists probable changes in a consumer service (like WKS service) to react to OSDU notification update events based on content of recordBlocks.

recordBlocks WKS service
metadata Fetch the latest version of the record and fix appropriate metadata blocks in the derived record.
data Recreate the corresponding WKS record using WKS mapping

In case of changes in data as well as metadata blocks, actions taken by WKS service would be to regenerate the derived records and update metadata blocks for them accordingly.

Consequences

  • Record change event needs to send new attributes suggested above in case of create and update events.
  • Register service and accompanying documentation needs to update the example record changed event information to reflect the new attributes in list topics API.
  • Consumer applications need to react to the updated event information from OSDU notification service.
Edited by Mandar Kulkarni
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information