Skip to content

Bulk update api & GCP implementation

Hong Yan requested to merge bulk-update-api into master

Type of change

  • Bug Fix
  • Feature

Please provide link to gitlab issue or ADR(Architecture Decision Record)
[System/Storage] Bulk Update for ACLs on Records

Does this introduce a change in the core logic?

  • [YES]

Does this introduce a change in the cloud provider implementation, if so which cloud?

  • AWS
  • Azure
  • GCP
  • IBM

The API is fully implemented for GCP. In storage core, the new added interface methods

org.opengroup.osdu.storage.provider.interfaces.ICloudStorage.updateObjectMetadata
org.opengroup.osdu.storage.provider.interfaces.ICloudStorage.revertObjectMetadata

A new method is added in API and two methods added in SPI. We have overridden the method in all clouds which throws NotImplementedException, this allows us to review and merge the changes now and then cloud providers can implement depends on the availability.

Does this introduce a breaking change?

  • [YES]

What is the current behavior?

Records that are ingested with ACLs that are incorrect can be painful to correct in the current system. There is no update mechanism for the ACLs, leading to mass re-ingestions of records for the sole purpose of correcting ACLs when the data has not changed.

What is the new/expected behavior?

Add a new API (PATCH /records)to do PATCH updates of just the ACLs of a batch of data so that the only input needs to be the Record Identifiers and the new ACLs.

Have you added/updated Unit Tests and Integration Tests?

Yes, new unit tests are added for storage-core logic and gcp implementation. Integration tests are also added, but need to be tested in a trusted branch to ensure they work properly. For all providers other than GCP, the integration tests are overridden by placeholder tests which only check if the api returns 501 Not Implemented.

Any other useful information

Edited by ethiraj krishnamanaidu

Merge request reports