[BUG] Incorrect Operation Type Published When Updating Record Kind When OPA is Enabled
When OPA is enabled for Storage service and updating a record's kind field, the record's previous kind is still observable in Search service.
For example, creating the following record using PUT {{base_url}}/api/storage/v2/records:
[
{
"id":"{{data_partition_id}}:dataset--File.Generic:1000",
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
"data": {
"Endian": "BIG",
"Name": "dummy",
"DatasetProperties.FileSourceInfo.FileSource": "",
"DatasetProperties.FileSourceInfo.PreloadFilePath": ""
},
"namespace": "osdu:wks",
"legal": {
"legaltags": [
"{{data_partition_id}}-public-usa-dataset-1"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"acl": {
"viewers": [
"data.default.viewers@{{data_partition_id}}.{{domain}}"
],
"owners": [
"data.default.owners@{{data_partition_id}}.{{domain}}"
]
},
"type": "dataset--File.Generic",
"version": 1620833190423950
}
]
And updating the kind to be:
[
{
"id":"{{data_partition_id}}:dataset--File.Generic:1000",
"kind": "osdu:wks:dataset--File.Generic:1.0.1",
...
}
]
It is expected that Search service would return the following result when making the request to POST {{base_url}}/api/search/v2/query:
Body:
{
"kind": "osdu:wks:dataset--File.Generic:1.0.0"
}
Expected Result:
{
"results": [],
"aggregations": [],
"totalCount": 0
}
However, the un-updated result is returned:
{
"results": [
{
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
"source": "wks",
"acl": {
"viewers": [
"data.default.viewers@osdu.example.com"
],
"owners": [
"data.default.owners@osdu.example.com"
]
},
"type": "dataset--File.Generic",
"version": 1663008428769106,
"tags": null,
"modifyUser": "admin@testing.com",
"modifyTime": "2022-09-12T18:47:08.806Z",
"createTime": "2022-09-09T19:12:46.378Z",
"authority": "osdu",
"namespace": "osdu:wks",
"legal": {
"legaltags": [
"osdu-public-usa-dataset-1"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "admin@testing.com",
"id": "osdu:dataset--File.Generic:1000"
}
],
"aggregations": null,
"totalCount": 1
}
Performing a record query on Storage service returns the expected response, as well as Search service when searching for the updated kind.
The issue appears to be caused by the incorrect message being published by Storage service resulting in Indexer service only creating the new index for the updated record and not removing the record from the previous kind. Indexer knows to remove a record from an index during an update operation when there is a previousVersionKind field that is not null or empty. Storage publishes a message with this field when the updated records kind is different than the existing records kind. The existing records kind is overwritten to match the updated record when validating the record using OPA, so when the kind's are compared by Storage's IngestService, the result always evaluates to be a match and, therefore, the previousVersionKind field is never populated.