Skip to content

Azure - fix legal tag expiration

Wesley Evans requested to merge azure/fix-legal-tag-expiration into master

Issue description

This MR is intended to solve the issue described here

If there are large number of records associated with a legalTag that expires after running the cron job, we are seeing availability issues and inconsistent result in terms of record searchability.

Changes

  • Overloaded RecordMetadataRepository.queryByLegalTagName to accept an array of legal tags
  • Updated LegalComplianceChangeServiceAzureImpl.updateComplianceOnRecords to use overloaded queryByLegalTagName to search for multiple legal tags at once
  • Updated corresponding unit tests

Change description

Each Cosmos DB search by legal tag name is a cross-partition search. When searching for a large number (hundreds) of legal tags, this results in hundreds of cross-partition queries, exceeding the RU's for the search service. The solution simply reworks the query to search for multiple legal tags in a single query. This results in 1 or 2 cross-partition queries rather than hundreds.

Log from call to /api/legal/v1/jobs/updateLegalTagStatus demonstrating successful deletion of incompliant tag

2024-11-11 18:19:11.200  INFO storage-577cbcd9df-qpvrc --- [p-nio-80-exec-4] TxnLogger                                correlation-id=44f1f29a-3e9d-4779-b6c6-d948c096821f data-partition-id=opendes api-method=POST operation-name={POST [/push-handlers/legaltag-changed]} user-id=8b2a56ba-edf5-47ce-94b6-42c336ec8172 app-id=678fadf8-e5a8-46cd-a75d-4d6cc95d9bc9:storage.app Start Web-API POST /push-handlers/legaltag-changed Headers: {correlation-id:44f1f29a-3e9d-4779-b6c6-d948c096821f,data-partition-id:opendes,content-type:application/json} {correlation-id=44f1f29a-3e9d-4779-b6c6-d948c096821f, data-partition-id=opendes}
2024-11-11 18:19:11.861  WARN storage-577cbcd9df-qpvrc --- [p-nio-80-exec-4] o.o.o.a.l.Slf4JLogger                    correlation-id=44f1f29a-3e9d-4779-b6c6-d948c096821f data-partition-id=opendes api-method=POST operation-name={POST [/push-handlers/legaltag-changed]} user-id=8b2a56ba-edf5-47ce-94b6-42c336ec8172 app-id=678fadf8-e5a8-46cd-a75d-4d6cc95d9bc9:storage.app Inconsistency between pubsub message and legal: opendes-storage-1731348999265random2. Expected incompliant. {correlation-id=44f1f29a-3e9d-4779-b6c6-d948c096821f, data-partition-id=opendes}
2024-11-11 18:19:12.263  INFO storage-577cbcd9df-qpvrc --- [p-nio-80-exec-4] o.o.o.s.p.a.MessageBusImpl               correlation-id=44f1f29a-3e9d-4779-b6c6-d948c096821f data-partition-id=opendes api-method=POST operation-name={POST [/push-handlers/legaltag-changed]} user-id=8b2a56ba-edf5-47ce-94b6-42c336ec8172 app-id=678fadf8-e5a8-46cd-a75d-4d6cc95d9bc9:The correlation id for this is 44f1f29a-3e9d-4779-b6c6-d948c096821f
2024-11-11 18:19:12.300  INFO storage-577cbcd9df-qpvrc --- [p-nio-80-exec-4] .s.LegalComplianceChangeServiceAzureImpl correlation-id=44f1f29a-3e9d-4779-b6c6-d948c096821f data-partition-id=opendes api-method=POST operation-name={POST [/push-handlers/legaltag-changed]} user-id=8b2a56ba-edf5-47ce-94b6-42c336ec8172 app-id=678fadf8-e5a8-46cd-a75d-4d6cc95d9bc9:1 Records deleted successfully [opendes:endtoend:1.1.1731348999265] for legal tag(s) opendes-storage-1731348999265
2024-11-11 18:19:12.301  INFO storage-577cbcd9df-qpvrc --- [p-nio-80-exec-4] TxnLogger                                correlation-id=44f1f29a-3e9d-4779-b6c6-d948c096821f data-partition-id=opendes api-method=POST operation-name={POST [/push-handlers/legaltag-changed]} user-id=8b2a56ba-edf5-47ce-94b6-42c336ec8172 app-id=678fadf8-e5a8-46cd-a75d-4d6cc95d9bc9:storage.app End Web-API POST /push-handlers/legaltag-changed Headers: {correlation-id:44f1f29a-3e9d-4779-b6c6-d948c096821f} status=200 time=1101 ms {correlation-id=44f1f29a-3e9d-4779-b6c6-d948c096821f, data-partition-id=opendes}
Edited by Wesley Evans

Merge request reports

Loading