F1 (Java) Story 23 [Storage + Indexer Queue]: write to records-changed-2 topic implementation for AWS

Description

When a record is changed in the custom namespace, a message should be sent to the "Record Changed V2" topic on message broker instead of the usual "Record Changed V1" topic. The requirement is mentioned in the following ADRs:

• ADR: Namespacing storage records
• ADR - Project & Workflow Services - Core Services Integration - Search Service Support

It has already been implemented for Azure CSP, see MR !553

Acceptance criteria

• Changing a record in a custom namespace in AWS environment causes sending a message to "Record Changed V2" topic

Testing scenarios

• change a record in SOR -> a message goes to the "Record Changed V1" topic
• change a record in a custom NS -> a message goes to the "Record Changed V2" topic

Technical notes

Storage

Implement AWS RecordChanged V2 publisher.

• The event should be sent to a new topic, we cannot reuse the existing topic, as it may disrupt existing consumers. Records copies are supposed to be sent to a new topic, so we don't want them to show up in the RecordsChanged V1 topic.
• Update infrastructure scripts, a new topic should become a default of environment bootstrapping.
• Enable the Collaboration feature in the dev environment.
• Update readme if needed.

Indexer Queue

Implement RecordChanged V2 consumer.

• Indexer Queue should be able to consume events from a new topic.
• Indexer Queue should send events to the Indexer service, using the same endpoint /api/indexer/v2/_dps/task-handlers/index-worker
• collaboration-id validation happens in the Storage service, if necessary add validation in Indexer Queue.

RecordChanged V2 event example:

curl --location 'https://INDEXER/api/indexer/v2/_dps/task-handlers/index-worker' \
--header 'data-partition-id: osdu' \
--header 'correlation-id: 74c20433-544f-46e3-a215-c059b2ca6810' \
--data '{
    "data": "[{\"id\":\"osdu:test-data--Integration:test1\",\"kind\":\"osdu:indexer:test-data--Integration:1.0.0\",\"op\":\"create\"}]",
    "attributes": {
        "account-id": "osdu",
        "x-collaboration": "id={{collab-id}},application={{app-id}}" 
    }
}'
}