F1 (Java) Story 27 [Indexer]: Implement indexing respecting x-collaboration property

Description

When Indexer receives a message from "Record Changed V2" topic, the message contains x-collaboration property that points on record in a custom NS. The indexing logic for records in custom namespace should be implemented in compliance with the recuirements in the following ADR: ADR - Project & Workflow Services - Core Services Integration - Search Service Support

Acceptance criteria

  • a specific indexing logic is implemented for indexing records lying in custom NSs.

Testing scenarios

  • all necessary unit tests
  • when a indexing a record in a custom NS, it should be indexed with "Collaboration" property set in the indexed document

Technical notes

Key points:

  1. For the new RecordChangedV2 event the same consumer entry point can be used, as the event body can be extended with new properties without code changes:
  
curl --location 'https://INDEXER/api/indexer/v2/_dps/task-handlers/index-worker' \
--header 'data-partition-id: osdu' \
--header 'correlation-id: 74c20433-544f-46e3-a215-c059b2ca6810' \
--data '{
    "data": "[{\"id\":\"osdu:test-data--Integration:test1\",\"kind\":\"osdu:indexer:test-data--Integration:1.0.0\",\"op\":\"create\"}]",
    "attributes": {
        "account-id": "osdu",
        "x-collaboration": "id={{collab-id}},application={{app-id}}" 
    }
}'
}

  1. We should reuse existing indices from SOR, updating them as needed. Untitled_Diagram.drawio_31_

  2. When we receive the RecordChangedV2 event we will push it into the index of SOR, with the same kind.

    • Thus mappings will have the collaboration-id property in the meta block.
    • Records from the SOR won't have values for that property "collaboration-id": null.
    • Records from the WIP context will have "collaboration-id": "12345"

  3. Later in the Search service SOR and WIP records can be filtered by that property. Exist query, when searching in WIP context:

curl --location 'https://nice-etching-277309-system.es.us-central1.gcp.cloud.es.io:9243/osdu-test-test-search-2.0.0/_search' \
--header 'Housekeeping: yes' \
--header 'Content-Type: application/json' \
--data '{
  "query": {
    "exists": {
      "field": "collaboration-id"
    }
  }
}'

Does not exist when searching in SOR:

{
  "query": {
    "bool": {
      "must_not": {
        "exists": {
          "field": "collaboration-id"
        }
      }
    }
  }
}
  • We do not need to update the Indexer /index-worker endpoint as it consumes a model that is suitable for RecordChanged V2 event, it has a map of attributes, where collaboration id could be added.
  • The entry point for updates could be IndexerServiceImpl.
  • RecordInfo does not have properties to hold collaboration-id. So here we need to decide whether we will update this model(or create a new one) to pass it further with collaboration-id or construct and inject Spring Bean into context, produced only if collaboration-id is present. We'll reuse existing methods, so deeper in the call stack we need to have info about collaboration-id somehow.
  • Do not forget that functionality should be controlled via Feature Flag, so supplement all respective methods with a flag check.
  • The next place for updates is MappingService and it's method syncIndexMappingIfRequired.
  • Somewhere here we can inject collaboration-id property into the meta block.
  • Update ingested record metadata in getSourceMap method, add collaboration-id to the record meta.

Changes made for the feature BagOfWords could be studied as they are somewhat similar. indexer-service!649 (diffs)

Record reindex:

Endpoint to update: https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/indexer-core/src/main/java/org/opengroup/osdu/indexer/api/ReindexApi.java?ref_type=heads#L82

Required changes:

Edited by Rustam Lotsmanenko (EPAM)