NullPointerException in Indexer Service When Processing Multiple Records with Same Schema

Summary

Indexer service fails with NullPointerException when processing batches containing multiple records of the same kind/schema, but only when no Elasticsearch indexes exist prior to ingestion. This results in incomplete indexing of large data volumes.

Environment

Service: Indexer Service
Version: m25
Platform: All (discovered in Azure)

Severity

High - Not all data in bulk indexing operations will be available for search

Issue Details

Problem Description

When attempting to index larger volumes of data (e.g., Reference Data Values), not all records are successfully indexed. This issue only occurs when:

No Elasticsearch indexes exist prior to ingestion (fresh instance)
A single batch contains multiple change record notifications with the same kind/schema
Indexer cache is cleared

The operation succeeds if:

The same batch is retried after the initial failure
Indexes already exist before processing
Records have different schemas

Root Cause

The defect occurs due to mutation of shared reference objects during processing:

IndexerServiceImpl.cacheOrCreateElasticMapping() iterates through a list of IndexSchema objects
Multiple list items reference the same object instance (duplicate schemas for records of same kind)
During first iteration, TypeMapper.getDataAttributeIndexerMapping() modifies Map entries when processing Map-type fields
On subsequent iterations with the same object reference, the modified structure lacks expected "properties" key
This causes NullPointerException when trying to access propertiesMap.entrySet()

Key Code Path:

  IndexerServiceImpl.processRecordChangedMessages()
    ↓
  IndexerServiceImpl.cacheOrCreateElasticMapping()
    ↓
  IndexerMappingServiceImpl.getDataMapping()
    ↓
  TypeMapper.getDataAttributeIndexerMapping() // <- Mutates shared reference

Steps to Reproduce

Prerequisites

Clear indexer cache (set RUNTIME_ENV_LO=true and restart service)
Delete relevant Elasticsearch indexes:

# Enable wildcard deletion
$settingsUrl = "$baseUrl/_cluster/settings"
$enableWildcardBody = @{
    transient = @{
        "action.destructive_requires_name" = $false
    }
} | ConvertTo-Json -Depth 3

Invoke-RestMethod -Uri $settingsUrl -Method "PUT" -Headers $authHeader -Body $enableWildcardBody

# Delete indexes
Invoke-RestMethod -Uri "$baseUrl/$IndexPattern" -Method "DELETE" -Headers $authHeader

Reproduction Steps

Execute the following request against indexer task worker endpoint:

  curl --request POST \
    --url http://localhost:8080/api/indexer/v2/_dps/task-handlers/index-worker \
    --header 'Authorization: Bearer <token>' \
    --header 'Content-Type: application/json' \
    --header 'data-partition-id: opendes' \
    --data '{
      "messageId": "test-message-id",
      "publishTime": "2025-09-08T03:01:57.892Z",
      "data": "[{\"id\":\"opendes:reference-data--ActivityCode:Test1\",\"kind\":\"osdu:wks:reference-data--ActivityCode:1.0.1\",\"op\":\"create\"},{\"id\":\"opendes:reference-data--ActivityCode:Test2\",\"kind\":\"osdu:wks:reference-d    
  ata--ActivityCode:1.0.1\",\"op\":\"create\"}]",
      "attributes": {
        "data-partition-id": "opendes",
        "correlation-id": "test-correlation-id"
      }
  }'

Expected vs Actual Results

Expected: Both records should be successfully indexed in Elasticsearch

Actual:

HTTP 500 response
NullPointerException in logs
Elasticsearch index created but no records inserted

Error Details

  java.lang.NullPointerException: Cannot invoke "java.util.Map.entrySet()" because "propertiesMap" is null
      at org.opengroup.osdu.indexer.util.TypeMapper.getDataAttributeIndexerMapping(TypeMapper.java:153)
      at org.opengroup.osdu.indexer.util.TypeMapper.getDataAttributeIndexerMapping(TypeMapper.java:155)
      at org.opengroup.osdu.indexer.service.IndexerMappingServiceImpl.getDataMapping(IndexerMappingServiceImpl.java:177)
      at org.opengroup.osdu.indexer.service.IndexerMappingServiceImpl.syncMetaAttributeIndexMappingIfRequired(IndexerMappingServiceImpl.java:239)
      at org.opengroup.osdu.indexer.service.IndexerServiceImpl.cacheOrCreateElasticMapping(IndexerServiceImpl.java:612)

Impact

Data unavailable for search: Records fail to index during bulk operations
Operational Impact: Reference data ingestion failures on fresh instances
Reliability: Inconsistent behavior requiring manual retry operations

Workarounds

Retry the same batch after initial failure
Pre-create Elasticsearch indexes before bulk ingestion
Process records with different schemas in separate batches

Solution Implemented

Root Cause: The getIndexerPayload() method in IndexerServiceImpl.java was adding duplicate IndexSchema objects to the schemas list when multiple records shared the same kind/schema, leading to mutation issues during processing.

Fix Applied: Modified the getIndexerPayload() method (lines 382-421) to deduplicate schemas before adding them to the schemas list. This ensures each unique schema is only processed once, preventing the shared reference mutation
issue.

Code Changes:

File: IndexerServiceImpl.java
Method: getIndexerPayload()
Change: Implemented schema deduplication logic to add each unique schema only once to the schemas list
Benefit: Eliminates processing of duplicate schema objects, preventing mutation of shared references

Technical Details: The fix ensures that when multiple records in a batch share the same schema (same kind), only one instance of that schema is added to the processing pipeline. This prevents the scenario where:

First schema instance gets mutated during mapping generation
Subsequent identical schema instances fail due to the mutated state

This solution maintains the existing processing logic while eliminating the root cause of the shared reference mutation.

Verification

After implementing the fix:

Multiple records with the same schema can be processed successfully in a single batch
Fresh Elasticsearch instances no longer encounter this error
Bulk reference data ingestion operations complete successfully

Edited Sep 08, 2025 by Charles Zipp