x-osdu-indexing changes are breaking

Context:

Indexing hints in the OSDU schemas are considered decorations and not taken into account when schemas versions are validated for 'breaking changes'.

Downstream indexing changes from any state to any other state are considered breaking changes:

  • Breaking changes for the indexer: changes from flattened to nested require the re-indexing of the kind in question.
  • Consuming applications must use a different query syntax.

How it's done today:

The process depends on human interaction (assuming OSDU well-known schemas here, but this is no different for custom schemas):

  • Stakeholders ask for an indexing behavior change, OSDU Data Definition reacts by changing the x-osdu-indexing extension tag values in the schema.
  • OSDU Data Definition Release notes identify the kinds, which are to be re-indexed.
    • In M10 virtually all kinds had to be re-indexed
    • In M11 type reference-data--QualityDataRuleSet requires re-indexing
  • During deployment the records for the affected kinds must be re-indexed.

Issue with current design:

Upon deployment of a new milestone (or custom schemas),

  1. for all involved data-partitions, delete the index for the changed kind and trigger re-indexing. This can take - depending on the number of records per kind - a very log time and cause serious down-time.
  2. Applications have no good way of understanding that the query syntax has changed. Applications may no longer find data if they depended on queries into data structures affected by the change.

Proposal:

PUBLISHED Schema Status

  1. For schemas with state PUBLISHED treat changes to the x-osdu-indexing extension tag values in the schema as _ breaking changes_.
  2. Breaking changes require an incremented major schema version number.
  3. Schema Validation Changes during schema creation:
    • Changes to the x-osdu-indexing extension tag values in PUBLISHED schemas with same major schema version numbers will be rejected. I.e., the attempted registration of such schema will fail with error.

DEVELOPMENT Schema Status

  1. The validation for DEVELOPMENT status schemas for incremental versions on top of or between existing minor or patch versions follows the same rules as for PUBLISHED schemas. Attempts to change the x-osdu-indexing extension tag values will be rejected by the Schema service.
  2. For 'single' version schemas in DEVELOPMENT, the updates of the x-osdu-indexing extension tag values are permitted.
    • It is the responsibility of the schema authors to communicate the impact to deployment and consumers. This is expected to be acceptable during the development phase.

CC @nthakur @ChrisZhang @chad @pbehede