x-osdu-indexing changes are breaking
Context:
Indexing hints in the OSDU schemas are considered decorations and not taken into account when schemas versions are validated for 'breaking changes'.
Downstream indexing changes from any state to any other state are considered breaking changes:
- Breaking changes for the indexer: changes from
flattened
tonested
require the re-indexing of the kind in question. - Consuming applications must use a different query syntax.
How it's done today:
The process depends on human interaction (assuming OSDU well-known schemas here, but this is no different for custom schemas):
- Stakeholders ask for an indexing behavior change, OSDU Data Definition reacts by changing the
x-osdu-indexing
extension tag values in the schema. - OSDU Data Definition Release notes identify the kinds, which are to be re-indexed.
- In M10 virtually all kinds had to be re-indexed
- In M11 type
reference-data--QualityDataRuleSet
requires re-indexing
- During deployment the records for the affected kinds must be re-indexed.
Issue with current design:
Upon deployment of a new milestone (or custom schemas),
- for all involved data-partitions, delete the index for the changed kind and trigger re-indexing. This can take - depending on the number of records per kind - a very log time and cause serious down-time.
- Applications have no good way of understanding that the query syntax has changed. Applications may no longer find data if they depended on queries into data structures affected by the change.
Proposal:
PUBLISHED
Schema Status
- For schemas with state
PUBLISHED
treat changes to thex-osdu-indexing
extension tag values in the schema as _ breaking changes_. - Breaking changes require an incremented major schema version number.
- Schema Validation Changes during schema creation:
- Changes to the
x-osdu-indexing
extension tag values inPUBLISHED
schemas with same major schema version numbers will be rejected. I.e., the attempted registration of such schema will fail with error.
- Changes to the
DEVELOPMENT
Schema Status
- The validation for
DEVELOPMENT
status schemas for incremental versions on top of or between existing minor or patch versions follows the same rules as forPUBLISHED
schemas. Attempts to change thex-osdu-indexing
extension tag values will be rejected by the Schema service. - For 'single' version schemas in
DEVELOPMENT
, the updates of thex-osdu-indexing
extension tag values are permitted.- It is the responsibility of the schema authors to communicate the impact to deployment and consumers. This is expected to be acceptable during the development phase.