Skip to content

fix: azure cosmos DB redesign to improve general performances

Diego Molteni requested to merge slb/dm3/cosmos-db-design into master

The existing design of cosmos DB could not meet the required performances. All datasets that belong to the same subproject are stored with/under the same physical partition key. A physical partition set a limit of 10k RU/s (cannot be modified). To enable cosmos horizontal scaling and higher RU/s the DB has been redesigned so every entity (tenant/subproject/dataset/app) will have its own partition key. This will allow cosmos to spread entities across multiple physical partitions

  • tenant -> tn-<tenant_namme> (1 per partition)
  • subproject -> sp-<subproject_name> (subproject belong to a tenant )
  • dataset -> ds-<subprpoject_name>-<dataset_path>-<dataset_name> (datasets belong to a subproject... the subproject name is required)
  • app -> app-<app_name> (apps belong to a tenant)

ref: https://docs.microsoft.com/en-us/azure/cosmos-db/partitioning-overview#choose-partitionkey

An issue has been fixed on how global tags are parsed from input requests.

Added support for managing azure migration:

  • The service will work with the older index model if exist only the old cosmos database
  • The service will work with the new index model if exist only the new cosmos database (partition migrated or newly created)
  • The service will throw an error if both datasets exist at the same time (migration in progress)

The correlation id propagation has been fixed. The Azure insight telemetry process has been fixed so probes do not get included as consumer in the tracking (as does not have auth info)

Edited by Diego Molteni

Merge request reports