Skip to content
GitLab
  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • S seismic-dms-service
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 34
    • Issues 34
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 17
    • Merge requests 17
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe Software
  • Platform
  • Domain Data Mgmt Services
  • Seismic
  • Seismic DMS Suite
  • seismic-dms-service
  • Merge requests
  • !304

fix: azure cosmos DB redesign to improve general performances

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Diego Molteni requested to merge slb/dm3/cosmos-db-design into master Jan 24, 2022
  • Overview 8
  • Commits 52
  • Pipelines 46
  • Changes 41

The existing design of cosmos DB could not meet the required performances. All datasets that belong to the same subproject are stored with/under the same physical partition key. A physical partition set a limit of 10k RU/s (cannot be modified). To enable cosmos horizontal scaling and higher RU/s the DB has been redesigned so every entity (tenant/subproject/dataset/app) will have its own partition key. This will allow cosmos to spread entities across multiple physical partitions

  • tenant -> tn-<tenant_namme> (1 per partition)
  • subproject -> sp-<subproject_name> (subproject belong to a tenant )
  • dataset -> ds-<subprpoject_name>-<dataset_path>-<dataset_name> (datasets belong to a subproject... the subproject name is required)
  • app -> app-<app_name> (apps belong to a tenant)

ref: https://docs.microsoft.com/en-us/azure/cosmos-db/partitioning-overview#choose-partitionkey

An issue has been fixed on how global tags are parsed from input requests.

Added support for managing azure migration:

  • The service will work with the older index model if exist only the old cosmos database
  • The service will work with the new index model if exist only the new cosmos database (partition migrated or newly created)
  • The service will throw an error if both datasets exist at the same time (migration in progress)

The correlation id propagation has been fixed. The Azure insight telemetry process has been fixed so probes do not get included as consumer in the tracking (as does not have auth info)

Edited Mar 16, 2022 by Diego Molteni
Assignee
Assign to
Reviewer
Request review from
Time tracking
Source branch: slb/dm3/cosmos-db-design