Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Register
  • Sign in
  • S seismic-dms-service
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
    • Locked files
  • Issues 49
    • Issues 49
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 12
    • Merge requests 12
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Terraform modules
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe SoftwareOpen Subsurface Data Universe Software
  • Platform
  • Domain Data Management Services
  • Seismic
  • Seismic DMS Suite
  • seismic-dms-service
  • Merge requests
  • !304

fix: azure cosmos DB redesign to improve general performances

  • Review changes

  • Download
  • Patches
  • Plain diff
Merged Diego Molteni requested to merge slb/dm3/cosmos-db-design into master Jan 24, 2022
  • Overview 8
  • Commits 52
  • Pipelines 46
  • Changes 41

The existing design of cosmos DB could not meet the required performances. All datasets that belong to the same subproject are stored with/under the same physical partition key. A physical partition set a limit of 10k RU/s (cannot be modified). To enable cosmos horizontal scaling and higher RU/s the DB has been redesigned so every entity (tenant/subproject/dataset/app) will have its own partition key. This will allow cosmos to spread entities across multiple physical partitions

  • tenant -> tn-<tenant_namme> (1 per partition)
  • subproject -> sp-<subproject_name> (subproject belong to a tenant )
  • dataset -> ds-<subprpoject_name>-<dataset_path>-<dataset_name> (datasets belong to a subproject... the subproject name is required)
  • app -> app-<app_name> (apps belong to a tenant)

ref: https://docs.microsoft.com/en-us/azure/cosmos-db/partitioning-overview#choose-partitionkey

An issue has been fixed on how global tags are parsed from input requests.

Added support for managing azure migration:

  • The service will work with the older index model if exist only the old cosmos database
  • The service will work with the new index model if exist only the new cosmos database (partition migrated or newly created)
  • The service will throw an error if both datasets exist at the same time (migration in progress)

The correlation id propagation has been fixed. The Azure insight telemetry process has been fixed so probes do not get included as consumer in the tracking (as does not have auth info)

Edited Mar 16, 2022 by Diego Molteni
Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: slb/dm3/cosmos-db-design