Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Register
  • Sign in
  • Wellbore Domain Services Wellbore Domain Services
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
    • Locked files
  • Issues 17
    • Issues 17
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 5
    • Merge requests 5
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe SoftwareOpen Subsurface Data Universe Software
  • Platform
  • Domain Data Management Services
  • Wellbore
  • Wellbore Domain ServicesWellbore Domain Services
  • Merge requests
  • !613

Load save catalog async storage

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Yannick requested to merge load_save_catalog_async_storage into master Sep 14, 2022
  • Overview 9
  • Commits 8
  • Pipelines 4
  • Changes 14

Main goal make the save and the load of the bulk catalog outside of the Dask worker and perform the I/O asynchronously. Note that the json dumps and load are done inside the main thread, by the way, for testing, even with dealing with many column (tested with 150k columns) is quick enough (actually it's not heavier than the current dump/load done for the meta data).

To do so, some side changes:

  • [no mandatory] tenant resolved in middleware and then put in the context
  • join builder reviewed to always use '/' as separator to be compatible to all known cases so far (including windows fs or Azurite emulator)
  • some test fixtures updated
Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: load_save_catalog_async_storage