OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2021-06-16T22:18:58Zhttps://community.opengroup.org/osdu/platform/deployment-and-operations/multi-region-deployment/multi-region/-/issues/1[Multi-Region Deployment] Support OSDU Multi-Region Deployment : Requirements2021-06-16T22:18:58ZStephen Whitley (Invited Expert)[Multi-Region Deployment] Support OSDU Multi-Region Deployment : Requirements**Background:**
Subsurface exploration and production is a global enterprise, hence most operating companies have a need for subsurface data availability across multiple geographic regions. Strict latency, data sovereignty, or local regu...**Background:**
Subsurface exploration and production is a global enterprise, hence most operating companies have a need for subsurface data availability across multiple geographic regions. Strict latency, data sovereignty, or local regulatory/legislative requirements may result in the need for numerous distributed, high-availability OSDU regions. Sharing and managing data across multiple distributed installations is a core capability for OSDU.
An OSDU Region is defined as an independent and highly available geographical deployment of the full OSDU design. It provides full functionality within a given region and there is no need for deploying multiple OSDU environments for high availability or disaster recovery requirements. Strict latency, data sovereignty or local legislation requirements may result into the need for additional OSDU deployments. OSDU deployments are preferably public cloud based. In situation where in-country OSDU deployment is required and no public cloud region is available, physical edge capabilities from the public cloud provider will be used. In any case, the exposed OSDU APIs deployed in the different OSDU regions will provide identical functionality.
**Objective:**
- This requirement is to ensure OSDU data platform can support highly available multi-region/geographical deployment.
**Business Case:**
1. Adopting company has subsurface professionals interacting with OSDU around the world. All expect optimal performance and high availability when interacting with data and applications.
2. Adopting company has stringent availability/up-time and fault tolerance demands on the data and applications hosted in OSDU.
3. Adopting company have disaster recovery and business continuity requirements that need to be fulfilled. A multi-region deployment is essential to the goal of maximum redundancy and resilience to catastrophic failure.
4. Regulatory restrictions governing the adopting company's activities mandate regional data never travels beyond the region's geographic borders.
**Summary of Requirements**
OSDU must enable OSDU Using Companies to stand-up and operate multiple concurrent, homogeneous instances of OSDU in different geographic regions. Each OSDU deployment instance is stand-alone/independent, but all are federated to provide global access to the underlying subsurface data hosted and delivered via the OSDU Data Platform in each region. OSDU shall deliver users no more than approx. 40 millisecond latency between the end user and the OSDU co-resident applications and services; therefore, multiple regional deployments shall have globally replicated metadata and regionally originated and stored actual data files (datasets, etc.), including raw data, processed data and interpreted data.
**Requirements**
*General Requirements*
1. Using companies shall be able to deploy any number of independent, complete, and homogeneous instances of OSDU in different geographic regions.
2. Each regional deployment includes identical functionalities in the main architectural components: platform services, loading/enrichment/consumption data services, search engine, graph and NoSQL databases, and object stores.
3. Each deployment is 'stand-alone': applications, services and data can be independently deployed, accessed, and utilized by real-world users within the hosting region.
4. Each regional deployment can access the shared 'System of Record' i.e., the data platform including all the globally replicated metadata and reference data.
5. Stand up of a new deployment may require administrative capabilities residing in another regional deployment
*Administration Requirements*
1. One region shall be designated as the 'Admin Region' which provides administrative functions allowing configuration, monitoring, and control of all regional instances.
2. OSDU shall provide the ability to designate and configure one and only one deployment as an 'Admin Region'.
3. Once an OSDU deployment is designated as the 'Admin Region', all administrative API calls throughout all OSDU regional deployments are re-rerouted to the administrative services hosted in the Admin Region
4. From the ‘Admin Region’, the health and availability of other OSDU regional deployments can be accessed and logged.
5. Administrative entitlements shall be defined to govern which users/roles can access which administrative capabilities.
*Infrastructure Requirements*
1. One region (not necessarily the 'Admin Region') can be designated a 'Central Region' in order to host the development and deployment infrastructure (dev/test environments, CI/CD tooling, etc.)
2. Using companies shall be able to install and configure development + deployment infrastructure to a regional instance for development, acceptance test, and CI/CD activities.
3. Deployment infrastructure accommodates setup of the OSDU deployment instances (platform services, identity provider, data platform, search capabilities, etc.) It also must accommodate deployment of applications and services from the OSDU marketplace as well as Using Company proprietary works. All regional deployments are homogeneous and the deployment infrastructure accommodates making new regional deployments appear similar to pre-existing deployments.
4. Using companies shall be able to setup and configure connectivity between an OSDU instance and Operator's on-premise data center(s)
5. Each OSDU deployment must allow connections to/from a co-hosted 'Transit VPC' which facilitates data connections between on-premise / in-country installations and OSDU
*Data Platform Requirements*
1. Replication of data and services are always subject to administratively-defined replication policies.
2. Replication policies are persisted and distributed as master reference data in the OSDU Data Platform.
3. Global access to metadata and underlying data records is governed by the entitlements’ policies / rules from the OSDU Data Platform.
4. All metadata is replicated across all nodes; at every node access to all metadata for that multi- region OSDU implementation. Single-cloud provider within an OSDU Using Company (i.e., replicating metadata between multiple cloud provider solutions is out of scope for R3).
5. Transactional data is replicated by exception, not by default. It is expected that the 'home' for data is well-defined and mapped between its geographical region and its usage. By exception and on-request, transactional data can be replicated to host regions to achieve efficient consumption.
6. Data is to be regionally bound and tagged. Data resides by default in the OSDU “home” region in which the loading/ingestion workflow occurred, which will be the region associated with the data from an earth location perspective. Since the describing metadata is replicated globally, any OSDU user can discern in which region the described data resides.
7. The OSDU DP Search API can be used to locate search results based on globally replicated master-data, reference-data, and metadata. As stated elsewhere in the requirements, metadata are not protected and a user accessing any OSDU regional deployment can see all metadata. However, this is only true for metadata and of course not for the data content files / datasets / etc. (raw, processed, interpreted, etc.).
8. Facilities shall be provided to enable on-demand replication of content data files from ‘home’ region to a ‘host’ region’: selected data can be transported via request from its home region to any other OSDU region. Subsequent changes, that is, new items of data that serve as successor versions or new derivations, are populated to the home region and, if requested, to the same ‘host region’.
*Standout Non-functional Requirements (**TBD**)*
1. High-availability: Are OSDU services and data always available to end users, despite failure conditions?
2. Performance: Does the distributed solution meet the in-region performance goals for the data platform?
3. Latency: How much latency is there between the creation of content and it's availability across all regions?
4. Security: How much does the solution increase the threat surface of the OSDU data platform?
5. Ease of Use: How easy is it configure and operate?
6. Observability: How easy is it to test, identify, and resolve incorrect behavior?
7. Maintainability: How much does the solution introduce complexity in development, deployment and operations of the data platform?
**Additional Use Cases**
1. Automated replication of metadata across all regions.
(applies to: work-product-metadata, work-product-component-metadata, associated file metadata)
* "Easy" - leverage CSP storage technologies
* Leads to 'eventual consistency' of search result indices which crawl the metadata at different intervals
2. Master-Reference Data - also needs to be automatically replicated across all regions (to enable context and validation for transactional data.)
3. On-demand request for a WP located in a different region:
* Can be requested by a user via search UI or by a microservice via API
* WP is copied to the requestor region
- Subject to E&O contract + replication policy. If the replication policy states the data cannot travel to the requestor region, then user is notified - confirm?
- Metadata has a 'locator' indicating the WP's home region
- Lineage preserved, but it's not a deep copy of the references
- WPCs are 'deep copied' to the requestor region with read-only access
* Any derivative WPs made in the requestor region are loaded/ingested back in the 'home' region as identified in the metadata 'locator'
- Lineage is updated as usual
- WPs are loaded/ingested in the 'home' region
4. Repartitioning (example: grow from 2-regions to 3-regions) - �be able to reallocate home partition boundaries via 'staging'
* Creating a new region -- provides instant accessibility to the 'common' partition.
* Administrative functions to be able to move/copy a partition between regions. Some using companies may require the ability to replicate a single partition across several regions. Each region has a replica with only the data applicable to its users / assets (geographic area) and can transfer changes of this data to the 'home' region. This allows the 'home' region to perform analysis on data that is up to date across the entire extent.
**Architecture Principles / Guidelines**
1. Synchronous cross-regional calls should be avoided when possible. Applications should use regional resources.
2. Embrace asynchronous systems and replication - high availability / deferred consistency
3. Leverage the principles of Isolation and Redundancy: a failure of any kind in one Region should not affect services running in another.
**OSDU Regions**
An OSDU Region is an independent and complete deployment of all infrastructure and application services required to provide OSDU Data Platform (DP) services. Whilst every OSDU Region will have a homogeneous DP service deployment, a few types of OSDU regions can be identified:
- “*OSDU Standard Hosted Region*”: This is the common type of OSDU Region and will host all data platform services, including security, ingest, logging and monitoring. Platform, application and workflow services will be deployed based on need in this standard OSDU Region.
- “*OSDU Central Region*”: One of the standard OSDU regions will be assigned as a central region and will be hosting the development, integration test and performance test environments and Continuous Integration (CI)/ Continuous Delivery (CD) services. If required, it may host a central logging service. It will also host global OSDU application.
- “*OSDU Admin Region*”: One of the standard OSDU regions will be assigned as an admin region and this is a configured parameter in the OSDU metadata, namely the 'Admin Management' parameters. Subsurface Master Data System (SMDS) Master and Reference Data as well as OSDU management parameters stored in SMDS can only be updated in this configured OSDU Admin Region. All Admin APIs are available in all OSDU regions, though the call will be rejected (or redirected) if the API is not hosted in the OSDU Admin Region.
- “*OSDU In-Country Region*”: When public cloud infrastructure cannot be leveraged to deploy an OSDU Region for data sovereignty or latency reasons, an in-country OSDU deployment takes place with the consideration of incurring least management and operating overhead. Public cloud edge capabilities will be leveraged to most possible extent to increase the consistency and lower the overhead. The closest “OSDU Standard Region” will be configured as a proxy for in-country region's Subsurface Work Product Services (SWPS) metadata represent files within the in-country OSDU Region and providing home region replication.
List of OSDU Regions, and the identification of the Central and Admin OSDU Region will be part of the SMDS Master Data and will be administered as part of the Admin service.M1 - Release 0.1Stephen Whitley (Invited Expert)Hrvoje MarkovicFerris ArgyleRaj KannanDania Kodeih (Microsoft)JoeAlan DonigerBrian KirklandStephen Whitley (Invited Expert)https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/issues/7[Platform] Support OSDU Multi-Region Deployment2020-09-15T20:43:37ZStephen Whitley (Invited Expert)[Platform] Support OSDU Multi-Region Deployment**Background:**
Subsurface exploration and production is a global enterprise, hence most operating companies have a need for subsurface data availability across multiple geographic regions. Strict latency, data sovereignty, or local regu...**Background:**
Subsurface exploration and production is a global enterprise, hence most operating companies have a need for subsurface data availability across multiple geographic regions. Strict latency, data sovereignty, or local regulatory/legislative requirements may result in the need for numerous distributed, high-availability OSDU regions. Sharing and managing data across multiple distributed installations is a core capability for OSDU.
An OSDU Region is defined as an independent and highly available geographical deployment of the full OSDU design. It provides full functionality within a given region and there is no need for deploying multiple OSDU environments for high availability or disaster recovery requirements. Strict latency, data sovereignty or local legislation requirements may result into the need for additional OSDU deployments. OSDU deployments are preferably public cloud based. In situation where in-country OSDU deployment is required and no public cloud region is available, physical edge capabilities from the public cloud provider will be used. In any case, the exposed OSDU APIs deployed in the different OSDU regions will provide identical functionality.
**Objective:**
- This requirement is to ensure OSDU data platform can support highly available multi-region/geographical deployment.
**Business Case:**
1. Adopting company has subsurface professionals interacting with OSDU around the world. All expect optimal performance and high availability when interacting with data and applications.
2. Adopting company has stringent availability/up-time and fault tolerance demands on the data and applications hosted in OSDU.
3. Adopting company have disaster recovery and business continuity requirements that need to be fulfilled. A multi-region deployment is essential to the goal of maximum redundancy and resilience to catastrophic failure.
4. Regulatory restrictions governing the adopting company's activities mandate regional data never travels beyond the region's geographic borders.
**Summary of Requirements**
OSDU must enable OSDU Using Companies to stand-up and operate multiple concurrent, homogeneous instances of OSDU in different geographic regions. Each OSDU deployment instance is stand-alone/independent, but all are federated to provide global access to the underlying subsurface data hosted and delivered via the OSDU Data Platform in each region. OSDU shall deliver users no more than approx. 40 millisecond latency between the end user and the OSDU co-resident applications and services; therefore, multiple regional deployments shall have globally replicated metadata and regionally originated and stored actual data files (datasets, etc.), including raw data, processed data and interpreted data.
**Requirements**
*General Requirements*
1. Using companies shall be able to deploy any number of independent, complete, and homogeneous instances of OSDU in different geographic regions.
2. Each regional deployment includes identical functionalities in the main architectural components: platform services, loading/enrichment/consumption data services, search engine, graph and NoSQL databases, and object stores.
3. Each deployment is 'stand-alone': applications, services and data can be independently deployed, accessed, and utilized by real-world users within the hosting region.
4. Each regional deployment can access the shared 'System of Record' i.e., the data platform including all the globally replicated metadata and reference data.
5. Stand up of a new deployment may require administrative capabilities residing in another regional deployment
*Administration Requirements*
1. One region shall be designated as the 'Admin Region' which provides administrative functions allowing configuration, monitoring, and control of all regional instances.
2. OSDU shall provide the ability to designate and configure one and only one deployment as an 'Admin Region'.
3. Once an OSDU deployment is designated as the 'Admin Region', all administrative API calls throughout all OSDU regional deployments are re-rerouted to the administrative services hosted in the Admin Region
4. From the ‘Admin Region’, the health and availability of other OSDU regional deployments can be accessed and logged.
5. Administrative entitlements shall be defined to govern which users/roles can access which administrative capabilities.
*Infrastructure Requirements*
1. One region (not necessarily the 'Admin Region') can be designated a 'Central Region' in order to host the development and deployment infrastructure (dev/test environments, CI/CD tooling, etc.)
2. Using companies shall be able to install and configure development + deployment infrastructure to a regional instance for development, acceptance test, and CI/CD activities.
3. Deployment infrastructure accommodates setup of the OSDU deployment instances (platform services, identity provider, data platform, search capabilities, etc.) It also must accommodate deployment of applications and services from the OSDU marketplace as well as Using Company proprietary works. All regional deployments are homogeneous and the deployment infrastructure accommodates making new regional deployments appear similar to pre-existing deployments.
4. Using companies shall be able to setup and configure connectivity between an OSDU instance and Operator's on-premise data center(s)
5. Each OSDU deployment must allow connections to/from a co-hosted 'Transit VPC' which facilitates data connections between on-premise / in-country installations and OSDU
*Data Platform Requirements*
1. Replication of data and services are always subject to administratively-defined replication policies.
2. Replication policies are persisted and distributed as master reference data in the OSDU Data Platform.
3. Global access to metadata and underlying data records is governed by the entitlements’ policies / rules from the OSDU Data Platform.
4. All metadata is replicated across all nodes; at every node access to all metadata for that multi- region OSDU implementation. Single-cloud provider within an OSDU Using Company (i.e., replicating metadata between multiple cloud provider solutions is out of scope for R3).
5. Transactional data is replicated by exception, not by default. It is expected that the 'home' for data is well-defined and mapped between its geographical region and its usage. By exception and on-request, transactional data can be replicated to host regions to achieve efficient consumption.
6. Data is to be regionally bound and tagged. Data resides by default in the OSDU “home” region in which the loading/ingestion workflow occurred, which will be the region associated with the data from an earth location perspective. Since the describing metadata is replicated globally, any OSDU user can discern in which region the described data resides.
7. The OSDU DP Search API can be used to locate search results based on globally replicated master-data, reference-data, and metadata. As stated elsewhere in the requirements, metadata are not protected and a user accessing any OSDU regional deployment can see all metadata. However, this is only true for metadata and of course not for the data content files / datasets / etc. (raw, processed, interpreted, etc.).
8. Facilities shall be provided to enable on-demand replication of content data files from ‘home’ region to a ‘host’ region’: selected data can be transported via request from its home region to any other OSDU region. Subsequent changes, that is, new items of data that serve as successor versions or new derivations, are populated to the home region and, if requested, to the same ‘host region’.
*Standout Non-functional Requirements (**TBD**)*
1. High-availability: Are OSDU services and data always available to end users, despite failure conditions?
2. Performance: Does the distributed solution meet the in-region performance goals for the data platform?
3. Latency: How much latency is there between the creation of content and it's availability across all regions?
4. Security: How much does the solution increase the threat surface of the OSDU data platform?
5. Ease of Use: How easy is it configure and operate?
6. Observability: How easy is it to test, identify, and resolve incorrect behavior?
7. Maintainability: How much does the solution introduce complexity in development, deployment and operations of the data platform?
**Additional Use Cases**
1. Automated replication of metadata across all regions.
(applies to: work-product-metadata, work-product-component-metadata, associated file metadata)
* "Easy" - leverage CSP storage technologies
* Leads to 'eventual consistency' of search result indices which crawl the metadata at different intervals
2. Master-Reference Data - also needs to be automatically replicated across all regions (to enable context and validation for transactional data.)
3. On-demand request for a WP located in a different region:
* Can be requested by a user via search UI or by a microservice via API
* WP is copied to the requestor region
- Subject to E&O contract + replication policy. If the replication policy states the data cannot travel to the requestor region, then user is notified - confirm?
- Metadata has a 'locator' indicating the WP's home region
- Lineage preserved, but it's not a deep copy of the references
- WPCs are 'deep copied' to the requestor region with read-only access
* Any derivative WPs made in the requestor region are loaded/ingested back in the 'home' region as identified in the metadata 'locator'
- Lineage is updated as usual
- WPs are loaded/ingested in the 'home' region
4. Repartitioning (example: grow from 2-regions to 3-regions) - �be able to reallocate home partition boundaries via 'staging'
* Creating a new region -- provides instant accessibility to the 'common' partition.
* Administrative functions to be able to move/copy a partition between regions. Some using companies may require the ability to replicate a single partition across several regions. Each region has a replica with only the data applicable to its users / assets (geographic area) and can transfer changes of this data to the 'home' region. This allows the 'home' region to perform analysis on data that is up to date across the entire extent.
**Architecture Principles / Guidelines**
1. Synchronous cross-regional calls should be avoided when possible. Applications should use regional resources.
2. Embrace asynchronous systems and replication - high availability / deferred consistency
3. Leverage the principles of Isolation and Redundancy: a failure of any kind in one Region should not affect services running in another.
**OSDU Regions**
An OSDU Region is an independent and complete deployment of all infrastructure and application services required to provide OSDU Data Platform (DP) services. Whilst every OSDU Region will have a homogeneous DP service deployment, a few types of OSDU regions can be identified:
- “*OSDU Standard Hosted Region*”: This is the common type of OSDU Region and will host all data platform services, including security, ingest, logging and monitoring. Platform, application and workflow services will be deployed based on need in this standard OSDU Region.
- “*OSDU Central Region*”: One of the standard OSDU regions will be assigned as a central region and will be hosting the development, integration test and performance test environments and Continuous Integration (CI)/ Continuous Delivery (CD) services. If required, it may host a central logging service. It will also host global OSDU application.
- “*OSDU Admin Region*”: One of the standard OSDU regions will be assigned as an admin region and this is a configured parameter in the OSDU metadata, namely the 'Admin Management' parameters. Subsurface Master Data System (SMDS) Master and Reference Data as well as OSDU management parameters stored in SMDS can only be updated in this configured OSDU Admin Region. All Admin APIs are available in all OSDU regions, though the call will be rejected (or redirected) if the API is not hosted in the OSDU Admin Region.
- “*OSDU In-Country Region*”: When public cloud infrastructure cannot be leveraged to deploy an OSDU Region for data sovereignty or latency reasons, an in-country OSDU deployment takes place with the consideration of incurring least management and operating overhead. Public cloud edge capabilities will be leveraged to most possible extent to increase the consistency and lower the overhead. The closest “OSDU Standard Region” will be configured as a proxy for in-country region's Subsurface Work Product Services (SWPS) metadata represent files within the in-country OSDU Region and providing home region replication.
List of OSDU Regions, and the identification of the Central and Admin OSDU Region will be part of the SMDS Master Data and will be administered as part of the Admin service.M1 - Release 0.1Stephen Whitley (Invited Expert)Hrvoje MarkovicFerris ArgyleRaj KannanDania Kodeih (Microsoft)JoeAlan DonigerBrian KirklandStephen Whitley (Invited Expert)2020-08-03https://community.opengroup.org/osdu/platform/home/-/issues/14[Platform] Support OSDU Global Deployment2020-06-18T18:25:21ZStephen Whitley (Invited Expert)[Platform] Support OSDU Global Deployment**Objective:**
- This requirement is to ensure OSDU data platform can support highly available multi-region/geographical deployment.
An OSDU Region is defined as an independent and highly available geographical deployment of the full O...**Objective:**
- This requirement is to ensure OSDU data platform can support highly available multi-region/geographical deployment.
An OSDU Region is defined as an independent and highly available geographical deployment of the full OSDU design. It provides full functionality and there is no need for deploying multiple OSDU regions for high availability or disaster recovery requirements. Strict latency, data sovereignty or local legislation requirements may result into the need for additional OSDU deployments. OSDU deployments are preferably public cloud based. In situation where in-country OSDU deployment is required and no public cloud region is available, physical edge capabilities from the public cloud provider will be used. In any case, the exposed OSDU APIs deployed in the different OSDU regions will provide identical functionality.
**OSDU Regions**
An OSDU Region is an independent and complete deployment of all infrastructure and application services required to provide OSDU Data Platform (DP) services. Whilst every OSDU Region will have a homogeneous DP service deployment, a few types of OSDU regions can be identified:
- “*OSDU Standard Hosted Region*”: This is the common type of OSDU Region and will host all data platform services, including security, ingest, logging and monitoring. Platform, application and workflow services will be deployed based on need in this standard OSDU Region.
- “*OSDU Central Region*”: One of the standard OSDU regions will be assigned as a central region and will be hosting the development, integration test and performance test environments and Continuous Integration (CI)/ Continuous Delivery (CD) services. If required, it may host a central logging service. It will also host global OSDU application.
- “*OSDU Admin Region*”: One of the standard OSDU regions will be assigned as an admin region and this is a configured parameter in the OSDU metadata, namely the 'Admin Management' parameters. Subsurface Master Data System (SMDS) Master and Reference Data as well as OSDU management parameters stored in SMDS can only be updated in this configured OSDU Admin Region. All Admin APIs are available in all OSDU regions, though the call will be rejected (or redirected) if the API is not hosted in the OSDU Admin Region.
- “*OSDU In-Country Region*”: When public cloud infrastructure cannot be leveraged to deploy an OSDU Region for data sovereignty or latency reasons, an in-country OSDU deployment takes place with the consideration of incurring least management and operating overhead. Public cloud edge capabilities will be leveraged to most possible extent to increase the consistency and lower the overhead. The closest “OSDU Standard Region” will be configured as a proxy for in-country region's Subsurface Work Product Services (SWPS) metadata represent files within the in-country OSDU Region and providing home region replication.
List of OSDU Regions, and the identification of the Central and Admin OSDU Region will be part of the SMDS Master Data and will be administered as part of the Admin service.
M1 - Release 0.1Stephen Whitley (Invited Expert)Hrvoje MarkovicFerris ArgyleRaj KannanDania Kodeih (Microsoft)JoeStephen Whitley (Invited Expert)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/29SEGYExport won't export datasets where Dimensions_012 is unavailable2020-03-27T08:48:38ZMorten OfstadSEGYExport won't export datasets where Dimensions_012 is unavailableThis prevents SEGYExport from working correctly on 2D datasets or in other cases where Dimensions_01 is available but Dimensions_012 is not. It also won't work with crossline-oriented prestack or poststack seismic where the primary key c...This prevents SEGYExport from working correctly on 2D datasets or in other cases where Dimensions_01 is available but Dimensions_012 is not. It also won't work with crossline-oriented prestack or poststack seismic where the primary key corresponds to dimension 1 instead of dimension 2. We should identify which dimensions correspond to the sample dimension and always loop over the primary dimension. We should also read either from the 3D dimension group or the 2D dimension group that contains the first two dimensions of the 3D dimension group.Morten OfstadJørgen Lindjorgen.lind@3lc.aiMorten Ofstadhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/30SEGYExport doesn't work with other formats than IBM-float2020-03-27T12:41:37ZMorten OfstadSEGYExport doesn't work with other formats than IBM-floatThere are no checks to see what DataSampleFormat the original file used, and there are no checks for the format of the input VDS. The default should be to export as integer data when the VDS is U8 or U16 with bias corresponding to signed...There are no checks to see what DataSampleFormat the original file used, and there are no checks for the format of the input VDS. The default should be to export as integer data when the VDS is U8 or U16 with bias corresponding to signed integer, and to honor any DataSampleFormatCode metadata in the SEGY category and failing that we can peek at the original binary header to see if that indicates using IEEE.Morten OfstadMorten Ofstadhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/31AWS authentication fails with assumed roles2020-03-10T14:40:43ZMorten OfstadAWS authentication fails with assumed rolesThis is actually caused by an issue in the AWS C++ SDK where configurations with assumed roles are ignored by the DefaultCredentialsProviderChain: https://github.com/aws/aws-sdk-cpp/issues/150
However, we need to find a workaround as th...This is actually caused by an issue in the AWS C++ SDK where configurations with assumed roles are ignored by the DefaultCredentialsProviderChain: https://github.com/aws/aws-sdk-cpp/issues/150
However, we need to find a workaround as this was promised to be fixed "incredibly soon" back in 2016 and still isn't fixed.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/32SEGYImport should be able to import SEG-Y from S3 or Azure Blob Storage2020-03-10T10:26:24ZMorten OfstadSEGYImport should be able to import SEG-Y from S3 or Azure Blob StorageThis requires making it possible to instantiate and use an IOManager from a client of the OpenVDS API (in this case the SEGYImport command-line tool). It also requires adding command line options to provide the required OpenOptions to th...This requires making it possible to instantiate and use an IOManager from a client of the OpenVDS API (in this case the SEGYImport command-line tool). It also requires adding command line options to provide the required OpenOptions to that IOManager to the SEGYImport command and matching the input files to a pattern so s3:// is recognized as a file in an S3 Blob and getting some advice from Microsoft on how to refer to data on Azure Blob Storage.Version 1.0https://community.opengroup.org/osdu/data/open-test-data/-/issues/67[Open Test Data] OpenID Connect frontchannel logout does not work reliably2021-06-16T22:20:38ZSteven Reynolds[Open Test Data] OpenID Connect frontchannel logout does not work reliablyAfter experience supporting OSDU R1, we can see that OpenID Connect frontchannel logout does not work reliably. All of the service providers send an HTML header X-Frame-Options deny. However, the OpenID specifications recommend to use an...After experience supporting OSDU R1, we can see that OpenID Connect frontchannel logout does not work reliably. All of the service providers send an HTML header X-Frame-Options deny. However, the OpenID specifications recommend to use an iFrame when implementing the frontchannel logout.
So unfortunately, this is a conflict. The X-Frame-Options header of deny is meant to prevent code from using an iFrame.
For some cases this mostly works but with sporadic errors. In other cases this never works.https://community.opengroup.org/osdu/platform/system/storage/-/issues/3[Storage] Record Delete/Undelete2021-06-16T22:19:40ZAn Ngo[Storage] Record Delete/Undelete**Delete**: Current implementation of record soft delete is marking the current record version inactive.
**Undelete**: Currently, upserting a deleted record would re-activate it, hence undeleting it.
Both of these behaviors are not co...**Delete**: Current implementation of record soft delete is marking the current record version inactive.
**Undelete**: Currently, upserting a deleted record would re-activate it, hence undeleting it.
Both of these behaviors are not correct, and need to be handled better.M1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/issues/4[System/Storage]Publish event for record update2021-06-16T22:19:39ZAn Ngo[System/Storage]Publish event for record updateCurrently, updating a record also gets an "Op: create" event.
We need to send out "Op: Update" event for updating events, so consumer can differentiate between create and update events.Currently, updating a record also gets an "Op: create" event.
We need to send out "Op: Update" event for updating events, so consumer can differentiate between create and update events.M1 - Release 0.1https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/issues/3[Wellbore DDMS] Wellbore data object2022-08-23T10:47:24ZDror Dahan[Wellbore DDMS] Wellbore data object* Provide the schema for the Wellbore data object.
* Provide Create, Read, Update, Delete API's for the wellbore data object
* Provide an API to get a wellbore by version* Provide the schema for the Wellbore data object.
* Provide Create, Read, Update, Delete API's for the wellbore data object
* Provide an API to get a wellbore by versionM1 - Release 0.1Dror DahanDror Dahanhttps://community.opengroup.org/osdu/platform/home/-/issues/15Wellbore data object2020-03-25T12:49:43ZDror DahanWellbore data objectProvide CRU APIs for the Wellbore data object Provide CRU APIs for the Wellbore data object M1 - Release 0.1Dror DahanDror Dahan2020-06-01https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/issues/2[Wellbore DDMS] Logset Data object APIs2022-08-23T10:47:23ZDror Dahan[Wellbore DDMS] Logset Data object APIs* Provide schema for the Logset data object
* Provide Create, Read, Update, Delete API's for the Logset data object
* Provide an API to get a Logset by version* Provide schema for the Logset data object
* Provide Create, Read, Update, Delete API's for the Logset data object
* Provide an API to get a Logset by versionM1 - Release 0.1Dror DahanDror Dahanhttps://community.opengroup.org/osdu/platform/home/-/issues/16Logset Data object2020-03-25T12:49:23ZDror DahanLogset Data objectProvide CRU APIs to manipulate the Logset data objectProvide CRU APIs to manipulate the Logset data objectM1 - Release 0.1Dror DahanDror Dahan2020-06-01https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/issues/1[Wellbore DDMS] Log Data object2022-08-23T10:47:23ZDror Dahan[Wellbore DDMS] Log Data object* Provide the schema for the Log data object
* Provide Create, Read, Update, Delete API's for the Log object
* Provide Create, Update, read API's for the bulk of a Log object
* For large Logs (where bulk data is larger than 1MB), provide...* Provide the schema for the Log data object
* Provide Create, Read, Update, Delete API's for the Log object
* Provide Create, Update, read API's for the bulk of a Log object
* For large Logs (where bulk data is larger than 1MB), provide the capacity to get the log decimated (for quick data access).M1 - Release 0.1Dror DahanDror Dahanhttps://community.opengroup.org/osdu/platform/home/-/issues/17Log Data object2020-03-25T12:48:23ZDror DahanLog Data objectProvide various APIs to manipulate Log data:
The specific APIs will be listed shortly.
Provide various APIs to manipulate Log data:
The specific APIs will be listed shortly.
M1 - Release 0.1Dror DahanDror Dahan2020-06-01https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/22Orchestration and Ingestion Services and API specification2022-06-28T19:39:46ZFerris ArgyleOrchestration and Ingestion Services and API specificationOpenDES requires an ingestion service to ingest data and enrich its metadata. The [Orchestration and Ingestion flow](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-(C)/Design-and-Implementation/Ingestion-and-Enrichment-...OpenDES requires an ingestion service to ingest data and enrich its metadata. The [Orchestration and Ingestion flow](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-(C)/Design-and-Implementation/Ingestion-and-Enrichment-Detail/R2-Ingestion-Workflow-Orchestration-non-Spike):
* Provides an ingestion flow which unifies OSDU R1 and DELFI Data Ecosystem requirements
* Refactors the orchestration policy from the DELFI Data Ecosystem application code into a workflow engine
## Status
- [ ] Proposed
- [ ] Trialing
- [X] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
Current DELFI orchestration is explicit: each service publishes an event, which is expected by the next service
* There’s no way to inject custom logic into the process
* There’s no way to customize existing process for particular client/data type
We need a mechanism to support agile ingestion workflow development which supports different data types and enrichment requirements.
The scope of this decision is to agree on the [services and their APIs](/osdu/documentation/-/wikis/OSDU-(C)/Roadmap/OSDU-Release-2-Details/R2-Orchestration-and-Ingestion-non-Spike)
## Decision
## Rationale
## Consequences
## When to revisit
---
# Tradeoff Analysis - Input to decision
Without these services and APIs, there isn't an API, nor a standardized mechanism to ingest data; instead there is a collection of ad-hoc approaches and scripts.
Some operators which to test OSDU by ingesting data from a number of systems, and then consuming it in others. Half of this use case is unaddressed without an Ingestion flow.
## Alternatives and implications
* The [Orchestration and Ingestion flow](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-(C)/Design-and-Implementation/Ingestion-and-Enrichment-Detail/R2-Ingestion-Workflow-Orchestration-non-Spike) proposes and trials an approach.
* No alternatives have been proposed.
## Decision criteria and tradeoffs
## Decision timeline
R2Ferris ArgyleFerris Argylehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/33Parallel write support and example2024-03-07T09:17:38ZMorten OfstadParallel write support and exampleFor R3, the HPC group is interested in developing an example of how to do parallel write (e.g. using MPI). This probably requires adding some support functions to make it easier to send metadata to a central coordinator and write the chu...For R3, the HPC group is interested in developing an example of how to do parallel write (e.g. using MPI). This probably requires adding some support functions to make it easier to send metadata to a central coordinator and write the chunk-metadata pages separately.Morten OfstadMorten Ofstadhttps://community.opengroup.org/osdu/platform/system/search-service/-/issues/11Search - Map aggregate functionality (R2)2021-06-25T12:42:58ZFerris ArgyleSearch - Map aggregate functionality (R2)## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
* Displays wells on map to ensure return result is as soon as possible; restricts # of fields and wells. Allows returning well...## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
* Displays wells on map to ensure return result is as soon as possible; restricts # of fields and wells. Allows returning wells based on searches of child data.
* If query is geographic boundary and otherwise minimal, return all wells within that boundary: three fields (well name, unique id, …?)
* This functionality is a popular request, but implemented as special case in the data platform; not clear whether it can be generalized - Alan
## Decision
Retain map aggregate functionality through a more generalized mechanism than R1
* Replace with injection of Elastic spatial filter
## Rationale
Supports ISVs required functionality in a way that's not over-fitted for INT and SSIO
## Consequences
INT and SSIO need to re-work R1 implementations.
Implementation Tasks:
* Gap fit on this use case
* Test according to the definition of done (Write test cases)
* Add a user story to the project ADO
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
* Retain R1 map aggregate functionality: no porting required, but overfits for INT and SSIO
* Retain map aggregate functionality through a more generalized mechanism: supports ISVs required functionality in a way that's not over-fitted for INT and SSIO
* Deprecate map aggregate functionality: simplifies OSDU but causes excessive burden on ISV client and networking
## Decision criteria and tradeoffs
## Decision timelineethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/search-service/-/issues/12Search - Tree walking2021-06-25T12:43:55ZFerris ArgyleSearch - Tree walking## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
Want to be able to find all the children of a well in a single query: well bores, trajectories, etc.
Have an UI area on left-ha...## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
Want to be able to find all the children of a well in a single query: well bores, trajectories, etc.
Have an UI area on left-hand side which has a data tree, populate in lazy manner: once someone clicks on item in tree, load those
## Decision
Retain R1 tree-walking in a lazy fashion
## Rationale
Supports ISVs currently implemented functionality without undue burden on service.
## Consequences
How to address potential large scale of children? - Raj
* Could perhaps just return children unique IDs in response…
Implementation Tasks:
* Gap fit on this use case
* Test according to the definition of done (Write test cases)
* Add a user story to the project ADO
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timeline