OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2021-06-16T22:17:32Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/68Backup and Restore support2021-06-16T22:17:32ZKateryna Kurach (EPAM)Backup and Restore supporthttps://community.opengroup.org/osdu/platform/system/reference/unit-service/-/issues/14Unit Service Routes fail when unit measure has '/'2020-10-01T19:40:59ZMatt WiseUnit Service Routes fail when unit measure has '/'When using routes with inline parameters, slash ("/") is treated as an path terminator (even escaped) and therefore any units of measure such as ft/s cause the unit service to generate 400 errors.
The integration tests are failing for t...When using routes with inline parameters, slash ("/") is treated as an path terminator (even escaped) and therefore any units of measure such as ft/s cause the unit service to generate 400 errors.
The integration tests are failing for the service as they tests conversion of ft/s to m/s.
**Example: abcd unit conversion route failure**
The following integration test tries to convert ft/s to m/s:
```python
def test_get_conversion_as_abcd_by_namespace_and_symbols(self):
"""Test get_conversion_as_abcd_by_namespace_and_symbols"""
try:
api_response = self.api_instance.get_conversion_as_abcd_by_namespace_and_symbols(
namespaces='Energistics_UoM,RP66', from_symbol='ft/s', to_symbol='m/s',
data_partition_id=self.env.data_partition_id)
self.assertIsNotNone(api_response)
self.assertIsInstance(api_response, ConversionResult)
self.assertIsNotNone(api_response.abcd)
self.assertIsNone(api_response.scale_offset)
self.assertEqual(0.0, api_response.abcd.a)
self.assertEqual(0.3048, api_response.abcd.b)
self.assertEqual(1.0, api_response.abcd.c)
self.assertEqual(0.0, api_response.abcd.d)
except ApiException as e:
self.fail(str(e))
```
The following route is expected to be called by the integration test:
```java
@GetMapping("/conversion/abcd/{namespaces}/{fromSymbol}/{toSymbol}")
public ConversionResult getConversionABCDBySymbols(@PathVariable("namespaces") String namespaces,
@PathVariable("fromSymbol") String fromSymbol,
@PathVariable("toSymbol") String toSymbol) {
try {
return catalog.getConversionABCDBySymbols(namespaces, fromSymbol, toSymbol);
}
catch (Exception ex) {
throw AppException.createBadRequest(ex.getMessage());
}
}
```
The integration test result is as follows:
```
======================================================================
ERROR [0.006s]: test_get_conversion_as_abcd_by_namespace_and_symbols (unit_test_core.test_unit_service_v2.TestConversions)
Test get_conversion_as_abcd_by_namespace_and_symbols
----------------------------------------------------------------------
Traceback (most recent call last):
File "../unit_test_core/test_unit_service_v2.py", line 122, in test_get_conversion_as_abcd_by_namespace_and_symbols
api_response = self.api_instance.get_conversion_as_abcd_by_namespace_and_symbols(
unit_test_core.v2.swagger_client.rest.ApiException: (400)
Reason:
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/html;charset=utf-8', 'Content-Language': 'en', 'Content-Length': '435', 'Date': 'Tue, 01 Sep 2020 16:03:40 GMT', 'Connection': 'close'})
HTTP response body: <!doctype html><html lang="en"><head><title>HTTP Status 400 – Bad Request</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 400 – Bad Request</h1></body></html>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "../unit_test_core/test_unit_service_v2.py", line 134, in test_get_conversion_as_abcd_by_namespace_and_symbols
self.fail(str(e))
AssertionError: (400)
Reason:
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/html;charset=utf-8', 'Content-Language': 'en', 'Content-Length': '435', 'Date': 'Tue, 01 Sep 2020 16:03:40 GMT', 'Connection': 'close'})
HTTP response body: <!doctype html><html lang="en"><head><title>HTTP Status 400 – Bad Request</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 400 – Bad Request</h1></body></html>
```
I also tested the route via postman and saw the same result:
Postman Call:
![image](/uploads/0235b43793c8b5d9fc65477fb1173ce3/image.png)
The route does work when calling it with a non / unit:
![image](/uploads/b6fd7ea5748430835194cce61ecc0029/image.png)M1 - Release 0.1ethiraj krishnamanaiduJoeethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/6AWS Implementation of Energistics Parsers2021-07-28T13:30:51ZAsh SathyaseelanAWS Implementation of Energistics ParsersSample issueSample issueM7 - Release 0.10GregGreg2021-04-09https://community.opengroup.org/osdu/platform/system/file/-/issues/10Add driver field in /getLocation API response2022-09-27T11:43:47ZWei SunAdd driver field in /getLocation API responseIn current File service API design, the getLocation API is used to return signed URL only, but the backend storage providers have some http headers to optimize the data operation. I suggest to add new field for driver in getLocation resp...In current File service API design, the getLocation API is used to return signed URL only, but the backend storage providers have some http headers to optimize the data operation. I suggest to add new field for driver in getLocation response to enable client applications to upload file to the signed URL with optimized way.
origin:
```json
{
"FileID": "file ID",
"Location": {
"SignedURL": "GCS signed URL"
}
}
```
change to:
```json
{
"FileID": "file ID",
"Location": {
"SignedURL": "GCS signed URL"
"Driver": "GCS"
}
}
```https://community.opengroup.org/osdu/ui/data-loading/bulk-loader/-/issues/1How to load Missing Manifests using bulk-loader scripts2021-06-16T22:17:25ZsrinivasHow to load Missing Manifests using bulk-loader scriptsHi,
When Ingesting sample dataset using Bulk-loader scripts, some manifests are missing.
Ex: Total Trajectories are 5944, but few documents are not loaded in ElasticSearch. Please refer below:
yellow open opendes-osdu-wellboretrajector...Hi,
When Ingesting sample dataset using Bulk-loader scripts, some manifests are missing.
Ex: Total Trajectories are 5944, but few documents are not loaded in ElasticSearch. Please refer below:
yellow open opendes-osdu-wellboretrajectory-wp-0.2.0 UhWBAa4oRRy4C7DGNsiuTg 1 1 **5943 **0 1.9mb 1.9mb
yellow open opendes-osdu-wellboretrajectory-wpc-0.2.0 Fod9BmhHQvyIwmUHi7EKcA 1 1 **5942 **0 2.2mb 2.2mb
How to find missing manifest and load the same?https://community.opengroup.org/osdu/platform/system/file/-/issues/11ADR Master and Reference Schema versioning; SRN format2023-07-05T10:09:40ZKateryna Kurach (EPAM)ADR Master and Reference Schema versioning; SRN format### Change Type:
- [X] Feature
- [ ] Bugfix
- [ ] Refactoring
### Context and Scope
## 1. Reference and Master Data Schema version format
Different aspects of OSDU Reference Schemas (RS), Reference OSDU Resources populated with speci...### Change Type:
- [X] Feature
- [ ] Bugfix
- [ ] Refactoring
### Context and Scope
## 1. Reference and Master Data Schema version format
Different aspects of OSDU Reference Schemas (RS), Reference OSDU Resources populated with specific Reference Value Lists, and other OSDU schemas can change with time. It was discussed on the Data Definitions team and Reference Data Ingestion meetings that there are requirements to track these different categories of change/versioning. Many of the identified categories are below. We have added other versioning categories and clarifications as well.
1.1 For any OSDU schema, capture:
- **Schema version** - Describes the version of the Schema structure. Usually a new schema structure version will be delivered together with a new OSDU release, but minor schema versions may also be released (e.g. a schema change that simply adds a property (which is a non-breaking change)).
*Question: Is governance established that the schema version will be tracked by the schema name, or was this a temporary solution by Thomas Gehrmann? Is this documented somewhere? If not, then OSDU needs to establish the proper governance on this and document it.*
*Question: Are we capturing minor and major schema changes? If yes, how is each defined?*
- **Resource version** - Data change within the same schema version. The schema/structure itself didn't change, but a new version of the Resource was added to OSDU (e.g. because one or more property values needed to be updated). For most schemas, it is understood that data change simply creates a new Resource, with incremented version, with the different data values. However, this concept deserves special attention with Reference Data Values/Lists since changing some Reference Data Values can sometimes have massive and breaking data management consequences (e.g. Reference Lists classified by the DD&M subcommittee as “fixed” are defined by OSDU. This exact list is critical either to system functionality or to industry interoperability).
This version number must be incremented regardless of what the reason was for any change to the contents of the data, including the categories below in the Reference Values section.
- **Source** –Uniquely describes the system and/or organization from which this data object comes. Many different source-versions can attempt to identify the same real-world object (such as Wellbore) or activity (such as Production Volume reporting). (For a Wellbore, for example, this would be similar to PPDM’s WELL_VERSION.)
Ideally, we could track:
- Source to my organization (value would capture an outside organization) “data.DataSourceOrganisationID” property?
- Source system/application/database “source” property?
*Notes: This identifies a version data that attempts to define a real-world object or measurement, not a version of a data object that would need to be numerically incremented like the other version categories here.*
1.2 For Reference Values:
- **Reference Value data changes** – In addition to the general “version” resource property, the following properties are needed to better govern Reference Value lists:
- OSDU-governed: You might create a new version because of an OSDU-governed change to a reference list. The OSDU Reference List version must be captured, and incremented whenever an updated OSDU-governed list is published and subsequently used in a Reference Data resource. This applies to the OSDU-governed reference values in an “open” list, and to “fixed” governance categories of reference value lists, as determined by the OSDU Reference Values team. A way to capture this does not exist yet.
- Locally governed: You might create a new version because a governed Reference List for a particular implementation was updated, like at an operator (i.e. “open” and “local” reference list category). The locally-governed Reference List version must be captured and incremented whenever the local data governance group publishes and the list is susbsequently used in a Reference Data resource. This applies to the reference values in an “open” list, and to “local” governance categories of reference value lists, as determined by the OSDU Reference Values team. A way to capture this does not exist yet.
- **Attribution Authority**: For any reference value or reference list, those values and descriptions may have been created by OSDU or by an outside organization (such as PPDM or Energistics). Both OSDU and outside standards may change over time, so it is critical to capture both the source organization and the publication version of those outside standards used. This is already accommodated by the Attribution Authority, Publication, and Revision properties which are standard Reference Resource properties.
*Note: this is different than the “OSDU-governed” versioning category mentioned above. The OSDU-governed versioning category refers to a complete list of Reference Values for a particular reference object. The attribution authority is captured to each value individually. In other words, an OSDU-governed reference list could potentially include some values created by OSDU attribution authority and some from an outside attribution authority, but the list as a whole will be considered “OSDU-governed”.*
Summary: OSDU should establish clear governance to appropriately and consistently track these categories of versioning:
For any resource:
- Schema Version (might exist in schema name format; needs confirmation)
- Resource Version (exists)
Additional to Reference List resources:
- OSDU-governed list version (does not exist)
- Locally-governed list version (does not exist)
- Attribution Authority + Publication + Revision (exists)
The best solution would be to create appropriate properties for the version categories that do not yet exist.
In addition, OSDU should also capture the OSDU governance category of Reference Value Lists within the reference schema and resource itself: “Fixed”, “Open”, or “Local”. A way to capture this does not exist yet.
## 2. SRN format
Also, decision has to be taken regarding SRN format. It must be decided whether it has to contain corresponding schema version or not. Currently SRN doesn't contain a version (e.g. "srn:<namespace>:reference-data/VerticalCRS:MSL:").
*Note: Tentatively, we think that capturing Schema Version + Resource Version in the schema name would uniquely identify resource referenced (like a foreign key).*
For reference lists, you want to be able to identify the specific version of the reference list that a WPC (e.g.) references.
However, for a WPC (Marker, e.g.) to reference a parent Master object (Wellbore, e.g.), it doesn’t need to reference a specific point-in-time version of it; It should reference the most recent version.
If this is true, SRNs for Reference Data would need to include Schema + Resource Version in the SRN, but SRN would be more generic for all other group types.
Problem: SRN identity is uncertain.
A. Is SRN intended to uniquely define the physical real-world object in the case of Master Data (like a Wellbore)? If yes, then SRN should not contain version for Master Data references.
B. Or is SRN intended to uniquely define a data record with its version (like a GUID)? If yes, then Master Data Version should be included in the SRN.
It should not be used for both, but both must be accommodated by OSDU.
Some additional condiderations:
A. Version is NOT included in an SRN.
Pros:
- It simplifies end-user aggregation of data to a single parent record. Your WPCs, created at different times will be referencing the same Master data record, not a point-in-time older version of that Master record. Existing WPC are always in the "current" state and users do not have to enrich and create a new version of WPC each time corresponding RS or Master Data Schema changes.
Cons:
- It leaves the question open as to how you could have different Wellbore Versions (similar to WELL_VERSION in PPDM). It seems that this is not currently supported by the OSDU canonical schemas, but is a real use case – similar to the way you can have different versions of Trajectories in WPC.
- You can loose aspects of historical parent-child relationships/data lineage. For example, a Trajectory might have TVD calculated based on the “active” elevation of a particular Wellbore resource version. Then that Wellbore gets updated, and the newest version of that Wellbore record has a different active elevation type or active elevation value. Now the Trajectory file is out-of-sync in this regard with its parent Wellbore from that point-in-time.
B. Version is included in an SRN.
Pros:
- It it potentially allows you to have different Wellbore Versions (using UWI and Source, for example, as the natural key)
- Traceability and lineage of the data
Cons:
- Raises the question of how to uniquely identify the one physical wellbore, or the “gold” Wellbore record (similar to WELL in PPDM)
- Complexities with updating existing WPCs that have links to older versions of MDS. End-user aggregations can be disaligned if there are WPCs in the system that are linked to different schema versions.
- Another consideration is related to possible future search complexity. If SRN value changes, some WPC could be found using "new" SRN value and some WPCs should be found using "old" SRN value.
Users will have to implement additional enrichment workflows to solve the issues related to SRN version descrepancies (and probably develop some functions that will detect all "outdated" SRN links). That leads to high usage of computing resources (e.g. to change all WPC SRNs to point to the new version etc).
### Decision
- There is a requirement to track Schemas versioning. Decision has to be taken on the Schema version format (especially for Reference Schemas)
- Decision has to be taken on the SRN format: will it contain Schema version or not ("srn:<namespace>:reference-data/VerticalCRS:MSL:" VS "srn:<namespace>:reference-data/VerticalCRS.1.0.0:MSL:"
### Rationale
### Consequence
- No consequences for CSPs
- Consequences for majority of the OSDU services. Change in the Schema definition will lead to the change in the Manifest creation process as well as in Enrichment and Delivery API.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/63SEGYExport failing with "Inconsistent metadata" error2020-09-04T09:05:41ZsrinivasSEGYExport failing with "Inconsistent metadata" error1. SEGYImport, with headers option, created a VDS file
2. SEGYExport, is failing to create SEGY file using the VDS file created in step#1
Attached screen-shot for commands and Error:
![SegyExport](/uploads/854506a504936134a49736476cc18d...1. SEGYImport, with headers option, created a VDS file
2. SEGYExport, is failing to create SEGY file using the VDS file created in step#1
Attached screen-shot for commands and Error:
![SegyExport](/uploads/854506a504936134a49736476cc18d7e/SegyExport.png)https://community.opengroup.org/osdu/platform/system/storage/-/issues/31[Storage Service] Integration tester ACL hardcoded for the bulk-acl integrati...2021-06-16T22:19:35ZRucha Deshpande[Storage Service] Integration tester ACL hardcoded for the bulk-acl integration testThe integration tester ACL used in the new bulk acl integration test is hardcoded in org.opengroup.osdu.storage.util\TestUtils.java.
This assumes the integration test user is already in that entitlements group.
` public static final S...The integration tester ACL used in the new bulk acl integration test is hardcoded in org.opengroup.osdu.storage.util\TestUtils.java.
This assumes the integration test user is already in that entitlements group.
` public static final String getIntegrationTesterAcl() {
return String.format("data.integration.test@%s", getAclSuffix());
}
`
An environment variable should be used instead.M1 - Release 0.1ethiraj krishnamanaiduJoeRucha DeshpandeMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/32[Storage Service] Bulk ACL Integration tests do not cover revertObjectMetadat...2021-06-16T22:19:34ZRucha Deshpande[Storage Service] Bulk ACL Integration tests do not cover revertObjectMetadata functionalityThe Bulk ACL integration tests do not cover the revertObjectMetadata functionality.The Bulk ACL integration tests do not cover the revertObjectMetadata functionality.M1 - Release 0.1ethiraj krishnamanaiduJoeRucha DeshpandeMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/6shouldParseDateInOldFormat unit test fails in LegalTagSerializationTest2021-06-16T22:19:54ZChristian LecknershouldParseDateInOldFormat unit test fails in LegalTagSerializationTestFailed tests:
shouldParseDateInOldFormat(org.opengroup.osdu.core.common.model.legal.LegalTagSerializationTest)
Tests run: 149, Failures: 1, Errors: 0, Skipped: 2
Failed since it expected the date to be 2019-10-11. I changed it locall...Failed tests:
shouldParseDateInOldFormat(org.opengroup.osdu.core.common.model.legal.LegalTagSerializationTest)
Tests run: 149, Failures: 1, Errors: 0, Skipped: 2
Failed since it expected the date to be 2019-10-11. I changed it locally to 2019-10-11 in my branch to get it to pass. However, this should be looked at
@Test
public void shouldParseDateInOldFormat() throws IOException {
final String input = "{\"id\":123,\"name\":\"name\",\"description\":\"desc\"," +
"\"properties\":{\"countryOfOrigin\":[\"US\",\"BY\"],\"contractId\":\"contrId\"," +
"\"expirationDate\": 1570863500000,\"originator\":\"company\"," +
"\"dataType\":\"dataType\",\"securityClassification\":\"securityClassification\"," +
"\"personalData\":\"data\",\"exportClassification\":\"exportClassification\"}," +
"\"isValid\":false}";
LegalTag legalTag = objectMapper.readValue(input, LegalTag.class);
Assert.assertEquals(Date.valueOf("2019-10-12").toLocalDate(), legalTag.getProperties().getExpirationDate().toLocalDate());
}https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/69Enhanced API Specs for Workflow Service2021-01-15T21:31:44ZSwapnilEnhanced API Specs for Workflow ServiceThis ADR is version 1 for Workflow Service R3 open specification.
Proposed changes are required for CSV Ingestor Horizon 1 :
- https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/5
- https:/...This ADR is version 1 for Workflow Service R3 open specification.
Proposed changes are required for CSV Ingestor Horizon 1 :
- https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/5
- https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/4
Story related to utilization of the changes made by proposal:
- https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/8
At high level, the proposal talks about:
- More aligned entity definitions for Ingestion Workflow Service R2.Proposal is to define 2 entities - workflow and workflow run and APIs defining CRUD operations against those entities.
- Introduction of one new API i.e POST /workflow to register a dag with standard operators.
We are proposing the first set of APIs and we agree that there are many more APIs needed to support future requirements like composability and reusability of the workflow components. These requirements are out of the scope of this ADR and subsequent ADRs will be created for them.
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
In OSDU R2, there is Ingestion Workflow Service(https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow) for orchestration tool( airflow) specific managerial operations.
The API specs are tightly coupled with Airflow's way of doing things. We need a more generic entity defining - workflow and workflow run.Also, R2 version does not have the capability to register a dag with standard operators.
## Decision
Align on the first set of OpenAPI specs for Workflow Service.
Considering the requirements for the OSDU R3 Ingestion framework we propose to introduce four APIs.
The functionality of three of the four APIs that we are proposing already exists in OSDU R2 and we intend to keep the functionality intact and just realign those APIs with Workflow and WorkflowRun entities. Also, fully decouple those APIs from the Airflow framework. We propose to provide continued support for OSDU R2 APIs.
The fourth new API that we are introducing provides an ability for a workflow provider to register a workflow (dag) with standard airflow operators.
Supported workflow operations by the new version are as follows:
CRUD operations for Workflow:
- Creation\registration of workflow. (new)
CRUD operations for Workflow Run:
- Triggering a workflow. (similar functionality exist in R2) - Similar API exists in R2 which has very limited scope i.e for each combination of the user, data, and workflow types, the API identifies a suitable DAG and then calls Airflow. Proposed API is more generic where we explicitly pass the workflowID to be triggered.
- Querying details of a workflow run. (similar functionality exist in R2)
- Updation attributes of workflow run(similar functionality exist in R2)
## Rationale
Workflow Service is a wrapper functionality on the orchestrator. We need to define entities which are not specific to one orchestrator. Also, add 1 API (create workflow) to support CSV Ingestor Horizon 1.
## Consequences
High-level alignment on the workflow and workflowRun entities enable us to incrementally add many more future requirements on the workflows and workflow runs. Also, the new proposed API provides an ability for a workflow provider to register a workflow (dag) with standard airflow operators.
# Tradeoff Analysis - Input to decision
The new proposed version will incorporate below user actions in true sense:
- Ability to register a workflow.
- Streamlined entity definitions.
This version of service will have no negative impact when compared to current version of Workflow Service in terms of:
- Performance
- Scalability
- Reliability
## Decision criteria and tradeoffs
- Usability of the APIs
- Extensibility of the APIs
- Cost of ImplementationJoeMatt WiseJoe2020-09-18https://community.opengroup.org/osdu/platform/system/home/-/issues/50Enhanced API Specs for Ingestion Workflow Service2020-09-07T18:38:17ZSwapnilEnhanced API Specs for Ingestion Workflow Service## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
In OSDU R2, there is Ingestion Workflow Service(https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-work...## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
In OSDU R2, there is Ingestion Workflow Service(https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow) for orchestration tool( airflow) specific managerial operations. In OSDU R3, the proposal is to have an enhanced version of Ingestion Workflow Service. Both serve a similar purpose i.e to provide a wrapper functionality for orchestrator tools and is designed to carry out CRUD operations of domain workflows and domain workflow runs. R2 version of the service talks about only 3 APIs – startWorkflow, updateWorfklowStatus, getStatus. R2 version of the service is tightly dependent on Apache Airflow as the orchestration tool. This ADR introduces an enhanced version of Ingestion Workflow Service API to cater to more complex workflow scenarios in Ingestion workflows. Also, the aim is to have an orchestrator tool independent specification.
## Decision
In OSDU R3 framework, Ingestion Workflow Service will be responsible for end to end management (creation, modification, execution and monitoring) of ingestion workflows from user perspective. This will become the way to create domain workflows in OSDU Data Platform. Users with workflow creation & triggers roles are completely abstracted from technical complexities in the orchestration tool used.
Supported workflow operations by the new version are as follows:
CRUD operations for Workflow:
- Creation\registration of workflow. (new)
- Updation\editing of workflow (new)
- Querying details of a workflow. (new)
- Listing all configured workflows. (new)
- Deletion of workflow. (new)
CRUD operations for Workflow Run:
- Triggering a workflow. (already exist)
- Querying details of all workflow runs for a workflow.
- Querying details of a workflow run. (already exist)
## Rationale
The Domain Workflow expectations from an orchestration tool are not supported out of the box by options available in the market. Different domain workflows have different expectations on case by case basis. It is important to have a wrapper layer which casts the behaviour of the orchestrator tool to a custom behaviour suited for domain workflows executions on Data Platform. This will also enable loose coupling of orchestrator tools (currently airflow) with Data Platform. Adoption of alternate orchestration tools will be better managed.
## Consequences
Data Platform's workflow expectations and orchestrator tool behaviour mismatch will be better managed. In case of modifications to Orchestrator tools, changes can be incorporated without multiple touch points spread across the Data Platform. Breaking changes while version upgrades (if any) and alternative tool implementation will be a controlled activity.
# Tradeoff Analysis - Input to decision
The new proposed version will incorporate below user actions in true sense:
- User is completely abstracted from the underlying orchestrator tool. Workflow editors can create and manage complex workflow without being technical experts on the orchestrator tool.
- OSDU data platform will be easily able to integrate future changes to Orchestrator framework. Technical changes will be within periphery of the Workflow Ingestion Service.
- Users can query into historical runs of the workflows, and the status of ingestion can be tracked at right granularity.This will end user domain workflow success\failure\in progress reporting.
- Users will have fine grained control over the attributes of workflow (for example – max concurrency, active\inactive features).
This version of service will have no negative impact when compared to current version of Ingestion Workflow Service in terms of:
- Performance
- Scalability
- Reliability
## Decision criteria and tradeoffs
- Usability
- Cost of Implementationhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/20Make default account id configurable for Schema service2020-09-22T03:33:05ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comMake default account id configurable for Schema service## Context and Scope
Hardcoded shared account id value as "common" in org.opengroup.osdu.schema.constants.SchemaConstants makes it difficult to run Schema service without a configured account. Currently, its presence is obligatory. Howev...## Context and Scope
Hardcoded shared account id value as "common" in org.opengroup.osdu.schema.constants.SchemaConstants makes it difficult to run Schema service without a configured account. Currently, its presence is obligatory. However, an absence of "common" leads to a lot of exceptions related to NPE, when Schema service is trying to reach storage services with a help of this account.
## Decision
move
public static final String ACCOUNT_ID_COMMON_PROJECT = "common"
to schema-core application.properties file
account.id.common.project = common
## Rational
Hardcoded account id won't allow us to run Schema service without "common" account.
## Consequences
Use a spring annotation "@value" where account id is required instead of SchemaConstants.ACCOUNT_ID_COMMON_PROJECT.
Providers should change SchemaConstants.ACCOUNT_ID_COMMON_PROJECT to new value from properties file where they need to use it.Dmitriy RudkoDmitriy Rudko2020-09-22https://community.opengroup.org/osdu/platform/system/partition/-/issues/1Partition API routes are missing standard OSDU Roles2023-09-27T12:19:38ZMatt WisePartition API routes are missing standard OSDU RolesRoutes in the partition service use a custom authorization filter that does not use Roles, but rather hardcodes a specific account type 'DomainAdminServiceAccount'.
pre-auth block in API methods:
```java
@PreAuthorize("@authorizationFil...Routes in the partition service use a custom authorization filter that does not use Roles, but rather hardcodes a specific account type 'DomainAdminServiceAccount'.
pre-auth block in API methods:
```java
@PreAuthorize("@authorizationFilter.hasPermissions()")
```
Authorization filter has the following function:
```java
public boolean hasPermissions() {
return authorizationService.isDomainAdminServiceAccount();
}
```
**Should the partition service use standard roles rather than user types to be compliant with other services?**
Other services use the following:
```java
@PreAuthorize("@authorizationFilter.hasRole('" + DeliveryRole.VIEWER + "')")
```M1 - Release 0.1ethiraj krishnamanaiduJoeMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/partition/-/issues/2Partition Service Should Support Updating Existing Secrets in a Partition2020-12-02T21:49:32ZMatt WisePartition Service Should Support Updating Existing Secrets in a Partition## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The Partition Service currently supports 3 API Operations.
1. Create Partition (POST)
2. Get Partition (GET)
3. Delete Partition...## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The Partition Service currently supports 3 API Operations.
1. Create Partition (POST)
2. Get Partition (GET)
3. Delete Partition (DELETE)
Currently, the only way to add new secrets or update existing ones to the KV store for a partition is to delete it and recreate it. This leads to extra API calls and risks missing putting back some data that was not meant to be touched.
## Decision
If this is implemented, KV stores will be simpler to update existing values on without having to delete existing data.
## Rationale
Partitions need to be updated with new secrets and edits to existing ones. In order to minimize the complexity of this operation for external callers, having a route to support this will simplify partition editing processes.
## Consequences
New API route in partition has to be supported/tested by all implementors of the serviceM1 - Release 0.1ethiraj krishnamanaiduDania Kodeih (Microsoft)JoeChris ZhangRucha DeshpandeMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/search-service/-/issues/24ADR Support self-signed certificates for elasticsearch2020-09-10T18:16:40ZRiabokon Stanislav(EPAM)[GCP]ADR Support self-signed certificates for elasticsearch## Context and Scope
Search Service does not support HTTPS connections with self-signed certificates for elastic search.
## Decision
Add a property 'security.https.certificate.trust' into application-*.properties.
If it is 'true', we ...## Context and Scope
Search Service does not support HTTPS connections with self-signed certificates for elastic search.
## Decision
Add a property 'security.https.certificate.trust' into application-*.properties.
If it is 'true', we will use `TrustSelfSignedStrategy()` in `org.opengroup.osdu.search.util.ElasticClientHandler` that trusts self-signed certificates.
## Rational
We would use self-signed certificates for elastic search instead of signed certificates.
## Consequences
These changes could affect all providers due to these things will be implemented in search-core.Dmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/partition/-/issues/3ADR Support self-signed certificates for elasticsearch2020-09-10T18:00:18ZRiabokon Stanislav(EPAM)[GCP]ADR Support self-signed certificates for elasticsearch## Context and Scope
Search Service does not support HTTPS connections with self-signed certificates for elastic search.
## Decision
Add a property 'security.https.certificate.trust' into application-*.properties.
If it is 'true', we ...## Context and Scope
Search Service does not support HTTPS connections with self-signed certificates for elastic search.
## Decision
Add a property 'security.https.certificate.trust' into application-*.properties.
If it is 'true', we will use `TrustSelfSignedStrategy()` in `org.opengroup.osdu.search.util.ElasticClientHandler` that trusts self-signed certificates.
## Rational
We would use self-signed certificates for elastic search instead of signed certificates.
## Consequences
These changes could affect all providers due to these things will be implemented in search-core.Dmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/search-service/-/issues/23ADR Support self-signed certificates for elasticsearch2020-09-10T17:57:27ZRiabokon Stanislav(EPAM)[GCP]ADR Support self-signed certificates for elasticsearch## Context and Scope
Search Service does not support HTTPS connections with self-signed certificates for elastic search.
## Decision
Add a property 'security.https.certificate.trust' into application-*.properties.
If it is 'true', we ...## Context and Scope
Search Service does not support HTTPS connections with self-signed certificates for elastic search.
## Decision
Add a property 'security.https.certificate.trust' into application-*.properties.
If it is 'true', we will use `TrustSelfSignedStrategy()` in `org.opengroup.osdu.search.util.ElasticClientHandler` that trusts self-signed certificates.
## Rational
We would use self-signed certificates for elastic search instead of signed certificates.
## Consequences
These changes could affect all providers due to these things will be implemented in search-core.Dmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/home/-/issues/52ADR Support self-signed certificates for elasticsearch2020-09-29T11:23:02ZRiabokon Stanislav(EPAM)[GCP]ADR Support self-signed certificates for elasticsearch## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
Search and Indexer Service does not support HTTPS connections with self-signed certificates (SSC) for Elastic search.
## Decision
Add a new property tha...## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
Search and Indexer Service does not support HTTPS connections with self-signed certificates (SSC) for Elastic search.
## Decision
Add a new property that will control trust/not-to-trust SSC.
- Module: search-core
- Affected Class: `org.opengroup.osdu.search.util.ElasticClientHandler`
- Property: `security.https.certificate.trust`
- Default value `false`
```java
public class ElasticClientHandler {
...
@Value("#{new Boolean('${security.https.certificate.trust:false}')}")
private Boolean securityHttpsCertificateTrust;
...
if ("https".equals(protocolScheme) && securityHttpsCertificateTrust) {
...
```
## Rational
To use self-signed certificates for Elastic on non-production environments.
## Consequences
No changes from other CSP will be required. Default strategy still will be `Not to trust SSC`.JoeDmitriy RudkoJoehttps://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/4ADR Support self-signed certificates for elasticsearch2020-09-10T17:58:47ZRiabokon Stanislav(EPAM)[GCP]ADR Support self-signed certificates for elasticsearch## Context and Scope
Indexer Service does not support HTTPS connections with self-signed certificates for elastic search.
## Decision
Add a property 'security.https.certificate.trust' into application-*.properties.
If it is 'true', we...## Context and Scope
Indexer Service does not support HTTPS connections with self-signed certificates for elastic search.
## Decision
Add a property 'security.https.certificate.trust' into application-*.properties.
If it is 'true', we will use `TrustSelfSignedStrategy()` in `org.opengroup.osdu.search.util.ElasticClientHandler` that trusts self-signed certificates.
## Rational
We would use self-signed certificates for elastic search instead of signed certificates.
## Consequences
These changes could affect all providers due to these things will be implemented in indexer-core.Dmitriy RudkoDmitriy Rudko