OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2023-09-26T14:57:13Zhttps://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/issues/41GET /api/legal/v1/jobs/updateLegalTagStatus timeout2023-09-26T14:57:13ZShane HutchinsGET /api/legal/v1/jobs/updateLegalTagStatus timeoutI'm getting a timeout from https://osdu.r3m18.preshiptesting.osdu.aws/api/legal/v1/jobs/updateLegalTagStatus
Run this curl command to reproduce this failure:
curl --connect-timeout 60 --max-time 60 -X GET -H 'Authorization: Bearer ...I'm getting a timeout from https://osdu.r3m18.preshiptesting.osdu.aws/api/legal/v1/jobs/updateLegalTagStatus
Run this curl command to reproduce this failure:
curl --connect-timeout 60 --max-time 60 -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/legal/v1/jobs/updateLegalTagStatus
GET /api/legal/v1/jobs/updateLegalTagStatus on Azure
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: opendes' https://osdu-ship.msft-osdu-test.org/api/legal/v1/jobs/updateLegalTagStatushttps://community.opengroup.org/osdu/platform/system/register/-/issues/45M18 GET and DELETE /api/register/v1/action/{id}2023-06-15T13:35:40ZShane HutchinsM18 GET and DELETE /api/register/v1/action/{id}Received a response with 5xx status code: 500
{"timestamp":"2023-06-14T14:29:55.655+00:00","status":500,"error":"Internal Server Error","path":"/api/register/v1/action/00%3B"}
Expected a 404 error or other 4xx.
Run this get curl comman...Received a response with 5xx status code: 500
{"timestamp":"2023-06-14T14:29:55.655+00:00","status":500,"error":"Internal Server Error","path":"/api/register/v1/action/00%3B"}
Expected a 404 error or other 4xx.
Run this get curl command to reproduce this failure:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/register/v1/action/00%3B
Run this delete curl command to reproduce this failure:
curl -X DELETE -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/register/v1/action/00%25
I was able to reproduce this on AWS and Azure.
DELETE /api/register/v1/action/{id}
curl -X DELETE -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: opendes' https://osdu-ship.msft-osdu-test.org/api/register/v1/action/00%25
GET /api/register/v1/action/{id}
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: opendes' https://osdu-ship.msft-osdu-test.org/api/register/v1/action/00%3Bhttps://community.opengroup.org/osdu/platform/system/register/-/issues/44M18 GET and DELETE /api/register/v1/ddms/{id}2023-06-15T14:01:56ZShane HutchinsM18 GET and DELETE /api/register/v1/ddms/{id}Received a response with 5xx status code: 500
{"timestamp":"2023-06-14T14:29:41.188+00:00","status":500,"error":"Internal Server Error","path":"/api/register/v1/ddms/00%3B"}
This should have returned a 404 or maybe a 401, but not 500.
...Received a response with 5xx status code: 500
{"timestamp":"2023-06-14T14:29:41.188+00:00","status":500,"error":"Internal Server Error","path":"/api/register/v1/ddms/00%3B"}
This should have returned a 404 or maybe a 401, but not 500.
Run this get curl command to reproduce this failure:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/register/v1/ddms/00%3B
Run this delete curl command to reproduce this failure:
curl -X DELETE -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/register/v1/ddms/00%0A
Was able to produce this issue in AWS and Azure.
GET /api/register/v1/ddms/{id}
Run this curl command to reproduce this failure:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: opendes' https://osdu-ship.msft-osdu-test.org/api/register/v1/ddms/00%3B
curl -X DELETE -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: opendes' https://osdu-ship.msft-osdu-test.org/api/register/v1/ddms/00%0Ahttps://community.opengroup.org/osdu/platform/system/file/-/issues/85Duplicate PreloadFilePath value on ingestion of File.Generics2023-06-16T13:17:04ZGorm-Erik AarsheimDuplicate PreloadFilePath value on ingestion of File.GenericsWhen ingesting File.Generics into OSDU, we are using the {OSDU_BASE_URL}/file/v2/files/metadata endpoint to create the File.Generic. In the body we are setting the PreloadFilePath value on data.DatasetProperties.FileSourceInfo.PreloadFil...When ingesting File.Generics into OSDU, we are using the {OSDU_BASE_URL}/file/v2/files/metadata endpoint to create the File.Generic. In the body we are setting the PreloadFilePath value on data.DatasetProperties.FileSourceInfo.PreloadFilePath as described in the schema. The file is created successfully but when we read out the record from Storage API there is a duplicate value for PreloadFilePath where it has both 'PreloadFilePath' with a capital 'P', and 'preloadFilePath' with a non-capital 'P'. We've tried to ingest a file with a non-capital 'preloadFilePath' as well, but it still comes out with both values when reading it back. We have also tried generating the File.Generic raw json object from our code and using this to create it directly from the metadata endpoint in Postman to disclose that our code is somehow meddeling with the object. This also comes out with the same result. It's also worth mentioning that we are running OSDU through Microsoft ADME.Om Prakash GuptaOm Prakash Guptahttps://community.opengroup.org/osdu/platform/system/home/-/issues/103ADR: Persisting / Querying Status messages2024-03-26T11:25:30Zdevesh bajpaiADR: Persisting / Querying Status messages# Decision Title
Persisting / Querying Status messages to do a post query or analysis on status of a workflow that is either executing or was executed in past.
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
-...# Decision Title
Persisting / Querying Status messages to do a post query or analysis on status of a workflow that is either executing or was executed in past.
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Overview
With ADR#80 only real time updates of status was possible via subscribing to events. However, there is no mechanism to do a post query or analysis on status of a workflow that is either executing or was executed in past. This is the problem statement the current ADR is trying to address, and this can be done by persisting the status events and exposing a type safe API on top of this persisted content.
## Context & Scope
ADR#80 talks about Global Status Monitoring framework that provide a mechanism to track status of data journey/dataflows on the data platform. More details can be found in here https://community.opengroup.org/osdu/platform/system/home/-/issues/80#
Data journey/Dataflows - A typical Dataflow can be expressed as shown
![Untitled_Diagram.drawio](/uploads/d524fc5901b602ff2e19bcfa45478124/Untitled_Diagram.drawio.png)
Dataflow could have millions of records spread across multiple datasets to ingest in Data Platform. Consumer of this status model should have way to decide
whether dataflow has finished or not
if it is Successful or Failed
if failed then why? what's the reason?
at what stage it is in currently?
The most important aspect of that ADR#80 was agreeing on the contract of status message. To ensure that every OSDU service emitting a status message abides by the contract and there is a standard data model. Please view raised MR for the same.
This MR defines two type of GSM status messages.
DataSet Details - Dataset pertains to any data, such as file, collection of files, etc. DatasetId is a metadata record id that is returned in the response by the Metadata API while the metadata record is created. User can use Dataset Id to find correlation id of workflows initiated and track their progress using status detail messages.
Status Details - Holds the status of multiple stages in a dataflow initiated.
As the decision of ADR it was agreed that every OSDU service will publish their status to message queue, against a CorrelationId. Consumers can simply subscribe to that message queue or notification service to get events of status change for that CorrelationId. All our OSDU services are making use of CorrelationId and propagate the same for further REST calls. We leverage CorrelationId for tying all related status changed notifications.
In the same ADR it was also mentioned that, Status data processor service can be built to listen to these status changed events and put into persistent store to make them accessible for future references using querying capabilities.
## Solution
This ADR talks about contributing two services to OSDU community.
[Status Collector Service](https://community.opengroup.org/osdu/platform/system/status-collector)
Status Collector is an internal service (not exposed for external calls) and reacts to status messages that are published to status message queue. It picks up messages published to status message queue from every stage(OSDU Service) of the process and normalizes them to store in persistent storage for future reference.
[Status Processor Service](https://community.opengroup.org/osdu/platform/system/status-processor)
Status Processor Service provides APIs that allow users to query persisted status with multiple filters like correlationId, recordId, stage and status.
![collector-Page-2.drawio](/uploads/cb271b507c45834220b9d7ba32fb43ab/collector-Page-2.drawio.png)
Status Processor endpoints
The Status Processor service will support the operations listed below via different endpoints:
Query Dataset details
![image-2023-5-28_17-10-5](/uploads/a4adf3a935c1c1439bafb5f51958fab8/image-2023-5-28_17-10-5.png)
![image-2023-5-28_17-10-53](/uploads/fe17ce5edcaf27ac61456b984bfeb1bf/image-2023-5-28_17-10-53.png)
Query Status
![worddavf79f5c218b10163e8f8a12f5087a2ac1](/uploads/fd3fd8e4e608fcff39d53354a16933ef/worddavf79f5c218b10163e8f8a12f5087a2ac1.png)
![worddava18d304169b02aa8fca8394959add2f3](/uploads/ead6c6245f9bbd047f462d789c58190d/worddava18d304169b02aa8fca8394959add2f3.png)
## Decision
Implement services to persist and query GSM messages.
## Rationale
At present only real time updates of status was possible via subscribing to events. However, there is no mechanism to do a post query or analysis on status of a workflow that is either executing or was executed in past.
## Consequences
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelinedevesh bajpaidevesh bajpaihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/28ADR: EDS DMS - Bulk Data Ingestion and Dataset ID Naturalization2023-11-14T15:34:11ZNisha ThakranADR: EDS DMS - Bulk Data Ingestion and Dataset ID Naturalization**Introduction:**
The purpose of this ADR is to address the advanced use case of EDS DMS to Add bulk data into the OSDU Platform. Additionally, it focuses on naturalizing the dataset ID associated with the relevant schemas for Work-Prod...**Introduction:**
The purpose of this ADR is to address the advanced use case of EDS DMS to Add bulk data into the OSDU Platform. Additionally, it focuses on naturalizing the dataset ID associated with the relevant schemas for Work-Product-Component (WPC) record that has linked data files. By importing and naturalizing the dataset IDs, we aim to improve the capabilities of handling bulk data efficiently and to ensure that the data files are properly added to the OSDU Platform and the WPCs' child datasets are converted from "external" to "internal" type, providing improved accessibility and integration of the data within the platform.
**Objective:**
Currently, the EDS fetch-and-ingest process only copies the metadata, and the child dataset of the WPC at the operator's end is flagged as "external." While operators can use EDS DMS to obtain the actual data file when needed, but it is not properly added to the Data Platform. This approach proposes a solution to address these limitations by appropriately adding the data file to the OSDU Platform and converting the WPC's child dataset from "external" to "internal.”
**Status**
- [x] Proposed
- [x] Trialing
- [x] Under review
- [ ] Approved
- [ ] Retired
**Scope**
The scope of this ADR includes the following scenarios:
- Importing Bulk Data: Importing bulk data from the provider end into the Data Platform using EDS DMS.
- File Storage in OSDU Instance: The use of the dataset API to store the fetched files back to the operator's OSDU instance.
- Naturalizing Dataset IDs and re ingestion: The naturalization of dataset IDs from external to internal to align them with the relevant schemas within the Data Platform.
**Given /assumptions:**
- WPC metadata is already present at the operator end.
**Required Changes:**
- eds_dataset: New Dag will be introduced to perform the naturalization of the bulk data after the upload of the data files at the operator’s end.
- The DAG will be executed on-demand without the involvement of a scheduler.
- An additional boolean parameters should be introduced in the internal dataset schemas(File.Generic) to indicate whether the dataset id is internal or naturalized, to keep a track of the naturalized dataset ids.we can achieve this by using the ExtensionProperties of the dataset schemas.
**DatasetExternal:True**
* During the naturalization process, it is important to maintain a similar dataset ID while allowing for a conversion of the data type i.e from external to internal, it will be easier for reverse mapping. Example: opendes:dataset--ConnectedSource.Generic:test123 will be converted to: opendes:dataset--File.Generic:test123
we can also have additional parameters within the ExternalProperties like pointer to the external dataset id (connectedSource.Generic) with the version.
eg: Parent_dataset_id:opendes:dataset--ConnectedSource.Generic:test123
Parent_dataset_id_version: 1614105463059152
**Inputs:**
The inputs for the naturalization process should be an array of WPC IDs. Each WPC ID represents a specific work-product-component. Here is an example of the input structure:
[ "osdu:work-product-component--WellLog:7fdf1681b7ed1a1d54046ca1c2438add13719fafd18295a5e35d7bbdb45a53e4", "osdu:work-product-component--WellLog:7fdf1681b7ed1a1d54046ca1c2438ad89cf67"]
**Implementation:**
- Retrieve the ConnectedSource.Generic by providing the specified input IDs (well Log id, wellboreTrajectory id).
- Initiate the EDS DMS Operator, passing the ConnectedSource.Generic as input, and retrieve the signed URL.
- Download the LAS file at the operator's end using the obtained signed URL.
- Utilize the Dataset File service to upload the downloaded LAS file to the OSDU instance for the operator.
- Convert the child dataset of the WPC from "external" to "internal" by reassigning it to the cloud location mentioned above, such as changing ConnectedSource.Generic to File.Generic.
**Exception Handling:**
- Dataset ID is not present with the given WPC ids: if dataset id is missing, raise an exception or return an error indicating the issue.
- Invalid or Unrecognized IDs WPC ids or Dataset IDs: Handle cases where the provided WPC ID or dataset ID is invalid or unrecognized at operator’s end.
- Data Retrieval Errors: Implement error handling for situations where the dataset associated with the ID cannot be retrieved.
**Sequence Diagram**
![image](/uploads/f82c52840380e323c48b7a3e7f0084ab/image.png)
**Functional Requirements:**
- Data Import:
- The system should provide functionality to import bulk data from the provider's end using EDS DMS.
- EDS DMS Integration:
- The system should integrate with the EDS DMS to establish a connection with the provider's data
source.
- Dataset ID Naturalization:
- The system should support the naturalization of dataset IDs (external to internal) associated with
different schemas by uploading the imported data back to the Operator's OSDU instance and creating the
internal id.
- Mapping and re ingest:
- Mapping the transformed dataset id back to the schema id and and proceed to re-ingest the
transformed data back to the instance, ensuring its seamless integration.
**Non-functional Requirements**
- Performance:
- The system should be capable of handling large volumes of wellbore data efficiently, providing fast response times for data retrieval and analysis.
- It should be able to handle concurrent user interactions and maintain performance under peak load conditions.
- Scalability:
- The system should be scalable to accommodate increasing amounts of wellbore data and growing user bases.
- It should be able to handle additional data sources and support a high number of concurrent users without significant degradation in performance.https://community.opengroup.org/osdu/platform/system/file/-/issues/84File Service rejects Record ID from dataset--File.Generic manifest and genera...2023-06-09T15:27:51ZSamiullah GhousudeenFile Service rejects Record ID from dataset--File.Generic manifest and generate always GUID as record id**File Service rejects Record ID from dataset--File.Generic manifest and generate always GUID as record id**
Please check below CURL request to register metadata record for file dataset, I have included ID in the manifest, however all t...**File Service rejects Record ID from dataset--File.Generic manifest and generate always GUID as record id**
Please check below CURL request to register metadata record for file dataset, I have included ID in the manifest, however all time it rejects this id and create system generated GUID for this registered record.
**{+ Create File metadata - Request +}**
curl --location 'https://osdu-ship.msft-osdu-test.org/api/file/v2/files/metadata' \
--header 'Data-Partition-Id: opendes' \
--header 'Authorization: Bearer ...' \
--header 'Content-Type: application/json' \
--header 'Cookie: JSESSIONID=F4452A7D9F8752E8A82DC6E354D29B26' \
--data-raw '{
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
{+ "id":"opendes:dataset--File.Generic:sami-test1",+}
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.viewers@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-Test-Legal-Tag-4766549"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"data": {
"Endian": "BIG",
"DatasetProperties": {
"FileSourceInfo": {
"FileSource": "/osdu-user/1686225883215-2023-06-08-12-04-43-215/4a62ec123d43427e93af2a4a1c515a6b"
}
}
}
}'
**{+ Create File metadata - Response +}**
{
{+"id": "opendes:dataset--File.Generic:d5c226d6-3eb2-4825-8b9b-e0834d0464cb"+}
}
cc- @todaiks @debasisc @chadhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-cpp-lib/-/issues/23Bug in escape function defined in utils.cc2023-06-13T20:11:56ZMichaelBug in escape function defined in utils.ccIn the file https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-cpp-lib/-/blob/master/src/src/lib/shared/utils.cc, line 264 there's escape function defined. It contains a bug in...In the file https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-cpp-lib/-/blob/master/src/src/lib/shared/utils.cc, line 264 there's escape function defined. It contains a bug in line 270:
`if (c < 0 && (isalnum(c) || c == '-' || c == '_' || c == '.' || c == '~'))`
Specifically, the "if c < 0" condition shouldn't be there, at least from my point of view. As a result, the following code of sdapi library:
```
http.set_url(_sdmanager->getSDUrl() +
"/dataset/tenant/" + escape(tenant) +
"/subproject/" + escape(subproject) +
"/exist");
```
produces the following URL:
`"https://osdu-ship.msft-osdu-test.org/seistore-svc/api/v3/dataset/tenant/%6F%70%65%6E%64%65%73/subproject/%6D%69%63%68%61%65%6C%76%31/exist"`
instead of
`"https://osdu-ship.msft-osdu-test.org/seistore-svc/api/v3/dataset/tenant/opendes/subproject/michaelv1/exist"`https://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/268Data - SPIKE - Investigate if GCZ can work without GridGain2023-10-18T15:22:59ZNoel OkanyaData - SPIKE - Investigate if GCZ can work without GridGainAs a Developer, I want to investigate if GCZ can work without GridGain to address the below concerns:
1. GridGain is a third party library. What if it stops to exist?
2. Apache Ignite is also a third party open source library.As a Developer, I want to investigate if GCZ can work without GridGain to address the below concerns:
1. GridGain is a third party library. What if it stops to exist?
2. Apache Ignite is also a third party open source library.Levi RemingtonLevi Remingtonhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/issues/34[Analytical Queries] Implement search of RCA by its measurements2023-08-21T19:03:14ZEmile Veliyev[Analytical Queries] Implement search of RCA by its measurementsBusiness goal:
To implement a simplified way of searching and retrieving Sample information by entering one or several search criteria in one request.
Implementation details:
Data model: https://kb.epam.com/pages/viewpage.action?space...Business goal:
To implement a simplified way of searching and retrieving Sample information by entering one or several search criteria in one request.
Implementation details:
Data model: https://kb.epam.com/pages/viewpage.action?spaceKey=XOMROCK&title=Data+Types#DataTypes-RCA-RoutineCoreAnalysis
In a search query a user should be able to specify:
attribute by which the search should be performed (mandatory)
operator (mandatory)
value (mandatory)
List of the RCA data attributes that should be used for search:
SampleDepth
Permeability
Porosity
GrainDensity
Saturation
Pressure
Temperature
User should be able to specify a search parameter and a value of this parameter. The value operator may be:
=
>
<
< AND >
>=
<=
<= AND >=
User should be able to combine multiple search criteria.
Using AND - search queries should work as multiple filters then.
Using OR
Response:
Response should be in the form of list with RCA WPC ID and corresponding bulk data
Response should include multiple objects of RCA data if that satisfies the queried condition
Response should also include the RockSampleID.
// Example of response
{
"RCAID": "opendes:work-product-component--RockSampleAnalysis:dd76cf6c-226f-5636-ad1b-1ca0f8249cc8",
"RockSampleID": "namespace:master-data--RockSample:SomeUniqueRockSampleID:226f-5636-ad1b-1ca0f8249cc8",
"RCAData": {
"Conditions": [
{
"Pressure": 800,
"PressureMeasurementTypeID": "opendes:reference-data--PressureMeasurementType:Overburden:",
"Temperature": 12345.6
}
],
"Measurements": [
{
"Permeability": 4410.0,
"PermeabilityMeasurementTypeID": "opendes:reference-data--PermeabilityMeasurementType:Gas:",
"Porosity": 35.7,
"PorosityMeasurementTypeID": "opendes:reference-data--PorosityMeasurementType:HeliumInjection:"
}
]
}
}
Only data that the user has permission to view or edit should be displayed (user is included into corresponding RockSampleAnalysis WPC "owners" or "viewers" ACL groups).
Codes:
200 - query was successfully executed and 0+ results were returned
422 - validation error
if parameter name specified does not match the one from the list, message "Search cannot be executed by the specified parameter or parameter does not exist. Allowed parameters: SampleDepth, Permeability, Porosity, GrainDensity, Saturation" should be displayed
if not a numeric value was specified, message "Search value is not of numeric type" message should be displayed
if specified operator does not match one from the list, message "Specified search cannot be executed, allowed operators: =, >, <, < AND >, >=, <=, <= AND >=" should be displayed
if any of the mandatory request parameters were not specified, message "Mandatory request parameters are not specified" should be displayed.
Acceptance Criteria
User is able to perform Sample search using Rock Sample DDMS endpoint by the following Sample parameters:
SampleDepth
Permeability
Porosity
GrainDensity
Saturation
Pressure
Temperature
Query should return RCA with all its data based on the provided params and conditions
User is able to combine different search requests using AND / OR operators to get narrowed or broader search results.
Validation error should be displayed with an informative message in case search request is entered not according to the specified format.
Descriptionhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/issues/33[Analytical Queries] Implement search of rock samples by formation of the wel...2023-08-21T19:03:20ZEmile Veliyev[Analytical Queries] Implement search of rock samples by formation of the wellboreContext: Users should be able to search for RCA data by formation name
Use Case Scenario
User executes a query to find the RCA data that belongs Formation Name = WolfStock
System identifies all existing Rock Sample records that have st...Context: Users should be able to search for RCA data by formation name
Use Case Scenario
User executes a query to find the RCA data that belongs Formation Name = WolfStock
System identifies all existing Rock Sample records that have stored WellboreID within
From identified WellboreIDs system retrieves the linked WellboreMarkerSet WPC
Within identified WellboreMarkerSet WPC system filters these that have data.Markers[].FeatureTypeID = "data-partition:reference-data–FeatureTypeID:Top"
System creates an array of existing formations that should include the following data
WellBoreID
Formations
FormationName = data.Markers[].FeatureName
FormationTopDepth = data.Markers[].MarkerMeasuredDepth
FormationBottomDepth = will be defined in step #6
System executes the check to identify if there are multiple objects within each WellboreID:
If there are multiple objects, then sort the values of MarkerMeasuredDepth in ascending order.
From the sorted list of MarkerMeasuredDepth values identify the closest biggest value to each MarkerMeasuredDepth.
Identified closest values should be FormationBottomDepth. FormationBottomDepth of the Formation with the largest MarkerMeasuredDepth should be set to "30000"
If there is only one object within the Wellbore ID array, then default FormationBottomDepth is set to "30000"
System selects only records where FormationName from the search request = FormationName in the array
System selects all Samples that are linked to the WellboreID from the previous step (#7) subset using RockSample master metadata records (data.WellboreID = WellboreID from an array)
System checks to which RCA identified Samples belong and check the SampleDepth parameter in identified RCA
System checks the following condition: FormationTopDepth <= SampleDepth <= FormationBottomDepth
If condition is true, then system associates RockSample with the Formation
System returns Rock Samples that match the condition
System return RCA data which belongs to identified samples
Response
// Example of response
{
"RCAID": "opendes:work-product-component--RockSampleAnalysis:dd76cf6c-226f-5636-ad1b-1ca0f8249cc8",
"RockSampleID": "namespace:master-data--RockSample:SomeUniqueRockSampleID:226f-5636-ad1b-1ca0f8249cc8",
"FormationName": "Wolfstock",
"RCAData": {
"Conditions": [
{
"Pressure": 800,
"PressureMeasurementTypeID": "opendes:reference-data--PressureMeasurementType:Overburden:",
"Temperature": 12345.6
}
],
"Measurements": [
{
"Permeability": 4410.0,
"PermeabilityMeasurementTypeID": "opendes:reference-data--PermeabilityMeasurementType:Gas:",
"Porosity": 35.7,
"PorosityMeasurementTypeID": "opendes:reference-data--PorosityMeasurementType:HeliumInjection:"
}
]
}
}https://community.opengroup.org/osdu/data/open-test-data/-/issues/91TNO dataset, master-data missing required value2023-06-05T10:24:14ZDo DangTNO dataset, master-data missing required valueHi,
I'm using [generate_manifests.sh](https://community.opengroup.org/osdu/platform/data-flow/data-loading/open-test-data/-/blob/master/rc--3.0.0/6-data-load-scripts/scripts/generate-manifests.sh) to generate manifests. However, there ar...Hi,
I'm using [generate_manifests.sh](https://community.opengroup.org/osdu/platform/data-flow/data-loading/open-test-data/-/blob/master/rc--3.0.0/6-data-load-scripts/scripts/generate-manifests.sh) to generate manifests. However, there are some missing properties in the provided TNO dataset, in this [file](https://community.opengroup.org/osdu/platform/data-flow/data-loading/open-test-data/-/blob/master/rc--3.0.0/1-data/3-provided/TNO/master-data/Well/well_tno_csv_0915_2021.csv)
Log
```Creating manifest using /Users/dodang/workspace/free/osdu/open-test-data-master/rc--3.0.0/6-data-load-scripts/scripts/../../1-data/3-provided/TNO/master-data/Well/well_tno_csv_0915_2021.csv with template /Users/dodang/workspace/free/osdu/open-test-data-master/rc--3.0.0/6-data-load-scripts/scripts/../../5-templates/master_data/template_maWell.1.0.0.json
Not using group file
2023-06-04 23:07:55 INFO input_csv: /Users/dodang/workspace/free/osdu/open-test-data-master/rc--3.0.0/1-data/3-provided/TNO/master-data/Well/well_tno_csv_0915_2021.csv
2023-06-04 23:07:55 INFO template_json: /Users/dodang/workspace/free/osdu/open-test-data-master/rc--3.0.0/5-templates/master_data/template_maWell.1.0.0.json
2023-06-04 23:07:55 INFO output_path: /Users/dodang/workspace/free/osdu/open-test-data-master/rc--3.0.0/generated-manifests/tno/master-well-data-manifests
2023-06-04 23:07:55 INFO schema_path: /Users/dodang/workspace/free/osdu/open-test-data-master/rc--3.0.0/3-schema
2023-06-04 23:07:55 INFO schema_ns_name: <namespace>
2023-06-04 23:07:55 INFO schema_ns_value: --required_template
2023-06-04 23:07:55 INFO
2023-06-04 23:07:56 ERROR Unable to process data row: 1
Traceback (most recent call last):
File "/Users/dodang/workspace/free/osdu/open-test-data-master/rc--3.0.0/2-scripts/load_manifest_scripts/src/loading_manifest/csv_to_json.py", line 631, in create_manifest_from_csv
jsonschema.validate(to_be_validated, schema, resolver=resolver)
File "/Users/dodang/workspace/arm-venv/lib/python3.9/site-packages/jsonschema/validators.py", line 1121, in validate
raise error
jsonschema.exceptions.ValidationError: 'properties' is a required property
Failed validating 'required' in schema['properties']['data']['allOf'][1]['properties']['SpatialLocation']['properties']['Wgs84Coordinates']['properties']['features']['items']:
{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'geometry': {'oneOf': [{'type': 'null'},
{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'coordinates': {'items': {'type': 'number'},
'minItems': 2,
'type': 'array'},
'type': {'enum': ['Point'],
'type': 'string'}},
'required': ['type',
'coordinates'],
'title': 'GeoJSON Point',
'type': 'object'},
{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'coordinates': {'items': {'items': {'type': 'number'},
'minItems': 2,
'type': 'array'},
'minItems': 2,
'type': 'array'},
'type': {'enum': ['LineString'],
'type': 'string'}},
'required': ['type',
'coordinates'],
'title': 'GeoJSON LineString',
'type': 'object'},
{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'coordinates': {'items': {'items': {'items': {'type': 'number'},
'minItems': 2,
'type': 'array'},
'minItems': 4,
'type': 'array'},
'type': 'array'},
'type': {'enum': ['Polygon'],
'type': 'string'}},
'required': ['type',
'coordinates'],
'title': 'GeoJSON Polygon',
'type': 'object'},
{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'coordinates': {'items': {'items': {'type': 'number'},
'minItems': 2,
'type': 'array'},
'type': 'array'},
'type': {'enum': ['MultiPoint'],
'type': 'string'}},
'required': ['type',
'coordinates'],
'title': 'GeoJSON MultiPoint',
'type': 'object'},
{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'coordinates': {'items': {'items': {'items': {'type': 'number'},
'minItems': 2,
'type': 'array'},
'minItems': 2,
'type': 'array'},
'type': 'array'},
'type': {'enum': ['MultiLineString'],
'type': 'string'}},
'required': ['type',
'coordinates'],
'title': 'GeoJSON '
'MultiLineString',
'type': 'object'},
{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'coordinates': {'items': {'items': {'items': {'items': {'type': 'number'},
'minItems': 2,
'type': 'array'},
'minItems': 4,
'type': 'array'},
'type': 'array'},
'type': 'array'},
'type': {'enum': ['MultiPolygon'],
'type': 'string'}},
'required': ['type',
'coordinates'],
'title': 'GeoJSON MultiPolygon',
'type': 'object'},
{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'geometries': {'items': {'oneOf': [{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'coordinates': {'items': {'type': 'number'},
'minItems': 2,
'type': 'array'},
'type': {'enum': ['Point'],
'type': 'string'}},
'required': ['type',
'coordinates'],
'title': 'GeoJSON '
'Point',
'type': 'object'},
{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'coordinates': {'items': {'items': {'type': 'number'},
'minItems': 2,
'type': 'array'},
'minItems': 2,
'type': 'array'},
'type': {'enum': ['LineString'],
'type': 'string'}},
'required': ['type',
'coordinates'],
'title': 'GeoJSON '
'LineString',
'type': 'object'},
{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'coordinates': {'items': {'items': {'items': {'type': 'number'},
'minItems': 2,
'type': 'array'},
'minItems': 4,
'type': 'array'},
'type': 'array'},
'type': {'enum': ['Polygon'],
'type': 'string'}},
'required': ['type',
'coordinates'],
'title': 'GeoJSON '
'Polygon',
'type': 'object'},
{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'coordinates': {'items': {'items': {'type': 'number'},
'minItems': 2,
'type': 'array'},
'type': 'array'},
'type': {'enum': ['MultiPoint'],
'type': 'string'}},
'required': ['type',
'coordinates'],
'title': 'GeoJSON '
'MultiPoint',
'type': 'object'},
{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'coordinates': {'items': {'items': {'items': {'type': 'number'},
'minItems': 2,
'type': 'array'},
'minItems': 2,
'type': 'array'},
'type': 'array'},
'type': {'enum': ['MultiLineString'],
'type': 'string'}},
'required': ['type',
'coordinates'],
'title': 'GeoJSON '
'MultiLineString',
'type': 'object'},
{'properties': {'bbox': {'items': {'type': 'number'},
'minItems': 4,
'type': 'array'},
'coordinates': {'items': {'items': {'items': {'items': {'type': 'number'},
'minItems': 2,
'type': 'array'},
'minItems': 4,
'type': 'array'},
'type': 'array'},
'type': 'array'},
'type': {'enum': ['MultiPolygon'],
'type': 'string'}},
'required': ['type',
'coordinates'],
'title': 'GeoJSON '
'MultiPolygon',
'type': 'object'}]},
'type': 'array'},
'type': {'enum': ['GeometryCollection'],
'type': 'string'}},
'required': ['type',
'geometries'],
'title': 'GeoJSON '
'GeometryCollection',
'type': 'object'}]},
'properties': {'oneOf': [{'type': 'null'},
{'type': 'object'}]},
'type': {'enum': ['Feature'], 'type': 'string'}},
'required': ['type', 'properties', 'geometry'],
'title': 'GeoJSON Feature',
'type': 'object'}
On instance['data']['SpatialLocation']['Wgs84Coordinates']['features'][0]:
{'geometry': {'coordinates': [3.51906683, 55.68101428],
'type': 'Point'},
'type': 'Feature'}```https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/497Erroneous Inconsistent search results when adding or removing records2024-02-22T15:26:10ZMichaelErroneous Inconsistent search results when adding or removing recordsWe are adding and removing records from OSDU Storage, and we see erroneous results where a record that we add is not shown in the search result or a record that we delete is shown.
We tried waiting for the expected result from the sear...We are adding and removing records from OSDU Storage, and we see erroneous results where a record that we add is not shown in the search result or a record that we delete is shown.
We tried waiting for the expected result from the search and then trying again. We see erroneous results sometimes when we find it in the search results, but the next search does not find it. Similarly, for deleted records, we sometimes see the deletion only to see it not deleted in the next search.
Here is a video that demonstrates the inconsistent search results after a record has been created: [osdu_indexing_consistency.zip](/uploads/88858771b3a497a7336dd3bf47bb0886/osdu_indexing_consistency.zip)
Here is the postman collection that is used in the video: [Indexing_Test.postman_collection.json](/uploads/6958cd6c0b3da081a54a46488e1a2497/Indexing_Test.postman_collection.json)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/190Add OSDU manifest generation capability2023-06-02T14:03:18ZMorten OfstadAdd OSDU manifest generation capabilityVDSInfo (or maybe a new vds utility) should be able to generate OSDU compliant JSON manifest for ingesting VDS. This was suggested by Juliana Fernandes (@fernandes_jfa) and would make life a lot easier when interacting with the OSDU inge...VDSInfo (or maybe a new vds utility) should be able to generate OSDU compliant JSON manifest for ingesting VDS. This was suggested by Juliana Fernandes (@fernandes_jfa) and would make life a lot easier when interacting with the OSDU ingestion service.https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/66Tubular data type - adjust to recent change in group type (Wpc -> Master)2023-06-01T12:42:42ZDebasis ChatterjeeTubular data type - adjust to recent change in group type (Wpc -> Master)Please see this issue.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/551
TubularAssembly and TubularComponentPlease see this issue.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/551
TubularAssembly and TubularComponenthttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/issues/71IBM Wellbore Domain Services Integration test cases are failing.2023-07-10T12:49:38Zvikas ranaIBM Wellbore Domain Services Integration test cases are failing.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/jobs/1992734.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/jobs/1992734.M18 - Release 0.21vikas ranavikas ranahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/153IBM workflow integration test failing - for M182023-06-01T09:52:21Zvikas ranaIBM workflow integration test failing - for M18https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/jobs/1996356https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/jobs/1996356M18 - Release 0.21vikas ranavikas ranahttps://community.opengroup.org/osdu/platform/system/storage/-/issues/173Does not detect mismatch of entity name between "kind" and "id"2023-06-06T00:42:59ZDebasis ChatterjeeDoes not detect mismatch of entity name between "kind" and "id"I made this test case to create record directly by using Storage service and then the same record by using Manifest-based Ingestion.
```
"kind": "osdu:wks:work-product-component--TubularComponent:1.0.0",
"id": "osdu:work...I made this test case to create record directly by using Storage service and then the same record by using Manifest-based Ingestion.
```
"kind": "osdu:wks:work-product-component--TubularComponent:1.0.0",
"id": "osdu:work-product-component--TubularAssembly:TUBULARDC31May",
```
As you can see "kind" speaks of **TubularComponent** whereas "id" speaks of **TubularAssembly**.
Storage service seems very forgiving. It creates the record and also Indexer replicates the record in Index store. So, we can also retrieve by using Search service.
Whereas Manifest-based Ingestion rejects this JSON payload wit suitable reason, as expected.
See this document.
https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M17/Test_Plan_Results_M17/Core_Services/M17-AWS-Storage-service-test-sanity.docxhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/issues/27Refactor CMD argument parsing2023-06-13T20:19:05ZYan Sushchynski (EPAM)Refactor CMD argument parsingIt would be nice to use the built-in Python library [argparse](https://docs.python.org/3/library/argparse.html) to parse arguments from the command line. The current approach looks a little messy, not declarative, and not flexible enough...It would be nice to use the built-in Python library [argparse](https://docs.python.org/3/library/argparse.html) to parse arguments from the command line. The current approach looks a little messy, not declarative, and not flexible enough.
Let's say, we want to add an extra parameter to parse (e.g., data-partition-id), and this is difficult to do since there are a lot places where we have to add the changes.
But if we have a single file/entry-point where we describe expected commands with [argparse](https://docs.python.org/3/library/argparse.html) the action of adding a new parameter will be much more pleasant.Diego MolteniMark YanDiego Moltenihttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/issues/24simple_osdu_docker_desktop - CrashLoopBackOff, Error in several pods2023-08-08T19:54:41ZChad Leongsimple_osdu_docker_desktop - CrashLoopBackOff, Error in several podsHi - following the guide https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/tree/master/examples/simple_osdu_docker_desktop, with the following `custom-values.yaml`:
```yaml
global:
doma...Hi - following the guide https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/tree/master/examples/simple_osdu_docker_desktop, with the following `custom-values.yaml`:
```yaml
global:
domain: "localhost"
# Configuration parameter to switch between HTTP and HTTPS mode for external endpoint.
# Default - HTTP. HTTPS requires additional configuration
useHttps: false
keycloak:
auth:
# Fill in variable value, the value should contain only alphanumerical characters and should be at least 8 symbols
adminPassword: "admin12345"
# This value should be set to 'none' unless https is used (global.useHttps = true)
proxy: none
minio:
auth:
# Fill in variable value
rootPassword: "admin12345"
persistence:
size: 30Gi
# This value should be set to 'true' when using self-signed certificates or installing on minikube and docker desktop
useInternalServerUrl: true
postgresql:
global:
postgresql:
auth:
# Fill in variable value
postgresPassword: "admin12345"
persistence:
size: 8Gi
airflow:
externalDatabase:
# Fill in variable value
password: "admin12345"
auth:
# Fill in variable value
password: "admin12345"
elasticsearch:
security:
# Fill in variable value
elasticPassword: "admin12345"
master:
persistence:
size: 8Gi
data:
persistence:
size: 8Gi
# Configuration parameter to enable data bootstrap in storage service
conf:
enableDataBootstrap: &enable_data_bootstrap false
gc_legal_deploy:
conf:
bootstrapEnabled: *enable_data_bootstrap
gc_storage_deploy:
conf:
bootstrapEnabled: *enable_data_bootstrap
```
I seem to be getting several `errors` and `CrashLoopBackOff` in different pods:
```
airflow-bootstrap-deployment-5755bb555f-xwv8j 1/2 Error 4 7m31s
airflow-scheduler-6594fc8fb7-7jtj9 2/2 Running 0 7m30s
airflow-web-787f64645f-27x2b 1/2 Running 0 7m29s
crs-catalog-6fd7748997-rq5c2 2/2 Running 0 7m33s
crs-conversion-f5f6fdcb8-rlv9b 2/2 Running 0 7m31s
dataset-7bd48f55c-wg8hm 2/2 Running 0 7m32s
eds-dms-7889fcb467-6pxws 2/2 Running 0 7m31s
elastic-bootstrap-deployment-5548867bd4-j69kj 1/2 Error 3 (4m7s ago) 7m31s
elasticsearch-master-0 1/2 Running 0 7m32s
entitlements-556bdb4bd-q7r6r 0/2 Pending 0 7m27s
entitlements-bootstrap-6685b84c96-4qlmv 1/2 Error 1 (2m22s ago) 7m31s
file-865dcd77fc-4k949 2/2 Running 0 7m27s
indexer-5bc6dd8fc8-xdksw 0/2 Pending 0 7m26s
keycloak-0 1/2 Running 0 7m32s
keycloak-bootstrap-deployment-7455cd6448-d64tb 1/2 Error 3 7m28s
legal-68f8d464bc-jx2c2 2/2 Running 0 7m30s
minio-95957bb-pldbg 1/2 Running 1 (2m32s ago) 7m30s
minio-bootstrap-deployment-6556948c7f-mthdm 0/2 Pending 0 7m27s
notification-5d7484fd7-rh4vc 1/2 Error 0 7m32s
opa-8d9d54c46-ldjjk 1/2 Running 0 7m30s
partition-748bbb8b77-4gkcp 2/2 Running 0 7m31s
partition-bootstrap-5fdf48dd59-jpbb8 1/2 Error 2 (3m22s ago) 7m31s
policy-bd6bfb75b-2f58n 2/2 Running 0 7m33s
policy-bootstrap-55f74957f4-j84bf 1/2 Running 2 (101s ago) 7m31s
postgres-bootstrap-deployment-84d45f87d5-fxmbt 1/2 Running 1 (2m39s ago) 7m30s
postgresql-db-0 1/2 Running 0 7m32s
rabbitmq-0 1/2 Running 0 7m32s
rabbitmq-bootstrap-deployment-5d957b54cd-qvtc9 1/2 Error 5 (4m39s ago) 7m33s
redis-dataset-5d75c9cbcb-7rhld 2/2 Running 0 7m29s
redis-entitlements-589bddff6-rn5rm 2/2 Running 0 7m28s
redis-indexer-7c5d65b7f6-jxclx 2/2 Running 0 7m29s
redis-notification-5564799dc6-dnzfp 2/2 Running 0 7m32s
redis-search-686c9dfd87-fh4l8 0/2 Pending 0 7m27s
redis-seismic-store-855fbd99bd-98p7w 2/2 Running 0 7m31s
redis-storage-7d948cd6bb-vbwzm 0/2 Pending 0 7m26s
register-778bffc499-6kjt2 1/2 Error 0 7m29s
schema-69f66d8d8c-cj8nr 2/2 Running 0 7m33s
schema-bootstrap-6557dc76d7-xfgjm 1/2 CrashLoopBackOff 2 (2m16s ago) 7m30s
search-7f48df6c6b-lwjw8 0/2 Pending 0 7m27s
secret-8f8dd48f6-jgjkl 2/2 Running 0 7m33s
seismic-store-7f49bc88d4-66cmk 2/2 Running 0 7m29s
storage-75d9786668-b8gdh 2/2 Running 0 7m28s
unit-fb59b9999-8wgj6 2/2 Running 0 7m28s
well-delivery-784b676d97-8bmz7 2/2 Running 0 7m28s
wellbore-55d756b874-724x6 0/2 Pending 0 7m27s
workflow-5d56b64678-27v54 2/2 Running 0 7m32s
workflow-bootstrap-6d9f99c499-8fqx5 1/2 CrashLoopBackOff 5 (2m7s ago) 7m32s
```
Any idea how I can resolve these?Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)