OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2024-03-12T06:17:17Zhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-production-source-service/-/issues/9[REQ] Create SPI for Google Cloud2024-03-12T06:17:17ZAsan Arifov[REQ] Create SPI for Google CloudPDMS MVP1, phase2https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-pss-entities-service/-/issues/31Swagger for Entity Management Service2024-03-26T04:53:34ZAsan ArifovSwagger for Entity Management Service* [x] Add Info endpoint.
* [x] Add Health endpoints.
* [x] Update title and description (import from PDF if necessary).
* [x] Add contact and license links.
* [x] Update error messages according to new standard.
* [x] Remove defaults.
* ...* [x] Add Info endpoint.
* [x] Add Health endpoints.
* [x] Update title and description (import from PDF if necessary).
* [x] Add contact and license links.
* [x] Update error messages according to new standard.
* [x] Remove defaults.
* [x] Change data-partition-id label to Tenant Id.
* [x] Remove appkey authorizationPDMS MVP1, phase2Asan ArifovAsan Arifovhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-timeseries-ingestion-service/-/issues/29Swagger for Time-series Management Service2024-03-26T04:54:28ZAsan ArifovSwagger for Time-series Management Service* [x] Add Info endpoint.
* [x] Add Health endpoints.
* [x] Update title and description (import from PDF if necessary).
* [x] Add contact and license links.
* [x] Update error messages according to new standard.
* [x] Remove defaults.
* ...* [x] Add Info endpoint.
* [x] Add Health endpoints.
* [x] Update title and description (import from PDF if necessary).
* [x] Add contact and license links.
* [x] Update error messages according to new standard.
* [x] Remove defaults.
* [x] Change data-partition-id label to Tenant Id.
* [x] Remove appkey authorizationPDMS MVP1, phase2Asan ArifovAsan Arifovhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-production-source-service/-/issues/8Swagger for Source Management Service2024-03-26T04:55:12ZAsan ArifovSwagger for Source Management Service* [x] Add Info endpoint.
* [x] Add Health endpoints.
* [x] Update title and description (import from PDF if necessary).
* [x] Add contact and license links.
* [x] Update error messages according to new standard.
* [x] Remove defaults.
* ...* [x] Add Info endpoint.
* [x] Add Health endpoints.
* [x] Update title and description (import from PDF if necessary).
* [x] Add contact and license links.
* [x] Update error messages according to new standard.
* [x] Remove defaults.
* [x] Add data-partition-id.
* [x] Change POST endpoint to \\sources.PDMS MVP1, phase2Asan ArifovAsan Arifovhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/238Segmentation fault when opening a corrupted VDS file2024-03-22T12:47:53ZSveinung RundhovdeSegmentation fault when opening a corrupted VDS fileVersion: 3.4.0
When opening an invalid VDS file a pointer to 0x0 is returned in src/OpenVDS/VDS/VolumeDataStoreVDSFile.cpp:645. There is an assert on the next line checking for this. Perhaps this should be checked for in release mode as...Version: 3.4.0
When opening an invalid VDS file a pointer to 0x0 is returned in src/OpenVDS/VDS/VolumeDataStoreVDSFile.cpp:645. There is an assert on the next line checking for this. Perhaps this should be checked for in release mode as well to provide a better error message?https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/166Add incoming request validation in Add Member API2024-03-26T11:38:19ZDeepa KumariAdd incoming request validation in Add Member APICurrently there are no validations on the email coming inside the request body for Azure.
Recently, we've enabled OID Validation: https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure/-/merge_request...Currently there are no validations on the email coming inside the request body for Azure.
Recently, we've enabled OID Validation: https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure/-/merge_requests/783
This will be a **breaking change** for AAD users: only OIDs will be accepted inside the Add member API.
Proposed Solution:
1. Add a validation inside azure layer to do the OID Validation: If the token is issued by AAD, then the incoming value inside email parameter should be a valid OID.
2. Graph API is going to be used to query the user information.M23 - Release 0.26Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-production-source-service/-/issues/7[BUG] Remove redundant code param from update source endpoint2024-03-27T07:49:41ZAsan Arifov[BUG] Remove redundant code param from update source endpointRemove redundant code param from update source endpoint
![2.png](/uploads/f63f4943f19555eb913e4eb8b5a14e4b/2.png)Remove redundant code param from update source endpoint
![2.png](/uploads/f63f4943f19555eb913e4eb8b5a14e4b/2.png)PDMS MVP1, phase2https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-structure-storage-service/-/issues/16[BUG] Register Entity Endpoint Not Working2024-03-13T06:07:38Zaliuddin abd rauf[BUG] Register Entity Endpoint Not Workingerror happened when tried to add new entity. the status code is 200, but the response message show 500, and yes the entity when try to retrieve it back, its not there
![image.png](/uploads/cd60750cfb7dd4846e36561dddac6243/image.png)
...error happened when tried to add new entity. the status code is 200, but the response message show 500, and yes the entity when try to retrieve it back, its not there
![image.png](/uploads/cd60750cfb7dd4846e36561dddac6243/image.png)
![image.png](/uploads/bb40573b00652dc615941a0a383f7ceb/image.png)PDMS MVP1, phase2https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-pss-entities-service/-/issues/30[BUG] Register Entity Endpoint Not Working2024-03-11T04:34:13Zaliuddin abd rauf[BUG] Register Entity Endpoint Not Workingerror happened when tried to add new entity. the status code is 200, but the response message show 500, and yes the entity when try to retrieve it back, its not there
![image.png](/uploads/e125902e9f0e54bff042a5548f9dc3a4/image.png){wid...error happened when tried to add new entity. the status code is 200, but the response message show 500, and yes the entity when try to retrieve it back, its not there
![image.png](/uploads/e125902e9f0e54bff042a5548f9dc3a4/image.png){width=577 height=380}
![image.png](/uploads/73652e844d4b512ac110b98f6079e0c6/image.png)PDMS MVP1, phase2https://community.opengroup.org/osdu/platform/system/reference/schema-upgrade/-/issues/30setup azure helm charts2024-03-26T11:37:15ZVikas Hoode [BP]vikas.hoode@bp.comsetup azure helm chartssetup standardized azure helm chartssetup standardized azure helm chartsM23 - Release 0.26Deepanshu SinghalDeepanshu Singhal2024-03-18https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/issues/343PVTModel Schema Updates2024-03-14T23:55:09ZMichael JonesPVTModel Schema UpdatesComponents array object - Add the following attribute
* Parachor - Number - unitless
ViscosityCoefficients object
* Convert to an array object
* Add FluidType attribute to the element object
* Uses SampleType reference-data, specifi...Components array object - Add the following attribute
* Parachor - Number - unitless
ViscosityCoefficients object
* Convert to an array object
* Add FluidType attribute to the element object
* Uses SampleType reference-data, specifically for values "oil" and "gas"M23 - Release 0.26Ernesto GutierrezMichael JonesMykhailo BuriakErnesto Gutierrezhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/220storage record with no acl owners become ghost record if OPA service is enabled.2024-03-28T06:20:14ZOm Prakash Guptastorage record with no acl owners become ghost record if OPA service is enabled.Storage records become inaccessible if OPA is enabled in case there is no ACL group associated with the record.
# Scenario:
Usually, when we create a record we define the owners and viewers group and the member associated with the group...Storage records become inaccessible if OPA is enabled in case there is no ACL group associated with the record.
# Scenario:
Usually, when we create a record we define the owners and viewers group and the member associated with the group can access the record. However, it is possible to delete the group and even disassociate ACL groups from the storage record. there is no validation as of now for a must-required single record. eventually record becomes a ghost record and nobody can access it.
There was a fix provided to users. data. root members can still access the group and add ACLs if needed.
it is discussed in this ADR
https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/141
# Findings
We have seen that code works fine and still users.data.root members can access the record if there is no associated ACL members for the record but if OPA is enabled we can not access the record even member is associated to users.data.root group.
code below checks if OPA is enabled and get access rights from OPA service
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/storage-core/src/main/java/org/opengroup/osdu/storage/service/IngestionServiceImpl.java#L198
OPA service returns with false access rites. However, if OPA is disabled the flow works because we have code added to return true if the member belongs to users.data.root.
We have found this not working in the Azure OSDU instance and need to know if requires a policy file fix or shall be handled in code to stop records from becoming ghost in case OPA is enabled.Dadong ZhouKelly ZhouShane HutchinsDeepa KumariDadong Zhouhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/155Augmented Index - Use Case 3 (Wellbore Name) not working in M22 Preship Envir...2024-03-07T11:27:28ZNorman MedinaAugmented Index - Use Case 3 (Wellbore Name) not working in M22 Preship EnvironmentI was testing out the augmented index feature on the M22 Preship environment. I was trying to implement the use cases documented in this [tutorial](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/docs/t...I was testing out the augmented index feature on the M22 Preship environment. I was trying to implement the use cases documented in this [tutorial](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/docs/tutorial/IndexAugmenter.md#use_cases). Use cases 1 and 5 worked for me. Use case 3 did not, as the field `WellboreName` wasn't coming out of search after reindexing. I tried testing this out three times, but it still didn't work.
I used the snippet that was provided in the tutorial page and didn't modify anything in it.https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/154Augmented Index - Use Case 2 (Country Names) not working in M22 Preship Envir...2024-03-07T11:27:59ZNorman MedinaAugmented Index - Use Case 2 (Country Names) not working in M22 Preship EnvironmentI was testing out the augmented index feature on the M22 Preship environment. I was trying to implement the use cases documented in this [tutorial](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/docs/t...I was testing out the augmented index feature on the M22 Preship environment. I was trying to implement the use cases documented in this [tutorial](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/docs/tutorial/IndexAugmenter.md#use_cases). Use cases 1 and 5 worked for me. Use case 2 did not, as the field `CountryNames` wasn't coming out of search after reindexing.
I also tried this on `osdu:wks:master-data--Wellbore:1.0.0` but replaced the item in `RelatedConditionMatches` with `^[\\w\\-\\.]+:reference-data--GeoPoliticalEntityType:Province:$` and the Name to `ProvinceNames`, but the custom `ProvinceNames` field is not appearing.
Please see below the reference data I used:
```
[
{
"acl": {
"owners": [
"{{New_OwnerDataGroup}}@{{data-partition-id}}{{domain}}"
],
"viewers": [
"{{New_ViewerDataGroup}}@{{data-partition-id}}{{domain}}"
]
},
"legal": {
"legaltags": [
"{{LegalTagNameExists}}"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"meta": [],
"data": {
"Code": "osdu:wks:master-data--Wellbore:1.",
"Configurations": [
{
"Name": "ProvinceNames",
"Policy": "ExtractAllMatches",
"Paths": [
{
"RelatedObjectsSpec": {
"RelatedObjectID": "data.GeoContexts[].GeoPoliticalEntityID",
"RelatedObjectKind": "osdu:wks:master-data--GeoPoliticalEntity:1.",
"RelatedConditionMatches": [
"^[\\w\\-\\.]+:reference-data--GeoPoliticalEntityType:Province:$"
],
"RelatedConditionProperty": "data.GeoContexts[].GeoTypeID"
},
"ValueExtraction": {
"ValuePath": "data.GeoPoliticalEntityName"
}
}
],
"UseCase": "As a user I want to find objects by a province name."
}
]
},
"id": "{{data-partition-id}}:reference-data--IndexPropertyPathConfiguration:wks:master-data--Wellbore:1.",
"kind": "osdu:wks:reference-data--IndexPropertyPathConfiguration:1.0.0",
"version": 0
}
]
```https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/237Buffer over-read2024-03-07T10:00:39ZSveinung RundhovdeBuffer over-readThere is a buffer over-read in src/OpenVDS/VDS/ConvertValues.h:
template <typename T>
static void CopyFrom1Bit(void * __restrict voiddst, const void* __restrict voidsrc, int32_t count)
{
T* target = (T*)voiddst;
uint8_t* source = (u...There is a buffer over-read in src/OpenVDS/VDS/ConvertValues.h:
template <typename T>
static void CopyFrom1Bit(void * __restrict voiddst, const void* __restrict voidsrc, int32_t count)
{
T* target = (T*)voiddst;
uint8_t* source = (uint8_t*)voidsrc;
uint8_t bits = *source;
int32_t mask = 1;
for (int i = 0; i < count; i++)
{
*target = (bits & mask)? T(1) : T(0);
target++;
mask <<= 1;
if (mask == 0x100)
{
source++;
bits = *source; <-- Read one past end of buffer on this line.
mask = 1;
}
}
}
I assume this happens on last iteration of the loop so the value is never used. It is highly unlikely that this will cause any issues, but I guess there is a theoretical possibility that this can cause a segmentation fault.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/236Unexpected errors when accessing dataset with OpenVDS 3.4.02024-03-22T12:46:03ZAlexander JaustUnexpected errors when accessing dataset with OpenVDS 3.4.0Hi,
I am currently testing out OpenVDS 3.4.0 and I really appreciate the increase amount of error messages. (Un)fortunately, I am also hit by some of the error messages. I am not sure if I am doing anything wrong or if our data set in t...Hi,
I am currently testing out OpenVDS 3.4.0 and I really appreciate the increase amount of error messages. (Un)fortunately, I am also hit by some of the error messages. I am not sure if I am doing anything wrong or if our data set in the Azure storage account is broken. The error messages I see do not appear when using in OpenVDS 3.3.3.
I have attached some Python scripts that I used to create the error messages at the bottom of this issues. I can also split this issue into several if it is too much content.
Any hints to what is going wrong and on how to debug this would be greatly appreciated.
## Versions, storage, dataset
* OpenVDS 3.4.0
* Azure blob store
* Dataset: Volve `ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536` with RLE compression and brick size 32. The file was created in June 2023 with OpenVDS/SEGYImport, but cannot tell which OpenVDS version it was.
This is the only file we did some testing on. I noticed that there are a few blobs with size 0. I am not sure if this is relevant.
## Observed errors
### Invalid headers
I see several messages about invalid headers, but I am not sure which parts is invalid. I don't understand why this only affects a few blobs (2-4) and not all blobs.
```text
[...]
Request was canceled.
OpenVDS reported error:
Invalid header (e.g. unsupported Wavelet compression version) for chunk: Dimensions_012LOD0/2647
[...]
```
### Invalid HTTP request range (HTTP 416 error) and logging of SAS token
1. The [range header of the HTTP request](https://learn.microsoft.com/en-us/rest/api/storageservices/specifying-the-range-header-for-blob-service-operations) seems to be wrong.
2. The error message exposes the SAS token. I am not sure if this is done on purpose. At least I see the risk that the token may appear unexpectedly in log files
```text
[...]
Http error response: 416 -> https://STORAGEACCOUNT/volve/ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536/vds_32_RLE/Dimensions_012LOD0/6945?SASTOKEN: The range specified is invalid for the current size of the resource.
RequestId: REQUESTID
Time:2024-03-06T13:27:02.0923545Z
[...]
```
I retracted potentially sensitive information.
None of the blobs mentioned in the error message has zero size:
```text
Blob name, Access tier, Blob type, Size
7747, Hot (Inferred), Block blob, 68.04 KiB
5470, Hot (Inferred), Block blob, 128.04 KiB
4302, Hot (Inferred), Block blob, 128.04 KiB
```
I could reproduce this error if requesting VolumeSamples so far. My script for request VolumeSubsets hangs after inline 42 and several error messages about invalid headers.
## Python script for reporduction
Below you find the Python code to request data from the VDS data set. The Python script needs the Python packages `openvds`, `python-dotenv`
and `numpy`.
### Requesting subsamples
```python
#!/usr/bin/env python3
import os
import numpy as np
import openvds
from dotenv import load_dotenv
load_dotenv()
SAS_TOKEN = os.getenv("SAS_TOKEN")
BLOB_URL = os.getenv("BLOB_URL")
if __name__ == "__main__":
print("Accessing samples from Azure blob")
try:
with openvds.open(url=BLOB_URL, connectionString=SAS_TOKEN) as vds:
manager = openvds.getAccessManager(vds)
layout = manager.volumeDataLayout
# Voxel extents are hardcoded for volve cube
# ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536
for depth in range(850):
samplePositions = []
for il in range(401):
for xl in range(720):
samplePositions.append((depth + 0.5, xl + 0.5, il + 0.5))
samplePositions = np.array(samplePositions)
print(f"Request at depth voxel {depth+0.5}")
request = manager.requestVolumeSamples(
samplePositions=samplePositions,
dimensionsND=openvds.DimensionsND.Dimensions_012,
lod=0,
channel=0,
interpolationMethod=openvds.InterpolationMethod.Cubic,
)
success = request.waitForCompletion()
if success:
print("Request was successfull")
else:
print(f"request:\n{request}")
if request.isCanceled:
print("Request was canceled.\n OpenVDS reported error:\n"
f" {request.errorMessage}"
)
else:
print("Request failed due to timeout")
except RuntimeError as error:
print(f"Could not open VDS: {error}")
```
### Requesting subsets
```python
#!/usr/bin/env python3
import os
import numpy as np
import openvds
from dotenv import load_dotenv
load_dotenv()
SAS_TOKEN = os.getenv("SAS_TOKEN")
BLOB_URL = os.getenv("BLOB_URL")
if __name__ == "__main__":
print("Accessing samples from Azure blob")
try:
with openvds.open(url=BLOB_URL, connectionString=SAS_TOKEN) as vds:
manager = openvds.getAccessManager(vds)
layout = manager.volumeDataLayout
sampleDimension, crosslineDimension, inlineDimension = (0, 1, 2)
for inlineIndex in range(401):
voxelMin = (0, 0, inlineIndex)
voxelMax = (
layout.getDimensionNumSamples(sampleDimension),
layout.getDimensionNumSamples(crosslineDimension),
inlineIndex + 1,
)
buffer = np.empty(
(
layout.getDimensionNumSamples(crosslineDimension),
layout.getDimensionNumSamples(sampleDimension),
)
)
print(f"Request at inline {inlineIndex}")
request = manager.requestVolumeSubset(
data_out=buffer,
dimensionsND=openvds.DimensionsND.Dimensions_012,
min=voxelMin,
max=voxelMax,
lod=0,
channel=0,
)
success = request.waitForCompletion()
if success:
print("Request was successfull")
else:
print(f"request:\n{request}")
if request.isCanceled:
print("Request was canceled.\n OpenVDS reported error:\n"
f"{request.errorMessage}"
)
else:
print("Request failed due to timeout")
except RuntimeError as error:
print(f"Could not open VDS: {error}")
```https://community.opengroup.org/osdu/platform/system/project-and-workflow/-/issues/73F1 (Java) Story 17 Service and basic data access layer for getting WIP resou...2024-03-20T12:13:41ZRostislav Dublin (EPAM)F1 (Java) Story 17 Service and basic data access layer for getting WIP resourcesDescription
---
See #72 for understanding API that requires the service
Also see the ADR "#108 [Core Services Integration - Search Service Support](https://community.opengroup.org/osdu/platform/system/home/-/issues/108)"
Implement the...Description
---
See #72 for understanding API that requires the service
Also see the ADR "#108 [Core Services Integration - Search Service Support](https://community.opengroup.org/osdu/platform/system/home/-/issues/108)"
Implement the Service
- It takes the CP ID as a parameter
- It gets the CP record and finds its `data.Namespace`
- It requests OSDU Search service API and queries the CP namespace for all indexed record IDs in all available kinds (one-by-one)
- It composes a solid list of all find record IDs
- It returns the list to the caller
Acceptance criteria
---
- The service created and capable to consume the OSADU Search service for getting ALL kinds data from a given namespace
Testing scenarios
---
- all necessary unit tests
- integration tests with existing and absent CP ID
Technical notes
---
The current version of the OSDU Search service is not capable of running queries in a custom namespace context.
So, the service can be fed with the mock data. And put TODO for improving it in the future.
Again, see the ADR "#108 [Core Services Integration - Search Service Support](https://community.opengroup.org/osdu/platform/system/home/-/issues/108)"Pavel BarzouPavel Barzouhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-timeseries-ingestion-service/-/issues/28[BUG] Ingestion Not Working2024-03-06T03:22:41Zaliuddin abd rauf[BUG] Ingestion Not WorkingError happened for both batch and single endpoint when trying to add new data to the system as per image below:
![image.png](/uploads/7eb395ffd79b37d8ff919386cf4700ba/image.png){width=822 height=378}
I tried to check with the log, belo...Error happened for both batch and single endpoint when trying to add new data to the system as per image below:
![image.png](/uploads/7eb395ffd79b37d8ff919386cf4700ba/image.png){width=822 height=378}
I tried to check with the log, below is the log that i found in the ingestion-service
![image.png](/uploads/53e422f07c4d76d0f772a9a6097eba4c/image.png){width=906 height=232}PDMS MVP1, phase2https://community.opengroup.org/osdu/platform/system/reference/schema-upgrade/-/issues/29Update, consolidate, format documentation2024-03-05T19:49:03ZRasheed Nagoor GaniUpdate, consolidate, format documentationhttps://osdu.pages.opengroup.org/platform/system/reference/schema-upgrade/
~~Documentation exists in many places. It will be much nicer to bring it all together in a Gitlab pages format. (As done in policy service)~~https://osdu.pages.opengroup.org/platform/system/reference/schema-upgrade/
~~Documentation exists in many places. It will be much nicer to bring it all together in a Gitlab pages format. (As done in policy service)~~M23 - Release 0.26Mark ChanceMark Chancehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-pss-properties-service/-/issues/15Swagger for Property Discovery Service2024-03-26T04:56:14ZAsan ArifovSwagger for Property Discovery Service* [x] Add Info endpoint.
* [x] Add Health endpoints.
* [x] Update title and description (import from PDF if necessary).
* [x] Add contact and license links.
* [x] Update error messages according to new standard.
* [x] Remove defaults.
* ...* [x] Add Info endpoint.
* [x] Add Health endpoints.
* [x] Update title and description (import from PDF if necessary).
* [x] Add contact and license links.
* [x] Update error messages according to new standard.
* [x] Remove defaults.
* [x] Change data-partition-id label to Tenant Id.
* [x] Remove appkey authorizationPDMS MVP1, phase2Asan ArifovAsan Arifov