OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2023-10-02T13:38:59Zhttps://community.opengroup.org/osdu/platform/system/search-service/-/issues/109Spatial filter with longitude out of range [-180;180] considered as invalid2023-10-02T13:38:59ZYauheni LesnikauSpatial filter with longitude out of range [-180;180] considered as invalidIf a rectangle spatial filter larger than 180 on longitude, and the rectangle across the international date change line, the following behavior is observed:
Longitude of right side of the rectangle will be converted from "-165" to "195"...If a rectangle spatial filter larger than 180 on longitude, and the rectangle across the international date change line, the following behavior is observed:
Longitude of right side of the rectangle will be converted from "-165" to "195"and search API gives 400 error, because it constraints the longitude to be within [-180,180]
`{"field":"data.SpatialLocation.Wgs84Coordinates","byGeoPolygon":{"points":[{"longitude":5.648519075025133,"latitude":47.33389841046996},{"longitude":5.648519075025133,"latitude":-80.31395034614398},{"longitude":194.6133628249749,"latitude":-80.31395034614398},{"longitude":194.6133628249749,"latitude":47.33389841046996},{"longitude":5.648519075025133,"latitude":47.33389841046996}]}}}`
I would suggest to relax the validation to range [-360, 360]. Because the it is difficult to guarantee the efficiency of the changes for all Elastics in all CSPs, it is recommended to use some feature flag. Each CSP could switch on and test this validation on one's own.Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/deployment-and-operations/multi-region-deployment/multi-region/-/issues/13Federated Search PoC Design for Multi-Region OSDU2023-08-15T07:55:35ZWan Ahmad Zaki Wan Mohammad NoorFederated Search PoC Design for Multi-Region OSDU# Background
Multi-Region OSDU has long been an important yet complex community initiative since it was first introduced in Feb 2020 in this ADR. It was brought back to community focus in the June 2022 EA sub-committee F2F meeting. Since...# Background
Multi-Region OSDU has long been an important yet complex community initiative since it was first introduced in Feb 2020 in this ADR. It was brought back to community focus in the June 2022 EA sub-committee F2F meeting. Since August of 2022, PETRONAS initiated the conversation with MSFT and SLB to collaborate and started brainstorming and researching requirements and feasible solutions. In the meantime, Shell has done a thorough and wide-reaching user voice gathering and produced a comprehensive and detailed “Multi Region Use Cases” document. Based on these efforts, MSFT has created a phased design proposal for multi-region OSDU that aligns with customer requirements as closely as possible and can be tackled in an incremental fashion. The outcome and test results from a prior phase PoC can determine whether a next phase is needed, and which direction should be taken in the next phase if needed. Once a PoC with acceptable SLA is reached, the implementation can begin. This phased approach will optimize the efforts and outcome.
# Scope and Purpose
This doc is intended to be used as a guide for the PoC work for the first phase, Federated Search approach. It includes detailed descriptions of the Federated Search approach, what new APIs are needed and how to implement them, what tests are needed, and what performance data will be collected to come up with estimated latency numbers under typical use scenarios. These numbers will determine whether the Federated Search approach meets user expectations. If it does, it can be implemented with the lowest cost compared with other approaches and deliver benefits to customers quickly. If not, we can move on to the next architectural model. Even if it is determined to be not meeting user expectations, the PoC work will be useful to pinpoint bottlenecks and gaps and establish baseline latency data for the next phase implementation.
# Federated Search Details
![image](/uploads/8336bc57d788888612ca0460c09a3dd2/image.png)
The Federated Search approach builds on top of the current OSDU implementation and requires no change to the existing APIs in the existing OSDU services. Several new federated search APIs will be added to Search service, Storage service, Well Delivery DDMS and Wellbore DDMS, and a new multi-region configuration service will be created for multi-region administration and configuration tasks. Three of the new APIs are in scope of the PoC. The new configuration service is not in scope of the PoC; manual configuration will be done instead.
### How Federated Search Works
**Deployment**
A global OSDU Administrator deploys multiple instances of OSDU to multiple CSP regions, each instance with its own data partitions. Data ingestion works the same as today. A data record only exists in the OSDU instance in the home region where the data is ingested. There is no raw data replication or catalog data replication or search index replication between instances.
**Region and Group Configuration**
The Administrator creates a global “Regions” table to hold all deployed OSDU instances’ IDs, endpoints, and data partition names. The Administrator configures “Cross-Region Partition Groups” to specify which partitions are logically associated to form a cross region search group. Multiple of such groups can be configured. The partition ID in the “Cross-Region Partition Groups” is in the format of “Instance ID: Partition Name” to guarantee its uniqueness. The configuration APIs will be included in the new Multi-region Configuration Service in actual implementation. In PoC, the configuration will be done by manually creating two Cosmos DB tables.
**Two New Global Search APIs and One New Global Storage Query API**
A new “POST /api/global_search/v2/query” and “POST /api/global_search/v2/query_with_cursor” APIs will be added to the core Search Service. They are the counter APIs for the current local versions of the “POST /api/search/v2/query” and “POST /api/search/v2/query_with_cursor” APIs that only search in a local partition. The Global Search APIs conduct search in a “Cross-Region Partition Group” across multiple regions.
In Storage Service, there are a set of four query APIs that retrieve data records in a local partition. Four global versions of these query APIs will be implemented to retrieve records from a “Cross-Region Partition Group”. For simplicity, in PoC, only one new global query API “GET /api/storage/v2/global_query/kinds” will be created in the Storage Service as the global counter for local API “GET /api/storage/v2/query/kinds”.
In Wellbore DDMS and Well Delivery DDMS, there are several local query APIs. The global version of these APIs will need to be added in actual implementation. For simplicity, they will not be included in PoC.
The request body and request headers for the new global APIs are the same as the local versions except the Partition ID in the request header. The new APIs require a “Cross-Region Partition Group” ID instead of a single Partition ID. This ID specifies the default partitions for the search. An optional query parameter “remote_partitions” allows user to select a subset of the default partitions to search. For example, if a group includes “Instance 1: Partition A, Instance 2: Partition A and Instance 3: Partition A” and “remote_partitions” query parameter has value of “Instance 1: Partition A, Instance 3: Partition A”, the global search will skip searching in Instance 2: Partition A. Optional query parameter provides flexibility for users to skip remote partitions to optimize performance.
The global APIs return the aggregation of the search results from all the partitions belonging to the group or the subset specified in the optional query parameter. The local partition is always implicitly included in the search. Optional query parameter will be validated against the “Cross-Region Partition Group” ID in the header. If any of the remote partitions does not belong to the group, “400 Bad Request” error is returned. For simplicity, validation will not be implemented in PoC.
```
POST /api/global_search/v2/query?remote_partitions=p1, p2, …
Header: cross-region-partition-group-id
```
```
POST /api/global_search/v2/query_with_cursor?remote_partitions=p1, p2, …
Header: cross-region-partition-group-id
```
```
GET /api/storage/v2/global_query/kinds?remote_partitions=p1, p2, …
Header: cross-region-partition-group-id
```
For the new global APIs, the user must meet the entitlement requirements for all the participating partitions’ data access. Otherwise, global APIs will return “401 Unauthorized” error.
# Testing Federated Search under Multi-Region Deployment
To adequately test latency numbers, one multi-region deployment and multiple tests will be carried out for PoC.
### Deployment and configuration
Three OSDU on Azure instances will be created in three different Azure regions across different continents (West US, West Europe, and East Asia) with each instance having one data partition. Manually create the “Regions” table and a “Cross-Region Partition Groups” table with one group that includes the three partitions from the three instances. Configure remote settings for the three Elastic Search clusters to have two remote clusters for each using Elastic Cluster Update Settings API.
### Tests
Upload different test data sets to each instance and run manual tests in Postman. Test local search and storage query APIs and their counter global APIs with the same request body and log response time for each test. When running global API tests, choose different optional query parameters to select different number of remote partitions. Test with different kinds and availability of data. The delta between the response time from local request and that from global request can be a rough estimate for latency caused by cross region search.
| Local | Global |
| ------ | ------ |
| Search/query | Search/query with 1 remote partition |
| | Search/query with 2 remote partitions |
| Search/query_with_cursor Initial request| Search/query_with_cursor Initial request with 1 remote partition |
|| Search/query_with_cursor Initial request with 2 remote partitions |
| Search/query_with_cursor request |Search/query_with_cursor request with 1 remote partition |
| | Search/query_with_cursor request with 2 remote partitions |
| Storage/query/kinds| Storage/query/kinds with 1 remote partition |
| | Storage/query/kinds with 2 remote partitions |
### Next Step After Testing
After the above tests are done and latency data is collected, the direction for the next step will be determined from the latency data.
If the latency data does not meet SLA, it indicates Federated Search approach is not an acceptable option and we will move to the next architecture model of “External Partition” that requires EDS shadow record replication to decrease search latency.
On the other hand, if latency data is within acceptable SLA, an additional PoC step can be done, which is to test fetching relatively large amount of raw data from a remote region using File service. The details about this PoC step will be created later when deemed necessary.
### POC Code Branch
- Search POC branch – [Code repository](https://community.opengroup.org/osdu/platform/system/search-service/-/tree/multiregion_poc2)
- Storage POC branch – [Code repository](https://community.opengroup.org/osdu/platform/system/storage/-/tree/multiregion_poc2?ref_type=heads)https://community.opengroup.org/osdu/platform/deployment-and-operations/terraform-deployment-aws/-/issues/3Getting error while pulling images2023-01-10T10:52:30ZSudhakar AGetting error while pulling images
Hi,
when running the step 5 (Steps to Manually Deploy the Infrastructure with Make) and we got the below error.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++...
Hi,
when running the step 5 (Steps to Manually Deploy the Infrastructure with Make) and we got the below error.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Error: osdu-ingest/os-ingestion-workflow failed to fetch resource from kubernetes: context deadline exceeded
│
│ with module.ingest.module.ingestion_worflow.kubectl_manifest.deployment,
│ on ingest/ingestion_workflow/kubernetes.tf line 41, in resource "kubectl_manifest" "deployment":
│ 41: resource "kubectl_manifest" "deployment" {
│
╵╷
│ Error: osdu-seismic-ddms/os-seismic-store failed to fetch resource from kubernetes: context deadline exceeded
│
│ with module.seismic_store_services.module.seismic_store.kubectl_manifest.deployment,
│ on seismic_store_services/seismic_store/kubernetes.tf line 31, in resource "kubectl_manifest" "deployment":
│ 31: resource "kubectl_manifest" "deployment" {
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
we found that the images.json in location “terraform-deployment-aws/deployment” having docker service images which are pointing to repo (registry.repo.osdu.aws/airflow-dag-upload:latest) with image tag latest is giving the error. We tried pulling the images manually and it returned 404 not found error.
====================================================================================================================
docker pull registry.repo.osdu.aws/os-policy-service:latest
Error response from daemon: error parsing HTTP 404 response body: no error details found in HTTP response body: "{}"
====================================================================================================================
Can we please get some assistance on solving this error.https://community.opengroup.org/osdu/platform/system/storage/-/issues/159Storage adds null meta to record ingested without2023-03-22T04:11:53ZAn NgoStorage adds null meta to record ingested without1. Record was ingested without specifying "meta" block. PUT api was successful.
2. Fetch the ingested record. Notice that Storage added "meta": null to the record.
**Checking with Search.**
Search indexed successfully. Status code was 2...1. Record was ingested without specifying "meta" block. PUT api was successful.
2. Fetch the ingested record. Notice that Storage added "meta": null to the record.
**Checking with Search.**
Search indexed successfully. Status code was 200.
Search result does not return the meta.
The current behavior is challenged saying that Meta block shouldn't have been added. Or if added, then it should be empty and not null.
So instead of adding:
"meta": null
It should be:
"meta": []
Upon creating or updating a record, providing an empty meta block should also be allowed.https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/issues/8Validation for whole schema2023-01-05T11:31:37ZChad LeongValidation for whole schema## Summary
As highlighted in preship testing for M15 issue - https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/405#note_190480, user is able to pass in non-master data (`"id": "{{data_partition_id}}:work-product-compon...## Summary
As highlighted in preship testing for M15 issue - https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/405#note_190480, user is able to pass in non-master data (`"id": "{{data_partition_id}}:work-product-component--WellLog:{{NewWellLogUWI}}"`) into the `MasterData` field and ingestion runs successfully without throwing error.
## Steps:
1. `curl --location --request POST 'https://r3m15.preshiptesting.osdu.aws/api/workflow/v1/workflow/Osdu_ingest/workflowRun'`
Response
```json
{
"runId": "{{$guid}}",
"executionContext": {
"acl": {
"owners": [
"{{New_OwnerDataGroup}}@{{data_partition_id}}{{domain}}"
],
"viewers": [
"{{New_ViewerDataGroup}}@{{data_partition_id}}{{domain}}"
]
},
"legal": {
"legaltags": [
"{{tagName}}"
],
"otherRelevantDataCountries": [
"US"
]
},
"Payload": {
"AppKey": "test-app",
"data-partition-id": "{{data_partition_id}}"
},
"manifest": {
"kind": "{{authority}}:{{schemaSource}}:Manifest:1.0.0",
"MasterData": [
{
"id": "{{data_partition_id}}:master-data--Organisation:Auto_Test_{{NewWellLogUWI}}",
"kind": "{{authority}}:{{schemaSource}}:master-data--Organisation:1.0.0",
"acl": {
"owners": [
"{{New_OwnerDataGroup}}@{{data_partition_id}}{{domain}}"
],
"viewers": [
"{{New_ViewerDataGroup}}@{{data_partition_id}}{{domain}}"
]
},
"legal": {
"legaltags": [
"{{tagName}}"
],
"otherRelevantDataCountries": [
"US"
]
},
"data": {
"Source": "AUTO test",
"OrganisationName": "Auto_Test_{{NewWellLogUWI}}"
}
},
{
"id": "{{data_partition_id}}:work-product-component--WellLog:{{NewWellLogUWI}}",
"kind": "{{authority}}:{{schemaSource}}:work-product-component--WellLog:1.0.0",
"acl": {
"owners": [
"{{New_OwnerDataGroup}}@{{data_partition_id}}{{domain}}"
],
"viewers": [
"{{New_ViewerDataGroup}}@{{data_partition_id}}{{domain}}"
]
},
"legal": {
"legaltags": [
"{{tagName}}"
],
"otherRelevantDataCountries": [
"US"
]
},
"data": {
"Source": "AUTO test NL_TNO",
"SpatialLocation": {
"Wgs84Coordinates": {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
3.51906683,
55.68101428
]
},
"properties": {}
}
]
}
},
"FacilityID": "FaciltyIdAutoTest_{{NewWellLogUWI}}",
"FacilityTypeID": "{{FacilityTypeRecordID}}:",
"FacilityOperator": [
{
"FacilityOperatorID": "FacilityOperatorIdAutoTest_{{NewWellLogUWI}}",
"FacilityOperatorOrganisationID": "{{data_partition_id}}:master-data--Organisation:Auto_Test_{{NewWellLogUWI}}:"
}
],
"FacilityName": "FacilityNameAutoTest_{{NewWellLogUWI}}",
"FacilityNameAlias": [
{
"AliasName": "Alias_Auto_Test_{{NewWellLogUWI}}",
"AliasNameTypeID": "{{AliasNameTypeRecordID}}:"
}
],
"FacilityEvent": [
{
"FacilityEventTypeID": "{{FacilityEventTypeSpudDateRecordID}}:",
"EffectiveDateTime": "1999-06-03T00:00:00"
}
],
"VerticalMeasurements": [
{
"VerticalMeasurementID": "Kelly Bushing",
"VerticalMeasurement": 36.6,
"VerticalMeasurementPathID": "{{VerticalMeasPathRecordID}}:"
}
],
"NameAliases": [],
"GeoContexts": []
}
}
]
}
}
}
```
## Expected Behavior:
Ingestion workflow should throw a validation error because `"id": "{{data_partition_id}}:work-product-component--WellLog:{{NewWellLogUWI}}"` belongs to the work product component and should be in the `Data` field.
## Observed Behavior:
Ingestion runs successfully and no error is shownhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/issues/55Slow performance for fetch curve data requests2023-02-26T18:10:03ZMichaelSlow performance for fetch curve data requestsFetching curve data in Wellbore DDMS curve data request appears to take at least 800 milliseconds, even if the size of the curve data is small.
Ideally, the typical fetch curve data request should take less than 500 milliseconds.
The f...Fetching curve data in Wellbore DDMS curve data request appears to take at least 800 milliseconds, even if the size of the curve data is small.
Ideally, the typical fetch curve data request should take less than 500 milliseconds.
The following Wellbore DDMS curve data request will fetch curve data for a well log in the Azure M15 pre-shipping environment. This well log only has 3 curves with 5 indices, however, it can take 1600 milliseconds to complete.
```
curl --location --request GET 'https://osdu-ship.msft-osdu-test.org/api/os-wellbore-ddms/ddms/v3/welllogs/opendes:work-product-component--WellLog:AutoTest_999265713361/data' \
--header 'data-partition-id: opendes' \
--header 'offset: 0' \
--header 'limit: 100' \
--header 'curves;' \
--header 'describe: false' \
--header 'orient: split' \
--header 'Authorization: Bearer {{access_token}}'
```https://community.opengroup.org/osdu/platform/deployment-and-operations/terraform-deployment-aws/-/issues/2Where we will get the OSDU deployment urls2022-12-22T16:24:14ZsudheerWhere we will get the OSDU deployment urlsI deployed the code but am unable to find the OSDU url.
Where will get the url and provide the complete url list for all services.I deployed the code but am unable to find the OSDU url.
Where will get the url and provide the complete url list for all services.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/92Refactor base images and npm dependencies version2023-03-24T15:54:04ZAliaksandr Ramanovich (EPAM)Refactor base images and npm dependencies versionIt seems it's time to update versions of base images used in Dockerfiles and dependencies in the package.json fileIt seems it's time to update versions of base images used in Dockerfiles and dependencies in the package.json fileSacha BrantsSacha Brantshttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/120System endpoint should also be accessible to user with admin roles2023-11-29T21:16:40ZAbhishek Kumar (SLB)System endpoint should also be accessible to user with admin rolesAs per initial design, there was a need for reserved partition to store **SHARED** schemas so that users can share schemas across partitions. For example, boostrapped schemas can be accessed by any user from any partition. This design do...As per initial design, there was a need for reserved partition to store **SHARED** schemas so that users can share schemas across partitions. For example, boostrapped schemas can be accessed by any user from any partition. This design does not restrict users to create and share their schemas directly rather going through the bootstrapping route.
Initially, creation of all the schemas were through _**common PUT/POST endpoint**_ and the decision of whether schema is **SHARED** or **INTERNAL** was decided based on the data-partition-id mentioned in the header. This design had its own challenges:
- Necessity of maintaining dedicated partition which all end users must be aware about
- Due to common endpoint for both **SHARED** and **INTERNAL** schemas, the decision making was driven through the data-partition-id in the header.
- Could not put any governance at the URL level for better access management.
![image](/uploads/44297a56933116a349c97b9f3e718c38/image.png)
Considering above challenges separate system endpoint was introduced to bifurcate **INTERNAL** and **SHARED** schema creation. Which means SHARED schemas can only be created using system endpoint without specifying any data-partition-id. This, however, was a breaking change and hence it was achieved in two phases:
1. **Creation of System partition:**
At this stage, only system partition was introduced without impacting the user experience. As a result, all existing **SHARED** partitions were moved to system partition and new requests were re-directed to system partitions i.e no new SHARED schemas were getting created into reserved partition (default)
![image](/uploads/42e8bb9df4441cbc7d319101f41582ae/image.png)
2. **Introduction of new endpoint** (https://{{domain}}/api/schema-service/v1/schemas/system):
Because of a dedicated endpoint, **SHARED** schemas are created without providing any data-partition-id in the header.
![image](/uploads/e76b6b085b9e4213d7178a5d8cddc8ec/image.png)
But there is a issue with this design, it is always expected to create SHARED schemas only through bootstrap script which uses service principal. As a result of this, users who were previously able to create and share their schemas are impacted. They now only need to go through bootstrapping process.
To fix this, we need to restrict the access to this endpoint through some entitlement service group role like service **schema-service.admin**.https://community.opengroup.org/osdu/platform/system/storage/-/issues/156ADR: Recover a soft deleted record in storage2023-09-11T08:27:45ZAbhishek NandaADR: Recover a soft deleted record in storageAbility to recover a soft deleted record in storage service
# Decision Title
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The storage service provides 2 ways to delete a r...Ability to recover a soft deleted record in storage service
# Decision Title
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The storage service provides 2 ways to delete a record. One way is to logically delete the record in which the record with same id can be revived later because its version history is maintained and the other one is to purge the record in which case, the record's version history is deleted too. In both types of deletions, the record cannot be accessed using storage or search service.
Today there is no easy way to query or recover the soft-deleted records. Providing admin-only APIs will help admins to search, view and recover the soft-deleted data if required.
# Tradeoff Analysis - Input to decision
Today users have to maintain the soft deleted record IDs on their own. Below is the workaround available today to attempt recovery of such records
1. Recreate the record with existing id and random/empty data and meta blocks. This will mark the record as active.
2. Fetch all versions of the record.
3. Fetch the latest version prior to the one just created to get back the actual record data and meta blocks.
4. Recreate the record using the response to create a new version of the record with the appropriate data.
## Decision
Create 3 new APIs as below
1. Fetch deleted records (accessible to _users.datalake.admins_) -> This will fetch a list of records. Since the list can be very long it should return a maximum of 100 records and support a from and to deletion dates filter along with pagination.
![image](/uploads/ca34cf94f3184fba05d2ade6bb502a90/image.png)
2. Recover deleted records by id (accessible to _users.datalake.admins_) -> This will take a list of record ids (max 500) that are to be recovered and return the list of record ids that succeeded as well as failed.
![image](/uploads/ae448c5fb9ed5803101aeba51a4fd7b4/image.png)
3. Recover deleted records by metadata filters (Currently support for only fromDeletedDate and toDeletedDate) (accessible to _users.datalake.admins_) -> This will take filter criteria of records that are to be recovered and return the list of record ids that succeeded as well as failed.
![image](/uploads/2b1d373eed8513e166fba784be4b3250/image.png)
## Consequences
1. This will help users to bulk recover deleted records in a single go.
2. The APIs will help prevent having garbage record versions that had to be created just to make the record active.
3. This will help users to fetch a list of soft deleted records which was not possible earlier.
Open API spec for the service
[storage-recover-swagger.yaml](/uploads/396cc62881dfe5f075f0e987f0313472/storage-recover-swagger.yaml)https://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/200Deployment - SPIKE: Investigate client id and client secret retrieval2022-12-14T16:53:18ZJoel RomeroDeployment - SPIKE: Investigate client id and client secret retrievalhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/issues/25implications associated with Python(pickle) module.2022-12-30T09:03:08ZJayesh Bagulimplications associated with Python(pickle) module.I am working on a vulnerability issue of the Pickle module in open-zgy.
https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/security/vulnerabilities/18266
The pickle library’s documentation discour...I am working on a vulnerability issue of the Pickle module in open-zgy.
https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/security/vulnerabilities/18266
The pickle library’s documentation discourages the unpickling of untrusted data. Currently, deserialization is happening with a simple approach.
To prevent unsafe deserialization there are multiple approaches are there.
1) Implementing a message authentication code (MAC) to ensure the data integrity of the payload. (hmac and hashlib)
2) Run the deserialization code with limited access permissions.
3) Validate Inputs.
I would like to hear which will best suit it as well as compatibility option for all existing things.
CC: @Srinivasan_Narayanan @nursheikh @chadJayesh BagulJayesh Bagulhttps://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/issues/33LegalTag name field max length is too small2023-03-03T05:54:14ZDadong ZhouLegalTag name field max length is too smallShell is currently migrating the SDU E&O contracts to OSDU LegalTags. We use long human-readable names as the ids and the names are much longer then the current LegalTag name field max length(100). Is it possible to increase the name fie...Shell is currently migrating the SDU E&O contracts to OSDU LegalTags. We use long human-readable names as the ids and the names are much longer then the current LegalTag name field max length(100). Is it possible to increase the name field max length to like 400? Thanks.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/162Byte swap bug in Ibm2ieee2023-06-20T12:16:12ZArtem ВByte swap bug in Ibm2ieeeWe found that `Ibm2ieee` implementation does not take into account file and host endianness.
For example:
```
template<SEGY::Endianness ENDIANNESS, SEGY::BinaryHeader::DataSampleFormatCode FORMAT>
void copySamples(float * prTarget, con...We found that `Ibm2ieee` implementation does not take into account file and host endianness.
For example:
```
template<SEGY::Endianness ENDIANNESS, SEGY::BinaryHeader::DataSampleFormatCode FORMAT>
void copySamples(float * prTarget, const unsigned char * puSource, int iSampleMin, int iSampleMax)
{
if (ENDIANNESS == SEGY::Endianness::BigEndian)
{
nValue = (int)(puSource[iSample * 4 + 0] << 24 | puSource[iSample * 4 + 1] << 16 | puSource[iSample * 4 + 2] << 8 | puSource[iSample * 4 + 3]);
}
else
{
nValue = (int)(puSource[iSample * 4 + 3] << 24 | puSource[iSample * 4 + 2] << 16 | puSource[iSample * 4 + 1] << 8 | puSource[iSample * 4 + 0]);
}
}
```
use native conversion with real file endianess.
The original 'Ibm2ieee' form SEGY.cpp always swaps bytes:
```
template<SEGY::Endianness ENDIANNESS, SEGY::BinaryHeader::DataSampleFormatCode FORMAT>
void copySamples(float * prTarget, const unsigned char * puSource, int iSampleMin, int iSampleMax)
{
if (FORMAT == SEGY::BinaryHeader::DataSampleFormatCode::IBMFloat)
{
SEGY::Ibm2ieee(prTarget, puSource + iSampleMin * 4, iSampleMax - iSampleMin);
return;
}
}
void ibm2ieee(void * to, const void * from, size_t len)
{
...
#ifdef WIN32
fr = _byteswap_ulong(fr);
#else
fr = __builtin_bswap32(fr);
#endif // WIN32
...
}
```
It is needed to check real file and hosts endianess before swap bytes.
---
It makes wrong traces data for little-endian SEG-Ys with IBMFloats data formathttps://community.opengroup.org/osdu/platform/deployment-and-operations/terraform-deployment-aws/-/issues/1Deployment instructions have some issues2022-12-22T16:16:51ZDaryl GunnDeployment instructions have some issuesI've been trying to deploy OSDU platform to AWS (main and release 0.17) but had some challenges:
- Terraform validate shows a couple of undefined provider warnings and some deprecated argument warnings (different numbers of each in main...I've been trying to deploy OSDU platform to AWS (main and release 0.17) but had some challenges:
- Terraform validate shows a couple of undefined provider warnings and some deprecated argument warnings (different numbers of each in main or 0.17)- I'm not familiar with terraform to know if these have consequences.
- There's a typo: Command `terraform deploy -var-file=../../<shared-resource-prefix>.tfvars.json -auto-approve` should be terraform 'apply'.
- There's a missing folder and file: Instructions reference `python3 ./devops/scripts/create-helm-values-file.py <resource-prefix> --region=<deploy-region>` but I can't find that folder or file.
- The helm chart command is described as `helm repo add core $HELM_REPO_PATH/core`, but no HELM_REPO_PATH parameter is set in the instructions. I assumed it was s3://osdu-artifacts but get access denied when trying to get charts from s3://osdu-artifacts/core. Although instructions state the bucket is public, it cannot be accessed or have its contents listed.https://community.opengroup.org/osdu/platform/data-flow/ingestion/opc-ua-ingestion/-/issues/17OPC-UA-PROD : Pass node ids from post service and get values2022-12-02T12:31:50ZAshutosh KumarOPC-UA-PROD : Pass node ids from post service and get valuesWrite a post service which accepts dynamic number of node ids from request body and gets the relevant values from server.Write a post service which accepts dynamic number of node ids from request body and gets the relevant values from server.Ashutosh KumarAshutosh Kumarhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/91The v3 to v4 sync process needs to be implemented for all models2023-09-20T02:16:49ZSacha BrantsThe v3 to v4 sync process needs to be implemented for all modelshttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/90Fix deployment to allow access to in https://osdu-glab.msft-osdu-test.org/sei...2023-06-15T14:28:28ZSacha BrantsFix deployment to allow access to in https://osdu-glab.msft-osdu-test.org/seistore-svc/api/v4/swagger-ui.htmlhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/89Implement dataset storage for AWS2023-09-27T13:19:38ZSacha BrantsImplement dataset storage for AWShttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/86osdu:wks:work-product-component--NotionalSeismicLine:1.0.02023-03-24T15:58:23ZSacha Brantsosdu:wks:work-product-component--NotionalSeismicLine:1.0.0