OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2023-11-21T09:49:45Zhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/issues/28SASSA Status Check2023-11-21T09:49:45ZGhost UserSASSA Status CheckThe SASSA Status Check service is a pivotal online tool in South Africa, offering beneficiaries a streamlined means to monitor the progress of their social security grants. By entering essential details and application information, users...The SASSA Status Check service is a pivotal online tool in South Africa, offering beneficiaries a streamlined means to monitor the progress of their social security grants. By entering essential details and application information, users can easily track the status of their applications, including processing stages and upcoming payment dates. This transparent and accessible platform not only reduces uncertainty for recipients but also promotes accountability within the social security system. The [SASSA Status Check](https://sassastatuscheck.org.za/) service is a testament to the agency's commitment to efficiency and open communication, ensuring that those relying on social assistance can stay informed and have confidence in the grant distribution process.https://community.opengroup.org/osdu/platform/home/-/issues/44[ADR] Avoid the need to provide persistable reference information (Unit syste...2023-11-17T17:48:56ZDebasis Chatterjee[ADR] Avoid the need to provide persistable reference information (Unit system, Coordinate Reference System)## Status
- [x] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context
Currently, user needs to provide both the Reference Entity information and the persistable reference.
Evident when user needs to specify...## Status
- [x] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context
Currently, user needs to provide both the Reference Entity information and the persistable reference.
Evident when user needs to specify unit of measure or coordinate reference system.
This is inefficient and is also error prone.
What if the Data Loader or the User makes a mistake in persistable reference value and the values are inconsistent?
The proposal is to save the user this trouble. Let the user provide link to existing Reference entity and ID.
However, programs such as Manifest-based Ingestion could query Reference value and add required line in JSON file being used to actually store/populate record.
See this high level diagram for an understanding of this proposal.
[ADR-for-persistableReference-issue.pptx](/uploads/95b748500e8f64544ff64a492836515d/ADR-for-persistableReference-issue.pptx)
You can find historical context in the following issues:
- https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/92
- https://community.opengroup.org/osdu/platform/system/reference/unit-service/-/issues/24
- https://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service/-/issues/25
## Scope
Make suitable change in code :
- Manifest-based Ingestion (both unit and CRS)
- Unit conversion
- CRS conversion
Likely impact for CSV Parser and WITSML Parser.
## Decision
Analyze impact (if adverse) caused by this suggested change :
1. Approve change to Manifest-based Ingestion
2. Approve change to Unit Conversion API
3. Approve change to CRS Conversion API
## Rationale
The proposal is cleaner approach since the lengthy persistable reference can be daunting for some of the users.
There is chance of user making mistake in the string of persistableReference.
Also, there can be contradiction in provided input (such as reference to certain entry in Reference data but different persistableReference).
## Consequences
- Revise sample JSON files (used for loading TNO, Volve data)
- Revise test Postman collections (Platform Validation team)
- Revise documentation (Data Loading)
- Revise API Swagger documentationhttps://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/315Remove callback url from gcz Authentication implementation2023-11-17T16:14:58ZAnkita SrivastavaRemove callback url from gcz Authentication implementationRemove callback url from gcz Authentication implementationRemove callback url from gcz Authentication implementationGCZ Sprint 53https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/54Enabling User Context in Ingestion, Google Cloud approach2023-11-17T10:16:09ZAndrei Dalhikh [EPAM/GC]Enabling User Context in Ingestion, Google Cloud approach## User impersonation approach
To address the [highlighted security concerns](https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52#note_134717) in the related [ADR](https://community.opengroup.org/osdu/plat...## User impersonation approach
To address the [highlighted security concerns](https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52#note_134717) in the related [ADR](https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52) we make user impersonation the responsibility of the Entitlements service. To have this we do the following:
- Limit Airflow SA with single role to impersonate users (group: **users.datalake.delegation**)
- Add special group (**users.datalake.impersonation**) to the users, which MAY be impersonated (in general - workflow service users)
- Implement the following authorisation flow:
## Flow diagram
![user_impersonation_flow.drawio](/uploads/b870acc57f2450139ecd827e60d37a0a/user_impersonation_flow.drawio.png)
1. User initiates DAG execution through the Workflow service API request
2. Workflow service makes state record in database and stores user id as workflow run id (submittedBy)
3. DAG obtains user id from workflow db record by Workflow service API request (implemented in Python SDK)
4. DAG performs a call to some OSDU service, Python SDK library code injects all outgoing requests with the Airflow service account token and the special "on-behalf-of" header. The "on-behalf-of" header value is equal to the user id obtained at the previous step
5. OSDU service passes incoming request headers to the Entitlements service to authorise user request
6. Entitlements service performs the following flow:
- Only `/group` endpoint will support impersonation flow, Entitlements management endpoints like add member, create group, etc will ignore "on-behalf-of" header.
- If request to Entitlements endpont `/groups` contains special "on-behalf-of" header and the user (Airflow SA) that is willing to act on behalf is NOT a member of the special group to impersonate users (group: users.datalake.delegation) then service returns HTTP 403 Forbidden status
- If request contains special "on-behalf-of" header and the user (Airflow SA) that is willing to act on behalf belongs to special group to impersonate users (group: users.datalake.delegation) then it collects groups for the impersonated user specified in this header.
- If the impersonated user (that initially triggered workflow, and should be acknowledged as an owner/creator of ongoing changes) group list does NOT contain special group (users.datalake.impersonation) then Entitlements service returns HTTP 403 Forbidden.
- Else the impersonated user group list returned to the calling OSDU service
7. OSDU service performs usual check of the returned groups list for the presence of specific rights to make the call.
- Entitlements service response should NOT be cached due to security reasons in the case of "on-behalf-of" request header presence
See also: [Entitlements service documentation](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/tree/master/provider/entitlements-v2-jdbc#authorisation-flow-for-impersonated-users)M20 - Release 0.23Andrei Dalhikh [EPAM/GC]Andrei Dalhikh [EPAM/GC]https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/220Add CRS database lookup to SEGYImport tool2023-11-16T12:21:10ZMorten OfstadAdd CRS database lookup to SEGYImport toolIt would be so much easier if it was possible to specify CRS by using the EPSG ID or UTM zone to look up the CRSWkt automatically, e.g. like this: --crs EPSG:23031 or --crs UTM-31N to get the WKT
```
PROJCS["ED50 / UTM zone 31N",GEOGCS["...It would be so much easier if it was possible to specify CRS by using the EPSG ID or UTM zone to look up the CRSWkt automatically, e.g. like this: --crs EPSG:23031 or --crs UTM-31N to get the WKT
```
PROJCS["ED50 / UTM zone 31N",GEOGCS["ED50",DATUM["European_Datum_1950",SPHEROID["International 1924",6378388,297],TOWGS84[-87,-98,-121,0,0,0,0]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4230"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",3],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","23031"]]
```
I looked up this at [https://epsg.io/23031](https://epsg.io/23031)https://community.opengroup.org/osdu/platform/system/partition/-/issues/34Change response code for RequestRejectedException2023-11-16T10:54:27ZNeha KhandelwalChange response code for RequestRejectedExceptionCurrently, when a request URL contains an unknown or potentially malicious string, Spring Security utilizes a HttpFirewall interface to reject the request with a _org.springframework.security.web.firewall.RequestRejectedException._ This ...Currently, when a request URL contains an unknown or potentially malicious string, Spring Security utilizes a HttpFirewall interface to reject the request with a _org.springframework.security.web.firewall.RequestRejectedException._ This exception will return as 500 Internal Server Error with the message "The request was rejected because the URL contained a potentially malicious String \[string\]." An example of such a string is "//".
Since this error is caused by a bad request from the user, the retuned response should instead be a 400 Client Error. Furthermore, keeping the response as a 500 error can impact the SLIs/SLOs of both SDMS and the Partition Service.
The purposed solution is to implement a RequestRejectedHandler to change the response code to 400 when there is a RequestRejectedException.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/issues/86Provide Data R/W endpoint for Marker "content" (not "catalog") data (see use ...2023-11-15T23:47:19ZDebasis ChatterjeeProvide Data R/W endpoint for Marker "content" (not "catalog") data (see use case of AvailableMarkerPropertiesThis link will show typical use case where such external data may be persisted in "optimized content" similar to Well Log Curve or Trajectory Station properties.
https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E...This link will show typical use case where such external data may be persisted in "optimized content" similar to Well Log Curve or Trajectory Station properties.
https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/Examples/WorkedExamples/WellboreMarkerSet/README.mdhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/189[SAST] Vue_DOM_XSS in file index.html2023-11-15T10:54:25ZYauhen Shaliou [EPAM/GCP][SAST] Vue_DOM_XSS in file index.html**Description**
The method m-1"\> embeds untrusted data in generated output with href, at line 36 of \\storage\\provider\\storage-azure\\src\\main\\resources\\static\\index.html. This untrusted data is embedded into the output without p...**Description**
The method m-1"\> embeds untrusted data in generated output with href, at line 36 of \\storage\\provider\\storage-azure\\src\\main\\resources\\static\\index.html. This untrusted data is embedded into the output without proper sanitization or encoding, enabling an attacker to inject malicious code into the generated web-page.
# **Location:**
<table>
<tr>
<th> </th>
<th>Source</th>
<th>Destination</th>
</tr>
<tr>
<th>File</th>
<td>storage/provider/storage-azure/src/main/resources/static/index.html</td>
<td>storage/provider/storage-azure/src/main/resources/static/index.html</td>
</tr>
<tr>
<th>Line number</th>
<td>92</td>
<td>36</td>
</tr>
<tr>
<th>Object</th>
<td>pathname</td>
<td>href</td>
</tr>
<tr>
<th>Code line</th>
<td>return location.protocol + '//' + location.host + location.pathname</td>
<td>
\<a :href="signInUrl" class="btn btn-primary" v-if="!token" class="col-2"\>Login\</a\>
</td>
</tr>
</table>M21 - Release 0.24https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/136[SAST] SQL_Injection in the ClusterGremlinConnector file2023-11-15T10:54:14ZYauhen Shaliou [EPAM/GCP][SAST] SQL_Injection in the ClusterGremlinConnector file<table>
<tr>
<th> </th>
<th>
</th>
<th>Destination</th>
</tr>
<tr>
<th>File</th>
<td>
</td>
<td>entitlements/provider/entitlements-v2-azure/src/main/java/org/opengroup/osdu/entitlements/v2/azure/spi/gremlin/connection/ClusterGremlinCon...<table>
<tr>
<th> </th>
<th>
</th>
<th>Destination</th>
</tr>
<tr>
<th>File</th>
<td>
</td>
<td>entitlements/provider/entitlements-v2-azure/src/main/java/org/opengroup/osdu/entitlements/v2/azure/spi/gremlin/connection/ClusterGremlinConnector.java</td>
</tr>
<tr>
<th>Line number</th>
<td>
</td>
<td>102</td>
</tr>
<tr>
<th>Object</th>
<td>
</td>
<td>getResultList</td>
</tr>
<tr>
<th>Code line</th>
<td>
</td>
<td>return getResultList(client.submit(query));</td>
</tr>
</table>
**Description**
The application's submitTraversalAsQueryString method executes an SQL query with getResultList, at line 102 of \\entitlements\\provider\\entitlements-v2-azure\\src\\main\\java\\org\\opengroup\\osdu\\entitlements\\v2\\azure\\spi\\gremlin\\connection\\ClusterGremlinConnector.java. The application constructs this SQL query by embedding an untrusted string into the query without proper sanitization. The concatenated string is submitted to the database, where it is parsed and executed accordingly.An attacker would be able to inject arbitrary syntax and data into the SQL query, by crafting a malicious payload and providing it via the input groupEmail; this input is then read by the listGroupMembers method at line 58 of \\entitlements\\entitlements-v2-core\\src\\main\\java\\org\\opengroup\\osdu\\entitlements\\v2\\api\\ListMemberApi.java. This input then flows through the code, into a query and to the database server - without sanitization.This may enable an SQL Injection attack.M21 - Release 0.24Srinivasan NarayananSrinivasan Narayananhttps://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure/-/issues/30istio2023-11-15T09:30:33ZJANRAJ CJistiohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/111is_batch option still supporting2023-11-14T17:13:24ZBruce Jinis_batch option still supportingIn `manifest_ingestion` dags, we have 2 payload types, which leads to 2 flows, one is batch upload, another one is 3 step with some validations with schema and integrity.
My question is, are we still using the batch_upload option on the ...In `manifest_ingestion` dags, we have 2 payload types, which leads to 2 flows, one is batch upload, another one is 3 step with some validations with schema and integrity.
My question is, are we still using the batch_upload option on the left?
Since we are not checking for schema and integrity on that flow, and we have option to split 1 large file in batches in `processing_single_manifest_file_task![Screenshot_2023-11-13_at_3.24.18_PM](/uploads/8a9242b16809d5dcaf1237c4c9152049/Screenshot_2023-11-13_at_3.24.18_PM.png).https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/28ADR: EDS DMS - Bulk Data Ingestion and Dataset ID Naturalization2023-11-14T15:34:11ZNisha ThakranADR: EDS DMS - Bulk Data Ingestion and Dataset ID Naturalization**Introduction:**
The purpose of this ADR is to address the advanced use case of EDS DMS to Add bulk data into the OSDU Platform. Additionally, it focuses on naturalizing the dataset ID associated with the relevant schemas for Work-Prod...**Introduction:**
The purpose of this ADR is to address the advanced use case of EDS DMS to Add bulk data into the OSDU Platform. Additionally, it focuses on naturalizing the dataset ID associated with the relevant schemas for Work-Product-Component (WPC) record that has linked data files. By importing and naturalizing the dataset IDs, we aim to improve the capabilities of handling bulk data efficiently and to ensure that the data files are properly added to the OSDU Platform and the WPCs' child datasets are converted from "external" to "internal" type, providing improved accessibility and integration of the data within the platform.
**Objective:**
Currently, the EDS fetch-and-ingest process only copies the metadata, and the child dataset of the WPC at the operator's end is flagged as "external." While operators can use EDS DMS to obtain the actual data file when needed, but it is not properly added to the Data Platform. This approach proposes a solution to address these limitations by appropriately adding the data file to the OSDU Platform and converting the WPC's child dataset from "external" to "internal.”
**Status**
- [x] Proposed
- [x] Trialing
- [x] Under review
- [ ] Approved
- [ ] Retired
**Scope**
The scope of this ADR includes the following scenarios:
- Importing Bulk Data: Importing bulk data from the provider end into the Data Platform using EDS DMS.
- File Storage in OSDU Instance: The use of the dataset API to store the fetched files back to the operator's OSDU instance.
- Naturalizing Dataset IDs and re ingestion: The naturalization of dataset IDs from external to internal to align them with the relevant schemas within the Data Platform.
**Given /assumptions:**
- WPC metadata is already present at the operator end.
**Required Changes:**
- eds_dataset: New Dag will be introduced to perform the naturalization of the bulk data after the upload of the data files at the operator’s end.
- The DAG will be executed on-demand without the involvement of a scheduler.
- An additional boolean parameters should be introduced in the internal dataset schemas(File.Generic) to indicate whether the dataset id is internal or naturalized, to keep a track of the naturalized dataset ids.we can achieve this by using the ExtensionProperties of the dataset schemas.
**DatasetExternal:True**
* During the naturalization process, it is important to maintain a similar dataset ID while allowing for a conversion of the data type i.e from external to internal, it will be easier for reverse mapping. Example: opendes:dataset--ConnectedSource.Generic:test123 will be converted to: opendes:dataset--File.Generic:test123
we can also have additional parameters within the ExternalProperties like pointer to the external dataset id (connectedSource.Generic) with the version.
eg: Parent_dataset_id:opendes:dataset--ConnectedSource.Generic:test123
Parent_dataset_id_version: 1614105463059152
**Inputs:**
The inputs for the naturalization process should be an array of WPC IDs. Each WPC ID represents a specific work-product-component. Here is an example of the input structure:
[ "osdu:work-product-component--WellLog:7fdf1681b7ed1a1d54046ca1c2438add13719fafd18295a5e35d7bbdb45a53e4", "osdu:work-product-component--WellLog:7fdf1681b7ed1a1d54046ca1c2438ad89cf67"]
**Implementation:**
- Retrieve the ConnectedSource.Generic by providing the specified input IDs (well Log id, wellboreTrajectory id).
- Initiate the EDS DMS Operator, passing the ConnectedSource.Generic as input, and retrieve the signed URL.
- Download the LAS file at the operator's end using the obtained signed URL.
- Utilize the Dataset File service to upload the downloaded LAS file to the OSDU instance for the operator.
- Convert the child dataset of the WPC from "external" to "internal" by reassigning it to the cloud location mentioned above, such as changing ConnectedSource.Generic to File.Generic.
**Exception Handling:**
- Dataset ID is not present with the given WPC ids: if dataset id is missing, raise an exception or return an error indicating the issue.
- Invalid or Unrecognized IDs WPC ids or Dataset IDs: Handle cases where the provided WPC ID or dataset ID is invalid or unrecognized at operator’s end.
- Data Retrieval Errors: Implement error handling for situations where the dataset associated with the ID cannot be retrieved.
**Sequence Diagram**
![image](/uploads/f82c52840380e323c48b7a3e7f0084ab/image.png)
**Functional Requirements:**
- Data Import:
- The system should provide functionality to import bulk data from the provider's end using EDS DMS.
- EDS DMS Integration:
- The system should integrate with the EDS DMS to establish a connection with the provider's data
source.
- Dataset ID Naturalization:
- The system should support the naturalization of dataset IDs (external to internal) associated with
different schemas by uploading the imported data back to the Operator's OSDU instance and creating the
internal id.
- Mapping and re ingest:
- Mapping the transformed dataset id back to the schema id and and proceed to re-ingest the
transformed data back to the instance, ensuring its seamless integration.
**Non-functional Requirements**
- Performance:
- The system should be capable of handling large volumes of wellbore data efficiently, providing fast response times for data retrieval and analysis.
- It should be able to handle concurrent user interactions and maintain performance under peak load conditions.
- Scalability:
- The system should be scalable to accommodate increasing amounts of wellbore data and growing user bases.
- It should be able to handle additional data sources and support a high number of concurrent users without significant degradation in performance.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/issues/84Challenge ingesting certain subfiles in LIS well log format2023-11-14T14:35:23ZBjørn Harald FotlandChallenge ingesting certain subfiles in LIS well log formatHi,
we are working with an operator performing bulk ingestion of different types of well log file formats (LAS, DLIS, LIS). We have met some challenges with the LIS files originating from from DISKOS.
A LIS file may consist of several ...Hi,
we are working with an operator performing bulk ingestion of different types of well log file formats (LAS, DLIS, LIS). We have met some challenges with the LIS files originating from from DISKOS.
A LIS file may consist of several subfiles with metadata and data, each one corresponding to an OSDU WellLog record and data for each curve.
What we observe is that a number of LIS subfiles do not fulfil the requirements of the Wellbore DDMS regarding ReferenceCurve properties. The requirements being that the ReferenceCurve is sorted (decreasingly/increasingly) and that the reference curve samples are unique. The reference curve are typically depth samples.
Example: Repeated depth samples with other curve samples have different values for the same depth.
DEPTH GR CALI
2299.0 .. ..
2300.0 10.0 9.0
2300.0 20.0 8.0
2301.0 .. ..
In this case it is challenging to find an approach to ingest the curve data into the Wellbore DDMS without losing data fidelity or introducing manual processes.
Given that the data need to be loaded into the Wellbore DDMS, what would be the recommended approach?
Ideally keeping the data in the first version of the WellLog for updating/fixing the reference curve in a later WellLog version/new WellLog record using applications/workflows performed on top of OSDU.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/105GET /v1/workflow/{workflow_name}/workflowRun getAllRunInstances parameters2023-11-14T14:35:21ZAalekh JainGET /v1/workflow/{workflow_name}/workflowRun getAllRunInstances parameters1. Please specify the expected behaviour from the following parameters present in the api spec for get all run instances of a workflow
* [boolean] **partial**
* [string] **conf**
2. What is the expected format in which the start dat...1. Please specify the expected behaviour from the following parameters present in the api spec for get all run instances of a workflow
* [boolean] **partial**
* [string] **conf**
2. What is the expected format in which the start date and end date is provided. Is it going to be Unix epoch time? In case it's going to be unix epoch time, maybe it's better to name it `startTimeStamp` rather than `startDate` and `endTimeStamp` rather than `endDate`. A minor nit, the type of `endDate` is mentioned as boolean in the api spec. Should be string.
3. Currently, the response only returns the list of workflow runs and the cursor for the subsequent requests is not being returned. In such a scenario, how can one obtain the cursor for the subsequent request. Shouldn't we pass the cursor as part of the response? How do we plan to handle this scenario?
4. What is the expected behaviour when an invalid parameter is passed (apart from `limit`, `cursor`, `startDate`, `endDate` etc.)?
cc: @kibattulhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/121[SAST] Client_Privacy_Violation in file queue.ts2023-11-13T15:53:55ZYauhen Shaliou [EPAM/GCP][SAST] Client_Privacy_Violation in file queue.ts**Description**
Method setup at line 42 of \\seismic-store-service\\app\\sdms\\src\\cloud\\shared\\queue.ts sends user information outside the application. This may constitute a Privacy Violation.
<table>
<tr>
<th> </th>
<th>Source</th...**Description**
Method setup at line 42 of \\seismic-store-service\\app\\sdms\\src\\cloud\\shared\\queue.ts sends user information outside the application. This may constitute a Privacy Violation.
<table>
<tr>
<th> </th>
<th>Source</th>
<th>Destination</th>
</tr>
<tr>
<th>File</th>
<td>seismic-store-service/app/sdms/src/cloud/shared/queue.ts</td>
<td>seismic-store-service/app/sdms/src/cloud/providers/azure/insights.ts</td>
</tr>
<tr>
<th>Line number</th>
<td>42</td>
<td>129</td>
</tr>
<tr>
<th>Object</th>
<td>password</td>
<td>log</td>
</tr>
<tr>
<th>Code line</th>
<td>redisOptions.password = cacheParams.KEY;</td>
<td>console.log(data);</td>
</tr>
</table>https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/120[SAST] SSL_Verification_Bypass in file cosmosdb.ts2023-11-13T15:22:49ZYauhen Shaliou [EPAM/GCP][SAST] SSL_Verification_Bypass in file cosmosdb.ts# **Location:**
<table>
<tr>
<th> </th>
<th>
</th>
<th>Destination</th>
</tr>
<tr>
<th>File</th>
<td>
</td>
<td>seismic-store-service/app/sdms/src/cloud/providers/azure/cosmosdb.ts</td>
</tr>
<tr>
<th>Line number</th>
<td>
</td>
<td>...# **Location:**
<table>
<tr>
<th> </th>
<th>
</th>
<th>Destination</th>
</tr>
<tr>
<th>File</th>
<td>
</td>
<td>seismic-store-service/app/sdms/src/cloud/providers/azure/cosmosdb.ts</td>
</tr>
<tr>
<th>Line number</th>
<td>
</td>
<td>67</td>
</tr>
<tr>
<th>Object</th>
<td>
</td>
<td>rejectUnauthorized</td>
</tr>
<tr>
<th>Code line</th>
<td>
</td>
<td>rejectUnauthorized: false</td>
</tr>
</table>
**Description**
\\seismic-store-service\\app\\sdms\\src\\cloud\\providers\\azure\\cosmosdb.ts relies HTTPS requests, in constructor. The rejectUnauthorized parameter, at line 67, effectively disables verification of the SSL certificate trust chain.
JavaScript Explicitly Disabling Certificate Verification var https = require('https'); var options = { hostname: 'domain.com', port: 443, path: '/', method: 'GET', rejectUnauthorized: false; }; options.agent = new https.Agent(options); var req = https.request(options, function(res) { res.on('data', function(d) { handleRequest(d); }); }); req.end();https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/36[SAST] Missing HSTS Header in file AuthSecurityConfig.java2023-11-13T15:14:10ZYauhen Shaliou [EPAM/GCP][SAST] Missing HSTS Header in file AuthSecurityConfig.java**Description**
The web-application does not define an HSTS header, leaving it vulnerable to attack.
crs-catalog-service/provider/crs-catalog-aws/src/main/java/org/opengroup/osdu/crs/security/AuthSecurityConfig.java
line number: 114
...**Description**
The web-application does not define an HSTS header, leaving it vulnerable to attack.
crs-catalog-service/provider/crs-catalog-aws/src/main/java/org/opengroup/osdu/crs/security/AuthSecurityConfig.java
line number: 114
Setting an HSTS Header in an HTTP Response response.setHeader("Strict-Transport-Security", "max-age=31536000; includeSubDomains");https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/109Confusion about the parent-child relationship for user.data.root group2023-11-10T23:01:40ZShuai LiConfusion about the parent-child relationship for user.data.root groupI am very confused about the parent-child relationship for _user.data.root_ group.
According to the documentation (https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/blob/master/docs/bootstrap/bootstrap-...I am very confused about the parent-child relationship for _user.data.root_ group.
According to the documentation (https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/blob/master/docs/bootstrap/bootstrap-groups-structure.md), the _user.data.root_ group is "_A group will be automatically added to all data groups so that the member of it has the permission to all data on the partition_". It means _user.data.root_ group is a member of all other data groups. In the OSDU implementation of parent-child relationship of roles, the "member" should be "child" and the target group should be "parent". The Add Member API works exactly in this way. "child" aggregates all rights of its parents. So in theory, _user.data.root_ group should be "child" of all other data groups. In this way, _user.data.root_ group will have all the rights of all other data groups.
But the implementation of parent-child relationship for _user.data.root_ group is the reversed order, i.e. _user.data.root_ group is parents of all other data groups. I think this implementation is wrong. It is not consistent with the documentation and the _user.data.root_ group cannot aggregate the rights of other data groups.
The wrong code logic is in addRootGroupNodeAsMemberOfGroupNewGroup method of org.opengroup.osdu.entitlements.v2.jdbc.spi.jdbc.creategroup.CreateGroupRepoJdbc. The name of this method indicates it will add root group as a member of a new data group, but its codes shows it adds the root group as parents of a new data group.
private void addRootGroupNodeAsMemberOfGroupNewGroup(GroupInfoEntity createdGroup, CreateGroupRepoDto createGroupRepoDto) {
GroupInfoEntity parentGroup = groupRepository
.findByEmail(createGroupRepoDto.getDataRootGroupNode().getNodeId())
.stream()
.findFirst()
.orElseThrow(() ->
new DatabaseAccessException(
HttpStatus.NOT_FOUND,
"Could not find the group with email: " +
createGroupRepoDto.getDataRootGroupNode().getNodeId()));
groupRepository.addChildGroupById(parentGroup.getId(), createdGroup.getId());
}
How could the root group aggregate the rights of all other data groups if it is the parent of other data groups.
**Besides the entitlement-v2-jdbc, entitlement-v2-AWS also implements the same wrong logic**. I have not checked the codes of other providers, I am not sure if they implement the logic correctly.
There is another evidence which can prove the logic of adding _user.data.root_ group as parents is wrong.
The run method in org.opengroup.osdu.entitlements.v2.service.CreateGroupService class has this logic (check the code below).
- when you are adding a new data group, it will check the existing parents of user.data.root group. If the number of parents of user.data.root group is larger than the quota, it will throw an exception. It implicitly indicates the user.data.root group should be child, not the parent. If user.data.root group is parent of every other data group, there will be no need to add this parent amount check logic in this method.
```
Set<ParentReference> allExistingParentsOfRootDataGroup = retrieveGroupRepo.loadAllParents(dataRootGroupNode).getParentReferences();
if (allExistingParentsOfRootDataGroup.size() >= dataRootGroupQuota) {
log.error(String.format("Identity %s already belong to %d groups", dataRootGroupNode.getNodeId(), allExistingParentsOfRootDataGroup.size()));
throw new AppException(HttpStatus.PRECONDITION_FAILED.value(), HttpStatus.PRECONDITION_FAILED.getReasonPhrase(), String.format("%s's group quota hit. Identity can't belong to more than %d groups",
dataRootGroupNode.getNodeId(), dataRootGroupQuota));
}
```
Maybe I have a wrong understanding, hope someone can give me some clarification. Thanks.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/reservoir/open-etp-server/-/issues/96Dockerfile.azure build error: ERROR: failed to solve: process "/bin/sh -c ech...2023-11-10T11:01:43ZBenjamin LaGroneDockerfile.azure build error: ERROR: failed to solve: process "/bin/sh -c echo \"== Install Core dependencies ==\" && tdnf install -yIn our deployment pipeline, in a Github Action, during the build and push to our container registry step:
**I have tried both below to the same effect:
- name: Build and push image to ACR
run: |
cd community/re...In our deployment pipeline, in a Github Action, during the build and push to our container registry step:
**I have tried both below to the same effect:
- name: Build and push image to ACR
run: |
cd community/reservoir-ddms/
docker build . -t ${{ vars.AZURE_CONTAINER_REGISTRY }}/open-etp-server:epam --file Dockerfile.azure
# az acr build --registry ${{ vars.AZURE_CONTAINER_REGISTRY }} \
# --image reservoirddms:latest \
# --file community/reservoir-ddms/Dockerfile.azure ./community/reservoir-ddms/
I get the error:
ERROR: failed to solve: process "/bin/sh -c echo \"== Install Core dependencies ==\" && tdnf install -y alsa-lib alsa-lib-devel autoconf automake binutils bison build-essential clang clang-devel cmake dbus dbus-devel dbus-glib dbus-glib-devel diffutils elfutils-devel file-libs flex fontconfig-devel gawk gcc gettext git glibc-devel glib-schemas gobject-introspection gobject-introspection-devel harfbuzz harfbuzz-devel kernel-headers intltool libatomic_ops libcap-devel libffi libffi-devel libgudev libgudev-devel libjpeg-turbo libjpeg-turbo-devel libltdl libltdl-devel libpng-devel libtiff libtiff-devel libusb libusb-devel libwebp libwebp-devel libxml2 libxml2-devel make meson newt nss nss-libs openldap openssl-devel pam-devel pango pango-devel patch perl-XML-Parser polkit-devel python2-devel python3-mako sqlite-devel systemd-devel UI-cairo UI-cairo-devel unzip vala vala-devel vala-tools zip" did not complete successfully: exit code: 21
Error: Process completed with exit code 1.
Has anyone seen this and resolved it?https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/reservoir/open-etp-client/-/issues/19Standardization of endpoints2023-11-09T14:26:30ZChad LeongStandardization of endpoints# Summary
Based on the recent M19 pre-shipping testing, we identified a few discrepancies in the Reservoir DDMS endpoints for both ETP and REST servers. We need to follow the standard convention of OSDU.
## Actual behavior
These are t...# Summary
Based on the recent M19 pre-shipping testing, we identified a few discrepancies in the Reservoir DDMS endpoints for both ETP and REST servers. We need to follow the standard convention of OSDU.
## Actual behavior
These are the endpoints observed today.
ETP Server:
- [AWS] wss://osdu.r3m18.preshiptesting.osdu.aws/api/oreservoir-ddms-etp/v2/
- [Azure] wss://osdu-ship.msft-osdu-test.org/oetp/reservoir-ddms/
- [Google] wss://preship.gcp.gnrg-osdu.projects.epam.com/api/oetp-server/v2/
- [IBM] ?
REST Server:
- [AWS] https://osdu.r3m18.preshiptesting.osdu.aws/api/reservoir-ddms/v2
- [Azure] https://osdu-ship.msft-osdu-test.org/api/oetp-client/v2
- [Google] https://preship.gcp.gnrg-osdu.projects.epam.com/api/reservoir-ddms/v2
- [IBM] ?
## Intended behavior
We should follow the standard endpoint convention in OSDU.
ETP Server:
- `{baserurl}/api/reservoir-ddms-etp/v2/`
REST server:
- `{baserurl}/api/reservoir-ddms/v2/{methods}`
## Pending Implementation
- [X] AWS
- [ ] Azure
- [X] GC
- [ ] IBMM21 - Release 0.24