OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2024-03-28T11:04:41Zhttps://community.opengroup.org/osdu/platform/system/project-and-workflow/-/issues/65F1 (Java) Story 13 Upgrade CollaborationProject CREATE operation business logic.2024-03-28T11:04:41ZRostislav Dublin (EPAM)F1 (Java) Story 13 Upgrade CollaborationProject CREATE operation business logic.Description
---
On project creation, the input JSON payload is a document of the CollaborationProject kind.
We need to validate the provided content and specifically augment it:
1. If `data.LifecycleStatus` is provided, it MUST be `name...Description
---
On project creation, the input JSON payload is a document of the CollaborationProject kind.
We need to validate the provided content and specifically augment it:
1. If `data.LifecycleStatus` is provided, it MUST be `namespace:reference-data--CollaborationProjectStatus:Open:`. Or we raise error.
2. If `data.LifecycleStatus` is NOT provided, we set it to `namespace:reference-data--CollaborationProjectStatus:Open:`.
3. If non-empty `data.LifecycleEvents` is provided, we raise an error.
4. We add a new [CollaborationLifecycleEvent](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blob/644-collaboration-project/E-R/abstract/AbstractCollaborationLifecycleEvent.1.0.0.md) to the `data.LifecycleEvents`:
```
{
"Name": "Create",
"Date": "<current timestamp>"
}
```
5. If "data.Namespace" is provided, we check its uniqueness and format correctness. If it is not unique or incorrect, we raise an error.
6. If "data.Namespace" is NOT provided, we generate a value, check its uniqueness, and re-generate it if is not unique. Then we set the value.
7. If `data.TrustedResourcesID` is provided - we raise an error.
Acceptance Criteria
---
All data validation and augmentation requirements listed in the description are implemented and tested.
Testing Scenarios
---
1. Test the `CREATE operation of the `ProjectService` class with both valid and invalid payloads.
2. Each validator should be intensively tested with varied payloads to ensure correct validations and modifications.
Technical Notes
---
1. For data store-read from OSDU use Storage client from os-core-common libVladimir Korolevskii (EPAM)Vladimir Korolevskii (EPAM)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/core/dspdm-services/-/issues/1Service should use only Community Maven Repositories2024-03-28T09:18:49ZDanylo Vanin (EPAM)Service should use only Community Maven Repositories## `pom.xml` files should point to only Community repositories.
In the current state several `pom.xml` files point to non-Community Maven Repositories.
Example (`src/pom.xml`, lines 192+):
```xml
<distributionManagement>
<rep...## `pom.xml` files should point to only Community repositories.
In the current state several `pom.xml` files point to non-Community Maven Repositories.
Example (`src/pom.xml`, lines 192+):
```xml
<distributionManagement>
<repository>
<id>dspdm-release</id>
<name>dspdm-release</name>
<url>https://repo.ds365.ai/artifactory/dspdm-maven-release</url>
</repository>
<snapshotRepository>
<id>dspdm-snapshots</id>
<name>dspdm-snapshots</name>
<url>https://repo.ds365.ai/artifactory/dspdm-maven-snapshots</url>
</snapshotRepository>
</distributionManagement>
```
Example (`src/dspdm.msp.mainservice/pom.xml`, line 52-53, but used as property in other parts of file):
```xml
<release.url>https://artifacts.repo.openearth.community/artifactory/distservices-maven-staging</release.url> <snapshot.url>https://artifacts.repo.openearth.community/artifactory/distservices-maven-snapshots</snapshot.url>
```
To make service compliant with the Community approach, please do the following:
1. Add `.mvn` folder with correct profile settings. Examples can be found in other Java services such as Storage (https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/.mvn/community-maven.settings.xml) or Search (https://community.opengroup.org/osdu/platform/system/search-service). Note the project id in the following URL types: `https://community.opengroup.org/api/v4/projects/**44**/packages/maven`. It should be configured to id of current repository (id of current project is 1245).
2. Remove any references to non-Community repositories inside `pom.xml` files. Examples of pom.xml files can be found in other Java-based services (e.g. [here](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/storage-core/pom.xml?ref_type=heads))
3. Each pom.xml should be configured so each artifact is published to Community Maven repository. References to this logic can be also found in other Java-based services.
Additional information on how Maven building and publishing jobs are configured in CI/CD can be found here: https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/blob/master/build/maven.ymlRiabokon Stanislav(EPAM)[GCP]Danylo Vanin (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/issues/220storage record with no acl owners become ghost record if OPA service is enabled.2024-03-28T06:20:14ZOm Prakash Guptastorage record with no acl owners become ghost record if OPA service is enabled.Storage records become inaccessible if OPA is enabled in case there is no ACL group associated with the record.
# Scenario:
Usually, when we create a record we define the owners and viewers group and the member associated with the group...Storage records become inaccessible if OPA is enabled in case there is no ACL group associated with the record.
# Scenario:
Usually, when we create a record we define the owners and viewers group and the member associated with the group can access the record. However, it is possible to delete the group and even disassociate ACL groups from the storage record. there is no validation as of now for a must-required single record. eventually record becomes a ghost record and nobody can access it.
There was a fix provided to users. data. root members can still access the group and add ACLs if needed.
it is discussed in this ADR
https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/141
# Findings
We have seen that code works fine and still users.data.root members can access the record if there is no associated ACL members for the record but if OPA is enabled we can not access the record even member is associated to users.data.root group.
code below checks if OPA is enabled and get access rights from OPA service
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/storage-core/src/main/java/org/opengroup/osdu/storage/service/IngestionServiceImpl.java#L198
OPA service returns with false access rites. However, if OPA is disabled the flow works because we have code added to return true if the member belongs to users.data.root.
We have found this not working in the Azure OSDU instance and need to know if requires a policy file fix or shall be handled in code to stop records from becoming ghost in case OPA is enabled.Dadong ZhouKelly ZhouShane HutchinsDeepa KumariDadong Zhouhttps://community.opengroup.org/osdu/platform/system/project-and-workflow/-/issues/44F1 DevOps Story 4: Set up a Basic CI/CD Pipeline2024-03-27T12:08:32ZRostislav Dublin (EPAM)F1 DevOps Story 4: Set up a Basic CI/CD PipelineDescription
---
Set up a fully functioning CI/CD pipeline that facilitates continuous
integration and deployment of the project code. The pipeline should cater
to the specific needs of the project, including building the application,
...Description
---
Set up a fully functioning CI/CD pipeline that facilitates continuous
integration and deployment of the project code. The pipeline should cater
to the specific needs of the project, including building the application,
running linters, checkers, unit and integration tests, building the Docker
image, placing the image into the OSDU GitLab container registry, running a
Docker container.
Acceptance Criteria
---
- A functioning CI/CD pipeline is established that can:
- [x] - Build the application.
- [x] - Run all linters and checkers.
- [x] - Run unit tests.
- [x] - Build the Docker image.
- [x] - Place the Docker image into the OSDU GitLab container registry.
- [x] - Run the Docker container.
- [x] - Run integration tests.
- The pipeline should trigger on any branch push on GitLab and also on the
merge request (MR) merge into the main branch.
Testing Scenarios
---
- Confirm that the pipeline successfully executes all the defined stages:
application build, linters and checkers, unit tests, Docker image build,
image placement into GitLab container, Docker container run, and
integration tests.
- Validate that the pipeline triggers and executes successfully post
a branch push or an MR merge on GitLab.
Technical Notes
---
- Consult documentation for the selected CI/CD tool for guidance on
setting up the pipeline.
- Align the setup with other OSDU services' pipelines like Storage to ensure
consistency and maintain OSDU standards.
- The pipeline implementation should adhere to the OSDU rules.
- Use "os-core-common" and "os-core-lib-aws" libraries in line with how
other OSDU core services use them to reduce boilerplate in the pipeline
scripts.Siarhei Symanovich (EPAM)Siarhei Symanovich (EPAM)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-timeseries-ingestion-service/-/issues/26[BUG] Single Endpoint Error Response Value Incorrect2024-03-27T07:57:51Zaliuddin abd rauf[BUG] Single Endpoint Error Response Value Incorrectthe error response format for the single timeseries ingestion is correct, but the content or value of the json body seems not correct for the reason and message property.
![image.png](/uploads/3b80ecfb5fcf32719d4724c205180193/image.png)...the error response format for the single timeseries ingestion is correct, but the content or value of the json body seems not correct for the reason and message property.
![image.png](/uploads/3b80ecfb5fcf32719d4724c205180193/image.png)For the reason, following standard error format, it should contain the reason for the code, it usually same as the reason for the httpstatuscode, eg: 400, reason is Bad Request. And for the message part, it should clarify what trigger the error, instead of passing stringfy json value.PDMS MVP1, phase2https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-production-source-service/-/issues/7[BUG] Remove redundant code param from update source endpoint2024-03-27T07:49:41ZAsan Arifov[BUG] Remove redundant code param from update source endpointRemove redundant code param from update source endpoint
![2.png](/uploads/f63f4943f19555eb913e4eb8b5a14e4b/2.png)Remove redundant code param from update source endpoint
![2.png](/uploads/f63f4943f19555eb913e4eb8b5a14e4b/2.png)PDMS MVP1, phase2https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-pps-timeseries-service/-/issues/10[REQ] "Source" Query String To Path2024-03-27T07:39:32Zaliuddin abd rauf[REQ] "Source" Query String To PathNeed to move source param from params to path variables:
{{baseUrl}}/:entityRecID/:propertyDescriptor?start=0&end=1872513910000&source=mysrc1 -> {{baseUrl}}/:entityRecID/:propertyDescriptor/:source?start=0&end=1872513910000Need to move source param from params to path variables:
{{baseUrl}}/:entityRecID/:propertyDescriptor?start=0&end=1872513910000&source=mysrc1 -> {{baseUrl}}/:entityRecID/:propertyDescriptor/:source?start=0&end=1872513910000PDMS MVP1, phase2https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-timeseries-dms/-/issues/9[REQ] API Versioning standardization2024-03-27T07:38:12ZAsan Arifov[REQ] API Versioning standardizationAll the internal and public services endpoints versions must be reset to version 1
* LatestStreamDetail [http://{{PDMS_TIMESERIES_QUERY_HOST}}/timeseries/api/v2/streams/:streamId?start=1684724571425&end=1684724571426&version=16847245714...All the internal and public services endpoints versions must be reset to version 1
* LatestStreamDetail [http://{{PDMS_TIMESERIES_QUERY_HOST}}/timeseries/api/v2/streams/:streamId?start=1684724571425&end=1684724571426&version=1684724571426](http://%7B%7BPDMS_TIMESERIES_QUERY_HOST%7D%7D/timeseries/api/v2/streams/:streamId?start=1684724571425&end=1684724571426&version=1684724571426)
Update the modified endpoints calls in osdu-production-timeseries-service
![osdu-pdms-architecture.png](/uploads/984844ffed1be4886cc9749d5a8c0bf8/osdu-pdms-architecture.png)PDMS MVP1, phase2Danh Nguyennguyencong.danh@petronas.comDanh Nguyennguyencong.danh@petronas.comhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-pdm-service/-/issues/9[REQ] API Versioning standardization2024-03-27T07:35:54ZAsan Arifov[REQ] API Versioning standardizationAll the internal and public services endpoints versions must be reset to version 1
* ALL the endpoints
Update the modified endpoints calls in
* osdu-hierarchy-storage-service
* osdu-production-properties-service
* osdu-production-time...All the internal and public services endpoints versions must be reset to version 1
* ALL the endpoints
Update the modified endpoints calls in
* osdu-hierarchy-storage-service
* osdu-production-properties-service
* osdu-production-timeseries-service
* osdu-timeseries-writevack-service
* osdu-timeseries-writeback-publisher
![osdu-pdms-architecture.png](/uploads/759bd7679b28ca970b200cb08f9ffc98/osdu-pdms-architecture.png)PDMS MVP1, phase2khoi huynh anhanhkhoi.huynh@petronas.comkhoi huynh anhanhkhoi.huynh@petronas.comhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-structure-query-service/-/issues/8[REQ] API Versioning standardization2024-03-27T07:33:05ZAsan Arifov[REQ] API Versioning standardizationAll the internal and public services endpoints versions must be reset to version 1
GetElementsByType [http://{{PDMS_STRUCTURE_QUERY_HOST}}/structurequery/structure/api/v2/elements?version=1784135848279&start=0&end=1784135848279&ofType=F...All the internal and public services endpoints versions must be reset to version 1
GetElementsByType [http://{{PDMS_STRUCTURE_QUERY_HOST}}/structurequery/structure/api/v2/elements?version=1784135848279&start=0&end=1784135848279&ofType=FIELD](http://%7B%7BPDMS_STRUCTURE_QUERY_HOST%7D%7D/structurequery/structure/api/v2/elements?version=1784135848279&start=0&end=1784135848279&ofType=FIELD)
Update the modified endpoints calls in:
* osdu-pps-timeseries-service-cache
* osdu-timeseries-writeback-service
See the diagram below
![osdu-pdms-architecture.png](/uploads/ac30f3c5b4111e3a63b25179cac9ed0b/osdu-pdms-architecture.png)PDMS MVP1, phase2Danh Nguyennguyencong.danh@petronas.comDanh Nguyennguyencong.danh@petronas.comhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-timeseries-dms/-/issues/12[REQ] Consistent health check endpoints names and response format2024-03-27T07:28:43ZAsan Arifov[REQ] Consistent health check endpoints names and response formatSee EPIC for detailsSee EPIC for detailsPDMS MVP1, phase2https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-pss-properties-service/-/issues/11[REQ] Consistent health check endpoints names and response format2024-03-27T07:26:19ZAsan Arifov[REQ] Consistent health check endpoints names and response formatSee EPIC for detailsSee EPIC for detailsPDMS MVP1, phase2https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-pdm-service/-/issues/12[REQ] Consistent health check endpoints names and response format2024-03-27T07:21:58ZAsan Arifov[REQ] Consistent health check endpoints names and response formatSee EPIC for detailsSee EPIC for detailsPDMS MVP1, phase2https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-structure-query-service/-/issues/11[REQ] Consistent health check endpoints names and response format2024-03-27T07:20:06ZAsan Arifov[REQ] Consistent health check endpoints names and response formatSee EPIC for detailsSee EPIC for detailsPDMS MVP1, phase2Duy Daodaokhanh.duy@petronas.comDuy Daodaokhanh.duy@petronas.comhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-timeseries-ingestion-service/-/issues/21[PERF] Slow Endpoint2024-03-27T07:10:50Zaliuddin abd rauf[PERF] Slow EndpointI've noticed, since the changes made to fix the batch endpoint, the endpoints for both single and batch taking more time to response the 202 Accepted message. The function seems still working, but the performance become slower, it took, ...I've noticed, since the changes made to fix the batch endpoint, the endpoints for both single and batch taking more time to response the 202 Accepted message. The function seems still working, but the performance become slower, it took, fastest I got to ingest 3 timeseries record is 3++ seconds. Before this, it didn't even exceed 1 second.
![image.png](/uploads/1b3b77542d5a23d63ef165a4dc9ac8b6/image.png)PDMS MVP1, phase2Duy Daodaokhanh.duy@petronas.comDuy Daodaokhanh.duy@petronas.comhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-pss-entities-service/-/issues/10[BUG] Hierarchy check always returns "No entities hierarchy in cache for rela...2024-03-27T07:08:11Zaliuddin abd rauf[BUG] Hierarchy check always returns "No entities hierarchy in cache for relationshipType ORG_RELATIONSHIPS"**endpoint**: {{baseUrl}}/entities?hierarchy=ORGANIZATION&types=WELL
The endpoint for hierarchy check not working ({{baseUrl}}/entities?hierarchy=ORGANIZATION&types=WELL). always return always reports "No entities hierarchy in cache for...**endpoint**: {{baseUrl}}/entities?hierarchy=ORGANIZATION&types=WELL
The endpoint for hierarchy check not working ({{baseUrl}}/entities?hierarchy=ORGANIZATION&types=WELL). always return always reports "No entities hierarchy in cache for relationshipType ORG_RELATIONSHIPS"PDMS MVP1, phase2Jozsef BarczaJozsef Barczahttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-production-source-service/-/issues/4[REQ] Consistent health check endpoints names and response format2024-03-27T06:33:18ZAsan Arifov[REQ] Consistent health check endpoints names and response formatSee EPIC for detailsSee EPIC for detailsPDMS MVP1, phase2https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-timeseries-ingestion-service/-/issues/17[REQ] Consistent health check endpoints names and response format2024-03-27T06:32:55ZAsan Arifov[REQ] Consistent health check endpoints names and response formatSee EPIC for detailsSee EPIC for detailsPDMS MVP1, phase2https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-production-stream-storage-service/-/issues/2[REQ] Consistent health check endpoints names and response format2024-03-27T06:32:53ZAsan Arifov[REQ] Consistent health check endpoints names and response formatSee EPIC for detailsSee EPIC for detailsPDMS MVP1, phase2https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-pps-timeseries-service-cache/-/issues/9[REQ] Consistent health check endpoints names and response format2024-03-27T06:32:38ZAsan Arifov[REQ] Consistent health check endpoints names and response formatSee EPIC for detailsSee EPIC for detailsPDMS MVP1, phase2