OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2023-03-24T21:39:41Zhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/53Provide Domain API to read/write Seismic 2D Navigation data (sourced from mul...2023-03-24T21:39:41ZDebasis ChatterjeeProvide Domain API to read/write Seismic 2D Navigation data (sourced from multiple formats)This will be very similar to approach in Wellbore DDMS.
Access to Well Log data via API, although source data can be LAS, DLIS, LIS, WITSML.
Similarly in this case, source data can be UKOOA< SegP1, IOGP format.
But uniform set of Domai...This will be very similar to approach in Wellbore DDMS.
Access to Well Log data via API, although source data can be LAS, DLIS, LIS, WITSML.
Similarly in this case, source data can be UKOOA< SegP1, IOGP format.
But uniform set of Domain API should allow programmatic access to this information to help applications.
Common use case being an application that wants to display 2D Navigation on Map, with option to show SP labels at certain zoom level.
Please also check this issue for Data Definition.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/348
cc - @Keith_Wall (for information)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/52Difference in documentation and functionality of utility cp endpoint2023-03-24T19:32:53ZWalter DDifference in documentation and functionality of utility cp endpointThe documentation for Utility CP endpoint mentions 'The source and destination dataset must be in the same sub-project.' However, the endpoint returns 202 Accepted response even when the source and destination sub-project are not same.The documentation for Utility CP endpoint mentions 'The source and destination dataset must be in the same sub-project.' However, the endpoint returns 202 Accepted response even when the source and destination sub-project are not same.M11 - Release 0.14Diego MolteniDiego Moltenihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/57Utilizing Standard Pipelines2023-03-24T19:24:00ZDavid Diederichd.diederich@opengroup.orgUtilizing Standard PipelinesI'd like this project to consider merging your CI pipeline work with the osdu/platform/ci-cd-pipelines> project, and utilize more jobs by includes than using local CI config.
### Some Reasons to Consider
**Copy/paste code is hard to ke...I'd like this project to consider merging your CI pipeline work with the osdu/platform/ci-cd-pipelines> project, and utilize more jobs by includes than using local CI config.
### Some Reasons to Consider
**Copy/paste code is hard to keep maintained**
Most of your CI logic appears to have started as a copy/paste from the main repository, anyway.
But keeping it local means that developers need to update changes in multiple places, and when they're working on the improvements they don't have your use case in mind.
This included some recent developments to get the dev2 environment going, but it also includes the changes to the FOSSA scanning -- you're still using an older, unmaintained image for the scanning.
And, when I did the changes, I worked test examples for maven and pip, the two supported build systems.
If npm had been there, I would have had it in mind.
**You miss new pipeline developments**
I'm moving pieces of the release management scripts into the pipeline to make more aspects of the tagging process happen automatically from branch creation.
For now, it's only dependency scanning data, but upgrades are planned to do more stages from there.
The GitLab Ultimate scanners check for security vulnerabilities, and the InfoSec team utilizes these results to plan their work.
These scanners aren't running on your project, but would be if included the appropriate CI configuration -- or at least, we'd see what needs to be improved on those scanners to function if they don't work out of the box.
**Your improvements aren't available to others**
Any improvements you make to the CI process after you've copied it remains in your local repository.
Others could benefit from having this available in a common location.
Supporting another language gives future OSDU projects more capabilities right at the start.
You'd even get to define the basic processes for these.
### Open to Discussion
I'd like to hear more about how the custom pipelines came to be, and if they are serving a need that can't be generalized.
For steps that are truly custom and unique to your project, it makes sense to have them as local CI config files.
If we do decide to start using more of the standard pipeline logic, I think we'll need to implement it slowly, a piece at a time.
Of course, if you think a big bang MR is better, I'd consider that, too.
Thank you in advance for your thoughts.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/58For Tenant there is no endpoint that can be used to list all the available te...2023-03-24T19:22:43ZKamlesh TodaiFor Tenant there is no endpoint that can be used to list all the available tenantsThere should be a way to list all the tenants to which the user has access. At present, there is no way to do that. If one had created the tenant in the past and cannot remember the name, then there is no way to find that name.There should be a way to list all the tenants to which the user has access. At present, there is no way to do that. If one had created the tenant in the past and cannot remember the name, then there is no way to find that name.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/62Release process does not follow general approach2023-03-24T19:22:10ZMikhail Piatliou (EPAM)Release process does not follow general approachThe pipelines for the `release` branches and `tags` https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/v0.15.0/devops/osdu/cloud-providers/gcp.yml use reference ...The pipelines for the `release` branches and `tags` https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/v0.15.0/devops/osdu/cloud-providers/gcp.yml use reference to the `master` branch in https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/tree/master/cloud-providers.
It is incorrect behavior since the service release/tags pipelines should have reference to the release/tags in ci-cd common pipelines accordingly.
For example, Partition service https://community.opengroup.org/osdu/platform/system/partition/-/blob/v0.15.1/.gitlab-ci.yml#L51.
The issue is valid for all CSPs.
Cc: @divido @Kateryna_Kurach @Oleksandr_KosseDaniel PerezDaniel Perezhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/65Make cloud interfaces and abstract classes less GCP specific2023-03-24T19:13:34ZYan Sushchynski (EPAM)Make cloud interfaces and abstract classes less GCP specificHello!
During implementing `Anthos` provider we faced troubles with creating the concrete `Journal` class: [here](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/...Hello!
During implementing `Anthos` provider we faced troubles with creating the concrete `Journal` class: [here](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/feat/Anthos_GCP/app/sdms/src/cloud/providers/anthos/postgresql.ts#L96).
If we understand correctly, `AbstractJournal` and `AbstractJournalTransaction` classes simply reproduce GCP Datastore interfaces. It is ok for GCP implementation, because there is no extra effort needed for implementing concrete journal classes. However, it is hard to implement concrete journal classes for other CSPs. This becomes obvious when we compare the number of lines for GCP and other CSPs particular journal classes (e.g., [GCP](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/feat/Anthos_GCP/app/sdms/src/cloud/providers/google/datastore.ts) and [Azure](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/feat/Anthos_GCP/app/sdms/src/cloud/providers/azure/cosmosdb.ts)).
Also, using Datastore "low-level" logic in the core code makes this code hard to read and debug. E.g., for Anthos we use PostgreSQL database and the concrete implementation required a lot of workarounds to fit Datastore "low-level" methods into SQL.
For example, there are specific Datastore operators in the abstract class ([here](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/feat/Anthos_GCP/app/sdms/src/cloud/journal.ts#L25)).
I'd suggest refactoring the common code and switch from using Datastore methods to focusing on more general and high-level logic.
For example, instead of using [this](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/master/app/sdms/src/services/dataset/dao.ts#L46) in the common code
```ts
let query = journalClient.createQuery(
Config.SEISMIC_STORE_NS + '-' + dataset.tenant + '-' + dataset.subproject, Config.DATASETS_KIND);
query = query.filter('name', dataset.name).filter('path', dataset.path);
const [entities] = await journalClient.runQuery(query);
```
We could use something like this:
```ts
// Just an example
const entity = await journalClient.getEntity(dataset.path, dataset.name);
```
In this case, we could be more concise and cleaner in concrete CSP implementations. Also, implementing high-level classes lets us use the best practices for each particular data base.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/66[GCP] Docker Image loading time is very long2023-03-24T19:13:05ZMaksimelyan Tamashevich (EPAM)[GCP] Docker Image loading time is very long![image](/uploads/014b51a301ac9672301c884b1f9414da/image.png)
As you can see, the image loading time is more than 15 minutes. This problem is temporary, it was noticed several times in the evening (UTC+2).
And as a result, our deploymen...![image](/uploads/014b51a301ac9672301c884b1f9414da/image.png)
As you can see, the image loading time is more than 15 minutes. This problem is temporary, it was noticed several times in the evening (UTC+2).
And as a result, our deployment job fails because the default helm timeout of 5 minutes has been exceeded.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/68Utility LS endpoint doesn't work for directories2023-03-24T19:11:22ZKonstantin KhottchenkovUtility LS endpoint doesn't work for directoriesNew test scenario was added for UTILITY LS endpoint. The feature of filtering the output for only datasets, only folders or both datasets and folders was added and tested.
The result of tests shows that use of "wmode" parameter with valu...New test scenario was added for UTILITY LS endpoint. The feature of filtering the output for only datasets, only folders or both datasets and folders was added and tested.
The result of tests shows that use of "wmode" parameter with values "dirs" and "all" that filter response to receive only names of directories or both datasets and directories correspondingly fails for AWS and ANTHOS. We couldn't check if Google also affected because Google environment is broken at all. Thus this tests were disabled for mentioned CSPs.
[Pipeline run](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/jobs/1458328)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/69Subproject creation Bad Request2023-03-24T16:02:13ZDenis Karpenok (EPAM)Subproject creation Bad RequestGCP preshipping environment.
Tenant was created:
`{
"name": "autotesttenantid436502",
"esd": "odesprod.osdu-gcp.go3-nrg.projects.epam.com",
"gcpid": "osdu-data-prod",
"default_acls": "users.datalake.admins@odesprod.osdu...GCP preshipping environment.
Tenant was created:
`{
"name": "autotesttenantid436502",
"esd": "odesprod.osdu-gcp.go3-nrg.projects.epam.com",
"gcpid": "osdu-data-prod",
"default_acls": "users.datalake.admins@odesprod.osdu-gcp.go3-nrg.projects.epam.com"
}`
Trying to create subproject.
Request:
`curl --location --request POST 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/seismic-store/v3/subproject/tenant/autotesttenantid436502/subproject/subprojectodi725168' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: odesprod' \
--header 'ltag: odesprod-SeismicDMS-Legal-Tag-Test7116874' \
--header 'Authorization: Bearer ID_TOCKEN' \
--data-raw '{
"admin": "admin@odesprod.osdu-gcp.go3-nrg.projects.epam.com",
"storage_class": "MULTI_REGIONAL",
"storage_location": "US",
"legal": {
"legaltags": [
"odesprod-SeismicDMS-Legal-Tag-Test7116874"
],
"otherRelevantDataCountries": [
"US"
]
}
}'`
Response:
`[seismic-store-service] Bad Request`
Seismic-store logs:
`2022-10-21 15:40:40.798 EET{"error":{"code":400,"message":"[seismic-store-service] Bad Request","status":"BAD_REQUEST"}}`Sacha BrantsSacha Brantshttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/70Subproject deletion bad request2023-03-24T16:00:19ZYan Sushchynski (EPAM)Subproject deletion bad requestWe found a bug connected with deleting subprojects. It seems that when we delete them the call to Entitlements service has a wrong URL.
We can see from the logs that Seismic sends request to the following URL:
https://entitlements/api/...We found a bug connected with deleting subprojects. It seems that when we delete them the call to Entitlements service has a wrong URL.
We can see from the logs that Seismic sends request to the following URL:
https://entitlements/api/entitlements/v2/groups/data/<goupname>.
The __data__ is extra here.
The bug is here: https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/master/app/sdms/src/cloud/providers/google/config.ts#L131M15 - Release 0.18Diego MolteniSacha BrantsDiego Moltenihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/86osdu:wks:work-product-component--NotionalSeismicLine:1.0.02023-03-24T15:58:23ZSacha Brantsosdu:wks:work-product-component--NotionalSeismicLine:1.0.0https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/92Refactor base images and npm dependencies version2023-03-24T15:54:04ZAliaksandr Ramanovich (EPAM)Refactor base images and npm dependencies versionIt seems it's time to update versions of base images used in Dockerfiles and dependencies in the package.json fileIt seems it's time to update versions of base images used in Dockerfiles and dependencies in the package.json fileSacha BrantsSacha Brantshttps://community.opengroup.org/osdu/platform/system/notification/-/issues/24Support pull messaging model2023-03-24T06:36:10Zashley kelhamSupport pull messaging model##Status
- [X] Proposed
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context and scope
Currently Notification service in OSDU supports a push messaging model where the server pro-actively sends messages to subscribing clients but ...##Status
- [X] Proposed
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context and scope
Currently Notification service in OSDU supports a push messaging model where the server pro-actively sends messages to subscribing clients but not a pull model where client's are in control of retrieving messages from the server.
This ADR proposes to enable clients to choose whether to pull messages or receive them via push from the notification service when registering for messages.
The registration service could be extended to support an extra API to allow clients to register for pull messages. The user then chooses whether to use this API or the existing register API for push messages
```
/pullsubscription:
post:
tags:
- Subscription
summary: Create a subscription
description: "Create a subscription. Required roles: 'users.datalake.editors' or
'users.datalake.admins'"
operationId: Create a subscription
parameters:
- $ref: "#/components/parameters/data-partition-id"
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/pullSubscription"
responses:
"201":
description: Created
content:
application/json:
schema:
$ref: "#/components/schemas/pullSubscriptionCreateResult"
components:
parameters:
pullSubscription:
type: object
required:
- schema
properties:
name:
type: string
pattern: ^[A-Za-z0-9- ]{2,50}
example: test-subscription
description:
type: string
pattern: ^[A-Za-z0-9. ]{0,255}
example: test description
topic:
type: string
example: data-changed-v1
pullSubscriptionCreatedResult:
type: object
required:
- schema
properties:
id:
type: string
example: dGVzdC1uYW1l
name:
type: string
pattern: ^[A-Za-z0-9- ]{2,50}
example: test-subscription
description:
type: string
pattern: ^[A-Za-z0-9. ]{0,255}
example: test description
topic:
type: string
example: data-changed-v1
createdBy:
type: string
example: test@myapp.com
createdOnEpoch:
$ref: "#/components/schemas/createdOnEpoch"
```
The response of the API then tells the client their unique subscription ID. The client uses this to pull messages. The notification service would need to be extended to have an exposed API which clients would call with their subscription ID.
```
/pullsubscription/{id}:
get:
tags:
- Subscription
summary: Get messages for subscription
description: "Gets an array of messages for the given subscription id if any. Required roles: 'users.datalake.editors' or
'users.datalake.admins'"
operationId: Create a subscription
parameters:
- name: id
in: path
required: true
schema:
type: string
pattern: ^[A-Za-z0-9-]{2,128}
- $ref: "#/components/parameters/data-partition-id"
responses:
"200":
description: OK
content:
application/json:
schema:
type: array
```
The GET and DELETE subscription registration API would work with an ID that was either from a pull or push subscription
## Trade-offs
The main benefit of pull over push is in scenarios where the client want's to control the request rate at which it receives messages from the server. This may be in scenarios where the server pushes to aggressively for the client to handle causing potential, errors or even crashes on the client side. Also if the client can retrieve messages at a higher rate than the server can pushes then in a pull model the client can potentially increase the rate at which it can retrieve and process messages.
Another potential benefit is with security. In the push model the client must expose a public endpoint for the server to push messages to. In OSDU we secure this with a given client credential to generate a jwt token or secret to generate a HMAC signature on the HTTPS requests pushed. This forces the client to expose a public endpoint and have a secret rotation strategy in place. This is negated in the pull model where the client users the common authorization scheme when calling any of the OSDU APIs to pull messages putting no extra burden on the client to maintain a secure system. We also ask clients to expose an extra endpoint to test registrations for notifications putting more burden on the client to setup maintain and secure.
## Decisionashley kelhamashley kelhamhttps://community.opengroup.org/osdu/platform/system/search-service/-/issues/100Search should gracefully handle non 200-OK responses from policy service2023-03-23T18:53:40ZShane HutchinsSearch should gracefully handle non 200-OK responses from policy serviceRequest:
After https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/290 I would like to see an enhancement to search.
If the policy service delivers anything other than a HTTP_200_OK response, that search...Request:
After https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/290 I would like to see an enhancement to search.
If the policy service delivers anything other than a HTTP_200_OK response, that search handles this and fail gracefully. Currently it will deliver a failed message to elastic search and this is far from optimal.
Related background:
Policy Service currently handles a special use-case with translate API, if the policy-id is not found it returns {"query": {"match_none": {}}} instead of a HTTP_404_NOT_FOUND. But search should handle all the other use-cases, for example when Policy service is not available (or OPA is not available) - i.e. a HTTP_500_INTERNAL_SERVER_ERROR. This feature was added as a result of https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/72https://community.opengroup.org/osdu/platform/system/storage/-/issues/168Storage should allow empty data block upon record creation/update2023-03-22T04:13:47ZAn NgoStorage should allow empty data block upon record creation/updateStorage PUT api should allow empty data block upon record creation/update if that is compliant with the schema being defined.
Currently, data block is required.
data: {}
This is a breaking change since it changes the behavior of the ...Storage PUT api should allow empty data block upon record creation/update if that is compliant with the schema being defined.
Currently, data block is required.
data: {}
This is a breaking change since it changes the behavior of the API.
Indexer service needs to be checked to ensure empty data block is being handled correctly.https://community.opengroup.org/osdu/platform/system/storage/-/issues/159Storage adds null meta to record ingested without2023-03-22T04:11:53ZAn NgoStorage adds null meta to record ingested without1. Record was ingested without specifying "meta" block. PUT api was successful.
2. Fetch the ingested record. Notice that Storage added "meta": null to the record.
**Checking with Search.**
Search indexed successfully. Status code was 2...1. Record was ingested without specifying "meta" block. PUT api was successful.
2. Fetch the ingested record. Notice that Storage added "meta": null to the record.
**Checking with Search.**
Search indexed successfully. Status code was 200.
Search result does not return the meta.
The current behavior is challenged saying that Meta block shouldn't have been added. Or if added, then it should be empty and not null.
So instead of adding:
"meta": null
It should be:
"meta": []
Upon creating or updating a record, providing an empty meta block should also be allowed.https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/18EDS - Adding Logger to give details about Osdu_ingest run id and Sample fetch...2023-03-20T13:36:39ZPriyanka BhongadeEDS - Adding Logger to give details about Osdu_ingest run id and Sample fetched data record_ Add Logger to display Osdu_ingest run id in below format
Osdu_ingest runId=xxxx
- Correction in logger while dsplaying sample fetched data record
currently logger has " Record 1 :"
To make the message more clearer , changing the disp..._ Add Logger to display Osdu_ingest run id in below format
Osdu_ingest runId=xxxx
- Correction in logger while dsplaying sample fetched data record
currently logger has " Record 1 :"
To make the message more clearer , changing the display message in logs as "Displaying only one Sample Record"M17 - Release 0.20Nisha ThakranPriyanka BhongadeNisha Thakranhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/issues/26memory leak with data chunking2023-03-16T20:34:29ZYunhua Koglinmemory leak with data chunkingHello, When we have dask for data chunking features, memory is not released
![image__1_](/uploads/7e3da10faa83fe89d243992cddfad3d7/image__1_.png)Hello, When we have dask for data chunking features, memory is not released
![image__1_](/uploads/7e3da10faa83fe89d243992cddfad3d7/image__1_.png)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/151Misleading message in Xcom summary when legal tag is missing2023-03-16T11:31:11ZDebasis ChatterjeeMisleading message in Xcom summary when legal tag is missingI was running a simple test case in AWS/M16/Preship.
I was getting this message.
Now, I get failure. \[{'id': 'osdu:reference-data--FacilityEventType:DC13MAR', 'kind': 'osdu:wks:reference-data--FacilityEventType:1.0.0', 'reason': '400 C...I was running a simple test case in AWS/M16/Preship.
I was getting this message.
Now, I get failure. \[{'id': 'osdu:reference-data--FacilityEventType:DC13MAR', 'kind': 'osdu:wks:reference-data--FacilityEventType:1.0.0', 'reason': '400 Client Error: Bad Request for url: [http://os-storage.osdu-services:8080/api/storage/v2/records'}](http://os-storage.osdu-services:8080/api/storage/v2/records'%7D)\]
Turns out (thanks to AWS Support Nazeem Akbar Ali) that this is because legal tag was not defined and he found the reason by chckingr elevant log file.
See details here -
https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/470#note_207244https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/resqml-parser/-/issues/1(SpatialPoint/SpatialArea) What to do with big dataset ?2023-03-15T10:01:23ZValentin Gauthier(SpatialPoint/SpatialArea) What to do with big dataset ?For now the entire points list is required to be load in single list to compute the bounding box and the central point.
It could fail if the data is too heavy.
We should be able to handle thatFor now the entire points list is required to be load in single list to compute the bounding box and the central point.
It could fail if the data is too heavy.
We should be able to handle thatValentin GauthierValentin Gauthier