OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2022-08-23T11:18:45Zhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/105Offboard Delivery Service2022-08-23T11:18:45ZJasonOffboard Delivery ServiceDelivery service no longer exists independently as its functionality will be merged into file service. We need to remove references to delivery service from our documentation since it no longer exists.Delivery service no longer exists independently as its functionality will be merged into file service. We need to remove references to delivery service from our documentation since it no longer exists.M6 - Release 0.9 - removeMANISH KUMARMANISH KUMARhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/107External Data Service Onboarding2022-08-23T10:47:30ZGarrett EdmondsonExternal Data Service Onboarding**Service name**: `External Data Service`
[External Data Service Repo](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/tree/master)
[External Data Service CI/CD bran...**Service name**: `External Data Service`
[External Data Service Repo](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/tree/master)
[External Data Service CI/CD branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/tree/CI-CD)
**Infrastructure and Initial Requirements**
- [ ] Add any additional Azure cloud infrastructure (Cosmos containers, Storage containers, fileshares, etc.) to the Terraform template. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp). Note that if the infrastructure is a part of the data-partition template, you may need to add secrets to the keyvault that are partition specific; if doing so, update the createPartition REST request to include the keys that you have added so they are accessible in service code. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/rest/partition.http#L48)
- [ ] Create an ingress point for the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/appgw-ingress.yaml)
- [ ] Add any test data that is required for the service integration tests. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/test_data)
- [ ] Update `upload-data.py` to upload any new test data files you created. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/upload-data.py).
- [ ] Update the integration tester with any entitlements required to test the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/user_info_1.json)
- [ ] Add in any new secrets that the service needs to run. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/kv-secrets.yaml)
- [ ] Create environment variable script to generate .yaml files to be used with Intellij [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) plugin and .envrc files to be used with [direnv](https://direnv.net/). [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/variables)
**Gitlab Code and Documentation**
- [ ] Complete the service code such that it passes all integration tests locally. There is some documentation on starting off implementing an Azure provider. [Link](./gitlab-service-readme-template.md)
- [ ] Create helm charts for service. The charts for each service are located in the `devops/azure` directory. You can look at charts from other services as a model. The charts will be nearly identical except for the different environment variables, values, etc each service needs to run. [Link](./gitlab-service-guide.md)
- [ ] Implement Istio for the service if this has not already been done. Here is an example MR that shows what steps are required. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/64)
- [ ] Create an Istio auth policy in the `devops/azure/chart/templates` directory. Here is an example of an Istio auth policy that is generic and can be used by other services. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/devops/azure/chart/templates/azure-istio-auth-policy.yaml)
- [ ] Add any variables that are required for the service integration tests to the Azure CI-CD file. [Link](https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/blob/master/cloud-providers/azure.yml)
- [ ] Verify that the README for the Azure provider correctly and clearly describes how to run and test the service. There is a README template to help. [Link](./gitlab-service-readme-template.md)
- [ ] Push any changes and verify that the Gitlab pipeline is passing in master.
**Development and Demo Azure Devops Pipelines**
- [ ] Create development ADO pipeline at `devops/azure/development-pipeline.yml` in the service repo.
- [ ] Verify development pipeline passes in ADO.
- [ ] Create Demo ADO pipeline at `devops/azure/pipeline.yml` in the service repo.
- [ ] Verify demo pipeline is passing in ADO.
**User Documentation**
- [ ] Add the service to the mirror pipeline instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/code-mirroring.md)
- [ ] Add the service to the manual deployment instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts)
- [ ] Add any required variables to the already existing variable group instructions for automated deployment. You should know if any variables need to be added to existing variable groups from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [ ] Add a variable group `Azure Service Release - $SERVICE_NAME` to the documentation. You should know what values to set for this variable group from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [ ] Add a step for creating the service pipeline at the bottom of the service-automation page. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [ ] Create a rest script with sample calls to the service for users. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/rest)Krishna Nikhil VedurumudiVivek OjhaKrishna Nikhil Vedurumudihttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/31Bean with hard-coded properties imposes policy that limits application/servic...2022-11-24T12:34:33ZSherman YangBean with hard-coded properties imposes policy that limits application/service flexibilityIn OS Core Common repo's src/main/java/org/opengroup/osdu/core/common/http/HttpConfiguration.java, a bean is automatically created with hard-coded properties that impose policy which limits service flexibility.
For example, the new ent...In OS Core Common repo's src/main/java/org/opengroup/osdu/core/common/http/HttpConfiguration.java, a bean is automatically created with hard-coded properties that impose policy which limits service flexibility.
For example, the new entitlement v2 service needs to accept case insensitive Json input but with the hard-coded ObjectMapper bean that got introduced since osdu core common version 0.0.17, the entitlement service's case insensitive json configuration is no longer available as the common library creates its own ObjectMapper which disables/overrides the service's settings and imposes case sensitivity.
Need to relax this limitation to allow application flexibility.https://community.opengroup.org/osdu/platform/system/file/-/issues/22File service record id generation2022-09-30T11:54:43Zethiraj krishnamanaiduFile service record id generationWe have logic in File service to validate and generate the RecordId, this duplicate logic that is already in storage service. We should let storage service generated RecordId and validation.We have logic in File service to validate and generate the RecordId, this duplicate logic that is already in storage service. We should let storage service generated RecordId and validation.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/issues/1e2e tests: setup step must create the subproject2023-03-30T16:57:09ZRucha Deshpandee2e tests: setup step must create the subprojectThe e2e tests assume that a subproject exists. Just as some files are uploaded in the 'setup' step, the subproject must also be created as part of the setup step here.
https://community.opengroup.org/osdu/platform/domain-data-mgmt-servic...The e2e tests assume that a subproject exists. Just as some files are uploaded in the 'setup' step, the subproject must also be created as part of the setup step here.
https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/blob/master/test/e2e/conftest.pyRucha DeshpandeDiego MolteniYunhua KoglinRucha Deshpandehttps://community.opengroup.org/osdu/platform/system/storage/-/issues/46Design consideration: Support transactions2022-09-29T13:41:40ZAlan HensonDesign consideration: Support transactionsOne of the requirements made of ingestion is to support transactional capabilities while storing records. This would allow any process interacting with the Storage API to perform the following:
- Begin a transaction
- Modify one or more ...One of the requirements made of ingestion is to support transactional capabilities while storing records. This would allow any process interacting with the Storage API to perform the following:
- Begin a transaction
- Modify one or more records (Create, Update, Delete, Purge) without any additional activity occurring (i.e., sending notifications)
- Commit the transaction (persist all modifications in entirety with anyone error resulting in a rollback) - on success, release all notifications generated through the record modifications
- Rollback the transaction (undo all modifications)
I'm sure there are other considerations involved with adding transactional support. These are some early thoughts to initiate the discussion.
The expectation is that this is feature is not considered until after Release 3.Chris ZhangChris Zhanghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/86Human-readable Reference Values in Manifests2021-03-11T13:11:55ZKeith WallHuman-readable Reference Values in ManifestsR3 requires that reference values substitute %20 for any spaces in the value, when used in a manifest. For example: ": "osdu:reference-data--CoordinateReferenceSystem:WGS%2084:". Previous versions of OSDU did not require the substitut...R3 requires that reference values substitute %20 for any spaces in the value, when used in a manifest. For example: ": "osdu:reference-data--CoordinateReferenceSystem:WGS%2084:". Previous versions of OSDU did not require the substitution. It adds work and makes QC more difficult.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/87Parsing error at AirflowWorkflowEngineServiceImpl class2021-02-11T14:28:10ZBhushan RadeParsing error at AirflowWorkflowEngineServiceImpl classIssue - getting parsing [error](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/blob/master/workflow-core/src/main/java/org/opengroup/osdu/workflow/service/AirflowWorkflowEngineServiceImpl.java#L86)...Issue - getting parsing [error](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/blob/master/workflow-core/src/main/java/org/opengroup/osdu/workflow/service/AirflowWorkflowEngineServiceImpl.java#L86) while converting Airflow response JSON String into [TriggerWorkflowResponse.java](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/blob/master/workflow-core/src/main/java/org/opengroup/osdu/workflow/model/ClientResponse.java) class
Reason -
[at the tim](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/blob/master/workflow-core/src/main/java/org/opengroup/osdu/workflow/service/AirflowWorkflowEngineServiceImpl.java#L86)e of parsing it expects JSON string as per TriggerWorkflowResponse.java format. but [This line ](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/blob/master/workflow-core/src/main/java/org/opengroup/osdu/workflow/service/AirflowWorkflowEngineServiceImpl.java#L141) collecting airflow response in the wrong format.
propose change -
replace the 141st [line](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/blob/master/workflow-core/src/main/java/org/opengroup/osdu/workflow/service/AirflowWorkflowEngineServiceImpl.java#L141) from AirflowWorkflowEngineServiceImpl.java with the following -
```
.responseBody(response.getEntity(String.class));
```ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/88The workflow service should provide outcomes and error messages produced by w...2022-08-08T11:24:21ZAlan HensonThe workflow service should provide outcomes and error messages produced by workflow instancesThe Workflow Service is capable of launching registered workflows. Those workflows presently run inside Airflow containers, which are likely implemented using containers. As these workflows run, they log information to the Airflow log fi...The Workflow Service is capable of launching registered workflows. Those workflows presently run inside Airflow containers, which are likely implemented using containers. As these workflows run, they log information to the Airflow log files. They might also implement a logging strategy that sends logs to the underlying CSP's logging framework. However, there is no programmatic way to fetch the outcomes of a workflow.
We need a uniform way to allow external processes to inquire information about a workflow beyond its execution status. Things that could be fetched via a run id, or some other mechanism, might include:
- Log messages
- Error messages (including partial failures)
- Workflow status (beyond the execution status - workflows might have their own managed state)
- Workflow results (i.e., records processed or stored [including record IDs], activities performed, etc.)
- Other details
This solution might require a broader perspective at the core service level, or it might be a solution unique to workflow service. Either way, we need to move beyond Airflow logs for a better user experience.
Additionally, as part of this story, the Workflow Service should consider statuses that indicate partial failures - meaning part of the job was successful, but part of the job also failed (even if gracefully failed).Raj KannanJane McConnellAsh SathyaseelanAlan HensonRaj Kannanhttps://community.opengroup.org/osdu/platform/system/register/-/issues/27GET /subscription by notificationID should be restricted2021-08-27T21:34:31ZAliaksei DarafeyeuGET /subscription by notificationID should be restrictedRight now, to perform GET `/subscription?notificationId=X` operation user should has one of role (`users.datalake.ops`, `users.datalake.admins`, `users.datalake.editors`) but according security check access to this operation should be re...Right now, to perform GET `/subscription?notificationId=X` operation user should has one of role (`users.datalake.ops`, `users.datalake.admins`, `users.datalake.editors`) but according security check access to this operation should be restricted to only `users.datalake.ops`.ethiraj krishnamanaiduChris ZhangGary MurphyNitin-slbNeelesh ThakurRobert Chadwick [Schlumberger]ethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/117Feature Change - Enable Pod to Pod transport security2021-06-14T04:26:42ZDaniel SchollFeature Change - Enable Pod to Pod transport securityCurrent implementation terminates and ssl offloads transport security at the Load Balancer. Transport Security should exist all the way to the Kubernetes Pod and between Pods.
OSDU Security - Universal Encryption - B2
Acceptance Crite...Current implementation terminates and ssl offloads transport security at the Load Balancer. Transport Security should exist all the way to the Kubernetes Pod and between Pods.
OSDU Security - Universal Encryption - B2
Acceptance Criteria
---
1. A design decision should be made on the best way to handle this feature.
2. Infrastructure/Helm automation should automatically configure and enable this feature.
3. Service Helm charts should be changed to move to httpshttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/118Feature Change - Enable AKS host-based encryption2022-08-23T11:17:12ZDaniel SchollFeature Change - Enable AKS host-based encryptionCurrently the AKS nodes aren't configured to use host-based encryption which needs to be enabled to support encryption at rest security requirements.
OSDU Security - Universal Encryption - A2
> Host-based encryption on Azure Kubernetes...Currently the AKS nodes aren't configured to use host-based encryption which needs to be enabled to support encryption at rest security requirements.
OSDU Security - Universal Encryption - A2
> Host-based encryption on Azure Kubernetes Service is in Preview. [Link](https://docs.microsoft.com/en-us/azure/aks/enable-host-encryption#:~:text=With%20host%2Dbased%20encryption%2C%20the,encrypted%20to%20the%20Storage%20service.&text=The%20cache%20of%20OS%20and,type%20set%20on%20those%20disks.)
Acceptance Criteria
---
1. Infrastructure Automation should automatically configure this feature.
2. Unit and Integration Tests Should passM7 - Release 0.10.0 - removeVivek OjhaVivek Ojhahttps://community.opengroup.org/osdu/platform/system/storage/-/issues/49Add Storage endpoint to create ids on request2022-09-29T13:41:40ZAlan HensonAdd Storage endpoint to create ids on requestAs part of storing records in OSDU, the Storage service creates an `id` value when the Record stored is absent an `id` value. The only way to get the `id` is to store the record. There are use cases, such as processing `surrogate-key` va...As part of storing records in OSDU, the Storage service creates an `id` value when the Record stored is absent an `id` value. The only way to get the `id` is to store the record. There are use cases, such as processing `surrogate-key` values prior to storing records, that warrant having an `id` created prior to storing the record so processing can occur external to storage while still using storage-generated (aka system-generated) `ids`.
This would function similarly to how database sequences might work. The API should support a request for one or more `id`s and it should support creating these `id`s for more than one type of `kind`.https://community.opengroup.org/osdu/platform/data-flow/real-time/consumers/python-cli-kafka-consumer/-/issues/1Nobis eum quia dolorum.2021-02-15T16:01:20ZDouglas DohmeyerNobis eum quia dolorum.## Ut
Odit cum earum. Fuga esse minus. Natus laboriosam iste. In enim voluptas. Dolores nobis similique.
#### Et## Ut
Odit cum earum. Fuga esse minus. Natus laboriosam iste. In enim voluptas. Dolores nobis similique.
#### EtSample GitLab Project Milestone 12021-10-15https://community.opengroup.org/osdu/platform/data-flow/real-time/processors/pipe/-/issues/1Nobis eum quia dolorum.2021-02-15T16:01:20ZDmitry KniazevNobis eum quia dolorum.## Ut
Odit cum earum. Fuga esse minus. Natus laboriosam iste. In enim voluptas. Dolores nobis similique.
#### Et## Ut
Odit cum earum. Fuga esse minus. Natus laboriosam iste. In enim voluptas. Dolores nobis similique.
#### EtSample GitLab Project Milestone 12021-10-14https://community.opengroup.org/osdu/platform/data-flow/real-time/consumers/python-cli-kafka-consumer/-/issues/2Excepturi qui tempore officiis.2021-02-15T16:01:25ZDouglas DohmeyerExcepturi qui tempore officiis.###### Voluptas
Molestiae nostrum aut. Nisi voluptatem natus. Voluptas accusamus voluptatum. Fuga dolor dignissimos. Natus consequatur adipisci.
## Sit
Repudiandae delectus quia. Vitae non ad. Quod neque perferendis.
###### Dolor
Earum r...###### Voluptas
Molestiae nostrum aut. Nisi voluptatem natus. Voluptas accusamus voluptatum. Fuga dolor dignissimos. Natus consequatur adipisci.
## Sit
Repudiandae delectus quia. Vitae non ad. Quod neque perferendis.
###### Dolor
Earum reprehenderit eum. Quia qui voluptatem. Cum voluptatibus eligendi.
#### Laboriosam
At facilis a. Recusandae hic quasi. Voluptas alias eum.
Maiores voluptatem rerum. Modi deleniti veniam. *Nemo* rerum minus.Sample GitLab Project Milestone 22021-10-22https://community.opengroup.org/osdu/platform/data-flow/real-time/processors/pipe/-/issues/2Excepturi qui tempore officiis.2021-02-15T16:01:25ZDmitry KniazevExcepturi qui tempore officiis.###### Voluptas
Molestiae nostrum aut. Nisi voluptatem natus. Voluptas accusamus voluptatum. Fuga dolor dignissimos. Natus consequatur adipisci.
## Sit
Repudiandae delectus quia. Vitae non ad. Quod neque perferendis.
###### Dolor
Earum r...###### Voluptas
Molestiae nostrum aut. Nisi voluptatem natus. Voluptas accusamus voluptatum. Fuga dolor dignissimos. Natus consequatur adipisci.
## Sit
Repudiandae delectus quia. Vitae non ad. Quod neque perferendis.
###### Dolor
Earum reprehenderit eum. Quia qui voluptatem. Cum voluptatibus eligendi.
#### Laboriosam
At facilis a. Recusandae hic quasi. Voluptas alias eum.
Maiores voluptatem rerum. Modi deleniti veniam. *Nemo* rerum minus.Sample GitLab Project Milestone 22021-10-21https://community.opengroup.org/osdu/platform/data-flow/real-time/consumers/python-cli-kafka-consumer/-/issues/3Deserunt expedita dolorum distinctio.2021-02-15T16:01:31ZDouglas DohmeyerDeserunt expedita dolorum distinctio.## Et
Nihil quasi neque. Possimus architecto cumque. Rerum fugit velit. Optio cum iure. Fuga aspernatur delectus.
## Consectetur
Eos et facere. Explicabo officia mollitia. Quo est cum.
`Molestiae.`## Et
Nihil quasi neque. Possimus architecto cumque. Rerum fugit velit. Optio cum iure. Fuga aspernatur delectus.
## Consectetur
Eos et facere. Explicabo officia mollitia. Quo est cum.
`Molestiae.`Sample GitLab Project Milestone 32021-10-29https://community.opengroup.org/osdu/platform/data-flow/real-time/processors/pipe/-/issues/3Deserunt expedita dolorum distinctio.2021-02-15T16:01:31ZDmitry KniazevDeserunt expedita dolorum distinctio.## Et
Nihil quasi neque. Possimus architecto cumque. Rerum fugit velit. Optio cum iure. Fuga aspernatur delectus.
## Consectetur
Eos et facere. Explicabo officia mollitia. Quo est cum.
`Molestiae.`## Et
Nihil quasi neque. Possimus architecto cumque. Rerum fugit velit. Optio cum iure. Fuga aspernatur delectus.
## Consectetur
Eos et facere. Explicabo officia mollitia. Quo est cum.
`Molestiae.`Sample GitLab Project Milestone 32021-10-28https://community.opengroup.org/osdu/platform/data-flow/real-time/consumers/python-cli-kafka-consumer/-/issues/4Reprehenderit est sunt est.2021-02-15T15:56:01ZDouglas DohmeyerReprehenderit est sunt est.#### Est
Sed voluptas corrupti. Aliquid ipsam eaque. Voluptatibus pariatur voluptatem. Et odit id. Architecto aut maiores.
* Consequuntur.
* Et.
* Reprehenderit.
* Ipsa.
* Et.
* Eveniet.
* Voluptatem.
* Tempora.
* Expedita.#### Est
Sed voluptas corrupti. Aliquid ipsam eaque. Voluptatibus pariatur voluptatem. Et odit id. Architecto aut maiores.
* Consequuntur.
* Et.
* Reprehenderit.
* Ipsa.
* Et.
* Eveniet.
* Voluptatem.
* Tempora.
* Expedita.