Home issueshttps://community.opengroup.org/osdu/platform/home/-/issues2023-03-30T08:11:31Zhttps://community.opengroup.org/osdu/platform/home/-/issues/51Update Info API to support Core Version and implementation Version2023-03-30T08:11:31ZKrishna Nikhil VedurumudiUpdate Info API to support Core Version and implementation Versionhttps://gitlab.opengroup.org/osdu/subcommittees/ea/work-products/adr-elaboration/-/issues/78https://gitlab.opengroup.org/osdu/subcommittees/ea/work-products/adr-elaboration/-/issues/78Krishna Nikhil VedurumudiKrishna Nikhil Vedurumudihttps://community.opengroup.org/osdu/platform/home/-/issues/50Release management for Core Libraries.2023-03-30T08:14:20ZKrishna Nikhil VedurumudiRelease management for Core Libraries.Change in release process for Core Libraries to have reduced impact on Code tagging process for Milestone releases
Ref: https://gitlab.opengroup.org/osdu/subcommittees/ea/work-products/adr-elaboration/-/issues/79Change in release process for Core Libraries to have reduced impact on Code tagging process for Milestone releases
Ref: https://gitlab.opengroup.org/osdu/subcommittees/ea/work-products/adr-elaboration/-/issues/79Krishna Nikhil VedurumudiKrishna Nikhil Vedurumudihttps://community.opengroup.org/osdu/platform/home/-/issues/48Removing Airflow 1.x support in favor Airflow 2.x for M12+2023-12-05T10:33:23ZChad LeongRemoving Airflow 1.x support in favor Airflow 2.x for M12+## Existing Practice / Background
Airflow 2.x was introduced in M10 release for various improvements over Airflow 1.x. In M10 Airflow 2.x was released with feature flags so that customers can choose to stay at Airflow 1.x as well.
## M...## Existing Practice / Background
Airflow 2.x was introduced in M10 release for various improvements over Airflow 1.x. In M10 Airflow 2.x was released with feature flags so that customers can choose to stay at Airflow 1.x as well.
## Motivation
The OSDU community has decided to deprecate the support for Airflow 1.x and will completely remove the ability to stay at Airflow 1.x starting in M12 milestone.
CSP Decision Approvals
- [x] AWS
- [x] Azure
- [x] IBM
- [x] GCPhttps://community.opengroup.org/osdu/platform/home/-/issues/47Swagger UI throws "Whitelabel Error Page" for several services2023-08-02T10:04:35ZAn NgoSwagger UI throws "Whitelabel Error Page" for several servicesFor several services, the swagger page is showing "Whitelabel Error Page".
![image](/uploads/dc287779032213e02f5056e4390a23bb/image.png)
This error is observed after the swagger upgrade.
This is due to the reason that there are some m...For several services, the swagger page is showing "Whitelabel Error Page".
![image](/uploads/dc287779032213e02f5056e4390a23bb/image.png)
This error is observed after the swagger upgrade.
This is due to the reason that there are some missing/ inconsistent updates related to swagger for every service.
For example - In case of Entitlements Service, the swagger commit involved changes in the **[azure-istio-auth-policy.yaml](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/merge_requests/175/diffs?commit_id=56fbdf335f8ab29fd794314118b2fb3e5654ad39#acaaad215c25b431962f64748aa1c65fe3699abd)** but [storage service](https://community.opengroup.org/osdu/platform/system/storage/-/commit/191597a47d10ba3f69e42148e3c5c1e7c7ca04f9#022b6d114052cad09a79e365f1af9879bf04fe63) is missing this update.
Same is the case for the other services.
- It is mostly because in some services, this file does not exist within the service repository but in the [**infra-azure-provisioning**](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts/osdu-istio-auth/templates).
- For some services like `Notification` - this same [**file**](https://community.opengroup.org/osdu/platform/system/notification/-/tree/master/devops/azure/chart/templates) exists with a different name. So its missed.
- Same for `Register` service - the [**file**](https://community.opengroup.org/osdu/platform/system/register/-/blob/master/devops/azure/chart/templates/osdu-istio-policy.yaml) exists with a different name
This inconsistent updates are likely causing the swagger page to be broken.
**Impacted services:**
- CRS Catalog Service
- CRS Conversion Service
- Legal Service
- Partition Service
- Register Service
- Seismic Data Management Service
- Search Service
- Spatial Ref Service (no swagger endpoint)
- Storage Service
- Unit Service
- Workflow Service
**These ones are working:**
- Entitlements
- Indexer
- Search Extension
- Schema
- Wellbore DMS
- File
- WorkflowSrinivasan NarayananSrinivasan Narayananhttps://community.opengroup.org/osdu/platform/home/-/issues/45Critical security vulnerability about Swagger2022-03-01T13:25:46ZChris ZhangCritical security vulnerability about SwaggerThis is to track the security vulnerability about Swagger. https://nvd.nist.gov/vuln/detail/CVE-2019-17495
This applies to all OSDU services, and it is reported in OSDU security dashboard as well. This is one example: https://community...This is to track the security vulnerability about Swagger. https://nvd.nist.gov/vuln/detail/CVE-2019-17495
This applies to all OSDU services, and it is reported in OSDU security dashboard as well. This is one example: https://community.opengroup.org/osdu/platform/system/dataset/-/security/vulnerabilities/16194M10 - Release 0.13https://community.opengroup.org/osdu/platform/home/-/issues/43Security Vulnerability issues for M9 (Assigned to SLB team)2021-09-10T22:13:24ZChris ZhangSecurity Vulnerability issues for M9 (Assigned to SLB team)Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track SLB team's tasks on:
1. Open ZGY - Sacha Brants(SLB)
1. S...Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track SLB team's tasks on:
1. Open ZGY - Sacha Brants(SLB)
1. Seismic-dms-service - Sacha Brants(SLB)M9 - Release 0.12https://community.opengroup.org/osdu/platform/home/-/issues/42Security Vulnerability issues for M9 (Assigned to MSFT team)2023-10-05T11:16:18ZChris ZhangSecurity Vulnerability issues for M9 (Assigned to MSFT team)Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track MSFT team's tasks on:
1. Schema - @Madhur Tanwani(MSFT)
1...Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track MSFT team's tasks on:
1. Schema - @Madhur Tanwani(MSFT)
1. Infra-azure-provisioning - @Madhur Tanwani(MSFT)
**Additional Information**
The link below contains information on the vulnerability remediation process. For additional information on how to close out a vulnerability after it has been resolved please review step 7.
https://community.opengroup.org/groups/osdu/platform/-/wikis/Vulnerability-RemediationM9 - Release 0.12Madhur Tanwani [Microsoft]Krishnan GanesanMadhur Tanwani [Microsoft]https://community.opengroup.org/osdu/platform/home/-/issues/41Security Vulnerability issues for M9 (Assigned to GCP team)2021-10-04T18:39:16ZChris ZhangSecurity Vulnerability issues for M9 (Assigned to GCP team)Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track GCP team's tasks on:
1. IndexerQueue - @Kateryna Kurach(G...Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track GCP team's tasks on:
1. IndexerQueue - @Kateryna Kurach(GCP)
1. File - @Kateryna Kurach (GCP)
**Additional Information**
The link below contains information on the vulnerability remediation process. For additional information on how to close out a vulnerability after it has been resolved please review step 7.
https://community.opengroup.org/groups/osdu/platform/-/wikis/Vulnerability-RemediationM9 - Release 0.12Kateryna Kurach (EPAM)Kateryna Kurach (EPAM)https://community.opengroup.org/osdu/platform/home/-/issues/40Security Vulnerability issues for M9 (Assigned to IBM team)2023-10-05T11:16:27ZChris ZhangSecurity Vulnerability issues for M9 (Assigned to IBM team)Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track IBM team's tasks on:
1. CRS Conversion - @JINGDONG SUN(IB...Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track IBM team's tasks on:
1. CRS Conversion - @JINGDONG SUN(IBM)
1. Unit - @JINGDONG SUN(IBM)
**Additional Information**
The link below contains information on the vulnerability remediation process. For additional information on how to close out a vulnerability after it has been resolved please review step 7.
https://community.opengroup.org/groups/osdu/platform/-/wikis/Vulnerability-RemediationM9 - Release 0.12jingdong sunjingdong sunhttps://community.opengroup.org/osdu/platform/home/-/issues/39Security Vulnerability issues for M9 (Assigned to AWS team)2021-10-20T13:58:00ZChris ZhangSecurity Vulnerability issues for M9 (Assigned to AWS team)Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track AWS team's tasks on:
1. CRS Catalog - @Greg Wibben(AWS)
1...Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track AWS team's tasks on:
1. CRS Catalog - @Greg Wibben(AWS)
1. Entitlements - @Greg Wibben(AWS)
**Additional Information**
The link below contains information on the vulnerability remediation process. For additional information on how to close out a vulnerability after it has been resolved please review step 7.
https://community.opengroup.org/groups/osdu/platform/-/wikis/Vulnerability-RemediationM9 - Release 0.12GregGreghttps://community.opengroup.org/osdu/platform/home/-/issues/38IBM services to use partition service for tenant validation rather than isola...2021-08-25T05:15:02ZShrikant GargIBM services to use partition service for tenant validation rather than isolated tenatinfo datastoreIBM services to use partition service for tenant validation rather than isolated tenatinfo datastore
IBM will be using partition service to validate tenant information passed from other servicesIBM services to use partition service for tenant validation rather than isolated tenatinfo datastore
IBM will be using partition service to validate tenant information passed from other servicesM8 - Release 0.11Shrikant GargShrikant Garghttps://community.opengroup.org/osdu/platform/home/-/issues/37Version endpoint - Part 1 (Storage Service)2021-08-13T12:21:04ZKateryna Kurach (EPAM)Version endpoint - Part 1 (Storage Service)Issue that tracks M8 activities: https://community.opengroup.org/osdu/platform/home/-/issues/36
I'd like to propose a new endpoint for all services to retrieve the version information. I'm most interested in the tag version / upcoming t...Issue that tracks M8 activities: https://community.opengroup.org/osdu/platform/home/-/issues/36
I'd like to propose a new endpoint for all services to retrieve the version information. I'm most interested in the tag version / upcoming tag version. My thinking is maven-centric, but I'd like to get the artifact version injected into the jar files and available via a simple GET endpoint.
I have several use cases in mind right now:
1. The end customer should have a way to query their environment to know what versions they are running. Then they would know that patches they see coming in on community have been applied to their environment. The Admin UI may be able to use this endpoint to make that query easier.
1. Application developers would be able to query versions of the services they are working with to determine compatibility.
1. The CI pipeline can query the running instances of dependent services, and issue a warning if the major/minor doesn't match the currently executing one. Have branch names or commit hashes would further improve this, but isn't part of my initial thinking.
What complexities and challenges do you see in trying to provide this information?M7 - Release 0.10https://community.opengroup.org/osdu/platform/home/-/issues/36Version Endpoint [GONRG-2681]2023-06-13T20:06:12ZDavid Diederichd.diederich@opengroup.orgVersion Endpoint [GONRG-2681]I'd like to propose a new endpoint for all services to retrieve the version information. I'm most interested in the tag version / upcoming tag version. My thinking is maven-centric, but I'd like to get the artifact version injected into ...I'd like to propose a new endpoint for all services to retrieve the version information. I'm most interested in the tag version / upcoming tag version. My thinking is maven-centric, but I'd like to get the artifact version injected into the jar files and available via a simple GET endpoint.
I have several use cases in mind right now:
1. The end customer should have a way to query their environment to know what versions they are running. Then they would know that patches they see coming in on community have been applied to their environment. The Admin UI may be able to use this endpoint to make that query easier.
2. Application developers would be able to query versions of the services they are working with to determine compatibility.
3. The CI pipeline can query the running instances of dependent services, and issue a warning if the major/minor doesn't match the currently executing one. Have branch names or commit hashes would further improve this, but isn't part of my initial thinking.
What complexities and challenges do you see in trying to provide this information?
https://jiraeu.epam.com/browse/GONRG-2681
Scope for M7:
storage,
search,
Indexer-queue,
Indexer,
LegalM8 - Release 0.11Kateryna Kurach (EPAM)Kateryna Kurach (EPAM)https://community.opengroup.org/osdu/platform/home/-/issues/35OSDU community reports2022-06-22T17:17:56Zethiraj krishnamanaiduOSDU community reports1. Community size/trend (no. active contributors and demographics, e.g. company)
2. Activity stats
- No. commits trends (by time, by commitors, by company)
- No. merge requests (by time, by commitors, by company)
3. Repo stats
- No of R...1. Community size/trend (no. active contributors and demographics, e.g. company)
2. Activity stats
- No. commits trends (by time, by commitors, by company)
- No. merge requests (by time, by commitors, by company)
3. Repo stats
- No of Repo
- Size (in LOC)
- Churn (add/update/remove), would be good to know where these churns are (e.g. on which repo, a heat map)
- Complexity (any other standard code stats we are tracking to indicate the health of the code base)
4. Test stats
- Tests (different levels, unit, integration, e2e) and coveragesDavid Diederichd.diederich@opengroup.orgethiraj krishnamanaiduDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/home/-/issues/33Springboot version upgrade2020-08-31T19:16:05ZNitin-slbSpringboot version upgradeThis is needed to fix the whitesource scan issues in core and cp modules.This is needed to fix the whitesource scan issues in core and cp modules.ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/home/-/issues/31Partition service2023-10-20T12:34:01Zashley kelhamPartition service## Status
- [X] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
The OSDU data platform supports the concept of data partitions and the availability of multiple partitions for a single deploymen...## Status
- [X] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
The OSDU data platform supports the concept of data partitions and the availability of multiple partitions for a single deployment.
A data partition provides the highest level of data isolation within a single OSDU deployment. All access rights are governed at a partition level and data is separated in a way that allows for the partitions life cycle and deployment to be handled independently from one another.
![n](/uploads/841b498ac93e5da86571878308db378e/n.png)
A data partition helps solve many concerns
- Data Access – More sensitive data can be deployed to a separate partition providing an isolated access boundary separate to the rest of their data providing higher security
- Data Management – Organizations can distribute partitions to different segments/units for self-management where appropriate for reasons of budgeting, access rights, rights of use, licensing etc.
- Joint Ventures - Partitions can be set up for joint ventures between different organizations
- Data Residency – An organization can deploy a partition to a location with unique residency/sovereignty concerns to the rest of their data. Data partitions are an orthogonal concern to the region/location of a deployment but can be used to help simplify concerns in cases of data sovereignty and residency concerns.
- Development - Organizations want to develop applications on top of their deployments. Data partitions allow the test data used in development to be kept completely separate from their real data stopping pollution scenarios in their workflows.
- SaaS delivery - ability to deliver different partitions for different customers. Supporting encryption of data in different partitions with customer specific keys for additional data protection.
Most of the APIs in OSDU expose a custom header 'data-partition-id'. The value of this points to the data partition the client is requesting access to.
In the current implementation the specific properties that relate to a partition are largely captured through the shared ‘TenantInfo’ data object. Other properties specific to a partition e.g. secrets are often stored in different key vaults or databases and also need to be retrieved.
These are then high-frequency usage points in the system and we want to rationalize these concerns in a central service to help maintain the integrity of the system.
## Decision
For R3 we will encapsulate the storage and retrieval of partition specific properties within its own service.
## Rationale
This will encapsulate the storage and retrieval of partition specific information. This will decouple implementations away from shared databases improving maintainability.
We can also encapsulate the performance, reliability, and scaling needs behind this service. Retrieving partition information is a high frequency, single point of failure in the system so having each service duplicate this increases the risk of service availability.
## Consequences
Deployments will need to switch to store the partition specific properties via the service.
Services will need to be switched to retrieve partition specific properties via the service.
These changes can happen gradually overtime and should be specific to each provider. One providers consumption of this new service should not affect another providers implementation.
# Trade-off Analysis - Input to decision
An alternative approach is the current Java shared code implementation. Here we see that a large amount of the partition specific properties are retrieved in a single POJO called ‘TenantInfo’. Other partition specific properties like Elastic Search credentials are retrieved from secret stores. There may be other variations on where this data may be retrieved from.
All of these are single points of failure in the system that are retrieved on every request. If you cannot get these properties, the request cannot be completed for any service that supports the data-partition-id header.
Therefore, each service must implement the logic to retrieve this information or at least each language/framework in OSDU needs a shared component for the most common properties.
These properties are needed on every request and having multiple implementations make a performant, elastic and reliable implementation both less likely and harder to achieve.
Similarly coupling all services to specific databases in the system makes maintainability worse as there is tighter coupling of the system so any changes at the data layer end up requiring cascading changes in multiple services.
However, these implementations do already exist and are working today. There is an effort involved in both creating this service and then migrating the other services to use this instead.
## Decision criteria and trade-offs
- Elasticity
- Performance
- Reliability
- Maintainability
- Cost of changeM1 - Release 0.1ethiraj krishnamanaiduFerris ArgyleDania Kodeih (Microsoft)Wladmir FrazaoJoeMatt Wiseethiraj krishnamanaidu2020-08-28https://community.opengroup.org/osdu/platform/home/-/issues/30Partition service2020-08-11T23:10:56Zashley kelhamPartition service## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The OSDU data platform supports the concept of data partitions and the availability of multiple partitions for a single deploymen...## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The OSDU data platform supports the concept of data partitions and the availability of multiple partitions for a single deployment.
A data partition is the highest level of data isolation within an OSDU deployment and keeps data separated from other data partitions.
![n](/uploads/4f16b2ac7078a49f7d857978f9cc2f1e/n.png)
Today the specific properties that relate to a partition are largely captured through the shared ‘TenantInfo’ data object. Other properties specific to a partition e.g. secrets are often stored in different key vaults or databases and also need to be retrieved.
These are then high-frequency usage oints in the system and we want to rationalize these concerns in a central service to help maintain the integrity of the system.
## Decision
For R3 we will encapsulate the storage and retrieval of partition specific properties within its own service.
## Rationale
This will encapsulate the storage and retrieval of partition specific information. This will decouple implementations away from shared databases improving maintainability.
We can also encapsulate the performance, reliability, and scaling needs behind this service. REtrieving partition information is a high frequency, single point of failure in the system so having each service duplicate this increases the risk of service availability.
## Consequences
Deployments will need to switch to store the partition specific properties via the service.
Services will need to be switched to retrieve partition specific properties via the service.
# Trade-off Analysis - Input to decision
An alternative approach is the current Java shared code implementation. Here we see that a large amount of the partition specific properties are retrieved in a single POJO called ‘TenantInfo’. However other partition specific properties like Elastic Search credentials are retrieved from secret stores. There may be other variations on where this data may be retrieved from.
All of these are single points of failure in the system that are retrieved on every request. If you cannot get these properties, the request cannot be completed for any service that supports the data-partition-id header.
Therefore, each service must implement the logic to retrieve this information or at least each language/framework in OSDU needs a shared component for the most common properties.
These properties are needed on every request and therefore having multiple implementations make having a performant, elasticity and reliable implementation both less likely and harder to achieve.
Similarly coupling all services to specific databases in the system makes maintainability worse as there is tighter coupling of the system so any changes at the data layer end up requiring cascading changes in multiple services.
However, these implementations do already exist and are working today. There is an effort involved in both creating this service and then migrating the other services to use this instead.
## Decision criteria and trade-offs
- Elasticity
- Performance
- Reliability
- Maintainability
- Cost of changeM1 - Release 0.1ethiraj krishnamanaiduethiraj krishnamanaidu2020-08-28https://community.opengroup.org/osdu/platform/home/-/issues/32Define and validate Python Standards for GitLab to be used for the rest of th...2021-06-16T22:20:00ZStephen Whitley (Invited Expert)Define and validate Python Standards for GitLab to be used for the rest of the OSDU contributionsWe need the OSDU data platform standards for Python
Repo layout
Code scanning
Logging
....We need the OSDU data platform standards for Python
Repo layout
Code scanning
Logging
....Raj KannanJoeRaj Kannanhttps://community.opengroup.org/osdu/platform/home/-/issues/27Formalize need for BYOC backend for new project/service contributions.2024-03-08T12:59:35ZRaj KannanFormalize need for BYOC backend for new project/service contributions.When we have a contribution of a new project or a new service to OSDU, different dev teams need to be able to quickly try out the code and understand the behavior beyond the overview/walkthrough done by the contributing team.
However in...When we have a contribution of a new project or a new service to OSDU, different dev teams need to be able to quickly try out the code and understand the behavior beyond the overview/walkthrough done by the contributing team.
However in order for our dev teams to be able to run/debug the service and associated integration test themselves, requires the dev team to have access to the deployment environment of the contributor and their underlying CSP secrets, keys etc., which causes latency in understanding and adoption of code within OSDU platform.
BYOC back-end is intended for this purpose, but we haven't required this as part of any contribution. It may be worth considering this as a first step in project contribution after the code is authorized for inclusion in community Gitlab, so we can make the above easier.
### Requirements
* BYOC back-end implementation for any new service or project contribution to OSDU platform
* Ability for all CSPs and other platform developers to be able to run the service, test-suite independent of the original contributor's infrastructure
* Associated CI/CD updates to ensure it can be run independently w/BYOC and (ideally) on a developer workstation.
### Operator Input
* Pending, but this is more of a PMC governance and process streamlining issue.
### Definition of Done
* After Code is scanned and validated for contribution, contributing team works on a BYOC backend
* new service and/or project is able to pass the integration tests with a BYOC backend
* basic documentation is included on how to build/run against BYOC to learn about this service or projectM1 - Release 0.1Raj KannanJoeRaj Kannan2020-07-10https://community.opengroup.org/osdu/platform/home/-/issues/26Trade off analysis between generic Plug & Play vs Service Replacement2023-10-20T12:32:14ZMeena RathinavelTrade off analysis between generic Plug & Play vs Service ReplacementWe need to analyze and align on Plug & Play service vs Service Replacement
Also understanding whether this would be canonical for all customers or will be within **only OSDU **
The above is an action item coming from our weekly PMC Pr...We need to analyze and align on Plug & Play service vs Service Replacement
Also understanding whether this would be canonical for all customers or will be within **only OSDU **
The above is an action item coming from our weekly PMC Program Review meeting on 06/16M1 - Release 0.1Stephen Whitley (Invited Expert)Stephen Whitley (Invited Expert)2020-06-30