Home issueshttps://community.opengroup.org/osdu/platform/home/-/issues2024-03-08T12:59:35Zhttps://community.opengroup.org/osdu/platform/home/-/issues/27Formalize need for BYOC backend for new project/service contributions.2024-03-08T12:59:35ZRaj KannanFormalize need for BYOC backend for new project/service contributions.When we have a contribution of a new project or a new service to OSDU, different dev teams need to be able to quickly try out the code and understand the behavior beyond the overview/walkthrough done by the contributing team.
However in...When we have a contribution of a new project or a new service to OSDU, different dev teams need to be able to quickly try out the code and understand the behavior beyond the overview/walkthrough done by the contributing team.
However in order for our dev teams to be able to run/debug the service and associated integration test themselves, requires the dev team to have access to the deployment environment of the contributor and their underlying CSP secrets, keys etc., which causes latency in understanding and adoption of code within OSDU platform.
BYOC back-end is intended for this purpose, but we haven't required this as part of any contribution. It may be worth considering this as a first step in project contribution after the code is authorized for inclusion in community Gitlab, so we can make the above easier.
### Requirements
* BYOC back-end implementation for any new service or project contribution to OSDU platform
* Ability for all CSPs and other platform developers to be able to run the service, test-suite independent of the original contributor's infrastructure
* Associated CI/CD updates to ensure it can be run independently w/BYOC and (ideally) on a developer workstation.
### Operator Input
* Pending, but this is more of a PMC governance and process streamlining issue.
### Definition of Done
* After Code is scanned and validated for contribution, contributing team works on a BYOC backend
* new service and/or project is able to pass the integration tests with a BYOC backend
* basic documentation is included on how to build/run against BYOC to learn about this service or projectM1 - Release 0.1Raj KannanJoeRaj Kannan2020-07-10https://community.opengroup.org/osdu/platform/home/-/issues/48Removing Airflow 1.x support in favor Airflow 2.x for M12+2023-12-05T10:33:23ZChad LeongRemoving Airflow 1.x support in favor Airflow 2.x for M12+## Existing Practice / Background
Airflow 2.x was introduced in M10 release for various improvements over Airflow 1.x. In M10 Airflow 2.x was released with feature flags so that customers can choose to stay at Airflow 1.x as well.
## M...## Existing Practice / Background
Airflow 2.x was introduced in M10 release for various improvements over Airflow 1.x. In M10 Airflow 2.x was released with feature flags so that customers can choose to stay at Airflow 1.x as well.
## Motivation
The OSDU community has decided to deprecate the support for Airflow 1.x and will completely remove the ability to stay at Airflow 1.x starting in M12 milestone.
CSP Decision Approvals
- [x] AWS
- [x] Azure
- [x] IBM
- [x] GCPhttps://community.opengroup.org/osdu/platform/home/-/issues/31Partition service2023-10-20T12:34:01Zashley kelhamPartition service## Status
- [X] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
The OSDU data platform supports the concept of data partitions and the availability of multiple partitions for a single deploymen...## Status
- [X] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
The OSDU data platform supports the concept of data partitions and the availability of multiple partitions for a single deployment.
A data partition provides the highest level of data isolation within a single OSDU deployment. All access rights are governed at a partition level and data is separated in a way that allows for the partitions life cycle and deployment to be handled independently from one another.
![n](/uploads/841b498ac93e5da86571878308db378e/n.png)
A data partition helps solve many concerns
- Data Access – More sensitive data can be deployed to a separate partition providing an isolated access boundary separate to the rest of their data providing higher security
- Data Management – Organizations can distribute partitions to different segments/units for self-management where appropriate for reasons of budgeting, access rights, rights of use, licensing etc.
- Joint Ventures - Partitions can be set up for joint ventures between different organizations
- Data Residency – An organization can deploy a partition to a location with unique residency/sovereignty concerns to the rest of their data. Data partitions are an orthogonal concern to the region/location of a deployment but can be used to help simplify concerns in cases of data sovereignty and residency concerns.
- Development - Organizations want to develop applications on top of their deployments. Data partitions allow the test data used in development to be kept completely separate from their real data stopping pollution scenarios in their workflows.
- SaaS delivery - ability to deliver different partitions for different customers. Supporting encryption of data in different partitions with customer specific keys for additional data protection.
Most of the APIs in OSDU expose a custom header 'data-partition-id'. The value of this points to the data partition the client is requesting access to.
In the current implementation the specific properties that relate to a partition are largely captured through the shared ‘TenantInfo’ data object. Other properties specific to a partition e.g. secrets are often stored in different key vaults or databases and also need to be retrieved.
These are then high-frequency usage points in the system and we want to rationalize these concerns in a central service to help maintain the integrity of the system.
## Decision
For R3 we will encapsulate the storage and retrieval of partition specific properties within its own service.
## Rationale
This will encapsulate the storage and retrieval of partition specific information. This will decouple implementations away from shared databases improving maintainability.
We can also encapsulate the performance, reliability, and scaling needs behind this service. Retrieving partition information is a high frequency, single point of failure in the system so having each service duplicate this increases the risk of service availability.
## Consequences
Deployments will need to switch to store the partition specific properties via the service.
Services will need to be switched to retrieve partition specific properties via the service.
These changes can happen gradually overtime and should be specific to each provider. One providers consumption of this new service should not affect another providers implementation.
# Trade-off Analysis - Input to decision
An alternative approach is the current Java shared code implementation. Here we see that a large amount of the partition specific properties are retrieved in a single POJO called ‘TenantInfo’. Other partition specific properties like Elastic Search credentials are retrieved from secret stores. There may be other variations on where this data may be retrieved from.
All of these are single points of failure in the system that are retrieved on every request. If you cannot get these properties, the request cannot be completed for any service that supports the data-partition-id header.
Therefore, each service must implement the logic to retrieve this information or at least each language/framework in OSDU needs a shared component for the most common properties.
These properties are needed on every request and having multiple implementations make a performant, elastic and reliable implementation both less likely and harder to achieve.
Similarly coupling all services to specific databases in the system makes maintainability worse as there is tighter coupling of the system so any changes at the data layer end up requiring cascading changes in multiple services.
However, these implementations do already exist and are working today. There is an effort involved in both creating this service and then migrating the other services to use this instead.
## Decision criteria and trade-offs
- Elasticity
- Performance
- Reliability
- Maintainability
- Cost of changeM1 - Release 0.1ethiraj krishnamanaiduFerris ArgyleDania Kodeih (Microsoft)Wladmir FrazaoJoeMatt Wiseethiraj krishnamanaidu2020-08-28https://community.opengroup.org/osdu/platform/home/-/issues/26Trade off analysis between generic Plug & Play vs Service Replacement2023-10-20T12:32:14ZMeena RathinavelTrade off analysis between generic Plug & Play vs Service ReplacementWe need to analyze and align on Plug & Play service vs Service Replacement
Also understanding whether this would be canonical for all customers or will be within **only OSDU **
The above is an action item coming from our weekly PMC Pr...We need to analyze and align on Plug & Play service vs Service Replacement
Also understanding whether this would be canonical for all customers or will be within **only OSDU **
The above is an action item coming from our weekly PMC Program Review meeting on 06/16M1 - Release 0.1Stephen Whitley (Invited Expert)Stephen Whitley (Invited Expert)2020-06-30https://community.opengroup.org/osdu/platform/home/-/issues/28Issue Taxonomy2023-10-20T12:32:04ZStephen Whitley (Invited Expert)Issue TaxonomyWe are often left to address the gaps from architectural principles (which stay at a pretty high and abstract level) to the actual implementation detail. Here is an attempt to bridge that gap by providing a set of Lightweight Architectur...We are often left to address the gaps from architectural principles (which stay at a pretty high and abstract level) to the actual implementation detail. Here is an attempt to bridge that gap by providing a set of Lightweight Architecture Decision Records (LADRs) which are simple to follow and can be implemented in a given team/project by the developers
# Decision Title
Taxonomy for issues to ensure efficient tracking and governance of PMC projects
## Status
- [x] Initiated
- [x] Proposed
- [x] Trialing
- [x] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The proposal can be found [**here**](https://community.opengroup.org/osdu/governance/project-management-committee/-/wikis/PMC-Issue-Taxonomy) for review.
It is suggested that after a first pass review of the proposal to refine, that this is put to trial for the System and Data flow projects using the proposed methodology.
The taxonomy guideline can be further refined based on practical application in these projects and then baselined.
## Decision
Stage-1: Review with CSP leads, program managers and OSDU lead to get first pass feedback.
## Rationale
Gitlab issue list and boards are very generic and we need to be able have visibility at program level for macro tasks and be able to define, assign and manage micro level tasks within a repo or sub-project level as well using the same system.
## Consequences
The current issue list is unwieldy and therefore becomes a management and governance challenge. A working model is in order to get us to a productive environment in Gitlab.
## When to revisit
Suggest 3 sprints of practical application in System and Data flow projects and to review after that
Aug 2020
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
* Ability to manage and track ADR record lifecycle
* Ability to manage and track macro backlog items for program level reporting, detailed backlog and defect items at project level for assignments
* Ability to define CSP specific issues and defects and track their assignments and progress
## Decision timeline
Aug 2020M1 - Release 0.1Raj KannanJoeRaj Kannanhttps://community.opengroup.org/osdu/platform/home/-/issues/24OSDU Code tagging and versioning2023-10-20T12:31:37Zethiraj krishnamanaiduOSDU Code tagging and versioningWe are often left to address the gaps from architectural principles (which stay at a pretty high and abstract level) to the actual implementation detail. Here is an attempt to bridge that gap by providing a set of Lightweight Architectur...We are often left to address the gaps from architectural principles (which stay at a pretty high and abstract level) to the actual implementation detail. Here is an attempt to bridge that gap by providing a set of Lightweight Architecture Decision Records (LADRs) which are simple to follow and can be implemented in a given team/project by the developers
# Decision Title
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
## Decision
## Rationale
## Consequences
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelineethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/home/-/issues/40Security Vulnerability issues for M9 (Assigned to IBM team)2023-10-05T11:16:27ZChris ZhangSecurity Vulnerability issues for M9 (Assigned to IBM team)Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track IBM team's tasks on:
1. CRS Conversion - @JINGDONG SUN(IB...Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track IBM team's tasks on:
1. CRS Conversion - @JINGDONG SUN(IBM)
1. Unit - @JINGDONG SUN(IBM)
**Additional Information**
The link below contains information on the vulnerability remediation process. For additional information on how to close out a vulnerability after it has been resolved please review step 7.
https://community.opengroup.org/groups/osdu/platform/-/wikis/Vulnerability-RemediationM9 - Release 0.12jingdong sunjingdong sunhttps://community.opengroup.org/osdu/platform/home/-/issues/42Security Vulnerability issues for M9 (Assigned to MSFT team)2023-10-05T11:16:18ZChris ZhangSecurity Vulnerability issues for M9 (Assigned to MSFT team)Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track MSFT team's tasks on:
1. Schema - @Madhur Tanwani(MSFT)
1...Address all ‘critical’ and ‘high’ severity vulnerabilities in this report: https://community.opengroup.org/groups/osdu/platform/-/security/vulnerabilities.
This issue is to track MSFT team's tasks on:
1. Schema - @Madhur Tanwani(MSFT)
1. Infra-azure-provisioning - @Madhur Tanwani(MSFT)
**Additional Information**
The link below contains information on the vulnerability remediation process. For additional information on how to close out a vulnerability after it has been resolved please review step 7.
https://community.opengroup.org/groups/osdu/platform/-/wikis/Vulnerability-RemediationM9 - Release 0.12Madhur Tanwani [Microsoft]Krishnan GanesanMadhur Tanwani [Microsoft]https://community.opengroup.org/osdu/platform/home/-/issues/47Swagger UI throws "Whitelabel Error Page" for several services2023-08-02T10:04:35ZAn NgoSwagger UI throws "Whitelabel Error Page" for several servicesFor several services, the swagger page is showing "Whitelabel Error Page".
![image](/uploads/dc287779032213e02f5056e4390a23bb/image.png)
This error is observed after the swagger upgrade.
This is due to the reason that there are some m...For several services, the swagger page is showing "Whitelabel Error Page".
![image](/uploads/dc287779032213e02f5056e4390a23bb/image.png)
This error is observed after the swagger upgrade.
This is due to the reason that there are some missing/ inconsistent updates related to swagger for every service.
For example - In case of Entitlements Service, the swagger commit involved changes in the **[azure-istio-auth-policy.yaml](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/merge_requests/175/diffs?commit_id=56fbdf335f8ab29fd794314118b2fb3e5654ad39#acaaad215c25b431962f64748aa1c65fe3699abd)** but [storage service](https://community.opengroup.org/osdu/platform/system/storage/-/commit/191597a47d10ba3f69e42148e3c5c1e7c7ca04f9#022b6d114052cad09a79e365f1af9879bf04fe63) is missing this update.
Same is the case for the other services.
- It is mostly because in some services, this file does not exist within the service repository but in the [**infra-azure-provisioning**](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts/osdu-istio-auth/templates).
- For some services like `Notification` - this same [**file**](https://community.opengroup.org/osdu/platform/system/notification/-/tree/master/devops/azure/chart/templates) exists with a different name. So its missed.
- Same for `Register` service - the [**file**](https://community.opengroup.org/osdu/platform/system/register/-/blob/master/devops/azure/chart/templates/osdu-istio-policy.yaml) exists with a different name
This inconsistent updates are likely causing the swagger page to be broken.
**Impacted services:**
- CRS Catalog Service
- CRS Conversion Service
- Legal Service
- Partition Service
- Register Service
- Seismic Data Management Service
- Search Service
- Spatial Ref Service (no swagger endpoint)
- Storage Service
- Unit Service
- Workflow Service
**These ones are working:**
- Entitlements
- Indexer
- Search Extension
- Schema
- Wellbore DMS
- File
- WorkflowSrinivasan NarayananSrinivasan Narayananhttps://community.opengroup.org/osdu/platform/home/-/issues/36Version Endpoint [GONRG-2681]2023-06-13T20:06:12ZDavid Diederichd.diederich@opengroup.orgVersion Endpoint [GONRG-2681]I'd like to propose a new endpoint for all services to retrieve the version information. I'm most interested in the tag version / upcoming tag version. My thinking is maven-centric, but I'd like to get the artifact version injected into ...I'd like to propose a new endpoint for all services to retrieve the version information. I'm most interested in the tag version / upcoming tag version. My thinking is maven-centric, but I'd like to get the artifact version injected into the jar files and available via a simple GET endpoint.
I have several use cases in mind right now:
1. The end customer should have a way to query their environment to know what versions they are running. Then they would know that patches they see coming in on community have been applied to their environment. The Admin UI may be able to use this endpoint to make that query easier.
2. Application developers would be able to query versions of the services they are working with to determine compatibility.
3. The CI pipeline can query the running instances of dependent services, and issue a warning if the major/minor doesn't match the currently executing one. Have branch names or commit hashes would further improve this, but isn't part of my initial thinking.
What complexities and challenges do you see in trying to provide this information?
https://jiraeu.epam.com/browse/GONRG-2681
Scope for M7:
storage,
search,
Indexer-queue,
Indexer,
LegalM8 - Release 0.11Kateryna Kurach (EPAM)Kateryna Kurach (EPAM)https://community.opengroup.org/osdu/platform/home/-/issues/50Release management for Core Libraries.2023-03-30T08:14:20ZKrishna Nikhil VedurumudiRelease management for Core Libraries.Change in release process for Core Libraries to have reduced impact on Code tagging process for Milestone releases
Ref: https://gitlab.opengroup.org/osdu/subcommittees/ea/work-products/adr-elaboration/-/issues/79Change in release process for Core Libraries to have reduced impact on Code tagging process for Milestone releases
Ref: https://gitlab.opengroup.org/osdu/subcommittees/ea/work-products/adr-elaboration/-/issues/79Krishna Nikhil VedurumudiKrishna Nikhil Vedurumudihttps://community.opengroup.org/osdu/platform/home/-/issues/51Update Info API to support Core Version and implementation Version2023-03-30T08:11:31ZKrishna Nikhil VedurumudiUpdate Info API to support Core Version and implementation Versionhttps://gitlab.opengroup.org/osdu/subcommittees/ea/work-products/adr-elaboration/-/issues/78https://gitlab.opengroup.org/osdu/subcommittees/ea/work-products/adr-elaboration/-/issues/78Krishna Nikhil VedurumudiKrishna Nikhil Vedurumudihttps://community.opengroup.org/osdu/platform/home/-/issues/22Definition of Done for R3 Clarification2022-09-15T23:49:36ZDania Kodeih (Microsoft)Definition of Done for R3 ClarificationWhat is the meaning of "at runtime" in this page: https://community.opengroup.org/osdu/platform/home/-/wikis/Planning/R3/DoneWhat is the meaning of "at runtime" in this page: https://community.opengroup.org/osdu/platform/home/-/wikis/Planning/R3/DoneStephen Whitley (Invited Expert)Stephen Whitley (Invited Expert)https://community.opengroup.org/osdu/platform/home/-/issues/20OSDU Data Platform Documentation2022-09-15T23:49:36ZStephen Whitley (Invited Expert)OSDU Data Platform DocumentationOSDU Data Platform Documentation
- [ ] OSDU System Architecture and Design
- [ ] Service Provider API Standards for OSDU (cloud-vendor specific)OSDU Data Platform Documentation
- [ ] OSDU System Architecture and Design
- [ ] Service Provider API Standards for OSDU (cloud-vendor specific)M1 - Release 0.12020-07-31https://community.opengroup.org/osdu/platform/home/-/issues/10Complete Entitlements APIs for R3 development2022-08-23T13:29:45ZStephen Whitley (Invited Expert)Complete Entitlements APIs for R3 developmentM1 - Release 0.1Dania Kodeih (Microsoft)JoeChris ZhangDania Kodeih (Microsoft)2020-03-31https://community.opengroup.org/osdu/platform/home/-/issues/2Populate Backlog from SLB Contributions2022-08-23T13:29:45ZStephen Whitley (Invited Expert)Populate Backlog from SLB ContributionsPopulate the Backlog issues list with the list of additional services that SLB is contributing during the R3 development phase.Populate the Backlog issues list with the list of additional services that SLB is contributing during the R3 development phase.M1 - Release 0.1Chris ZhangAn NgoMichael CleminsonChris Zhang2020-02-28https://community.opengroup.org/osdu/platform/home/-/issues/18[Platform] Move common code base to GitLab2022-08-23T13:29:45ZStephen Whitley (Invited Expert)[Platform] Move common code base to GitLabMigrate from Azure DevOps (code) and internal gitlab (data models, data) to community gitlab for R3 development.
Move the Core services for Legal and Entitlements (4 providers) ported in Release 2 from ADO to GitLab
- System Services
...Migrate from Azure DevOps (code) and internal gitlab (data models, data) to community gitlab for R3 development.
Move the Core services for Legal and Entitlements (4 providers) ported in Release 2 from ADO to GitLab
- System Services
- [x] Core Common
- [x] AWS Lib
- [x] Azure Lib
- [x] GCP Lib
- [x] IBM Lib
- [x] Storage
- [x] Search
- [x] Indexer
- [x] indexer queue
- Security and Compliance
- [x] Legal
- [x] AWS Entitlement
- [x] AZure Entitlement
- [x] GCP Entitlement
- [x] IBM Entitlement
Participants:
- @dkodeih, MSFT
- @fargyle, GCP
- @babeal, AWS
- @wladmirf. IBM
See [Code Freeze](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-R2-code-freeze) ActivitiesM1 - Release 0.1David Diederichd.diederich@opengroup.orgethiraj krishnamanaiduFerris ArgyleDania Kodeih (Microsoft)Wladmir FrazaoJoeBrandt BealDavid Diederichd.diederich@opengroup.org2020-05-29https://community.opengroup.org/osdu/platform/home/-/issues/21Tag platform projects to capture the migration and validation of R2 code to G...2022-08-23T13:29:44ZStephen Whitley (Invited Expert)Tag platform projects to capture the migration and validation of R2 code to GitLabWe need to tag the R3 code at the point of validation migration from ADOWe need to tag the R3 code at the point of validation migration from ADOM1 - Release 0.1David Diederichd.diederich@opengroup.orgethiraj krishnamanaiduDavid Diederichd.diederich@opengroup.org2020-05-15https://community.opengroup.org/osdu/platform/home/-/issues/35OSDU community reports2022-06-22T17:17:56Zethiraj krishnamanaiduOSDU community reports1. Community size/trend (no. active contributors and demographics, e.g. company)
2. Activity stats
- No. commits trends (by time, by commitors, by company)
- No. merge requests (by time, by commitors, by company)
3. Repo stats
- No of R...1. Community size/trend (no. active contributors and demographics, e.g. company)
2. Activity stats
- No. commits trends (by time, by commitors, by company)
- No. merge requests (by time, by commitors, by company)
3. Repo stats
- No of Repo
- Size (in LOC)
- Churn (add/update/remove), would be good to know where these churns are (e.g. on which repo, a heat map)
- Complexity (any other standard code stats we are tracking to indicate the health of the code base)
4. Test stats
- Tests (different levels, unit, integration, e2e) and coveragesDavid Diederichd.diederich@opengroup.orgethiraj krishnamanaiduDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/home/-/issues/45Critical security vulnerability about Swagger2022-03-01T13:25:46ZChris ZhangCritical security vulnerability about SwaggerThis is to track the security vulnerability about Swagger. https://nvd.nist.gov/vuln/detail/CVE-2019-17495
This applies to all OSDU services, and it is reported in OSDU security dashboard as well. This is one example: https://community...This is to track the security vulnerability about Swagger. https://nvd.nist.gov/vuln/detail/CVE-2019-17495
This applies to all OSDU services, and it is reported in OSDU security dashboard as well. This is one example: https://community.opengroup.org/osdu/platform/system/dataset/-/security/vulnerabilities/16194M10 - Release 0.13