OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2020-06-24T20:30:44Zhttps://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/12Kubernetes Security Requirements2020-06-24T20:30:44ZPaco Hope (AWS)Kubernetes Security Requirements### Input from Operators
These requirements provided by
- **Chevron**
- **Total**
- **Repsol**
- **Shell**
### Requirements
- K8s diagnostic settings must be enabled and forwarded to Log Analytics
- Azure AD should be enabled in Kuber...### Input from Operators
These requirements provided by
- **Chevron**
- **Total**
- **Repsol**
- **Shell**
### Requirements
- K8s diagnostic settings must be enabled and forwarded to Log Analytics
- Azure AD should be enabled in Kubernetes Service
- Cluster RBAC must be enabled in Kubernetes Service
- Do not directly or indirectly grant cluster admin level access to developers
- The latest version of Kubernetes should be used . Autoupdate daemon to be used
- Ensure containers listen only on allowed ports
- Do not allow privileged containers in AKS
- Container build file must be auditable/visible
- Allow Secrets Injection into containersM1 - Release 0.1https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/13Static Code Analysis during CI/CD Pipeline2021-06-28T17:21:55ZPaco Hope (AWS)Static Code Analysis during CI/CD PipelineThis probably belongs in the [CI-CD pipelines](https://community.opengroup.org/osdu/platform/ci-cd-pipelines) board.
### Requirements
- Java code needs to pass through a static code analysis tool.
- Infrastructure code needs to pass th...This probably belongs in the [CI-CD pipelines](https://community.opengroup.org/osdu/platform/ci-cd-pipelines) board.
### Requirements
- Java code needs to pass through a static code analysis tool.
- Infrastructure code needs to pass through a static code analysis tool.
- Third party / open source code libraries that are used need to be scanned for vulnerabilities and for compatible open source licenses.
### Operator Input
* **Chevron** lists this as a requirement
* **Shell** lists this as a requirement
* **ExxonMobil** lists this as a requirement
* **BP** lists this as a requirement
### Definition of Done
- Code is scanned as a standard set of tests during the standard pipeline build
- Security tests failing can cause the build to failM1 - Release 0.1https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/15Operating System Security Requirements2020-06-24T20:30:16ZPaco Hope (AWS)Operating System Security RequirementsOperators have specific requirements for the operating systems that they run. Whether that's the OS version (Windows versions, Linux distributions, etc.) and customizations that are unique to them. E.g., installing agents for data collec...Operators have specific requirements for the operating systems that they run. Whether that's the OS version (Windows versions, Linux distributions, etc.) and customizations that are unique to them. E.g., installing agents for data collection or security. E.g., disabling services, configuring services like SSH, subsystems like DNS, etc.
The data platform needs to allow these customizations to be applied and maintained when deploying. Whether that is operator-selected machine images, operator-selected container platforms, or installing operator-provided.
### Operator Inputs
- **BP**: Permit custom security policies / modules in the OS builds that are deployed in the CSP
- **BP**: Permit specific agents (e.g., Azure) loaded into OS builds that are deployed.
- **Noble Energy**: Windows 10, Windows 2019 Server, Centos and Oracle Linux. Pre-fer kernel 7 for driver support. M1 - Release 0.1https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/16Futureproofing: Cryptographic agility2023-07-05T10:09:41ZPaco Hope (AWS)Futureproofing: Cryptographic agilityThis is **not** an R3 issue.
**ExxonMobil** has raised the concern of quantum cryptography. For the future we need to make sure that the OSDU Data Platform can shift to quantum-safe encryption mechanisms when those mechanisms exist.This is **not** an R3 issue.
**ExxonMobil** has raised the concern of quantum cryptography. For the future we need to make sure that the OSDU Data Platform can shift to quantum-safe encryption mechanisms when those mechanisms exist.https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/17Network Security Requirements2020-09-22T03:40:20ZPaco Hope (AWS)Network Security RequirementsThe infrastructure that runs the OSDU Data platform must have limited outbound connectivity to the Internet.
How this is accomplished will vary by cloud provider. But it is important that individual containers, services, and hosts cann...The infrastructure that runs the OSDU Data platform must have limited outbound connectivity to the Internet.
How this is accomplished will vary by cloud provider. But it is important that individual containers, services, and hosts cannot make arbitrary connections to arbitrary outside IP addresses on arbitrary ports.
# Network security controls
## Egress
What network egress requirements are there for OSDU Data Platform services? E.g., if the data platform needs to/wants to connect to an outside data source, what network security requirements does the Operator impose?
Example controls include:
- Whitelisted domains and/or IP ranges.
- Proxying outbound HTTPS requests through an Operator-supplied proxy.
- For example, if the Load Service calls the Search Service, can that HTTPS API call pass through an HTTPS proxy.
- Egress to the Operator's on-premises network via a dedicated VPN channel
- Some operators want to block all outbound connection to internet IPs
## Ingress
Connections inbound are usually monitored.
- What level of logging / information is required for inbound connections?
## Transit
What network security controls are required inside the internal cloud network (e.g., between services)?
### Operator Input
- **Wintershall Dea**: And the Server(s) will have restrictions to access the Internet, at least it will be strictly controlled. HTTPS outbound only connections are possible. 2) The Deployment run in our on-premises Datacenter(s), here we have only Proxy-based Internet-Access for Servers. HTTPS outbound should be also no problem when it’s agreed.
- **ExxonMobil**: It is a requirement to have any outbound HTTPS pass through a proxy.
- **Shell**: WAF or L7 security inbound is required.https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/18Code Integrity, Access Control, Change Management2020-06-26T21:16:14ZPaco Hope (AWS)Code Integrity, Access Control, Change ManagementOperators want to know who has access/authority to commit code to the OSDU Data Platform. This doesn't need to be complex or long. But they need to know who can commit and how those permissions are handled. The process for accepting pull...Operators want to know who has access/authority to commit code to the OSDU Data Platform. This doesn't need to be complex or long. But they need to know who can commit and how those permissions are handled. The process for accepting pull requests/commits and making changes to the code should also be documented.
### Operator Inputs
- **Shell** lists this as a requirement.
### Definition of Done
Documentation and/or a link is probably sufficient. E.g., a link to the current list of approved committers, and a statement like "The OSDU OMC approves committers based on X, Y, Z". A statement about which workstream has authority over change management will satisfy that requirement, too.M1 - Release 0.1Raj KannanJoeRaj Kannanhttps://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/19Selectable External Content Encryption2020-06-19T14:07:41ZPaco Hope (AWS)Selectable External Content EncryptionSome sensitive content must be configured to be encrypted using an external encryption key source. Data that has the attribute **[TBD:ExternalEncryption]** must be encrypted prior to storage in the underlying cloud provider's storage sys...Some sensitive content must be configured to be encrypted using an external encryption key source. Data that has the attribute **[TBD:ExternalEncryption]** must be encrypted prior to storage in the underlying cloud provider's storage system. The encryption key for this encryption is pre-established by the operator. The external key providing system is reached via API call from the OSDU data platform service prior to encrypting/decrypting. This call can fail (because the operator has withdrawn consent to decrypt this data in the data platform), thus failures must be handled gracefully.
### Operator Input
- **ExxonMobil** lists this as a requirement to store sensitive data (e.g., annotations, commentary) in the OSDU data platform.
### Example: Export data
1. End user requests sensitive data to be exported
2. Data platform service retrieves encrypted data from cloud platform storage
3. Data platform service contacts external key provider to retrieve data key (that decrypts this data element)
4. Choice:
a. External key service replies negative: no key found / available. Data platform returns an error code.
b. External key service replies with a data key: Data platform decrypts the data and continues normally
### Example: Load data
1. End user requests sensitive data to be loaded. Manifest sets the **[TBD:ExternalEncryption]** in the manifest.
2. Data platform service contacts external key provider to retrieve **new** data key (that will encrypts this data element)
3. Choice:
a. External key service replies negative: no key available. Data platform returns an error code for this file.
b. External key service replies with a data key: Data platform encrypts the data with the provided data key and continues normallyhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/1Missing clear test strategy and test framework on Airflow DAG development2021-06-14T16:07:20ZWei SunMissing clear test strategy and test framework on Airflow DAG developmentNeed the clarity for DAG test framework (unit test and integration test) to ensure the code quality and no DAG broken.Need the clarity for DAG test framework (unit test and integration test) to ensure the code quality and no DAG broken.Ferris ArgyleDania Kodeih (Microsoft)Wladmir FrazaoJoeDmitriy RudkoAlan BrazKateryna Kurach (EPAM)Ferris Argylehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/1[Ingestion Workflow] OSDU services and Airflow two-way secure communications2022-04-07T18:20:38ZPingjiang Wang[Ingestion Workflow] OSDU services and Airflow two-way secure communicationsProvide clear authentication and authorization model and specification on OSDU services and Airflow two-way secure communications.
- How does Workflow service call into Airflow?
- Note: use Airflow basic auth loses the identity of the ...Provide clear authentication and authorization model and specification on OSDU services and Airflow two-way secure communications.
- How does Workflow service call into Airflow?
- Note: use Airflow basic auth loses the identity of the original caller, and increase the potential security risk if the Workflow service is compromised
- How does Airflow call into Storage service?
- Note: should caller identity been passed to Storage service to carry out record insertion? If not, what identity should be used to call Storage service from Airflow?
@dkodeih @stephenwhitleyM1 - Release 0.1Stephen Whitley (Invited Expert)Dania Kodeih (Microsoft)Nick PickusStephen Whitley (Invited Expert)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/2Please explore Apache Airflow limitations to finalize design for R32021-03-02T12:48:31ZRaj KannanPlease explore Apache Airflow limitations to finalize design for R3There have been a few concerns raised by Airflow as a generic DAG/orchestration solution for data flow in OSDU. It would be good to capture these issues here and to respond back with observations/solutions so the decision decision can be...There have been a few concerns raised by Airflow as a generic DAG/orchestration solution for data flow in OSDU. It would be good to capture these issues here and to respond back with observations/solutions so the decision decision can be properly captured.
1. Airflow is cloud-native only in GCP, which can make it cumbersome to host in other CSPs where the management of the infrastructure becomes a platform/operator responsibility unlike PaaS solutions.
1. With Airflow, it will be quite hard to isolate workflows as the workflows are within the same execution environment. As OSDU approaches "OSDU SaaS" and OSDU for smaller operators where it may be hosted by a SI or CSP, this can make it challenging for multi-tenant deployments.
1. Airflow DAGs are python only and some parsers and libraries can be Java or C++. Just as a comparison something like Argo which is kubernetes based could help have worksteps in different language/environments as each becomes a separate container instance rather than a python script.
1. Airflow apparently has an execution delay between tasks - it is unclear if this is a framework limitation or specific experience of a setup, but perhaps worth capturing to analyze.
1. Similarly there are concerns about temporary state/data and an intermediary persistence to hold across DAG worksteps. Beyond what can be held in memory, does Airflow provide a persistable temporary cache for such state?
1. Is the Airflow DSL cumbersome to author for ingestion/enrichment workflow providers (ISVs, SIs, operators). In comparison to YAML or other alternatives is this a good choice.
Once the elaboration work is complete, kindly capture this as a LADR for the Data flow project. Thanks for the advice on these issues.M1 - Release 0.1Ferris ArgyleFerris Argylehttps://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/20Don't preclude Endpoint Detection & Response2020-06-19T14:29:48ZPaco Hope (AWS)Don't preclude Endpoint Detection & ResponseWhere instances / VMs are deployed, do not preclude the use of operator-provided endpoint detection & response software installed on the nodes.
### Operator Inputs
- **Shell** lists this as a requirementWhere instances / VMs are deployed, do not preclude the use of operator-provided endpoint detection & response software installed on the nodes.
### Operator Inputs
- **Shell** lists this as a requirementM1 - Release 0.1https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/18Ingestion Framework Guiding Principles2023-03-09T18:15:53ZStephen Whitley (Invited Expert)Ingestion Framework Guiding Principles
## Ingestion Framework Guiding Principles
These guiding principles are shared by all the SLB authored user stories reflecting the capabilities of OpenDES
* Simple, Dedicated and Efficient APIs as ingestion entry points for each data f...
## Ingestion Framework Guiding Principles
These guiding principles are shared by all the SLB authored user stories reflecting the capabilities of OpenDES
* Simple, Dedicated and Efficient APIs as ingestion entry points for each data format
* Store original high-fidelity data as is. Data in its original form must land first in the most appropriate store. Ex: - A DLIS file must land in the File DMS, A ZGY seismic survey must land in the Seismic DMS, etc. Once the data, in its original form, lands in the appropriate DMS, Parsers/Scanners/Enrichment processes can be applied to that data. The original data is stored as-is and parsers/scanner/clean-up processes will output derived entities.
* Framework should inherently enforce data lifecycle, Original ---> Well Known Structure ---> Well known Entity
* Extensibility of Parsers/Scanner/Enrichment processes through Configurations/Registrations
* Track flow and lifecycle of data in the data platformM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/home/-/issues/33[OSDU Core Services] Project Consensus Model and Voting2020-07-07T00:19:29ZStephen Whitley (Invited Expert)[OSDU Core Services] Project Consensus Model and Voting## Consensus model and Voting
Ideally Open Source projects operate on a daily basis with very little formality. However, the diversity of deployment environments and associated impact on technical architecture and sustainability will le...## Consensus model and Voting
Ideally Open Source projects operate on a daily basis with very little formality. However, the diversity of deployment environments and associated impact on technical architecture and sustainability will lead to inevitable conflict. When these arise, we will follow a decision process that is driven by trade-off analysis, resolved through voting and documented via decision records.
## Trade-off analysis
When consensus is not met we will follow a trade-off analysis in which competing ideas are assessed against **project goals** on the principle that the health of the entire system and subsequent and sustainable adoption by customers is more valuable than any single technical decision.
## Voting
- Simple majority (51%) for any decision that does not introduce a breaking change to a commitment or deployed production version of the system
- Super majority (75%) plus escalation to the PMC for any decision that does introduce a breaking change to a commitment or deployed production version of the systemhttps://community.opengroup.org/osdu/platform/system/home/-/issues/34[OSDU Core Services] Governance Model2020-07-07T00:17:34ZStephen Whitley (Invited Expert)[OSDU Core Services] Governance ModelProject Governance is based on meritocracy and contribution. For Release 3, contribution is largely provided by the following companies (alphabetical) at roughly equal levels of commitment and thus represents the voting block for the pro...Project Governance is based on meritocracy and contribution. For Release 3, contribution is largely provided by the following companies (alphabetical) at roughly equal levels of commitment and thus represents the voting block for the project.
### Voting Committers
The following are our *voting committers* for significant decisions:
- Amazon Web Services
- Google
- IBM
- Microsoft
- Schlumberger (initial contributor of the core services)
The makeup of the voting committers will evolve over time if the investment model changes.
### Maintainer Committers
To create scale in the project, we also have *Maintainer Committers* who are granted authority
* to submit and approve changes to the master branch.
* vet and approve new contributors (developers) on the project
These Committers are
* Members of the OSDU Forum
* selected according to the [PMC Committer model](https://community.opengroup.org/osdu/governance/project-management-committee/-/wikis/Charter#pmc-project-committer)
* are chosen by other committers based purely on merit
* identified by their maintainer status in the [GitLab System Group](https://community.opengroup.org/groups/osdu/platform/system/-/group_members)
### Contributors
Anyone can ask to contribute to the project as a GitLab "developer" whether they are members of the OSDU Forum or not. However, since developer status in GitLab carries permissions; each developer will have to be vetted and approved by one of the committers.
https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/24[OSDU Data Flow] Governance Model2020-08-28T14:39:51ZStephen Whitley (Invited Expert)[OSDU Data Flow] Governance ModelProject Governance is based on meritocracy and contribution. For Release 3, contribution is largely provided by the following companies (alphabetical) at roughly equal levels of commitment and thus represents the voting block for the pro...Project Governance is based on meritocracy and contribution. For Release 3, contribution is largely provided by the following companies (alphabetical) at roughly equal levels of commitment and thus represents the voting block for the project.
### Voting Committers
The following are our *voting committers* for significant decisions:
- Amazon Web Services
- Google
- IBM
- Microsoft
- Schlumberger (initial contributor of the core services)
The makeup of the voting committers will evolve over time if the investment model changes.
### Maintainer Committers
To create scale in the project, we also have *Maintainer Committers* who are granted authority
* to submit and approve changes to the master branch.
* vet and approve new contributors (developers) on the project
These Committers are
* Members of the OSDU Forum
* selected according to the [PMC Committer model](https://community.opengroup.org/osdu/governance/project-management-committee/-/wikis/Charter#pmc-project-committer)
* are chosen by other committers based purely on merit
* identified by their maintainer status in the [GitLab System Group](https://community.opengroup.org/groups/osdu/platform/system/-/group_members)
### Contributors
Anyone can ask to contribute to the project as a GitLab "developer" whether they are members of the OSDU Forum or not. However, since developer status in GitLab carries permissions; each developer will have to be vetted and approved by one of the committers.https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/25[OSDU Data Flow] Project Consensus Model and Voting2020-08-28T14:40:01ZStephen Whitley (Invited Expert)[OSDU Data Flow] Project Consensus Model and Voting## Consensus model and Voting
Ideally Open Source projects operate on a daily basis with very little formality. However, the diversity of deployment environments and associated impact on technical architecture and sustainability will le...## Consensus model and Voting
Ideally Open Source projects operate on a daily basis with very little formality. However, the diversity of deployment environments and associated impact on technical architecture and sustainability will lead to inevitable conflict. When these arise, we will follow a decision process that is driven by trade-off analysis, resolved through voting and documented via decision records.
## Trade-off analysis
When consensus is not met we will follow a trade-off analysis in which competing ideas are assessed against **project goals** on the principle that the health of the entire system and subsequent and sustainable adoption by customers is more valuable than any single technical decision.
## Voting
- Simple majority (51%) for any decision that does not introduce a breaking change to a commitment or deployed production version of the system
- Super majority (75%) plus escalation to the PMC for any decision that does introduce a breaking change to a commitment or deployed production version of the systemhttps://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/23WAF / Layer 7 Protection2020-06-24T22:08:36ZPaco Hope (AWS)WAF / Layer 7 Protection## Summary
External-facing API endpoints must have some sort of web application firewall / Layer 7 security mechanism in place. External endpoints are those invoked by end users. This doesn't include cloud-native services (e.g., calling...## Summary
External-facing API endpoints must have some sort of web application firewall / Layer 7 security mechanism in place. External endpoints are those invoked by end users. This doesn't include cloud-native services (e.g., calling the cloud's object storage or the cloud's APIs). This does not include data platform APIs that are strictly internal (if such things exist).
### Operator Input
* **Shell** requires thisM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/home/-/issues/36[OSDU Core Service] Definition of Done2020-06-26T12:24:06ZStephen Whitley (Invited Expert)[OSDU Core Service] Definition of Done### Definition of Done
The OSDU Data Platform Core Services has the ambition of continuous delivery and thus has the ambition of continuous done. Because of this, *Done* has a stronger contract at a feature level than it does at a serv...### Definition of Done
The OSDU Data Platform Core Services has the ambition of continuous delivery and thus has the ambition of continuous done. Because of this, *Done* has a stronger contract at a feature level than it does at a service level.
#### Feature Done
A feature is not done until it is running and validated on multiple cloud platforms and ready for consumption. A feature will be:
- implemented
- tested
- Unit
- Compliance
- Integration
- Workflow
- documented
- Internal for code maintenance
- API
- Usage
- deployed
Given the multi-platform deployment requirements, we will consider feature done from a common code perspective once it is demonstrated on two or more cloud platforms. It will be done from the OSDU Forum perspective once available on all supported platforms. This distinction allows OSDU platform deployments to move a slightly different speeds to allow early feedback in a production environment.
##### Service Done
Continuous delivery requires continuous backlog pruning and thus there is no preconceive notion of done. That is a idealized statement that is only true once a system has reached a level of maturity allowing autonomous delivery of services.
For Release 3 we will define a specific scope of work which will be documented both in narrative form as well as issues. See [Release 3 Planning](https://community.opengroup.org/osdu/platform/system/home/-/wikis/Planning/R3).https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/24Document Open Group security methods/responsibilities2020-06-29T16:17:05ZPaco Hope (AWS)Document Open Group security methods/responsibilitiesProvide documentation on the Open Group's responsibilities with respect to security of the code provided.Provide documentation on the Open Group's responsibilities with respect to security of the code provided.M1 - Release 0.1https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/25Entitlement Inheritance2020-09-22T03:40:17ZStephen Whitley (Invited Expert)Entitlement Inheritance## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The OSDU data platform supports data derivatives as part of enrichment steps. These derivatives can be created from multiple p...## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The OSDU data platform supports data derivatives as part of enrichment steps. These derivatives can be created from multiple parent sources; each with a different entitlement model.
``` mermaid
graph LR
P[Enrichment]
S1((Source 1))
S2((Source 2))
S3((Source 3))
style S1 fill:#f9f,stroke:#333,stroke-width:2px
style S2 fill:#9ff,stroke:#333,stroke-width:2px
style S3 fill:#ff9,stroke:#333,stroke-width:2px
R((Result))
S1 --> P
S2 --> P
S3 --> P
P --> R
```
The entitlement of the Result can be a function of the Entitlements of each of the sources, the enrichment that was performed and the organization that created it.
## Decision
For R3 we will explicitly define the entitlements for the Result instead of trying to compute it from its source(s) or traversal of its source(s).
## Rationale
This will ensure that entitlement and performance of evaluation is deterministic.
## Consequences
Derivative data from multiple sources will have to be tagged explicitly.
## When to revisit
After R3 when we have experience and better understand the implications on
- usability
- performance
- maintainability
# Trade-off Analysis - Input to decision
We could try to compute the entitlements based on lineage; however, there is no guarantee that the information required to do this properly would be both available and complete.
This places the burden on establishing access rights for derivative data on the producer of this derivative. This producer would then explicitly access to the new data to other interested parties.
## Alternatives and implications
- **Lineage traversal**: Looking at all ancestor data in real time to evaluate right of access. This would rely on having all the information required to calculate access rights correctly and would have performance implications.
- **Calculated Entitlement**: This would avoid the performance issues of the above; but would still require access to all the information required to assess access rights; including from the producer of the derivative.
## Decision criteria and trade-offs
- Correctness
- Determinism
- Usability
- Performance
## Decision timeline
July 2020M1 - Release 0.1Hrvoje MarkovicHrvoje Markovic