Documentation issueshttps://community.opengroup.org/osdu/documentation/-/issues2020-03-27T06:51:06Zhttps://community.opengroup.org/osdu/documentation/-/issues/67Ingestion - move createRecord into File Service instead of OSDU DAG2020-03-27T06:51:06ZFerris ArgyleIngestion - move createRecord into File Service instead of OSDU DAGThe intent is to expose this more transparently to ISVs and get them thinking in terms of file record.
* AWS sees an issue with requiring with ISVs having to provide FileIDs, though this is optional
## Status
- [ ] Proposed
- [ ] Tr...The intent is to expose this more transparently to ISVs and get them thinking in terms of file record.
* AWS sees an issue with requiring with ISVs having to provide FileIDs, though this is optional
## Status
- [ ] Proposed
- [ ] Trialing
- [X] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
[OSDU DAG](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-(C)/Design-and-Implementation/Ingestion-and-Enrichment-Detail/R2-Ingestion-Workflow-Orchestration-non-Spike#example-of-osdu-ingestion-dag)
In future, when have Event-based model, may be able to turn Ingestion steps around since assured that file already exists when creating record.
## Decision
## Rationale
## Consequences
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
Moving create record out of the DAG:
* System still should create records for WP/WPC. If this logic is moved into the service, we don't need DAG anymore
Leaving create record in the DAG:
* Future-proof for extending the logic (extra indexers, etc.)
## Decision criteria and tradeoffs
## Decision timelineRelease 2https://community.opengroup.org/osdu/documentation/-/issues/66Ingestion - replace FileID with JSON FileRecord2020-03-27T06:51:16ZFerris ArgyleIngestion - replace FileID with JSON FileRecord* File record will point to a manifest file, or to an opaque file, in which case no further processing required. This is a replacement for the FileID in the “Submit Pre-Loaded”
* It’s then up to the application to create the file record...* File record will point to a manifest file, or to an opaque file, in which case no further processing required. This is a replacement for the FileID in the “Submit Pre-Loaded”
* It’s then up to the application to create the file record and hand it to the ingestion service; will then either create a manifest or create an opaque file record
* The DAG type (DataType) would be passed along with this so knows which type of DAG to run.
* Also add another method called createFileRecord(filename, unsignedURL, datatype, ???) which would return a FileRecord as created in the Storage service.
## Status
- [ ] Proposed
- [ ] Trialing
- [X] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
This pattern more closely matches the ISV bulk loader pattern implemented by EPAM.
## Decision
## Rationale
## Consequences
This would require re-orienting everything else in the Ingestion service which relies on FileID.
* Because of the way the OSDU DAG is implemented, the Ingestion service needs to call the OSDU (aka Manifest) DAG.
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
Replace FileID with JSON FileRecord
* Breaks referential integrity: creates records even if File doesn't exist because wasn't successfully loaded
* Aligns better with with ISV bulk loader test cases implemented by EPAM, simplifying testing
* Assumes ingestions is used only for the file type data
Implement Ingestion service as designed
* Preserves referential integrity
* Doesn't require re-working existing Ingestion services
* Requires greater alignment of test cases with ISV bulk loader test cases implemented by EPAM
## Decision criteria and tradeoffs
## Decision timelineRelease 2JoeJoehttps://community.opengroup.org/osdu/documentation/-/issues/65File service - incorporate Delivery service functionality2020-03-23T18:36:28ZFerris ArgyleFile service - incorporate Delivery service functionality## Status
- [ ] Proposed
- [ ] Trialing
- [X] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The Delivery service was intended as a general-purpose delivery mechanism for OSDU assets; in R2, the only assets being deliver...## Status
- [ ] Proposed
- [ ] Trialing
- [X] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The Delivery service was intended as a general-purpose delivery mechanism for OSDU assets; in R2, the only assets being delivered are files.
The proposal is to incorporate the additional file delivery methods required into the File service, and push a more general Delivery service to R3.
Details
* Add a method to the FileService to retrieve the signedURL for a Manifest file, postponing the need for a Delivery Service until R3 or beyond when processing non-file based data, and simplifying the client application calls
* Originally the unsigned to signedURL conversion was supposed to be in the Delivery service.
## Decision
## Rationale
## Consequences
Implementation Task:
* Gap fit on this use case
* Test according to the definition of done (Write test cases)
* Add a user story to the project ADO
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
Implement Delivery service as designed
* More complex implementation
* More future-proof, gets us closer to R3
Postpone Delivery service to R3, and include file delivery requirements in File Service
* Simpler implementation
* More complex File service which includes methods which aren't required by some services (eg. Ingestion)
## Decision criteria and tradeoffs
## Decision timelineRelease 2JoeJoehttps://community.opengroup.org/osdu/documentation/-/issues/54Ingestion reference data validation2020-03-23T18:35:20ZFerris ArgyleIngestion reference data validation## Status
- [ ] Proposed
- [ ] Trialing
- [x] Under review
- [ ] Approved
- [ ] Retired
### History
This was approved in [January OSDU R2 Decisions review](https://docs.google.com/presentation/d/13muxPb8gnZ8P_mwFvGu0K3rWaqoHD5z5uNx2-Lb...## Status
- [ ] Proposed
- [ ] Trialing
- [x] Under review
- [ ] Approved
- [ ] Retired
### History
This was approved in [January OSDU R2 Decisions review](https://docs.google.com/presentation/d/13muxPb8gnZ8P_mwFvGu0K3rWaqoHD5z5uNx2-LbmN2k/edit#slide=id.g7628e71997_0_358) and has since been revisited.
## Context & Scope
* Currently we are only validating manifests, not reference targets; we should validate that reference data to which an SRN refers exists, as well as other relationships
* How should this work? OpenDES model is to load and cleanup after based on batch scan checks; OSDU philosophy tends to be to only allow valid data in; if they’re doing as part of pre-processing, then capture as long-term backlog issue.
## Decision
## Rationale
## Consequences
Implementation Tasks:
* Gap Fit: Understand how validation is done today (ex, is it well formed JSON)
* Create an ADO story
* Dependency on Schema Service developed by SLB Pune Team
* Ethiraj to update on readiness
* Cross-Cloud Testing and Validation
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
Propose postponing post-R2; this implies that bad data can be loaded.
## Decision criteria and tradeoffs
## Decision timelineRelease 2JoeJoehttps://community.opengroup.org/osdu/documentation/-/issues/52Delivery access tokens2020-03-23T16:08:19ZFerris ArgyleDelivery access tokens## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
Retaining OpenDES functionality in R2 and expanding it
Supporting R1 Behavior
WPC types (Kinds) as it relates to real content ar...## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
Retaining OpenDES functionality in R2 and expanding it
Supporting R1 Behavior
WPC types (Kinds) as it relates to real content are identified by A LIST OF SRNs
## Decision
Return a read access token, and then use the cloud blob API and then use a convenient function API that rides on top of the data platform and carries through the logic
* 1 SRN -- 1 signed URL
* X SRNs --- X signed URLs with transformation
* Metadata - if you want to get to the S3 path, in the future we would get an S3 prefix to the VDS object
## Rationale
## Consequences
Implementation Tasks:
* Gap fit on this use case
* Test according to the definition of done (Write test cases)
* Add a user story to the project ADO
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelineRelease 2JoeJoehttps://community.opengroup.org/osdu/documentation/-/issues/50Entitlements Filter2020-03-23T16:09:14ZFerris ArgyleEntitlements Filter## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
* Retaining OpenDES functionality in R2 and expanding it
* Supporting R1 Behavior
## Decision
Add Entitlements Filter
* Use ...## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
* Retaining OpenDES functionality in R2 and expanding it
* Supporting R1 Behavior
## Decision
Add Entitlements Filter
* Use a default group or a default company policy to grant wider access to meta-data
* Meta-data to be carved out to go into an explicit group and that group would have general access
* Meta-data includes: group-type, master data, reference data
## Rationale
Supports ISVs currently implemented functionality without undue burden on service.
## Consequences
None.
Implementation Tasks:
* Discussion Jan 9 in OSDU Authorization and Entitlements workshop.
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelineRelease 2ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/documentation/-/issues/48OpenDES record/resource id and group semantics (SRN)2020-03-23T18:36:07ZFerris ArgyleOpenDES record/resource id and group semantics (SRN)## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
SRN has record/resource id and group encapsulated, whereas GUID doesn’t - Alan
## Decision
Add semantics to OpenDES for record...## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
SRN has record/resource id and group encapsulated, whereas GUID doesn’t - Alan
## Decision
Add semantics to OpenDES for record/resource id and group
## Rationale
## Consequences
Implementation Tasks:
* Review and formalize the gap fit
* Add a service to map the SRN to the identifier to the ingestion workflow in OpenDES
* Add test
* Define new services to fill gap
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelineRelease 2ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/documentation/-/issues/47Search - Return entities name2020-03-23T18:39:10ZFerris ArgyleSearch - Return entities name## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
## Decision
Retain name for all return entities: these are there in R1, but weren’t in R0 in some cases
## Rationale
Support...## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
## Decision
Retain name for all return entities: these are there in R1, but weren’t in R0 in some cases
## Rationale
Supports ISVs currently implemented functionality.
## Consequences
None.
Implementation Tasks:
* Test according to the definition of done (Write test cases)
* Add a user story to the project ADO
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelineRelease 2ethiraj krishnamanaiduethiraj krishnamanaidu