Documentation issueshttps://community.opengroup.org/osdu/documentation/-/issues2020-03-23T16:03:50Zhttps://community.opengroup.org/osdu/documentation/-/issues/64Self-service deployability2020-03-23T16:03:50ZFerris ArgyleSelf-service deployabilityFigure out what self-service deployability looks like between R2 and R3
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
Infosys will help with what deployment at scale looks l...Figure out what self-service deployability looks like between R2 and R3
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
Infosys will help with what deployment at scale looks like, how much is OSDU vs. provider
## Decision
## Rationale
## Consequences
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelineRelease 3ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/documentation/-/issues/63Delivery resource type2020-03-23T18:36:48ZFerris ArgyleDelivery resource typeIntroduce the concept of a delivery/consumption resource type.
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
Key Use Case
* I want the data and this transformation applied ...Introduce the concept of a delivery/consumption resource type.
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
Key Use Case
* I want the data and this transformation applied to it
Need to have a place to map delivery/consumption resource types to parsers/converters
## Decision
## Rationale
## Consequences
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelineRelease 3JoeJoehttps://community.opengroup.org/osdu/documentation/-/issues/58Search - Map aggregate functionality (R3)2020-03-23T19:02:07ZFerris ArgyleSearch - Map aggregate functionality (R3)Redesign for generic case rather than putting in code
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
* Displays wells on map to ensure return result is as soon as possible; ...Redesign for generic case rather than putting in code
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
* Displays wells on map to ensure return result is as soon as possible; restricts # of fields and wells. Allows returning wells based on searches of child data.
* If query is geographic boundary and otherwise minimal, return all wells within that boundary: three fields (well name, unique id, …?)
* This functionality is a popular request, but implemented as special case in the data platform; not clear whether it can be generalized - Alan
## Decision
## Rationale
Supports ISVs required functionality in a way that's not over-fitted for INT and SSIO
## Consequences
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
* Retain R2 injection of Elastic spatial filter
* Redesign for generic case rather than putting in code
## Decision criteria and tradeoffs
## Decision timelineRelease 3ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/documentation/-/issues/57Search convenience queries2020-03-23T18:38:17ZFerris ArgyleSearch convenience queries## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
R1 implemented map aggregate, zoom, and other convenience methods to address INT and SSIO requests.
## Decision
Deprecate plat...## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
R1 implemented map aggregate, zoom, and other convenience methods to address INT and SSIO requests.
## Decision
Deprecate platform support for consumer-specific convenience methods other than map aggregates, eg. zoom
## Rationale
## Consequences
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
* Deprecate platform support for consumer-specific convenience methods other than map aggregates, eg. zoom
* Retain these
## Decision criteria and tradeoffs
* Over-fitting against INT and SSIO
* Client convenience vs. service burden
## Decision timeline
R3Release 3ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/documentation/-/issues/67Ingestion - move createRecord into File Service instead of OSDU DAG2020-03-27T06:51:06ZFerris ArgyleIngestion - move createRecord into File Service instead of OSDU DAGThe intent is to expose this more transparently to ISVs and get them thinking in terms of file record.
* AWS sees an issue with requiring with ISVs having to provide FileIDs, though this is optional
## Status
- [ ] Proposed
- [ ] Tr...The intent is to expose this more transparently to ISVs and get them thinking in terms of file record.
* AWS sees an issue with requiring with ISVs having to provide FileIDs, though this is optional
## Status
- [ ] Proposed
- [ ] Trialing
- [X] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
[OSDU DAG](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-(C)/Design-and-Implementation/Ingestion-and-Enrichment-Detail/R2-Ingestion-Workflow-Orchestration-non-Spike#example-of-osdu-ingestion-dag)
In future, when have Event-based model, may be able to turn Ingestion steps around since assured that file already exists when creating record.
## Decision
## Rationale
## Consequences
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
Moving create record out of the DAG:
* System still should create records for WP/WPC. If this logic is moved into the service, we don't need DAG anymore
Leaving create record in the DAG:
* Future-proof for extending the logic (extra indexers, etc.)
## Decision criteria and tradeoffs
## Decision timelineRelease 2https://community.opengroup.org/osdu/documentation/-/issues/66Ingestion - replace FileID with JSON FileRecord2020-03-27T06:51:16ZFerris ArgyleIngestion - replace FileID with JSON FileRecord* File record will point to a manifest file, or to an opaque file, in which case no further processing required. This is a replacement for the FileID in the “Submit Pre-Loaded”
* It’s then up to the application to create the file record...* File record will point to a manifest file, or to an opaque file, in which case no further processing required. This is a replacement for the FileID in the “Submit Pre-Loaded”
* It’s then up to the application to create the file record and hand it to the ingestion service; will then either create a manifest or create an opaque file record
* The DAG type (DataType) would be passed along with this so knows which type of DAG to run.
* Also add another method called createFileRecord(filename, unsignedURL, datatype, ???) which would return a FileRecord as created in the Storage service.
## Status
- [ ] Proposed
- [ ] Trialing
- [X] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
This pattern more closely matches the ISV bulk loader pattern implemented by EPAM.
## Decision
## Rationale
## Consequences
This would require re-orienting everything else in the Ingestion service which relies on FileID.
* Because of the way the OSDU DAG is implemented, the Ingestion service needs to call the OSDU (aka Manifest) DAG.
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
Replace FileID with JSON FileRecord
* Breaks referential integrity: creates records even if File doesn't exist because wasn't successfully loaded
* Aligns better with with ISV bulk loader test cases implemented by EPAM, simplifying testing
* Assumes ingestions is used only for the file type data
Implement Ingestion service as designed
* Preserves referential integrity
* Doesn't require re-working existing Ingestion services
* Requires greater alignment of test cases with ISV bulk loader test cases implemented by EPAM
## Decision criteria and tradeoffs
## Decision timelineRelease 2JoeJoehttps://community.opengroup.org/osdu/documentation/-/issues/65File service - incorporate Delivery service functionality2020-03-23T18:36:28ZFerris ArgyleFile service - incorporate Delivery service functionality## Status
- [ ] Proposed
- [ ] Trialing
- [X] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The Delivery service was intended as a general-purpose delivery mechanism for OSDU assets; in R2, the only assets being deliver...## Status
- [ ] Proposed
- [ ] Trialing
- [X] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The Delivery service was intended as a general-purpose delivery mechanism for OSDU assets; in R2, the only assets being delivered are files.
The proposal is to incorporate the additional file delivery methods required into the File service, and push a more general Delivery service to R3.
Details
* Add a method to the FileService to retrieve the signedURL for a Manifest file, postponing the need for a Delivery Service until R3 or beyond when processing non-file based data, and simplifying the client application calls
* Originally the unsigned to signedURL conversion was supposed to be in the Delivery service.
## Decision
## Rationale
## Consequences
Implementation Task:
* Gap fit on this use case
* Test according to the definition of done (Write test cases)
* Add a user story to the project ADO
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
Implement Delivery service as designed
* More complex implementation
* More future-proof, gets us closer to R3
Postpone Delivery service to R3, and include file delivery requirements in File Service
* Simpler implementation
* More complex File service which includes methods which aren't required by some services (eg. Ingestion)
## Decision criteria and tradeoffs
## Decision timelineRelease 2JoeJoehttps://community.opengroup.org/osdu/documentation/-/issues/54Ingestion reference data validation2020-03-23T18:35:20ZFerris ArgyleIngestion reference data validation## Status
- [ ] Proposed
- [ ] Trialing
- [x] Under review
- [ ] Approved
- [ ] Retired
### History
This was approved in [January OSDU R2 Decisions review](https://docs.google.com/presentation/d/13muxPb8gnZ8P_mwFvGu0K3rWaqoHD5z5uNx2-Lb...## Status
- [ ] Proposed
- [ ] Trialing
- [x] Under review
- [ ] Approved
- [ ] Retired
### History
This was approved in [January OSDU R2 Decisions review](https://docs.google.com/presentation/d/13muxPb8gnZ8P_mwFvGu0K3rWaqoHD5z5uNx2-LbmN2k/edit#slide=id.g7628e71997_0_358) and has since been revisited.
## Context & Scope
* Currently we are only validating manifests, not reference targets; we should validate that reference data to which an SRN refers exists, as well as other relationships
* How should this work? OpenDES model is to load and cleanup after based on batch scan checks; OSDU philosophy tends to be to only allow valid data in; if they’re doing as part of pre-processing, then capture as long-term backlog issue.
## Decision
## Rationale
## Consequences
Implementation Tasks:
* Gap Fit: Understand how validation is done today (ex, is it well formed JSON)
* Create an ADO story
* Dependency on Schema Service developed by SLB Pune Team
* Ethiraj to update on readiness
* Cross-Cloud Testing and Validation
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
Propose postponing post-R2; this implies that bad data can be loaded.
## Decision criteria and tradeoffs
## Decision timelineRelease 2JoeJoehttps://community.opengroup.org/osdu/documentation/-/issues/52Delivery access tokens2020-03-23T16:08:19ZFerris ArgyleDelivery access tokens## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
Retaining OpenDES functionality in R2 and expanding it
Supporting R1 Behavior
WPC types (Kinds) as it relates to real content ar...## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
Retaining OpenDES functionality in R2 and expanding it
Supporting R1 Behavior
WPC types (Kinds) as it relates to real content are identified by A LIST OF SRNs
## Decision
Return a read access token, and then use the cloud blob API and then use a convenient function API that rides on top of the data platform and carries through the logic
* 1 SRN -- 1 signed URL
* X SRNs --- X signed URLs with transformation
* Metadata - if you want to get to the S3 path, in the future we would get an S3 prefix to the VDS object
## Rationale
## Consequences
Implementation Tasks:
* Gap fit on this use case
* Test according to the definition of done (Write test cases)
* Add a user story to the project ADO
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelineRelease 2JoeJoehttps://community.opengroup.org/osdu/documentation/-/issues/50Entitlements Filter2020-03-23T16:09:14ZFerris ArgyleEntitlements Filter## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
* Retaining OpenDES functionality in R2 and expanding it
* Supporting R1 Behavior
## Decision
Add Entitlements Filter
* Use ...## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
* Retaining OpenDES functionality in R2 and expanding it
* Supporting R1 Behavior
## Decision
Add Entitlements Filter
* Use a default group or a default company policy to grant wider access to meta-data
* Meta-data to be carved out to go into an explicit group and that group would have general access
* Meta-data includes: group-type, master data, reference data
## Rationale
Supports ISVs currently implemented functionality without undue burden on service.
## Consequences
None.
Implementation Tasks:
* Discussion Jan 9 in OSDU Authorization and Entitlements workshop.
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelineRelease 2ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/documentation/-/issues/48OpenDES record/resource id and group semantics (SRN)2020-03-23T18:36:07ZFerris ArgyleOpenDES record/resource id and group semantics (SRN)## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
SRN has record/resource id and group encapsulated, whereas GUID doesn’t - Alan
## Decision
Add semantics to OpenDES for record...## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
SRN has record/resource id and group encapsulated, whereas GUID doesn’t - Alan
## Decision
Add semantics to OpenDES for record/resource id and group
## Rationale
## Consequences
Implementation Tasks:
* Review and formalize the gap fit
* Add a service to map the SRN to the identifier to the ingestion workflow in OpenDES
* Add test
* Define new services to fill gap
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelineRelease 2ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/documentation/-/issues/47Search - Return entities name2020-03-23T18:39:10ZFerris ArgyleSearch - Return entities name## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
## Decision
Retain name for all return entities: these are there in R1, but weren’t in R0 in some cases
## Rationale
Support...## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
## Decision
Retain name for all return entities: these are there in R1, but weren’t in R0 in some cases
## Rationale
Supports ISVs currently implemented functionality.
## Consequences
None.
Implementation Tasks:
* Test according to the definition of done (Write test cases)
* Add a user story to the project ADO
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelineRelease 2ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/documentation/-/issues/94Core services overview page needs to be updated to reflect the latest API doc2023-06-06T20:29:36ZChad LeongCore services overview page needs to be updated to reflect the latest API docCore services overview [page](https://community.opengroup.org/osdu/documentation/-/wikis/Core-Services-Overview) needs to be updated to reflect the latest API doc as per https://community.opengroup.org/osdu/platform/pre-shipping/-/issues...Core services overview [page](https://community.opengroup.org/osdu/documentation/-/wikis/Core-Services-Overview) needs to be updated to reflect the latest API doc as per https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/480 .
We need to update to maintain both static and dynamic page.Debasis ChatterjeeChad LeongDebasis Chatterjeehttps://community.opengroup.org/osdu/documentation/-/issues/93flip order of CRS Convert and Catlog2023-05-22T22:03:25ZBert Kampesflip order of CRS Convert and Catlog@chad, a few comments. Very minor, but you may be able to make the improvements :
[ ] for the CRS services it makes sense to first list Catalog, then Convert. At the same time, the swagger calls it "CRS Convert" and this page "Conver...@chad, a few comments. Very minor, but you may be able to make the improvements :
[ ] for the CRS services it makes sense to first list Catalog, then Convert. At the same time, the swagger calls it "CRS Convert" and this page "Conversion". I would change it to "CRS Convert" on this page. Perhaps you just want to close this issue without action, but if it is not a trouble when you are editing the page, I suggest it is better to flip the rows in the table around.
[ ] For the Geospatial Consumption Zone, it says "code repo". You may want to change this to "Geospatial Consumption Zone" to be consistent with others.
[ ] perhaps more important, the CRS Convert and Catolog services v2 are deprecated for use. The swagger pages/docs don't really seem to indicated that well. Would it be possible to remove v2 or add "(deprecated for use)" after the services? The thing to remember perhaps is that particularly for the CRS Catalog service it is a complete break with the past.https://community.opengroup.org/osdu/documentation/-/issues/90Storage service documentation - Adding information on records insert limit [D...2021-10-26T10:09:35ZNaufal Mohamed NooriStorage service documentation - Adding information on records insert limit [Document Enhancement]Please see the following link:
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md
We dont have any information related to the limit of record insertion using storage service. I ha...Please see the following link:
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md
We dont have any information related to the limit of record insertion using storage service. I have tried from my end and seeing this message:
PUT: BASE_URL/api/storage/v2/records
```
{
"code": 400,
"reason": "Validation error.",
"message": "createOrUpdateRecords.records: Up to 500 records can be ingested at a time"
}
```
It will be beneficial if the documentation shows this limit.
cc: @debasischttps://community.opengroup.org/osdu/documentation/-/issues/89R3M7 - Authentication information2021-08-15T17:52:24ZDebasis ChatterjeeR3M7 - Authentication informationHi Aliaksandr,
I am checking this site. URL includes R2, but I presume that it is valid for R3. Right?
https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/GCP/GCP-Pre-Prod-Onboarding-Documentation
I also have an e...Hi Aliaksandr,
I am checking this site. URL includes R2, but I presume that it is valid for R3. Right?
https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/GCP/GCP-Pre-Prod-Onboarding-Documentation
I also have an earlier email from Maksimelyan.
My questions –
1. Username – presumably use my existing username (debasis_chatterjee@osdu-gcp.go3-nrg.projects.epam.com) that I used prior to M7. Right?
2. Instruction says – client_secret will be sent by separate email. Should I use what was provided to me prior to M7?
3. Where is clear step by step instruction as to how one should authenticate in Postman?
4. We also need separate Smoke test collection for running standard Core services API (ex: Search, Legal, Entitlements..).
5. Can you confirm if my username is privileged to perform Seismic DDMS steps?
Please check and advise.
Thank you
Debasis
cc - @maksimelyan_tamashevichhttps://community.opengroup.org/osdu/documentation/-/issues/88Show "permissions required" for all services in documentation in consistent m...2021-06-28T21:11:49ZDebasis ChatterjeeShow "permissions required" for all services in documentation in consistent mannerExample from Search service -
https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/tutorial/SearchService.md
Scroll at the very end. It shows you that minimum permissions required is "users.osdu.viewer...Example from Search service -
https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/tutorial/SearchService.md
Scroll at the very end. It shows you that minimum permissions required is "users.osdu.viewers".
Else, when someone gets 401 (Unauthorized) error from a service, he/she does not have much clue to figure out which role/group is missed from his/her user access credentials.
GET {{osduonaws_base_url}}/api/entitlements/v1/groupshttps://community.opengroup.org/osdu/documentation/-/issues/87R3 - Unit Service documentation2021-06-29T05:32:00ZDebasis ChatterjeeR3 - Unit Service documentationCan you please provide some high level overview, by using simple example of unit conversion?
Such as feet -> meters.
Temperature is also good candidate as this will show sue of multiplier and added value.
Such as from centigrade and fa...Can you please provide some high level overview, by using simple example of unit conversion?
Such as feet -> meters.
Temperature is also good candidate as this will show sue of multiplier and added value.
Such as from centigrade and farenheit.
As far as I see now, the documentation lists the API calls with brief explanation of each.https://community.opengroup.org/osdu/documentation/-/issues/86Allowed kind in storage create storage api2021-06-28T21:10:29ZRupali KombadeAllowed kind in storage create storage apiWhere can I get information about allowed kind in storage create schema API. if I want to specify 'array of objects' how to specify it?
Consider the data format is as below:
data {
name: "area name",
country: "xyz",
wells:[
name: "we...Where can I get information about allowed kind in storage create schema API. if I want to specify 'array of objects' how to specify it?
Consider the data format is as below:
data {
name: "area name",
country: "xyz",
wells:[
name: "well1",
type: "planning",
coordinates: {
latitute:12345,
longitude: 1235
}
]
}
to make above data searchable I need to connect it with storage schema. When I tried to create schema using below service, its giving me error that - **Schema item 'wells' has an invalid data type 'object'**, cay you please suggest how to add array or list in schema kind for wells array?
api/storage/v2/schemas
{
"kind": "common:wks:welldata:1.0.0",
"schema": [
{
"path": "name",
"kind": "string"
},
{
"path": "country",
"kind": "string"
},
{
"path": "wells",
"kind": "object"
}
]
}https://community.opengroup.org/osdu/documentation/-/issues/85Storage service documentation - using the term "dataset-name" for explanation...2021-06-28T21:08:24ZDebasis ChatterjeeStorage service documentation - using the term "dataset-name" for explanation of "kind"Please see this link
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md
Explains a few basic concepts like "id", "kind".
**kind**: *(mandatory)* Kind of data being ingested. Must...Please see this link
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md
Explains a few basic concepts like "id", "kind".
**kind**: *(mandatory)* Kind of data being ingested. Must follow the naming convention: `{Data-Partition-Id}:{dataset-name}:{record-type}:{version}`.
The problem is using the term "**dataset**" in second component of "kind".
As you know, right now the term "dataset" is used in a different way (Data Definition such as for File.Generic, File.Collection and also new Services).
Can you plan to make suitable corrections?
Sample "kind" from json files in wiki of Data Loading team.
`"kind": "osdu:wks:work-product-component--WellLog:1.0.0",`
Thank you