Documentation issueshttps://community.opengroup.org/osdu/documentation/-/issues2021-06-28T21:10:29Zhttps://community.opengroup.org/osdu/documentation/-/issues/86Allowed kind in storage create storage api2021-06-28T21:10:29ZRupali KombadeAllowed kind in storage create storage apiWhere can I get information about allowed kind in storage create schema API. if I want to specify 'array of objects' how to specify it?
Consider the data format is as below:
data {
name: "area name",
country: "xyz",
wells:[
name: "we...Where can I get information about allowed kind in storage create schema API. if I want to specify 'array of objects' how to specify it?
Consider the data format is as below:
data {
name: "area name",
country: "xyz",
wells:[
name: "well1",
type: "planning",
coordinates: {
latitute:12345,
longitude: 1235
}
]
}
to make above data searchable I need to connect it with storage schema. When I tried to create schema using below service, its giving me error that - **Schema item 'wells' has an invalid data type 'object'**, cay you please suggest how to add array or list in schema kind for wells array?
api/storage/v2/schemas
{
"kind": "common:wks:welldata:1.0.0",
"schema": [
{
"path": "name",
"kind": "string"
},
{
"path": "country",
"kind": "string"
},
{
"path": "wells",
"kind": "object"
}
]
}https://community.opengroup.org/osdu/documentation/-/issues/85Storage service documentation - using the term "dataset-name" for explanation...2021-06-28T21:08:24ZDebasis ChatterjeeStorage service documentation - using the term "dataset-name" for explanation of "kind"Please see this link
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md
Explains a few basic concepts like "id", "kind".
**kind**: *(mandatory)* Kind of data being ingested. Must...Please see this link
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md
Explains a few basic concepts like "id", "kind".
**kind**: *(mandatory)* Kind of data being ingested. Must follow the naming convention: `{Data-Partition-Id}:{dataset-name}:{record-type}:{version}`.
The problem is using the term "**dataset**" in second component of "kind".
As you know, right now the term "dataset" is used in a different way (Data Definition such as for File.Generic, File.Collection and also new Services).
Can you plan to make suitable corrections?
Sample "kind" from json files in wiki of Data Loading team.
`"kind": "osdu:wks:work-product-component--WellLog:1.0.0",`
Thank youhttps://community.opengroup.org/osdu/documentation/-/issues/84Ingestion Workflow - provide an option to run integrity test on previous load...2021-06-28T22:14:54ZDebasis ChatterjeeIngestion Workflow - provide an option to run integrity test on previous loaded data (with integrity test option disabled)This option may allow the Data Manager/Loader to run integrity test on previous loaded data (a set of records), and result may show which ones are still invalid.
For the valid ones (ex: reference check is now valid as suitable "referen...This option may allow the Data Manager/Loader to run integrity test on previous loaded data (a set of records), and result may show which ones are still invalid.
For the valid ones (ex: reference check is now valid as suitable "referenced data" now exists), the system property can be reset as the record is no longer invalid from integrity perspective.
Please refer to this issue.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/203https://community.opengroup.org/osdu/documentation/-/issues/83Ingestion workflow - provide an option for "Dry run" simply for validation ch...2021-06-28T22:13:19ZDebasis ChatterjeeIngestion workflow - provide an option for "Dry run" simply for validation checksThis option can be helpful to alert the Data Loader of issues in input data, simply from integrity tests.
Then he/she can take corrective action up front and then attempt to actually load the large data set.This option can be helpful to alert the Data Loader of issues in input data, simply from integrity tests.
Then he/she can take corrective action up front and then attempt to actually load the large data set.https://community.opengroup.org/osdu/documentation/-/issues/82Ingestion workflow - provide option to enable, disable validation checks (ref...2021-03-19T11:38:38ZDebasis ChatterjeeIngestion workflow - provide option to enable, disable validation checks (referenced information)Starting with recent version of "Ingestion Workflow", we see that integrity checks are enabled. This is very useful.
However, please provide option of setting true/false for "integrity checks" to the end user (Data Loader) who may or ma...Starting with recent version of "Ingestion Workflow", we see that integrity checks are enabled. This is very useful.
However, please provide option of setting true/false for "integrity checks" to the end user (Data Loader) who may or may not have required skills to tweak python code inside the DAG. Default can be left as "True".
Excerpt from Airflow Log (GCP environment) is shown below for quick reference -
```
[2021-03-18 17:45:08,504] {base_task_runner.py:113} INFO - Job 27967: Subtask provide_manifest_integrity_task [2021-03-18 17:45:08,503]
{validate_referential_integrity.py:156} DEBUG - Extracted reference ids:
['osdu:reference-data--AliasNameType:WELL_NAME',
'osdu:reference-data--VerticalMeasurementPath:DEPTH_DATUM_ELEV',
'osdu:reference-data--ResourceSecurityClassification:Public',
'osdu:reference-data--FacilityEventType:SPUD_DATE',
'osdu:reference-data--FacilityType:WELLBLABLA',
'osdu:master-data--Organisation:HESS']
```
In this example, all checks failed as the environment lacked standard Reference values at the time of this run. Else, I would only expect one reference check to fail (FacilityType = "WELLBLABLA" instead of "WELL").
```
[2021-03-18 17:45:44,405] {base_task_runner.py:113} INFO - Job 27967: Subtask provide_manifest_integrity_task [2021-03-18 17:45:44,405]
{validate_referential_integrity.py:177} WARNING - The next ids are absent in the system:
['osdu:reference-data--FacilityType:WELLBLABLA',
'osdu:reference-data--FacilityEventType:SPUD_DATE',
'osdu:reference-data--ResourceSecurityClassification:Public',
'osdu:reference-data--VerticalMeasurementPath:DEPTH_DATUM_ELEV',
'osdu:master-data--Organisation:HESS',
'osdu:reference-data--AliasNameType:WELL_NAME']
[2021-03-18 17:45:44,413] {base_task_runner.py:113} INFO - Job 27967: Subtask provide_manifest_integrity_task [2021-03-18 17:45:44,411]
{validate_referential_integrity.py:231} WARNING - Resource with kind odesprod:wks:master-data--Well:1.0.0 was rejected
```https://community.opengroup.org/osdu/documentation/-/issues/81R3 Entitlement Service - Update content to OSDU context, fix broken link2021-06-28T22:10:58ZDebasis ChatterjeeR3 Entitlement Service - Update content to OSDU context, fix broken linkThe content seems very Schlumberger-oriented. Please change to general context, and in line with upcoming Admin UI.
The link below is broken.
`All the API's can be found [here](/osdu/documentation/-/wikis/solutions/dataecosystem/apis)`...The content seems very Schlumberger-oriented. Please change to general context, and in line with upcoming Admin UI.
The link below is broken.
`All the API's can be found [here](/osdu/documentation/-/wikis/solutions/dataecosystem/apis)`
Also provide link to Swagger documentation as in other services.
Also, I did not see clear documentation about any service to "get details of a group".https://community.opengroup.org/osdu/documentation/-/issues/80Capture "Processing application" as a new field over and above usual audit tr...2021-07-06T15:30:37ZDebasis ChatterjeeCapture "Processing application" as a new field over and above usual audit trail fields (created-by, created-on-date)As we start to see many alternatives for loading data such as "Ingestion workflow", "CSV Parser", "WITSML parser", "LAS parser", this will be useful addition. First we need suitable field common for Work-product-component, Master data, R...As we start to see many alternatives for loading data such as "Ingestion workflow", "CSV Parser", "WITSML parser", "LAS parser", this will be useful addition. First we need suitable field common for Work-product-component, Master data, Reference data.
This information will help us to troubleshoot if we see any problem with ingested data.
Copying to @alan.henson and @james.pipe for awareness.https://community.opengroup.org/osdu/documentation/-/issues/79R3 API documentation - Storage service2021-06-28T21:00:52ZDebasis ChatterjeeR3 API documentation - Storage service![R3-core-Storage-service](/uploads/771723f9250969752e7386e773b01e1c/R3-core-Storage-service.PNG)
Start with this page -
`https://community.opengroup.org/osdu/documentation/-/wikis/Core-Services-Overview`
Click on "How to" for "Storage...![R3-core-Storage-service](/uploads/771723f9250969752e7386e773b01e1c/R3-core-Storage-service.PNG)
Start with this page -
`https://community.opengroup.org/osdu/documentation/-/wikis/Core-Services-Overview`
Click on "How to" for "Storage service". Takes us to
`https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md`
Please provide some recommendation about "addition of record into Document store" - as to end users may now leverage "Ingestion workflow" (DAG based approach) for creation of Master data, Reference data and also the chain wp-wpc-dataset/file.
I refer to this -
`https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/blob/master/README.md#post-v1workflowworkflow_nameworkflowrun`
Also provide similar clarification regarding apparent overlap between "Schema" features documented here with the new Schema service (yet to be documented properly).
In section on "Partition", this mentions DELFI. And also here is an opportunity to provide a good lead to the new "Partition service".https://community.opengroup.org/osdu/documentation/-/issues/78R3 Ingestion - alternatives2021-06-28T21:40:35ZDebasis ChatterjeeR3 Ingestion - alternativesAs per recent discussion with @alan.henson , I would like to request a clear statement in this page stating which alternative is recommended and will be fully supported by 4 CSPs at the time of R3 release.
![R3-Ingestion-alternatives](/...As per recent discussion with @alan.henson , I would like to request a clear statement in this page stating which alternative is recommended and will be fully supported by 4 CSPs at the time of R3 release.
![R3-Ingestion-alternatives](/uploads/6149088d7608bb90e404abe47a3f2482/R3-Ingestion-alternatives.PNG)
The following wiki page mentioned two entries under "Ingestion".
`https://community.opengroup.org/osdu/documentation/-/wikis/Core-Services-Overview`
Thank youhttps://community.opengroup.org/osdu/documentation/-/issues/77R3 File service - "Notes" link is broken2021-06-28T20:59:20ZDebasis ChatterjeeR3 File service - "Notes" link is broken![Core-service-File](/uploads/1d5ed33d77b8300ad9f1ef8caabeed0e/Core-service-File.PNG)
Please check this link and fix as needed.
Thank you![Core-service-File](/uploads/1d5ed33d77b8300ad9f1ef8caabeed0e/Core-service-File.PNG)
Please check this link and fix as needed.
Thank youhttps://community.opengroup.org/osdu/documentation/-/issues/76R3 Search Service - typo error for message when resource does not exist2021-06-29T15:19:44ZDebasis ChatterjeeR3 Search Service - typo error for message when resource does not existSee this message -
POST {{base_url}}/api/search/v2/query
Body
```
{
"kind": "opendes:wks:master-data--wellbore:1.0.0",
"query": "*"
}
```
Response -
```
{
"code": 404,
"reason": "Not Found",
"message": "Resource yo...See this message -
POST {{base_url}}/api/search/v2/query
Body
```
{
"kind": "opendes:wks:master-data--wellbore:1.0.0",
"query": "*"
}
```
Response -
```
{
"code": 404,
"reason": "Not Found",
"message": "Resource you are trying to find does not exists"
}
```
Also note that the error message is the same when user mistypes entity name (ex: Wellbore1 instead of Wellbore) and when entity is empty i.e. no data loaded yet.
I have earlier verified using Schema service that this entity exists -
```
{
"schemaIdentity": {
"authority": "opendes",
"source": "wks",
"entityType": "master-data--Wellbore",
"schemaVersionMajor": 1,
"schemaVersionMinor": 0,
"schemaVersionPatch": 0,
"id": "opendes:wks:master-data--Wellbore:1.0.0"
},
"createdBy": "e09bb2a5-337f-48a7-8f11-13f939b8a6ee",
"dateCreated": "2021-02-18T03:53:57.738+00:00",
"status": "DEVELOPMENT",
"scope": "SHARED"
},
```https://community.opengroup.org/osdu/documentation/-/issues/75R3 Storage service - Documentation - explanation of "kind"2021-02-26T22:38:17ZDebasis ChatterjeeR3 Storage service - Documentation - explanation of "kind"Please check the following.
`https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#create-schema`
**kind**: *(mandatory)* Kind of data being ingested. Must follow the naming conventi...Please check the following.
`https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#create-schema`
**kind**: *(mandatory)* Kind of data being ingested. Must follow the naming convention: `{Data-Partition-Id}:{dataset-name}:{record-type}:{version}`.
Suggest use "schema" rather than "dataset" for the second part. In R3, the term "Dataset" is used separately to refer to File, Collection of files etc.
See example of "kind" from GCP R3-M3 environment.
> "kind": "odesprod:wks:work-product-component--WellboreMarkerSet:1.0.0",https://community.opengroup.org/osdu/documentation/-/issues/74R3 Schema service - Documentation link is broken2021-02-26T22:31:16ZDebasis ChatterjeeR3 Schema service - Documentation link is brokenStarted from the following link.
`https://community.opengroup.org/osdu/documentation/-/wikis/Core-Services-Overview`
Clicked on "Notes" in Documentation column. The link is broken.
Please check.
And while on the subject, please menti...Started from the following link.
`https://community.opengroup.org/osdu/documentation/-/wikis/Core-Services-Overview`
Clicked on "Notes" in Documentation column. The link is broken.
Please check.
And while on the subject, please mention overlap (if any) with some functions of Storage service.
At a first glance, it appears that some of the functions from Storage may have been deprecated?
Perhaps a clear explanation of the difference/relationship between schema (from Schema-service) and storage schema (from Storage service)?
`POST https://{{domain}}/api/schema-service/v1/schema`
and
`POST https://{{domain}}/api/storage/v2/schemas`
`https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md`
Example -
`GET {{schema_api_url}}/schema?limit=10000`
and
`GET /api/storage/v2/query/kinds`https://community.opengroup.org/osdu/documentation/-/issues/73Create/Update Storage Record API response should include the Record version t...2021-06-16T22:18:59Zashley kelhamCreate/Update Storage Record API response should include the Record version to prevent possible race conditionCurrently this API returns the following response body
```
{
"recordCount": 0,
"recordIds": [
"string"
],
"skippedRecordIds": [
"string"
]
}
```
The client then needs to read the Record to get the **version** of the Re...Currently this API returns the following response body
```
{
"recordCount": 0,
"recordIds": [
"string"
],
"skippedRecordIds": [
"string"
]
}
```
The client then needs to read the Record to get the **version** of the Record they created/updated. In this time another client could have created a new version of the Record and so the original client will not retrieve the correct version they created.
If the response body of the API included the version of the ID then the client knows what version of the record they created. The suggestion is to add a new property to the response body that includes this
```
{
"recordCount": 0,
"recordIds": [
"string"
],
"recordIdVersions": [
"string"
],
"skippedRecordIds": [
"string"
]
}
```
where each recordIdVersions value will be in the format
```
<id>:<version>
e.g.
"common:welldb:123456:2300942424909"
```https://community.opengroup.org/osdu/documentation/-/issues/72Get wells in polygon | Spatial Search2020-06-26T02:05:07ZKushalGet wells in polygon | Spatial SearchHello Everyone ,
Get wells in polygon has permission issues
{
"code": 401,
"reason": "Access denied",
"message": "The user is not authorized to perform this action"
}
I get the following both in GCP , Azure haven't tried ...Hello Everyone ,
Get wells in polygon has permission issues
{
"code": 401,
"reason": "Access denied",
"message": "The user is not authorized to perform this action"
}
I get the following both in GCP , Azure haven't tried in AWS
Post parameters used :
{
"kind": "opendes:osdu:seismictracedata-wpc:0.2.0",
"spatialFilter": {
"field": "data.Data.ExtensionProperties.locationWGS84",
"byGeoPolygon": {
"points": [
{
"longitude": 1.813767236,
"latitude": 58.42946998
},
{
"longitude": 1.949673713,
"latitude": 58.4041567
},
{
"longitude": 1.978324105,
"latitude": 58.45614207
},
{
"longitude": 1.84227351,
"latitude": 58.48147214
},
{
"longitude": 1.813767236,
"latitude": 58.42946998
}
]
}
},
"offset": 0,
"limit": 30
}
{
"kind": "opendes:osdu:*:0.2.1",
"spatialFilter": {
"field": "data.GeoLocation",
"byGeoPolygon": {
"points": [
{
"longitude": 5.1580810546875,
"latitude": 52.859180945520826
},
{
"longitude": 4.9493408203125,
"latitude": 52.75956761546834
},
{
"longitude": 5.064697265625,
"latitude": 52.579688026538726
},
{
"longitude": 5.372314453125,
"latitude": 52.68970242806752
},
{
"longitude": 5.1580810546875,
"latitude": 52.859180945520826
}
]
}
},
"offset": 0,
"limit": 30
}
(borrowed the example from https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/OSDU-Query-Syntax#spatial-queries , please use points and not coordinates)
@dkodeih @danielscholl @ethirajhttps://community.opengroup.org/osdu/documentation/-/issues/71Incorrect examples for geospatial queries | https://community.opengroup.org/o...2021-06-16T22:19:00ZKushalIncorrect examples for geospatial queries | https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/OSDU-Query-Syntax#spatial-querieshttps://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/OSDU-Query-Syntax#spatial-queries
Should be point/points
and
not cordinate/coordinates
@dmitry-kniazevhttps://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/OSDU-Query-Syntax#spatial-queries
Should be point/points
and
not cordinate/coordinates
@dmitry-kniazevhttps://community.opengroup.org/osdu/documentation/-/issues/70Create OSDU GCP Security Manual2020-08-21T00:02:39ZElizaveta Zeldina (EPAM)Create OSDU GCP Security ManualCreate OSDU GCP Security Manual and submit it to https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/GCPCreate OSDU GCP Security Manual and submit it to https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/GCPDmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/documentation/-/issues/68ISV Bulk Loader Scripts - leverage Ingestion service2020-05-19T12:38:32ZFerris ArgyleISV Bulk Loader Scripts - leverage Ingestion serviceThis will ensure that the Ingestion service is able to pass the tests which are based on the Loader.
Use
* Ingestion SDK
* File id as part of the manifest
Keep the rest of the script processing
## Status
- [ ] Proposed
- [ ] Trialing...This will ensure that the Ingestion service is able to pass the tests which are based on the Loader.
Use
* Ingestion SDK
* File id as part of the manifest
Keep the rest of the script processing
## Status
- [ ] Proposed
- [ ] Trialing
- [X] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The Ingestion test cases are based on the ISV bulk loader test cases; the Loader scripts are changed frequently, which affects the test results, and the Ingestion service will be perpetually out of sync.
## Decision
## Rationale
## Consequences
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelineRelease 2Dmitry KniazevDmitry Kniazevhttps://community.opengroup.org/osdu/documentation/-/issues/67Ingestion - move createRecord into File Service instead of OSDU DAG2020-03-27T06:51:06ZFerris ArgyleIngestion - move createRecord into File Service instead of OSDU DAGThe intent is to expose this more transparently to ISVs and get them thinking in terms of file record.
* AWS sees an issue with requiring with ISVs having to provide FileIDs, though this is optional
## Status
- [ ] Proposed
- [ ] Tr...The intent is to expose this more transparently to ISVs and get them thinking in terms of file record.
* AWS sees an issue with requiring with ISVs having to provide FileIDs, though this is optional
## Status
- [ ] Proposed
- [ ] Trialing
- [X] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
[OSDU DAG](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-(C)/Design-and-Implementation/Ingestion-and-Enrichment-Detail/R2-Ingestion-Workflow-Orchestration-non-Spike#example-of-osdu-ingestion-dag)
In future, when have Event-based model, may be able to turn Ingestion steps around since assured that file already exists when creating record.
## Decision
## Rationale
## Consequences
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
Moving create record out of the DAG:
* System still should create records for WP/WPC. If this logic is moved into the service, we don't need DAG anymore
Leaving create record in the DAG:
* Future-proof for extending the logic (extra indexers, etc.)
## Decision criteria and tradeoffs
## Decision timelineRelease 2https://community.opengroup.org/osdu/documentation/-/issues/66Ingestion - replace FileID with JSON FileRecord2020-03-27T06:51:16ZFerris ArgyleIngestion - replace FileID with JSON FileRecord* File record will point to a manifest file, or to an opaque file, in which case no further processing required. This is a replacement for the FileID in the “Submit Pre-Loaded”
* It’s then up to the application to create the file record...* File record will point to a manifest file, or to an opaque file, in which case no further processing required. This is a replacement for the FileID in the “Submit Pre-Loaded”
* It’s then up to the application to create the file record and hand it to the ingestion service; will then either create a manifest or create an opaque file record
* The DAG type (DataType) would be passed along with this so knows which type of DAG to run.
* Also add another method called createFileRecord(filename, unsignedURL, datatype, ???) which would return a FileRecord as created in the Storage service.
## Status
- [ ] Proposed
- [ ] Trialing
- [X] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
This pattern more closely matches the ISV bulk loader pattern implemented by EPAM.
## Decision
## Rationale
## Consequences
This would require re-orienting everything else in the Ingestion service which relies on FileID.
* Because of the way the OSDU DAG is implemented, the Ingestion service needs to call the OSDU (aka Manifest) DAG.
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
Replace FileID with JSON FileRecord
* Breaks referential integrity: creates records even if File doesn't exist because wasn't successfully loaded
* Aligns better with with ISV bulk loader test cases implemented by EPAM, simplifying testing
* Assumes ingestions is used only for the file type data
Implement Ingestion service as designed
* Preserves referential integrity
* Doesn't require re-working existing Ingestion services
* Requires greater alignment of test cases with ISV bulk loader test cases implemented by EPAM
## Decision criteria and tradeoffs
## Decision timelineRelease 2JoeJoe