OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2021-06-14T12:19:20Zhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/171Adding new in partition service for Workflow Ingestion Service (storage accont)2021-06-14T12:19:20ZAalekh JainAdding new in partition service for Workflow Ingestion Service (storage accont)In order to support multi partition for storage account in workflow ingestion service, we need to add the following properties to partition service -
1. `ingest-storage-account-name`
2. `ingest-storage-account-key`
MR is raised here: ...In order to support multi partition for storage account in workflow ingestion service, we need to add the following properties to partition service -
1. `ingest-storage-account-name`
2. `ingest-storage-account-key`
MR is raised here: !317
Change in core lib azure are introduced here: https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/110M7 - Release 0.10https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/173opendes hardcoded in http scripts2021-06-23T09:23:50ZKishore Battulaopendes hardcoded in http scriptsopendes is hardcoded in the http scripts even though a variable exists at the top of the scripts. This is resulting unexpected behavior when changing the data-partition-id.
One of the hardcoded locations: https://community.opengroup.org...opendes is hardcoded in the http scripts even though a variable exists at the top of the scripts. This is resulting unexpected behavior when changing the data-partition-id.
One of the hardcoded locations: https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/rest/check.http#L202M7 - Release 0.10https://community.opengroup.org/osdu/platform/system/file/-/issues/30Need for DELETE endpoint2021-06-29T09:41:44ZParesh BehedeNeed for DELETE endpointCurrently there is no way to delete already uploaded file by user from data platform, in case user uploads wrong file by mistake that file can not be deleted by user.
We must give ability to delete metadata record and file associated wi...Currently there is no way to delete already uploaded file by user from data platform, in case user uploads wrong file by mistake that file can not be deleted by user.
We must give ability to delete metadata record and file associated with that metadata record to user, so that user can delete file uploaded by him/her when ever its necessary.
New endpoint in File Service could be DELETE /v2/files/{id}/metadataM7 - Release 0.10Paresh BehedeParesh Behedehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/well-delivery/well-delivery/-/issues/1SLB Specific Variable defined in ADO pipeline yaml files2021-06-29T22:00:07ZJasonSLB Specific Variable defined in ADO pipeline yaml filesThere are currently two issues preventing generic customers to deploy the ADO pipelines in this repo:
- Non-development pipeline is currently hard coded with an SLB-specific repo name [here](https://community.opengroup.org/osdu/platform/...There are currently two issues preventing generic customers to deploy the ADO pipelines in this repo:
- Non-development pipeline is currently hard coded with an SLB-specific repo name [here](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/well-delivery/well-delivery/-/blob/master/devops/azure/pipeline.yml#L37). This causes the ADO pipelines to fail for other users because it can't find the repo titled `security-infrastructure`
- Non-development pipeline is hard coded with SLB environment names: https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/well-delivery/well-delivery/-/blob/master/devops/azure/pipeline.yml#L89
We will need to remove these SLB-specific lines in to make the pipelines work for everyone.M7 - Release 0.10JasonJasonhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/10CSV parser program uses File service but should use Dataset service2021-07-01T03:03:41ZSpencer Suttonsuttonsp@amazon.comCSV parser program uses File service but should use Dataset serviceIt looks like this uses the File service to pull down a CSV before doing the parsing logic. Dataset service should be used when interacting with any bulk data or files via OSDU.
**I'm planning on updating this code to use Dataset servic...It looks like this uses the File service to pull down a CSV before doing the parsing logic. Dataset service should be used when interacting with any bulk data or files via OSDU.
**I'm planning on updating this code to use Dataset service instead, would this be mergeable when done?**M7 - Release 0.10ethiraj krishnamanaiduDania Kodeih (Microsoft)Joeethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/14CSV parser not deleting the workflow that is registered after integration tests2021-07-01T03:03:44ZKishore BattulaCSV parser not deleting the workflow that is registered after integration testsCSV Parser in azure registered the parser through workflow service register workflow API. After integration test, the workflow service must be deleted otherwise each run will create csv workflows in the system which will slow down the ai...CSV Parser in azure registered the parser through workflow service register workflow API. After integration test, the workflow service must be deleted otherwise each run will create csv workflows in the system which will slow down the airflow to load huge number of DAGs at runtime.M7 - Release 0.10SwapnilSwapnilhttps://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/23Spotbugs Failing in some Services with Out of Memory Exception2021-07-06T15:38:33ZDavid Diederichd.diederich@opengroup.orgSpotbugs Failing in some Services with Out of Memory ExceptionSome services, such as [Partition](https://community.opengroup.org/osdu/platform/system/partition/-/jobs/404507) fail in the spotbugs step. If re-run with `SECURE_LOG_LEVEL` set to `"debug"`, we see that the [failure](https://community.o...Some services, such as [Partition](https://community.opengroup.org/osdu/platform/system/partition/-/jobs/404507) fail in the spotbugs step. If re-run with `SECURE_LOG_LEVEL` set to `"debug"`, we see that the [failure](https://community.opengroup.org/osdu/platform/system/partition/-/jobs/404635#L878) is `java.lang.OutOfMemoryError`
From that [same debug output](https://community.opengroup.org/osdu/platform/system/partition/-/jobs/404635#L868), spotbugs is run with `java -Xmx1900M`.M7 - Release 0.10David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/63Entitlement V2 API support2021-07-07T12:11:27Zjingdong sunEntitlement V2 API supportM7 - Release 0.10Anuj GuptaShaonjingdong sunAnuj Guptahttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/issues/3Deployment framework [H1] [GONRG-619]2021-07-07T13:41:23ZSergey Krupenin (EPAM)Deployment framework [H1] [GONRG-619]https://jiraeu.epam.com/browse/GONRG-619https://jiraeu.epam.com/browse/GONRG-619M7 - Release 0.10Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/65GCP OSDU Entitlement V2 API [GONRG-226]2021-07-07T13:41:51ZSergey Krupenin (EPAM)GCP OSDU Entitlement V2 API [GONRG-226]https://jiraeu.epam.com/browse/GONRG-226https://jiraeu.epam.com/browse/GONRG-226M7 - Release 0.10Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/40Schema Version Update Issue2021-07-08T06:20:43ZYunhua KoglinSchema Version Update IssueTo reproduce the issue:
1. Register a new schema
2. Update the schema by bumping the patch (or minor) version and adding a new data property
3. Call update schema endpoint
{
"error":{
"code":400,
"message":"Breaking chang...To reproduce the issue:
1. Register a new schema
2. Update the schema by bumping the patch (or minor) version and adding a new data property
3. Call update schema endpoint
{
"error":{
"code":400,
"message":"Breaking changes found, please change schema major version",
"errors":[
{
"domain":"global",
"reason":"badRequest",
"message":"Breaking changes found, please change schema major version"
}
]
}
}M7 - Release 0.10Abhishek Kumar (SLB)Abhishek Kumar (SLB)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/62Add cache for Search ID handler [GONRG-2593]2021-07-08T13:21:54ZKateryna Kurach (EPAM)Add cache for Search ID handler [GONRG-2593]https://jiraeu.epam.com/browse/GONRG-2593https://jiraeu.epam.com/browse/GONRG-2593M7 - Release 0.10Kateryna Kurach (EPAM)Kateryna Kurach (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/21CSV Parser Enhancement - Token generation for long-running jobs2021-07-08T16:33:07ZFernando Nahu Cantera RubioCSV Parser Enhancement - Token generation for long-running jobs## Token generation for long-running jobs
An interface AuthJwtToken was added for generating tokens, the following classes have dummy implementations for it, and until reworked the request token will be used.
- AwsServiceAccountAuthToke...## Token generation for long-running jobs
An interface AuthJwtToken was added for generating tokens, the following classes have dummy implementations for it, and until reworked the request token will be used.
- AwsServiceAccountAuthToken
- ServiceAccountAuthToken
- IBMServicePrincipalAuthTokenM7 - Release 0.10SwapnilFernando Nahu Cantera RubioSwapnilhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/22CSV Parser Enhancement - Spatial data handler2021-07-08T16:33:16ZFernando Nahu Cantera RubioCSV Parser Enhancement - Spatial data handler## Spatial data handler
### Pre-requisities:
* Schema used to ingest the data has Spatial reference.
* CSV file has the Spatial data attributes.
* ExtensionProperties Block is used to provide content details of the file, the Workflow S...## Spatial data handler
### Pre-requisities:
* Schema used to ingest the data has Spatial reference.
* CSV file has the Spatial data attributes.
* ExtensionProperties Block is used to provide content details of the file, the Workflow Service uses this same block to provide Spatial data information.
* SpatialMapping: This section is used to create the Spatial data block in the ingested records.
* type: This field refers to the type of the Spatial data; currently the Workflow Service only supports point.
* latitude: This field refers to the Latitude of the point.
* longitude: This field refers to the Longitude of the point.
```
{
"ExtensionProperties": {
"FileContentsDetails": {
"TargetKind": "<<authority>:<source>:<entityType>:<version>>",
"FileType": "csv",
"SpatialMapping":{
"type": "point",
"latitude": "Column name of the CSV which contains the LATITUDE value",
"longitude": "Column name of the CSV which contains the LONGITUDE value"
},
"FrameOfReference": [
{
"kind": "CRS",
"name": "GCS_WGS_1984",
"persistableReference": "{\"wkt\":\"GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433],AUTHORITY[\\\"EPSG\\\",4326]]\",\"ver\":\"PE_10_3_1\",\"name\":\"GCS_WGS_1984\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"4326\"},\"type\":\"LBC\"}",
"propertyNames": [
"Column name of the CSV which contains the LATITUDE value",
"Column name of the CSV which contains the LONGITUDE value"
],
"propertyValues": [
"deg"
],
"uncertainty": 0
}
}
}
}M7 - Release 0.10SwapnilFernando Nahu Cantera RubioSwapnilhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/23CSV Parser Enhancement - Nested Schema2021-07-08T16:33:21ZFernando Nahu Cantera RubioCSV Parser Enhancement - Nested Schema## Nested Schema
* To support the ingestion of data into nested attributes, the headers of the uploaded csv header should match the nested attributes of the target schemas, using the delimiter characters defined on the metadata file.
*...## Nested Schema
* To support the ingestion of data into nested attributes, the headers of the uploaded csv header should match the nested attributes of the target schemas, using the delimiter characters defined on the metadata file.
* The ```nestedFieldDelimiter``` attribute in file metadata is used to define which character is going to be used on the csv file header to describe the different levels of nested attributes while the ingestor parses the files.
* The delimiter character used to define nested structures on the csv file header must match the one defined by the ```nestedFieldDelimiter``` on the file metadata record, otherwise the attributes on the csv file will not be considered nested.
```
{
"ExtensionProperties": {
"FileContentsDetails": {
"TargetKind": "<<authority>:<source>:<entityType>:<version>>",
"nestedFieldDelimiter":".",
"FileType": "csv"
}
}
}M7 - Release 0.10SwapnilFernando Nahu Cantera RubioSwapnilhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/24CSV Enhancement - Relationships2021-07-08T16:33:26ZFernando Nahu Cantera RubioCSV Enhancement - Relationships## Relationships
* CSV ingestion supports two kinds of relationships:
1. **Deterministic (Schema-driven)**
These relationships require that the entity be referred to in the record's targetKind schema under an attribute having ```...## Relationships
* CSV ingestion supports two kinds of relationships:
1. **Deterministic (Schema-driven)**
These relationships require that the entity be referred to in the record's targetKind schema under an attribute having ```x-osdu-relationship``` tag. Because they are present in the schema, they are represented directly as attributes in the ```data``` block of the record.
2. **Non Deterministic (Data-driven)**
These relationships do not require any mention in the schema. They are represented within the ```data.relationships``` block of the record.
* ExtensionProperties block in the file metadata record is used to provide additional information for ingestion. We can use this block to provide relationship information. There are three ways of providing this information:
* In the ```relationships``` block, with the entity name and a list of parent record ID(s). The ID(s) provided here are directly used to establish relationships.
* In the ```relatedNaturalKey``` block, as an entity that requires a search of the targetKind using the natural keys provided to establish a relationship.
* _sourceColumn_: Column name of the CSV file which refers to the key parent attribute.
* _targetKind_: Schema ID of the parent record.
* _targetAttribute_: The key attribute of the parent record which is used to search the parent record.
* _**Pre-requisites**_: CSV file should have the key attributes of the parent records.
```
{
"ExtensionProperties": {
"relationships": {
"project": {
"ids": [
"<recordId1>"
]
},
"well": {
"ids": [
"<recordId2>"
"<recordId3>"
]
}
},
"relatedNaturalKey": {
"wellbore": {
"targetKind":"<<authority>:<source>:<entityType>:<version>>",
"keys": [
{
"sourceColumn":"UWI",
"targetAttribute":"uwi"
}
]
}
}
}
}
```
* The schema of the record should have information about attributes that contain deterministic relationships.
* The _EntityType_ field within the ```x-osdu-relationship``` block should contain the entity that needs to be matched from the ExtensionProperties block.
```
{
"properties": {
"wellId": {
"type":"string",
"pattern":"^[\\w\\-\\.]+:\\-\\-well:[\\w\\-\\.\\:\\%]+:[0-9]*$",
"x-osdu-relationship": [
{
"GroupType":"master-data",
"EntityType":"well"
}
]
},
"wellboreId": {
"type":"string",
"pattern":"^[\\w\\-\\.]+:\\-\\-wellbore:[\\w\\-\\.\\:\\%]+:[0-9]*$",
"x-osdu-relationship": [
{
"GroupType":"master-data",
"EntityType":"wellbore"
}
]
}
}
}
```
* The final record will then have the relationships defined as below:
```
{
"data": {
"relationships": {
"project": {
"ids": [
"<recordId1>"
]
}
},
"wellId":"<recordId2>",
"wellboreId":"<recordId5>"
}
}M7 - Release 0.10SwapnilFernando Nahu Cantera RubioSwapnilhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/25CSV Enhancement - Id generation change2021-07-08T16:33:30ZFernando Nahu Cantera RubioCSV Enhancement - Id generation change## Id generation change
Change in the ID generation to follow OSDU pattern ```<authority/data-partition-id>:<source>:<entity-type>:<base64-of-xosdu-natural-keys>```
* authority/data-partition-id is taken from the request triggering the ...## Id generation change
Change in the ID generation to follow OSDU pattern ```<authority/data-partition-id>:<source>:<entity-type>:<base64-of-xosdu-natural-keys>```
* authority/data-partition-id is taken from the request triggering the workflowM7 - Release 0.10SwapnilFernando Nahu Cantera RubioSwapnilhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/26CSV Enhancement - Multithread optimization2021-07-08T16:33:39ZFernando Nahu Cantera RubioCSV Enhancement - Multithread optimization## Multithread optimization
Each record is read and added as a task in an executor service to be enriched and stored parallel with other records.## Multithread optimization
Each record is read and added as a task in an executor service to be enriched and stored parallel with other records.M7 - Release 0.10SwapnilFernando Nahu Cantera RubioSwapnilhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/20CSV Parser Enhancement - Improvement of search client to escape special chara...2021-07-08T16:33:45ZSwapnilCSV Parser Enhancement - Improvement of search client to escape special characters
## Improvement of search client to escape special characters
Change in the Search Client to escape special character reserved by the Search Service when building queries.
The special characters are: ~ ` ! @ # $ % ^ * ( ) - _ + = { } [ ...
## Improvement of search client to escape special characters
Change in the Search Client to escape special character reserved by the Search Service when building queries.
The special characters are: ~ ` ! @ # $ % ^ * ( ) - _ + = { } [ ] | \ / : ; ' < > , . ?M7 - Release 0.10SwapnilFernando Nahu Cantera RubioSwapnilhttps://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/66OSDU GCP Migrate R2 services to Entitlement v2 (integrate services to use ent...2021-07-14T12:24:51ZSergey Krupenin (EPAM)OSDU GCP Migrate R2 services to Entitlement v2 (integrate services to use entitlement v2) [GONRG-2647]https://jiraeu.epam.com/browse/GONRG-2647https://jiraeu.epam.com/browse/GONRG-2647M7 - Release 0.10Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]