OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2023-12-26T14:20:13Zhttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/94OPA Breaks Adding a New Partition to OSDU2023-12-26T14:20:13ZShane HutchinsOPA Breaks Adding a New Partition to OSDUCurrently there is only a manual workaround to add a new partition to OSDU https://osdu.pages.opengroup.org/platform/security-and-compliance/policy/bundles/#adding-a-new-partition-to-osdu
Currently if you do not follow these manual step...Currently there is only a manual workaround to add a new partition to OSDU https://osdu.pages.opengroup.org/platform/security-and-compliance/policy/bundles/#adding-a-new-partition-to-osdu
Currently if you do not follow these manual steps. There will never be a bundle for the partition and Policy Service will error on all requests for that partition. OPA configuration (which generally comes from a kubernetes config map) isn't updated to know to attempt to read bundle for that partition.
This is known to break Policy, Storage, Search and Seismic DMS (seismic-store-service v4).
Impacts Milestone releases: M14-M18M20 - Release 0.23Hrvoje MarkovicNeelesh ThakurRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comDadong ZhouYauhen Shaliou [EPAM/GCP]Shane HutchinsSrinivasan NarayananYong Zengvikas ranaHrvoje Markovichttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/issues/23Issues 17/22: OSDU pods are in CrashLoopBackOff state2023-08-16T13:00:20Zm sIssues 17/22: OSDU pods are in CrashLoopBackOff stateLocal HTTP minikube cluster with issues 17/22 errors, documentation(odt), custom-values.yaml and logs attached:
[commTicket.tar.gz](/uploads/74cda85e867b3addebf02ccac559f86f/commTicket.tar.gz)Local HTTP minikube cluster with issues 17/22 errors, documentation(odt), custom-values.yaml and logs attached:
[commTicket.tar.gz](/uploads/74cda85e867b3addebf02ccac559f86f/commTicket.tar.gz)Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/255Do not pass sensitive information using configuration files2023-05-11T20:06:33ZMorris EstepaDo not pass sensitive information using configuration filesThe client secret is currently expected to be passed in as a value through the application's configuration file. I suggest that we should instead follow established patterns used in other OSDU services to pass around sensitive informatio...The client secret is currently expected to be passed in as a value through the application's configuration file. I suggest that we should instead follow established patterns used in other OSDU services to pass around sensitive information to the running application.
See: https://community.opengroup.org/osdu/platform/consumption/geospatial/-/blob/master/gcz-transformer-core/config/application.yml#L83https://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/254Data - Load SLB New Zealand data into OSDU2024-03-12T05:04:37ZJoel RomeroData - Load SLB New Zealand data into OSDUAs a GCZ Product Owner, I want SLB New Zealand test data set to be loaded, so that the data is available for GCZ to develop with.
Note:
- Suggest Data Manager to load the data on IBM pre-ship (OSDU instance)
- Reach out to Operators fo...As a GCZ Product Owner, I want SLB New Zealand test data set to be loaded, so that the data is available for GCZ to develop with.
Note:
- Suggest Data Manager to load the data on IBM pre-ship (OSDU instance)
- Reach out to Operators for Data Managers that have this expertise
Acceptance Criteria:
- Data loaded to IBM pre-ship
- GCZ ran on loaded data and possible issues captured in the backlogMichael WilhiteMichael Wilhitehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/commons/-/issues/9Correct CRS transformation to WGS842023-06-07T14:24:52ZValentin GauthierCorrect CRS transformation to WGS84The function `search_bound_projected_for_projected_epsg_code` searches a BoundProject in the CRS catalog.
After some tests on the results, it seems that the translation is not correct (data that should layed in the North Sea is layed in...The function `search_bound_projected_for_projected_epsg_code` searches a BoundProject in the CRS catalog.
After some tests on the results, it seems that the translation is not correct (data that should layed in the North Sea is layed in Africa).
The search of the BoundProjectedCRS must be fixed.
See :
- https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/commons/-/blob/main/app/osdu_utils.py#L393
- https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/Guides/Chapters/04-FrameOfReference.md#443-finding-the-coordinatetransformation-to-wgs-84https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/262Feature - multiregion private endpoints2023-05-10T16:53:32ZArturo Hernandez [EPAM]Feature - multiregion private endpointsAs for now when private endpoints were introduced, the main restriction it is that private endpoints are restricted to the same region.
* https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-overview#private-endpoint-pr...As for now when private endpoints were introduced, the main restriction it is that private endpoints are restricted to the same region.
* https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-overview#private-endpoint-properties
> The private endpoint must be deployed in the same region and subscription as the virtual network.
This means that if we plan to deploy another partition on different region, it will not be possible due this limitation.
Ideally, I think this would be best approach:
* CR and SR should be deployed in the same region (No need for new virtual network)
* SR will need new subnet for network peering.
* DP should have own net and subnet which will peer to the SR subnet. All the private endpoints will be created in the DP attached to the dedicated virtual network of the DP resources.
I think this is not priority to be implemented on specific Milestone, I guess there are few use cases on which we may want to have partitions on different region than the control plane.Arturo Hernandez [EPAM]Arturo Hernandez [EPAM]https://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/252Transformer - Limit potential ports to 1 to prevent decoupling of Transformer...2023-05-17T15:44:02ZLevi RemingtonTransformer - Limit potential ports to 1 to prevent decoupling of Transformer/Provider Ignite configurations due to dynamic port handling when configured port is in useIn Local Profile, if Transformer is configured to start Ignite on port 10800, but port 10800 is in use, then Transformer will automatically roll over into the next port 10801. This is normally fine, but this port change is not communicat...In Local Profile, if Transformer is configured to start Ignite on port 10800, but port 10800 is in use, then Transformer will automatically roll over into the next port 10801. This is normally fine, but this port change is not communicated to the user or Provider, so Provider may still attempt to connect to Ignite on Port 10800, but fail inexplicably.
I suggest we limit the port rollover so that Transformer **must connect to the configured port** or else it fails. This way, its behavior is more predictable.
Acceptance Criteria:
- Transformer should not run and log an error if configured port already in usehttps://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/issues/37Provide clear documentation about legal tag component "Expiration date"2023-05-18T20:38:23ZDebasis ChatterjeeProvide clear documentation about legal tag component "Expiration date"Legal tag property "expiration date".
https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/blob/master/docs/tutorial/ComplianceService.md#expiration-date
Let's say, one data record has legal tag attached, where...Legal tag property "expiration date".
https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/blob/master/docs/tutorial/ComplianceService.md#expiration-date
Let's say, one data record has legal tag attached, where expiration date is 1-May-2023.
Now that today's date is 6-May-2023, should that data record be still accessible?
Should Operator (its Data Manager) execute some process in the background to detect such records and do something with it?
Will Indexer pick up such record and replicate in Elasticsearch, at the very onset i.e. when the record is first created?
I could not find clear documentation about handling of this property in Data Platform.
Please check and advise.
Thank youhttps://community.opengroup.org/osdu/platform/system/register/-/issues/43Provide clear "end to end" documentation of typical use case of Register service2024-01-29T10:51:46ZDebasis ChatterjeeProvide clear "end to end" documentation of typical use case of Register serviceSuggest creation of simple "Hello world" style example for real life utilization of this service.
Mention clearly if some steps cannot be tested from Postman (like other services) and will require some coding.
Also bring "notification"...Suggest creation of simple "Hello world" style example for real life utilization of this service.
Mention clearly if some steps cannot be tested from Postman (like other services) and will require some coding.
Also bring "notification" service in picture, if that is relevant.
For example, can this help in use case such as this? Data Manager is concerned about "Well" record in OSDU instance and would like to be notified if and when any user adds a new record, deletes an existing record or updates an existing record?
If yes, describe how one can implement this with Register service and possibly other related service.
If not, please pick a different, meaningful real life use case and describe the full implementation.
You may do this separately for two cases which are already documented - (a) Action service, (b) How to become DDMS.https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/493Dataset API to get storageInstructions for kindSubType=dataset--FileCollectio...2024-02-08T12:24:12ZKamlesh TodaiDataset API to get storageInstructions for kindSubType=dataset--FileCollection.Generic is not consistent across all the CSPsThe implementation of GET storageInstruction for DATASET API is not consistent across all the CSPs
e.g. https://{{DATASET_HOST}}/storageInstructions?kindSubType=dataset--FileCollection.Generic when it is executed the output is various am...The implementation of GET storageInstruction for DATASET API is not consistent across all the CSPs
e.g. https://{{DATASET_HOST}}/storageInstructions?kindSubType=dataset--FileCollection.Generic when it is executed the output is various among all the CSPs. The usage example provided by AWS works in AWS environment but cannot get it to work in other CSP's environments, because other CSPs are not returning the same kind of information when request is made for storageInstructions as
the documentation is lacking and the working example is missing, on how to consume/use the information to create the collection of multiple files (FileCollection.Generic) using the registerDataset
**Details**
`**AWS**
**curl --location --request POST 'https://r3m17ps.preshiptesting.osdu.aws/api/dataset/v1/storageInstructions?kindSubType=dataset--FileCollection.Generic'** \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer eyJraWQiOi...g0QA8ImMw'
Response 200 OK
{
"providerKey": "AWS_S3",
"storageLocation": {
**"unsignedUrl"**: "s3://r3m17ps-shared-561735291427-ca-central-1-file/osdu/FHugmJALXGTlPTC5VpxBxEmQfGy68jXp/",
"createdAt": "2023-05-06T11:32:37.167Z",
"connectionString": "AccessKeyId=ASIAYFSQIKIRU352WEF6;SecretAccessKey=n0QRlrK/YwDooN9+n2q8ky1whKhjFUCLK05U5aAZ;SessionToken=IQoJb3JpZ2luX2VjEBMaDGNhLWNlbnRyYWwtMSJHMEUCIQD2F8q01x6RAwLUrEhxH64m3FEVUtf058bPkrnOZfKjQwIgNuLtSmO7hIExCuSquCOq5+K1DyXgVxuiGNezuAkDzIsqpgUILRACGgw1NjE3MzUyOTE0MjciDL77xgCuhM4FLyVm3CqDBZw9vX/uBEhZ2RpwCmld1E6qSYgE2rKqdeNlv8JLGjENSq66gw3VA7Rw73UiQfq4ilOp32zf5tSvXN2P4BNZbtuv+t2bgjGHMhqH0eaQn+y5VJHMJOVJJuI7aajgFr6c0bGE2zxi9RrHX7d7luV79arzBlIO1XeeSLlDYlyPVHRB+pc4VmAXslpwNeFkHh/ih1P+6sy/VkHlmwYMTuz2q0rG0M0ME+SwK1udd0jAukLbAu0Ao2zAJ3yQ5QXq/L8DtHOx8nc4w40p/eocLcy4HOwKyY0omeM5QfQ2gCrPcBPS0jLwkUCnSjDVODffuakXhT9pALeqoX7stqDf2F2PrTxt8J+XMTRxo4tIMMtpzpiJzGpGxuqQIDlroCYf+AUmg4GxyCXm0oCNxH8uoSF8pnxsu3dQvSO8+dXf6QzOzWCmyyh9YjRrZLveITJ0Y08vcbBQGCw04HVfexhFxDOmaNf8X4PfSQr6Czwnbxl7p5S3NLVo56lxEe195+nPk9KHFOHBHHmq3du/gy2IrE0a4zMZdMG/p8vSAFvux8VUqR2dB9FJlhh3DiKq3MoBvFBo/D4MCI4JZ7v3hUJFo+fZP2ew981vz5OT1xGLH2MvY+53/jtMjYvt7HqN1HNv3SZuAzboqxfb79G7Y09ZiKjKiincxt6/FVM0jVolXXQeyaZgZMkh+asxiFlztZAL8KueGxXnHrPlsiUD+nFuWYv4Qz3vmBTNjNK1gqCczg4rgb7kM/AwNZ4U3MUanVVbXUX15T0gOeu826e2Adv+aJy5twR37Gi1+1Zp4tN4/Q+liI5ms3xQJAwg4BJpwVptNMAXaWSkojhnFvvNt8vgfQnBPy9aPdQw1fXYogY6nQFSWKU1AthSv4P/CbCc5mauZyAT8m5UYU3DoNu07Lb2+rPgiRji/H0cvCalDFPa4Typ4vzYWWjWNxOHkc3lWbmI3tI69IRBVjdTGBeA5P706YTb3h00M8CbwpTGI3WkR9FEKgSTtv0tgEZHt8dy89BqyG3R/5IMZrTS2HRMht/kDzwVoFEo6ruAA+Na89DF/V8d6T8WW45aPBZSR1lu;Expiration=2023-05-06T12:32:37Z",
"credentials": {
**"accessKeyId"**: "ASIAYFSQIKIRU352WEF6",
**"secretAccessKey"**: "n0QRlrK/YwDooN9+n2q8ky1whKhjFUCLK05U5aAZ",
**"sessionToken"**: "IQoJb3JpZ2luX2VjEBMaDGNhLWNlbnRyYWwtMSJHMEUCIQD2F8q01x6RAwLUrEhxH64m3FEVUtf058bPkrnOZfKjQwIgNuLtSmO7hIExCuSquCOq5+K1DyXgVxuiGNezuAkDzIsqpgUILRACGgw1NjE3MzUyOTE0MjciDL77xgCuhM4FLyVm3CqDBZw9vX/uBEhZ2RpwCmld1E6qSYgE2rKqdeNlv8JLGjENSq66gw3VA7Rw73UiQfq4ilOp32zf5tSvXN2P4BNZbtuv+t2bgjGHMhqH0eaQn+y5VJHMJOVJJuI7aajgFr6c0bGE2zxi9RrHX7d7luV79arzBlIO1XeeSLlDYlyPVHRB+pc4VmAXslpwNeFkHh/ih1P+6sy/VkHlmwYMTuz2q0rG0M0ME+SwK1udd0jAukLbAu0Ao2zAJ3yQ5QXq/L8DtHOx8nc4w40p/eocLcy4HOwKyY0omeM5QfQ2gCrPcBPS0jLwkUCnSjDVODffuakXhT9pALeqoX7stqDf2F2PrTxt8J+XMTRxo4tIMMtpzpiJzGpGxuqQIDlroCYf+AUmg4GxyCXm0oCNxH8uoSF8pnxsu3dQvSO8+dXf6QzOzWCmyyh9YjRrZLveITJ0Y08vcbBQGCw04HVfexhFxDOmaNf8X4PfSQr6Czwnbxl7p5S3NLVo56lxEe195+nPk9KHFOHBHHmq3du/gy2IrE0a4zMZdMG/p8vSAFvux8VUqR2dB9FJlhh3DiKq3MoBvFBo/D4MCI4JZ7v3hUJFo+fZP2ew981vz5OT1xGLH2MvY+53/jtMjYvt7HqN1HNv3SZuAzboqxfb79G7Y09ZiKjKiincxt6/FVM0jVolXXQeyaZgZMkh+asxiFlztZAL8KueGxXnHrPlsiUD+nFuWYv4Qz3vmBTNjNK1gqCczg4rgb7kM/AwNZ4U3MUanVVbXUX15T0gOeu826e2Adv+aJy5twR37Gi1+1Zp4tN4/Q+liI5ms3xQJAwg4BJpwVptNMAXaWSkojhnFvvNt8vgfQnBPy9aPdQw1fXYogY6nQFSWKU1AthSv4P/CbCc5mauZyAT8m5UYU3DoNu07Lb2+rPgiRji/H0cvCalDFPa4Typ4vzYWWjWNxOHkc3lWbmI3tI69IRBVjdTGBeA5P706YTb3h00M8CbwpTGI3WkR9FEKgSTtv0tgEZHt8dy89BqyG3R/5IMZrTS2HRMht/kDzwVoFEo6ruAA+Na89DF/V8d6T8WW45aPBZSR1lu",
"expiration": "2023-05-06T12:32:37.000+00:00"
},
"createdBy": "admin@testing.com",
**"region":** "ca-central-1"
}
}
**Using the information from the response of the previous request upload the file
Authorization Tab (Settings)
Type: AWS Signature
AccessKey
SecretKey
AWS Region
Service Name: s3
Session Token**
**curl --location --request PUT 'https://r3m17ps-shared-561735291427-ca-central-1-file.s3.amazonaws.com/osdu/FHugmJALXGTlPTC5VpxBxEmQfGy68jXp/file1' \**
--header 'Content-Type: text/csv' \
--header 'X-Amz-Security-Token: IQoJb3JpZ2luX2VjEBMaDGNhLWNlbnRyYWwtMSJHMEUCIQD2F8q01x6RAwLUrEhxH64m3FEVUtf058bPkrnOZfKjQwIgNuLtSmO7hIExCuSquCOq5+K1DyXgVxuiGNezuAkDzIsqpgUILRACGgw1NjE3MzUyOTE0MjciDL77xgCuhM4FLyVm3CqDBZw9vX/uBEhZ2RpwCmld1E6qSYgE2rKqdeNlv8JLGjENSq66gw3VA7Rw73UiQfq4ilOp32zf5tSvXN2P4BNZbtuv+t2bgjGHMhqH0eaQn+y5VJHMJOVJJuI7aajgFr6c0bGE2zxi9RrHX7d7luV79arzBlIO1XeeSLlDYlyPVHRB+pc4VmAXslpwNeFkHh/ih1P+6sy/VkHlmwYMTuz2q0rG0M0ME+SwK1udd0jAukLbAu0Ao2zAJ3yQ5QXq/L8DtHOx8nc4w40p/eocLcy4HOwKyY0omeM5QfQ2gCrPcBPS0jLwkUCnSjDVODffuakXhT9pALeqoX7stqDf2F2PrTxt8J+XMTRxo4tIMMtpzpiJzGpGxuqQIDlroCYf+AUmg4GxyCXm0oCNxH8uoSF8pnxsu3dQvSO8+dXf6QzOzWCmyyh9YjRrZLveITJ0Y08vcbBQGCw04HVfexhFxDOmaNf8X4PfSQr6Czwnbxl7p5S3NLVo56lxEe195+nPk9KHFOHBHHmq3du/gy2IrE0a4zMZdMG/p8vSAFvux8VUqR2dB9FJlhh3DiKq3MoBvFBo/D4MCI4JZ7v3hUJFo+fZP2ew981vz5OT1xGLH2MvY+53/jtMjYvt7HqN1HNv3SZuAzboqxfb79G7Y09ZiKjKiincxt6/FVM0jVolXXQeyaZgZMkh+asxiFlztZAL8KueGxXnHrPlsiUD+nFuWYv4Qz3vmBTNjNK1gqCczg4rgb7kM/AwNZ4U3MUanVVbXUX15T0gOeu826e2Adv+aJy5twR37Gi1+1Zp4tN4/Q+liI5ms3xQJAwg4BJpwVptNMAXaWSkojhnFvvNt8vgfQnBPy9aPdQw1fXYogY6nQFSWKU1AthSv4P/CbCc5mauZyAT8m5UYU3DoNu07Lb2+rPgiRji/H0cvCalDFPa4Typ4vzYWWjWNxOHkc3lWbmI3tI69IRBVjdTGBeA5P706YTb3h00M8CbwpTGI3WkR9FEKgSTtv0tgEZHt8dy89BqyG3R/5IMZrTS2HRMht/kDzwVoFEo6ruAA+Na89DF/V8d6T8WW45aPBZSR1lu' \
--header 'X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' \
--header 'X-Amz-Date: 20230506T113518Z' \
--header 'Authorization: AWS4-HMAC-SHA256 Credential=ASIAYFSQIKIRU352WEF6/20230506/ca-central-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=46fda8f99c22d1e881f37b12b94724079a1f2c6559a639fe86799a671f26cb91' \
--data '@Postman Collection/31_CICD_Setup_CSVIngestion/wellbore.csv'
**Azure**
curl --location --request POST 'https://osdu-ship.msft-osdu-test.org/api/dataset/v1/storageInstructions?kindSubType=dataset--FileCollection.Generic' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer eyJ0eXAiOiJ...e9tIuUDbg'
Response 200 OK
{
"providerKey": "AZURE",
"storageLocation": {
**"signedUrl"**: "https://osdumvpdp1shipbbnghiercl.dfs.core.windows.net/datalake-staging-area/osdu-user-1683373961973-2023-05-06-11-52-41-973-276de2f88be64f29861d7f23de482330?sv=2020-08-04&spr=https&st=2023-05-06T11%3A52%3A42Z&se=2023-05-13T11%3A52%3A42Z&sr=d&sp=racwl&sdd=1&sig=Si5JnpGmg15ToaXDlz3voXPPmcWdgEOYzGvjqR%2BTEdk%3D",
**"fileCollectionSource"**: "/osdu-user-1683373961973-2023-05-06-11-52-41-973-276de2f88be64f29861d7f23de482330",
"createdBy": "osdu-user"
}
}
**How to use the information to load the multiple files to create a collection**__
**GC**
curl --location --request POST 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/dataset/v1/storageInstructions?kindSubType=dataset--FileCollection.Generic' \
--header 'data-partition-id: odesprod' \
--header 'Authorization: Bearer ya29.a0AWY7Ckmv_...thKfg0169'
Response 200 OK
{
"providerKey": "GCP",
"storageLocation": {
**"url"**: "https://storage.googleapis.com/osdu-data-prod-odesprod-staging-area/",
**"signingOptions"**: {
**"bucket"**: "osdu-data-prod-odesprod-staging-area",
**"connectionString"**: "ya29.dr.AfMbl1iSb-YCAWAR64myZ33i8nYVBFRLX7BT4G4G5x7CbCe6EeJBioAzBgbut_O0LVCynVd9ebQSVIwMYy_O6O22cM43CFt-AC5S_FHD_Up_OS1pDiBJ0SS-rAAK052taCvek_vAspNC1kFDEAZ-bgOMGSTe5uLYhzfjLX2v89JDPO2I_ytLQkQGGjG9sn8wR2xuSFfEfOhH3O9eGa5r9LwCAUsVus7j1ZKjvVnAoiFVBSWcg-lrOiG8yUycp-xAfHdYNErxELPq_yBeIqlgnOkF3Di5JZDNZSDQrp8n6CCnUTYIXiBgUl-E193cTHb5UwnYVatn1oMUmpN3MP76yMdCvMPbgmxLZCB1fMGQOCjNF49AwoibPZbyk1FU7jr9OlAiJ-lmGimx5V3V6Vp1NbGHLlA_UAbIcvmtFE4oOAIo9NXvIohL9p4gaktCbJbCde11xDw-87OUT7XnmBxsT7fG0HTXx6tpmdDqh9URPUhot-bj2n67fjUYirIO1EgCpQc-YNaX8pSB03DVFsdeNw33xWKJ7WiU4U0sP4gVr0iKzCjP5LFmxUpHDIcx2dHkNRKEYdDTyn3947rsXQEdYrgW-fSp1hFvwgAcRTqW_qIcWZmRt15OAoPZJompCEUnslqfH6SkSESXVlsWtME4a2mgaYvzRMCfuvtmsMQpAR8a0o6MbqxnQ6Lj5EUkMt_kqtv7CgW1XuKvkqhA4gkZPoJRKu55PtrEvjEpOgzxBbG0e4MZT68KngAtSsfO0IuG7bPXb1_KQrHBOpRYObe2RTAquj9tTGOCXnEWm-evKN3MBiHbdJsLXtbFSODcD9HqdJM8UsTBotigEAMX_7S6bkzE9JMpc9j3g2xrBbpHhDkR7iSB71TSxhLalFC_IeUTj9XBlHd3WmyqYa1JCmGyj8vrckQiR8yYfyUyHpdfFe2EtczeA10323UAnI2G8HZw2M2ViHKdBDfTk-AfruUR5iU8T-0nBko761nWJ1R7zPg1d50XCdiWAxu9g_9Q8qW-s9ss3DVAmhBb_QPUQblRIGWmawZQIUJPPpzSOGNinR9MH4R_un2vpA1j1M5zRZU4td1yzDUkyLbC2-X8GrKtZAnBwZC9GGu_6TOMwe0wIb0RQY2yGBR0-MdBIiDTVkwVyZQpGHZb6JTlSGB4sl3H_1qXp58F2koDKbHKLNdt7eRryWjPQ-9VQIFH2V1CELo05_0Sc0sCqBNAwHf9XD_r93DPgLlwpv-FOc8Zl4lJi48-WmFwNhTTON8juKT7MNmUS9UFI0nDKHTpWVBeGKbT-xjNt75BzunsVdM8ZuWW-i6CzO7KJaFGzL_7l8-1pyayBm0sXqz1T4j8KLlUPsCsV01RY5516OwR2YVI0araTVnDrqaUW4TiAFnxfxIIsNeTYXSvI436ityUFXjCEp4mAxnid5fEjZAI2ujQCDYjjSzwNqXBfmkswK6fh8NOMl8J8BGHp90FiP_j2HHPYngGzo6LP8Ry4-bkutUQqBGxu5v2lbzMKsNePO5vxzvd7W_7vavycmQEmDtsomnZtEBFt5xQpVQe1frMr0h6raXb-O_Oo_Z6ctPNgOQ30VFy3gMAfj0Bav2B9KWWVa5Gdw-p_einENiJzrVp_qax3A-ui7-B_KvqtMdDAnDjyr0jrpT2-wUubcmjFK3Ado_sn64aEuHWo8x9j0Fzdgv559ddUC-h6gQ7t3MqaJBHUcAVoqd0C38IrvtogE3lMTJAIVLEPhiTaC_6vlTw6l57ujDip4mNn1hdODN-N2XmbLtFq6aw886oZP1lheJr7uJfY5igKXKMlgyIbiykeOEYxZMERpkf5BDEdmpiEixAFuUC0JehKeZ8e4xTld2LcPGqZNyPm_Mt-gHlRUxdXbHhe-ID-yLVzYDmbTOp5-zvqA4A_vIMpV2dRWVUaXznmF_F7Y90Ou1SGKNZ2uLL6_1UXDvJeGF9dR_CI3OlK9ojY48D0FGMrua0kjOY5Pqe76dqIFL3Q2KPB2Q-tlgz-EHei4QQmZFzfwhMXYtCVPnvwHKT8PmLQopBrmYQVVDF9inew0iI2eZYjIsRqGSCQvpBBOt43Pk1OJJ4-D3xicu-45m4xgPn58wfoHyBjf1VYjFeTXrXQ-0qTNSN_IVYaSEtlthtB6YgfDjUWAoIs1Zpl-chhLjd3nEmXvuHT3vqph-PUx_raodOb1LQ9nUw6Ck0t1-x3XQTecaIGvBJUF-cBXSu7Ahz3QNwW4nshIvnt3u_jAijfbDPibJyRymFTVqNh5w-TNqXKWiL7PwrBlHAUlfF4FY-OTl8rA7DesF-t_2semd5BVLQQUnfUOBoi1rejFFXznEOsi4-ikRp5KmDC6OvZycKcZM8S0Jm2XH97yDp8ZzBXxcYX7BubcqSBc23nZs8ItQV_suikKE6x0Sshz5fEtQo4vltc4EaPZF82sT1YsZGGzLe6NdBLB4jmF5rnvepiNfOUtf-XgHG6BwDqzpkA9FIFGfRxjV3d4hVKHjJ0Qn827nCISpSQjjoEuj4OYW7npoC",
**"filepath"**: "70219d0ce05a4e14b828e719eaf6f552/"
},
**"fileCollectionSource"**: "70219d0ce05a4e14b828e719eaf6f552",
"createdBy": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com"
}
}
**How to use the information to load the multiple files to create a collection**__
`
Attached is the Postman collection that works on AWS
[DatatsetFileCollection_Test.postman_collection.json](/uploads/805790ab17f8fecae4c7bc557125da24/DatatsetFileCollection_Test.postman_collection.json)M20 - Release 0.23Denis Karpenok (EPAM)Dzmitry Malkevich (EPAM)Yauhen Shaliou [EPAM/GCP]Denis Karpenok (EPAM)https://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/249Make it easier to inject CSP-specific authentication logic2023-05-09T17:34:44ZMorris EstepaMake it easier to inject CSP-specific authentication logicThe authenticateOsdu method found [here](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/blob/master/gcz-transformer-core/src/main/java/org/osdu/gcz/transformer/configuration/TransformerConfiguration.java#L80) retu...The authenticateOsdu method found [here](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/blob/master/gcz-transformer-core/src/main/java/org/osdu/gcz/transformer/configuration/TransformerConfiguration.java#L80) returns a concrete class. This is causing issues for CSP-specific authentication logic.
I suggest that the class be changed so that a new private auto-wired variable of type IAuthenticate is used instead. Each CSP will have to provide their own implementation of that interface if they choose to do so. Otherwise, a default implementation of the interface is injected into the variable.https://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/248Ignite cache lost2023-07-12T07:15:38ZMorris EstepaIgnite cache lostThe ignite cache partition seems to get lost at some point. I receive the following error when hitting the Provider service with a feature query:
```
{
"error": "50000: class org.apache.ignite.internal.processors.cache.CacheInvalidSta...The ignite cache partition seems to get lost at some point. I receive the following error when hitting the Provider service with a feature query:
```
{
"error": "50000: class org.apache.ignite.internal.processors.cache.CacheInvalidStateException: Failed to execute query because cache partition has been lostPart [cacheName=osdu_wks_master-data--Well_1.0.0, part=0]"
}
```
Restarting the ignite and transformer pods temporarily fixes the issue.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/issues/28Please update OpenVDS dependencies in dll-hell.md2023-05-04T14:03:12ZJørgen Lindjorgen.lind@3lc.aiPlease update OpenVDS dependencies in dll-hell.mdVersion 3.2 and newer of OpenVDS does not have any visible dependencies, except sytem libraries such as:
- linux-vdso.so.1
- libpthread.so.0
- libdl.so.2
- libstdc++.so.6
- libm.so.6
- libgcc_s.so.1
- libc.so.6
- ld-linux-x86-64.so.2
T...Version 3.2 and newer of OpenVDS does not have any visible dependencies, except sytem libraries such as:
- linux-vdso.so.1
- libpthread.so.0
- libdl.so.2
- libstdc++.so.6
- libm.so.6
- libgcc_s.so.1
- libc.so.6
- ld-linux-x86-64.so.2
The situation is very similar on Windows. Therefor the mentioning of OpenVDS in the dll-hell section is outdated and should be revised.https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/26EDS M17 Features and Fixes details2023-05-11T05:56:21ZPriyanka BhongadeEDS M17 Features and Fixes detailsThe significant features and fixes of EDS M17 are listed below:
Features:
1. PasswordCredentials OAuth Flow Type has been introduced, which allows EDS M17 to generate an access token for data providers using this flow type for authoriz...The significant features and fixes of EDS M17 are listed below:
Features:
1. PasswordCredentials OAuth Flow Type has been introduced, which allows EDS M17 to generate an access token for data providers using this flow type for authorization. To generate the access token, the parameters required are username, password, client ID, client secret, and scopes. The "FlowTypeID": "{{data_partition_id}}:reference-data--OAuth2FlowType:PasswordCredentials:" is added to the ConnectedSourceRegistryEntry record.https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/home/-/issues/267
2. EDS M17 now validates the expiry of the refresh token and auto-generates a new refresh token while updating the secret vault. If the refresh token value in the secret vault is expired, the eds_ingest fails to generate an access token, and the run fails. To handle this situation, eds_ingest verifies if the refresh token is expired and generates a new refresh token value following PasswordCredentials authentication grant type. The secret service then accesses the new refresh token value to update the old/expired value with the newly generated refresh token value. The "FlowTypeID": "{{data_partition_id}}:reference-data--OAuth2FlowType:RefreshTokenKeyName:" is added to the ConnectedSourceRegistryEntry (CSRE). The data provider for this feature is Katalyst. https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/19
3. Parent data mapping is now handled in EDS M17, which includes keeping the source identifier ("id" of the parent data) in NameAlias of the parent record during ingestion into the operator environment. This helps the operator to find the source of each record and group them. When ingesting child data (e.g., Well log data) into the target environment, the child data is tagged to the right master data (e.g., Wellbore) in the target environment, and there is no name mismatch. This feature helps to identify a unique well using external rules between the external source and the target environment. https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/home/-/issues/268
Fixes:
A logger has been added to detail the Osdu_ingest run id and the sample-fetched data record. The message displayed in eds_ingest Airflow Logs includes Osdu_ingest Run Id and one Sample data fetched from the data provider with the text "Displaying only one Sample Record." https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/23
The conversion of ConnectedSourceDataPartitionID to OnIngestionDataPartitionID for Array Datatype has been fixed. While ingestion, ConnectedSourceDataPartitionID (provider’s data partition id) is replaced with the OnIngestionDataPartitionID (operator’s data partition id) for all the parameters of the record with different datatypes (arrays, dicts). Each conversion is handled differently based on its datatype. For example, the conversion of string parameters from 'ResourceHomeRegionID': 'osdu:reference-data--OSDURegion:AWSEastUSA:' to 'ResourceHomeRegionID': 'opendes:reference-data--OSDURegion:AWSEastUSA:' is done similarly to the conversion of the array datatype. https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/home/-/issues/261
The Dynamic Schema Authority for Kind of CSRE, CSDJ, and ExternalReferenceValueMapping is now added from Airflow Variable. The constant file has Kind of few eds dependent schemas, such as ConnectedSourceRegistryEntry, ConnectedSourceDataJob, and ExternalReferenceValueMapping. The Schema_Authority value was static in the Kind, which is now replaced with the Schema_authority value fetched from the Airflow Variable. https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/22
EDS now raises an exception when Airflow Variable is not found or None. Eds_ingest fails with KeyError if any of the important Airflow variable values are missing. https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/21M17 - Release 0.20Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/25EDS - Include Password Credentials OAuth Flow Type2023-05-04T08:33:42ZPriyanka BhongadeEDS - Include Password Credentials OAuth Flow Type- [x] Identify the changes
- [x] create a function/POC to handle Password Credentials OAuth Flow Type
- [x] create a unit test case
- [x] Test the functionality
- [x] code review- [x] Identify the changes
- [x] create a function/POC to handle Password Credentials OAuth Flow Type
- [x] create a unit test case
- [x] Test the functionality
- [x] code reviewM17 - Release 0.20Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/24EDS - Display Osdu_ingest run ID in eds_ingest Xcom Summary2023-05-23T08:14:18ZPriyanka BhongadeEDS - Display Osdu_ingest run ID in eds_ingest Xcom SummaryM18 - Release 0.21Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/491EDS - M17 feature and fixes Testing on Azure Preship with detailed observatio...2023-06-13T10:59:51ZPriyanka BhongadeEDS - M17 feature and fixes Testing on Azure Preship with detailed observation and findings- [x] eds_scheduler run details after changes in constant file and including airflow utility
- eds_scheduler dag link : https://osdu-ship.msft-osdu-test.org/airflow2/graph?dag_id=eds_scheduler
- [x] Logger for eds_ingest - Display...- [x] eds_scheduler run details after changes in constant file and including airflow utility
- eds_scheduler dag link : https://osdu-ship.msft-osdu-test.org/airflow2/graph?dag_id=eds_scheduler
- [x] Logger for eds_ingest - Display message for sample record - https://github.com/ExxonMobil/osdu-platform/issues/318
- [x] Logger for eds_ingest - Display Osdu_ingest run ID explicitly in logs - https://github.com/ExxonMobil/osdu-platform/issues/318
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T07%3A59%3A56%2B00%3A00
- ![image](/uploads/bcf7d5da196ebef814667114a3bc337d/image.png)
- [x] Password Credential OAuth flow type - EDS - Include Password Credentials OAuth Flow Type #303
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T11%3A45%3A47%2B00%3A00
CSRE : opendes:master-data--ConnectedSourceRegistryEntry:KatalystTestingM17
![image](/uploads/7178379b3df27dfc0b32f2e411503bcb/image.png)
![image](/uploads/f56a814c55a2c69300f916f1f46eba64/image.png)
- [x] Validate the expiry of refresh token and generate a new refresh token and update secret vault - https://github.com/ExxonMobil/osdu-platform/issues/324
- eds_ingest dag link :
- [x] EDS - Adding Dynamic Schema Authority for Kind of CSRE ,CSDJ and ExternalReferenceValueMapping from Airflow Variable - https://github.com/ExxonMobil/osdu-platform/issues/360
- [x] EDS - Raise exception when Airflow Variable not found or None - https://github.com/ExxonMobil/osdu-platform/issues/359
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T08%3A10%3A55%2B00%3A00
-
![image](/uploads/0e48aa06635e320ea51b9b130bd28fa6/image.png)
- [ ] EDS: Conversion of the ConnectedSourceDataPartitionID to OnIngestionDataPartitionID for Array Datatype - https://github.com/ExxonMobil/osdu-platform/issues/322
- OSDU Ingest Link:
- [x] Test on Non-OSDU compliant Provider for master and WPC data
- Master Data -- SeismicAcquisitionSurvey
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T11%3A54%3A02%2B00%3A00
- Osdu_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=Osdu_ingest&task_id=provide_manifest_integrity_task&execution_date=2023-05-02T11%3A54%3A15.263543%2B00%3A00
- Work Product Component -- issue with Wrapper - unable to provide data as "()" in the query
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T12%3A00%3A21%2B00%3A00
- Osdu_ingest dag link :
- [x] Test on OSDU Compliant Provider for master and WPC data - AWS M16
- Master Data -- Well
- eds ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T07%3A59%3A56%2B00%3A00
- Osdu_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=Osdu_ingest&task_id=process_single_manifest_file_task&execution_date=2023-05-02T08%3A00%3A28.724429%2B00%3A00
- Work Product Component Data -- WellLog
- eds ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T08%3A06%3A35%2B00%3A00
- Osdu_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=Osdu_ingest&task_id=process_single_manifest_file_task&execution_date=2023-05-02T08%3A06%3A48.542915%2B00%3A00
- [ ] Ingest any record and it will be added with "NameAliases" like below which has the source record id - https://github.com/ExxonMobil/osdu-platform/issues/289
"NameAliases": [
{
"AliasName": "odesprod:master-data--Basin:Kam2_02",
"AliasNameTypeID": ":reference-data--AliasNameType:EDSConnectedSourceIdentifier:",
"DefinitionOrganisationID": null
}
]
*Removed data-partition-id to avoid osdu_ingestion error
- [ ] Create provider specific External Reference Value Mapping - https://github.com/ExxonMobil/osdu-platform/issues/376https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/484M17 AWS - WITSML Parser - Trajectory data type - failure at schema validation...2023-05-02T21:04:01ZDebasis ChatterjeeM17 AWS - WITSML Parser - Trajectory data type - failure at schema validation stage[M17-AWS-WITSML-Trajectory-steps-and-data-Debasis.zip](/uploads/59370f20510a3deae8f2dd1398e39e3d/M17-AWS-WITSML-Trajectory-steps-and-data-Debasis.zip)
Please note that this is different problem than what we experienced earlier with work...[M17-AWS-WITSML-Trajectory-steps-and-data-Debasis.zip](/uploads/59370f20510a3deae8f2dd1398e39e3d/M17-AWS-WITSML-Trajectory-steps-and-data-Debasis.zip)
Please note that this is different problem than what we experienced earlier with work-product schema 1.1.0.
This time, it fails in schema validation stage. for Trajectory station type.https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/92Update Policy service to support latest python2024-03-15T15:40:16ZShane HutchinsUpdate Policy service to support latest pythonPolicy Service requires Python 3.9.x.
Update Policy service to use a more recent version of Python.
Created from https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/91Policy Service requires Python 3.9.x.
Update Policy service to use a more recent version of Python.
Created from https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/91M24 - Release 0.27Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/reservoir/open-etp-server/-/issues/54Add challenges response used when the client does not have a token2023-04-27T13:34:39ZFabiola RiveraAdd challenges response used when the client does not have a tokenNeed to add the AuthorizationDetails endpoint capability which has the information the client needs to find the authorization server and get a bearer token.
Also add the challenges field to AuthorizeResponse (which is the same informat...Need to add the AuthorizationDetails endpoint capability which has the information the client needs to find the authorization server and get a bearer token.
Also add the challenges field to AuthorizeResponse (which is the same information that is specified in an endpoint's AuthorizationDetails endpoint capability).
Can add a new environment variable with the bearer auth param.
See ETP specs for details.