OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2024-03-13T14:17:00Zhttps://community.opengroup.org/osdu/platform/data-flow/real-time/RTDIP/rtdip/-/issues/6Finalize RTDIP-OSDU architecture2024-03-13T14:17:00ZMikhail TeplitskiyFinalize RTDIP-OSDU architecturehttps://community.opengroup.org/osdu/platform/home/-/issues/55Azure GLAB/Pre-ship environments - change in authentication process2024-03-12T14:54:20Zsaketh somarajuAzure GLAB/Pre-ship environments - change in authentication process## Background
- Due to recent security changes in the azure tenant, to access the osdu environments, one must have personal access token to test the services and workflows.
## Prerequisites
- Person who wants to access the environment...## Background
- Due to recent security changes in the azure tenant, to access the osdu environments, one must have personal access token to test the services and workflows.
## Prerequisites
- Person who wants to access the environment (**Azure GLAB / Azure Pre-ship** should be invited to the azure tenant, Post accepting the invitation, one should complete the onboading process which includes setting up authenticator application/MFA.
- Person should have `tenant_id`, `client_id`, `client_secret` handy, corresponding to the environement which he/she/they are generating access token.
### How request
- Approach azure team in slack or comment on this issue
## Procedure to create Personal Access token ( access_token )
- Get `tenant_id`, `client_id`, `client_secret` handy of the specific environment (**Azure GLAB / Azure Pre-ship**) to which access token is being generated.
- In case of required access for both GLAB and preship environments, this process should be followed seperately with two different sets of corressponding `client_id`, `client_secret`.
- **Step 1.1**: Prepare the below URL replacing the parameters **<tenant_id>**, **<client_id>** with actual values of targetted environment.
```bash
https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/authorize?client_id=<client_id>&response_type=code&redirect_uri=http://localhost:8080&response_mode=query&scope=<client_id>%2f.default&state=12345&sso_reload=true
```
- **Step 1.2**: After you replace the parameters, you can paste the request in the URL of any browser (incognito window/private tab) and select Enter.
- **Step 1.3**: Sign in to your Azure portal with your organisation email id and its corressponding password.
- **Step 2.1**: Now the browser authenticates the user and You might see the "Hmmm...can't reach this page" error message in the browser. You can ignore it. <br>
![localhost-redirection-error](/uploads/cec250436b964112198d055cb3dac4ce/localhost-redirection-error.png)
- The browser redirects to `http://localhost:8080/?code={authorization code}&state=...` upon successful authentication.
- **Step 2.2**: Copy the response from the URL bar of the browser and fetch the text between **code=** and **&state**. This is known as **code** <br>
ex: http://localhost:8080/?code=**0.BRoAv4j5cvGGr0...au78f**&state=12345&session....
- Save the **code** as a postman environment variable `code` which will be used in next step.
- **Step 2.3:** Replace **<tenant_id>**, **< code >** (code in step 2.2), **<client_id>** and **<clent_secret>** in the following curl request, hit this curl request using postman tool.<br>
- On postman click on new request.
![new_postman_request](/uploads/6518b2026194ffbbf02e32c307e39f80/new_postman_request.JPG)
- Paste the following curl request in the URL section of the new request
![curl_request](/uploads/56d05fc44b1c81321521921fc9e13fa0/curl_request.JPG)
``` bash
curl --location --request POST 'https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data grant_type="authorization_code" \
--data redirect_uri=http://localhost:8080 \
--data client_id="<client_id>" \
--data client_secret="<client_secret>" \
--data scope="<client_id>/.default openid profile offline_access" \
--data code={{code}}
```
- Sample response
```bash
{
"token_type": "Bearer",
"scope": ".....",
"expires_in": 4557,
"access_token": "eyJ0eXAiOiJKV1QiLCJub25jZSI6IkJuUXdJd0ZFc...",
"refresh_token": "0.ARoAv4j5cvGGr0GRqy180BHbR8lB8cvIWGtHpawGN..."
}
```
- **Step 2.4:** Save **access_token** , **refresh_token** in your local postman environment
- **Step 3.1:** Test couple of OSDU services with this **access_token** to make sure you can access the OSDU environment with newly generated token.
---
## Common issues one can face during this process
1. Not replacing placeholders ( **<tenant_id>**, **< code >**, **<client_id>** and **<clent_secret>** )with correct values in the URL/Curl request
**Resolution:** Double check url and curl request and make sure correct values are being replaced in the placeholders.
2. **< code >** generated during step 2.2 will expire after 1 hour. Follow the process again from Step 1.1 if the **< code >** expires.
3. It is recommended to use postman tool for step 2.3 rather than git bash, windows cmd, etc.
4. It is expected to face error "Hmmm...can't reach this page/ Can't found the URL" in the browser for step 2.1https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/693Pre Shipping GCP (https://preship.gcp.gnrg-osdu.projects.epam.com) search ser...2024-03-12T05:40:46ZRakesh SharmaPre Shipping GCP (https://preship.gcp.gnrg-osdu.projects.epam.com) search service failing with 'Tenant configuration issue'Hi,
GCP pre shipping instance https://preship.gcp.gnrg-osdu.projects.epam.com search service is failing with Internal Server Error and following is the response
{"code":500,"reason":"Tenant configuration issue.","message":"Tenant config...Hi,
GCP pre shipping instance https://preship.gcp.gnrg-osdu.projects.epam.com search service is failing with Internal Server Error and following is the response
{"code":500,"reason":"Tenant configuration issue.","message":"Tenant configuration issue."}
We are using service account and questlabs@osdu-service-prod.iam.gserviceaccount.com is the client email id.https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/165ADR - Entitlements - Additional attributes in list groups on behalf of API2024-03-29T05:41:50ZEirik HaughomADR - Entitlements - Additional attributes in list groups on behalf of API# ADR - Entitlements - Expose ROLE in list all groups API
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The `/members/<member_email>/groups` API allows to query entitlement ...# ADR - Entitlements - Expose ROLE in list all groups API
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The `/members/<member_email>/groups` API allows to query entitlement group memberships for a specific user.
```
"groups": [
{
"name": "service.search.user",
"description": "Datalake Search users",
"email": "service.search.user@opendes.contoso.com"
},
{
"name": "users.datalake.ops",
"description": "Datalake ops",
"email": "users.datalake.ops@opendes.contoso.com"
}
```
## Decision
Expand response with the attribute `ROLE` for each group. Example:
```
"groups": [
{
"name": "service.search.user",
"description": "Datalake Search users",
"email": "service.search.user@opendes.contoso.com",
"role": "OWNER"
},
{
"name": "users.datalake.ops",
"description": "Datalake ops",
"email": "users.datalake.ops@opendes.contoso.com",
"role": "MEMBER"
}
```
## Rationale
Increase governance capabilities.
Improve OSDU Admin UI which now uses statically configured "MEMBER" as role for each group, even though that is not the true role:
```
private getUserGroups(type: string = 'none') {
this.restService
.getUsersAccessRights(this.user.id, type)
.subscribe((groups: OsduGroup[]) => {
console.log('Got ', groups.length, ' groups');
this.groups = groups.map((g) => ({
email: g.email,
role: 'MEMBER',
}));
this.filterGroups(this.searchControl.value);
console.log(this.filteredGroups);
});
console.log(this.filteredGroups);
}
```https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/156Augmented Index - parent-child use case - trailing colon (":") in relationshi...2024-03-08T16:00:02ZDebasis ChatterjeeAugmented Index - parent-child use case - trailing colon (":") in relationship field in child recordSee my test case in AWS/M22/Preship.
https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/Test_plan_Results_M22/Core%20Services/M22-AWS-Augmented-Index-parent-child-steps-Debasis.docx
This test case involves "We...See my test case in AWS/M22/Preship.
https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/Test_plan_Results_M22/Core%20Services/M22-AWS-Augmented-Index-parent-child-steps-Debasis.docx
This test case involves "Well" (parent) and "Wellbore" (child).
When troubleshooting with @zhibinmai , we tried by removing trailing colon from relationship field of child record Wellbore.
With that change, I could get clean run and "virtual field" appears in search response as expected.
In child Wellbore record -
"WellID": "osdu:master-data--Well:WELL07MARDC"
whereas it should be
"WellID": "osdu:master-data--Well:WELL07MARDC:"
(this is the convention when referencing field from another entity - like "foreign key" relationship)
I earlier ran similar test case in Azure and did not see the impact of trailing colon there.
@ydzeng - how to ensure feature parity between version of code (for augmented index) in M22/AWS/Preship with other CSPs?
cc @chad and @sjtomlinsonhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/692Augmented Index - Use Case 2 (Country Names) not working in M22 Preship Envir...2024-03-07T11:28:12ZNorman MedinaAugmented Index - Use Case 2 (Country Names) not working in M22 Preship EnvironmentI was testing out the augmented index feature on the M22 Preship environment. I was trying to implement the use cases documented in this [tutorial](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/docs/t...I was testing out the augmented index feature on the M22 Preship environment. I was trying to implement the use cases documented in this [tutorial](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/docs/tutorial/IndexAugmenter.md#use_cases). Use cases 1 and 5 worked for me. Use case 2 did not, as the field `CountryNames` wasn't coming out of search after reindexing.
I also tried this on `osdu:wks:master-data--Wellbore:1.0.0` but replaced the item in `RelatedConditionMatches` with `^[\\w\\-\\.]+:reference-data--GeoPoliticalEntityType:Province:$` and the Name to `ProvinceNames`, but the custom `ProvinceNames` field is not appearing.
Please see below the reference data I used:
```
[
{
"acl": {
"owners": [
"{{New_OwnerDataGroup}}@{{data-partition-id}}{{domain}}"
],
"viewers": [
"{{New_ViewerDataGroup}}@{{data-partition-id}}{{domain}}"
]
},
"legal": {
"legaltags": [
"{{LegalTagNameExists}}"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"meta": [],
"data": {
"Code": "osdu:wks:master-data--Wellbore:1.",
"Configurations": [
{
"Name": "ProvinceNames",
"Policy": "ExtractAllMatches",
"Paths": [
{
"RelatedObjectsSpec": {
"RelatedObjectID": "data.GeoContexts[].GeoPoliticalEntityID",
"RelatedObjectKind": "osdu:wks:master-data--GeoPoliticalEntity:1.",
"RelatedConditionMatches": [
"^[\\w\\-\\.]+:reference-data--GeoPoliticalEntityType:Province:$"
],
"RelatedConditionProperty": "data.GeoContexts[].GeoTypeID"
},
"ValueExtraction": {
"ValuePath": "data.GeoPoliticalEntityName"
}
}
],
"UseCase": "As a user I want to find objects by a province name."
}
]
},
"id": "{{data-partition-id}}:reference-data--IndexPropertyPathConfiguration:wks:master-data--Wellbore:1.",
"kind": "osdu:wks:reference-data--IndexPropertyPathConfiguration:1.0.0",
"version": 0
}
]
```https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/691Augmented Index - Use Case 3 (Wellbore Name) not working in M22 Preship Envir...2024-03-07T11:28:19ZNorman MedinaAugmented Index - Use Case 3 (Wellbore Name) not working in M22 Preship EnvironmentI was testing out the augmented index feature on the M22 Preship environment. I was trying to implement the use cases documented in this [tutorial](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/docs/t...I was testing out the augmented index feature on the M22 Preship environment. I was trying to implement the use cases documented in this [tutorial](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/docs/tutorial/IndexAugmenter.md#use_cases). Use cases 1 and 5 worked for me. Use case 3 did not, as the field `WellboreName` wasn't coming out of search after reindexing. I tried testing this out three times, but it still didn't work.
I used the snippet that was provided in the tutorial page and didn't modify anything in it.https://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui/-/issues/39Export - Select data (via search or from a map) & export metadata index (in t...2024-03-14T11:44:57ZAndrew FlackExport - Select data (via search or from a map) & export metadata index (in tabular or json format)**I want to**...Be able to search and find data and select **multiple datasets** to export in a particular file format
**So that**...I may analyse this outside of the tool**I want to**...Be able to search and find data and select **multiple datasets** to export in a particular file format
**So that**...I may analyse this outside of the toolhttps://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui/-/issues/38Data Quality - Duplication check within OSDU core catalogue and DDMS2024-03-14T10:49:12ZAndrew FlackData Quality - Duplication check within OSDU core catalogue and DDMS**I want to**...Query for duplicate data items stored within the Core Catalogue or within an DDMS
**So that**...I can resolve the duplication issues using this UI.**I want to**...Query for duplicate data items stored within the Core Catalogue or within an DDMS
**So that**...I can resolve the duplication issues using this UI.https://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui/-/issues/37Data Quality - Identify records in DDMS that conflict with (or are orphaned f...2024-03-14T10:51:24ZAndrew FlackData Quality - Identify records in DDMS that conflict with (or are orphaned from) records in Core Catalogue**I want to**...Identify records in a DDMS that have no corresponding record within the Core Catalogue
**So that**...I can remedy the issue or validate that the data ought not to have a core catalogue entry.**I want to**...Identify records in a DDMS that have no corresponding record within the Core Catalogue
**So that**...I can remedy the issue or validate that the data ought not to have a core catalogue entry.https://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui/-/issues/36Data Quality - Retrospectively assign & edit the Technical Assurance associat...2024-03-14T11:47:06ZAndrew FlackData Quality - Retrospectively assign & edit the Technical Assurance associated to a data record.**I want to**...Be able to retrospectively update an incorrectly assigned tech assurance value or update it's assurance value
**So that**...All key data in OSDU is tagged with Technical Assurance and that this assurance is kept current ...**I want to**...Be able to retrospectively update an incorrectly assigned tech assurance value or update it's assurance value
**So that**...All key data in OSDU is tagged with Technical Assurance and that this assurance is kept current and correct.https://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui/-/issues/35Data Quality - Define Tech Assurance Rules / Criteria / Personnas2024-03-14T11:47:56ZAndrew FlackData Quality - Define Tech Assurance Rules / Criteria / Personnas**I want to**...Be able to define technical assurance terms, personnas , criteria to assign to data
**So that**...An end user understands what the data can or cannot be used for (and therefore trusts the data)**I want to**...Be able to define technical assurance terms, personnas , criteria to assign to data
**So that**...An end user understands what the data can or cannot be used for (and therefore trusts the data)https://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui/-/issues/34Visualisation - Visualise OSDU Entity Relationship Diagram/ERD or equivalent2024-03-14T11:48:19ZAndrew FlackVisualisation - Visualise OSDU Entity Relationship Diagram/ERD or equivalent**I want to**...Be able to easily visualise the OSDU ERD for a selected data type
**So that**...I can easily understand any interdependency of schemas/Master/Reference data**I want to**...Be able to easily visualise the OSDU ERD for a selected data type
**So that**...I can easily understand any interdependency of schemas/Master/Reference datahttps://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui/-/issues/33Visualisation - Visualise connections between data in a DDMS & it's associate...2024-03-14T11:48:45ZAndrew FlackVisualisation - Visualise connections between data in a DDMS & it's associated record in the core catalogue**I want to**...Easily identify records in a DDMS that a connection/are the same records as data held within the Core Catalogue (ideally via a UI)
**So that**...I can be clear on the relationship between the data held in a DDMS and it's...**I want to**...Easily identify records in a DDMS that a connection/are the same records as data held within the Core Catalogue (ideally via a UI)
**So that**...I can be clear on the relationship between the data held in a DDMS and it's association to the wider catalogue of data held within OSDU.https://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui/-/issues/32Visualisation - UI to understand Ancestry (identifies child/parent relationsh...2024-03-14T11:49:32ZAndrew FlackVisualisation - UI to understand Ancestry (identifies child/parent relationship and their legal tags) and Lineage Assertion (identifies Direct, Indirect & Reference relationships to derivates)**I want to**...Be able to easily determine the origin of each data item
**So that**...I can easily understand how each data item is related to others/where it was derived from (so that I may trust the data).**I want to**...Be able to easily determine the origin of each data item
**So that**...I can easily understand how each data item is related to others/where it was derived from (so that I may trust the data).https://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui/-/issues/31Visualisation - Graphically display multiple different data types together in...2024-03-14T11:50:18ZAndrew FlackVisualisation - Graphically display multiple different data types together in the same UI**I want to**...Easily find and preview multiple, related data types in the same UI, without the need to toggle between tabs.
**So that**...I can, for example, visualise a well log along side it's log print pdf, or a seismic volume alon...**I want to**...Easily find and preview multiple, related data types in the same UI, without the need to toggle between tabs.
**So that**...I can, for example, visualise a well log along side it's log print pdf, or a seismic volume alongside it's processing report.https://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui/-/issues/30Visualisation - Graphically display the documents/unstructured files held in ...2024-03-14T11:50:46ZAndrew FlackVisualisation - Graphically display the documents/unstructured files held in OSDU in a UI (particularly all MS Office Formats, PDF, csv, JPG/PNG/TIFF)**I want to**...Be able to view documents/unstructured data (e.g. EoWR, Daily Reports, Log Prints, Images, Surveys,etc.)
**So that**...I can quickly QC or review the data available in OSDU in order to edit/delete/export/import/load the...**I want to**...Be able to view documents/unstructured data (e.g. EoWR, Daily Reports, Log Prints, Images, Surveys,etc.)
**So that**...I can quickly QC or review the data available in OSDU in order to edit/delete/export/import/load the data I need.https://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui/-/issues/29Visualisation - Graphically display Seismic data available in OSDU in a UI2024-03-14T11:51:11ZAndrew FlackVisualisation - Graphically display Seismic data available in OSDU in a UI**I want to**...Be able to view seismic data (e.g. a IL/XL/Time-Slice/Depth-Slice, Survey Outlines, Trace Headers, Histogram, etc.) for a given Seismic Survey or Volume within OSDU.
**So that**...I can quickly QC or review the data ava...**I want to**...Be able to view seismic data (e.g. a IL/XL/Time-Slice/Depth-Slice, Survey Outlines, Trace Headers, Histogram, etc.) for a given Seismic Survey or Volume within OSDU.
**So that**...I can quickly QC or review the data available in OSDU in order to edit/delete/export/import/load the data I need.https://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui/-/issues/28Visualisation - Graphically display Well data available in OSDU2024-03-14T11:51:46ZAndrew FlackVisualisation - Graphically display Well data available in OSDU**I want to**...Be able to view well data (e.g. a well trajectory in 3D space, or the well log curves available) for a given well (or multiple wells, for comparison) within OSDU.
**So that**...I can quickly QC or review the data availa...**I want to**...Be able to view well data (e.g. a well trajectory in 3D space, or the well log curves available) for a given well (or multiple wells, for comparison) within OSDU.
**So that**...I can quickly QC or review the data available in OSDU in order to edit/delete/export/import/load the data I need.https://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui/-/issues/27Update/Enrich - Bulk update multiple records2024-03-14T11:52:12ZAndrew FlackUpdate/Enrich - Bulk update multiple records**I want to**...Edit metadata in bulk for multiple existing records in OSDU and commit changes as new version
**So that**...The data is kept up to date.**I want to**...Edit metadata in bulk for multiple existing records in OSDU and commit changes as new version
**So that**...The data is kept up to date.