OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2024-01-18T12:44:05Zhttps://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/issues/38ADR: Provide search capability in legal tags2024-01-18T12:44:05ZSrabana GuhaADR: Provide search capability in legal tags## Context
Is it possible to add the search capability for legal tags based on the legaltag attributes including the ones in extensionProperties? There might be hundreds of thousands of Legaltags. Need to count on the number of legaltag...## Context
Is it possible to add the search capability for legal tags based on the legaltag attributes including the ones in extensionProperties? There might be hundreds of thousands of Legaltags. Need to count on the number of legaltags while designing the solutions.
For more details please check #36
#### Problem Statement
Is it possible to add the search capability for legal tags based on the legaltag attributes including the ones in extensionProperties? There might be hundreds of thousands of Legaltags. Need to count on the number of leagltags while designing the solutions.
#### Potential Solution Approach
There is a thought to implement the solution using Regex. Using Regex seems a pretty simple approach both for usage and implementation. "offset" and "sort" will still require further discussion and can be implemented in iteration.
This solution calls for retrieving each legal tag document from the datastore which might cause a potential performance issue.
Either ES can be used to index and search for Legaltags documents.
Post our team discussion on May, 9 these are the various approaches that came up:
1. As the underlying architecture or Legal Service is specific to CSPs, we need to implement the search specific to CSP. in AWS, the legal datastore in DynamoDB with Mongo DB as an additional config option. In Azure it's the Cosmos Datastore. Google stores as an Object Datastore. For AWS, the DynamoDB table currently has no index. In that case, full table scan is the only option to run search on legal tags. While this may seem easiest approach solution wise, there might come performance issue if the legal tags grow potentially higher in number. Whether we should go for this approach would depend on the POC being conducted on the level of hundreds and thousands of legaltags inn Dynamo.
1. Next approach could be to add in index to the datastore. This might have to be considered if there is considerable performance overhead found in the POC. Details of the POC provided below. This might not be the quickest one to implement considering different CSPs that need to be involved while creating the legal tag datastore table.
1. Another approach is to totally delegate the search operation of the Legaltags to something like ElasticSearch. In that case need to following the same as Storage service and Search service.
1. Use the List LegalTags API to fetch the legal tags record and search on each of the Tags. After initial analysis we need to work more on search on each of these LegalTag objects. Post detailed requirements discussion it was observed that complex search operations need to be supported. To satisfy the requirement it's better to use a service which supports search. Implementing in application tier is not a feasible option.
#### Tasks
* Identify the persistent layer of the legal tag service - Identified. CSP specific.
* run POC to implement different use cases so as to justify the decision which should be undertaken for the solution - Ongoing
* Run the Legal Service locally.
* Run PoC on Elastic Search to index legal tags and then execute search on legal tags.
#### POC
POC is being implemented to fetch legal tags from the datastore. Currently it is being conducted considering Dynamo as the AWS Implementation. The upper limit to be considered is few hundreds of thousands of data store. Currently this is a standalone Java application.
Next phase of POC would involve invoking OSDU Core API Services to retrieve > 10k records from Dynamo DB and noting the performance.
Another scenario to consider once data records are retrieved how is the search especially on extension properties tag coming along.
#### Open Questions and answers based on discussions on May 25, 2023
* Structure of ExtensionProperties attributes has no definitive format. It will be different for different companies. Shell might follow a structure which might be different for other companies. So extensionproperties need to be pretty much flexible
* The formatting of all LegalTags are pretty much standardized. We are referring to the structure outside of the extensionproperties. Currently, it’s having Name, Description, properties, isValid attributes other that id and dataPartitionId
* Search should support all attribute search in legaltags
* Expected new API to have POST request with body which will contain the query. Query can be pretty complex.
* Mostly the feature will be available as a separate API under legal services.
* Multi- attribute search along with complex queries
* Sort and Search together need to be supported along with limit and offset
* The response should return the entire legalTag with all attributes.
* DSL preferred to Lucene syntax – need to revisit this implementationwise.
#### Findings
We have implemented a POC to run a full DynamoDB table scan. Currently the Dynamo DB is deployed I M16 environment ad the POC is being run from local laptop. This is only for the AWS implementation. We have still not researched the idea for other CSPs. We are running different scenarios and recording the time the POC is taking. The scenario
| Scenario | Time Recorded |
|----------|---------------|
| Full table scan. Currently table has 10,528 items | < 0.020Sec |
| Full table scan along with filter condition (isValid = 1) | <0.020sec |
| Program involving full table scan along with flyer condition and comparing values in the entire table. Example search for 'US' under properties.countryOfOrigin. Table scan time remains same. | Program execution time: 18.2sec |
| Program involving full table scan along with flyer condition and comparing values from extension properties attribute in the entire table. Example search for 'RDS' under properties.extensionproperties.AgreementParty. Table scan time remains same. This includes the user input time for the search pattern(as "RDS") and search attribute (as properties.extensionproperties.AgreementParty) | Program execution time: \~14sec |
| Executed the OSDU API (List LegalTags) to retrieve the legaltags. Total program execution time includes establishing the connection to the remote API, and printing the response to standard output. | 3.598Sec |https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-pss-entities-service/-/issues/17osdu-structure-storage-service - healthcheck: redis & JanusGraph2024-01-18T09:48:20ZJozsef Barczaosdu-structure-storage-service - healthcheck: redis & JanusGraphhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-pss-entities-service/-/issues/16osdu-structure-query-service - redis & janusgraph integrated to the healthcheck2024-01-18T09:42:03ZJozsef Barczaosdu-structure-query-service - redis & janusgraph integrated to the healthcheckhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-pss-entities-service/-/issues/15osdu-production-timeseries-service - redis added to healthcheck2024-01-18T09:40:51ZJozsef Barczaosdu-production-timeseries-service - redis added to healthcheckhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production/historian/services/osdu-pss-entities-service/-/issues/14pps-timeseries-service: add redis to healtcheck page2024-01-18T08:20:51ZJozsef Barczapps-timeseries-service: add redis to healtcheck pagehttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/611M20 Azure RDDMS Performance is 3/4 times slower than M19 while PUSH/PULL Resq...2024-01-18T04:25:35ZSoumik DuttaM20 Azure RDDMS Performance is 3/4 times slower than M19 while PUSH/PULL Resqml filesPUSH / PULL resqml files to Azure RDDMS instance using Open etp ssl client. Try it for M19 and M20 for all the epc files available (https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/reservoir/open-etp-server/-/tree/...PUSH / PULL resqml files to Azure RDDMS instance using Open etp ssl client. Try it for M19 and M20 for all the epc files available (https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/reservoir/open-etp-server/-/tree/main/data?ref_type=heads) and note the time.
An example of PUSH Data (command to be run using open etp ssl client):
docker run -it --rm -v /mnt/d/OES/open-etp-server/data:/data open-etp:ssl-client openETPServer space -S wss://${RDDMS_URL} -s **demo/volve_wells** --import-epc /data/**Volve_Demo_Wells_Depth.epc** --data-partition-id ${PARTITION} --auth bearer --jwt-token ${TOKEN} --time
An example of PULL Data (command to be run using open etp ssl client):
docker run -it --rm -v /mnt/d/OES/Reservoir_DMS_Suite/open-etp-server-main/data:/data open-etp:ssl-client openETPServer space -S wss://${RDDMS_URL} -s demo/volve_wells --export-epc /data/demo_volve_wells.epc --data-partition-id ${PARTITION} --auth bearer --jwt-token ${TOKEN} --time
In bold are the dataspace created and the resqml file used
"https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/reservoir/open-etp-server" One can consult this page t know more about the commands used to PUSH / PULL data.Soumik DuttaSoumik Duttahttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/issues/125Validation of SampleAnalysisTypeIDs array values in SamplesAnalysis and Sampl...2024-01-17T22:03:08ZMichael JonesValidation of SampleAnalysisTypeIDs array values in SamplesAnalysis and SamplesAnalysesReport against SampleAnalysisType reference-data- SamplesAnalysis Work Product Components contain an array object 'SampleAnalysisTypeIDs.'
- SamplesAnalysesReport Work Product Components also contain this array.
- The values within this array in both WPCs must be validated against val...- SamplesAnalysis Work Product Components contain an array object 'SampleAnalysisTypeIDs.'
- SamplesAnalysesReport Work Product Components also contain this array.
- The values within this array in both WPCs must be validated against values within the Reference Data manifest 'SampleAnalysisType.'
[Unified Sample](https://gitlab.opengroup.org/osdu/subcommittees/data-def/projects/Petrophysics/docs/-/tree/master/Design%20Documents/UnifiedSample_brainstorming_proposals)
- If found and valid, proceed with ingest/load.
- If there is no value provided, stop ingest/load and provide error to user
- If value is not found within the reference list, stop ingest/load and provide error to user.
NOTE: The attached SampleAnalysisType reference list is a condensed version of the list currently in discussion within OSDU and as such will in the future need to be updated/replaced, and values will need to be validated again.RAFS DDMS Sprint 13Ernesto GutierrezErnesto Gutierrezhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/issues/310Add Reference List for "VitriniteMountTechnique"2024-01-17T21:35:35ZMichael JonesAdd Reference List for "VitriniteMountTechnique"VitriniteMountTechnique was identified as needing a reference list, but is currently free text.
Values have been identified and the list has been drafted at the link below
https://gitlab.opengroup.org/osdu/subcommittees/data-def/projec...VitriniteMountTechnique was identified as needing a reference list, but is currently free text.
Values have been identified and the list has been drafted at the link below
https://gitlab.opengroup.org/osdu/subcommittees/data-def/projects/RAFSDDMSDEV/docs/-/blob/main/Design%20Documents/ReferenceValues/Manifests/reference-data/OPEN/VitriniteMountTechnique.json
We also need to modify existing Vitrinite Reflectance content schema attribute **'VitriniteMountTechnique'** to
```json
"VitriniteMountTechnique": {
"type": "string",
"description": "Preparation technique for vitrinite mount"
"pattern": "^[\\w\\-\\.]+:reference-data\\-\\-VitriniteMountTechnique:[\\w\\-\\.\\:\\%]+:[0-9]*$"
},
```Ernesto GutierrezErnesto Gutierrezhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/665Azure: CERT DATE INVALID2024-01-17T20:56:41ZChad LeongAzure: CERT DATE INVALIDVisiting any [Swagger API doc](https://community.opengroup.org/groups/osdu/platform/-/wikis/Core-Services-API-Docs) page will result in NET::ERR_CERT_DATE_INVALID
Example: https://osdu-ship.msft-osdu-test.org/api/entitlements/v2/swagger...Visiting any [Swagger API doc](https://community.opengroup.org/groups/osdu/platform/-/wikis/Core-Services-API-Docs) page will result in NET::ERR_CERT_DATE_INVALID
Example: https://osdu-ship.msft-osdu-test.org/api/entitlements/v2/swagger-ui/index.html
![image](/uploads/1fde24217984c922d0db6af8f83b707d/image.png)M22 - Release 0.25Srinivasan NarayananSrinivasan Narayananhttps://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/318CI/CD - Produce .tgz artefact for GCZ Provider node dependencies2024-01-17T15:52:24ZLevi RemingtonCI/CD - Produce .tgz artefact for GCZ Provider node dependenciesLevi RemingtonLevi Remingtonhttps://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/319Update Recommended Java Version2024-01-17T15:52:18ZLevi RemingtonUpdate Recommended Java VersionIn GCZ Transformer docs, we recommend JDK11 Java Version. However, our pom.xml compiler, the docker images, and OSDU Core Services all use Java 1.8 / JDK8. In addition, we've found that recent versions of JDK11 (`11.0.21`) have presented...In GCZ Transformer docs, we recommend JDK11 Java Version. However, our pom.xml compiler, the docker images, and OSDU Core Services all use Java 1.8 / JDK8. In addition, we've found that recent versions of JDK11 (`11.0.21`) have presented error during runtime. For these reasons we have decided to switch our recommended Java version from JDK11 to JDK8. This should have minimal to no impact, given that deployments thus far have leveraged JDK8 or earlier non-problematic versions of JDK11.Levi RemingtonAnkita SrivastavaLevi Remingtonhttps://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/316Fix generateGeoSON in Search2024-01-17T15:52:10ZAnkita SrivastavaFix generateGeoSON in SearchCurrently Search.generateGeoJSON is loading some properties with assumption of certain keys availability. In case of key absence, it results in Null Pointer Exception and that record is skipped.
Acceptance:
If any key is unavailable, s...Currently Search.generateGeoJSON is loading some properties with assumption of certain keys availability. In case of key absence, it results in Null Pointer Exception and that record is skipped.
Acceptance:
If any key is unavailable, skip that property but still load the recordGCZ Sprint 56 & 57Ankita SrivastavaAnkita Srivastavahttps://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/293Documentation - Create Documents for GCZ Security Pattern2024-01-17T15:52:07ZNoel OkanyaDocumentation - Create Documents for GCZ Security PatternAs a GCZ Product Owner, I want to understand potential GCZ Security Patterns, so that users can be aware of security options.
Acceptance Criteria:
- How to consume GCZ
- Roles, Privileges, and Responsibilities
- WorkaroundsAs a GCZ Product Owner, I want to understand potential GCZ Security Patterns, so that users can be aware of security options.
Acceptance Criteria:
- How to consume GCZ
- Roles, Privileges, and Responsibilities
- WorkaroundsBrianAnkita SrivastavaDavid JacobBrianhttps://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/304Development Environment Discussion2024-01-17T15:52:04ZNoel OkanyaDevelopment Environment DiscussionAs a GCZ Product Owner, I want to understand how we can provide a sustainable development environment that is continually available for the GCZ dev team, including the extra data provided by SLB (New Zealand). The goal is to meet and dis...As a GCZ Product Owner, I want to understand how we can provide a sustainable development environment that is continually available for the GCZ dev team, including the extra data provided by SLB (New Zealand). The goal is to meet and discuss how other OSDU dev teams are meeting this need.
Attendees:
Ankita, Brian, Levi, Michael, Debasis, ChadDebasis ChatterjeeBrianDebasis Chatterjeehttps://community.opengroup.org/osdu/platform/system/home/-/issues/97ADR - Swagger: Using springdoc-openapi for swaggers APIs2024-01-17T15:28:41ZRene von Borstel [EPAM]ADR - Swagger: Using springdoc-openapi for swaggers APIs## Decision Title
Swagger Sanity Phase 2: Using springdoc-openapi for swaggers APIs
## Status
- [x] Proposed
- [x] Approved
- [ ] Implementing (incl. documenting)
- [ ] Testing
- [ ] Released
## Context
- The swagger APIs are maintain...## Decision Title
Swagger Sanity Phase 2: Using springdoc-openapi for swaggers APIs
## Status
- [x] Proposed
- [x] Approved
- [ ] Implementing (incl. documenting)
- [ ] Testing
- [ ] Released
## Context
- The swagger APIs are maintained as part of each service, for instance [Storage Swagger Controller](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/storage-core/src/main/java/org/opengroup/osdu/storage/swagger/HomeController.java).
- The swagger doc is handwritten and manually maintained, for instance [Storage Service Swagger](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/api/storage_openapi.yaml).
## Assumption
The swagger for all services is in version 3 already.
## Problem statement
The cost of maintenance of the swagger doc is high. We have stale swaggers in the system already.
## Proposed solution
We have the following frameworks that will help lower the cost of upkeep of swagger.
1. ```springdoc-openapi```
2. ```springdoc-openapi-webflux-ui```
## Scope / Acceptance Criteria
The above effort will encapsulate the following
1. **Controller, Model improvement to retain the swagger information that is present today.**
2. The swagger endpoint path will be a consistent as we see it [today](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/storage-core/src/main/java/org/opengroup/osdu/storage/swagger/HomeController.java#L24).
3. The deprecation of the swagger controllers we maintain should be seamless experience for the user.
# Target Release
TBD
# FAQ
**1. How will we manage migration to next generation of swaggers, when available. **
We will count on Spring Boot's adaptation of the next generation to be back compatible.
**2. Will there be regressions in the existing experience? **
No, as specified in the Scope.
## Useful references
1. https://www.baeldung.com/spring-rest-openapi-documentation
2. Adding annotations that will contain the description which the current Swagger has. https://www.baeldung.com/swagger-set-example-description#1-add-description-to-methods-and-parameters
# Owner
Please contact @komakkarM16 - Release 0.19Komal MakkarKomal Makkarhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/58Incorrect handling of Geospatial fields causes EDS-ingested records to not ma...2024-01-17T15:19:18ZBen GreerIncorrect handling of Geospatial fields causes EDS-ingested records to not match schema and not have geospatial metadata indexedWhen a record is ingested through EDS Fetch & Ingest process, the Spatial Point Wgs84 Coordinates fields are not transformed as expected to match the schema representation for spatial locations. This causes the records ingested through E...When a record is ingested through EDS Fetch & Ingest process, the Spatial Point Wgs84 Coordinates fields are not transformed as expected to match the schema representation for spatial locations. This causes the records ingested through EDS Fetch & Ingest to not have their geospatial fields at the same JSON property path as regularly ingested records.
The issue is related to Search service flattening certain fields and EDS not undoing it when creating the manifests for ingestion.
See M22 testing writeup here: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/Test_plan_[…]22-EDS-testing-aws-spatial-fields-bfg.docx?ref_type=headsM23 - Release 0.26Ashish SaxenaAshish Saxenahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/666RI Ingestion by Reference failed ---> 06 - Workflow - Get Work Status for mas...2024-01-17T14:42:46ZRanjith KumarRI Ingestion by Reference failed ---> 06 - Workflow - Get Work Status for master dataIngestion By Reference CI-CD v3.0
06 - Workflow - Get Work Status for master data![Screenshot_2024-01-17_104929]
&
06 - Workflow - Get Work Status reference data
(/uploads/1b54c90c75f894e9030975e4b9399bec/Screenshot_2024-01-17_104929.png...Ingestion By Reference CI-CD v3.0
06 - Workflow - Get Work Status for master data![Screenshot_2024-01-17_104929]
&
06 - Workflow - Get Work Status reference data
(/uploads/1b54c90c75f894e9030975e4b9399bec/Screenshot_2024-01-17_104929.png)
The above 2 workflow keep on failing in RI.https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/655IBM M22 - Wellbore DDMS /api/os-wellbore-ddms/about & /api/os-wellbore-ddms/v...2024-01-16T10:26:13ZChad LeongIBM M22 - Wellbore DDMS /api/os-wellbore-ddms/about & /api/os-wellbore-ddms/version```
GET https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/about
GET https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/version
```
Error: getaddrinfo ENOTFOUND cpd-osdu.apps.o...```
GET https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/about
GET https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/version
```
Error: getaddrinfo ENOTFOUND cpd-osdu.apps.osdu-preship.ibmodi.com
are not respondingAnuj Guptavikas ranaAnuj Guptahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/654M22 IBM Unauthorized error when getting upload signed URL in CSV ingestion2024-01-15T19:16:00ZTaylor GraberM22 IBM Unauthorized error when getting upload signed URL in CSV ingestionWhen following step 5 in the csv ingestion collection, I receive the following. Please let me know if there is something I need to do on my side that's causing the issue.
GET {{FILE_HOST}}/files/uploadURL
Response:
{
"error": {
...When following step 5 in the csv ingestion collection, I receive the following. Please let me know if there is something I need to do on my side that's causing the issue.
GET {{FILE_HOST}}/files/uploadURL
Response:
{
"error": {
"code": 401,
"message": "Unauthorized",
"errors": [
{
"domain": "global",
"reason": "unauthorized",
"message": "Unauthorized"
}
]
}
}
Test Results:
Couldn't evaluate the test script:
TypeError: Cannot read properties of undefined (reading 'SignedURL')
cURL:
curl --location 'https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-file/api/file/v2/files/uploadURL' \
--header 'data-partition-id: opendes' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJ2UmhTM0hBSnB5TU5jSTkwMko2UjR0R0tYYk0zTE94UWhSdkNIMFN5RDJvIn0.eyJleHAiOjE3MDQ5OTUyOTEsImlhdCI6MTcwNDk4NDQ5MSwianRpIjoiMzhmZWYxOGQtMWUxNy00NDJhLWI4MzAtOGMxYjk3YTZhOGY5IiwiaXNzIjoiaHR0cHM6Ly9rZXljbG9hay1vc2R1LmFwcHMub3NkdS1wcmVzaGlwLmlibW9kaS5jb20vYXV0aC9yZWFsbXMvT1NEVSIsInN1YiI6ImE5OWFmMTEzLTBkODktNDI1Ny1iZDIwLWE5Mzg0OWI4MjkwYSIsInR5cCI6IkJlYXJlciIsImF6cCI6Im9zZHUtbG9naW4iLCJzZXNzaW9uX3N0YXRlIjoiNzY3ZDYzZDctYjFjOC00MzQwLWFjZTQtZmU3OGNjYTVmNzJmIiwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbImRlZmF1bHQtcm9sZXMtb3NkdSJdfSwic2NvcGUiOiJvcGVuaWQgZW1haWwgcHJvZmlsZSIsInNpZCI6Ijc2N2Q2M2Q3LWIxYzgtNDM0MC1hY2U0LWZlNzhjY2E1ZjcyZiIsInJvb3RVc2VyIjp0cnVlLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsIm5hbWUiOiJvc2R1IGJ2dCIsInByZWZlcnJlZF91c2VybmFtZSI6Im9zZHUtYnZ0IiwiZ2l2ZW5fbmFtZSI6Im9zZHUiLCJmYW1pbHlfbmFtZSI6ImJ2dCIsImVtYWlsIjoib3NkdS1idnRAb3NkdS5vcGVuZ3JvdXAub3JnIn0.Snc2NEovp1pNXZIuV_gnfC_GxRJ0tNQv7rNY6hiqKc0WI_H7q7HeYyOy_wtGPzw1Na_30rKi4pnCVE8JA3tBYR6NXI8p8vbFQ3mNRt0u7UXFk8-EzOA_hCiYnyEq69lXLMcbHB2MJYxds_D8GsqW_QFV7YvG1OXxBGQ3KnHuQMks1VNU6Pci3MLgujUI5B-NPg3MYNQzzXpQaVNCpZgsFedEXL1I0RJKxtzcGpC3JvvCbWE5S0w8__TssgNeWIeqgVzgVWgq61DwIEne_mxYevn6OMDHssbiZ5gag9KlZGlsN8lKBRD5j1JxSJeT5ILwKj6f-vpxi8v1ZUVbc4W3eg' \
--header 'Cookie: 5e7c4a992d2558462d1c9fcde6f1cee7=4ec498d778b559cf74a987cfbc06b21b'M22 - Release 0.25https://community.opengroup.org/osdu/platform/system/search-service/-/issues/106ADR: Additional attribute in Sort query to filter the records2024-01-15T11:54:09ZMandar KulkarniADR: Additional attribute in Sort query to filter the recordsAdditional attribute in Sort query to filter the records
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
The search service accepts sort query as of now where the caller can s...Additional attribute in Sort query to filter the records
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
The search service accepts sort query as of now where the caller can specify 2 attributes.
- field : This is a list of fields to sort the results.
- order : This is a list of orders to sort the results. This requires fix value as either ASC or DESC.
More can be read from this search [documentation](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/tutorial/SearchService.md#sort).
The SortQuery model supported in the search can be seen [here](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/model/search/SortQuery.java).
Elasticsearch supports filter attribute as well inside the sort field to filter the objects inside nested path, please refer the ES documentation [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/sort-search-results.html#_nested_sorting_examples).
In the absence of support for 'filter' attribute in OSDU search, we have a limitation to sort results of nested fields as mentioned in this [issue](https://community.opengroup.org/osdu/platform/system/search-service/-/issues/101).
## Tradeoff Analysis
This will be a non-breaking change and a feature supported in Elasticsearch will be supported with OSDU search as well.
## Decision
The /query and /query_with_cursor APIs in search service accepts a SortQuery model like below:
```
SortQuery:
type: object
properties:
field:
type: array
description: 'The list of fields to sort the results.'
items:
type: string
order:
type: array
description: 'The list of orders to sort the results. The element must be either ASC or DESC.'
items:
type: string
```
The proposal is to add an **optional** new attribute called 'filter' in the SortQuery model in OSDU search service. The filter string passed by the caller will be passed to underneath Elasticsearch.
If the filter string is not passed in the incoming request, existing behavior shall be maintained.
The 'filter' string will be used to filter records while searching and sorting the records based on attributes inside nested objects.
```
SortQuery:
type: object
properties:
field:
type: array
description: 'The list of fields to sort the results.'
items:
type: string
order:
type: array
description: 'The list of orders to sort the results. The element must be either ASC or DESC.'
items:
type: string
filter:
type: array
description: 'A filter that the inner objects inside the nested path should match with, in order for its field values to be taken into account by sorting.'
items:
type: string
```
Below is a sample query to sort records of kind osdu:wks:master-data--Wellbore:1.0.0.
The sorting is based on the values from VerticalMeasurements array where the VerticalMeasurementTypeID matches with the value in filter.
```
{
"kind": "osdu:wks:master-data--Wellbore:1.0.0",
"sort": {
"field": [
"nested(data.VerticalMeasurements, VerticalMeasurement, min)"
],
"order": [
"ASC"
],
"filter":["nested(data.VerticalMeasurements, VerticalMeasurementTypeID:\"tenant1:reference-data--VerticalMeasurementType:KB\", match)"]
}
}
```
This query will be transformed into an elastic search query like below:
```
"sort" : [
{
"data.VerticalMeasurements.VerticalMeasurement" : {
"mode" : "min",
"order" : "asc",
"nested": {
"path": "data.VerticalMeasurements",
"filter": {
"match" : { "data.VerticalMeasurements.VerticalMeasurementTypeID" : "tenant1:reference-data--VerticalMeasurementType:KB" }
}
}
}
}
]
```
The path for the filter would be taken from the nested field inside the 'filter'.
## Consequences
- Change in core-common to update the [SortQuery](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/model/search/SortQuery.java) model.
- Change in [search-core](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/search-core/src/main/java/org/opengroup/osdu/search/util/SortParserUtil.java#L94) to set filter in nested sort queries.
- Search service documentation and Open API specs need to be updated.M20 - Release 0.23Chad LeongMark ChanceChad Leong