Search issueshttps://community.opengroup.org/osdu/platform/system/search-service/-/issues2023-05-25T09:06:52Zhttps://community.opengroup.org/osdu/platform/system/search-service/-/issues/119Improve bad response error messages2023-05-25T09:06:52ZMichaelImprove bad response error messagesA generic "Invalid parameters were given on search request" is given when the records requested exceed the 10,000 limit. For example the following request:
```
curl --location --request POST 'https://r3m15.preshiptesting.osdu.aws/api/se...A generic "Invalid parameters were given on search request" is given when the records requested exceed the 10,000 limit. For example the following request:
```
curl --location --request POST 'https://r3m15.preshiptesting.osdu.aws/api/search/v2/query' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer ...' \
--data-raw '{
"kind": "osdu:wks:master-data--Well:*",
"returnedFields": [
"id",
"kind",
"data.FacilityID",
"data.FacilityName",
"data.SpatialLocation.Wgs84Coordinates"
],
"sort": {
"field": [
"id"
],
"order": [
"ASC"
]
},
"limit": 1000,
"offset": 9001
}'
```
Response:
`{
"code": 400,
"reason": "Bad Request",
"message": "Invalid parameters were given on search request"
}`
For this case it would be good to have a specific error message like "limit and offset parameters exceeds maximum limit".https://community.opengroup.org/osdu/platform/system/search-service/-/issues/101Sorting on nested array attributes2023-06-08T12:55:46ZMandar KulkarniSorting on nested array attributesI have read this [documentation](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/tutorial/ArrayOfObjects.md#sort)and this [ADR](https://community.opengroup.org/osdu/platform/system/search-service/-/...I have read this [documentation](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/tutorial/ArrayOfObjects.md#sort)and this [ADR](https://community.opengroup.org/osdu/platform/system/search-service/-/issues/38) related to sorting.
The question I have is do we support sorting of records based on particular elements in the array with a filter condition?
We have a use case to sort the Wellbore ("osdu:wks:master-data--Wellbore:1.0.0") records based on the VerticalMeasurement value of type KB.
VerticalMeasurements is an array in the data block in Wellbore records. Each record can have an array of VerticalMeasurements
So if I have 5 records which contain values like below for VerticalMeasurements:
```
"id":"wellbore1",
"VerticalMeasurements": [
{
"VerticalMeasurementID": "well header elevation KB",
"VerticalMeasurementPathID": "tenant1:reference-data--VerticalMeasurementPath:ELEV:",
"VerticalMeasurement": 10,
"VerticalMeasurementTypeID": "tenant1:reference-data--VerticalMeasurementType:KB:"
},
{
"VerticalMeasurementID": "well header elevation GL",
"VerticalMeasurementPathID": "tenant1:reference-data--VerticalMeasurementPath:ELEV:",
"VerticalMeasurement": 50,
"VerticalMeasurementTypeID": "tenant1:reference-data--VerticalMeasurementType:GR:",
}]
"id":"wellbore2",
"VerticalMeasurements": [
{
"VerticalMeasurementID": "well header elevation KB",
"VerticalMeasurementPathID": "tenant1:reference-data--VerticalMeasurementPath:ELEV:",
"VerticalMeasurement": 20,
"VerticalMeasurementTypeID": "tenant1:reference-data--VerticalMeasurementType:KB:"
},
{
"VerticalMeasurementID": "well header elevation GL",
"VerticalMeasurementPathID": "tenant1:reference-data--VerticalMeasurementPath:ELEV:",
"VerticalMeasurement": 40,
"VerticalMeasurementTypeID": "tenant1:reference-data--VerticalMeasurementType:GR:",
}]
"id":"wellbore3",
"VerticalMeasurements": [
{
"VerticalMeasurementID": "well header elevation KB",
"VerticalMeasurementPathID": "tenant1:reference-data--VerticalMeasurementPath:ELEV:",
"VerticalMeasurement": 30,
"VerticalMeasurementTypeID": "tenant1:reference-data--VerticalMeasurementType:KB:"
},
{
"VerticalMeasurementID": "well header elevation GL",
"VerticalMeasurementPathID": "tenant1:reference-data--VerticalMeasurementPath:ELEV:",
"VerticalMeasurement": 30,
"VerticalMeasurementTypeID": "tenant1:reference-data--VerticalMeasurementType:GR:",
}]
"id":"wellbore4",
"VerticalMeasurements": [
{
"VerticalMeasurementID": "well header elevation KB",
"VerticalMeasurementPathID": "tenant1:reference-data--VerticalMeasurementPath:ELEV:",
"VerticalMeasurement": 40,
"VerticalMeasurementTypeID": "tenant1:reference-data--VerticalMeasurementType:KB:"
},
{
"VerticalMeasurementID": "well header elevation GL",
"VerticalMeasurementPathID": "tenant1:reference-data--VerticalMeasurementPath:ELEV:",
"VerticalMeasurement": 20,
"VerticalMeasurementTypeID": "tenant1:reference-data--VerticalMeasurementType:GR:",
}]
"id":"wellbore5",
"VerticalMeasurements": [
{
"VerticalMeasurementID": "well header elevation KB",
"VerticalMeasurementPathID": "tenant1:reference-data--VerticalMeasurementPath:ELEV:",
"VerticalMeasurement": 50,
"VerticalMeasurementTypeID": "tenant1:reference-data--VerticalMeasurementType:KB:"
},
{
"VerticalMeasurementID": "well header elevation GL",
"VerticalMeasurementPathID": "tenant1:reference-data--VerticalMeasurementPath:ELEV:",
"VerticalMeasurement": 10,
"VerticalMeasurementTypeID": "tenant1:reference-data--VerticalMeasurementType:GR:",
}]
```
then we want to sort the records based on value of data.VerticalMeasurements.VerticalMeasurement where data.VerticalMeasurement.VerticalMeasurementTypeID is tenant1:reference-data--VerticalMeasurementType:KB
So the **EXPECTED** result of search query
```
{
"kind": "osdu:wks:master-data--Wellbore:1.0.0",
"query":"nested(data.VerticalMeasurements, (VerticalMeasurementTypeID:\"tenant1:reference-data--VerticalMeasurementType:KB\"))",
"sort": {
"field": [
"nested(data.VerticalMeasurements, VerticalMeasurement, min)"
],
"order": [
"DESC"
]
}
}
```
should be like below
```
{
"id":"wellbore5",
"id":"wellbore4",
"id":"wellbore3",
"id":"wellbore2",
"id":"wellbore1"
}
```
This currently doesn't seem possible today.
So the records get sorted on the minimum values from the VerticalMeasurements array and the **ACTUAL** response is
```
{
"id":"wellbore1",
"id":"wellbore2",
"id":"wellbore3",
"id":"wellbore4",
"id":"wellbore5"
}
```
This is because minimum values from VerticalMeasurements array are for VerticalMeasurementType:GR and sorting happens on the mode specified in the query which is **'min'**.
The below query which specifies filter within sort clause also gives a similar response.
```
{
"kind": "osdu:wks:master-data--Wellbore:1.0.0",
"query":"nested(data.VerticalMeasurements, (VerticalMeasurementTypeID:\"tenant1:reference-data--VerticalMeasurementType:KB\"))",
"sort": {
"field": [
"nested(data.VerticalMeasurements, VerticalMeasurement, min)"
],
"order": [
"ASC"
],
"filter":["nested(data.VerticalMeasurements, VerticalMeasurementTypeID:\"tenant1:reference-data--VerticalMeasurementType:KB\", match)"]
}
}
```
Can the sorting be supported via search service where sorting of records happens on the field specified inside query or inside sort clause?
Let me know if this needs more clarification and requires an ADR or if this is something that can be fixed as an issue.https://community.opengroup.org/osdu/platform/system/search-service/-/issues/90OSDU’s ISV un-friendliness linked to the complexity of the Schemas2022-08-23T21:28:22ZPierrick GaudinOSDU’s ISV un-friendliness linked to the complexity of the SchemasI have brought up OSDU’s ISV un-friendliness linked to the complexity of the Schemas (and hence payloads). My feedback was given as an SI working with ISVs and noting their reticence to go too deep into integration. Simple things like “w...I have brought up OSDU’s ISV un-friendliness linked to the complexity of the Schemas (and hence payloads). My feedback was given as an SI working with ISVs and noting their reticence to go too deep into integration. Simple things like “why is it so difficult to locate the UWI”, and thereafter “what do we do if different companies use different alias names for UWI (such a CUWI, PUWI, or TGlobal)”. The point being that an ISV needs a standard way of finding UWI irrespective of how the Platform owner has organised content (and we did suggest to some ISVs that they use a rules-based configuration themselves to help keep integration generic).
On a couple of occasions I did suggest that we revisit the ISV friendliness (special attributes that are always available in any search response) that was implicit with earlier versions of OSDU (R0, R1, and Wipro’s Azure R1+) This friendliness was put aside (temporarily at the time, but it has become permeant) during the OSDU / OpenDES friction point discussions late 2019 / early 2020 just ahead of moving into R2/R3 development with the new code base.
In fact, in July 2019, just after the first OSDU demo version (R0) was made available on Azure and AWS, it was agreed that the Data Definitions Committee would take responsibility for defining such “special attributes” that would always be available in a search response. The reasoning was simple. There were dependencies that were related to the relationships between different schemas; Well and Wellbore for example (namely Parent and Child). Also, since these attributes were NOT attributes from the Schemas themselves, they would need to be separately documented (but also in a Schema format). Finally, there were implications for the ingestion service. As such, there was a strong need to standardise the way such “special attributes” were declared in order to ensure that they could be handled in a generic manner by the ingestion service. This latter point was never really discussed in much detail thereafter.
The point about special attributes is that they are computed by the ingestion service based upon a set of rules. Wipro did independently implement these features into the OSDU ingestion engine of R1 to prove it could be done. We called it R1+ at the time, and it was actually delivered to several Operators. In fact, as the code-base changed going into R2/R3, Wipro also aligned the “rules definitions” with the new Schema format with a view to keeping the “special attributes” feature available as an option for R3. The other thing that Wipro built into its R1+ version was the ability to auto-generate the Index Schemas for elastic search (something that was done manually in R0 and R1). These Index Schemas did not require Hints in the Schemas, although they did faithfully reproduce and rendered searchable everything in the original Schema, along with the “special attributes”.
Here are some examples of “special attributes” that were associated with a Wellbore in R1 (not all was perfect, but the intent is clear). There were special attributes for Name and UWBI. There were attributes indicating the Well name and UWI. You will also note by deduction that ingesting Logs, Markersets and Trajectories, for example, caused the Wellbore record to be updated with counters and pointer arrays.
"WellCommonName": "G16-02",
"WellUWI": "1028",
"WellSRN": "srn:master-data/Well:1028:",
"SpudDate": "1983-02-13T00:00:00",
"CommonName": "AME-102-S1",
"UWBI": "5017",
"WellboreSRN": "srn:master-data/Wellbore:7607:",
"WellCommonName": "AME-102",
"WellLogCount": 0,
"WellLogSRN": [],
"WellboreMarkerCount": 1,
"WellboreMarkerSRN": [
"srn:work-product-component/WellboreMarker:67657777573843378679176b3a1ddda9:1"
],
"WellborePathCount": 1,
"WellborePathSRN": [
"srn:work-product-component/WellboreTrajectory:2b293633fea6464086b67ecd8016adcc:1"
]
}
With all that said, my point is that instead of raising issues to add an attribute or two and then needing to decide where to put them, perhaps we should simply revisit the past features outlined above with a view to bringing some of it back into play.https://community.opengroup.org/osdu/platform/system/search-service/-/issues/88Search service returns date-time fields with an incorrect format2022-08-24T10:48:48ZAn NgoSearch service returns date-time fields with an incorrect formatI created a record in Storage containing a date-time field. When I get the record from OSDU Storage, it returns correctly the ISO 8601 / RFC 3339 format I inserted.
Problem: When I retrieve the same record from the OSDU Search service, ...I created a record in Storage containing a date-time field. When I get the record from OSDU Storage, it returns correctly the ISO 8601 / RFC 3339 format I inserted.
Problem: When I retrieve the same record from the OSDU Search service, the date has an unexpected format, which causes unmarshalling to fail.
If you fetch it from the OSDU Storage service, the two date-time fields are displayed correctly as inserted.
**What I expect:**
Date fields in the format "2021-06-25T19:16:55+00:00" or "2021-06-25T19:16:55Z"
**What I got instead:**
"2021-06-25T19:16:55+**0000**" (missing colon at the decimal fraction of a second)
**per documentation, the correct format should be:**
**Complete date plus hours, minutes and seconds:**
YYYY-MM-DDThh:mm:ssTZD (eg 1997-07-16T19:20:30+01:00)
**Complete date plus hours, minutes, seconds and a decimal fraction of a second**
YYYY-MM-DDThh:mm:ss.sTZD (eg 1997-07-16T19:20:30.45+01:00)https://community.opengroup.org/osdu/platform/system/search-service/-/issues/87Can't determine when page limit is for search request based on error message2022-08-24T10:50:28ZMichaelCan't determine when page limit is for search request based on error messageThe Search API is limited to only returning a maximum number of matching results for a query (default is 10,000 results total).
When this limit is reached a 400 error is returned however no details are provided about what parameters are...The Search API is limited to only returning a maximum number of matching results for a query (default is 10,000 results total).
When this limit is reached a 400 error is returned however no details are provided about what parameters are invalid.
Below is an example search request that retrieves records beyond the max allowed limit:
```
curl --location --request POST 'https://r3m11.preshiptesting.osdu.aws/api/search/v2/query' \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer ...' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "osdu:wks:work-product-component--WellboreTrajectory:*",
"limit": 1000,
"offset": 10000
}'
```
Response:
```
{
"code": 400,
"reason": "Bad Request",
"message": "Invalid parameters were given on search request"
}
```
The generic 400 error message "Invalid parameters were given on search request" makes it difficult for the user to determine what parameters were specified incorrectly in the request.
Ideally, the message should be more specific about what parameters are invalid and why.