Search service does not return all null objects when the key object datatype has value not equal to string
Search service should return all object key and return them null if the object is not populated during ingestion/storage insertion. However, we found that if keys have non-string value type from schema definition it will return null from the search - as opposed to when the object has value type string where non populated key will return 'null'
Example, I have ingested the following payload in R3M20 AWS preship:
{ "runId": "{{$guid}}", "executionContext": { "acl": { "viewers": [ "data.default.viewers@osdu.example.com" ], "owners": [ "data.default.owners@osdu.example.com" ] }, "legal": { "legaltags": [ "osdu-public-usa-dataset" ], "otherRelevantDataCountries": [ "US" ] }, "Payload": { "AppKey": "test-app", "data-partition-id": "osdu" }, "manifest": { "kind": "osdu:wks:Manifest:1.0.0", "Data": { "WorkProductComponents": [ { "id": "osdu:work-product-component--SeismicTraceData:TEST_ISSUE_1", "kind": "osdu:wks:work-product-component--SeismicTraceData:1.4.0", "acl": { "viewers": [ "data.default.viewers@osdu.example.com" ], "owners": [ "data.default.owners@osdu.example.com" ] }, "legal": { "legaltags": [ "osdu-public-usa-dataset" ], "otherRelevantDataCountries": [ "US" ] }, "data": { "Name": "TESTT_ISSUE", "StartTime": 0, "EndTime": 10000 }, "meta": [ { "kind": "Unit", "name": "ms", "persistableReference": "{\"abcd\":{\"a\":0.0,\"b\":0.001,\"c\":1.0,\"d\":0.0},\"symbol\":\"ms\",\"baseMeasurement\":{\"ancestry\":\"T\",\"type\":\"UM\"},\"type\":\"UAD\"}", "propertyNames": [ "StartTime", "EndTime", "SampleCount" ], "unitOfMeasureID": "osdu:reference-data--UnitOfMeasure:ms:" } ] } ] } } } }
During search query, the data.SampleCount and others non-string keys are not shown up from search return. Look at the sample below traceDomainUoM (type: string) is visible as null but the traceLength (type: number) is not visible:
{ "results": [ { "data": { "SpatialArea.QuantitativeAccuracyBandID": null, "VirtualProperties.DefaultLocation.QuantitativeAccuracyBandID": null, "LiveTraceOutline.CoordinateQualityCheckPerformedBy": null, "SpatialArea.SpatialParameterTypeID": null, "Difference": null, "ResourceCurationStatus": null, "SpatialArea.SpatialGeometryTypeID": null, "SortOrderID": null, "IsExtendedLoad": null, "Name": "TESTT_ISSUE", "SeismicFilteringTypeID": null, "VirtualProperties.DefaultName": "TESTT_ISSUE", "VirtualProperties.DefaultLocation.CoordinateQualityCheckPerformedBy": null, "ResourceSecurityClassification": null, "VerticalMeasurementTypeID": null, "SeismicStackingTypeID": null, "ExistenceKind": null, "ProcessingProjectID": null, "SeismicDomainTypeID": null, "Preferred2DInterpretationSetID": null, "HorizontalCRSID": null, "SeismicAttributeTypeID": null, "BinGridID": null, "SeismicProcessingStageTypeID": null, "StartTime": 0.0, "LiveTraceOutline.SpatialParameterTypeID": null, "SpatialArea.QualitativeSpatialAccuracyTypeID": null, "SpatialPoint.SpatialGeometryTypeID": null, "LiveTraceOutline.QuantitativeAccuracyBandID": null, "LiveTraceOutline.SpatialGeometryTypeID": null, "IsDiscoverable": null, "SeismicWaveTypeID": null, "Precision.WordFormat": null, "VirtualProperties.DefaultLocation.QualitativeSpatialAccuracyTypeID": null, "SubmitterName": null, "TraceDomainUOM": null, "SeismicPhaseID": null, "SpatialPoint.QualitativeSpatialAccuracyTypeID": null, "PrincipalAcquisitionProjectID": null, "GatherTypeID": null, "Description": null, "Phase": null, "EndTime": 10.0, "TimeLapse.TimeSeriesID": null, "ResourceLifecycleStatus": null, "SeismicTraceDataDimensionalityTypeID": null, "TechnicalAssuranceID": null, "VirtualProperties.DefaultLocation.SpatialGeometryTypeID": null, "Source": null, "SeismicLineGeometryID": null, "LiveTraceOutline.QualitativeSpatialAccuracyTypeID": null, "SpatialPoint.CoordinateQualityCheckPerformedBy": null, "TraceRelationFileID": null, "Polarity": null, "SpatialPoint.SpatialParameterTypeID": null, "SpatialPoint.QuantitativeAccuracyBandID": null, "SpatialArea.CoordinateQualityCheckPerformedBy": null, "SeismicPolarityID": null, "Seismic2DName": null, "VirtualProperties.DefaultLocation.SpatialParameterTypeID": null, "ResourceHomeRegionID": null, "Preferred3DInterpretationSetID": null, "SeismicMigrationTypeID": null }, "kind": "osdu:wks:work-product-component--SeismicTraceData:1.4.0", "source": "wks", "acl": { "viewers": [ "data.default.viewers@osdu.example.com" ], "owners": [ "data.default.owners@osdu.example.com" ] }, "type": "work-product-component--SeismicTraceData", "version": 1703067759404129, "tags": { "normalizedKind": "osdu:wks:work-product-component--SeismicTraceData:1" }, "modifyUser": "admin-main@testing.com", "modifyTime": "2023-12-20T10:22:40.387Z", "createTime": "2023-12-20T10:17:21.256Z", "authority": "osdu", "namespace": "osdu:wks", "legal": { "legaltags": [ "osdu-public-usa-dataset" ], "otherRelevantDataCountries": [ "US" ], "status": "compliant" }, "createUser": "serviceprincipal-main@testing.com", "id": "osdu:work-product-component--SeismicTraceData:TEST_ISSUE_1" } ], "aggregations": null, "totalCount": 1 }
We do hope the search index will return consistently regardless the type of keys input either string, boolean, number or integer.
cc @debasisc