Similar to issue I have in #587 we would like to have IBM to open up a console monitor for IBM to be used for logging and monitoring procedure in preship
OR
You have your personel to test out based on the existing test case here: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M16/Test_Plan_Results_M16/Operational%20Procedures%20Test/R3M16_IBM_OSDU_PTP_TeamA_Non_Destructive_Operational_Procedure_Test_NN.docx
cc @davidglass
@anujgupta Test case results are consistent and can be downloaded from the directory given by @ishakumari So far everything is good. Closing this ticket
@dzmitry_malkevich Here is the zipped payload
I did perform manifest ingestion with bulk and batch manifest ingestion with huge size payload, the Airflow process failed with the following log:
*** No logs found on s3 for ti=<TaskInstance: Osdu_ingest.process_manifest_task_2 07a62538-1c1e-4200-9d63-f880bde99362 [failed]> *** Could not read served logs: [Errno -2] Name or service not known
The list of runID wiht huge payload (more than 10 MB):
I have performed a smaller payload (less than 10mb json) and they were successful
Do slack me in person or via Preship channel I can share the json payload in person (I cant attach the json payload here due to limit of 10mb attachment)
Naufal Mohamed Noori (1fea4f40) at 17 Jan 03:55
Update 2 files
Naufal Mohamed Noori (32ef83c4) at 17 Jan 03:50
Update file RI_M22_Performance_Loading_Test_2024-01-17_10-55-42_log...
Naufal Mohamed Noori (cb50dab4) at 12 Jan 12:58
Update file CSPs_Bulk_Batch_Load_Summary.jpg
Naufal Mohamed Noori (e6655e50) at 12 Jan 12:57
Update 2 files
Naufal Mohamed Noori (8d799559) at 12 Jan 08:20
Manifest bulk batch AWS and Azure
Naufal Mohamed Noori (37f7aa91) at 12 Jan 08:09
Update 3 files
Similar to issue I have in #587 we would like to have IBM to open up a console monitor for IBM to be used for logging and monitoring procedure in preship
OR
You have your personel to test out based on the existing test case here: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M16/Test_Plan_Results_M16/Operational%20Procedures%20Test/R3M16_IBM_OSDU_PTP_TeamA_Non_Destructive_Operational_Procedure_Test_NN.docx
cc @davidglass
I used to have access to IBM RedHat console to complete preship test logging and monitoring but after m19 deployment seems that the old URL is not anymore working: https://console-openshift-console.odi-og-osdu-ba8e38d4c011d627379af1a4280c4e35-0000.us-south.containers.appdomain.cloud/
Can you grant me a new access to new RedHat console portal URL for the latest M19 release
My email is naufal.noori@katalystdm.com
Search service should return all object key and return them null if the object is not populated during ingestion/storage insertion. However, we found that if keys have non-string value type from schema definition it will return null from the search - as opposed to when the object has value type string where non populated key will return 'null'
Example, I have ingested the following payload in R3M20 AWS preship:
{ "runId": "{{$guid}}", "executionContext": { "acl": { "viewers": [ "data.default.viewers@osdu.example.com" ], "owners": [ "data.default.owners@osdu.example.com" ] }, "legal": { "legaltags": [ "osdu-public-usa-dataset" ], "otherRelevantDataCountries": [ "US" ] }, "Payload": { "AppKey": "test-app", "data-partition-id": "osdu" }, "manifest": { "kind": "osdu:wks:Manifest:1.0.0", "Data": { "WorkProductComponents": [ { "id": "osdu:work-product-component--SeismicTraceData:TEST_ISSUE_1", "kind": "osdu:wks:work-product-component--SeismicTraceData:1.4.0", "acl": { "viewers": [ "data.default.viewers@osdu.example.com" ], "owners": [ "data.default.owners@osdu.example.com" ] }, "legal": { "legaltags": [ "osdu-public-usa-dataset" ], "otherRelevantDataCountries": [ "US" ] }, "data": { "Name": "TESTT_ISSUE", "StartTime": 0, "EndTime": 10000 }, "meta": [ { "kind": "Unit", "name": "ms", "persistableReference": "{\"abcd\":{\"a\":0.0,\"b\":0.001,\"c\":1.0,\"d\":0.0},\"symbol\":\"ms\",\"baseMeasurement\":{\"ancestry\":\"T\",\"type\":\"UM\"},\"type\":\"UAD\"}", "propertyNames": [ "StartTime", "EndTime", "SampleCount" ], "unitOfMeasureID": "osdu:reference-data--UnitOfMeasure:ms:" } ] } ] } } } }
During search query, the data.SampleCount and others non-string keys are not shown up from search return. Look at the sample below traceDomainUoM (type: string) is visible as null but the traceLength (type: number) is not visible:
{ "results": [ { "data": { "SpatialArea.QuantitativeAccuracyBandID": null, "VirtualProperties.DefaultLocation.QuantitativeAccuracyBandID": null, "LiveTraceOutline.CoordinateQualityCheckPerformedBy": null, "SpatialArea.SpatialParameterTypeID": null, "Difference": null, "ResourceCurationStatus": null, "SpatialArea.SpatialGeometryTypeID": null, "SortOrderID": null, "IsExtendedLoad": null, "Name": "TESTT_ISSUE", "SeismicFilteringTypeID": null, "VirtualProperties.DefaultName": "TESTT_ISSUE", "VirtualProperties.DefaultLocation.CoordinateQualityCheckPerformedBy": null, "ResourceSecurityClassification": null, "VerticalMeasurementTypeID": null, "SeismicStackingTypeID": null, "ExistenceKind": null, "ProcessingProjectID": null, "SeismicDomainTypeID": null, "Preferred2DInterpretationSetID": null, "HorizontalCRSID": null, "SeismicAttributeTypeID": null, "BinGridID": null, "SeismicProcessingStageTypeID": null, "StartTime": 0.0, "LiveTraceOutline.SpatialParameterTypeID": null, "SpatialArea.QualitativeSpatialAccuracyTypeID": null, "SpatialPoint.SpatialGeometryTypeID": null, "LiveTraceOutline.QuantitativeAccuracyBandID": null, "LiveTraceOutline.SpatialGeometryTypeID": null, "IsDiscoverable": null, "SeismicWaveTypeID": null, "Precision.WordFormat": null, "VirtualProperties.DefaultLocation.QualitativeSpatialAccuracyTypeID": null, "SubmitterName": null, "TraceDomainUOM": null, "SeismicPhaseID": null, "SpatialPoint.QualitativeSpatialAccuracyTypeID": null, "PrincipalAcquisitionProjectID": null, "GatherTypeID": null, "Description": null, "Phase": null, "EndTime": 10.0, "TimeLapse.TimeSeriesID": null, "ResourceLifecycleStatus": null, "SeismicTraceDataDimensionalityTypeID": null, "TechnicalAssuranceID": null, "VirtualProperties.DefaultLocation.SpatialGeometryTypeID": null, "Source": null, "SeismicLineGeometryID": null, "LiveTraceOutline.QualitativeSpatialAccuracyTypeID": null, "SpatialPoint.CoordinateQualityCheckPerformedBy": null, "TraceRelationFileID": null, "Polarity": null, "SpatialPoint.SpatialParameterTypeID": null, "SpatialPoint.QuantitativeAccuracyBandID": null, "SpatialArea.CoordinateQualityCheckPerformedBy": null, "SeismicPolarityID": null, "Seismic2DName": null, "VirtualProperties.DefaultLocation.SpatialParameterTypeID": null, "ResourceHomeRegionID": null, "Preferred3DInterpretationSetID": null, "SeismicMigrationTypeID": null }, "kind": "osdu:wks:work-product-component--SeismicTraceData:1.4.0", "source": "wks", "acl": { "viewers": [ "data.default.viewers@osdu.example.com" ], "owners": [ "data.default.owners@osdu.example.com" ] }, "type": "work-product-component--SeismicTraceData", "version": 1703067759404129, "tags": { "normalizedKind": "osdu:wks:work-product-component--SeismicTraceData:1" }, "modifyUser": "admin-main@testing.com", "modifyTime": "2023-12-20T10:22:40.387Z", "createTime": "2023-12-20T10:17:21.256Z", "authority": "osdu", "namespace": "osdu:wks", "legal": { "legaltags": [ "osdu-public-usa-dataset" ], "otherRelevantDataCountries": [ "US" ], "status": "compliant" }, "createUser": "serviceprincipal-main@testing.com", "id": "osdu:work-product-component--SeismicTraceData:TEST_ISSUE_1" } ], "aggregations": null, "totalCount": 1 }
We do hope the search index will return consistently regardless the type of keys input either string, boolean, number or integer.
cc @debasisc
Naufal Mohamed Noori (fec826cb) at 17 Nov 09:50
Upload New File
Naufal Mohamed Noori (f2338bf3) at 17 Nov 09:44
Update 2 files
Naufal Mohamed Noori (f7051d9b) at 17 Nov 08:42
Update 4 files
Naufal Mohamed Noori (39595973) at 17 Nov 08:40
Update 2 files
Naufal Mohamed Noori (86277a25) at 17 Nov 08:37
Add new directory
I was testing out logging and monitoring preship milestone 18 and found that Azure portal application insight is not capturing any logs related to any endpoint (manifest ingestion/search query) except for GET record.
Endpoint request with log was recorded from Application Insight --> Transaction (osdu-mvp-crship-9vns-ai): https://osdu-ship.msft-osdu-test.org/api/storage/v2/records/opendes:master-dataa--Agreement:WesternGeco
Endpoint request with log were not recorded from Application Insight --> Transaction (osdu-mvp-crship-9vns-ai):
Screenshot below shows only GET record request was recorded from transaction log. However, the manifest ingestion/search query were not recorded:
Tested and verified in R3M21 Azure preship. All OSDU basic core log services are captured in Azure portal application:
Core | Result in transaction search | Correlation id |
---|---|---|
storage | log visible | 8634b94e-ebb2-4428-aef1-3d73ec32c35d |
search | log visible | fa565117-b583-4cc9-9bd3-ed2fa2bebf30 |
entitlement | log visible | a4474368-3e61-414a-a9c5-3a2024ccbc0a |
legal | log visible | 0b16507a-030a-4556-8ab5-2cd408ca289a |
schema | log visible | d6022ba6-331f-4d44-95ed-ae6ad1d94618 |
workflow | log visible | 84dd8ca6-74ae-4ef3-a2ec-0df278d9e06e |
dataset | log visible | 4d9ec096-2d10-409b-977e-051a85c7f1a3 |
unit | log visible | 05a6d33b-f064-447c-a5eb-275996507ed7 |
crs | log visible | ed5b126f-8846-493d-99cd-69d012ee0d0c |
legal | log visible | 0b16507a-030a-4556-8ab5-2cd408ca289a |
schema | log visible | d6022ba6-331f-4d44-95ed-ae6ad1d94618 |