Pre Shipping issueshttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues2023-03-08T16:39:34Zhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/440400 error occurs when running multiple search requests2023-03-08T16:39:34ZMichael400 error occurs when running multiple search requestsWhen executing multiple requests one after another, eventually the search requests start failing with a 400 error: Failed to derive xcontent
Here is an example search request:
curl --location 'https://r3m15.preshiptesting.osdu.aws/api/...When executing multiple requests one after another, eventually the search requests start failing with a 400 error: Failed to derive xcontent
Here is an example search request:
curl --location 'https://r3m15.preshiptesting.osdu.aws/api/search/v2/query' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer ...' \
--data '{
"kind": "osdu:wks:*:*",
"query": "(data.WellboreID:(\"osdu:master-data--Wellbore:4901\"))",
"limit": 100,
"offset": 0
}'M15 - Release 0.18https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/55Access of preshipping env, (Need service account key)2021-09-21T13:06:04Zupendra kumarAccess of preshipping env, (Need service account key)## Summary
(Summarize the bug encountered concisely)
## Steps to reproduce
(How one can reproduce the issue - this is very important)
## Example Environment(Tenant)
(Tenant name)
## What is the current bug behavior?
(What actually...## Summary
(Summarize the bug encountered concisely)
## Steps to reproduce
(How one can reproduce the issue - this is very important)
## Example Environment(Tenant)
(Tenant name)
## What is the current bug behavior?
(What actually happens)
## What is the expected correct behavior?
(What you should see instead)
## Relevant logs and/or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code, as
it's very hard to read otherwise.)
## Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
/cc @kateryna_kurach
/cc @Aliaksandr_Ramanovich1
Need access of pre shipping env, (Need service account key )https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/587Access to RedHat IBM Console2024-01-12T08:02:29ZNaufal Mohamed NooriAccess to RedHat IBM ConsoleI used to have access to IBM RedHat console to complete preship test logging and monitoring but after m19 deployment seems that the old URL is not anymore working: https://console-openshift-console.odi-og-osdu-ba8e38d4c011d627379af1a4280...I used to have access to IBM RedHat console to complete preship test logging and monitoring but after m19 deployment seems that the old URL is not anymore working: https://console-openshift-console.odi-og-osdu-ba8e38d4c011d627379af1a4280c4e35-0000.us-south.containers.appdomain.cloud/
Can you grant me a new access to new RedHat console portal URL for the latest M19 release
My email is naufal.noori@katalystdm.comAnuj Guptavikas ranaAnuj Guptahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/477Add all released new schema versions to test data2023-04-25T09:38:01ZMichaelAdd all released new schema versions to test dataNew schemas are released (for instance master-data--Wellbore:1.3.0), however, these new schemas are not added to the pre-shipping environments for testing.
There should at least be a few records that use the latest released schemas for ...New schemas are released (for instance master-data--Wellbore:1.3.0), however, these new schemas are not added to the pre-shipping environments for testing.
There should at least be a few records that use the latest released schemas for each major data type (aster-data--Well, master-data--Wellbore, wpc--SeismicTraceData, wpc--SeismicHorizon, wpc--WellLog, etc) available in the pre-shipping environments by addition to the test dataset.https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/49Admin UI2022-08-23T13:29:59ZSehubo AkinyanmiAdmin UIAdmin UI
(All CSPs should now be on EV2; potential to see same behaviour across all CSPs)Admin UI
(All CSPs should now be on EV2; potential to see same behaviour across all CSPs)Pre-Shipping R3-M8Grant MarblestoneGrant Marblestonehttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/48Admin UI2022-08-23T13:29:56ZSehubo AkinyanmiAdmin UIAdmin UI
(All CSPs should now be on EV2; potential to see same behaviour across all CSPs)Admin UI
(All CSPs should now be on EV2; potential to see same behaviour across all CSPs)Pre-Shipping R3-M8Grant MarblestoneGrant Marblestonehttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/47Admin UI2022-08-23T13:29:56ZSehubo AkinyanmiAdmin UIAdmin UI Test Validation (All CSPs should now be on EV2; potential to see same behaviour across all CSPs)Admin UI Test Validation (All CSPs should now be on EV2; potential to see same behaviour across all CSPs)Pre-Shipping R3-M8Grant MarblestoneGrant Marblestonehttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/46Admin UI2022-08-23T13:29:55ZSehubo AkinyanmiAdmin UIAdmin UI Test Validation
(All CSPs should now be on EV2; potential to see same behaviour across all CSPs)Admin UI Test Validation
(All CSPs should now be on EV2; potential to see same behaviour across all CSPs)Pre-Shipping R3-M8Grant MarblestoneGrant Marblestonehttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/51ADR: E2E preshipment team A workflow bot2023-08-09T08:47:29Zetienne peyssonADR: E2E preshipment team A workflow bot
## Status
- [x] Draft
- [ ] Proposed
- [x] Trialing
- [ ] Under Review
- [ ] Approved
- [ ] Retired
## Context & Scope
Following the Preshipment validation dashboard :
![preship-validation-scope](/uploads/1205798b98bfc6cbc4b2f341fe...
## Status
- [x] Draft
- [ ] Proposed
- [x] Trialing
- [ ] Under Review
- [ ] Approved
- [ ] Retired
## Context & Scope
Following the Preshipment validation dashboard :
![preship-validation-scope](/uploads/1205798b98bfc6cbc4b2f341fe68eb45/preship-validation-scope.png)
There are multiple manual steps to achieve in order to test each workflow.
This seems to be error prone and time consuming.
This ADR focuses on the following steps :
- Authenticate to any CSP
- File uploading whenever required
- Trigger DAG
- Validate the workflow
- Generate a report
- Clean up
That work could be also be extended with the following :
- Bulk loading
Another ADR has been approved for maintaining Postman collections to be integrated for testing on DAGs and services endpoints end to end executions.
There might be some overlapping with the ![Postman Collection ADR](https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/49).
## Proposition
Implement a script/framework so we can test each use case independently from any computer but also a Gitlab pipeline (e2e tests).
- Pros :
Get a clear report of the process so we can more easily provide feedbacks.
Have a common place with configuration templates to fill in for CSPs (another script could also help on that part).
Add other workflows using the existing framework.
Test using multiple source files.
Match the framework version with the releases so pipelines could be run in multiple test environments at the same time (as long as environments are available).
Easy onboarding for new developers.
- Cons :
Maintenance of the configuration parameters (CSPs) should follow releases cadence.
Needs developers
## Decision
## Rationale
## Consequences
- Direct consequence on the Preshipment team A.
## When to revisit
## Tradeoff analysis - input to decision
## Decision timelinehttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/108Airflow Console/log2022-08-23T13:29:57ZEsmira RafigayevaAirflow Console/logM9 - Release 0.12Esmira RafigayevaEsmira Rafigayevahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/23Airflow Console/Log2022-09-03T02:21:05ZEsmira RafigayevaAirflow Console/LogPre-Shipping R3-M8Yanbin ZhangYanbin Zhanghttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/22Airflow Console/Log2022-09-03T02:21:59ZEsmira RafigayevaAirflow Console/LogPre-Shipping R3-M8Naufal Mohamed NooriNaufal Mohamed Noorihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/21Airflow Console/Log2022-09-03T02:23:16ZEsmira RafigayevaAirflow Console/LogPre-Shipping R3-M8ivar Soerheimivar Soerheimhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/85Airflow regex issue - likely not CSP related (?)2021-10-01T12:15:53ZEsmira RafigayevaAirflow regex issue - likely not CSP related (?)
Encountered issue through manifest ingestion inside airflow DAG:
"UnitQuantityID": "opendes:reference-data--UnitQuantity:1:", The error here seems to be regex related, as airflow does the search for: "opendes:reference-data--UnitQuanti...
Encountered issue through manifest ingestion inside airflow DAG:
"UnitQuantityID": "opendes:reference-data--UnitQuantity:1:", The error here seems to be regex related, as airflow does the search for: "opendes:reference-data--UnitQuantity"
It seems that the number 1 at the end is identified as a version and removed. Who can fix this?
This issue is CSP independent and therefore a bug across all CSPs
Edited by ivar Soerheim just now @s0rhe1mivar SoerheimMANISH KUMARVivek Ojhaivar Soerheimhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/138Audit & Metrics2022-09-03T02:22:12ZSehubo AkinyanmiAudit & MetricsKPI Batch 1 - Scope validation from Project Lead, @Srinivasan Ramamoorthi [LTI]KPI Batch 1 - Scope validation from Project Lead, @Srinivasan Ramamoorthi [LTI]M9 - Release 0.12Naufal Mohamed NooriNaufal Mohamed Noorihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/230AWS - 50,000 Performance Loading Fails2022-08-23T12:47:15ZSehubo AkinyanmiAWS - 50,000 Performance Loading Fails
#50000 Records fails - below is the error from the python shell console.
"kind": "osdu:wks:master-data--Organisation:1.0.0", "acl": {"viewers": ["data.default.viewers@osdu.example.com"], "owners": ["data.default.owners@osdu.example.com...
#50000 Records fails - below is the error from the python shell console.
"kind": "osdu:wks:master-data--Organisation:1.0.0", "acl": {"viewers": ["data.default.viewers@osdu.example.com"], "owners": ["data.default.owners@osdu.example.com"]}, "legal": {"legaltags": ["osdu-public-usa-dataset"], "otherRelevantDataCountries": ["US"]}, "data": {"OrganisationName": "Name_TEST_BULK_2022-03-01_174_49999", "Source": "Source_TEST_BULK_2022-03-01_174_49999"}}, {"id": "osdu:master-data--Organisation:TEST_BULK_2022-03-01_174_50000", "kind": "osdu:wks:master-data--Organisation:1.0.0", "acl": {"viewers": ["data.default.viewers@osdu.example.com"], "owners": ["data.default.owners@osdu.example.com"]}, "legal": {"legaltags": ["osdu-public-usa-dataset"], "otherRelevantDataCountries": ["US"]}, "data": {"OrganisationName": "Name_TEST_BULK_2022-03-01_174_50000", "Source": "Source_TEST_BULK_2022-03-01_174_50000"}}]}}}, "headers": {"data-partition-id": "osdu", "Accept": "application/json", "Authorization": "Bearer {hidded}"}, "verify": false}
DEBUG:root:Response: 401
DEBUG:root:json = {"status": 401, "message": "The user is not authorized to perform this action", "reason": "Access denied"}
>>>
Please note, #1,000 records passed ok; and was able to retrieve the records.
>>> aws_client.get_record('osdu:master-data--Organisation:TEST_BULK_2022-03-01_233_999').dump()
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): r3m10.preshiptesting.osdu.aws:443
DEBUG:urllib3.connectionpool:https://r3m10.preshiptesting.osdu.aws:443 "GET /api/storage/v2/records/osdu:master-data--Organisation:TEST_BULK_2022-03-01_233_999 HTTP/1.1" 200 567
DEBUG:root:HTTP GET https://r3m10.preshiptesting.osdu.aws/api/storage/v2/records/osdu:master-data--Organisation:TEST_BULK_2022-03-01_233_999
DEBUG:root:args = {"headers": {"data-partition-id": "osdu", "Accept": "application/json", "Authorization": "Bearer {Hidden}"}, "verify": false}
DEBUG:root:Response: 200
DEBUG:root:json = {"data": {"OrganisationName": "Name_TEST_BULK_2022-03-01_233_999", "Source": "Source_TEST_BULK_2022-03-01_233_999"}, "meta": [], "id": "osdu:master-data--Organisation:TEST_BULK_2022-03-01_233_999", "version": 1646147532729232, "kind": "osdu:wks:master-data--Organisation:1.0.0", "acl": {"viewers": ["data.default.viewers@osdu.example.com"], "owners": ["data.default.owners@osdu.example.com"]}, "legal": {"legaltags": ["osdu-public-usa-dataset"], "otherRelevantDataCountries": ["US"], "status": "compliant"}, "createUser": "serviceprincipal@testing.com", "createTime": "2022-03-01T15:12:12.736Z"}
{
"data": {
"OrganisationName": "Name_TEST_BULK_2022-03-01_233_999",
"Source": "Source_TEST_BULK_2022-03-01_233_999"
},
"meta": [],
"id": "osdu:master-data--Organisation:TEST_BULK_2022-03-01_233_999",
"version": 1646147532729232,
"kind": "osdu:wks:master-data--Organisation:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@osdu.example.com"
],
"owners": [
"data.default.owners@osdu.example.com"
]
},
"legal": {
"legaltags": [
"osdu-public-usa-dataset"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "serviceprincipal@testing.com",
"createTime": "2022-03-01T15:12:12.736Z"
}
>>>M10 - Release 0.13GregMarc Burnie [AWS]Okoun-Ola Fabien HouetoJevon WilliamsGreghttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/183AWS: 500 records loading test failed2022-08-23T12:47:12ZWira YarmanAWS: 500 records loading test failedLoading test failed after an hour for 500 records - run id: 2021-12-10T06:35:17.498980+00:00Loading test failed after an hour for 500 records - run id: 2021-12-10T06:35:17.498980+00:00M9 - Release 0.12GregGreghttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/524AWS Cloud Watch Does not Capturing Any Logs/Information Related to Preship R3M182023-06-19T12:09:15ZNaufal Mohamed NooriAWS Cloud Watch Does not Capturing Any Logs/Information Related to Preship R3M18When trying to capture logs for API request sent out from Postman (Search Service/Storage Service/Workflow Service endpoints):
1. I did not see any related log visible in AWS console --> CloudWatch --> Logs --> Log Group
![image](/upl...When trying to capture logs for API request sent out from Postman (Search Service/Storage Service/Workflow Service endpoints):
1. I did not see any related log visible in AWS console --> CloudWatch --> Logs --> Log Group
![image](/uploads/934641474a9603a7d636605e5d97eb8e/image.png)
2. No cluster related to r3m18 running from AWS Console --> CloudWatch --> Insight --> Container Inisght
![image](/uploads/347dd3b11e8da2f543903c051b7d40e5/image.png)
The console was tagged into OSDU preshipping test account (US-east-2)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/275AWS - Core Basic Checks - SEARCH - not getting the desired answer2022-08-23T11:25:12ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brAWS - Core Basic Checks - SEARCH - not getting the desired answerHello,
I'm trying to do the basic checks on the Core Smoke Collection and I'm not getting the desired answer on one request from SEARCH.
The request is POST Get Wells with FacilityEvents Effective Date between X and Y (Range)
curl --l...Hello,
I'm trying to do the basic checks on the Core Smoke Collection and I'm not getting the desired answer on one request from SEARCH.
The request is POST Get Wells with FacilityEvents Effective Date between X and Y (Range)
curl --location --request POST 'https://r3m11.preshiptesting.osdu.aws/api/search/v2/query
The Body of the request is:
```
{
"kind": "{{authority}}:{{schemaSource}}:master-data--Well:{{standard_schema_version}}",
"query": "data.FacilityEvents.EffectiveDateTime: [1910-01-19 TO 2008-12-31]",
"sort": {
"field": ["data.FacilityEvents.EffectiveDateTime"],
"order": ["DESC"]
},
"limit": 30
}
```
and the response is:
```
{
"results": [],
"aggregations": [],
"totalCount": 0
}
```
The status of this request is 200 Ok But the rest results are:
![result](/uploads/7baac79a362a5d42d34f63ea8a3502d0/result.PNG)
An observation about this: in IBM and GCP this request is working.... And Debasis just sent to me the body synthax used by Naufal and it worked (just not sure if it is the right response). The body synthax that he sent to me was:
```
{
"kind": "osdu:wks:master-data--Well:1.0.0",
"query":"createTime: [2022-02-27 TO *]",
"returnedFields": [
"id", "createTime"
]
}
```https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/398AWS - CSV Ingestion - custom schema - record is created, but data cannot be s...2022-12-23T17:31:30ZDebasis ChatterjeeAWS - CSV Ingestion - custom schema - record is created, but data cannot be seen by using Search for retrievingSame issue as was reported earlier with R3M14.
[M15-AWS-CSV-Ingest-custom-schema-Debasis-problem.docx](/uploads/e959c60c3f226eacf67fb86927afdb25/M15-AWS-CSV-Ingest-custom-schema-Debasis-problem.docx)
cc @sami_gSame issue as was reported earlier with R3M14.
[M15-AWS-CSV-Ingest-custom-schema-Debasis-problem.docx](/uploads/e959c60c3f226eacf67fb86927afdb25/M15-AWS-CSV-Ingest-custom-schema-Debasis-problem.docx)
cc @sami_g