OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2023-10-25T14:52:30Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/commons/-/issues/11Search referenced Id into the catalog2023-10-25T14:52:30ZValentin GauthierSearch referenced Id into the catalogDuring translation, some references to other data are written in xml (RESQML/WITSML/PRODML) files.
For now, the code use the function : `get_obj_osdu_id` from the namespace_function.py` [file](https://community.opengroup.org/osdu/platfor...During translation, some references to other data are written in xml (RESQML/WITSML/PRODML) files.
For now, the code use the function : `get_obj_osdu_id` from the namespace_function.py` [file](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/commons/-/blob/main/commons_parser/namespace_function.py?ref_type=heads#L246).
A good improvement would be : after checking in the current data input to translate, if a reference is not found, check inside the catalog with the SearchService, to see if there is any data allready ingested that corresponds to the specific UUID (and maybe with more information like the ObjectVersion/Version). And if the data is found in the catalog, its id is returned.
We could also do the same with reference names (like "FeatureName" in manifests)https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/41Add helm repo version to the helm chart version in azure_helm_package2023-10-24T08:54:57ZKonstantin GukovAdd helm repo version to the helm chart version in azure_helm_package# Context
When packaging the helm chart on the pipeline, we first clone the helm repo by the branch name:
```
git clone --single-branch --branch ${COMMUNITY_HELM_BRANCH}
```
Then set the chart version using the commit SHA of the repos...# Context
When packaging the helm chart on the pipeline, we first clone the helm repo by the branch name:
```
git clone --single-branch --branch ${COMMUNITY_HELM_BRANCH}
```
Then set the chart version using the commit SHA of the repository:
```
CI_COMMIT_SHORT_SHA=$(echo "$CI_COMMIT_SHORT_SHA" | sed 's/^0*//')
helm_version=$(helm show chart ${helm_dir} | awk '/^version/{print $2}')-${CI_COMMIT_SHORT_SHA}
```
# Problem
CI_COMMIT_SHORT_SHA is not the SHA from the helm repository, it is the SHA in the source repository.
If I re-run the pipeline from the same commit in the source repo but update that COMMUNITY_HELM_BRANCH in the helm chart repo,
the pipeline will build a different helm chart but assign it the same version number.
If the helm chart version included SHAs of the both repositories, then it would be possible to back-track the exact source codes from the chart version number. This extra information would help troubleshooting pipeline failures.
# Solution
For example:
```
CI_COMMIT_SHORT_SHA=$(echo "$CI_COMMIT_SHORT_SHA" | sed 's/^0*//')
HELM_REPO_SHORT_SHA=$(git rev-parse --short HEAD | sed 's/^0*//')
helm_version=$(helm show chart ${helm_dir} | awk '/^version/{print $2}')-${CI_COMMIT_SHORT_SHA}-${HELM_REPO_SHORT_SHA}
```https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/119Rename "IStorage" methods for v42023-10-24T09:09:14ZYan Sushchynski (EPAM)Rename "IStorage" methods for v4Hello,
I noticed that the cloud-storage [interface](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/master/app/sdms-v4/src/cloud/storage.ts?ref_type=heads#L1...Hello,
I noticed that the cloud-storage [interface](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/master/app/sdms-v4/src/cloud/storage.ts?ref_type=heads#L19) has the following methods:
```
createBucket(bucketName: string): Promise<void>;
bucketExists(bucketName: string): Promise<boolean>;
deleteBucket(bucketName: string): Promise<void>;
```
These method names suggest that new buckets are getting created, checked for existence, or deleted within a single data-partition. However, the GC and Baremetal implementations are different -- a data-partition is expected to work with its own pre-created bucket instead of creating new ones. This discrepancy between the method names and their actual functionality could lead to confusion and misunderstanding.
A similar situation exists in the AWS implementation, where comments had to be added to clarify that 'bucketNames' are actually BLOBs, which can be seen [here](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/master/app/sdms-v4/src/cloud/providers/aws/storage.ts?ref_type=heads#L45).
I propose that we consider renaming these methods to more accurately reflect their functionality and create a better alignment with the actual implementation.
Thank you.Diego MolteniYunhua KoglinSacha BrantsMark YanDiego Moltenihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/eds-dms/-/issues/17[M18] 500 Internal Server Error when endpoint /eds/v1/retrievalInstructions i...2023-10-20T14:34:59ZDominik Kolasa[M18] 500 Internal Server Error when endpoint /eds/v1/retrievalInstructions is calledWhen provided with record which is not of kind: dataset--ConnectedSource.GenericStack 500 error is returned. Better exception handling should be added.
```
2023-10-06 21:34:35.229 INFO 6 --- [nio-8080-exec-2] o.o.o.edsdms.service.EdsDms...When provided with record which is not of kind: dataset--ConnectedSource.GenericStack 500 error is returned. Better exception handling should be added.
```
2023-10-06 21:34:35.229 INFO 6 --- [nio-8080-exec-2] o.o.o.edsdms.service.EdsDmsServiceImpl : Retrieved 1 records from storage
2023-10-06 21:34:35.231 ERROR 6 --- [nio-8080-exec-2] o.o.o.e.m.e.ExternalDataset : Problem with parsing data for external dataset record: osdu:dataset--File.Generic:73e309d55f347cf16401540eb2a7410bdbafaa24d895fcaba457939e5b3c0332
2023-10-06 21:34:35.231 ERROR 6 --- [nio-8080-exec-2] o.o.o.e.m.e.ExternalDataset : null
java.lang.NullPointerException
at org.opengroup.osdu.edsdms.util.RecordIdUtil.ordinalIndexOf(RecordIdUtil.java:20)
at org.opengroup.osdu.edsdms.util.RecordIdUtil.ignoreRecordVersion(RecordIdUtil.java:27)
at org.opengroup.osdu.edsdms.model.externaldataset.ExternalDataset.<init>(ExternalDataset.java:55)
at org.opengroup.osdu.edsdms.service.EdsDmsServiceImpl.lambda$modelRecords$0(EdsDmsServiceImpl.java:107)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
...
```https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/43[M18] EDS ingest fails on task status check2023-10-26T12:06:58ZDominik Kolasa[M18] EDS ingest fails on task status checkEDS ingest fails randomly on task status checks.
```
{{src_dags_fetch_and_ingest.py:158}} ERROR - {'status': 'error', 'message': Attribute
Traceback (most recent call last):
File "/usr/local/airflow/dags/eds_ingest/libs/src_dags_fetch...EDS ingest fails randomly on task status checks.
```
{{src_dags_fetch_and_ingest.py:158}} ERROR - {'status': 'error', 'message': Attribute
Traceback (most recent call last):
File "/usr/local/airflow/dags/eds_ingest/libs/src_dags_fetch_and_ingest.py", line 118, in fetch_and_ingest
if task_status.lower() == 'success' or current_try >= Constant.MAX_TRY:
```Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/ui/data-loading/osdu-cli/-/issues/22Unable to update existing records using OSDU CLI2023-11-23T14:55:49ZDurga Prasad Reddy NadavaluriUnable to update existing records using OSDU CLII am trying to ingest few records belongs to a reference schema using OSDU CLI. I have observed that OSDU CLI creates a new record if the record is not existed in the instance but fails to modify the existing record. I have searched for ...I am trying to ingest few records belongs to a reference schema using OSDU CLI. I have observed that OSDU CLI creates a new record if the record is not existed in the instance but fails to modify the existing record. I have searched for the open/closed issues but couldn't find anything related to update a record.
But in the verify.py file, its mentioned that it doesn't support versioning.
**"""Verify if records exist in OSDU.
This command will check whether id's in the specified manifest files exist in OSDU.
Note that this doesn't support versioning - success indicates that a record's id
is found, however there is no check of the contents so it could be an older version
if you have done multiple uploads of the same item with different content."""
```**
My question is that how we need to handle if user wants to update an existing record using manifest ingestion through OSDU CLI?https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/110Provide suitable clue (error message) when the batch size is large and no rec...2023-10-19T15:42:06ZDebasis ChatterjeeProvide suitable clue (error message) when the batch size is large and no records are processed/createdThe JSON payloads are clean. Actually, for my test cases these were JSON files from OSDU Reference data.
And Policy service is not enabled in my OSDU instance.
I run the payload the first time. It fails to create records.
Checked all th...The JSON payloads are clean. Actually, for my test cases these were JSON files from OSDU Reference data.
And Policy service is not enabled in my OSDU instance.
I run the payload the first time. It fails to create records.
Checked all the log files and they are all clean. So, that is very misleading.
I then split the initial payload into smaller pieces and then the job goes through smoothly.
What we need is a clear error message such as "in your current System configuration, it is not possible to handle payload of size larger than XXX".https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/116Add a sample DataAuthz policy using LegalTag extension properties for policy ...2023-10-31T22:24:02ZDadong ZhouAdd a sample DataAuthz policy using LegalTag extension properties for policy integration testingAdd a sample DataAuthz policy using LegalTag extension properties which will be included in policy integration testing.Add a sample DataAuthz policy using LegalTag extension properties which will be included in policy integration testing.Dadong ZhouDadong Zhouhttps://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/308Unit Test cases2023-10-18T16:04:47Zvikas ranaUnit Test casesData preparation should also part of unit test cases.Data preparation should also part of unit test cases.https://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/307'host' in koop-config.json - Configurable.2023-10-18T16:03:48Zvikas rana'host' in koop-config.json - Configurable.The 'host' value in the koop-config.json file is currently hard-coded as "127.0.0.1." We need to make this value configurable .The 'host' value in the koop-config.json file is currently hard-coded as "127.0.0.1." We need to make this value configurable .https://community.opengroup.org/osdu/data/open-test-data/-/issues/93Volve Stratigraphy2023-10-25T05:58:29ZThomas Gehrmann [slb]Volve StratigraphyAdd a complete stratigraphic column for Volve. This is based on the spreadsheet provided by Bjarne Bøklepp [Equinor] in the member GitLab [here](https://gitlab.opengroup.org/osdu/subcommittees/data-def/projects/well-delivery/docs/-/blob/...Add a complete stratigraphic column for Volve. This is based on the spreadsheet provided by Bjarne Bøklepp [Equinor] in the member GitLab [here](https://gitlab.opengroup.org/osdu/subcommittees/data-def/projects/well-delivery/docs/-/blob/master/Design%20Documents/WellLogExtensions/Volve_WellLog_examples/stratigraphy/volve_lithostratigraphy.xlsx?ref_type=heads).Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/issues/271"add data" API to cross-check Analysis properties and show error if any mismatch2024-01-11T12:39:45ZDebasis Chatterjee"add data" API to cross-check Analysis properties and show error if any mismatch## Initial request
Refer to related issue in Data Definition space. https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/619
We need to have enough information in the catalog (Work-product-component) a...## Initial request
Refer to related issue in Data Definition space. https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/619
We need to have enough information in the catalog (Work-product-component) about what measurements to expect in the "content". Plus strict check (as we find in Wellbore DDMS) to show error when we try to use more fields (than stated in WPC) for "add content data" step. WPC also does not provide much clue about what to expect in the "content" - single-value, multi-value and so on.
For the NMR example, there are more details when we see JSON payload.
```plaintext
"AvailableSampleAnalysisProperties": [
"SamplesAnalysisID",
"SampleID",
"FreshState",
"FullBrineSaturation",
"PartialBrineSaturation",
"LiquidFilledPorosity",
"Porosity",
"EffectivePorosity",
"BVI",
"FFI",
"Swi",
"T2Cutoff"
],
```
Not quite Page-92 of sample Kentish report.
```plaintext
"Saturation": {
"Value": 100.0,
"UnitOfMeasure": "{{data-partition-id}}:reference-data--UnitOfMeasure:%25:"
},
"T2": {
"Value": 5.01,
"UnitOfMeasure": "{{data-partition-id}}:reference-data--UnitOfMeasure:ms:"
},
"Porosity": {
"Value": 0.136,
"UnitOfMeasure": "{{data-partition-id}}:reference-data--UnitOfMeasure:%25:"
}
},
```
---
# Proposed Solution
**Precondition**: Content schema is recorded and existng SampleAnalysisID is defined in the content schema
* System counts the names of attributes, object names and array names specified in the content schema
* System drops duplicate names and updates AvailableSampleAnalysisProperties array in samplesanalysis record with names from content schema
* If array already included some values already, then they should preserve after updateMykhailo BuriakMykhailo Buriakhttps://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/306Seismic 3D Polygon and attributes2023-10-12T19:16:03ZDebasis ChatterjeeSeismic 3D Polygon and attributesInitial request to @LeviRemi
```
{
"attributes": {
"ProjectName": "LOCKWOOD 3D",
"ProjectEndDate": 1319040000000,
"Operator": "opendes:master-data--Organisation:CHESA...Initial request to @LeviRemi
```
{
"attributes": {
"ProjectName": "LOCKWOOD 3D",
"ProjectEndDate": 1319040000000,
"Operator": "opendes:master-data--Organisation:CHESAPEAKE:",
"OperatingEnvironmentID": "opendes:reference-data--OperatingEnvironment:Onshore:",
"SeismicGeometryTypeID": "opendes:reference-data--SeismicGeometryType:3D:",
"SampleInterval": "0.002",
"RecordLength": "3.0",
"FoldCount": "999",
"esriOid": 10
},
```
Do you think there is option to truncate value when appropriate?
Ex: Operator = CHESAPEAKE.
Operating Environment = Onshore
SeismicGeometry type = 3D
Also, check the ProjectEndDate.
And how about units of measure for fields such as Sample Interval, Record Length?
Was there any consideration to expose AreaCalculated or AreaNominal?https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/323SEGY to VDS DAG and SEGY to ZGY DAG deployment failing2023-10-12T07:23:53ZDiya AliasSEGY to VDS DAG and SEGY to ZGY DAG deployment failingHi, I am deploying OSDU M20 using Manual deployment. When I am trying to deploy SEGY to VDS DAG and SEGY to ZGY DAG with below commands its failing. Looks like the docker image tag specified in the manual deployment documentation is very...Hi, I am deploying OSDU M20 using Manual deployment. When I am trying to deploy SEGY to VDS DAG and SEGY to ZGY DAG with below commands its failing. Looks like the docker image tag specified in the manual deployment documentation is very old one. Please can anyone help to find the newest tag for deploying these DAGS.
Images used to deploy as per documentation:
- **msosdu.azurecr.io/segy-to-zgy-conversion-dag:0.11.0**
- **msosdu.azurecr.io/segy-to-vds-conversion-dag:0.11.0**
- Documentation Link: https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/release/0.23/docs/service-data.md?ref_type=heads
Error received for both DAG deployments:
Error while registering DAG {"code":403,"reason":"Non empty dag content obtained","message":"Setting dag content not allowed"}https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/40Code coverage reporting in CI/CD pipeline2024-02-27T09:10:33ZOm Prakash GuptaCode coverage reporting in CI/CD pipelineCurrently, We do not see any adoption for using code coverage in any of the pipelines. It's very important to maintain good code quality as OSDU platform gets adopted by different vendors/clients.
Observation:
- we do see that there ar...Currently, We do not see any adoption for using code coverage in any of the pipelines. It's very important to maintain good code quality as OSDU platform gets adopted by different vendors/clients.
Observation:
- we do see that there are unit test cases already written but not updated diligently.
- there were issues created long back to have code coverage in OSDU standard pipelines but No action on implementation e.g. here https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/19.
- Not all projects have unit test coverage written which shall be standard practice.
Implementation:
- We got to use code coverage in the pipeline itself. We could enable Git lab code coverage in the pipeline Documentation here https://docs.gitlab.com/ee/ci/testing/code_coverage.html
- Get contributions from vendors to make existing unit test cases updated with the current code.
- Circulate standard guidelines to contribute to any project with unit test cases included and up-to-date with at least 80% code coverage.
- Once unit tests are updated keep failures in the pipeline at least at 70%.
tools from git lab documentation can be used:
- gcovr (C/C++)
- JaCoCo (Java/Kotlin)
- pytest-cov (Python)Om Prakash GuptaOm Prakash Guptahttps://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/305Data - Provide Support for Grav/Mag data2023-10-11T15:42:22ZNoel OkanyaData - Provide Support for Grav/Mag datahttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/117[ADR] Advanced filters for dataset search2023-12-04T14:23:23ZAlexandre Gattiker[ADR] Advanced filters for dataset search# Introduction
We need additional filtering support to be able to filter the `POST /dataset/tenant/{tenantid}/subproject/{subprojectid}` and `PUT /operation/bulk-delete` (added in [!891](https://community.opengroup.org/osdu/platform/dom...# Introduction
We need additional filtering support to be able to filter the `POST /dataset/tenant/{tenantid}/subproject/{subprojectid}` and `PUT /operation/bulk-delete` (added in [!891](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/891/diffs#fafb01a8314993d61fca390beef912c7813278eb)) operations by metadata fields with more complex expressions than a single key-value match.
# Status
* [x] Initiated
* [x] Proposed
* [x] Under Review
* [ ] Approved
* [ ] Rejected
# Problem statement
The SDMS API `POST /dataset/tenant/{tenantid}/subproject/{subprojectid}` currently accepts the following body parameters, among others:
* `search`, a single SQL-like search parameter, for example: `search=name=file%`
* `gtags`, an array of strings matching tags associated with dataset metadata.
The `search` field does not support more than one field, or more than one possible value for a field.
The SDMS API `PUT /operation/bulk-delete` (added in [!891](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/891/diffs#fafb01a8314993d61fca390beef912c7813278eb)) requires a `path` parameter containing `tenantid`, `subprojectid` and `path` but does not support filtering by metadata fields or tags.
For both search and delete, we need to be able to filter by more than one field, or more than one possible value for a field.
Furthermore, we expect a need for more complex filter solutions, such as combining `AND`, `OR` and `NOT` operators. The proposed solution should ideally be extensible to support additional expressions and operators in the future if needed.
# Proposed solution
Add an optional `filter` parameter to the `POST /dataset/tenant/{tenantid}/subproject/{subprojectid}` and `PUT /operation/bulk-delete` API endpoints.
The `search` and `gtags` parameters are to be deprecated.
## Overview
The `filter` parameter can take a payload with a variable format, allowing expressing a simple filter on a single field, as well as logical combinations of filters with arbitrary complexity.
The `POST /dataset/tenant/{tenantid}/subproject/{subprojectid}` operation has been selected for extension because:
* Advanced metadata filtering, encompassing select and search functionalities, has already been incorporated into that operation.
* The SDMS API also accepts the `GET` method for the operation with parameters provided in the query string, as a legacy endpoint. The `POST` version of the endpoints has been introduced to address issues related to handling large request parameters, where sending the cursor as a query parameter can lead to oversized requests and subsequent failures.
## Examples
Example value for the `filter` parameter:
```json
{
"and": [
{
"not": {
"property": "gtags",
"operator": "CONTAINS",
"value": "tagA"
}
},
{
"or": [
{
"property": "name",
"operator": "LIKE",
"value": "test.%"
},
{
"property": "name",
"operator": "=",
"value": "dataset.sgy"
}
]
}
]
}
```
This is equivalent to the following pseudo-SQL statement:
```sql
SELECT * FROM datasets d WHERE
NOT (EXISTS (SELECT VALUE 1 FROM t IN d.data.gtags WHERE t = 'tagA')
OR (IS_STRING(d.data.gtags) AND STRINGEQUALS(d.data.gtags, 'tagA')))
AND (
d.name LIKE 'test.%'
OR d.name = 'dataset.sgy'
)
```
## Details
The `filter` parameter can be:
* A **property match filter**:
```json
{
"property": "...",
"operator": "...",
"value": "..."
}
```
The implementation will be extensible with additional keys if needed in the future, e.g. to specify case sensitivity.
* An **`and` or `or` filter**, i.e. an object containing only the key `and` or `or`, of which the value is an array of one or more filters (i.e. a property match filter or an `and`, `or` or `not` filter)
```json
{
"and": [...]
}
```
* A **`not` filter**, i.e. an object containing only the key `not`, of which the value is a filter (i.e. a property match filter or an `and`, `or` or `not` filter)
```json
{
"not": ...
}
```
# Out of scope / limitations
The operations at `GET /utility/ls` and `POST /utility/ls` can also be used for retrieving datasets, but will not be extended with advanced filtering at the moment. That functionality can be added later if required.Diego MolteniDiego Moltenihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/issues/264Extensibility - provide way to add new test types (V2 API)2023-11-28T15:52:39ZDebasis ChatterjeeExtensibility - provide way to add new test types (V2 API)Following up from discussion of 5-Oct-2023, we are looking for ways to add new test types' support.
see enclosed -[rafs-debasis.xlsx](/uploads/db2f18cb58a7b294ce5457dfeea17976/rafs-debasis.xlsx)
It would be ideal if this can be achieve...Following up from discussion of 5-Oct-2023, we are looking for ways to add new test types' support.
see enclosed -[rafs-debasis.xlsx](/uploads/db2f18cb58a7b294ce5457dfeea17976/rafs-debasis.xlsx)
It would be ideal if this can be achieved by simply adding suitable values in Reference data, showing new test type and fields obtained from measurement and/or analysis.
In the worst case, it would be useful to document the approach about what kind of additional coding would be required.
With "Hello world" style example.
We can assume simple "content schema" for this requirement.
Catalog would have list of measurement fields and content would have exactly that.
Example of wellboreTrajectory schema in Wellbore DDMS.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blob/master/Examples/work-product-component/WellboreTrajectory.1.3.0.json
```
"AvailableTrajectoryStationProperties": [
{
"TrajectoryStationPropertyTypeID": "partition-id:reference-data--TrajectoryStationPropertyType:AzimuthTN:",
"StationPropertyUnitID": "partition-id:reference-data--UnitOfMeasure:dega:",
"Name": "AzimuthTN"
}
],
```https://community.opengroup.org/osdu/platform/system/storage/-/issues/185ADR: API to retrieve past events of storage records2023-10-11T16:28:52ZYifan YeADR: API to retrieve past events of storage recordsNew API in Storage service to rehydrate past creation and last modified events for a given kind within the given time range.
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [ ] Approved
* [ ] Retired
## Context & Scope
The OSDU Sto...New API in Storage service to rehydrate past creation and last modified events for a given kind within the given time range.
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [ ] Approved
* [ ] Retired
## Context & Scope
The OSDU Storage service does not provide a way to retrieve past events of the records been created/modified. Many OSDU applications would be interested in retrieving the past events of the records that happened before the application subscribed to the notification service. This new API proposed in the ADR will provide the concerned applications with a way to backtrack the events.
The proposal is to provide an API on storage service to support retrieving past events of records of a kind that happened in the given time range, where the events will be returned in a paginated format and ascending chronological order based on the timestamp.
The new API will retrieve the first and the last events of the record, filter the events by the start date and end date provided by the user, and then return the filtered events.
## Tradeoff Analysis
The new API does not represent a breaking change of any other API, and consequently neither for the consuming applications. Only concerned-consuming applications would benefit from this new feature, while it remains entirely transparent for others.
## Decision
Provide an API to query past events of records of the given kind and return the events in paginated ascending chronological order.
{
“id”: \<RECORD_ID\>
“kind”: \<KIND\>
“op”: \<CREATE|UPDATE|DELETE, etc\>
"version": \<VERSION\>
"timestamp": \<TIMESTAMP\>\
}
## Consequences
* A new API on the Storage service would be available.
* Documentation of the Storage service should be modified with details for the new API.Yifan YeYifan Yehttps://community.opengroup.org/osdu/platform/deployment-and-operations/base-containers-gcp/airflow-helm-chart/-/issues/1Airflow task are spending lot of time in scheduled state2023-11-27T15:27:34Zdevesh bajpaiAirflow task are spending lot of time in scheduled state**Steps to reproduce:** Upload 1000 small csv files (5 raw per CSV) and trigger SLB CSV workflow
**Expectation:** Jobs should get completed in 10- 15 minutes.
**What actually happened:** Jobs are taking hours to complete.
**Observation:*...**Steps to reproduce:** Upload 1000 small csv files (5 raw per CSV) and trigger SLB CSV workflow
**Expectation:** Jobs should get completed in 10- 15 minutes.
**What actually happened:** Jobs are taking hours to complete.
**Observation:** Airflow task are waiting for long time (25-30 minutes) in scheduled state before getting queued. Once queued task takes few seconds to execute.
Please find our analysis here
[Airflow_performance_issue_investigation_results.pdf](/uploads/71c3f37a2987d720c8cc20dbcbf5595e/Airflow_performance_issue_investigation_results.pdf)Nur SheikhLucy LiuNur Sheikh