OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2021-10-28T13:19:37Zhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/206Azure provisioning requires a trusted domain for central_resources2021-10-28T13:19:37ZChris SmithAzure provisioning requires a trusted domain for central_resourcesWhen running the `central_resources` terraform scripts for Azure, the build fails with the following error:
```
Value of identifierUris property must use a verified domain of the organization or subdomain: 'http://osdu-mvp-crdemo-iesk-a...When running the `central_resources` terraform scripts for Azure, the build fails with the following error:
```
Value of identifierUris property must use a verified domain of the organization or subdomain: 'http://osdu-mvp-crdemo-iesk-app'"
{"item": "PropertyErrorCode", "value": "HostNameNotOnVerifiedDomain"}
{"item": "HostName", "value": "http://osdu-mvp-crdemo-iesk-app"}
with module.ad_application_azuread_application.main[0].
on ../../../modules/provides/azure/ad-application/main.tf line 20, in resource "azuread_application" "main":
20: resource azuread_application" "main" {
```
Microsoft [enforced this change on Oct 15, 2021](https://docs.microsoft.com/en-us/azure/active-directory/develop/reference-breaking-changes#appid-uri-in-single-tenant-applications-will-require-use-of-default-scheme-or-verified-domains).
We worked some updates based on the `release/0.11` tag and will push a branch PR shortly for review.https://community.opengroup.org/osdu/platform/home/-/issues/44[ADR] Avoid the need to provide persistable reference information (Unit syste...2023-11-17T17:48:56ZDebasis Chatterjee[ADR] Avoid the need to provide persistable reference information (Unit system, Coordinate Reference System)## Status
- [x] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context
Currently, user needs to provide both the Reference Entity information and the persistable reference.
Evident when user needs to specify...## Status
- [x] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context
Currently, user needs to provide both the Reference Entity information and the persistable reference.
Evident when user needs to specify unit of measure or coordinate reference system.
This is inefficient and is also error prone.
What if the Data Loader or the User makes a mistake in persistable reference value and the values are inconsistent?
The proposal is to save the user this trouble. Let the user provide link to existing Reference entity and ID.
However, programs such as Manifest-based Ingestion could query Reference value and add required line in JSON file being used to actually store/populate record.
See this high level diagram for an understanding of this proposal.
[ADR-for-persistableReference-issue.pptx](/uploads/95b748500e8f64544ff64a492836515d/ADR-for-persistableReference-issue.pptx)
You can find historical context in the following issues:
- https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/92
- https://community.opengroup.org/osdu/platform/system/reference/unit-service/-/issues/24
- https://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service/-/issues/25
## Scope
Make suitable change in code :
- Manifest-based Ingestion (both unit and CRS)
- Unit conversion
- CRS conversion
Likely impact for CSV Parser and WITSML Parser.
## Decision
Analyze impact (if adverse) caused by this suggested change :
1. Approve change to Manifest-based Ingestion
2. Approve change to Unit Conversion API
3. Approve change to CRS Conversion API
## Rationale
The proposal is cleaner approach since the lengthy persistable reference can be daunting for some of the users.
There is chance of user making mistake in the string of persistableReference.
Also, there can be contradiction in provided input (such as reference to certain entry in Reference data but different persistableReference).
## Consequences
- Revise sample JSON files (used for loading TNO, Volve data)
- Revise test Postman collections (Platform Validation team)
- Revise documentation (Data Loading)
- Revise API Swagger documentationhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/58CSV: Ability to define relationships based on multiple keys2021-10-27T09:54:20ZFernando Nahu Cantera RubioCSV: Ability to define relationships based on multiple keysIngestion workflow improvement
CSV ingestor will allow us to define a key composed of multiple attributes to search a related objectIngestion workflow improvement
CSV ingestor will allow us to define a key composed of multiple attributes to search a related objectM10 - Release 0.13Fernando Nahu Cantera RubioFernando Nahu Cantera Rubiohttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/39[ADR] seismic storage tiers support2023-09-08T06:37:16ZDiego Molteni[ADR] seismic storage tiers support# Introduction
This ADR proposes how support to multi storage tiers should be enabled in SDMS to better manage storage costs.
## Status
- [x] Initiated
- [ ] Proposed
- [ ] Under Review
- [ ] Approved
- [ ] Rejected
## SDMS dataset c...# Introduction
This ADR proposes how support to multi storage tiers should be enabled in SDMS to better manage storage costs.
## Status
- [x] Initiated
- [ ] Proposed
- [ ] Under Review
- [ ] Approved
- [ ] Rejected
## SDMS dataset concepts and ingestion overview
A dataset resource, in SDMS, is identified by the follow URI string
**`sd://tenant/subproject/path/dataset`** where:
- **tenant**: is the unique data-partition-id.
- **subproject**: is the name of the data group.
- **path**: is a virtual path in the subproject (a folder three).
- **dataset**: is the name of the dataset.
For example in the dataset **`sd://opendes/sandbox/processing/2023/result.zgy`**
- **tenant** = opendes
- **subproject** = sandbox
- **path** = /processing/2023/
- **dataset** = result.zgy
A dataset in SDMS is composed by a metadata descriptor, maintained in the SDMS db catalogue, and a set of objects saved in a cloud storage resource. A dataset through SDMS is always seen as a single entity even if its data content has been split into multiple objects.
In general, SDMS, has a dedicated storage resource per partition and all objects composing datasets are stored into it. All objects composing a dataset are saved into the storage account in different way based on the storage policy applied at data group level (subproject):
- **subproject access policy = "uniform"**: access is granted at data group level. A subproject's writer/reader can write-read/read any dataset in the subproject. For each subproject a dedicated storage resource is created and all objects composing the dataset are saved under a virtual folder path. For example, in Azure, a dataset is saved into storage-account(per partition)\container(per subproject)\virtual-folder(per dataset)\object_0...object_N.
- **subproject access policy = "dataset"**: access is granted at dataset level. A dataset writer/reader can write-read/read only the datasets he has access too. For each dataset a dedicated storage resource is created and all objects composing the dataset are saved under a virtual folder path. For example, in Azure, a dataset is saved into storage-account(per partition)\container(per dataset)\virtual-folder(per dataset)\object_0...object_N.
When a dataset is uploaded to SDMS, the following ingestion flow is executed:
1. Client register the dataset in SDMS. SDMS will create a dataset descriptor in the internal catalogue and reserve a storage area where upload dataset composing objects.
1. SDMS returns the descriptor metadata to the client.
1. Client request a connection string to SDMS for the reserved storage resource.
1. SDMS returns the generated connection string to the Client.
1. Client split the dataset into multiple object and upload them to the reserved storage resource.
1. Client request to SDMS to close the dataset.
![image](/uploads/3efc31ddabf2b406b942c6cdce0b2594/image.png)
## Storage Tiers and SDKs support
To provide a cost-effective solution, SDMS must enable storage tiers management features in order to save dataset's composing objects to a specific storage tier class. For example, in Azure, supported tiers class are **Hot**, **Cold**, **Archive**. If data objects can be saved into a **Cold** tier instead of a **Hot**, a cost saving will be generated for clients.
An object's tier can be set or updated directly by calling cloud storage methods when an object is uploaded or manipulated.
![image](/uploads/26c2ecf612ac2ff9f0a48aa249489824/image.png)
These operations are executed from client applications through CSP provided SDKs. The SDMS suite offers 2 clients libraries: a C++ client library **SDAPI** and a Python command line utilities **SDUTIL**.
This ADR presents how SDMS service and provided client libraries, **SDAPI** and **SDUTIL**, should be enhanced to support object's tiering features.
## Set storage tier class
This features enables consumer to set the desired storage tier class when objects are uploaded.
In SDAPI, we will add a storage tier class argument to the generic dataset opening method. This will be set as default storage tier class when a dataset object is uploaded. If not set, the default storage tier class will be used (Hot for Azure). In addition the tiering argument will be added to both dataset and utility upload method provided to ingest in a local dataset in SDMS as single object.
```c++
// open a dataset specifying the storage tier class.
SDGenericDataset dataset(&manager, "sd://tenant/subproject/path/dataset");
dataset.open(SDDatasetDisposition::CREATE|OVERRIDE, {
{ api::json::Constants::tier, Tier::<tier-class>}});
// object will be uploaded with the dataset specified <tier-class>
dataset.write("object_name", data, size);
// save the storage tier information in the manifest
dataset.close();
// upload a dataset: generic dataset class
SDGenericDataset dataset(&manager, "sd://tenant/subproject/path/dataset");
dataset.upload("fileToUploadPath", Tier::Cold);
// save the storage tier information in the manifest
dataset.close();
// upload a dataset: utility class
SDUtils utils(&manager);
utils.upload("sd://tenant/subproject/path/dataset", "fileToUploadPath", Tier::Cold);
```
In case the dataset already exist and is opened with a READ_WRITE disposition, the tier should be set as the one specified in the manifest. In case this is not present the default one should be applied.
The SDUTIL utility, does not provide methods to manipulate objects. Dataset are uploaded to SDMS via **cp** command that automatically split the dataset into multiple objects and upload them in the storage resource. The tier class can be specified in the upload version of the **cp** command. All dataset's composing objects will be uploaded to the **cp** command specified tier class.
```python
sdutil cp data sd://tenant/subproject/path/data --tier="<tier_class>"
```
In both SDUTIL and SDAPI, when dataset is closed (at the end of an upload operation), the storage tier class value must be set in the dataset manifest in order to make the tier information available and trusted by ingestion and consuming applications.
```json
manifest: {
type: "most of the times set to \"GENERIC\"",
nojbects: "number of dataset's objects",
size: "the dataset size",
checksum: "the dataset checksum",
tier_class: "the storage tier class"
}
```
Please note that SDMS is in control of changes made by SDAPI and SDUTIL applications only. if other apps are used to change the dataset object's storage tier class, these must also updated the dataset manifest(by calling the PATCH /dataset endpoint).
## Update storage tier class
This features enables consumer to updated the desired storage tier class to an uploaded object.
In SDAPI, we will add a new method to the generic dataset and the utility classes to update the dataset object's storage tier class.
```c++
// uploaded a dataset tier class: generic dataset class
SDGenericDataset dataset(&manager, "sd://tenant/subproject/path/dataset");
dataset.open(SDDatasetDisposition::READ_WRITE);
// update the storage class tier of all dataset's objects
dataset.update(Tier::<Tier::class>);
// save the storage tier information in the manifest
dataset.close()
// uploaded a dataset tier class: utility class
SDUtils utils(&manager);
utils.update("sd://tenant/subproject/path/dataset", Tier::Cold);
```
In **SDUTIL**, we will update the **patch** command to update the storage tier class to all dataset's objects :
```python
sdutil patch sd://tenant/subproject/path/dataset --tier=<tier_class>
```
## Retrieve storage tier class
To know what is the dataset's storage tier class, client applications can retrieve the dataset descriptor and read the content of the associated value in the manifest. both SDAPI and SDUTIL should be updated to expose the new value. In SDAPI the dataset model will be updated adding the extra property and in SDUTIL we will enhanced the **stat** command by adding the tier class information to the detailed command output.
```
- Name: sd://test-partition/sandbox/cube.zgy
- Created By: dmolteni3@.com
- Created Date: Tue May 16 2023 11:16:08 GMT+0000 (Coordinated Universal Time)
- Size: 36.0 MB
- No of Objects: 2
- Legal Tag: test-partition-default-legal
- Storage Tier Class: Hot
```Diego MolteniDiego Moltenihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/40Indexer, Frame of Reference, DateTime conversion2022-09-29T13:41:07ZDebasis ChatterjeeIndexer, Frame of Reference, DateTime conversionCan you please share a working example of DataTime conversion?
Load manifest (JSON) showing actual data and also matching "meta" block for a date field.
**I tried the following in Load Manifest (AWS R3M8 Preship environment) -**
Data...Can you please share a working example of DataTime conversion?
Load manifest (JSON) showing actual data and also matching "meta" block for a date field.
**I tried the following in Load Manifest (AWS R3M8 Preship environment) -**
Data has this line -
` "ProjectEndDate": "2008-02-01T14:00:00.000+03:00",`
Meta block has this for DateTime conversion -
```
{
"kind": "DateTime",
"name": "UTC-ISO8601",
"persistableReference": "{\"format\":\"yyyy-MM-ddTHH:mm:ss.fffZ\",\"timeZone\":\"UTC\",\"type\":\"DTM\"}",
"propertyNames": [
"ProjectEndDate ]
}
```
**I see this error when I query for troubleshooting (Search service) .**
For now, we can simply discuss problem of DateTime conversion.
```
{
"kind": "osdu:wks:master-data--SeismicAcquisitionSurvey:1.0.0",
"query": "id: \"osdu:master-data--SeismicAcquisitionSurvey:ST0202R08-DC-23Oct\"",
"returnedFields": [
"id",
"index"
]
}
```
Response
```
{
"results": [
{
"index": {
"trace": [
"Unit conversion: persistableReference not valid",
"Unit conversion: persistableReference not valid",
"DateTime conversion: Frame of reference does not match given data for property ProjectEndDate, no conversion applied."
],
"statusCode": 400,
"lastUpdateTime": "2021-10-23T10:13:21.496Z"
},
"id": "osdu:master-data--SeismicAcquisitionSurvey:ST0202R08-DC-23Oct"
}
],
"totalCount": 1
}
```
Thanks for your help.
cc - @gehrmann for information
cc - @jingdongsun , @anujgupta and @shamazum (since some work was done by IBM resources in this area, as far as I remember)https://community.opengroup.org/osdu/platform/system/home/-/issues/91ADR: Address 5000 groups quota limit by providing new APIs to give status on ...2022-06-20T11:31:53ZAn NgoADR: Address 5000 groups quota limit by providing new APIs to give status on group usage# Handle 5000 group quota
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
Currently, we do not have a way to identify beforehand that the 5000 groups quota is about to be hit...# Handle 5000 group quota
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
Currently, we do not have a way to identify beforehand that the 5000 groups quota is about to be hit. Currently users reach out to us once this quota is hit and they are being blocked because of this until the cleanup of groups is done. We need a better way to identify this before the quota is actually hit.
There are 2 quotas that need to be addressed:
- 5000 group membership per identity.
- 5000 user + data group existence per data partition.
Proposed API changes:
- Update existing API (Entitlement) CreatedByApp, CreatedBy with CreatedGroup
- New API (Entitlement):
- Aggregation of number of Groups per partition
- Aggregation of number of Groups per group types
- Aggregation of number of Groups per application
- CreateByApp, CreatedBy
## Decision
## Rationale
The consuming applications should be alerting their users when the quota is about to hit, ideally when 90% of the quota is reached. Data Platform will provide the APIs for checking group usage. Then the consuming applications can leverage this to alert their users, or do the clean-up.
## Consequences
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
## Decision criteria and tradeoffs
## Decision timelinehttps://community.opengroup.org/osdu/platform/system/storage/-/issues/94Remove deprecated storage schema api feature flag and relevant integration tests2022-11-21T11:04:04ZLarissa PereiraRemove deprecated storage schema api feature flag and relevant integration testsThe following endpoints are currently driven by feature flag "schema_endpoints_disabled". MR (https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/251)
* POST endpoint(**/api/storage/v2/schema**) in storage servi...The following endpoints are currently driven by feature flag "schema_endpoints_disabled". MR (https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/251)
* POST endpoint(**/api/storage/v2/schema**) in storage service.
* GET endpoint(**/api/storage/v2/schema**) in storage service
* DELETE endpoint(**/api/storage/v2/schema**) in storage service
* GET endpoint(**/api/storage/v2/query/kinds**) in storage service
When the feature flag is removed please ensure to also update integration tests to fix/remove any tests that are using these deprecated endpoints. Currently these tests are being ignored if the feature flag is true (viz. schema_endpoints_disabled = true) as part of MR (https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/257)
**Update** -- the GET /query/kinds endpoint ended up being taken out from under the feature flag and retained as it is needed for the storage concern around what kinds are actually in storage vs. merely defined in the Schema Service.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/issues/27Log recognition - to make use of Logging company2023-06-19T12:17:49ZDebasis ChatterjeeLog recognition - to make use of Logging companyIt seems the current implementation does not take into account Logging company.
```
{
"description": "LDTD Gamma Ray",
"label": "GRD",
"log_unit": "GAPI"
}
```
As far as I know from Energistics/PWLS (@jayholl), recognition proces...It seems the current implementation does not take into account Logging company.
```
{
"description": "LDTD Gamma Ray",
"label": "GRD",
"log_unit": "GAPI"
}
```
As far as I know from Energistics/PWLS (@jayholl), recognition process should take two inputs -
- Logging company
- Curve Mnemonic
Output -
Either "**Not available**" (PWLS dictionary does not have this combination)
or
the triplet (**LogCurveMainFamily -> LogCurveFamily -> LogCurveType**)
I am showing this sample ID from the leaf entity.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blob/master/E-R/reference-data/LogCurveType.1.0.0.md
**BakerHughesInteq:A6A0R**
As you can see, this is combination of Contractor and also Curve Mnemonic.
cc - @chad and @s0rhe1m for informationhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/37Support auth with access_token2023-03-27T19:17:39ZAleksandr Spivakov (EPAM)Support auth with access_tokenCurrently service supports only id_token for authorization. It will be good to have support for access_token.Currently service supports only id_token for authorization. It will be good to have support for access_token.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/issues/26memory leak with data chunking2023-03-16T20:34:29ZYunhua Koglinmemory leak with data chunkingHello, When we have dask for data chunking features, memory is not released
![image__1_](/uploads/7e3da10faa83fe89d243992cddfad3d7/image__1_.png)Hello, When we have dask for data chunking features, memory is not released
![image__1_](/uploads/7e3da10faa83fe89d243992cddfad3d7/image__1_.png)https://community.opengroup.org/osdu/platform/deployment-and-operations/audit-and-metrics/-/issues/39Elasticity: Autoscaling2021-10-18T11:43:23ZStephen Whitley (Invited Expert)Elasticity: Autoscaling# SLI Title
## SLI Group
- [ ] Data Access & Egress
- [ ] Data Access Rights
- [ ] Data Governance
- [ ] Data Ingest
- [ ] Data Quality
- [ ] Data Volume
- [ ] Delivery
- [X] Platform Performance
- [X] Platform Cost
- [ ] Platform Trac...# SLI Title
## SLI Group
- [ ] Data Access & Egress
- [ ] Data Access Rights
- [ ] Data Governance
- [ ] Data Ingest
- [ ] Data Quality
- [ ] Data Volume
- [ ] Delivery
- [X] Platform Performance
- [X] Platform Cost
- [ ] Platform Traction
- [ ] Search
- [ ] Other
## How Measured
Capture Autoscaling events (scale up/scale down)
## Dependencieshttps://community.opengroup.org/osdu/platform/deployment-and-operations/audit-and-metrics/-/issues/38Data Ingestion Status2021-10-18T11:41:34ZStephen Whitley (Invited Expert)Data Ingestion Status# Breakdown of Data Ingestion by Status
Error : violates ingestion rule
Failure : The workflow fails
Success : The ingestion job completes correctly.
## SLI Group
- [ ] Data Access & Egress
- [ ] Data Access Rights
- [ ] Data Governanc...# Breakdown of Data Ingestion by Status
Error : violates ingestion rule
Failure : The workflow fails
Success : The ingestion job completes correctly.
## SLI Group
- [ ] Data Access & Egress
- [ ] Data Access Rights
- [ ] Data Governance
- [X] Data Ingest
- [ ] Data Quality
- [x] Data Volume
- [ ] Delivery
- [ ] Platform Performance
- [ ] Platform Traction
- [ ] Search
- [ ] Other
## How Measured
Ingestion Logs
## Dependencieshttps://community.opengroup.org/osdu/platform/data-flow/real-time/streams/stream-admin-service/-/issues/13Source / Fake ETP Producer2021-12-10T14:23:57ZDmitry KniazevSource / Fake ETP ProducerTahir AliTahir Alihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/39Enhance documentation to explain how one can troubleshoot issues in Indexer (...2023-03-09T18:15:53ZDebasis ChatterjeeEnhance documentation to explain how one can troubleshoot issues in Indexer (Normalizer)Please see related issue #32 .
Suggest adding suitable notes in Core Services (Indexer) documentation.
https://community.opengroup.org/osdu/documentation/-/wikis/Core-Services-Overview
cc - @nthakur for informationPlease see related issue #32 .
Suggest adding suitable notes in Core Services (Indexer) documentation.
https://community.opengroup.org/osdu/documentation/-/wikis/Core-Services-Overview
cc - @nthakur for informationhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/issues/12[GCP] Can't download data from Seismic Store if it consists of more than one ...2023-03-30T16:54:59ZYan Sushchynski (EPAM)[GCP] Can't download data from Seismic Store if it consists of more than one fileAfter uploading oVDS dataset to Seismic Store with SEGY->oVDS converter, I want to download the result to my local machine.
But I got this error
![image](/uploads/ebb3058d89b9dd27ddf937bd259b8125/image.png)
As I understand, `sdutil` us...After uploading oVDS dataset to Seismic Store with SEGY->oVDS converter, I want to download the result to my local machine.
But I got this error
![image](/uploads/ebb3058d89b9dd27ddf937bd259b8125/image.png)
As I understand, `sdutil` uses `gcsurl` of the dataset and attempts to download it directly from the bucket, but it can't do this if the dataset consists of more than one file.
This is how the oVDS dataset looks in the bucket
![image](/uploads/361a48c79afa88e2542dd269d4cb94b8/image.png)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/home/-/issues/19sdutil help to show recursive listing "ls -r"2022-07-27T10:22:38ZDebasis Chatterjeesdutil help to show recursive listing "ls -r"When working in GCP Preship environment, @Yan_Sushchynski showed me this possibility.
I think this could be useful for "Help" to show as advanced option.
```
> python sdutil [command]
available commands:
* app : application auth...When working in GCP Preship environment, @Yan_Sushchynski showed me this possibility.
I think this could be useful for "Help" to show as advanced option.
```
> python sdutil [command]
available commands:
* app : application authorization utilities
* auth : authentication utilities
* config : manage the utility configuration
* cp : copy data to(upload)/from(download)/in(copy) seismic store
* ls : list subprojects and datasets
* mk : create a subproject resource
* mv : move a dataset in seismic store
* patch : patch a seismic store subproject or dataset
* rm : delete a subproject or a space separated list of datasets
* stat : print information like size, creation date, legal tag(admin) for a space separated list of tenants, subprojects or datasets
* unlock : remove a lock on a seismic store dataset
* user : user authorization utilities
* version : print the sdutil version
(sdutilenv) C:\seismic-store-sdutil-master>
```https://community.opengroup.org/osdu/platform/data-flow/real-time/streams/stream-admin-service/-/issues/12Admin / Info Stream2021-11-12T15:02:24ZDmitry KniazevAdmin / Info StreamImplement the GET /stream/{id}/info method of the streaming API in the [controller](https://community.opengroup.org/osdu/platform/data-flow/real-time/streams/stream-admin-service/-/blob/develop/src/main/java/org/opengroup/osdu/streaming/...Implement the GET /stream/{id}/info method of the streaming API in the [controller](https://community.opengroup.org/osdu/platform/data-flow/real-time/streams/stream-admin-service/-/blob/develop/src/main/java/org/opengroup/osdu/streaming/api/StreamingAdminControllerImpl.java) using the DeploymentAdminService (to be created):
- [ ] get stream definition record from the storage service using the stream id (input parameter)
- [ ] extract deployment id from ExtensionProperties
__need brainstorming how to collect the status of the deployment__
- [ ] handle exceptions and return the appropriate HTTP code
- [ ] create tests for each possible return codeStephen NimmoStephen Nimmohttps://community.opengroup.org/osdu/platform/data-flow/real-time/streams/stream-admin-service/-/issues/10Sink / Web Messages Consumer2021-10-29T16:40:48ZDmitry KniazevSink / Web Messages ConsumerWrite a simple web app (Python|Node|Java) **packaged as a docker container** that connects to Kafka, consumes messages from a topic and visualizes this information on a web page.
Input parameters are passed as env variables:
```
bootst...Write a simple web app (Python|Node|Java) **packaged as a docker container** that connects to Kafka, consumes messages from a topic and visualizes this information on a web page.
Input parameters are passed as env variables:
```
bootstrap.servers - list of brokers to bootstrap kafka connection
OSDU_STREAMS_SUBSCRIBEIDS - the list of message keys to monitor in the source topic and route to sink topic
OSDU_STREAMS_SOURCEBINDINGS - the list of source topics to read messages from
OSDU_STREAMS_SINKBINDINGS - the list of sink topics to write the messages to
```
- [ ] On start-up extract the parameters from env variables:
- [ ] bootstrap.servers = localhost:9092 (list of brokers to bootstrap Kafka connection)
- [ ] OSDU_STREAMS_SUBSCRIBEIDS = "opendes:work-product-component--WellLog:be54a691c0384182944d71c6b2b6f699"
- [ ] OSDU_STREAMS_SOURCEBINDINGS = "opendes_myApp--WellLog_1.0.0"
- [ ] OSDU_STREAMS_SINKBINDINGS = "http://localhost:8080" (not used in this application)
- [ ] establish connection with Kafka using bootstrap servers and topics information
- [ ] run infinite loop of reading messages from the source topic, extract the measurements from the avro message and visualize it on the page
- [ ] create tests for the appDouglas DohmeyerDouglas Dohmeyerhttps://community.opengroup.org/osdu/platform/data-flow/real-time/streams/stream-admin-service/-/issues/9Processor / Messages Router2021-10-25T16:53:20ZDmitry KniazevProcessor / Messages RouterThis is a simple Kafka Stream App (Java) that reads messages from Source topic and routes them (with no change) to a sink topic.
Input parameters are passed as env variables:
```
bootstrap.servers - list of brokers to bootstrap kafka c...This is a simple Kafka Stream App (Java) that reads messages from Source topic and routes them (with no change) to a sink topic.
Input parameters are passed as env variables:
```
bootstrap.servers - list of brokers to bootstrap kafka connection
OSDU_STREAMS_SUBSCRIBEIDS - the list of message keys to monitor in the source topic and route to sink topic
OSDU_STREAMS_SOURCEBINDINGS - the list of source topics to read messages from
OSDU_STREAMS_SINKBINDINGS - the list of sink topics to write the messages to
```
- [ ] On start-up extract the parameters from env variables:
- [ ] bootstrap.servers = localhost:9092 (list of brokers to bootstrap Kafka connection)
- [ ] OSDU_STREAMS_SUBSCRIBEIDS = "opendes:work-product-component--WellLog:be54a691c0384182944d71c6b2b6f699"
- [ ] OSDU_STREAMS_SOURCEBINDINGS = "opendes_wks_work-product-component--WellLog_1.0.0"
- [ ] OSDU_STREAMS_SINKBINDINGS = "opendes_myApp--WellLog_1.0.0"
- [ ] establish connection with Kafka using bootstrap servers and topics information
- [ ] create the topology to route messages from source topic to sink topic with no changes
- [ ] create tests for the Kafka Streams AppDmitry KniazevDmitry Kniazevhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/106Loading of 50,000 records using Osdu_ingest DAG for IBM platform - job did no...2022-08-23T12:49:24ZKamlesh TodaiLoading of 50,000 records using Osdu_ingest DAG for IBM platform - job did not finish was left running for more than 48 hoursThe job submitted to ingest 50000 records did not complete after letting it run for more than 48 hours. Looking at the airflow log it appeared to be saving the records, but it did not return status of finished, failed, or success. It alw...The job submitted to ingest 50000 records did not complete after letting it run for more than 48 hours. Looking at the airflow log it appeared to be saving the records, but it did not return status of finished, failed, or success. It always returned status of running and one could see the airflow log getting updated.