OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2024-01-10T12:38:59Zhttps://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure/-/issues/32Update deployments to support PodDisruptionBudget & TopologySpreadConstraints...2024-01-10T12:38:59ZRitesh KoulUpdate deployments to support PodDisruptionBudget & TopologySpreadConstraints for airflowNeed to update airflow-8.5.2.tgz file to support **PodDisruptionBudget** & **TopologySpreadConstraints** features.
More details regarding implementation can be found under comments section given for an existing MR
**PodDisruptionBudget*...Need to update airflow-8.5.2.tgz file to support **PodDisruptionBudget** & **TopologySpreadConstraints** features.
More details regarding implementation can be found under comments section given for an existing MR
**PodDisruptionBudget** - https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure/-/merge_requests/749#note_274646
**TopologySpreadConstraints** - https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure/-/merge_requests/749#note_275017https://community.opengroup.org/osdu/platform/system/reference/unit-service/-/issues/57Removal of CSPs Modules and Main Class Reassignment to the Core2024-01-10T10:00:08ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRemoval of CSPs Modules and Main Class Reassignment to the Core# ADR: Remove provider modules from the Unit service.
Simplify the Development and maintenance of the Unit service by removing CSP modules.
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## ...# ADR: Remove provider modules from the Unit service.
Simplify the Development and maintenance of the Unit service by removing CSP modules.
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The CRS-Conversion service operates independently of cloud-specific technologies, conducting all calculations at runtime and storing necessary data within bundled service files. However, within the OSDU Community, four distinct artifacts are generated, each designated per CSP, tested, maintained, and patched for vulnerabilities separately.
## Decision
- Delete provider modules.
- Move the main class to the Unit Core.
- Merge and move Spring Security Configurations to the Unit Core. These configurations are used for service request handling and are independent of cloud technologies. Despite minimal differences, these configurations are dispersed across CSPs, leading to inconsistency in handling and increasing the risk of service misconfiguration.
- Merge and move properties files to the Unit Core.
- Determine the necessity of incorporating CSP libraries as pluggable utilities. These libraries could serve as background utilities for tasks such as log formatting and trace capture. If utilized within the CRS Catalog, establish a method to independently integrate them. This approach could subsequently be adopted for the Community Implementation.
## Rationale
The existing setup of the Unit multiplies the effort needed for maintenance and release processes without visible benefits. This service contains minimal cloud-specific code, primarily limited to occasional utilities from libraries. By excluding CSP modules, the OSDU Community can offer sustainable, thoroughly tested artifacts for the CRS Catalog, significantly reducing the necessary effort.
## Consequences
* Deletion of provider modules.
* Minor CI/CD refactoring to transition to a single artifact (JAR file) from four different artifacts.
* (Optional, pending agreement) Implement a solution for abstracting utility libraries used by CSPs, which could be beneficial in the future.
## Tradeoff Analysis
Beyond defining an abstraction mechanism for CSP utility libraries, the proposal aims to decrease the effort needed for support, releases, and vulnerability management. But if abstraction for libraries is needed it definitely should not introduce more complexity, as it would contradict the main goal of this ADR.
## Alternatives and implications
- Introducing the Core Plus module during the Community Implementation phase. Similar to existing CSP modules, it would be a shallow module. However, introducing custom utilities might pose complexities. On the other hand, it won't be hard to create the same shallow modules elsewhere, but Community OSDU is moving towards maintaining only a single version of the platform.
- Alternatively, we could include the main class, properties, and security configs in the Core, making those components optional without disrupting existing CSP providers.Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service/-/issues/91Removal of CSPs Modules and Main Class Reassignment to the Core2024-03-25T12:11:26ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRemoval of CSPs Modules and Main Class Reassignment to the Core# ADR: Remove provider modules from the CRS Conversion.
Simplify the Development and maintenance of the CRS Conversion service by removing CSP modules.
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] ...# ADR: Remove provider modules from the CRS Conversion.
Simplify the Development and maintenance of the CRS Conversion service by removing CSP modules.
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The CRS-Conversion service operates independently of cloud-specific technologies, conducting all calculations at runtime and storing necessary data within bundled service files. However, within the OSDU Community, four distinct artifacts are generated, each designated per CSP, tested, maintained, and patched for vulnerabilities separately.
## Decision
- Delete provider modules.
- Move the main class to the CRS Conversion Core.
- Merge and move Spring Security Configurations to the CRS Conversion Core. These configurations are used for service request handling and are independent of cloud technologies. Despite minimal differences, these configurations are dispersed across CSPs, leading to inconsistency in handling and increasing the risk of service misconfiguration.
- Merge and move properties files to the CRS Conversion Core.
- Determine the necessity of incorporating CSP libraries as pluggable utilities. These libraries could serve as background utilities for tasks such as log formatting and trace capture. If utilized within the CRS Catalog, establish a method to independently integrate them. This approach could subsequently be adopted for the Community Implementation.
## Rationale
The existing setup of the CRS Conversion multiplies the effort needed for maintenance and release processes without visible benefits. This service contains minimal cloud-specific code, primarily limited to occasional utilities from libraries. By excluding CSP modules, the OSDU Community can offer sustainable, thoroughly tested artifacts for the CRS Catalog, significantly reducing the necessary effort.
## Consequences
* Deletion of provider modules.
* Minor CI/CD refactoring to transition to a single artifact (JAR file) from four different artifacts.
* (Optional, pending agreement) Implement a solution for abstracting utility libraries used by CSPs, which could be beneficial in the future.
## Tradeoff Analysis
Beyond defining an abstraction mechanism for CSP utility libraries, the proposal aims to decrease the effort needed for support, releases, and vulnerability management. But if abstraction for libraries is needed it definitely should not introduce more complexity, as it would contradict the main goal of this ADR.
## Alternatives and implications
- Introducing the Core Plus module during the Community Implementation phase. Similar to existing CSP modules, it would be a shallow module. However, introducing custom utilities might pose complexities. On the other hand, it won't be hard to create the same shallow modules elsewhere, but Community OSDU is moving towards maintaining only a single version of the platform.
- Alternatively, we could include the main class, properties, and security configs in the Core, making those components optional without disrupting existing CSP providers.Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/653M22 - RAFS-Azure - HTTPS Code : 422 Data validation failed2024-01-19T05:12:04ZEsakkiprem SubramaniyanM22 - RAFS-Azure - HTTPS Code : 422 Data validation failedAPI Details:
Version : V2
Folder : Samples Analysis [Wettability Index]
API : Add Data
Method : Post
URL : https://{{RAFS_DDMS_HOST}}/v2/samplesanalysis/{{wettabilityindex_record_id}}/data/wettabilityindex
Response :
{"code...API Details:
Version : V2
Folder : Samples Analysis [Wettability Index]
API : Add Data
Method : Post
URL : https://{{RAFS_DDMS_HOST}}/v2/samplesanalysis/{{wettabilityindex_record_id}}/data/wettabilityindex
Response :
{"code":422,"reason":"Data validation failed.","errors":{"Invalid type":[{"DataSchema":"{'WettabilityIndexData': {'CapillaryPressureAnalysisID': '{{cp_record_id}}:', 'ForcedImbibedBrineVolume': {'Value': 12, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:cm3:'}, 'ForcedImbibedOilVolume': {'Value': 32, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:cm3:'}, 'Temperature': {'Value': 1.0, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:degF:'}, 'InitialBrineSaturation': {'Value': 0.457, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:m3%2Fm3:'}, 'InitialOilSaturation': {'Value': 0.98, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:m3%2Fm3:'}, 'SpontaneousImbibedBrineVolume': {'Value': 6, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:cm3:'}, 'DisplacedOilVolume': {'Value': 0.7768, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:cm3:'}, 'BrineImbibitionBrineSaturation': {'Value': 7, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:m3%2Fm3:'}, 'BrineDisplacementOilSaturation': {'Value': 0.36, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:m3%2Fm3:'}, 'SpontaneousImbibedOilVolume': {'Value': 0.7768, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:cm3:'}, 'OilImbibitionBrineSaturation': {'Value': 3.98, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:m3%2Fm3:'}, 'DisplacedBrineVolume': {'Value': 0.24234000000000003, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:cm3:'}, 'DisplacementRatio': {'Value': 5, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:cm3%2Fcm3:'}, 'OilImbibitionOilSaturation': {'Value': 8.1, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:m3%2Fm3:'}, 'WettabilityIndex': [{'Value': 4.67, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:m3%2Fm3:', 'WettabilityIndexType': 'opendes:reference-data--WettabilityIndexType:AmottHarvey:'}, {'Value': 3.2, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:m3%2Fm3:', 'WettabilityIndexType': 'opendes:reference-data--WettabilityIndexType:AmottHarvey:'}], 'ConfiningPressure': {'Value': 0.453, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:psi:'}, 'DesaturationMethod': 'opendes:reference-data--DesaturationMethod:CentrifugeOilWater:', 'DisplacingFluid': 'opendes:reference-data--DisplacingFluidType:Decane:', 'DisplacedFluid': 'opendes:reference-data--DisplacedFluidType:Carnation:', 'FluidSystem': 'opendes:reference-data--FluidSystemType:GasOil:', 'InitialWaterSaturation': {'Value': 32, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:ppk:'}, 'InitialCapillaryPressure': {'Value': 65, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:bar:'}, 'ForcedWaterImbibition': {'Value': 12, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:m3%2Fm3:'}, 'ForcedOilImbibition': {'Value': 5.9, 'UnitOfMeasure': 'opendes:reference-data--UnitOfMeasure:m3%2Fm3:'}}}"}]}}https://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/323Testing - Demonstrate potential OAuth Flow in JS WebApp for secure GCZ Service2024-01-10T16:18:38ZLevi RemingtonTesting - Demonstrate potential OAuth Flow in JS WebApp for secure GCZ ServiceTo support OSDU PreShip testing, we can develop a version of our sample JS WebApp which prompts user for login and authenticates with OSDU to enable access.
Acceptance Criteria:
- New JS WebAPP that authenticates with AWS preship automa...To support OSDU PreShip testing, we can develop a version of our sample JS WebApp which prompts user for login and authenticates with OSDU to enable access.
Acceptance Criteria:
- New JS WebAPP that authenticates with AWS preship automaticallyhttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/79Removal of CSPs Modules and Main Class Reassignment to the Core2024-03-25T12:11:21ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRemoval of CSPs Modules and Main Class Reassignment to the Core# ADR: Remove provider modules from the CRS Catalog.
Simplify the Development and maintenance of the CRS Catalog service by removing CSP modules.
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retire...# ADR: Remove provider modules from the CRS Catalog.
Simplify the Development and maintenance of the CRS Catalog service by removing CSP modules.
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The CRS-Catalog service operates independently of cloud-specific technologies, conducting all calculations at runtime and storing necessary data within bundled service files. However, within the OSDU Community, four distinct artifacts are generated, each designated per CSP, tested, maintained, and patched for vulnerabilities separately.
## Decision
- Delete provider modules.
- Move the main class to the CRS Catalog Core. ([Azure](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-azure/crs-catalog-aks/src/main/java/org/opengroup/osdu/crs/CrsAksApplication.java), [AWS](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-aws/src/main/java/org/opengroup/osdu/crs/CrsApplicationAWS.java), [GC](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-gc/crs-catalog-gke/src/main/java/org/opengroup/osdu/crs/CRSGKEApplication.java), [IBM](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-ibm/crs-catalog-ocp/src/main/java/org/opengroup/osdu/crs/CrsOcpApplication.java))
- Merge and move Spring Security Configurations to the CRS Catalog Core. These configurations are used for service request handling and are independent of cloud technologies. Despite minimal differences, these configurations are dispersed across CSPs, leading to inconsistency in handling and increasing the risk of service misconfiguration. ([Azure](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-azure/crs-catalog-aks/src/main/java/org/opengroup/osdu/crs/security/SecurityConfig.java),[AWS](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-aws/src/main/java/org/opengroup/osdu/crs/security/AuthSecurityConfig.java),[GC](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-gc/crs-catalog-gke/src/main/java/org/opengroup/osdu/crs/security/AuthSecurityConfig.java),[IBM](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-ibm/crs-catalog-ocp/src/main/java/org/opengroup/osdu/crs/security/SecurityConfig.java))
- Merge and move properties files to the CRS Catalog Core.
- Determine the necessity of incorporating CSP libraries as pluggable utilities. These libraries could serve as background utilities for tasks such as log formatting and trace capture. If utilized within the CRS Catalog, establish a method to independently integrate them. This approach could subsequently be adopted for the Community Implementation.
## Rationale
The existing setup of the CRS Catalog multiplies the effort needed for maintenance and release processes without visible benefits. This service contains minimal cloud-specific code, primarily limited to occasional utilities from libraries. By excluding CSP modules, the OSDU Community can offer sustainable, thoroughly tested artifacts for the CRS Catalog, significantly reducing the necessary effort.
## Consequences
* Deletion of provider modules.
* Minor CI/CD refactoring to transition to a single artifact (JAR file) from four different artifacts.
* (Optional, pending agreement) Implement a solution for abstracting utility libraries used by CSPs, which could be beneficial in the future.
## Tradeoff Analysis
Beyond defining an abstraction mechanism for CSP utility libraries, the proposal aims to decrease the effort needed for support, releases, and vulnerability management. But if abstraction for libraries is needed it definitely should not introduce more complexity, as it would contradict the main goal of this ADR.
## Alternatives and implications
- Introducing the Core Plus module during the Community Implementation phase. Similar to existing CSP modules, it would be a shallow module. However, introducing custom utilities might pose complexities. On the other hand, it won't be hard to create the same shallow modules elsewhere, but Community OSDU is moving towards maintaining only a single version of the platform.
- Alternatively, we could include the main class, properties, and security configs in the Core, making those components optional without disrupting existing CSP providers.Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/search-service/-/issues/154Align the Search Service Code Base with OSDU Platform Development Principles.2024-01-28T16:10:10ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comAlign the Search Service Code Base with OSDU Platform Development Principles.# ADR: Move Code Duplicates from CSP Modules to the Core Module.
Enhance Search service maintenance, align with the future ElasticSearch 8 migration, and minimize the effort needed for introducing Community implementation by reducing co...# ADR: Move Code Duplicates from CSP Modules to the Core Module.
Enhance Search service maintenance, align with the future ElasticSearch 8 migration, and minimize the effort needed for introducing Community implementation by reducing code duplication in CSPs modules.
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The Search service contains duplicated code for constructing Elasticsearch queries within CSP modules, in classes such as QueryBase.java and QueryServiceImpl.java. These redundancies add complexity to code maintenance without offering visible benefits. Query builders do not have CSP-specific code, additionally, differences have emerged in these classes over time:
https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/provider/search-azure/src/main/java/org/opengroup/osdu/search/provider/azure/provider/impl/QueryBase.java
https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/provider/search-aws/src/main/java/org/opengroup/osdu/search/provider/aws/provider/impl/QueryBase.java
https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/provider/search-gc/src/main/java/org/opengroup/osdu/search/provider/gcp/provider/impl/QueryBase.java
https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/provider/search-ibm/src/main/java/org/opengroup/osdu/search/provider/ibm/provider/impl/QueryBase.java?ref_type=heads
## Decision
Identify delta in search service across different providers. Prioritize the most advanced version if significant differences exist and move it to the Core module. For instance, we previously migrated the optimized geo query builders from the Azure provider to the core: https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/556 Following the same principle, we can eliminate other existing duplicates.
## Rationale
Aside from the current increased cost and complexity of maintenance, we have at least two major tasks pending in the Search service:
- The migration to ElasticSearch 8 will require migrating all Elasticsearch query builders. Currently, the required effort will increase proportionally with the number of providers. https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/111
- For the Community Implementation of the Search service, selecting a version of Query Builders will be required. It could be a copy of the GC provider, which diverges from OSDU Development principles (like all providers currently), or a robust, reusable solution. https://gitlab.opengroup.org/osdu/pmc/community-implementation/-/issues/9
## Consequences
* Removal of code duplicates in provider modules.
* Introduction of a consolidated ElasticSearch query builder in the core module.
* Potential impact on features currently in development due to substantial codebase changes.
## Tradeoff Analysis
While this won't break API behavior, it could be seen as disruptive in the development process due to significant codebase changes. Tweaks and improvements made by CSPs to their modules might be overlooked during refactoring if not captured through integration testing.
## Alternatives and implications
An alternative to the current ADR involves relocating code duplicates to the Community Implementation module rather than the Core, designated for use in the Community implementation of the Search service. However, this would require developers to support five modules if new features are introduced or if migration to ElasticSearch 8 begins.M23 - Release 0.26Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/search-service/-/issues/153LQCIndicator.1.0.0 Custom Reference Schema OSDU Search API Issue results in W...2024-01-09T12:13:57ZVedantha Sampekai SampekaiLQCIndicator.1.0.0 Custom Reference Schema OSDU Search API Issue results in WellLogTypeID field not appearing in search result where data is available at storageHello OSDU Forum Team,
I from Shell LQC OSDU Migration Project reaching out to you regarding an OSDU Search Issue facing with an LQC Custom Reference Schema/Data type. Jake from DP team (Jake.J.Pearce@shell.com) is working on this issue...Hello OSDU Forum Team,
I from Shell LQC OSDU Migration Project reaching out to you regarding an OSDU Search Issue facing with an LQC Custom Reference Schema/Data type. Jake from DP team (Jake.J.Pearce@shell.com) is working on this issue, he suggested us to raise this as an issue with OSDU Forum to get some assistance on this. I have explained the whole issue in step-by-step as below, hope it helps you in understanding it.
In case if need any more information this issue, I am happy to explain the issue in detail over a call. Please share me the required associate's names and your convenient time to setup a call. My Email-id: vedantha.gowda@shell.com
PFA LQCIndicator Schema JSON file for your reference.
[LQCIndicator_Custom_Reference_Schema.json](/uploads/26ecd1320e94666823cfb60a70ebb55b/LQCIndicator_Custom_Reference_Schema.json)
**Quick Summary of the issue**
Field "WellLogTypeID" displaying "null" for LQCIndicator:1.0.0 custom reference schema when results are returned for the kind via the Search API. The custom schema is referencing OSDU public data.
**Please find the below details regarding the issue,**
- LQCIndicator.1.0.0 is an LQC Custom Reference Schema/Data Type designed and approved from Shell Data Architect team.
- DP Team has ingested/registered the schema in all OSDU Environments – US Instance.
- LQC Team prepared the reference data as per the business requirement and handed over it to RDM Team in an Excel Sheet to ingest into LQCIndicator Schema.
- RDM Team has built an Informatica Pipeline, prepared an manifest payload and ingested the reference data into LQCIndicator Schema in OSDU Acceptance Environment US Instance.
- RDM team were using OSDU Manifest Ingestion API Service for ingesting the LQCIndicator Reference data into OSDU through Informatica pipeline.
- LQCIndicator has total 42 records and all of them successfully ingested into OSDU Acc Env, but for only 8 records the WellLogTypeID field is appearing as NULL, even though expected data is available in storage.
- Out of 42 LQCIndicator records 8 records are referencing to LOGTYPE:Interpreted and rest 34 records referencing to ConveyanceMethod:LoggingWhileDrilling and ConveyanceMethod:ElectricWirelineConveyed. All of the records referencing LogType:Interpreted are displaying "null" in the WellLogTypeID field. The other referenced fields (for ConveyanceMethod:LoggingWhileDrilling and ConveyanceMethod:ElectricWirelineConveyed) are displaying values as they should.
- When we queried the LQCIndicator schema kind through OSDU Search API service the WellLogTypeID field is showing NULL
- When we queried the LQCIndicator schema kind through OSDU Storage API Service the WellLogTypeID field is showing the correct data value - ("WellLogTypeID": "osdu:reference-data--LogType:Interpreted:").
- The 8 records (id) where we are facing WellLogTypeID field NULL issue are listed at the bottom.
- The WellLogTypeID & ConveyanceMethodID both are similar fields as per schema design. Both are referencing to OSDU Forum Schema’s. ConveyanceMethod referencing to schema – “osdu:wks:reference-data--ConveyanceMethod:1.0.0” and WellLogTypeID referencing to schema “osdu:wks:reference-data--LogType:1.0.0”.
- ConveyanceMethodID attribute works off the same referencing mechanism as WellLogTypeID, however this field is displaying values properly when obtained through the Search API - "ConveyanceMethodID": "osdu:reference-data--ConveyanceMethod:LoggingWhileDrilling:", as per the business requirement (The rest 34 LQCIndicator records).
- A new OSDU environment was created at Shell and we have successfully re-created the LQCIndicator Schema WellLogTypeID field NULL Issue in the new OSDU Environment.
- As suggested by DP team we are raising this issue to OSDU Forum and requesting you to help us on resolving this issue.
- I have attached the LQC Custom Reference Schema for your reference.
- I am happy to supply any further information as required
**Screenshot's of WellLogTypeID field NULL issue in LQCIndicator.1.0.0 Schema/Data type**
Queried through - OSDU Search API Service:
![image](/uploads/7a1863246448f997a3ba12dd8a318bdd/image.png)
Queried through - OSDU Storage API Service:
![image](/uploads/bfc4e4ab14fe38bef9962ec6be9262bc/image.png)
**Below is Search Query to find it in AWS@shell OSDU Acceptance Env - US Instance**
{
"kind": "shell:wks:reference-data--LQCIndicator:1.0.0",
"returnedFields": ["id", "data.WellLogTypeID"]
}
**Below are the 8 - LQCIndicator id’s out of 42 the WellLogTypeID data NULL issue** (for the rest 34 LQCIndicator – id’s the WellLogTypeID should be NULL, where ConveyanceMethodID attribute will be populated)
osdu:reference-data--LQCIndicator:24
osdu:reference-data--LQCIndicator:25
osdu:reference-data--LQCIndicator:26
osdu:reference-data--LQCIndicator:27
osdu:reference-data--LQCIndicator:28
osdu:reference-data--LQCIndicator:29
osdu:reference-data--LQCIndicator:30
osdu:reference-data--LQCIndicator:31
Looking forward for your guidance and help in resolving this issue. Thank youhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/652M22/AWS/Preship - Wellbore DDMS - "Create markertset" step fails2024-01-10T03:39:14ZDebasis ChatterjeeM22/AWS/Preship - Wellbore DDMS - "Create markertset" step failsStep 7.1 from the collection which is provided by CSP.
https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/AWS-M22/DDMS%20Wellbore/AWS_OSDU_R3M22_WellboreDDMS_Collection.postman_collection.json
POST {{osduonaws...Step 7.1 from the collection which is provided by CSP.
https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/AWS-M22/DDMS%20Wellbore/AWS_OSDU_R3M22_WellboreDDMS_Collection.postman_collection.json
POST {{osduonaws_base_url}}/api/os-wellbore-ddms/ddms/v3/wellboremarkersets
fails with legal tag problem.
{
"origin": "osdu-data-ecosystem-storage",
"errors": [
{
"code": 400,
"reason": "Invalid legal tags",
"message": "Invalid legal tags found on record"
}
]
}
Checked from curl code and found that the proper tag is being used. I also tried by forcing the legal tag name instead of variable.
Even then it fails with the same error.
cc @ydzeng , @cailletg and @Diddakuntlahttps://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/139Too many results returned after bagofwords feature2024-01-19T19:47:34ZGuillaume CailletToo many results returned after bagofwords featureHi,
When enabling the [BagOfWords feature](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/113), some search query with a "query" filter return too many results.
I've reproduced the issue on several AWS env...Hi,
When enabling the [BagOfWords feature](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/113), some search query with a "query" filter return too many results.
I've reproduced the issue on several AWS environment, and I don't have this issue if the indexer is deployed with the Feature flag `featureFlag.bagOfWords.enabled` set to False.
I have attached the 3 records and schema I used (these are from the `os-search` integration tests in `testing/integration-tests/search-test-core/src/main/resources/testData/records_1.json`)
[records.json](/uploads/196fce2d3f739b3c4349bd4e5075aeed/records.json)
[schema.json](/uploads/990d8ac4242d6a09921e16236f6a72e5/schema.json)
( I didn't delete these 3 records from the `main.osdu-gl.osdu.aws` environment, so if you have access to it, you should be able to reproduce these queries )
Once the records are indexed :
Issue a `search` query with the following payload:
```
{
"kind": "opendes:search1704732571020:test-data--Integration:1.0.1",
"query": "OFFICE9"
}
```
I have all 3 records returned, instead of 0 (there are no "OFFICE9" text in the 3 records)
Same if I use a "valid" query matching at least one record, for example
```
{
"kind": "opendes:search1704732571020:test-data--Integration:1.0.1",
"query": "OFFICE4"
}
```
Also returns 3 records instead of one.
This issue seems to occurs only when using digit suffix. If I use a letter, it works properly, for example
```
{
"kind": "opendes:search1704732571020:test-data--Integration:1.0.1",
"query": "OFFICEZ"
}
```
Properly returns 0 results.
I have managed to reproduce the issue directly on the elasticsearch server by using their REST API, so the issue is not with the Search service I think :
POST https://localhost:9200/opendes-search1704732571020-test-data--integration-1.0.1/_search (I'm using k8s port-forwarding to dircetly connect to the ES server)
with the following payload
```{
"from": 0,
"size": 10,
"timeout": "1m",
"query": {
"bool": {
"must": [
{
"bool": {
"must": [
{
"query_string": {
"query": "OFFICE9"
}
}
],
"adjust_pure_negative": true,
"boost": 1.0
}
}
]
}
}
}
```
Returns 3 results when BagOfWords is enabled, only 1 if not.M22 - Release 0.25Mark ChanceStanisław BienieckiMark Chancehttps://community.opengroup.org/osdu/platform/consumption/geospatial/-/issues/322Transformer - SPIKE: Evaluate potential support for Wildcard Element in Confi...2024-02-05T16:02:50ZNoel OkanyaTransformer - SPIKE: Evaluate potential support for Wildcard Element in Configuired OSDU KindsAs a GCZ Developer, I would like to evaluate potential support for Wildcard Element in configured OSDU Kinds, so that we can understand the level of effort it would take for GCZ to support Wildcard Kind ingestion.
This request is from P...As a GCZ Developer, I would like to evaluate potential support for Wildcard Element in configured OSDU Kinds, so that we can understand the level of effort it would take for GCZ to support Wildcard Kind ingestion.
This request is from Petronas SLB Deployment. Customer is looking for automatic cache update as updated data is loaded in OSDU without manually updating application.yml
Customer needs latest version of data.
Presently once OSDU is updated with new version data, transformer needs below workflow:
- Update application.yml with new kind names.
- Restart transformer so that transformer can use updated application.yml
(if it is remote profile which is mostly used for customer deployments, transformer only connects to existing cache and data is not reloaded)
- Data will be reloaded as per cron job or use Ambassdor updateCache for each updated kindAnkita SrivastavaAnkita Srivastavahttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/reservoir/open-etp-server/-/issues/110GetDataspaceInfo is wrong when called after DeleteDataspaces2024-01-30T15:42:42ZVirginie MarcoutGetDataspaceInfo is wrong when called after DeleteDataspacesThe openETPServer does not update the dataspace cache after deleting a just created dataspace, so after we create and delete we are able to get cached dataspace information.The openETPServer does not update the dataspace cache after deleting a just created dataspace, so after we create and delete we are able to get cached dataspace information.M23 - Release 0.26Virginie MarcoutVirginie Marcouthttps://community.opengroup.org/osdu/platform/system/dataset/-/issues/78Static Dataset API doc is outdated2024-01-08T12:42:45ZChad LeongStatic Dataset API doc is outdatedFix static API doc as part of https://community.opengroup.org/groups/osdu/platform/-/epics/19Fix static API doc as part of https://community.opengroup.org/groups/osdu/platform/-/epics/19https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/328Central Resources2024-02-01T15:04:22ZKallesh DCentral ResourcesWhile creating central resources i am getting below error.Please check and let me know
│
│ with module.ad_application.azuread_application.main[0],
│ on ../../../modules/providers/azure/ad-application/main.tf line 22, in resource "...While creating central resources i am getting below error.Please check and let me know
│
│ with module.ad_application.azuread_application.main[0],
│ on ../../../modules/providers/azure/ad-application/main.tf line 22, in resource "azuread_application" "main":
│ 22: resource "azuread_application" "main" {
│
│ ApplicationsClient.BaseClient.Patch(): unexpected status 403 with OData error: Authorization_RequestDenied: Insufficient privileges to
│ complete the operation.https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/651RI: Entitlement swagger UI is down2024-01-08T11:27:49ZChad LeongRI: Entitlement swagger UI is downEntitlement swagger UI is down : https://osdu.bm22.gcp.gnrg-osdu.projects.epam.com/api/entitlements/v2/swagger-ui/index.html
Expected behavior:
https://osdu-ship.msft-osdu-test.org/api/entitlements/v2/swagger-ui/index.htmlEntitlement swagger UI is down : https://osdu.bm22.gcp.gnrg-osdu.projects.epam.com/api/entitlements/v2/swagger-ui/index.html
Expected behavior:
https://osdu-ship.msft-osdu-test.org/api/entitlements/v2/swagger-ui/index.htmlDzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/650String array becomes String after index2024-01-08T08:56:09ZChad LeongString array becomes String after indexThe String array becomes String after it is indexed. Bug should be introduced by [MR 649](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/649)
See issue created under Indexer https://community.openg...The String array becomes String after it is indexed. Bug should be introduced by [MR 649](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/649)
See issue created under Indexer https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/137https://community.opengroup.org/osdu/platform/system/search-service/-/issues/152ADR: Ability to get all the records of a given Persisted Collection from search2024-01-11T09:20:16ZJuilee PaluskarADR: Ability to get all the records of a given Persisted Collection from search## Status
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [ ] Approved
* [ ] Retired
## Context & Scope
A persisted collection can aggregate objects of different nature including master data, work-product-component, reference data....## Status
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [ ] Approved
* [ ] Retired
## Context & Scope
A persisted collection can aggregate objects of different nature including master data, work-product-component, reference data. It could contain collection of records of heterogenous kind. At a given point, MemberIDs field of PersistedCollection maintains list of objects which are part of the collection.
More can be read from this [schema](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/Generated/work-product-component/PersistedCollection.1.2.0.json) .
Problem :
Today, there is no way to get all the records which belongs to a particular Persisted Collection. Today, user has to perform atleast 2 search queries to get the records of a Persisted Collection.
1st Query - To get the Persisted Collection record and retrieve record IDs are from MemeberID field.
2nd Query - To get the actual record from retrieved record Ids in the 1st query.
For the 2nd query, to get multiple records in 1 search query, user has to form a query with OR operator. E.g.
{
"Query" : recordId-1 **OR** recordId-2 **OR** recordId-3 .. recordId-1000.
}
Here ElasticSearch has limitation of max usage of **OR** conditions in 1 query.
So if a PersistedCollection contains more than 1000 records , user has to invoke multiple search queries to get all the records.
## Possible Solution
One of the possible solution to address this requirement could be adding Persisted Collection record id in the record's data. So whenever records get added to the Persisted Collection , record's data should be updated with the information of Persisted Collection id.
This could be done by listening to record change event for PersistedCollection kind .
## Consequences
* This will help users to get the records of Persisted Collection in a single go.
* This will help users to get the records to which he/she has access.
* This will help users to form queries to get desire records from PeristedCollection such as “Give all the records of a persisted collection where data.\<someproperty\> is \<xyz\>“ in one go.
* This will help users to make filters based on different objects in the collection.https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/125Upgrade SHA1 to SHA2 or SHA2562024-02-13T14:11:22ZShane HutchinsUpgrade SHA1 to SHA2 or SHA256Policy Service currently uses SHA1 for logging purposes with changes to policies. This SHA1 is returned in json response when changes are made as well.
While it's not used for security it would be nice to upgrade from SHA1.
Created bas...Policy Service currently uses SHA1 for logging purposes with changes to policies. This SHA1 is returned in json response when changes are made as well.
While it's not used for security it would be nice to upgrade from SHA1.
Created based upon https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/124Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/138Datetime formatting/parsing issues result in field not appearing in search index2024-03-04T12:23:12ZMark ChanceDatetime formatting/parsing issues result in field not appearing in search index**Subject:** Certain "date" type attributes unavailable via SEARCH API but available by STORAGE API
QA team just highlighted that some “date” related fields have gone missing again from the SEARCH API services. I have posted the sna...**Subject:** Certain "date" type attributes unavailable via SEARCH API but available by STORAGE API
QA team just highlighted that some “date” related fields have gone missing again from the SEARCH API services. I have posted the snapshot below. Please note that no schema updates/changes has happened. QA (as end users) are ingesting and retrieving data (CRUD) to and from the schema.\
\
{
"kind": "tenant1:wks:work-product-component--Sheet:1.0.0",
"query": "\\"tenant1:work-product-component--Sheet:d92b4ff85fd040dba9009209e85a3c31\\""
}\
\
Through SEARCH:
![Search.png](/uploads/8c66dfb8d694298852a39f7d7eb50918/Search.png)Through STORAGE:
![Storage.png](/uploads/7af8e15227941a43f1a3a8b6440931aa/Storage.png)This is fixed by https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/694M23 - Release 0.26Mark ChanceMark Chancehttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/327Upgrade Terraform version to the latest stable version2024-02-01T20:12:39ZAnastasiia DolotUpgrade Terraform version to the latest stable version[Terraform 1.3.5](https://releases.hashicorp.com/terraform/1.3.5/) (2 years ago) is the version we use to create the infrastructure.
To upgrade terraform to the [latest stable version 1.6.6](https://releases.hashicorp.com/terraform/1.6.6...[Terraform 1.3.5](https://releases.hashicorp.com/terraform/1.3.5/) (2 years ago) is the version we use to create the infrastructure.
To upgrade terraform to the [latest stable version 1.6.6](https://releases.hashicorp.com/terraform/1.6.6/)
https://github.com/hashicorp/terraform/releases/tag/v1.6.0
The azurerm provider was updated from version 3.39.1 (1 year ago) to version 3.79.0 in [MR_913](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/913/diffs?commit_id=8e976a4f09f8693a0b19819c8264eb709cd83395#e23442233c363f9a6c2d80f0962c6a14c7349f1a)
Terraform version upgrade is needed for compliancy.Arturo Hernandez [EPAM]Arturo Hernandez [EPAM]