OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2024-03-25T12:11:21Zhttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/79Removal of CSPs Modules and Main Class Reassignment to the Core2024-03-25T12:11:21ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRemoval of CSPs Modules and Main Class Reassignment to the Core# ADR: Remove provider modules from the CRS Catalog.
Simplify the Development and maintenance of the CRS Catalog service by removing CSP modules.
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retire...# ADR: Remove provider modules from the CRS Catalog.
Simplify the Development and maintenance of the CRS Catalog service by removing CSP modules.
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The CRS-Catalog service operates independently of cloud-specific technologies, conducting all calculations at runtime and storing necessary data within bundled service files. However, within the OSDU Community, four distinct artifacts are generated, each designated per CSP, tested, maintained, and patched for vulnerabilities separately.
## Decision
- Delete provider modules.
- Move the main class to the CRS Catalog Core. ([Azure](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-azure/crs-catalog-aks/src/main/java/org/opengroup/osdu/crs/CrsAksApplication.java), [AWS](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-aws/src/main/java/org/opengroup/osdu/crs/CrsApplicationAWS.java), [GC](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-gc/crs-catalog-gke/src/main/java/org/opengroup/osdu/crs/CRSGKEApplication.java), [IBM](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-ibm/crs-catalog-ocp/src/main/java/org/opengroup/osdu/crs/CrsOcpApplication.java))
- Merge and move Spring Security Configurations to the CRS Catalog Core. These configurations are used for service request handling and are independent of cloud technologies. Despite minimal differences, these configurations are dispersed across CSPs, leading to inconsistency in handling and increasing the risk of service misconfiguration. ([Azure](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-azure/crs-catalog-aks/src/main/java/org/opengroup/osdu/crs/security/SecurityConfig.java),[AWS](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-aws/src/main/java/org/opengroup/osdu/crs/security/AuthSecurityConfig.java),[GC](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-gc/crs-catalog-gke/src/main/java/org/opengroup/osdu/crs/security/AuthSecurityConfig.java),[IBM](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/provider/crs-catalog-ibm/crs-catalog-ocp/src/main/java/org/opengroup/osdu/crs/security/SecurityConfig.java))
- Merge and move properties files to the CRS Catalog Core.
- Determine the necessity of incorporating CSP libraries as pluggable utilities. These libraries could serve as background utilities for tasks such as log formatting and trace capture. If utilized within the CRS Catalog, establish a method to independently integrate them. This approach could subsequently be adopted for the Community Implementation.
## Rationale
The existing setup of the CRS Catalog multiplies the effort needed for maintenance and release processes without visible benefits. This service contains minimal cloud-specific code, primarily limited to occasional utilities from libraries. By excluding CSP modules, the OSDU Community can offer sustainable, thoroughly tested artifacts for the CRS Catalog, significantly reducing the necessary effort.
## Consequences
* Deletion of provider modules.
* Minor CI/CD refactoring to transition to a single artifact (JAR file) from four different artifacts.
* (Optional, pending agreement) Implement a solution for abstracting utility libraries used by CSPs, which could be beneficial in the future.
## Tradeoff Analysis
Beyond defining an abstraction mechanism for CSP utility libraries, the proposal aims to decrease the effort needed for support, releases, and vulnerability management. But if abstraction for libraries is needed it definitely should not introduce more complexity, as it would contradict the main goal of this ADR.
## Alternatives and implications
- Introducing the Core Plus module during the Community Implementation phase. Similar to existing CSP modules, it would be a shallow module. However, introducing custom utilities might pose complexities. On the other hand, it won't be hard to create the same shallow modules elsewhere, but Community OSDU is moving towards maintaining only a single version of the platform.
- Alternatively, we could include the main class, properties, and security configs in the Core, making those components optional without disrupting existing CSP providers.Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/search-service/-/issues/154Align the Search Service Code Base with OSDU Platform Development Principles.2024-01-28T16:10:10ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comAlign the Search Service Code Base with OSDU Platform Development Principles.# ADR: Move Code Duplicates from CSP Modules to the Core Module.
Enhance Search service maintenance, align with the future ElasticSearch 8 migration, and minimize the effort needed for introducing Community implementation by reducing co...# ADR: Move Code Duplicates from CSP Modules to the Core Module.
Enhance Search service maintenance, align with the future ElasticSearch 8 migration, and minimize the effort needed for introducing Community implementation by reducing code duplication in CSPs modules.
## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The Search service contains duplicated code for constructing Elasticsearch queries within CSP modules, in classes such as QueryBase.java and QueryServiceImpl.java. These redundancies add complexity to code maintenance without offering visible benefits. Query builders do not have CSP-specific code, additionally, differences have emerged in these classes over time:
https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/provider/search-azure/src/main/java/org/opengroup/osdu/search/provider/azure/provider/impl/QueryBase.java
https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/provider/search-aws/src/main/java/org/opengroup/osdu/search/provider/aws/provider/impl/QueryBase.java
https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/provider/search-gc/src/main/java/org/opengroup/osdu/search/provider/gcp/provider/impl/QueryBase.java
https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/provider/search-ibm/src/main/java/org/opengroup/osdu/search/provider/ibm/provider/impl/QueryBase.java?ref_type=heads
## Decision
Identify delta in search service across different providers. Prioritize the most advanced version if significant differences exist and move it to the Core module. For instance, we previously migrated the optimized geo query builders from the Azure provider to the core: https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/556 Following the same principle, we can eliminate other existing duplicates.
## Rationale
Aside from the current increased cost and complexity of maintenance, we have at least two major tasks pending in the Search service:
- The migration to ElasticSearch 8 will require migrating all Elasticsearch query builders. Currently, the required effort will increase proportionally with the number of providers. https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/111
- For the Community Implementation of the Search service, selecting a version of Query Builders will be required. It could be a copy of the GC provider, which diverges from OSDU Development principles (like all providers currently), or a robust, reusable solution. https://gitlab.opengroup.org/osdu/pmc/community-implementation/-/issues/9
## Consequences
* Removal of code duplicates in provider modules.
* Introduction of a consolidated ElasticSearch query builder in the core module.
* Potential impact on features currently in development due to substantial codebase changes.
## Tradeoff Analysis
While this won't break API behavior, it could be seen as disruptive in the development process due to significant codebase changes. Tweaks and improvements made by CSPs to their modules might be overlooked during refactoring if not captured through integration testing.
## Alternatives and implications
An alternative to the current ADR involves relocating code duplicates to the Community Implementation module rather than the Core, designated for use in the Community implementation of the Search service. However, this would require developers to support five modules if new features are introduced or if migration to ElasticSearch 8 begins.M23 - Release 0.26Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/search-service/-/issues/153LQCIndicator.1.0.0 Custom Reference Schema OSDU Search API Issue results in W...2024-01-09T12:13:57ZVedantha Sampekai SampekaiLQCIndicator.1.0.0 Custom Reference Schema OSDU Search API Issue results in WellLogTypeID field not appearing in search result where data is available at storageHello OSDU Forum Team,
I from Shell LQC OSDU Migration Project reaching out to you regarding an OSDU Search Issue facing with an LQC Custom Reference Schema/Data type. Jake from DP team (Jake.J.Pearce@shell.com) is working on this issue...Hello OSDU Forum Team,
I from Shell LQC OSDU Migration Project reaching out to you regarding an OSDU Search Issue facing with an LQC Custom Reference Schema/Data type. Jake from DP team (Jake.J.Pearce@shell.com) is working on this issue, he suggested us to raise this as an issue with OSDU Forum to get some assistance on this. I have explained the whole issue in step-by-step as below, hope it helps you in understanding it.
In case if need any more information this issue, I am happy to explain the issue in detail over a call. Please share me the required associate's names and your convenient time to setup a call. My Email-id: vedantha.gowda@shell.com
PFA LQCIndicator Schema JSON file for your reference.
[LQCIndicator_Custom_Reference_Schema.json](/uploads/26ecd1320e94666823cfb60a70ebb55b/LQCIndicator_Custom_Reference_Schema.json)
**Quick Summary of the issue**
Field "WellLogTypeID" displaying "null" for LQCIndicator:1.0.0 custom reference schema when results are returned for the kind via the Search API. The custom schema is referencing OSDU public data.
**Please find the below details regarding the issue,**
- LQCIndicator.1.0.0 is an LQC Custom Reference Schema/Data Type designed and approved from Shell Data Architect team.
- DP Team has ingested/registered the schema in all OSDU Environments – US Instance.
- LQC Team prepared the reference data as per the business requirement and handed over it to RDM Team in an Excel Sheet to ingest into LQCIndicator Schema.
- RDM Team has built an Informatica Pipeline, prepared an manifest payload and ingested the reference data into LQCIndicator Schema in OSDU Acceptance Environment US Instance.
- RDM team were using OSDU Manifest Ingestion API Service for ingesting the LQCIndicator Reference data into OSDU through Informatica pipeline.
- LQCIndicator has total 42 records and all of them successfully ingested into OSDU Acc Env, but for only 8 records the WellLogTypeID field is appearing as NULL, even though expected data is available in storage.
- Out of 42 LQCIndicator records 8 records are referencing to LOGTYPE:Interpreted and rest 34 records referencing to ConveyanceMethod:LoggingWhileDrilling and ConveyanceMethod:ElectricWirelineConveyed. All of the records referencing LogType:Interpreted are displaying "null" in the WellLogTypeID field. The other referenced fields (for ConveyanceMethod:LoggingWhileDrilling and ConveyanceMethod:ElectricWirelineConveyed) are displaying values as they should.
- When we queried the LQCIndicator schema kind through OSDU Search API service the WellLogTypeID field is showing NULL
- When we queried the LQCIndicator schema kind through OSDU Storage API Service the WellLogTypeID field is showing the correct data value - ("WellLogTypeID": "osdu:reference-data--LogType:Interpreted:").
- The 8 records (id) where we are facing WellLogTypeID field NULL issue are listed at the bottom.
- The WellLogTypeID & ConveyanceMethodID both are similar fields as per schema design. Both are referencing to OSDU Forum Schema’s. ConveyanceMethod referencing to schema – “osdu:wks:reference-data--ConveyanceMethod:1.0.0” and WellLogTypeID referencing to schema “osdu:wks:reference-data--LogType:1.0.0”.
- ConveyanceMethodID attribute works off the same referencing mechanism as WellLogTypeID, however this field is displaying values properly when obtained through the Search API - "ConveyanceMethodID": "osdu:reference-data--ConveyanceMethod:LoggingWhileDrilling:", as per the business requirement (The rest 34 LQCIndicator records).
- A new OSDU environment was created at Shell and we have successfully re-created the LQCIndicator Schema WellLogTypeID field NULL Issue in the new OSDU Environment.
- As suggested by DP team we are raising this issue to OSDU Forum and requesting you to help us on resolving this issue.
- I have attached the LQC Custom Reference Schema for your reference.
- I am happy to supply any further information as required
**Screenshot's of WellLogTypeID field NULL issue in LQCIndicator.1.0.0 Schema/Data type**
Queried through - OSDU Search API Service:
![image](/uploads/7a1863246448f997a3ba12dd8a318bdd/image.png)
Queried through - OSDU Storage API Service:
![image](/uploads/bfc4e4ab14fe38bef9962ec6be9262bc/image.png)
**Below is Search Query to find it in AWS@shell OSDU Acceptance Env - US Instance**
{
"kind": "shell:wks:reference-data--LQCIndicator:1.0.0",
"returnedFields": ["id", "data.WellLogTypeID"]
}
**Below are the 8 - LQCIndicator id’s out of 42 the WellLogTypeID data NULL issue** (for the rest 34 LQCIndicator – id’s the WellLogTypeID should be NULL, where ConveyanceMethodID attribute will be populated)
osdu:reference-data--LQCIndicator:24
osdu:reference-data--LQCIndicator:25
osdu:reference-data--LQCIndicator:26
osdu:reference-data--LQCIndicator:27
osdu:reference-data--LQCIndicator:28
osdu:reference-data--LQCIndicator:29
osdu:reference-data--LQCIndicator:30
osdu:reference-data--LQCIndicator:31
Looking forward for your guidance and help in resolving this issue. Thank youhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/652M22/AWS/Preship - Wellbore DDMS - "Create markertset" step fails2024-01-10T03:39:14ZDebasis ChatterjeeM22/AWS/Preship - Wellbore DDMS - "Create markertset" step failsStep 7.1 from the collection which is provided by CSP.
https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/AWS-M22/DDMS%20Wellbore/AWS_OSDU_R3M22_WellboreDDMS_Collection.postman_collection.json
POST {{osduonaws...Step 7.1 from the collection which is provided by CSP.
https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/AWS-M22/DDMS%20Wellbore/AWS_OSDU_R3M22_WellboreDDMS_Collection.postman_collection.json
POST {{osduonaws_base_url}}/api/os-wellbore-ddms/ddms/v3/wellboremarkersets
fails with legal tag problem.
{
"origin": "osdu-data-ecosystem-storage",
"errors": [
{
"code": 400,
"reason": "Invalid legal tags",
"message": "Invalid legal tags found on record"
}
]
}
Checked from curl code and found that the proper tag is being used. I also tried by forcing the legal tag name instead of variable.
Even then it fails with the same error.
cc @ydzeng , @cailletg and @Diddakuntlahttps://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/139Too many results returned after bagofwords feature2024-01-19T19:47:34ZGuillaume CailletToo many results returned after bagofwords featureHi,
When enabling the [BagOfWords feature](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/113), some search query with a "query" filter return too many results.
I've reproduced the issue on several AWS env...Hi,
When enabling the [BagOfWords feature](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/113), some search query with a "query" filter return too many results.
I've reproduced the issue on several AWS environment, and I don't have this issue if the indexer is deployed with the Feature flag `featureFlag.bagOfWords.enabled` set to False.
I have attached the 3 records and schema I used (these are from the `os-search` integration tests in `testing/integration-tests/search-test-core/src/main/resources/testData/records_1.json`)
[records.json](/uploads/196fce2d3f739b3c4349bd4e5075aeed/records.json)
[schema.json](/uploads/990d8ac4242d6a09921e16236f6a72e5/schema.json)
( I didn't delete these 3 records from the `main.osdu-gl.osdu.aws` environment, so if you have access to it, you should be able to reproduce these queries )
Once the records are indexed :
Issue a `search` query with the following payload:
```
{
"kind": "opendes:search1704732571020:test-data--Integration:1.0.1",
"query": "OFFICE9"
}
```
I have all 3 records returned, instead of 0 (there are no "OFFICE9" text in the 3 records)
Same if I use a "valid" query matching at least one record, for example
```
{
"kind": "opendes:search1704732571020:test-data--Integration:1.0.1",
"query": "OFFICE4"
}
```
Also returns 3 records instead of one.
This issue seems to occurs only when using digit suffix. If I use a letter, it works properly, for example
```
{
"kind": "opendes:search1704732571020:test-data--Integration:1.0.1",
"query": "OFFICEZ"
}
```
Properly returns 0 results.
I have managed to reproduce the issue directly on the elasticsearch server by using their REST API, so the issue is not with the Search service I think :
POST https://localhost:9200/opendes-search1704732571020-test-data--integration-1.0.1/_search (I'm using k8s port-forwarding to dircetly connect to the ES server)
with the following payload
```{
"from": 0,
"size": 10,
"timeout": "1m",
"query": {
"bool": {
"must": [
{
"bool": {
"must": [
{
"query_string": {
"query": "OFFICE9"
}
}
],
"adjust_pure_negative": true,
"boost": 1.0
}
}
]
}
}
}
```
Returns 3 results when BagOfWords is enabled, only 1 if not.M22 - Release 0.25Mark ChanceStanisław BienieckiMark Chancehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/reservoir/open-etp-server/-/issues/110GetDataspaceInfo is wrong when called after DeleteDataspaces2024-01-30T15:42:42ZVirginie MarcoutGetDataspaceInfo is wrong when called after DeleteDataspacesThe openETPServer does not update the dataspace cache after deleting a just created dataspace, so after we create and delete we are able to get cached dataspace information.The openETPServer does not update the dataspace cache after deleting a just created dataspace, so after we create and delete we are able to get cached dataspace information.M23 - Release 0.26Virginie MarcoutVirginie Marcouthttps://community.opengroup.org/osdu/platform/system/dataset/-/issues/78Static Dataset API doc is outdated2024-01-08T12:42:45ZChad LeongStatic Dataset API doc is outdatedFix static API doc as part of https://community.opengroup.org/groups/osdu/platform/-/epics/19Fix static API doc as part of https://community.opengroup.org/groups/osdu/platform/-/epics/19https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/328Central Resources2024-02-01T15:04:22ZKallesh DCentral ResourcesWhile creating central resources i am getting below error.Please check and let me know
│
│ with module.ad_application.azuread_application.main[0],
│ on ../../../modules/providers/azure/ad-application/main.tf line 22, in resource "...While creating central resources i am getting below error.Please check and let me know
│
│ with module.ad_application.azuread_application.main[0],
│ on ../../../modules/providers/azure/ad-application/main.tf line 22, in resource "azuread_application" "main":
│ 22: resource "azuread_application" "main" {
│
│ ApplicationsClient.BaseClient.Patch(): unexpected status 403 with OData error: Authorization_RequestDenied: Insufficient privileges to
│ complete the operation.https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/651RI: Entitlement swagger UI is down2024-01-08T11:27:49ZChad LeongRI: Entitlement swagger UI is downEntitlement swagger UI is down : https://osdu.bm22.gcp.gnrg-osdu.projects.epam.com/api/entitlements/v2/swagger-ui/index.html
Expected behavior:
https://osdu-ship.msft-osdu-test.org/api/entitlements/v2/swagger-ui/index.htmlEntitlement swagger UI is down : https://osdu.bm22.gcp.gnrg-osdu.projects.epam.com/api/entitlements/v2/swagger-ui/index.html
Expected behavior:
https://osdu-ship.msft-osdu-test.org/api/entitlements/v2/swagger-ui/index.htmlDzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/650String array becomes String after index2024-01-08T08:56:09ZChad LeongString array becomes String after indexThe String array becomes String after it is indexed. Bug should be introduced by [MR 649](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/649)
See issue created under Indexer https://community.openg...The String array becomes String after it is indexed. Bug should be introduced by [MR 649](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/649)
See issue created under Indexer https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/137https://community.opengroup.org/osdu/platform/system/search-service/-/issues/152ADR: Ability to get all the records of a given Persisted Collection from search2024-01-11T09:20:16ZJuilee PaluskarADR: Ability to get all the records of a given Persisted Collection from search## Status
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [ ] Approved
* [ ] Retired
## Context & Scope
A persisted collection can aggregate objects of different nature including master data, work-product-component, reference data....## Status
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [ ] Approved
* [ ] Retired
## Context & Scope
A persisted collection can aggregate objects of different nature including master data, work-product-component, reference data. It could contain collection of records of heterogenous kind. At a given point, MemberIDs field of PersistedCollection maintains list of objects which are part of the collection.
More can be read from this [schema](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/Generated/work-product-component/PersistedCollection.1.2.0.json) .
Problem :
Today, there is no way to get all the records which belongs to a particular Persisted Collection. Today, user has to perform atleast 2 search queries to get the records of a Persisted Collection.
1st Query - To get the Persisted Collection record and retrieve record IDs are from MemeberID field.
2nd Query - To get the actual record from retrieved record Ids in the 1st query.
For the 2nd query, to get multiple records in 1 search query, user has to form a query with OR operator. E.g.
{
"Query" : recordId-1 **OR** recordId-2 **OR** recordId-3 .. recordId-1000.
}
Here ElasticSearch has limitation of max usage of **OR** conditions in 1 query.
So if a PersistedCollection contains more than 1000 records , user has to invoke multiple search queries to get all the records.
## Possible Solution
One of the possible solution to address this requirement could be adding Persisted Collection record id in the record's data. So whenever records get added to the Persisted Collection , record's data should be updated with the information of Persisted Collection id.
This could be done by listening to record change event for PersistedCollection kind .
## Consequences
* This will help users to get the records of Persisted Collection in a single go.
* This will help users to get the records to which he/she has access.
* This will help users to form queries to get desire records from PeristedCollection such as “Give all the records of a persisted collection where data.\<someproperty\> is \<xyz\>“ in one go.
* This will help users to make filters based on different objects in the collection.https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/125Upgrade SHA1 to SHA2 or SHA2562024-02-13T14:11:22ZShane HutchinsUpgrade SHA1 to SHA2 or SHA256Policy Service currently uses SHA1 for logging purposes with changes to policies. This SHA1 is returned in json response when changes are made as well.
While it's not used for security it would be nice to upgrade from SHA1.
Created bas...Policy Service currently uses SHA1 for logging purposes with changes to policies. This SHA1 is returned in json response when changes are made as well.
While it's not used for security it would be nice to upgrade from SHA1.
Created based upon https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/124Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/138Datetime formatting/parsing issues result in field not appearing in search index2024-03-04T12:23:12ZMark ChanceDatetime formatting/parsing issues result in field not appearing in search index**Subject:** Certain "date" type attributes unavailable via SEARCH API but available by STORAGE API
QA team just highlighted that some “date” related fields have gone missing again from the SEARCH API services. I have posted the sna...**Subject:** Certain "date" type attributes unavailable via SEARCH API but available by STORAGE API
QA team just highlighted that some “date” related fields have gone missing again from the SEARCH API services. I have posted the snapshot below. Please note that no schema updates/changes has happened. QA (as end users) are ingesting and retrieving data (CRUD) to and from the schema.\
\
{
"kind": "tenant1:wks:work-product-component--Sheet:1.0.0",
"query": "\\"tenant1:work-product-component--Sheet:d92b4ff85fd040dba9009209e85a3c31\\""
}\
\
Through SEARCH:
![Search.png](/uploads/8c66dfb8d694298852a39f7d7eb50918/Search.png)Through STORAGE:
![Storage.png](/uploads/7af8e15227941a43f1a3a8b6440931aa/Storage.png)This is fixed by https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/694M23 - Release 0.26Mark ChanceMark Chancehttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/327Upgrade Terraform version to the latest stable version2024-02-01T20:12:39ZAnastasiia DolotUpgrade Terraform version to the latest stable version[Terraform 1.3.5](https://releases.hashicorp.com/terraform/1.3.5/) (2 years ago) is the version we use to create the infrastructure.
To upgrade terraform to the [latest stable version 1.6.6](https://releases.hashicorp.com/terraform/1.6.6...[Terraform 1.3.5](https://releases.hashicorp.com/terraform/1.3.5/) (2 years ago) is the version we use to create the infrastructure.
To upgrade terraform to the [latest stable version 1.6.6](https://releases.hashicorp.com/terraform/1.6.6/)
https://github.com/hashicorp/terraform/releases/tag/v1.6.0
The azurerm provider was updated from version 3.39.1 (1 year ago) to version 3.79.0 in [MR_913](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/913/diffs?commit_id=8e976a4f09f8693a0b19819c8264eb709cd83395#e23442233c363f9a6c2d80f0962c6a14c7349f1a)
Terraform version upgrade is needed for compliancy.Arturo Hernandez [EPAM]Arturo Hernandez [EPAM]https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/137String array becomes String after index2024-01-24T08:54:10ZZhibin MaiString array becomes String after indexThe String array becomes String after it is indexed. Bug should be introduced by [MR 649](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/649)
To illustrate the problem, I used one example from Augm...The String array becomes String after it is indexed. Bug should be introduced by [MR 649](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/649)
To illustrate the problem, I used one example from Augmenter Configuration that has String array attributes.
- Storage Format of part of data payload:
![image](/uploads/9dacff15a729788fffb02e916b704569/image.png)
- Index (document) Format of part of data payload returned by method in class StorageIndexerPayloadMapper
```
public Map<String, Object> mapDataPayload(ArrayList<String> asIngestedCoordinatesPaths, IndexSchema storageSchema, Map<String, Object> storageRecordData,
String recordId) {
Map<String, Object> dataCollectorMap = new HashMap<>();
//..
mapDataPayload(storageSchema.getDataSchema(), storageRecordData, recordId, dataCollectorMap);
//...
return dataCollectorMap;
}
```
![image](/uploads/dfe1df18988936c5b137c542edd58c96/image.png)
- Search result before re-index from local indexer service with the [MR 649](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/649):
![image](/uploads/7714272f8aa0286c90b278e7546d8b33/image.png)
- Search result after re-index from local indexer service with the [MR 649](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/649):
![image](/uploads/b51dbecdc83cc6279b71017d1f8f1b61/image.png)M22 - Release 0.25Mark ChanceStanisław BienieckiMark Chancehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/issues/323Electrical Properties - Additional attributes2024-01-12T18:37:55ZMichael JonesElectrical Properties - Additional attributesAdd the following attributes to the Electrical Properties content schema at the same level as SampleID:
```JSON
"Temperature": {
"type": "object",
"properties": {
"Value": {
"type": "number"
},
"UnitOfMeasure": {
...Add the following attributes to the Electrical Properties content schema at the same level as SampleID:
```JSON
"Temperature": {
"type": "object",
"properties": {
"Value": {
"type": "number"
},
"UnitOfMeasure": {
"type": "string",
"pattern": "^[\\w\\-\\.]+:reference-data\\-\\-UnitOfMeasure:[\\w\\-\\.\\:\\%]+:[0-9]*$"
}
},
"NetConfiningStress": {
"type": "object",
"properties": {
"Value": {
"type": "number"
},
"UnitOfMeasure": {
"type": "string",
"pattern": "^[\\w\\-\\.]+:reference-data\\-\\-UnitOfMeasure:[\\w\\-\\.\\:\\%]+:[0-9]*$"
}
},
"Frequency": {
"type": "object",
"properties": {
"Value": {
"type": "number"
},
"UnitOfMeasure": {
"type": "string",
"pattern": "^[\\w\\-\\.]+:reference-data\\-\\-UnitOfMeasure:[\\w\\-\\.\\:\\%]+:[0-9]*$"
}
}
```M23 - Release 0.26Ernesto GutierrezErnesto Gutierrezhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/136The augmented attributes are not searchable2024-01-26T18:29:27ZZhibin MaiThe augmented attributes are not searchableA common issue as mentioned in the [IndexAugmenter.md](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/docs/tutorial/IndexAugmenter.md), the augmented attributes are not searchable. It requires the reco...A common issue as mentioned in the [IndexAugmenter.md](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/docs/tutorial/IndexAugmenter.md), the augmented attributes are not searchable. It requires the records of the augmented kind(s) to be re-indexed.
It is understandable that in order to augment the existing records, it needs to re-index the existing records of the augmented kind(s). However, there is a common scenario, data admins/managers want to verify the effect/result of the augmenter configuration immediately after they deploy the augmenter configuration. They normally don't have permission to trigger re-index. Furthermore, in this scenario, they should not trigger re-index of the whole kind(s) before they finalize the augmenter configuration for given kind(s).
If the indexer can automatically update the schema mapping of the augmented kind(s) in the ElasticSearch when it detects that the augmented configuration was updated. Then data admins/managers can see the effect/result of the augmenter configuration immediately by updating one of the existing data records or inserting a new data record. It will tremendously reduce the time on troubleshooting as well as developing and deploying new/updated augmenter configurations.
Given updating the schema mapping of the augmented kind(s) in the ElasticSearch is a lightweight operation as comparing to re-index the whole kind(s), I think it is worth to make this enhancement.M23 - Release 0.26Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/issues/322Preship postman collection is hardcoded to Azure ACLs2024-01-03T22:14:12ZBryan DawsonPreship postman collection is hardcoded to Azure ACLsThe postman collection for preshipping hardcodes the ACLs to a `contoso.com` domain.
![image.png](/uploads/e693f39caae11c3f4fa043628a023e2a/image.png)
Instead, this should use a variable for whole part after the `@` so that AWS and G...The postman collection for preshipping hardcodes the ACLs to a `contoso.com` domain.
![image.png](/uploads/e693f39caae11c3f4fa043628a023e2a/image.png)
Instead, this should use a variable for whole part after the `@` so that AWS and GC preshipping can reuse the same collection.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/issues/321Inconsistent variable usage in reference value postman collection2024-01-03T19:52:31ZBryan DawsonInconsistent variable usage in reference value postman collectionIn the postman collection for loading the reference data it uses the variable `WORKFLOW_URL` for most of the requests
![image.png](/uploads/1cc45ca96af511b64707de142f8418fe/image.png)
However, some of the newer requests added use a di...In the postman collection for loading the reference data it uses the variable `WORKFLOW_URL` for most of the requests
![image.png](/uploads/1cc45ca96af511b64707de142f8418fe/image.png)
However, some of the newer requests added use a different variable of `osdu_endpoint`
![image.png](/uploads/eb3ef72ada79756b33fe130b0f5fbb7f/image.png)
We should be consistent and use `WORKFLOW_URL` for all.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/issues/320Inconsistent variable usage in schema registration postman collection2024-01-03T19:52:31ZBryan DawsonInconsistent variable usage in schema registration postman collectionMost of the requests in the [schema collection](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/blob/main/deployments/rafsddms_schemas_mvp.postman_collection.json?ref_typ...Most of the requests in the [schema collection](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/blob/main/deployments/rafsddms_schemas_mvp.postman_collection.json?ref_type=heads) use the variable `SCHEMA_HOST` for the URL:
![image.png](/uploads/e488c000e08eac35ea04c155c873acae/image.png)
but a few do not and require setting up a separate variable for `OSDU_BASE_HOST`
![image.png](/uploads/0ae7881531b0a26d1c78e955a8d0f7e3/image.png)
We should change the collection to be consistent.