seismic-dms-service merge requestshttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests2023-08-18T10:25:39Zhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/354ci: fixing gcp broken config map2023-08-18T10:25:39ZDaniel Perezci: fixing gcp broken config mapUpdating variables in GCP config mapUpdating variables in GCP config mapM12 - Release 0.15Daniel PerezDaniel Perezhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/304fix: azure cosmos DB redesign to improve general performances2022-06-20T10:58:40ZDiego Moltenifix: azure cosmos DB redesign to improve general performancesThe existing design of cosmos DB could not meet the required performances. All datasets that belong to the same subproject are stored with/under the same physical partition key. A physical partition set a limit of 10k RU/s (cannot be mod...The existing design of cosmos DB could not meet the required performances. All datasets that belong to the same subproject are stored with/under the same physical partition key. A physical partition set a limit of 10k RU/s (cannot be modified). To enable cosmos horizontal scaling and higher RU/s the DB has been redesigned so every entity (tenant/subproject/dataset/app) will have its own partition key. This will allow cosmos to spread entities across multiple physical partitions
* tenant -> tn-<tenant_namme> (1 per partition)
* subproject -> sp-<subproject_name> (subproject belong to a tenant )
* dataset -> ds-<subprpoject_name>-<dataset_path>-<dataset_name> (datasets belong to a subproject... the subproject name is required)
* app -> app-<app_name> (apps belong to a tenant)
ref: https://docs.microsoft.com/en-us/azure/cosmos-db/partitioning-overview#choose-partitionkey
An issue has been fixed on how global tags are parsed from input requests.
Added support for managing azure migration:
- The service will work with the older index model if exist only the old cosmos database
- The service will work with the new index model if exist only the new cosmos database (partition migrated or newly created)
- The service will throw an error if both datasets exist at the same time (migration in progress)
The correlation id propagation has been fixed.
The Azure insight telemetry process has been fixed so probes do not get included as consumer in the tracking (as does not have auth info)M12 - Release 0.15https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/489Cherry pick: Merge branch 'fix_Anthos_logger_minio' into 'release/0.16'2022-12-09T13:49:22ZChad LeongCherry pick: Merge branch 'fix_Anthos_logger_minio' into 'release/0.16'fix: Anthos logger, MinIO list objects, and search Datasets by gtags
See merge request osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service!467
(cherry picked from commit 6253eb78dfe2240529e65214e8815...fix: Anthos logger, MinIO list objects, and search Datasets by gtags
See merge request osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service!467
(cherry picked from commit 6253eb78dfe2240529e65214e881539dd48627ea)
c802dabf fix: Anthos logger and Minio
e3d5324d fix: linters
6cabc36d fix: search tags
9eb8031c Merge branch 'master' of...M13 - Release 0.16David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/484Cherry pick: Merge branch 'slb/dm3/force-subproject-key' into 'master'2022-08-17T08:32:45ZChad LeongCherry pick: Merge branch 'slb/dm3/force-subproject-key' into 'master'fix: (azure) forcing usage old key model when using a single partition model db in azure comsos
See merge request osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service!479
(cherry picked from commit 1e...fix: (azure) forcing usage old key model when using a single partition model db in azure comsos
See merge request osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service!479
(cherry picked from commit 1e7082a0e4207f518daec9d08f4a4d5ba1f86cba)
dd2623eb fix: forcing usage old key in azure old dbM13 - Release 0.16David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/483ci: update alpine version to fix busybox vulnerability2022-08-17T08:34:59ZYifei Xuci: update alpine version to fix busybox vulnerabilityOriginal MR: !444Original MR: !444M13 - Release 0.16Yifei XuOkoun-Ola Fabien HouetoYifei Xuhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/482Cherry-pick 'feat: added list endpoint as post' into release/0.162022-08-15T18:32:31ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'feat: added list endpoint as post' into release/0.16Original MR: !436Original MR: !436M13 - Release 0.16David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/479fix: (azure) forcing usage old key model when using a single partition model ...2022-08-15T16:22:58ZDiego Moltenifix: (azure) forcing usage old key model when using a single partition model db in azure comsosforcing usage old key model when using a single partition model db in azure comsosforcing usage old key model when using a single partition model db in azure comsosM13 - Release 0.16Diego MolteniDiego Moltenihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/468Cherry-pick 'ci: fixed pipeline condition' into release/0.162022-08-11T03:27:40ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'ci: fixed pipeline condition' into release/0.16Original MR: !464Original MR: !464M13 - Release 0.16David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/467fix: Anthos logger, MinIO list objects, and search Datasets by gtags2022-08-17T09:01:02ZYan Sushchynski (EPAM)fix: Anthos logger, MinIO list objects, and search Datasets by gtagsFix two issues:
1. log4js doesn't write logs in GCP, so it was changed to usual `console`
1. Different versions of MinIO return different responses for `ListObjects` with and without `Contents` field, so we added handling such a situati...Fix two issues:
1. log4js doesn't write logs in GCP, so it was changed to usual `console`
1. Different versions of MinIO return different responses for `ListObjects` with and without `Contents` field, so we added handling such a situation.
2. Add searching in JSON arrays in the DatabaseM13 - Release 0.16https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/464ci: fixed pipeline condition2022-08-16T14:50:45ZDaniel Perezci: fixed pipeline conditionM13 - Release 0.16https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/455fix: defined the model for file-metadata record field in the azure sidecar2022-08-03T14:02:56ZDiego Moltenifix: defined the model for file-metadata record field in the azure sidecarThe filemetadata model was modeled as generic object in the sidecar and this was not allowing the conversion in the service.
this MR will provide the data model for the file-metadata field for the dataset record.The filemetadata model was modeled as generic object in the sidecar and this was not allowing the conversion in the service.
this MR will provide the data model for the file-metadata field for the dataset record.M13 - Release 0.16Diego MolteniDiego Moltenihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/450ci: migrate ubuntu and node images to use MCR for seismic sdms and filemetadata2022-08-18T23:43:44ZMarija Dukicci: migrate ubuntu and node images to use MCR for seismic sdms and filemetadata# Overview
- Copied [Dockerfile](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/master/app/filemetadata/Dockerfile) for Seismic Filemetadata to **devops/azur...# Overview
- Copied [Dockerfile](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/master/app/filemetadata/Dockerfile) for Seismic Filemetadata to **devops/azure** location and updated `FROM` in new Dockerfile to use ubuntu image from Microsoft Container Registry (this is to avoid other cloud providers using MCR)
- Copied [runtime.Dockerfile](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/master/app/sdms/docker/runtime.Dockerfile) for Seismic SDMS to **devops/azure** location and updated `FROM` in new Dockerfile to use node:16-alpine image from Microsoft Container Registry (this is to avoid other cloud providers using MCR)
- Updated **filemetadata_azure_containerize** job in pipeline to build image from new Dockerfile location
- Updated **sdms_azure_containerize** job in pipeline to build image from new Dockerfile location
- ADO work item: [Securing the containers supply chain - Seismic](https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/_workitems/edit/7918)
# Testing
- Local testing has passed successfully and new image is being used
- **filemetadata_azure_containerize** step is pulling _ubuntu:20.04_ image from MCR in MR pipeline
![image](/uploads/9b09260045683ddca74c79e23a00576f/image.png)
- **sdms_azure_containerize** step is pulling _node:14-alpine_ image from MCR in MR pipeline
![image](/uploads/b6dcff935597bc410f48b0b846469c9c/image.png)
- Filemetadata and SDMS Azure integration tests have passed successfully in MR pipeline
- SDMS ibm and gcp deployment is failing due to invalid or expired token (known issue)
- Additional testing will be done on Azure side with build pipeline against **m12-master** branchM13 - Release 0.16Marija DukicMarija Dukichttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/449ci: GONRG-5336 - add var for env id2022-07-29T08:09:38ZAliaksandr Ramanovich (EPAM)ci: GONRG-5336 - add var for env idadd var to define env id in gcp seismic configmapadd var to define env id in gcp seismic configmapM13 - Release 0.16Aliaksandr Ramanovich (EPAM)Aliaksandr Ramanovich (EPAM)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/446feat: Remove favicon because it is not licensed2022-07-27T19:37:50ZDuo Chenfeat: Remove favicon because it is not licensedRemove favicon because it is not licensed.Remove favicon because it is not licensed.M13 - Release 0.16https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/445fix: updated sidecar to support directories list in a subproject2022-07-28T12:47:25ZDiego Moltenifix: updated sidecar to support directories list in a subprojectThe sidecar was working by querying and converting full datasets records.
In case the directory list is required, the query selects only the "path" filed of the record and, because of this selection, the result cannot be converted.
This ...The sidecar was working by querying and converting full datasets records.
In case the directory list is required, the query selects only the "path" filed of the record and, because of this selection, the result cannot be converted.
This problem has been fixed by adding a specific endpoint in the sidecar that is called from the path selection query from SDMS.M13 - Release 0.16Diego MolteniDiego Moltenihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/444ci: update alpine version to fix busybox vulnerability2023-05-11T12:38:37ZYifei Xuci: update alpine version to fix busybox vulnerabilityM13 - Release 0.16Yifei XuYifei Xuhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/442fix: list sub-projects returns datasets from other sub-projects2022-07-25T14:44:37ZDiego Moltenifix: list sub-projects returns datasets from other sub-projectsReplaced the generic LIKE rule with a RegExMatch in the SQL query and URI Encode it before sending it to the sidecar.
The SQL query select datasets in a subproject using the rule "id LIKE sp-<tenant_name>-<subproject_name>-%" In case a ...Replaced the generic LIKE rule with a RegExMatch in the SQL query and URI Encode it before sending it to the sidecar.
The SQL query select datasets in a subproject using the rule "id LIKE sp-<tenant_name>-<subproject_name>-%" In case a subproject contains a "-" in the name, it can be wrongly selected.
For example:
tenant_name = "opendes"
subproject_name_01= "sandbox"
subproject_name_02 = "sandbox-test"
To select datasets in "sandbox" using the rule "id LIKE sp-opendes-sandbox-%" also wrongly select dataset in "sandbox-test"
replacing the SQL LIKE rule with the RegexMath fixes the issueM13 - Release 0.16Diego MolteniDiego Moltenihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/439feat: add data partition id logging2022-07-28T14:58:28ZKonstantin Khottchenkovfeat: add data partition id loggingM13 - Release 0.16Konstantin KhottchenkovKonstantin Khottchenkovhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/438ci: adding files for cicd pipeline2022-07-29T07:57:37ZDaniel Perezci: adding files for cicd pipelineAdding files to be used in cicd pipeline.Adding files to be used in cicd pipeline.M13 - Release 0.16Daniel PerezDaniel Perezhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/merge_requests/437fix: refactor vars to camelcase2022-07-29T07:56:22ZKrisztian Molnar (EPAM)fix: refactor vars to camelcaseM13 - Release 0.16Krisztian Molnar (EPAM)Krisztian Molnar (EPAM)