Indexer merge requestshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests2022-01-20T18:01:43Zhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/267Draft: fix whitesource issues2022-01-20T18:01:43ZYauheni LesnikauDraft: fix whitesource issuesYauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/262Reindex API Improvement2022-01-21T10:22:44ZVibhuti Sharma [Microsoft]Reindex API ImprovementAddresses https://community.opengroup.org/osdu/platform/system/storage/-/issues/106Addresses https://community.opengroup.org/osdu/platform/system/storage/-/issues/106Vibhuti Sharma [Microsoft]Vibhuti Sharma [Microsoft]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/226Upgrade OSDU dependencies2022-01-25T17:37:00ZDavid Diederichd.diederich@opengroup.orgUpgrade OSDU dependenciesThis includes !165 in the commit graph, so that MR will close when this one does.
Also, that will close #25.
This upgrade does more than just upgrade the core-common, but it also does that.This includes !165 in the commit graph, so that MR will close when this one does.
Also, that will close #25.
This upgrade does more than just upgrade the core-common, but it also does that.David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/239tomcat embed core vuln fix2022-01-28T09:12:44ZGokul Nagaretomcat embed core vuln fixfixes issue - https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/45fixes issue - https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/45M10 - Release 0.13Anuj GuptaShrikant GargAnuj Guptahttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/165Resolve "Upgrade Core Common Dependency"2022-02-08T12:52:02ZDavid Diederichd.diederich@opengroup.orgResolve "Upgrade Core Common Dependency"Closes #21
See osdu/platform&9 for more details.
## MR's Raison d'ĂȘtre
This MR updates the core common dependency to use the latest released version of the library. Keeping our library usage at the latest helps avoid subtle incompati...Closes #21
See osdu/platform&9 for more details.
## MR's Raison d'ĂȘtre
This MR updates the core common dependency to use the latest released version of the library. Keeping our library usage at the latest helps avoid subtle incompatibility bugs from creeping in, and ensures that all Data Platform code is staying current with bug fixes and performance improvements in the core libraries.
*Dependency Table, after the application of this MR's changes*
| Maven Dependencies | _Root POM_ | testing/ |
| ------------------------ | ------------------ | ---------- |
| os-core-common | 0.12.0-rc3 | 0.12.0-rc3 |
| core.aws.os-core-lib-aws | 0.11.0 | 0.3.16 |
| core-lib-azure | 0.11.0-rc5 | 0.11.0-rc4 |
| core-lib-gcp | 0.10.0, 0.12.0-rc3 | |
| os-core-lib-ibm | 0.12.0-SNAPSHOT | 0.7.0 |
## Housekeeping Updates
### FOSSA Update
This MR also updates the FOSSA attributions, since these need to be kept up to date on every MR. Changes are expected whenever the library dependencies are updated.
### Update Deprecated Pipeline Includes
This MR also updates the `.gitlab-ci.yml` file to use the newer pipeline includes, rather than relying on deprecated includes that mimic old behavior. These housekeeping updates should be done regularly on MRs when they are discovered so the old CI include files can be deleted and no longer supported.M9 - Release 0.12David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/290Fix retry for deleted records when index not exist2022-03-04T12:18:19ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comFix retry for deleted records when index not exist# Description:
When the indexer receives an event with Op type DELETE, it deletes provided records at elasticsearch.
During integration tests scenario when index completely deleted at elasticsearch are often, and in that case,
if index ...# Description:
When the indexer receives an event with Op type DELETE, it deletes provided records at elasticsearch.
During integration tests scenario when index completely deleted at elasticsearch are often, and in that case,
if index receives DELETE events for specific records in that index, they will always fail:
~~~
{
"data": "[{\"id\":\"osdu:work-product-component--WellboreMarkerSet:60182840a28944bb81d81cc4d4c60136\",\"kind\":\"testtest:wks:work-product-component--WellboreMarkerSet:1.0.0\",\"op\":\"delete\"}]",
"attributes": {
"account-id": "{{data-partition-id}}"
}
}
~~~
But due to NOT_FOUND response from elasticsearch this task will go for the next round of retry
~~~
"{auditLog={resources=[RecordStatus(id=osdu:inttest:164613302639927, kind=osdu:ds:inttest:1.0.1646133014242, operationType=delete, status=FAIL, message=[osdu-ds-inttest-1.0.1646133014242] ElasticsearchException[Elasticsearch exception [type=index_not_found_exception, reason=no such index [osdu-ds-inttest-1.0.1646133014242]]])], action=DELETE, actionId=IN003, message=Failed deleting record in index, user=workload-indexer-gcp@nice-etching-277309.iam.gserviceaccount.com, status=FAILURE}} {account-id=osdu, correlation-id=8729b1d4-17c9-4141-9257-5fd63ec9c373, data-partition-id=osdu}"
~~~
Currently, this forms a cycle at GCP env with endlessly task retry.
# How to test:
Does functionality was tested and how?
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [ ] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [ ] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M11 - Release 0.14Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/204Draft: Update pom.xml2022-03-16T17:09:52ZSmitha ManjunathDraft: Update pom.xmlupdate dependencyupdate dependencySmitha ManjunathSmitha Manjunathhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/298PBI-5653: Upgrading spring boot dependencies. Spring boot dependencies wil be...2022-03-28T11:03:12ZHarsheet ShahPBI-5653: Upgrading spring boot dependencies. Spring boot dependencies wil be...PBI-5653: Upgrading spring boot dependencies. Spring boot dependencies wil be fetched from os-common-core.PBI-5653: Upgrading spring boot dependencies. Spring boot dependencies wil be fetched from os-common-core.Harsheet ShahHarsheet Shahhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/301Upgrade First Party Library Dependencies for Release 0.142022-04-05T10:39:28ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.14This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Preparing packages...
Branch: master
SHA: cfdbcd6a9f0d1a7df0604d1d838cdbd88163c4fb
Maven: 0.14.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| --------------------------------------------------- | ------------------ | ---------------------- |
| core-lib-azure | 0.14.0-rc2 | 0.12.0-rc10 |
| core-lib-gcp | 0.14.0-rc2, 0.10.0 | |
| os-core-lib-aws | 0.13.0 | 0.3.16 |
| obm | 0.13.1-SNAPSHOT | |
| oqm | 0.13.0-SNAPSHOT | |
| os-core-common | 0.14.0-rc4 | 0.14.0-rc4 |
| os-core-lib-ibm | 0.13.0 | 0.13.0 |
| osm | 0.13.0-SNAPSHOT | |
| (3rd Party) net.minidev.json-smart | 2.4.7 | 2.4.6 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |
### Dependency Information After the Upgrade
```
Preparing packages...
Branch: dependency-upgrade
SHA: f812a5cadee154e2e4b7e1d2bb4b4370bc441d14
Maven: 0.14.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| --------------------------------------------------- | -------------- | ---------------------- |
| core-lib-azure | 0.14.0 | 0.12.0-rc10 |
| core-lib-gcp | 0.14.0, 0.10.0 | |
| os-core-lib-aws | 0.14.0 | 0.3.16 |
| obm | 0.14.0 | |
| oqm | 0.14.0 | |
| os-core-common | 0.14.0 | 0.14.0 |
| os-core-lib-ibm | 0.14.0 | 0.14.0 |
| osm | 0.14.0 | |
| (3rd Party) net.minidev.json-smart | 2.4.7 | 2.4.6 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |M11 - Release 0.14https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/302Upgrade First Party Library Dependencies for Release 0.14 (Except Azure)2022-04-05T10:39:33ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.14 (Except Azure)This is related to !301, except it holds back the Azure dependency -- based on the result that the Azure tests failed in !201.
### Dependency Information After the UpgradeThis is related to !301, except it holds back the Azure dependency -- based on the result that the Azure tests failed in !201.
### Dependency Information After the UpgradeM11 - Release 0.14David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/295(GONRG-4427) helm on prem2022-04-08T13:36:03ZVladislav Shishko (EPAM)(GONRG-4427) helm on premVladislav Shishko (EPAM)Vladislav Shishko (EPAM)https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/197unflatten data attribute keys to get pretty response2022-04-18T13:57:19ZNeelesh Thakurunflatten data attribute keys to get pretty responseMake Search service response pretty:
Current sample response:
```json
{
"results": [
{
"data": {
"relationships.definitiveTrajectory.id": "opendes:trajectory:0ed0164d-bd45-43ad-8fef-0fcec0d9fd87_trajectory",
...Make Search service response pretty:
Current sample response:
```json
{
"results": [
{
"data": {
"relationships.definitiveTrajectory.id": "opendes:trajectory:0ed0164d-bd45-43ad-8fef-0fcec0d9fd87_trajectory",
"relationships.tieInWellbore.version": 0,
"relationships.tieInWellbore.confidence": 0,
"relationships.definitiveTrajectory.confidence": 0,
"relationships.definitiveTrajectory.version": 0,
"name": "4900105093"
},
"kind": "opendes:petrel:wellbore:1.0.6"
}
]
}
```
With this change, formatted search service response:
```json
{
"results": [
{
"data": {
"relationships": {
"definitiveTrajectory": {
"id": "opendes:trajectory:0ed0164d-bd45-43ad-8fef-0fcec0d9fd87_trajectory",
"confidence": 0,
"version": 0
},
"tieInWellbore": {
"version": 0,
"confidence": 0
}
},
"name": "4900105093"
},
"kind": "opendes:petrel:wellbore:1.0.6"
}
]
}
```
Related core-common MR: [117](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/117)M9 - Release 0.12https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/335Update core lib azure version2022-06-21T14:02:34ZHarsheet ShahUpdate core lib azure version(cherry picked from commit 1bd4a6e6aaf681ac0b14c12b0c288dd1587258e2)
Conflicts:
.gitlab-ci.yml(cherry picked from commit 1bd4a6e6aaf681ac0b14c12b0c288dd1587258e2)
Conflicts:
.gitlab-ci.ymlHarsheet ShahHarsheet Shahhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/339Draft:Ibm indexer helm2022-07-01T06:09:57ZShrikant GargDraft:Ibm indexer helminitial chart for indexerinitial chart for indexerShrikant GargShrikant Garghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/344Fixed jackson conflict version issue2022-07-12T08:07:52ZDmitrii Novikov (EPAM)Fixed jackson conflict version issueDmitrii Novikov (EPAM)Dmitrii Novikov (EPAM)https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/234Fix a false cache hit in a multipartition scenario2022-07-12T15:41:06ZSabarish K R EFix a false cache hit in a multipartition scenarioFixes a scenario where an index created in DP 1, but when same index name's records are put in DP2, the index mapping doesn't get created due to cache hit.Fixes a scenario where an index created in DP 1, but when same index name's records are put in DP2, the index mapping doesn't get created due to cache hit.https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/211Autoscaling and resource limits based on feature flag2022-07-12T15:41:08ZAlok JoshiAutoscaling and resource limits based on feature flagAdd autoscaling configuration with hpa and pod resource limits based on environment feature flagAdd autoscaling configuration with hpa and pod resource limits based on environment feature flagM9 - Release 0.12Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/244Update pom.xml2022-07-12T15:41:10ZSmitha ManjunathUpdate pom.xmlhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/265Remove Page Size check while querying Storage API2022-07-12T15:41:14ZVibhuti Sharma [Microsoft]Remove Page Size check while querying Storage APIVibhuti Sharma [Microsoft]Vibhuti Sharma [Microsoft]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/281Adding rate limit filter2022-08-05T06:24:18ZSHEFFALI JAINAdding rate limit filter## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [ ] YES
- [x] NO
## Does this introduce a change in the...## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [ ] YES
- [x] NO
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [ ] YES
- [x] NO
## Scope and Context
We are trying to provide design via which rate limiting can be applied to any service if enabled via flag and will be disabled by default. By setting a limit on how many requests a consumer is allowed to make in a given unit of time. We reject any requests above the limit with an appropriate response, like HTTP status 429 (Too Many Requests).
## What is the current behavior?
Currently, no rate limiting is applied on service which can limit the count of users accessing it.
## What is the new/expected behavior?
The service will have specific token count which will set limit to restrict number of users using the service per cycle.
## Have you added/updated Unit Tests and Integration Tests?
No.
## Any other useful information
Added Envoy filter to apply rate limiting. Added support to generate yaml file via Helm in deployment itself.
Initially value of applying rate limit filter is set to false.
It can be set while installing helm command using following instruction :
--set envoyFilter.enabled=trueSHEFFALI JAINSHEFFALI JAIN