Indexer merge requestshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests2023-08-18T15:05:03Zhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/525AWS: Fixes user token timeout when a lengthy reindexing is started2023-08-18T15:05:03ZDerek HudsonAWS: Fixes user token timeout when a lengthy reindexing is startedFixes user token timeout when a lengthy reindexing procedure is initiated by using a service principal rather than the user's more temporary tokens.
Note: this is based on a stale master branch. The pipeline stages for GCP are failing b...Fixes user token timeout when a lengthy reindexing procedure is initiated by using a service principal rather than the user's more temporary tokens.
Note: this is based on a stale master branch. The pipeline stages for GCP are failing because this is based on a stale master branch. The branch this was based on is [here](https://community.opengroup.org/osdu/platform/system/indexer-service/-/pipelines/179562), where the pipeline is also failing for those same issues.M18 - Release 0.21Yong ZengDerek HudsonYong Zenghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/386AWS Helm Build Update2022-09-20T22:24:56ZMarc Burnie [AWS]AWS Helm Build UpdateM14 - Release 0.17Marc Burnie [AWS]Marc Burnie [AWS]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/19Aws integration2023-08-18T15:18:59ZMatt WiseAws integrationM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/18Aws integration2020-08-06T14:31:56ZMatt WiseAws integrationhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/503Aws integration merge2023-03-10T17:38:30ZManish JangidAws integration mergeUpdated container memoryUpdated container memoryM16 - Release 0.19https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/647AWS - Merge from dev2023-11-27T19:07:40ZGuillaume CailletAWS - Merge from devAWS changes only. No changes on core logic or other CSP.
* Add or update License header
* Upgrade buildspec
* Update security configAWS changes only. No changes on core logic or other CSP.
* Add or update License header
* Upgrade buildspec
* Update security configM22 - Release 0.25Guillaume CailletGuillaume Caillethttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/116Aws reindex support2023-08-18T15:08:34ZMatt WiseAws reindex supportM5 - Release 0.8Matt WiseBill WangMatt Wisehttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/715Aws side code quality2024-02-26T20:24:57ZBruce JinAws side code qualityThis MR will improve AWS code quality, fix some codesmellThis MR will improve AWS code quality, fix some codesmellM23 - Release 0.26Bruce JinBruce Jinhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/145AWS Support ECK/Elasticsearch on EKS2023-08-18T15:08:09ZMatt WiseAWS Support ECK/Elasticsearch on EKScommit b3a5f318
Author: Matt Wise <wsmatth@amazon.com>
Date: Mon Mar 22 2021 11:44:35 GMT-0500 (Central Daylight Time)
fix int tests for new elasticsearch deployment
commit 71ba4de3
Author: Matt Wise <wsmatth@amazon.com>
Date...commit b3a5f318
Author: Matt Wise <wsmatth@amazon.com>
Date: Mon Mar 22 2021 11:44:35 GMT-0500 (Central Daylight Time)
fix int tests for new elasticsearch deployment
commit 71ba4de3
Author: Matt Wise <wsmatth@amazon.com>
Date: Thu Mar 18 2021 14:02:48 GMT-0500 (Central Daylight Time)
support new elasticsearch connection model
commit 4d512443
Author: Matt Wise <wsmatth@amazon.com>
Date: Thu Mar 18 2021 13:57:26 GMT-0500 (Central Daylight Time)
Merge remote-tracking branch 'origin/master' into dev
commit bbaaa4af
Author: zhijie wang <wanzhiji@amazon.com>
Date: Thu Mar 04 2021 11:52:26 GMT-0600 (Central Standard Time)
re-enable test
commit 8e187e1f
Author: zhijie wang <wanzhiji@amazon.com>
Date: Wed Mar 03 2021 16:24:12 GMT-0600 (Central Standard Time)
merge dev
commit b8bde92f
Author: zhijie wang <wanzhiji@amazon.com>
Date: Wed Mar 03 2021 15:26:52 GMT-0600 (Central Standard Time)
add reindex task
commit 86590ad2
Author: zhijie wang <wanzhiji@amazon.com>
Date: Fri Feb 12 2021 16:35:55 GMT-0600 (Central Standard Time)
remove uncessary hard code
commit 94db5a97
Author: zhijie wang <wanzhiji@amazon.com>
Date: Fri Feb 12 2021 16:28:58 GMT-0600 (Central Standard Time)
use correct indexerQueueTaskBuilder for AWS
commit 5d95151c
Author: zhijie wang <wanzhiji@amazon.com>
Date: Fri Feb 12 2021 11:48:00 GMT-0600 (Central Standard Time)
Solve DependencyInjection for ReIndexService (previously using IndexerQueueTaskBuilder from Core)
commit f3824aff
Author: wanzhiji <wanzhiji@amazon.com>
Date: Fri Feb 05 2021 16:27:31 GMT-0600 (Central Standard Time)
Storage 404 error causing message to not retry. Add fix
commit 149d03fc
Author: Matt Wise <wsmatth@amazon.com>
Date: Fri Feb 05 2021 10:17:20 GMT-0800 (Pacific Standard Time)
Merge remote-tracking branch 'origin/dev' into storage-404
commit f478d4d3
Author: zhijie wang <wanzhiji@amazon.com>
Date: Fri Feb 05 2021 09:10:16 GMT-0800 (Pacific Standard Time)
fix storage 404
commit 3580a510
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com>
Date: Wed Feb 03 2021 08:13:36 GMT-0800 (Pacific Standard Time)
Merge branch 'dev' of codecommit://os-indexer into dev
commit e17cd0f9
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com>
Date: Wed Feb 03 2021 08:13:24 GMT-0800 (Pacific Standard Time)
ubuntu expired key fix
commit 8623a8c6
Author: Spencer Sutton <suttonsp@amazon.com>
Date: Wed Feb 03 2021 07:21:23 GMT-0800 (Pacific Standard Time)
SSL AWS
commit 78cab700
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com>
Date: Fri Jan 29 2021 12:41:00 GMT-0600 (Central Standard Time)
Adding ssl props
commit 82a15c82
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com>
Date: Fri Jan 29 2021 12:20:36 GMT-0600 (Central Standard Time)
Adding SSL
commit cd820a21
Author: Matt Wise <wsmatth@amazon.com>
Date: Tue Feb 02 2021 07:57:36 GMT-0800 (Pacific Standard Time)
disable test failures
commit a3dacfb7
Author: Spencer Sutton <suttonsp@amazon.com>
Date: Wed Feb 03 2021 09:21:23 GMT-0600 (Central Standard Time)
SSL AWS
commit 78cab700
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com>
Date: Fri Jan 29 2021 12:41:00 GMT-0600 (Central Standard Time)
Adding ssl props
commit 82a15c82
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com>
Date: Fri Jan 29 2021 12:20:36 GMT-0600 (Central Standard Time)
Adding SSL
commit 40af5d94
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com>
Date: Wed Feb 03 2021 10:13:24 GMT-0600 (Central Standard Time)
ubuntu expired key fix
commit 56b3f539
Author: Matt Wise <wsmatth@amazon.com>
Date: Tue Feb 02 2021 09:57:36 GMT-0600 (Central Standard Time)
disable test failuresM5 - Release 0.8Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/602aws-sync-dev-to-master2023-08-19T02:34:31ZLong Chengaws-sync-dev-to-masterM20 - Release 0.23Long ChengLong Chenghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/172AWS Updates2023-08-18T15:07:46ZSpencer Suttonsuttonsp@amazon.comAWS Updatescommit 54e13552
Author: Spencer Sutton <suttonsp@amazon.com>
Date: Thu May 27 2021 10:10:45 GMT-0500 (Central Daylight Time)
"Excluding springs default security, locking down mvn central"commit 54e13552
Author: Spencer Sutton <suttonsp@amazon.com>
Date: Thu May 27 2021 10:10:45 GMT-0500 (Central Daylight Time)
"Excluding springs default security, locking down mvn central"M7 - Release 0.10Spencer Suttonsuttonsp@amazon.comSpencer Suttonsuttonsp@amazon.comhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/171AWS Updates2021-07-13T17:44:22ZSpencer Suttonsuttonsp@amazon.comAWS Updatescommit 54e13552
Author: Spencer Sutton <suttonsp@amazon.com>
Date: Thu May 27 2021 10:10:45 GMT-0500 (Central Daylight Time)
"Excluding springs default security, locking down mvn central"commit 54e13552
Author: Spencer Sutton <suttonsp@amazon.com>
Date: Thu May 27 2021 10:10:45 GMT-0500 (Central Daylight Time)
"Excluding springs default security, locking down mvn central"Spencer Suttonsuttonsp@amazon.comSpencer Suttonsuttonsp@amazon.comhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/91AWS Updates2023-08-18T15:08:55ZSpencer Suttonsuttonsp@amazon.comAWS UpdatesM4 - Release 0.7ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoJoeDmitriy Rudkoethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/722Aws upgrade aws lib2024-03-11T14:07:40ZDerek HudsonAws upgrade aws libM23 - Release 0.26Derek HudsonDerek Hudsonhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/394AWS Using Helm to Deploy2022-10-01T00:00:19ZMarc Burnie [AWS]AWS Using Helm to DeployM14 - Release 0.17Marc Burnie [AWS]Marc Burnie [AWS]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/11Azure - (4/4) - Adding missing context path for indexer2020-07-16T23:03:22ZDuvelis CaraoAzure - (4/4) - Adding missing context path for indexer**This only impacts Azure implementation.**
This MR is part of group of MRs with dependencies between them
0. Update infrastructure (Azure / osdu-infrastructure) to have reference to the new URLs (https://github.com/Azure/osdu-infr...**This only impacts Azure implementation.**
This MR is part of group of MRs with dependencies between them
0. Update infrastructure (Azure / osdu-infrastructure) to have reference to the new URLs (https://github.com/Azure/osdu-infrastructure/pull/23)
1. Adding missing context path for entitlements.
2. Adding missing context path for legal.
3. Adding missing context path for storage.
4. Adding missing context path for indexer
After merging this MR, the new indexer URL will be:
https://{host}**/api/indexer/v2/**
Before merging this MR:
- Please make sure the dependent MR is done: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/33
--------
This PR is related to 6 other requests for changing a URL structure and adding in context paths for services.
1. osdu-infrastructure 1/6 PR https://github.com/Azure/osdu-infrastructure/pull/23
1. CI-CD Pipelines 2/6 This PR
1. Entitlements Azure 3/6 https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-azure/-/merge_requests/11
1. Legal 4/6 https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/merge_requests/32
1. Storage 5/6 https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/33
1. Indexer 6/6 https://community.opengroup.org/osdu/platform/system/indexer-service
They are all required and any one but not the other will break builds and tests across the entire ecosystem for Azure.M1 - Release 0.1ethiraj krishnamanaiduDania Kodeih (Microsoft)Daniel Schollethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/215[Azure] Added health checks for indexer service2023-08-18T15:06:56ZAalekh Jain[Azure] Added health checks for indexer serviceAdded liveness and readiness probes in azure for indexer service.Added liveness and readiness probes in azure for indexer service.M9 - Release 0.12https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/126Azure: Fixing reindex2023-08-18T15:08:24ZJasonAzure: Fixing reindexFixing Azure reindex so that it is able to interact with entitlements service of http within the k8s cluster. Related to #14
From initial investigation, it seems like the issue was in part that the `x-functions-key` header was not bein...Fixing Azure reindex so that it is able to interact with entitlements service of http within the k8s cluster. Related to #14
From initial investigation, it seems like the issue was in part that the `x-functions-key` header was not being set properly, and so the value was coming through as null when indexer-service was checking authorization with entitlements. This is because Istio was preventing the request from even hitting the service container for entitlements. When we turn off Istio, we see some informative error logs from tomcat in the entitlements pod:
```
2021-03-17 13:11:46.830 INFO entitlements-azure-57764d67f8-6jthb --- [p-nio-80-exec-4] o.a.c.h.Http11Processor correlation-id= data-partition-id=: Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: The HTTP header line [x-functions-key0x0d] does not conform to RFC 7230 and has been ignored.
at org.apache.coyote.http11.Http11InputBuffer.skipLine(Http11InputBuffer.java:1041)
at org.apache.coyote.http11.Http11InputBuffer.parseHeader(Http11InputBuffer.java:893)
at org.apache.coyote.http11.Http11InputBuffer.parseHeaders(Http11InputBuffer.java:593)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:284)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:880)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1601)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
```
This is interesting for two reason: (1) it seems that Istio and Tomcat perform some header validation. (2) For whatever reason, the validation is only behaving this way over http and not https. I did a bit of reading on the Tomcat code that is throwing the error. I think the relevant part is [here](https://github.com/apache/tomcat/blob/9.0.x/java/org/apache/coyote/http11/Http11InputBuffer.java#L891). So since there is this illegal character in the header, Tomcat throws a 400 on behalf of entitlements service for the request. It seems Istio must do something similar since the request does not even reach the service container when Istio is on.
Interestingly, both Istio and Tomcat only seem to be doing this for http and not https. We were able to successfully make the call (even with the null header) before when we referred to entitlements service by its DNS name rather than the internal access point within the cluster (`http://entitlements-azure/entitlements/v1`).
This MR resolves this issue by directly hardcoding the value for the `x-functions-key` header. We were already hardcoding the value to be `abcd` in `application.properties`. However, it was not being set properly because the code block the defines what the header value should be set to:
```
@Value("${indexer.queue.key}")
private String queueKey;
```
was running after the variable was used, so it was coming through as null.M5 - Release 0.8JasonJasonhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/113Azure || Pre-Scale Indexer to match Storage2023-08-18T15:08:39ZKrishna Nikhil VedurumudiAzure || Pre-Scale Indexer to match Storage## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provi...## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
The min of 2 containers are not enough to handle the load that 10 storage service containers can handle.
## What is the new/expected behavior?
Pre-scaling the Indexer service to 10 containers.
Note: The values can be fine-tuned once container bench marking is done for Indexer Service
## Have you added/updated Unit Tests and Integration Tests?
N/A
## Any other useful informationM4 - Release 0.7https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/564Azure reindex topic2023-06-13T11:15:23ZZhibin MaiAzure reindex topicThis is follow-up of [Update the re-index topic name to 'reindextopic'](https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure/-/merge_requests/621). Set env. variables of re-index topic in SLB Azure Sa...This is follow-up of [Update the re-index topic name to 'reindextopic'](https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure/-/merge_requests/621). Set env. variables of re-index topic in SLB Azure SaaS deployment.M19 - Release 0.22Zhibin MaiZhibin Mai