OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2023-01-19T08:04:43Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/10EDS - Keep some identification/flag in created records2023-01-19T08:04:43ZNisha ThakranEDS - Keep some identification/flag in created records* To identify the provider for the ingested records(well,wellbore etc)
* while ingesting Override the Data.Source with Data.SourceOrganisationId (CSRE's Parameters)
* Data.SourceOrganisationId will be a referenced value eg:(opendes:maste...* To identify the provider for the ingested records(well,wellbore etc)
* while ingesting Override the Data.Source with Data.SourceOrganisationId (CSRE's Parameters)
* Data.SourceOrganisationId will be a referenced value eg:(opendes:master-data--Organisation:AWS-PRESHIP:)
https://osdu.aha.io/bookmarks/custom_pivots/7174030888388275981/7190270164658735517M16 - Release 0.19Nisha ThakranNisha Thakranhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/150Misleading log statements2022-12-12T15:35:32ZMaksim MalkovMisleading log statementsWorkflow service search for a triggered workflow first in provided data partition. System workflow like CSV would not be available in data partition. In such cases service publish logs "workflow not found"
Next same workflow is searched ...Workflow service search for a triggered workflow first in provided data partition. System workflow like CSV would not be available in data partition. In such cases service publish logs "workflow not found"
Next same workflow is searched in system db and it is found there and processing completes
But these logs are creating a confusion that some workflow is not found by workflow service, but actually there is no such issue.M16 - Release 0.19https://community.opengroup.org/osdu/platform/system/reference/schema-upgrade/-/issues/2Roll back feature for JSON upgrade2023-03-09T18:43:47ZVikas Hoode [BP]vikas.hoode@bp.comRoll back feature for JSON upgraderequired to design a roll back plan for JSON migration.required to design a roll back plan for JSON migration.M16 - Release 0.19Vikas Hoode [BP]vikas.hoode@bp.comVikas Hoode [BP]vikas.hoode@bp.com2022-12-19https://community.opengroup.org/osdu/platform/system/reference/schema-upgrade/-/issues/1Multi threaded processing of JSON records2023-03-09T18:43:52ZVikas Hoode [BP]vikas.hoode@bp.comMulti threaded processing of JSON recordsJSOn migration is now a sequential process. To speed up process execution we need a solution via a multi threaded program.JSOn migration is now a sequential process. To speed up process execution we need a solution via a multi threaded program.M16 - Release 0.19Vikas Hoode [BP]vikas.hoode@bp.comVikas Hoode [BP]vikas.hoode@bp.com2022-12-16https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/15ADR: Static code analysis for Python libraries2023-10-30T14:43:02ZYan Sushchynski (EPAM)ADR: Static code analysis for Python libraries## Context
Python is a dynamically typed language, so developers don't need to worry about types. This works well if a project is small and a few developers work on it.
However, once the project gets bigger, and involves a lot of engi...## Context
Python is a dynamically typed language, so developers don't need to worry about types. This works well if a project is small and a few developers work on it.
However, once the project gets bigger, and involves a lot of engineers, understanding how code works becomes the cornerstone of the further development. Python has type annotations designed to help developers to understand code. Now, these type annotations in our Python libraries are kind of hints for developers and their IDEs, but following them is not mandatory, and they can be simply ignored.
As a result, we face issues when some methods are called with arguments with wrong types. And these bugs unexpectedly show in runtime under certain conditions.
It is not so rare to get the following runtime error: `AttributeError: 'dict' object has no attribute 'to_JSON'`
However, these bugs could be easily catch with any static analyzer.
## Decision
Add a static analysis step for type checking to CI/CD pipelines right before unit-tests. The step will be run on the container with preinstalled tools for Python static analysis (e.g., [pytype](https://github.com/google/pytype) or [mypy](https://github.com/python/mypy)).
At first, we are going to add static analysis to the following libraries:
1. https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/tree/master/osdu_api - excluding CSP-specific code from `osdu_api/providers`;
1. https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib;
1. https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib.
Further, we can cover other Python libraries with static analysis.
## Consequences
Pros:
1. It will be much easier to catch subtle bugs without writing extra unit-tests;
2. Developers will be forced to follow type annotations that will make code more readable and understandable.
Cons:
1. The existing code should be refactored to pass static analysis validations;
2. Some developers might find obeying these new rules too strict.M16 - Release 0.19https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/113ADR: API to query Entitlement groups2022-12-13T00:08:54ZMandar KulkarniADR: API to query Entitlement groupsNew query parameter to query the Entitlement groups within a data Partition
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
The GET groups API as of now gives the groups that ...New query parameter to query the Entitlement groups within a data Partition
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
The GET groups API as of now gives the groups that user is part of.
The new GET groups/all API as per approved [ADR](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/93) will provide all the groups within the data partition.
The proposal is to provide a **new** query parameter in the GET groups/all API to get groups containing search term provided by the user.
## Tradeoff Analysis
As of now there is no feature available on Entitlement groups to query the groups based on a string. So the consumers have to get ALL the groups from the partition and then filter the groups on their own.
A data partition may have thousands of groups at a time and a consumer application may be interested only in a subset of the groups and not all.
Updating the GET /groups/all API to accept a query parameter to filter the entitlement groups will help such consumer applications.
## Decision
We can add a new **optional** query parameter called as 'query' to GET /groups/all API which accepts query string from the user. The API will return the Entitlement groups from the data partition which contain the string passed by user in the group name.
The new parameter 'query' will be in addition to the query parameters 'cursor', 'limit' and 'type' proposed in an already approved [ADR](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/93).
For example if user passes 'query' as 'tenant1', then the Entitlement groups containing tenant1 in the group name will be returned by this API.
```
GET <env-bas-url>/api/entitlements/v2/groups/all?query=tenant1"
```
Response of the API will be the list of Group details.
```
{
"groups": [
{
"name": "users.tenant1-example.viewers,",
"email": "users.tenant1-example.viewers@{datapartition}.{domain}.com,",
"description": "This is an user group for example viewers of tenant1."
},
{
"name": "users.tenant1-default.viewers,",
"email": "users.tenant1-default.viewers@{datapartition}.{domain}.com,",
"description": "This is an user group for default viewers of tenant1."
}
]
}
```
## Consequences
- Entitlements service documentation needs to be updated.M16 - Release 0.19https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/issues/3ADR : New Airflow operator (and respective DAG) to get information about Airf...2022-12-09T10:13:37ZYan Sushchynski (EPAM)ADR : New Airflow operator (and respective DAG) to get information about Airflow environment**Context:**
===
If Airflow environments are created not with fully managed services (e.g., Google Composer or Amazon MWAA) it is difficult to check what versions of `osdu-airflow-lib`, `osdu-api`, `osdu-ingestion` and other libs are use...**Context:**
===
If Airflow environments are created not with fully managed services (e.g., Google Composer or Amazon MWAA) it is difficult to check what versions of `osdu-airflow-lib`, `osdu-api`, `osdu-ingestion` and other libs are used. For the instance, if Airflow is deployed in K8S the only way to check the versions is to go into running pods and call `pip list`.
Also, `Airflow variables` might be set as environmental variables, and they don't appear in Web UI that makes it hard to debug.
**Proposal:**
===
The proposal is to create a new Airflow operator and the DAG that will return the environmental information of Airflow instances.
- The DAG should be triggered with a system workflow (ADR: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/118)
- The operator should return Airflow variables and the versions of `osdu-*` libs.M16 - Release 0.19https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/26EDS M17 Features and Fixes details2023-05-11T05:56:21ZPriyanka BhongadeEDS M17 Features and Fixes detailsThe significant features and fixes of EDS M17 are listed below:
Features:
1. PasswordCredentials OAuth Flow Type has been introduced, which allows EDS M17 to generate an access token for data providers using this flow type for authoriz...The significant features and fixes of EDS M17 are listed below:
Features:
1. PasswordCredentials OAuth Flow Type has been introduced, which allows EDS M17 to generate an access token for data providers using this flow type for authorization. To generate the access token, the parameters required are username, password, client ID, client secret, and scopes. The "FlowTypeID": "{{data_partition_id}}:reference-data--OAuth2FlowType:PasswordCredentials:" is added to the ConnectedSourceRegistryEntry record.https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/home/-/issues/267
2. EDS M17 now validates the expiry of the refresh token and auto-generates a new refresh token while updating the secret vault. If the refresh token value in the secret vault is expired, the eds_ingest fails to generate an access token, and the run fails. To handle this situation, eds_ingest verifies if the refresh token is expired and generates a new refresh token value following PasswordCredentials authentication grant type. The secret service then accesses the new refresh token value to update the old/expired value with the newly generated refresh token value. The "FlowTypeID": "{{data_partition_id}}:reference-data--OAuth2FlowType:RefreshTokenKeyName:" is added to the ConnectedSourceRegistryEntry (CSRE). The data provider for this feature is Katalyst. https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/19
3. Parent data mapping is now handled in EDS M17, which includes keeping the source identifier ("id" of the parent data) in NameAlias of the parent record during ingestion into the operator environment. This helps the operator to find the source of each record and group them. When ingesting child data (e.g., Well log data) into the target environment, the child data is tagged to the right master data (e.g., Wellbore) in the target environment, and there is no name mismatch. This feature helps to identify a unique well using external rules between the external source and the target environment. https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/home/-/issues/268
Fixes:
A logger has been added to detail the Osdu_ingest run id and the sample-fetched data record. The message displayed in eds_ingest Airflow Logs includes Osdu_ingest Run Id and one Sample data fetched from the data provider with the text "Displaying only one Sample Record." https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/23
The conversion of ConnectedSourceDataPartitionID to OnIngestionDataPartitionID for Array Datatype has been fixed. While ingestion, ConnectedSourceDataPartitionID (provider’s data partition id) is replaced with the OnIngestionDataPartitionID (operator’s data partition id) for all the parameters of the record with different datatypes (arrays, dicts). Each conversion is handled differently based on its datatype. For example, the conversion of string parameters from 'ResourceHomeRegionID': 'osdu:reference-data--OSDURegion:AWSEastUSA:' to 'ResourceHomeRegionID': 'opendes:reference-data--OSDURegion:AWSEastUSA:' is done similarly to the conversion of the array datatype. https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/home/-/issues/261
The Dynamic Schema Authority for Kind of CSRE, CSDJ, and ExternalReferenceValueMapping is now added from Airflow Variable. The constant file has Kind of few eds dependent schemas, such as ConnectedSourceRegistryEntry, ConnectedSourceDataJob, and ExternalReferenceValueMapping. The Schema_Authority value was static in the Kind, which is now replaced with the Schema_authority value fetched from the Airflow Variable. https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/22
EDS now raises an exception when Airflow Variable is not found or None. Eds_ingest fails with KeyError if any of the important Airflow variable values are missing. https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/21M17 - Release 0.20Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/25EDS - Include Password Credentials OAuth Flow Type2023-05-04T08:33:42ZPriyanka BhongadeEDS - Include Password Credentials OAuth Flow Type- [x] Identify the changes
- [x] create a function/POC to handle Password Credentials OAuth Flow Type
- [x] create a unit test case
- [x] Test the functionality
- [x] code review- [x] Identify the changes
- [x] create a function/POC to handle Password Credentials OAuth Flow Type
- [x] create a unit test case
- [x] Test the functionality
- [x] code reviewM17 - Release 0.20Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/22EDS - Adding Dynamic Schema Authority for Kind of CSRE ,CSDJ and ExternalRefe...2023-05-04T10:42:00ZPriyanka BhongadeEDS - Adding Dynamic Schema Authority for Kind of CSRE ,CSDJ and ExternalReferenceValueMapping from Airflow VariableM17 - Release 0.20Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/21EDS - Raise exception when Airflow Variable not found or None2023-05-04T10:42:00ZPriyanka BhongadeEDS - Raise exception when Airflow Variable not found or NoneTo raise exception when Airflow Variable are not found or None value for the eds ingest and eds SchedulerTo raise exception when Airflow Variable are not found or None value for the eds ingest and eds SchedulerM17 - Release 0.20Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/19EDS - Auto generate Refresh Token for RefreshTokenKeyName OAuth Flow type whe...2023-05-04T11:38:41ZPriyanka BhongadeEDS - Auto generate Refresh Token for RefreshTokenKeyName OAuth Flow type when expired- [x] Identify the changes
- [x] create a function/POC to handle Auto generation of Refresh Token for RefreshTokenKeyName OAuth Flow type when expired
- [x] create a unit test case
- [x] Test the functionality
- [x] code review- [x] Identify the changes
- [x] create a function/POC to handle Auto generation of Refresh Token for RefreshTokenKeyName OAuth Flow type when expired
- [x] create a unit test case
- [x] Test the functionality
- [x] code reviewM17 - Release 0.20Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/18EDS - Adding Logger to give details about Osdu_ingest run id and Sample fetch...2023-03-20T13:36:39ZPriyanka BhongadeEDS - Adding Logger to give details about Osdu_ingest run id and Sample fetched data record_ Add Logger to display Osdu_ingest run id in below format
Osdu_ingest runId=xxxx
- Correction in logger while dsplaying sample fetched data record
currently logger has " Record 1 :"
To make the message more clearer , changing the disp..._ Add Logger to display Osdu_ingest run id in below format
Osdu_ingest runId=xxxx
- Correction in logger while dsplaying sample fetched data record
currently logger has " Record 1 :"
To make the message more clearer , changing the display message in logs as "Displaying only one Sample Record"M17 - Release 0.20Nisha ThakranPriyanka BhongadeNisha Thakranhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/166Need example of how to use the POST /query/records:batch Fetch multiple rec...2023-04-20T03:00:55ZKamlesh TodaiNeed example of how to use the POST /query/records:batch Fetch multiple recordsThe Storage API documentation mention about
POST /query/records/batch Fetch multiple records. Would like to get the sample of how is this feature expected to be used.
Need clarification on
Account ID is the active OSDU account (OSDU ...The Storage API documentation mention about
POST /query/records/batch Fetch multiple records. Would like to get the sample of how is this feature expected to be used.
Need clarification on
Account ID is the active OSDU account (OSDU account or customer's account) which the users choose to use with the Search API.
frame-of-reference: This value indicates whether normalization applies, should be either 'none' or 'units=SI;crs=wgs84;elevation=msl;azimuth=true north;dates=utc;'
@chad @debasiscM17 - Release 0.20https://community.opengroup.org/osdu/platform/system/search-service/-/issues/123Search service does not ignore unmapped fields (records without spatial attri...2023-03-13T11:02:45ZAn NgoSearch service does not ignore unmapped fields (records without spatial attributes are returned regardless)The following request returns all records in that kinds I can access, but none of them actually has SpatialLocation attribute.
```
curl --location '<baseUrl>/search/v2/query' \
--header 'data-partition-id: partitionID' \
--header 'Auth...The following request returns all records in that kinds I can access, but none of them actually has SpatialLocation attribute.
```
curl --location '<baseUrl>/search/v2/query' \
--header 'data-partition-id: partitionID' \
--header 'Authorization: Bearer ' \
--header 'Content-Type: application/json' \
--data '{
"kind": "osdu:test:Hello:1.0.0",
"query": "*",
"spatialFilter": {
"field": "data.SpatialLocation.Wgs84Coordinates",
"byIntersection": {
"polygons": [
{
"points": [
{
"longitude": -180,
"latitude": 90
},
{
"longitude": 180,
"latitude": 90
},
{
"longitude": 180,
"latitude": -90
},
{
"longitude": -180,
"latitude": -90
},
{
"longitude": -180,
"latitude": 90
}
]
}
]
}
}
}'
```
However, the following request returns 0 record which is expected.
```
curl --location '<baseUrl>/search/v2/query' \
--header 'data-partition-id: partitionID' \
--header 'Authorization: Bearer ' \
--header 'Content-Type: application/json' \
--data '{
"kind": "osdu:test:Hello:1.0.0",
"query": "_exists_:data.SpatialLocation"
}'
```
**Fix**: Ignore Unmapped fields in Elastic SearchM17 - Release 0.20Neelesh ThakurNeelesh Thakurhttps://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/132Project Vulnerability Scanning: osdu/platform/data-flow/data-loading/osdu-cli2023-04-05T13:20:19Zdesman boldenProject Vulnerability Scanning: osdu/platform/data-flow/data-loading/osdu-cli**Why did I receive this?**
In efforts to increase security on the OSDU platform we must ensure all projects containing source code are being scanned on a regular basis. You are receiving this notification because you have been identifi...**Why did I receive this?**
In efforts to increase security on the OSDU platform we must ensure all projects containing source code are being scanned on a regular basis. You are receiving this notification because you have been identified as an owner of a project in Gitlab that isn't being scanned for vulnerabilities.
**What do I need to do?**
Please include gitlab-ultimate.yml (https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/blob/master/scanners/gitlab-ultimate.yml) to your project so it can be scanned for vulnerabilities.
**Project(s) in Scope:**
osdu/platform/data-flow/data-loading/osdu-cliM17 - Release 0.20Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/system/storage/-/issues/160ADR - Clean OpenAPI 3.0 Documentation using 'Code First Approach'2023-07-10T08:02:52ZOm Prakash GuptaADR - Clean OpenAPI 3.0 Documentation using 'Code First Approach'## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
While adopting **OpenAPI 3.0** standards using `springdoc`, we end up adding lot of documentation to native controller of each AP...## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
While adopting **OpenAPI 3.0** standards using `springdoc`, we end up adding lot of documentation to native controller of each API.
- API contract is not clearly visible
- reduces the readability of the API
- business logic & documentation at the same place
## Tradeoff Analysis
- To maintain clean API documentation
- API, Controller segregation
- adopt future changes w.r.t to documentation or contract change
## Proposed Solution:
- Introduce API, Controller Layer Segregation
- API will have contract, definitions & OpenAPI documentation
- Controller will implement the API contract with clean code
#References:
1. [‘Code First’ API Documentation](https://reflectoring.io/spring-boot-springdoc/)
## Sample Refactor in Storage Patch API
- [Patch API](https://community.opengroup.org/osdu/platform/system/storage/-/blob/az/td-codefirst/storage-core/src/main/java/org/opengroup/osdu/storage/api/PatchApi.java)
- [Patch Controller](https://community.opengroup.org/osdu/platform/system/storage/-/blob/az/td-codefirst/storage-core/src/main/java/org/opengroup/osdu/storage/api/PatchController.java)
## Sample Example code
Lets consider a TODO API with normal Crud operation
First we write Interface and define necessary annotations.
```
@RequestMapping("/api/todos")
@Tag(name = "Todo API", description = "euismod in pellentesque ...")
interface TodoApi {
@GetMapping
@ResponseStatus(code = HttpStatus.OK)
List<Todo> findAll();
@GetMapping("/{id}")
@ResponseStatus(code = HttpStatus.OK)
Todo findById(@PathVariable String id);
@PostMapping
@ResponseStatus(code = HttpStatus.CREATED)
Todo save(@RequestBody Todo todo);
@PutMapping("/{id}")
@ResponseStatus(code = HttpStatus.OK)
Todo update(@PathVariable String id, @RequestBody Todo todo);
@DeleteMapping("/{id}")
@ResponseStatus(code = HttpStatus.NO_CONTENT)
void delete(@PathVariable String id);
}
```
##
Then we derive existing controllers from interface for controller implementation
```
@RestController
class TodoController implements TodoApi {
// method implementations
}
```
## Consequences
- Requires changes across services and code refactoring.
- No Breaking functional changes.M17 - Release 0.20Chad LeongOm Prakash GuptaChad Leonghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/64Refactor DAG related code2023-04-04T10:49:00ZYan Sushchynski (EPAM)Refactor DAG related code### Introduction
There is DAG related code that is executed in the container during a DAG run. The code is [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src...### Introduction
There is DAG related code that is executed in the container during a DAG run. The code is [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src/witsml_parser/main.py) and [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src/witsml_parser/energistics/libs/create_energistics_manifest.py). And this code looks messy and outdated, and requires some refactoring.
### What should be done?
1. Update the code to make it work with the most recent `osdu-*` Python libs. The dependencies are here https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/build/requirements.txt
2. Delete deprecated functionality of processing files by `preload_file_path` [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src/witsml_parser/energistics/libs/create_energistics_manifest.py#L314).
3. Add the static-analysis step in the CI/CD.
4. Add possibility to pass the user's access/id token to the DAG
5. Common refactoring, because the code is messy now (a lot of "ifs" and lines of code in a single function)M17 - Release 0.20Vadzim Kulybaharshit aggarwalWalter Detienne peyssonMarc Burnie [AWS]Vadzim Kulybahttps://community.opengroup.org/osdu/platform/system/search-service/-/issues/113Enhancement for a new way to apply Search service policy rules - metadata res...2023-02-17T09:11:57ZDadong ZhouEnhancement for a new way to apply Search service policy rules - metadata restrictionFrom Policy side requirement: https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/64
From recent internal meetings/discussions, we learnt a new requirement: we would like to use the policy rules to cont...From Policy side requirement: https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/64
From recent internal meetings/discussions, we learnt a new requirement: we would like to use the policy rules to control what metadata fields the Search service will return back to the user based on the data record legal tags and the login user attributes (ie country etc). In this scenario, the user may not have access to the data record (permission controlled by the Storage policy rules) but will allow the user to search for the data record with limited visibility of the metadata. See if this is possible for a future enhancement.M17 - Release 0.20https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/118[Azure] Unnecessary cronjob increasing latency2023-02-20T17:07:15ZThiago Senador[Azure] Unnecessary cronjob increasing latencyWe have a [cron job](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/blob/master/provider/entitlements-v2-azure/src/main/java/org/opengroup/osdu/entitlements/v2/azure/service/PartitionCacheTtlService....We have a [cron job](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/blob/master/provider/entitlements-v2-azure/src/main/java/org/opengroup/osdu/entitlements/v2/azure/service/PartitionCacheTtlService.java#L95) running every 5 minutes just to update some ttl values from Partition service. Since we don't need to update such values at runtime this cronjob is unnecessary, and more than that, we strongly believe it is causing a significant multi-threading overhead to the service: after analyzing some entitlements requests extremely slow (latency > 1 minute), we realized some threads hanging around the same time the cronjob executes. As an empirical analysis, I removed the cronjob and its related config/annotations. After a couple of days running the entitlements version without the cronjob I noticed improvements in 99 request latency percentile as well as the absence of those lengthy requests.
FYI, some useful references related to the issue.
- the [`@EnableAsync`](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/blob/master/provider/entitlements-v2-azure/src/main/java/org/opengroup/osdu/entitlements/v2/azure/EntitlementsV2Application.java#L16) tag is misplaced. It should be used in a [`@Configuration`](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/scheduling/annotation/EnableAsync.html) class and not in an application level.
- the [`@EnableScheduling`](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/blob/master/provider/entitlements-v2-azure/src/main/java/org/opengroup/osdu/entitlements/v2/azure/EntitlementsV2Application.java#L17) tag is misplaced. It should be uses as localized as possible (method level) and not at application level.
Can we completely eliminate this cronjob from Azure deployment?M17 - Release 0.20Chad LeongChad Leong