File issueshttps://community.opengroup.org/osdu/platform/system/file/-/issues2023-08-07T11:13:22Zhttps://community.opengroup.org/osdu/platform/system/file/-/issues/87Apply role-based access to File V2 endpoints.2023-08-07T11:13:22ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comApply role-based access to File V2 endpoints.File V2/DMS API doesn't use Authorization filters (@PreAuthorize), and doesn't evaluate the roles of the requester, which could lead to data leaks.
Also, it was marked as Hidden but this rule was not applied to the Infra level automatic...File V2/DMS API doesn't use Authorization filters (@PreAuthorize), and doesn't evaluate the roles of the requester, which could lead to data leaks.
Also, it was marked as Hidden but this rule was not applied to the Infra level automatically.
https://community.opengroup.org/osdu/platform/system/file/-/blob/master/file-core/src/main/java/org/opengroup/osdu/file/api/FileDmsApi.java#L57
Potential issues:
- If not closed from Istio, data leaks are possible.
- Even if closed from the outside, authorization of internal requests will not be evaluated.M19 - Release 0.22Oleksandr Kosse (EPAM)Riabokon Stanislav(EPAM)[GCP]Andrei Dalhikh [EPAM/GC]Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/file/-/issues/90Fixing sonar quality issues2023-07-28T09:39:01ZGauri ChitaleFixing sonar quality issuesSonarqube analysis has reported multiple code smells in the file service code.Sonarqube analysis has reported multiple code smells in the file service code.https://community.opengroup.org/osdu/platform/system/file/-/issues/88[ADR] Dataset service security enhancments2023-07-10T10:43:58ZOm Prakash Gupta[ADR] Dataset service security enhancments# Decision Title
Security Enhancements for Dataset Service's Signed URL APIs
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
A customer has voiced a security concern about Fi...# Decision Title
Security Enhancements for Dataset Service's Signed URL APIs
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
A customer has voiced a security concern about File Service's `POST GetStorageInstructions` and `POST GetReterievalInstructions` APIs under the scenario of a malicious user getting hold of the generated signed URLs and using them to access files from storage. When Private Link is not a desired option to mitigate these concerns for the customer due to policy and deployment complexity reasons, the following enhancements are proposed to the two existing APIs and introducing a new API to alleviate the customer's security concerns.
## Decision
### Proposed Changes
1. For `POSTS GetStorageInstructions` API: Change default TTL from 7 days to 1 hour and make TTL configurable through a query Paramater `expiryTime` in Time Units Minutes,Hours,Days. The expiry time is capped at 7 Days if the time provided by the User exceeds the capped value. In absence of this parameter, the Signed URL would be valid for 1 Hour by default.
2. For `POST GetReterievalInstructions` API: Change default TTL from 7 days to 1 hour. and make TTL configurable through a query Paramater `expiryTime` in Time Units Minutes,Hours,Days. The expiry time is capped at 7 Days if the time provided by the User exceeds the capped value.
These two changes make the two APIs behave consistently also.
3. New API to revoke all Signed URLs generated for a specified storage account. Storage account is specified through a query parameter `storageAccount`. User can grab the storageAccount from the `GetReterievalInstructions` or `GetStorageInstructions` response.
POST api/Dateset/v1/revokeURLs
This API will use the `StorageAccountRevokeUserDelegationKeys` to revoke all the User Delegation Keys for the storage account and that will revoke all the User Delegation SAS tokens and thus invalidate all the Signed URLs.
4. Start using user-defined delegation keys for storage accounts rather than using storage account keys.
## Rationale
Shortened TTL for the Signed URLs decreases the Window of opportunities for a malicious user to use the Sighed URLs to access any sensitive information; Additional Revoking API provides customers a capability to mitigate the risk at the earliest moment if Signed URL leaking is detected.
## Consequences
**Caution**: SAS token in a Signed URL cannot be individually revoked. This API will revoke all SAS tokens generated and invalidate all signed URLs for that storage account. A user needs to send `GET uploadURL` and `GET downloadURL` requests again to generate new URLs. It should only be used when the customer knows for sure a signed URL has been compromised.
**Caution**: User Delegation Keys are cached by Azure Storage, so there may be a delay between when the user initiates the process of revocation and when an existing user delegation SAS becomes invalid. So after calling `POST revokeURLs`, wait for sometime and verify the compromised URL no longer works before sending `GET uploadURL` and `GET downloadURL` requests again.
These cautions need to be included in the file service open API spec and be communicated to customers clearly.
## Backward Compatibility
This is NOT a breaking change.https://community.opengroup.org/osdu/platform/system/file/-/issues/79File ci cd pipelines do not use file-test-core-bdd with vital test cases.2022-11-04T10:15:34ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comFile ci cd pipelines do not use file-test-core-bdd with vital test cases.There is a BDD tests defined in the File testing module: <br/>
https://community.opengroup.org/osdu/platform/system/file/-/tree/master/testing/file-test-core-bdd <br/>
They get test case updates with new feature introductions, for exampl...There is a BDD tests defined in the File testing module: <br/>
https://community.opengroup.org/osdu/platform/system/file/-/tree/master/testing/file-test-core-bdd <br/>
They get test case updates with new feature introductions, for example: <br/>
https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/138/diffs#d67c53013c6814c8d874d0daf0cffc9179ad1d00 <br/>
But they are not used in cicd pipelines, which left those features not cowered. <br/>
And it looks like because of ignoring them for a long time, tests have some compatibility issues which leads to runtime errors like
~~~
java.lang.NoClassDefFoundError: Could not initialize class io.restassured.RestAssured
at org.opengroup.osdu.file.util.test.RestAssuredClient.<init>(RestAssuredClient.java:30)
at org.opengroup.osdu.file.util.test.HttpClientFactory.getInstance(HttpClientFactory.java:8)
at org.opengroup.osdu.file.stepdefs.FileStepDef_GET.lambda$new$1(FileStepDef_GET.java:76)
~~~
Keeping them ignored may cause issues with feature introduction and verification. <br/>
There are several possible solutions: <br/>
- Fix and enable file-test-core-bdd tests in the integration step
- Copy missing tests from to file-test-CSP_PROVIDER_MODULERustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/file/-/issues/71File core module Junits are not getting executed2022-06-17T11:26:17ZAbhishek Kumar (SLB)File core module Junits are not getting executedAbhishek Kumar (SLB)Abhishek Kumar (SLB)https://community.opengroup.org/osdu/platform/system/file/-/issues/69File Service: Requests to POST metadata are taking long time2022-08-23T21:03:43ZSachin JaiswalFile Service: Requests to POST metadata are taking long time### Problem Statement
Request to post metadata takes long time when try to calculate the checksum for larger files.
### Solution
We can overcome this problem by reading bytes from the input stream and storing them into the buffer array.### Problem Statement
Request to post metadata takes long time when try to calculate the checksum for larger files.
### Solution
We can overcome this problem by reading bytes from the input stream and storing them into the buffer array.https://community.opengroup.org/osdu/platform/system/file/-/issues/63Preloadfilepath & ExtensionProperties removed from file Metadata API2022-11-28T14:10:32Zivar SoerheimPreloadfilepath & ExtensionProperties removed from file Metadata APIDuring ingestion of file metadata under /files/metadata using POST command the Preloadfilepath and ExtensionProperties are not persisted when returning the record post ingest.
This seems like strange behaviour to me. I would like to ei...During ingestion of file metadata under /files/metadata using POST command the Preloadfilepath and ExtensionProperties are not persisted when returning the record post ingest.
This seems like strange behaviour to me. I would like to either understand why this happens, or extend the file metadata api so these properties are not removed.
This is the workflow:
1. Get Signed URL for upload
2. Upload file using signed URL
3. Upload file metadata using file api (this returns ID of created record and can be searched in storage)
4. Refer to this ID when creating well log record or any other record
The problem with this workflow is that:
- PreloadFilePath and ExtensionProperties are removed from the record during metadata uploadhttps://community.opengroup.org/osdu/platform/system/file/-/issues/59Using Publisher Facade to publish status messages2022-02-24T08:50:26ZTsvetelina IvanovaUsing Publisher Facade to publish status messagesAzure core lib introduces a publisher facade which can be used across services in order to publish messages to message brokers(Service Bus/ Event Grid).It will help to manage and update at a single source instead of each service doing it...Azure core lib introduces a publisher facade which can be used across services in order to publish messages to message brokers(Service Bus/ Event Grid).It will help to manage and update at a single source instead of each service doing it individually.The pub sub configuration can be used to configure publishing for Event grid and Service bus.
Link of related issue:
https://community.opengroup.org/osdu/platform/system/notification/-/issues/25Tsvetelina IvanovaTsvetelina Ivanovahttps://community.opengroup.org/osdu/platform/system/file/-/issues/54Upgrade to Log4J 2.172021-12-21T03:09:43ZDavid Diederichd.diederich@opengroup.orgUpgrade to Log4J 2.17The Apache Foundation released another Log4j2 update, version 2.17, which address a denial of service vulnerability.
This issue tracks progress to upgrade this dependency for this project.The Apache Foundation released another Log4j2 update, version 2.17, which address a denial of service vulnerability.
This issue tracks progress to upgrade this dependency for this project.https://community.opengroup.org/osdu/platform/system/file/-/issues/53Log4J Expedient Updates and Patches2021-12-17T11:10:27ZDavid Diederichd.diederich@opengroup.orgLog4J Expedient Updates and PatchesThis issue associates MRs that were applied to this project quickly to get a patched version ready as soon as possible. The intent is to provide a reference point for later, more thoughtful, analysis.This issue associates MRs that were applied to this project quickly to get a patched version ready as soon as possible. The intent is to provide a reference point for later, more thoughtful, analysis.David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/file/-/issues/52Log4J CVE-2021-442282021-12-17T06:39:56ZTsvetelina IvanovaLog4J CVE-2021-44228Apache Log4j2 <=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters...Apache Log4j2 <=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default. In previous releases (>2.10) this behavior can be mitigated by setting system property "log4j2.formatMsgNoLookups" to “true” or it can be mitigated in prior releases (<2.10) by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-\*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class).Tsvetelina IvanovaTsvetelina Ivanovahttps://community.opengroup.org/osdu/platform/system/file/-/issues/47KeyVault Quota Issue2021-11-23T08:35:00ZTsvetelina IvanovaKeyVault Quota IssueFile Service makes a large number of calls to Key Vault to retrieve storage account name and key.
When connecting to storage the core-lib-azure library should be used to create Blob clients and use the implemented cache for keyvault sec...File Service makes a large number of calls to Key Vault to retrieve storage account name and key.
When connecting to storage the core-lib-azure library should be used to create Blob clients and use the implemented cache for keyvault secrets.Tsvetelina IvanovaTsvetelina Ivanovahttps://community.opengroup.org/osdu/platform/system/file/-/issues/45File integration test are failing2021-10-18T04:53:46Zsachin GuptaFile integration test are failingIT for DMS API is getting failed in our pipeline. please see the attached image.
![image](/uploads/d998012b6f862c595d56593f0d312dba/image.png)IT for DMS API is getting failed in our pipeline. please see the attached image.
![image](/uploads/d998012b6f862c595d56593f0d312dba/image.png)Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/file/-/issues/44Compilation failure in Master2021-09-29T12:53:40ZAbhishek Kumar (SLB)Compilation failure in MasterThere is a compilation issue in the master branch.
@ethiraj : Please assign it to the right person.
Job [#619104](https://community.opengroup.org/osdu/platform/system/file/-/jobs/619104) failed for 384101a9fe1c781d5e2783fce313803344c62...There is a compilation issue in the master branch.
@ethiraj : Please assign it to the right person.
Job [#619104](https://community.opengroup.org/osdu/platform/system/file/-/jobs/619104) failed for 384101a9fe1c781d5e2783fce313803344c6276a:ethiraj krishnamanaiduRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRiabokon Stanislav(EPAM)[GCP]ethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/file/-/issues/42Implement status publishing method in IBM2022-07-22T09:15:22ZParesh BehedeImplement status publishing method in IBMWe have recently implemented code for publishing several status of metadata api endpoint as per the decided in approved ADR https://community.opengroup.org/osdu/platform/system/home/-/issues/80
changes done in file core can be found her...We have recently implemented code for publishing several status of metadata api endpoint as per the decided in approved ADR https://community.opengroup.org/osdu/platform/system/home/-/issues/80
changes done in file core can be found here along with its publish method implementation for Azure provider https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/124
This issue has been created for IBM team to implement that publish method to publish the status events in message queue.Anuj GuptaShaonjingdong sunAnuj Guptahttps://community.opengroup.org/osdu/platform/system/file/-/issues/41Implement status publishing method in AWS2022-09-27T11:24:19ZParesh BehedeImplement status publishing method in AWSWe have recently implemented code for publishing several status of metadata api endpoint as per the decided in approved ADR https://community.opengroup.org/osdu/platform/system/home/-/issues/80
changes done in file core can be found her...We have recently implemented code for publishing several status of metadata api endpoint as per the decided in approved ADR https://community.opengroup.org/osdu/platform/system/home/-/issues/80
changes done in file core can be found here along with its publish method implementation for Azure provider https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/124
This issue has been created for AWS team to implement that publish method to publish the status events in message queue.GregGreghttps://community.opengroup.org/osdu/platform/system/file/-/issues/40Implement status publishing method in GCP2022-09-27T11:24:30ZParesh BehedeImplement status publishing method in GCPWe have recently implemented code for publishing several status of metadata api endpoint as per the decided in approved ADR
changes done in file core can be found here along with its publish method implementation for Azure provider http...We have recently implemented code for publishing several status of metadata api endpoint as per the decided in approved ADR
changes done in file core can be found here along with its publish method implementation for Azure provider https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/124
This issue has been created for GCP team to implement that publish method to publish the status events in message queue.Kateryna Kurach (EPAM)Kateryna Kurach (EPAM)https://community.opengroup.org/osdu/platform/system/file/-/issues/36Request for HTTP header Content-Disposition (optionally) for file download2022-09-27T11:33:23ZSteven ReynoldsRequest for HTTP header Content-Disposition (optionally) for file downloadWe implemented this Download feature in our application, that will return a download URL that points to the OSDU download API, but the download behaviour in browsers is not consistent between csv and las files for instance. Some browsers...We implemented this Download feature in our application, that will return a download URL that points to the OSDU download API, but the download behaviour in browsers is not consistent between csv and las files for instance. Some browsers will actually trigger a download, others will open in a new tab. We would like for the user experience to be consistent regardless of the file type.
We had a look at it and it's up to the the platform to add the following http headers to these downloads http response, so that the browser knows it must initiate a file download :
header("Content-Disposition", "attachment; filename=myfilename.myextension");
Maybe we could also think about an extra parameter to the request to let the server know it must add this header.https://community.opengroup.org/osdu/platform/system/file/-/issues/34Upgrade Core Azure Dependency2022-02-11T22:00:53ZDavid Diederichd.diederich@opengroup.orgUpgrade Core Azure Dependencyhttps://community.opengroup.org/osdu/platform/system/file/-/issues/33Upgrade Core IBM Dependency2022-02-11T22:01:07ZDavid Diederichd.diederich@opengroup.orgUpgrade Core IBM Dependency