File issueshttps://community.opengroup.org/osdu/platform/system/file/-/issues2023-11-29T09:58:29Zhttps://community.opengroup.org/osdu/platform/system/file/-/issues/78ADR: Security Enhancements for File Service's Signed URL APIs2023-11-29T09:58:29ZLucy LiuADR: Security Enhancements for File Service's Signed URL APIs
# Decision Title
Security Enhancements for File Service's Signed URL APIs
## Status
- [X] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
A customer has voiced a security concern about File...
# Decision Title
Security Enhancements for File Service's Signed URL APIs
## Status
- [X] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
A customer has voiced a security concern about File Service's `GET uploadURL` and `GET downloadURL` APIs under the scenario of a malicious user getting hold of the generated signed URLs and using them to access files from storage. When Private Link is not a desired option to mitigate these concerns for the customer due to policy and deployment complexity reasons, the following enhancements are proposed to the two existing APIs and introducing a new API to alleviate the customer's security concerns.
## Decision
### Proposed Changes
1. For `GET uploadURL` API: Change default TTL from 7 days to 1 hour and make TTL configurable through a query Paramater `expiryTime` in Time Units Minutes,Hours,Days. The expiry time is capped at 7 Days if the time provided by the User exceeds the capped value. In absence of this parameter, the Signed URL would be valid for 1 Hour by default.
2. For `GET downloadURL` API: Change default TTL from 7 days to 1 hour. TTL is already configurable through the query Paramater `expiryTime`.
These two changes make the two APIs behave consistently also.
3. New API to revoke all Signed URLs generated for a specified storage account. Storage account is specified through a query parameter `storageAccount`. User can grab the storageAccount from the `GET uploadURL` or `GET downloadURL` response.
POST api/file/v2/files/revokeURLs
This API will use the `StorageAccountRevokeUserDelegationKeys` to revoke all the User Delegation Keys for the storage account and that will revoke all the User Delegation SAS tokens and thus invalidate all the Signed URLs.
## Rationale
Shortened TTL for the Signed URLs decreases the Window of opportunities for a malicious user to use the Sighed URLs to access any sensitive information; Additional Revoking API provides customers a capability to mitigate the risk at the earliest moment if Signed URL leaking is detected.
## Consequences
**Caution**: SAS token in a Signed URL cannot be individually revoked. This API will revoke all SAS tokens generated and invalidate all signed URLs for that storage account. A user needs to send `GET uploadURL` and `GET downloadURL` requests again to generate new URLs. It should only be used when the customer knows for sure a signed URL has been compromised.
**Caution**: User Delegation Keys are cached by Azure Storage, so there may be a delay between when the user initiates the process of revocation and when an existing user delegation SAS becomes invalid. So after calling `POST revokeURLs`, wait for sometime and verify the compromised URL no longer works before sending `GET uploadURL` and `GET downloadURL` requests again.
These cautions need to be included in the file service open API spec and be communicated to customers clearly.
## Backward Compatibility
This is NOT a breaking change.M18 - Release 0.21Om Prakash GuptaOm Prakash Guptahttps://community.opengroup.org/osdu/platform/system/file/-/issues/69File Service: Requests to POST metadata are taking long time2022-08-23T21:03:43ZSachin JaiswalFile Service: Requests to POST metadata are taking long time### Problem Statement
Request to post metadata takes long time when try to calculate the checksum for larger files.
### Solution
We can overcome this problem by reading bytes from the input stream and storing them into the buffer array.### Problem Statement
Request to post metadata takes long time when try to calculate the checksum for larger files.
### Solution
We can overcome this problem by reading bytes from the input stream and storing them into the buffer array.https://community.opengroup.org/osdu/platform/system/file/-/issues/37GET /DownloadURL api to support generating signedURL with custom expiry time2021-08-27T21:29:39ZOrsu AkhilGET /DownloadURL api to support generating signedURL with custom expiry timeCurrently in cloud provider implementation of File Application for Azure , GCP the /DownloadURL api would return signedURL to access file stored in persistent location and the expiry time of this URL is by default set to 7 Days. This may...Currently in cloud provider implementation of File Application for Azure , GCP the /DownloadURL api would return signedURL to access file stored in persistent location and the expiry time of this URL is by default set to 7 Days. This mayn't be desirable and giving flexibility to User to determine when the URL should expire would be a good feature add-on for this api. This information can be passed by User via GET query Parameter in the api specification.M8 - Release 0.11Orsu AkhilParesh BehedeOrsu Akhilhttps://community.opengroup.org/osdu/platform/system/file/-/issues/34Upgrade Core Azure Dependency2022-02-11T22:00:53ZDavid Diederichd.diederich@opengroup.orgUpgrade Core Azure Dependencyhttps://community.opengroup.org/osdu/platform/system/file/-/issues/29ADR: Rationalizing File and File DMS services.2023-08-08T11:23:16ZKrishna Nikhil VedurumudiADR: Rationalizing File and File DMS services.# Decision Title
Rationalizing File and File DMS services.
## Status
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [x] Approved
* [ ] Retired
## Context & Scope
The Dataset service in its implementation of the DMS APIs (getStor...# Decision Title
Rationalizing File and File DMS services.
## Status
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [x] Approved
* [ ] Retired
## Context & Scope
The Dataset service in its implementation of the DMS APIs (getStorageInstructions and getRetrievalInstructions) delegates the call to specific DMS's implementation of the `get*Instructons` API. That lead to creation of new service called `File-DMS` which supports the APIs that Dataset Service requires.
> Note: The File-DMS does not have an approved ADR, and is not an approved by the community yet.
On the other hand, we already have a File Service that owns the responsibility for the management of files. Also, the design and APIs of the File service are approved by a thorough process of ADR.
So, instead of having two services who own the similar responsibility of managing files (File and File-DMS) we should rationalize both the services and move the APIs that Dataset Service requires to File Service itself. It would reduce the additional maintenance overhead of managing another service.
## Decision
The decision is to merge File and File DMS services and consolidate all the File management APIs in **File Service** keeping inline with the File Service ADR that was approved [File Service ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/47).
## Rationale
Having multiple services with similar functionalities and responsibilities is an additional overhead w.r.t maintenance. Since there is already one approved service for supporting Files, **the Additional APIs on Files required to support DMS Functionalities** should be hosted by File Service.
## Consequences
> The **DMS APIs** **getStorageInstructions** and **getRetrievalInstructions** for Files and File-Collections will be moved to File Service.
| Functionality | API to be Added in File Service | Existing API in File-DMS Service | Status | Capability |
|---------------------|-------------------------|-------------------------|--------|--------------------------------------------------|
| File Get-Storage Instructions | POST /files/storageInstructions | GET /file/getStorageInstructions | Moved from File-DMS Service to File Service | Generates the Signed URLs / temporary access tokens required to upload Files |
| File Get-Retrieval Instructions | POST /files/retrievalInstructions | POST /file/getRetrievalInstructions | Moved from File-DMS Service to File Service | Generates the Signed URLs / temporary access tokens required to download Files |
| File Collection Get-Storage Instructions | POST /file-collections/storageInstructions | GET /file-collection/getStorageInstructions | Moved from File-DMS Service to File Service | Generates the Signed URLs / temporary access tokens required to upload File Collections |
| File Collection Get-Retrieval Instructions | POST /file-collections/retrievalInstructions | POST /file-collection/getRetrievalInstructions | Moved from File-DMS Service to File Service | Generates the Signed URLs / temporary access tokens required to download File Collections |
### Pros of merging File DMS APIs into File Service
- Existing File Service clients will not be impacted because the existing APIs will continue to stay.
- The Core-Logic of handling File uploads and downloads already exist in File Service and are well tested. So, the DMS APIs that are required for [Dataset ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/57) will be able to directly hook into it reducing the development time.
- Features such as "Uploading to staging container" will be available out of the box. This reduces gaps between Dataset Architecture and existing File Service, which in-turn enables easier migration for clients.JoeMadhur Tanwani [Microsoft]Joehttps://community.opengroup.org/osdu/platform/system/file/-/issues/23Delivery API not working on Azure2021-03-01T21:46:28ZHrvoje MarkovicDelivery API not working on AzureDelivery API of file service returns
`{
"code": 415,
"reason": "Unsupported media type",
"message": "upstream server responded with unsupported media type: text/html"
}` irrespectively if records with provided srns exist or ...Delivery API of file service returns
`{
"code": 415,
"reason": "Unsupported media type",
"message": "upstream server responded with unsupported media type: text/html"
}` irrespectively if records with provided srns exist or not.<br>
The root cause seems to be faulty config for search query API url for Azure. There is a missing "/" in https://community.opengroup.org/osdu/platform/system/file/-/blob/master/provider/file-azure/src/main/resources/application.properties#L93.https://community.opengroup.org/osdu/platform/system/file/-/issues/21Inconsistent URL versioning2021-03-05T09:56:56ZKishore BattulaInconsistent URL versioning# Context & Scope
The following endpoints have both v1/v2 in their path which is not a clear API versioning
- /api/file/v2/v1/files/{id}/downloadURL
- /api/file/v2/v1/files/uploadURL
- /api/file/v2/v1/files/metadata
- /api/file/v2/v1/fi...# Context & Scope
The following endpoints have both v1/v2 in their path which is not a clear API versioning
- /api/file/v2/v1/files/{id}/downloadURL
- /api/file/v2/v1/files/uploadURL
- /api/file/v2/v1/files/metadata
- /api/file/v2/v1/files/{id}/metadata
# Decision
Change the Above APIs respectively
- /api/file/v1/files/{id}/downloadURL
- /api/file/v1/files/uploadURL
- /api/file/v1/files/metadata
- /api/file/v1/files/{id}/metadata
# Rational
Consistent and clear API versioning.
# Consequences
Users of these APIs need to change the paths for above APIs
# Impact
The changes would be in core module and no needed for CSPs to change any codehttps://community.opengroup.org/osdu/platform/system/file/-/issues/18If partitonId == null, a endpoint will return 403 status instead of 401.2020-11-13T12:25:12ZRiabokon Stanislav(EPAM)[GCP]If partitonId == null, a endpoint will return 403 status instead of 401.This problem has been identified when this MR https://community.opengroup.org/osdu/platform/system/file/-/commit/4ddd2e7228252edef7474ecad0c86d31c3c3598e had been merged into master
commit: https://community.opengroup.org/osdu/platform/...This problem has been identified when this MR https://community.opengroup.org/osdu/platform/system/file/-/commit/4ddd2e7228252edef7474ecad0c86d31c3c3598e had been merged into master
commit: https://community.opengroup.org/osdu/platform/system/file/-/commit/4ddd2e7228252edef7474ecad0c86d31c3c3598e#7e5fc265ef88187dd3731e9bcceee01ff99f7569_44_44
A new behavior "If partitonId == null, a endpoint will return 403 status instead of 401"
I am not sure that it is a correct response from an architecture perspective.
cc: @Dmitriy_Rudko @rostislav.dublinRiabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/file/-/issues/15Update naming conventions2020-09-17T17:18:18ZNicholas KarskyUpdate naming conventionsFile deliverable currently comes as a .jar file for provider/file-azure. Update so that the deliverable is a spring-boot.jar like other services. @ethiraj @dkodeihFile deliverable currently comes as a .jar file for provider/file-azure. Update so that the deliverable is a spring-boot.jar like other services. @ethiraj @dkodeihDania Kodeih (Microsoft)Dania Kodeih (Microsoft)https://community.opengroup.org/osdu/platform/system/file/-/issues/13Complete CI/CD and Integration Tests for Azure2020-10-15T19:39:39ZDania Kodeih (Microsoft)Complete CI/CD and Integration Tests for AzureM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/file/-/issues/12Complete Azure File Service CI/CD and Tests2020-09-16T20:28:29ZDania Kodeih (Microsoft)Complete Azure File Service CI/CD and TestsM1 - Release 0.1Dania Kodeih (Microsoft)Dania Kodeih (Microsoft)