File issueshttps://community.opengroup.org/osdu/platform/system/file/-/issues2023-08-08T16:30:43Zhttps://community.opengroup.org/osdu/platform/system/file/-/issues/89Unable to download the file2023-08-08T16:30:43ZSachin JaiswalUnable to download the file**Problem**
File download fails if file name contains the special character like ","
**Steps to reproduce**
- Generate a signedurl to upload a file (tesing,copy).
- Upload a file
- create a file metadata and use same filename (tesing,...**Problem**
File download fails if file name contains the special character like ","
**Steps to reproduce**
- Generate a signedurl to upload a file (tesing,copy).
- Upload a file
- create a file metadata and use same filename (tesing,copy)
- Generate a signedurl to download a file
- Copy and paste the signedurl in browser
**Error notice on bowser** - `<storage-url> sent an invalid response. ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION `
**Proposed Solutions**
- File service - wrap quotes to file name before passing it to OS Core Lib Azure **OR**
- OS Core Lib Azure - Below changes in [BlobStore class](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/blob/master/src/main/java/org/opengroup/osdu/azure/blobstorage/BlobStore.java#:~:text=581-%2cblobServiceSasSignatureValues.setContentDisposition%28%22attachment%3b%20filename%3D%20%22%20%2B%20fileName%29%3b%2c-582) of OS Core Lib Azure repo.
`blobServiceSasSignatureValues.setContentDisposition("attachment; filename=\"" + fileName + "\"")`;https://community.opengroup.org/osdu/platform/system/file/-/issues/86/api/file/v2/files bugs2023-09-26T14:22:03ZShane Hutchins/api/file/v2/files bugsReceived a response with 5xx status code: 500. I expected a 404 from these APIs but got 500 Internal Server Error.
Run this curl command to reproduce this failure:
GET /api/file/v2/files/%0A/metadata:
curl -X GET -H 'Authorization...Received a response with 5xx status code: 500. I expected a 404 from these APIs but got 500 Internal Server Error.
Run this curl command to reproduce this failure:
GET /api/file/v2/files/%0A/metadata:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/file/v2/files/%0A/metadata
DELETE /api/file/v2/files/%3B/metadata:
curl -X DELETE -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/file/v2/files/%3B/metadata
GET /api/file/v2/files/%3B/downloadURL:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/file/v2/files/%3B/downloadURL
Confirmed this issue on AWS and Azure.
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'Cookie: JSESSIONID=SESSIONIDHERE' -H 'data-partition-id: opendes' https://osdu-ship.msft-osdu-test.org/api/file/v2/files/%0A/metadatahttps://community.opengroup.org/osdu/platform/system/file/-/issues/85Duplicate PreloadFilePath value on ingestion of File.Generics2023-06-16T13:17:04ZGorm-Erik AarsheimDuplicate PreloadFilePath value on ingestion of File.GenericsWhen ingesting File.Generics into OSDU, we are using the {OSDU_BASE_URL}/file/v2/files/metadata endpoint to create the File.Generic. In the body we are setting the PreloadFilePath value on data.DatasetProperties.FileSourceInfo.PreloadFil...When ingesting File.Generics into OSDU, we are using the {OSDU_BASE_URL}/file/v2/files/metadata endpoint to create the File.Generic. In the body we are setting the PreloadFilePath value on data.DatasetProperties.FileSourceInfo.PreloadFilePath as described in the schema. The file is created successfully but when we read out the record from Storage API there is a duplicate value for PreloadFilePath where it has both 'PreloadFilePath' with a capital 'P', and 'preloadFilePath' with a non-capital 'P'. We've tried to ingest a file with a non-capital 'preloadFilePath' as well, but it still comes out with both values when reading it back. We have also tried generating the File.Generic raw json object from our code and using this to create it directly from the metadata endpoint in Postman to disclose that our code is somehow meddeling with the object. This also comes out with the same result. It's also worth mentioning that we are running OSDU through Microsoft ADME.Om Prakash GuptaOm Prakash Guptahttps://community.opengroup.org/osdu/platform/system/file/-/issues/78ADR: Security Enhancements for File Service's Signed URL APIs2023-11-29T09:58:29ZLucy LiuADR: Security Enhancements for File Service's Signed URL APIs
# Decision Title
Security Enhancements for File Service's Signed URL APIs
## Status
- [X] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
A customer has voiced a security concern about File...
# Decision Title
Security Enhancements for File Service's Signed URL APIs
## Status
- [X] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
A customer has voiced a security concern about File Service's `GET uploadURL` and `GET downloadURL` APIs under the scenario of a malicious user getting hold of the generated signed URLs and using them to access files from storage. When Private Link is not a desired option to mitigate these concerns for the customer due to policy and deployment complexity reasons, the following enhancements are proposed to the two existing APIs and introducing a new API to alleviate the customer's security concerns.
## Decision
### Proposed Changes
1. For `GET uploadURL` API: Change default TTL from 7 days to 1 hour and make TTL configurable through a query Paramater `expiryTime` in Time Units Minutes,Hours,Days. The expiry time is capped at 7 Days if the time provided by the User exceeds the capped value. In absence of this parameter, the Signed URL would be valid for 1 Hour by default.
2. For `GET downloadURL` API: Change default TTL from 7 days to 1 hour. TTL is already configurable through the query Paramater `expiryTime`.
These two changes make the two APIs behave consistently also.
3. New API to revoke all Signed URLs generated for a specified storage account. Storage account is specified through a query parameter `storageAccount`. User can grab the storageAccount from the `GET uploadURL` or `GET downloadURL` response.
POST api/file/v2/files/revokeURLs
This API will use the `StorageAccountRevokeUserDelegationKeys` to revoke all the User Delegation Keys for the storage account and that will revoke all the User Delegation SAS tokens and thus invalidate all the Signed URLs.
## Rationale
Shortened TTL for the Signed URLs decreases the Window of opportunities for a malicious user to use the Sighed URLs to access any sensitive information; Additional Revoking API provides customers a capability to mitigate the risk at the earliest moment if Signed URL leaking is detected.
## Consequences
**Caution**: SAS token in a Signed URL cannot be individually revoked. This API will revoke all SAS tokens generated and invalidate all signed URLs for that storage account. A user needs to send `GET uploadURL` and `GET downloadURL` requests again to generate new URLs. It should only be used when the customer knows for sure a signed URL has been compromised.
**Caution**: User Delegation Keys are cached by Azure Storage, so there may be a delay between when the user initiates the process of revocation and when an existing user delegation SAS becomes invalid. So after calling `POST revokeURLs`, wait for sometime and verify the compromised URL no longer works before sending `GET uploadURL` and `GET downloadURL` requests again.
These cautions need to be included in the file service open API spec and be communicated to customers clearly.
## Backward Compatibility
This is NOT a breaking change.M18 - Release 0.21Om Prakash GuptaOm Prakash Guptahttps://community.opengroup.org/osdu/platform/system/file/-/issues/69File Service: Requests to POST metadata are taking long time2022-08-23T21:03:43ZSachin JaiswalFile Service: Requests to POST metadata are taking long time### Problem Statement
Request to post metadata takes long time when try to calculate the checksum for larger files.
### Solution
We can overcome this problem by reading bytes from the input stream and storing them into the buffer array.### Problem Statement
Request to post metadata takes long time when try to calculate the checksum for larger files.
### Solution
We can overcome this problem by reading bytes from the input stream and storing them into the buffer array.https://community.opengroup.org/osdu/platform/system/file/-/issues/62POST /metadata endpoint returns 500 error when Blob Not Found2022-09-27T11:52:47ZTsvetelina IvanovaPOST /metadata endpoint returns 500 error when Blob Not FoundPOST /metadata endpoint returns a BlobStorageExcepiton with message "Error occurred while creating file metadata Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does no...POST /metadata endpoint returns a BlobStorageExcepiton with message "Error occurred while creating file metadata Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist._RequestId:f60ebd44-b01e-0012-532e-3236bf000000_Time:2022-03-07T14:23:07.9454432Z</Message></Error>" Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.
RequestId:f60ebd44-b01e-0012-532e-3236bf000000
Time:2022-03-07T14:23:07.9454432Z</Message></Error>"" as a 500 Internal Server Error when calling deleteFile method to delete file from the blob storage.
The file is copied successfully from staging to the persistent area, but the delete of the file from staging fails with "The specified blob does not exist".
For exception logs see the attached file:
[File_Service_Azure_Logs_-_BlobStorageException.csv](/uploads/2cfc5a0baf2616fb456791ce8412babb/File_Service_Azure_Logs_-_BlobStorageException.csv)https://community.opengroup.org/osdu/platform/system/file/-/issues/38GET /DownloadURL api to support generating signedURL that can be accessible f...2022-09-29T13:41:08ZOrsu AkhilGET /DownloadURL api to support generating signedURL that can be accessible from select IP address.The /DownloadURL api would return signedURL to access file stored in persistent location based on File Metadata id .
Having the feature wherein User can specify the range of IP address , only through which when signedURL is used would re...The /DownloadURL api would return signedURL to access file stored in persistent location based on File Metadata id .
Having the feature wherein User can specify the range of IP address , only through which when signedURL is used would result in access to the file in persistent location would be good. This IP address or range of IP address can be passed by User via GET query Parameter in the api specification.
Supporting this sort of customization in generated signed url is possible in Azure Cloud and would like to know about this in other cloud providers , if is feasible in others as well then could come up with implementation supporting this feature.Orsu AkhilParesh BehedeOrsu Akhilhttps://community.opengroup.org/osdu/platform/system/file/-/issues/37GET /DownloadURL api to support generating signedURL with custom expiry time2021-08-27T21:29:39ZOrsu AkhilGET /DownloadURL api to support generating signedURL with custom expiry timeCurrently in cloud provider implementation of File Application for Azure , GCP the /DownloadURL api would return signedURL to access file stored in persistent location and the expiry time of this URL is by default set to 7 Days. This may...Currently in cloud provider implementation of File Application for Azure , GCP the /DownloadURL api would return signedURL to access file stored in persistent location and the expiry time of this URL is by default set to 7 Days. This mayn't be desirable and giving flexibility to User to determine when the URL should expire would be a good feature add-on for this api. This information can be passed by User via GET query Parameter in the api specification.M8 - Release 0.11Orsu AkhilParesh BehedeOrsu Akhilhttps://community.opengroup.org/osdu/platform/system/file/-/issues/34Upgrade Core Azure Dependency2022-02-11T22:00:53ZDavid Diederichd.diederich@opengroup.orgUpgrade Core Azure Dependencyhttps://community.opengroup.org/osdu/platform/system/file/-/issues/29ADR: Rationalizing File and File DMS services.2023-08-08T11:23:16ZKrishna Nikhil VedurumudiADR: Rationalizing File and File DMS services.# Decision Title
Rationalizing File and File DMS services.
## Status
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [x] Approved
* [ ] Retired
## Context & Scope
The Dataset service in its implementation of the DMS APIs (getStor...# Decision Title
Rationalizing File and File DMS services.
## Status
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [x] Approved
* [ ] Retired
## Context & Scope
The Dataset service in its implementation of the DMS APIs (getStorageInstructions and getRetrievalInstructions) delegates the call to specific DMS's implementation of the `get*Instructons` API. That lead to creation of new service called `File-DMS` which supports the APIs that Dataset Service requires.
> Note: The File-DMS does not have an approved ADR, and is not an approved by the community yet.
On the other hand, we already have a File Service that owns the responsibility for the management of files. Also, the design and APIs of the File service are approved by a thorough process of ADR.
So, instead of having two services who own the similar responsibility of managing files (File and File-DMS) we should rationalize both the services and move the APIs that Dataset Service requires to File Service itself. It would reduce the additional maintenance overhead of managing another service.
## Decision
The decision is to merge File and File DMS services and consolidate all the File management APIs in **File Service** keeping inline with the File Service ADR that was approved [File Service ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/47).
## Rationale
Having multiple services with similar functionalities and responsibilities is an additional overhead w.r.t maintenance. Since there is already one approved service for supporting Files, **the Additional APIs on Files required to support DMS Functionalities** should be hosted by File Service.
## Consequences
> The **DMS APIs** **getStorageInstructions** and **getRetrievalInstructions** for Files and File-Collections will be moved to File Service.
| Functionality | API to be Added in File Service | Existing API in File-DMS Service | Status | Capability |
|---------------------|-------------------------|-------------------------|--------|--------------------------------------------------|
| File Get-Storage Instructions | POST /files/storageInstructions | GET /file/getStorageInstructions | Moved from File-DMS Service to File Service | Generates the Signed URLs / temporary access tokens required to upload Files |
| File Get-Retrieval Instructions | POST /files/retrievalInstructions | POST /file/getRetrievalInstructions | Moved from File-DMS Service to File Service | Generates the Signed URLs / temporary access tokens required to download Files |
| File Collection Get-Storage Instructions | POST /file-collections/storageInstructions | GET /file-collection/getStorageInstructions | Moved from File-DMS Service to File Service | Generates the Signed URLs / temporary access tokens required to upload File Collections |
| File Collection Get-Retrieval Instructions | POST /file-collections/retrievalInstructions | POST /file-collection/getRetrievalInstructions | Moved from File-DMS Service to File Service | Generates the Signed URLs / temporary access tokens required to download File Collections |
### Pros of merging File DMS APIs into File Service
- Existing File Service clients will not be impacted because the existing APIs will continue to stay.
- The Core-Logic of handling File uploads and downloads already exist in File Service and are well tested. So, the DMS APIs that are required for [Dataset ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/57) will be able to directly hook into it reducing the development time.
- Features such as "Uploading to staging container" will be available out of the box. This reduces gaps between Dataset Architecture and existing File Service, which in-turn enables easier migration for clients.JoeMadhur Tanwani [Microsoft]Joehttps://community.opengroup.org/osdu/platform/system/file/-/issues/23Delivery API not working on Azure2021-03-01T21:46:28ZHrvoje MarkovicDelivery API not working on AzureDelivery API of file service returns
`{
"code": 415,
"reason": "Unsupported media type",
"message": "upstream server responded with unsupported media type: text/html"
}` irrespectively if records with provided srns exist or ...Delivery API of file service returns
`{
"code": 415,
"reason": "Unsupported media type",
"message": "upstream server responded with unsupported media type: text/html"
}` irrespectively if records with provided srns exist or not.<br>
The root cause seems to be faulty config for search query API url for Azure. There is a missing "/" in https://community.opengroup.org/osdu/platform/system/file/-/blob/master/provider/file-azure/src/main/resources/application.properties#L93.https://community.opengroup.org/osdu/platform/system/file/-/issues/21Inconsistent URL versioning2021-03-05T09:56:56ZKishore BattulaInconsistent URL versioning# Context & Scope
The following endpoints have both v1/v2 in their path which is not a clear API versioning
- /api/file/v2/v1/files/{id}/downloadURL
- /api/file/v2/v1/files/uploadURL
- /api/file/v2/v1/files/metadata
- /api/file/v2/v1/fi...# Context & Scope
The following endpoints have both v1/v2 in their path which is not a clear API versioning
- /api/file/v2/v1/files/{id}/downloadURL
- /api/file/v2/v1/files/uploadURL
- /api/file/v2/v1/files/metadata
- /api/file/v2/v1/files/{id}/metadata
# Decision
Change the Above APIs respectively
- /api/file/v1/files/{id}/downloadURL
- /api/file/v1/files/uploadURL
- /api/file/v1/files/metadata
- /api/file/v1/files/{id}/metadata
# Rational
Consistent and clear API versioning.
# Consequences
Users of these APIs need to change the paths for above APIs
# Impact
The changes would be in core module and no needed for CSPs to change any codehttps://community.opengroup.org/osdu/platform/system/file/-/issues/18If partitonId == null, a endpoint will return 403 status instead of 401.2020-11-13T12:25:12ZRiabokon Stanislav(EPAM)[GCP]If partitonId == null, a endpoint will return 403 status instead of 401.This problem has been identified when this MR https://community.opengroup.org/osdu/platform/system/file/-/commit/4ddd2e7228252edef7474ecad0c86d31c3c3598e had been merged into master
commit: https://community.opengroup.org/osdu/platform/...This problem has been identified when this MR https://community.opengroup.org/osdu/platform/system/file/-/commit/4ddd2e7228252edef7474ecad0c86d31c3c3598e had been merged into master
commit: https://community.opengroup.org/osdu/platform/system/file/-/commit/4ddd2e7228252edef7474ecad0c86d31c3c3598e#7e5fc265ef88187dd3731e9bcceee01ff99f7569_44_44
A new behavior "If partitonId == null, a endpoint will return 403 status instead of 401"
I am not sure that it is a correct response from an architecture perspective.
cc: @Dmitriy_Rudko @rostislav.dublinRiabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/file/-/issues/15Update naming conventions2020-09-17T17:18:18ZNicholas KarskyUpdate naming conventionsFile deliverable currently comes as a .jar file for provider/file-azure. Update so that the deliverable is a spring-boot.jar like other services. @ethiraj @dkodeihFile deliverable currently comes as a .jar file for provider/file-azure. Update so that the deliverable is a spring-boot.jar like other services. @ethiraj @dkodeihDania Kodeih (Microsoft)Dania Kodeih (Microsoft)https://community.opengroup.org/osdu/platform/system/file/-/issues/13Complete CI/CD and Integration Tests for Azure2020-10-15T19:39:39ZDania Kodeih (Microsoft)Complete CI/CD and Integration Tests for AzureM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/file/-/issues/12Complete Azure File Service CI/CD and Tests2020-09-16T20:28:29ZDania Kodeih (Microsoft)Complete Azure File Service CI/CD and TestsM1 - Release 0.1Dania Kodeih (Microsoft)Dania Kodeih (Microsoft)