Skip to content

[ADR] Dataset service security enhancments

Decision Title

Security Enhancements for Dataset Service's Signed URL APIs

Status

  • Proposed
  • Trialing
  • Under review
  • Approved
  • Retired

Context & Scope

This ADR to address the security concern mentioned and implemented for file service ADR here 'file#78 (closed)' . Similarly in dataset service, we have POST StorageInstructions and POST ReterievalInstructions APIs under the scenario of a malicious user getting hold of the generated signed URLs and using them to access files from storage. When Private Link is not a desired option to mitigate these concerns for the customer due to policy and deployment complexity reasons, the following enhancements are proposed to the two existing APIs and introducing a new API to alleviate the customer's security concerns.

Decision

Proposed Changes

DATASET Service:

  1. For POSTS StorageInstructions API: Adding a new optional query parameter for expiryTime, which will be passed on to File Service StorageInstructions endpoint.
  2. For POST ReterievalInstructions API: Adding a new optional query parameter for expiryTime, which will be passed on to File Service ReterievalInstructions endpoint.
  3. New API: New API to revoke all Signed URLs generated for a specified storage account. This will be passed on to File Service RevokeUrl. This is cloud agnostic and should be implemented by each CSP in File Service. 'POST api/Dateset/v1/revokeURL'

FILE Service: DMS API's: /v2/file-collections & v2/files(4 endpoints)

  1. For POSTS StorageInstructions API: Adding a new optional query parameter for expiryTime
  2. For POST ReterievalInstructions API: Adding a new optional query parameter for expiryTime
  3. For Additional user experience: We'll add expiryTime as a response parameter, it will be converted to either default or capped TTL. This is a good to have feature and can be implemented depending upon if we have the bandwidth in this release.

Rationale

Shortened TTL for the Signed URLs decreases the Window of opportunities for a malicious user to use the Sighed URLs to access any sensitive information; Additional Revoking API provides customers a capability to mitigate the risk at the earliest moment if Signed URL leaking is detected.

Consequences

Caution: SAS token in a Signed URL cannot be individually revoked. This API will revoke all SAS tokens generated and invalidate all signed URLs for that storage account. A user needs to send GetReterievalInstructions and GetStorageInstructions requests again to generate new URLs. It should only be used when the customer knows for sure a signed URL has been compromised.

These cautions need to be included in the file service open API spec and be communicated to customers clearly.

Backward Compatibility

This is NOT a breaking change.

Observations

The dataset is using file service APIs to generate sas urls and perform any storage-related operations. The dataset is acting as a wrapper or proxy API for file/storage service APIS. The dataset internally calls File service Apis and has a full dependency.

We need to move to utilize dataset and deprecate file service APis and merge them with dataset service or keep that as backend supportive service which shall be addressed through a separate ADR.

Edited by Deepa Kumari