[ADR] Dataset service security enhancments

Decision Title

Security Enhancements for Dataset Service's Signed URL APIs

Status

  • Proposed
  • Trialing
  • Under review
  • Approved
  • Retired

Context & Scope

A customer has voiced a security concern about File Service's POST GetStorageInstructions and POST GetReterievalInstructions APIs under the scenario of a malicious user getting hold of the generated signed URLs and using them to access files from storage. When Private Link is not a desired option to mitigate these concerns for the customer due to policy and deployment complexity reasons, the following enhancements are proposed to the two existing APIs and introducing a new API to alleviate the customer's security concerns.

Decision

Proposed Changes

  1. For POSTS GetStorageInstructions API: Change default TTL from 7 days to 1 hour and make TTL configurable through a query Paramater expiryTime in Time Units Minutes,Hours,Days. The expiry time is capped at 7 Days if the time provided by the User exceeds the capped value. In absence of this parameter, the Signed URL would be valid for 1 Hour by default.
  2. For POST GetReterievalInstructions API: Change default TTL from 7 days to 1 hour. and make TTL configurable through a query Paramater expiryTime in Time Units Minutes,Hours,Days. The expiry time is capped at 7 Days if the time provided by the User exceeds the capped value.

These two changes make the two APIs behave consistently also.

  1. New API to revoke all Signed URLs generated for a specified storage account. Storage account is specified through a query parameter storageAccount. User can grab the storageAccount from the GetReterievalInstructions or GetStorageInstructions response.

    POST api/Dateset/v1/revokeURLs

This API will use the StorageAccountRevokeUserDelegationKeys to revoke all the User Delegation Keys for the storage account and that will revoke all the User Delegation SAS tokens and thus invalidate all the Signed URLs.

  1. Start using user-defined delegation keys for storage accounts rather than using storage account keys.

Rationale

Shortened TTL for the Signed URLs decreases the Window of opportunities for a malicious user to use the Sighed URLs to access any sensitive information; Additional Revoking API provides customers a capability to mitigate the risk at the earliest moment if Signed URL leaking is detected.

Consequences

Caution: SAS token in a Signed URL cannot be individually revoked. This API will revoke all SAS tokens generated and invalidate all signed URLs for that storage account. A user needs to send GET uploadURL and GET downloadURL requests again to generate new URLs. It should only be used when the customer knows for sure a signed URL has been compromised.

Caution: User Delegation Keys are cached by Azure Storage, so there may be a delay between when the user initiates the process of revocation and when an existing user delegation SAS becomes invalid. So after calling POST revokeURLs, wait for sometime and verify the compromised URL no longer works before sending GET uploadURL and GET downloadURL requests again.

These cautions need to be included in the file service open API spec and be communicated to customers clearly.

Backward Compatibility

This is NOT a breaking change.

Edited by Om Prakash Gupta