ADR: Secret Storage Service
Secret Storage Service
Status
-
Proposed -
Trialing -
Under review -
Approved -
Retired
Context & Scope
There are a number of places in the data platform where key/value pairs need to be stored globally and referenceable later. Importantly, there will be a number of values that will be sensitive and need protection. For example: external data connections will store some kind of credential on system A, that grants it programmatic access to system B. This might be a database connection string, a client_id and client_secret for OAuth, a traditional user name and password, a public/private key pair, or some other material that grants access when used correctly.
This ADR proposes the creation of a secret storage service. It outlines common use cases as well as functional and non-functional expectations. This initial draft has a lot of question marks, meant to prompt discusison.
Use Cases
- I am an administrator on a data platform instance. I have received credentials from another data platform or external data source. I need to load those credentials into my instance, so that my instance can connect to the external data source.
- I am an administrator on a data platform instance. I need to set the value for CORS headers on my external API endpoints. I need to put this value in a central location so it can be retrieved as necessary by API endpoints.
- I am an administrator on a data platform instance. I have received the public key certificate that represents a trusted IdP. I want to store that public key certificate in a location, so that the cloud-provider services can pick it up and provision it/authorise it to access my data platform instance.
- I am an application implementor. My application needs to store secrets for access to online services (e.g., API keys). I need to know how to store these secrets in a given OSDU data platform instance, without needing to know cloud-specific APIs or mechanisms.
Functional Requirements
Overall we will define a few basic operations for secrets.
- Create: given a name for a key, store its value. Optionally overwrite an existing value with the same key.
- Read: return the plaintext, unprotected value associated with a key
- Delete: delete the key and value named
- List: return the list of keys that are known
- Encrypted in transit: Only HTTPS endpoints will be defined for this service. No non-HTTPS endpoints will be defined or listening.
Non-functional Requirements
- Consistent definition of consistency. Is it write-after-read consistent or eventually consistent? We must define what kind of consistency the service provides, then all CSPs must create implementations that deliver that kind of consistency. It's not configurable or optional, but it also doesn't vary from CSP to CSP.
- Encrypted storage at rest: CSPs must provide a cloud-specific implementation that always encrypts at rest.
- SLA? Is there a TP95 or other performance requirement? e.g., maximum latency?
- Capacity? This must be able to hold hundreds? thousands? millions? of secrets?
- Security? What is the authorisation model? The vast majority of reads are by internal services (e.g., external data connections, etc). However, the vast majority of writes are presumably by end-users. An administrative user gets the credentials out-of-band and must load them into their data platform instance.
- Backup/Restore? Do we need to be able to export these secrets in order to facilitate backup/restore/disaster recovery? How do we protect exports, since they contain sensitive data?
- Partition awareness? Is the secret storage service aware of data partitions? Is it instantiated per partition or is it global?
- Versioning? Are key/values versioned or is there just one current version?
Decision
We will encapsulate operation on sensitive key value pairs within its own service. This service will be used by other OSDU data platform services to fetch secrets as required. It also needs to permit authorised users (presumably administrators) to provision secrets into the store.
Rationale
Every implementation will have sensitive data in it. If it's not stored this way, it has to be stored some other way. If we don't describe the secure means of protecting these secrets, then the story varies by cloud provider. It's important to the overall data platform's security story that we have a strong story about how these secrets are stored.
Consequences
When to revisit
Tradeoff Analysis - Input to decision
Alternatives and implications
Alternative 1: CSP-specific implementations
Each cloud provider could just choose their own storage mechanism. Keys, values, and the details around their security will vary by CSP. Naming conventions for keys will vary. When a potential user of the data platform asks how the secrets are stored, they have to get a CSP-specific story about that.
Alternative 2: Use an existing store of some kind
An existing service that stores data (e.g., the partition service) could hold these kinds of data, also. That would require changing that existing service to meet a higher security bar, since it will be holding more sensitive information.
Alternative 3: Every service does a YOLO implementation
Every service that needs to store keys and values does so in its own way. Each one has a different API, CSP implementation, etc.