Google Cloud Secret Manager offers users a simple way to store sensitive values in cloud projects. These values, such as API keys or database credentials, are commonly accessed by other services running in the same cloud project, such as Cloud Run, and are passed as environment variables to running workloads.
In this article, you’ll learn about use cases for secrets, versioning, and lifecycles. You’ll also find out about the related administration decisions that surround storing and retrieving secrets in a Google Cloud environment. While the platform keeps the secrets secure by default, this article also reviews logging access to the secrets and building a Pub/Sub integration pattern to notify administrators about changes.
This article does not cover Cloud KMS, a separate API to store secrets in hardware security modules in Google Cloud.
Summary of key Google Cloud Secret Manager concepts
This article will cover the following five areas.
| Use cases | Learn how Google Cloud Secrets are used in enterprise IT deployments and the common pitfalls to avoid. |
| Versioning | Understand the purpose of secret versions and how they are used; they are not simply another copy of the same secret. |
| Quotas and limits | If you’re retrieving a secret frequently—for example, every time you make an API call—be sure that project-level quotas are high enough. |
| Pub/Sub integration | Review built-in Pub/Sub topic connectivity to be notified when a secret needs to be rotated or its version or value is changed. |
| Logging access to secrets | Accessing a secret is not logged by default, but there is a process to activate additional logging. |
Use cases
As more workloads move to cloud environments, securely storing and retrieving sensitive data such as API keys, passwords, or database credentials becomes an increasingly crucial component of platform functionality. Google Cloud’s Secret Manager is a centralized, global tool that provides a platform-native solution for managing small pieces of secret data.
If the majority of your workloads are in Google Cloud, unifying around a solution like Secret Manager is an excellent method to ensure IAM consistency, eliminate external network calls, and inject secret values into other platform services such as Google Kubernetes Engine or Cloud Run. However, if you operate a multi-cloud or hybrid cloud environment, you may prefer an open-source third party tool like Hashicorp Vault.
Secret Manager’s native functionality adheres to the principle of least privilege and includes granular IAM roles and policies, such as the Secret Manager Secret Accessor role applied to a single secret. In addition to applying specific roles to users, you can also add conditional access policies, such as date and time restrictions, or limit users’ access to a single secret.
Note that Secret Manager is different from Google’s Cloud Key Management service, also referred to as Cloud KMS. Cloud KMS offers hardware key security with HSM support for customer-managed encryption keys. Cloud KMS is usually only required for highly regulated industries that require compliance, and it can only manage keys, not general secret data such as credential files or passwords. Secret Manager meets most organizational requirements by always encrypting data before it is written to disk using the AES-256 encryption standard and communicating over HTTPS.
Versioning
Secret versions are subresources of a secret resource and contain the underlying secret data. They are commonly used during scheduled secret rotation. New versions are not simply a copy of the same secret but instead contain an updated value for the parent secret resource.
Versions can be enabled, disabled, and destroyed. A best practice is to disable secret versions prior to destroying the version or deleting the parent secret resource. This allows you to reverse the decision before permanently deleting the secret data in case a lagging dependency is still attempting to retrieve it.
Each version is identified by a version number, but if you prefer to use a text string to reference the secret, you can assign an alias to it. Google Cloud has a built-in alias called “latest” that always calls the latest version of the secret. However, if downstream systems are using the “latest” alias instead of a version number or user-defined alias, creating and destroying versions can lead to unintended downstream effects. Systems referencing secrets should always use the version number or alias to ensure that new secret versions don’t break existing applications.
In addition to aliases, secrets can also be assigned additional metadata via labels and annotations. These offer no additional security benefits but can be useful for audit logging or reporting purposes.
Quotas and limits
If you’re retrieving a secret frequently, such as every time you make an API call, be sure that the project-level quotas in the workload’s environment are high enough. Most customers will not run into the quotas or limits on accessing secrets, but if you include Secret Manager as a dependency in a SaaS application between API calls, you may hit the ceiling sooner than expected.
To illustrate this, imagine an application that calls a secure API downstream on behalf of a customer. The application may reference a customer-specific credential in Secret Manager as part of every API call to keep the credentials out of a local cache. This is known as an access request, and these are limited at 90,000 per minute per project. The 90,000 number can be raised, but that requires opening a Google Cloud support ticket.
Additionally, the secret value payload size can never exceed 64 KiB and a secret cannot have more than 50 aliases.
Pub/Sub integration
Secret Manager offers an optional tight integration pattern with Google Cloud’s Pub/Sub service. Many companies deploy Pub/Sub as part of a cloud foundation or landing zone to send logging and event data to downstream security systems, commonly a SIEM.
Secret Manager can connect a single secret to up to ten Pub/Sub topics to notify other systems when a secret needs to be rotated, the secret value changes, or a secret version changes. Integration with Pub/Sub is configured on a per-secret basis, and the Google-provided Secret Manager service account must have the Pub/Sub Publisher IAM role applied on the receiving topics.
As of this writing, there is no Organizational Policy that requires secrets to connect to Pub/Sub topics. If this is a security requirement for your organization, it’s best to use an infrastructure-as-code (IaC) tool to create secrets in a compliant fashion and monitor your cloud environment with a tool like Paladin Cloud, an extensible open-source security-as-code platform. Similarly, while Secret Manager can notify Pub/Sub topics of secret rotation expiration dates, the service does not currently have the ability to auto-rotate secrets.
Logging access to secrets
While the previous section outlined logging secret changes and rotation requirements via a Pub/Sub topic, accessing a secret is not logged by default. However, there is the capability to enable granular access logs on secrets and retrieve the data in Google Cloud’s Logs Explorer. To do this, you must first activate the relevant data access logs in the IAM Cloud Audit Logs service.
To perform this in the Google Cloud Console, follow these steps:
- Go to IAM & Admin → Audit Logs.
- Search for the Secret Manager API and select it.
- Check the Data Read option, and press Save.
To perform this in Terraform, add the following resource to your main.tf file:
resource "google_project_iam_audit_config" "secret_manager_access_logs" {
project = "your-project-name"
service = "secretmanager.googleapis.com"
audit_log_config {
log_type = "DATA_READ"
}
}
A note of caution: If you use Terraform in your environment, secret data may be visible in your Terraform state if you use Terraform to fully manage a secret’s lifecycle. This is primarily a concern when including sensitive values in a Terraform apply command. This issue is well-documented but unresolved. To limit exposure, set secret values outside Terraform and only use Terraform to reference the value, or add strict access policies on the underlying location where your state is stored.
Data read audit logs can also create large bills if the secrets are frequently accessed. Evaluate the tradeoffs and the necessity of logging every secret request before activating audit logging, and configure custom retention policies on log buckets that contain this data. If there is a specific compliance reporting standard you must meet, Google Cloud provides all of the related documentation, including Secret Manager best practices, on its compliance offerings page.
Identify, prioritize and remediate the most important cloud security risks.
Correlate findings across your existing tools: CSPM + Infrastructure + App Vulnerabilities
Reduce alert fatigue by up to 50% and lower your overall risk profile by up to 25%
Summary of key concepts
Storing secrets in Google Cloud is a convenient approach for applications to retrieve sensitive information from within the platform. However, it’s important to understand how secrets are managed, audited, and controlled for safety and scalability. This includes using versions to rotate secrets, leveraging aliases for easier management, and notifying administrators of changes and rotations using Pub/Sub. Third-party tools such as Paladin Cloud can help by making auditing secret policies and finding vulnerabilities easier across all environments that contain sensitive data.
The low cost and high default quota of retrieving secrets at scale make Google Cloud’s Secret Manager an attractive architecture decision when storing sensitive values such as API keys. That said, ensuring that your design respects Google Cloud platform limits is a necessity when using Secret Manager in a SaaS environment, along with evaluating whether enabling data read audit logs is truly required in your environment.