AWS Secrets Manager is an AWS service that enables you to rotate, manage, and retrieve secrets for applications, services, and IT components through their lifecycle. It can store various kinds of secrets, including API Keys, SSH Keys, and database credentials, and supports any key-value pair.
AWS Secrets Manager supports encryption at rest, encryption in transit, and applies the AWS Shared Responsibility Model. This allows the application to retrieve and manage its secrets securely through the service.
Apart from secure secrets storage, AWS Secrets Manager also provides:
- Automatic secrets rotation
- Automatic replication to multiple AWS regions
- Programmatic secrets retrieval
- Integration with other AWS services
Integrating AWS Secrets Manager with Amazon EKS allows your EKS pods to use Secrets Manager as a secrets management module. In this article, we will explain how to integrate AWS Secrets Manager with your Amazon EKS Clusters, explore the advantages of using a secrets management solution, best practices, and address some of the Secrets Manager’s current limitations.
EKS Secrets Manager executive summary
Below is an executive summary of the key areas covered in this article.
Integrating AWS Secrets Manager with Amazon EKS Key Concepts
| Concept | Summary |
|---|---|
| Advantages of using a secrets management solution on EKS | Secrets management solutions enable the secure storage, retrieval, and management of secrets like credentials and API keys in EKS. |
| Setting up the infrastructure for Secrets Manager with Amazon EKS integration | Reviews how to set up the infrastructure for Secrets Manager integration with EKS. |
| Best practices for Secrets Manager integrations | Following the shared responsibility model, rotating keys, limiting access to secrets, and effective use of encryption are all important Secrets Manager best practices. |
| Limitations on Secrets Manager integration | Discussion on limitations regarding AWS Secrets Manager and how you can work around them. |
Advantages of using a secrets management solution on EKS
Secrets are a vital part of application security. Your application will likely communicate with external services like a database, a 3rd party API, or another service of your application. Secrets, like passphrases and API keys, keep communication secure.
The process of fetching, managing, and rotating secrets for secure communication with external services can quickly become complex. A secrets management solution solves this problem by removing most of the complexity and overhead from your application.
Integrating a secrets management solution with your EKS Cluster also strengthens infrastructure security and compliance. AWS Secrets Manager integration with multiple AWS Services makes it a great solution for managing secrets.
Setting up the infrastructure for Secrets Manager with Amazon EKS integration
In the sections below, we will walk through how to set up the Kubernetes Secrets Store CSI Driver with AWS Secrets Manager on AWS. We will enable secret sync to define secrets set up on Secrets Manager as environment variables on our EKS Pods. We will use Helm to install the necessary packages, so make sure it is already installed on your system so you can follow along.
ASCP for Kubernetes Secrets Store CSI Driver
The AWS Secrets and Configuration Provider (ASCP) for the Kubernetes Secrets Store CSI Driver should be installed to enable integrating Secrets Manager on your EKS pods. ASCP allows your secrets to be stored as files mounted inside the pods which can then be used by creating a SecretProviderClass YAML file that contains information about your secrets and how your workload should use them.
To limit access for your secrets to a specific Amazon EKS Pod you can utilize AWS IAM through IAM Roles and policies. The diagram below details an implementation example.
Diagram of Secrets Manager integration with Amazon EKS
Installing Kubernetes Secrets Store CSI Driver
The commands on the snippet below add the repository for the CSI Driver on your machine, and from there you install the CSI Driver on your EKS Cluster using Helm. You can enable secret sync by setting its parameter to true.
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set syncSecret.enabled=true --namespace kube-system
Other features, which can be found in the Secrets Store installation guide, can also be enabled for the CSI Driver. For details on different values you can configure when running Helm install, see the Helm install configuration for the secrets store CSI driver.
Setting up AWS Secrets and Configuration Provider (ASCP) on EKS
To install ASCP, use this command:
kubectl apply -f
https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml
This will install the ASCP on your kube-system namespace and create the necessary permissions and resources to allow it to run.
Configuring the IAM Role and Policies
An IAM Role should be created to allow access to the AWS Secrets Manager from EKS. The IAM Role should be attached to a Kubernetes service account which we will set up later.
When creating secrets, you have the option of encrypting secrets with KMS. If your secrets are encrypted with KMS, your policy must contain a statement that decrypts the secret value. Otherwise, you can have a simple policy with minimal permissions to get secrets from Secrets Manager.
Policy for unencrypted secrets
The policy below grants minimal permissions to the secret for your Kubernetes service account.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ], "Resource": "arn:*:secretsmanager:*:*:secret:very-secret-value" } ] }
Policy for encrypted secrets with KMS
When adding multiple secrets you must add an array to the Resource on your policy so it can commit the policy statement on multiple secrets. Below is an example policy that adds multiple encrypted secrets.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "*", "Condition": { "StringEquals": { "kms:CallerAccount": [ "AWS_ACCOUNT_ID" ], "kms:ViaService": [ "secretsmanager.AWS_REGION.amazonaws.com" ] } } }, { "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ], "Resource": "arn:*:secretsmanager:*:*:secret:very-secret-value" } ] }
Creating a service account on your EKS Cluster
Create a Kubernetes service account so your pods can access Secret Manager. The IAM Role created previously will be attached to this service account to allow access.
apiVersion: v1 kind: ServiceAccount metadata: name: secret-manager-service-account annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<IAM_ROLE_NAME>
Create a YAML file with the snippet above and apply it on your EKS cluster so it can take effect using:
kubectl apply -f $FILENAME.yaml
Creating SecretProviderClass
To enable using the Secrets Store CSI driver, create a custom resource named SecretProviderClass (SPC) on your EKS Cluster – which will provide driver configurations and provider-specific parameters to the CSI Driver. You configure all the secrets your pod needs through the SecretProviderClass. The pod will mount the secrets as a volume from the SPC.
apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: secrets-demo spec: provider: aws secretObjects: - secretName: api_secret_key type: Opaque data: - objectName: "MySecretValue" key: password parameters: objects: | - objectName: arn:*:secretsmanager:*:*:secret:very-secret-value objectAlias: "MySecretValue"
The secretObjects value above helps to implement the secret sync with Kubernetes Secrets. That means it will sync changes to your secrets on Secrets Manager with the existing configuration on the pod. It only works when secret sync is enabled.
The objects field of the SecretProviderClass can contain multiple subfields, which can be found here. You can define specific key-value pairs to extract from JSON-formatted secrets using the jmesPath. The objectName and the objectType fields are both required. Using the example below, you can mount multiple secrets on the custom resource.
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1 kind: SecretProviderClass namespace : secrets-demo metadata: name: very-secret-credentials spec: provider: aws parameters: objects: | - objectName: "arn:*:secretsmanager:*:*:secret:very-secret-value" jmesPath: - path: username objectAlias: dbusername - path: password objectAlias: dbpassword - objectName: "arn:*:secrets manager:*:*:secret:very-secret-value" - objectName: "another-secret" objectType: "secrets manager"
Once you have setup the manifest for the SecretProviderClass you will have to run the file using:
kubectl apply -f $FILENAME.yaml
The SecretProviderClass must be in the same namespace as the Amazon EKS pod it references.
Mounting secrets on deployment
All that’s left now is to mount your secrets when deploying your application to Kubernetes. The SecretProviderClass above will contain information about the secrets that are setup and how to display them in the EKS pod. This allows us to mount the secrets as volumes to the deployment manifest. The values from secrets will be extended as files on the mounted volumes, which the deployment manifest can reference.
The following is a sample deployment YAML file that deploys an nginx server on Kubernetes with health checks and rolling updates. Here, we add our secrets as a mounted volume on it and reference the SecretProviderClass created before.
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: web spec: selector: matchLabels: app: web replicas: 2 strategy: type: RollingUpdate template: metadata: labels: app: web spec: serviceAccountName: secret-manager-service-account volumes: name: demo-secrets csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttibutes: secretProviderClass: secrets-demo spec: containers: —name: nginx image: nginx ports: —containerPort: 80 volumeMounts: - name: demo-secrets mountPath: “/mnt/db/secrets” readOnly: true
Applying the changes from your deployment manifest allows your Kubernetes deployment to securely extract and use secrets from AWS Secrets Manager.
Best practices for EKS Secrets Manager integrations
According to the Shared Responsibility Model by AWS, as secure as AWS Secrets Manager is, there are implementations users can follow to strengthen its security further. Following these best practices can help you improve infrastructure security:
- Encrypt secrets using KMS
- Ensure that AWS Secrets Manager enforces data-at-rest encryption using KMS Customer Managed Keys (CMK)
- Enable secrets rotation and rotation interval according to your compliance policies
- Define minimal access policies to access secrets
Limitations on Secrets Manager integrations
There are some limitations when it comes to AWS Secrets Manager that can impact certain use cases.
One such limitation is rate limiting. If your application or API makes frequent requests, you could be hit by rate limiting while trying to retrieve secrets on AWS Secrets Manager.
Secrets Manager is also as secure as you make it. Allowing overly-permissive policies could expose your secrets to the outside and intruders – putting your infrastructure at risk.
The Secrets Manager rotation supports rotating secrets through AWS Lambda as well. Doing so could log sensitive information to the lambda logs, like secret values, so be wary of this.
If you want to store binary data, you won’t be able to upload it using the Management Console, only SDK or the AWS CLI.
Identify, prioritize and remediate the most important cloud security risks.
Correlate findings across your existing tools: CSPM + Infrastructure + App Vulnerabilities
Reduce alert fatigue by up to 50% and lower your overall risk profile by up to 25%
Conclusion
AWS Secrets Manager makes cloud implementations easier thanks to its ease of integrating with other AWS Services. You should always be careful to avoid data leaks when implementing public cloud solutions. Tightening policies and mitigating threats is essential to enable application scalability while mitigating the risks posed by threat actors.
Paladin Cloud’s Cloud Security Posture Management (CSPM) platform could help you find blind spots in your cloud security. Its open-source solution could help you visualize, identify, prioritize and quickly remediate issues when implementing such solutions as integrating AWS Secrets Manager with Amazon EKS. The agentless open-source platform can be run against your full AWS, GCP, and Azure environments or through specific services like AWS Secrets Manager and EKS.