CDK Entest

Introduction#

Persistent volume and claim
AWS EBS CSI
Static provisioning
Dyanmic provisioning
NodePort, ClusterIP and LoadBalancer services

Host Path Volume#

Let's create a volume in a specified single EC2 host. First, let's remote access to the volume and add data.

sudo mkdir /mnt/data
sudo sh -c "echo 'Hello from Kubernetes storage' > /mnt/data/index.html"

Second, follow this docs to manually create a persistent volume.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: '/mnt/data'
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - ip-10-0-0-106.us-west-2.compute.internal

Let's create the persistent volume.

kubectl apply -f yaml/pv-host-path.yaml

Then check it.

get pv task-pv-volume

Third, create a persistent volume claim.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi

Then describe the claim.

kubectl get pvc task-pv-claim

Finally, let's create a pod.

apiVersion: v1
kind: Pod
metadata:
  name: task-pv-pod
spec:
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
        claimName: task-pv-claim
  containers:
    - name: task-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: 'http-server'
      volumeMounts:
        - mountPath: '/usr/share/nginx/html'
          name: task-pv-storage

Run the pod and exec into it to check, let's cat index.html and see it content.

kubectl exec -it task-pv-pod -- /bin/sh
cd /usr/share/nginx/html
cat index.html

The index.html content should be the same as created from the beginging.

NodePort Service#

First, let port forward to check the web running.

kubectl port-forward pod/task-pv-pod 8080:80

Second, let's expose the service via NodePort type.

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    name: task-pv-pod
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: NodePort

Let's find the assigned port. In this case the NodePort service port is 32195.

kubectl get services -n default

Access the EC2 node and check the service.

curl http://localhost:32195

ClusterIP Service#

Let's create a ClusterIP service.

apiVersion: v1
kind: Service
metadata:
  name: nginx-service-clusterip
spec:
  selector:
    name: task-pv-pod
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

Describe the service, note the allocated IP address.

kubectl describe service nginx-service-clusterip

Then go to an EC2 instance and curl to the assigned IP address.

curl http://10.0.0.40:80

How service ClusterIPs are allocated?

dynamically: the cluster's control plane automatically picks a free IP address from within the configured IP range for type: ClusterIP Services.
statically: you specify an IP address of your choice, from within the configured IP range for Services.

EBS CSI Driver#

Static provisioning
Dyanmic provisioning

Create a Persistent Volume from existing ebs volume. This method called static provisioning. Please make sure to create a EBS volume and provie its id and zone here.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-pv
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 5Gi
  csi:
    driver: ebs.csi.aws.com
    fsType: ext4
    volumeHandle: vol-03c604538dd7d2f41
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.ebs.csi.aws.com/zone
              operator: In
              values:
                - us-east-2c
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  storageClassName: '' # Empty string must be explicitly set otherwise default StorageClass will be set
  volumeName: test-pv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: centos
      command: ['/bin/sh']
      args:
        ['-c', 'while true; do echo $(date -u) >> /data/out.txt; sleep 5; done']
      volumeMounts:
        - name: persistent-storage
          mountPath: /data
  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: ebs-claim

Exec in to the pod and see the data output saved in /data/out.txt

kubectl exec -it app -- /bin/sh

Now let create a dynamic provisioning. This will create a EBS volume of 4GB, IOPS 3000, and a throughput of 125Mbps. Please note that for dynamic provisioning, when we delete the object yaml, the ebs volume will be deleted as well.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 4Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: centos
      command: ['/bin/sh']
      args:
        ['-c', 'while true; do echo $(date -u) >> /data/out.txt; sleep 5; done']
      volumeMounts:
        - name: persistent-storage
          mountPath: /data
  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: ebs-claim

S3 CSI Driver#

Let's update the iam-role-stack to create a role which will be assumed by the s3 csi driver.

iam-role.yaml

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  EksStackName:
    Type: String
    Default: eks-stack-eks-cluster
Resources:
  EksOIDCProvider:
    Type: AWS::IAM::OIDCProvider
    Properties:
      Url: !ImportValue
        Fn::Sub: ${EksStackName}-oidc-url
      ClientIdList:
        - sts.amazonaws.com
      ThumbprintList:
        - 6938fd4d98bab03faadb97b34396831e3780aea1
        - 9e99a48a9960b14926bb7f3b02e22da2b0ab7280
  # Role for eks csi driver add-on
  EksCsiDriverRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: eks-csi-driver-role
      AssumeRolePolicyDocument: !Sub
        - |
          {
            "Version": "2012-10-17",
            "Statement": [
              {
                "Effect": "Allow",
                "Principal": {
                  "Federated": "arn:${PARTITION}:iam::${ACCOUNT_ID}:oidc-provider/${OIDC}"
                },
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Condition": {
                  "StringEquals": {
                    "${OIDC}:aud": "sts.amazonaws.com",
                    "${OIDC}:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
                  }
                }
              }
            ]
          }
        - PARTITION: !Ref AWS::Partition
          ACCOUNT_ID: !Ref AWS::AccountId
          OIDC: !ImportValue
            Fn::Sub: ${EksStackName}-oidc-id
      ManagedPolicyArns:
        - !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy
  # Role for eks container insight add-on
  EksContainerInsightRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: eks-container-insight-role
      AssumeRolePolicyDocument: !Sub
        - |
          {
            "Version": "2012-10-17",
            "Statement": [
              {
                "Effect": "Allow",
                "Principal": {
                  "Federated": "arn:${PARTITION}:iam::${ACCOUNT_ID}:oidc-provider/${OIDC}"
                },
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Condition": {
                  "StringEquals": {
                    "${OIDC}:aud": "sts.amazonaws.com",
                    "${OIDC}:sub": "system:serviceaccount:amazon-cloudwatch:cloudwatch-agent"
                  }
                }
              }
            ]
          }
        - PARTITION: !Ref AWS::Partition
          ACCOUNT_ID: !Ref AWS::AccountId
          OIDC: !ImportValue
            Fn::Sub: ${EksStackName}-oidc-id
            # Fn::ImportValue: !Sub ${EksStackName}-oidc-id
      ManagedPolicyArns:
        - !Sub arn:${AWS::Partition}:iam::aws:policy/CloudWatchAgentServerPolicy
  # Role for eks csi s3 add-on
  EksS3CsiDriverRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: eks-s3-csi-driver-role
      AssumeRolePolicyDocument: !Sub
        - |
          {
            "Version": "2012-10-17",
            "Statement": [
              {
                "Effect": "Allow",
                "Principal": {
                  "Federated": "arn:${PARTITION}:iam::${ACCOUNT_ID}:oidc-provider/${OIDC}"
                },
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Condition": {
                  "StringEquals": {
                    "${OIDC}:aud": "sts.amazonaws.com",
                    "${OIDC}:sub": "system:serviceaccount:kube-system:s3-csi-driver-sa"
                  }
                }
              }
            ]
          }
        - PARTITION: !Ref AWS::Partition
          ACCOUNT_ID: !Ref AWS::AccountId
          OIDC: !ImportValue
            Fn::Sub: ${EksStackName}-oidc-id
      ManagedPolicyArns:
        - !Sub arn:${AWS::Partition}:iam::aws:policy/AmazonS3FullAccess

Then install the S3 CSI Driver from add-on. Below is an application example.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: s3-pv
spec:
  capacity:
    storage: 1200Gi # ignored, required
  accessModes:
    - ReadWriteMany # supported options: ReadWriteMany / ReadOnlyMany
  mountOptions:
    - allow-delete
    - region us-west-2
    - prefix some-s3-prefix/
  csi:
    driver: s3.csi.aws.com # required
    volumeHandle: s3-csi-driver-volume
    volumeAttributes:
      bucketName: s3-csi-driver-14062024
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: s3-claim
spec:
  accessModes:
    - ReadWriteMany # supported options: ReadWriteMany / ReadOnlyMany
  storageClassName: '' # required for static provisioning
  resources:
    requests:
      storage: 1200Gi # ignored, required
  volumeName: s3-pv
---
apiVersion: v1
kind: Pod
metadata:
  name: s3-app
spec:
  containers:
    - name: app
      image: centos
      command: ['/bin/sh']
      args:
        [
          '-c',
          "echo 'Hello from the container!' >> /data/$(date -u).txt; tail -f /dev/null"
        ]
      volumeMounts:
        - name: persistent-storage
          mountPath: /data
  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: s3-claim

Trobleshooting#

Delete persistent volume.

kubectl patch pv static-pv-ebs -p '{"metadata":{"finalizers":null}}'

eksctl utils associate-iam-oidc-provider --cluster eks-stack-eks-cluster --approve

eksctl create iamserviceaccount \
  --name cloudwatch-agent \
  --namespace amazon-cloudwatch --cluster eks-stack-eks-cluster \
  --role-name RoleForEKSContainerInsights \
  --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
  --role-only \
  --approve

eksctl create iamserviceaccount \
    --name ebs-csi-controller-sa \
    --namespace kube-system \
    --cluster eks-stack-eks-cluster \
    --role-name AmazonEKS_EBS_CSI_DriverRoleDemo \
    --role-only \
    --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
    --approve

aws eks delete-addon \
--cluster-name eks-stack-eks-cluster \
--addon-name amazon-cloudwatch-observability

ClusterName=eks-stack-eks-cluster
RegionName=us-west-2
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/g;s/{{region_name}}/'${RegionName}'/g' | kubectl delete -f -
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-custom-resource-definitions.yaml | kubectl delete -f -

The OIDC thumbprint list per region.

{"us-east-2": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"us-east-1": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"us-west-1": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"us-west-2": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"ap-east-1": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"ap-northeast-2": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"ap-southeast-1": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"ap-southeast-2": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"ap-northeast-1": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"eu-central-1": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"eu-west-1": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"eu-west-2": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"eu-west-3": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"eu-north-1": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"me-south-1": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}
{"sa-east-1": "9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"}

The script to get the thumbprint.

#!/bin/bash

set -e

if [ ! -z "$DEBUG" ] ; then
    set -x
fi

REGIONS="us-east-2
    us-east-1
    us-west-1
    us-west-2
    ap-east-1
    ap-northeast-2
    ap-southeast-1
    ap-southeast-2
    ap-northeast-1
    eu-central-1
    eu-west-1
    eu-west-2
    eu-west-3
    eu-north-1
    me-south-1
    sa-east-1"

for REGION in $REGIONS ; do
    JWKS_URI="oidc.eks.${REGION}.amazonaws.com"

    # Extract all certificates in separate files
    # https://unix.stackexchange.com/questions/368123/how-to-extract-the-root-ca-and-subordinate-ca-from-a-certificate-chain-in-linux
    TEMP=$(mktemp -d -t oidc-eks-XXXX)
    openssl s_client -servername $JWKS_URI -showcerts -connect $JWKS_URI:443 < /dev/null 2>/dev/null | awk -v dir="$TEMP" '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/{ if(/BEGIN/){a++}; out=dir"/cert00"a".crt"; print >out }'

    # Assume last found certificate in chain is the ROOT_CA
    ROOT_CA=$(ls -1 $TEMP/* | tail -1)

    # Extract fingerprint in desired format (no header, no colons)
    THUMBPRINT=$(openssl x509 -fingerprint -noout -in $ROOT_CA | sed 's/^.*=//' | sed 's/://g')
    printf '{"%s": "%s"}\n' $REGION $THUMBPRINT
    rm -rf $TEMP
done

Obtain Thumbprint#

First open this URL in a browser.

https://oidc.eks.us-west-2.amazonaws.com/id/B0A30A17C98DD4693DEBFAEA162620B8/.well-known/openid-configuration

Then take note the jwks_uri without https as the following.

oidc.eks.us-west-2.amazonaws.com/id/B0A30A17C98DD4693DEBFAEA162620B8/keys

Use the OpenSSL command line tool to run the following command.

openssl s_client -servername oidc.eks.us-west-2.amazonaws.com -showcerts -connect oidc.eks.us-west-2.amazonaws.com:443

Then save the certificate to certificate.crt

Next, run the following command.

openssl x509 -in certificate.crt -fingerprint -sha1 -noout

Then output should look like below.

sha1 Fingerprint=9E:99:A4:8A:99:60:B1:49:26:BB:7F:3B:02:E2:2D:A2:B0:AB:72:80

Just remove these colon character (:) to get the thumbprint.