Tag: k8s

Synchronizing files between local machine and Kubernetes Persistent Volume

How to rsync files to a Kubernetes Persistent Volume (PV)?

I need to rsync a recursive directory tree to a specific PV in a Kubernetes cluster.
Unfortunately, kubectl cp command does not support rsync behavior at the moment. So we have to do a simple hacking to do it.
rsync behavior to copy files from pod to pod, perhaps for kubectl cp · Issue #551 · kubernetes/kubectl · GitHub

TLDR;

We can use a rsync script to rsync files to a Kubernetes pod which mount specific PV as a volume and rsync is executable.

The same idea can be found here
rsync files to a kubernetes pod – Server Fault

Let’s say we have a recursive folder sample-dir and a PV named sample-pv which Persistent Volume Claim (PVC) named sample-pvc

Prepare working pod

The working pod has to (1) mount PVC sample-pvc as a volume; (2) has rsync is executable.

We will create a YAML file for working pod which mounts PVC sample-pvc as a volume. And for (2), we will use a rsync executable Docker image eeacms/rsync.

working-pod.yml

apiVersion: v1
kind: Pod
metadata:
    name: working-pod
spec:
    volumes:
    - name: sample-volume
      persistentVolumeClaim:
        claimName: sample-pvc # mount pvc as volume
    containers:
    - name: operator
      image: eeacms/rsync # rsync executable Docker image
      command: ['sh', '-c', 'sleep 3600'] # sleep 3600 secs for files copy
      volumeMounts:
      - mountPath: /var/files # mount volume to path /var/files
        name: sample-volume

Prepare rsync bash script

We will create a bash script name rsync-helper.sh for execute rsync on working pod.

#!/bin/bash
pod=$1;shift;kubectl exec -i $pod -- "$@"

Make sure rsync-helper.sh is executable.

$ chmod +x ./rsync-helper.sh

Deploy working pod to Kubernetes cluster

In next step, we will deploy working pod to Kubernetes cluster.

$ kubectl apply -f ./working-pod.yml

Execute rsyn command on remote working pod

After status of wroking-pod pod become running, we will execute rsync-helper.sh script on it remotely.

$ rsync -av --delete-after --progress -e './rsync-helper.sh' sample-dir working-pod:/var/files/

building file list ...
 0 files...
19 files to consider
sample-dir/
sample-dir/new-file
           6 100%    0.00kB/s    0:00:00
           6 100%    0.00kB/s    0:00:00 (xfer#1, to-check=16/19)
sample-dir/updated
           5 100%    4.88kB/s    0:00:00
           5 100%    4.88kB/s    0:00:00 (xfer#2, to-check=15/19)
deleting sample_dir/deleted

sent 689 bytes  received 70 bytes  1518.00 bytes/sec
total size is 98957  speedup is 130.38

Note, you can add –dryrun option for debuging

$ rsync -av --dry-run --delete-after --progress -e './rsync-helper.sh' sample_dir working-pod:/var/files/

Teardown

We will delete working pod by bellow command

$ kubectl delete --grace-period=1 -f ./working-pod.yml

Conclusion

In this blog, we have a work around solution for synchronizing files and directories between local machine and PV on Kubernetes cluster.

Goodluck, happy Hacking!

Additional, Other Kubernetes related topics is here. https://tuantranf.me/tag/kubernetes/

Kubernetes ConfigMap recursive directory issues

A workaround solution for using Kubernetes Configmap to map a recursive directory.

First, about ConfigMap recursive directory issues, you can find from below topics

or

However, unfortunately, it isn’t possible to build a Kubernetes ConfigMap from a directory, recursively [1] currently.

For example,

conf/
└── integrator
    └── conf
        ├── axis2
        │   └── axis2.xml
        ├── carbon.xml
        ├── datasources
        │   └── master-datasources.xml
        ├── registry.xml
        └── user-mgt.xml

Therefore, we will use a work-around to solve this issue.

Prepare a Helm template

First, let prepare a Helm template for Kubernetes deployment with configmap.yaml for ConfigMap and deployment.yaml for Deployment

$ tree -L 2 helm_templates/your-template
helm_templates/your-template
├── Chart.yaml
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── configmap.yaml  <--- ConfigMap
│   ├── deployment.yaml <--- Deployment
└── values.yaml

For instance, Configmap yaml for you Helm template

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ template "your-template.fullname" . }}-config
data:
# mapping configMaps from values.yml to ConfigMap
{{- range $key, $value := .Values.configMaps }}
  {{ $key }}: |-
{{ $value.content | indent 4 }}
{{- end }}

After that, a Deployment.yaml which mount all configMap data to volume using subPath. Please note that we using path properties for mount path setting.

spec:
      volumes:
        - name: {{ template "your-template.fullname" . }}-config
          configMap: { name: {{ template "your-template.fullname" . }}-config }
      containers:
        - volumeMounts:
            {{- range $key, $value := .Values.configMaps }}
            - name: {{ template "your-template.fullname" $ }}-config
              # note that we using path properties for mount path setting
              mountPath: /config/{{ $value.path }}
              subPath: {{ $key }}
            {{- end }}

Generate your Helm values.yaml by Python code

After that, we will prepare a simple Python code to read recursive directory and pass file’s content to values.yaml. For loading recursive directory from config_dir.

def get_config_maps(config_dir) -> dict:
         config_maps = {}
         filenames = [y for x in os.walk(config_dir) for y in glob.glob(os.path.join(x[0], '*')) if os.path.isfile(y)]

         for filename in filenames:
             # remove parent dir path
             relative_path = filename.replace(config_dir + os.path.sep, "")
             # replace path separator by "__" to prevent Kubernetes syntax error
             # currently file path doesn't validate '[-._a-zA-Z0-9]+'
             key = relative_path.replace(os.path.sep, "__")
             with open(filename, 'r') as file:
                 config_maps[key] = {
                     "path": relative_path, # a path that will be used for mount ConfigMap to Pod's volume
                     "content": file.read()
                 }
         return config_maps

The next step is, saving file’s content to values.yaml

 # load value template
 confif_dir = 'path/to/your/conf'
 with open('helm_templates/your-template/values.yaml', 'r') as file:
      deploy_values = yaml.safe_load(file)
      deploy_values['configMaps'] = get_config_maps(config_dir)

Deploy using your Helm template

helm install [your-helm-template-path] -f values.yaml

In conclusion, we can build a ConfigMap from a recursive directory by using a simple Python code with Helm template. Good luck & happy Kubernetes! :beer:

Additional, Other Kubernetes related topics is here. https://tuantranf.me/tag/kubernetes/

Original work is here https://gist.github.com/tuantranf/1faf61d0864353ed617af1f90f3ee453

How to configure access limitation for a Kubernetes Pod

This blog provide a guideline how to restrict pod communication using Network Policies on AWS EKS cluster.

About Network Policy

We will use network policy feature of Kubernetes.

A network policy is a specification of how groups of pods are allowed to communicate with each other and other network endpoints.
By default, the pod’s communication is open within themselves and other endpoints. In a production-level cluster, it is not secure to have an open pod to pod communication.

In order to implement network policies in your cluster, you must use a compatible container network plugin and Calico is one of the compatible technology.

For more information, Network Policies

Calico on your Amazon EKS cluster

Project Calico is a network policy engine for Kubernetes.

  1. Apply the Calico manifest from the aws/amazon-vpc-cni-k8s GitHub project. This manifest creates DaemonSets in the kube-system namespace.
kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.5/config/v1.5/calico.yaml
  1. Watch the kube-system DaemonSets and wait for the calico-node DaemonSet to have the DESIRED number of pods in the READY state. When this happens, Calico is working.
kubectl get daemonset calico-node --namespace kube-system

NAME          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
calico-node   1         1         1       1            1           beta.kubernetes.io/os=linux   27m

To uninstall Calico

kubectl delete -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.5/config/v1.5/calico.yaml

For more information from AWS Installing Calico on Amazon EKS

Create Network Policies to restrict access to your pod

Create manifest file

Please check and update correct value for your access source pod in target-sample-pod-network-policy.yaml

Temporarily, let say it’s name is source-sample-pod.

target-sample-pod-network-policy.yaml

# Deny all access to target-sample-pod
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-to-target-sample-pod
  namespace: default # namespace of target-sample-pod
spec:
  podSelector:
    matchLabels:
      # labels of target-sample-pod
      app: target-sample-pod
      release: target-sample-pod
  policyTypes:
    - Ingress

---
# Allow only access from specified Pod
# Web application API Pod
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-source-sample-pod-to-target-sample-pod
  namespace: default # namespace of target-sample-pod
spec:
  podSelector:
    matchLabels:
      # labels of target-sample-pod
      app: target-sample-pod
      release: target-sample-pod
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              # labels of source-sample-pod pod
              # please update with your label values
              app: source-sample-pod
      # Optional - using when restrict access to a specified port
      ports:
        - protocol: TCP
          port: 8080 # port of target-sample-pod

Create network policy

Use kubectl to create a NetworkPolicy from the above api-server-task-policy.yaml file:

kubectl apply -f kubernetes/target-sample-pod-network-policy.yaml

Please note that it may take 10 seconds to be effected.

Verification

You should see the following:

  • We cannot access target-sample-pod pod from another pod
  • source-sample-pod can now access target-sample-pod pod (on TCP port 8080 only).

Verify that we can not access from another pod

kubectl run --generator=run-pod/v1 busybox --rm -ti --image=busybox -- /bin/sh

/ # wget --spider --timeout=1 target-sample-pod:8080
Connecting to target-sample-pod:8080 (172.20.238.20:80)
wget: download timed out
/ #

Verify that we can access from pod with correct label app: source-sample-pod 

kubectl run --generator=run-pod/v1 busybox --rm -ti --labels="source-sample-pod" --image=busybox -- /bin/sh

/ # wget --spider --timeout=1 target-sample-pod:8080
Connecting to target-sample-pod:8080 (172.20.238.20:80)
remote file exists

How to delete network policy

kubectl delete -f kubernetes/target-sample-pod-network-policy.yaml

Reference