Docker, Kubernetes, and Co.

Prepare hosts

DNS Setup

root@master-node:~# cat /etc/hosts localhost master-node slave-node1 slave-node2 slave-node3

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Add kernel modules

root@master-node:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.

Install packages for VirtualBox additions

apt-get install linux-headers-$(uname -r)

apt-get install build-essential

Install VirtualBox additions after that.

Install NFS packages

Server (master-node)

apt install nfs-kernel-server

Worker nodes

apt install nfs-common

Configure NFS shares

Create volumes/directories to be shared

root@master-node:~# mkdir -p /var/nfs/vol
root@master-node:~# mkdir -p /var/nfs/vol1
root@master-node:~# mkdir -p /var/nfs/vol2
root@master-node:~# mkdir -p /var/nfs/vol3

Configure shares

root@master-node:~# cat /etc/exports

Configure service

root@master-node:~# systemctl is-enabled nfs-server
root@master-node:~# systemctl restart nfs-server

Verify access from the server and from the worker nodes

root@master-node:~# showmount --exports
Export list for
root@slave-node1:~# showmount --exports
Export list for

Disable IPv6

vi /etc/sysctl.conf


Start installing Kubernetes cluster


MetalLB (Updated)

Follow the instruction here:

To install MetalLB, apply the manifest:

kubectl apply -f

Instead of adding a configmap, we now add a resource definition for IP address pool:

kind: L2Advertisement
  name: l2-ip
  namespace: metallb-system
  - default-pool
kind: IPAddressPool
  name: default-pool
  namespace: metallb-system

Install Helm

Kubernetes NFS Subdir External Provisioner

root@master-node:~# helm repo add nfs-subdir-external-provisioner
"nfs-subdir-external-provisioner" has been added to your repositories

root@master-node:~# helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --set nfs.server= --set nfs.path=/var/nfs/vol1
NAME: nfs-subdir-external-provisioner
LAST DEPLOYED: Thu Apr 28 16:17:19 2022
NAMESPACE: default
STATUS: deployed

Make NFS storage class default

kubectl patch storageclass nfs-client -p ‘{“metadata”: {“annotations”:{“”:”true”}}}’

olegme@master-node:~/k0suser@computer$ kubectl get storageclass
nfs-client (default) cluster.local/nfs-release-nfs-subdir-external-provisioner Delete Immediate true 8m12s

Configure example POD to use the external NFS Provisioner

Vagualy based on

Crucial element is in the PV Claim definition:

root@master-node:~/k0s/manifests# cat pv-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
  name: task-pv-claim
  storageClassName: nfs-client
    - ReadWriteOnce
      storage: 1Gi

Note storageClassName attribute nfs-client, it is the same as the storageClass definition we created in the previous step.

Alternative to NFS Provisioner – Local Path Provisioner

To be found here –

Lately, I experienced some strange issues with PostgreSQL installation using NFS provisioner and couldn’t really find the root cause. So I decided to move over to this simplest one. Don’t forget to change the default storage class!

Install PostgreSQL cluster

PostgreSQL Operator

Start with the instruction from up to the point where you are ready to deploy the first PostgreSQL cluster via the GUI. Then use GUI to create a skeleton of the deployment YAML.

Create cluster

Start with the GUI:

More instances will create more standby pods and probably are not needed. Also add user(s) and database(s):

But don’t hit Create Cluster button, use Copy instead and put it into a YAML file:

root@master-node:~/k0s/manifests# vi pgtest4.yaml
kind: "postgresql"
apiVersion: ""

  name: "acid-pgtest4"
  namespace: "default"
    team: acid

  teamId: "acid"
    version: "14"
  numberOfInstances: 2
  enableMasterLoadBalancer: true
  enableConnectionPooler: false
    size: "20Gi"
    storageClass: nfs-client
    camunda: []
    camunda: camunda
    # IP ranges to access your cluster go here

      cpu: 100m
      memory: 100Mi
      cpu: 500m
      memory: 500Mi

Two critical attributes cannot be added via GUI. storageClass: nfs-clinet will make sure that the PV Claims created request volumes from the NFS Provisioner and allowedSourceRanges: will allow connection to the database cluster from any IP. If not specified, it will allow connections only from the localhost. This can be refined later.

Apply the created YAML file:

root@master-node:~/k0s/manifests# kubectl apply -f pgtest4.yaml created

And after a while observe your new PostgreSQL cluster be created:

root@master-node:~/k0s/manifests# kubectl get svc -w
NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)          AGE
acid-pgtest2           LoadBalancer   5432:32015/TCP   16h
acid-pgtest2-config    ClusterIP      None             <none>          <none>           16h
acid-pgtest2-pooler    ClusterIP    <none>          5432/TCP         16h
acid-pgtest2-repl      ClusterIP     <none>          5432/TCP         16h
acid-pgtest3           LoadBalancer   5432:32518/TCP   13h
acid-pgtest3-config    ClusterIP      None             <none>          <none>           13h
acid-pgtest3-repl      ClusterIP    <none>          5432/TCP         13h
acid-pgtest4           LoadBalancer   5432:32511/TCP   2m6s
acid-pgtest4-config    ClusterIP      None             <none>          <none>           2m
acid-pgtest4-repl      ClusterIP     <none>          5432/TCP         2m6s
kubernetes             ClusterIP        <none>          443/TCP          2d17h
my-nginx               LoadBalancer   80:31001/TCP     25h
nginx-deployment       NodePort    <none>          80:30598/TCP     2d
postgres-operator      ClusterIP    <none>          8080/TCP         17h
postgres-operator-ui   ClusterIP   <none>          80/TCP           17h

Our new Service is now up and available on the IP Let’s check it.
We will need a password for our user camunda. We can do the following:

root@master-node:~/k0s/manifests# export PGPASSWORD=$(kubectl get secret -o 'jsonpath={.data.password}' | base64 -d)               
root@master-node:~/k0s/manifests# echo $PGPASSWORD

In the first line above camunda is the user name and acid-pgtest4 is the name of the service. With the above we set an environment variable, which will be used in psql command:

root@master-node:~/k0s/manifests# psql -U camunda -h
psql (13.5 (Debian 13.5-0+deb11u1), server 14.2 (Ubuntu 14.2-1.pgdg18.04+1))
WARNING: psql major version 13, server major version 14.
         Some psql features might not work.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

camunda=> \l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
 camunda   | camunda  | UTF8     | en_US.utf-8 | en_US.utf-8 |
 postgres  | postgres | UTF8     | en_US.utf-8 | en_US.utf-8 |
 template0 | postgres | UTF8     | en_US.utf-8 | en_US.utf-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf-8 | en_US.utf-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
(4 rows)


The database can be now accessed from the host machine as well:

Camunda Engine

Install via Helm chart

Starting point:

Secrete creation differs a bit, different user and also the password, which we already setup for PostgreSQL access in the previous part.

root@master-node:~# kubectl create secret generic                   \
    camunda-bpm-platform-db-credentials             \
    --from-literal=DB_USERNAME=camunda \
secret/camunda-bpm-platform-db-credentials created

Number of instances can be adjusted in the values.yaml. The file will look like this:

root@master-node:~# cat k0s/manifests/values.yaml
# Default values for camunda-bpm-platform.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

  debug: false
  replicaCount: 3
  nameOverride: ""
  fullnameOverride: "camunda-bpm-platform"

  repository: camunda/camunda-bpm-platform
  tag: latest
  pullPolicy: IfNotPresent
  pullSecrets: []
  command: []
  args: []

# By default H2 database is used, which is handy for demos and tests,
# however, H2 is not supported in a clustered scenario.
# So for real-world workloads, an external database like PostgreSQL should be used.
  driver: org.postgresql.Driver
  url: jdbc:postgresql://acid-pgtest4:5432/camunda
  credentialsSecretName: camunda-bpm-platform-db-credentials

  type: LoadBalancer
  port: 8080
  portName: http

  enabled: false
    type: ClusterIP
    port: 9404
    portName: metrics
    annotations: "true" "/" "9404"


Note that the service type has to be set to LoadBalancer if we deploy more than one instance. This has one caveat, though. With the current Helm chart, it is impossible to specify the sessionAffinity attribute. The web interface will behave somewhat weirdly if we don’t do that. For example, you won’t be able to disable the Telemetry consent pop-up or won’t be able to log in at all. Obviously, the Helm chart has to be adjusted, but the short-term solution would be to patch the service configuration post-install. In my case, I was able to accomplish this directly from Lens.

Here is an excerpt from its manifest after the patching:

Update. Here is also a command line version of the patching process:

kubectl patch svc camunda-bpm-platform -p '{"spec": {"sessionAffinity": "ClientIP"}}'

Confirm we can access the engine

To be continued…

What we have achieved so far. We have installed a bare metal Kubernetes cluster with k0s, we have a working PostgreSQL cluster and a working Camunda 7.17 installation.
Let’s fill it with life and continue to the next article Docker, Kubernetes, and Co. – part II

You may also want to take a look at one of the previously published articles as we will be doing something similar: Camunda on Oracle Cloud – will it work?



Leave a Reply