Taint, Tolerations and Node Affinity

This is one of interesting topics of Kubernetes. It is about what pods can be loaded into a node.

Let me share a small example where in there is a person and a bug. Few bug tries to land on the person, the person is tainted with a repellant. Some bugs might be intolerant towards the taint and move away. Whereas some bug might land on the person as they are tolerant to the taint.

Taint

Moving forward we will corelate the example wherein person will be node and bug will pod. Remember taint is at node level and tolerations are at pod level.

Scenario #1:

We have 3 nodes Node1, Node2 and Node3. We also have 4 pods A, B, C and D. By default, Kubernetes scheduler will schedule pods equally across nodes based on availability.

Scenario #2:

We are tainting a node1 with app=blue. Now all pods are intolerant towards Node1 and will not be scheduled in it.

Command: kubectl taint nodes node1 key1=value1:NoSchedule
kubectl taint nodes node1 app=blue:NoSchedule        

Taint Effects:

1. NoSchedule – Does not schedule any pods which cannot tolerate this effect, existing pods without toleration will remain in the same node and will not be rescheduled.
2. PreferNoSchedule- Try not to schedule any pods which cannot tolerate this effect, if not schedule one.
3. NoExecute- Does not schedule any pods which cannot tolerate this effect, existing pods without toleration will rescheduled to different node.

Tolerations:

Scenario #3:

We are setting toleration for Pod A with app=blue inside the pod’s yaml. So this makes Pod A tolerant towards Node1 and the scheduler schedules it to Node1 and the rest of the nodes are scheduled based on nodes availability.

  apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  tolerations:
  - key: "app"
    operator: "Equal"
    value: "blue"
    effect: "NoSchedule"        

Note: Remember a node with toleration can still be scheduled on other nodes which does not have any taint applied to it as well. It is with the scheduler to decide which node a pod is scheduled. Taints and tolerations does not guarantee that only pods with tolerations be scheduled to tainted nodes. Hence for this purpose Kubernetes provides something called Node Affinity. 

There are additional properties like nodename which tells the scheduler to load the pod into a specific node. Nodeselector lets you load pods on to a node based on the labels of a node.

Node Affinity:

Node affinity allows a pod to be scheduled to a specific node based on the matching properties set on the both pod and node.

Node affinity types:

requiredDuringSchedulingIgnoredDuringExecution – Set of rules that should be met by pods while scheduling and ignored while pod execution.

preferredDuringSchedulingIgnoredDuringExecution - Set of rules that should be tried to meet by pods while scheduling and ignored while pod execution.

apiVersion: v1
kind: Pod
metadata:
  name: with-node-affinity
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/e2e-az-name
            operator: In
            values:
            - e2e-az1
            - e2e-az2
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: another-node-label-key
            operator: In
            values:
            - another-node-label-value
  containers:
  - name: with-node-affinity
    image: k8s.gcr.io/pause:2.0        

Anti-affinity

Anti-affinity allow you to constrain which nodes your pod is eligible to be scheduled based on labels on pods that are already running on the node rather than based on labels on nodes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-cache
spec:
  selector:
    matchLabels:
      app: store
  replicas: 3
  template:
    metadata:
      labels:
        app: store
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - store
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: redis-server
        image: redis:3.2-alpine        

This is all about the topic of Taints, Tolerations and Node Affinity.

Summary:

1.     Node affinity attracts a set of pods to a node based on the property that is set.

2.     Taint repels pods from a node based on the taint set on a node

3.     Tolerations allows pod to be scheduled on a node with matching taints

4.     Taints and Tolerations work together to ensure only appropriate pods are scheduled on the nodes