Chapter 1

• A brief introduction

• Pre-requirements

(auto-generated TOC)

Chapter 2

• What and why of orchestration

• Kubernetes concepts

(auto-generated TOC)

Chapter 3

• Getting a Kubernetes cluster for learning

• Docker Desktop (Windows 10/macOS)

• minikube (Windows 10 Home)

• MicroK8s (Linux)

• Web-based options

• shpod: For a consistent Kubernetes experience ...

(auto-generated TOC)

Chapter 4

• First contact with kubectl

• Running our first containers on Kubernetes

• Accessing logs from the CLI

• Assignment 1: first steps

(auto-generated TOC)

Chapter 5

• Exposing containers

• Kubernetes network model

• Assignment 2: more about deployments

(auto-generated TOC)

Chapter 6

• Our sample application

• Shipping images with a registry

• Running DockerCoins on Kubernetes

• Assignment 3: deploy wordsmith

(auto-generated TOC)

Chapter 7

• Scaling our demo app

• Deploying with YAML

• The Kubernetes Dashboard

• Security implications of kubectl apply

• Daemon sets

• Labels and selectors

• Assignment 4: custom load balancing

(auto-generated TOC)

Chapter 8

• Authoring YAML

• Using server-dry-run and diff

• Rolling updates

• Healthchecks

(auto-generated TOC)

Chapter 9

• Managing configuration

(auto-generated TOC)

Chapter 10

• Exposing HTTP services with Ingress resources

• Ingress in action: NGINX

• Swapping NGINX for Traefik

(auto-generated TOC)

Chapter 11

• Volumes

• Stateful sets

• Running a Consul cluster

• Persistent Volumes Claims

• Local Persistent Volumes

(auto-generated TOC)

Chapter 12

• Kustomize

• Managing stacks with Helm

• Helm chart format

• Creating a basic chart

• Creating better Helm charts

• Helm secrets

(auto-generated TOC)

Chapter 13

• Extending the Kubernetes API

• Operators

• Owners and dependents

(auto-generated TOC)

Chapter 14

• Centralized logging

• Collecting metrics with Prometheus

(auto-generated TOC)

Chapter 15

• Resource Limits

• Defining min, max, and default resources

• Namespace quotas

• Limiting resources in practice

• Checking pod and node resource usage

(auto-generated TOC)

Chapter 16

• Cluster sizing

• The Horizontal Pod Autoscaler

(auto-generated TOC)

Chapter 17

• Declarative vs imperative

• Kubernetes Management Approaches

• Recording deployment actions

• Git-based workflows

(auto-generated TOC)

Chapter 18

• Building images with the Docker Engine

• Building images with Kaniko

(auto-generated TOC)

Chapter 19

• Building our own cluster

• Adding nodes to the cluster

• API server availability

• Static pods

(auto-generated TOC)

Chapter 20

• Owners and dependents

• Exposing HTTP services with Ingress resources

• Upgrading clusters

• Backing up clusters

• The Cloud Controller Manager

(auto-generated TOC)

Chapter 21

• Namespaces

• Controlling a Kubernetes cluster remotely

• Accessing internal services

• Accessing the API with kubectl proxy

(auto-generated TOC)

Chapter 22

• The Container Network Interface

• Interconnecting clusters

(auto-generated TOC)

Chapter 23

• Network policies

• Authentication and authorization

• Pod Security Policies

• The CSR API

• OpenID Connect

• Securing the control plane

(auto-generated TOC)

Chapter 24

• Next steps

• Links and resources

(auto-generated TOC)

shared/toc.md

A brief introduction

• This was initially written by Jérôme Petazzoni to support in-person, instructor-led workshops and tutorials

• Credit is also due to multiple contributors — thank you!

• I recommend using the Slack Chat to help you ...

• ... And be comfortable spending some time reading the Kubernetes documentation ...

• ... And looking for answers on StackOverflow and other outlets

k8smastery/intro.md

Hands on, you shall practice

• Nobody ever became a Jedi by spending their lives reading Wookiepedia

• Likewise, it will take more than merely reading these slides to make you an expert

• These slides include tons of exercises and examples

• They assume that you have access to a Kubernetes cluster

k8smastery/intro.md

Pre-requirements

• Be comfortable with the UNIX command line

  • navigating directories

  • editing files

  • a little bit of bash-fu (environment variables, loops)

• Some Docker knowledge

  • docker run, docker ps, docker build

  • ideally, you know how to write a Dockerfile and build it
    (even if it's a FROM line and a couple of RUN commands)

• It's totally OK if you are not a Docker expert!

k8smastery/prereqs.md

Hands-on exercises

• The whole workshop is hands-on, with "exercies"

• You are invited to reproduce these exercises with me

• All exercises are identified with a dashed box plus keyboard icon

k8smastery/prereqs.md

What and why of orchestration

• There are many computing orchestrators

• They make decisions about when and where to "do work"

Container orchestrator

• Many open source projects have been created in the last 5 years to:

  • Schedule running of containers on servers

Major container orchestration projects

• Kubernetes, aka K8s

• Docker Swarm (and Swarm classic)

• Apache Mesos/Marathon

• Cloud Foundry

• Amazon ECS (not OSS, AWS-only)

• HashiCorp Nomad

Kubernetes distributions

• Kubernetes "vanilla upstream" (not a distribution)

Kubernetes concepts

• Kubernetes is a container management system

• It runs and manages containerized applications on a cluster (one or more servers)

• Often this is simply called "container orchestration"

• Sometimes shortened to Kube or K8s ("Kay-eights" or "Kates")

k8smastery/concepts-k8s.md

Basic things we can ask Kubernetes to do

Other things that Kubernetes can do for us

• Basic autoscaling

• Blue/green deployment, canary deployment

• Long running services, but also batch (one-off) and CRON-like jobs

• Overcommit our cluster and evict low-priority jobs

• Run services with stateful data (databases etc.)

• Fine-grained access control defining what can be done by whom on which resources

• Integrating third party services (service catalog)

• Automating complex tasks (operators)

k8smastery/concepts-k8s.md

Kubernetes architecture

k8smastery/concepts-k8s.md

Kubernetes architecture

• Ha ha ha ha

• OK, I was trying to scare you, it's much simpler than that ❤️

k8smastery/concepts-k8s.md

Credits

• The first schema is a Kubernetes cluster with storage backed by multi-path iSCSI

  (Courtesy of Yongbok Kim)

• The second one is a simplified representation of a Kubernetes cluster

  (Courtesy of Imesh Gunaratne)

k8smastery/concepts-k8s.md

Kubernetes architecture: the nodes

• The nodes executing our containers run a collection of services:

  • a container Engine (typically Docker)

  • kubelet (the "node agent")

  • kube-proxy (a necessary but not sufficient network component)

• Nodes were formerly called "minions"

  (You might see that word in older articles or documentation)

k8smastery/concepts-k8s.md

Kubernetes architecture: the control plane

• The Kubernetes logic (its "brains") is a collection of services:

  • the API server (our point of entry to everything!)

  • core services like the scheduler and controller manager

  • etcd (a highly available key/value store; the "database" of Kubernetes)

• Together, these services form the control plane of our cluster

• The control plane is also called the "master"

k8smastery/concepts-k8s.md

Interacting with Kubernetes

• We will interact with our Kubernetes cluster through the Kubernetes API

• The Kubernetes API is (mostly) RESTful

• It allows us to create, read, update, delete resources

• A few common resource types are:

  • node (a machine — physical or virtual — in our cluster)

  • pod (group of containers running together on a node)

  • service (stable network endpoint to connect to one or multiple containers)

k8smastery/concepts-k8s.md

Pods

• Pods are a new abstraction!

Credits

• The first diagram is courtesy of Lucas Käldström, in this presentation

  • it's one of the best Kubernetes architecture diagrams available!

• The second diagram is courtesy of Weaveworks

  • a pod can have multiple containers working together

  • IP addresses are associated with pods, not with individual containers

Both diagrams used with permission.

k8smastery/concepts-k8s.md

Getting a Kubernetes cluster for learning

• Best: Get a environment locally

  • Docker Desktop (Win/macOS/Linux), Rancher Desktop (Win/macOS/Linux), or microk8s (Linux)
  • Small setup effort; free; flexible environments
  • Requires 2GB+ of memory

Docker Desktop (Windows 10/macOS)

• Docker Desktop (DD) is great for a local dev/test setup

minikube (Windows 10 Home)

• A good local install option if you can't run Docker Desktop

MicroK8s (Linux)

• Easy install and management of local Kubernetes

MicroK8s Additional Info

• We'll need these later (these are done for us in Docker Desktop and minikube):

• You can also install other plugins this way like microk8s enable dashboard or microk8s enable ingress

k8smastery/install-microk8s.md

MicroK8s Troubleshooting

• Run a check for any config problems

• If you also have Docker installed, you can ignore warnings about iptables and registries

• See troubleshooting site if you have issues

k8smastery/install-microk8s.md

Web-based options

Last choice: Use a browser-based solution

shpod: For a consistent Kubernetes experience ...

• You can use shpod for examples

• shpod provides a shell running in a pod on the cluster

• It comes with many tools pre-installed (helm, stern, curl, jq...)

• These tools are used in many exercises in these slides

• shpod also gives you shell completion and a fancy prompt

• Create it with kubectl apply -f https://k8smastery.com/shpod.yaml

• Attach to shell with kubectl attach --namespace=shpod -ti shpod

• After finishing course kubectl delete -f https://k8smastery.com/shpod.yaml

k8smastery/install-shpod.md

First contact with kubectl

• kubectl is (almost) the only tool we'll need to talk to Kubernetes

• It is a rich CLI tool around the Kubernetes API

  (Everything you can do with kubectl, you can do directly with the API)

kubectl get

• Let's look at our Node resources with kubectl get!

k8s/kubectlget.md

Obtaining machine-readable output

• kubectl get can output JSON, YAML, or be directly formatted

k8s/kubectlget.md

(Ab)using kubectl and jq

• It's super easy to build custom reports

k8s/kubectlget.md

Viewing details

• We can use kubectl get -o yaml to see all available details

• However, YAML output is often simultaneously too much and not enough

• For instance, kubectl get node node1 -o yaml is:

  • too much information (e.g.: list of images available on this node)

  • not enough information (e.g.: doesn't show pods running on this node)

  • difficult to read for a human operator

• For a comprehensive overview, we can use kubectl describe instead

k8s/kubectlget.md

kubectl describe

• kubectl describe needs a resource type and (optionally) a resource name

• It is possible to provide a resource name prefix

  (all matching objects will be displayed)

• kubectl describe will retrieve some extra information about the resource

(We should notice a bunch of control plane pods.)

k8s/kubectlget.md

Type names

• The most common resource names have three forms:

  • singular (e.g. node, service, deployment)

  • plural (e.g. nodes, services, deployments)

  • short (e.g. no, svc, deploy)

• Some resources do not have a short name

• Endpoints only have a plural form

  (because even a single Endpoints resource is actually a list of endpoints)

k8s/kubectlget.md

More get commands: Services

• A service is a stable endpoint to connect to "something"

  (In the initial proposal, they were called "portals")

More get commands: Listing running containers

• Containers are manipulated through pods

• A pod is a group of containers:

  • running together (on the same node)

  • sharing resources (RAM, CPU; but also network, volumes)

Namespaces

• Namespaces allow us to segregate resources

Accessing namespaces

• By default, kubectl uses the default namespace

• We can see resources in all namespaces with --all-namespaces

Here are our system pods!

k8s/kubectlget.md

What are all these control plane pods?

• etcd is our etcd server

• kube-apiserver is the API server

• kube-controller-manager and kube-scheduler are other control plane components

• coredns provides DNS-based service discovery (replacing kube-dns as of 1.11)

• kube-proxy is the (per-node) component managing port mappings and such

• <net name> is the optional (per-node) component managing the network overlay

• the READY column indicates the number of containers in each pod

• Note: this only shows containers, you won't see host svcs (e.g. microk8s)

• Also Note: you may see different namespaces depending on setup

k8s/kubectlget.md

Scoping another namespace

• We can also look at a different namespace (other than default)

k8s/kubectlget.md

Namespaces and other kubectl commands

• We can use -n/--namespace with almost every kubectl command

• Example:

  • kubectl create --namespace=X to create something in namespace X

• We can use -A/--all-namespaces with most commands that manipulate multiple objects

• Examples:

  • kubectl delete can delete resources across multiple namespaces

  • kubectl label can add/remove/update labels across multiple namespaces

k8s/kubectlget.md

Services

• A service is a stable endpoint to connect to "something"

  (In the initial proposal, they were called "portals")

ClusterIP services

• A ClusterIP service is internal, available from the cluster only

• This is useful for introspection from within containers

The command above should either time out, or show an authentication error. Why?

k8s/kubectlget.md

Time out

• Connections to ClusterIP services only work from within the cluster

• If we are outside the cluster, the curl command will probably time out

  (Because the IP address, e.g. 10.96.0.1, isn't routed properly outside the cluster)

• This is the case with most "real" Kubernetes clusters

• To try the connection from within the cluster, we can use shpod

k8s/kubectlget.md

Authentication error

This is what we should see when connecting from within the cluster:

$ curl -k https://10.96.0.1
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
  },
  "code": 403
}

k8s/kubectlget.md

Explanations

• We can see kind, apiVersion, metadata

• These are typical of a Kubernetes API reply

• Because we are talking to the Kubernetes API

• The Kubernetes API tells us "Forbidden"

  (because it requires authentication)

• The Kubernetes API is reachable from within the cluster

  (many apps integrating with Kubernetes will use this)

k8s/kubectlget.md

DNS integration

• Each service also gets a DNS record

• The Kubernetes DNS resolver is available from within pods

  (and sometimes, from within nodes, depending on configuration)

• Code running in pods can connect to services using their name

  (e.g. https://kubernetes/...)

k8s/kubectlget.md

Running our first containers on Kubernetes

• First things first: we cannot run a container

Starting a simple pod with kubectl run

• We need to specify at least a name and the image we want to use

Behind the scenes of kubectl run

• Let's look at the resources that were created by kubectl run

What are these different things?

• A deployment is a high-level construct

  • allows scaling, rolling updates, rollbacks

  • multiple deployments can be used together to implement a canary deployment

  • delegates pods management to replica sets

• A replica set is a low-level construct

  • makes sure that a given number of identical pods are running

  • allows scaling

  • rarely used directly

• Note: A replication controller is the deprecated predecessor of a replica set

k8s/kubectlrun.md

Our pingpong deployment

• kubectl run created a deployment, deployment.apps/pingpong

NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/pingpong   1         1         1            1           10m

• That deployment created a replica set, replicaset.apps/pingpong-xxxxxxxxxx

NAME                                  DESIRED   CURRENT   READY     AGE
replicaset.apps/pingpong-7c8bbcd9bc   1         1         1         10m

• That replica set created a pod, pod/pingpong-xxxxxxxxxx-yyyyy

NAME                            READY     STATUS    RESTARTS   AGE
pod/pingpong-7c8bbcd9bc-6c9qz   1/1       Running   0          10m

• We'll see later how these folks play together for:

  • scaling, high availability, rolling updates

k8s/kubectlrun.md

Viewing container output

• Let's use the kubectl logs command

• We will pass either a pod name, or a type/name

  (E.g. if we specify a deployment or replica set, it will get the first pod in it)

• Unless specified otherwise, it will only show logs of the first container in the pod

  (Good thing there's only one in ours!)

k8s/kubectlrun.md

Streaming logs in real time

• Just like docker logs, kubectl logs supports convenient options:

  • -f/--follow to stream logs in real time (à la tail -f)

  • --tail to indicate how many lines you want to see (from the end)

  • --since to get logs only after a given timestamp

k8s/kubectlrun.md

Scaling our application

• We can create additional copies of our container (I mean, our pod) with kubectl scale

Note: what if we tried to scale replicaset.apps/pingpong-xxxxxxxxxx?

We could! But the deployment would notice it right away, and scale back to the initial level.

k8s/kubectlrun.md

Log streaming

• Let's look again at the output of kubectl logs

  (the one we started before scaling up)

• kubectl logs shows us one line per second

• We could expect 3 lines per second

  (since we should now have 3 pods running ping)

• Let's try to figure out what's happening!

k8s/kubectlrun.md

Streaming logs of multiple pods

• What happens if we restart kubectl logs?

kubectl logs will warn us that multiple pods were found, and that it's showing us only one of them.

Let's leave kubectl logs running while we keep exploring.

k8s/kubectlrun.md

Resilience

• The deployment pingpong watches its replica set

• The replica set ensures that the right number of pods are running

• What happens if pods disappear?

k8s/kubectlrun.md

What happened?

• kubectl delete pod terminates the pod gracefully

  (sending it the TERM signal and waiting for it to shutdown)

• As soon as the pod is in "Terminating" state, the Replica Set replaces it

• But we can still see the output of the "Terminating" pod in kubectl logs

• Until 30 seconds later, when the grace period expires

• The pod is then killed, and kubectl logs exits

k8s/kubectlrun.md

What if we wanted something different?

• What if we wanted to start a "one-shot" container that doesn't get restarted?

• We could use kubectl run --restart=OnFailure or kubectl run --restart=Never

• These commands would create jobs or pods instead of deployments

• Under the hood, kubectl run invokes "generators" to create resource descriptions

• We could also write these resource descriptions ourselves (typically in YAML),
  and create them on the cluster with kubectl apply -f (discussed later)

• With kubectl run --schedule=..., we can also create cronjobs

k8s/kubectlrun.md

Scheduling periodic background work

• A Cron Job is a job that will be executed at specific intervals

  (the name comes from the traditional cronjobs executed by the UNIX crond)

• It requires a schedule, represented as five space-separated fields:

  • minute [0,59]
  • hour [0,23]
  • day of the month [1,31]
  • month of the year [1,12]
  • day of the week ([0,6] with 0=Sunday)

• * means "all valid values"; /N means "every N"

• Example: */3 * * * * means "every three minutes"

k8s/kubectlrun.md

Creating a Cron Job

• Let's create a simple job to be executed every three minutes

• Cron Jobs need to terminate, otherwise they'd run forever

k8s/kubectlrun.md

Cron Jobs in action

• At the specified schedule, the Cron Job will create a Job

• The Job will create a Pod

• The Job will make sure that the Pod completes

  (re-creating another one if it fails, for instance if its node fails)

(It will take a few minutes before the first job is scheduled.)

k8s/kubectlrun.md

What about that deprecation warning?

• As we can see from the previous slide, kubectl run can do many things

• The exact type of resource created is not obvious

• To make things more explicit, it is better to use kubectl create:

  • kubectl create deployment to create a deployment

  • kubectl create job to create a job

  • kubectl create cronjob to run a job periodically
    (since Kubernetes 1.14)

• Eventually, kubectl run will be used only to start one-shot pods

  (see https://github.com/kubernetes/kubernetes/pull/68132)

k8s/kubectlrun.md

Various ways of creating resources

• kubectl run

  • easy way to get started
  • versatile

• kubectl create <resource>

  • explicit, but lacks some features
  • can't create a CronJob before Kubernetes 1.14
  • can't pass command-line arguments to deployments

• kubectl create -f foo.yaml or kubectl apply -f foo.yaml

  • all features are available
  • requires writing YAML

k8s/kubectlrun.md

Viewing logs of multiple pods

• When we specify a deployment name, only one single pod's logs are shown

• We can view the logs of multiple pods by specifying a selector

• A selector is a logic expression using labels

• Conveniently, when you kubectl run somename, the associated objects have a run=somename label

k8s/kubectlrun.md

Streaming logs of multiple pods

• Can we stream the logs of all our pingpong pods?

Note: combining -l and -f is only possible since Kubernetes 1.14!

Let's try to understand why ...

k8s/kubectlrun.md

Shortcomings of kubectl logs

• We don't see which pod sent which log line

• If pods are restarted / replaced, the log stream stops

• If new pods are added, we don't see their logs

• To stream the logs of multiple pods, we need to write a selector

• There are external tools to address these shortcomings

  (e.g.: Stern)

k8s/kubectlrun.md

Accessing logs from the CLI

• The kubectl logs command has limitations:

  • it cannot stream logs from multiple pods at a time

  • when showing logs from multiple pods, it mixes them all together

• We are going to see how to do it better

k8s/logs-cli.md

Doing it manually

• We could (if we were so inclined) write a program or script that would:

  • take a selector as an argument

  • enumerate all pods matching that selector (with kubectl get -l ...)

  • fork one kubectl logs --follow ... command per container

  • annotate the logs (the output of each kubectl logs ... process) with their origin

  • preserve ordering by using kubectl logs --timestamps ... and merge the output

Stern

Stern is an open source project originally by Wercker.

From the README:

Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod. Each result is color coded for quicker debugging.

The query is a regular expression so the pod name can easily be filtered and you don't need to specify the exact id (for instance omitting the deployment id). If a pod is deleted it gets removed from tail and if a new pod is added it automatically gets tailed.

Exactly what we need!

k8s/logs-cli.md

Checking if Stern is installed

• Run stern (without arguments) to check if it's installed:

  $ stern
  Tail multiple pods and containers from Kubernetes
  Usage:
  stern pod-query [flags]

• If it's missing, let's see how to install it

k8s/logs-cli.md

Installing Stern

• Stern is written in Go

• Go programs are usually very easy to install

  (no dependencies, extra libraries to install, etc)

• Binary releases are available on GitHub

• Stern is also available through most package managers

  (e.g. on macOS, we can brew install stern or sudo port install stern) k8s/logs-cli.md

Using Stern

• There are two ways to specify the pods whose logs we want to see:

  • -l followed by a selector expression (like with many kubectl commands)

  • with a "pod query," i.e. a regex used to match pod names

• These two ways can be combined if necessary

k8s/logs-cli.md

Stern convenient options

• The --tail N flag shows the last N lines for each container

  (Instead of showing the logs since the creation of the container)

• The -t / --timestamps flag shows timestamps

• The --all-namespaces flag is self-explanatory

k8s/logs-cli.md

Using Stern with a selector

• When specifying a selector, we can omit the value for a label

• This will match all objects having that label (regardless of the value)

• Everything created with kubectl run has a label run

• Everything created with kubectl create deployment has a label app

• We can use that property to view the logs of all the pods created with kubectl create deployment

Assignment 1: first steps

Answer these questions with the kubectl command you'd use to get the answer:

Cluster inventory

1.1. How many nodes does your cluster have?

1.2. What kernel version and what container engine is each node running?

(answers on next slide)

assignments/01kubectlrun.md

Assignment 1: first steps

Control plane examination

2.1. List only the pods in the kube-system namespace.

2.2. Explain the role of some of these pods.

2.3. If there are few or no pods in kube-system, why could that be?

(answers on next slide)

assignments/01kubectlrun.md

Assignment 1: first steps

Running containers

3.1. Create a deployment using kubectl create that runs the image bretfisher/clock and name it ticktock.

3.2. Start 2 more containers of that image in the ticktock deployment.

3.3. Use a selector to output only the last line of logs of each container.

(answers on next slide)

assignments/01kubectlrun.md

19,000 words

They say, "a picture is worth one thousand words."

The following 19 slides show what really happens when we run:

kubectl run web --image=nginx --replicas=3

k8s/deploymentslideshow.md

Exposing containers

• We can connect to our pods using their IP address

• Then we need to figure out a lot of things:

  • how do we look up the IP address of the pod(s)?

  • how do we connect from outside the cluster?

  • how do we load balance traffic?

  • what if a pod fails?

• Kubernetes has a resource type named Service

• Services address all these questions!

k8smastery/kubectlexpose.md

Services in a nutshell

• Services give us a stable endpoint to connect to a pod or a group of pods

• An easy way to create a service is to use kubectl expose

• If we have a deployment named my-little-deploy, we can run:

  kubectl expose deployment my-little-deploy --port=80

  ... and this will create a service with the same name (my-little-deploy)

• Services are automatically added to an internal DNS zone

  (in the example above, our code can now connect to http://my-little-deploy/)

k8smastery/kubectlexpose.md

Advantages of services

• We don't need to look up the IP address of the pod(s)

  (we resolve the IP address of the service using DNS)

• There are multiple service types; some of them allow external traffic

  (e.g. LoadBalancer and NodePort)

• Services provide load balancing

  (for both internal and external traffic)

• Service addresses are independent from pods' addresses

  (when a pod fails, the service seamlessly sends traffic to its replacement)

k8smastery/kubectlexpose.md

Many kinds and flavors of service

• There are different types of services:

  ClusterIP, NodePort, LoadBalancer, ExternalName

• There are also headless services

• Services can also have optional external IPs

• There is also another resource type called Ingress

  (specifically for HTTP services)

• Wow, that's a lot! Let's start with the basics ...

k8smastery/kubectlexpose.md

ClusterIP

• It's the default service type

• A virtual IP address is allocated for the service

  (in an internal, private range; e.g. 10.96.0.0/12)

• This IP address is reachable only from within the cluster (nodes and pods)

• Our code can connect to the service using the original port number

• Perfect for internal communication, within the cluster

k8smastery/kubectlexpose.md

LoadBalancer

• An external load balancer is allocated for the service

  (typically a cloud load balancer, e.g. ELB on AWS, GLB on GCE ...)

• This is available only when the underlying infrastructure provides some kind of "load balancer as a service"

• Each service of that type will typically cost a little bit of money

  (e.g. a few cents per hour on AWS or GCE)

• Ideally, traffic would flow directly from the load balancer to the pods

• In practice, it will often flow through a NodePort first

k8smastery/kubectlexpose.md

NodePort

• A port number is allocated for the service

  (by default, in the 30000-32767 range)

• That port is made available on all our nodes and anybody can connect to it

  (we can connect to any node on that port to reach the service)

• Our code needs to be changed to connect to that new port number

• Under the hood: kube-proxy sets up a bunch of iptables rules on our nodes

• Sometimes, it's the only available option for external traffic

  (e.g. most clusters deployed with kubeadm or on-premises)

k8smastery/kubectlexpose.md

Running containers with open ports

• Since ping doesn't have anything to connect to, we'll have to run something else

• We could use the nginx official image, but ...

  ... we wouldn't be able to tell the backends from each other!

• We are going to use bretfisher/httpenv, a tiny HTTP server written in Go

• bretfisher/httpenv listens on port 8888

• It serves its environment variables in JSON format

• The environment variables will include HOSTNAME, which will be the pod name

  (and therefore, will be different on each backend)

k8smastery/kubectlexpose.md

Creating a deployment for our HTTP server

• We could do kubectl run httpenv --image=bretfisher/httpenv ...

• But since kubectl run is changing, let's see how to use kubectl create instead

k8smastery/kubectlexpose.md

Exposing our deployment

• We'll create a default ClusterIP service

k8smastery/kubectlexpose.md

Services are layer 4 constructs

• You can assign IP addresses to services, but they are still layer 4

  (i.e. a service is not an IP address; it's an IP address + protocol + port)

• This is caused by the current implementation of kube-proxy

  (it relies on mechanisms that don't support layer 3)

• As a result: you have to indicate the port number for your service

  (with some exceptions, like ExternalName or headless services, covered later)

k8smastery/kubectlexpose.md

Testing our service

• We will now send a few HTTP requests to our pods

k8smastery/kubectlexpose.md

Kubernetes network model

• TL,DR:

  Our cluster (nodes and pods) is one big flat IP network.

Kubernetes network model: the good

• Everything can reach everything

• No address translation

• No port translation

• No new protocol

• The network implementation can decide how to allocate addresses

• IP addresses don't have to be "portable" from a node to another

  (For example, We can use a subnet per node and use a simple routed topology)

• The specification is simple enough to allow many various implementations

k8s/kubenet.md

Kubernetes network model: the less good

• Everything can reach everything

  • if you want security, you need to add network policies

  • the network implementation you use needs to support them

• There are literally dozens of implementations out there

  (15 are listed in the Kubernetes documentation)

• Pods have level 3 (IP) connectivity, but services are level 4 (TCP or UDP)

  (Services map to a single UDP or TCP port; no port ranges or arbitrary IP packets)

• kube-proxy is on the data path when connecting to a pod or container,
  and it's not particularly fast (relies on userland proxying or iptables)

k8s/kubenet.md

Kubernetes network model: in practice

• The nodes we are using have been set up to use kubenet, Calico, or something else

• Don't worry about the warning about kube-proxy performance

• Unless you:

  • routinely saturate 10G network interfaces
  • count packet rates in millions per second
  • run high-traffic VOIP or gaming platforms
  • do weird things that involve millions of simultaneous connections
    (in which case you're already familiar with kernel tuning)

• If necessary, there are alternatives to kube-proxy; e.g. kube-router

k8s/kubenet.md

Assignment 2: more about deployments

1. Create a deployment called littletomcat using the tomcat image.

2. What command will help you get the IP address of that Tomcat server?

3. What steps would you take to ping it from another container?

   (Use the shpod environment if necessary.)

4. What command would delete the running pod inside that deployment?

5. What happens if we delete the pod that holds Tomcat, while the ping is running?

(answers on next two slides)

assignments/02kubectlexpose.md

Assignment 2: first service

1. What command can give our Tomcat server a stable DNS name and IP address?

   (An address that doesn't change when something bad happens to the container.)

2. What commands would you run to curl Tomcat with that DNS address?

   (Use the shpod environment if necessary.)

3. If we delete the pod that holds Tomcat, does the IP address still work?

(answers on next slide)

assignments/02kubectlexpose.md

What's this application?

DockerCoins in the microservices era

• DockerCoins is made of 5 services:

  • rng = web service generating random bytes

  • hasher = web service computing hash of POSTed data

  • worker = background process calling rng and hasher

  • webui = web interface to watch progress

  • redis = data store (holds a counter updated by worker)

• These 5 services are visible in the application's Compose file, dockercoins-compose.yml

shared/sampleapp.md

How DockerCoins works

• worker invokes web service rng to generate random bytes

• worker invokes web service hasher to hash these bytes

• worker does this in an infinite loop

• Every second, worker updates redis to indicate how many loops were done

• webui queries redis, and computes and exposes "hashing speed" in our browser

(See diagram on next slide!)

shared/sampleapp.md

Service discovery in container-land

How does each service find out the address of the other ones?

Example in worker/worker.py

redis = Redis("redis")
def get_random_bytes():
    r = requests.get("http://rng/32")
    return r.content
def hash_bytes(data):
    r = requests.post("http://hasher/",
                      data=data,
                      headers={"Content-Type": "application/octet-stream"})

(Full source code available here)

shared/sampleapp.md

DockerCoins at work

• worker will log HTTP requests to rng and hasher

• rng and hasher will log incoming HTTP requests

• webui will give us a graph on coins mined per second

shared/sampleapp.md

Check out the app in Docker Compose

• Compose is (still) great for local development

• You can test this app if you have Docker and Compose installed

• If not, remember play-with-docker.com

• View the webui on localhost:8000 or click the 8080 link in PWD shared/sampleapp.md

Stopping the application

• If we interrupt Compose (with ^C), it will politely ask the Docker Engine to stop the app

• The Docker Engine will send a TERM signal to the containers

• If the containers do not exit in a timely manner, the Engine sends a KILL signal

Shipping images with a registry

• For development using Docker, it has build, ship, and run features

• Now that we want to run on a cluster, things are different

• Kubernetes doesn't have a build feature built-in

• The way to ship (pull) images to Kubernetes is to use a registry

k8s/shippingimages.md

How Docker registries work (a reminder)

• What happens when we execute docker run alpine ?

• If the Engine needs to pull the alpine image, it expands it into library/alpine

• library/alpine is expanded into index.docker.io/library/alpine

• The Engine communicates with index.docker.io to retrieve library/alpine:latest

• To use something else than index.docker.io, we specify it in the image name

• Examples:

  docker pull gcr.io/google-containers/alpine-with-bash:1.0
  docker build -t registry.mycompany.io:5000/myimage:awesome .
  docker push registry.mycompany.io:5000/myimage:awesome

k8s/shippingimages.md

Building and shipping images

• There are many options!

• Manually:

  • build locally (with docker build or otherwise)

  • push to the registry

• Automatically:

  • build and test locally

  • when ready, commit and push a code repository

  • the code repository notifies an automated build system

  • that system gets the code, builds it, pushes the image to the registry

k8s/shippingimages.md

Which registry do we want to use?

• There are SAAS products like Docker Hub, Quay, GitLab ...

• Each major cloud provider has an option as well

  (ACR on Azure, ECR on AWS, GCR on Google Cloud...)

Running DockerCoins on Kubernetes

• Create one deployment for each component

  (hasher, redis, rng, webui, worker)

• Expose deployments that need to accept connections

  (hasher, redis, rng, webui)

• For redis, we can use the official redis image

• For the 4 others, we need to build images and push them to some registry

k8s/shippingimages.md

Using images from the Docker Hub

• For everyone's convenience, we took care of building DockerCoins images

• We pushed these images to the DockerHub, under the dockercoins user

• These images are tagged with a version number, v0.1

• The full image names are therefore:

  • dockercoins/hasher:v0.1

  • dockercoins/rng:v0.1

  • dockercoins/webui:v0.1

  • dockercoins/worker:v0.1

k8s/buildshiprun-dockerhub.md

Running DockerCoins on Kubernetes

• We can now deploy our code (as well as a redis instance)

k8s/ourapponkube.md

Is this working?

• After waiting for the deployment to complete, let's look at the logs!

  (Hint: use kubectl get deploy -w to watch deployment events)

Connecting containers together

• Three deployments need to be reachable by others: hasher, redis, rng

• worker doesn't need to be exposed

• webui will be dealt with later

k8s/ourapponkube.md

Is this working yet?

• The worker has an infinite loop, that retries 10 seconds after an error

Exposing services for external access

• Now we would like to access the Web UI

• We will expose it with a NodePort

  (just like we did for the registry)

k8s/ourapponkube.md

Accessing the web UI

• We can now connect to any node, on the allocated node port, to view the web UI

Assignment 3: deploy wordsmith

• Let's deploy another application called wordsmith

• Wordsmith has 3 components:

  • a web frontend: bretfisher/wordsmith-web

  • a API backend: bretfisher/wordsmith-words (NOTE: won't run on Raspberry Pi's arm/v7 yet GH Issue)

  • a postgres database: bretfisher/wordsmith-db

• We have built images for these components, and pushed them on the Docker Hub

• We want to deploy all 3 components on Kubernetes

• We want to be able to connect to the web frontend with our browser

assignments/03deploywordsmith.md

Wordsmith details

• Here are all the network flows in the app:

  • the web frontend listens on port 80

  • the web frontend connects to the API at the address http://words:8080

  • the API backend listens on port 8080

  • the API connects to the database with the connection string pgsql://db:5432

  • the database listens on port 5432

assignments/03deploywordsmith.md

Winning conditions

• After deploying and connecting everything together, open the web frontend

• This is what we should see:

  

  (You will probably see a different sentence, though.)

• If you see empty LEGO bricks, something's wrong ...

assignments/03deploywordsmith.md

Scaling things up

• If we reload that page, we get the same sentence

• And that sentence repeats the same adjective and noun anyway

• Can we do better?

• Yes, if we scale up the API backend!

• Try to scale up the API backend and see what happens

assignments/03deploywordsmith.md

Scaling our demo app

• Our ultimate goal is to get more DockerCoins

  (i.e. increase the number of loops per second shown on the web UI)

• Let's look at the architecture again:

  

Adding another worker

• All we have to do is scale the worker Deployment

Adding more workers

• If 2 workers give us 2x speed, what about 3 workers?

Adding even more workers

• Let's see if 10 workers give us 10x speed!

Why are we stuck at 10-12 hashes per second?

• If this was high-quality, production code, we would have instrumentation

  (Datadog, Honeycomb, New Relic, statsd, Sumologic, ...)

• It's not!

• Perhaps we could benchmark our web services?

  (with tools like ab, or even simpler, httping)

k8s/scalingdockercoins.md

Benchmarking our web services

• We want to check hasher and rng

• We are going to use httping

• It's just like ping, but using HTTP GET requests

  (it measures how long it takes to perform one GET request)

• It's used like this:

  httping [-c count] http://host:port/path

• Or even simpler:

  httping ip.ad.dr.ess

• We will use httping on the ClusterIP addresses of our services

k8s/scalingdockercoins.md

Obtaining ClusterIP addresses

• We can simply check the output of kubectl get services

• Or do it programmatically, as in the example below

Now we can access the IP addresses of our services through $HASHER and $RNG.

k8s/scalingdockercoins.md

Checking hasher and rng response times

Let's draw hasty conclusions

• The bottleneck seems to be rng

• What if we don't have enough entropy and can't generate enough random numbers?

• We need to scale out the rng service on multiple machines!

Note: this is a fiction! We have enough entropy. But we need a pretext to scale out.

Deploying with YAML

• So far, we created resources with the following commands:

  • kubectl run

  • kubectl create deployment

  • kubectl expose

• We can also create resources directly with YAML manifests

k8s/yamldeploy.md

kubectl apply vs create

• kubectl create -f whatever.yaml

  • creates resources if they don't exist

  • if resources already exist, don't alter them
    (and display error message)

• kubectl apply -f whatever.yaml

  • creates resources if they don't exist

  • if resources already exist, update them
    (to match the definition provided by the YAML file)

  • stores the manifest as an annotation in the resource

k8s/yamldeploy.md

Creating multiple resources

• The manifest can contain multiple resources separated by ---

 kind: ...
 apiVersion: ...
 metadata:
   name: ...
   ...
 spec:
   ...
 ---
 kind: ...
 apiVersion: ...
 metadata:
   name: ...
   ...
 spec: 
   ...

k8s/yamldeploy.md

Creating multiple resources

• The manifest can also contain a list of resources

 apiVersion: v1
 kind: List
 items:
 - kind: ...
   apiVersion: ...
   ...
 - kind: ...
   apiVersion: ...
   ...

k8s/yamldeploy.md

Deploying DockerCoins with YAML

• Here's a YAML manifest with all the resources for DockerCoins

  (Deployments and Services)

• We can use it if we need to deploy or redeploy DockerCoins

• Yes YAML file commands can use URL's!

k8s/yamldeploy.md

Apply errors for create or run resources

• Note the warnings if you already had the resources created

• This is because we didn't use apply before

• This is OK for us learning, so ignore the warnings

• Generally in production you want to stick with one method or the other

k8s/yamldeploy.md

Deleting resources

• We can also use a YAML file to delete resources

• kubectl delete -f ... will delete all the resources mentioned in a YAML file

  (useful to clean up everything that was created by kubectl apply -f ...)

• The definitions of the resources don't matter

  (just their kind, apiVersion, and name)

k8s/yamldeploy.md

Pruning¹ resources

• We can also tell kubectl to remove old resources

• This is done with kubectl apply -f ... --prune

• It will remove resources that don't exist in the YAML file(s)

• But only if they were created with kubectl apply in the first place

  (technically, if they have an annotation kubectl.kubernetes.io/last-applied-configuration)

¹If English is not your first language: to prune means to remove dead or overgrown branches in a tree, to help it to grow.

k8s/yamldeploy.md

YAML as source of truth

• Imagine the following workflow:

  • do not use kubectl run, kubectl create deployment, kubectl expose ...

  • define everything with YAML

  • kubectl apply -f ... --prune --all that YAML

  • keep that YAML under version control

  • enforce all changes to go through that YAML (e.g. with pull requests)

• Our version control system now has a full history of what we deploy

• Compares to "Infrastructure-as-Code", but for app deployments

k8s/yamldeploy.md