A brief introduction

• This was initially written by Jérôme Petazzoni to support in-person, instructor-led workshops and tutorials

• Credit is also due to multiple contributors — thank you!

• You can also follow along on your own, at your own pace

• We included as much information as possible in these slides

• We recommend having a mentor to help you ...

• ... Or be comfortable spending some time reading the Kubernetes documentation ...

• ... And looking for answers on StackOverflow and other outlets

k8s/intro.md

Accessing these slides now

• We recommend that you open these slides in your browser:

  http://container.training/

• Use arrows to move to next/previous slide

  (up, down, left, right, page up, page down)

• Type a slide number + ENTER to go to that slide

• The slide number is also visible in the URL bar

  (e.g. .../#123 for slide 123)

shared/about-slides.md

Accessing these slides later

• Slides will remain online so you can review them later if needed

  (let's say we'll keep them online at least 1 year, how about that?)

• You can download the slides using that URL:

  http://container.training/slides.zip

  (then open the file kube-selfpaced.yml.html)

• You will to find new versions of these slides on:

  https://container.training/

shared/about-slides.md

These slides are open source

• You are welcome to use, re-use, share these slides

• These slides are written in markdown

• The sources of these slides are available in a public GitHub repository:

  https://github.com/jpetazzo/container.training

• Typos? Mistakes? Questions? Feel free to hover over the bottom of the slide ...

👇 Try it! The source file will be shown and you can view it on GitHub and fork and edit it.

shared/about-slides.md

Chapter 1

• Pre-requirements

• Our sample application

• Kubernetes concepts

(auto-generated TOC)

Chapter 2

• First contact with kubectl

• Running our first containers on Kubernetes

• Accessing logs from the CLI

• Declarative vs imperative

(auto-generated TOC)

Chapter 3

• Kubernetes network model

• Exposing containers

• Shipping images with a registry

• Running DockerCoins on Kubernetes

• Deploying with YAML

(auto-generated TOC)

Chapter 4

• Setting up Kubernetes

• The Kubernetes Dashboard

• Security implications of kubectl apply

• Scaling our demo app

• Daemon sets

• Labels and selectors

• Authoring YAML

(auto-generated TOC)

Chapter 5

• Rolling updates

• Healthchecks

• Recording deployment actions

(auto-generated TOC)

Chapter 6

• Namespaces

• Controlling a Kubernetes cluster remotely

• Accessing internal services

• Accessing the API with kubectl proxy

(auto-generated TOC)

Chapter 7

• Exposing HTTP services with Ingress resources

• Kustomize

• Managing stacks with Helm

• Helm chart format

• Creating a basic chart

• Creating better Helm charts

• Helm secrets

(auto-generated TOC)

Chapter 8

• Network policies

• Authentication and authorization

• Pod Security Policies

• The CSR API

• OpenID Connect

• Securing the control plane

(auto-generated TOC)

Chapter 9

• Volumes

• Building images with the Docker Engine

• Building images with Kaniko

(auto-generated TOC)

Chapter 10

• Managing configuration

• Stateful sets

• Running a Consul cluster

• Persistent Volumes Claims

• Local Persistent Volumes

• Highly available Persistent Volumes

(auto-generated TOC)

Chapter 11

• Centralized logging

• Collecting metrics with Prometheus

• Resource Limits

• Defining min, max, and default resources

• Namespace quotas

• Limiting resources in practice

• Checking pod and node resource usage

• Cluster sizing

• The Horizontal Pod Autoscaler

(auto-generated TOC)

Chapter 12

• Extending the Kubernetes API

• Operators

• Owners and dependents

(auto-generated TOC)

Chapter 13

• Building our own cluster

• Adding nodes to the cluster

• The Container Network Interface

• API server availability

• Static pods

(auto-generated TOC)

Chapter 14

• Upgrading clusters

• Backing up clusters

• The Cloud Controller Manager

• Git-based workflows

(auto-generated TOC)

Chapter 15

• Last words

• Links and resources

(auto-generated TOC)

shared/toc.md

Pre-requirements

• Be comfortable with the UNIX command line

  • navigating directories

  • editing files

  • a little bit of bash-fu (environment variables, loops)

• Some Docker knowledge

  • docker run, docker ps, docker build

  • ideally, you know how to write a Dockerfile and build it
    (even if it's a FROM line and a couple of RUN commands)

• It's totally OK if you are not a Docker expert!

shared/prereqs.md

Hands-on sections

• The whole workshop is hands-on

• We are going to build, ship, and run containers!

• You are invited to reproduce all the demos

• All hands-on sections are clearly identified, like the gray rectangle below

shared/prereqs.md

Doing or re-doing the workshop on your own?

• Use something like Play-With-Docker or Play-With-Kubernetes

  Zero setup effort; but environment are short-lived and might have limited resources

• Create your own cluster (local or cloud VMs)

  Small setup effort; small cost; flexible environments

• Create a bunch of clusters for you and your friends (instructions)

  Bigger setup effort; ideal for group training

shared/connecting.md

For a consistent Kubernetes experience ...

• If you are using your own Kubernetes cluster, you can use shpod

• shpod provides a shell running in a pod on your own cluster

• It comes with many tools pre-installed (helm, stern...)

• These tools are used in many exercises in these slides

• shpod also gives you completion and a fancy prompt

shared/connecting.md

We will (mostly) interact with node1 only

These remarks apply only when using multiple nodes, of course.

• Unless instructed, all commands must be run from the first VM, node1

• We will only check out/copy the code on node1

• During normal operations, we do not need access to the other nodes

• If we had to troubleshoot issues, we would use a combination of:

  • SSH (to access system logs, daemon status...)

  • Docker API (to check running containers and container engine status)

shared/connecting.md

Terminals

Once in a while, the instructions will say:
"Open a new terminal."

There are multiple ways to do this:

• create a new window or tab on your machine, and SSH into the VM;

• use screen or tmux on the VM and open a new window from there.

You are welcome to use the method that you feel the most comfortable with.

shared/connecting.md

Tmux cheatsheet

Tmux is a terminal multiplexer like screen.

You don't have to use it or even know about it to follow along.
But some of us like to use it to switch between terminals.
It has been preinstalled on your workshop nodes.

• Ctrl-b c → creates a new window
• Ctrl-b n → go to next window
• Ctrl-b p → go to previous window
• Ctrl-b " → split window top/bottom
• Ctrl-b % → split window left/right
• Ctrl-b Alt-1 → rearrange windows in columns
• Ctrl-b Alt-2 → rearrange windows in rows
• Ctrl-b arrows → navigate to other windows
• Ctrl-b d → detach session
• tmux attach → reattach to session

shared/connecting.md

Versions installed

• Kubernetes 1.17.2
• Docker Engine 19.03.5
• Docker Compose 1.24.1

k8s/versions-k8s.md

Kubernetes versioning and cadence

• Kubernetes versions are expressed using semantic versioning

  (a Kubernetes version is expressed as MAJOR.MINOR.PATCH)

• There is a new patch release whenever needed

  (generally, there is about 2 to 4 weeks between patch releases, except when a critical bug or vulnerability is found: in that case, a patch release will follow as fast as possible)

• There is a new minor release approximately every 3 months

• At any given time, 3 minor releases are maintained

  (in other words, a given minor release is maintained about 9 months)

k8s/versions-k8s.md

Kubernetes version compatibility

Should my version of kubectl match exactly my cluster version?

• kubectl can be up to one minor version older or newer than the cluster

  (if cluster version is 1.15.X, kubectl can be 1.14.Y, 1.15.Y, or 1.16.Y)

• Things might work with larger version differences

  (but they will probably fail randomly, so be careful)

• This is an example of an error indicating version compability issues:

  error: SchemaError(io.k8s.api.autoscaling.v2beta1.ExternalMetricStatus):
  invalid object doesn't have additional properties

• Check the documentation for the whole story about compatibility

k8s/versions-k8s.md

What's this application?

DockerCoins in the microservices era

• DockerCoins is made of 5 services:

  • rng = web service generating random bytes

  • hasher = web service computing hash of POSTed data

  • worker = background process calling rng and hasher

  • webui = web interface to watch progress

  • redis = data store (holds a counter updated by worker)

• These 5 services are visible in the application's Compose file, dockercoins-compose.yml

shared/sampleapp.md

How DockerCoins works

• worker invokes web service rng to generate random bytes

• worker invokes web service hasher to hash these bytes

• worker does this in an infinite loop

• Every second, worker updates redis to indicate how many loops were done

• webui queries redis, and computes and exposes "hashing speed" in our browser

(See diagram on next slide!)

shared/sampleapp.md

Service discovery in container-land

How does each service find out the address of the other ones?

Example in worker/worker.py

redis = Redis("redis")
def get_random_bytes():
    r = requests.get("http://rng/32")
    return r.content
def hash_bytes(data):
    r = requests.post("http://hasher/",
                      data=data,
                      headers={"Content-Type": "application/octet-stream"})

(Full source code available here)

shared/sampleapp.md

DockerCoins at work

• worker will log HTTP requests to rng and hasher

• rng and hasher will log incoming HTTP requests

• webui will give us a graph on coins mined per second

shared/sampleapp.md

Check out the app in Docker Compose

• Compose is (still) great for local development

• You can test this app if you have Docker and Compose installed

• If not, remember play-with-docker.com

• View the webui on localhost:8000 or click the 8080 link in PWD shared/sampleapp.md

Stopping the application

• If we interrupt Compose (with ^C), it will politely ask the Docker Engine to stop the app

• The Docker Engine will send a TERM signal to the containers

• If the containers do not exit in a timely manner, the Engine sends a KILL signal

Kubernetes concepts

• Kubernetes is a container management system

• It runs and manages containerized applications on a cluster

Basic things we can ask Kubernetes to do

Other things that Kubernetes can do for us

• Autoscaling

  (straightforward on CPU; more complex on other metrics)

• Resource management and scheduling

  (reserve CPU/RAM for containers; placement constraints)

• Advanced rollout patterns

  (blue/green deployment, canary deployment)

k8s/concepts-k8s.md

More things that Kubernetes can do for us

• Batch jobs

  (one-off; parallel; also cron-style periodic execution)

• Fine-grained access control

  (defining what can be done by whom on which resources)

• Stateful services

  (databases, message queues, etc.)

• Automating complex tasks with operators

  (e.g. database replication, failover, etc.)

k8s/concepts-k8s.md

Kubernetes architecture

k8s/concepts-k8s.md

Kubernetes architecture

• Ha ha ha ha

• OK, I was trying to scare you, it's much simpler than that ❤️

k8s/concepts-k8s.md

Credits

• The first schema is a Kubernetes cluster with storage backed by multi-path iSCSI

  (Courtesy of Yongbok Kim)

• The second one is a simplified representation of a Kubernetes cluster

  (Courtesy of Imesh Gunaratne)

k8s/concepts-k8s.md

Kubernetes architecture: the nodes

• The nodes executing our containers run a collection of services:

  • a container Engine (typically Docker)

  • kubelet (the "node agent")

  • kube-proxy (a necessary but not sufficient network component)

• Nodes were formerly called "minions"

  (You might see that word in older articles or documentation)

k8s/concepts-k8s.md

Kubernetes architecture: the control plane

• The Kubernetes logic (its "brains") is a collection of services:

  • the API server (our point of entry to everything!)

  • core services like the scheduler and controller manager

  • etcd (a highly available key/value store; the "database" of Kubernetes)

• Together, these services form the control plane of our cluster

• The control plane is also called the "master"

k8s/concepts-k8s.md

Interacting with Kubernetes

• We will interact with our Kubernetes cluster through the Kubernetes API

• The Kubernetes API is (mostly) RESTful

• It allows us to create, read, update, delete resources

• A few common resource types are:

  • node (a machine — physical or virtual — in our cluster)

  • pod (group of containers running together on a node)

  • service (stable network endpoint to connect to one or multiple containers)

k8s/concepts-k8s.md

Scaling

• How would we scale the pod shown on the previous slide?

• Do create additional pods

  • each pod can be on a different node

  • each pod will have its own IP address

• Do not add more NGINX containers in the pod

  • all the NGINX containers would be on the same node

  • they would all have the same IP address
    (resulting in Address alreading in use errors)

k8s/concepts-k8s.md

Together or separate

• Should we put e.g. a web application server and a cache together?
  ("cache" being something like e.g. Memcached or Redis)

• Putting them in the same pod means:

  • they have to be scaled together

  • they can communicate very efficiently over localhost

• Putting them in different pods means:

  • they can be scaled separately

  • they must communicate over remote IP addresses
    (incurring more latency, lower performance)

• Both scenarios can make sense, depending on our goals

k8s/concepts-k8s.md

Credits

• The first diagram is courtesy of Lucas Käldström, in this presentation

  • it's one of the best Kubernetes architecture diagrams available!

• The second diagram is courtesy of Weave Works

  • a pod can have multiple containers working together

  • IP addresses are associated with pods, not with individual containers

Both diagrams used with permission.

k8s/concepts-k8s.md

First contact with kubectl

• kubectl is (almost) the only tool we'll need to talk to Kubernetes

• It is a rich CLI tool around the Kubernetes API

  (Everything you can do with kubectl, you can do directly with the API)

kubectl get

• Let's look at our Node resources with kubectl get!

k8s/kubectlget.md

Obtaining machine-readable output

• kubectl get can output JSON, YAML, or be directly formatted

k8s/kubectlget.md

(Ab)using kubectl and jq

• It's super easy to build custom reports

k8s/kubectlget.md

Viewing details

• We can use kubectl get -o yaml to see all available details

• However, YAML output is often simultaneously too much and not enough

• For instance, kubectl get node node1 -o yaml is:

  • too much information (e.g.: list of images available on this node)

  • not enough information (e.g.: doesn't show pods running on this node)

  • difficult to read for a human operator

• For a comprehensive overview, we can use kubectl describe instead

k8s/kubectlget.md

kubectl describe

• kubectl describe needs a resource type and (optionally) a resource name

• It is possible to provide a resource name prefix

  (all matching objects will be displayed)

• kubectl describe will retrieve some extra information about the resource

(We should notice a bunch of control plane pods.)

k8s/kubectlget.md

Type names

• The most common resource names have three forms:

  • singular (e.g. node, service, deployment)

  • plural (e.g. nodes, services, deployments)

  • short (e.g. no, svc, deploy)

• Some resources do not have a short name

• Endpoints only have a plural form

  (because even a single Endpoints resource is actually a list of endpoints)

k8s/kubectlget.md

More get commands: Services

• A service is a stable endpoint to connect to "something"

  (In the initial proposal, they were called "portals")

More get commands: Listing running containers

• Containers are manipulated through pods

• A pod is a group of containers:

  • running together (on the same node)

  • sharing resources (RAM, CPU; but also network, volumes)

Namespaces

• Namespaces allow us to segregate resources

Accessing namespaces

• By default, kubectl uses the default namespace

• We can see resources in all namespaces with --all-namespaces

Here are our system pods!

k8s/kubectlget.md

What are all these control plane pods?

• etcd is our etcd server

• kube-apiserver is the API server

• kube-controller-manager and kube-scheduler are other control plane components

• coredns provides DNS-based service discovery (replacing kube-dns as of 1.11)

• kube-proxy is the (per-node) component managing port mappings and such

• <net name> is the optional (per-node) component managing the network overlay

• the READY column indicates the number of containers in each pod

• Note: this only shows containers, you won't see host svcs (e.g. microk8s)

• Also Note: you may see different namespaces depending on setup

k8s/kubectlget.md

Scoping another namespace

• We can also look at a different namespace (other than default)

k8s/kubectlget.md

Namespaces and other kubectl commands

• We can use -n/--namespace with almost every kubectl command

• Example:

  • kubectl create --namespace=X to create something in namespace X

• We can use -A/--all-namespaces with most commands that manipulate multiple objects

• Examples:

  • kubectl delete can delete resources across multiple namespaces

  • kubectl label can add/remove/update labels across multiple namespaces

k8s/kubectlget.md

Services

• A service is a stable endpoint to connect to "something"

  (In the initial proposal, they were called "portals")

ClusterIP services

• A ClusterIP service is internal, available from the cluster only

• This is useful for introspection from within containers

The command above should either time out, or show an authentication error. Why?

k8s/kubectlget.md

Time out

• Connections to ClusterIP services only work from within the cluster

• If we are outside the cluster, the curl command will probably time out

  (Because the IP address, e.g. 10.96.0.1, isn't routed properly outside the cluster)

• This is the case with most "real" Kubernetes clusters

• To try the connection from within the cluster, we can use shpod

k8s/kubectlget.md

Authentication error

This is what we should see when connecting from within the cluster:

$ curl -k https://10.96.0.1
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
  },
  "code": 403
}

k8s/kubectlget.md

Explanations

• We can see kind, apiVersion, metadata

• These are typical of a Kubernetes API reply

• Because we are talking to the Kubernetes API

• The Kubernetes API tells us "Forbidden"

  (because it requires authentication)

• The Kubernetes API is reachable from within the cluster

  (many apps integrating with Kubernetes will use this)

k8s/kubectlget.md

DNS integration

• Each service also gets a DNS record

• The Kubernetes DNS resolver is available from within pods

  (and sometimes, from within nodes, depending on configuration)

• Code running in pods can connect to services using their name

  (e.g. https://kubernetes/...)

k8s/kubectlget.md

Running our first containers on Kubernetes

• First things first: we cannot run a container

Starting a simple pod with kubectl run

• We need to specify at least a name and the image we want to use

Behind the scenes of kubectl run

• Let's look at the resources that were created by kubectl run

What are these different things?

• A deployment is a high-level construct

  • allows scaling, rolling updates, rollbacks

  • multiple deployments can be used together to implement a canary deployment

  • delegates pods management to replica sets

• A replica set is a low-level construct

  • makes sure that a given number of identical pods are running

  • allows scaling

  • rarely used directly

• Note: A replication controller is the deprecated predecessor of a replica set

k8s/kubectlrun.md

Our pingpong deployment

• kubectl run created a deployment, deployment.apps/pingpong

NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/pingpong   1         1         1            1           10m

• That deployment created a replica set, replicaset.apps/pingpong-xxxxxxxxxx

NAME                                  DESIRED   CURRENT   READY     AGE
replicaset.apps/pingpong-7c8bbcd9bc   1         1         1         10m

• That replica set created a pod, pod/pingpong-xxxxxxxxxx-yyyyy

NAME                            READY     STATUS    RESTARTS   AGE
pod/pingpong-7c8bbcd9bc-6c9qz   1/1       Running   0          10m

• We'll see later how these folks play together for:

  • scaling, high availability, rolling updates

k8s/kubectlrun.md

Viewing container output

• Let's use the kubectl logs command

• We will pass either a pod name, or a type/name

  (E.g. if we specify a deployment or replica set, it will get the first pod in it)

• Unless specified otherwise, it will only show logs of the first container in the pod

  (Good thing there's only one in ours!)

k8s/kubectlrun.md

Streaming logs in real time

• Just like docker logs, kubectl logs supports convenient options:

  • -f/--follow to stream logs in real time (à la tail -f)

  • --tail to indicate how many lines you want to see (from the end)

  • --since to get logs only after a given timestamp

k8s/kubectlrun.md

Scaling our application

• We can create additional copies of our container (I mean, our pod) with kubectl scale

Note: what if we tried to scale replicaset.apps/pingpong-xxxxxxxxxx?

We could! But the deployment would notice it right away, and scale back to the initial level.

k8s/kubectlrun.md

Log streaming

• Let's look again at the output of kubectl logs

  (the one we started before scaling up)

• kubectl logs shows us one line per second

• We could expect 3 lines per second

  (since we should now have 3 pods running ping)

• Let's try to figure out what's happening!

k8s/kubectlrun.md

Streaming logs of multiple pods

• What happens if we restart kubectl logs?

kubectl logs will warn us that multiple pods were found, and that it's showing us only one of them.

Let's leave kubectl logs running while we keep exploring.

k8s/kubectlrun.md

Resilience

• The deployment pingpong watches its replica set

• The replica set ensures that the right number of pods are running

• What happens if pods disappear?

k8s/kubectlrun.md

What happened?

• kubectl delete pod terminates the pod gracefully

  (sending it the TERM signal and waiting for it to shutdown)

• As soon as the pod is in "Terminating" state, the Replica Set replaces it

• But we can still see the output of the "Terminating" pod in kubectl logs

• Until 30 seconds later, when the grace period expires

• The pod is then killed, and kubectl logs exits

k8s/kubectlrun.md

What if we wanted something different?

• What if we wanted to start a "one-shot" container that doesn't get restarted?

• We could use kubectl run --restart=OnFailure or kubectl run --restart=Never

• These commands would create jobs or pods instead of deployments

• Under the hood, kubectl run invokes "generators" to create resource descriptions

• We could also write these resource descriptions ourselves (typically in YAML),
  and create them on the cluster with kubectl apply -f (discussed later)

• With kubectl run --schedule=..., we can also create cronjobs

k8s/kubectlrun.md

Scheduling periodic background work

• A Cron Job is a job that will be executed at specific intervals

  (the name comes from the traditional cronjobs executed by the UNIX crond)

• It requires a schedule, represented as five space-separated fields:

  • minute [0,59]
  • hour [0,23]
  • day of the month [1,31]
  • month of the year [1,12]
  • day of the week ([0,6] with 0=Sunday)

• * means "all valid values"; /N means "every N"

• Example: */3 * * * * means "every three minutes"

k8s/kubectlrun.md

Creating a Cron Job

• Let's create a simple job to be executed every three minutes

• Cron Jobs need to terminate, otherwise they'd run forever

k8s/kubectlrun.md

Cron Jobs in action

• At the specified schedule, the Cron Job will create a Job

• The Job will create a Pod

• The Job will make sure that the Pod completes

  (re-creating another one if it fails, for instance if its node fails)

(It will take a few minutes before the first job is scheduled.)

k8s/kubectlrun.md

What about that deprecation warning?

• As we can see from the previous slide, kubectl run can do many things

• The exact type of resource created is not obvious

• To make things more explicit, it is better to use kubectl create:

  • kubectl create deployment to create a deployment

  • kubectl create job to create a job

  • kubectl create cronjob to run a job periodically
    (since Kubernetes 1.14)

• Eventually, kubectl run will be used only to start one-shot pods

  (see https://github.com/kubernetes/kubernetes/pull/68132)

k8s/kubectlrun.md

Various ways of creating resources

• kubectl run

  • easy way to get started
  • versatile

• kubectl create <resource>

  • explicit, but lacks some features
  • can't create a CronJob before Kubernetes 1.14
  • can't pass command-line arguments to deployments

• kubectl create -f foo.yaml or kubectl apply -f foo.yaml

  • all features are available
  • requires writing YAML

k8s/kubectlrun.md

Viewing logs of multiple pods

• When we specify a deployment name, only one single pod's logs are shown

• We can view the logs of multiple pods by specifying a selector

• A selector is a logic expression using labels

• Conveniently, when you kubectl run somename, the associated objects have a run=somename label

k8s/kubectlrun.md

Streaming logs of multiple pods

• Can we stream the logs of all our pingpong pods?

Note: combining -l and -f is only possible since Kubernetes 1.14!

Let's try to understand why ...

k8s/kubectlrun.md

Shortcomings of kubectl logs

• We don't see which pod sent which log line

• If pods are restarted / replaced, the log stream stops

• If new pods are added, we don't see their logs

• To stream the logs of multiple pods, we need to write a selector

• There are external tools to address these shortcomings

  (e.g.: Stern)

k8s/kubectlrun.md

Accessing logs from the CLI

• The kubectl logs command has limitations:

  • it cannot stream logs from multiple pods at a time

  • when showing logs from multiple pods, it mixes them all together

• We are going to see how to do it better

k8s/logs-cli.md

Doing it manually

• We could (if we were so inclined) write a program or script that would:

  • take a selector as an argument

  • enumerate all pods matching that selector (with kubectl get -l ...)

  • fork one kubectl logs --follow ... command per container

  • annotate the logs (the output of each kubectl logs ... process) with their origin

  • preserve ordering by using kubectl logs --timestamps ... and merge the output

Stern

Stern is an open source project originally by Wercker.

From the README:

Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod. Each result is color coded for quicker debugging.

The query is a regular expression so the pod name can easily be filtered and you don't need to specify the exact id (for instance omitting the deployment id). If a pod is deleted it gets removed from tail and if a new pod is added it automatically gets tailed.

Exactly what we need!

k8s/logs-cli.md

Checking if Stern is installed

• Run stern (without arguments) to check if it's installed:

  $ stern
  Tail multiple pods and containers from Kubernetes
  Usage:
  stern pod-query [flags]

• If it's missing, let's see how to install it

k8s/logs-cli.md

Installing Stern

• Stern is written in Go

• Go programs are usually very easy to install

  (no dependencies, extra libraries to install, etc)

• Binary releases are available on GitHub

• Stern is also available through most package managers

  (e.g. on macOS, we can brew install stern or sudo port install stern) k8s/logs-cli.md

Using Stern

• There are two ways to specify the pods whose logs we want to see:

  • -l followed by a selector expression (like with many kubectl commands)

  • with a "pod query," i.e. a regex used to match pod names

• These two ways can be combined if necessary

k8s/logs-cli.md

Stern convenient options

• The --tail N flag shows the last N lines for each container

  (Instead of showing the logs since the creation of the container)

• The -t / --timestamps flag shows timestamps

• The --all-namespaces flag is self-explanatory

k8s/logs-cli.md

Using Stern with a selector

• When specifying a selector, we can omit the value for a label

• This will match all objects having that label (regardless of the value)

• Everything created with kubectl run has a label run

• Everything created with kubectl create deployment has a label app

• We can use that property to view the logs of all the pods created with kubectl create deployment

Declarative vs imperative

• Our container orchestrator puts a very strong emphasis on being declarative

• Declarative:

  I would like a cup of tea.

• Imperative:

  Boil some water. Pour it in a teapot. Add tea leaves. Steep for a while. Serve in a cup.

Declarative vs imperative

• What declarative would really be:

  I want a cup of tea, obtained by pouring an infusion¹ of tea leaves in a cup.

Declarative vs imperative

• Imperative systems:

  • simpler

  • if a task is interrupted, we have to restart from scratch

• Declarative systems:

  • if a task is interrupted (or if we show up to the party half-way through), we can figure out what's missing and do only what's necessary

  • we need to be able to observe the system

  • ... and compute a "diff" between what we have and what we want

shared/declarative.md

Declarative vs imperative in Kubernetes

• With Kubernetes, we cannot say: "run this container"

• All we can do is write a spec and push it to the API server

  (for example, by creating a resource like a Pod or a Deployment)

• The API server will validate that spec (and reject it if it's invalid)

• Then it will store it in etcd

• A controller will "notice" that spec and act upon it

k8s/declarative.md

Reconciling state

• Watch for the spec fields in the YAML files later!

• The spec describes how we want the thing to be

• Kubernetes will reconcile the current state with the spec
  (technically, this is done by a number of controllers)

• When we want to change some resource, we update the spec

• Kubernetes will then converge that resource

k8s/declarative.md

19,000 words

They say, "a picture is worth one thousand words."

The following 19 slides show what really happens when we run:

kubectl run web --image=nginx --replicas=3

k8s/deploymentslideshow.md

Kubernetes network model

• TL,DR:

  Our cluster (nodes and pods) is one big flat IP network.

Kubernetes network model: the good

• Everything can reach everything

• No address translation

• No port translation

• No new protocol

• The network implementation can decide how to allocate addresses

• IP addresses don't have to be "portable" from a node to another

  (For example, We can use a subnet per node and use a simple routed topology)

• The specification is simple enough to allow many various implementations

k8s/kubenet.md

Kubernetes network model: the less good

• Everything can reach everything

  • if you want security, you need to add network policies

  • the network implementation you use needs to support them

• There are literally dozens of implementations out there

  (15 are listed in the Kubernetes documentation)

• Pods have level 3 (IP) connectivity, but services are level 4 (TCP or UDP)

  (Services map to a single UDP or TCP port; no port ranges or arbitrary IP packets)

• kube-proxy is on the data path when connecting to a pod or container,
  and it's not particularly fast (relies on userland proxying or iptables)

k8s/kubenet.md

Kubernetes network model: in practice

• The nodes we are using have been set up to use kubenet, Calico, or something else

• Don't worry about the warning about kube-proxy performance

• Unless you:

  • routinely saturate 10G network interfaces
  • count packet rates in millions per second
  • run high-traffic VOIP or gaming platforms
  • do weird things that involve millions of simultaneous connections
    (in which case you're already familiar with kernel tuning)

• If necessary, there are alternatives to kube-proxy; e.g. kube-router

k8s/kubenet.md

Exposing containers

• We can connect to our pods using their IP address

• Then we need to figure out a lot of things:

  • how do we look up the IP address of the pod(s)?

  • how do we connect from outside the cluster?

  • how do we load balance traffic?

  • what if a pod fails?

• Kubernetes has a resource type named Service

• Services address all these questions!

k8s/kubectlexpose.md

Services in a nutshell

• Services give us a stable endpoint to connect to a pod or a group of pods

• An easy way to create a service is to use kubectl expose

• If we have a deployment named my-little-deploy, we can run:

  kubectl expose deployment my-little-deploy --port=80

  ... and this will create a service with the same name (my-little-deploy)

• Services are automatically added to an internal DNS zone

  (in the example above, our code can now connect to http://my-little-deploy/)

k8s/kubectlexpose.md

Advantages of services

• We don't need to look up the IP address of the pod(s)

  (we resolve the IP address of the service using DNS)

• There are multiple service types; some of them allow external traffic

  (e.g. LoadBalancer and NodePort)

• Services provide load balancing

  (for both internal and external traffic)

• Service addresses are independent from pods' addresses

  (when a pod fails, the service seamlessly sends traffic to its replacement)

k8s/kubectlexpose.md

Many kinds and flavors of service

• There are different types of services:

  ClusterIP, NodePort, LoadBalancer, ExternalName

• There are also headless services

• Services can also have optional external IPs

• There is also another resource type called Ingress

  (specifically for HTTP services)

• Wow, that's a lot! Let's start with the basics ...

k8s/kubectlexpose.md

ClusterIP

• It's the default service type

• A virtual IP address is allocated for the service

  (in an internal, private range; e.g. 10.96.0.0/12)

• This IP address is reachable only from within the cluster (nodes and pods)

• Our code can connect to the service using the original port number

• Perfect for internal communication, within the cluster

k8s/kubectlexpose.md

LoadBalancer

• An external load balancer is allocated for the service

  (typically a cloud load balancer, e.g. ELB on AWS, GLB on GCE ...)

• This is available only when the underlying infrastructure provides some kind of "load balancer as a service"

• Each service of that type will typically cost a little bit of money

  (e.g. a few cents per hour on AWS or GCE)

• Ideally, traffic would flow directly from the load balancer to the pods

• In practice, it will often flow through a NodePort first

k8s/kubectlexpose.md

NodePort

• A port number is allocated for the service

  (by default, in the 30000-32767 range)

• That port is made available on all our nodes and anybody can connect to it

  (we can connect to any node on that port to reach the service)

• Our code needs to be changed to connect to that new port number

• Under the hood: kube-proxy sets up a bunch of iptables rules on our nodes

• Sometimes, it's the only available option for external traffic

  (e.g. most clusters deployed with kubeadm or on-premises)

k8s/kubectlexpose.md

Running containers with open ports

• Since ping doesn't have anything to connect to, we'll have to run something else

• We could use the nginx official image, but ...

  ... we wouldn't be able to tell the backends from each other!

• We are going to use bretfisher/httpenv, a tiny HTTP server written in Go

• bretfisher/httpenv listens on port 8888

• It serves its environment variables in JSON format

• The environment variables will include HOSTNAME, which will be the pod name

  (and therefore, will be different on each backend)

k8s/kubectlexpose.md

Creating a deployment for our HTTP server

• We could do kubectl run httpenv --image=bretfisher/httpenv ...

• But since kubectl run is changing, let's see how to use kubectl create instead

k8s/kubectlexpose.md

Exposing our deployment

• We'll create a default ClusterIP service

k8s/kubectlexpose.md

Services are layer 4 constructs

• You can assign IP addresses to services, but they are still layer 4

  (i.e. a service is not an IP address; it's an IP address + protocol + port)

• This is caused by the current implementation of kube-proxy

  (it relies on mechanisms that don't support layer 3)

• As a result: you have to indicate the port number for your service

  (with some exceptions, like ExternalName or headless services, covered later)

k8s/kubectlexpose.md

Testing our service

• We will now send a few HTTP requests to our pods

k8s/kubectlexpose.md

Shipping images with a registry

• For development using Docker, it has build, ship, and run features

• Now that we want to run on a cluster, things are different

• Kubernetes doesn't have a build feature built-in

• The way to ship (pull) images to Kubernetes is to use a registry

k8s/shippingimages.md

How Docker registries work (a reminder)

• What happens when we execute docker run alpine ?

• If the Engine needs to pull the alpine image, it expands it into library/alpine

• library/alpine is expanded into index.docker.io/library/alpine

• The Engine communicates with index.docker.io to retrieve library/alpine:latest

• To use something else than index.docker.io, we specify it in the image name

• Examples:

  docker pull gcr.io/google-containers/alpine-with-bash:1.0
  docker build -t registry.mycompany.io:5000/myimage:awesome .
  docker push registry.mycompany.io:5000/myimage:awesome

k8s/shippingimages.md

Building and shipping images

• There are many options!

• Manually:

  • build locally (with docker build or otherwise)

  • push to the registry

• Automatically:

  • build and test locally

  • when ready, commit and push a code repository

  • the code repository notifies an automated build system

  • that system gets the code, builds it, pushes the image to the registry

k8s/shippingimages.md

Which registry do we want to use?

• There are SAAS products like Docker Hub, Quay, GitLab ...

• Each major cloud provider has an option as well

  (ACR on Azure, ECR on AWS, GCR on Google Cloud...)

Running DockerCoins on Kubernetes

• Create one deployment for each component

  (hasher, redis, rng, webui, worker)

• Expose deployments that need to accept connections

  (hasher, redis, rng, webui)

• For redis, we can use the official redis image

• For the 4 others, we need to build images and push them to some registry

k8s/shippingimages.md

Self-hosting our registry

Note: this section shows how to run the Docker open source registry and use it to ship images on our cluster. While this method works fine, we recommend that you consider using one of the hosted, free automated build services instead. It will be much easier!

If you need to run a registry on premises, this section gives you a starting point, but you will need to make a lot of changes so that the registry is secured, highly available, and so that your build pipeline is automated.

k8s/buildshiprun-selfhosted.md

Using the open source registry

• We need to run a registry container

• It will store images and layers to the local filesystem
  (but you can add a config file to use S3, Swift, etc.)

• Docker requires TLS when communicating with the registry

  • unless for registries on 127.0.0.0/8 (i.e. localhost)

  • or with the Engine flag --insecure-registry

• Our strategy: publish the registry container on a NodePort,
  so that it's available through 127.0.0.1:xxxxx on each node

k8s/buildshiprun-selfhosted.md

Deploying a self-hosted registry

• We will deploy a registry container, and expose it with a NodePort

k8s/buildshiprun-selfhosted.md

Connecting to our registry

• We need to find out which port has been allocated

k8s/buildshiprun-selfhosted.md

Testing our registry

• A convenient Docker registry API route to remember is /v2/_catalog

Testing our local registry

• We can retag a small image, and push it to the registry

k8s/buildshiprun-selfhosted.md

Checking again what's on our local registry

• Let's use the same endpoint as before

The curl command should now output:

{"repositories":["busybox"]}

k8s/buildshiprun-selfhosted.md

Building and pushing our images

• We are going to use a convenient feature of Docker Compose

Let's have a look at the dockercoins.yml file while this is building and pushing.

k8s/buildshiprun-selfhosted.md

version: "3"
services:
  rng:
    build: dockercoins/rng
    image: ${REGISTRY-127.0.0.1:5000}/rng:${TAG-latest}
    deploy:
      mode: global
  ...
  redis:
    image: redis
  ...
  worker:
    build: dockercoins/worker
    image: ${REGISTRY-127.0.0.1:5000}/worker:${TAG-latest}
    ...
    deploy:
      replicas: 10

Just in case you were wondering ... Docker "services" are not Kubernetes "services".

k8s/buildshiprun-selfhosted.md

Checking the content of the registry

• All our images should now be in the registry

In these slides, all the commands to deploy DockerCoins will use a $REGISTRY environment variable, so that we can quickly switch from the self-hosted registry to pre-built images hosted on the Docker Hub. So make sure that this $REGISTRY variable is set correctly when running the exercises! k8s/buildshiprun-selfhosted.md

Using images from the Docker Hub

• For everyone's convenience, we took care of building DockerCoins images

• We pushed these images to the DockerHub, under the dockercoins user

• These images are tagged with a version number, v0.1

• The full image names are therefore:

  • dockercoins/hasher:v0.1

  • dockercoins/rng:v0.1

  • dockercoins/webui:v0.1

  • dockercoins/worker:v0.1

k8s/buildshiprun-dockerhub.md

Running DockerCoins on Kubernetes

• We can now deploy our code (as well as a redis instance)

k8s/ourapponkube.md

Is this working?

• After waiting for the deployment to complete, let's look at the logs!

  (Hint: use kubectl get deploy -w to watch deployment events)

Connecting containers together

• Three deployments need to be reachable by others: hasher, redis, rng

• worker doesn't need to be exposed

• webui will be dealt with later

k8s/ourapponkube.md

Is this working yet?

• The worker has an infinite loop, that retries 10 seconds after an error

Exposing services for external access

• Now we would like to access the Web UI

• We will expose it with a NodePort

  (just like we did for the registry)

k8s/ourapponkube.md

Accessing the web UI

• We can now connect to any node, on the allocated node port, to view the web UI

Deploying with YAML

• So far, we created resources with the following commands:

  • kubectl run

  • kubectl create deployment

  • kubectl expose

• We can also create resources directly with YAML manifests

k8s/yamldeploy.md

kubectl apply vs create

• kubectl create -f whatever.yaml

  • creates resources if they don't exist

  • if resources already exist, don't alter them
    (and display error message)

• kubectl apply -f whatever.yaml

  • creates resources if they don't exist

  • if resources already exist, update them
    (to match the definition provided by the YAML file)

  • stores the manifest as an annotation in the resource

k8s/yamldeploy.md

Creating multiple resources

• The manifest can contain multiple resources separated by ---

 kind: ...
 apiVersion: ...
 metadata:
   name: ...
   ...
 spec:
   ...
 ---
 kind: ...
 apiVersion: ...
 metadata:
   name: ...
   ...
 spec: 
   ...

k8s/yamldeploy.md

Creating multiple resources

• The manifest can also contain a list of resources

 apiVersion: v1
 kind: List
 items:
 - kind: ...
   apiVersion: ...
   ...
 - kind: ...
   apiVersion: ...
   ...

k8s/yamldeploy.md

Deploying DockerCoins with YAML

• Here's a YAML manifest with all the resources for DockerCoins

  (Deployments and Services)

• We can use it if we need to deploy or redeploy DockerCoins

• Yes YAML file commands can use URL's!

k8s/yamldeploy.md

Apply errors for create or run resources

• Note the warnings if you already had the resources created

• This is because we didn't use apply before

• This is OK for us learning, so ignore the warnings

• Generally in production you want to stick with one method or the other

k8s/yamldeploy.md

Deleting resources

• We can also use a YAML file to delete resources

• kubectl delete -f ... will delete all the resources mentioned in a YAML file

  (useful to clean up everything that was created by kubectl apply -f ...)

• The definitions of the resources don't matter

  (just their kind, apiVersion, and name)

k8s/yamldeploy.md

Pruning¹ resources

• We can also tell kubectl to remove old resources

• This is done with kubectl apply -f ... --prune

• It will remove resources that don't exist in the YAML file(s)

• But only if they were created with kubectl apply in the first place

  (technically, if they have an annotation kubectl.kubernetes.io/last-applied-configuration)

¹If English is not your first language: to prune means to remove dead or overgrown branches in a tree, to help it to grow.

k8s/yamldeploy.md

YAML as source of truth

• Imagine the following workflow:

  • do not use kubectl run, kubectl create deployment, kubectl expose ...

  • define everything with YAML

  • kubectl apply -f ... --prune --all that YAML

  • keep that YAML under version control

  • enforce all changes to go through that YAML (e.g. with pull requests)

• Our version control system now has a full history of what we deploy

• Compares to "Infrastructure-as-Code", but for app deployments

k8s/yamldeploy.md

Setting up Kubernetes

• How did we set up these Kubernetes clusters that we're using?

kubeadm drawbacks

• Doesn't set up Docker or any other container engine

• Doesn't set up the overlay network

• Doesn't set up multi-master (no high availability)

Other deployment options

• AKS: managed Kubernetes on Azure

• GKE: managed Kubernetes on Google Cloud

• EKS, eksctl: managed Kubernetes on AWS

• kops: customizable deployments on AWS, Digital Ocean, GCE (beta), vSphere (alpha)

• minikube, kubespawn, Docker Desktop, kind: for local development

• kubicorn, the Cluster API: deploy your clusters declaratively, "the Kubernetes way"

k8s/setup-k8s.md

Even more deployment options

• If you like Ansible: kubespray

• If you like Terraform: typhoon

• If you like Terraform and Puppet: tarmak

• You can also learn how to install every component manually, with the excellent tutorial Kubernetes The Hard Way

  Kubernetes The Hard Way is optimized for learning, which means taking the long route to ensure you understand each task required to bootstrap a Kubernetes cluster.

• There are also many commercial options available!

• For a longer list, check the Kubernetes documentation:
  it has a great guide to pick the right solution to set up Kubernetes.

k8s/setup-k8s.md

The Kubernetes Dashboard

• Kubernetes resources can also be viewed with an official web UI

• That dashboard is usually exposed over HTTPS

  (this requires obtaining a proper TLS certificate)

• Dashboard users need to authenticate

• We are going to take a dangerous shortcut

k8s/dashboard.md

The insecure method

• We could (and should) use Let's Encrypt ...

• ... but we don't want to deal with TLS certificates

• We could (and should) learn how authentication and authorization work ...

• ... but we will use a guest account with admin access instead

Yes, this will open our cluster to all kinds of shenanigans. Don't do this at home.

k8s/dashboard.md

Running a very insecure dashboard

• We are going to deploy that dashboard with one single command

• This command will create all the necessary resources

  (the dashboard itself, the HTTP wrapper, the admin/guest account)

• All these resources are defined in a YAML file

• All we have to do is load that YAML file with with kubectl apply -f

k8s/dashboard.md

Connecting to the dashboard

You'll want the 3xxxx port.

The dashboard will then ask you which authentication you want to use.

k8s/dashboard.md

Dashboard authentication

• We have three authentication options at this point:

  • token (associated with a role that has appropriate permissions)

  • kubeconfig (e.g. using the ~/.kube/config file)

  • "skip" (use the dashboard "service account")

• Let's use "skip": we're logged in!

Running the Kubernetes Dashboard securely

• The steps that we just showed you are for educational purposes only!

• If you do that on your production cluster, people can and will abuse it

• For an in-depth discussion about securing the dashboard,
  check this excellent post on Heptio's blog

Other dashboards

• Kube Web View

  • read-only dashboard

  • optimized for "troubleshooting and incident response"

  • see vision and goals for details

• Kube Ops View

  • "provides a common operational picture for multiple Kubernetes clusters"

Security implications of kubectl apply

• When we do kubectl apply -f <URL>, we create arbitrary resources

• Resources can be evil; imagine a deployment that ...

kubectl apply is the new curl | sh

• curl | sh is convenient

• It's safe if you use HTTPS URLs from trusted sources

Scaling our demo app

• Our ultimate goal is to get more DockerCoins

  (i.e. increase the number of loops per second shown on the web UI)

• Let's look at the architecture again:

  

Adding another worker

• All we have to do is scale the worker Deployment

Adding more workers

• If 2 workers give us 2x speed, what about 3 workers?

Adding even more workers

• Let's see if 10 workers give us 10x speed!

Why are we stuck at 10-12 hashes per second?

• If this was high-quality, production code, we would have instrumentation

  (Datadog, Honeycomb, New Relic, statsd, Sumologic, ...)

• It's not!

• Perhaps we could benchmark our web services?

  (with tools like ab, or even simpler, httping)

k8s/scalingdockercoins.md

Benchmarking our web services

• We want to check hasher and rng

• We are going to use httping

• It's just like ping, but using HTTP GET requests

  (it measures how long it takes to perform one GET request)

• It's used like this:

  httping [-c count] http://host:port/path

• Or even simpler:

  httping ip.ad.dr.ess

• We will use httping on the ClusterIP addresses of our services

k8s/scalingdockercoins.md

Obtaining ClusterIP addresses

• We can simply check the output of kubectl get services

• Or do it programmatically, as in the example below

Now we can access the IP addresses of our services through $HASHER and $RNG.

k8s/scalingdockercoins.md