In the first Kubernetes installment, we built a simple app that had a web server running on Nginx with a MySQL database. If you’re planning to follow the steps in this post, that environment should already be up and running!
Kubernetes Deployments
A deployment is used to control updating a pod or replica set. You might be thinking “why not just delete the pod and spin up the new one?” Well, there are two main reasons for this approach. First, deleting the pod and spinning up a new one causes downtime of the application. The second reason is you could have tens, hundreds, or thousands of pods behind a load balancer, and that would take a lot of time. Generally, this technique is used to deploy changes in an environment while ensuring minimal downtime.
Example Scenario
First, let’s dive into an example scenario highlighting why a deployment is needed. In the example from the last post we deployed an Nginx web server. For this example, let’s say we need to make a change to the Nginx config that allows it to support https. To do this, we would need to expose port 443 from the pod. The nginx.yaml will look something like this:
nginx.yaml:
Just like the example in the first installment, apply this with: kubectl apply -f nginx.yaml
Wait a second….what is this?!
This response tells us that Kubernetes doesn’t know what to do. Essentially, it’s saying that we are only able to update certain fields of the pod after it has been deployed.
So, how do we fix it? One solution is to manually delete the old pod, and then reinstall the new one:
kubectl delete pod example-nginx
kubectl install -f example-nginx
This would work, but what if we are in a real-world environment where downtime would have missed thousands+ connections?
This is where deployments come in. The first thing we need to do is convert our pod into a deployment:
nginx.yaml:
For good measure, let’s go over the new fields we’ve added:
- Kind: Deployment - instead of being “Pod” it is a now a deployment
- Spec.replicas: 1 - the number of pods to put in this deployment, more on this later
- Spec.strategy.type: RollingUpdate - how the update will take place. The idea of a rolling update is to slowly interchange the pods instead of hard stopping the pods and spinning up the changed ones.
- Spec.strategy.rollingUpdate.maxSurge: 1 - how many “extra” pods we can have, meaning if we have replicas as 1, it will allow up to 1 extra pod to spin up.
- Spec.strategy.rollingUpdate.maxUnavailable: 0 - how many of the replicas we asked for can be unavailable at once. At 0, it will always have at least the number of replicas specified up.
- Spec.selector: this is the selector for how the deployment knows which pods belong to it. It is important that spec.template.metadata.labels is the same as spec.selector.matchLabels.
Everything else included above was covered in the last post. You can find that here.
Before deploying, it’s important to delete the old one-off pod (if it’s still running):
kubectl delete pod example-nginx
Now, let’s spin up this deployment:
kubectl apply -f nginx.yaml
Once that’s done, we can get the pods:
kubectl get pods
This is different, isn’t it? During a deployment, Kubernetes labels pods in this format:
name-REVISION-XXX.
So, why do we need all this extra data? Well, let’s say we want 3 Nginx servers behind the load balancer service we deployed last time instead of 1.
nginx.yaml:
This is where we’ll see the first benefit of a deployment. We don’t need to delete the old deployment before applying this file as long as we don’t change the deployment metadata.name.
Run: kubectl apply -f nginx.yaml
The result will look something like this:
There are a couple of important things to note here. First, the original pod (example-nginx-748b8d846c-tt7w9) is still running. This means that Kubernetes recognized that it didn’t need to turn it down and then spin up a new one, it just spun up two more.
The second thing to note is that they all share the same revision (in my case 748b8d846c). This is one way to verify that the deployment is complete. The other way to verify the deployment is by using this command:
kubectl rollout status deployment example-nginx
The result will look something like this:
Another command that is useful, particularly if you have multiple deployments, is:
kubectl get deployments
The result of this command will show something like this:
Now, let’s demonstrate a very cool feature of a deployment that I like to call “the very hard to kill an entire deployment” feature.
Try to delete a pod:
kubectl delete pod example-nginx-748b8d846c-tt7w9
Since this is a blocking command, it will likely take some time to return. This happens because the deployment is waiting to kill this pod, and spin back up another, before returning. You can use ctrl+c to exit the blocking command and it will still come back up. However, you will need to keep checking.
Assuming you waited for the command to return, you can now get the pods:
kubectl get pods
Well, well, well, what have we here? The old pod (example-nginx-748b8d846c-tt7w9) is gone and without running any other command, a new one has popped up (example-nginx-748b8d846c-2cpgb).
That is partly why it is called a deployment controller - it’s much smarter than your normal pod. ;)
Ok, so let’s go over another scenario that really highlights the usage of the “rolling” part of the update. To do this, we are going to assume that we have decided to remove https support from the container for whatever reason (certain to be a bad one, but nonetheless).
Our Nginx file should look similar, but without port 443 exposed:
nginx.yaml:
Again, we don’t need to delete the old one. Run:
kubectl apply -f nginx.yaml
The first thing that will be seen with “kubectl get pods” is:
Remember how we mentioned that “maxSurge: 1”? This is where we will see that in action. The deployment is surging the replicas by 1, for a total of 4, while keeping the maximum unavailable (set by maxUnavailable) at 0.
At the same time, with “kubectl get deployments” we will see:
This is showing that we want 3 pods. We have 4 total (remember the surge) and 2 of them are up to date, but only 3 of 4 are available (this means 1 pod is still old, and 1 pod is terminating).
Eventually, we will see this:
Just like that, we 3/3 pods are ready!
Let’s take a closer look at these new pods:
Do you notice how the revision is different? We went from revision 748b8d846c to revision 7f8747fb49 (I wish they were a bit more different, but it is very random and out of the user's control).
Imagine how much time you could save if you had something with 10+ replicas?
Now, the load balancer service (included in the last post) will distribute the load equally amongst the three running pods!
One Last Note...
By default, Kubernetes is sending a TERM signal to the processes in the pod when it is deleted. How these processes handle that signal is out of the control of Kubernetes, and may come with undesired consequences (like killing all db connections instead of letting them finish). If you would like more control over what happens when the termination signal is sent, look into PreStop.
About Automox
Facing growing threats and a rapidly expanding attack surface, understaffed and alert-fatigued organizations need more efficient ways to eliminate their exposure to vulnerabilities. Automox is a modern cyber hygiene platform that closes the aperture of attack by more than 80% with just half the effort of traditional solutions.
Cloud-native and globally available, Automox enforces OS & third-party patch management, security configurations, and custom scripting across Windows, Mac, and Linux from a single intuitive console. IT and SecOps can quickly gain control and share visibility of on-prem, remote and virtual endpoints without the need to deploy costly infrastructure.
Experience modern, cloud-native patch management today with a 15-day free trial of Automox and start recapturing more than half the time you're currently spending on managing your attack surface. Automox dramatically reduces corporate risk while raising operational efficiency to deliver best-in-class security outcomes, faster and with fewer resources.
