The case for learning Kubernetes at a company that doesn’t use it

Heap’s CI/CD pipelines and cloud-based applications are masterfully architected and maintained by highly skilled infrastructure engineers. Kubernetes, however, is not a major feature of this arsenal, at least not yet. Nevertheless, I decided to use the Certified Kubernetes Administrator (CKA) exam as a tool to study Kubernetes, and in doing so uncovered several benefits for software engineers that go beyond just improving their DevOps skills.

Why Kubernetes?

As a software engineer with limited DevOps experience, I spent my first few months at Heap learning as much as I could about AWS—Heap’s chosen cloud provider. I used certifications as a tool to structure my learning and immerse myself in the knowledge pool, which proved to be a good way to build a foundation for learning various technologies.

I made a list of a few other certifications I wanted to study for, with Kubernetes interestingly coming in dead last on that list. My list, however, quickly got flipped upside down when one of our brilliant new engineering-hires demoed a Kubernetes cluster for Heap’s version of the Hackathon – Shipweek.

Seeing the production containers for key components—particularly the ones my team handles—deployed to an isolated environment within the span of Ship Week got me thinking. With my team’s services sitting on the front lines of Heap’s data capture, they make first contact with billions of incoming event data daily, making it tricky to isolate and reproduce production issues when they occur. Integration testing can also be a challenge due to the disparate services making up our pipelines.

Though I didn’t have a concrete plan for how Kubernetes could help, I quickly promoted the Certified Kubernetes Administrator (CKA) exam to the next certification on my list and started studying. In doing so, I uncovered four benefits for software engineers beyond improving DevOps skills.

1. Get better at application development on a Unix OS

With Kubernetes certifications being practical instead of multiple choice, you'll need some dexterity within the Unix shell and operating system to pass the exams. Unless your development environment runs on Windows, chances are you've had to edit files using Vim, create aliases for often-used commands, and curl URLs or tail logs during your application development cycles. Exposure to Kubernetes will repeatedly put these skills to use, sharpening them as you go. 

You’ll encounter several scenarios where you need advanced knowledge of the Unix OS to solve problems. For instance, when installing a Kubernetes cluster from scratch, several components, including the etcd key-value data store—used by Kubernetes to store state—are installed as services. Consider the following abbreviated list of steps involved in installing etcd for a Kubernetes cluster:

1. Use the wget command to download the etcd binaries

2. Use the tar utility to extract the binaries from the downloaded gzip file

3. Configure the etcd.service systemd unit file

4. Start the etcd service  

This pattern of manually downloading binaries and setting them up as a service is also used to install many other utilities outside the Kubernetes ecosystem, making it useful for software engineers. 

For engineers who develop on Unix, a micro-productivity booster you might pick up while studying includes advanced techniques with editors like Vim. Deploying Kubernetes primitives like pods, services, and configmaps involves running commands in the shell to generate them imperatively or spawn yaml files that define them, similar to terraform or cloudformation templates.

In cases where you need complex configurations that you can’t generate from the command-line, updating the yaml files in editors like Vim is the de facto method for completing configurations. Since the CKA exam requires speed to get through all questions in the allocated time, advanced knowledge of Vim is the only way to carry out edits efficiently. Some tips and tricks you might pick up include:

• Activating line numbers for Vim files:

:set nu

This can also be set up in the ~/.vimrc on your workstation

• Navigating to specific lines in navigation mode:

[line-number]gg

• Executing case-insensitive find and replace in a Vim file:

:%s/term/replace/gi

• Copying and pasting blocks of code with the cursor positioned on the desired line:

[number-of-lines]yy #then typing `p` to paste

As a software engineer, it can be hard to justify learning a few shortcuts for Vim. Especially when you have other things to learn, like a few dozen javascript libraries. Knowing a study path shares a tip or two on doing even the most mundane things more efficiently should be a good motivator.

2. Enforce CloudOps concepts

AWS features such as IAM policies, tasks, and ECS services loosely map to Kubernetes concepts like role-based access controls (RBAC), pods, and deployments (as shown in the chart below). 

If you’re a software engineer who is new to DevOps but has a fair amount of AWS knowledge, learning Kubernetes will sometimes feel like a second pass for concepts you’ve learned about the AWS cloud. But nothing will improve your expertise with a given technology, like frequent hands-on usage. And you will get a lot more comfortable working with cloud infrastructure after encountering certain concepts from a different perspective, courtesy of studying Kubernetes.  

3. Improve troubleshooting skills

One key argument for software engineers to be more ops savvy is that it encourages them to be more responsive to issues occurring in production. Heap’s incident management response gleans heavily from Site Reliability Engineering (SRE) best practices, and there is no shortage of demand for sharp troubleshooting instincts when incidents occur. 

Troubleshooting Kubernetes clusters involves a much larger surface area than a single application or service. Consider a case where you are troubleshooting a failure of the API server—a Kubernetes component that serves as the communications entry point from the worker node to control plane. There are several places you would look to find a root cause of the issue, including:

• The /var/log/pods and /var/log/containers folders

• The container runtime logs using crictl ps or crictl logs

• Logs for the kernel, daemons, and other components at /var/log/syslog

• The service logs using the journalctl command-line utility

The subject of troubleshooting in Kubernetes is an extensive one with principles that shadow troubleshooting just about any other service or application. And because troubleshooting infrastructure can be a challenge for a software engineer new to DevOps, the fact that studying Kubernetes teaches a structured approach to the subject should be a great motivator for anyone looking to improve. If you’re interested in learning more, read about the methodologies involved in the official documentation. 

4. Stimulate creativity as a software engineer

While studying Kubernetes, the final benefit I found is somewhat philosophical and applies exclusively to software engineers. It hinges on broadening your perspectives as an application developer by studying complex software.

Technologies like Kubernetes are so sophisticated that some might forget at their core they are just applications written by other software engineers. Like most software projects you’ve worked on, they went through planning, development, and some kind of quality control before releasing. 

Over the course of your career, there will be times you or your teammates avoid tackling certain projects due to perceived complexity. This might be justifiable in most cases, but in a few others, what masquerades as complexity is only a knowledge gap you can bridge with exposure to concepts documented somewhere or applied in another application like Kubernetes. 

To a DevOps professional, Kubernetes is a means to an end—a tool that performs a given function. But to a curious software engineer, it can be a treasure trove of clever implementation decisions. The curious engineer might find themselves browsing the Kubernetes GitHub repository, poking around the Golang code used to build it. Even without looking at code, they might ponder certain things, like how the Kubelet agent on worker nodes communicates with the API server on the control plane. They might even find inspiration in mundane details like how thorough the kubectl command-line help resources are. Creative thinking like this will profoundly impact any software engineer’s application development game in the long run. 

Conclusion 

Learning is a nonstop endeavor. At Heap, and increasingly many other technology companies running infrastructure in the cloud, software engineers are expected to be more involved with infrastructure provisioning for the applications they build. It’s no longer enough to throw applications and services over the wall to a DevOps team after writing code.

So, if like me, you find yourself looking to improve your DevOps knowledge, studying Kubernetes using the Certified Kubernetes Administrator (CKA) exam as a guide has a lot of benefits worth the trouble—even if your current company doesn’t use it.