When you hand over the reins of Kubernetes to your developers, kubectl is always a friction point for them.

I think that developers should get to know kubectl, just like they should know how to use other command-line tools, but having said that, you can and SHOULD implement tools that will take kubectl out of the daily workflow.

Central Logging

With Pods and Nodes going up and down, you want one central location for storing your logs.
There is nothing worse than trying to figure out a bug by running the kubectl logs roulette. Did we log this? Or maybe the Pod that received the traffic is now long gone?

The joys of troubleshooting. So the first step for making your developers happy is to have an accessible and easy to use central logging solution that they can work with.

Now, for that to work well, you will have to follow the best practices for working with containers and direct all of your logs to STDOUT instead of files.

There are a few ways to do that:

1. Get your loggers to write to STDOUT instead of files.
2. If you can’t avoid writing log to files, use Sidecar containers to pull the logs and then write them to STDOUT.

Check out:

1. EFK (Elasticsearch, Fluentd, Kibana). Pretty much everything is based on this stack, even the managed cloud solutions.
2. DataDog - A cloud service that also offers monitoring and security features.
3. Logz.io - Again, another cloud solution that covers logging and monitoring.

Monitoring

A good monitoring setup is a another must have. Not only will it alert you when things go wrong, but it can also help you be more proactive when dealing with issues.

When you are trying to solve a production issue, even from the developer’s point of view, it’s helpful to have a single dashboard with current and historical data about your application.
This includes the usual suspects like CPU and Memory, but also application-specific metrics like database connections, error count, queue size, etc.

Instead of going over the pods one by one trying to figure out what is going on, you have one central interface that gives you the big picture and allows you to drill down to the low-level data.

Check out

1. Prometheus - Works great for Kubernetes, and combined with Grafana, gives you a whole lot of value out of the box. Almost every public Helm chart also includes a ServiceMonitor object, which you can implement to monitor the application.
2. DataDog - A cloud solution that has everything - APM capabilities, Security monitoring, and some good old resource monitoring. It seems as if everyone is raving about this tool, and there’s a reason for that. It’s not cheap, but so is doing everything yourself.

Making changes - GitOps

GitOps is more than just a buzzword, and a methodology that can help your developers deploy and make changes with a toolset that is already on their belt - Git.

The idea behind it is that instead of running commands for deploying or updating the environment, everything moves to git push.
Want to deploy a new version of the app? Git push.
Change an environment variable for a pod? Git push.
Rollback? You guessed it, git push.

Using the GitOps methodology will give you more clarity as to what is deployed and have a commit log of the changes over time. 
It also prevents your team from introducing bad habits related to manual updates.

You can read more about GitOps here.

Faster CI / CD

When developers trust the CI / CD process, they tend to use it more instead of trying to hack something together to get around it.
I’ve seen examples where developers, out of frustration, well, let’s just say that they found a way to work with the cluster that is a bit hacky.

If they had a reliable and fast process to test their code and get feedback, they wouldn’t have to go around it.

So optimize your pipelines to the point that they provide quick feedback to your developers, exposing test results and logs when failing.

Better UI tools

I started as a network engineer working with Cisco equipment. Back then, it was all about the CLI. A GUI was for lightweights.

But the days of the old crappy JAVA GUI are behind us, and there are so many good GUI options for just about anything, including Kubernetes.

Check out:

1. k9s - a neat little tool that runs in your terminal and lets you interact with Kubernetes.
2. Cloud Providers - If you are using a managed service like GKE, you will be able to manage and view your cluster workloads from their web console.
3. The Kubernetes Web Interface - Not my favorite but it’s very easy to get it running.