K8s Troubleshooting handbook — Recently, our K8s cluster run into some Zombie process issues. Pods cannot be deleted or created, and even can’t SSH into the node. We found a lots of defunct process in many Pods. The symptom in Pod looks like: CPU: 0% usr 0% sys…

Linux zombie process deep dive — Before we introduce and troubleshoot Zombie process, let’s first understand what is “process status”. Process Status When iowait is increasing, the process is likely to be in an uninterruptible state for a long time because it cannot get a response from the hardware. From the output of ps or top command, you…

An interesting ChatGPT bot project for K8s — Today I’d like to introduce an interesting projected called “K8s ChatGPT Bot”. The purpose of this project is to deploy a ChatGPT bot for your K8s issues. You can ask ChatGPT how to solve your Prometheus alerts, get pithy responses, no more solving alerts alone in the darkness! This setup…

Go code snippet for AWS operations — A code snippet in Golang to query all AWS EC2 “Reserved Instances” and get information. This code can be used to extract the “Expires” date of AWS reserved instances programmably. You can easily extend the purpose and functionality of this code. AWS EC2 Reserved Instances Amazon EC2 Reserved Instances (RI) provide a significant discount…

Deep dive into Linux /proc directory — /proc/pid/schedstat This file contains process level scheduling information. There are three fields in this file correlating for that process to: Time spent on the cpu (in nanoseconds), this value is the same as the se.sum_exec_runtime Time spent waiting on a runqueue (in nanoseconds), this value is the same as se.wait_sum Number…

Robusta, the open source K8s troubleshooting platform intro — What is Robusta Robusta (https://home.robusta.dev/) is an open source platform for K8s troubleshooting. Like many other cloud-antive apps, it is installed and managed with Helm3, and sits on top of the monitoring stack (Prometheus, Elasticsearch, etc.) and tells you why alerts occurred and how to fix them. Robusta comes with the following five…

How does Linux File system work? — Like Linux CPU and memory, the management of disks and file systems is also the core function of the OS. Disk provides the most basic persistent storage for the system. The file system provides a tree-like structure for managing files on the disk. Inodes and Directory Entries The file system itself is a mechanism…

What is helmfile and why use it? — Helmfile adds additional functionality to Helm by wrapping it in a declarative spec that allows you to compose several charts together to create a comprehensive deployment artifact for anything from a single application to your entire infrastructure stack. Helmfile lets you: Keep a directory of chart value files and maintain…

Can you tell the difference between the two? — What is the difference between HTTP Keep-Alive and TCP Keepalive? In fact, these two are completely different things, and the implementation levels are also different: HTTP’s Keep-Alive is implemented in OSI Layer 7 (Application layer, user mode), and it is called HTTP long connection TCP’s Keepalive is implemented in OSI…