Numerous Kubernetes distributions (e.g., k0s, K3s, Rancher, etc.) and cloud services offering Kubernetes (e.g., GKE, AKS, EKS) are available today. But have you ever wondered why these communities or cloud providers claim to provide Kubernetes? Could I also claim to offer Kubernetes?
本篇文章為英文版本，繁體中文版本請點擊這裡。
The answer is that you must pass the Kubernetes Conformance Test!
What is Kubernetes Conformance?#
Kubernetes Conformance checks whether a Kubernetes distribution adheres to the specifications, ensuring compatibility across different systems. This includes APIs, networking, storage, resource scheduling, and security. Only distributions that pass all tests can be considered Certified Kubernetes.
What are the benefits of becoming Certified Kubernetes? According to CNCF’s official website:
- Consistency: Users expect consistent operations across any Kubernetes version.
- Timely Updates: To maintain certification, providers must deliver the latest Kubernetes versions annually or more frequently, ensuring users can access the latest features.
- Verifiability: End-users can verify compliance by running the same open-source conformance application (e.g., Sonobuoy) used for certification.
Is certification free for everyone?#
Nonprofit organizations and CNCF members are not charged certification fees.
Companies that are not CNCF members but wish to be certified must pay a membership fee to CNCF.
Nonprofits must complete a certification agreement, including project contact information, before certification.
For more details, visit CNCF’s official page on Kubernetes Conformance：https://cncf.io/ck
Introduction to Sonobuoy#
Sonobuoy is a diagnostic tool that runs plugins (including Kubernetes conformance tests) in a non-destructive and easily understandable manner to help you assess the state of a Kubernetes cluster. It offers a customizable, extensible, and cluster-agnostic approach to generate rich, clear cluster reports.
Sonobuoy supports the following use cases through selective data dumping of Kubernetes resource objects and cluster nodes:
- Integrated end-to-end (e2e) conformance testing
- Workload debugging
- Custom data collection via extensible components
Project website: https://sonobuoy.io/ GitHub: https://github.com/vmware-tanzu/sonobuoy
Next, we’ll use Sonobuoy’s “E2E & Conformance” plugin to perform a conformance test on our Kubernetes cluster!
Using Sonobuoy: Example with Kubespray v2.26.0#
On September 6, 2024, Kubespray released version 2.26.0, corresponding to Kubernetes v1.30.4. Let’s use this version for demonstration and upload the results to cncf/k8s-conformance!
Demo Environment#
For this demo, I used a MacBook Pro M2 Max. When downloading programs or images, use the arm64 version. If using Intel architecture, replace with amd64 accordingly.
- MacBook Pro 16" 2023 (M2 Max)
- Host OS: macOS Sonoma 14.6.1
- Guest OS: Ubuntu 24.04
- Python 3.12.5 (Virtualenv)
- Parallels Desktop Pro 19.4.1 (54985)
- Vagrant 2.4.1
For Python environment isolation, I recommend using Virtualenv.
Setting Up the Environment#
First, download Kubespray v2.26.0:
git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray
git checkout v2.26.0
Assuming Python is already set up, install Kubespray’s dependencies:
pip install -r requirements.txt
Kubespray provides Vagrant for testing. Configure it as follows:
mkdir -p vagrant
cat > ./vagrant/config.rb <<__EOF__
$instance_name_prefix = "kube"
$vm_cpus = 4
$vm_memory = 4096
$num_instances = 3
$os = "ubuntu2404"
$subnet = "10.2.20"
$inventory = "inventory/k8s-conformance"
$network_plugin = "calico"
__EOF__
mkdir -p inventory/k8s-conformance
Now, simply run vagrant up to start the machines and use Kubespray to install Kubernetes.
vagrant up
Once setup is complete, SSH into the kube-1 machine using vagrant and switch to the root user:
vagrant ssh kube-1
sudo -i
Verify the setup with kubectl get node:
root@kube-1:~# kubectl get node
NAME STATUS ROLES AGE VERSION
kube-1 Ready control-plane 19h v1.30.4
kube-2 Ready control-plane 19h v1.30.4
kube-3 Ready <none> 19h v1.30.4
If it looks good, install Sonobuoy to begin testing.
Installing and Running Sonobuoy#
Download the ARM version of Sonobuoy and extract it:
curl -sfL -O https://github.com/vmware-tanzu/sonobuoy/releases/download/v0.57.2/sonobuoy_0.57.2_linux_arm64.tar.gz
tar -xzvf sonobuoy_0.57.2_linux_arm64.tar.gz
Once everything is confirmed to be fine, you can start testing with sonobuoy run. Set the --mode option to certified-conformance for conformance test.
sonobuoy/systemd-logs (without the architecture suffix) for the --systemd-logs-image argument.export SONOBUOY_IMAGE_VERSION=v0.57
export SONOBUOY_LOGS_IMAGE_VERSION=v0.4
./sonobuoy run --mode=certified-conformance \
--sonobuoy-image=sonobuoy/sonobuoy:$SONOBUOY_IMAGE_VERSION \
--systemd-logs-image=sonobuoy/systemd-logs-arm64:$SONOBUOY_LOGS_IMAGE_VERSION \
--wait
Next comes a long wait. Previously, using CNTUG Infra Labs to run 16 vCPUs and 16G RAM with VirtualBox took about 2 hours. You might consider taking a nap and checking the results later.
During the wait, you can open another terminal window and use sudo kubectl get pod -A -w to see what is currently being tested:
vagrant@kube-1:~$ sudo kubectl get pod -A -w
NAMESPACE NAME READY STATUS RESTARTS AGE
cronjob-5525 concurrent-28764978-rddh8 1/1 Running 0 35s
kube-system calico-kube-controllers-b5f8f6849-5lb4z 1/1 Running 0 20m
kube-system calico-node-bxmn5 1/1 Running 0 20m
kube-system calico-node-dj4gm 1/1 Running 0 20m
kube-system calico-node-vlpkl 1/1 Running 0 20m
kube-system coredns-776bb9db5d-jdwqm 1/1 Running 0 20m
kube-system coredns-776bb9db5d-n7jp5 1/1 Running 0 20m
kube-system dns-autoscaler-6ffb84bd6-gwwpb 1/1 Running 0 20m
kube-system kube-apiserver-kube-1 1/1 Running 0 21m
kube-system kube-apiserver-kube-2 1/1 Running 0 21m
kube-system kube-controller-manager-kube-1 1/1 Running 1 21m
kube-system kube-controller-manager-kube-2 1/1 Running 1 21m
kube-system kube-proxy-2mprz 1/1 Running 0 21m
kube-system kube-proxy-7h4c6 1/1 Running 0 21m
kube-system kube-proxy-8mdcd 1/1 Running 0 21m
kube-system kube-scheduler-kube-1 1/1 Running 1 21m
kube-system kube-scheduler-kube-2 1/1 Running 1 21m
kube-system nginx-proxy-kube-3 1/1 Running 0 20m
kube-system nodelocaldns-9mkb8 1/1 Running 0 20m
kube-system nodelocaldns-hh8tw 1/1 Running 0 20m
kube-system nodelocaldns-wf9m4 1/1 Running 0 20m
sonobuoy sonobuoy 1/1 Running 0 10m
sonobuoy sonobuoy-e2e-job-7653863eaff14dfe 2/2 Running 0 10m
sonobuoy sonobuoy-systemd-logs-daemon-set-22d198ef6e024c5d-2fbhr 2/2 Running 0 10m
sonobuoy sonobuoy-systemd-logs-daemon-set-22d198ef6e024c5d-nf5sq 2/2 Running 0 10m
sonobuoy sonobuoy-systemd-logs-daemon-set-22d198ef6e024c5d-sbdcw 2/2 Running 0 10m
cronjob-5525 concurrent-28764979-bkt42 0/1 Pending 0 0s
cronjob-5525 concurrent-28764979-bkt42 0/1 Pending 0 0s
cronjob-5525 concurrent-28764979-bkt42 0/1 ContainerCreating 0 0s
cronjob-5525 concurrent-28764979-bkt42 0/1 ContainerCreating 0 0s
cronjob-5525 concurrent-28764978-rddh8 1/1 Terminating 0 60s
cronjob-5525 concurrent-28764979-bkt42 0/1 Terminating 0 0s
cronjob-5525 concurrent-28764978-rddh8 1/1 Terminating 0 60s
var-expansion-681 var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d 0/1 Pending 0 0s
var-expansion-681 var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d 0/1 Pending 0 0s
var-expansion-681 var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d 0/1 ContainerCreating 0 0s
cronjob-5525 concurrent-28764979-bkt42 0/1 Terminating 0 0s
var-expansion-681 var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d 0/1 ContainerCreating 0 0s
cronjob-5525 concurrent-28764979-bkt42 1/1 Terminating 0 1s
var-expansion-681 var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d 0/1 CreateContainerConfigError 0 1s
var-expansion-681 var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d 0/1 Terminating 0 2s
var-expansion-681 var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d 0/1 Terminating 0 3s
var-expansion-681 var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d 0/1 Terminating 0 3s
Alternatively, you can use sonobuoy status to check the execution status. However, the progress bar seems to be broken in my case, but it doesn’t affect the process.
root@kube-1:~# ./sonobuoy status
PLUGIN STATUS RESULT COUNT PROGRESS
e2e running 1 Passed: 0, Failed: 0, Remaining:402
systemd-logs complete 3
Sonobuoy is still running. Runs can take 60 minutes or more depending on cluster and plugin configuration.
Once the test is complete, this message will appear:
14:10:07 Sonobuoy has completed. Use `sonobuoy retrieve` to get results.
root@kube-1:~#
Retrieve Sonobuoy Test Results#
Next, use sonobuoy retrieve to fetch the results:
root@kube-1:~# ./sonobuoy retrieve
202409091608_sonobuoy_f07c551b-1479-42b0-8290-8bb54e8516cb.tar.gz
The result is a tar.gz file. Copy it to a regular user directory and change ownership to ensure it can be transferred successfully from the virtual machine to the host machine:
root@kube-1:~# cp 202409091608_sonobuoy_f07c551b-1479-42b0-8290-8bb54e8516cb.tar.gz /home/vagrant/result.tar.gz
root@kube-1:~# exit
logout
vagrant@kube-1:~$ sudo chown vagrant:vagrant result.tar.gz
vagrant@kube-1:~$ exit
logout
Back on the host machine, retrieve the test results from the virtual machine kube-1:
vagrant scp kube-1:result.tar.gz .
After extracting the result.tar.gz file, navigate to the folder plugins/e2e/results/global/. The files e2e.log and junit_01.xml are the required results for submission.
Before submission, check the last few lines of e2e.log to ensure there are no failures. The output should look something like this:
Ran 402 of 7199 Specs in 6126.157 seconds
SUCCESS! -- 402 Passed | 0 Failed | 0 Pending | 6797 Skipped
PASS
Ginkgo ran 1 suite in 1h42m6.592562066s
Test Suite Passed
If there are no failures, you’re ready to upload the results!
Upload Results to GitHub#
First, fork the cncf/k8s-conformance repository and select the certification version.
Important: Each pull request (PR) should include results for only one certification version. For multiple certifications, create separate PRs.
For example, if Kubespray v2.26.0 corresponds to Kubernetes v1.30, create a new folder v1.30/kubespray and include the e2e.log and junit_01.xml files. Since Kubespray has previously submitted results, you only need to update the PRODUCT.yaml version and the description in README.md.
Add a commit message: Conformance results for v1.30/kubespray。
Submit the PR to CNCF, ensuring the Pre-submission checklist is completed correctly.
For reference, here’s the PR link for Kubespray - Kubernetes v1.30 certification: https://github.com/cncf/k8s-conformance/pull/3373
Once submitted, wait for the kubernetes-conformance-bot to validate the results. This usually takes 1–2 hours.
After bot validation, CNCF staff will review the submission, which may take 1–2 weeks.
Conclusion#
After reading this blog about Conformance, it’s clear that the SIG Architecture team invests significant effort in maintaining the certification process.
This certification acts as a “specification manual” for Kubernetes. Passing it means the distribution can be considered a Kubernetes release, ensuring compatibility with features described in the official Kubernetes documentation. However, some distributions offer additional or extended features, requiring careful consideration when switching between distributions.
For example, similar to C++ compilers, anyone can implement a C++ compiler as long as they follow the C++ specification. Such a compiler would be compliant. However, compilers like GNU GCC include extended features (e.g., PBDS), which require additional attention when switching to another compiler.