Kubernetes Conformance Test - Sonobuoy

Numerous Kubernetes distributions (e.g., k0s, K3s, Rancher, etc.) and cloud services offering Kubernetes (e.g., GKE, AKS, EKS) are available today. But have you ever wondered why these communities or cloud providers claim to provide Kubernetes? Could I also claim to offer Kubernetes?

This article is in English. For the Traditional Chinese version, please click here.
本篇文章為英文版本，繁體中文版本請點擊這裡。

The answer is that you must pass the Kubernetes Conformance Test!

What is Kubernetes Conformance?

Kubernetes Conformance checks whether a Kubernetes distribution adheres to the specifications, ensuring compatibility across different systems. This includes APIs, networking, storage, resource scheduling, and security. Only distributions that pass all tests can be considered Certified Kubernetes.

What are the benefits of becoming Certified Kubernetes? According to CNCF’s official website:

  1. Consistency: Users expect consistent operations across any Kubernetes version.
  2. Timely Updates: To maintain certification, providers must deliver the latest Kubernetes versions annually or more frequently, ensuring users can access the latest features.
  3. Verifiability: End-users can verify compliance by running the same open-source conformance application (e.g., Sonobuoy) used for certification.

Is certification free for everyone?

Nonprofit organizations and CNCF members are not charged certification fees.

Companies that are not CNCF members but wish to be certified must pay a membership fee to CNCF.

Nonprofits must complete a certification agreement, including project contact information, before certification.

For more details, visit CNCF’s official page on Kubernetes Conformance：https://cncf.io/ck

Introduction to Sonobuoy

Sonobuoy

Sonobuoy is a diagnostic tool that runs plugins (including Kubernetes conformance tests) in a non-destructive and easily understandable manner to help you assess the state of a Kubernetes cluster. It offers a customizable, extensible, and cluster-agnostic approach to generate rich, clear cluster reports.

Sonobuoy supports the following use cases through selective data dumping of Kubernetes resource objects and cluster nodes:

  • Integrated end-to-end (e2e) conformance testing
  • Workload debugging
  • Custom data collection via extensible components

Project website: https://sonobuoy.io/
GitHub: https://github.com/vmware-tanzu/sonobuoy

Next, we’ll use Sonobuoy’s “E2E & Conformance” plugin to perform a conformance test on our Kubernetes cluster!

Using Sonobuoy: Example with Kubespray v2.26.0

On September 6, 2024, Kubespray released version 2.26.0, corresponding to Kubernetes v1.30.4. Let’s use this version for demonstration and upload the results to cncf/k8s-conformance!

Demo Environment

For this demo, I used a MacBook Pro M2 Max. When downloading programs or images, use the arm64 version. If using Intel architecture, replace with amd64 accordingly.

  • MacBook Pro 16” 2023 (M2 Max)
  • Host OS: macOS Sonoma 14.6.1
  • Guest OS: Ubuntu 24.04
  • Python 3.12.5 (Virtualenv)
  • Parallels Desktop Pro 19.4.1 (54985)
  • Vagrant 2.4.1

For Python environment isolation, I recommend using Virtualenv.

Setting Up the Environment

First, download Kubespray v2.26.0:

1
2
3
git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray
git checkout v2.26.0

Assuming Python is already set up, install Kubespray’s dependencies:

1
pip install -r requirements.txt

Kubespray provides Vagrant for testing. Configure it as follows:

1
2
3
4
5
6
7
8
9
10
11
12
mkdir -p vagrant
cat > ./vagrant/config.rb <<__EOF__
$instance_name_prefix = "kube"
$vm_cpus = 4
$vm_memory = 4096
$num_instances = 3
$os = "ubuntu2404"
$subnet = "10.2.20"
$inventory = "inventory/k8s-conformance"
$network_plugin = "calico"
__EOF__
mkdir -p inventory/k8s-conformance

Now, simply run vagrant up to start the machines and use Kubespray to install Kubernetes.

Details for configuring Vagrant providers are omitted here; please refer to online resources for setting up Vagrant providers.

1
vagrant up

Once setup is complete, SSH into the kube-1 machine using vagrant and switch to the root user:

1
2
vagrant ssh kube-1
sudo -i

Verify the setup with kubectl get node:

1
2
3
4
5
root@kube-1:~# kubectl get node
NAME     STATUS   ROLES           AGE   VERSION
kube-1   Ready    control-plane   19h   v1.30.4
kube-2   Ready    control-plane   19h   v1.30.4
kube-3   Ready    <none>          19h   v1.30.4

If it looks good, install Sonobuoy to begin testing.

Installing and Running Sonobuoy

Download the ARM version of Sonobuoy and extract it:

1
2
curl -sfL -O https://github.com/vmware-tanzu/sonobuoy/releases/download/v0.57.2/sonobuoy_0.57.2_linux_arm64.tar.gz
tar -xzvf sonobuoy_0.57.2_linux_arm64.tar.gz

Once everything is confirmed to be fine, you can start testing with sonobuoy run. Set the --mode option to certified-conformance for conformance test.

For Intel versions, use sonobuoy/systemd-logs (without the architecture suffix) for the --systemd-logs-image argument.

1
2
3
4
5
6
export SONOBUOY_IMAGE_VERSION=v0.57
export SONOBUOY_LOGS_IMAGE_VERSION=v0.4
./sonobuoy run --mode=certified-conformance \
    --sonobuoy-image=sonobuoy/sonobuoy:$SONOBUOY_IMAGE_VERSION \
    --systemd-logs-image=sonobuoy/systemd-logs-arm64:$SONOBUOY_LOGS_IMAGE_VERSION \
    --wait

Next comes a long wait. Previously, using CNTUG Infra Labs to run 16 vCPUs and 16G RAM with VirtualBox took about 2 hours. You might consider taking a nap and checking the results later.

During the wait, you can open another terminal window and use sudo kubectl get pod -A -w to see what is currently being tested:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
vagrant@kube-1:~$ sudo kubectl get pod -A -w
NAMESPACE      NAME                                                      READY   STATUS    RESTARTS   AGE
cronjob-5525   concurrent-28764978-rddh8                                 1/1     Running   0          35s
kube-system    calico-kube-controllers-b5f8f6849-5lb4z                   1/1     Running   0          20m
kube-system    calico-node-bxmn5                                         1/1     Running   0          20m
kube-system    calico-node-dj4gm                                         1/1     Running   0          20m
kube-system    calico-node-vlpkl                                         1/1     Running   0          20m
kube-system    coredns-776bb9db5d-jdwqm                                  1/1     Running   0          20m
kube-system    coredns-776bb9db5d-n7jp5                                  1/1     Running   0          20m
kube-system    dns-autoscaler-6ffb84bd6-gwwpb                            1/1     Running   0          20m
kube-system    kube-apiserver-kube-1                                     1/1     Running   0          21m
kube-system    kube-apiserver-kube-2                                     1/1     Running   0          21m
kube-system    kube-controller-manager-kube-1                            1/1     Running   1          21m
kube-system    kube-controller-manager-kube-2                            1/1     Running   1          21m
kube-system    kube-proxy-2mprz                                          1/1     Running   0          21m
kube-system    kube-proxy-7h4c6                                          1/1     Running   0          21m
kube-system    kube-proxy-8mdcd                                          1/1     Running   0          21m
kube-system    kube-scheduler-kube-1                                     1/1     Running   1          21m
kube-system    kube-scheduler-kube-2                                     1/1     Running   1          21m
kube-system    nginx-proxy-kube-3                                        1/1     Running   0          20m
kube-system    nodelocaldns-9mkb8                                        1/1     Running   0          20m
kube-system    nodelocaldns-hh8tw                                        1/1     Running   0          20m
kube-system    nodelocaldns-wf9m4                                        1/1     Running   0          20m
sonobuoy       sonobuoy                                                  1/1     Running   0          10m
sonobuoy       sonobuoy-e2e-job-7653863eaff14dfe                         2/2     Running   0          10m
sonobuoy       sonobuoy-systemd-logs-daemon-set-22d198ef6e024c5d-2fbhr   2/2     Running   0          10m
sonobuoy       sonobuoy-systemd-logs-daemon-set-22d198ef6e024c5d-nf5sq   2/2     Running   0          10m
sonobuoy       sonobuoy-systemd-logs-daemon-set-22d198ef6e024c5d-sbdcw   2/2     Running   0          10m
cronjob-5525   concurrent-28764979-bkt42                                 0/1     Pending   0          0s
cronjob-5525   concurrent-28764979-bkt42                                 0/1     Pending   0          0s
cronjob-5525   concurrent-28764979-bkt42                                 0/1     ContainerCreating   0          0s
cronjob-5525   concurrent-28764979-bkt42                                 0/1     ContainerCreating   0          0s
cronjob-5525   concurrent-28764978-rddh8                                 1/1     Terminating         0          60s
cronjob-5525   concurrent-28764979-bkt42                                 0/1     Terminating         0          0s
cronjob-5525   concurrent-28764978-rddh8                                 1/1     Terminating         0          60s
var-expansion-681   var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d        0/1     Pending             0          0s
var-expansion-681   var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d        0/1     Pending             0          0s
var-expansion-681   var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d        0/1     ContainerCreating   0          0s
cronjob-5525        concurrent-28764979-bkt42                                 0/1     Terminating         0          0s
var-expansion-681   var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d        0/1     ContainerCreating   0          0s
cronjob-5525        concurrent-28764979-bkt42                                 1/1     Terminating         0          1s
var-expansion-681   var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d        0/1     CreateContainerConfigError   0          1s
var-expansion-681   var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d        0/1     Terminating                  0          2s
var-expansion-681   var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d        0/1     Terminating                  0          3s
var-expansion-681   var-expansion-bb921fe1-aac4-4040-9659-f0c893bfb41d        0/1     Terminating                  0          3s

Alternatively, you can use sonobuoy status to check the execution status. However, the progress bar seems to be broken in my case, but it doesn’t affect the process.

1
2
3
4
5
6
root@kube-1:~# ./sonobuoy status
         PLUGIN     STATUS   RESULT   COUNT                                PROGRESS
            e2e    running                1   Passed:  0, Failed:  0, Remaining:402
   systemd-logs   complete                3

Sonobuoy is still running. Runs can take 60 minutes or more depending on cluster and plugin configuration.

2000 years later......

Once the test is complete, this message will appear:

1
2
14:10:07 Sonobuoy has completed. Use `sonobuoy retrieve` to get results.
root@kube-1:~#

Retrieve Sonobuoy Test Results

Next, use sonobuoy retrieve to fetch the results:

1
2
root@kube-1:~# ./sonobuoy retrieve
202409091608_sonobuoy_f07c551b-1479-42b0-8290-8bb54e8516cb.tar.gz

The result is a tar.gz file. Copy it to a regular user directory and change ownership to ensure it can be transferred successfully from the virtual machine to the host machine:

1
2
3
4
5
6
root@kube-1:~# cp 202409091608_sonobuoy_f07c551b-1479-42b0-8290-8bb54e8516cb.tar.gz /home/vagrant/result.tar.gz
root@kube-1:~# exit
logout
vagrant@kube-1:~$ sudo chown vagrant:vagrant result.tar.gz 
vagrant@kube-1:~$ exit
logout

Back on the host machine, retrieve the test results from the virtual machine kube-1:

1
vagrant scp kube-1:result.tar.gz .

After extracting the result.tar.gz file, navigate to the folder plugins/e2e/results/global/. The files e2e.log and junit_01.xml are the required results for submission.

Before submission, check the last few lines of e2e.log to ensure there are no failures. The output should look something like this:

1
2
3
4
5
6
Ran 402 of 7199 Specs in 6126.157 seconds
SUCCESS! -- 402 Passed | 0 Failed | 0 Pending | 6797 Skipped
PASS

Ginkgo ran 1 suite in 1h42m6.592562066s
Test Suite Passed

If there are no failures, you’re ready to upload the results!

Upload Results to GitHub

First, fork the cncf/k8s-conformance repository and select the certification version.

Important: Each pull request (PR) should include results for only one certification version. For multiple certifications, create separate PRs.

For example, if Kubespray v2.26.0 corresponds to Kubernetes v1.30, create a new folder v1.30/kubespray and include the e2e.log and junit_01.xml files. Since Kubespray has previously submitted results, you only need to update the PRODUCT.yaml version and the description in README.md.

Add a commit message: Conformance results for v1.30/kubespray。

Submit the PR to CNCF, ensuring the Pre-submission checklist is completed correctly.

For reference, here’s the PR link for Kubespray - Kubernetes v1.30 certification: https://github.com/cncf/k8s-conformance/pull/3373

Once submitted, wait for the kubernetes-conformance-bot to validate the results. This usually takes 1–2 hours.

After bot validation, CNCF staff will review the submission, which may take 1–2 weeks.

Conclusion

After reading this blog about Conformance, it’s clear that the SIG Architecture team invests significant effort in maintaining the certification process.

This certification acts as a “specification manual” for Kubernetes. Passing it means the distribution can be considered a Kubernetes release, ensuring compatibility with features described in the official Kubernetes documentation. However, some distributions offer additional or extended features, requiring careful consideration when switching between distributions.

For example, similar to C++ compilers, anyone can implement a C++ compiler as long as they follow the C++ specification. Such a compiler would be compliant. However, compilers like GNU GCC include extended features (e.g., PBDS), which require additional attention when switching to another compiler.

References

  1. Certified Kubernetes Software Conformance - CNCF
  2. cncf/k8s-conformance - GitHub
  3. Spotlight on SIG Architecture: Conformance - Kubernetes
  4. The Kubernetes End-To-End Testing Plugin - Sonobuoy
  5. vmware-tanzu/sonobuoy - GitHub
  6. Kubespray v2.26.0 - GitHub
Buy Me A Coffee
Kubernetes
#kubernetes #conformance #sonobuoy #cncf
Kubernetes Conformance Test - Sonobuoy
https://blog.yangjerry.tw/k8s-conformance-sonobuoy-en/
作者
Jerry Yang
發布於
2024年9月11日
許可協議