kubernetes-by-hand

Part 3 — kubectl & auth

Kubernetes By Hand

⏱️ Estimated time: 90–150 min · Difficulty: most moving parts so far Certs are where people lose 20–30 min — a mismatched CA or a wrong subject name is the usual culprit. The 401-vs-403 distinction in Troubleshooting is your compass. Times assume “understand and move,” not deep rabbit-holing.

At the end of Part 2 you had a working API server with two embarrassing holes: identity was a single shared token anyone could copy (me, via --token-auth-file), and authorization was --authorization-mode=AlwaysAllow — once you badged in, you could do anything. In this chapter we replace both with the real thing, and in doing so you’ll learn the two questions the API server asks about every single request:

  1. Authentication — “Who are you?” (prove your identity)
  2. Authorization — “Are you allowed to do this?” (RBAC decides)

You’ll build a tiny certificate authority, mint yourself a client certificate that is your identity, switch the API server from “allow everyone” to “check RBAC,” and watch your own requests get denied — then write the rule that allows them and watch them succeed. Feeling a request go from 403 Forbidden to 200 OK because of a rule you wrote is how RBAC stops being abstract.

Only at the very end do we introduce kubectl — and by then you’ll see it clearly for what it is: a comfortable client wrapped around the exact HTTP API you’ve been driving by hand with curl. No magic. Just a nicer curl.

[fwd] flags continue: a term marked [fwd] is a placeholder you’ll fully meet later.

Priming questions

Guess first. Don’t look anything up.

  1. In Part 2 you proved who you were with a shared bearer token — a password anyone holding the file could use, with no way to tell two people apart. How could an HTTP client prove its identity using the TLS connection itself, so the credential is per-user and unforgeable? (You already made a serving cert in Part 2 to prove the server to you. What if a cert could run the other direction and carry your name?)
  2. “Authentication” and “authorization” sound like synonyms in English but are two distinct steps here. What’s the difference, and why must they be separate? What breaks if you conflate them?
  3. RBAC stands for Role-Based Access Control. Before reading its details: if you had to design “who can do what” for a system with thousands of object types, what are the minimum pieces you’d need? (Hint: a verb, a thing, and a someone — how do you connect them?)
  4. kubectl get pods and curl -k https://.../api/v1/.../pods return the same data. So what, precisely, is kubectl doing that curl isn’t — and what must it read to know where your API server is and who you are?
  5. In Part 2 the audit log’s user field said me — the shared token user, identical for anyone holding the token. After this chapter it’ll say your real per-user identity, and records will show allowed vs denied. Why does giving each caller a real identity (and real permissions) make the audit log meaningfully more useful than it was with the shared token?

Assumed state

Resuming from a passed Part 2 gate.

Note on what carries over. Part 2’s serving cert and SA keypair stay exactly as they are — we keep using them. This chapter adds a real CA and client certs, and swaps AlwaysAllow for RBAC. The shared token still works for now; by the end you’ll have replaced it with a proper client-certificate identity and can retire it.

Check on cp:

hostname                                  # -> cp
etcdctl endpoint health                   # healthy (start etcd if needed)
# start the apiserver from Part 2 if needed, then:
curl -k https://127.0.0.1:6443/api/v1 >/dev/null && echo "apiserver OK"

The mental model

Every request to the API server runs a gauntlet, in this fixed order:

request --> [ Authentication ] --> [ Authorization ] --> [ Admission (fwd) ] --> etcd
             "who are you?"         "may you do it?"       "any last edits?"

RBAC itself is four object types, and they’re simpler than they sound (Q3):

Hold the shape: Role = what may be done; Binding = who may do the Role. Every RBAC setup you’ll ever see is combinations of those two ideas.

The build

Step 1 — Become a certificate authority

A CA is just a keypair you decide to trust as the root of identity. The API server will be told “trust certs signed by this CA,” and anyone holding a CA-signed cert gets the identity written inside it. On cp:

cd ~/pki

# The CA: a self-signed root you control.
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -days 3650 \
  -subj "/CN=kubernetes-by-hand-ca" \
  -out ca.crt

ls -1 ca.crt ca.key      # your certificate authority

ca.crt is public (the API server needs it to verify signatures). ca.key is the crown jewels — whoever holds it can mint any identity. In a real cluster this key is guarded fiercely; here it lives next to everything else because the whole lab is disposable.

Step 2 — Mint YOUR identity as a client certificate

Now issue yourself a client cert whose CN is your username and whose O is your group. This cert literally is your identity to the cluster.

cd ~/pki

# 1) your private key
openssl genrsa -out argv.key 2048

# 2) a signing request: CN=argv (username), O=byhand-admins (group)
openssl req -new -key argv.key \
  -subj "/CN=argv/O=byhand-admins" \
  -out argv.csr

# 3) the CA signs it -> a cert the API server will trust and read your name from
openssl x509 -req -in argv.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -days 365 -out argv.crt

# inspect what identity it encodes:
openssl x509 -in argv.crt -noout -subject
# -> subject=CN=argv, O=byhand-admins

You now hold three files that matter: argv.crt + argv.key (your identity) and ca.crt (to verify the server). Keep the names — CN=argv, O=byhand-admins — in mind; they’ll appear as your username and group everywhere from here on.

Step 3 — Restart the API server to trust your CA and enforce RBAC

Two changes from Part 2’s command: point the API server at your CA for client authentication, and flip authorization from AlwaysAllow to RBAC. Stop the API server (Ctrl-C) and restart with:

cd ~
sudo kube-apiserver \
  --etcd-servers=http://127.0.0.1:2379 \
  --service-cluster-ip-range=10.96.0.0/16 \
  --authorization-mode=Node,RBAC \
  --bind-address=127.0.0.1 \
  --secure-port=6443 \
  --tls-cert-file=$HOME/pki/apiserver.crt \
  --tls-private-key-file=$HOME/pki/apiserver.key \
  --client-ca-file=$HOME/pki/ca.crt \
  --service-account-key-file=$HOME/pki/sa.pub \
  --service-account-signing-key-file=$HOME/pki/sa.key \
  --service-account-issuer=https://kubernetes.default.svc \
  --audit-policy-file=$HOME/audit-policy.yaml \
  --audit-log-path=/tmp/audit.log \
  2>&1 | tee /tmp/apiserver.log

The lines that changed everything:

Note we also dropped --token-auth-file — that shared Part 2 token is retired here on purpose; from now on you identify with a client certificate, not a shared password. (You could keep it, but retiring it is the point.)

Leave it running.

Step 4 — Feel the denial (this is the lesson)

First, present your certificate identity to a normally-safe read. Note we now provide three PEM files: your cert, your key, and the CA (so curl trusts the server too — no more -k):

cd ~/pki

curl --cert argv.crt --key argv.key --cacert apiserver.crt \
  https://127.0.0.1:6443/api/v1/namespaces

You are authenticated (the server knows you’re argv) but not authorized — RBAC grants nothing by default. Expect a 403 Forbidden whose message names you:

... "forbidden: User \"argv\" cannot list resource \"namespaces\" ... "

Read that message closely — it’s RBAC being explicit and honest: this user, this verb, this resource, denied. You have a valid identity and zero permissions. That’s the secure default: deny unless explicitly allowed. Now go grant yourself something.

Step 5 — Grant a permission, watch 403 become 200

We need to write RBAC objects — but you can’t yet (you have no permissions), a classic bootstrapping knot. We cut it the way real clusters bootstrap: use a superuser the API server trusts implicitly. The simplest lab approach: temporarily talk to the API as a member of the built-in system:masters group, which is hardwired to full access.

Mint a one-off admin cert in system:masters (this group is the break-glass identity):

cd ~/pki
openssl genrsa -out admin.key 2048
openssl req -new -key admin.key -subj "/CN=admin/O=system:masters" -out admin.csr
openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 -out admin.crt

As that admin, create a ClusterRole (what may be done) and a ClusterRoleBinding (who may do it) that grant argv read access to namespaces. We POST these as JSON, same as any object:

cd ~/pki

# ClusterRole: "get,list,watch on namespaces"
curl --cert admin.crt --key admin.key --cacert apiserver.crt \
  -X POST https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles \
  -H 'Content-Type: application/json' \
  -d '{
    "apiVersion":"rbac.authorization.k8s.io/v1",
    "kind":"ClusterRole",
    "metadata":{"name":"ns-reader"},
    "rules":[{"apiGroups":[""],"resources":["namespaces"],"verbs":["get","list","watch"]}]
  }'

# ClusterRoleBinding: bind that role to user "argv"
curl --cert admin.crt --key admin.key --cacert apiserver.crt \
  -X POST https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings \
  -H 'Content-Type: application/json' \
  -d '{
    "apiVersion":"rbac.authorization.k8s.io/v1",
    "kind":"ClusterRoleBinding",
    "metadata":{"name":"argv-ns-reader"},
    "subjects":[{"kind":"User","name":"argv","apiGroup":"rbac.authorization.k8s.io"}],
    "roleRef":{"kind":"ClusterRole","name":"ns-reader","apiGroup":"rbac.authorization.k8s.io"}
  }'

Now repeat the exact request that was denied in Step 4, as argv:

curl --cert argv.crt --key argv.key --cacert apiserver.crt \
  https://127.0.0.1:6443/api/v1/namespaces | jq '.items[].metadata.name'

403 → 200. You now see the namespace list. Nothing about your identity changed — the same cert, the same user. The only difference is a rule you wrote connecting the verb list, the resource namespaces, and the subject argv. That is the entire mental model of RBAC, felt directly: Role = permitted verbs+resources; Binding = attach them to a someone.

Try a verb you didn’t grant and watch the denial return — proving grants are precise:

# you were granted get/list/watch, NOT delete:
curl --cert argv.crt --key argv.key --cacert apiserver.crt \
  -X DELETE https://127.0.0.1:6443/api/v1/namespaces/byhand
# -> 403 Forbidden: argv cannot delete namespaces

Step 6 — Confirm the audit log now names you (closing Part 2’s thread)

Priming Q5. In Part 2 the audit user said me — the shared token user, identical for anyone holding the token, and every action showed as allowed (there was nothing to deny). Look now:

tail -n 20 /tmp/audit.log | jq -r 'select(.objectRef.resource=="namespaces") | "\(.verb) \(.objectRef.resource) by \(.user.username) -> \(.responseStatus.code // "?")"' | tail -5

You’ll see lines attributing actions to argv — your real per-user identity — and your denied delete showing a 403. Two things got richer at once: the user is now a specific person rather than a shared password, and the ledger now records allowed vs denied. The audit machinery didn’t change — authentication and authorization did. That’s the whole reason audit logging lived in the API server chapter: it was only ever waiting for identity and permissions to become meaningful.

Step 7 — Finally, kubectl: a client of everything you just built

You’ve been the client by hand. kubectl automates it. It needs to know three things — where the API server is, how to trust it (CA), and who you are (your cert+key) — all of which live in a single file called a kubeconfig. Install kubectl and build one.

# same KVER as your apiserver!
KVER=v1.31.0
cd /tmp
curl -LO "https://dl.k8s.io/${KVER}/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/
kubectl version --client

Assemble a kubeconfig from the pieces you already have — notice every value maps to a concept you now understand:

cd ~/pki

kubectl config set-cluster byhand \
  --server=https://127.0.0.1:6443 \
  --certificate-authority=$HOME/pki/apiserver.crt \
  --embed-certs=true \
  --kubeconfig=$HOME/.kube/config

kubectl config set-credentials argv \
  --client-certificate=$HOME/pki/argv.crt \
  --client-key=$HOME/pki/argv.key \
  --embed-certs=true \
  --kubeconfig=$HOME/.kube/config

kubectl config set-context byhand \
  --cluster=byhand --user=argv \
  --kubeconfig=$HOME/.kube/config

kubectl config use-context byhand --kubeconfig=$HOME/.kube/config

Open ~/.kube/config and read it. There are no secrets you haven’t met: a cluster (server URL

Prove the equivalence:

kubectl get namespaces

Same list as your curl in Step 5 — because it is the same request. And RBAC still applies to kubectl exactly as it did to curl (same identity, same rules):

kubectl delete namespace byhand
# -> Error from server (Forbidden): ... argv cannot delete ... namespaces

kubectl didn’t grant itself anything. It’s a client. The API server is still the one deciding. That’s the sentence to leave this chapter with.

Optional, to feel that kubectl is a thin wrapper: ask it to show the raw HTTP it makes:

kubectl get namespaces -v=8 2>&1 | grep -E 'GET https|Request Headers|Response Status' | head

You’ll see it performing the very GET https://127.0.0.1:6443/api/v1/namespaces you did by hand.

Verification gate

You pass Part 3 when, without copying:

  1. You can explain the difference between authentication and authorization in one sentence each, and name what carries your identity (a client cert; CN=user, O=group).
  2. You make a request as argv that is denied by RBAC (403), then write a Role/Binding that makes the same request succeed (200) — and can articulate that only the rule changed, not your identity.
  3. You can state the RBAC shape: Role/ClusterRole = permitted verbs on resources; Binding = attaches a Role to a subject.
  4. kubectl get namespaces works through your kubeconfig, and you can name the three things a kubeconfig holds (cluster/CA, user cert+key, context).
  5. You can explain why kubectl delete is also denied — i.e. why kubectl has no more power than your curl did.

If #2 and #5 are fluent, RBAC and the client/server relationship are yours.

Troubleshooting

What just happened (close the loop)

Step back and see how far the picture has come. You have: a durable store (etcd), a single guarded front door that types/validates/persists/streams and now knows who’s calling and enforces what they may do (the API server), and a proper client (kubectl) that is nothing but a well-dressed HTTP caller. That is a real, if minimal, control plane — a database with a guarded API in front of it.

But it’s still just a database with an API. Nothing is acting on the objects yet. If you created a Pod [fwd] object right now, it would land in etcd as a key and then… sit there. Nothing schedules it; nothing runs it. The cluster can remember desired state but can’t yet move reality toward it. In Part 4 we add the first of the loops from Part 0’s mental model — the scheduler — and you’ll watch it do exactly one thing that suddenly makes the whole “reconcile reality to desired state” idea concrete: take a Pod that’s sitting unassigned and decide which node runs it.

Lab log

Snapshot cp as part-03-kubectl-auth and add a row to LAB_LOG.md:

| 2025-XX-XX | 03 | Y | part-03-kubectl-auth | built CA; argv identity via client cert; RBAC deny->allow; kubectl via kubeconfig; audit now names argv |

Next → Part 4 — the scheduler

← Part 2 · Index

This site is open source. Improve this page.