⏱️ Estimated time: 90–120 min Difficulty: Moderate
curl the API server with Authorization: Bearer $TOKEN.openssl ≥ 3.0 (openssl version).Part 3 replaces the bearer token with a CA, client certs, and RBAC. If the mechanics of a CA aren’t solid, every command in Part 3 is a spell you’re copying, not something you understand. This part builds a CA, a server cert, and a client cert by hand, breaks them on purpose, and runs a real TLS handshake — until none of it is magic.
server-a. What actually stops you from copying it onto server-b and using it there?You’ll answer all four cold by the end. Work happens on the control-plane node, ~/pki-lab/.
mkdir -p ~/pki-lab && cd ~/pki-lab
A private key signs. The matching public key verifies. You cannot derive the private key from the public one, and you cannot forge a valid signature without the private key. That’s it — that’s the entire mathematical trust anchor. Everything below is what you build on top of that one fact. You don’t need to know how the math does this, only that it does.
A CA is nothing but a private key plus a self-signed certificate. “Self-signed” means Issuer == Subject — it vouches for itself because nothing vouches for it. It’s the root; trust starts there or it doesn’t start.
# CA private key
openssl genrsa -out ca.key 2048
# self-signed CA certificate (this key vouching for itself)
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 \
-out ca.crt -subj "/CN=kubernetes-by-hand-ca"
Inspect it:
openssl x509 -in ca.crt -text -noout
Find and understand two lines:
Issuer and Subject are identical → self-signed.X509v3 Basic Constraints: CA:TRUE → this cert is allowed to sign other certs. Without this flag, a cert can’t act as a CA even if you have its key.Checkpoint: you now hold the only thing that can turn a CSR into a trusted cert — ca.key. Guard it like you’d guard a root password, because it functionally is one.
A CSR is a request: “I generated this keypair, here’s my public key, here’s who I claim to be — please sign it.” The CA never sees or needs the requester’s private key.
# server's own keypair
openssl genrsa -out server.key 2048
# CSR, with SAN — modern TLS clients ignore CN and check SAN instead
openssl req -new -key server.key -out server.csr \
-subj "/CN=server-a" \
-addext "subjectAltName=DNS:server-a,IP:10.0.0.11"
# CA signs it → this is the moment trust gets conferred
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-out server.crt -days 365 -sha256 -copy_extensions copy
Inspect the result:
openssl x509 -in server.crt -text -noout
Now Issuer = kubernetes-by-hand-ca, Subject = server-a, and the SAN block is present. The CA never touched server.key — it only ever saw the CSR (public info) and signed it with its own private key.
openssl verify -CAfile ca.crt server.crt
# -> server.crt: OK
Now break it two ways, and watch it fail both times. This is answering priming question 3 experientially.
# tamper: flip a byte in a copy, verify against the original
cp server.crt server-tampered.crt
sed -i 's/A/B/1' server-tampered.crt
openssl verify -CAfile ca.crt server-tampered.crt # fails — signature no longer matches content
rm server-tampered.crt # server.crt itself was never touched, no need to regenerate
# wrong CA: sign with an unrelated CA and verify against the original
openssl genrsa -out rogue-ca.key 2048
openssl req -x509 -new -nodes -key rogue-ca.key -days 365 -out rogue-ca.crt -subj "/CN=rogue-ca"
openssl x509 -req -in server.csr -CA rogue-ca.crt -CAkey rogue-ca.key -CAcreateserial -out rogue-server.crt -days 365 -copy_extensions copy
openssl verify -CAfile ca.crt rogue-server.crt # fails — signed by a key ca.crt doesn't trust
Verification is arithmetic, not vibes. A cert is only ever as trustworthy as the CA key that signed it — copying a valid cert to another machine (question 3) doesn’t help you without the matching private key, which never left server.key.
This is what happens every time you hit an HTTPS site. Only the server proves identity.
Terminal A:
openssl s_server -accept 8443 -cert server.crt -key server.key -CAfile ca.crt
Terminal B:
openssl s_client -connect localhost:8443 -CAfile ca.crt
Look for Verify return code: 0 (ok) in terminal B’s output. What just happened:
server.crt.ca.crt)? Yes.localhost vs server-a — in a real setup this must match, that’s answering question 4).Ctrl+C both sides.
Same pattern, other direction: the client also gets a cert, and the server demands it.
# client keypair + CSR + sign — note the O= field, you'll need it in a minute
openssl genrsa -out client.key 2048
openssl req -new -key client.key -out client.csr -subj "/CN=client-a/O=developers"
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-out client.crt -days 365 -sha256 -copy_extensions copy
Terminal A — server now requires a verified client cert:
openssl s_server -accept 8443 -cert server.crt -key server.key -CAfile ca.crt -Verify 1
Terminal B — client presents its cert:
openssl s_client -connect localhost:8443 -CAfile ca.crt -cert client.crt -key client.key
Both sides now show verify OK. Both sides proved identity with the same mechanism, just in opposite directions. This is mTLS — nothing more exotic happens inside a Kubernetes cluster.
| What you just built | What it becomes in Part 3 |
|---|---|
ca.key / ca.crt |
The cluster CA. kube-apiserver --client-ca-file=ca.crt |
Server cert + -Verify 1 requirement |
kube-apiserver’s own serving cert, and its requirement that clients present a cert |
client.crt with CN=client-a |
The CN of a client cert becomes the Kubernetes username — no separate user database exists |
client.crt with O=developers |
The O field becomes group membership, which RBAC RoleBindings bind against |
client.key + client.crt, presented on connect |
Exactly the client-certificate / client-key fields in your kubeconfig |
That CN → username, O → group mapping is the single biggest unlock for Part 3. Kubernetes doesn’t have a user table. Identity is whatever the CA vouched for in the cert.
Answer the four priming questions now, out loud, no notes:
server.crt let you impersonate the original server?If any answer is shaky, redo only that section — not the whole part. If all four are solid, stop here. Do not chase revocation (CRL/OCSP), cert-manager, Vault’s PKI engine, or intermediate CA chains right now — those are real topics, but they’re not blocking Part 3, and they’re exactly the kind of thing that eats a weekend if you let curiosity drive instead of the stop condition. Park them in BACKLOG.md if you want a marker.
YYYY-MM-DD - Part 2.5 (PKI interlude) - built CA, server cert, client cert by hand;
verified chain, broke it twice on purpose, ran live one-way and mutual TLS handshakes
with openssl s_server/s_client. Key unlock: CN->k8s username, O->k8s group, both come
from the CA-signed cert, not a user database.
Snapshot the control-plane node now, before touching Part 3: pki-interlude-complete. Everything in ~/pki-lab/ was scratch work for this part only — Part 3 builds its own CA and certs from scratch inside the actual cluster config paths, it doesn’t reuse these files.
Next → Part 3 — kubectl & auth