A complete Kubernetes tutorial built specifically for SDET / QA automation engineers — not platform engineers. Every section is framed around testing: spinning up ephemeral environments, running suites as Jobs, debugging failing pods, and validating deployments. The second half applies all of it to testing AI/LLM services running on Kubernetes.
kubectl commands you'll use dailyReal Kubernetes clusters can't run inside a browser tab, so this is a simulated demo-cluster
with pre-seeded pods, deployments, and services — including one intentionally broken pod so you can practice
debugging. Try a sample command or type your own kubectl command below.
Click a sample command, or type your own and hit "Run Command".
Kubernetes (K8s) is a system for running and managing containerized applications across a cluster of machines. It decides where each container runs, restarts it if it crashes, and gives it a stable way to talk to other containers. For an SDET, this matters because the application under test almost certainly runs on Kubernetes in staging and production — and increasingly, the test infrastructure itself (ephemeral environments, test runners, even AI model-serving) runs there too.
A quick mental model, from smallest to largest:
| Concept | What it is |
|---|---|
| Container | A single packaged process (your app + its dependencies) — same as plain Docker |
| Pod | The smallest deployable unit in K8s — one or more containers that share networking and storage |
| Node | A physical or virtual machine that runs pods |
| Cluster | A set of nodes managed together, plus the control plane that schedules and coordinates everything |
| Control plane | The "brain" — the API server, scheduler, and controllers that keep the cluster in its desired state |
These five objects cover the vast majority of what you'll read and write day to day.
| Object | Purpose | SDET-relevant note |
|---|---|---|
Pod | Runs one or more containers | You rarely create these directly — a Deployment manages them |
ReplicaSet | Keeps N identical pod replicas running | Managed automatically by a Deployment; you'll see it in describe output |
Deployment | Manages ReplicaSets, handles rolling updates | What you scale, roll back, and check rollout status on |
Service | Stable network endpoint routing to a set of pods | What your test client actually connects to, not individual pods |
Namespace | A virtual cluster / isolation boundary | Perfect for per-branch or per-PR ephemeral test environments |
# see everything running in a namespace
kubectl get pods,deployments,services -n staging
# pods aren't permanent -- their names include a random suffix
# e.g. orders-api-5d4f9c8b7-a1b2c
# ^ deployment ^ replicaset ^ pod
kubectl is the CLI you'll use for almost everything: inspecting state, reading logs, applying
changes, and shelling into a running container. These commands cover roughly 90% of real day-to-day usage.
# inspect
kubectl get pods # list pods in the current namespace
kubectl get pods -o wide # + node, IP, container info
kubectl describe pod <name> # full detail: events, conditions, image, probes
kubectl get events --sort-by=.lastTimestamp
# logs & shell access
kubectl logs <pod> # stdout/stderr of the main container
kubectl logs <pod> -c <container> -f # specific container, follow (tail -f style)
kubectl exec -it <pod> -- /bin/sh # shell into a running container
# change state
kubectl apply -f deployment.yaml # create or update from a manifest
kubectl delete pod <name> # delete -- a Deployment will recreate it
kubectl scale deployment/<name> --replicas=3
kubectl rollout status deployment/<name>
kubectl rollout undo deployment/<name> # roll back to the previous version
# networking (great for hitting an app under test)
kubectl port-forward svc/<name> 8080:80
Try several of these right now in the simulator above — describe and
logs against the intentionally broken pod are worth running first.
Every Kubernetes object can be described declaratively in YAML. Reading and writing these fluently — not just clicking through a dashboard — is a baseline expectation in interviews.
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-api
labels:
app: orders-api
spec:
replicas: 2
selector:
matchLabels:
app: orders-api
template:
metadata:
labels:
app: orders-api
spec:
containers:
- name: orders-api
image: registry.example.com/orders-api:1.4.2
ports:
- containerPort: 8080
resources:
requests: { cpu: "100m", memory: "128Mi" }
limits: { cpu: "500m", memory: "256Mi" }
apiVersion: v1
kind: Service
metadata:
name: orders-api-svc
spec:
selector:
app: orders-api # routes traffic to any pod with this label
ports:
- port: 80
targetPort: 8080
type: ClusterIP
requests is what the scheduler guarantees when placing your pod on a node.
limits is the hard ceiling — exceed the memory limit and the container gets OOMKilled. Forgetting
to set these is a classic cause of noisy-neighbor test flakiness in shared clusters.Hardcoding config into a container image means rebuilding an image every time an environment variable changes. ConfigMaps and Secrets decouple configuration from the image itself — directly relevant to injecting test-specific config (base URLs, feature flags, API keys) without touching application code.
apiVersion: v1
kind: ConfigMap
metadata:
name: orders-api-config
data:
LOG_LEVEL: "debug"
FEATURE_NEW_CHECKOUT: "true"
---
apiVersion: v1
kind: Secret
metadata:
name: orders-api-secret
type: Opaque
stringData:
DB_PASSWORD: "test-only-password" # base64-encoded automatically from stringData
Both are referenced from a pod spec the same way — as environment variables or mounted files:
envFrom:
- configMapRef: { name: orders-api-config }
- secretRef: { name: orders-api-secret }
Labels are how loosely-coupled Kubernetes objects find each other — a Service doesn't know about specific pods by name, it finds them by matching labels. Understanding this wiring is essential for debugging "why isn't my Service routing traffic to my pods."
# filter by label -- very useful in a shared/noisy test namespace
kubectl get pods -l app=orders-api
kubectl get pods -l 'environment in (staging, qa)'
# when a Service isn't routing traffic, this is usually why:
# the Service's spec.selector doesn't match any pod's metadata.labels
kubectl get pods --show-labels
kubectl describe svc orders-api-svc # check the "Selector" and "Endpoints" fields
Annotations look similar but are for non-identifying metadata (build info, tooling config) — Kubernetes never uses annotations for selection the way it uses labels.
Pods are ephemeral — anything written to a container's filesystem disappears when the pod is recreated. For anything that needs to survive restarts (a test database's data, uploaded test fixtures), you need a PersistentVolumeClaim, which is how a pod requests durable storage without caring exactly which disk it lands on.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-db-storage
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests: { storage: "5Gi" }
---
# referenced from a pod spec:
volumes:
- name: db-data
persistentVolumeClaim: { claimName: test-db-storage }
containers:
- name: postgres
volumeMounts:
- { name: db-data, mountPath: /var/lib/postgresql/data }
emptyDir volume (deleted with the pod) is often the right choice instead of a PVC. Reach for a
real PVC only when you specifically need data to outlive a single pod, like a shared fixture-seeded DB across
a whole test day.kubectl get pvc # check binding status: Pending means no PV satisfies the claim yet
kubectl describe pvc test-db-storage
A pod can run more than one container. This explains a status you'll see often —
0/2 Ready instead of the expected 1/1 — and two patterns worth recognizing on sight.
spec:
initContainers: # run to completion, in order, BEFORE main containers start
- name: wait-for-db
image: busybox
command: ['sh', '-c', 'until nc -z db-svc 5432; do sleep 2; done']
containers:
- name: orders-api # the main application container
image: orders-api:1.4.2
- name: log-shipper # sidecar -- runs ALONGSIDE the main container
image: fluent-bit
| Pattern | Lifecycle | Typical use |
|---|---|---|
| Init container | Runs to completion first, one at a time, before any main container starts | Waiting for a dependency, running a DB migration, seeding test fixtures |
| Sidecar | Runs continuously, alongside the main container, for the pod's whole lifetime | Log shipping, service-mesh proxies, metrics exporters |
If a pod is stuck and never reaches Running, check init containers first — a pod won't even
start its main containers until every init container has exited successfully.
You've already seen how a StatefulSet differs from a Deployment conceptually — stable identity and storage per replica. Here's what that looks like as an actual manifest, useful when your test suite needs its own dedicated database instance rather than a shared one.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: test-postgres
spec:
serviceName: test-postgres # must match a headless Service for stable DNS per replica
replicas: 1
selector: { matchLabels: { app: test-postgres } }
template:
metadata: { labels: { app: test-postgres } }
spec:
containers:
- name: postgres
image: postgres:16
volumeMounts:
- { name: data, mountPath: /var/lib/postgresql/data }
volumeClaimTemplates: # each replica gets its OWN PVC, automatically
- metadata: { name: data }
spec:
accessModes: ["ReadWriteOnce"]
resources: { requests: { storage: "2Gi" } }
Notice the pod this creates is named test-postgres-0, not a random suffix — and each replica
would reliably be addressable at test-postgres-0.test-postgres.<namespace>.svc.cluster.local,
which is exactly the stable identity a Deployment can't offer.
A Service (specifically ClusterIP) is only reachable inside the cluster. Ingress is the layer
that routes external HTTP(S) traffic in — by hostname and path — to the right internal Service, and is usually
where TLS termination happens too.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: staging-ingress
annotations:
cert-manager.io/cluster-issuer: "letsencrypt"
spec:
tls:
- hosts: ["staging.example.com"]
secretName: staging-tls
rules:
- host: staging.example.com
http:
paths:
- path: /api/orders
pathType: Prefix
backend: { service: { name: orders-api-svc, port: { number: 80 } } }
- path: /
pathType: Prefix
backend: { service: { name: frontend-svc, port: { number: 80 } } }
For testing purposes, this means path-based routing rules are themselves testable — hit
https://staging.example.com/api/orders from outside the cluster and confirm it actually reaches
orders-api-svc and not frontend-svc, exactly as the rule intends.
When a CI pipeline or a Python script authenticates to the cluster (as in the Python K8s client section later), it does so as a ServiceAccount — and Role-Based Access Control determines exactly what that identity is allowed to do. Least-privilege here matters: a test-automation pipeline should not have cluster-admin access.
apiVersion: v1
kind: ServiceAccount
metadata: { name: test-runner, namespace: pr-4521 }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role # scoped to ONE namespace (ClusterRole would be cluster-wide)
metadata: { name: test-runner-role, namespace: pr-4521 }
rules:
- apiGroups: ["", "batch", "apps"]
resources: ["pods", "pods/log", "jobs", "deployments"]
verbs: ["get", "list", "watch", "create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: test-runner-binding, namespace: pr-4521 }
subjects:
- { kind: ServiceAccount, name: test-runner, namespace: pr-4521 }
roleRef: { kind: Role, name: test-runner-role, apiGroup: rbac.authorization.k8s.io }
Role + RoleBinding grants permissions within one namespace; a
ClusterRole + ClusterRoleBinding grants them cluster-wide. A test-automation
pipeline should almost always use a namespaced Role scoped to exactly the verbs and resources it needs — not
a ClusterRole out of convenience."This is the part most Kubernetes tutorials skip, and the part SDET interviews actually probe: not "can you deploy an app" but "can you use the cluster to make testing faster, more isolated, and more reliable."
Namespaces are cheap to create and delete, which makes them ideal for spinning up an isolated environment per pull request or per test run — no shared staging environment to fight over.
# create an isolated namespace for this PR/branch
kubectl create namespace pr-4521
# deploy the full stack into it
kubectl apply -f k8s/ -n pr-4521
# point your test suite's BASE_URL at the ephemeral service, run tests...
# then tear it all down in one shot -- no leftover resources
kubectl delete namespace pr-4521
finally-equivalent step) — orphaned test namespaces are one of the most common sources of
runaway cloud costs on a team.A Job runs a pod to completion — perfect for a one-off test suite run inside the cluster,
close to the services it's testing. A CronJob does the same on a schedule, e.g. a nightly
regression run.
apiVersion: batch/v1
kind: Job
metadata:
name: pytest-regression-run
spec:
backoffLimit: 1 # don't endlessly retry a failing suite
template:
spec:
containers:
- name: test-runner
image: registry.example.com/sdet-suite:latest
command: ["pytest", "-m", "regression", "--junitxml=/results/out.xml"]
env:
- name: BASE_URL
value: "http://orders-api-svc.pr-4521.svc.cluster.local"
restartPolicy: Never
apiVersion: batch/v1
kind: CronJob
metadata:
name: nightly-regression
spec:
schedule: "0 2 * * *" # 2 AM daily, standard cron syntax
jobTemplate:
spec:
template:
spec:
containers:
- name: test-runner
image: registry.example.com/sdet-suite:latest
command: ["pytest", "-m", "regression"]
restartPolicy: Never
Notice the service DNS name in the Job — <service>.<namespace>.svc.cluster.local is
how anything inside the cluster reaches a Service by name, without hardcoding IPs.
When you're running tests from outside the cluster (your laptop, a CI runner without cluster
access) but the app under test is only reachable inside it, port-forward bridges the gap
without exposing anything publicly.
# forward local:8080 -> the Service's port 80 inside the cluster
kubectl port-forward svc/orders-api-svc 8080:80
# now, from another terminal (or your test suite):
curl http://localhost:8080/health
Inside the cluster, pods talk to each other via Service DNS names (as shown in the Job example above), not port-forwarding — port-forwarding is specifically a developer/tester convenience for reaching in from outside.
This is one of the most valuable things an SDET can understand about Kubernetes: a pod showing
Running does not mean the application inside it is actually ready to serve
traffic. Chasing this gap with time.sleep(30) in test setup is exactly the anti-pattern probes
exist to solve.
readinessProbe: # "should this pod receive traffic right now?"
httpGet:
path: /health/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe: # "is this pod still healthy, or should it be restarted?"
httpGet:
path: /health/live
port: 8080
periodSeconds: 15
failureThreshold: 3
# wait for real readiness instead of a fixed sleep before running tests
kubectl wait --for=condition=Ready pod -l app=orders-api --timeout=60s
kubectl logs is fine for one pod, but real debugging usually spans multiple pods and multiple
restarts.
# logs from a crashed container's PREVIOUS run -- critical for CrashLoopBackOff
kubectl logs <pod> --previous
# a specific container in a multi-container (sidecar) pod
kubectl logs <pod> -c <container-name>
# logs from every pod matching a label, tailed together
kubectl logs -l app=orders-api --all-containers=true -f --prefix
# in real clusters, logs are usually shipped off-pod entirely --
# a sidecar or node-level agent (Fluent Bit, Fluentd) forwards them
# to a central store (Elasticsearch/Loki) so they survive pod deletion
Try --previous against the broken pod in the simulator — it's the first thing you'd reach for
on a real CrashLoopBackOff pod, since the current container instance may not have logs yet.
A repeatable triage sequence beats guessing. This is also almost verbatim what a strong interview answer to "walk me through debugging a failing pod" sounds like.
# 1. what's the pod's status, and how many times has it restarted?
kubectl get pods
# 2. describe -- check Events at the bottom, and container "Last State"
kubectl describe pod <pod>
# 3. logs from the CURRENT and PREVIOUS container instance
kubectl logs <pod>
kubectl logs <pod> --previous
# 4. if it's running but misbehaving, get a shell inside it
kubectl exec -it <pod> -- /bin/sh
| Status you see | Likely cause |
|---|---|
ImagePullBackOff | Wrong image tag/name, or missing registry credentials |
CrashLoopBackOff | The app inside the container is exiting immediately — check logs --previous |
Pending | Scheduler can't place the pod — often insufficient CPU/memory on any node |
OOMKilled | Container exceeded its memory limit and was killed |
0/1 Ready but Running | Readiness probe is failing — app started but isn't passing its own health check |
Run kubectl describe pod orders-api-5d4f9c8b7-c3d4e in the simulator above to see this exact
triage flow against a realistic CrashLoopBackOff.
Helm packages a set of related manifests (Deployment, Service, ConfigMap, etc.) into one reusable, templated "chart." As an SDET, you'll mostly need to validate that a chart renders correctly and that a deployed release actually works.
# render the chart's templates locally WITHOUT deploying -- catch YAML/templating bugs early
helm template ./orders-api-chart --values values-staging.yaml
# lint for common mistakes
helm lint ./orders-api-chart
# install into a namespace
helm install orders-api ./orders-api-chart -n pr-4521 --create-namespace
# run the chart's built-in test hooks (pods annotated as helm.sh/hook: test)
helm test orders-api -n pr-4521
apiVersion: v1
kind: Pod
metadata:
name: "{{ .Release.Name }}-smoke-test"
annotations:
"helm.sh/hook": test # only runs on `helm test`, not `helm install`
spec:
containers:
- name: smoke-test
image: curlimages/curl
command: ['curl', '-f', 'http://{{ .Release.Name }}-svc/health']
restartPolicy: Never
Once an app is split across many small services, each deployed as its own set of pods, "does it work" stops being a single question. Two complementary strategies handle this without needing every service running at once.
A healthy pipeline usually runs contract tests on every commit (fast feedback) and full in-cluster integration tests less frequently — nightly, or before a release — since they're slower and costlier to run.
By default, any pod in a cluster can talk to any other pod. A NetworkPolicy restricts that — and as an SDET, verifying those restrictions actually hold (a service that shouldn't reach the payments service, truly can't) is a legitimate, testable security requirement, not just a config file to trust blindly.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: payments-allow-orders-only }
spec:
podSelector: { matchLabels: { app: payments } }
policyTypes: ["Ingress"]
ingress:
- from:
- podSelector: { matchLabels: { app: orders-api } } # only orders-api may reach payments
# from a pod WITHOUT the orders-api label, this request should time out / be refused
def test_frontend_cannot_reach_payments_directly():
with pytest.raises(requests.exceptions.ConnectionError):
requests.get("http://payments-svc.default.svc.cluster.local", timeout=3)
Note that this test has to run from inside the cluster (e.g. as a Job, using the same pattern from the Jobs section) as a pod carrying the "wrong" label — a policy's effect can't be observed from outside.
On a shared test cluster, one team's runaway test suite can starve everyone else's. Quotas and LimitRanges
cap total and per-container resource usage at the namespace level — and explain a specific, confusing failure
mode: a pod stuck Pending even though the cluster overall looks healthy.
apiVersion: v1
kind: ResourceQuota
metadata: { name: pr-namespace-quota, namespace: pr-4521 }
spec:
hard:
requests.cpu: "4"
requests.memory: "8Gi"
pods: "20" # hard cap on pod count in this namespace
---
apiVersion: v1
kind: LimitRange
metadata: { name: default-container-limits, namespace: pr-4521 }
spec:
limits:
- type: Container
default: { cpu: "250m", memory: "256Mi" } # applied when a pod spec omits limits
Pending and describe shows no scheduling error about
node capacity, check kubectl describe resourcequota -n <namespace> next — the namespace
itself may have hit its quota, which is invisible if you're only looking at node-level resources."kubectl exec -it only works if the target container has a shell — and many production and
AI-serving images are built on minimal/distroless base images with no shell at all, specifically to shrink
attack surface. kubectl debug is the modern workaround.
# attach a temporary debug container WITH a shell to a running pod that has none
kubectl debug -it orders-api-5d4f9c8b7-a1b2c --image=busybox --target=orders-api
# or spin up a full copy of a pod with an interactive shell, without touching the original
kubectl debug orders-api-5d4f9c8b7-a1b2c -it --image=busybox --copy-to=orders-api-debug --container=orders-api -- sh
The ephemeral debug container shares the target pod's network namespace, so you can curl localhost
exactly as the real application would see it — invaluable for diagnosing "the health check fails but I can't
get a shell to find out why."
Most "my service can't reach another service" issues are actually DNS issues in disguise. This is the classic, reliable technique for isolating them.
# launch a disposable pod with basic network tools, auto-deleted on exit
kubectl run dns-debug --rm -it --image=busybox --restart=Never -- sh
# from inside that pod:
nslookup orders-api-svc # same-namespace short name
nslookup orders-api-svc.default.svc.cluster.local # fully-qualified, cross-namespace
wget -qO- http://orders-api-svc/health # confirm the resolved IP actually answers
orders-api-svc only resolves correctly for pods in the same
namespace as the Service. A pod in a different namespace must use the fully-qualified form
(orders-api-svc.default.svc.cluster.local) — forgetting this is one of the most common causes of
mysterious cross-namespace connection failures.Beyond logs, most services expose a /metrics endpoint that Prometheus scrapes — request counts,
latency histograms, error rates. As a tester, you can assert directly against these metrics as part of a test,
not just against individual HTTP responses.
import requests
def parse_prometheus_metric(text, metric_name):
for line in text.splitlines():
if line.startswith(metric_name):
return float(line.split()[-1])
return None
def test_error_rate_stays_low_after_load():
# ... generate some load against the service first ...
metrics_text = requests.get("http://orders-api-svc/metrics").text
error_count = parse_prometheus_metric(metrics_text, "http_requests_total{status=\"500\"}") or 0
total_count = parse_prometheus_metric(metrics_text, "http_requests_total{status=\"200\"}") or 1
assert (error_count / total_count) < 0.01, "Error rate exceeded 1% under load"
This pattern generalizes directly to the HPA load-testing project later in this tutorial — the metrics endpoint is also how you'd confirm the autoscaler is reacting to real request volume, not just guessing.
Everything above is useful manually, but the real payoff is wiring it into a pipeline that runs automatically on every commit — this is where "SDET who understands Kubernetes" becomes "SDET who builds infrastructure."
name: Deploy to Ephemeral Namespace & Test
on: [pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: azure/setup-kubectl@v4
- name: Configure cluster access
run: echo "${{ secrets.KUBE_CONFIG }}" > kubeconfig.yaml
- name: Deploy to ephemeral namespace
run: |
export NS=pr-${{ github.event.pull_request.number }}
kubectl --kubeconfig kubeconfig.yaml create namespace $NS
kubectl --kubeconfig kubeconfig.yaml apply -f k8s/ -n $NS
kubectl --kubeconfig kubeconfig.yaml wait --for=condition=Ready pod -l app=orders-api -n $NS --timeout=90s
- name: Run test suite as a Job
run: kubectl --kubeconfig kubeconfig.yaml apply -f k8s/test-job.yaml -n $NS
- name: Tear down
if: always()
run: kubectl --kubeconfig kubeconfig.yaml delete namespace pr-${{ github.event.pull_request.number }}
Note the if: always() on teardown — this is what prevents orphaned namespaces from piling up
when a job fails partway through.
Before a new image gets anywhere near a test cluster, a fast security gate scans it for known-vulnerable dependencies — this typically sits right next to the test stage in a pipeline, and interviewers sometimes probe whether you think about this as part of "testing" a deployment, not just functional correctness.
- name: Scan image for vulnerabilities
run: |
trivy image --severity HIGH,CRITICAL --exit-code 1 \
registry.example.com/orders-api:${{ github.sha }}
# exit-code 1 fails the pipeline on any HIGH/CRITICAL finding,
# before the image is ever deployed to the test namespace
trivy (or grype) scans a container image layer-by-layer against known CVE
databases. Running this before deployment, rather than after, keeps a vulnerable image out of even an ephemeral
test environment.
You don't need a cloud cluster to develop and test Kubernetes manifests. These tools run a real (if small) Kubernetes cluster locally or inside a CI runner, in seconds.
| Tool | What it is | Best for |
|---|---|---|
kind | Runs a cluster inside Docker containers | CI pipelines — fast, disposable, no VM overhead |
minikube | Runs a single-node cluster in a local VM/container | Local development with a fuller feature set (dashboards, addons) |
k3d | Runs lightweight k3s in Docker | Very fast local clusters, low resource usage |
# spin up a throwaway cluster for a CI run, in seconds
kind create cluster --name ci-test-cluster
kubectl apply -f k8s/
# ... run tests ...
kind delete cluster --name ci-test-cluster
Copy-pasting a Deployment YAML three times for dev/staging/prod (and slowly drifting them apart) is a common
early mistake. kustomize — built directly into kubectl — lets you keep one base
manifest and layer small, environment-specific overlays on top.
resources:
- ../../base # the shared Deployment/Service manifests
namespace: staging
replicas:
- name: orders-api
count: 2
patches:
- target: { kind: Deployment, name: orders-api }
patch: |-
- op: replace
path: /spec/template/spec/containers/0/env/0/value
value: "staging"
# preview the fully-rendered manifest for an environment, without applying it
kubectl kustomize overlays/staging/
# apply it directly -- kubectl understands kustomize natively
kubectl apply -k overlays/staging/
The practical benefit for testing: your ephemeral PR namespace and your staging environment can both derive from the exact same base manifest, so what you tested is structurally the same thing that ships — just with a different overlay.
Beyond shelling out to kubectl, the official kubernetes Python package lets you
orchestrate test infrastructure programmatically — spinning up namespaces, launching test Jobs, and polling
status, all from a pytest fixture or CI script.
from kubernetes import client, config
config.load_kube_config() # or load_incluster_config() when running inside K8s
v1 = client.CoreV1Api()
def wait_for_pods_ready(namespace, label_selector, timeout=60):
import time
deadline = time.time() + timeout
while time.time() < deadline:
pods = v1.list_namespaced_pod(namespace, label_selector=label_selector)
if pods.items and all(
c.ready for p in pods.items for c in (p.status.container_statuses or [])
):
return True
time.sleep(2)
raise TimeoutError(f"Pods not ready in {namespace} after {timeout}s")
def create_ephemeral_namespace(name):
body = client.V1Namespace(metadata=client.V1ObjectMeta(name=name))
v1.create_namespace(body)
def delete_namespace(name):
v1.delete_namespace(name)
This is exactly the kind of helper you'd drop into a conftest.py as a session-scoped fixture —
the ephemeral-namespace pattern from earlier, expressed as reusable pytest code instead of shell commands.
Increasingly, teams don't run kubectl apply from CI at all — instead, a GitOps controller
(ArgoCD or Flux) continuously watches a Git repo and reconciles the cluster to match whatever's committed
there. This changes what "deploying" even means for a pipeline, and it's worth recognizing the pattern.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: orders-api-staging }
spec:
source:
repoURL: https://github.com/example-org/k8s-manifests
path: overlays/staging
targetRevision: main
destination: { namespace: staging }
syncPolicy:
automated: { prune: true, selfHeal: true } # auto-reverts manual kubectl edits
argocd app get orders-api-staging) rather than assuming the deploy finished the moment CI
finished — with selfHeal: true active, even a manual kubectl apply would get
silently reverted, which is a common source of confusion the first time someone hits it.Both strategies reduce the blast radius of a bad deployment — and both need automated tests gating the transition, not just a human eyeballing a dashboard.
| Strategy | How it works | Automated test's job |
|---|---|---|
| Rolling update | Gradually replaces old pods with new ones (Kubernetes default) | Readiness probes gate traffic to new pods automatically |
| Blue-green | Deploy the new version fully alongside the old, then switch the Service's selector all at once | Run a smoke-test suite against "green" before flipping the switch |
| Canary | Route a small % of traffic to the new version, monitor, then ramp up | Automated checks on error rate/latency decide whether to proceed or roll back |
# blue-green: deploy "green", smoke-test it, then repoint the Service
kubectl apply -f deployment-green.yaml
kubectl wait --for=condition=Ready pod -l version=green --timeout=60s
pytest -m smoke --base-url=http://green-preview-svc
# only if smoke tests pass:
kubectl patch service orders-api-svc -p '{"spec":{"selector":{"version":"green"}}}'
The kubectl patch service trick above is a manual, all-or-nothing traffic switch. A service
mesh makes fine-grained traffic splitting a first-class, declarative feature — genuinely finer canary control
than a bare Service can offer, and worth knowing about even if you never operate one yourself.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata: { name: orders-api }
spec:
hosts: ["orders-api-svc"]
http:
- route:
- { destination: { host: orders-api-svc, subset: v1 }, weight: 90 }
- { destination: { host: orders-api-svc, subset: v2 }, weight: 10 } # 10% canary
Beyond traffic splitting, a mesh commonly adds automatic mutual TLS between services and rich per-request metrics — meaning some of the observability and NetworkPolicy-style testing covered earlier can be validated through the mesh's own telemetry instead of custom test code.
Kubernetes makes it trivially easy to kill things on purpose — which is exactly what chaos testing needs. The goal isn't to break things randomly; it's to verify your system actually behaves the way its design claims it will under failure.
# the simplest possible chaos test -- no extra tooling required
kubectl delete pod -l app=orders-api --grace-period=0 --force
# then assert your test suite still gets a 200 -- the Deployment
# should have already recreated the pod, and the Service should
# have routed around the gap
Purpose-built tools like Chaos Mesh or Litmus go further: injecting network latency, simulating a full node failure, or corrupting DNS — all declaratively, as Kubernetes custom resources, so the chaos experiment itself is version-controlled alongside your test suite.
AI-powered features increasingly ship as their own microservices — a model-serving pod behind a Service, just like any other backend. Testing them on Kubernetes combines everything above with a few AI-specific wrinkles: non-determinism, latency-sensitive autoscaling, and using AI itself to make Kubernetes testing smarter.
Structurally, an AI microservice looks like any other Deployment + Service — the difference is usually heavier resource requirements and longer startup time while a model loads into memory.
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm-inference
namespace: ai-services
spec:
replicas: 1
selector:
matchLabels: { app: llm-inference }
template:
metadata:
labels: { app: llm-inference }
spec:
containers:
- name: llm-inference
image: registry.example.com/llm-inference:2.1.0
resources:
requests: { cpu: "2", memory: "8Gi" }
limits: { cpu: "4", memory: "16Gi" }
readinessProbe: # don't route traffic until the model is loaded
httpGet: { path: /health/ready, port: 8000 }
initialDelaySeconds: 30
periodSeconds: 10
initialDelaySeconds will cause the probe to fail repeatedly and
can even trigger unwanted restarts — tune probe timing (or use a startupProbe) around real model
load time, not typical web-app startup time.The Kubernetes-level testing (readiness, Service routing, logs) is identical to any other service. What's different is what you assert on once you get a response.
import requests, time
BASE_URL = "http://localhost:8080" # reached via kubectl port-forward svc/llm-inference-svc
def test_inference_returns_valid_schema():
resp = requests.post(f"{BASE_URL}/generate", json={"prompt": "Say hello"}, timeout=30)
assert resp.status_code == 200
body = resp.json()
assert "text" in body and isinstance(body["text"], str) # structure, not exact content
def test_inference_latency_is_acceptable():
start = time.perf_counter()
requests.post(f"{BASE_URL}/generate", json={"prompt": "Say hello"}, timeout=30)
elapsed = time.perf_counter() - start
assert elapsed < 5.0, f"Inference took {elapsed:.2f}s, expected < 5s"
AI inference is often bursty and resource-hungry, which makes autoscaling both more important and trickier than for a typical stateless web service.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: llm-inference-hpa
namespace: ai-services
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: llm-inference
minReplicas: 1
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
target: { type: Utilization, averageUtilization: 70 }
For GPU-backed inference, scheduling is resource-request-based just like CPU/memory — a pod requests a GPU and the scheduler only places it on a node that has one available:
resources:
limits:
nvidia.com/gpu: 1 # requires the NVIDIA device plugin installed on the cluster
As a tester, the practical implication is straightforward: your load tests against an AI service should specifically validate that the HPA actually scales up under load and that new replicas pass their readiness probe (i.e., finish loading the model) before traffic ramps up to them.
The same "constrained system prompt in, structured output out" pattern from Python-side AI testing applies directly to generating Kubernetes test scaffolding — a Job manifest, a Helm test hook, or a smoke-test pod spec — from a plain description of what needs testing.
import os, requests
SYSTEM_PROMPT = """You are a senior SDET who writes Kubernetes Job manifests for
smoke-testing a Service. Given a service name and its health endpoint,
output ONLY a valid Kubernetes Job YAML manifest that curls the endpoint
and fails the Job (non-zero exit) if the response is not HTTP 200."""
def generate_smoke_test_job(service_name: str, health_path: str) -> str:
resp = requests.post(
"https://api.anthropic.com/v1/messages",
headers={
"x-api-key": os.environ["ANTHROPIC_API_KEY"],
"anthropic-version": "2023-06-01",
"content-type": "application/json",
},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 600,
"system": SYSTEM_PROMPT,
"messages": [{"role": "user",
"content": f"service: {service_name}, health_path: {health_path}"}],
},
)
return resp.json()["content"][0]["text"]
if __name__ == "__main__":
yaml_manifest = generate_smoke_test_job("orders-api-svc", "/health")
print(yaml_manifest) # review it, then kubectl apply -f -
kubectl apply --dry-run=client -f - or kubeconform to validate the schema
before it ever touches a real cluster.Goal: feed a failing pod's describe output, recent events, and logs into an
LLM, and get back a plain-English likely root cause and a next debugging command — extending the same failure
triage pattern to be cluster-aware.
import os, json, subprocess, requests
TRIAGE_SYSTEM_PROMPT = """You are a senior SDET/SRE triaging a failing Kubernetes pod.
Given `kubectl describe pod` output, recent events, and logs, respond
with JSON containing: likely_cause (one sentence), category (one of:
image_issue, crash_on_startup, resource_limit, readiness_failure,
scheduling_issue), and next_command (one concrete kubectl command)."""
def gather_pod_context(pod_name, namespace="default") -> str:
describe = subprocess.run(
["kubectl", "describe", "pod", pod_name, "-n", namespace],
capture_output=True, text=True,
).stdout
logs = subprocess.run(
["kubectl", "logs", pod_name, "-n", namespace, "--previous", "--tail=50"],
capture_output=True, text=True,
).stdout
return f"DESCRIBE OUTPUT:\n{describe}\n\nRECENT LOGS:\n{logs}"
def triage_pod(pod_name, namespace="default") -> dict:
context = gather_pod_context(pod_name, namespace)
resp = requests.post(
"https://api.anthropic.com/v1/messages",
headers={
"x-api-key": os.environ["ANTHROPIC_API_KEY"],
"anthropic-version": "2023-06-01",
"content-type": "application/json",
},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 400,
"system": TRIAGE_SYSTEM_PROMPT,
"messages": [{"role": "user", "content": context}],
},
)
return json.loads(resp.json()["content"][0]["text"])
if __name__ == "__main__":
result = triage_pod("orders-api-5d4f9c8b7-c3d4e")
print(f"[{result['category']}] {result['likely_cause']}")
print(f"Try next: {result['next_command']}")
kubectl describe output nobody reads at 2am.Goal: a script that watches a rollout, waits for real readiness (not a fixed sleep), runs a quick smoke-test suite against the new pods, and reports a clear pass/fail — the automated gate a blue-green or canary deployment needs.
import subprocess, sys, requests, time
def wait_for_rollout(deployment, namespace="default", timeout="120s"):
result = subprocess.run(
["kubectl", "rollout", "status", f"deployment/{deployment}",
"-n", namespace, f"--timeout={timeout}"],
capture_output=True, text=True,
)
if result.returncode != 0:
raise RuntimeError(f"Rollout failed: {result.stderr}")
def run_smoke_tests(base_url) -> bool:
checks = [
("/health", 200),
("/api/orders?limit=1", 200),
]
all_passed = True
for path, expected in checks:
try:
resp = requests.get(base_url + path, timeout=10)
ok = resp.status_code == expected
except requests.RequestException:
ok = False
print(f"{'✅' if ok else '❌'} {path}")
all_passed &= ok
return all_passed
if __name__ == "__main__":
deployment, base_url = sys.argv[1], sys.argv[2]
print(f"Waiting for rollout of {deployment}...")
wait_for_rollout(deployment)
print("Rollout complete. Running smoke tests...")
passed = run_smoke_tests(base_url)
sys.exit(0 if passed else 1) # non-zero exit fails the CI step / blocks the traffic switch
Extend this with the failure-triage function from Project 1: on a failed smoke test, automatically gather pod context and get an LLM-generated likely cause attached to the failure report.
Goal: deploy a small Retrieval-Augmented-Generation (RAG) stack — a vector DB pod plus an inference pod — into an ephemeral namespace, and run functional and latency tests against the whole pipeline in-cluster.
# two Deployments + two Services in one file, both in the ai-services namespace
apiVersion: apps/v1
kind: Deployment
metadata: { name: vector-db, namespace: ai-services }
spec:
replicas: 1
selector: { matchLabels: { app: vector-db } }
template:
metadata: { labels: { app: vector-db } }
spec:
containers:
- name: vector-db
image: qdrant/qdrant:latest
ports: [{ containerPort: 6333 }]
---
apiVersion: v1
kind: Service
metadata: { name: vector-db-svc, namespace: ai-services }
spec:
selector: { app: vector-db }
ports: [{ port: 6333, targetPort: 6333 }]
# rag-inference Deployment/Service (same shape) points RAG_VECTOR_DB_URL
# at "http://vector-db-svc.ai-services.svc.cluster.local:6333"
import requests
BASE_URL = "http://localhost:8080" # port-forwarded to rag-inference-svc
def test_rag_retrieves_relevant_context():
resp = requests.post(f"{BASE_URL}/ask", json={"question": "What is our refund policy?"})
body = resp.json()
assert resp.status_code == 200
assert len(body["sources"]) > 0, "Expected at least one retrieved source document"
assert "refund" in body["answer"].lower() # loose relevance check, not exact match
def test_rag_pipeline_end_to_end_latency():
import time
start = time.perf_counter()
requests.post(f"{BASE_URL}/ask", json={"question": "What is our refund policy?"})
assert time.perf_counter() - start < 8.0 # retrieval + generation, generous budget
This project is the most complete demonstration of the whole tutorial in one place: multi-service YAML, Service-to-Service DNS, readiness gating, ephemeral-namespace deployment, and AI-aware assertions, all in a single in-cluster test run.
Goal: generate a realistic, bursty request pattern (the kind real AI-feature traffic actually looks like), throw it at the LLM-inference Service, and assert that the HorizontalPodAutoscaler from earlier actually scales replicas up in response — closing the loop between autoscaling config and observed behavior instead of just trusting the YAML.
import subprocess, time, requests
from concurrent.futures import ThreadPoolExecutor
def get_replica_count(deployment, namespace="ai-services") -> int:
out = subprocess.run(
["kubectl", "get", "deployment", deployment, "-n", namespace,
"-o", "jsonpath={.status.availableReplicas}"],
capture_output=True, text=True,
).stdout
return int(out or 0)
def generate_bursty_load(base_url, total_requests=200, concurrency=20):
def _send_one(_):
try:
requests.post(f"{base_url}/generate", json={"prompt": "Summarize this order"}, timeout=15)
except requests.RequestException:
pass
with ThreadPoolExecutor(max_workers=concurrency) as pool:
list(pool.map(_send_one, range(total_requests))) # AI features see bursty, not steady, traffic
def test_hpa_scales_up_under_burst_load():
baseline = get_replica_count("llm-inference")
generate_bursty_load("http://localhost:8080") # port-forwarded to llm-inference-svc
scaled_up = False
for _ in range(30): # poll for up to ~2.5 minutes; HPA reacts on a delay
if get_replica_count("llm-inference") > baseline:
scaled_up = True
break
time.sleep(5)
assert scaled_up, f"Expected replica count to exceed baseline of {baseline} under load"
A rapid-fire review of the questions most likely to come up, plus the comparison table interviewers love to draw on a whiteboard.
Rolling update replaces pods gradually, keeping some old pods serving traffic until new ones are ready — zero downtime, but briefly runs two versions at once. Recreate kills all old pods before creating new ones — guaranteed no version overlap, but causes downtime.
By label selector, not pod name. The Service watches for any pod whose labels match its spec.selector and adds it to its list of Endpoints; kube-proxy then load-balances traffic across those endpoints.
Kubernetes uses a three-way merge (last-applied config, current live state, and the new manifest) to figure out what changed. Conflicting concurrent updates can trigger a resource-version conflict error, which is why automation should retry idempotently rather than assume every apply succeeds first try.
Most often insufficient resources on any node to satisfy the pod's requests, but also possible: no node matches a required node selector/affinity rule, a taint the pod doesn't tolerate, or an unbound PersistentVolumeClaim.
Combine a chaos test (delete the pod while a load generator is mid-flight) with an assertion on the client side — no failed requests if the Service has other healthy replicas, or a clean retry/error if it's a single-replica dependency. This also validates preStop hooks and graceful shutdown handling in the app itself.
Without requests, the scheduler can over-pack a node, and one noisy pod can starve others of CPU/memory — a frequent, hard-to-diagnose cause of "my tests are flaky in CI but pass locally."
| Comparison | Key difference |
|---|---|
| Docker vs Kubernetes | Docker builds and runs a single container. Kubernetes orchestrates many containers across many machines — scheduling, scaling, networking, and self-healing on top of a container runtime (which is often, but not always, Docker under the hood). |
| Deployment vs StatefulSet | Deployment pods are interchangeable — any replica can be replaced by any other. StatefulSet pods have stable, unique identities and storage (pod-0, pod-1...) — used for databases and anything that needs consistent identity across restarts. |
| Deployment vs DaemonSet | Deployment runs N replicas wherever the scheduler decides. DaemonSet runs exactly one pod per node (or per matching node) — used for node-level agents like log collectors or monitoring. |
| Job vs Deployment | Job runs a pod to completion and stops. Deployment keeps its pods running indefinitely, restarting them if they exit. |
Write a minimal Deployment named catalog-api running 2 replicas of image
myorg/catalog-api:1.0 on port 8000, plus a matching Service exposing port 80 → 8000.
apiVersion: apps/v1
kind: Deployment
metadata: { name: catalog-api }
spec:
replicas: 2
selector: { matchLabels: { app: catalog-api } }
template:
metadata: { labels: { app: catalog-api } }
spec:
containers:
- name: catalog-api
image: myorg/catalog-api:1.0
ports: [{ containerPort: 8000 }]
---
apiVersion: v1
kind: Service
metadata: { name: catalog-api-svc }
spec:
selector: { app: catalog-api }
ports: [{ port: 80, targetPort: 8000 }]A pod shows STATUS: CrashLoopBackOff and RESTARTS: 8. List the exact sequence of
kubectl commands you'd run, in order, to find the root cause.
kubectl get pods # confirm status + restart count
kubectl describe pod <pod> # check Events + "Last State: Terminated" reason
kubectl logs <pod> --previous # logs from the crashed instance, not the new restart attempt
kubectl get pod <pod> -o yaml | grep -A5 resources # rule out OOMKilled from a too-low memory limitUsing the kubernetes Python package, write a function count_ready_pods(namespace, label_selector) that returns how many pods matching the selector currently have all containers reporting ready.
from kubernetes import client
def count_ready_pods(namespace, label_selector):
v1 = client.CoreV1Api()
pods = v1.list_namespaced_pod(namespace, label_selector=label_selector)
ready_count = 0
for pod in pods.items:
statuses = pod.status.container_statuses or []
if statuses and all(c.ready for c in statuses):
ready_count += 1
return ready_countAn LLM-backed /summarize endpoint returns different valid summaries for the same input each
time. Write two assertions you could make that don't rely on exact string matching.
assert 10 < len(response["summary"].split()) < 100 # reasonable length bounds
assert all(kw in response["summary"].lower() for kw in ["refund"]) # key topic present29 questions spanning Kubernetes fundamentals, testing patterns, CI/CD, and AI workloads. Your score appears at the end.
kind locally and actually run the Deployment/Service YAML from this tutorial against a real (if tiny) cluster