Part II — Applied Engineering | Prerequisites: Chapter 3 | Difficulty: Intermediate
The infrastructure layer — how code gets from a developer's machine to production, and the platforms that make it repeatable, scalable, and observable.
- Infrastructure as Code (IaC)
- Containers & Orchestration
- CI/CD Philosophies
- Cloud-Native: The 12-Factor App
- Networking for Backend Engineers
- Platform Engineering
- Kubernetes Deep Dive
- Chapter 3 (architecture deployed)
- Chapter 12b (Docker/Terraform/kubectl hands-on)
- Chapter 15 (CI/CD pipelines)
- Chapter 19 (AWS infrastructure)
- Chapter 35 (Everything as Code — extends IaC to policies, secrets, observability, compliance, GitOps, and platform abstractions)
Picture this: it's 11 PM on a Friday. You've been called in because the app is down. You SSH into the production server — the one production server, the one that everyone's been treating like a beloved pet for three years — and you start poking around. Some config file was edited by hand six months ago. Nobody remembers why. The deployment process is a 47-step wiki page that hasn't been updated since your predecessor left. Two of the steps contradict each other. Step 31 just says "do the thing."
Does this feel familiar? Maybe it's not quite that bad where you work. But there's a spectrum, and a lot of teams are much closer to that end than they'd like to admit.
Here's what changed the game: the realization that infrastructure is just code. The servers, the networks, the load balancers, the deployment pipelines — all of it can be described, versioned, reviewed, tested, and reproduced. The 2 AM firefighting sessions? Optional. The snowflake production server that only Dave knows how to fix? Extinct.
This chapter is about the tools and mindsets that took us from "we SSH into prod and pray" to "we push to git and sleep soundly." And here's the thing — once you've lived on the right side of this transition, you never want to go back.
Here's the question that unlocks IaC: Why should your servers be any different from your application code?
Your app code lives in version control. It gets reviewed before it ships. If something breaks, you can roll it back. You can spin up a fresh copy of your app on any machine that has the right dependencies. But your infrastructure? In the old world, it lives in the heads of two engineers who've been at the company for seven years, plus a bunch of manual click-throughs in the AWS console that nobody documented.
Infrastructure as Code flips this. Your entire cloud environment — VPCs, subnets, security groups, EC2 instances, RDS clusters, IAM roles, the whole deal — lives in .tf files or TypeScript or Python. It gets checked into git. It gets code-reviewed. It has a history. If someone deletes a subnet by accident, you know exactly when it happened, who did it, and you can restore it in minutes.
The practical superpower: you can spin up a complete replica of production for a staging environment, or for an integration test, or just to experiment — and then tear it down. No orphaned resources, no forgotten $300/month EC2 instance from a POC two years ago. You know exactly what you have because you declared exactly what you have.
There are two fundamental approaches to describing infrastructure, and they reflect a genuinely interesting philosophical difference about how computers should understand your intentions.
Declarative (Terraform, CloudFormation, HCL): You describe the desired end state. "I want three EC2 instances of type t3.medium, in these subnets, with this security group." You don't say how to get there. The tool figures out what currently exists, computes the diff, and makes the minimum set of changes to reach your declared state.
This is powerful because the tool does the reasoning. If you've already got two instances and you want three, it adds one. If you change an instance type, it knows whether it can do that in-place or needs to recreate. The engineer's job is to describe the world they want, not to choreograph every API call.
Imperative (Pulumi, AWS CDK): You write actual code — TypeScript, Python, Go — that describes the steps to build your infrastructure. This gives you the full power of a real programming language: loops, conditionals, functions, type safety, unit tests, IDE autocompletion.
// Pulumi: real TypeScript, real logic
const instances = Array.from({ length: instanceCount }, (_, i) =>
new aws.ec2.Instance(`web-${i}`, {
ami: latestAmi.id,
instanceType: env === "production" ? "t3.large" : "t3.micro",
subnetId: privateSubnets[i % privateSubnets.length].id,
tags: { Name: `web-${i}`, Environment: env },
})
);You can do this in Terraform too (with HCL's count and for_each), but it's not as natural. The risk with imperative IaC is "too-clever infrastructure code" — engineers treating IaC as an opportunity to write abstract, generic frameworks when a simpler, more readable declaration would suffice. Infrastructure is already complex enough without adding layers of abstraction.
Terraform is the industry standard for a reason. The HCL DSL is readable even to people who don't write it. The provider ecosystem is massive — AWS, GCP, Azure, Datadog, PagerDuty, GitHub, basically anything with an API has a Terraform provider. State management is mature. The plan → apply workflow is predictable.
The concern: HashiCorp changed Terraform's license to BSL (Business Source License) in 2023, which restricts commercial use of the tool itself (though not your IaC code). This spawned the OpenTofu fork, which is the community's open-source continuation. For most teams, Terraform/OpenTofu is still the right call. Just know the license situation if you're building tooling around it.
# Classic Terraform: readable, explicit, stateful
resource "aws_instance" "web" {
ami = data.aws_ami.ubuntu.id
instance_type = var.instance_type
vpc_security_group_ids = [aws_security_group.web.id]
subnet_id = aws_subnet.private.id
tags = {
Name = "web-server"
Environment = var.environment
ManagedBy = "terraform"
}
}Pulumi shines when your infrastructure logic is genuinely complex — when you need to pull data from an API to determine resource counts, or when you want to share infrastructure components as versioned npm packages across teams. The TypeScript types catch errors at compile time that Terraform would only catch at apply time.
AWS CDK is essentially Pulumi but AWS-only and compiling to CloudFormation. Deep AWS integration is the upside. Vendor lock-in and CloudFormation's sometimes-tortured limits are the downside. If you're all-in on AWS and want to write Python, CDK is excellent.
The honest take: start with Terraform. Learn it well. Graduate to Pulumi if you hit its limitations.
Terraform's state file is the map between your HCL declarations and the real resources in your cloud provider. When Terraform runs, it reads the state to know what exists, then compares to your config to know what to change, then updates the state after making changes.
If you run Terraform from your laptop with a local terraform.tfstate, you've already found a footgun. The moment a second engineer runs terraform apply, they have different state, and now your resources and your state are out of sync. Things get deleted that shouldn't. Resources get recreated. It's chaos.
Remote state is not optional for teams. The canonical AWS setup:
terraform {
backend "s3" {
bucket = "my-company-terraform-state"
key = "production/terraform.tfstate"
region = "us-east-1"
encrypt = true # State files contain secrets
# DynamoDB table prevents two people from applying simultaneously
dynamodb_table = "terraform-state-lock"
}
}The DynamoDB lock table is critical. It prevents two engineers from running terraform apply simultaneously and corrupting the state. Without it, you're relying on coordination and trust — which works fine until it doesn't, usually at the worst possible moment.
State files contain secrets. Database passwords, API keys, private key material — it ends up in the state file in plaintext. Always encrypt at rest (the encrypt = true flag with S3 + KMS). Always enable versioning on the S3 bucket so you can recover from a corrupt state.
Terraform Cloud and Atlantis are popular solutions that wrap the state backend with a more complete workflow: locking, audit logs, per-PR plan previews, approval gates before applies.
Here's a scenario that happens constantly: someone manually changes a security group rule in the AWS console because there's a production incident and there's no time to go through the IaC workflow. The incident is resolved. The manual change is never codified. Now your Terraform state says one thing, your IaC config says a second thing, and the actual cloud resource is a third thing. This is called drift.
Periodic terraform plan in CI (even without applying) will show you what's drifted. Run it on a schedule — daily at minimum, hourly for critical infrastructure — and alert when there are unexpected changes.
For more sophisticated drift detection, AWS Config continuously monitors resource configurations and can alert on rules violations. In the GitOps world (more on this in a moment), you have a controller that continuously reconciles desired state against actual state, making drift detection nearly real-time.
Old way: treat servers like pets. Give them names. Nurse them back to health when they're sick. SSH in and patch them, upgrade packages, tweak configs. Every server accumulates a unique history.
New way: treat servers like cattle. They're numbered, not named. When one gets sick, you don't fix it — you replace it. You never mutate a running server in production.
The workflow for immutable infrastructure:
- Change the Dockerfile or base AMI (in code, in git)
- Build a new immutable image
- Deploy the new image alongside the old one
- Route traffic to the new deployment
- Verify health, then terminate the old instances
No more configuration drift. No more "it works on staging because staging was provisioned six months ago with different packages." Production and staging were built from the same image artifact at the same point in time. The image is your source of truth.
This is why containers have become so dominant — a Docker image is the ultimate immutable artifact. It contains the app, its runtime, its dependencies, all baked in. If it passes tests in CI, it will behave the same way in production.
GitOps takes IaC one step further: git isn't just where you store your infrastructure code, it's the authoritative source of truth for the current desired state of your infrastructure. Nothing gets deployed except through a git commit. Every change is a pull request. Your audit log is your git history.
Two deployment models:
Push-based: Your CI system (GitHub Actions, Jenkins) pushes changes to the cluster after a merge. Simple, familiar. The downside: your CI system needs credentials to reach your cluster — and those credentials, if leaked, give an attacker write access to production.
Pull-based: An agent running inside the cluster (Argo CD, Flux) watches the git repo and pulls changes in. The cluster doesn't need to be reachable from the outside. Your git repo doesn't need credentials to access the cluster. The attack surface is smaller, and the reconciliation is continuous — if someone manually changes something in the cluster, the GitOps controller will revert it. The cluster self-heals toward the desired state in git.
Pull-based GitOps is the more secure model, and for production Kubernetes workloads, it's become close to the standard approach.
To understand why containers mattered so much, you need to understand what software deployment looked like before them.
The Before Times, in depth:
The year is 2012. You have a Python web application. To deploy it, you need to:
- Provision a server (either physical hardware or a virtual machine — both require a ticket to your ops team, or a lengthy AWS console session)
- SSH in and install the OS dependencies: Python 3.8 (but the OS comes with Python 2.7), libpq for PostgreSQL support, libxml2 for XML parsing, maybe a C compiler to build some packages from source
- Set up a Python virtualenv to isolate your dependencies from the system Python
- Clone your application code and install requirements
- Configure your application: environment variables, config files, database URLs
- Set up a process supervisor (supervisord, or SystemD) to keep your app running after crashes
- Configure nginx as a reverse proxy in front of your app
- Add your SSL certificates
- Set up logrotate so your log files don't fill the disk
- Configure firewall rules
That's for one server. Now imagine doing it across 10 servers. And keeping all 10 consistent as you install security patches, upgrade Python versions, add new dependencies, and change configuration. Now imagine doing this across staging and production environments that need to be equivalent but keep drifting apart because different engineers made different manual changes.
This is not a hypothetical. This was the reality for most software teams.
What changed with containers:
Docker, released in 2013, popularized a solution to this entire class of problems. The core insight: rather than deploying code onto a server, deploy a complete, self-contained environment that includes the code.
A Docker image is built from a Dockerfile — a precise, reproducible recipe that starts from a known base (Ubuntu 22.04, Python 3.11-slim, Node 20-alpine) and layers on every dependency, configuration, and application file in a specified order. The result is an immutable artifact: a content-addressed, versioned snapshot of everything needed to run the application.
When that image runs as a container on any machine that has a container runtime:
- The same exact Python version runs (not whatever was installed on the host)
- The same exact library versions are present (not whatever was installed for another app)
- The same exact file system layout exists (not the accumulated history of six years of manual changes)
- The same exact environment variables are set (not leaked from the host shell)
This solved something that organizations had been quietly suffering with for decades: environment parity. Your CI server, your staging environment, and your production servers now all run identical environments. The bug you reproduce locally is the same bug in production, because local is the same as production.
The second-order effect: the deployment unit changed.
Before containers, the deployment unit was a server. You updated servers by mutating them — SSHing in, pulling new code, restarting processes. The server accumulated history. It had "drift" — subtle differences from a clean build that nobody remembered introducing.
With containers, the deployment unit is an image. You don't mutate running containers; you replace them. Update your application? Build a new image, pull it to your servers, stop the old container, start the new one. Roll back? Start the old image. The image is immutable and its behavior is deterministic.
This shift from mutable servers to immutable images cascaded through the entire industry:
- Local development became reliable:
docker-compose upgives every developer the same stack. "It works on my machine" became "it works in the container" — which means it works everywhere. - CI/CD became consistent: Your CI environment is a container. Your tests run in the same environment as production. The infamous "it passes locally but fails in CI" bug became rare.
- Scaling became trivial: Starting 10 more instances of a container takes seconds. Kubernetes can do it automatically. There's no "set up a new server" step.
- Security posture improved: Containers have isolation boundaries. An application running in one container can't directly access the filesystem or processes of another container.
- Rollbacks became instant: Keep the last known-good image tag. If the new version has a problem, deploy the old tag. No "undo the last 23 manual changes."
The Docker ecosystem effect:
Docker also standardized how software is distributed. Before Docker Hub, installing something like Elasticsearch involved downloading a tarball, configuring Java, setting up environment variables, and hoping you followed the right documentation for your OS. After Docker:
docker run -p 9200:9200 elasticsearch:8.11That's it. The same command works on macOS, Linux, Windows. The Elasticsearch developers control exactly what's in that image. There's no "it depends on your system configuration." This made local development stacks (Compose files with Postgres, Redis, Kafka, Elasticsearch) straightforward in a way that previously required dedicated "DevOps" knowledge.
The effect on the industry was compounding: as containers became standard, the tooling improved (Docker Desktop, multi-platform builds, BuildKit), the registries improved (Docker Hub, ECR, GCR, GHCR), and the orchestration problem became the next thing to solve — which is where Kubernetes enters the story.
A Dockerfile that works and a Dockerfile that's production-ready are different things. Here's what separates them:
Multi-stage builds are the single biggest improvement you can make to most Dockerfiles. The idea: use one stage with build tools to compile your app, then copy only the output into a clean, minimal runtime image. The build tools — compilers, test frameworks, dev dependencies — never ship to production.
# Stage 1: Build
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
RUN npm run build
# Stage 2: Runtime
# Only the build output lands here — no node_modules from dev deps,
# no source code, no build tools
FROM node:20-alpine AS runtime
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
EXPOSE 3000
# Run as non-root — because running as root inside a container
# is a significant security risk
USER node
CMD ["node", "dist/index.js"]The resulting image is dramatically smaller (often 90% smaller) and has a vastly reduced attack surface. Smaller images pull faster, scan faster, and start faster.
Pin base image digests, not just tags. FROM node:20-alpine looks pinned, but it's not. The node:20-alpine tag can be updated, and the image you pull today might not be the one you pull tomorrow. Use the digest: FROM node:20-alpine@sha256:abc123.... Now the exact image is pinned forever. CI should rebuild when you intentionally update, not randomly when upstream updates.
Layer ordering is not just style, it's performance. Docker caches layers. If a layer changes, all subsequent layers are invalidated. Put the things that change rarely at the top (base image, system packages, lockfiles), and the things that change frequently at the bottom (your source code). In a Node.js project:
COPY package*.json ./ # Changes rarely: cache until deps change
RUN npm ci # Cached when lockfile unchanged
COPY . . # Changes constantly: always rebuilds from hereIf you copy source code before npm ci, every code change invalidates the npm install layer. Every build installs all dependencies from scratch. That 2-minute CI pipeline becomes 8 minutes for no reason.
One process per container. If you need a web server and a background worker, run two containers. This is one of the core container design principles — it makes scaling, monitoring, and failure handling infinitely cleaner. You can scale the web servers without scaling the workers, restart a crashed worker without restarting the web servers.
Scan images for vulnerabilities. Trivy and Grype are excellent. Run them in CI and fail the build on high/critical CVEs. Base images accumulate vulnerabilities over time — update regularly, and automated scanning is how you know when "regularly" needs to mean "now."
Kubernetes is a container orchestration platform. It answers questions like: which node should this container run on? What happens when a node goes down? How do you roll out a new version without downtime? How do containers find each other on the network?
Understanding the architecture isn't just academic — when things go wrong (and they will), knowing what each component does is the difference between a 5-minute fix and a 3-hour mystery.
The Control Plane is the brain of the cluster:
- kube-apiserver: Every interaction with Kubernetes goes through here.
kubectl, operators, CI systems — they all talk to the API server. It validates requests, stores state in etcd, and broadcasts changes. - etcd: The distributed key-value store that holds all cluster state. If etcd is healthy, your cluster can recover from almost anything. If etcd is gone, your cluster is gone. Back it up.
- kube-scheduler: Watches for pods that haven't been assigned to a node and picks the best node for each one, based on resource requests, node affinity rules, taints, and tolerations.
- kube-controller-manager: Runs the control loops that reconcile desired state with actual state. The Deployment controller, for example, watches Deployments and ensures the right number of pods are running.
The Node is where your workloads actually run:
- kubelet: The agent on each node. Talks to the API server, ensures containers are running as specified.
- kube-proxy: Handles network routing for Services — maintains iptables or IPVS rules that route traffic to the right pods.
- container runtime: Usually containerd (Docker used to be here, was replaced). Actually starts and stops containers.
The Core Resources you'll use every day:
| Resource | Purpose |
|---|---|
| Pod | Smallest deployable unit (1+ containers sharing network and storage) |
| Deployment | Declarative updates, rolling deployments, rollback |
| Service | Stable network endpoint for a set of pods (ClusterIP, NodePort, LoadBalancer) |
| StatefulSet | Like Deployment but with stable identity and ordered operations — databases |
| DaemonSet | One pod per node — log collectors, monitoring agents, network plugins |
| Job / CronJob | Batch and scheduled workloads |
| ConfigMap / Secret | Configuration and sensitive data injected into pods |
| Ingress | HTTP routing rules, typically managed by an Ingress Controller |
Kubernetes is the right answer for many production deployments. It is not the right answer for all of them, and the industry's enthusiasm for Kubernetes has resulted in a significant amount of over-engineering at smaller scales. Let's be explicit about when Kubernetes isn't worth it.
The operational complexity is real and significant.
Kubernetes has a steep learning curve. Understanding Kubernetes networking (CNI plugins, ClusterIP vs NodePort vs LoadBalancer, kube-proxy iptables rules, CoreDNS, Ingress controllers) takes weeks of study and months of hands-on experience. Understanding resource management (requests vs limits, QoS classes, eviction policies, OOMKill forensics) takes more. Understanding security (RBAC, NetworkPolicy, PodSecurityStandards, Secrets management, IRSA) takes more still.
For a team of 5 engineers building a product, having even one person spend 30-40% of their time managing Kubernetes infrastructure is expensive. That's engineering time not spent on features. Before adopting Kubernetes, ask honestly: do we have the bandwidth to operate this well?
Kubernetes has a meaningful minimum viable infrastructure cost.
A minimally viable production Kubernetes cluster (EKS, GKE, or AKS) with a managed control plane, a minimum of 2-3 worker nodes for high availability, an Ingress controller, a certificate manager, a monitoring stack, and a GitOps tool runs $300-600/month before you put any workloads on it. For a startup with 1000 users, that's often more expensive than the workloads themselves. Compare: a single EC2 t3.medium running your application + RDS PostgreSQL + ElastiCache Redis might total $80-150/month and handle the same traffic with a fraction of the operational overhead.
The breakeven point for Kubernetes:
Kubernetes starts to make sense when multiple or more of these conditions are true:
- You have multiple services (>5) that need independent scaling, independent deployment cadences, and resource isolation
- You have meaningful traffic that requires horizontal scaling of individual services
- You have a team with someone who either already knows Kubernetes or has the time to learn it properly
- You need the specific features: advanced deployment strategies (canary, blue-green), auto-scaling based on custom metrics, workload isolation via namespaces, sophisticated health checking
- Your cloud spend makes the cluster overhead worthwhile: if you're spending $3000+/month on compute anyway, the $400 cluster overhead is noise
What to use instead of Kubernetes:
AWS App Runner / Google Cloud Run / Azure Container Apps: Managed container platforms that take your Docker image, handle scaling (including scale-to-zero), and charge by actual usage. The operational overhead is near-zero. You don't configure load balancers, auto-scalers, or health checks — the platform handles it. For services that are stateless and HTTP-based, this is often the right answer up to significant scale.
ECS (Elastic Container Service): AWS's container orchestration service that's simpler than EKS. No control plane to manage, no etcd to worry about, deep AWS integration (IAM, ALB, CloudWatch). ECS Fargate removes even the node management overhead. ECS isn't as capable as Kubernetes for complex workloads, but for many teams it's 80% of the benefit with 20% of the complexity.
Fly.io / Railway / Render: For smaller applications, these platforms deploy Docker containers without any infrastructure management. You fly deploy and it runs. The operational overhead is minimal. Not suitable for enterprise-scale workloads, but excellent for side projects, internal tools, and early-stage products.
Single well-configured EC2 instances: Don't laugh. A single EC2 t3.large with Docker Compose running your entire stack, behind a load balancer, with automated backups and monitoring, can handle tens of thousands of daily active users. Simple to understand, simple to debug, simple to operate. The horizontal scaling story is weaker, but for many products, vertical scaling and a reliable single server is sufficient and worth the simplicity.
The honest take: Start without Kubernetes. Graduate to it when you have evidence that your current infrastructure is insufficient for your needs, and when you have the team bandwidth to operate it properly. The engineers who've operated Kubernetes in production for years are rightfully enthusiastic about it — it solves real problems at scale. But the problems it solves are problems you should confirm you have before paying the complexity tax.
Don't use Kubernetes unless your monthly compute bill justifies the operational overhead. A good rule of thumb: if you're spending less than $1,000/month on compute, the $400+ baseline K8s overhead (and the engineering time to run it) is almost certainly not worth it. If you're at $3,000+/month in compute, the cluster overhead starts to become noise.
All figures are ballpark estimates as of 2025 — check current pricing before budgeting.
| Option | Approximate Monthly Cost | Operational Overhead |
|---|---|---|
| EKS/GKE control plane fee | ~$75/mo | — (managed) |
| Minimum viable K8s cluster (3 nodes, t3.medium) | ~$300–600/mo before workloads | High — networking, RBAC, upgrades, monitoring |
| Production K8s (3 nodes + ingress + monitoring + GitOps) | ~$600–1,200/mo baseline | Very high |
| ECS Fargate (equivalent stateless workload) | ~$150–400/mo | Low — no node management |
| Google Cloud Run (equivalent HTTP workload) | ~$50–200/mo | Near-zero |
| AWS App Runner (containerized HTTP service) | ~$25–150/mo | Near-zero |
The Fargate and Cloud Run rows aren't inferior options — they're often the right option. Cloud Run can auto-scale to zero (great for low-traffic services), handles HTTPS automatically, and charges per request-millisecond. For stateless services getting less than a few million requests per day, Cloud Run or App Runner typically costs less and requires less engineering time than Kubernetes.
If you're at a company that's already on Kubernetes, learn it deeply and use it well. If you're at an early-stage company choosing your stack, strongly consider starting with something simpler and revisiting the choice when you have real scaling requirements.
A concrete migration path that works:
-
Start on a managed container platform (Cloud Run, App Runner, Railway). Zero infrastructure management, reasonable cost, automatic scaling. This handles most early-stage companies indefinitely.
-
Graduate to ECS Fargate when you need more control: custom networking, spot instances, scheduled tasks, VPC integration, more complex IAM requirements. Still no node management. ECS can carry you to significant scale.
-
Graduate to EKS (managed Kubernetes) when you genuinely need: multi-container pods with shared networking, custom schedulers, complex operator patterns, advanced traffic management, or when you have specific workload types (ML training, GPU workloads, batch processing with complex dependencies) that need Kubernetes-specific capabilities.
At each stage, the migration is motivated by a specific technical requirement you couldn't meet at the previous level — not by a desire to use the most sophisticated tool available. This approach keeps your infrastructure complexity proportional to your actual needs, which is one of the hardest engineering disciplines to maintain.
A Service Mesh sounds intimidating, but the core idea is simple: instead of building security, observability, and traffic management into every service, you inject a sidecar proxy into every pod and let the proxy handle all of that at the network layer.
Istio and Linkerd are the two main players. Both work by injecting a lightweight proxy (Envoy in Istio's case) alongside your application container. All network traffic flows through the proxy. The proxy can:
- mTLS: Automatically encrypt and authenticate all service-to-service traffic. No more "well, anyone inside the cluster can talk to anything" security model.
- Observability: Emit metrics, traces, and logs for every request, without any instrumentation code in your services.
- Traffic management: Weighted routing (send 5% of traffic to a new version), circuit breaking, retry policies, timeouts — all configurable without changing application code.
- Authorization policies: "Service A is allowed to call Service B on these paths, Service C is not."
The honest cost assessment: service meshes add complexity, latency (every request goes through two proxy hops), and a significant operational burden. Istio, in particular, has a steep learning curve and a history of difficult upgrades.
Use a service mesh when: You have many services (10+), you need consistent mTLS across your entire mesh, and you have the platform engineering bandwidth to operate it.
Avoid a service mesh when: You have 5 services and you're implementing one because it sounds cool. The operational cost will exceed the benefit for years.
A Kubernetes Operator is a custom controller that extends Kubernetes with operational knowledge for a specific domain. The concept: Kubernetes is great at running stateless apps, but running a Postgres cluster requires knowledge about backups, failover, schema migrations, replication topology — things Kubernetes itself doesn't understand.
An Operator encodes that knowledge. You declare a Postgres custom resource, and the Postgres Operator knows how to:
- Bootstrap a primary and replicas
- Handle failover when the primary goes down
- Manage backups on a schedule
- Apply schema migrations safely
From your perspective, managing Postgres in Kubernetes looks the same as managing a Deployment. You declare what you want; the Operator makes it happen. The complexity is encapsulated in the Operator, not scattered across runbooks.
Popular production-grade Operators: cert-manager (TLS certificates), Prometheus Operator (monitoring), Zalando Postgres Operator, Strimzi (Kafka), Argo CD (GitOps).
For hands-on GitHub Actions mastery — reusable workflows, OIDC, self-hosted runners, and more — see Chapter 33: GitHub Actions Core and Chapter 33b: Advanced GitHub Actions.
These three terms are often conflated, but they describe importantly different points on a spectrum of automation confidence:
Continuous Integration (CI) is the foundation. Every developer merges to mainline frequently — at least daily, ideally multiple times a day. Every commit triggers an automated build and test run. The invariant: the main branch is always passing. When it's not, fixing it is the team's highest priority. CI is about integration hygiene — catching the merge conflicts and compatibility breaks early, when they're cheap to fix.
Continuous Delivery extends CI to deployment: every change that passes CI is releasable. The deploy to production is a one-click (or one-command) operation. You might not deploy every commit, but you could. The decision of when to release is a business decision, not a technical one held hostage by complex, risky deployment procedures.
Continuous Deployment removes the human from the release decision entirely. Every commit that passes CI is automatically deployed to production. No approval gate, no manual trigger. This is only viable when you have excellent test coverage, solid monitoring, fast rollbacks, and a high deployment frequency (high frequency makes each individual deployment low-risk).
The progression isn't purely technical — it's cultural. CD requires trusting your tests enough to deploy without human review. That trust is earned over time with good test practices, good monitoring, and a track record of reliable automated deployments.
Long-lived feature branches are the enemy of continuous integration. If you're working on a branch for two weeks, you're not integrating — you're diverging. The longer the branch lives, the bigger the merge, the more conflicts, the higher the risk.
Trunk-based development means everyone commits to main (or very short-lived branches that merge within 1-2 days). But how do you merge incomplete features? Feature flags.
Feature flags decouple deployment from release. You deploy the code for a new feature, but it's behind a flag that's off. It ships to production harmlessly. When you're ready to release (after QA, after a stakeholder review, in a controlled rollout), you flip the flag. If something goes wrong, you flip it back.
This is powerful beyond just trunk-based development:
- Percentage rollouts: Enable the feature for 1% of users, watch metrics, ramp up
- User targeting: Enable for internal users first, then beta users, then everyone
- Kill switches: If a feature causes problems in production, turn it off in 30 seconds without a deploy
LaunchDarkly, Unleash, Flagsmith — there are good managed and self-hosted options. For many teams, a simple Redis-backed feature flag implementation serves the need without the added dependency.
Choosing how you deploy is choosing how you manage risk. There's no universally correct answer — the right strategy depends on your risk tolerance, infrastructure budget, and rollback requirements.
| Strategy | Downtime | Infra Cost | Rollback | Risk |
|---|---|---|---|---|
| Blue-Green | Zero | 2x | Instant | Low |
| Canary | Zero | +small % | Fast | Lowest |
| Rolling | Zero | +1 instance | Slow | Medium |
| Recreate | Yes | 1x | Redeploy | Highest |
Deployment Strategies in Practice — A Walkthrough
Understanding the strategies conceptually is the easy part. Understanding how they actually feel to run in production, and what can go wrong, is what separates engineers who've done them from engineers who've only read about them.
Blue-Green in practice:
Blue-green gives you the fastest possible rollback (load balancer flip, typically a DNS change or ALB target group swap that takes seconds) at the cost of running double the infrastructure continuously.
The setup in AWS: two Auto Scaling Groups (or ECS Services), both behind an ALB. Blue is the current production, receiving 100% of traffic via a target group. Green is idle (potentially scaled down to minimum capacity to save cost). Your deployment pipeline:
- Scale green group up to full production capacity
- Deploy the new version to green
- Run smoke tests against green (using an internal DNS entry that points directly to green)
- Flip the ALB listener to point at green's target group (takes effect in seconds)
- Watch your monitors for 5-10 minutes
- If healthy: deregister blue. If problem: flip back.
The hidden complexity: session state. If your application stores sessions in memory rather than in Redis or a database, users hitting the new blue/green boundary will lose their sessions. Blue-green assumes stateless services. Sticky sessions at the ALB can help short-term but don't solve the fundamental problem.
Database migrations are the other challenge. Blue-green deployment of your application layer doesn't help if your database schema change breaks the old version. The rule: database changes must be backward-compatible before you swap. Deploy a schema that works with both old and new app versions, then swap, then (optionally) clean up the compatibility shim.
Canary in practice:
Canary is more sophisticated than blue-green but gives you risk management that blue-green doesn't: you control exactly what percentage of users experience the new version, and you can make data-driven decisions about whether to proceed.
The mechanics in Kubernetes:
# Production: 9 replicas of the stable version, 1 of the canary
# Kubernetes Services load-balance across all Pods matching the selector
# stable-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-stable
spec:
replicas: 9
selector:
matchLabels:
app: api
track: stable
template:
metadata:
labels:
app: api
track: stable
version: v1.5.2
spec:
containers:
- name: api
image: myapp/api:v1.5.2
---
# canary-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-canary
spec:
replicas: 1 # 1 of 10 total = 10% canary traffic
selector:
matchLabels:
app: api
track: canary
template:
metadata:
labels:
app: api
track: canary
version: v1.6.0
spec:
containers:
- name: api
image: myapp/api:v1.6.0
---
# service.yaml - routes to ALL pods matching app: api, regardless of track
apiVersion: v1
kind: Service
metadata:
name: api
spec:
selector:
app: api # matches both stable and canary podsThis gives you a 90/10 traffic split with 9 stable pods and 1 canary pod. The split adjusts by changing replica counts.
For more granular traffic control (sending specific users to canary, or doing 1% without needing 100x more pods), you need a traffic management layer — Istio's VirtualService, AWS ALB weighted routing, or Argo Rollouts. These operate at the request level rather than the pod count level:
# Argo Rollouts: gradually shift traffic without changing replica counts
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: api
spec:
strategy:
canary:
canaryService: api-canary
stableService: api-stable
steps:
- setWeight: 5 # 5% to canary
- pause: {duration: 10m}
- setWeight: 20
- pause: {duration: 10m}
- setWeight: 50
- pause: {duration: 10m}
- setWeight: 100 # full promotion
analysis:
templates:
- templateName: error-rate-check
startingStep: 1What you monitor during a canary deployment:
This is the piece most teams underinvest in. The canary is only useful if you have the right metrics to make the go/no-go decision. At minimum:
- Error rate (5xx responses): canary should not have a materially higher error rate than baseline
- p99 latency: canary latency should not degrade significantly relative to stable
- Business metrics: for revenue-impacting flows, monitor conversion rate, cart abandonment, successful payment rate
If your error rate alert fires for the canary, you have two options:
- If the error rate is low (< 1%): continue monitoring, likely noise
- If the error rate is elevated (> 2-3x baseline): roll back immediately by scaling canary to 0 replicas
Rolling deployment in practice:
Rolling is the Kubernetes default for Deployments and is what you get if you don't configure anything else:
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # allow 1 extra pod during rollout (so you never drop below replica count)
maxUnavailable: 0 # never take a pod down before a new one is readyWith maxUnavailable: 0 and maxSurge: 1, the rollout process is: start a new pod with the new version → wait for it to be ready (readiness probe passes) → terminate one old pod → repeat until all pods are updated.
The critical thing that makes rolling updates work: readiness probes. If your readiness probe is too simple (just checks if the process is running, not if it's actually ready to serve traffic), you'll get traffic sent to pods that are booting up and not ready. The readiness probe should check all the dependencies your pod needs to be ready: can it connect to the database? Can it reach the cache? Is the warm-up complete?
readinessProbe:
httpGet:
path: /health/ready # returns 200 only when truly ready to serve traffic
port: 8080
initialDelaySeconds: 10 # give the app time to start before first check
periodSeconds: 5
failureThreshold: 3 # fail after 3 consecutive failures (15 seconds)Rolling update + good readiness probes = zero-downtime deployments with no extra infrastructure cost. This is the correct default for most stateless services.
The 12-Factor App methodology, articulated by Heroku engineers over a decade ago, is a collection of principles for building apps that run well in modern cloud environments. It sounds abstract until you try to run an app that violates one of the factors — then you understand exactly why each one exists.
The methodology emerged from painful experience: apps that couldn't scale horizontally because they stored state on disk. Apps that broke in staging because config was hardcoded. Apps that leaked processes because they didn't handle shutdown signals. Each factor is a lesson learned from something that went wrong at scale.
Here's the full set, with the why behind each one:
| Factor | Principle | Why It Matters |
|---|---|---|
| I. Codebase | One repo, many deploys | Multiple repos per app creates dependency hell and makes it hard to reason about what's deployed where |
| II. Dependencies | Explicitly declare and isolate | Never assume system packages exist. Declare everything in a manifest (package.json, requirements.txt, go.mod) |
| III. Config | Store in environment variables | Config that differs between environments (dev/staging/prod) must not be in the code. If your config is in a file that gets committed, you'll eventually commit a secret |
| IV. Backing Services | Treat as attached resources (swap via config) | Your database, your cache, your message queue — treat them all as services you connect to via URL. Swap prod for a local instance just by changing an env var |
| V. Build, Release, Run | Strictly separate stages | Build produces an artifact. Release combines the artifact with config. Run executes the release. Never mix these — you should be able to roll back a release without rebuilding |
| VI. Processes | Stateless, share-nothing | If your process dies and restarts, nothing should be lost. Don't store state in memory or on the local filesystem |
| VII. Port Binding | Self-contained, export via port | The app is responsible for starting its own HTTP server and binding to a port. No dependence on injected runtime |
| VIII. Concurrency | Scale out via process model | Scale by running more processes, not by making processes bigger or more complex |
| IX. Disposability | Fast startup, graceful shutdown | Processes can be started or stopped at any moment. Fast startup enables rapid deployment. Graceful shutdown (finishing in-flight requests) prevents data corruption |
| X. Dev/Prod Parity | Minimize gaps between environments | The same backing services, the same OS, the same runtime. The smaller the gap, the smaller the surprise when you deploy |
| XI. Logs | Treat as event streams (stdout) | Don't write to log files. Don't manage log rotation. Write to stdout; let the platform collect, route, and store |
| XII. Admin Processes | Run as one-off processes in same environment | Database migrations, one-time scripts — run them in the same environment as your app, not on a special "admin server" |
The factors that trip up teams most often: III (Config) — secrets keep ending up in code. VI (Processes) — stateful session handling that breaks when you scale to more than one instance. IX (Disposability) — apps that take 90 seconds to start, making rolling deployments painful.
The consequences when each factor is violated:
I (Codebase): Multiple repos that share code via copy-paste lead to divergence. You fix a bug in the payments module, but it exists in 3 repos, and you only fix 2 of them. The third bites you in production 6 months later. One repo, many deploys.
II (Dependencies): An app that assumes curl is on the system PATH breaks on the one EC2 AMI that doesn't have it installed. An app that assumes a specific version of libssl breaks when the OS is patched. Declare everything in a manifest (package.json, requirements.txt, go.mod) and use containers to isolate from host packages.
III (Config): A developer commits a production database URL to git to "quickly test something." That URL contains credentials. Git history is forever. The credentials are rotated, but the git history remains, and someone finds them a year later. The violation: config that differs between environments must not live in code. Environment variables, secrets managers, or runtime config injection — not hardcoded values, not config files committed to the repository.
IV (Backing Services): Hardcoding postgresql://prod-db.internal/app means that when you want to run integration tests against a test database, you have to change code. Treat the database URL as an attached resource: postgresql://{DATABASE_URL}. Swapping to a local database, a test database, or a replica is a config change, not a code change.
V (Build, Release, Run): An application that builds its configuration at runtime (fetching secrets, generating config files during startup) violates the separation between release and run. If the configuration generation fails, the app won't start — and you can't easily reproduce the exact release artifact that ran in production last Tuesday. The build produces an artifact; the release combines the artifact with the configuration; the run executes. These stages must be strictly separate and the release artifact must be immutable.
VI (Processes): The classic violation: storing user sessions in memory. Works fine with one server. The moment you add a second server for load, 50% of users get logged out on every request that routes to the wrong server. Store sessions in Redis. Store uploaded files in S3. Store generated reports in object storage. If your app restarts, users should not notice.
VII (Port Binding): An app that requires being deployed behind Apache or Nginx to function is harder to run locally (you need to set up nginx) and harder to containerize (you need nginx in the container too). An app that binds its own HTTP server to a port (Express, Gunicorn, Jetty) is self-contained and can be tested with curl localhost:3000. Keep the reverse proxy as a separate, composable layer.
VIII (Concurrency): An app that spawns internal threads for background work can't be scaled at the individual thread level. You scale the whole process. If your background jobs need more capacity than your web serving, you can't independently scale them. Structure work as processes: a web process type, a worker process type, a scheduler process type. Scale each independently.
IX (Disposability): Slow startup means slow deployments, slow auto-scaling responses, and slow recovery from crashes. Fast startup is a feature with concrete, measurable consequences:
- A 3-minute startup + 10 instances = 30-minute deployments (users experience degradation throughout)
- A 10-second startup + 10 instances = 90-second deployments (users barely notice)
- Auto-scaling that takes 5 minutes to add capacity during a traffic spike is nearly useless for handling spike traffic
- A crashed instance that takes 3 minutes to restart has users experiencing errors for 3 minutes
Graceful shutdown matters equally. On SIGTERM, finish in-flight requests, close database connections, commit work-in-progress. An app that exits immediately on SIGTERM has in-flight requests that fail with connection reset errors. Handle the signal, drain gracefully (typically within 30 seconds), then exit.
X (Dev/Prod Parity): The "works on my machine" problem is often a dev/prod parity problem in disguise. Different OS versions, different database major versions, different time zones, different locale settings. Containers dramatically help here (same image in dev and prod), but they don't solve everything — production has production data volumes, production network latency, production memory pressure. Keep the gap as small as possible.
XI (Logs): An app that writes logs to /var/log/app.log requires the server to have that directory, requires log rotation configuration (or the disk fills up), and requires SSH access to read the logs. An app that writes to stdout requires nothing from the server and plugs into any log aggregation system (ELK, Loki, CloudWatch) without modification. Write to stdout. Let the platform collect it.
XII (Admin Processes): Running database migrations from your laptop against the production database, with production credentials on your local machine, is a 2 AM incident waiting to happen. Run migrations from the same image that runs your app, in the same environment, as an automated step in your deployment pipeline. The migration is code that should be versioned, tested, and deployed the same way as application code.
Factor IX deserves extra attention because its consequences are concrete. If your app takes 3 minutes to start, a rolling deployment on 10 instances takes 30 minutes and leaves users experiencing slow responses throughout. If it takes 10 seconds, the whole deployment is done in under 2 minutes. Fast startup is a feature. Design for it.
Backend engineers who don't understand networking are constantly mystified by a class of problems that are actually straightforward once you have the right mental model. Why is this request slow even though the server is fast? (Head-of-line blocking.) Why does the app work but users in Asia get 800ms latency? (Missing CDN edge.) Why does DNS change propagation take forever? (TTL.)
Let's build those mental models.
HTTP/1.1 is how the web ran for decades. One request at a time per connection (mostly). To parallelize, browsers opened multiple connections — 6, 8, sometimes more. Servers handled hundreds of half-idle connections. It worked, but inefficiently.
HTTP/2 fixed the major bottlenecks with multiplexing: many requests over a single TCP connection. Requests don't block each other at the HTTP layer. Headers are compressed (HPACK). The protocol is binary, not text.
But there was a subtle problem: TCP head-of-line blocking. TCP guarantees ordered delivery. If a packet is lost, TCP waits for retransmission before delivering anything that came after it — even data from completely unrelated HTTP/2 streams. One lost packet blocks all streams.
HTTP/3 solves this by abandoning TCP entirely. It uses QUIC, a protocol built on UDP. QUIC implements its own reliable delivery per stream. A lost packet on stream 1 doesn't block streams 2 and 3. QUIC also includes TLS 1.3 baked in (reducing handshake round trips) and supports connection migration — if your phone switches from WiFi to cellular, the QUIC connection survives without a new handshake.
| Feature | HTTP/1.1 | HTTP/2 | HTTP/3 (QUIC) |
|---|---|---|---|
| Multiplexing | No | Yes (single TCP) | Yes (per-stream) |
| Head-of-line blocking | HTTP + TCP | TCP only | None |
| Header compression | None | HPACK | QPACK |
| Connection setup | 2-3 RTT | 1-2 RTT | 0-1 RTT (0-RTT resumption) |
| Transport | TCP | TCP | UDP (QUIC) |
For your backend: most modern load balancers and API gateways support HTTP/2 to clients automatically. HTTP/3 adoption is growing fast — Cloudflare, Fastly, and major cloud CDNs support it. Your app server typically speaks HTTP/1.1 or HTTP/2 to the proxy; the protocol to clients is handled at the edge.
DNS resolution is one of those things engineers know exists but rarely think about deeply — until it causes a problem.
The resolution chain for api.example.com:
- Check local OS cache (and browser cache)
- Ask the recursive resolver (usually your ISP's or 8.8.8.8 or 1.1.1.1)
- Recursive resolver checks its cache; if miss, asks the Root DNS servers
- Root servers say "for
.com, talk to these nameservers" .comTLD nameservers say "forexample.com, talk to these nameservers"- Authoritative nameserver for
example.comreturns the record forapi.example.com - Result is cached for the duration of the TTL
The TTL (Time to Live) is the most important operational concept in DNS. If your A record has a TTL of 3600 (one hour), a change to that record takes up to an hour to propagate to all users. Before a migration or traffic cutover, lower your TTLs (to 60-300 seconds) a day in advance. After the migration, raise them back.
Key record types you need to know:
- A / AAAA: Map hostname to IPv4/IPv6 address. The most common record.
- CNAME: Alias.
api.example.com→api.us-east.internal.example.com. Can't be used at zone apex (the root domain) — that's what ALIAS/ANAME records are for. - SRV: Service locator record. Specifies hostname and port. Used in service discovery, Kubernetes, and some protocols.
- TXT: Text records. Used for domain verification (SPF, DKIM, domain ownership proofs).
- MX: Mail exchange. Which servers handle email for the domain.
A CDN (Content Delivery Network) is a globally distributed network of servers that caches your content close to your users. Instead of a user in Singapore hitting your us-east-1 origin server (150ms round trip), they hit a CDN edge in Singapore (5ms round trip), which has your content cached.
But modern CDNs do much more than caching:
- DDoS protection: Edge nodes absorb volumetric attacks before they reach your origin
- TLS termination: The edge handles TLS handshakes; your origin serves plain HTTP internally
- Edge compute: Run code at the edge (Cloudflare Workers, Fastly Compute@Edge) for auth, A/B testing, personalization — without the round trip to origin
- Image optimization: Automatically convert, resize, and serve WebP/AVIF
- Origin shield: A second tier of caching between edge nodes and origin, dramatically reducing origin requests
For most web apps, the CDN configuration that matters most:
- Cache-Control headers: Your app controls what the CDN caches and for how long.
public, max-age=31536000, immutablefor hashed static assets.no-cachefor HTML that needs to always be fresh. - Cache invalidation: Purge specific paths when content changes. For immutable assets (with content hashes in filenames), you don't need to invalidate — the URL changes when the content changes.
When you have 50 microservices, how does Service A know where to find Service B? Hardcoded IPs are obviously wrong. Even hostnames are fragile if the set of instances changes dynamically.
Three patterns:
Client-side discovery: The service queries a service registry (like Consul or Eureka), gets a list of healthy instances, and load-balances itself. More control, more complexity in each service.
Server-side discovery: The service sends its request to a load balancer or service proxy, which queries the registry and routes to a healthy instance. The service doesn't need to know about discovery at all. This is how Kubernetes Services work — you call http://user-service:8080 and kube-proxy routes to a healthy pod.
DNS-based: Services register themselves as DNS records. Callers use standard DNS resolution. Universal and simple. The downside: DNS caching can serve stale records for up to the TTL after an instance becomes unhealthy.
L4 (Transport layer): Routes based on IP and TCP/UDP port. Doesn't look inside the packets. Very fast, minimal overhead. Can't route based on URL, headers, or application-layer content. Used for non-HTTP protocols, or when you need maximum throughput with minimal overhead.
L7 (Application layer): Understands HTTP. Routes based on path (/api → API servers, / → frontend servers), headers, cookies, or request body content. Enables path-based routing, host-based routing, sticky sessions, content-based routing. Slightly higher overhead, much more flexible. This is what nginx, HAProxy, Envoy, and AWS ALB implement.
For most web applications, L7 load balancing at the entry point (your Ingress controller or ALB) is the right choice. L4 load balancing makes sense for TCP-level work (database proxies, raw TCP services, NLBs for network performance).
Here's a thought experiment: what would it mean to 10x the productivity of every engineer on your team simultaneously? You could hire 10x more engineers. Or you could build better tools.
Platform Engineering is the discipline of building internal platforms that make every developer more productive, more autonomous, and more able to focus on business problems rather than infrastructure problems. Done well, it's the highest-leverage engineering investment a company can make.
The alternative — every team managing their own infrastructure, their own CI/CD, their own secret management, their own deployment processes — is the tragedy of the commons at scale. Everyone doing it themselves, everyone solving the same problems slightly differently, everyone accumulating slightly different technical debt.
An Internal Developer Platform (IDP) is the set of tools and workflows that developers use to go from code to running-in-production. It typically includes:
- Service catalog: A directory of all services, their owners, their dependencies, their SLOs, their runbooks
- CI/CD templates: Golden-path pipelines that handle build, test, scan, and deploy for common service types
- Environment provisioning: Self-service creation of isolated dev/test environments
- Observability: Pre-configured dashboards, alerting, and log aggregation for every service that onboards
- Secret management: Integrated access to Vault, AWS Secrets Manager, or equivalent — without each team implementing their own secret handling
- Ephemeral environments: Per-pull-request environments that spin up for review and tear down on merge
Backstage (by Spotify, now CNCF) has become the standard foundation for IDPs. It's a developer portal framework that you extend with plugins for your specific toolchain. Building an IDP on Backstage is significantly faster than building from scratch, with the tradeoff that Backstage is complex to operate and customize.
The central concept of platform engineering is the golden path: a well-supported, well-documented, opinionated way to accomplish a task. Not mandated. Not the only way. Just the paved road.
If you need to deploy a new microservice, the golden path says: here's the service template, here's the CI/CD pipeline, here's how you get into the service catalog, here's how observability gets set up automatically. You can follow the golden path and have a production-ready service in an afternoon.
You can also go off-path. Maybe your service has unusual requirements. Maybe you're doing something experimental. The platform team doesn't block you — but they're not responsible for supporting you either. The golden path is where the support, the documentation, and the organizational knowledge lives.
The design goal: 80% of services should fit comfortably on the golden path. If you're forcing everything into one template and 40% of services are fighting the template, the template is wrong. If nothing fits and every service is a special snowflake, you have no platform, you have chaos with documentation.
Platform quality isn't measured by how sophisticated the platform is. It's measured by how it feels to use it. The research on developer experience has converged on three dimensions that predict developer productivity:
Feedback loops: How quickly do you know if something worked? In a bad setup, you push code, wait 20 minutes for CI, wait another 10 for deployment, then check if your change was correct. In a good setup, unit tests run in seconds locally. CI takes under 5 minutes. You know almost immediately. Short feedback loops allow rapid iteration. Long feedback loops force context-switching while you wait, which kills flow.
Cognitive load: How much do you have to keep in your head to do your work? A developer who needs to understand Kubernetes networking, Terraform modules, five different IAM policies, and a custom deployment orchestration script just to deploy a feature has too much cognitive load. The platform's job is to hide irrelevant complexity. You shouldn't need to understand how Helm charts work to deploy your service — you should just tell the platform what you want.
Flow state: How many interruptions break your concentration? A development environment that crashes twice a day, a CI system that has flaky tests, a local setup that takes 4 hours to configure — these are flow killers. The platform's job is to keep developers in flow.
The most important thing to understand about building an internal developer platform: it's a product, and developers are your customers. If you build features that nobody uses, you've wasted the time. If developers have to file tickets to provision environments or get secrets, you've built a bureaucracy, not a platform.
Platform team KPIs:
- Adoption rates: What percentage of teams are using each platform capability?
- Time-to-first-deploy: How long does it take a new service to reach production for the first time?
- Mean time to recover: When something breaks, how fast can teams fix it using platform tooling?
- Developer satisfaction: Regular surveys. Quarterly NPS. Direct feedback channels. Developers should be vocal if the platform is slowing them down.
The platform team that asks for feedback, ships improvements fast, and treats developers as customers will earn trust and drive adoption. The platform team that mandates adoption of tools that don't work well will create a culture of workarounds and resentment.
Beyond kubectl — the knowledge needed to operate Kubernetes in production. This section goes deep. If you're running Kubernetes (or planning to), this is the stuff that separates "we have a cluster" from "our cluster is reliable and secure."
Kubernetes networking has one foundational rule that underpins everything: every pod gets its own IP address, and pods can communicate directly without NAT. This is called the flat network model, and it simplifies service discovery dramatically compared to systems where port mapping and NAT are everywhere.
CNI (Container Network Interface) plugins implement this flat network on whatever physical or cloud network you have:
- Calico: eBPF-based, high performance, feature-rich network policy enforcement
- Cilium: eBPF-based, emerging as the choice for sophisticated network policy and observability
- Flannel: Simple overlay network, easy to understand, limited features
- Weave: Encrypted overlay, simpler than Calico
Service networking is how pods find each other via stable names:
- ClusterIP: A virtual IP (within the cluster only) that routes to healthy pods. kube-proxy maintains iptables or IPVS rules to do this routing. When you call
http://user-service:8080, it resolves to a ClusterIP, which routes to an actual pod. - NodePort: Exposes the service on a high-numbered port on every node. External traffic can reach it at
<node-ip>:<node-port>. Useful for development, not great for production. - LoadBalancer: Provisions a cloud load balancer (AWS ALB/NLB, GCP LB) automatically. The right way to expose services to external traffic.
DNS inside the cluster is handled by CoreDNS. The pattern: <service-name>.<namespace>.svc.cluster.local. Within the same namespace, you can just use <service-name>. This is how microservices find each other — no hard-coded IPs, no external DNS required.
Ingress is the standard way to do HTTP routing into the cluster. You define routing rules (path, host), and an Ingress Controller (nginx, Traefik, Envoy/Contour, AWS ALB Ingress) implements them. One LoadBalancer service → Ingress Controller → many services, with routing by hostname and path.
NetworkPolicy is the firewall between pods. The default in Kubernetes: every pod can talk to every other pod. That's a terrible security posture for production. NetworkPolicy lets you restrict:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-api-to-db
spec:
podSelector:
matchLabels:
app: database
ingress:
- from:
- podSelector:
matchLabels:
app: api
ports:
- port: 5432This says: the database pods only accept traffic from API pods on port 5432. Everything else is denied. Apply network policies as default-deny-all, then explicitly allow what's needed. This is defense in depth — even if an attacker compromises one service, they can't freely pivot to others.
Role-Based Access Control is Kubernetes' authorization system. Getting RBAC right is the difference between a secure cluster and one where a compromised pod can do anything to anything.
The model is three parts:
-
ServiceAccount: Every pod runs as a ServiceAccount. If you don't specify one, it runs as the
defaultServiceAccount. ThedefaultServiceAccount has no permissions by default, but organizations often grant it too much. -
Role / ClusterRole: Defines a set of permissions — which API verbs (
get,list,watch,create,update,delete) on which resources (pods,deployments,secrets). A Role is namespace-scoped; a ClusterRole is cluster-wide. -
RoleBinding / ClusterRoleBinding: Assigns a Role to a subject (ServiceAccount, User, or Group). This is what actually grants the permissions.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: production
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: production
name: read-pods
subjects:
- kind: ServiceAccount
name: monitoring-agent
namespace: production
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.ioPrinciple of least privilege: Only grant what's actually needed. A web application shouldn't need any Kubernetes API access at all in most cases. A monitoring agent needs read access to pods. Nothing gets cluster-admin except the platform team, and even then, only for break-glass scenarios.
The common mistake that security auditors find everywhere: verbs: ["*"] on resources: ["*"] with apiGroups: ["*"]. This is cluster-admin under a different name. Any pod running as this ServiceAccount can create, delete, or modify any resource in the cluster — including creating new admin accounts. It's a complete cluster takeover in the event of a compromised pod.
If Kubernetes manifests are the "config files," Helm is the "package manager." Helm bundles related manifests into a "chart" — a parameterized, versioned package that can be installed, upgraded, and rolled back.
Why you need Helm: deploying a non-trivial application to Kubernetes typically involves 10-20 YAML files — Deployments, Services, ConfigMaps, Secrets, Ingress, HPA, PDB, ServiceAccount, RBAC, and more. Managing these by hand across dev/staging/production, each with slightly different configuration, becomes untenable quickly.
Helm's chart structure:
my-app/
├── Chart.yaml # Metadata (name, version, dependencies)
├── values.yaml # Default configuration values
├── templates/
│ ├── deployment.yaml # Go templates referencing {{ .Values.* }}
│ ├── service.yaml
│ ├── ingress.yaml
│ ├── configmap.yaml
│ ├── _helpers.tpl # Reusable template functions
│ └── NOTES.txt # Post-install message
└── charts/ # Subcharts (dependencies)
values.yaml is where you define defaults. Overrides come in at deploy time:
helm upgrade --install my-app ./my-app \
-f production-values.yaml \
--set image.tag=v1.2.3 \
--set replicaCount=5Key commands in the Helm workflow:
helm template— render manifests locally without applying them. Invaluable for debugging template issues.helm diff— (plugin) preview what will change before upgrading. Liketerraform planbut for Helm.helm rollback— roll back to any previous release. The history is preserved in Secrets in the namespace.
The Helm vs Kustomize question comes up often:
| Aspect | Helm | Kustomize |
|---|---|---|
| Approach | Templating (Go templates) | Patching (overlays on base manifests) |
| Complexity | Higher (template logic) | Lower (plain YAML) |
| Ecosystem | Massive chart library (Artifact Hub) | Built into kubectl |
| Best for | Distributing apps, complex configs | Internal apps, simple overrides |
The pragmatic recommendation: Helm for third-party apps (nginx-ingress, cert-manager, Prometheus, Grafana), Kustomize for your own services. Third-party charts are battle-tested, parameterizable, and maintained. Your own services are usually simpler and benefit from Kustomize's plain-YAML approach.
CRDs (Custom Resource Definitions) let you add your own resource types to Kubernetes. Once you define a CRD, you can create instances of it with kubectl, list them, watch them, apply RBAC to them — just like built-in resources.
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: databases.example.com
spec:
group: example.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
engine: { type: string, enum: [postgres, mysql] }
version: { type: string }
storage: { type: string }
scope: Namespaced
names:
plural: databases
singular: database
kind: DatabaseAfter applying this, kubectl get databases works. You can create a Database object. But nothing happens yet — you need a controller to watch for Database objects and act on them.
That controller is the Operator. The Operator Maturity Model describes the levels of operational knowledge an operator encodes:
- Install: Can deploy the application from a CRD
- Upgrade: Can upgrade the application, including rolling upgrades
- Lifecycle management: Handles backup/restore, failure recovery
- Deep insights: Exposes metrics, alerts, SLO tracking for the managed app
- Autopilot: Auto-scales, auto-tunes, auto-heals based on workload
Production operators worth knowing: cert-manager (TLS certificates, automatic renewal), Prometheus Operator (manages Prometheus configurations as CRDs), Zalando Postgres Operator (production Postgres with HA and backups), Strimzi (Kafka). These encode years of operational experience.
Resources (CPU and memory) are how Kubernetes decides where to schedule pods and how to handle resource contention. Getting this right is one of the most impactful things you can tune in a running cluster.
Requests are the minimum guaranteed resources. The scheduler uses requests to decide which node to place a pod on. If a node has 2 CPU available and a pod requests 4 CPU, it won't be scheduled there.
Limits are the maximum allowed resources. A pod that exceeds its memory limit is OOMKilled (killed by the out-of-memory killer). A pod that exceeds its CPU limit is throttled.
resources:
requests:
cpu: "250m" # 0.25 CPU cores (250 millicores)
memory: "512Mi" # 512 MiB
limits:
cpu: "1000m" # 1 CPU core max
memory: "1Gi" # 1 GiB maxThe counterintuitive recommendation about CPU limits: consider not setting them. CPU throttling is insidious. A pod at its CPU limit has its execution periodically paused, even if there's idle CPU on the node. This shows up as increased latency, not as obvious failures. Setting CPU requests without limits lets pods burst when CPU is available and only contend when the node is actually busy. Memory limits matter much more — OOMKilled is at least a clear signal.
HPA (Horizontal Pod Autoscaler): Scale the number of pods based on metrics. CPU utilization is the basic case; custom metrics from Prometheus enable scaling on request rate, queue depth, or any business metric.
VPA (Vertical Pod Autoscaler): Recommends or automatically adjusts requests and limits based on actual usage. Run in recommendation mode first — it will tell you if your requests are over- or under-provisioned without actually changing anything.
PDB (Pod Disruption Budget): Specifies the minimum number of pods that must remain available during voluntary disruptions (node upgrades, cluster maintenance). Without a PDB, Kubernetes might drain two nodes simultaneously and take your 3-replica service to 1 replica.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-pdb
spec:
minAvailable: 2 # Always keep at least 2 pods running
selector:
matchLabels:
app: apiLimitRange and ResourceQuota are namespace-level guardrails. LimitRange sets default requests and limits for pods that don't specify them. ResourceQuota caps the total resources a namespace can consume. Both are important in multi-tenant clusters where one team's memory-hungry deployment shouldn't starve everyone else.
The principle: even if an attacker gets code execution inside a container, the container's privileges should be minimal enough that they can't do significant damage.
The SecurityContext defines what a container is allowed to do:
securityContext:
runAsNonRoot: true
runAsUser: 1000
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
allowPrivilegeEscalation: falseBreaking this down:
runAsNonRoot: Don't run as UID 0. A container running as root that escapes container isolation has root on the node.readOnlyRootFilesystem: The container can't write to its own filesystem. Malware can't persist, can't modify its own binaries.capabilities.drop: ["ALL"]: Linux capabilities grant specific root-like powers (binding low ports, changing file ownership, managing network interfaces). Drop all of them unless you specifically need one.allowPrivilegeEscalation: false: Preventssetuidbinary exploits from escalating privileges.
Pod Security Standards are Kubernetes' built-in policy framework (replaced the deprecated PodSecurityPolicies):
- Privileged: No restrictions. For system components only.
- Baseline: Prevents the most obvious escapes. Minimum for any multi-tenant cluster.
- Restricted: Enforces all security best practices. Start here for new workloads.
Secrets management — don't put secrets in Kubernetes Secrets if you can avoid it. Kubernetes Secrets are base64-encoded (not encrypted) by default in etcd. The better approaches:
- external-secrets-operator: Syncs secrets from Vault, AWS Secrets Manager, GCP Secret Manager into Kubernetes Secrets automatically
- sealed-secrets: Encrypts secrets using a cluster-specific key; the encrypted form can be safely committed to git
Production problems in Kubernetes have a known playbook. Most issues fall into a handful of categories, and knowing the diagnostic commands gets you to the root cause fast.
# Pod not starting? Start with describe — events at the bottom tell you why
kubectl describe pod my-pod-xyz
# Common reasons decoded:
# - ImagePullBackOff → wrong image name/tag, or missing imagePullSecret
# - CrashLoopBackOff → app crashes on startup, logs will show the error
# - Pending → insufficient resources, or node affinity/taint preventing scheduling
# - OOMKilled → memory limit is too low for what the app actually needs
# Container logs — the first thing to check for running apps
kubectl logs my-pod-xyz # Current logs
kubectl logs my-pod-xyz --previous # Logs from the last crashed container
kubectl logs -l app=my-app --all-containers # All pods matching a label
# Shell access — when logs aren't enough
kubectl exec -it my-pod-xyz -- /bin/sh # Shell into the container
kubectl debug my-pod-xyz --image=busybox --target=app # Ephemeral debug container
# Network debugging — the netshoot image is your Swiss Army knife
kubectl run debug --image=nicolaka/netshoot -it --rm -- bash
# Inside: curl, dig, nslookup, tcpdump, ping, iperf, and more
# Resource usage — is the node or pod starved?
kubectl top pods # CPU/memory per pod
kubectl top nodes # CPU/memory per nodeEphemeral debug containers (the kubectl debug approach) deserve special mention. Modern, minimal containers often don't have shells or debugging tools. Rather than building debug tools into your production images (bloating them, adding attack surface), ephemeral containers let you attach a debug image to a running pod temporarily. The debug container shares the process namespace with the target container, so you can inspect running processes, network connections, and file descriptors.
This is the checklist you run through before calling a service production-ready. Not optional, not aspirational — these are the things that will bite you if you skip them:
- Resource requests and limits set on all pods
- Liveness and readiness probes configured
- Pod Disruption Budgets for critical services
- NetworkPolicies restricting pod-to-pod traffic (default-deny is the goal)
- RBAC with least privilege — no cluster-admin for application service accounts
- Secrets in external secret manager (not plaintext in Kubernetes Secrets)
- HPA configured for variable-traffic services
- Node anti-affinity for replicas — spread across nodes and availability zones
- Monitoring: Prometheus + Grafana dashboards per service, not just cluster-level
- Logging: centralized (Loki/ELK/OpenSearch) with structured JSON logs
- SecurityContext: non-root, read-only rootfs, dropped capabilities
- Image scanning in CI (trivy/grype) with build failure on high/critical CVEs
- Rollback strategy tested —
helm rollbackorkubectl rollout undoverified to work
Understanding where your organization sits on the DevOps maturity spectrum is essential for knowing which investments will have the highest leverage. The model below is a pragmatic framework based on observable practices and capabilities, not certifications or self-assessments.
Characteristics:
- Deployments are manual: SSH into servers, pull code, restart processes
- No CI pipeline; builds happen on developer laptops
- No automated tests running in any automated pipeline
- Infrastructure is documented (if at all) in wikis or tribal knowledge
- One or two "server whisperers" who understand the production environment
- Incidents are resolved by those individuals, often with no post-mortem
What's painful: Deployments are high-stress events. Bugs are discovered in production, not before. When a server whisperer leaves or is unavailable, the team is stuck. Rollbacks mean SSHing back in and hoping you can reverse the manual changes you made. Staging and production have drifted — what works in staging frequently breaks in production.
What to fix first: A single automated deployment mechanism. Even a shell script that deploys from a git tag is dramatically better than manual SSH. This is your week-one improvement.
Characteristics:
- Source control for all application code (git)
- Automated build from a commit (at minimum, a build script)
- Some automated tests (unit tests, even if sparse)
- Basic CI: tests run automatically on push to main
- Deployments are scripted (even if not automated)
- One or two production environments (staging + prod)
What's painful: Deployments still require a human to trigger the script. Test coverage is low — most bugs are still discovered in production. Infrastructure changes still require the "I know how to do this" person. On-call is reactive: you find out about incidents when users complain.
What to fix next: Invest in test coverage for your most critical business flows. Make deployments fully automated (push to main → automatic deploy to staging, one-click deploy to production). Add basic monitoring (uptime check, error rate alerting).
Characteristics:
- CI pipeline: every commit triggers build + tests
- Automated deployment to staging on every merge
- Production deployment automated or semi-automated (one-click)
- Infrastructure is described in code (at least partially — VMs, basic networking)
- Tests cover the critical user paths (unit + integration, maybe some E2E)
- Monitoring and alerting for key metrics (error rate, latency, availability)
- Post-mortems happen after significant incidents
What's good here: Engineers trust CI — a green build means something. Deployments happen more frequently (weekly instead of monthly). New team members can get a development environment running without help from a senior engineer. Incidents are detected by monitoring, not users.
What's painful: Infrastructure is "code" in theory but still has significant manual steps. Secrets management is ad-hoc (env files checked into repos, secrets in CI environment variables without rotation). There's no self-service for developers to provision environments. Cross-team coordination is required for database changes.
What to fix next: Full IaC (Terraform or CDK for all infrastructure). Secrets management (Vault, AWS Secrets Manager, external-secrets-operator). Automated database migrations as part of the deployment pipeline. A PR-per-environment ephemeral testing setup.
Characteristics:
- Trunk-based development with feature flags
- Continuous deployment to staging; one-click (or automated) to production
- All infrastructure declared in IaC, in code review, in git history
- Secrets management fully automated (rotation, access control, audit logs)
- Containerized workloads, probably on Kubernetes or a managed container platform
- Developer self-service: new services, new environments, new secrets — no tickets required
- SLOs defined and monitored; incident management process is mature
- Post-mortems are blameless and produce tracked action items
- Observability: structured logging, distributed tracing, metrics dashboards per service
What's good here: Deployments are routine, not events. The team deploys multiple times per day. Incidents are detected by SLO alerts before customers notice. New engineers are productive within days, not months. Infrastructure changes are code-reviewed like application changes.
What's painful: Platform complexity is high — someone needs to own and maintain the Kubernetes cluster, the CI platform, the secrets management, the observability stack. The cognitive load for engineers who need to understand infrastructure is non-trivial. Security posture may have gaps in supply chain (container image scanning, dependency vulnerabilities).
Characteristics:
- Platform Engineering team: internal developer platform abstracts infrastructure from product engineers
- Golden paths for every common service type (API service, background worker, scheduled job)
- Progressive delivery as standard: canary deployments with automated SLO-based promotion/rollback
- DORA metrics tracked and improving: deployment frequency (multiple per day), lead time for changes (hours), MTTR (< 1 hour), change failure rate (< 5%)
- Security as code: policy-as-code (OPA/Gatekeeper), image signing, software bill of materials (SBOM)
- Cost optimization: FinOps practice, right-sizing automated, unused resources detected and cleaned up
- Chaos engineering: controlled failure injection to validate resilience assumptions
What this looks like: A developer writes code and merges to main. Within minutes, it's deployed to staging automatically. Within 15 minutes, a canary in production is serving 5% of traffic. Automated analysis of error rates and latency compares the canary to the baseline. If healthy, it automatically promotes to 100% over the next hour. If unhealthy, it automatically rolls back with an alert to the developer. No human made a deployment decision.
New services are created by filling in a web form in the internal developer portal (Backstage or equivalent). CI/CD, monitoring dashboards, service catalog registration, PagerDuty integration — all happen automatically as part of service creation.
DORA Metrics as your north star:
The DevOps Research and Assessment (DORA) research identified four metrics that most reliably predict organizational performance:
| Metric | Elite | High | Medium | Low |
|---|---|---|---|---|
| Deployment Frequency | Multiple/day | Daily-weekly | Weekly-monthly | < monthly |
| Lead Time for Changes | < 1 hour | 1 day-1 week | 1 week-1 month | > 1 month |
| MTTR | < 1 hour | < 1 day | 1 day-1 week | > 1 week |
| Change Failure Rate | < 5% | < 10% | 10-15% | > 15% |
These metrics are correlated with software delivery performance and organizational performance — teams that score as "Elite" across all four metrics ship more, break less, and recover faster. They're useful as leading indicators for your DevOps investments: if deployment frequency is low, the bottleneck is in your deployment pipeline. If MTTR is high, the bottleneck is in observability and incident response. Fix the bottleneck, watch the metric improve.
The journey is the point:
Most organizations exist between Levels 1 and 3. The goal is not to reach Level 4 — it's to continuously improve. Each level of maturity enables faster, safer delivery of software. The investments in automation, tooling, and process are not overhead — they're the infrastructure that lets product teams move fast without breaking things.
The single highest-leverage investment at any level: reduce the feedback loop. Faster CI means bugs are caught sooner. Faster deployments mean smaller, safer releases. Faster on-call response means less user impact. Follow the feedback loops and shorten them.
Experienced infrastructure engineers have a set of questions they ask before committing to any architectural decision. These aren't philosophical — they're practical checklists that prevent expensive mistakes.
Before choosing a container orchestration platform:
- How many containers will you run in 6 months? (< 20: probably don't need Kubernetes. 20-100: evaluate ECS vs K8s. > 100: Kubernetes probably pays for itself.)
- Do you have a dedicated platform engineer, or will product engineers maintain this? (Be honest about operational capacity.)
- What's your cloud spend today vs projected? (Kubernetes cluster overhead is ~$300-600/month minimum. Is that proportional?)
- Do you need multi-region? (Kubernetes multi-cluster is significantly harder than single-cluster.)
Before implementing a CI/CD strategy:
- What's your current deployment frequency? (If monthly, don't invest in the tooling for daily deploys yet — invest in reducing the friction first.)
- Where do deployments fail most often? (Answer this with data from your last 20 deployments, not gut feel.)
- What does a rollback currently look like? How long does it take? (That number should inform your deployment strategy choice.)
- Do you have feature flags? (Without feature flags, trunk-based development is genuinely risky.)
Before adopting IaC:
- Is your current infrastructure stable and well-understood? (IaC-ing chaos just codifies the chaos.)
- Who will maintain the IaC modules after the person who wrote them leaves? (Unusually clever Terraform is a future maintenance burden.)
- How will you handle state for resources created before IaC? (
terraform importexists but is painful at scale.) - What's your plan for secrets in the state file? (This question needs an answer before day one, not month six.)
Before adding a service mesh:
- Can you name a specific problem you're trying to solve that requires a service mesh? (If the answer is vague, the service mesh is premature.)
- Does your team have someone who has operated Istio or Linkerd before? (If not, the learning curve is a real cost.)
- Have you profiled the latency impact of the sidecar proxy on your most latency-sensitive flows? (Typically 1-3ms round-trip — often acceptable, occasionally not.)
These questions don't have universal answers. They're prompts to think carefully before defaulting to whatever is considered "best practice" this year. Best practices are useful as starting points; your specific context determines whether they apply.
Here's what the full picture looks like for a production-grade deployment at a mid-sized company:
Development happens in containers locally (Docker Compose or kind). Environment variables come from .env.local (not committed). Feature flags are on in development, off in staging by default.
Pull requests trigger a CI pipeline: build, unit tests, integration tests, container build, image scan, Helm chart lint. A preview environment (ephemeral, per-PR) is provisioned via Terraform and gets the new version deployed by Argo CD watching the PR's branch.
Merging to main triggers the staging deployment automatically — Argo CD pulls the updated image tag, applies the Helm release, and waits for the rollout. If the rollout fails (readiness probes don't pass), Argo CD marks the sync as failed and alerts.
Production promotion is a merge or tag in the gitops repo — sometimes manual (for careful services), sometimes automated after staging verification. A canary deployment sends 5% of production traffic to the new version. After 15 minutes of watching error rates and latency (Prometheus metrics, Datadog, whatever you use), the canary is either promoted to 100% or rolled back.
The entire infrastructure is declared in Terraform. Modules are versioned. Changes go through code review. terraform plan previews are posted to pull requests via Atlantis. Nothing touches the cloud console except in break-glass incidents, and those incidents are post-hoc codified in IaC.
That stack didn't exist five years ago in this form. It exists now, it's achievable for any team willing to invest in it, and the engineering velocity it enables is genuinely transformative. The 2 AM SSH sessions? They still happen — but they're rarer, shorter, and better-understood. That's what this chapter is about.
Next: Chapter 8 — Data Engineering & Databases. Because all that infrastructure needs to store and serve data reliably.
Want to put this into practice? The TicketPulse course has hands-on modules that build on these concepts:
- L1-M14: First Deployment — Ship TicketPulse to a real cloud environment for the first time: Docker, CI pipeline, and zero-downtime deploys
- L2-M43: Kubernetes Fundamentals — Deploy TicketPulse on Kubernetes with Deployments, Services, ConfigMaps, and HPA autoscaling
- L2-M44: Terraform and IaC — Declare TicketPulse's entire cloud infrastructure in Terraform and practice the plan/apply/review workflow
- L3-M83: Advanced Kubernetes — Tackle multi-cluster deployments, GitOps with Argo CD, and progressive delivery for TicketPulse's production environment
- Dockerize one service in your project if it isn't already — write a
Dockerfile, build it locally, run it withdocker run, and verify the service starts correctly. Bonus: add a.dockerignoreand make sure the image is as small as possible. - Write a health check endpoint — add a
/healthzor/healthroute to one of your services that returns HTTP 200 with a JSON body confirming the service is healthy and its dependencies (database, cache) are reachable. Wire it to your load balancer or readiness probe. - Review your CI pipeline and time each step — look at your last successful CI run and record how long each stage takes. Identify the single slowest step and research one concrete optimization (caching, parallelism, test splitting) that would reduce it.