Common Gotchas

Pipelines rarely fail loudly. They rot — through small tolerated dysfunctions that compound until the automation exists but nobody trusts it. These are the five rots to catch early, because each is cheap to fix young and brutal to fix old.

1. Flaky tests

A test that fails one run in ten — timing assumptions, test-order dependence, real network calls — trains the whole team on one lesson: red might mean nothing. Then a real failure sails through on the second re-run. Policy that works: a flaky test is quarantined (skipped, ticketed, visibly) the day it's identified, then fixed or deleted. "Just re-run it" is how pipelines die.

2. Secrets in the pipeline

Deploy stages need credentials, and pipelines leak them in ways laptops don't: a debug echo that prints an env var into a log half the company can read, a secret committed to the repo "temporarily," a token with production write access used in every PR build. Rules: secrets live in the CI system's secret store (masked in logs — a feature the GitHub Actions track shows in detail), scoped to the narrowest stage that needs them, and logs are treated as public-ish. If a secret ever hits a log: rotate it, don't just delete the line.

3. The snowflake build agent

A build server hand-configured over the years — that one Node version, that manually installed library, those cached credentials — is a "works on this machine" problem wearing a CI costume. When it dies, nobody can rebuild it; until then, builds pass there and fail everywhere else. The fix is the Docker track's lesson applied to CI itself: build environments should be declared (containers, config files), disposable, and identical on every run.

4. The permanent "temporary" bypass

Production is on fire, so someone deploys manually, skipping the pipeline — defensible at 3 a.m., corrosive at the retro. Now production runs code the pipeline never saw; the next automated deploy may silently undo the hotfix; and everyone learned the pipeline is optional under pressure. Better: make the pipeline itself fast enough for emergencies (a skip-the-slow-tests emergency lane that still builds, records, and deploys the artifact properly), so the fast path and the safe path are the same path.

5. Deploys nobody can observe

The pipeline says "deployed ✅" — but is the app actually working? Teams that stop at "the copy succeeded" find out about broken releases from customers. Close the loop: the deploy stage should end with a smoke test (hit the health endpoint, check a core flow) and the dashboards from Lesson 4's canary discussion should be one click from the pipeline page. "Deployed" and "verified in production" are two different stages; treat them that way.

1. Last 20 runs: any failure that "fixed itself" on re-run?   → flaky
2. Search the logs for anything that shouldn't be public.     → secrets
3. Could you recreate the build machine from files in git?    → snowflake
4. When did someone last deploy around the pipeline? Why?     → bypass
5. After "deploy ✅", what *proves* the app works?            → observability

That's the track. The recap ties the discipline together — and points at the hands-on GitHub Actions track where these concepts become working YAML.