Operations Interview Questions

ThrottledRequests, SystemErrors, UserErrors, Consumed vs provisioned capacity, and replication or PITR-related failures if you use those features. Latency p99 on the API is the user view. Alarming only on CPU-like host metrics does not apply; DynamoDB is a service, not your instance.

Continuous backups so you can restore to any second in the retention window (typically 35 days) into a new table. It protects against bad deploys and accidental deletes. PITR is not a substitute for testing that restore into a new table actually works.

On-demand snapshots mark a known-good point before a migration or schema-risking change. PITR covers “unknown moment of corruption.” Restore still creates a new table; practice renaming and cutover. Backups that have never been restored are unproven.

It ranks hottest keys and most throttled items so you can see skew that table-level graphs hide. Turn it on when throttling is mysterious. It is a diagnosis tool, not a fix; you still change keys or cache after you see USER#42 at the top.

Most UpdateTable changes happen while the table stays available; status may be UPDATING. Some combinations are limited (you cannot change certain properties at once). Plan for throttling or brief elevation in errors during capacity mode switches, but it is not a maintenance-window outage like a SQL dump.

Export (PITR-based) dumps table data to S3 for analytics or DR copies without a Scan from the app. Import creates a new table from S3. Use them for warehouse feeds and rebuilds; they are not a millisecond failover. IAM on the bucket is part of the runbook.

ACTIVE is ready for traffic. UPDATING means a change (GSI create, capacity, class) is in progress; most reads/writes still succeed. Creating a GSI while UPDATING can take a long time on large tables—do not stack conflicting updates or CI will race you.

Restore to a timestamp, enable the same GSIs/streams/auto scaling, dual-write or freeze writes if you must, replay the small gap, then switch endpoints. DNS/config flags beat renaming in place. Skipping GSI rebuild on the restored table is a common cutover bug.

Throttles are throughput/hot-key. UserErrors are client validation, missing tables, or condition failures. SystemErrors are AWS-side and should be rare; retry them. Treating every 400 as throttle hides bad keys in the request.

Add first, wait until ACTIVE, shift traffic, then delete the old index. Backfill consumes capacity—raise headroom. Deleting a GSI that still serves queries is an instant incident. Monitor OnlineIndexPercentageProgress rather than guessing by wall clock.

Infrequently accessed tables where storage dominates cost and you accept higher per-request prices. It is not a performance tier for hot paths. Moving a high-QPS session table to IA to “save money” usually increases the bill and latency.

Environment, owner, service, PII classification, and cost-center tags. Cost Explorer without tags cannot tell staging Scan jobs from prod. Tags also drive backup plans and encryption exceptions in larger orgs.

Revert the UpdateTable, watch ThrottledRequests, and temporarily switch to on-demand if the revert is slow. Client backoff should already be in place. Capacity-as-code reviews belong in CI so the next Terraform apply cannot set WCUs to 1 on checkout.

Watch ReplicationLatency, conflict counts, and that IAM/encryption match per region. Failover is not instant magic—run a regional drill. Last-writer-wins means you need application rules for concurrent updates on the same item across regions.

New table with the target keys, dual-write, backfill, compare counts/checksums, shift reads with a flag, then stop writes to green’s predecessor. Keep PITR on both until soak time ends. In-place key changes are not blue/green; they are a copy.

Check AWS health, account quotas, on-demand ramp, a new GSI backfill, a Scan job, and SDK retry storms. If all keys throttle, it is often quota or a table-level limit, not one celebrity PK. Mitigate with cache, shed load, and raise quotas; then find the deploy that changed access patterns.

User-facing availability and latency SLOs, plus an error budget that includes throttles and dependency faults. Track iterator age if streams are on the critical path. DynamoDB’s own 99.99% is not your SLO if your Lambda and DAX sit in front.

Delete unused GSIs, right-size provisioned vs on-demand, TTL junk data, Standard-IA for cold tables, and tighter backups. Cost spikes often come from Scan, ALL projections, or PITR on huge tables. Show unit cost per million requests so “just add an index” has a price.

Scheduled restore to a scratch table in a non-prod account, assert item counts and a sample GetItem, then delete. Alert if the drill fails. Documentation of RPO/RTO without a last successful restore date is theater.

Dashboards for iterator age, DLQ depth, and consumer error rate next to table throttles. A table that is “healthy” while search is 40 minutes behind is not healthy for users. Runbooks should include pausing a bad consumer without dropping the 24-hour window.