Enabling on-demand for DynamoDB (DDB) can prevent throttling when there’s high load. Or at least that’s what I thought. There was a relatively large file being processed which resulted in doing bursty batch writes to a single partition key (PK). Throughput spiked, requests got throttled even though on-demand was turned on.
When on-demand is turned on, there is still a throughput limit that is enforced at the partition level for both the table and GSI. Each partition in the table can simultaneously serve up to 1,000 write request units and 3,000 read request units per second. Exceeding these limits on the partition level leads to hot partitioning resulting in database requests getting throttled. To reduce the risk of hot partitioning, here are some measures that we can account for to ensure that we do not exceed the maximum throughput threshold:
- Reduce concurrent operations to avoid a single burst when performing batch operations on a single PK to spread the load over a period of time
- Use exponential backoff to avoid immediate retries that could increase the number of requests made per second
- Better DDB design strategy to reduce load on the PK. For example, having a good sort key (SK) design can also leverage DDB’s “split for heat” mechanism
Designing for what works is relatively easy. But designing for scale requires you to understand the constraints of the database to design an optimal schema and architecture such that it doesn’t break even under high load. In this case, turning on on-demand isn’t always the silver bullet for your features to work at scale. A good database design and architecture is.