DynamoDB Bulk Executor: Lessons From Building Glue Bulk Jobs

CloudWatch Live Tail displays at most 500 events per second, and beyond that it streams only a sample. The team behind AWS’s open-source Bulk Executor for DynamoDB learned that when large result sets came back incomplete, one of several gotchas in their “lessons we learned the hard way” write-up. For platform engineers building internal tools that run bulk jobs against DynamoDB, the post is a checklist of failures that only show up at scale.

How It Works

Bulk Executor runs actions against many or all items in a table, such as mass deletes or comparing two tables. The team wanted three things: commands that feel like a local script with live output and ^C to stop, scale to hundreds of parallel workers without managing infrastructure, and easy-to-add commands. They chose AWS Glue over SQS with Lambda, Fargate, EMR, and Ray because it offered serverless parallelism and a built-in DynamoDB connector.

The lessons from the write-up:

• Quote the cost first. Reads are priced from table metadata, which updates about every six hours, so a freshly loaded table can look smaller than it is. For provisioned tables they take two-thirds of the on-demand price, since quoting a little high is safer. Deletes are priced in two phases: read cost up front, write cost once matches are known.
• Make ^C stop both sides. Interrupting the Python client left the Glue job running until timeout. A signal handler now sets a flag that a watcher loop uses to stop the Glue job before exiting.
• Require point-in-time recovery. Any mutating command needs PITR enabled on the table, so a bad --where clause can be rolled back.
• Don’t trust a log stream to be complete. Because of the Live Tail cap, large output goes to S3 with only the first 10 results shown. Events can also arrive out of order or after the job finishes, so the client stops listening only after a few seconds of silence.
• Collapse errors. A Spark accumulator collects task exceptions so a bad clause prints one message, not hundreds of identical stack traces.
• Rate-limit direct calls. The connector can’t delete items, so some commands call DynamoDB directly, with no built-in throttling. A distributed rate limiter coordinates all workers.

What This Means for You

Going direct often paid off. A scancount command using segmented scans with Select: COUNT was consistently faster than counting through the connector, because only the count leaves the database. The team also found that for large jobs, Glue compute costs tend to be a small fraction of DynamoDB costs.

AI helped with maintenance. The team used AI-generated stress tests to compare connectors and finish the Glue 4 to Glue 5 upgrade in one working session, then moved to Glue 6 for its 30% lower DPU-hour cost, which Stack Archive covered in the Glue 6.0 price cut. They also built an AI linter of markdown rules that checks, for example, that IAM permissions described in the README match what the code creates and verifies.

Source: AWS Architecture Blog

Content created with AI assistance and reviewed for accuracy.