CSIRO's Serverless Genomic Architecture on AWS: 18-Second Ingestion, 1.5-Second Queries, $0.33 per 1,000 Requests
Cloud Platform EngineeringGenomic data has a set of architectural constraints that make it a useful design reference for any workload with similar requirements: large datasets (billions of base positions per genome), sensitive access (regulatory requirements on response granularity), distributed ownership (institutions need to share without centralizing custody), and highly variable query patterns — long periods of near-zero traffic with occasional bursts.
CSIRO’s Serverless Beacon (sBeacon) is a production implementation of the Beacon protocol standard on AWS that addresses all four constraints simultaneously. The architecture and cost figures make it one of the more instructive serverless case studies published this year.
The Architecture
Four managed services carry the workload:
Amazon S3 — stores raw genomic data (VCF files) and metadata. Data ownership stays with the originating institution; each organization maintains its own S3 bucket and grants federated access rather than copying data to a central store.
AWS Lambda — handles indexing and querying logic. Ephemeral by design: no persistent state, scales to zero, scales up on demand. The genomic access pattern (sparse queries, variable burst) is a near-perfect fit for Lambda’s billing and scaling model.
Amazon DynamoDB — manages ontology term lookups. The Beacon protocol uses controlled vocabularies (disease ontologies, phenotype terms) to standardize what queries mean across institutions. DynamoDB handles the lookup tables that translate incoming query terms to standardized identifiers.
Amazon Athena — runs structural queries against metadata tables. The queries that identify which samples carry a given variant in a genomic region run against Athena’s serverless SQL interface, billing per query rather than per instance-hour.
The Performance Numbers
Tested against chromosome 1 from the 1000 Genomes Project (2,504 samples — the standard benchmarking dataset):
| Metric | Value |
|---|---|
| Ingestion time (chromosome 1) | 18 seconds |
| Query response time (10,000 bp region) | ~1.5 seconds |
| Data onboarding cost | Less than $0.01 |
| Monthly idle cost (1.1 GB storage) | $0.03 |
| Cost per 1,000 queries (Lambda + Athena + DynamoDB) | $0.33 |
Stated scale ceiling: “hundreds of millions of individuals and billions of genomic locations” — designed for eventual mega-biobank datasets rather than single-study scales.
Zero-Trust Access Control
Three elements compose the security model:
Amazon Cognito JWT tokens — authentication at the API boundary. Every request carries a short-lived token; no persistent sessions.
Role-based response granularity — the Beacon protocol supports three levels of response detail: boolean (yes/no a variant exists), count (how many samples carry it), or full record (which specific samples). The RBAC layer enforces that a given caller receives only the granularity their role permits.
Ephemeral Lambda compute — query execution has no persistent state. Lambda functions start, execute, and terminate. There is no running instance to compromise between queries.
The decentralized ownership model is the final security layer: VCF files stay at the originating institution’s S3 bucket. The network queries across institutional stores through the Beacon protocol without any single node holding the complete dataset.
The Pattern This Demonstrates
The CSIRO architecture generalizes beyond genomics. The combination of constraints it solves — large datasets with sensitive access control, distributed ownership, variable query patterns, cost-proportional-to-use billing — appears in any multi-tenant research or healthcare platform where institutions need to share data without consolidating custody.
Platform teams building research data-sharing infrastructure or multi-institution analytics pipelines on AWS will recognize the architectural decisions: Lambda for compute isolation and burst absorption, Athena for serverless SQL against object storage, DynamoDB for lookup tables, Cognito for federated identity. The implementation details are genomics-specific; the pattern is reusable.
The So What
The cost numbers are the headline: $0.33 per 1,000 queries, $0.03/month at idle. These are the figures that make serverless compelling for research workloads with uncertain query volumes — the architecture costs essentially nothing when no one is querying and scales to handle burst access to a dataset spanning hundreds of millions of genomes.
For platform engineers designing data-sharing infrastructure with multi-institution constraints: the decentralized ownership model (data stays at source, queries federate across nodes) solves the governance problem while reducing data egress costs. Centralizing everything in one store is the instinctive architectural choice; for this class of workload, federated serverless is both cheaper and more politically viable across institutional boundaries.
Source: AWS Architecture Blog
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →