How to Build a Scalable User Search Layer on Amazon Cognito

Summary

• Amazon Cognito’s native ListUsers API only handles exact-match lookups on standard attributes — it breaks down for partial matching, fuzzy search, or complex multi-field filtering at scale
• This stack adds an event-driven synchronization layer: Cognito triggers and CloudTrail events feed DynamoDB Streams, which automatically index users into OpenSearch Serverless
• Result: sub-second full-text search across any combination of user attributes, at any user pool size, without manual sync jobs

Purpose

Extend Cognito user management with production-grade search. Use when you need partial matching, multi-field filters (email domain, registration date range, custom attributes), or fuzzy search across a large user pool. Not needed for simple exact-match lookups on email or phone.

Prerequisites

• AWS account with permissions for: Cognito, Lambda, DynamoDB, OpenSearch Serverless, CloudTrail, EventBridge, API Gateway
• Existing Cognito User Pool (or creating one)
• Basic familiarity with Lambda function deployment

Steps

1. Create the DynamoDB user profile table

Create a DynamoDB table (UserProfiles) with userId as the partition key. Enable DynamoDB Streams with NEW_AND_OLD_IMAGES stream type — this is the mechanism that triggers OpenSearch indexing automatically on every write.

2. Configure Cognito Lambda triggers

Add two Lambda triggers to your Cognito User Pool:

• Post Confirmation: fires after a user confirms their account; writes the new user profile to DynamoDB
• Pre Token Generation: fires at authentication; updates the DynamoDB profile with any attribute changes

These two triggers handle the authentication event stream. Admin operations (creating users via the Admin API) produce no authentication events, so they need separate handling.

3. Capture admin operations via CloudTrail + EventBridge

Enable CloudTrail for your account and create an EventBridge rule that matches AdminCreateUser, AdminUpdateUserAttributes, and AdminDeleteUser API calls. Route matching events to a Lambda function that syncs the corresponding DynamoDB record. This closes the gap that the Cognito triggers leave open.

4. Deploy the OpenSearch indexing Lambda

Create a Lambda function triggered by the DynamoDB Stream. For each stream record:

• On INSERT or MODIFY: upsert the user document into your OpenSearch Serverless index
• On REMOVE: delete the document from the index

The stream handles all writes automatically — no batch jobs, no manual sync.

5. Create the OpenSearch Serverless collection and index

Create an OpenSearch Serverless collection with the SEARCH type. Define an index mapping for your user attributes (email, phone, custom attributes, registration timestamp). Configure field types appropriately: text with keyword sub-field for email and phone to support both full-text and exact matching.

6. Build the search Lambda and API Gateway endpoint

Create a Lambda function that accepts search query parameters and executes an OpenSearch query against your index. Wire it to an API Gateway endpoint protected by a Cognito authorizer — so search access requires valid authentication. Return paginated results with configurable page size.

The Trap

The most common mistake is over-indexing: mapping every Cognito attribute as a searchable field. OpenSearch Serverless pricing scales with indexed data volume. Index only the attributes your application actually searches on, and store the rest as non-indexed metadata. Audit your query patterns before finalizing the index mapping.

Cited References

• AWS Architecture Blog: Building a Scalable User Search Layer on Amazon Cognito
• Amazon OpenSearch Serverless documentation
• DynamoDB Streams and Lambda integration

Source: AWS Architecture Blog

Content created with AI assistance and reviewed for accuracy.