S3 Vectors Gets Metadata Pre-Filtering: Up to 5× More Matching Vectors on Selective Queries

Vector similarity search with metadata filtering has a well-known failure mode: apply the metadata filter after the similarity search, and you get fewer results than expected — sometimes zero — on highly selective filters. Amazon S3 Vectors is addressing this with an ENHANCED index mode that flips the order of operations, evaluating metadata filters before similarity search runs.

The result on selective filters: up to 5× more matching vectors returned.

The Problem Pre-Filtering Solves

In a standard (CLASSIC) vector similarity search with metadata filtering:

1. Similarity search runs across all indexed vectors
2. Metadata filter is applied to the candidate set
3. Results are returned

When the metadata filter is highly selective — filtering for a specific document type, date range, user ID, or category — post-filtering can eliminate most or all of the similarity search candidates. A query for “concepts similar to X, but only from this client’s document corpus” might return 100 similarity candidates, of which 2 pass the metadata filter. The recall problem compounds on large indexes with narrow filter criteria.

ENHANCED mode inverts the sequence: metadata filters are evaluated first to identify the candidate space, then similarity search runs within that candidate space. For the same selective query, the similarity search operates on a pre-qualified corpus rather than the full index — and has access to far more genuinely matching vectors.

How ENHANCED Mode Works

ENHANCED mode adds several capabilities on top of the CLASSIC filter set:

Filter-first execution: Metadata conditions are evaluated before similarity search. On a query filtering for client_id = "acme" AND doc_type = "contract", the similarity search runs only against vectors matching those conditions rather than against all vectors.

Expanded filter complexity: Up to 100 filter constraints per query (a significant increase from CLASSIC mode). This enables compound filtering on large enterprise datasets where multiple categorical conditions must be satisfied simultaneously.

Hierarchy navigation with $startsWith: Document hierarchies (e.g. project/phase/component) can be queried using $startsWith, returning all vectors at or below a specified path prefix. This supports agentic systems that need to scope a search to a particular project, team, or knowledge domain without enumerating every document path.

Filterable metadata per vector: Up to 2KB of filterable metadata per vector — sufficient for enterprise use cases with extended tagging (document type, author, date, project, access control labels, domain taxonomy).

Upgrading Without Re-Ingestion

Existing S3 Vectors indexes can upgrade to ENHANCED mode via the UpdateIndexMode API call. No re-ingestion of vectors is required. The index conversion runs in the background; queries continue to return results during the conversion using CLASSIC mode until the upgrade completes.

aws s3vectors update-index-mode \
  --index-name my-production-index \
  --index-mode ENHANCED

This matters for production deployments: a mode upgrade that required full re-ingestion would be prohibitive for large indexes. The in-place upgrade path means teams can test ENHANCED mode on existing production indexes without a parallel rebuild cycle.

Where the 5× Improvement Applies

The up-to-5× recall improvement is specific to highly selective filter conditions — scenarios where a small fraction of the total vector index matches the metadata criteria. On broad filters (filtering half or more of the index), the difference between pre- and post-filtering is less pronounced because the candidate set is already large.

The use cases where ENHANCED mode has the highest impact:

• Multi-tenant RAG pipelines where each query must be scoped to a specific tenant’s documents
• Access-controlled knowledge bases where user permissions limit the accessible document set
• Domain-partitioned retrieval where a specialized agent should only search its domain (legal, financial, clinical, technical)
• Temporal filtering in agentic systems that need to reason over recent documents specifically

These are exactly the agentic system architectures where naive post-filtering fails most visibly, and where 5× recall improvement translates directly into fewer hallucinations caused by inadequate retrieval.

Source: AWS News Blog

Content created with AI assistance and reviewed for accuracy.