Reranking: Improving Search Relevance with the AI SDK

Reranking improves search relevance by reordering documents based on their relevance to a query. Unlike embedding-based similarity search, reranking models are specifically trained to understand the relationship between queries and documents, often producing more accurate relevance scores.

The Problem: Vector Search vs Query Relevance

Vector search finds semantically similar documents but may miss the most relevant one when the query uses different vocabulary. For example, a query like "How do I get a refund?" might match documents containing "billing" or "payment" (similar financial vocabulary) rather than the document that actually answers the question.

Two-Stage Retrieval Pipeline

The recommended approach combines fast vector search with precise reranking:

┌───────────────────────────────────────────────────────────────┐
│                    All Documents (1000s)                      │
└──────────────────────────┬────────────────────────────────────┘
                           │
                           ▼
              ┌───────────────────────┐
              │   Vector Search       │
              │   (Fast, Broad)       │
              └──────────┬────────────┘
                         │
                         ▼
              ┌─────────────────────────┐
              │   Top 20-100 Candidates │
              └──────────┬──────────────┘
                         │
                         ▼
              ┌────────────────────────┐
              │   Reranking            │
              │   (Precise, Slower)    │
              └──────────┬─────────────┘
                         │
                         ▼
              ┌─────────────────────────┐
              │   Top 3-10 Results      │
              └─────────────────────────┘

1. Broad retrieval: Vector search returns top 20-100 candidates quickly (tune for latency/recall)
2. Precise reranking: Re-score candidates for query relevance using the rerank function

Example: Customer Support Knowledge Base

Using a knowledge base with 6 articles, we compare vector search with reranking.

For clarity, examples below rerank small document sets directly. In production, rerank only the top candidates from vector search (see Two-Stage Retrieval Pipeline).

Vector Search Results

import { embed, cosineSimilarity } from 'ai';
import { ollama } from 'ai-sdk-ollama';

const query = 'How do I get a refund?';
const queryEmbedding = await embed({
  model: ollama.textEmbeddingModel('nomic-embed-text'),
  value: query,
});

// Compare with pre-embedded articles
const scores = articles.map((article, i) => ({
  article,
  score: cosineSimilarity(queryEmbedding.embedding, articleEmbeddings[i].embedding),
}));

const topResults = scores.sort((a, b) => b.score - a.score).slice(0, 3);

Results:

Vector search ranks "Billing and Payment Methods" #1 because it shares financial vocabulary, but it doesn't answer the refund question.

After Reranking

Using the rerank function:

import { rerank } from 'ai';
import { ollama } from 'ai-sdk-ollama';

const { ranking, rerankedDocuments } = await rerank({
  model: ollama.embeddingReranking('embeddinggemma'),
  documents: supportArticles.map(a => ({
    id: a.id,
    title: a.title,
    content: a.content,
  })),
  query: 'How do I get a refund?',
  topN: 3,
});

Results:

Reranking correctly identifies KB003 (which contains "Refunds are available within 14 days") as the most relevant result.

Comparison: Vector Search vs Reranking

Testing 7 queries designed to expose vocabulary mismatch issues:

Results:

• Vector Search accuracy: 5/7 (71%)
• After Reranking: 7/7 (100%)
• Queries improved: 2 (both vocabulary mismatch cases)

Working with Object Documents

Reranking supports structured documents (JSON objects), making it ideal for searching through databases, emails, or other structured content:

import { rerank } from 'ai';
import { ollama } from 'ai-sdk-ollama';

const emails = [
  { from: 'aws-billing@amazon.com', subject: 'Your AWS Invoice', date: '2024-12-15' },
  { from: 'team@linear.app', subject: 'Weekly Project Update', date: '2024-12-14' },
  { from: 'noreply@github.com', subject: 'Security alert: new sign-in', date: '2024-12-14' },
  { from: 'sales@datadog.com', subject: 'Your Datadog Quote - Enterprise Plan', date: '2024-12-13' },
  { from: 'no-reply@vercel.com', subject: 'Deployment failed: main branch', date: '2024-12-13' },
  { from: 'hr@company.com', subject: 'Holiday Schedule Reminder', date: '2024-12-12' },
];

const { ranking, rerankedDocuments } = await rerank({
  model: ollama.embeddingReranking('embeddinggemma'),
  documents: emails,
  query: 'Find the pricing quote from Datadog',
  topN: 3,
});

Results:

Use Case: RAG Context Selection

For RAG applications, reranking selects the most relevant documents to include in the LLM context:

import { rerank } from 'ai';
import { ollama } from 'ai-sdk-ollama';

const docs = [
  'The createClient() function initializes a new API client...',
  'Authentication uses Bearer tokens...',
  'Rate limiting returns HTTP 429 when exceeded...',
  'Webhooks are sent via POST to your configured endpoint...',
  'Error responses follow RFC 7807 Problem Details format...',
  'Pagination uses cursor-based navigation...',
];

const { rerankedDocuments, ranking } = await rerank({
  model: ollama.embeddingReranking('embeddinggemma'),
  documents: docs,
  query: 'How do I handle API errors?',
  topN: 2,
});

// Use rerankedDocuments for LLM context
const context = rerankedDocuments.join('\n\n');

Results:

These top documents would be passed to the LLM for answer generation.

Understanding the Results

The rerank function returns a comprehensive result object:

import { rerank } from 'ai';
import { ollama } from 'ai-sdk-ollama';

const { ranking, rerankedDocuments, originalDocuments } = await rerank({
  model: ollama.embeddingReranking('embeddinggemma'),
  documents: supportArticles,
  query: 'How do I get a refund?',
  topN: 3,
});

Return value properties:

Each item in the ranking array contains:

• originalIndex: Position in the original documents array
• score: Relevance score (higher is more relevant)
• document: The original document

Note: Scores are not comparable across models or providers; only relative ordering within a single rerank call matters.

Filtering by Score Threshold

In production, you may want to discard results with low relevance scores. Filter the ranking array to only include results above a threshold:

const { ranking } = await rerank({
  model: cohere.reranking('rerank-v3.5'),
  documents,
  query,
  topN: 10,
});

// Filter to only include results with score > 0.4
const relevantResults = ranking.filter(item => item.score > 0.4);
// Returns only documents with relevance score above threshold

Reranking Providers & Models

The AI SDK supports multiple reranking providers. You can use reranking models from:

Cohere

import { rerank } from 'ai';
import { cohere } from '@ai-sdk/cohere';

const { ranking } = await rerank({
  model: cohere.reranking('rerank-v3.5'),
  documents,
  query,
  topN: 3,
});

Amazon Bedrock

import { rerank } from 'ai';
import { bedrock } from '@ai-sdk/amazon-bedrock';

const { ranking } = await rerank({
  model: bedrock.reranking('cohere.rerank-v3-5:0'),
  documents,
  query,
  topN: 3,
});

Ollama (Local)

import { rerank } from 'ai';
import { ollama } from 'ai-sdk-ollama';

const { ranking } = await rerank({
  model: ollama.embeddingReranking('embeddinggemma'),
  documents,
  query,
  topN: 3,
});

Note: Some reranking models (e.g., Cohere rerankers) are cross-encoders that jointly score query–document pairs, while others (e.g., embedding-based rerankers) trade some accuracy for speed and locality. Cross-encoders typically provide higher accuracy but require API calls, while embedding-based rerankers can run locally with lower latency.

Settings

Top-N Results

Use topN to limit the number of results returned. This is useful for retrieving only the most relevant documents:

import { rerank } from 'ai';
import { cohere } from '@ai-sdk/cohere';

const { ranking } = await rerank({
  model: cohere.reranking('rerank-v3.5'),
  documents: ['doc1', 'doc2', 'doc3', 'doc4', 'doc5'],
  query: 'relevant information',
  topN: 3, // Return only top 3 most relevant documents
});

Provider Options

Reranking model settings can be configured using providerOptions for provider-specific parameters. This is particularly useful for handling long documents that might otherwise be truncated:

import { rerank } from 'ai';
import { cohere } from '@ai-sdk/cohere';

const { ranking } = await rerank({
  model: cohere.reranking('rerank-v3.5'),
  documents: ['long document 1...', 'long document 2...'],
  query: 'relevant information',
  providerOptions: {
    cohere: {
      maxTokensPerDoc: 1000, // Limit tokens per document
    },
  },
});

Examples (provider-specific; check provider docs for available options):

• Cohere: maxTokensPerDoc - Maximum tokens per document (default varies by model)
• Other providers may expose additional options via providerOptions

Best Practices

1. Use vector search for broad retrieval first: Retrieve top 20-100 candidates before reranking (tune for latency/recall trade-offs). In practice, reranking typically adds tens of milliseconds for 20–50 documents, depending on model and provider.

2. Tune topN based on use case:

   • RAG: topN: 3 (small context window)
   • Search results: topN: 10
   • Auto-reply selection: topN: 1

3. Monitor performance: Track click-through rates, RAG answer accuracy, and queries requiring refinement.

Summary

Reranking improves search relevance by reordering documents based on query-document relationships rather than just semantic similarity. For search systems where precision matters customer support, RAG, email, documentation, reranking improves result quality with minimal code changes.

See the ai-sdk-reranking-example repository for runnable code.