How Full-Text Search Works

PostgreSQL's full-text search feature answers a deceptively simple question: does this document contain these words, in some meaningful linguistic sense, and how well does it match compared to other documents?

Under the hood that question is answered by a text search pipeline that parses, classifies, and normalizes words before any matching or ranking happens, which is what separates full-text search from a plain string comparison.

This page builds the conceptual model behind that pipeline: how raw text becomes a tsvector, why matching uses lexemes instead of substrings, and how ranking turns a boolean match into an ordered result set.

Summary

• Core Idea: Full-text search converts document text and query text into normalized linguistic units called lexemes, then matches and ranks documents based on those units rather than raw characters.
• Why It Matters: Users expect "running", "runs", and "ran" to all match a search for "run", and plain string matching cannot do that without an explicit linguistic normalization step.
• Key Concepts: parser, token, lexeme, dictionary, tsvector, tsquery, ts_rank, ts_rank_cd.
• When to Use: Reach for this mental model when a search query is missing documents you expect it to find, or when ranking order looks wrong and you need to reason about why.
• Limitations / Trade-offs: The same normalization that makes search forgiving of word forms also makes it blind to exact substrings, so full-text search cannot replace LIKE for prefix or code-fragment matching.
• Related Topics: tsvector/tsquery mechanics, GIN index internals, hybrid keyword-plus-vector search, Postgres FTS versus Elasticsearch.

Foundations

Full-text search begins with a parser, a component that breaks raw text into a stream of classified tokens rather than a flat list of words.

The parser does not just split on whitespace; it recognizes categories like words, numbers, email addresses, URLs, and hyphenated compounds, and it tags each token with its category.

Each token category is then routed through a dictionary, a pluggable normalization step that can stem a word to its root, discard it as a stop word, or fold it to a canonical spelling.

The output of that dictionary stage is a lexeme, the normalized unit that full-text search actually indexes and matches, which is why "databases" and "database" both normalize down to the same lexeme under the English configuration.

A tsvector is simply the sorted, deduplicated collection of lexemes produced for one piece of text, optionally carrying weight labels and position information for each lexeme.

A tsquery goes through the same parsing and normalization pipeline, which is exactly why a raw string built by hand can produce a query that never matches anything meaningful.

The match itself, expressed with the @@ operator, is a boolean test of whether the query's lexeme expression is satisfied by the lexemes present in the document's tsvector.

Because matching happens on normalized lexemes rather than raw text, full-text search is fundamentally a linguistic operation, not a character-comparison operation.

Mechanics & Interactions

The parsing and normalization pipeline runs in a strict order every time a tsvector or tsquery is built, and understanding that order explains most "why didn't this match" questions.

First the parser tokenizes and classifies raw text into typed tokens; second, each token type is mapped to an ordered list of dictionaries through a text search configuration like english or simple; third, the first dictionary in that list that recognizes the token either normalizes it, discards it as a stop word, or passes it to the next dictionary in the chain.

This is why swapping configurations changes results without changing a single row of data: simple skips stemming entirely and is well suited to product codes, while english stems aggressively and suits prose.

Positions matter too, because a tsvector retains where each lexeme occurred in the source text, which is what makes phrase operators like <-> possible at query time.

Weights layer on top of positions: a lexeme can be labeled A, B, C, or D, typically to mark that it came from a title versus a body versus metadata, and ranking functions read those labels to bias scores.

-- The same words, two configurations, two different lexeme sets
SELECT to_tsvector('simple', 'Running the databases');
-- 'databases':3 'running':1 'the':2
SELECT to_tsvector('english', 'Running the databases');
-- 'databas':3 'run':1

Ranking is where full-text search moves from a boolean yes/no into an ordered result set, and PostgreSQL ships two ranking functions built on different theories of relevance.

ts_rank is essentially a weighted term-frequency score: it counts how many query lexemes matched, factors in their position weights, and produces a score that treats matches anywhere in the document roughly equally.

ts_rank_cd, the cover-density variant, additionally rewards documents where the matched lexemes appear close together, on the theory that clustered matches indicate a more focused, relevant passage.

Neither function considers document length normalization or corpus-wide term rarity the way a full information-retrieval engine's BM25 algorithm does, which is a deliberate simplicity trade-off in exchange for running entirely inside SQL.

Advanced Considerations & Applications

The @@ match operator alone would force a sequential scan over every row's tsvector, so full-text search leans on a GIN index to make matching fast at scale, and the conceptual trick is the same one every inverted-index search engine uses.

A GIN index inverts the relationship between documents and lexemes: instead of storing "this document contains these lexemes," it stores "this lexeme appears in these documents," which turns a match query into a fast lookup rather than a per-row scan.

That inversion is precisely what distinguishes full-text search from LIKE '%term%' or a regular expression, both of which must inspect the actual bytes of every candidate row because neither has a pre-built structure connecting terms to rows.

LIKE and regex are substring-exact and case/accent-sensitive by default, while full-text search is linguistically normalized and position-aware, so the two techniques solve genuinely different problems and are often combined rather than chosen exclusively.

External search engines like Elasticsearch build on the same inverted-index and term-frequency concepts but add a much richer analyzer ecosystem, distributed sharding, and more sophisticated relevance models, at the cost of running and operating a second system outside the transactional database.

The practical decision point is usually not "which technique is smarter" but "does the relevance and scale ceiling of an in-database inverted index meet the product's needs without paying for a second cluster."

Common Misconceptions

• "Full-text search is just LIKE with an index" - it is a linguistic pipeline built on parsed, normalized lexemes, while LIKE is a raw substring comparison with no normalization at all.
• "A tsvector stores the original text" - it stores a deduplicated, sorted set of normalized lexemes plus optional weights and positions, and the original words are not recoverable from it in general.
• "Higher ts_rank always means a better match for the user" - ts_rank reflects term frequency and weight labels, not semantic relevance, so a document can rank highly while missing the user's actual intent.
• "ts_rank and ts_rank_cd will always agree on ordering" - they encode different theories of relevance, term frequency versus match clustering, so they can and do produce different orderings for the same query.
• "Stemming works the same for every language" - each text search configuration has its own dictionary chain, so stemming, stop words, and even tokenization boundaries differ by configuration.

FAQs

Related

• Full-Text Search Basics - the step-by-step recipe for building a searchable table this page explains the theory behind
• tsvector & tsquery - deeper mechanics on dictionaries, operators, and ranking functions
• GIN Indexes for FTS - how the inverted index described here is built and maintained in practice
• Hybrid Search - combining this lexical model with vector embeddings for semantic recall
• Full-Text Search Best Practices - the operational checklist that complements this conceptual model

Stack versions: This page was written for PostgreSQL 18.4 (stable 18, maintenance 17), pgvector 0.8+, PostGIS 3.5+, pgbouncer 1.x, and Patroni 3.x.