Embeddings and vector databases

Why this matters

Embeddings are how computers measure meaning. Vector DBs are how we store and search billions of those meanings in milliseconds. Together they power 80% of every enterprise AI system: search, RAG, recommendations, anomaly detection, and agent memory.

In 2026, the pay-related skill is not "I have heard of vectors." It is "I picked Chroma over pgvector for X reasons, tuned the index for Y, and our recall@10 went from 0.71 to 0.94."

Learning objectives

1. Generate embeddings with OpenAI, Voyage, and a local sentence-transformers model.
2. Build a working semantic search index with ChromaDB 1.x (the most used local DB).
3. Build the same with pgvector on Postgres (the production default in 2026 enterprise).
4. Pick distance metrics, index types, and collection settings consciously.
5. Visualise embedding clusters with UMAP.

1. What is an embedding, really?

An embedding is a fixed-length list of floats (a vector) representing a piece of text's meaning.

Rule of thumb: start with text-embedding-3-small (it is cheap and good). Move to voyage-3-large when you need every percentage of recall. Move to a local model when privacy or volume forces it.

from openai import OpenAI
client = OpenAI()

vec = client.embeddings.create(
    model="text-embedding-3-small",
    input="agentic AI is the future of software",
).data[0].embedding
print(len(vec))  # 1536

Distance / similarity metrics

To compare two vectors:

• Cosine similarity (default): measures angle, ignores magnitude. Use this 95% of the time.
• Dot product: like cosine if vectors are normalized. Slightly faster.
• L2 / Euclidean: distance in space. Sometimes used for image embeddings.

import numpy as np
def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

OpenAI / Voyage embeddings come already normalised, so cosine = dot product = same ranking.

2. ChromaDB — the easiest local vector DB

ChromaDB (1.x in 2026) is the default for prototyping and small production. The whole API is 4 functions.

# uv add chromadb
import chromadb
from chromadb.config import Settings

# Persistent on disk:
client = chromadb.PersistentClient(path="./chroma_db")

# Or in-memory for tests:
# client = chromadb.Client()

collection = client.get_or_create_collection(
    name="docs",
    metadata={"hnsw:space": "cosine"},  # or "l2", "ip" (dot product)
)

collection.add(
    ids=["doc1", "doc2", "doc3"],
    documents=[
        "LangGraph models agents as state machines.",
        "CrewAI uses role-based agent crews.",
        "MCP standardises tool integration for LLMs.",
    ],
    metadatas=[
        {"source": "blog", "year": 2026, "tags": ["langgraph", "agents"]},
        {"source": "docs", "year": 2026, "tags": ["crewai"]},
        {"source": "spec", "year": 2026, "tags": ["mcp"]},
    ],
)

# Chroma will embed query_texts automatically using its default embedder
# (ONNX MiniLM-L6-v2). Override with `embedding_function` for production.
results = collection.query(
    query_texts=["how do AI agents communicate with tools?"],
    n_results=2,
    where={"year": 2026},
    where_document={"$contains": "tool"},
)
print(results["documents"])

Picking the embedding function

Default is fine for demos. For production, you almost always want OpenAI or your own:

from chromadb.utils import embedding_functions

oa_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.environ["OPENAI_API_KEY"],
    model_name="text-embedding-3-small",
)
collection = client.get_or_create_collection("docs", embedding_function=oa_ef)

You can also pass your own pre-computed embeddings via embeddings=[...] — useful when you embed elsewhere (e.g. a queue job).

Metadata filters (the secret weapon)

ChromaDB supports $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $contains, $not_contains, $and, $or, $regex. Combine for surgical retrieval:

collection.query(
    query_texts=["password reset"],
    n_results=5,
    where={"$and": [
        {"locale": {"$eq": "en"}},
        {"updated_at": {"$gte": "2026-01-01"}},
    ]},
    where_document={"$contains": "password"},
)

Use metadata to enforce multi-tenancy (filter by tenant_id), freshness (filter by date), and permissions (filter by allowed groups). Cheap and effective.

Persistence and scaling

PersistentClient writes a SQLite + Parquet store under ./chroma_db. For multi-process or production, run a Chroma server:

chroma run --path ./chroma_db --port 8000

client = chromadb.HttpClient(host="localhost", port=8000)

For real scale (>10M docs) most teams move to Qdrant, Weaviate, Milvus, or pgvector.

3. pgvector — the production default in 2026

You already have a Postgres in your stack. Adding pgvector means one less moving part, real ACID transactions, joins with relational data, easy backups. That is why enterprise AI shops moved to it.

Setup (Docker)

docker run -d --name pgvec -p 5432:5432 -e POSTGRES_PASSWORD=secret pgvector/pgvector:pg17

Schema

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id          BIGSERIAL PRIMARY KEY,
  doc_id      TEXT NOT NULL,
  text        TEXT NOT NULL,
  metadata    JSONB,
  embedding   VECTOR(1536)  -- match your model dim
);

-- HNSW index for fast cosine search
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

-- GIN index for metadata
CREATE INDEX ON chunks USING gin (metadata);

Insert + query

# uv add psycopg2-binary "psycopg[binary]" sqlalchemy
import psycopg
from openai import OpenAI

oc = OpenAI()
def embed(t: str): return oc.embeddings.create(model="text-embedding-3-small", input=t).data[0].embedding

conn = psycopg.connect("dbname=postgres user=postgres password=secret host=localhost")
cur = conn.cursor()

# insert
text = "LangGraph models agents as state machines."
cur.execute(
    "INSERT INTO chunks (doc_id, text, metadata, embedding) VALUES (%s,%s,%s,%s)",
    ("doc1", text, {"source": "blog"}, embed(text)),
)
conn.commit()

# query — cosine distance is `<=>`
q = embed("how do AI agents communicate?")
cur.execute(
    """
    SELECT id, text, 1 - (embedding <=> %s::vector) AS score
    FROM chunks
    WHERE metadata->>'source' = 'blog'
    ORDER BY embedding <=> %s::vector
    LIMIT 5
    """,
    (q, q),
)
for row in cur.fetchall():
    print(row)

Index choices

• HNSW (default in 2026): graph-based, sub-millisecond, slightly slower writes. Pick this.
• IVFFlat: simpler, faster writes, slightly slower reads. Use when corpus is small or rebuild-heavy.
• No index: exact brute-force. Fine up to 100k vectors.

When to pick which DB

4. Visualising embeddings (a portfolio-impressing 30 minutes)

Hiring managers love a UMAP plot of your embeddings. Run once per project.

# uv add umap-learn matplotlib pandas scikit-learn
import umap, matplotlib.pyplot as plt
import numpy as np
from openai import OpenAI

oc = OpenAI()
texts = [...]            # 200-1000 docs
embs = np.array([oc.embeddings.create(model="text-embedding-3-small", input=t).data[0].embedding for t in texts])

reducer = umap.UMAP(n_neighbors=15, min_dist=0.1, metric="cosine", random_state=42)
coords = reducer.fit_transform(embs)

plt.figure(figsize=(8,6))
plt.scatter(coords[:,0], coords[:,1], s=4)
plt.title("Document semantic space"); plt.tight_layout(); plt.savefig("umap.png")

Use cluster colors (KMeans labels) and sample nearest-neighbours to gut-check that semantically similar docs sit near each other. Drop the screenshot into your README.

Hands-on lab (4 hours)

Build mini_search.py that:

1. Loads 500 random arXiv abstracts (use the arxiv Python client or a CSV).
2. Embeds them with text-embedding-3-small and stores in ChromaDB with metadata {category, year}.
3. Mirrors the same data into pgvector via Docker.
4. Implements a CLI: python mini_search.py "agentic ai" --year 2025 --top 5 returning results from both backends.
5. Benchmarks: query latency p50/p95 on 100 random queries — Chroma vs pgvector.
6. Generates a UMAP scatter colored by category.

Acceptance criteria:

• Identical top-1 result on 80%+ queries between Chroma and pgvector.
• p95 < 60ms on a laptop, n=500.
• README with index choices justified ("we picked HNSW because...").

Common pitfalls

1. Mismatched embedding model and dim. Migrating from small (1536) to large (3072) requires re-embedding everything.
2. Storing the document text inside the vector field. Use a separate column.
3. Forgetting normalised embeddings. Cosine ranking will look weird with un-normalised vectors.
4. No index. A 1M-vector brute-force query takes seconds.
5. Cosine vs L2 mistakes. Pick once, document it.

Self-check

1. Why is 1 - (embedding <=> q) the cosine similarity?
2. When would you NOT pick pgvector?
3. What does HNSW stand for and why is it fast?
4. Why does "default" Chroma embedding fail in production?
5. What does UMAP do that PCA does not?

References

• Chroma docs
• Chroma metadata filtering
• pgvector GitHub
• pgvector HNSW vs IVFFlat
• Voyage embeddings benchmarks
• OpenAI embeddings
• UMAP docs