Architect Approximate Retrieval Over More Than 100 Million Embeddings

Brute force does not work: 100M vectors times full inner product per query blows up latency and cost. You need approximate nearest neighbor (ANN) - trade a little recall for orders of magnitude in speed.

Index choices: IVF (cluster into buckets, search only nearby buckets) as the base, PQ to compress vectors and save memory, HNSW graph index for high recall at high memory cost. Explain the recall/latency/memory tradeoff for each.

System side: Sharding and replication, incremental index updates (new items go to a hot table, merged on offline rebuild), and inner product vs. cosine (normalization matters).

Report all three acceptance metrics together: recall@K against brute-force ground truth, P99 latency, and cost per QPS - not just speed.