ML System Design Interview Questions (2026)

Practice end-to-end ML system design: recommendation, ranking, RAG, LLM agents, training pipelines, serving, and evaluation — the round that filters most MLE candidates. This bank has 158 ml system design questions (51 with free model answers).

Other topics: ML Theory · ML Coding · Behavioral · All questions

Practice this question with an AI interviewer

Get asked follow-ups live, then receive a scored report — like a real MLE interview loop.

Start AI mock interview

#17 Architect a Recommender for an Autoplay Short-Video Feed ★
#18 Architect the Retrieval and Ranking Stack for Search ★
#19 Architect a Low-Latency Model for Predicting Ad Click Probability ★
#20 Architect a Customer-Support Assistant Grounded in Company Knowledge ★
#21 Architect Monitoring and Alerts for a Production Model ★
#30 Architect a Shared Platform for Creating and Serving Features ★
#31 Architect an Experimentation Platform with Sound Assignment and Analysis ★
#32 Architect Approximate Retrieval Over More Than 100 Million Embeddings ★
#40 Architect a Production Service for Autoregressive LLM Generation ★
#165 Contrast the prefill and decode phases and explain why prefill is compute-bound while decode is memory-bandwidth-bound. ★
#166 Explain continuous (in-flight) batching and why it replaced static batching. ★
#167 Explain PagedAttention and the KV-cache fragmentation problem it solves. ★
#168 Explain speculative decoding: why output quality is preserved and when it does not help. ★
#169 Explain prefix/prompt caching: when to use it and what invalidates a cached prefix. ★
#170 Compare FP16, BF16, FP8, INT8, INT4, and FP4 for serving and name what breaks at each step down. ★
#171 Compare tensor, pipeline, data, sequence, and expert parallelism and explain when to combine them. ★
#172 Estimate the GPU memory needed to serve a 70B model, accounting for weights, KV cache, activations, and fragmentation. ★
#173 Define TTFT, TPOT, ITL, and throughput, and explain how they trade against each other. ★
#174 Do the roofline math: how many tokens/sec can one H100 produce for a 70B model at batch size 1? ★
#175 Decide when to choose vLLM, SGLang, TensorRT-LLM, or a custom stack. ★
#176 Enumerate and rank every lever for cutting LLM serving cost by 10x. ★
#177 Your p99 latency doubled after a deploy with no model change — walk through the diagnosis. ★
#178 Explain chunked prefill and why it improves tail latency under mixed traffic.
#179 Explain disaggregated prefill/decode serving and when it pays for itself. ★
#180 Design a batched inference system in which 100 requests take the same time as one. ★
#181 Design the serving stack for a Claude-scale LLM API that maximizes GPU utilization without wrecking p99 latency. ★
#182 Estimate the KV-cache memory to serve a 70B-class model at 128k context, and say what you do when it does not fit. ★
#183 Design a rate limiter for an LLM API where cost scales with tokens rather than requests. ★
#184 Design the scheduler for a continuous-batching inference engine. ★
#185 Explain when speculative decoding helps, when it hurts, and why you would adapt the speculator to live traffic. ★
#186 Price a dedicated 70B endpoint: estimate cost per million output tokens and explain the throughput-latency trade. ★
#187 Design a serverless inference platform serving 100+ open models on a shared GPU fleet. ★
#188 Design the serving stack for a ChatGPT-scale consumer assistant: hundreds of millions of weekly users, streaming chat, multiple model tiers. ★
#189 Design a 70B+ model serving stack for hundreds of millions of assistant users and explain where the money goes. ★
#190 Design the parallelism strategy for serving a 405B dense model: where TP, PP, and EP go and why. ★
#191 Design the serving stack for a consumer chatbot with real-time search over a social-media firehose. ★
#192 Sketch how you would serve a 671B-parameter MoE with low latency under GPU-memory constraints. ★
#193 Walk through serving a 70B model on a single 80GB GPU: whether it fits and the expected single-stream tokens/sec. ★
#194 Explain what TensorRT/TensorRT-LLM actually does to speed up a model and when it will not help. ★
#195 Estimate what one message costs to serve and identify which lever moves that cost most. ★
#196 Our serving cost is dominated by KV cache, not weights — cut it by an order of magnitude and name what you give up.
#197 Design the cache that sits between turns for dialogues averaging ~180 messages.
#198 Design a routing layer that decides whether a request runs on-device, on a first-party server model, or on a third-party model.
#199 Explain how you would run a ~3B model on a phone with tight memory and power budgets versus in a datacenter.
#200 Contrast post-training quantization with quantization-aware training, and explain what breaks at 2–4 bits and how to recover quality.
#201 Walk through diagnosing and fixing an on-device feature whose time-to-first-token is 1.8s.
#202 Explain why serving real-time TTS is a different capacity problem from serving a text LLM, and how you would plan capacity.
#203 Budget end-to-end latency for a real-time voice agent and explain why time-to-first-audio differs from time-to-first-token.
#204 Budget latency for a real-time voice agent across VAD, ASR, LLM, TTS, and network, and say where the time goes. ★
#205 Quantize and prune a vision model for a fixed onboard compute budget without losing recall on small objects.
#206 Redo the decode roofline argument for an LPU with no HBM — only on-die SRAM — and explain what changes.
#207 Walk through deploying a 70B dense model at 8-bit weights on chips with ~230MB of SRAM each, including unit economics.
#208 A customer's distributed training job gets 55% scaling efficiency at 64 nodes — debug it.
#209 Explain what breaks when scaling LLM training from 8 GPUs to thousands, and how modern stacks handle it.
#210 You are training on tens of thousands of GPUs with constant hardware failures — how do you keep goodput high?
#211 Diagnose and fix a pretraining loss that suddenly diverges at step 300k of a long run.
#212 Design the training setup for a 70B-parameter model that does not fit on one accelerator.
#213 Design a deduplication pipeline for a web-scale pretraining corpus that runs as a streaming process.
#214 Walk through building a web-scale pretraining corpus pipeline and deciding whether each filter earns its place.
#215 Choose a chunking strategy for a large technical-documentation corpus and justify it.
#216 Decide between a sparse BM25 retriever and a dense retriever, and explain when you need both. ★
#217 Explain what a reranker is, when to use one, and what a cross-encoder costs you. ★
#218 Evaluate a RAG pipeline's retrieval and generation quality separately.
#219 Design permission-aware retrieval so users never see content they cannot access in the source system. ★
#220 Compare HNSW, IVF-PQ, and flat indexes: how to pick, and what recall@k costs in latency.
#221 Design an enterprise RAG assistant over 10M documents with per-user permissions. ★
#222 Explain the full RAG pipeline plus embedding training details.
#223 Design the ranking system combining BM25, dense retrieval, and LLM reranking across multiple indexes.
#224 With 50 retrieved passages but a useful context budget of ~10, explain how you choose and how you know the choices are good.
#225 Design an answer engine that returns a cited, streamed answer within a 3-second end-to-end budget.
#226 Build retrieval over a 200-page credit agreement where the operative clause on page 140 depends on a defined term on page 8.
#227 Argue with numbers for when to put a whole contract in context versus retrieving over it.
#228 Explain why plain RAG gets "how many open orders are blocked on a supplier issue?" wrong, and name the right architecture.
#229 Merge ranked results from N connector shards into a global top-k with a per-user permission filter, efficiently.
#230 Design a connector framework syncing content and permissions from 100+ SaaS apps into one index.
#231 Design cross-lingual RAG where the knowledge base is English and Hindi but users ask in Tamil, Telugu, or transliterated Hinglish.
#232 Explain when a 1M-token context window is the right tool versus retrieval for a long-context assistant.
#233 Explain what causes semantic drift in embedding search and how to detect it.
#234 Design a two-stage retrieval pipeline with an embedding model and a reranker, and explain when the reranker earns its latency.
#235 Defend RAG over fine-tuning for an enterprise assistant, and say where RAG breaks.
#236 Explain how an answer engine handles breaking news — a query about something that happened 20 minutes ago.
#237 Explain the ReAct pattern and what it solves beyond chain-of-thought alone. ★
#238 Describe how to make a production tool-calling agent loop reliable enough to ship.
#239 Design the harness for an agent making multi-file changes from a natural-language task without wrecking the codebase.
#240 Design human-in-the-loop approval for an agent that takes consequential actions. ★
#241 Walk through the prompt-injection attack surface and defenses for an agent that reads inbound email and can reply and search internal docs.
#242 Design guardrails for a consumer-facing assistant: input filters, output filters, or both? ★
#243 Explain how to detect and measure hallucinations in a production RAG system. ★
#244 Design an LLM-as-judge evaluation, name its known biases, and explain how to correct them.
#245 Explain how you would evaluate an autonomous software-engineering agent and why SWE-bench pass rates mislead.
#246 Design the evaluation system for an AI assistant before and after each model release.
#247 Design an evaluation framework for ads ranking.
#248 A customer says "the model got worse" after a model-version upgrade — how do you verify and respond?
#249 An eval pipeline reports a 6-point drop for a customer's model between Tuesday and Wednesday with no model change — debug it.
#250 Explain how benchmark scores can improve while users say the system got worse.
#251 Design the safety system for open-ended character chat.
#252 Design the voice-cloning safety stack: consent, watermarking, and abuse response.
#253 Treat a generated note containing a medication the patient never mentioned as a safety incident: how do you detect it before a clinician sees it?
#254 Investigate and mitigate a conversational AI model that gives overly confident but factually wrong answers in high-risk contexts.
#255 Design the harmful-content detection system for Facebook and Instagram uploads.
#256 Design a content-moderation system combining classifiers and LLMs.
#257 Design the Instagram Reels recommendation system.
#258 Design a personalized news-feed ranking system for Facebook's feed.
#259 Design a recommendation system for Facebook Ads.
#260 Your ads CTR model shows a 2% offline AUC gain but the online A/B is revenue-neutral with worse calibration — explain what is going on and what you do.
#261 Enumerate the reasons an offline metric can improve while the online A/B does not, and explain how to tell them apart.
#262 Explain position bias in ranking data and how you would debias training.
#263 Explain how to design a feature store and what causes training/serving skew.
#264 Design an ETA prediction system for a ride-hailing marketplace: features, model, and sub-100ms serving.
#265 Design a personalized feed ranking system with two-stage candidate generation and ranking.
#266 Design a streaming-catalogue content recommendation system including cold start for new titles and users.
#267 Design a fraud-detection system with heavy class imbalance and an adversarial opponent. ★
#268 Explain how to monitor a deployed ranking model for drift and what triggers a retrain.
#269 Given a user and a product, model the probability of viewing, clicking, and purchasing: choose features and decide between joint vs independent models.
#270 Explain how you would train a model on millions of Amazon data points, including downsampling choices.
#271 Design an ML system that predicts which links to show for a search query, given user_id, timestamp, query, and clicked links.
#272 Design the data engine: fleet triggers → upload → labelling → retraining → shadow-mode validation → release.
#273 Explain how you would actually measure whether an autonomy release is safer than the last one, given disengagement rate is a weak proxy.
#274 Design a system that finds driving segments similar to a given one across the entire fleet archive.
#275 Design Duolingo's spaced-repetition scheduler at 100M DAU scale.
#276 Design the gamification events pipeline (streaks, leagues, gems).
#277 Propose an ML solution for detecting spam in user reports.
#278 Design an A/B test to evaluate a new AI tutor's effect on 30-day retention.
#279 Turn a clinical conversation into billable diagnosis codes: define the accuracy bar and how to build to it.
#280 Explain how to improve transcription of drug names using the patient's chart medications without it backfiring.
#281 Design an RLHF preference-data pipeline producing 100k prompt-response comparisons a week with quality guarantees.
#282 With no ground truth on subjective preference judgments, explain how to measure and improve label quality.
#283 Design an RL environment for booking a multi-city trip in a web travel app: specify the reward and stop the policy from hacking it.
#284 Design a document-intelligence pipeline: scanned PDFs in, structured fields out, at 10M documents.
#285 Design a Text-to-SQL system over a warehouse with thousands of tables. ★
#286 Design a Copilot feature answering questions over a user's work email, documents, and meetings without leaking inaccessible content.
#287 Explain how you would evaluate a meeting-summarization feature before shipping it to a hundred million users.
#288 Design an LLM gateway with routing across providers, failover, caching, budgets, and rate limits. ★
#289 Design a code assistant: repo indexing, context assembly, edit application, and evaluation.
#290 Design Cursor's tab (next-edit prediction) system with sub-100ms perceived latency for millions of daily users.
#291 Explain how to backfill six months of history without blocking today's loads.
#292 Explain how you decide between a t-test and a non-parametric test.
#293 Our signup conversion dropped 8% last week — explain how you would investigate.
#417 Design an image-generation service (Midjourney/DALL-E style): queue management, GPU scheduling, and content-safety filtering at scale
#418 Design an autonomous travel-booking agent: orchestrator, tool gateway, credential management, rollback, and risk-classified actions
#419 Design a hospital voice assistant: real-time STT+LLM+TTS pipeline, HIPAA compliance, and medical latency budgets
#420 Design a conversational recommender system that fuses dialogue state tracking with recommendation engines
#421 Design a meeting assistant: recording, speaker diarization, summarization, action-item extraction, and calendar integrations
#422 Design a distributed search system serving 1 billion documents at 1 million QPS
#423 Design the tool surface for a coding agent: which tools exist, what their schemas look like, and how results come back
#424 Design a multi-agent system where specialized agents collaborate over MCP, and explain how you evaluate it in production
#425 Design the memory architecture for a long-running agent: short-term, long-term, shared, and episodic memory with freshness guarantees
#426 Design multi-tenant isolation for an AI platform: data, access, compute, and cache isolation across tenants
#427 Design an ML experiment-tracking platform: run logging, metric comparison, and artifact management at org scale
#428 Design visual search: from image embeddings to index to ranking for visually similar products
#429 Design a landmark-recognition system over a global photo corpus
#430 Design automatic face and license-plate blurring for street-level imagery at planetary scale
#431 Design over-the-air model-update distribution across millions of on-device models: delta updates, staged rollout, and rollback
#432 Decide when a task needs agentic RAG over single-shot retrieval, and design the iterative retrieve-plan-act loop
#433 Your RAG system works on 10K documents but breaks at 100M — explain what changes in the architecture
#434 Design conflict resolution for RAG when retrieved sources disagree: versioning, source authority, and surfacing disagreement
#435 Design a hallucination-free banking chatbot: domain-constrained generation, citation verification, and regulatory compliance
#436 Your RAG chatbot gives fluent but factually wrong answers — explain how you isolate retrieval failures from generation failures and fix each