ML Theory Interview Questions (2026)

Master the fundamentals interviewers actually probe: loss functions, optimization, regularization, probability, and generalization — with free model answers. This bank has 165 ml theory questions (59 with free model answers).

Other topics: ML System Design · ML Coding · Behavioral · All questions

Practice this question with an AI interviewer

Get asked follow-ups live, then receive a scored report — like a real MLE interview loop.

Start AI mock interview

#1 Explain a Single K-Means Update Step by Step ★
#2 How Can Softmax Be Calculated Without Numerical Overflow? ★
#3 For Classification, Why Is Cross-Entropy Usually a Better Loss Than Squared Error? ★
#4 What Causes Gradient Signals to Vanish or Blow Up, and How Can You Stabilize Them? ★
#5 Compare Plain SGD, Momentum, and Adam—Where Can Adam Fall Short? ★
#6 How Do L1 and L2 Penalties Shape a Model Differently? ★
#7 Validation Error Starts Rising While Training Error Keeps Falling—What Should You Change? ★
#8 How Should You Explain the Trade-Off Between Model Bias and Variance? ★
#9 Which Forms of Data Leakage Matter Most, and How Can You Block Them? ★
#10 How Does XGBoost Use Curvature and Regularization in Its Objective? ★
#11 In What Ways Does LightGBM Differ from XGBoost? ★
#12 How Should an MLE Interpret the Meaning of ROC-AUC? ★
#13 How Would You Tackle a Severely Imbalanced Classification Dataset? ★
#14 How Would You Engineer Numeric, Categorical, and Temporal Inputs?
#15 When Should You Use Built-In Importance, Permutation Tests, or SHAP?
#16 Compare Precision@K, Recall@K, and NDCG for Ranking Evaluation ★
#25 During Training and Serving, How Should Dropout Be Applied? ★
#26 When Should a Network Use Batch Normalization Rather Than Layer Normalization?
#27 Derive the Main Steps and Complexity of Self-Attention
#28 Explain a Transformer from Input Embeddings Through Its Output Head
#29 What Makes Skip Connections Effective in Very Deep Networks? ★
#39 Connect Cross-Entropy, KL Divergence, and Maximum Likelihood ★
#42 Derive scaled dot-product attention and explain why dividing by √d_k is essential. ★
#43 Describe the KV cache, derive its memory formula, and discuss what it costs at production scale. ★
#44 Contrast multi-query and grouped-query attention with standard multi-head attention, and name what is sacrificed. ★
#45 Explain multi-head latent attention and the motivation DeepSeek had for introducing it. ★
#46 Walk through FlashAttention and account for its speedup given that it performs the same FLOPs.
#47 Describe how byte-pair encoding works and where it breaks down (numbers, code, non-Latin scripts). ★
#48 Trace the evolution of positional encodings from sinusoidal and learned schemes through RoPE and ALiBi, and explain why transformers need them.
#49 Explain rotary position embeddings and how position interpolation or YaRN stretches context past the training length. ★
#50 State what the Chinchilla scaling laws claim and how they revised earlier intuitions about scale.
#51 Describe mixture-of-experts and how it grows model capacity without growing per-token compute. ★
#52 Distinguish pretraining, supervised fine-tuning, and preference optimization. ★
#53 Compare greedy, beam, top-k, top-p, and temperature decoding, and identify when each one fails. ★
#54 Explain the lost-in-the-middle phenomenon in long contexts and how you would mitigate it. ★
#55 Justify placing LayerNorm before each transformer block in modern architectures and describe RMSNorm.
#56 Explain SwiGLU and why gated activations displaced ReLU/GELU in current LLM feed-forward blocks. ★
#57 Trace a single forward pass of a decoder-only transformer tensor by tensor.
#58 Sketch the architecture of a Transformer model.
#59 Explain self-attention, its computational complexity, and your options as contexts grow long.
#60 Clarify how cross-entropy, KL divergence, and perplexity relate, and why cross-entropy is the standard LM training loss.
#61 Explain why transformers displaced RNNs for language modeling and what the KV cache buys you at inference.
#62 Justify the architectural choices of a Llama-class model — grouped-query attention, RoPE, SwiGLU — against the original 2017 Transformer.
#63 Describe Mistral 7B's grouped-query plus sliding-window attention and what each choice gains and costs.
#64 Explain how a Mixtral-style sparse MoE works and why a ~47B-parameter model runs at roughly 13B cost.
#65 Walk through DeepSeekMoE and how it differs from a conventional top-2 MoE such as Mixtral.
#66 Explain DeepSeek-V3's auxiliary-loss-free load balancing: what was wrong with the auxiliary loss and how the bias trick works.
#67 Describe multi-token prediction and the reason for training with it.
#68 Explain what breaks first when pushing Kimi from 8K to hundreds of thousands of tokens, and why.
#69 Explain Kimi K2's MLA and compare it with GQA for KV-cache reduction.
#70 Contrast GLM's autoregressive blank-infilling pretraining objective with BERT and GPT, and why the team claimed it unifies understanding and generation.
#71 Explain Qwen's byte-level BPE with a ~151K vocabulary and single-character digits: why those choices and what trade-offs they carry.
#72 Explain why tokenization is the first bottleneck for Indian-language LLMs and how a low-fertility tokenizer changes the economics.
#73 Name the key components of a Transformer and explain why each one matters.
#74 Explain attention-free transformer architectures and their trade-offs.
#75 Explain what "attention" means in ML models and what happens to a network with all hidden layers removed.
#76 A GPU/TPU rated at 500 TFLOPS sustains only ~50 TFLOPS on a large-model kernel — what chip-level factors could explain the 10x drop?
#77 Given D=4096, F=4D, V=32000, L=64, compute a transformer's parameter count and the fraction that are attention parameters.
#78 Compare fine-tuning T5, RoBERTa, and Llama 2 7B for multi-turn conversational chatbots and argue which is most likely to perform best.
#79 Argue whether encoder-decoder models are still needed when decoder-only models can do everything.
#80 Contrast BERT and GPT architecturally and explain why causal masking suits generation while bidirectional masking suits understanding.
#81 Trace a single token's journey through a Transformer, layer by layer, until its prediction.
#82 Explain the fundamental difference between a vision-language model and a plain LLM.
#83 Walk through RLHF end to end: reward model, policy optimization, and the KL penalty. ★
#84 Explain DPO, why it displaced PPO-based RLHF at many labs, and when online RL is still preferable. ★
#85 Explain GRPO and why dropping the value network matters at scale. ★
#86 Derive the LoRA decomposition mathematically, explain why it works, and describe how you pick the rank r. ★
#87 Explain how QLoRA achieves its memory reduction and what quantization trade-offs it involves.
#88 Compare LoRA, prefix tuning, prompt tuning, and full fine-tuning, and state when each is the right choice. ★
#89 Define catastrophic forgetting and describe how to mitigate it during fine-tuning. ★
#90 Give a decision framework for choosing between prompting, RAG, and fine-tuning, with cost and latency attached. ★
#91 Do the GPU memory math for full fine-tuning a 7B model in bf16 with Adam, then redo it with LoRA. ★
#92 Explain RLVR and where it beats a learned reward model. ★
#93 Explain reward hacking in RLHF and how labs address it.
#94 Describe knowledge distillation and how you would build a strong small model from a large one.
#95 R1-Zero was trained with RL and essentially no SFT — what did that demonstrate, and why did full R1 bring SFT back?
#96 Explain what actually breaks in FP8 training at 671B scale and how you make it stable.
#97 Walk through a post-training recipe that turns a pretrained base model into a personalized assistant.
#98 Compare SFT, RLHF, DPO, and RLVR for improving an instruction-tuned model: what data each needs and when to pick which.
#99 After fine-tuning for a customer, target accuracy rose but the model got worse at everything else — explain what happened and how you respond.
#100 Explain Constitutional AI, what it buys over vanilla RLHF, and what it does not solve.
#101 Fine-tune an 8B model on a single 24GB GPU: walk through the memory math and the exact stack you would use.
#102 Describe a time you tuned or fine-tuned a model and what the hardest part was.
#103 Define the bias-variance trade-off and discuss how the two relate.
#104 State the assumptions of linear regression.
#105 Distinguish regularization from validation and explain when each is the right tool.
#106 Derive the gradient of cross-entropy loss with softmax inputs and explain why the two are fused numerically.
#107 Explain the SVD and give two places it appears in modern deep learning.
#108 On average, how many fair coin flips until two heads in a row? Walk through the reasoning.
#109 Decide when Q-learning beats policy gradients and vice versa.
#110 With only limited access to feature weights, how would you explain a loan rejection from a binary classifier?
#111 Give the closed-form solution of linear regression and explain when you would use gradient descent instead.
#112 Contrast L1 and L2 regularization in logistic regression.
#113 Write the logistic regression loss and prove it has a global minimum.
#114 Differentiate KL-divergence loss from cross-entropy loss and from contrastive loss.
#115 Contrast bagging with boosting and explain the computational difference between XGBoost and Random Forest. ★
#116 Explain the bias-variance trade-off, cross-validation, and the curse of dimensionality together.
#117 Describe how GRU cells work, how they address vanishing gradients, and how a BiLSTM works.
#118 Explain when you would prioritize precision, recall, or F1 over the others.
#119 Describe your approach to data imbalance, collinearity, feature selection, and regularization.
#120 Explain how you would design and evaluate an A/B test, and interpret a p-value in that setting.
#121 Define maximum likelihood estimation and explain how it differs from Bayesian inference.
#122 Compute a KL divergence given different random variables.
#123 If a classifier's accuracy is 1, what are the lower and upper bounds on the loss for a single training example?
#124 Given two models at 85% and 82% accuracy, explain which you would pick and why.
#125 Describe how you handle missing data in Pandas.
#126 Explain strategies for combating overfitting in tree-based classification models.
#127 Summarize how the Adam optimizer differs from other methods and its benefits for image classification networks.
#128 A network confuses pugs with pit bulls and some training labels are wrong — how do you change the model and the data?
#129 Explain how to evaluate a clustering model when no pre-labelled groups exist.
#130 Explain the architecture of an LSTM network and the mathematical equations behind it.
#131 Explain precision and recall, and how to avoid underfitting and overfitting.
#132 Define regularization and dropout and explain what each does.
#133 Explain gradient descent and describe k-fold validation.
#134 Explain why gradient descent is needed instead of directly computing the minimum of the loss surface.
#135 Distinguish L2 regularization from weight decay in neural networks, and state when they can be equivalent.
#136 Explain how an L1 or L2 regularizer changes the minimizer of a regression problem, and whether ridge can beat plain linear regression on training loss.
#137 State whether weight sharing raises a model's bias or its variance, and justify the answer.
#138 Explain why a variational autoencoder needs its "variation" and what is lost without it, relating this to the NLU-to-NLG shift.
#139 Given a friend training from scratch on 100 labeled examples versus your transfer-learning approach from 1M examples, name a problem they will hit and how your approach fixes it.
#140 Contrast modeling P(y|x) with modeling P(y,x): advantages of each and which algorithms are typically used.
#141 Contrast MLE and MAP for linear regression, and state what each of linear and logistic regression assumes about P(y|x).
#142 State Bayes' rule and explain the posterior, likelihood, and prior.
#143 Manually compute entropy, Gini index, and information gain on a hypothetical dataset to find the best split. ★
#144 Contrast batch normalization with layer normalization. ★
#145 Explain contrastive learning.
#146 Explain how you would auto-label a fleet dataset and what quality controls you would apply. ★
#147 Detect and handle distribution shift between fleet data and the training set. ★
#148 Handle extreme class imbalance in rare-event detection (e.g., a child running into the road). ★
#149 Design the neural network architecture for multi-camera 3D object detection without lidar. ★
#150 Identify the well-known failure mode of behavior cloning on teleoperation data and how you address it on a real humanoid. ★
#151 Explain what transfers and what does not when a whole-body controller trained purely in simulation runs on real hardware, and how to close the gap. ★
#152 Describe where reinforcement learning fits on top of imitation learning for manipulation and why the reward is the hard part. ★
#153 Argue the case for modular perception-prediction-planning versus end-to-end learned driving, then say what you would actually build. ★
#154 Explain where vision-language and foundation models genuinely help in an autonomy stack and where they are a liability. ★
#155 Argue both sides of cascaded ASR+LLM+TTS versus native speech-to-speech. ★
#156 Explain how to evaluate ASR quality beyond WER and TTS quality when no single correct output exists. ★
#157 Explain how to build ASR that handles code-mixed speech such as Hinglish, given Whisper's failures there. ★
#158 Explain the hard parts of text normalization and prosody for code-mixed, mixed-script TTS. ★
#159 Explain how vision-language models get images into an LLM — projector, cross-attention, or native tokens — and what changes for video. ★
#160 Explain what a vision-language-action model is and how it differs from an LLM with tools. ★
#161 Explain action chunking and why predicting a sequence of future actions beats predicting only the next one. ★
#162 Explain how to fuse camera, radar, and IMU into one perception stack and where the fusion should happen. ★
#163 Justify carrying lidar, radar, and cameras rather than cameras alone, and say where you would fuse them. ★
#164 Explain how scaling laws should influence safety evaluation of large models. ★
#397 How would you explain the difference between an AI agent and a fixed LLM workflow, and when is the agentic approach the wrong choice?
#398 Walk through the ReAct loop: how does the model decide what to do next, and what stops it from looping forever?
#399 Your agent keeps calling the wrong tool with malformed arguments — do you fix the prompt or the tools first, and why?
#400 Explain the 'lethal trifecta' for AI agents and how you would break it.
#401 Why can't a system prompt reliably defend against prompt injection, even in principle?
#402 How would you evaluate an AI agent — what do trajectory evals catch that outcome-only evals miss?
#403 Why does AlpacaEval 2.0 report length-controlled win rate instead of raw win rate?
#404 What is a diffusion model, and how does its training objective differ from a GAN's?
#405 Walk through the forward and reverse processes of a DDPM — what exactly is the network trained to predict?
#406 Compare GPTQ and AWQ: what does each method exploit to preserve quality at 4-bit precision?
#407 Explain speculative decoding — why does it speed up generation without changing the output distribution?
#408 What is self-consistency decoding, and why does it improve over plain chain-of-thought?
#409 What does it mean for a chain-of-thought trace to be unfaithful, and why does that matter for safety monitoring?
#410 Explain HyDE (hypothetical document embeddings): when does it help retrieval, and when does it backfire?
#411 Why does weight quantization speed up token decoding far more than prompt prefill?
#412 Explain selective state-space models (Mamba) and how they achieve linear-time sequence modeling.
#413 How would you calibrate a classifier's predicted probabilities, and how do you measure calibration quality?
#414 Explain MCP — what problem does it solve, and when would you not use it?
#415 For an agent SLA, why is pass^k a better reliability metric than pass@k?
#416 What is test-time compute scaling, and when does spending more inference compute beat using a bigger model?