Together AI MLE Interview Questions (2026)

Questions reported from Together AI machine learning engineer interviews. This bank tracks 14 questions asked at Together AI (12 with free model answers). Practice them live with DeepOffer's AI voice interviewer.

← All 465 questions

Practice this question with an AI interviewer

Get asked follow-ups live, then receive a scored report — like a real MLE interview loop.

Start AI mock interview

#43 Describe the KV cache, derive its memory formula, and discuss what it costs at production scale. ★
#46 Walk through FlashAttention and account for its speedup given that it performs the same FLOPs.
#165 Contrast the prefill and decode phases and explain why prefill is compute-bound while decode is memory-bandwidth-bound. ★
#166 Explain continuous (in-flight) batching and why it replaced static batching. ★
#167 Explain PagedAttention and the KV-cache fragmentation problem it solves. ★
#168 Explain speculative decoding: why output quality is preserved and when it does not help. ★
#170 Compare FP16, BF16, FP8, INT8, INT4, and FP4 for serving and name what breaks at each step down. ★
#174 Do the roofline math: how many tokens/sec can one H100 produce for a 70B model at batch size 1? ★
#175 Decide when to choose vLLM, SGLang, TensorRT-LLM, or a custom stack. ★
#184 Design the scheduler for a continuous-batching inference engine. ★
#185 Explain when speculative decoding helps, when it hurts, and why you would adapt the speculator to live traffic. ★
#186 Price a dedicated 70B endpoint: estimate cost per million output tokens and explain the throughput-latency trade. ★
#187 Design a serverless inference platform serving 100+ open models on a shared GPU fleet. ★
#208 A customer's distributed training job gets 55% scaling efficiency at 64 nodes — debug it.