Tag: ml-infrastructure

When you type a prompt into an image or video generator, the model is only one stage. A map of what surrounds it: input safety checks, prompt rewriting and expansion, generation, output safety checks and watermarking, and the newer agentic pipelines that plan shots and call several models. Built from Google's documentation and open-source releases from Google, Alibaba and Tencent.

Why a large language model can check several guessed tokens for about the cost of generating one, how the acceptance rule keeps the output exactly the same, the main ways to make guesses (draft models, n-grams, Medusa, EAGLE, multi-token prediction), and when it stops helping, including for vision-language and mixture-of-experts models.

Where the time goes when you send a prompt to a model in production: the network, the queue, tokenization, batching, the two very different GPU phases, and streaming the answer back. Following one request from start to finish.