Intermediate MLOps & Production
9 MIN READ
Life of a Request: Following One Prompt Through an Inference Server
Where the time goes when you send a prompt to a model in production: the network, the queue, tokenization, batching, the two very different GPU phases, and streaming the answer back. Following one request from start to finish.