A document assistant takes 18 seconds to prepare a reply. The team wants to replace the model. But if most of that time is spent waiting for a document search, changing the model may leave the problem untouched.

Start by measuring where the time goes. The numbers below are illustrative, not benchmarks.

Record the timeline for one request

Give the request an identifier and record when it entered the queue, when retrieval started and ended, when the model began responding, and when any tool calls completed. Also record when the user saw the first useful result and when the whole task finished.

Suppose a request spends 2 seconds in a queue, 9 retrieving documents, 5 generating a draft, and 2 saving it. Retrieval is the first place to investigate. A faster draft generator could still help, but it cannot recover time spent elsewhere.

Do not add durations blindly when steps overlap. Measure the overall elapsed time as well as each stage.

Compare like with like

Use a small set of representative tasks: a short question, a longer document request, and a request that needs an external tool. Keep track of document size, retries, and whether a result was already cached. A fast repeated question does not tell you what happens on a first visit.

Review both typical and slow requests. Averages can hide the handful of waits users remember most. Store timing and safe identifiers rather than copying prompts or customer documents into performance logs.

Fix the bottleneck without lowering the quality bar

  • Queue: check whether work waits behind unrelated jobs.
  • Retrieval: check whether the workflow asks for more sources than the answer needs.
  • Generation: test a shorter input or a faster model against the same quality examples.
  • Tool calls: avoid repeating the same lookup; only parallelize steps that do not depend on each other.
  • Waiting experience: show progress for real stages, and say when a result is ready for review.

Do not remove evidence, validation, or approval simply to make a demonstration feel faster. After each change, rerun the same cases and compare timing and correctness together.

See our AI quality testing guide for the other half of that comparison, or contact us to discuss a workflow review.