A familiar prompt can produce a noticeably different answer after a model update. That does not automatically mean the new version is better or worse overall; it means the comparison needs a consistent starting point.
Keep the comparison fair
Use the same representative prompts, inputs, and success criteria with both versions. For a team using a model to summarize meeting notes, that might mean checking whether key decisions and owners are preserved, not simply whether the summary reads smoothly. Record the settings too, since a changed system prompt or tool configuration can affect the result.
Look beyond the best answer
A few polished examples do not show how consistently a model handles ordinary requests, edge cases, or incomplete information. Review failures as well as successes, and note whether the update changes response time, output format, or tool use in ways that affect the workflow. A small, repeatable test set will not answer every question, but it can expose changes that a showcase misses.
Make the update useful
When reporting a model change, say what was compared, what improved in your test, and what remains uncertain. Keep the baseline so the next update can be judged against the same tasks. That turns a new-version announcement into a signal readers and practitioners can actually use.