Building the Scorecard Before Trying to Move the Score

The challenge

The existing engine was proven but not consistent enough to run without supervision. Launching on it would have meant either constant manual intervention or letting customers see inconsistent results. And there was no way to tell whether any given change made it better, because nobody had defined what better meant in testable terms.

Where we came in

The usual instinct is to start tuning prompts and see what feels improved. That produces motion without progress, because with a generative system you cannot tell a real gain from a lucky sample. We insisted on the unglamorous half first: define compliance rules in testable form, curate a validation set, and run a baseline. Only then does week-over-week improvement mean anything.

What we did

• Defined the brand and layout compliance rules in a form a machine can test, not a form a person interprets
• Curated and labelled a validation image set with acceptance criteria defined per image
• Built a scoring pipeline and ran a documented baseline before changing anything
• Added boundary and overlap validation, with automatic corrective regeneration when a guardrail rejects an output
• Introduced feature-flag rollout and model routing so changes ship progressively rather than all at once
• Reported measured progress on a fixed cadence rather than on request

Where it landed

• A repeatable scoring harness that judges each release against the same labelled validation set
• Guardrails catch invalid output and trigger regeneration instead of passing it downstream
• Improvement is reported against a documented baseline, so progress is demonstrable rather than asserted