Moving from fast expert decisions to taxonomy-informed specialist judges and selective human review.

TLDR: AI video evaluation should operate as a structured test suite—combining deterministic code checks and taxonomy-informed specialist LLM judges—rather than relying on a single, vague "quality" critic. In this workflow, automated fail-fast gates filter out unusable candidates early, reserving expert human review to audit edge cases, define rubrics, and approve high-value survivors.

Generating a plausible video is no longer the hard part. Deciding whether it is the right video still is.

A candidate can look polished and remain unusable. It may have the wrong aspect ratio for its placement. The colour scheme may drift. The wrong type of hero may appear. The product can look convincing while behaving in a way that the real product never would. Those failures do not belong in one vague category called "quality". Some are properties that code can check exactly. Others depend on product knowledge, creative intent or visual interpretation. They have different costs and need different evaluators.

The production question is simple: should this candidate continue through the workflow, or should we stop spending time and compute on it? Our answer has evolved from a low-friction human decision into a set of explicit tests backed by specialised human judgement.

Start with the decision you already make

Our first evaluation interface worked like Tinder. A subject-matter expert watched a candidate and swiped one way to approve it or the other to reject it.

That was deliberately simple. The label corresponded to a real production decision, and experts could provide it quickly. We did not begin by asking reviewers to fill out a long annotation form for every video or invent a scoring system before we understood the workflow.

The binary label had an obvious limit. It told us that an expert rejected a video, but not why.

Two candidates could receive the same verdict for unrelated reasons. One might violate a hard delivery constraint. Another might depict the wrong hero. A third might satisfy every written rule and still feel wrong to an experienced reviewer. The swipe captured the outcome of the judgement while hiding its structure.

That is useful at the beginning. It is not enough for an evaluator that needs to diagnose a failure or improve the next candidate.

Our next step is to collect a textual rejection reason alongside the decision. This is planned work rather than a finished learning loop. The reason matters because it can connect a failed asset to the part of the evaluation system that needs attention.

Expert judgement needs a specification

The expert role is broader than labelling outputs. Subject-matter experts define, at least, four inputs to the evaluation system: taxonomy, rules, rubrics and examples.

• A taxonomy names the concepts that matter. It might distinguish types of heroes, properties of a scene or classes of generation failure.
• Rules state what is required or prohibited. They can express a delivery constraint, a visual requirement or something that must appear in the creative.
• Rubrics tell an evaluator how to apply a criterion. A label such as "brand consistency" is too broad on its own. The rubric needs to explain what evidence counts, which deviations are acceptable and what should cause a failure.
• Examples ground the specification in real assets. They show the difference between a clear pass, a clear failure and the cases where even qualified reviewers may disagree.

These are different artefacts. Mixing them into one large prompt makes changes difficult to interpret.

If a judge improves after an update, we should know whether the change came from a clearer rule, a better rubric, a more representative example or a different model.

The subject-matter experts remain the source of the specification. An LLM can apply that specification at scale, but it does not decide which creative standard the business should care about.

Replace the general critic with specialist tests

Our later evaluation approach uses multiple LLM judges. They see the complete video or image together with the relevant taxonomy and rules. Different judges focus on different tasks, and each test returns pass or fail.

Reported examples include aspect ratio, hero type and colour scheme. They also illustrate why the mechanism should follow the criterion.

Aspect ratio normally does not need an LLM. If the source dimensions are available, a deterministic check is cheaper, faster and easier to trust. Hero type or product fidelity may require visual and semantic interpretation, which makes them better candidates for a multimodal judge.

The useful abstraction is not "send the video to several models". It is a test suite in which each test has:

• a named criterion;
• a defined input;
• a versioned rule or rubric;
• an expected pass/fail output;
• evidence supporting the result;
• a known action when the test fails.

A collection of specialist judges is not automatically more accurate than one general judge. Several models can share the same blind spot. They can also produce consistent answers for the wrong reason. Multiple outputs become useful when they isolate failure modes and can be checked against qualified reviewers.

Fail early, but keep the evidence

When a candidate fails a test in the current workflow, it is likely to be dropped immediately. This is a sensible production policy when the failure is blocking. There is little value in sending an invalid candidate through more generation stages or asking an expert to review a video that already violates a known requirement.

Fail-fast evaluation changes the economics of the workflow. The system spends expert attention on the candidates most likely to be usable rather than treating every generation equally.

It also creates a risk. If a semantic judge rejects a good candidate incorrectly, the candidate disappears before the final reviewer sees it.

Reviewing only the survivors tells us about false acceptances: bad candidates that slipped through. It tells us little about false rejections. To find those, experts sometimes inspect candidates outside the selected set to check that the automated evaluation remains aligned.

That sampling should be a deliberate part of evaluator validation. The rejected pool is not waste. It contains evidence about judge errors, missing rubric detail and edge cases that the taxonomy did not represent well. For the production path, a failed candidate can stop. For the learning path, we should preserve the failed test, its evidence, the evaluator configuration and any later expert correction.

We plan to use failed candidates and textual rejection reasons to generate targeted improvements. The mechanism matters. A correction might change a rule, rubric, example, threshold, judge configuration or generation step. Merely storing the comment does not establish that the system learned from it. The change has to improve representative held-out work without introducing a new regression.

Humans define the standard and audit the system

Subject-matter experts appear at several points in this workflow.

They define the taxonomy, rules, rubrics and examples. Their initial swipe decisions provide a direct acceptance signal. They review the best candidates that survive the automated tests. They also inspect some other candidates to detect drift and false rejection.

This is a more useful human-in-the-loop design than asking experts to approve every intermediate artefact. Automation handles repetitive checks and narrows the candidate set. Experts work where their judgement changes the decision or provides evidence that the evaluator itself needs correction.

The allocation should change with the reliability and cost of each test. A deterministic dimension check can usually run without escalation. A semantic test with poor agreement on a new class of video may need frequent expert review. A high-cost false acceptance may justify mandatory review even when the judge performs well on average.

Human review is therefore part of the measurement system, not a fallback added after automation fails.

Validate each test against the decision it controls

An evaluator should be measured at the level where it acts. If one failed test can remove a candidate, an aggregate agreement score is not enough. We need to know, for each criterion:

• how often the evaluator agrees with qualified reviewers;
• how often it passes a blocking failure;
• how often it rejects an acceptable candidate;
• which briefs, visual styles or generation sources cause errors;
• whether reviewers themselves agree on the criterion;
• how results change when the model, rubric or examples change;
• what the test costs and how long it takes.

The error costs are asymmetric. A missed format violation wastes downstream effort and may produce an unusable deliverable. A false failure on a promising creative removes a potentially strong candidate from consideration. Different tests may therefore need different thresholds, audit rates and deployment policies. A representative evaluation set should include ordinary passes, ordinary failures and difficult boundary cases. It should also contain the types of candidates the system will see after the next model or workflow change. Testing only the examples used to write the rubric measures familiarity, not generalisation. We have not included performance numbers in this article. Multiple judges do not establish correctness by themselves, and a credible result needs a defined human protocol, representative cases and per-test error analysis. Those measurements are part of the engineering work, not a decorative benchmark to add after deployment.

Passing pre-launch tests is not campaign performance

A video can satisfy every pre-launch requirement and still perform poorly with an audience. Another can violate a creative convention and perform unexpectedly well.

Campaign outcomes depend on targeting, placement, spend, audience and other factors outside the asset. Taxonomy compliance, expert acceptance and observed campaign performance are separate claims.

This evaluation system addresses the first two. It checks whether a candidate satisfies known requirements and whether its assessment remains aligned with subject-matter experts. Any claim that the same tests predict click-through rate, conversion or return on ad spend requires a separate validation design.

Keeping those claims separate makes the pre-launch evaluator more useful. We can improve it against a clear target instead of asking one score to stand in for every meaning of "good".

Evaluation is production infrastructure

Generating more candidates only helps when the workflow can reject unsuitable ones without multiplying review work.

The path we are taking starts with the expert decision, turns expert knowledge into explicit tests, uses deterministic or model-based evaluation according to the criterion, and sends the strongest candidates to final human review. Sampled failures check whether the gates remain aligned.

The next step is richer rejection evidence. Textual reasons can make failures useful beyond the current candidate, provided they lead to scoped changes that survive regression testing.

A video judge should not be treated as an oracle or a general-purpose critic. It is one component in a versioned test suite whose behaviour can be inspected, challenged and improved. That is what makes automated evaluation usable in a creative production workflow.