How to test AI culling accuracy on your own shoot

Every culling tool publishes an accuracy number, and not one of them can tell you whether it would have picked your gallery. The only test that answers that question uses a shoot you already delivered as the answer key. It takes about half an hour of your attention, it costs nothing, and here is exactly how to run it.

A published accuracy number is not about your work

Accuracy sounds like a property of the software. It is not. It is a score against an answer key, and the answer key belongs to whoever ran the test. When a vendor says its model is accurate, what it means is that the model agreed with a set of judgments made by people you have never met, on photos you did not shoot, for clients you do not have.

Your question is different and much narrower. You want to know whether this tool would have found the frames you delivered last spring. Nobody can answer that but you.

Be especially careful with cross-brand comparisons. FilterPixel publishes accuracy figures for its own product alongside figures for its rivals. That is a company testing itself and grading its competition, published as marketing. It may well be sincere. It is still not evidence, and neither is the equivalent chart from anyone else.

The wider argument about whether these models can be trusted with creative judgment at all is a separate piece, and we wrote it in can you trust AI to pick your photos. This article assumes you have stopped reading opinions and want a number of your own.

Why a delivered gallery is the perfect test material

A shoot you have already delivered is a labeled data set. You went through every frame at full attention, you made a decision on each one, and a client paid you for the result. That is a better answer key than anything a vendor could build, because it encodes your taste, your standard, and the things you know about that family.

Pick your test shoot with a little care:

• Delivered within the last year. Older than that and you are testing the tool against a version of your taste you have already moved on from.
• Not one you remember frame by frame. If you can still picture the gallery, you will unconsciously grade the tool generously.
• You still have the complete original folder. This is the one that stops most people. If your archive holds only the delivered selects, there is nothing to test against, because the tool has to be given the same 2,000 to 4,000 frames you started with. If you archive the whole card, you can run this test on any job in your history.
• An ordinary job, not your best one. A shoot that went perfectly, in good light, with an easy couple, will flatter every tool you point at it.

If you have never written down what you are actually selecting for, do that before you test anything, because you cannot grade a tool against a standard you have not stated. That exercise is its own job and we walked through it in building your culling criteria.

What you need before you start the clock

Four things: the complete original folder, a list of the filenames you delivered, a trial account, and a pass mark you write down before you look at anything.

The trial accounts are the easy part. Every serious tool in this category will let you test it properly for free, which is the whole reason this test is worth running rather than arguing about.

Trial terms checked August 2026 from each vendor's own page: Aftershoot, Narrative, Imagen, FilterPixel.

Note the third column, because it changes how you plan the half hour. A local tool starts working the moment you point it at a folder. A cloud tool has to receive the photos first, and on an ordinary home connection a full wedding is not a coffee break. That is a real part of the cost of using it, so let it show up in your test rather than starting your timer after the upload finishes.

The half hour test, step by step

The thirty minutes is your attention, not the computer's. The analysis run happens without you.

1. Copy the folder. Work on a duplicate of the original shoot. Not because these tools are dangerous, but because a test you are nervous about is a test you will cut short.
2. Build your answer key. Get the filenames of the images you actually delivered into a list. Your gallery host will export a file list, or you can pull the names straight from your delivered folder. Put them in column A of a spreadsheet.
3. Run the tool on everything. The whole shoot, every frame, at default settings. Do not hand it a curated subset, and do not skip the getting-ready coverage because it is boring. Start it and walk away.
4. Export what it picked. Every one of these tools can give you its selection as a list of filenames, or as flags you can filter in Lightroom and then export. Put that list in column B.
5. Match the two lists. In column C, next to each delivered filename, use =IF(COUNTIF(B:B,A2)>0,"picked","missed") and fill it down. That gives you your misses in one pass instead of an hour of squinting.
6. Look at every miss with your own eyes. This is the step that matters and it is the step people skip. The count tells you almost nothing. The frames tell you everything.

Run it a second time with the tool's sensitivity turned up, if it has that control. Most do, and the default is a setting somebody chose for an average photographer who is not you. A tool that fails at default and passes when you loosen it is a tool that works, with a note in the manual.

The scoring sheet, blank

Four boxes. Fill them in from your spreadsheet, for your shoot.

One division gives you the only headline figure worth having. Take your agreed keepers, divide by agreed keepers plus misses, and you have the share of your real gallery the tool found on its own. Call it keeper recall. It answers the question you actually asked.

The second number, extras divided by everything the tool picked, tells you how much padding you have to read past. It is worth writing down but it is worth far less worry, and that asymmetry is the most important thing on this page.

The two errors do not cost the same. An extra costs you about a second: you glance at a frame, you do not pick it, you move on. A miss costs you a photograph, and if you have trusted the shortlist and never opened the rest, it costs you that photograph permanently, in a gallery a client keeps forever. A tool that hands you a slightly bloated shortlist is doing its job. A tool that quietly loses the shot of the groom's father is not, however good its headline percentage looks.

How to read your own numbers

There is no industry pass mark, and anyone who quotes you one is quoting their own marketing. That is exactly why you wrote yours down in advance. Compare your result to that, not to a number from a chart.

Then put the percentage aside and go through the missed frames one at a time, asking three questions:

• Would I fight for this frame? A gallery has a handful of photographs that make the job, and a long tail of good ones that could be swapped for their neighbors without anyone noticing. Losing one from the tail is a shrug. Losing one from the handful is disqualifying.
• Is there a pattern? Twenty misses scattered at random is a different problem from twenty misses that are all backlit, all in the same dark reception, or all of the same guest. Patterns are predictable, and predictable is manageable.
• Why did it fail? Usually the answer is that the frame is technically weaker than the one beside it and carries meaning the model cannot see: a face nobody told it mattered, a hand on a shoulder, a moment that reads only if you were in the room.

Now look at the other side. Go through the extras, the frames it picked that you did not deliver. Some will be duplicates you had already chosen between, and you can ignore those. But a few will be genuinely good frames you walked past at midnight with three hundred to go, and finding one of those is worth more than any percentage on your sheet. It is not a defect in the tool. It is a defect in tired human attention, which is the actual thing you are trying to buy your way out of.

One more read of the sheet: your extras pile is also a report on your own consistency. If the tool keeps offering you frames you cannot articulate a reason for skipping, your standard may be looser than you think.

Five ways this test quietly lies to you

The method is simple. Fooling yourself with it is also simple.

• You test one shoot and decide. One shoot is one data point. Run it on two that are unalike, a bright outdoor family session and a dim reception, and you will learn more from the gap between the two results than from either one.
• You test your best-lit job. Everything works in good light. Give it the hard one, because the hard one is where the hours actually go.
• You grade against memory. Comparing the picks to what you think you delivered rather than to an exported list will flatter the tool every time. Export the list.
• You only look at the burst groups. Choosing between six near-identical frames is the part these tools are best at, so a test built only from bursts is a test designed to be passed. Judge the whole shoot, not the easy stretch of it. The narrower skill is worth its own attention, and we covered it in choosing the best frame from a burst.
• You measure accuracy and forget speed. They are separate questions and a tool can win one and lose the other badly. The companion exercise, walking one shoot through both routes and watching where the minutes actually go, is in AI versus manual culling on the same shoot.

When you have your sheet filled in for two shoots, you have something almost nobody in this market has: an accuracy figure for your own work, produced by a method you can explain. That is the basis for a decision, and it turns choosing culling software from a matter of taste into a matter of evidence.

FAQ