Generic AI will not run your CRO program. Give it no real knowledge of your customers, your data quality or your commercial limits, and it falls back on standard best practices and conclusions that arrive too fast. As a first pass that can be useful. It is not research, and it does not replace understanding the people who use your site or funnel.
I use AI in my work. I do not hand it the program. I started in design and UX, and I have been in digital since 2002, with UX and art direction work at Albelli, Makro, Nuon/Vattenfall and fonQ. Today I focus on conversion optimization for e-commerce and lead-generation teams. I find where journeys underperform and help decide what is worth testing or changing. The pattern I keep seeing is simple. Companies ask a general-purpose model, or an automated audit tool, to do the thinking, and then treat the output as if it knew the audience.
Why does generic AI fall back on best practices?
A general-purpose model learns from a wide mix of public writing: blog posts, playbooks and checklists, plus the same conversion heuristics that have circulated for years. Give it a URL or a screenshot, or just a thin brief, and it produces the sentences it has seen most often. Add social proof. Shorten the form. Make the call to action clearer. Reduce friction in checkout.
None of that is automatically wrong, but all of it is generic. The model has no durable knowledge of your customers, meaning why they hesitate, what they compare you with, which segments make margin, and which best practice would hurt one specific journey.
Google says something related about its own limits. Generative models do not look facts up the way a database does. They predict a likely sequence of words, so the output can contain inaccuracies, often called hallucinations. That is why Google tells site owners to fact-check AI-generated content before publishing it (see its guidance on using generative AI content). The same applies when the output is an audit, a list of hypotheses or a priority score instead of a blog post.
Without unique input (first-hand research, reliable analytics, session evidence, commercial context), the model has nothing better to lean on than the average advice on the public web.
What is the black-box risk in a CRO context?
The danger is a confident answer you cannot trace.
In a CRO audit that shows up in familiar ways. A model can pick the wrong conversion definition, or call a form submission a success when sales quality is what matters. It can ignore sample size and seasonality, or miss that tracking changed. And it can find a relationship between a page element and conversion and report it as a cause.
Search Engine Land describes this well in its piece on using Claude for audits. A model can produce a report that sounds credible and still get the conversion definition, the reporting period, the sample size or the page behavior wrong. It can find a relationship in a spreadsheet and a screenshot. It cannot prove that one caused the other.
CXL makes a similar point about scores. In a 2026 workshop on AI for CRO research, CRO consultant Ruben de Boer and the CXL team noted that an AI-generated score like 42% or 53% looks precise but is a subjective judgment, not a measured conversion effect. It also cannot know whether a familiar persuasion pattern matters to your visitors. The full write-up is AI for CRO research: what to automate (and what not to).
Then there is a failure mode that is easy to miss. Models often land on a best-practice answer first and go looking for evidence afterwards. CXL calls this “research as decoration.” The order should be evidence, then insight, then hypothesis, then test.
The black box also becomes a statistical problem. A paper on AI-assisted A/B experimentation at scale (Daley, Gorbachev and colleagues) describes a tension that any team testing on a live site should care about. Language models are nondeterministic and can hallucinate, which clashes with the trust an experimentation system has to provide. The authors also found that an agent’s eagerness to help can threaten statistical validity. It becomes almost costless to hunt for a significant window, add a post-hoc exclusion or re-cut the data until a test “wins.” Their response was not to pretend the model is deterministic. They limited where the model may choose a procedure and let deterministic code compute every number the system reports.
If you cannot see the definition, the data, the method and the limits, you do not have a finding. You have a fluent suggestion.
Why do in-house teams end up outsourcing their thinking to AI?
Corporate teams often hire specialists in-house and still end up with generic output. I put that down to pressure more than laziness.
The backlog is long. Stakeholders want a recommendation this week. A model always has an answer ready. A careful program will sometimes say that we do not know yet, that the data is messy, or that an idea should wait until the audience is better understood. In a meeting, the ready answer usually wins.
Over time the team stops doing the slow work that changes the quality of decisions. Watching sessions. Talking to customers. Checking that the GA4 event is the event the business cares about. Writing a hypothesis that can be wrong. The model becomes the first and last opinion in the room, and the thinking has been outsourced even though the in-house hire is still there.
That costs more than any tool invoice shows. You get plenty of activity (audits, decks, test ideas) and very little learning, plus a false sense of coverage. A complete-looking report can hide the fact that nobody has spent serious time with the actual customer journey.
For teams in exactly that spot I offer CRO coaching for in-house teams, where your specialists keep the program and I check their AI-assisted ideas against real evidence before anything reaches the roadmap.
If this sounds familiar, an automated pass on a public homepage is still a fair first look. I built a free Growth Scan for that. It checks conversion basics, trust, speed, accessibility and how ready a homepage is for AI search. Treat it as a starting point to review, not a finished strategy.
What does a proper experimentation program include?
A program is not a pile of AI-generated test ideas, and it is not a one-off audit that gets filed and forgotten. For an e-commerce or lead-generation site it needs a few unglamorous things in place.
Start with one primary metric and a guardrail. In a shop that might be revenue per visitor or conversion rate, with refunds, margin or average order value watched so a “win” does not quietly destroy value. For lead generation a form submit is rarely enough. You need a view of quality further down the funnel.
Evidence comes before opinions. Quantitative data shows where people drop out. Behavioral evidence and customer feedback, plus a structured walkthrough of the site, help explain why. I describe that mix on my research and CRO audits page. You do not need every possible data source. You need to understand one audience well enough to decide better than a generic checklist would.
Hypotheses have to be able to fail.
“Add trust badges” is not a hypothesis. “Returning mobile visitors abandon checkout after seeing shipping costs, and showing those costs earlier and more clearly will increase completed orders without lowering average order value” is one. You can test it. It can lose.
The backlog needs priorities a human can defend. Impact, confidence and effort are useful, and a model can help organize the list. A person still has to challenge weak evidence, drop ideas that exist only because they are fashionable, and keep the work tied to the business problem.
Agree the rules before you look at the numbers: primary metric, guardrails, sample size or duration, and what you will do with a win, a loss or an inconclusive result. Write them down. That is how you avoid the fishing expeditions the paper above warns about.
Last, keep learning instead of just shipping winners. The value of a program is the compounding knowledge about your audience. A one-off audit can start that. The audit should not be the whole relationship.
If you want a senior, human-led review of a journey, with the evidence behind it and a roadmap you can act on, I do that as an expert CRO audit. This is a defined piece of work and does not replace a longer program. It is not a template dressed up as one.
Where does AI help?
Teams should use AI. The question is where in the process.
It works when the task is wide, repetitive or structural and a person reviews the result. It can help you:
- Summarize a long analytics export or a set of research notes.
- Compare segments or pages once you have defined the metric and the period.
- Draft a heuristic review of a journey, as a list of possible issues, not as measured impact.
- Group findings from different research methods into themes, as long as the original sources stay attached.
- Turn a validated theme into a first draft of a hypothesis or test brief, or of the report.
CXL’s workshop reached a similar practical conclusion. Start with your process, not the model. Use AI to speed up parts of research such as heuristic review and thematic grouping. Put a human review at every handoff, so a weak assumption in step one does not become a whole backlog of bad tests. Search Engine Land’s audit guidance points the same way: give the model narrow questions, reliable evidence and clear rules for uncertainty, and keep diagnosis, prioritization and the final decision with a strategist.
AI is a poor replacement for the judgment calls: deciding what “conversion” means for the business, judging whether the data can be trusted, and understanding an audience’s motives and constraints. It also cannot rank work under real commercial and technical limits, confirm that a change caused a result, or sit with customers and watch them struggle with the site, which is sometimes exactly what the question needs.
The better your process and your inputs, the more useful AI gets. With a thin process you only automate thin work, faster.
A short checklist before you accept an AI CRO recommendation
| Check | If you cannot answer this, do not treat the output as a decision |
|---|---|
| Conversion definition | Is the model optimizing what the business values, such as quality, revenue or margin, and not only a convenient on-site event? |
| Data quality | Have you confirmed that tracking, time range and sample size are sound, and checked for recent site or tagging changes? |
| Audience evidence | Is there evidence about your visitors, such as analytics, behavior or feedback, or only generic UX heuristics? |
| Alternative explanations | Have you ruled out traffic mix, offer and seasonality, as well as bugs and measurement issues, before blaming a page element? |
| Causation | Is the next step a test or an investigation, rather than a permanent change justified by a correlation? |
| Pre-committed rules | If you are reading a test, were the metric, guardrails and stopping rules agreed before anyone saw results? |
| Human review | Has someone who knows the journey and the business signed off, or has the team only accepted a fluent report? |
Frequently asked questions
Can we use ChatGPT or Claude to run a CRO audit?
You can use them to organize evidence, draft a page review and produce a first list of questions. The draft is not the audit. As Search Engine Land notes, a model can write a convincing report and still be wrong about the conversion definition, the data or the page. A person still has to validate the tracking, inspect the experience and decide what is worth testing.
Should we hire a CRO specialist or just use AI?
Use AI for speed. Hire or embed a specialist when you need judgment and audience understanding, plus a program that lasts longer than a report. For e-commerce and lead-generation sites, acting on a fluent but shallow recommendation usually costs more than doing the thinking properly. Many in-house teams already have talented people. What they often lack is time, a clear process and someone who will challenge the first answer the model gives.
Is an AI “score” a measure of conversion impact?
No. A score from a heuristic review or an automated checker is a structured opinion. A score can help you notice possible problems. It is not a measured effect on conversion, revenue or lead quality. CXL has warned about the false precision these numbers create. Check the important items against real behavior and, where you have the traffic, against a test.
How should we use an automated scan or an AI audit tool?
Use it as a first pass on a public page or site, then investigate. My Growth Scan is built for that. If you need a deeper, senior reading of the experience and a roadmap, use a human-led expert audit. Do not jump from a generated list of issues to a redesign.
Where should AI stop inside an experimentation program?
It should stop before the numbers you report and before the decision you ship. Going by the Meta paper on AI-assisted experimentation, the model can help you retrieve, draft and diagnose, but the statistics should be computed in a controlled way and the launch decision should stay with the team. It also has to stop short of inventing customer evidence. AI can summarize interviews you actually ran. It should not replace them.
What is the first step if our team already relies on AI for CRO?
Slow the last step down. Keep using AI to gather and structure material, but add a review gate. The gate should cover the definition of success, data quality and audience evidence, and end with a human decision about what happens next. For a second pair of eyes on the site itself, start with the Growth Scan or an expert audit. If you need ongoing help inside the team, I work with companies on a freelance, interim or fractional basis. You can see how on Work with Pim.
Keep the program human
I am not against AI. I do not think companies should resist it for its own sake. I want teams to lift the metrics that matter for longer than a single audit cycle. That takes someone who still does the deep work: understanding the audience, checking the evidence and running a program that can learn.
Generic AI will help you move faster. It will not tell you what is true about your customers. That part is still the work.



