Skip to content

Code Review Is Not Testing, and AI Code Needs Both

AI code can look right in review and still fail when it runs. Real examples, what review catches, what only testing proves, and how to get both.

• Regressionly Team
Title image: Code review is not testing. AI code needs both. Two developers work at laptops at night; a pull request card shows Approved while a test run card shows the login test failed
Table of Contents

TL;DR: Code review tells you whether code is written well. Only testing tells you whether the application works. AI makes code look correct faster than ever, so the gap between “approved” and “working” is easy to miss. In this article we share real examples where reviewed code failed the moment it ran, and what good test evidence looks like. Regressionly closes that gap with human-in-the-loop testing pods that run your regression suite and verify every release, starting with a free two-week pilot.

AI can generate code that looks correct. A developer can review it and approve it. And the application can still fail the first time it runs.

We see this almost every week. At Regressionly, we test software for teams that build with AI. Our human testers manage test cases, run manual and automated tests, track every execution result and dig into every failure. The test management platform we built for that work keeps all of it in one place, so we are not depending on scattered reports and spreadsheets. Most of the examples in this article come from that daily work.

The lesson is always the same. Code review and testing answer different questions. With AI-generated code, you need both answers.

In this article we will cover:

Quick summary

QuestionCode reviewTesting
What does it look at?The codeThe running application
What does it catch well?Duplicate code, unnecessary logic, security, maintainabilityWrong behavior, missing values, broken workflows
What can it miss?How the app behaves in real situationsBadly written code that happens to work
What does it prove?The code is written properlyThe feature actually works
How much do we trust AI-written tests?-Useful, but in our experience about 60 to 70% complete
Who signs off on a release?Developers confirm the codeQA confirms the behavior; AI supports, never decides alone

What code review is good at

Code review is not the problem. It is a valuable practice, and we would never ship without it.

A good review helps a team find duplicate code, unnecessary logic, security concerns and maintenance problems. It checks whether the code is written properly, whether it follows the team’s standards, and whether the next developer will be able to understand it.

That is a lot of value. But notice what all of those have in common. They are about the code itself.

What review cannot see

A reviewer reads the code. They do not watch the application run.

So review cannot always tell you how the application will behave in real situations. Unexpected user actions. Data that is slightly different from what the developer imagined. A change in one feature that quietly affects another feature somewhere else in the application.

Those problems only show up when the code is executed. That is why testing exists.

Comparison: code review checks the code (duplicate code, unnecessary logic, security concerns, maintainability); testing checks the running app (real behavior, missing values, unexpected user actions, side effects on other features). Review asks: is it written properly? Testing asks: does it work?
Review and testing answer different questions. A release needs both answers.

AI makes this gap bigger, not smaller. AI-generated code usually looks clean. It is well formatted, well named and logically structured. That makes it easy to approve. The better the code looks, the easier it is to forget that nobody has seen it run yet.

The login test that looked correct

Here is a recent example from our own automation work.

We wrote automation code to log in to an application, with help from AI. The code was reviewed, and it looked correct. The navigation logic was right. The steps were in the right order. Nothing in the code looked suspicious.

Then we executed it, and the login failed.

To find out why, we checked the logs, the screenshots and the login data stored during the run. That showed the problem. The application URL needs an important parameter. The full URL, including that parameter, was stored correctly in the project’s .env file. But the automation code did not use the full URL when it navigated to the application. The parameter was simply missing, so the login could never succeed.

The review was focused on the implementation and whether the navigation logic was correct. And it was. Nobody verified the actual application behavior during the review, so nobody noticed the missing parameter. It was only found when we performed the steps manually and compared them with the screenshots, logs and stored login data.

Two panels. In code review, the login automation code looks correct and is approved. In the test run, the URL in the .env file includes a required parameter, but the URL the test navigated to is missing it, so login failed. Logs, screenshot and stored login data showed the cause
The code was right on paper. The run showed what it actually did.

This is a small bug. It took one missing value. But it is a perfect example of why the code review was correct and the code still did not work. A reviewer reads what the code says. Execution shows what the code does.

A passing test is not a working feature

The opposite problem is even more dangerous: a test that passes while the feature is broken.

We have seen this happen many times, and it almost always has the same cause. The test checks that an action was completed, not that the expected result happened.

Clicking the Submit button successfully does not mean the data was saved correctly. A test that only checks the click will pass even if the save failed, or if the wrong data was saved. A good test goes further. It verifies that the data was saved, and that it is reflected correctly in the application.

This matters even more with AI. AI is very good at writing tests that run green. It is not always good at choosing the right thing to assert. When you review an AI-written test, the most important question is not “does it pass?” It is “what exactly is it proving?”

Not sure your green tests are proving anything? Regressionly’s testing pods review your existing suite, find the tests that only check clicks, and add the assertions that check real results. Start with a free two-week pilot.

How far we trust AI-written test cases

We use AI to support both development and test automation. It helps us generate code, create test cases, improve existing automation and investigate failures. It saves a lot of time.

But we do not trust AI-written test cases blindly. In our experience, we can trust them maybe 60 to 70%.

Here is a typical example. We had a new application, and we asked AI to write test cases for it. AI gave us tests for navigation, login, forgot password and reset password. All useful. But it did not cover integrations, and it did not test the actual functionality the application was built for. The tests were about getting into the app, not about what the app does.

So we did the missing work ourselves. We gave AI the important functionality of the application, with example steps: perform these steps, and write test cases for this flow. With that context, AI produced much better test cases. But a human still had to know which functionality mattered and tell AI about it.

How we review an AI-written test

When we review a test case written by AI, we check things in this order:

  1. The test case itself. Does it cover the actual business requirement and the real functionality of the application, or just the easy parts?
  2. The assertions. Does it check the real expected result, or only that an action completed?
  3. The selectors and locators. Are they stable, or will they break on the next UI change?
  4. The test data. Is it realistic, and does it make the result meaningful?

Then we run it. A review of a test is still just a review. Only the execution tells you whether the test works and whether the result is correct.

What good test evidence looks like

When a test fails, the result “failed” is not enough. You need to know where it failed and why. We learned this the hard way while building our own platform.

When a change broke our own reports

This happened to us while working on Regressionly itself. We made some changes to how automation execution results are reported. The code was reviewed, and the overall test results were still correct. Every scenario that should pass passed, and every scenario that should fail failed.

But the individual step results were not showing correctly. For example, a test had three steps that passed and one step that failed. The overall status, failed, was right. Regressionly was not showing the correct status for each step, so anyone reading the report could not see where the test actually broke.

Our regression testing caught it. The changes to reporting had affected an existing feature, one that nobody was working on and nobody thought to check in review. That is exactly the kind of problem regression testing exists for: a change in one place quietly breaking something that already worked somewhere else.

We fixed it, and today every automated run shows the result of each individual step, along with the error message, stack trace and screenshot for the step that failed.

Before and after. Overall result failed in both. Broken: the four steps of the test do not show their real status. Correct: three steps passed and one step failed, with its error message, stack trace and screenshot. Regression testing caught the broken view
The overall result was right both times. Regression testing caught that the step-level view was not.

The lesson for every team is simple. Checking only the overall test result is not enough. You need to validate the individual steps and the complete execution workflow.

How we investigate a failure

With that evidence in place, this is how we investigate a failed test:

  1. Find the failed step. See exactly where the execution stopped.
  2. Look at the screenshot and the error message. See what the application showed at that moment.
  3. Read the code behind that step. The stack trace points to the step definition or page object where it failed.
  4. Check the execution history. Did this test pass before? Has it failed like this before?
  5. Decide what kind of problem it is. An application bug, a locator change, an environment issue or an automation stability issue.
Five-step failure investigation: find the failed step, check the screenshot and error, read the code via the stack trace, compare with execution history, then decide whether it is an application bug, a locator change, an environment issue or an automation stability issue

That last step matters. Not every failure is an application bug.

Flaky tests are real, so evidence matters

Automated tests can be flaky. A test sometimes fails because the application loaded slowly, an element took longer to appear, the network had a problem or the test environment was unstable. Then it passes on the next run.

So we never treat every failure as a bug. We check the execution history, the logs and the screenshots first. Only then do we decide whether it is an application issue, a locator change or an automation stability problem. Without that evidence, a team either wastes hours chasing noise or starts ignoring real failures.

Why execution history matters

A single run tells you what happened today. Execution history tells you how the application behaves over time.

With history, you can see repeated failures. You can identify flaky tests. You can see tests that used to pass and have now started failing, which is exactly what a regression is. You can compare results from one run with another, for example before and after a release. And over time, you can use all of that to make the application more stable.

Want this kind of evidence on every release, without building it yourself? A Regressionly pod runs your regression suite overnight, triages every failure and sends you a human-verified report each morning: what passed, what broke and why. Talk to us.

Seeing what has not been tested

Testing only gives you confidence for what was actually tested. So you also need to see what was not.

Keeping test cases and their execution history in one place makes that visible. You can find test cases that have never been executed. You can also mark each test case with its automation status, such as automated or manual, so you know how each one is tested, and which ones nobody is testing at all.

Manual or automated?

We do not automate everything. We automate the important functionality, and especially flows that involve integrations. In a test management application, for example, creating, editing and deleting a test case are core functions, so they are automated.

Repetitive variations are a different story. Take an input field that should accept only letters and reject special characters. We automate one or two of those checks. Entering many different inputs into the same field again and again is faster and cheaper to cover with manual testing. The goal is not maximum automation. It is the right coverage at the right cost.

The role of QA in AI-assisted development

AI has changed the job of QA. It is no longer only about finding bugs.

QA teams now need to understand the business requirements. They need to identify the test cases that are missing. They need to review the test cases AI generates and verify whether the application actually behaves correctly.

AI helps here too. It helps us understand error messages, suggests possible solutions and sometimes spots the issue quickly. But AI also makes assumptions without understanding the complete application flow. If it cannot access the application, it can only suggest a fix without checking the functionality. Sometimes that fix works. Sometimes it fixes one issue and creates another. That is why every suggested change still has to be verified by actually executing it.

AI can create test cases faster. It still needs human understanding.

How we decide a release is ready

Before a release, we look for a few things:

  • Critical test cases are passing.
  • There are no unresolved high-priority issues.
  • The regression execution was successful.
  • Newly added functionality has been properly validated.
  • Existing functionality has not been affected by the new changes.
  • Reports are correct, clear and easy to understand.
  • Every failure has enough detail to understand what went wrong.
Release readiness checklist: critical tests passing, no unresolved high-priority issues, regression run successful, new functionality validated, existing functionality unaffected, clear reports, enough failure detail. Developers confirm the code, QA confirms the behavior, AI supports

Who makes the final call?

Developers and QA, together. Developers make sure the code is implemented correctly. QA verifies the actual behavior of the application: whether the generated code and the functionality really work. AI can speed up the whole process, but it should support the decision, not make it on its own.

One habit to adopt today

If your team uses AI-generated code, adopt this one habit. Do not trust code just because it looks correct. Always execute it first. Validate the actual results: is the outcome correct, and are the changes really reflected in the application? Then test the existing functionality, to make sure the new changes did not break anything that already worked.

Frequently asked questions

Is code review the same as testing?

No. Code review checks how the code is written: duplicate code, unnecessary logic, security concerns and maintainability. Testing checks what the application actually does when it runs. Code that looks correct in review can still fail in execution, so a team needs both.

Why does AI-generated code pass review but fail in testing?

AI writes code that looks clean and logical, so it reads well in review. But a reviewer sees the code, not the running application. Small gaps, like a missing URL parameter or a check that only confirms a click, only show up when the code is executed.

Can you trust AI-written test cases?

Use them, but do not trust them blindly. In our experience, AI-written test cases are useful but cover maybe 60 to 70% of what is needed. They often cover login and navigation and miss the real business functionality, so a human has to give AI the important flows and review the result.

Why can a test pass when the feature is broken?

Because the test only checks that an action completed, not that the expected result happened. Clicking Submit successfully does not prove the data was saved. A good test checks that the data was saved and that it shows up correctly in the application.

What should you check first when an automated test fails?

Start with the step where the execution stopped. Then look at the screenshot and the error message, the code behind that step, the stack trace and the earlier execution history. Together they tell you whether it is an application bug, an automation issue or an environment issue.

Who should decide that an AI-generated feature is ready to release?

Developers and QA together. Developers make sure the code is implemented correctly, QA verifies how the application actually behaves, and AI supports the decision but does not make it on its own.

How does Regressionly help teams that build with AI?

Regressionly provides human-in-the-loop testing pods. They build and run your regression suite, review every run, triage every failure and send a human-verified report, so what reaches production has been tested, not only reviewed.

Where Regressionly fits

Your team can keep using AI to write code and tests. Keep your code reviews too. They catch real problems. But neither of them proves that the application works. Only execution does, with evidence a person has checked.

That is what Regressionly does. We give you a testing pod: human testers who work alongside your AI-driven development team. They:

  • Build your regression suite, or adopt and extend the Playwright, Cypress or Selenium suite you already have. You own all of the code.
  • Run it overnight on every build, so a human-verified report is ready before your morning stand-up.
  • Check what your tests actually prove, not just whether they pass.
  • Triage every failure with step results, screenshots, stack traces and history, and tell you whether it is a real bug.
  • Sign off on every run. A named senior tester owns the verdict.

AI can write the code, and review can check the code. Only proper testing can give you confidence that the application actually works.

If your team ships AI-generated code and wants that confidence on every release, start a free two-week pilot. Within two weeks, you get a running regression suite on your critical flows and your first human-verified report. See pricing for the plans after that.

References

About the authors

Regressionly provides human-in-the-loop testing pods for teams that build with AI: AI speed, human judgment, every release verified by a person.

Share Article

Read more by topic: