Code Review Is Not Testing, and AI Code Needs Both
AI code can look right in review and still fail when it runs. Real examples, what review catches, what only testing proves, and how to get both.
Table of Contents
- Quick summary
- What code review is good at
- What review cannot see
- The login test that looked correct
- A passing test is not a working feature
- How far we trust AI-written test cases
- How we review an AI-written test
- What good test evidence looks like
- When a change broke our own reports
- How we investigate a failure
- Flaky tests are real, so evidence matters
- Why execution history matters
- Seeing what has not been tested
- Manual or automated?
- The role of QA in AI-assisted development
- How we decide a release is ready
- Who makes the final call?
- One habit to adopt today
- Frequently asked questions
- Is code review the same as testing?
- Why does AI-generated code pass review but fail in testing?
- Can you trust AI-written test cases?
- Why can a test pass when the feature is broken?
- What should you check first when an automated test fails?
- Who should decide that an AI-generated feature is ready to release?
- How does Regressionly help teams that build with AI?
- Where Regressionly fits
- Related reading
- References
- About the authors
TL;DR: Code review tells you whether code is written well. Only testing tells you whether the application works. AI makes code look correct faster than ever, so the gap between “approved” and “working” is easy to miss. In this article we share real examples where reviewed code failed the moment it ran, and what good test evidence looks like. Regressionly closes that gap with human-in-the-loop testing pods that run your regression suite and verify every release, starting with a free two-week pilot.
AI can generate code that looks correct. A developer can review it and approve it. And the application can still fail the first time it runs.
We see this almost every week. At Regressionly, we test software for teams that build with AI. Our human testers manage test cases, run manual and automated tests, track every execution result and dig into every failure. The test management platform we built for that work keeps all of it in one place, so we are not depending on scattered reports and spreadsheets. Most of the examples in this article come from that daily work.
The lesson is always the same. Code review and testing answer different questions. With AI-generated code, you need both answers.
In this article we will cover:
- What code review is good at, and what it cannot see
- A real example: the login test that looked correct
- Why a passing test can still hide a broken feature
- How far we trust AI-written test cases
- What good test evidence looks like
- How to decide when a feature is ready to release
Quick summary
| Question | Code review | Testing |
|---|---|---|
| What does it look at? | The code | The running application |
| What does it catch well? | Duplicate code, unnecessary logic, security, maintainability | Wrong behavior, missing values, broken workflows |
| What can it miss? | How the app behaves in real situations | Badly written code that happens to work |
| What does it prove? | The code is written properly | The feature actually works |
| How much do we trust AI-written tests? | - | Useful, but in our experience about 60 to 70% complete |
| Who signs off on a release? | Developers confirm the code | QA confirms the behavior; AI supports, never decides alone |
What code review is good at
Code review is not the problem. It is a valuable practice, and we would never ship without it.
A good review helps a team find duplicate code, unnecessary logic, security concerns and maintenance problems. It checks whether the code is written properly, whether it follows the team’s standards, and whether the next developer will be able to understand it.
That is a lot of value. But notice what all of those have in common. They are about the code itself.
What review cannot see
A reviewer reads the code. They do not watch the application run.
So review cannot always tell you how the application will behave in real situations. Unexpected user actions. Data that is slightly different from what the developer imagined. A change in one feature that quietly affects another feature somewhere else in the application.
Those problems only show up when the code is executed. That is why testing exists.
AI makes this gap bigger, not smaller. AI-generated code usually looks clean. It is well formatted, well named and logically structured. That makes it easy to approve. The better the code looks, the easier it is to forget that nobody has seen it run yet.
The login test that looked correct
Here is a recent example from our own automation work.
We wrote automation code to log in to an application, with help from AI. The code was reviewed, and it looked correct. The navigation logic was right. The steps were in the right order. Nothing in the code looked suspicious.
Then we executed it, and the login failed.
To find out why, we checked the logs, the screenshots and the login data stored during the run. That showed the problem. The application URL needs an important parameter. The full URL, including that parameter, was stored correctly in the project’s .env file. But the automation code did not use the full URL when it navigated to the application. The parameter was simply missing, so the login could never succeed.
The review was focused on the implementation and whether the navigation logic was correct. And it was. Nobody verified the actual application behavior during the review, so nobody noticed the missing parameter. It was only found when we performed the steps manually and compared them with the screenshots, logs and stored login data.
This is a small bug. It took one missing value. But it is a perfect example of why the code review was correct and the code still did not work. A reviewer reads what the code says. Execution shows what the code does.
A passing test is not a working feature
The opposite problem is even more dangerous: a test that passes while the feature is broken.
We have seen this happen many times, and it almost always has the same cause. The test checks that an action was completed, not that the expected result happened.
Clicking the Submit button successfully does not mean the data was saved correctly. A test that only checks the click will pass even if the save failed, or if the wrong data was saved. A good test goes further. It verifies that the data was saved, and that it is reflected correctly in the application.
This matters even more with AI. AI is very good at writing tests that run green. It is not always good at choosing the right thing to assert. When you review an AI-written test, the most important question is not “does it pass?” It is “what exactly is it proving?”
Not sure your green tests are proving anything? Regressionly’s testing pods review your existing suite, find the tests that only check clicks, and add the assertions that check real results. Start with a free two-week pilot.
How far we trust AI-written test cases
We use AI to support both development and test automation. It helps us generate code, create test cases, improve existing automation and investigate failures. It saves a lot of time.
But we do not trust AI-written test cases blindly. In our experience, we can trust them maybe 60 to 70%.
Here is a typical example. We had a new application, and we asked AI to write test cases for it. AI gave us tests for navigation, login, forgot password and reset password. All useful. But it did not cover integrations, and it did not test the actual functionality the application was built for. The tests were about getting into the app, not about what the app does.
So we did the missing work ourselves. We gave AI the important functionality of the application, with example steps: perform these steps, and write test cases for this flow. With that context, AI produced much better test cases. But a human still had to know which functionality mattered and tell AI about it.
How we review an AI-written test
When we review a test case written by AI, we check things in this order:
- The test case itself. Does it cover the actual business requirement and the real functionality of the application, or just the easy parts?
- The assertions. Does it check the real expected result, or only that an action completed?
- The selectors and locators. Are they stable, or will they break on the next UI change?
- The test data. Is it realistic, and does it make the result meaningful?
Then we run it. A review of a test is still just a review. Only the execution tells you whether the test works and whether the result is correct.
What good test evidence looks like
When a test fails, the result “failed” is not enough. You need to know where it failed and why. We learned this the hard way while building our own platform.
When a change broke our own reports
This happened to us while working on Regressionly itself. We made some changes to how automation execution results are reported. The code was reviewed, and the overall test results were still correct. Every scenario that should pass passed, and every scenario that should fail failed.
But the individual step results were not showing correctly. For example, a test had three steps that passed and one step that failed. The overall status, failed, was right. Regressionly was not showing the correct status for each step, so anyone reading the report could not see where the test actually broke.
Our regression testing caught it. The changes to reporting had affected an existing feature, one that nobody was working on and nobody thought to check in review. That is exactly the kind of problem regression testing exists for: a change in one place quietly breaking something that already worked somewhere else.
We fixed it, and today every automated run shows the result of each individual step, along with the error message, stack trace and screenshot for the step that failed.
The lesson for every team is simple. Checking only the overall test result is not enough. You need to validate the individual steps and the complete execution workflow.
How we investigate a failure
With that evidence in place, this is how we investigate a failed test:
- Find the failed step. See exactly where the execution stopped.
- Look at the screenshot and the error message. See what the application showed at that moment.
- Read the code behind that step. The stack trace points to the step definition or page object where it failed.
- Check the execution history. Did this test pass before? Has it failed like this before?
- Decide what kind of problem it is. An application bug, a locator change, an environment issue or an automation stability issue.
That last step matters. Not every failure is an application bug.
Flaky tests are real, so evidence matters
Automated tests can be flaky. A test sometimes fails because the application loaded slowly, an element took longer to appear, the network had a problem or the test environment was unstable. Then it passes on the next run.
So we never treat every failure as a bug. We check the execution history, the logs and the screenshots first. Only then do we decide whether it is an application issue, a locator change or an automation stability problem. Without that evidence, a team either wastes hours chasing noise or starts ignoring real failures.
Why execution history matters
A single run tells you what happened today. Execution history tells you how the application behaves over time.
With history, you can see repeated failures. You can identify flaky tests. You can see tests that used to pass and have now started failing, which is exactly what a regression is. You can compare results from one run with another, for example before and after a release. And over time, you can use all of that to make the application more stable.
Want this kind of evidence on every release, without building it yourself? A Regressionly pod runs your regression suite overnight, triages every failure and sends you a human-verified report each morning: what passed, what broke and why. Talk to us.
Seeing what has not been tested
Testing only gives you confidence for what was actually tested. So you also need to see what was not.
Keeping test cases and their execution history in one place makes that visible. You can find test cases that have never been executed. You can also mark each test case with its automation status, such as automated or manual, so you know how each one is tested, and which ones nobody is testing at all.
Manual or automated?
We do not automate everything. We automate the important functionality, and especially flows that involve integrations. In a test management application, for example, creating, editing and deleting a test case are core functions, so they are automated.
Repetitive variations are a different story. Take an input field that should accept only letters and reject special characters. We automate one or two of those checks. Entering many different inputs into the same field again and again is faster and cheaper to cover with manual testing. The goal is not maximum automation. It is the right coverage at the right cost.
The role of QA in AI-assisted development
AI has changed the job of QA. It is no longer only about finding bugs.
QA teams now need to understand the business requirements. They need to identify the test cases that are missing. They need to review the test cases AI generates and verify whether the application actually behaves correctly.
AI helps here too. It helps us understand error messages, suggests possible solutions and sometimes spots the issue quickly. But AI also makes assumptions without understanding the complete application flow. If it cannot access the application, it can only suggest a fix without checking the functionality. Sometimes that fix works. Sometimes it fixes one issue and creates another. That is why every suggested change still has to be verified by actually executing it.
AI can create test cases faster. It still needs human understanding.
How we decide a release is ready
Before a release, we look for a few things:
- Critical test cases are passing.
- There are no unresolved high-priority issues.
- The regression execution was successful.
- Newly added functionality has been properly validated.
- Existing functionality has not been affected by the new changes.
- Reports are correct, clear and easy to understand.
- Every failure has enough detail to understand what went wrong.
Who makes the final call?
Developers and QA, together. Developers make sure the code is implemented correctly. QA verifies the actual behavior of the application: whether the generated code and the functionality really work. AI can speed up the whole process, but it should support the decision, not make it on its own.
One habit to adopt today
If your team uses AI-generated code, adopt this one habit. Do not trust code just because it looks correct. Always execute it first. Validate the actual results: is the outcome correct, and are the changes really reflected in the application? Then test the existing functionality, to make sure the new changes did not break anything that already worked.
Frequently asked questions
Is code review the same as testing?
No. Code review checks how the code is written: duplicate code, unnecessary logic, security concerns and maintainability. Testing checks what the application actually does when it runs. Code that looks correct in review can still fail in execution, so a team needs both.
Why does AI-generated code pass review but fail in testing?
AI writes code that looks clean and logical, so it reads well in review. But a reviewer sees the code, not the running application. Small gaps, like a missing URL parameter or a check that only confirms a click, only show up when the code is executed.
Can you trust AI-written test cases?
Use them, but do not trust them blindly. In our experience, AI-written test cases are useful but cover maybe 60 to 70% of what is needed. They often cover login and navigation and miss the real business functionality, so a human has to give AI the important flows and review the result.
Why can a test pass when the feature is broken?
Because the test only checks that an action completed, not that the expected result happened. Clicking Submit successfully does not prove the data was saved. A good test checks that the data was saved and that it shows up correctly in the application.
What should you check first when an automated test fails?
Start with the step where the execution stopped. Then look at the screenshot and the error message, the code behind that step, the stack trace and the earlier execution history. Together they tell you whether it is an application bug, an automation issue or an environment issue.
Who should decide that an AI-generated feature is ready to release?
Developers and QA together. Developers make sure the code is implemented correctly, QA verifies how the application actually behaves, and AI supports the decision but does not make it on its own.
How does Regressionly help teams that build with AI?
Regressionly provides human-in-the-loop testing pods. They build and run your regression suite, review every run, triage every failure and send a human-verified report, so what reaches production has been tested, not only reviewed.
Where Regressionly fits
Your team can keep using AI to write code and tests. Keep your code reviews too. They catch real problems. But neither of them proves that the application works. Only execution does, with evidence a person has checked.
That is what Regressionly does. We give you a testing pod: human testers who work alongside your AI-driven development team. They:
- Build your regression suite, or adopt and extend the Playwright, Cypress or Selenium suite you already have. You own all of the code.
- Run it overnight on every build, so a human-verified report is ready before your morning stand-up.
- Check what your tests actually prove, not just whether they pass.
- Triage every failure with step results, screenshots, stack traces and history, and tell you whether it is a real bug.
- Sign off on every run. A named senior tester owns the verdict.
AI can write the code, and review can check the code. Only proper testing can give you confidence that the application actually works.
If your team ships AI-generated code and wants that confidence on every release, start a free two-week pilot. Within two weeks, you get a running regression suite on your critical flows and your first human-verified report. See pricing for the plans after that.
Related reading
- The Tests Passed and Production Still Went Down: how AI rewrote a failing test to match a bug, and how to catch it
- Why AI-Generated Code Needs Human Testers: why the fix is a human in the loop, not more automation
- What Is Vibe Testing? A Practical Definition: what AI vibe testing is good at, and where it fails
- The Timezone Advantage: Why Overnight Regression Testing Works: how a team in India tests your builds while your engineers sleep
- Services · How it works · Pricing
References
- Google Engineering Practices: What to look for in a code review
- Cucumber: Gherkin reference, the Given / When / Then format for test scenarios
- Playwright: Locators, on writing selectors that survive UI changes
About the authors
Regressionly provides human-in-the-loop testing pods for teams that build with AI: AI speed, human judgment, every release verified by a person.
Share Article
Read more by topic:
Related Articles
Why AI-Generated Code Needs Human Testers
AI-generated code can pass every test and still break the business. Six blind spots only humans catch, and the review rules managers and QA leads need.
The Tests Passed and Production Still Went Down — How That Keeps Happening
AI writes the code and the tests, so both share the same blind spots. A real experiment, the bug it shipped, and how to catch the missing 5%.