Why AI-generated tests can pass with the wrong code

Two green tests accompany a proposed booking change. Before approving it, compare their inputs with the rule they are meant to check. This illustrative example is deliberately constructed; it is not a recorded AI failure or a measurement of any model.

Would you accept this change?#

The agreed rule is precise: cancellation is allowed before 18:00. For this small exercise, input is an already-validated integer number of minutes after midnight on one agreed local day. Dates, time zones and permission checks are outside the function. Compare the correct implementation with a plausible one-character mistake.

DEADLINE = 18 * 60  # Minutes after midnight; one already-agreed local day.


def can_cancel(minute: int) -> bool:
    return minute < DEADLINE


def can_cancel_buggy(minute: int) -> bool:
    return minute <= DEADLINE  # Deliberate mistake: accepts exactly 18:00.

The tests below check an allowed time and a rejected time. Their default selection is the correct function. The demonstration runner switches rule to can_cancel_buggy without changing either assertion.

class WeakDeadlineTests(unittest.TestCase):
    rule = staticmethod(can_cancel)

    def test_before_deadline_is_allowed(self) -> None:
        self.assertTrue(self.rule(17 * 60))

    def test_after_deadline_is_rejected(self) -> None:
        self.assertFalse(self.rule(19 * 60))

Choose the expected result before running the code#

Write the expected result from the agreed rule: exactly 18:00 is too late. Do not run the new implementation, observe true and copy true into the expectation. That would record what this version does while leaving the requirement unchecked.

The stronger suite retains the first two cases and checks the boundary and its neighbouring minutes. The example uses minutes, so these are its adjacent inputs; a system accepting seconds would need cases at that precision.

class BoundaryDeadlineTests(WeakDeadlineTests):
    # Inherits the 17:00 and 19:00 checks and adds the missing boundary.
    def test_exact_deadline_is_rejected(self) -> None:
        self.assertFalse(self.rule(18 * 60))

    def test_last_minute_before_deadline_is_allowed(self) -> None:
        self.assertTrue(self.rule(17 * 60 + 59))

    def test_first_minute_after_deadline_is_rejected(self) -> None:
        self.assertFalse(self.rule(18 * 60 + 1))

The exact-deadline assertion fails against the buggy function. This failure is useful because it names a disagreement with an already-decided rule. A failure caused by a missing Python interpreter or an unavailable database would not establish that disagreement.

There are occasions when recording existing output is exactly the job. Characterization tests preserve observed behaviour while refactoring. They answer a different question from whether a newly implemented rule is correct. State which question a test answers before judging its expectation.

Try a deliberate mistake#

Start with the correct function, then deliberately allow equality at the deadline. Do the tests complain? That is the central experiment behind mutation testing: introduce a small code change and check whether tests detect it. PIT’s documentation explains detected and surviving mutations, as well as mutations that do not change relevant behaviour.

Our demonstration manually selects one changed Python function. It does not run PIT, search a whole program for mutations or produce a mutation score. It establishes one narrower result: the weak suite misses this particular change and the boundary suite detects it.

Both weak tests execute the return expression. Merely knowing that the line ran would therefore miss the problem. Coverage asks which code was exercised; the assertion asks whether an observed result matches an expectation; reviewing the requirement asks whether that expectation is right. PIT’s introduction distinguishes coverage from the ability to detect introduced faults.

Even detecting every mutation tried would not settle an omitted feature. Our function has no caller identity, so no variation of its deadline comparison establishes who may cancel a booking. Add the relevant requirement and test at the appropriate boundary rather than treating one successful experiment as a complete review.

Run the passing and failing versions#

Save the complete Python example and run it from that folder with Python 3.11 or later and its standard sqlite3 module. No package installation, network connection or disk database is needed. The displayed Python fragments come from that same file.

python3 booking_tests.py –demo weak runs two checks against the buggy rule. Both pass; the process exits with code 0.

python3 booking_tests.py –demo boundary runs five checks against the same buggy rule. The exact-deadline check fails; the process exits with code 1. This red result is intentional.

python3 booking_tests.py selects the corrected versions across all three exercises. All 11 checks pass. The example notes explain the other modes and their expected results. These commands were executed with Python 3.13.5; the counts describe this fixture, not the website’s test suite.

Review the evidence, not the test count#

For a generated change, ask the author to connect each important rule to an observable check. Inspect where its expected value came from and name a plausible wrong implementation that the check would reject. The deadline example makes that inspection concrete: a count of tests is less informative than seeing whether exactly 18:00 was tried.

Use unit and integration tests to choose which components must participate in that check. Separately, a golden set evaluates the agent’s task completion. An agent can submit passing application tests and still fail an independently retained task expectation. Keep those expectations reviewable rather than deriving them from the submitted implementation. The same check applies when you review the whole pull request: read the tests last, as evidence rather than proof.