Characterization tests: make an AI refactor safer

A request to tidy existing code should not silently change what callers receive. In this illustrative booking example, the first cancellation still succeeds after cleanup. The difference appears only when a caller repeats the request.

Record the behaviour before the edit#

Before changing the implementation, exercise its public operation with a few deliberately chosen inputs. Record its actual outputs and relevant state changes. Give each observation a descriptive test name, then confirm the tests pass against the unchanged version.

Michael Feathers describes characterization testing as learning and documenting actual behaviour rather than assuming the behaviour we wish the system had. That distinction matters when the original requirements are incomplete. Passing these tests says the recorded observations still match; it does not settle whether those observations are desirable.

For our example, capture a first cancellation, a repeated request, and a booking that starts already cancelled. These are chosen cases, not a claim of complete coverage. The input and response are small enough to inspect directly, so there is no need for a large snapshot file.

Follow the second cancellation request#

The original operation returns cancelled and changes the booking’s state on its first call. A second call returns already_cancelled and leaves that state alone. The preserving refactor extracts the state change into a helper. The broken version additionally replaces the repeated-call result with an exception.

@dataclass
class Booking:
    cancelled: bool = False


def cancel_original(booking: Booking) -> str:
    if booking.cancelled:
        return "already_cancelled"
    booking.cancelled = True
    return "cancelled"


def mark_cancelled(booking: Booking) -> None:
    booking.cancelled = True


def cancel_refactored(booking: Booking) -> str:
    if booking.cancelled:
        return "already_cancelled"
    mark_cancelled(booking)
    return "cancelled"


def cancel_broken(booking: Booking) -> str:
    if booking.cancelled:
        raise ValueError("already cancelled")  # Unrequested behaviour change.
    mark_cancelled(booking)
    return "cancelled"

The tiny helper is here to make the comparison visible, not because every one-line assignment needs an abstraction. These functions model only cancellation state. Deadline eligibility, authorisation, database writes and notifications are separate concerns that this exercise does not implement.

Now save the observed return values and state in explicit assertions. The runner can apply these same expectations to any of the three versions without regenerating them from that version’s output.

class CancellationTests(unittest.TestCase):
    cancel = staticmethod(cancel_refactored)

    def test_first_request_cancels_booking(self) -> None:
        booking = Booking()
        self.assertEqual(self.cancel(booking), "cancelled")
        self.assertTrue(booking.cancelled)

    def test_second_request_returns_existing_result(self) -> None:
        booking = Booking()
        self.cancel(booking)
        self.assertEqual(self.cancel(booking), "already_cancelled")
        self.assertTrue(booking.cancelled)

    def test_previously_cancelled_booking_keeps_state(self) -> None:
        booking = Booking(cancelled=True)
        self.assertEqual(self.cancel(booking), "already_cancelled")
        self.assertTrue(booking.cancelled)

The first test alone would miss the change. The repeated-request test exercises a sequence: construct a booking, cancel it, cancel it again. The final test starts directly in the cancelled state. Both expose the new exception in the broken version.

Would you accept this refactor?#

Next, change the assumption: suppose the task explicitly requires repeat requests to return a documented error. Preserving already_cancelled is no longer the acceptance rule. The old test remains useful as a record of what changes, but the new requirement must determine the new assertion and any caller updates.

This distinction fits the coding agent’s repair loop. A failing check is a reason to investigate, not automatic permission to rewrite either the code or the expectation until the screen turns green.

Compare the original and changed versions#

Save the complete Python fixture. It uses Python 3.11 or later and the standard library, including sqlite3 for the neighbouring database exercise. The article’s fragments are taken from this file.

python3 booking_tests.py –demo legacy runs all three cancellation checks against the original function. They pass, establishing the recorded baseline for these cases.

python3 booking_tests.py –demo refactor selects the deliberately broken change. Three checks run; two report errors because the repeated and already-cancelled cases now raise an exception. The command intentionally exits with code 1.

python3 booking_tests.py selects the behaviour-preserving refactor, the corrected deadline rule and the SQLite adapter. All 11 checks pass. These results were reproduced with Python 3.13.5. The run notes list the expected results and the maintenance check.

Decide separately what should change#

Choose observations that matter to a caller: a return value, a visible state change, or whether an operation happened. Checking the extracted helper’s name would tie this example to its structure rather than the response it is meant to preserve. Larger tests also need stable starting data; an uncontrolled clock or a changing external service can make a before-and-after comparison ambiguous.

Keep the old implementation or a reproducible baseline while making the change. Do not let an agent replace every stored output as part of its cleanup. Inspect changed expectations individually and explain which requirement authorises each change. This is a proposed review practice, not a guarantee that a sufficiently large suite makes any rewrite safe.

The name characterization says why a test exists, not which components it runs. The unit-versus-integration example shows when a real database is needed to inspect a constraint. Our in-memory cancellation checks do not establish database rollback, notification delivery or safe retries across a network; idempotency and retries addresses that wider problem.

When implementing a new requirement instead of preserving an existing response, choose the expected values independently. The deadline testing exercise shows why copying the new implementation’s answer into its own test can leave the wrong behaviour unchallenged.