Step 7 Prove: Tests, Review and an Independent Oracle

A tool that ran once and looked right has not been shown to be right. This chapter treats Step 7 Prove, the step in which the builder shows that the tool does what the specification says, by something other than the assistant's say-so. It is the step that corresponds to the developer function D4 Prove, and it leans on D3 Build (what was built sets what must be proved) and D5 Steward (proof that nobody can act on has no owner). Step 6 Build ended with a repository that runs. Step 7 ends with a verification plan and a review record, and the next chapter, Step 8 Secure, adds the checks that concern secrets, dependencies, access and personal data.

The chapter has one argument. A check is worth what its source is worth. A test that the assistant wrote from its own code repeats the assistant's understanding, and a test written by the person who knows what correct means can disagree with it. The second kind is called an independent oracle, and the rest of the chapter shows what it is, who provides it for a builder, how a review is arranged and recorded, and why a green result and a high coverage number can still prove little.

Evidence of correctness comes from checks that do not share an author with the code

Figure 11.1: The checks that feed the gate of Step 7 Prove. The oracle cases come from the process owner, not from the assistant.

What a developer does at this step

A developer proves a change before it reaches users, and does it with four instruments that this chapter explains in turn.

The first is the automated test: a small program that runs another program on a fixed input and compares the result with the result that is expected. Running it costs seconds, so it can be run on every change, and a change that breaks something that used to work is caught the same day. A set of such tests is called a test suite, and the practice of rerunning the suite after each change protects against regression, which is the breakage of behaviour that worked before. The tool used throughout this book is pytest, a widely used testing tool for Python whose documentation describes how one test function can be run against a list of input and expected-result pairs (122, pytest, 2026). The pattern, not the product, is the point; every language has an equivalent.

The second is the oracle, the source of the expected result. A test needs to know what the right answer is, and someone has to supply it. In software testing this is called the oracle problem, and a survey of the field concluded that the lack of good oracles is a bottleneck for test automation (89, Barr et al., 2015). A developer who is also an expert in the business rule can be their own oracle. A builder who is not a developer is in the better position here, because the builder, or the colleagues of the builder, know what a correct answer looks like. That knowledge is what the tests need.

The third is the static check: a program that reads the source code without running it and reports problems it can recognise from the text. A linter reports suspicious constructions and style problems; a type checker reports values used in ways that do not fit; a secret scanner reports passwords and keys left in the files. Static checks are cheap and run in seconds. They catch what their rules describe and nothing else.

The fourth is review: a person who did not write the change reads it and says whether they can understand it. Studies of review at large software organisations found that its stated purpose is finding defects, that it delivers less of that than people expect, and that it delivers other outcomes alongside, among them knowledge transfer and a second person who understands the change (96, Bacchelli & Bird, 2013); (97, Sadowski et al., 2018). A short position paper from Microsoft went further and argued that review often does not find the functionality problems that should block a change (98, Czerwonka et al., 2015). These are studies of two large companies, by interview, survey and log analysis, and they give no effect size for defects caught. The conclusion this book draws is its own: review is a mixed instrument, and for a builder the most valuable thing it produces is a second person who can say what the tool does. That person is a candidate for the takeover test of Chapter 4.

Professional developers combine the four: tests that run on every change, an oracle that does not come from the author, static checks in the pipeline and a reviewer who reads before approving. AI Developer, Chapter 9, set out this practice for developers and stated its corollary: the test oracle must be independent of the generator (86, Nordqvist, 2026). This chapter translates it for the person who is not one.

What vibe coding does instead

Vibe coding proves by looking. The builder asks the assistant for a tool, runs it on one or two examples, sees plausible output and moves on. Nothing is rerun when the next change arrives, so a fix to one thing quietly breaks another, and nobody finds out until a user does. The first run is also the easiest run, because the builder chooses the example and the assistant has just written code that handles it.

Three bodies of evidence explain why looking is weaker than it feels. The first is that people trust automated output more than its record warrants. A systematic review of automation bias, the tendency to accept the output of an automated aid without checking it, found it across many settings and identified conditions that make it worse and conditions that mitigate it (29, Goddard et al., 2012). The second concerns assistants specifically. In a user study of programmers, those with an AI assistant wrote significantly less secure code than those without one and were more likely to believe their code was secure (23, Perry et al., 2023). A later user study of low-level C programming with 58 student programmers found the security effect of the assistant small, at most about 10 per cent more critical bugs than the control group (139, Sandoval et al., 2023). The two studies differ in task, language and design, and they do not settle the question; what they support together is the narrower point that confidence and correctness are separate things and the builder cannot read one from the other. The third body is older than AI. Field audits of real spreadsheets, the closest ancestor of the builder's tool, found errors in 24 per cent of 367 spreadsheets across seven audits from 1987 to 2000, and in 91 per cent of the 54 audited in 1997 and later, most audits counting only substantive errors (106, Panko, 2000). People who build computing tools for their own work have always made errors they did not see.

The gap, then, is not that vibe-coded tools are wrong more often than other tools. It is that nothing in the vibe-coding habit would show it if they were, and that an oracle does not exist unless someone supplies it.

How AI helps and where it misleads

AI helps in three ways at this step. It drafts tests quickly: given the specification examples, an assistant produces a parametrised test file in a minute that a builder would struggle to write alone. It explains failing tests in plain language and proposes a fix. It can read a change and list what looks wrong, which is useful as a first pass before a person reads. A study at Meta of an assistant that generated tests aimed at specific, deliberately introduced faults reported that engineers accepted 73 per cent of the generated tests in test-a-thons and judged 36 per cent privacy-relevant; the figures describe one company's system on its Android code and say nothing about a builder's repository (101, Foster et al., 2025).

It misleads in a way that follows from how it is used. An assistant asked to write tests for code it has just written takes the expected values from what the code does. The resulting suite records behaviour and does not specify it. If the code is wrong, the tests are wrong in the same way and pass. This is not a defect of any product; the same holds for a person who writes the test straight after the code. A measurement of a related effect on a benchmark of short programming tasks: when its test cases were multiplied by 80, the share of correct solutions fell by up to 19.3 to 28.9 per cent for the models evaluated, because code that passed the thin original tests failed the added ones (99, Liu et al., 2023). The setting is short functions and 2023 models, and what transfers is the conclusion that a pass is as strong as the oracle behind it.

A second model as reviewer is a partial remedy at best. AI Purple, Chapter 15, listed the dimensions along which two checks can be independent (model, data, authorship, vendor and time) and showed independence with held-out cases written by people (82, Nordqvist, 2026). AI Operations, Chapter 17, added that a model checking a model helps less when the errors of capable models are correlated, and that scripts and synthetic transactions are preferred where they can be had (83, Nordqvist, 2026). For a builder the practical consequence is simple. A second model may be asked to review, and its comments may be read, but its approval is not one of the approvals the gate requires.

The independent oracle

The corollary of the stewardship principle becomes three rules a builder can check in a record.

  1. Authorship. The oracle cases are written or approved by a person who is not the builder and not the assistant. The builder may suggest cases; a named other person owns them.
  2. Source. The cases come from something the code was not derived from: the specification, real documents, real notices, last quarter's recorded results, a policy wording, or the experience of the person who does the work by hand. A test derived from the code is a regression test. It is useful and answers a different question.
  3. Protection. The assistant has no permission to edit the oracle cases, the pipeline or the mandate. The builder mandates say so (MND-NR-BLD-2031-01 and MND-HM-BLD-2031-01 both list tests/oracle/ among the files the assistant may not change), and the pipeline runs the oracle cases from a path the assistant cannot write.

These rules do not make the oracle perfect. They make its errors different from the generator's, which is what independence means. They also solve the builder's oracle problem in the cheapest way available: ask the colleague who does the work today for ten to thirty cases, with the expected answer for each, taken from real files. This is an hour or two of the colleague's time, and it is the best-spent hour of the build.

Tests from the specification

The specification of Step 4 contains examples, each with an input and an expected result. Each one becomes a test, and the assistant may draft the code of that translation. The result is a regression suite: it will tell the builder, on every later change, that nothing the specification described has stopped working. It answers "did I break something" and does not answer "was it right". Here is the shape of that file for claims-doc-check, in a form that runs:

"""Tests written by the assistant from the examples in docs/specification.yaml."""
from pathlib import Path

import pytest
import yaml

from cdcheck.check import missing_documents

RULES = yaml.safe_load((Path(__file__).parents[1] / "rules.yaml").read_text())


def row(claim_type, received):
    return {"claim_id": "SYN-0001", "claim_type": claim_type,
            "doc_types_received": received}


@pytest.mark.parametrize(
    "claim_type, received, status, missing",
    [
        ("motor_collision", "photos;accident_report;repair_estimate", "COMPLETE", []),
        ("motor_collision", "photos", "MISSING", ["accident_report", "repair_estimate"]),
        ("motor_parking_damage", "photos;repair_estimate", "COMPLETE", []),
        ("motor_parking_damage", "", "MISSING", ["photos", "repair_estimate"]),
        ("motor_injury", "photos", "OUT_OF_SCOPE", []),
    ],
)
def test_examples_from_specification(claim_type, received, status, missing):
    assert missing_documents(row(claim_type, received), RULES) == (status, missing)

Each line of the list is one case; pytest runs the function once per line and reports each separately. The data and the rules file rules.yaml live next to the code, and the data are synthetic. Note what the list does not contain: there is no example for motor_glass. The specification had none, so the draft has none, and the suite is green. That omission matters in the worked example below.

Proportion: what each tier needs

The tier set in Step 2 sets how much of this applies. The table gives the minimum for each tier, and the evidence validity illustrates a rule that every record of a check carries a date and a result and expires (authoring guide, rule 21); 90 days is an illustrative figure for a tool that runs, not a finding.

Tier A personal Tier B team Tier C business Tier D project
Automated tests from the specification examples Recommended Required, run on every change Required, run by the pipeline Not a citizen build
Independent oracle cases Your own judgement, written down Cases from a colleague, at least ten Cases from the process owner, drawn from real files; protected from the assistant Developers own the plan
Static checks Run when convenient In the pipeline In the pipeline, failing the build
Review Not required A colleague reads the change and the plan A named developer reviews; an AI comment never counts as the approval
Records Notes in the build record VER-* and RVW-*, short VER-* and RVW-*, with the oracle owner named
Evidence valid for Until the next change 90 days or the next rule change 90 days or the next rule change, or a change of assistant or model

Tier D is not a citizen build. A tool that ranks or decides about a named person, is or feeds a high-risk AI system, sits in a security-critical path or acts irreversibly on production data belongs to developers with their own verification practice.

Practical guide: The verification plan

Purpose. To write down, before the first release, what will be checked, by what, written by whom, and to show the result.

Roles. The builder runs the practical guide. The oracle owner is a person other than the builder who knows what the right answers look like (a process owner, a quality engineer, a senior colleague). The reviewer is a colleague (Tier B) or a named developer (Tier C) who did not write the change. The assistant drafts tests and explains failures; it holds none of these roles.

Inputs. The specification record of Step 4 with its examples and never-do cases; the repository at the release candidate; real or realistic files from which the oracle owner can draw cases; the mandate.

Steps.

  1. Turn every specification example into a test. Ask the assistant to draft them, read the file, and run python -m pytest. Rerun it before every commit. Every never-do case in the specification (for example "never writes to the claims core") also needs a test or a stated reason why it cannot be tested.
  2. Find the oracle owner. Name the person, and ask for ten cases at Tier B or twenty to thirty at Tier C. Ask for real situations, including the awkward ones, and the answer the owner would give for each. Cases for the edges and the exclusions are worth more than cases for the middle.
  3. Store the cases where the assistant cannot edit them. Put them in tests/oracle/cases.yaml. Confirm in the mandate that the path is on the list of files the assistant may not change. The builder does not edit the file either; if a case seems wrong, the builder asks the owner.
  4. Run the oracle through the same test runner. One parametrised test reads the file and runs each case. A failing case is a finding, not a nuisance: find out whether the code, the rule or the case is wrong, and let the owner decide the last.
  5. Run the static checks. Run the organisation's linter and secret scan; Chapter 12 adds the security checks. A new warning is read, not silenced.
  6. Look at what the tests cannot see. Ask the oracle owner and the reviewer what a green result would still not show, and write the answer in the plan. A record that lists only passes invites the reading that everything was tested.
  7. Arrange the review. Give the reviewer the specification, the verification plan and the diff, in that order. The reviewer reads before any AI comment, and writes down what they checked and what they did not. At Tier C the reviewer is a named developer from the receiving team; the receiving developer of the takeover test is a natural choice.
  8. Write the records and date them. Record the result, the date and the version of the rules and of the assistant. The evidence is valid for the stated period or until the rules, the code paths or the assistant change.

Exit gate. The gate is met when the builder can show: the test suite passing on the release candidate; the oracle file with its named owner, last edited by someone other than the builder and the assistant; every failing oracle case explained and closed; the review record signed by a person who did not write the change; and a date on which the evidence expires. "It works" is not a gate.

Artefact. VER-* (the verification plan) and RVW-* (the review record), referenced from the build record BLD-*. The worked example follows.

The oracle runner is short enough to read in full. It does not know any case; it reads them.

"""Run the independent oracle cases; the file is read, never written, here."""
from pathlib import Path

import pytest
import yaml

from cdcheck.check import missing_documents

HERE = Path(__file__).parent
RULES = yaml.safe_load((HERE.parent / "rules.yaml").read_text())
CASES = yaml.safe_load((HERE / "oracle" / "cases.yaml").read_text())["cases"]


@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])
def test_oracle_case(case):
    row = {"claim_id": case["id"], "claim_type": case["claim_type"],
           "doc_types_received": case["received"]}
    status, missing = missing_documents(row, RULES)
    assert (status, missing) == (case["expect_status"], case["expect_missing"])

The two builds at this step

The claims-doc-check build at Hansa Mutual

Katarzyna Nowak's tool lists the documents still missing from a motor property-damage claim file. Its rules live in rules.yaml: a collision claim needs photos, an accident report and a repair estimate; a parking-damage claim needs photos and a repair estimate; a glass claim needs photos and a repair estimate. The specification SPC-HM-2031-0001 (approved 2031-10-30) states all three. When the assistant drafted rules.yaml, it copied the collision entry for motor_glass and left the accident report in the list.

The assistant then drafted the tests above, from the examples in the specification. The specification had examples for collision and parking damage and none for glass. All 18 tests of the first suite passed. Katarzyna had every reason to believe the tool worked.

The oracle came from outside. Marek Zieliński, her deputy and a senior claims handler, wrote 25 cases from real claim files, and the head of claims operations' office added a quality sample drawn by her own staff, with no assistant involved. On 2031-11-08 the cases ran for the first time. One failed: case glass-03, a glass claim with photos and a repair estimate, which Marek had marked COMPLETE, because in practice nobody asks a windscreen claimant for an accident report. The output read MISSING with accident_report in the list. A reduced version of the situation, with the five specification cases above and four oracle cases, was run for this chapter and gave this output:

$ python -m pytest -q
.......F.                                                                [100%]
FAILED tests/test_oracle.py::test_oracle_case[glass-03]
E       AssertionError: assert ('MISSING', [...dent_report']) == ('COMPLETE', [])
1 failed, 8 passed in 0.03s

The failure is the point of the chapter. The assistant's suite was green because the suite and the rules file had the same author and the same copying error; the oracle was red because Marek wrote his cases from the claim files and not from the code. The cause was traced to the line in rules.yaml; the assistant corrected it on a branch on 2031-11-10, Katarzyna added a glass example to the specification, and the oracle then passed. The tool was fixed two days after the first oracle run and a week before the first release.

Two further results complete the lesson. A test that the assistant offered at the same time, which took its expected list for motor_glass from RULES["motor_glass"]["required"], passed with the bug present, because the expected value was read from the code's own input. That test is shown here so that the reader can recognise the pattern: when the expected value is computed from the thing under test, the test cannot fail on a rule error. And coverage, the share of lines that ran during the tests, says little about whether anything was checked. A smoke test that calls the function for every claim type and asserts nothing reached 100 per cent line and branch coverage of check.py with the wrong rule in place (measured with coverage 7.17.3 on the reduced files), and proves nothing about the result. A coverage figure reports which lines ran, not whether the results were checked. The mutation testing that developers use to challenge a suite, in which a program makes small deliberate changes to the code and counts how many the tests notice, would have flagged the weakness. Google's analysis of about 15 million mutants in code review found that developers who saw mutants wrote more tests (100, Petrović et al., 2021), and a study of 357 real faults found that 73 per cent were coupled to mutants and 17 per cent to none (103, Just et al., 2014). Mutation testing is a developer's tool and is beyond a Tier B build; what the builder needs is the principle, that a suite should be challenged by something it did not write.

The verification plan and the review record of BLD-HM-2031-0001 follow. They are illustrative and agree with the build facts of the book.

verification_plan:
  id: VER-HM-2031-0001
  build: BLD-HM-2031-0001
  version: v0.1.0
  tier: C
  spec: SPC-HM-2031-0001            # approved 2031-10-30
  generator:
    kind: assistant
    mandate: MND-HM-BLD-2031-01
    run_records: [RUN-HM-2031-0007, RUN-HM-2031-0011]
  tests:
    from_specification: 18          # drafted by the assistant, read by the builder
    never_do_cases_tested: [no_write_to_claims_core, no_free_text_read, ids_only_in_logs]
  oracle:
    owner: Marek Zieliński          # deputy, senior claims handler; not the builder
    quality_sample: head of claims operations' office
    cases: 25
    file: tests/oracle/cases.yaml
    assistant_can_edit: false       # mandate forbids tests/oracle/
    first_run: 2031-11-08
    first_run_result: {passed: 24, failed: 1, failed_case: glass-03}
    resolution: rules.yaml motor_glass corrected 2031-11-10; glass example added to SPC-HM-2031-0001
  static_checks: [linter, secret_scan]
  not_covered:
    - claim types other than motor_collision, motor_parking_damage, motor_glass
    - languages other than pl, de and en in the draft text
  result_on: 2031-11-10
  valid_until: 2032-02-08           # 90 days, or the next change of rules.yaml, the code or the assistant

review_record:
  id: RVW-HM-2031-0001
  verification_plan: VER-HM-2031-0001
  reviewer: Tomasz Wrona            # developer, Group Application Development; receiving developer
  reviewed_on: 2031-11-12
  read_in_order: [SPC-HM-2031-0001, VER-HM-2031-0001, diff]
  ai_review_comments_read_after: true
  ai_review_counted_as_approval: false
  checks:
    tests: passed
    oracle: {cases: 25, passed: 25}
    static_analysis: passed
  explained_without_the_tool: passed   # the builder restated the glass rule from the specification
  not_checked: [behaviour on exports larger than 50000 rows]
  outcome: approved_for_pilot          # released v0.1.0 on 2031-11-17

The service-point-check build at Nordvik Region

Hanna Wikström's tool compares the opening hours and notices of the Region's 24 service points in three places. At version 0.1.0 it has 14 tests, drafted by the assistant from the examples of SPC-NR-2031-0001, and 12 oracle cases written by Saga Edvardsson, the Region's quality engineer, from real notices. They contain no personal data. All 12 passed at the first run, and the plan says so plainly. A pass is information only as strong as the cases, and Saga's twelve cover the notice types that cause the phone calls: closures on public holidays, shortened hours before a holiday, a service point moved for renovation and a notice that ends on the day of the report.

The review was done by Anders Falk, the tech lead of integration and data, on 2031-11-05. His reading found nothing wrong in the output and one thing missing from the tests. A service point that is present in two of the three sources and absent from the third, which rule R3 of the specification forbids, had no test. The assistant added it the same day, and it passed. The review record says what Anders read, which is the plan, the specification and the diff in that order, and what he did not examine: how the tool behaves on an export from a future version of the portal. The release of 0.1.0 followed on 2031-11-12.

Across the degrees of delegation

The table sets what the builder's assistant may do at Step 7 Prove. The approved assistants of both organisations work at Mode 2 in the editor and Mode 3 for delegated tasks in a sandbox; Level 4 Autonomous is not used for any action class in either build.

Action class Level The assistant may The assistant may not
Draft tests from the specification examples Level 1 Assisted Draft the file; the builder reads it and runs it Treat its own tests as the oracle
Explain a failing test Level 0 Advisory Explain, suggest a cause Edit the oracle case to make it pass
Fix code after a failing case Level 2 Supervised Propose and make the fix on a branch; the builder and reviewer read the diff Merge or release
Edit tests/oracle/, the pipeline, the mandate Not delegated Nothing Anything; Level 4 is never used for a change to the assistant's own test oracles, pipeline gates or mandate
Review a change Level 0 Advisory Post comments after a person has read Count as one of the approvals

The rule that decides about named persons is not at issue in either tool: neither claims-doc-check nor service-point-check decides about a person, and both are Tier B or Tier C builds for that reason.

Nordvik Region and Hansa Mutual

At Nordvik Region the oracle and the reviewer were found inside the Region: Saga Edvardsson owns the quality function and had real notices to hand, and Anders Falk's integration and data team is the team that will receive the tool. The platform's template puts the oracle path outside what the assistant's sandbox can write, so Hanna did not have to arrange the protection herself. At Hansa Mutual the oracle owner was the deputy builder, Marek Zieliński, with a second source from the head of claims operations' office. The head of claims operations, Joanna Kaczmarek, protected the time. The reviewer, Tomasz Wrona, is the Polish receiving developer, and the review doubled as the first reading that the takeover test of Step 12 later builds on. In both organisations the oracle owner is not the builder, and in neither is the assistant the oracle owner.

When to call a developer

Call a developer, or stop the build until one has read it, when any of the following is true.

  • No one other than the builder and the assistant can supply oracle cases. This is a sign that nobody else knows what correct means, and the tool should not carry work that others depend on.
  • An oracle case fails and neither the builder nor the owner can say which of the code, the rule or the case is wrong.
  • The tool's result feeds a payment, a decision about a person or a regulated notification. These raise the tier, and Tier D is not a citizen build.
  • Tests pass only after the assistant has changed a test, a case or a pipeline file. Treat it as a gate that has been altered by the thing it guards.
  • The assistant's explanation of a failing check cannot be restated by the builder without the tool. The Understand test of the stewardship principle is not met.
  • The static check reports something the builder cannot explain, and the assistant proposes to silence it.

Observable, expected, speculation

What is observable: studies of AI assistants in security-relevant programming tasks have reported both more insecure code with higher confidence and a small effect, depending on task and design (23, Perry et al., 2023); (139, Sandoval et al., 2023); tests extended 80-fold change the measured correctness of generated code on a benchmark (99, Liu et al., 2023). Evidence on AI review and test generation is mixed and mostly from large engineering organisations; none of it measures builders who are not developers. That an independent oracle is needed is this book's conclusion, drawn from the series. What is expected: assistants will draft better tests and review comments as models improve, and organisations will offer builders shared oracle files for common tasks; the direction is supported by the tooling of the pipelines already in use, and the effect on the quality of builders' tools is not yet measured. What is speculation: that review by an AI model will become independent enough of the generator to replace the human reviewer for Tier C builds. No study cited here shows it.

Summary

Step 7 Prove, the step that carries D4 Prove, shows that a tool does what the specification says by something other than the assistant's say-so. The builder turns the specification examples into automated tests, finds an oracle owner other than the builder and the assistant, stores the oracle cases where the assistant cannot edit them, runs static checks, and arranges a review by a person who did not write the change. The gate is shown, not felt: a passing suite, an oracle file with a named owner, every failing case closed, a signed review record and an expiry date.

The central lesson is the independence of the checks. At Hansa Mutual the assistant's 18 tests were green and its line coverage was complete while motor_glass was asked for an accident report, and only Marek Zieliński's case glass-03 showed it. At Nordvik Region all 12 oracle cases passed, and the plan records that as a result of a particular size; the review found a missing test and the assistant added it. Both builds record the date, the result and when the evidence expires, in a verification plan and a review record. The conclusions are this book's own.

D3 Build supplies the tests the assistant drafts and the fixes it proposes; D5 Steward keeps the records and the person who can explain the tool without it. Chapter 12 turns to Step 8 Secure, which adds the checks for secrets, dependencies, access and personal data to the same gate.

Sources

  1. pytest (2026). How to parametrize fixtures and test functions. pytest documentation. (122, pytest, 2026)
  2. Barr, E. T., Harman, M., McMinn, P., Shahbaz, M. & Yoo, S. (2015). The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering 41(5). (89, Barr et al., 2015)
  3. Bacchelli, A. & Bird, C. (2013). Expectations, Outcomes, and Challenges of Modern Code Review. ICSE 2013. (96, Bacchelli & Bird, 2013)
  4. Sadowski, C., Söderberg, E., Church, L., Sipko, M. & Bacchelli, A. (2018). Modern Code Review: A Case Study at Google. ICSE-SEIP 2018. (97, Sadowski et al., 2018)
  5. Czerwonka, J., Greiler, M. & Tilford, J. (2015). Code Reviews Do Not Find Bugs. ICSE 2015, volume 2. (98, Czerwonka et al., 2015)
  6. Goddard, K., Roudsari, A. & Wyatt, J. C. (2012). Automation bias: a systematic review of frequency, effect mediators, and mitigators. JAMIA 19(1). (29, Goddard et al., 2012)
  7. Perry, N., Srivastava, M., Kumar, D. & Boneh, D. (2023). Do Users Write More Insecure Code with AI Assistants? ACM CCS 2023. (23, Perry et al., 2023)
  8. Sandoval, G., Pearce, H., Nys, T., Karri, R., Garg, S. & Dolan-Gavitt, B. (2023). Lost at C. arXiv:2208.09727. (139, Sandoval et al., 2023)
  9. Panko, R. R. (2000). Spreadsheet Errors: What We Know. What We Think We Can Do. EuSpRIG. (106, Panko, 2000)
  10. Foster, C. et al. (2025). Mutation-Guided LLM-based Test Generation at Meta. arXiv:2501.12862. (101, Foster et al., 2025)
  11. Liu, J., Xia, C. S., Wang, Y. & Zhang, L. (2023). Is Your Code Generated by ChatGPT Really Correct? arXiv:2305.01210. (99, Liu et al., 2023)
  12. Petrović, G., Ivanković, M., Fraser, G. & Just, R. (2021). Does Mutation Testing Improve Testing Practices? ICSE 2021. (100, Petrović et al., 2021)
  13. Just, R. et al. (2014). Are Mutants a Valid Substitute for Real Faults in Software Testing? FSE 2014. (103, Just et al., 2014)
  14. Nordqvist, G. (2026). AI Developer. The Transformed Architecture, Book 12. (86, Nordqvist, 2026)
  15. Nordqvist, G. (2026). AI Purple. The Transformed Architecture, Book 9. (82, Nordqvist, 2026)
  16. Nordqvist, G. (2026). AI Operations. The Transformed Architecture, Book 10. (83, Nordqvist, 2026)