Michał Kornacki, PhD

Auditor Evolution: From Ad-Hoc Review to Cross-System Verification

Introduction

The auditing capability did not originate from a design document. It emerged from a failed first attempt and from successive discoveries of failure modes in other team systems. Between 2026-04-01 and 2026-05-12, the practice developed from ad-hoc file review into a systematic discipline with defined audit types, control families, risk codes, and a reproducible report format. This document records that evolution and the current state of the auditing system.

Historical Evolution

Phase 1: Emergence (2026-04-01)

The first audit failed. The internal auditor went overdue by more than an hour, and the orchestrator agent had to complete the work emergently. The only structural rule that emerged was the Time-Boxed Delivery Rule: when a deadline approaches, deliver partial findings with a revised ETA rather than letting perfection block delivery.

Phase 2: Formalisation (2026-04-06)

Two audits on the same day established the first repeatable patterns. A CLAUDE.md audit revealed that the router file had become an encyclopedia — 42,000+ characters with duplicated blocks — producing the principle that long-form protocols belong in workflow files. A mining workflow audit revealed that a data-flow rewrite had not been propagated to dependent files, producing the migration checklist pattern: list every file that references a changed concept, then verify each.

Phase 3: Methodology Maturation (2026-04-06 to 2026-04-28)

A series of deep technical audits transformed practice from spot-checking into systematic discipline. Key methodologies that emerged:

• grep-based extraction and diff for large-file audits — efficient for cross-referencing claims against sources at scale.
• "Read every instance, never spot-check" — a schema deviation found across seven files would have been missed by sampling.
• Bulk-reclass over per-claim correction — a single missing rule in a mining skill produced a 76% false-positive rate across thousands of claims; fixing the rule is more efficient than editing each claim.
• Final-sweep mandatory — folder overhauls consistently leave residue (ghost directories, stale references, orphaned files).

By the end of this phase, the auditor's memory contained codified Proven Patterns: a Migration Audit Protocol, KB Health Check Indicators, and Efficient File Comparison techniques.

Phase 4: System Upgrade (2026-05-11 to 2026-05-12)

The auditing system upgraded to cross-system verification with two major additions:

1. The auditing skill was formalised to version 1.0, introducing three audit types, eight control families, and a risk taxonomy of fourteen codes.
2. The Red-Team Challenge Phase was added: after draft findings, a second agent interrogates evidence sufficiency and alternative explanations; the auditor revises or rebuts each challenge before finalising the report.

Key lessons from this phase:

• Cross-system audits escalate severity for handoff gaps (single-agent gap = AMBER; handoff gap = RED, because risk propagates across boundaries).
• Consultation Panels extend audit reach: when operational logic exceeds file cross-referencing, the owning agent produces a domain brief that the auditor verifies independently.
• Baseline metrics must be quantitative and re-measurable (agent line counts, workflow line counts, skill line counts) so future audits can detect drift.

What Triggers an Audit

Audits are triggered by specific events:

1. A new team member is onboarded.
2. The governing document (CLAUDE.md) is modified.
3. Any team member profile is edited.
4. Folder structures or file conventions change.
5. The owner requests an ad hoc audit.

The Auditing Skill

The auditing skill (.claude/skills/auditing/SKILL.md, version 1.0) is the canonical operational document that governs how the internal auditor conducts every audit. It is loaded at the start of every audit task.

Audit Types

Control Families

Eight control families govern the scope of audit coverage:

1. Agent Governance — Profile completeness, tool restrictions, permission modes, model assignments, colour uniqueness.
2. Skill Security — Command injection risks, data exfiltration vectors, code execution paths. Graded PASS / WARN / FAIL.
3. KB Integrity — Claim accuracy, source-key validity, ID sequencing, duplicate detection, direct-quote verification, false-positive rates.
4. Workflow Handoff Safety — Documented transitions between agents, no dead ends, clear ownership, input/output specifications, confirmation gates.
5. Memory Hygiene — Temp file lifecycle compliance, archive pruning, standing instruction currency, memory accuracy.
6. Citation Accuracy — direct-quote verification against quotation marks, source traceability, in-text placement correctness.
7. Configuration Consistency — CLAUDE.md, settings, agent definitions, and workflows aligned. No silent divergence.
8. Session Provenance — Reproducibility metadata, version linkage (commit SHA, file hashes), audit trail completeness.

Risk Taxonomy

Fourteen risk codes map to the control families:

Standard Protocol (All Audit Types)

1. Scope Definition — The orchestrator agent specifies files, systems, and questions.
2. Evidence Collection — Read all files in scope. Use grep for pattern verification. Use wc, diff, hash comparison for file identity checks.
3. Consistency Matrix — Map rules from CLAUDE.md, workflows, and profiles against actual file content. Record: implemented, partial, missing, contradicted.
4. Five-Question Checklist per rule:
   • Stated? — Is the rule explicitly written?
   • Unambiguous? — Can it be interpreted in only one way?
   • Consistent? — Does it agree with all related rules?
   • Complete? — Does it cover all cases including edge cases?
   • Current? — Does it reflect actual state?
5. Gap Analysis — Trace realistic task scenarios through documented protocols. Identify dead ends, ambiguous responsibility, undocumented handoffs.
6. Severity Classification:
   • RED (Critical) — Rule violated, contradicted, or missing, causing incorrect output risk. Immediate resolution.
   • AMBER (Observation) — Inconsistency, ambiguity, or gap creating future risk. Reported immediately.
   • GREEN (Recommendation) — Improvement suggestion. Batched until five or more accumulate, or a full audit is requested.
7. Reporting — Audit Type, Scope, Findings (severity / location / evidence / assessment / recommended resolution), Workload Check, Folder Integrity Check, Summary.

Cross-System Methodology (Cross-System Audits Only)

8. Handoff Tracing — For each documented transition between agents/systems: Is the transition documented? Are inputs and outputs specified? Is there a dead end or ambiguity? Is there a confirmation gate?
9. Session Verification — Read session temp files and session database records to verify agents followed documented workflows. Check for silent failures, unauthorised deviations, skipped gates.
10. Multi-Agent Consistency — Verify that a rule stated in CLAUDE.md is implemented identically across all affected files. A single deviation is a typo; deviations across all instances are a structural problem.
11. Red-Team Challenge Phase — After draft findings:
    • The orchestrator agent spawns the argument architect or the research synthesizer to challenge each finding.
    • Challenger interrogates: evidence sufficiency, alternative explanations, missed angles.
    • The internal auditor revises or rebuts each challenge.
    • Final report includes Challenge Response appendix.
    • Severity escalation: gap in handoff between agents = RED (output risk propagates across boundaries).

Skill Security Audit Gate

When auditing any skill file, check for:

• Command injection — unsanitised shell variables, concatenated paths
• Code execution — eval, exec, dynamic code generation
• Data exfiltration — external network calls, file reads outside project scope
• Privilege escalation — commands that modify agent definitions, settings, or system files

Grade each skill: PASS (no issues), WARN (minor issue, documented), FAIL (critical issue, immediate remediation).

Evidence Collection Protocol

File-State Sampling:

• Read every instance of a class of artefacts (all agent profiles, all skill files) — never spot-check.
• For extracted documents, read standalone and flag every reference that fails to resolve within the file.
• For large files, use grep to locate target patterns before reading segments.

Reproducibility Metadata Template:

Every audit report must include:

• Audit date and auditor version (from profile)
• Scope file list with commit SHA from git rev-parse HEAD
• File hashes for all files in scope (via sha256sum or md5sum)
• External sources consulted (researcher brief reference, if applicable)
• Tools used (grep patterns, comparison commands)

External Research Integration

When audit scope requires external benchmarking:

1. The orchestrator agent defines scope and provides internal context.
2. The orchestrator agent spawns the researcher agent to produce a Structured Concept Brief.
3. The researcher agent delivers: Concept Inventory, Relevance Mapping, Gap Indicators, Source Provenance.
4. The internal auditor treats the brief as evidence input, not direction.
5. The internal auditor independently verifies the relevance mapping against internal files.
6. The internal auditor assigns severity and recommendations based on independent analysis.

Current State (2026-05-13)

Capabilities

The auditing system supports three audit types, eight control families, fourteen risk codes, and a reproducible six-section report format. Operational methodology includes the Five-Question Checklist, Consistency Matrix, Cross-System Methodology (Handoff Tracing, Session Verification, Multi-Agent Consistency, Red-Team Challenge Phase), Consultation Panels, External Benchmark Integration, and mandatory Reproducibility Metadata. A Skill Security Audit Gate grades skills PASS / WARN / FAIL. The internal auditor reports directly to the owner, preserving independence from the orchestrator agent.

Limitations

The internal auditor has no execution authority: it only produces reports; remediation depends on the configuration agent, the orchestrator agent, or owner action. It conducts no external research directly; all external benchmarking routes through the researcher agent. It modifies no file except its own audit reports. Its workload analysis is observational, not predictive.

Boundaries

1. No execution. Only output is audit reports.
2. No delegation. May recommend the orchestrator agent delegate; does not delegate itself.
3. No file modification. Does not edit any file other than audit reports in Owner's Inbox/.
4. No hiring. May recommend; does not initiate.
5. No filtering by the orchestrator agent. Reports go directly to owner.
6. No external research. Does not fetch external sources. When external benchmarking is required, the orchestrator agent provides a Structured Concept Brief from the researcher agent.

Cross-Cutting Patterns

Pattern 1: Migration Completeness Failures Recur

Every significant system overhaul left stale references behind. The lesson was learned incrementally and codified as the Migration Audit Protocol (five steps: grep -r for old pattern, categorise hits, update all active files, document historical hits, verify no critical paths stale). Yet the protocol was not always applied to the most consequential migrations. The gap between codified lessons and applied practice is itself a recurring theme.

Pattern 2: Spot-Checking Is Dangerous; Read Every Instance

A schema deviation found across multiple files would have been missed by sampling. A grep-based extraction of all source keys and IDs was necessary to catch orphan headers and broken references. The standing rule is explicit: "Read every instance, never spot-check." This represents a shift from sampling to census.

Pattern 3: Single-Rule Omissions Have System-Wide Effects

The absence of one quotation-mark verification rule in a mining skill produced a 76% false-positive rate across thousands of claims. The remediation philosophy shifted from per-claim correction to bulk-reclass + skill amendment: a single-rule omission with wide effect requires systemic remediation, not piecemeal editing. This pattern reappears in migration audits, where a single unpropagated path change breaks multiple operational files.

Pattern 4: Token Efficiency and Bloat Were Discovered Late, Then Quantified

Early audits used qualitative language ("~6 large blocks", "~70% is rarely-needed"). Later audits used wc, diff, and exact line counts. The KISS audit established the first quantitative baseline (agent line counts, workflow line counts, skill line counts). The progression shows a shift from qualitative intuition to metric-driven assessment.

Pattern 5: Residue Accumulates After Overhauls; Final-Sweep Is Mandatory

Folder overhauls consistently leave residue: ghost directories, stale references, orphaned files, empty sentinel files, binary debris. The standing rule: "Any future structural overhaul must include a final-sweep step verified by the internal auditor before declaring complete."

Pattern 6: Audit Independence Is Preserved Through Reporting Structure

From the first audit, reports were intended for the owner directly. The current profile formalises this: The internal auditor reports directly to the owner. The orchestrator agent notifies the internal auditor of auditable events but does not filter, prioritise, or suppress findings. The external audit pipeline reinforces this separation: the researcher agent provides research, the internal auditor judges; neither makes the other's findings.

Pattern 7: The First Failure Produced the Most Enduring Rule

The 2026-04-01 overdue incident was the only audit ever marked failed. It directly caused the Time-Boxed Delivery Rule. No subsequent audit has failed to deliver. Early failures, when honestly recorded and structurally addressed, produce more durable improvements than incremental successes.

Pattern 8: Boundary Files Accumulate Redundancy

Cross-system audits consistently reveal that boundary files (agent profiles, skill files) accumulate redundancy because they are edited incrementally without a canonical source. This explains why multiple agents carry near-identical blocks, why profiles duplicate skills, and why index tables require manual curation. The recommended fix — dynamic index generation, canonical boilerplate files — has been proposed but not implemented.

Pattern 9: Quantitative Rigor Improved Over Time

Early audits used approximations. By mid-2026, audits included exact line counts, file hashes, commit SHAs, and exact line references. The KISS audit even included a self-applied red-team challenge asking whether quantitative miscounts undermine findings.

Pattern 10: The Auditing System Grew by Absorbing Failures From Other Systems

Each major methodological upgrade was a direct response to a failure detected in another team system. The auditing system is essentially a compendium of every other system's failure modes, reverse-engineered into checklists. This is not a weakness; it is the reason the methodology is grounded in verified incidents rather than abstract theory.

Source material includes internal audit reports, profile definitions, and skill files from April–May 2026.