Skip to content
DivineSWE

DivineSWE Testing Guide

Wiring Tests vs Unit Tests

This guide explains how to author wiring vs unit tests for projects using DivineSWE. The structural enforcement of test-type selection lives in skills/avatar-forge/test-craft.md (Test Type Selection — Thoth’s test craft, read on demand by Thoth, Loki-Break and the Avatar) and agents/4eyes/AGENT.md (Bare-Mock Seam Check).

Wiring tests vs unit tests. A wiring test exercises an entry point (CLI command, HTTP route, message consumer, public API) end-to-end against a transport-layer fake. A unit test exercises a single pure module with no I/O. The project’s terminology favors “wiring” or “boundary” — NOT “smoke” alone (DivineSWE reserves “smoke” as one of five canonical tokens for test-type selection; “smoke” here is NOT the deploy-poke smoke test). Use the wiring/boundary terminology to avoid the deploy-poke confusion.

Typed test doubles (required). Wiring and unit tests MUST use typed test doubles. Bare mocks without type information (e.g., Mock() without spec_set=, vi.fn() without a typed generic, as any casts on fixtures, dynamic in C#) silently demote a wiring test to a unit test by erasing the contract. Use the typed-double form below.

Where wiring tests live. Place wiring tests in tests/integration/ (the project convention). Language defaults: Python tests/integration/, JS/TS tests/integration/, Go co-located as *_integration_test.go, Ruby spec/integration/, Java src/test/java/**/integration/.

Typed Test Doubles

Copy-pasteable examples:

Python (Mock(spec_set=Real))

from unittest.mock import Mock
from myapp.payment import PaymentClient


# Typed double: spec_set binds the mock to the real class's interface.
mock_client = Mock(spec_set=PaymentClient)
mock_client.charge.return_value = {"status": "ok"}

TypeScript (vi.mocked<Real>())

import { vi } from 'vitest';
import type { PaymentClient } from './payment-client';


// Typed mock: the generic binds the mock to the real interface.
const mockClient = vi.fn<PaymentClient['charge']>();
mockClient.mockResolvedValue({ status: 'ok' });

C# (Mock<T> — avoid dynamic)

using Moq;


// Typed Mock<T>: bound to the real interface. Avoid `dynamic` and `Mock<object>`.
var mockClient = new Mock<IPaymentClient>();
mockClient.Setup(c => c.Charge(It.IsAny<ChargeRequest>()))
          .Returns(new ChargeResponse { Status = "ok" });

Transport-Layer Mocking

Mock at the I/O boundary, not the type boundary:

Python with responses

import responses
import requests


@responses.activate
def test_payment_wiring_against_transport_fake():
    responses.add(
        responses.POST,
        "https://api.payments.example/charge",
        json={"status": "ok"},
        status=200,
    )
    # Production code under test calls the real HTTP client; the transport is faked.
    resp = requests.post("https://api.payments.example/charge", json={"amount": 100})
    assert resp.json()["status"] == "ok"

TypeScript with msw

import { setupServer } from 'msw/node';
import { http, HttpResponse } from 'msw';


const server = setupServer(
  http.post('https://api.payments.example/charge', () =>
    HttpResponse.json({ status: 'ok' })
  )
);


beforeAll(() => server.listen());
afterAll(() => server.close());


// Production code under test makes a real fetch(); msw intercepts at the transport.

DivineSWE Test Conventions

  • Test runner: npx vitest run
  • Test naming: should_<behavior>_when_<condition> with underscores
  • Structural tests (tests/structural/): dependency-free deterministic live-file scans. Pure fs reads + regex assertions. No mocks required.
  • Integration tests (tests/integration/): wiring tests that exercise the full pipeline against transport fakes.
  • Script tests (tests/scripts/): unit tests for scripts/*.mjs modules.
  • Agent/skill tests (tests/agents/, tests/skills/): read AGENT.md/SKILL.md content and assert regex patterns against it.

Generated-Artifact Tests Must Prove Real Loadability

Tests for generated platform artifacts must assert real loadability — TOML parseability, shell-command validity (no dangling tokens), hook command-path existence — never token/regex presence alone. Token-only tests have passed while the shipped artifact was broken. Corollaries:

  • Generate into a fresh mkdtemp (never a repo dist/). Because of this, an in-place-regen stale leak is invisible — add explicit in-place-regen wipe tests whenever a generator changes its output layout.
  • Empirically validate external-tool contracts: run the real binary against the actual artifact before claiming conformance. The real-binary copilot conformance tests (install @github/copilot, marketplace add the generated package, assert install + skill self-location + hook wiring execute) are the true loadability proof — stronger than any drift guard. They are gated on the binary + DIVINESWE_COPILOT_E2E, with a CI-only tripwire that FAILS-not-skips when CI is set but the binary/E2E flag is absent.

Doc-as-Contract Guards

When an AGENT.md documents a structured-output shape, a test (tests/structural/agent-doc-example-schema-valid.test.ts) must AJV-validate the documented example against its canonical schemas/<agent>-output.schema.json. Prose-grep “field is mentioned” tests let doc↔schema drift ship.

Boot-Smoke Contract

A Book whose deliverable is a runnable behavior must ship a no-mock end-to-end smoke authored as a Thoth TEST tagged with the canonical boot-smoke marker (distinct from the Rule B smoke token). The 5-clause contract is single-sourced in skills/avatar-forge/test-craft.md and referenced by name elsewhere (drift-guarded). Enforcement: the plan-time shape-check (scripts/verify-consultation-complete.mjs) fires only on a boot-smoke-marked task; the binding teeth are the 4eyes book-level audit (REQUEST_CHANGES on missing/vacuous/hand-wired smoke — per-prophecy audits cannot see a cross-prophecy smoke); Preacher is an advisory CAUTION tripwire; Styx PASS 14 runs the marked test timeout-wrapped (hang or non-zero = FAIL). Honest ceiling: a green smoke means “domain-wired”, NOT “runtime-verified” (depth-1; browser/presentation tier out of scope).

Doc-Relocation Test Idiom

Relocation/restructure tests live in tests/structural/ as dependency-free live-file scans. Assert content-preservation-not-mention: pick unique multi-line anchor strings per relocated block (not generic vocabulary), plus an explicit allow-list of terms legitimately shared by source and destination (lore, agent names, phase names). Add a link-resolution guard: for each relative markdown link, existsSync the target path (strip any #anchor first); for anchor links, GitHub-slugify the target file’s ##/### headings and assert the anchor is a member. Gotcha: a naive section-by-heading parser treats bash # comments inside fenced code blocks as headings — scope install-command scans carefully.

Test Baseline

Current test baseline: ~6187 tests. Pre-existing failures (~10) are NOT regressions — they are deprecated-marker live-file scans for removed skills. Localize regressions by failing-test SET diff vs baseline, never by failure count alone.

Relocated test-idiom history (from CLAUDE.md)

  • The site/ subtree has its own vitest runner (site tests run via cd site && npm run test, never the repo-root npx vitest run); whole-repo structural walkers must keep 'site' in their SKIP sets so the recursive copy never traverses site/node_modules.
  • The audited boot-smoke wiring gate: a Book whose deliverable is a runnable behavior must ship a no-mock end-to-end smoke authored as a Thoth TEST tagged with the canonical boot-smoke marker. Styx runs a 14-PASS protocol whose new PASS 14 (Boot-Smoke) runs the marked test timeout-wrapped (hang or non-zero exit = FAIL).
  • Doc-restructure test idiom: relocation tests live in tests/structural/ as dependency-free live-file scans. Assert content-PRESERVATION-not-mention — pick UNIQUE multi-line anchor strings per relocated block plus an allow-list of terms legitimately shared by README and docs.
  • Real-binary copilot conformance tests are the true loadability proof: they install @github/copilot, marketplace add the generated package, and assert install plus skill self-location plus hook wiring execute.
  • Doc-as-contract guard: when an AGENT.md documents a structured-output shape, a test must AJV-validate the documented example against its canonical schemas/<agent>-output.schema.json — prose-grep “field is mentioned” tests let doc-versus-schema drift ship.

Environmental Gotchas

  • A stray .git directory in the tmpdir (even an empty one) poisons every mkdtemp fixture: isRepositoryBoundary() trusts a .git directory unconditionally, so /tmp reads as a repo root and fixtures climb to it — surfacing as mass phantom “net-new” failures in the rite-gates/offering/teachings families. A/B-probe with git stash before attributing a regression to the working diff.
  • .mjs JSDoc must not contain TS-construct substrings — tests/scripts/mjs-conversion-adversarial.test.ts scans file text verbatim, so even a comment-only @typedef {{x: string}} trips it. The house pattern: hoist type shapes into the sibling .d.ts as named exported types and reference them via @typedef {import('./x.d.ts').T} T (the house pattern shipped in hermes-dispatch and spend-digest).

Source-Shape Gates (relocated from CLAUDE.md)

These guard the shape of the source itself, not the behaviour of a test. Each was found the hard way.

  • npm run lint (eslint scripts hooks) is a gate. It has shipped red on already-committed code because no phase gate ran it. Run it beside npm run typecheck, never instead of it. For the .mjs/JSDoc TS-construct rule it pairs with, see the .mjs JSDoc bullet under Environmental Gotchas above — that is its one home.
  • Math.ceil lives only in scripts/layout-hint.mjs. The Book-87 guard oracle-consult-step5a4-collapse-adversarial.test.ts flags any ceil(, Math.ceil, CEILING or “round up” anywhere under agents|skills|hooks|scripts|schemas — deliberately broad, so the layout algorithm’s ceiling rule has exactly one home. Express a quorum without rounding instead: seen * 3 >= total * 2.
  • consumedByOf (scripts/scry.mjs) honours a leading frontmatter fence ONLY when it is PREPENDED above the # title, and accepts any non-empty free text as the consumer. A fence after the title, body prose, or an empty value all leave the seed panelling as staged.

SET-diff blind spots

A name-keyed failing-test SET diff is blind two ways, both hit in a single cycle:

  • Payload drift. A test failing in BOTH the baseline and the current run whose failure MESSAGE changed reads as unchanged. This nearly lost a platform-regen obligation: the message grew to name new files while the test name stayed put. Compare failure messages for both-failing tests, not just the set of names.
  • beforeAll deaths. A suite whose setup throws marks its tests skipped, never failed, so nothing reaches status=failed to diff. Twenty passing tests silently stopped running this way. Scan for wholly-skipped suites explicitly.

Also check the collected FILE COUNT. A full run can silently under-collect — several runs collected 761-763 of 805 files while reporting ZERO collection errors. A short run reports FEWER failures than the truth, so it reads cleaner than reality, and it also FABRICATES payload drift (suites that generate into temp dirs produce different numbers when their siblings never ran). Assert the file count against an expected floor before trusting any verdict.

Control bytes in tests/**

tests/** carries no raw C0 bytes or DEL — \u escapes only, enforced by tests/structural/control-byte-guard.test.ts. The guard reads file BYTES via Node readFileSync; grep provably misses binary-classified files in this repo, so never implement this scan with grep. tests/structural/source-nul-bytes.test.ts is the complementary guard: NUL-only, over scripts/**+hooks/**. Complementary scopes, stated in both files.

Apparatus trap: \u escapes written through a shell heredoc can MATERIALIZE into literal control bytes at the tool boundary. Byte-scan any escape-heavy fixture after writing it.

Property and metamorphic tests

fast-check + @fast-check/vitest, with the run seed PINNED literally in the file so counterexamples reproduce. Do not set endOnFailure — it stops at the first failure WITHOUT shrinking, trading a one-character counterexample for a long one and defeating the point.

Derive oracles from the SPEC’s independent statement of the invariant. A property whose expected value is read off the implementation passes for every input by construction and proves only that the code equals itself; it exercises real code and is not a tautology, so both of the obvious guards admit it. A property must EXERCISE the implementation while deriving its EXPECTATION from the spec — those are different things. Record deliberate NON-properties (with the reason) so a later reader does not “restore” them.