Test
Test mode writes and proves the test suite of an existing Lex app across ten responsibilities, gated on a user-story document and on an effectiveness audit that shows each test can actually fail.
Test
Test mode writes the test suite for an existing LEX app and then proves that suite actually catches regressions. It is a flat workflow — a catalogue of responsibilities, not a numbered step loop — and it never commits or pushes.
Slash command: /lex-test.
The entry gate
kickstart_test_run refuses to open a session when the project has no user-story document, returning ok: false with prerequisite_blocked. The gate is non-waivable because an assertion derived from the implementation pins whatever the code does today, bugs included, so it can never fail. When it blocks, call dispatch_reverse_prerequisite: it hands the project to Backward for the missing documentation and records the pending return, so the test session resumes on the way back.
The flow
/lex-test
↓
kickstart_test_run(project_path=... OR github_url=...) # mandatory first call
list_responsibilities() / propose_test_plan() # discovery + heuristics
set_test_plan(responsibilities=[...], rationale="...") # mandatory ones injected
↓ per responsibility:
get_responsibility_brief(...) → runSubagent(brief.agent) → notify_responsibility_complete(...)
# inline: record_scenario_matrix() in design, record_test_execution() in verification
↓
finalize_test_run() # writes test-runs/TEST-REPORT.mdTen responsibilities in five phases
Phase | Responsibilities | Mandatory |
|---|---|---|
Ground truth |
| Yes |
Design |
| Yes |
Authoring |
| No |
Verification |
| The first two |
Reporting |
| Yes |
The five mandatory responsibilities are injected into every plan; waive_responsibility reaches only the optional ones.
In Lex, the action chain is the test
Test data is not a fixture the suite reads — the JSON create/update/delete chain is the test, replayed into a real database by ProcessAdminTestCase, with Python test files as an optional layer above it. There is no database isolation: ProcessAdminTestCase subclasses unittest.TestCase rather than django.test.TestCase, and lex pytest runs with keepdb=True, so nothing rolls back between methods, classes or runs. Order-dependence is a real failure mode here, and the effectiveness audit tests for it.
Proving the suite works
- Every scenario cites a criterion —
record_scenario_matrixrejects a row citing none. A scenario asserting a number must cite a ground-truth source or a question actually asked; reading it off a run is refused. - The audit perturbs and reverts — it breaks the fixture or code, confirms the test fails, reverts exactly, and issues a verdict of EFFECTIVE, WEAK or INEFFECTIVE. Reported coverage counts effective tests, not merely existing ones.
- Failures are triaged, never softened —
record_test_executionrequires a triage entry per failure: test-wrong, fixture-wrong, application-defect or environment.
finalize_test_run refuses a clean report while blockers stand: no committed plan, an incomplete mandatory responsibility, no scenario matrix, no recorded execution, an unproven acceptance criterion. incomplete_rationale does not bypass that — it stamps the report INCOMPLETE.
Where output lands
Tests go where Lex expects them: Tests/ for JSON scenario data and ProcessAdminTestCase modules, plus lex_test_config.yaml at the project root. Reports stay under test-runs/ — _index.md, TEST-REPORT.md, and a per-run folder with scenario-matrix.json and one report per responsibility. Never a lowercase tests/: on macOS and Windows that collides with Tests/ and merges reports into the fixture tree.
Commit the results yourself. See Outputs and artifacts.
Was this helpful?
Your feedback shapes what we write next.
