Test

Test mode writes and proves the test suite of an existing Lex app across ten responsibilities, gated on a user-story document and on an effectiveness audit that shows each test can actually fail.

A
Written By Atakan KayaLast updated 3 months ago

Test

Test mode writes the test suite for an existing LEX app and then proves that suite actually catches regressions. It is a flat workflow — a catalogue of responsibilities, not a numbered step loop — and it never commits or pushes.

Slash command: /lex-test.

The entry gate

kickstart_test_run refuses to open a session when the project has no user-story document, returning ok: false with prerequisite_blocked. The gate is non-waivable because an assertion derived from the implementation pins whatever the code does today, bugs included, so it can never fail. When it blocks, call dispatch_reverse_prerequisite: it hands the project to Backward for the missing documentation and records the pending return, so the test session resumes on the way back.

The flow

/lex-test
  ↓
kickstart_test_run(project_path=... OR github_url=...)   # mandatory first call
list_responsibilities() / propose_test_plan()            # discovery + heuristics
set_test_plan(responsibilities=[...], rationale="...")   # mandatory ones injected
  ↓ per responsibility:
get_responsibility_brief(...) → runSubagent(brief.agent) → notify_responsibility_complete(...)
# inline: record_scenario_matrix() in design, record_test_execution() in verification
  ↓
finalize_test_run()                     # writes test-runs/TEST-REPORT.md

Ten responsibilities in five phases

Phase

Responsibilities

Mandatory

Ground truth

story-assurance

Yes

Design

scenario-matrix

Yes

Authoring

chain-authoring, assertion-authoring, calculation-coverage, runner-config

No

Verification

execution, effectiveness-audit, chain-fidelity-audit

The first two

Reporting

gap-report

Yes

The five mandatory responsibilities are injected into every plan; waive_responsibility reaches only the optional ones.

In Lex, the action chain is the test

Test data is not a fixture the suite reads — the JSON create/update/delete chain is the test, replayed into a real database by ProcessAdminTestCase, with Python test files as an optional layer above it. There is no database isolation: ProcessAdminTestCase subclasses unittest.TestCase rather than django.test.TestCase, and lex pytest runs with keepdb=True, so nothing rolls back between methods, classes or runs. Order-dependence is a real failure mode here, and the effectiveness audit tests for it.

Proving the suite works

  • Every scenario cites a criterion — record_scenario_matrix rejects a row citing none. A scenario asserting a number must cite a ground-truth source or a question actually asked; reading it off a run is refused.
  • The audit perturbs and reverts — it breaks the fixture or code, confirms the test fails, reverts exactly, and issues a verdict of EFFECTIVE, WEAK or INEFFECTIVE. Reported coverage counts effective tests, not merely existing ones.
  • Failures are triaged, never softened — record_test_execution requires a triage entry per failure: test-wrong, fixture-wrong, application-defect or environment.

finalize_test_run refuses a clean report while blockers stand: no committed plan, an incomplete mandatory responsibility, no scenario matrix, no recorded execution, an unproven acceptance criterion. incomplete_rationale does not bypass that — it stamps the report INCOMPLETE.

Where output lands

Tests go where Lex expects them: Tests/ for JSON scenario data and ProcessAdminTestCase modules, plus lex_test_config.yaml at the project root. Reports stay under test-runs/ — _index.md, TEST-REPORT.md, and a per-run folder with scenario-matrix.json and one report per responsibility. Never a lowercase tests/: on macOS and Windows that collides with Tests/ and merges reports into the fixture tree.

Commit the results yourself. See Outputs and artifacts.

Was this helpful?

Your feedback shapes what we write next.