SmartTest
Roadmap to 1.6.0
1.0.0 froze SmartTest's contracts. The 1.x line keeps them: every version from 1.1.0 to 1.6.0 is additive (new flags, new config keys with defaults, new adapters), so a config, a script or a --json consumer written for 1.0.0 keeps working. Anything that would break a frozen contract waits for 2.0.0 (contracts).
The first real rollout after 1.0.0 set the order. On 1 November 2026 SmartTest went into four of OpenPro's repositories (front, admin, desktop monitor, MCP service): 0 missed failures and 42–96% less test time, but each repository needed hand work that init should have done, and a bug hid one test file. 1.1.0 adds Angular, proves selection is reliable with checks that inject real faults, brings Python forward as a preview, and makes that hand work unnecessary. OpenPro's backend needs PHP next, so 1.2.0 brings a PHP preview forward, with what the 1.1 rollout left open and the first precision stories. Then the rest of precision (1.3.0), the browser (1.4.0), the ecosystem (1.5.0) and the backend frameworks (1.6.0).
Planning is in story points; one sprint is about 40 points. A version ships when its gate is met, not on a date. Sprints continue from S27 (1.0.0).
Overview #
| Version | Theme | Sprints | Points | Gate in one line |
|---|---|---|---|---|
| 1.1.0 | Angular, reliability, Python preview, rollout (released) | S28–S31 | 152 | Angular and Python selected at 0 misses; a fault-injection audit finds no missed test; init alone reproduces the OpenPro setups |
| 1.2.0 | PHP preview, rollout follow-ups, first precision stories, hooks and CI, Laravel routes (released) | S32–S35 | 119 | PHP selection on OpenPro's backend at 0 misses with less test time; a branch's first push runs only its changes; live specs never run by accident; CI annotations |
| 1.3.0 | Precision | S36–S37 | 52 | Type-checker resolution; the 20% observed-execution target met on two benchmarks |
| 1.4.0 | Browser and end-to-end | S38–S39 | 64 | Source-mapped browser coverage; route-level selection for single-page apps |
| 1.5.0 | Ecosystem | S40–S41 | 60 | Third-party adapters installable from npm; a shared result cache |
| 1.6.0 | Backend frameworks (preview) | S42 | 28 | Laravel, Django and FastAPI edges; a migration selects the tests that touch its tables |
Total: 475 points, fifteen sprints (S28–S42).
1.1.0: Angular, reliability, Python preview, rollout (152 pts) #
Four sprints. S28 adds Angular. S29 turns the 1.0.2 audit into checks that run on every release, so "0 missed failures" is measured, not hoped. S30 brings Python forward from 1.5.0, built on S29's checks. S31 removes the hand work the OpenPro rollout needed.
S28: Angular (40 pts) #
Angular is the main TypeScript framework SmartTest does not understand. Most of an Angular app is ordinary TypeScript and is already analysed, but three things are missing. First, components load their template and styles by path (templateUrl, styleUrl), not by import. Today a change to foo.component.html selects no test at all, not even foo.component.spec.ts; that is a missed test, not only a missing feature. Second, tests run through ng test, not through a runner config. Third, angular.json holds what other projects keep in tsconfig and runner configs.
| Story | Pts |
|---|---|
Component resources are edges. templateUrl, styleUrl and styleUrls in @Component link the component to its template and styles, and Sass/Less @use/@import link a stylesheet to its partials. A template or style change selects the tests of that component and of its importers. A path that cannot be resolved is uncertainty. New edge type: a fixture where a template change selects only its component's specs. | 5 |
Angular workspaces detected. angular.json (and Nx project.json) read for each project: root, sourceRoot, the tsConfig of its build and test targets, fileReplacements (a change to environment.prod.ts selects the tests of environment.ts's importers) and global styles/scripts. detect and doctor report Angular and its version; init proposes the matching runner and scope. | 5 |
ng test as a runner. An adapter for the Angular CLI's test builders: @angular/build:unit-test (Vitest, the default since Angular 21) and the Karma builders. It reads include, exclude, setupFiles, providersFile, polyfills and tsConfig (all of them runner inputs, so a change selects that project's tests), and runs a selection with ng test --include per project. Karma reports per file only; the adapter declares it and does not emulate per-test results (ADR-0005). A run where ng test fails before reporting is a failure, never a pass. | 13 |
Jest and Vitest community setups. jest-preset-angular (setup-jest.ts, zone setup), @angular-builders/jest (Jest config inside angular.json) and @analogjs/vitest-angular discovered and run through the existing Jest and Vitest adapters. | 5 |
Angular code edges. Lazy routes (loadComponent, loadChildren with import()) link the route file to the loaded code; the legacy string form ('./x.module#XModule') is resolved, or treated as uncertainty when it cannot be. inject(X), constructor injection and providers need nothing new, since they import their tokens; a fixture proves it. | 5 |
Fixtures and a real-world check. Three fixtures: a standalone Angular 21 app on the unit-test builder, an NgModule app on Karma, and an Nx workspace on Jest. An open-source Angular application replayed with benchmark and added to the published benchmarks. The trace and resolution oracles from the 1.0.2 audit are run on it, with a trace that also records the non-code files a spec reads (templates, styles, JSON), which the 1.0.2 trace could not see. | 5 |
Angular guide. documentation/guides/angular.md: what is understood, what is uncertainty, the supported builders, and how to read explain on a template change. | 2 |
S28 gate: on the three fixtures, a template, style, component, service or angular.json change selects the affected specs and no others. The full suite through SmartTest reports the same tests and failures as ng test. benchmark on the open-source application shows 0 missed failures, and the trace oracle finds no file a spec loads that selection cannot reach.
S28 status (2 November 2026): delivered except two items, which moved.
- Delivered: component resources and stylesheet imports as edges;
angular.jsonread per project (test and build targets,fileReplacements, global styles); theangularrunner throughng testfor the unit-test builder and@angular-builders/jest;jest-preset-angularand Analog through the Jest and Vitest runners; lazy routes in both forms; theangular-vitestfixture in the regression harness; the real-installangular-clifixture (Angular 22) with its own CI job; the Angular guide. - Found while testing and fixed: a type error in an export no spec runs fails
ng test, so any non-cosmetic change to a file a spec imports selects it on builders that type-check; a VitestrunnerConfigis part ofng test, not a runner of its own. - Changed: Karma has no per-test report, so a Karma target is listed but never run (
ST_ANGULAR_KARMA_UNSUPPORTED), instead of results read from its exit code. - Moved: Nx workspaces (
project.json,nx test) to the monorepo orchestration story of 1.5.0. The NgModule-on-Karma fixture is dropped with Karma. - Real-world check: HMCTS's opal-frontend (354 spec files, 4,054 tests). The full suite through SmartTest gives the same results as
ng test(4,039 test ids: 15it.eachrows share a title, the S29 identity story). Ten commits replayed: 0 missed failures, 4% less time overall. Four of the five commits that ran everything changedyarn.lock(S31's lockfile story); on the other five, 19% less time. The largest selections come from one service injected into 145 specs, which 1.2.0's type-checker resolution addresses. swimlane's ngx-datatable (a Karma app beside a library on the unit-test builder in Chromium): both projects are detected and the library's 21 specs are listed throughng testand selected; their run in Chromium was not checked (the browser download did not complete on this machine).
S29: Reliability (40 pts) #
The 1.0.2 audit found about 30 ways a change could skip a test it affects. It was run by hand, once. This sprint makes those checks part of SmartTest and of every release, adds the one check that measures misses directly (inject a fault, see which tests fail), and closes the two correctness items the OpenPro rollout found.
| Story | Pts |
|---|---|
Fault injection. smarttest audit --mutate changes one function at a time in a real way (makes it throw, or return a wrong value), runs the full suite, and reports every test that failed without being selected. That is a missed test by definition, with no guessing from the graph. It samples functions across the repository within a time budget and never leaves a change behind (it works on a copy). | 8 |
Reachability audit as a command. The trace oracle from the 1.0.2 audit becomes smarttest audit --traces: every file a test loaded or read at run time (code and non-code: templates, styles, JSON, fixtures) must be reachable from that test in the graph, or reached by a runner input or uncertainty. The resolution oracle (each import resolved as the TypeScript compiler resolves it) becomes smarttest audit --resolution. | 8 |
| Audits in CI. A nightly job runs the three audits on every fixture and on the benchmark repositories; any missed test or unreachable file fails it. Each release runs them on the adopting projects and records the result in the delivery log. | 5 |
| Fail-safe under faults. Every internal error, timeout, corrupt or half-written state file, missing Git, killed runner and lost lock ends in a full run with a warning, never in a smaller selection or a pass. A fault-injection test suite proves each case. | 5 |
Flaky failures told apart. When benchmark or audit sees a failure, it reruns that test alone on the parent commit and on the change. A test that fails only under load is reported as load-flaky, separately, and never counted as a miss or as a pass. (1.0.2: MiaDock.test.tsx, tournament-pool.test.ts.) | 3 |
No test file is ever invisible. The built-in ignored directories (build, out, dist…) are skipped as build output, but a file the runner itself collects is always analysed: discovery's list wins over the walker's ignores. doctor and scan report any file a runner collects that the graph lacks. (OpenPro: src/tests/build/…test.ts was never selected.) | 5 |
it.each rows keep their own identity. A test id holds the row's formatted title, or its index when titles repeat, so reports, the result cache and flaky history count every row. (OpenPro: 8 passing rows counted as fewer.) | 3 |
Reliability report. audit writes one report: missed tests, unreachable files, flaky tests, fail-safe triggers, and the share of functions checked by fault injection. Published with the benchmarks. | 3 |
S29 gate: on the fixtures and the benchmark repositories (the five adopting projects and the Angular application), fault injection finds 0 missed tests, the reachability audit finds 0 unreachable files, and every fail-safe case ends in a full run.
S29 status (3 November 2026): delivered; gate met.
- Delivered:
smarttest audit --mutate,--tracesand--resolutionwith one report (.smarttest/reports/audit.{json,md}); a nightly workflow auditing every fixture that needs no browser (pnpm audit:fixtures); the fault-injection suite for fail-safe cases (integration-tests/fail-safe.test.ts); load-flaky failures told apart inbenchmarkandaudit; collected test files always analysed (src/tests/build/) and reported when they are not;it.eachrows numbered. - Found by the audits and fixed: Wasp load-time code in one operation broke tests importing the operations module for another (
--tracesonedges-vitest); a Python classmethod building an instance withcls(...)did not reach the constructor (--mutateon OpenProAuto, 2 missed tests); a unittest class-fixture error failed every file of the run (--mutate, 12 failures reported as load-flaky). Fail-safe: an unwritable state folder exited 70 and changes Git could not read stoppedrun; both now run, with a warning. - Measured: every fixture: 0 missed, 0 unreachable, 0 mismatched imports. Resolution against the compiler: openpro-front 13,037 imports, opal-frontend 8,455, Galdia 5,003, LeadsIt 4,904 (1 unresolved, as uncertainty), openpro-admin 674, OpenProMonitor 461, openpro-mcp-service 55: no mismatch. Fault injection: OpenProAuto 190 faults, Galdia 10, and 5 each in openpro-front, openpro-admin, openpro-mcp-service, OpenProMonitor, LeadsIt and opal-frontend: 0 missed (most uncaught faults are in code no test runs).
--tracesnow records the files a test reads and, for Python, the modules it imports. On the fixtures it found a Python module not linked to its own package's__init__.py(fixed). The trace check was run on the fixtures, not on the adopting projects (it runs every test file alone).
S30: Python preview (40 pts) #
Brought forward from 1.5.0. The Plugin SDK's language contract was designed for this; Python is the first language built on it, as an official plugin marked preview: it selects tests and never hides a failure, but its precision is not yet that of the TypeScript analyzer. Driven by OpenProAuto (Python, unittest).
| Story | Pts |
|---|---|
| Language plugin API in practice. Symbols, imports and tests from a language plugin feed the same graph, rules, safety modes and reports as TypeScript; a mixed repository (a Python service beside a TypeScript front end) is analysed in one run. | 13 |
Python analysis. Modules, classes and functions; import and from … import, relative imports, packages (__init__.py, namespace packages), src/ layouts and the roots in pyproject.toml; conftest.py fixtures as edges to the tests below them. Parsed without running Python. Dynamic imports (importlib, __import__, getattr on modules) are uncertainty and select the tests that reach them. | 13 |
pytest and unittest runners. Tests discovered from each runner's own configuration (pytest.ini, pyproject.toml, setup.cfg, unittest discovery patterns); selections run by node id; results read from JUnit XML; a collection error or "no tests collected" is a failure, never a pass. Configs and conftest.py are runner inputs. | 8 |
Fixtures and OpenProAuto. A pytest fixture and a unittest fixture; OpenProAuto replayed with benchmark and checked with S29's audits. | 4 |
Python guide. documentation/guides/python.md: what is understood, what is uncertainty, how to turn the preview off. | 2 |
S30 gate: on OpenProAuto, the full suite through SmartTest gives the same results as unittest; benchmark shows 0 missed failures with less test time than the full suite; fault injection finds 0 missed tests. The preview can be turned off per project.
S30 status (3 November 2026): delivered; gate met except the time target on OpenProAuto.
- Delivered:
@smarttest/language-python(modules, classes, methods, module values and load-time code, absolute and relative imports, packages,src/layouts andpythonpath,conftest.pyfiles and fixtures by name,importlibas uncertainty) and@smarttest/adapter-python(pytest through--collect-onlyand JUnit XML; unittest through a driver on its own loader); detection next to the JavaScript runners;python.enabledandpython.interpreter; two fixtures, a CI job on Linux and Windows, the Python guide. - OpenProAuto (295 unittest tests): the full suite through SmartTest gives unittest's 295 results; 29 replayed commits, 0 missed failures; 190 injected faults, 0 missed.
- Not met: less test time than the full suite. The six commits that touched OpenProAuto were large and selected 9–13 of its 14 test files, and its whole suite runs in 0.6 s, so SmartTest's analysis made those runs about 0.1 s slower than running everything.
- Mixed repository (
mixed-ts-python: a pytest service beside a Vitest library): one run, each side selects only its own tests; it found that a Python project in a subfolder did not resolve from its own folder (fixed).
S31: Rollout without hand work (32 pts) #
Every item comes from a problem met on OpenPro. Each was fixed by hand there; here it becomes the default. The two correctness items found there moved to S29.
| Story | Pts |
|---|---|
Pre-push reads Git's refs wherever the step runs. When the refs are missing, run --push falls back to the upstream merge-base with a warning instead of ST_HOOK_PUSH_REFS_MISSING; hook install rewrites a backgrounded test step (… &) to receive the refs it saved. | 5 |
init respects the hook a repository already has. It detects hook installers that copy scripts (cp scripts/pre-commit $(git rev-parse --git-path hooks)) and edits the source script, not the copy in .git; when an existing hook already runs related tests (vitest related, jest --findRelatedTests), it replaces that step in place, in the same hook, instead of adding a second one elsewhere. | 8 |
--runner <name> on run, impact, benchmark and validate-selection. Today keeping Playwright out of a parity check or a benchmark needs a copied config. benchmark follows hook.runners unless told otherwise. | 3 |
Lockfile changes select by package. A package-lock.json, pnpm-lock.yaml or yarn.lock change is diffed: only the packages whose resolved version changed select the code that imports them (and their tests). Today any lockfile change selects every test, which put a 4-minute estimate on a one-line push. A lockfile that cannot be parsed still selects everything. | 8 |
The hook budget explains itself. When an estimate passes hook.budgetMs, the warning names the trigger that widened the selection (a lockfile, a config file) and the files with no recorded duration. | 3 |
Rollout guide and case study. documentation/guides/rollout.md from the OpenPro migration: parity check, benchmark, hooks per repository, what to leave to CI; the four repositories added to the published benchmarks. | 5 |
S31 gate: on a fresh clone of each of OpenPro's four JavaScript repositories, smarttest init --yes alone produces a setup equivalent to the hand-tuned one (hook, scope, script placement), the full suite through SmartTest reports the same tests and failures as the runner, and benchmark shows 0 missed failures. No test file a runner collects is missing from the graph in any benchmark repository.
S31 status (3 November 2026): delivered; gate met for the setup, the published benchmarks not updated.
- Delivered: pre-push without Git's refs on stdin compares with the upstream merge-base, and
hook installgives the refs to a backgrounded test step; hooks copied into.git/hooksby a script are edited at their source, andinitreplaces a test step in place;--runneronrun,impact,validate-selectionandbenchmark; lockfile andpackage.jsonchanges select by package; the hook budget warning names what widened the selection; the rollout guide. - Lockfiles on opal-frontend's 12 dependency updates: a
@types/lodashbump selected 0 of 351 specs instead of all; the other 11 still run everything, each for a stated reason (the Angular builder and the test runner, packages a setup file or a script loads). init --yeson the commit before SmartTest in each of OpenPro's four repositories writes the hand-tuned config verbatim and the hook in the same file and stage. Two fixes came from it: a related-tests step kept in pre-push is replaced there (openpro-front), and a hooks folder held in a variable is followed (openpro-admin'ssetup-hooks). The one remaining difference: in openpro-front's parallel pre-push, SmartTest runs before the other steps, not beside them.- Not done: the four repositories in the published benchmarks.
1.2.0: PHP preview, rollout follow-ups, first precision stories, hooks and CI, Laravel routes (119 pts) #
Four sprints. S32 brings PHP forward from 1.5.0 because OpenPro needs it now. S33 closes what the 1.1 rollout left open and takes the precision stories that need no type checker. S34 brings forward what OpenPro's hooks and CI need from 1.4.0 and 1.5.0. S35 brings Laravel routes forward from 1.6.0, because without them the PHP preview selects every booting test on openpro-backend.
OpenPro's backend is Laravel 12 on PHPUnit 11: 2,001 test files, 4,669 PHP files under app/, and CaptainHook hooks that run a home-made scripts/test-changed.sh before each commit and push. PHP is the second language on the plugin path Python opened in S30, and ships the same way: an official plugin marked preview, which selects tests and never hides a failure.
S32: PHP preview (40 pts) #
SmartTest stays an npm package. It reads PHP source in TypeScript, without running PHP, and runs the project's own vendor/bin/phpunit (or php artisan test, or Pest) as a subprocess, as the Python adapter runs pytest. A PHP repository needs Node on the machine, not a package.json.
| Story | Pts |
|---|---|
PHP analysis. Namespaces; use statements (grouped, aliased, use function, use const); classes, interfaces, traits and enums with their methods, constants and properties; extends, implements and trait use as edges; names in the same namespace used without use. Files found through composer.json (autoload and autoload-dev: PSR-4, PSR-0, classmap, files) and require/include of literal paths. A method used through $this->, self::, static:: or a typed parameter links to that method; any other use links to the whole class. | 13 |
Laravel container edges. Facades link to the class behind them (getFacadeAccessor, the aliases in config/app.php); bind, singleton and scoped in service providers link an interface to its implementation; app(X::class), resolve() and make() link to X; the providers in bootstrap/providers.php are loaded by every test that boots the application. | 5 |
Unresolvable means selected. Dynamic names (new $class, $class::$method(), app($name), call_user_func with a variable, string class names built at run time) are uncertainty and select the tests that reach them, as in TypeScript and Python. | 3 |
PHPUnit and Pest runners. Tests discovered from phpunit.xml (testsuites, directory, suffix, exclude) and listed with --list-tests-xml; a selection run by file with results read from --log-junit; php artisan test and Pest detected and used when the project uses them. phpunit.xml, its bootstrap, tests/TestCase.php, .env.testing and database/migrations are runner inputs: a change to them selects every test that boots the application. A fatal error, a test file PHPUnit collects nothing from, or "No tests executed" is a failure, never a pass. | 8 |
Hooks in a PHP repository. init on a repository with composer.json and no package.json: SmartTest runs through npx, php.enabled and php.binary in the config, and a CaptainHook action that already runs changed tests (scripts/test-changed.sh) is replaced in place, as S31 does for Husky. | 3 |
Fixtures and OpenPro's backend. A plain PHPUnit fixture and a Laravel fixture (facade, container binding, feature test with RefreshDatabase); openpro-backend checked for parity, replayed with benchmark and audited with audit --resolution --mutate. | 5 |
PHP guide. documentation/guides/php.md: installing with Node beside PHP, what is understood, what is uncertainty, how to turn the preview off. | 3 |
S32 gate: on openpro-backend, the full suite through SmartTest gives the same results as php artisan test; benchmark shows 0 missed failures with less test time than the full suite; fault injection finds 0 missed tests; init --yes replaces test-changed.sh in its CaptainHook hooks. The preview can be turned off per project.
S32 status (6 November 2026): delivered; gate met for parity, misses and hooks, not for test time; fault injection on openpro-backend not measured.
- Delivered:
@smarttest/language-php(namespaces and every form ofuse; classes, interfaces, traits and enums with their members; member links through$this->,self::,static::,parent::, typed parameters and typed properties; Composer autoloading, literal includes; Laravel facades, bindings,app()/resolve()/make()and the application boot; dynamic names as uncertainty,// smarttest-loads:to resolve one) and@smarttest/adapter-phpunit(PHPUnit, Pest andphp artisan test:--list-tests-xml, a selection by file,--log-junit; a fatal error, an empty file or "No tests executed" fails);php.enabledandphp.binary;initwithoutpackage.jsonand CaptainHook actions replaced in place; two fixtures, a CI job on Linux and Windows, the PHP guide. - openpro-backend (Laravel 12, PHPUnit 11, 1,998 collected test files, on a clone): the full suite through SmartTest gives
php artisan test's results test for test (12,440 tests, the same 31 local failures; 272 s against 276 s).init --yesreplacesscripts/test-changed.shin both CaptainHook hooks.benchmarkon the last 5 commits: 0 missed failures, 1,830 of 1,998 files selected on every commit, 2% less test time (1,327.7 s against 1,351.1 s). - Not met: less test time. A test that boots the application loads every route, so it reaches every controller and what they use; four classes chosen at run time in
app/($class::query(),new $form,app($class)from config,$enum::tryFrom) andrequire $bootstrapFileintests/Support/TestBootCache.phpmake all of them uncertain, selected for any change. Thenaudit --mutateon the Laravel fixture found thatphp artisan testboots the application before PHPUnit starts, so a broken provider fails plain unit tests too: the boot is now an input of every test run through Artisan (fixed), and on openpro-backend the last commit selects all 1,998 files. Matching the URLs a feature test requests to the routes that serve them (1.6.0) is what narrows this. - Fault injection: both PHP fixtures, 0 missed (31 functions). On openpro-backend the
audit --resolution --mutaterun did not complete; it is not measured.
S33: Rollout follow-ups and first precision stories (37 pts) #
| Story | Pts |
|---|---|
A branch's first push runs only its changes. A push that creates a branch has no remote commit to compare with; today it compares with the whole history (deliberately, so nothing is missed). On openpro-front that meant 14,377 changed files and a 27-minute hook. It now compares with the last commit in the branch's history that a pre-push hook passed on, and falls back to the whole history only when there is none. A commit pushed with the hook bypassed has no passed record, so a new branch carrying it still checks it. The existing check that a new target is checked even when its upstream equals HEAD keeps passing. | 5 |
Windows CI green. pnpm verify passes on windows-latest (draft PR #8): paths, line endings and process handling in the tests and adapters, and the job required on main. | 3 |
OpenPro in the published benchmarks. Not done in S31: openpro-front, openpro-admin, OpenProMonitor and openpro-mcp-service replayed with benchmark and added to the published benchmarks; openpro-front's parallel pre-push runs SmartTest beside its other steps, as the hand-tuned hook did. | 5 |
Barrel and re-export pruning. A change behind a barrel (index.ts) selects only the importers of the symbol that changed, through any depth of re-exports, including export *. | 8 |
| A lazily loaded index. The in-process analysis at 10,000 modules under 500 ms (654 ms at 1.0.0): load the parts of the index a change reaches, not all of it. | 8 |
Selection precision report. benchmark adds the share of selected files that actually ran the changed code (from traces), so precision is measured, not only safety. | 8 |
S33 gate: a push creating a branch from main on openpro-front selects what the branch changed, not the whole history; pnpm verify green on Linux, macOS and Windows; the four OpenPro repositories published with 0 missed failures; barrel pruning at 0 misses on the published benchmarks, with the precision report showing the change; analysis at 10,000 modules under 500 ms in process.
S33 status (7 November 2026): delivered; gate met except the 500 ms target, which the released main misses (547–564 ms).
- First push of a branch: compares with the last commit a pre-push hook passed on; with no passed record, the whole history, as before. A commit pushed with the hook bypassed is still checked.
initkeeps openpro-front's parallel pre-push parallel (SmartTest starts as the background job the script waits for). - Windows CI:
pnpm verifypasses onwindows-latest(PR #8) and the job is required onmain, beside Linux and macOS. - OpenPro in the published benchmarks: openpro-front, openpro-admin, openpro-mcp-service and OpenProMonitor replayed on 5 November, 0 missed failures, 16–75% less test time (benchmarks for 1.2.0).
- Barrel pruning: namespace re-exports read member by member; a barrel edit that only changes named re-exports selects the importers of those names. Six commits on the
barrels-vitesthistory select 12 test files instead of 19, 0 missed; unjs/ufo's window is unchanged (107 of 168, its changes were already followed to the symbol); fault injection 0 missed on every fixture. - Lazy index: the in-process analysis at 10,000 modules took 466 ms (median of 10 runs at load about 1.2; 657 ms before on the same machine, 654 ms at 1.0.0), measured before the barrel-pruning and PHP merges. On the final
mainit measured 584–614 ms at load 12–15, then 547 ms and 564 ms in two runs on a quiet machine (load under 2): the later merges cost about 90 ms, and the target is still missed. - Precision report:
benchmark --tracesreports the share of measured selected files that ran the changed code: 55% onbarrels-vitest, 100% on ufo's window (8 of 8 measured).
S34: Hooks and CI (27 pts) #
| Story | Pts |
|---|---|
Commit trailer. A commit hook can add a trailer to the commit message with what the pre-commit run did (Tests: 12 run, 2,340 not run · 4 min saved), opt-in with hook.trailer; hook install adds the commit-msg step for Husky, lefthook, simple-git-hooks, CaptainHook and plain hooks. Replaces openpro-backend's scripts/test-trailer.sh. A commit whose tests were not run (bypassed, no record for the staged tree) gets no trailer, never a false one. | 5 |
Playwright projects understood; live specs. Moved from 1.4.0. Each project's baseURL, dependencies (setup projects) and testMatch read from the config; a spec that only targets a remote baseURL (https://…, not localhost) is marked live and never run by run, benchmark or a hook unless asked (--live). | 8 |
CI annotations. Moved from 1.5.0. GitHub Actions, GitLab CI and Bitbucket: selected and skipped tests as annotations on the pull request, with the explain reason; a step summary with time saved. | 8 |
Upgrade assistant. Moved from 1.5.0. smarttest sync proposes config changes for new 1.x features (lockfile selection, live specs, type-checker), shown as a diff, never applied silently. | 6 |
S34 gate: on openpro-front, the 78 Playwright specs that target the deployed environment are marked live and no hook, run or benchmark runs them without --live, while CI still can; a pull request on SmartTest's own repository shows the annotations and the summary; sync on each of the seven adopting repositories proposes the 1.2.0 changes as a diff and applies only what is accepted; openpro-backend's commits carry SmartTest's trailer instead of test-trailer.sh's.
S34 status (6 November 2026): delivered; gate met except the trailer on openpro-backend, which needs that repository's own change.
- Live specs: on openpro-front all 78 Playwright specs target
https://dev.openpro.aiand are classified live; withE2E_BASE_URLpointing at a local server, none are.run,ci, hooks,validate-selection,benchmarkandaudit --mutatereport them as "live, not run" unless--liveis given. Projects are read throughplaywright test --list, so environment variables in the config apply; a config Playwright cannot list makes no spec live and runs every selected spec. - CI annotations: on SmartTest's own pull request #11 the check run carried 10 notices (9 selected files and "116 more selected") and a fallback warning; GitLab and Bitbucket get report files, with no API call.
sync: dry-run on fresh clones of the seven adopting repositories proposed nothing for the five OpenPro repositories (their 1.2 setups were already made by hand),unittestfor LeadsIt andplaywrightfor Galdia (which leaves it out on purpose: noise from the existing drift check).hook.traileris proposed where a repository writes its own test trailer.- Commit trailer:
hook.traileradds acommit-msgstep for Husky, lefthook, simple-git-hooks, CaptainHook and plain hooks; no record for the committed tree, no trailer. Not done: openpro-backend still runsscripts/test-trailer.sh; switching it is part of its upgrade to 1.2.0.
S35: Laravel routes (15 pts) #
S32 made PHP selection correct on openpro-backend but not faster: every feature test boots the application, the application loads every route, and every route reaches its controller, so any change under app/ selects all 1,998 test files.
| Story | Pts |
|---|---|
Routes as edges. routes/*.php read statically (verbs, resource/apiResource, groups with prefix, middleware and controller, names, model binding): each route links to its controller action and the middleware, form requests and policies it names. A change to a route file, global middleware, a provider or the configuration still selects every booting test. | 5 |
Requests as edges. The URLs and route names a feature test requests (get, postJson, route('name'), action([…])) link the test to the matching routes, not to every route. A request whose URL cannot be read statically is uncertainty for that test only. | 5 |
Run-time class choices. $class::…, new $class and app($class) where $class comes from a literal map, match or config array of X::class values link to those classes; the rest keeps // smarttest-loads:. | 3 |
Fixture and openpro-backend. A routes fixture where a controller change selects only its route's tests; openpro-backend replayed with benchmark and audited. | 2 |
S35 gate: on openpro-backend, benchmark shows 0 missed failures and less test time than the full suite, with a controller change selecting only the feature tests that request its routes; fault injection on the PHP fixtures finds 0 missed tests.
S35 status (6 November 2026): delivered; gate met (openpro-backend measured on a clone with three smarttest-loads comments added).
- Delivered: route files read statically (verbs,
match,resource/apiResourcewithonly/except, groups with prefix, name, middleware and controller, fluent or array attributes, included route files, invokable, string and closure actions); each route is a symbol linked to its action, its controller's constructor and its middleware. Tests' requests (get/postJson/…,json/call, concatenated, interpolated,sprintf, constant,url()androute('name')urls) link the test to the routes that serve them; request helpers (authenticatedRequest($method, $uri)) take their callers' urls;$this->artisan('x:y')links to the command whose$signaturenames it. The boot loads route files, migrations, translations, views, commands and the providers list without using what they name. Class values read frommatcharms, config lists (config('prospection.connectors')) and private methods' literal callers. Found on the way:parent::setUp()made a change to one test'ssetUpreach every test (fixed for every language), and underphp artisan testthe boot is now reached through the graph instead of its import closure. - openpro-backend: the remaining uncertainty after the generic readings is three lines, which the maintainers would annotate (added on the clone only, applied under each replayed commit):
app/Services/Advertising/AdOwnership.phpbefore$class::query()—// smarttest-loads: app/Models/**/*.php;app/Support/AI/Live/Admin/Catalogue/AdminEndpointRules.phpbeforenew $form—// smarttest-loads: app/Http/Requests/**/*.php;tests/Support/TestBootCache.phpbeforerequire $bootstrapFile—// smarttest-loads: bootstrap/app.php. Without them every test that boots the application stays uncertain. benchmarkon the last 10 commits: 0 missed failures, 47% less test time (1,670.5 s against 3,146.1 s for the full runs; the full runs were slowed by load on two commits, 562 s and 346 s against about 265 s). Four commits select every test, each for a reason that holds: rate limiters registered byAppServiceProvider::boot, an edit ofroutes/web.php,config/crm.phpandconfig/prospection.php. The others select 182–537 of 1,998 files; 182 tests stay selected on any change (requests built at run time, routes registered inside tests, helpers called with a computed url).- Fault injection: the PHP fixtures, 0 missed (31 functions). openpro-backend (
audit --resolution --mutate --mutants 5, 27 minutes): 5 faults, 3 caught, 2 survived (both in vendored JavaScript underpublic/that no test runs), 0 missed; resolution 116 of 116 JavaScript imports agree.
1.3.0: Precision (52 pts) #
Run fewer tests for the same safety. Two items have been carried since 0.3.0 and 0.8.0. Barrel pruning, the lazy index and the precision report moved to S33.
| Story | Pts |
|---|---|
Type-checker resolution for calls local inference cannot type (values from factories, generics, injected dependencies). A call resolved by the checker selects the callers of that method instead of the whole class. Opt-in per project at first (analysis.typeChecker), with a time budget; on budget overrun it falls back to today's class-level selection, which is safe. | 21 |
Observed execution, second pass. Traces recorded per test file narrow a static selection only where both agree; trace --stale refreshes the traces a change invalidated. Target carried from 0.5.0: 20% fewer test files than static selection, at 0 misses. | 13 |
| Speed target carried from 0.8.0. 30% less wall-clock time than 1.0.0 on the published benchmarks, from the lazy index, earlier fail-fast ordering and runner warm-up reuse. | 13 |
TypeScript 7 parser. Adopted when its parser API leaves unstable/*; until then a watch item with a compatibility test against each 7.x release. | 5 |
Gate: type-checker resolution on at least two benchmark repositories with 0 misses; the 20% observed-execution reduction met on at least two of them.
1.4.0: Browser and end-to-end (64 pts) #
End-to-end specs are where test time goes, and where running the wrong one is costly: on OpenPro, the Playwright suite targets a live environment.
| Story | Pts |
|---|---|
| Browser coverage through source maps. Bundled and transpiled scripts mapped back to their sources, so a spec is linked to the components it actually rendered; Chromium first, then Firefox and WebKit where Playwright exposes coverage. | 13 |
Pages beyond the page fixture. New pages, popups and context.newPage() traced and mapped to their routes. | 5 |
| Route-level selection for single-page apps. React Router, TanStack Router and Vue Router routes mapped to the specs that visit them, as Next.js routes are today. | 13 |
| Cypress component tests selected like unit tests, with coverage. | 8 |
| End-to-end sharding by recorded duration across CI machines, with setup projects run once per shard. | 8 |
| Studio sign-off carried from 0.9.0. Full-screen design review in both themes, forced colours, text spacing, VoiceOver live regions. | 8 |
Visual and snapshot specs. Screenshot baselines (toHaveScreenshot) linked to the components and styles they cover; a style change selects them. | 9 |
Gate: on OpenPro's front, a component change selects the Playwright specs that render it, from source-mapped coverage; 0 missed failures across the end-to-end benchmarks.
1.5.0: Ecosystem (60 pts) #
| Story | Pts |
|---|---|
Third-party adapters from npm. The Plugin SDK's runner, VCS and language contracts (frozen in 1.0.0) become installable: plugins in the config loads them, doctor runs the conformance kit on each. Example plugins: a Mocha runner and an AVA runner. | 13 |
| A shared result cache for CI. Validated, signed results reused across CI runs of the same commit range (a cache directory or an S3-compatible bucket); never trusted without the HMAC signature 1.0.0 introduced. | 13 |
Monorepo orchestration. Nx, Turborepo and pnpm workspace task graphs read, so smarttest ci runs each affected package's tests with its own runner in the task graph's order. Includes Nx Angular projects (project.json, nx test), moved from S28. | 13 |
Bun and Deno test runners. bun test and deno test as built-in adapters. | 8 |
VS Code and JetBrains. The VS Code extension gains the impact map; a JetBrains plugin with "Run Affected Tests" and explain on hover. | 13 |
Gate: a third-party adapter published to npm passes the conformance kit and runs in a benchmark repository; a shared cache hit verified across two CI runs, with a forged entry rejected.
1.6.0: Backend frameworks, preview (28 pts) #
PHP moved to 1.2.0 (S32), as Python moved to 1.1.0 (S30). What is left is the framework knowledge for both languages, which turns the coarse edges of the previews (every test that boots the application) into precise ones.
| Story | Pts |
|---|---|
Framework edges. Laravel migrations and config keys (config('x.y'); Laravel routes moved to S35); Django and FastAPI routes and settings; a migration selects the tests that touch its tables instead of every database test. | 13 |
Hooks and CI for mixed repositories. pre-commit (the Python framework) installed like Husky and CaptainHook; smarttest ci runs PHP, Python and JavaScript selections side by side. | 5 |
| Benchmarks. The framework edges replayed on openpro-backend and OpenProAuto and checked with the S29 audits; results published with the rest. | 5 |
| Documentation. The framework edges in the PHP and Python guides. | 5 |
Gate: on openpro-backend, a migration selects only the tests that touch its tables, at 0 missed failures; fault injection finds 0 missed tests.
Not before 2.0.0 #
- AI test generation and audit (PRD Release 3), built on the generator's
it.todoscaffolds; the deterministic core never depends on it (ADR-0004). - Cloud and team analytics (PRD Release 5).
- Any change to a frozen contract: config schema, exit codes,
--jsonv1 schemas, Plugin SDK v1.
Languages after Python and PHP (Dart, Go, Rust, Kotlin) follow the same plugin path once their previews are stable.
Previous: Roadmap to 1.0.0 · Delivery log · Risks