Delivery log: 0.3.0 to 1.2.0

What was actually delivered against the roadmap, sprint by sprint. The foundation sprints S0 to S10 (0.1.0 and 0.2.0) are in the foundation backlog. Planning is in story points; one sprint is about 40 points. There are no dates: a sprint is a unit of work, and a version ships when its gate is met or the gap is written down.

Status words: done (shipped and tested), partial (shipped with a stated gap), moved (re-planned in a later version), not done (dropped from the version, reason given).

Summary #

The table covers 0.3.0 to 1.0.0. From 1.1.0 on, each sprint's status block in the 1.x roadmap records what was delivered and what moved; the sections below summarise each version.

Tests grew from 124 (0.2.0) to 434 repository tests plus 114 extension tests (0.9.0); the regression harness grew from 4 to 9 fixtures, all at 0 missed failures in BALANCED and SAFE mode.

S11–S12 · 0.3.0 Precise graph #

Goal: select by method, not by class.

Result: a change to TokenManager.verify no longer selects the tests that only use issue. LeadsIt measured with 0.5.0.

S13–S14 · 0.4.0 Complete graph #

Goal: edges for what is not TypeScript code.

Result: edges-vitest fixture with one targeted test per edge type and seven seeded regressions.

S15–S16 · 0.5.0 Runtime evidence #

Goal: combine static analysis with what tests actually run.

Gate: 0 missed failures over 12 + 40 LeadsIt commits; observed execution removed 1 of 11 test files on LeadsIt, short of the 20% target. Translation-namespace false positives found on LeadsIt were fixed (one commit went from 37 selected files to 4).

S17–S18 · 0.6.0 Adoption #

Goal: set up in any repository and any CI without editing configuration.

Gate: commander.js, path-to-regexp, ufo, defu and LeadsIt set up with init --yes, 0 missed failures (results). Bugs found on the way: fluent methods returning this, node --test directory globs, a local t mistaken for a translation function.

S19–S20 · 0.7.0 Adaptation and E2E #

Goal: stay correct when the project changes shape; select end-to-end specs.

Gate: met (integration-tests/adaptation.test.ts; next-playwright fixture; Playwright benchmark with 0 missed failures).

S21–S23 · 0.8.0 Speed #

Goal: run what remains faster.

Gate: not met. LeadsIt's 12-commit replay went from 47.2 s to 43.4 s (8% less, target 30%), 0 cache-integrity violations. A repeated run of an unchanged state on LeadsIt takes 1.2 s instead of 4.1 s (93 of 100 test files from the cache).

S24–S26 · 0.9.0 Hooks and interfaces #

Goal: make staged and pushed checks reliable and expose selection evidence in local interfaces.

Validation: pnpm verify passes 434 repository tests and 114 extension tests, TypeScript, Biome, package boundaries and generated-planning checks. Nine regression fixtures remain at zero missed failures. The packaged CLI was installed outside this checkout and served Studio assets and authenticated APIs. Its runtime audit reports zero advisories; the VSIX passes archive validation.

LeadsIt: all 714 assertions across 101 test files pass with 0.9.0 (4.78 s in this run). The staged hook reports 314 passed and 400 cached, zero failures, in 3.67 s including isolation. Its 12-commit benchmark reports 53.59 s full, 7.18 s project-gated, 34.38 s SmartTest, zero missed failures and zero cache violations. That is 36% faster than full runs and 379% slower than the project-gated baseline. These are local observations with a changing history/cache; they do not establish a release-to-release speed improvement.

The OpenPro-style fixture preserves typecheck, build and PID/wait semantics through hook install --replace. Hook duration was 1.88 s before and 2.70 s after; the two-commit benchmark found zero misses and zero cache violations. With no remote base, the new hook checks the complete committed tree conservatively. This small fixture does not represent OpenPro's full workload.

The LeadsIt dry run exposed replacement-string interpretation of shell $' sequences during an existing-hook upgrade. The installer now inserts the modified text through a callback; a regression test preserves all shell dollar sequences. The applied upgrade changes only the SmartTest invocation. A real adoption commit then exposed lost executable permissions during atomic replacement; atomic writes now preserve existing permissions, with an upgrade/uninstall regression test. The hook was restored before the final adoption commit.

Studio security checks cover token, Host, Origin, request methods, content type, source traversal and symlink boundaries, CSP and SSE shutdown. Missing benchmark values stay unavailable; incomplete runs cannot be labelled passed. Retained browser evidence covers five screens, both themes and 1440/768/390 widths without document overflow. The overview inspector has zero failures and three warnings; seven keyboard checks pass. These checks are not a complete WCAG 2.2 AA or all-screen design signoff. See the Studio critique.

Post-release audit (same tag, 2026-10-30): a performance and UX review before publication fixed N+1 work and repeated computation in hooks, Studio and the VS Code extension, and fixed Studio defects found only in a real browser.

• Hooks. The isolated checkout overwrote the index's file stats with its fresh mtimes, so the hook and the next normal run each re-read and re-hashed every source file. The checkout now reuses the working tree's stats for identical content. On a 4,000-file fixture, the next run's index update fell from about 175 ms to about 90 ms; the hook's own time is dominated by the checkout and the tests and did not change measurably (measurement). "Before" reads are memoised and fetched eight at a time, checks are bounded by CPU count, and a regression test covers the index stats.
• Explain. explain <file> --units explains every unit from one graph build. The extension uses it, so it makes one CLI call per file instead of one per symbol.
• Studio browser fixes. About 1,800 CSP errors from inline styles broke layout; Back/Forward did not work; re-renders lost focus and scroll; unit line numbers were off by one. All are fixed.
• Studio honesty. Trust and hook figures without data read as unmeasured, and an all-cached run has the verdict cached-pass. The overview inspector reports zero failures and zero warnings at 1440/768/390.
• VS Code. Analyses are cached until the graph changes; a file watcher replaces the 10-second expiry; concurrent requests for a file share one analysis.
• Tests. Tests now resolve workspace packages from source rather than dist/.

Feature review (same tag, 2026-10-30): a correctness review of every 0.9.0 feature, each defect reproduced before it was fixed and covered by a test that fails without the fix.

• Core. A file whose runner never started (missing Vitest) has no test records and was summed as a pass with exit 0; it now counts as an infrastructure error (exit 4).
• Hooks. Unstaged edits leaked into the staged run through workspace packages linked in node_modules, which could pass a commit whose staged content fails; the checkout now gets its own node_modules that points workspace packages at the staged copies. hook install --pre-push consumed the refs on stdin, which disabled the user's own pre-push guards; the refs are now handed back. Hooks were written where Git never runs them (project in a subdirectory, linked worktrees, absolute core.hooksPath). Other fixes: checks scoped to workspace packages were silently skipped; the user's post-checkout hook ran on every commit; Git inside the checkout saw HEAD as the index; delete-only pushes ran the whole history; SIGTERM/SIGHUP left the checkout behind; lefthook's pre-push entry got no stdin; non-shell hooks were broken by install; file names starting with - reached checks as options.
• Hooks, gaps closed afterwards. simple-git-hooks in a subdirectory project, links from anywhere in pnpm's store, and cleanup that could forget the user's own worktrees.
• Studio. A run in which no test passed (nothing selected, every test skipped) was "passed"; it is now no-tests. Results stream per runner group. Overlapping refreshes analysed the wrong source. Runs from the terminal or a hook were never picked up. A cancelled run was shown as finished, and the run drawer could stay "Running" after the event stream reconnected. An event-stream reader that stops reading is now disconnected instead of buffering without limit, and a run that finishes before the browser learns its id is caught up.
• VS Code. The run verdict now reads file and test statuses, so a run whose runner never started is no longer shown as passed. Runs exiting 4 or 70 keep their report, and the forced kill of a stuck CLI actually happens. Windows .cmd shims could not start, and "Run for this file" turned an explain error into "no path". A stale run verdict stayed in the status bar, CodeLens could bind to a nested function of the same name, and deactivation left processes running.

Validation round (same tag, 2026-10-31): the limits listed after the feature review, measured or tested instead of stated.

• OpenPro's pre-push workload. The front end (about 9,600 TypeScript files, 2,164 test files, npm) was tested in a separate clone with its existing pre-push hook: type-check, vitest related and a production build in parallel. Over three recent commits the hook as installed failed in 136–143 s, because vitest related ran out of Node's default 4 GB heap; with an 8 GB heap it passed in 125–176 s. With hook install --pre-push --replace the hook passed in 42–45 s, of which SmartTest took 3.4–6.0 s for 4–11 test files, the rest being type-check and build. A six-commit benchmark found 0 missed failures and 0 cache violations (95 failures already present in the parent commits on this machine): full runs 12 min 22 s, SmartTest 2 min 26 s, 80% less; the dependency-patch commit selected every test. Two findings: Vite refused ?url imports of installed packages inside the temporary checkout (fixed, with a test that fails without the fix), and SmartTest conservatively selects OpenPro's 21 Playwright specs on every push because their imports cannot be resolved; those need a browser and the deployed environment, so the measurement excluded e2e/** with exclude. A hook-only runner scope does not exist yet.
• dash. The pre-push and subdirectory blocks run under dash 0.5.12 (Debian's /bin/sh) and the system shell; CI on Ubuntu runs them under dash.
• VS Code. pnpm --filter smarttest-vscode test:host starts VS Code 1.143.1 with the extension against a Git copy of node-test-esm, trusted and in Restricted Mode: activation, all seven commands, status bar, CodeLens, hover, impact, a passing and a failing affected run, results going out of date on save, and no process in Restricted Mode (11 checks, passed twice locally). It found that a bin.js path in smarttest.executablePath did not start on Linux and macOS.
• CI. .github/workflows/ci.yml runs pnpm verify on Linux, and the extension's unit and real-host tests on Linux, Windows and macOS.
• Studio accessibility. pnpm studio:a11y drives every screen, the run drawer, the shortcuts dialog and pairing in Chromium, Firefox and WebKit (WPE on Linux), light and dark, with axe-core (WCAG 2.0–2.2 A/AA) plus keyboard, focus-return, live-region, reflow at 320 px and target-size checks, and keeps each screen's accessibility tree. Failures went from 39–41 per combination to 0. Not done: a pass with a real screen reader (NVDA, VoiceOver), real Safari, forced colours and text spacing.
• Studio refresh race. A refresh waiting behind another analysis could answer a staged request with the working tree; a deterministic test now covers it.

Gate: partly met. Features and limitations above are shipped explicitly as alpha; accessibility, complete keyboard/visual review, extension-host validation and remaining interface capabilities carry into 1.0.0. Source highlights indicate observed unit execution, not measured line coverage. Per-file results stream as each runner group finishes; cached results arrive with the summary.

S27 · 1.0.0 Stable #

Goal: freeze the public contracts, publish benchmarks, review security, and close the gates carried from 0.6.0 to 0.9.0.

Carried gates closed in this version:

• npm. ADR-0007 accepted: Apache-2.0, published from a local pnpm release:publish. npm refused the bare name smarttest as too similar to smart-test, and the @smarttest scope belongs to another account, so the package is @tkmremi/smarttest; the command is still smarttest.
• macOS and Windows. pnpm verify runs in CI on Linux, Windows and macOS. The first run had 86 failures on macOS and 142 on Windows. The real bugs were paths spelled differently from the runner's (/var vs /private/var, RUNNER~1) and pnpm's relative symlinks behind a junction. One test is skipped on Windows (signal cleanup; Windows has no POSIX signals).
• Cypress. A CI job installs Cypress 16.1.1 in fixtures/projects/cypress-e2e and runs it through SmartTest. It found four adapter bugs (test ids, missing skipped tests, CDATA failure text, specs that fail to load), all fixed.
• Studio with real assistive technology. CI runs Studio in real Safari (axe and keyboard), VoiceOver with Safari and NVDA with Chrome, through Guidepup, and keeps the spoken logs. NVDA announces landmarks, headings, run progress, the verdict and results. On hosted runners VoiceOver does not record live-region announcements at all (a plain calibration page is not heard either), so run progress and verdict are advisories there, not gated checks.
• Hook scope. hook.runners and hook.exclude leave browser specs to CI. They are reported as left to CI, never as passed.
• Extension file scope. --only restricts the change set, not the analysis.
• Studio. The impact map has filters and JSON, SVG and DOT export. The source viewer shows line and branch coverage. Chromium browser coverage is collected for Playwright.
• Defect closed for the gate. A selected test that no runner executes (for example, when playwright test --list fails) made the run pass with a warning. It now fails with exit code 4.

Gate: met. No open defect is known that can make a failing test be skipped. The regression harness and all six real-repository replays are at 0 missed failures.

Pre-publication audit (same tag, before npm). A code audit of the release diff, a first-user audit in fresh installs, and Galdia found defects; each is fixed with a regression test (changelog). Three could make a failing test look fine:

• a throwing initializer that a test reaches only through an import;
• two tests with the same name in one file;
• --shard advancing the native baseline.

Galdia's replay: 12 commits, 0 missed, 96% less test time (benchmarks).

1.0.1 · patch (1 November 2026) #

Checked on the six projects that installed 1.0.0 from npm (Galdia, LeadsIt, OpenPro's front, admin, MCP service and desktop monitor): doctor, scan, impact, explain, trace and real runs.

• Fixed: a file named like a test that no runner collects but other files import (src/ads/__tests__/renderManager.tsx in OpenPro's front end) stopped selection; the 12 specs importing it were never selected for changes under src/ads/. Selection now goes through such helpers. The run never passed (exit 4), but its hint would have hidden the specs.
• Fixed: Vitest and Jest discovery ignored the runner's config file (a Stryker config listed 2,298 files instead of 29); a failed listing's filename fallback took helpers as end-to-end specs; "Cannot be ruled out" now gives the graph's reason.
• Measured on 1.0.0: impact 0.65 s (Galdia, 1,400 files) to 1.3 s (OpenPro front, 10,600 files) warm, analysis itself under 25 ms; trace about 4–5 s per test file (one process each).

1.0.2 · patch (1 November 2026) #

An audit of 1.0.1 before more projects adopt it, so the fixes ship in one release. Two soundness oracles ran on the five adopting projects (every file a test loaded at run time must be reachable in the graph; every import must resolve as the TypeScript compiler resolves it), and four reviews reproduced each finding in a throwaway project.

• Fixed: four ways a run passed while the suite failed (Vitest hook failures, runner exit codes and run-level errors, failing Playwright setup projects, collected files not named like tests).
• Fixed: about 30 ways a change skipped a test it affects, across runner configuration (setup files, config imports, mapped modules), code analysis (interface-typed instances, doMock, workers and child processes), the graph walk, import resolution and its cache, the change set (deleted data files, subfolder projects, submodules, outside the project) and trace freshness (environment variables).
• Checked after the fixes: the trace oracle finds no file a test loaded that selection cannot reach (Galdia's ops scripts and LeadsIt's MCP server are now linked); resolution agrees with the compiler on 19,233 imports. Replaying 10 commits with full suites on each of the five projects gives 0 selection misses; the two failures the replay flagged (MiaDock.test.tsx in openpro-admin, tournament-pool.test.ts in Galdia) are tests that fail only under full-suite load, with no dependency on the change.
• Faster: 400 changed files 1.18 s to 0.51 s (one git cat-file --batch), path rules no longer quadratic, explain from the last decision 86 ms, traces.json about 18 times smaller.

1.2.0 · minor (7 November 2026) #

Sprints S32–S35 of the 1.x roadmap (119 points). Per-sprint status: the status blocks there.

• S32 PHP preview: PHPUnit, Pest and php artisan test, PHP analysed in TypeScript without running PHP, Composer autoloading, Laravel facades, bindings and the application boot; init in a repository without package.json, CaptainHook actions replaced in place. openpro-backend: php artisan test's 12,440 results test for test. Not met in S32: less test time (every booting test was uncertain), closed by S35.
• S33 Rollout follow-ups and precision: a branch's first push compares with the last commit a pre-push hook passed on; Windows CI green and required; OpenPro's four JavaScript repositories published at 0 missed failures, 16–75% less test time (benchmarks for 1.2.0); barrel and namespace re-export pruning; a lazily loaded index (466 ms at 10,000 modules when merged; 547–564 ms on the released main on a quiet machine, so the §31.3 target is still missed); selection precision in benchmark --traces.
• S34 Hooks and CI: live Playwright specs reported as "live, not run" unless --live; CI annotations and step summaries on GitHub Actions, GitLab CI and Bitbucket; sync proposes upgrades as diffs; an opt-in commit trailer (hook.trailer). Not done: openpro-backend still writes its own trailer, pending its upgrade.
• S35 Laravel routes: routes, requests, Artisan commands and run-time class choices as edges; parent:: calls no longer reach every subclass, in every language. openpro-backend (10 commits, on a clone with three smarttest-loads comments): 0 missed failures, 47% less test time; audit --resolution --mutate: 5 faults, 0 missed.
• Before the release: pnpm audit:fixtures 18 of 18 fixtures, 0 missed, 0 import mismatches (PHP fixtures included). smarttest audit --resolution --mutate (seed 1, built from 8feb0dd) on fresh clones of the seven adopting repositories: 24,550 of 24,551 imports agree with the compiler (1 unresolved in LeadsIt, counted as uncertainty), 55 faults injected, 22 caught, 38 survived, 0 missed. Survivors are in code no test reaches: vendored minified assets in openpro-backend, ui/ scripts in openpro-mcp-service. Not compared: tests failing before any fault (openpro-mcp-service: 3, which need the vite build its pretest runs; openpro-backend: 22, a service without installed dependencies and 5 PHP tests that need the network). One load-flaky test in Galdia (tournament-pool.test.ts, as in 1.0.2).

1.1.1 · patch (3 November 2026) #

• A file named like a test that no runner runs (a Vitest setup file under __tests__/, a *.live.test.ts the config excludes) is no longer selected as a test that cannot run. Found by openpro-front's pre-push when a push creating a branch compared with the whole history.

1.1.0 · minor (3 November 2026) #

Sprints S28–S31 of the 1.x roadmap. Per-sprint status and what was not done: the status blocks there.

• S28 Angular: component templates and styles and Sass/Less/CSS imports are edges; angular.json read per project; an angular runner through ng test for the unit-test builder and @angular-builders/jest; Karma listed, never run; Nx moved to 1.4.0. opal-frontend: the same results as ng test, 10 commits at 0 missed failures.
• S29 Reliability: smarttest audit --mutate | --traces | --resolution, a nightly fixture audit, a fault-injection suite for the fail-safe cases, load-flaky failures told apart, tests in folders named like build output analysed, it.each rows numbered. The audits found bugs fixed in this release (a Wasp load-time edge, Python cls(...) construction and package __init__.py links, a unittest fixture error failing every file, Yarn's vendored release indexed as source). On the seven real repositories, every import agrees with the compiler and no injected fault was missed.
• S30 Python preview: pytest and unittest, analysed without running Python. OpenProAuto: unittest's 295 results, 29 commits 0 missed, 190 faults 0 missed; not faster than its 0.6 s suite on the commits that touched it.
• S31 Rollout: lockfile and package.json changes select by package (opal-frontend: 1 of 12 dependency updates narrowed, the others run everything for a stated reason), --runner, pre-push without refs on stdin, copied hooks edited at their source, the hook budget explains itself, a rollout guide. init --yes reproduces the hand-tuned setups of OpenPro's four repositories.
• Hooks (from an adopting project's feedback): a missing install stops the commit or push; the estimate counts parallel workers; the pnpm isolation warning fires only when it applies.
• Fail-safe: changes that cannot be read, a failing plugin or an unexpected analysis error run every test with a warning; an unwritable state folder uses a temporary one.

After 1.0.0 #

Carried into 1.0.0 (closed in S27) #

Edit this page on GitHub