Case study: LeadsIt

LeadsIt was the first real repository migrated to SmartTest (commit d310fbac). It's a good test because it isn't a tidy fixture:

• No root package.json. Six independent projects (backend/, FrontLeadsIt/app/, mcp/, docs/growth/outbound/, …), each with its own lockfile.
• Three runners. Jest (backend, CommonJS JS), Vitest with a custom config (vitest.server.config.ts, TypeScript, Wasp) and Node's built-in node --test (twice, once with a directory argument).
• Tests loaded through computed paths. For example import(`${SCRIPTS}finalize-prerender.mjs`).
• Generated output and a submodule. Wasp's .wasp/ (with its own package.json files) and an apify submodule.
• A pre-commit hook that ran whole suites per touched project and skipped Redis-dependent tests with --testPathIgnorePatterns.

Rule of the exercise #

Only built-in commands. npm install, npx smarttest init --yes, smarttest run --staged in the hook, smarttest benchmark. Whenever SmartTest needed a custom flag or a hand-written config to work, that counted as a SmartTest bug, and it was fixed in SmartTest.

What it forced into the product (0.2.0) #

Result: parity #

npm run test:all through SmartTest: 706 passed, 0 failed. That's identical to the native runners: Jest 177 + Vitest 489 + MCP 30 + outbound 10, with the Redis suites skipped as before.

Result: history benchmark #

npx smarttest benchmark --commits 12 replays the last 12 commits in a temporary worktree. For each one it compares a full run, a project-gated run (the suites of every touched project, which is what the old hook did) and SmartTest. Every failure that wasn't selected is re-checked on the parent commit.

Missed failures: 0 · failures already present in the parent commit: 0

Times are wall-clock on this machine. SmartTest time includes its analysis overhead.

Update: 0.5.0 #

LeadsIt now vendors SmartTest 0.5.0 (tools/smarttest/smarttest-0.5.0.tgz). Parity is unchanged: npm run test:all passes 706 tests. smarttest trace records all 100 test files (Jest, Vitest and both node --test projects) in 23 seconds.

The benchmark window moved by two commits: the last two are the SmartTest migration itself, which change the root package.json and lockfile and therefore run everything.

What changed compared with 0.2.0:

• 0da20f4a and 42c7810f select more tests (2 → 4, 1 → 3). Both change schema.prisma and translation catalogs. 0.2.0 ignored schema changes entirely; 0.4.0 selects the code using the changed models and translation keys.
• The first 0.4.0 build selected 37 test files for 0da20f4a. Helpers such as getFooterNavigation(t) call t(item.nameKey), which counted as "reads any translation key", and the generated auth-critical and payment-critical rules then ran every auth and payment test. Translation keys are now qualified by the namespace bound with useTranslation("ns"), and a t received as a parameter reads in its callers' namespaces. The commit selects 4.
• Traces removed one test (47606fc6): the outbound test loads the changed file but never calls the changed function. Over the ten commits that do not touch the lockfile, observed execution reduced the selection from 11 to 10 test files. Static selection on LeadsIt is already narrow, so the 20% reduction targeted by the 0.5.0 gate is not reached here.

A wider run, smarttest benchmark --commits 200 --sample 40 (40 commits spread over the last 200), found 0 missed failures. SmartTest ran 522 of 3,052 test-file runs (17%), 354 of 2,884 on the 38 commits without a lockfile change, and saved 63% of the full-run time. Against the project-gated baseline it was 6% slower: on single-project commits the old hook runs one small suite, and SmartTest's analysis time is counted in full.

Update: 0.6.0 to 0.8.0 #

Update: 0.9.0 #

LeadsIt vendors tools/smarttest/smarttest-0.9.0.tgz. hook install --dry-run previews the upgrade and hook install --yes changes the existing invocation to run --staged --hook. Pelloux, i18n and TypeScript checks remain intact. Adoption caught and fixed a shell-text replacement bug in SmartTest before applying the hook.

The packaged CLI passes 714 assertions across 101 files in 4.78 s. The final adoption commit f21d47c1 also runs the executable Git hook: 714 passed, zero cached, in 5.3 s; the dirty apify submodule remains unstaged and untouched. A staged hook run takes 3.67 s including isolation: 314 passed, 400 cached, zero failed. Cached assertions were not executed in that run.

The 12-commit replay records 53.59 s full, 7.18 s project-gated and 34.38 s SmartTest, with zero missed failures and zero cache violations. SmartTest saves 36% against full runs but is 379% slower than project-gated execution on this window. It selects 526 of 1,204 test-file runs. History, cache and machine load differ from older measurements; this is not a controlled version comparison. The raw benchmark retains the per-commit results.

Update: 1.0.0 #

Measured on a fresh clone with the 1.0.0 candidate (all repositories, raw results):

• 12-commit replay: full runs 47.5 s, project-gated 12.8 s, SmartTest 6.9 s. That saves 85% against full runs and 46% against the project-gated baseline. SmartTest runs 5 of 1,225 test-file runs, with 0 missed failures, 0 already failing and 0 cache violations. The window (47835e1d to cac765a1) is newer than the 0.9.0 one and contains no SmartTest or lockfile commits, which explains most of the difference from 0.9.0's 34.4 s.
• Observed execution (0.5.0 target): with traces, the same 12 commits select 8 test files instead of 5. Traces ruled out none and added 3 on c4ad92a4. That change adds a function and a type alias to growth/gift/operations.ts, units with no V8 name at the parent, so every test that loads the file counts as running them. Target not met.
• Speed (0.8.0 target): SmartTest 0.7.0 takes 6.3 s on the same window and the 1.0.0 candidate 6.6–6.9 s. Target not met. Each replayed commit is a new state, so the result cache cannot help, and the time goes to analysis and runner start-up.
• Analysis time: cold scan 1.73 s (3.6 s in 0.6.0), warm scan 0.28 s. Cold impact 1.81 s. Warm impact 0.48 s: 471 ms of analysis, of which Git change detection takes 316 ms, and the graph loads from .smarttest/state/graph.json in 14 ms.
• Full suite through SmartTest: 724 tests in 103 files pass in 4.3 s.

Caveats #

• LeadsIt's suites are already fast (about 5–8 s for everything), so the absolute saving per commit is seconds, not minutes. The percentage is what carries over to larger suites.
• The first run in a fresh checkout pays a cold start: 1.7–1.8 s with the 1.0.0 candidate (3.6 s with 0.6.0, 5.5 s before that), for parsing about 1,600 files and asking each runner for its file list. Both are cached afterwards, and the graph is persisted.
• Commits outside tested projects run nothing. That's correct, but it also means "gated" can be 0 ms on those commits, and the old hook didn't run anything for them either.
• Root package.json, lockfile and tsconfig changes run the full suite or the project's suite. That's deliberate: they're mandatory triggers.

Edit this page on GitHub