Skip to content

feat: add MacOS support#81

Open
not-matthias wants to merge 12 commits into
mainfrom
cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos
Open

feat: add MacOS support#81
not-matthias wants to merge 12 commits into
mainfrom
cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos

Conversation

@not-matthias
Copy link
Copy Markdown
Member

No description provided.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Jun 2, 2026

Merging this PR will degrade performance by 93.19%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 5 improved benchmarks
❌ 19 regressed benchmarks
✅ 201 untouched benchmarks
🆕 1 new benchmark

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime iterative fibo 15 312 ns 369,169 ns -99.92%
WallTime iterative fibo 20 360 ns 367,274 ns -99.9%
WallTime short body 1.7 µs 377 µs -99.55%
WallTime short body 1.7 µs 379.6 µs -99.55%
WallTime short body 1.7 µs 373.4 µs -99.54%
WallTime short body 1.8 µs 373.7 µs -99.53%
WallTime short body 1.8 µs 373.8 µs -99.53%
WallTime short body 1.8 µs 376.5 µs -99.52%
WallTime fibo 10 2.1 µs 372.6 µs -99.43%
WallTime fibo 15 21.5 µs 392.1 µs -94.51%
WallTime recursive fibo 15 22.2 µs 396.5 µs -94.41%
WallTime end 35.5 µs 432 µs -91.79%
WallTime long body 214.3 µs 604.3 µs -64.53%
WallTime long body 211.7 µs 596.8 µs -64.52%
WallTime long body 217.2 µs 601.9 µs -63.91%
WallTime long body 217.3 µs 601.5 µs -63.87%
WallTime recursive fibo 20 239.5 µs 613.1 µs -60.94%
Simulation test_iterative_fibo_10 299.9 µs 547.2 µs -45.2%
WallTime wait 1ms 1 ms 1.4 ms -28.18%
Memory wait 1ms 16 B 10 B +60%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos (ad2b2e7) with main (4dae798)

Open in CodSpeed

@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch 2 times, most recently from f0b1349 to 3982c34 Compare June 2, 2026 10:36
Add benches/macos.bench.ts guarded with describe.skipIf(!darwin) so it
only runs on macOS, and wire the walltime-macos-test CI job to execute it
via a direct `node vitest.mjs bench --run macos` invocation.
@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch from 32c3b31 to 1d75a61 Compare June 3, 2026 13:23
@not-matthias not-matthias marked this pull request as ready for review June 3, 2026 15:22
@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented Jun 3, 2026

Greptile Summary

This PR adds macOS walltime support to the vitest and tinybench plugins by introducing per-iteration BENCHMARK_START/BENCHMARK_END FIFO markers in WalltimeRunner and a single per-run marker pair in the tinybench BaseBenchRunner, backed by a submodule bump to expose currentTimestamp and addMarker. A new codspeed-walltime-macos CI job is added that targets the sip-resign-exec-redirect runner, and a dedicated macOS-only bench file guards the darwin benchmark with describe.skipIf.

  • vitest-plugin/src/walltime/index.ts: __codspeed_root_frame__ now calls finishRound (2 FIFO round-trips) per iteration — this is new behaviour on the existing Linux codspeed-walltime path, not just macOS.
  • tinybench-plugin/src/shared.ts: sendRunMarkers emits one marker pair after all tasks complete (correctly walltime-gated) to avoid the per-iteration FIFO overflow documented in its inline comment.
  • ci.yml: Removes the old walltime-macos-test CI job; the macOS benchmark is now owned by codspeed.yml.

Confidence Score: 4/5

Safe to merge for the macOS-only path; the existing Linux walltime benchmarks now silently emit per-iteration FIFO markers that were not there before, matching a pattern the tinybench plugin explicitly removed to prevent CI timeouts.

The finishRound change in WalltimeRunner adds two blocking FIFO round-trips per tinybench iteration on every walltime run — including the four existing bench files on the Linux codspeed-walltime job. The tinybench plugin's own inline comment documents that this identical per-iteration pattern previously overwhelmed the runner and timed out CI, and that plugin was specifically refactored to avoid it. The macOS job uses a purpose-built runner branch, but the Linux job uses the existing runner with no changes and has no guard to skip the new markers.

packages/vitest-plugin/src/walltime/index.ts — the finishRound call inside codspeed_root_frame affects all walltime benchmark runs, not only the new macOS benchmark.

Important Files Changed

Filename Overview
packages/vitest-plugin/src/walltime/index.ts Adds per-iteration FIFO markers (BENCHMARK_START/END) in codspeed_root_frame; this is new behaviour on the Linux walltime path and contradicts the design rationale documented in the tinybench plugin.
packages/tinybench-plugin/src/shared.ts Adds sendRunMarkers emitting a single BENCHMARK_START/END pair bracketing the whole run; correctly guards on walltime mode and documents the per-iteration marker problem.
packages/tinybench-plugin/src/walltime.ts Switches warmup call from task.warmup() to task.warmupSync() for the sync path; cosmetic blank-line addition.
packages/tinybench-plugin/tests/index.integ.test.ts Adds currentTimestamp and addMarker mocks required by the new sendRunMarkers code path.
packages/vitest-plugin/benches/macos.bench.ts New macOS-only benchmark file; uses describe.skipIf(!isMacOS) to guard the darwin-only fibo bench.
.github/workflows/codspeed.yml Adds codspeed-walltime-macos job; still contains the diagnostic resign+exec step and a mutable branch:sip-resign-exec-redirect runner-version (both already flagged in previous threads).
.github/workflows/ci.yml Removes the old walltime-macos-test job; no new logic introduced.
packages/core/src/native_core/instruments/hooks Submodule bumped to b9ddb5b; new commit adds currentTimestamp and addMarker FIFO support required by this PR.

Sequence Diagram

sequenceDiagram
    participant TB as Tinybench Task.run()
    participant CF as __codspeed_root_frame__
    participant FN as benchmark fn()
    participant FIFO as Runner FIFO

    TB->>FIFO: startBenchmark() [ACK]
    loop N iterations
        TB->>CF: call fn
        CF->>CF: "roundStart = currentTimestamp()"
        CF->>FN: fn()
        FN-->>CF: result
        CF->>FIFO: addMarker(BENCHMARK_START, roundStart) [ACK]
        CF->>FIFO: addMarker(BENCHMARK_END, roundEnd) [ACK]
        CF-->>TB: result
    end
    TB->>FIFO: stopBenchmark() [ACK]
    TB->>FIFO: setExecutedBenchmark(pid, uri) [ACK]
Loading

Reviews (9): Last reviewed commit: "fix(ci): only run the macOS-only bench o..." | Re-trigger Greptile

Comment on lines +93 to +97
if (
result !== null &&
typeof result === "object" &&
typeof (result as PromiseLike<unknown>).then === "function"
) {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The multi-clause thenable check should be extracted into a named const per the project's rule that non-obvious or multi-clause boolean conditions must be named with an is*/has* prefix.

Suggested change
if (
result !== null &&
typeof result === "object" &&
typeof (result as PromiseLike<unknown>).then === "function"
) {
const isThenable =
result !== null &&
typeof result === "object" &&
typeof (result as PromiseLike<unknown>).then === "function";
if (isThenable) {

Rule Used: Extract any non-obvious or multi-clause boolean co... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment thread packages/vitest-plugin/benches/macos.bench.ts
Comment on lines +111 to +118
- name: Diagnose resign+exec on this runner
run: |
sw_vers; uname -m; sysctl -n hw.memsize hw.ncpu
cp /bin/sh /tmp/sh_copy
/usr/bin/codesign -s - -f /tmp/sh_copy
lipo -info /tmp/sh_copy
/tmp/sh_copy -c 'echo RESIGNED_ARM64E_OK; uname -m'
echo "exit=$?"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Diagnostic step left in production workflow

The "Diagnose resign+exec on this runner" step copies /bin/sh, codesigns it, and re-executes it — this is clearly debugging scaffolding for the SIP/resign investigation. Leaving it in the permanent workflow adds noise to every future CI run and exposes internal implementation details. It should be removed once the resign-exec behaviour is confirmed working.

working-directory: packages/vitest-plugin
run: pnpm turbo run bench --env-mode=loose --filter=@codspeed/vitest-plugin
mode: walltime
runner-version: branch:sip-resign-exec-redirect
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Mutable branch reference for runner-version

runner-version: branch:sip-resign-exec-redirect points to a live branch that can be force-pushed or rebased at any time. If the branch tip changes between two benchmark runs, the profiling behaviour changes silently, making measurement results non-comparable across commits. Consider pinning to a specific commit SHA or tag once the feature branch is stable, e.g. runner-version: sha:<commit>.

@not-matthias not-matthias force-pushed the cod-2720-bump-instrument-hooks-in-codspeed-node-to-support-macos branch 2 times, most recently from 888eaad to 9925d1a Compare June 4, 2026 13:17
not-matthias and others added 3 commits June 4, 2026 15:55
Per-iteration markers made marker volume proportional to the iteration
count (millions for fast benches): every addMarker call is a synchronous
FIFO round-trip to the runner, which overwhelmed it and timed out the
walltime CI job. Buffer round timestamps in JS instead and flush one
BENCHMARK_START/BENCHMARK_END pair bracketing the measured rounds after
the run.

Also exclude warmup from markers (the wrapper previously emitted markers
during warmup rounds) and fix the sync path to use warmupSync(): the
un-awaited async warmup() never actually ran before runSync().

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move marker emission from per-iteration (inside the task fn) to a single
BENCHMARK_START/BENCHMARK_END pair bracketing the whole bench run in
setupBenchMethods. Each addMarker call is a synchronous FIFO round-trip
to the runner, so per-iteration markers scaled with iteration count
(millions for fast benches), overwhelmed the runner, and timed out the
walltime CI job — while also inflating the measured samples.

Emission is gated to walltime mode so the analysis runner sharing
setupBenchMethods is unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pass the vitest file filter through turbo so the macOS job runs just
macos.bench.ts. The full vitest-plugin suite already runs on the linux
walltime job, and uploading the same benchmark twice for one commit
fails CodSpeed processing ("Found the same benchmark multiple times").

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant