Benchmark
What we measured, and how
Every number here was measured on the development laptop and written down in the project's decision records (D1, D2, …) with the method behind it. Your results will vary with your hardware, your screen and your network.
The machine
A laptop with an AMD Ryzen 7 7840HS, an NVIDIA RTX 4050 Laptop GPU with 6 GB of memory (CUDA 12.4), Windows 11 and Python 3.12. It often runs on battery, where Jimmy halves its optional background work.
Capture and speech
| What | Measured | How |
|---|---|---|
| Screen frame grab (DXGI) | 7–14 ms | 1920×1080 frames pulled from Windows' desktop duplication. |
| Window text (UI Automation) | ~230 ms | A live Electron window: 317 nodes, about 4,800 characters. Capped at 0.6 s; a busier window hit 655 nodes and stopped at the cap. |
| Browser text, before and after waking accessibility | 24 → 317 nodes | Chromium hides its interface from screen readers until one asks. Jimmy sends the standard signal once per window (D3). |
| Speech to text (Whisper large-v3-turbo) | 4 s in 0.6 s | About 1 GB of GPU memory, English and Hindi only (D36). |
| Change gate | ~49 % skipped | A 160×90 grey image; capture when ≥ 0.25 % of pixels move. A one-message chat scroll moves 2–15 %, a cursor blink 0.01 %. The old 64-bit hash missed 7 of 8 scrolls (D18). |
| Face detection | 20 ms | YuNet at 640 px on one CPU thread, against 65 ms on OpenCV's default 16 threads (D38). |
Search and answers
| What | Measured | How |
|---|---|---|
| Hybrid search (keywords + meaning) | 0.5 s | Was 2.8 s. Most of the gap was Windows trying IPv6 for "localhost" first, about 2.4 s per new connection; every local call now uses 127.0.0.1 with kept-open connections (D38). |
| Same results after that speed-up | 43 of 43 | Read paths compared before and after on a frozen copy of the real database (D38). |
| First spoken word, Gemini chain | 0.8–1.8 s | Six spoken questions in English and Hindi, all answered by Gemini 3.5 Flash-Lite after 3.8 Flash failed. The first took 7.6 s while 3.8 timed out (D58). |
| First word, NVIDIA chat model | 1.0–1.4 s | nemotron-3-ultra; a whole answer in 2.0–2.5 s (D43). |
| Recall answers in the stage 3 check | 8 / 8 | Correct, with nothing invented, on recorded history (D22). |
The card gate
The goal is few cards, each one defensible. Stage 3 was closed on recorded real history against a checklist fixed before evaluating (D22).
| What | Measured | How |
|---|---|---|
| Replay of 1.18 h of real history | 5 → 1 card | Local rules found 5 candidates; the local model (qwen2.5:3b) let one through. The bar was ≤ 10 cards in any hour. |
| Card-decision precision | 0.83 | Against the consensus of two blind AI judges. The bar was 0.80. |
| Card decision time | ~1 s | One yes/no question to qwen2.5:3b on the GPU, 3.3 GB of memory (D20). A 7B model was slower and worse. |
The agent
130 real commands, frozen from the owner's own sessions, are run through the real pipeline (fast rules first, then the agent), and the first decision is scored. Eight whole tasks run on invented screens that change after each action.
| System | 130 commands | Notes |
|---|---|---|
| Before the agent (D41) | 71 % | Rules plus a single tool pick. |
| Agent (D42) | 124 / 130 (95 %) | Median 0.8 s per decision; two runs gave 124 and 125. |
| NVIDIA gpt-oss-20b (D43) | 120 / 130 | Median 1.4 s per step. |
| Gemini chain (D58) | 122 / 130 | 124 picked the right tool; median 1.5 s. |
Whole tasks: on 2026-11-01 Gemini first managed 4 of 8 in 143 s and NVIDIA 5 of 8 in 167 s, under the same conditions. Both failed "close this tab" the same way, and the cause was Jimmy, not the model: "yes" pressed a different tab from the one the cursor showed. After that fix and a second one, Gemini did 8 of 8 in 42 s.
Tools per step: 19 core tools, about 1,400 tokens, instead of all 54 at about 3,300 (D48). A two-step screen task takes 2 model calls instead of 4 (D49).
The installer
Measured on the development laptop with its developer tools hidden (Python, Node and CUDA taken off the path), installing to a separate drive letter (D61):
| What | Measured | How |
|---|---|---|
| Install | 20 s | The 172 MB setup, per user, no admin. |
| First start, Lite profile | 71–436 s | On the processor; the range depends on how much was already downloaded. |
| Left on C: | 67 bytes | The 55-byte pointer to your data and a 12-byte path file, besides shortcuts and the uninstall entry. |
The same steps run on fresh GitHub Windows machines for every change: install to D:, setup, self-test, update, what's left on C:, and uninstall.
Storage
Screenshots are 1280 px wide so they're readable when opened full size, about 80–88 KB each. That is roughly 30–85 MB per active hour, or 90–250 GB a year at 8 hours a day (D38). The text database is small beside the images.
Weak spots
- Gemini 3.8 Flash on the free tier timed out or answered "high demand" on 5 of 5 calls when measured; Jimmy now treats slow as down and moves on within 6 seconds.
- The hosted NVIDIA model sometimes returns an empty answer, once 1 in 5 in a benchmark and in runs on bad days. Jimmy retries, then falls back.
- RECALL cards are proven only synthetically. The recorded data had no real repeat of an earlier page to find.
- Hindi accuracy on real speech has not been measured.
- A "full working day" has not been captured end to end in one run; the longest recorded run was 66 minutes.
- One machine. None of this has been measured on a second PC yet.
Run them yourself
From the Jimmy folder. The checks need no network; the evaluations call your model provider.
.\.venv\Scripts\python.exe tests\test_stage1.py # capture, blur, store .\.venv\Scripts\python.exe tests\test_stage3.py # the card gate .\.venv\Scripts\python.exe -m ambient replay # cards over your own history .\.venv\Scripts\python.exe tests\eval_agent.py # 130 real commands (live) .\.venv\Scripts\python.exe tests\eval_trajectory.py # 8 whole tasks (live)