Set up Jimmy

Overview

Jimmy runs in the background on your PC. It reads the text of the window in front of you, transcribes speech near your mic, and keeps both as one searchable history with a blurred screenshot for each moment. When you say "Jimmy, …", it searches that history, shows you what it found, and answers out loud.

Jimmy has two halves:

• A Python program (ambient and jimmy) that captures, stores, searches and talks to a language model.
• An overlay: a small pill at the top of your screen, plus cards and answers, built with Electron. The Python program starts it for you.

Install with the setup

For Windows 10 (22H2) or 11, 64-bit. You don't need Python, Node, CUDA or admin rights.

1. Download the latest Jimmy-Setup-<version>.exe (about 170 MB) and its .sha256 file from the project's Releases page on GitHub (the View on GitHub button on the home page, then Releases).
2. Check the file in PowerShell, from the folder you downloaded it to. The result must equal the value in the .sha256 file:

   (Get-FileHash .\Jimmy-Setup-<version>.exe).Hash

3. Run it. The setup isn't code-signed yet, so Windows shows "Windows protected your PC": choose More info, then Run anyway. Pick a folder on any drive. It installs for your Windows account only.

First start

Jimmy opens its own setup the first time. It checks your PC (Windows version, memory, free space, GPU, mic and camera), then asks:

• Where to keep your data, on any drive.
• How much of the PC it may use: Lite (about 2.5 GB), Balanced (about 4.5 GB) or Full (about 11 GB, with local AI for cards and search by meaning).
• A model key, optional: a free Gemini or NVIDIA key for spoken answers. Keys typed here are stored encrypted with Windows DPAPI, readable only by your Windows account.

Then it downloads its own Python and a speech model into your data folder. NVIDIA GPUs get fast GPU speech; other PCs use the processor.

What goes on C:: only a 55-byte file that remembers where your data is, the shortcuts and the uninstall entry. Everything else is in the folders you picked.

Storage and limits

Open the timeline (Ctrl + Alt + T) and choose the Settings tab. It shows what Jimmy uses, and lets you set how many days to keep (30 by default) and a size cap (20 GB by default). The oldest whole days go first, never today. You can also change the profile, clear caches and choose whether Jimmy starts with Windows.

Uninstalling (Windows Settings → Apps) removes Jimmy and its Python, and asks before deleting your data.

From source: what you need

The packages and models take several gigabytes. Captures then take roughly 30–85 MB per active hour.

Get the code

On the project's GitHub page (the View on GitHub button on the home page), choose Code → Download ZIP and unzip it to C:\Jimmy. If you use git, clone it to the same place instead. A short path avoids Windows' path-length problems.

Open PowerShell and go to that folder. Every command on this page runs from there:

cd C:\Jimmy

Packages and overlay

1. Make a Python 3.12 environment and install the packages:

py -3.12 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt

If py -3.12 says it can't find Python 3.12, install it from python.org and open a new PowerShell window.

2. Build the overlay once (and again only if you change its code):

cd overlay; npm install; npm run build; cd ..

Local models

With Ollama installed and running, pull the two models Jimmy uses on your PC:

ollama pull qwen2.5:3b
ollama pull bge-m3

qwen2.5:3b answers one yes/no question per possible card. bge-m3 turns text into vectors, so "consulting application" can find a page titled "McKinsey Forward". Whisper's speech model downloads by itself on the first run.

Model key (optional)

Spoken answers come from a cloud model. Pick one provider. Without any key, Jimmy still works: each answer shows what search found instead.

setx NVIDIA_API_KEY "nvapi-your-key-here"

GEMINI_API_KEY=your-key-here
JIMMY_PROVIDER=gemini

OPENAI_API_KEY=sk-your-key-here
JIMMY_PROVIDER=openai

Check and run

.\.venv\Scripts\python.exe -m ambient doctor
.\.venv\Scripts\python.exe -m ambient run

doctor checks every part and asks you to speak for 3 seconds to prove the mic hears you. run starts capture and opens the overlay; quit with Ctrl + C or the pill's Quit. Only one ambient run can go at a time.

Just want to see it first? .\.venv\Scripts\python.exe -m ambient run --demo plays a spoken, scripted tour of about 3 minutes that uses real capture, search, model and voice.

Two one-time steps make it work better for you: say "Jimmy, eye calibration" (about 20 seconds, guided on screen) so you can ask without the name by looking at the screen, and "Jimmy, remember my face" so the curtain knows you.

What you can say

Start with "Jimmy,". After it answers, keep talking for 10 seconds without the name; "thanks" ends it. Say just "Jimmy" to think first: it answers "Yes?" and listens. To type instead, press Ctrl + Alt + Space.

It understands times like today, yesterday afternoon, between 2 and 3, the last 20 minutes and last Friday. Say "stop" to end a task that's running.

Everyday controls

Exclusions

Password managers, banking and payment sites, private browser windows and Jimmy's own windows are never captured. To add your own, create data\exclusions.txt in the Jimmy folder with one rule per line, by program, window title or address:

exe:mysecret.exe
title:payroll
url:internal\.example\.com

Command line

Everything here runs with .\.venv\Scripts\python.exe -m … from the Jimmy folder.

Ask Jimmy from another assistant (MCP)

python -m jimmy mcp is a small MCP server (Model Context Protocol, the standard way assistants connect to tools) with exactly one read-only tool, recall. Claude Desktop, VS Code and others can ask "what was I reading yesterday?" and get the evidence Jimmy would use, capped at 6,000 characters and marked as data. It can't press, type or change anything. In Claude Desktop's claude_desktop_config.json:

{
  "mcpServers": {
    "jimmy": {
      "command": "C:\\Jimmy\\.venv\\Scripts\\python.exe",
      "args": ["-m", "jimmy", "mcp"],
      "env": { "PYTHONPATH": "C:\\Jimmy" }
    }
  }
}

Optional extras

Jimmy runs without any of these. Each is a setting in ambient\config.py, whose comment says what to measure before leaving it on.

If something's off

To see what Jimmy is doing right now, press the pill's Status button or say "Jimmy, show status". The Status dock shows whether it is listening (and for how many seconds you can talk without the name), the camera, Jimmy's own CPU and memory with a one-minute graph, any update with its download progress and an Update now button, and capture and the AI model.

Ask Jimmy. "Jimmy, what's wrong with the camera?" makes it check itself and tell you what it found: its own setting, Windows' camera privacy, another app holding the camera, and whether the camera gives pictures. "You can use the camera, fix it" repairs what is Jimmy's own on the spot, without a restart. Jimmy never changes Windows' own privacy settings itself: for those it opens the right Settings page and says which switch to turn on.

Troubleshooting

Current limits

• Windows only, and built and measured on one laptop. The setup was tested on that laptop with its developer tools hidden and on fresh GitHub Windows machines, not yet on many real PCs.
• The setup isn't code-signed, so Windows warns before running it. Check the SHA-256 as shown above.
• A source install keeps everything: the retention and size limits apply to the installed app. From source, captures grow by roughly 30–85 MB per active hour until you forget a span.
• Hindi accuracy is not measured on real speech yet.
• Canvas apps and video give no text unless you install Windows OCR, and the vision models a free key can reach describe pictures less exactly than a window's text.
• Gemini 3.8 Flash on the free tier was often unavailable when measured; Jimmy falls back to 3.5 Flash-Lite and then NVIDIA.