Local macOS companion
A signed menu-bar app owns Screen Recording, Screen Content, Accessibility, overlays, and visual capture.
A Codex-native visual context layer for macOS
Point at anything. Let Codex understand it.
ThisThat turns a Control+Shift highlight into structured visual context: crops, coordinates, source URLs and file paths when available, OCR, accessibility metadata, visual search, pixel probes, and change detection for Codex.
From TipTour to Codex
ThisThat is built from TipTour's strongest idea: the pointer/highlighter as an AI context primitive. TipTour explored a Gemini-style pointer assistant and required a Gemini API key. ThisThat keeps the pointing layer, removes the Gemini dependency, and routes AI work through Codex.
| Area | TipTour | ThisThat |
|---|---|---|
| AI route | Gemini / Gemini Live | Codex / ChatGPT |
| API key | Gemini API key required | No Gemini API key |
| Best retained idea | Pointer and highlight context | Pointer and highlight context optimized for Codex ingestion |
| Action routing | App-owned assistant behavior | Codex chooses Chrome or Computer Use after ThisThat explains the target |
This project is independent and is not affiliated with Google, Gemini, DeepMind, OpenAI, or Codex. It preserves attribution to TipTour and its upstream lineage.
Context, not control
ThisThat does not replace Codex's browser or desktop control tools. It gives Codex a precise answer to one question: what does the user mean by "this"?
MCP bridge
Codex launches the MCP daemon over stdio. The daemon talks to the app over a token-protected localhost bridge.
Visual packets
Codex receives focused crops, OCR, grids, coordinates, hashes, and routing hints only when needed.
Built for visual work
ThisThat is most useful when a target is visible but hard to explain precisely in text.
Games and moving UIs
Sample a changing scene, inspect HUD elements, compare before/after frames, and wait for visual state changes.
Fusion 360 and CAD
Use lossless crops, coordinate grids, and pixel-level inspection for tiny controls, edges, panels, and canvases.
Images and design tools
Point at artifacts, colors, visual defects, image regions, or interface elements that have no accessible label.
Browser and native workflows
Let ThisThat identify the target, then let Codex route the action through Chrome or Computer Use.
Install locally
ThisThat currently builds from source. A signed local app is important because macOS privacy permissions are tied to the app identity.
./script/build_and_run.sh --verify
./script/open_permission_setup.sh --wait 300 --interval 2 --restart-before-request --restart-after-ready
./scripts/install-thisthat-mcp.sh --force --app-path "$PWD/build/DerivedData/Build/Products/Debug/ThisThat.app"
No Gemini key. No constant screenshot stream.
ThisThat captures visual context when Codex asks or when you paint a highlight. The goal is compact, inspectable context that helps Codex use the right tool for the job.