Trained Eye · system walkthrough

How Trained Eye works

You look at something for ten seconds and say what to watch for. Meta's models turn that into a few hundred labelled training images, a small open model is fine-tuned on them, and from then on your glasses speak up when it happens. This page follows one request through every device and model.

"Tell me if I leave the burner on."

The skill this becomes is burner_left_on. It waits until the burner has been on for 10 seconds before it alerts, so lighting it to cook never triggers it.

The map

Five places do the work

Your glasses only see, hear and speak. Your iPhone carries frames to your Mac, and your Mac runs the whole project: it calls Meta's models while teaching, sends training to a rented GPU on Modal, and decides when to alert while watching.

On you: glasses and iPhone Your Mac Meta Model API Modal GPU Other services
Meta glasses camera · mic · speakers Meta DAT SDK 1.0.1 capture light stays on iPhone Trained Eye app SwiftUI · decodes HEVC speaks the alerts Your Mac te serve · port 8765 teach flow (te teach) watcher trigger: 3 of 4 + hold data/: logs, cache, ledger Meta Model API Voice Transcribe Muse Wonder · SAM 3.1 Muse Image Muse Spark Modal 1 × H100 GPU fine-tuning (Axolotl) student on vLLM W&B · Hugging Face logs · base model camera frames spoken alert 1 frame / s alert text teach calls answers train · serve p(yes) logs
The blue arrows are the live path while watching: glasses to iPhone to Mac, the frame scored by the fine-tuned model on Modal, and the probability back to your Mac, which decides whether to speak. The iPhone and the Mac talk over your home Wi-Fi (a WebSocket on port 8765); the glasses and the iPhone over Bluetooth and Wi-Fi through Meta's Wearables SDK.

Step 1 · Teach

Ten seconds of looking becomes a training set

Two things come in: your words and your view. The words become a precise skill spec, and the view becomes a few hundred training images. Every call goes through one client on your Mac that caches answers and keeps a spend ledger, so a rerun never pays twice.

YOUR WORDS YOUR VIEW Spoken request glasses mic Voice Transcribe speech to text Policy check on the Mac, free Muse Wonder writes the skill spec Skill spec read back, you confirm 10 s look glasses camera Keyframes 20 to 24 sharp frames Muse Image edits: lit and not Muse Spark labels, keeps agreeing Training set Axolotl JSONL, 512×512 SAM 3.1 box around the burner crops of the burner 3 keyframes what to draw what to ask
The spec is the hinge. Muse Wonder reads your words and three keyframes and writes what to watch ("the stove burner"), when to alert ("the burner is lit with no pan on it"), near-misses that must not count, the sentence to speak, a cooldown and the 10 second hold. Muse Image then edits your own keyframes into about 150 burner-on and 150 burner-off pictures under different light, angles and blur, with no people. Muse Spark labels each one, and a picture stays only when Spark agrees with what it was meant to show.

Step 2 · Train

A small open model learns the one question

The model that ends up watching is Muse Glimmer 30B, an open-weight Meta model, fine-tuned with Meta's own QLoRA recipe on one H100 GPU rented from Modal. Only adapter weights train; the vision tower stays frozen. A two-step smoke run comes first, and its timing prices the real run before you agree to it.

Training bundle set + session 1 clips te train Modal volume data + base weights from Hugging Face Fine-tune QLoRA · Muse Glimmer 30B 1 × H100 · Axolotl Export merge the adapter same answers on 50 Student endpoint vLLM on an H100 te serve-student smoke run first (2 steps) prices it · logs to W&B te calibrate: session 2 clips scored by the student set the threshold (at most 1 false alarm an hour)
Session 1 clips join the synthetic set for the "mixed" run; session 2 is never trained on, only used to calibrate the threshold; session 3 is the test set for the evaluation. The endpoint is protected by a key only your Mac holds, and it is billed only while a container is up.

Step 3 · Watch

One alert, second by second

While you wear the glasses, every frame of the burner gets a probability from the student. Your Mac only speaks when three of the last four frames agree, the burner has stayed on through its 10 second hold, and Muse Spark confirms the frame.

threshold 0 s 2 4 6 8 10 12 14 16 18 20 p(yes) hold: 10 s, from the first frame over 2 s: 3 of the last 4 over, so it is on 5 s: one under, still on 10 s: Spark checks the frame: yes glasses say "The burner is still on." cooldown: no repeat until it ends
Each bar is one frame (the iPhone sends one a second). A frame under the threshold never fires an alert, but a single dip does not restart the hold either. Had you turned the burner off at 7 seconds, nothing would have been said. The fridge works the same way with a 30 second hold and the tap with 20, so opening the fridge to grab milk stays silent. If Spark says no, the skill waits a minute and the hold starts again.

Sample flow

One evening with the burner skill

  1. Amaan opens Trained Eye on the iPhone, connects to the Mac and the glasses.

    iPhone + glasses: the app registers with the Meta AI app through Meta's Wearables SDK; the Mac runs te serve and the app checks it over Wi-Fi.

  2. On the Teach screen, he holds to talk: "Tell me if I leave the burner on," then looks at the stove for ten seconds as it goes from off to lit.

    Glasses mic and camera, iPhone: the request is recorded over Bluetooth; the capture goes to the Mac at three frames a second, with the audio.

  3. The phone shows the price of teaching it; he taps yes.

    Mac: nothing is paid before that yes. Anything over $15 always asks again.

  4. The phone reads back "I'll watch the front burner of the stove and tell you when the burner is lit with no pan on it. I'll say: The burner is still on." He confirms.

    Meta Model API: Voice Transcribe, then Muse Wonder writes the spec; the local policy check ran first, for free.

  5. The Mac builds the training set while he records a few minutes of normal cooking for testing.

    Meta Model API: SAM 3.1, Muse Image and Muse Spark. Mac: keyframes, crops and the dataset in data/skills/burner_left_on/.

  6. He starts training and goes back to his evening.

    Modal: te train fine-tunes Muse Glimmer 30B on one H100, then te serve-student puts it online. Mac: te calibrate sets the threshold from his session 2 clips.

  7. Days later he lights the burner for tea and walks off to answer the door.

    Glasses to iPhone to Mac: frames stream in; the student scores each one on Modal; the trigger waits out the 10 second hold.

  8. He hears "The burner is still on." in his glasses.

    Meta Model API: Muse Spark confirmed the last frame. iPhone: speaks it through the glasses' speakers over Bluetooth. The cooldown keeps it from repeating every second.

Who runs what

Every model and device in one table

JobRuns onModel or tech
See, hear and speakMeta glassesMeta Wearables DAT SDK 1.0.1; camera stream in HEVC; audio over Bluetooth
Carry frames, teach, speak alertsiPhoneTrained Eye app: SwiftUI, VideoToolbox decoding, a WebSocket to the Mac, speech synthesis
Backend, teach flow, watcher, triggerYour MacPython 3.12 te command; Starlette server on port 8765; cache and spend ledger
Speech to textMeta Model APImuse-voice-transcribe-1.0
Write the skill specMeta Model APImuse-wonder-1.0, structured output
Find the burner in each keyframeMeta Model APIsam-3.1
Generate training variationsMeta Model APImuse-image-1.0 edits of your keyframes
Label images, confirm alertsMeta Model APImuse-spark-1.3
Fine-tune the studentModal, 1 × H100meta-models/Muse-Glimmer-30B with Axolotl QLoRA (Meta's recipe)
Score frames while watchingModal, H100Your fine-tuned Glimmer on vLLM; the answer token's probability
Training logs, base weights, releaseW&B, Hugging FacePrivate W&B project; uploads only after you decide to publish

Step 4 · Prove it

Evaluation and the guard rails

How it is measured

te replay runs your labelled test clips through watch mode, and make eval scores them: frame accuracy, events caught on time (within the skill's window after its hold), false alarms per hour and dollars per hour. Experiment 1 compares real, synthetic and mixed training; Experiment 4 compares the fine-tuned student with base Glimmer and with Spark on every frame.

Privacy

Watch mode never stores frames, only times and scores. The glasses' capture light is never hidden. Generated images are never published, and faces are blurred in anything that is.

What it will not learn

No skills about specific people, bodies or health, covert watching, politics or war. The policy check refuses those on your Mac before any paid call. And the burner skill is a reminder, never a smoke alarm.