Video → Frames → ROI Crops

Why frames

• ACP has no video content block (v1 or v2); MCP has no Video type. Claude and GPT accept text + images (+ PDF for Claude). Only qwen3.x-max on DashScope takes video_url natively, and crow's pipeline is image-based anyway.
• Frames + crops work with EVERY vision model and with crow's existing pipeline: read_image_file persists the image into the session DB and hydrates it as a base64 data URL at LLM send (max dim capped at 1568px).

1. Get a video

A. Playwright recording via browser_run_code_unsafe (verified)

recordVideo can only be set at context creation, so spin up a NEW context from the managed page's browser. This snippet is verified end-to-end:

async (page) => {
  const browser = page.context().browser();
  const dir = '/tmp/crow-video';
  const ctx = await browser.newContext({
    recordVideo: { dir, size: { width: 800, height: 600 } },
  });
  const p = await ctx.newPage();
  await p.goto('https://example.com');   // do the thing you want captured
  await p.waitForTimeout(1000);
  const video = p.video();
  await p.close();                        // MUST close page to finalize video
  const dest = dir + '/demo.webm';
  await video.saveAs(dest);
  await ctx.close();
  return dest;
}

Gotchas (all hit in practice): - The run_code VM is sandboxed ESM: NO require, NO dynamic import. Don't touch fs — Playwright creates the dir itself; verify the file from terminal. - saveAs only works after p.close() (the webm is finalized on close). - Playwright records WebM/VP8 only. ffmpeg reads it fine. - Omit size to match the viewport.

B. --video on the playwright MCP server

If ~/.agents/crow/config.yaml has --video in the playwright mcpServers args, the managed context records automatically; grab the file with page.video().saveAs(...) before the session ends. Needs a fresh agent session to pick up (config is read at session start).

C. Screen / webcam via ffmpeg

ffmpeg -y -f x11grab -i :0 -t 5 screen.mp4            # 5s of the desktop
ffmpeg -y -f v4l2 -i /dev/video6 -t 5 cam.mp4         # webcam (index 6 on this box)

2. Split into frames

mkdir -p frames
# uniform sampling, N frames per second (1 fps is a good default)
ffmpeg -y -v error -i video.webm -vf fps=1 frames/f_%03d.png
# exact frame count: duration from ffprobe, fps = count/duration
ffprobe -v error -show_entries format=duration -of csv=p=0 video.webm
# only visual change points (UI transitions, page loads)
ffmpeg -y -v error -i video.webm -vf "select='gt(scene,0.3)'" -vsync vfr frames/scene_%03d.png
# one frame at a timestamp
ffmpeg -y -v error -ss 1.5 -i video.webm -frames:v 1 frames/t1.5.png

• -vsync vfr is REQUIRED with the scene select, or you get duplicated frames.
• Scene threshold: 0.3 for real cuts; static UI content needs 0.05–0.1 (a page reload scored ~0.1 here; at 0.3 you get NOTHING on a static page).
• Feed the vision model 4–12 frames, named in order, and ask temporal questions ("what changed between f_003 and f_007?"). Downscale huge frames first (-vf scale=1280:-2) — the 1568px cap applies at send time anyway.

3. Crop regions of interest

# ffmpeg: crop=W:H:X:Y
ffmpeg -y -v error -i frames/f_001.png -vf crop=400:300:200:150 roi.png
# ImageMagick (note +repage, or the crop keeps the old canvas offsets)
convert frames/f_001.png -crop 400x300+200+150 +repage roi.png

Finding X/Y: - Best: browser_evaluate → el.getBoundingClientRect() on the element of interest while the page is live; reuse those coords on same-size frames. - Otherwise eyeball the full frame with read_image_file and crop generous margins; re-crop tighter if the first pass misses. - Crop BEFORE sending, not after: a 4K frame downscaled to 1568px makes small UI text unreadable, but a tight crop of the same text is crisp.

4. Analyze

read_image_file each frame/crop. It persists the image into the session DB and the model sees it as an image block. For flows, send several frames in one turn in chronological order and ask what changed.

Environment notes

• ffmpeg is required (sudo apt install -y ffmpeg); verified v6.1.1 on this box. ImageMagick convert also present as a crop fallback.
• Verified test artifact layout: record → /tmp/crow-video/*.webm, frames → frames/, crops → roi*.png, then read_image_file.