Skip to content

model-probe

Why content-judging

  • /v1/models exposes no capability metadata on OpenAI-compatible gateways.
  • Status codes LIE (all observed live against DashScope):
  • text-only gateways silently DROP image blocks → 200 with an honest refusal ("no image was provided"), EMPTY content, or a confident hallucination ("white background, meerkat");
  • vision models can 400 a malformed probe image (1x1 px) while being fully vision-capable.
  • So the script sends the bundled assets/probe-image.png (ground truth: indigo/purple field, white "crow" wordmark, white crow/bird silhouette) and records the model's RAW answer.

1. Run the probe (evidence only — it decides nothing)

uv --project ~/.agents/skills/model-probe run python ~/.agents/skills/model-probe/scripts/probe_models.py
uv --project ~/.agents/skills/model-probe run python ~/.agents/skills/model-probe/scripts/probe_models.py --provider alibaba

The skill's pyproject pins crow-cli>=0.1.33, so Config.load reads the same ~/.agents/crow/config.yaml + .env the agent uses — no environment hunting. Each probe is one tiny request per model (max_tokens 64). Output columns: model, provider, http status, heuristic suggestion, raw answer.

2. YOU judge the modalities

The suggested column is a dumb heuristic (correct description → [text, image]; refusal/empty/hallucination/400 → [text]). Treat it as a first draft and sanity-check it:

  • "hmm, this doesn't seem right" → web_search the model's spec before applying (e.g. "deepseek-v3.2 vision", "qwen3.8-max video understanding"). Providers document modality support even when the API doesn't.
  • HTTP 429/5xx/connection errors (? suggestion) are NOT capability signals — leave those models untouched or rerun.
  • Image-GEN models (qwen-image-, wan) 400 on chat completions → [text] is correct: they must never receive routed images.
  • Add video/audio by hand for models known to take them natively (e.g. qwen3.x-max: [text, image, video]) — the probe only tests images.

3. Apply your verdicts with the skill's venv

apply_modalities() is line surgery: comments and formatting in config.yaml survive, reruns are idempotent. Feed it YOUR decided lists, not the raw suggestion column:

uv --project ~/.agents/skills/model-probe run python - <<'EOF'
import sys
from pathlib import Path
sys.path.insert(0, str(Path.home() / ".agents/skills/model-probe/scripts"))
from probe_models import apply_modalities

updates = {
    "qwen3.8-max-preview": ["text", "image", "video"],
    "deepseek-v3.2": ["text"],
    # ... one entry per model you probed and judged
}
p = Path.home() / ".agents/crow/config.yaml"
new, changed = apply_modalities(p.read_text(), updates)
p.write_text(new)
print(f"changed {changed} models")
EOF

Verify with crow-cli models (modality + sampling columns).

What the values mean

modality is a list of text | image | audio | video (per-model, in config.yaml). Default [text, image] = assume vision-capable until proven otherwise. [text] changes routing: image blocks get stripped / routed to a same-provider fallbacks: entry instead of hard-failing. temperature / reasoning_effort stay human choices — never write them from this skill.