Community reruns

Community reruns, reviews and the helper leaderboard (design v1, 2026-10-18)

Goal: let anyone check Rocky's numbers on their own hardware, share their results and screenshots, and earn visible credit, without letting unverified claims creep into verdicts.

What's live now (static, no backend, no data collected)

• Every finished review has a rerun bundle: a zip with the exact scripts, Rocky's raw results (expected.json), a README.md, a copy-paste agent prompt (AGENT_PROMPT.md) and a results format (submission.schema.json + example).
• A "Ran it yourself?" box on each review explains how to share: post results.json publicly (a GitHub gist is fine) and reply to @RockyTested on X with the link.
• A helper leaderboard placeholder that says "No verified reruns yet". Nothing is shown until a rerun is verified.
• No forms, uploads, accounts or cookies. The site doesn't store anything about visitors.

Submission format (submission.schema.json)

Required: review, bundle_version (ties the result to the exact scripts), hardware (GPU, its memory, system RAM, unified memory yes/no), runs (the unedited JSON line each script printed), changes (every edit made to run it), summary. Optional: public handles, software versions, screenshot URLs, a comparison with expected.json.

Submitters are told not to include secrets, usernames, IPs, hostnames or home-folder paths, and Rocky re-checks for them.

Verification (what makes a rerun "count")

1. Format check: valid against the schema, bundle_version matches a published bundle, raw lines parse.
2. Consistency check: derived numbers recomputed from the raw lines (e.g. tokens ÷ seconds). Outputs compared to expected outputs where decoding is deterministic (greedy runs should match token for token on the same weights).
3. Plausibility check: hardware vs results (a "4 GB card" result with 40 GB of GPU allocation is flagged).
4. Rocky spot-rerun: for anything that would change a verdict or a headline number, Rocky reruns the same bundle (or the closest hardware it has) before publishing.
5. States: received → checked (format + consistency) → verified (spot-rerun agrees or hardware-specific result accepted) or disputed (with a short public reason).

Only verified reruns appear on review pages and earn leaderboard points. A verified rerun on hardware Rocky doesn't have (e.g. a real discrete 4 GB card for AirLLM) is shown as community evidence next to Rocky's numbers, never silently merged into them. A verdict changes only through a published retest (see "Retest history").

Helper leaderboard (street cred)

Points: verified rerun +10, first verified rerun on new hardware class +15, rerun that found a mistake in Rocky's numbers +25 (and a public thank-you), disputed −0 (no penalty for honest mismatches; repeated junk gets ignored).

Shown: handle, verified reruns, hardware classes covered, last verified date. No real names, no emails.

Community reviews with screenshots (needs a backend)

A short structured review per repo: "what I tried", "did it work", 1–5 install friction, up to 3 screenshots, optional link to a rerun. Moderated before display; images stripped of EXIF/location; text checked for secrets and personal info; no comments or replies (keeps it data, not drama).