How scores work

Rocky Tested scorecard (v1)

Every review gets six sub-scores (0-100) that normally roll up to one Rocky Score (0-100) and a letter grade. Sub-scores come only

from measured data in results/<slug>.json; a sub-score with no measurement is null and its weight is redistributed

(never guessed). Per-review data: data/scores/<slug>.json. Category weights: data/scorecard-weights.json.

| Sub-score | What it measures | How it's computed |

|---|---|---|

| claim | Did the headline claim hold? | share of claim checks passed (counts, accuracy vs ground truth, claimed speed/memory reached). Exact-match claims: 100 hit, 50 partly, 0 miss. |

| performance | Speed / quality vs a baseline on the category packet | packet score (0-100) or speed ratio vs baseline mapped log-scale: >=1x baseline = 100, 10x slower = 50, 100x slower = 0 |

| install | Install friction | packets/install-friction.json rule: start 100, -10 per manual fix, -25 fails as written, -5 per 5 min beyond 5, -10 undeclared accounts/keys |

| reliability | Same result every time? | pass rate across repeats x (1 - coefficient of variation capped at 0.5) ; single run = capped at 60 |

| docs | README matches reality | 100 minus 20 per README instruction that was wrong/missing during the test |

| safety | Safe to run | 100 minus 30 per finding (exfil, undisclosed network calls, curl-pipe-bash installers, secret reads); scanners: false-positive/negative rates |

Rollup: score = sum(w_i * s_i) / sum(w_i over non-null s_i), rounded. Grades: A+ >=95, A >=90, A- >=85, B+ >=80, B >=75, B- >=70, C+ >=65, C >=60, C- >=55, D >=45, F <45.

Verdict link (guide, not automatic): REAL needs claim >=80; HALF-REAL claim 40-79; VAPOR claim <40. The verdict is still a written human call.

Confidence: high (claim test + repeats + install), medium (claim test, single run), low (smoke only). Low-confidence scores show as "prelim".

Grade pending A/B

Usefulness must be measured before the rollup becomes an official score or letter grade. If the overall verdict is

UNPROVEN, v2.works.verdict is unproven, or the usefulness text starts with Unproven, the page shows

Grade pending A/B instead of a grade or filled score dial. Measured sub-scores still appear.

In data/scores/<slug>.json, use "grade_status":"pending_ab", set score and grade to null, and retain any

measured-only rollup as partial_score. A partial score is context for the completed sub-scores, not a published Rocky

Score and not a basis for ranking the review.

Per-review JSON (data/scores/<slug>.json)

```json

{"slug":"...","version":1,"subscores":{"claim":90,"performance":null,"install":80,"reliability":60,"docs":80,"safety":100},

"score":82,"grade":"B+","confidence":"medium","basis":{"claim":"818 SKILL.md vs 817 claimed"},"tags":["skills","security"],"keywords":["..."]}

```

Workstream B renders these (badge, sort, filters). Tags: lowercase, from category + measured facts (e.g. count-verified, slow-but-works).

Method V2: four verdicts + Safety hard-cap

Method V2 reviews add four plain-English verdicts on top of the claim verdict. Each is a written human call

backed by a data file; the renderer never computes or defaults one. They live in data/verdicts.json under the

slug's v2 key — values yes / mixed / no / unverified, each with a one-line why:

| Verdict | Question it answers | Backing data |

|---|---|---|

| Works | Did it do the thing when Rocky ran it? | results/<slug>.json runs (A/B for usefulness, per docs/method-v2.md) |

| Safe | Safe to install and run (no exfil, no undisclosed network calls, no secret reads)? | safety checks in results/<slug>.json |

| Claims honest | Does the pitch match what was measured? | the claim ledger in data/claims/<slug>.json |

| Provenance | Does the repo's history check out (age, authorship, bursts, funnels)? | data/provenance/<slug>.json |

Rollup with Safety hard-cap: the four verdicts don't average into one number. Safe: no is a hard cap — the

overall Rocky Score is capped at 44 (grade F) and the review leads with the safety finding, no matter how well it

works or how honest the claims are. Safe: unverified caps confidence at medium. The other three verdicts stay

independent: a repo can be Works: yes and Claims honest: no at the same time, and the page says both.

A verdict that hasn't been assessed renders as "Not assessed" — it is never guessed from the others.