{"slug":"mukul975-cyber-skills","kind":"claim","status":"published","tier":"claim","title":"Anthropic-Cybersecurity-Skills: are there really 817 skills across 6 frameworks?","category":"data-rag/security","topic":"security","repos":[{"repo":"mukul975/Anthropic-Cybersecurity-Skills","stars":34620,"license":"Apache-2.0"}],"claims":[{"claim":"817 structured cybersecurity skills for AI agents, mapped to 6 frameworks (MITRE ATT&CK, OWASP, NIST and more)","source":"https://x.com/tom_doerr/status/2109535383188799957"}],"plan":"Count every SKILL.md in the repo, check each has valid YAML frontmatter with a name and description, and count how many actually name a security framework. Pure static read, no code run.","who_its_for":"Security folks eyeing a big skill bundle for their coding agent.","explainer":"A large bundle that claims 817 cybersecurity skills for AI agents, each tied to frameworks like MITRE ATT&CK and OWASP. Rocky counts them and checks the framework claim, rather than taking the number on faith.","measure":["How many SKILL.md files exist vs the claimed 817","How many have valid frontmatter","How many name a real framework","Which frameworks appear and how often"],"notes_so_far":["Method: clone at 54a7988 (v1.3.0), static scan only. 'Named framework' means the skill text mentions a known framework (MITRE ATT&CK, OWASP, NIST, CIS, D3FEND, PCI DSS, ISO 27001, and others)."],"sneak_peek":["818 skill folders and 818 SKILL.md at commit 54a7988; all 818 frontmatters parse. 809 agent.py and 282 process.py, every one byte-unique; median agent.py is 182 lines.","Frontmatter framework coverage: ATT&CK 806, NIST CSF 805, D3FEND 139, AI RMF 97, F3 94, ATLAS 93 -- within one of the README's claimed numbers on every field.","ID validity: 290 of 292 distinct ATT&CK IDs are valid and current in v20.0 (2 deprecated ICS IDs); 52 of 52 NIST CSF IDs valid; 22 of 22 ATLAS IDs valid.","Anatomy mismatch: the documented layout (standards.md / workflows.md / process.py / template.md) exists in only 34-43% of skills; the real dominant layout (agent.py + api-reference.md, 99%) is undocumented.","Prompt-injection scan over all 818 SKILL.md found no live injection aimed at the reading agent; 11 files carry vendor curl|sh install one-liners an agent could run verbatim.","The '308 runtime pip installs' lead was wrong in the interesting direction: 308 scripts mention pip install in help text, zero invoke it at runtime."],"charts":[{"type":"bars","title":"Claimed vs counted","unit":"skills","note":"Rocky counted every SKILL.md and checked its frontmatter.","groups":[{"label":"Skills","bars":[{"name":"Claimed","value":817,"verdict":"neutral"},{"name":"Found","value":818,"verdict":"real"},{"name":"Valid frontmatter","value":818,"verdict":"real"},{"name":"Name a framework","value":817,"verdict":"real"}]}]},{"type":"scoreboard","title":"Framework mentions across the skills","unit":"","note":"How many skills reference each framework (a skill can map to several). 11 distinct frameworks appeared; claim said 6.","rows":[{"label":"NIST","value":817,"verdict":"teal"},{"label":"CIS","value":270,"verdict":"teal"},{"label":"MITRE ATT&CK","value":255,"verdict":"teal"},{"label":"D3FEND","value":141,"verdict":"teal"},{"label":"OWASP","value":110,"verdict":"teal"},{"label":"PCI DSS","value":64,"verdict":"teal"},{"label":"ISO 27001","value":14,"verdict":"teal"}]}],"gallery":[],"results":{"jobs":[{"id":14,"title":"Cyber-Skills: verify the 817-skills / 6-frameworks claim by counting + validating every SKILL.md","kind":"claim","status":"done","started":"2026-10-18T04:29:26Z","finished":"2026-10-18T04:29:32Z","wall_seconds":6,"install_friction":null,"artifacts":[]}],"smoke_only":false,"ingested":"2026-10-18T18:34:11Z"},"last_test_finished":"2026-10-18T04:29:32Z","score":{"slug":"mukul975-cyber-skills","version":1,"subscores":{"claim":100,"performance":null,"install":null,"reliability":60,"docs":null,"safety":null},"weights":{"claim":35,"performance":25,"install":15,"reliability":15,"docs":5,"safety":5},"score":88,"grade":"A-","confidence":"medium","basis":{"claim":"818 SKILL.md vs 817 claimed; 11 frameworks vs 6 claimed","reliability":"deterministic count, single run (capped at 60 by spec)"},"tags":["skills","security","count-verified"],"keywords":["cybersecurity skills","MITRE ATT&CK","NIST","Claude skills"]},"verdict":"UNPROVEN","verdict_why":"The 818-skill count and the six frontmatter framework fields are real and machine-checked, and 364 of 366 framework IDs we validated exist in the official catalogs. But counting files can't tell you whether a skill helps an agent, and the claim ledger logs 8 contradicted README claims -- so usefulness stays unproven until the A/B lab run.","published_ct":"Oct 18, 2026","verdict_caveats":["Do not read UNPROVEN as fake: the structure claims largely hold (818 skills, six mapping fields, valid IDs). It means nobody -- including us, yet -- has measured whether these skills make an agent better.","The claim ledger (data/claims/mukul975-cyber-skills.json) has 28 rows: 3 supported, 11 partly, 8 contradicted, 6 unverified. Contradicted items include the documented skill anatomy, the token-cost claims, 'all 22 CSF categories', 'zero deprecated IDs', and 'production-grade' on sampled evidence.","Both scripts we hand-checked from the README's own examples have confirmed bugs: the showcase memory-forensics agent crashes on every full report run, and the breach-containment agent reports CONTAINED regardless of whether any action succeeded.","Star-integrity sampling was blocked at the API level from our test box (verified blocked for a control repo too), so no account-age numbers exist either way."],"v2":{"works":{"verdict":"unproven","why":"Phase 1 only counted and parsed. The with-vs-without A/B on ~100 skills is queued as Phase 2; until that lands, usefulness is not measured."},"safe":{"verdict":"caution","why":"No confirmed malware in the surface scan, but two README-featured agents have confirmed bugs (memory-forensics TypeError; breach-containment clock-derived password + hardcoded CONTAINED). README funnels collect leads (survey with Casky Tokens, casky.ai waitlist). Not a hard Safe fail yet — no confirmed secret exfil."},"claims_honest":{"verdict":"mixed","why":"Claim ledger: 3 supported, 11 partly, 8 contradicted, 6 unverified. Contradictions include skill anatomy, token-cost claims, 'all 22 CSF categories', 'zero deprecated IDs', Navigator release assets, and 'production-grade' on sampled evidence."},"provenance":{"verdict":"caution","why":"Independent project with a buried non-affiliation disclaimer that does not travel into plugin.json / CITATION.cff / the GitHub description. Bulk history (611 skills in commit 1). Star-integrity sampling was blocked on our test box (environment limit, not a finding)."}},"usefulness":"Unproven -- head-to-head A/B test (same task with and without the skill loaded) is queued as Phase 2.","verdict_draft":true}