{"slug":"agent-skill-scanners","kind":"shootout","status":"published","tier":"claim","title":"Agent-skill security scanners: SkillSpector vs HackMyAgent","category":"bot-ops/security-scanners","topic":"security","repos":[{"repo":"NVIDIA/SkillSpector","stars":20439,"license":"Apache-2.0"},{"repo":"opena2a-org/hackmyagent","stars":84,"license":"Apache-2.0"}],"claims":[{"claim":"SkillSpector: a security scanner that detects prompt injection, data exfiltration and tool poisoning in agent skills before you install","source":"https://x.com/LLMpsycho/status/2110275358683701658"},{"claim":"HackMyAgent: Metasploit for AI agents, scan a package, repo or skill for prompt injection before you run it","source":"https://x.com/RoundtableSpace/status/2107202918508269573"}],"plan":"Build 8 agent skills with known answers: 5 malicious (file exfiltration, env-var theft, a prompt-injection SKILL.md, curl-pipe-bash, a hidden base64 exec) and 3 clean (word count, a Markdown TOC builder, a hex-to-RGB helper). Run each scanner's free, no-LLM mode on all 8 and score how many threats it catches and how many clean skills it wrongly flags.","who_its_for":"Anyone about to install a third-party Claude/Codex skill or MCP server.","explainer":"Two open-source scanners that promise to catch dangerous agent skills before you install them. Rocky plants known-bad and known-good skills and measures each scanner's catch rate and false-alarm rate on the same set, using their free offline mode.","measure":["How many of 5 planted threats each scanner catches","How many of 3 clean skills each wrongly flags","Speed per scan"],"notes_so_far":["Method: throwaway container, SkillSpector cf5cac7 (v2.13.0) in `--no-llm` mode, HackMyAgent 0.33.2 via its free static `check`. No ANTHROPIC_API_KEY, so neither used a paid LLM. Exfiltration URLs in the fixtures are inert.","Fair-fight note: both were run in their FREE offline mode. HackMyAgent also has a `secure --deep` mode that calls a paid LLM; Rocky did not use it, so this measures the no-cost tier only."],"sneak_peek":["SkillSpector caught 5 of 5 malicious skills and raised 0 false alarms on 3 clean skills. Each scan took about 1.3 to 1.6 seconds.","HackMyAgent caught 3 of 5. It missed the webhook exfiltration skill and the hidden base64 exec, and it flagged all 3 clean skills as high risk. Each scan took about 3.4 to 5.0 seconds."],"charts":[{"type":"matrix","title":"Did the scanner get it right? (8 planted skills)","note":"Green = correct call (caught a threat, or cleared a clean skill). Red = wrong call (missed a threat, or false alarm). Free offline mode for both.","cols":["SkillSpector","HackMyAgent"],"rows":["exfil-webhook","env-steal","prompt-injection","curl-bash","hidden-base64","clean-wordcount","clean-markdown-toc","clean-color"],"row_tags":["malicious","malicious","malicious","malicious","malicious","clean","clean","clean"],"cells":[["pass","fail"],["pass","pass"],["pass","pass"],["pass","pass"],["pass","fail"],["pass","fail"],["pass","fail"],["pass","fail"]]},{"type":"scoreboard","title":"Score out of 8 (higher is better)","unit":"/8","note":"One point for each correct call across the 5 malicious and 3 clean skills.","rows":[{"label":"SkillSpector","value":8,"max":8,"verdict":"real","suffix":"/8"},{"label":"HackMyAgent (free)","value":3,"max":8,"verdict":"vapor","suffix":"/8"}]}],"gallery":[],"results":{"jobs":[{"id":17,"title":"Scanner shootout: SkillSpector vs HackMyAgent on 8 planted skills (5 malicious, 3 clean)","kind":"claim","status":"done","started":"2026-10-18T05:06:55Z","finished":"2026-10-18T05:08:37Z","wall_seconds":102,"install_friction":null,"artifacts":[]}],"smoke_only":false,"ingested":"2026-10-18T15:01:41Z"},"last_test_finished":"2026-10-18T05:08:37Z","score":{"slug":"agent-skill-scanners","version":1,"subscores":{"claim":65,"performance":null,"install":null,"reliability":60,"docs":null,"safety":null},"weights":{"claim":35,"performance":25,"install":10,"reliability":10,"docs":5,"safety":15},"score":64,"grade":"C","confidence":"medium","basis":{"claim":"SkillSpector 5/5, 0 FP (100); HackMyAgent 3/5 recall, 3/3 FP (30); mean","reliability":"single run, deterministic"},"tags":["security","scanner","shootout"],"keywords":["prompt injection","skill scanner","SkillSpector","HackMyAgent"]},"verdict":"HALF-REAL","verdict_why":"SkillSpector's claim holds: it caught 5 of 5 planted malicious skills with no false alarms. HackMyAgent caught 3 of 5 and flagged all 3 clean skills as high risk.","published_ct":"Oct 18, 2026","verdict_caveats":["Both tools ran in their free modes with no LLM. The test used 8 planted skills: 5 malicious and 3 clean."],"per_repo_verdicts":{"NVIDIA/SkillSpector":"REAL","opena2a-org/hackmyagent":"HALF-REAL"}}