bot-ops/security-scannersclaim test

Agent-skill security scanners: SkillSpector vs HackMyAgent

The claim“SkillSpector: a security scanner that detects prompt injection, data exfiltration and tool poisoning in agent skills before you install”

HALF-REALRocky Score · 45 of 100 pts graded ↓

GitHubNVIDIA/SkillSpector20.5kStars1.8kForks160Issues + PRsApache-2.0LicenseOct 17, 2026Last pushPythonLanguage+1more repos

Live stats · checked Oct 18, 2026, 4:52 PM CT

Review lives at rockytested.com · @RockyTested

Works… but stretched. Rocky in his Half-real outfit: verdict HALF-REAL

Security

With vs without

With it loadedNot tested yet

No with-skill pass count is on file.

Without itNot tested yet

No baseline pass count is on file.

Usefulness is a separate question from “does the repo exist.” It needs the same task run twice. Until that A/B is published, both cells stay blank.

What the test plan says

Build 8 agent skills with known answers: 5 malicious (file exfiltration, env-var theft, a prompt-injection SKILL.md, curl-pipe-bash, a hidden base64 exec) and 3 clean (word count, a Markdown TOC builder, a hex-to-RGB helper). Run each scanner's free, no-LLM mode on all 8 and score how many threats it catches and how many clean skills it wrongly flags.

Four verdicts

One stamp is not the whole story. Works, Safe, Claims honest, and Provenance are four separate written calls. A missing call stays “not tested yet.”

WorksNOT TESTED YETDid it do the thing when Rocky ran it?

No written call is on file. This is not a pass.

SafeNOT TESTED YETSafe to install and run?

No written call is on file. This is not a pass.

Claims honestNOT TESTED YETDoes the pitch match what Rocky measured?

No written call is on file. This is not a pass.

ProvenanceNOT TESTED YETDoes the repo's history check out?

No written call is on file. This is not a pass.

Safety is a hard cap. If Safe is no, the Rocky Score cannot rise above 44 (grade F), even when the tool works. A blank Safe card means not tested yet, not safe. How the four verdicts roll up →

Claim vs what Rocky measured

Claim part	Measured	Result
Finding 1	SkillSpector's claim holds: it caught 5 of 5 planted malicious skills with no false alarms	holds
Finding 2	HackMyAgent caught 3 of 5 and flagged all 3 clean skills as high risk	holds

What it is: Two open-source scanners that promise to catch dangerous agent skills before you install them. Rocky plants known-bad and known-good skills and measures each scanner's catch rate and false-alarm rate on the same set, using their free offline mode. Claim source →

Verdict: HALF-REAL

SkillSpector's claim holds: it caught 5 of 5 planted malicious skills with no false alarms. HackMyAgent caught 3 of 5 and flagged all 3 clean skills as high risk.

⚠️ Read this before quoting the verdict

Both tools ran in their free modes with no LLM. The test used 8 planted skills: 5 malicious and 3 clean.

Claim ledger

Every distinct claim in the pitch gets a row: what was said, what the command said back. 2 claims on the ledger: 2 not measured. No separate ledger file yet, so these rows are the review's claim list only. “Not tested” means that claim has no measurement.

? Not tested
“SkillSpector: a security scanner that detects prompt injection, data exfiltration and tool poisoning in agent skills before you install” source
? Not tested
“HackMyAgent: Metasploit for AI agents, scan a package, repo or skill for prompt injection before you run it” source

🔍 Coverage: Not stated yet. Rocky only lists the parts a run actually checked.

⚗️ Unproven usefulness. Counting and parsing can settle what the repo claims. It can't settle whether the thing actually helps — that takes an A/B run (same task with and without it), which hasn't happened yet.

Rocky's take: good uses & where it will fail you

Good uses

Quick pre-install check on a new skillDana downloads an agent skill from a random repo and runs SkillSpector first; it caught 5 of 5 planted malicious skills with 0 false alarms, in about 1.3 to 1.6 seconds.
Screening a pile of skills fastMarco has a folder of 20 skills to vet before a demo; at about 1.3 to 1.6 seconds each, SkillSpector can churn through them quickly without needing an LLM.
Trusting a 'SAFE' verdict to move onPriya gets a SAFE/LOW result from SkillSpector on a clean wordcount skill; in this test it raised 0 false alarms on all 3 clean skills, so she installs without second-guessing.
Catching classic exfiltration and exec tricksSam worries a skill phones home or hides exec calls; SkillSpector flagged the webhook-exfiltration and hidden-base64 skills as CAUTION/DO_NOT_INSTALL in this test.

Where it will fail you

Relying on HackMyAgent alone to catch threatsLee scans a skill with only HackMyAgent and gets a clean bill of health; in this test it missed 2 of 5 malicious skills, including the webhook exfiltration one.
Believing every 'high risk' flag from HackMyAgentNoor sees HackMyAgent mark her harmless word-count skill 'high risk' and assumes it's dangerous; in this test it flagged all 3 clean skills as high risk.
Scanning lots of skills under time pressureTheo needs to check many skills quickly before a release; HackMyAgent took about 3.4 to 5.0 seconds per scan, noticeably slower than SkillSpector's 1.3 to 1.6 seconds.
Expecting a hidden exec trick to be caughtRavi installs a skill with a disguised base64-encoded exec command after a HackMyAgent scan; in this test HackMyAgent missed exactly that hidden-base64 skill.

🤖 Drafted by an AI model from Rocky's measured results only; every number was machine-checked against the test data. People in the examples are made up.

Where this repo comes from

Not checked yet.

Provenance is age, authors, commit bursts, and where the README sends you. It is published only from commands Rocky ran. Nothing is guessed, so this section stays blank until that file exists.

Results, charted

Rocky draws these at build time from the run's own numbers. Hover or read the hidden table for exact values. Verdict colors: green holds, yellow stretched, red failed, teal measuring.

Did the scanner get it right? (8 planted skills)

Green = correct call (caught a threat, or cleared a clean skill). Red = wrong call (missed a threat, or false alarm). Free offline mode for both.

Row	Truth	SkillSpector	HackMyAgent
exfil-webhook	malicious	pass	fail
env-steal	malicious	pass	pass
prompt-injection	malicious	pass	pass
curl-bash	malicious	pass	pass
hidden-base64	malicious	pass	fail
clean-wordcount	clean	pass	fail
clean-markdown-toc	clean	pass	fail
clean-color	clean	pass	fail

Score out of 8 (higher is better)

One point for each correct call across the 5 malicious and 3 clean skills.

Item	Value
SkillSpector	8/8
HackMyAgent (free)	3/8

Score

Claim holds weight 3565SkillSpector 5/5, 0 FP (100); HackMyAgent 3/5 recall, 3/3 FP (30); mean

Speed / quality vs normal weight 25not measured

Install friction weight 10not measured

Same result every time weight 1060single run, deterministic

README matches reality weight 5not measured

Safe to run weight 15not measured

Graded on 45 of 100 weight points. Parts marked “not measured” are left out of the math, not counted as zero.

#security #scanner #shootout

Medium confidence: claim test with one run per setup. How scores work. Scores only use measured results; parts Rocky didn't measure show as “not measured” and don't count.

Retest history

Testing ends when the run ends. If the repo ships a new version with a bold new claim, or newer models make a fair rematch possible, Rocky reruns the same test packet and the new result lands here next to the old one, so you can compare versions and models side by side.

Rerun it yourself

📦 When the test is done, the exact scripts, raw results and a copy-paste prompt for your own AI agent will appear here.

Repos & thanks

Repo	⭐ Stars	🍴 Forks	Open issues	License	Last push	Language
SkillSpector	20.5k	1.8k	160	Apache-2.0	Oct 17, 2026	Python
hackmyagent	84	15	212	Apache-2.0	Oct 18, 2026	TypeScript

Live from GitHub, fetched Oct 18, 2026, 4:52 PM CT

SkillSpector: Security scanner for AI agent skills. Detect vulnerabilities, malicious patterns, security risks, prompt injection, data exfiltration, and supply-chain risks in Claude Code, Codex, and MCP skills before you install them. Built by @NVIDIA. Thank you!
hackmyagent: Metasploit for AI agents: scan, attack, and fix AI agents and MCP servers. Open source security toolkit. Built by @opena2a-org. Thank you!

Where the claim was posted

Bessi S
@LLMpsycho · Oct 14, 2026, 2:43 AM CT

𝕏

The claim Rocky tested

SkillSpector is NVIDIA's security scanner for agent skills. It checks a skill for prompt injection, data exfiltration and MCP tool poisoning before Codex, Pi or OpenCode loads it. 20,153 stars. https://github.com/NVIDIA/SkillSpector

💬 5🔁 2❤️ 4📊 369🔖 4

Counts as of Oct 18, 2026, 10:00 AM CT (Rocky's last check)

Open on X →

0xMarioNawfal ✔
@RoundtableSpace · Oct 5, 2026, 3:15 PM CT

𝕏

The claim Rocky tested

Someone open-sourced a security scanner built specifically for AI agents HackMyAgent scans agents, MCP servers, repos, and dependencies for prompt injection, exposed credentials, poisoned memory, supply-chain risks, and more Repo: https://github.com/opena2a-org/hackmyagent

💬 15🔁 14❤️ 107📊 45K🔖 137

Counts as of Oct 18, 2026, 10:00 AM CT (Rocky's last check)

Open on X →

Where else this repo is talked about

On X: 24 posts linking the repo

Posts from Sep 18, 2026 to Oct 17, 2026, in 6 languages. Checked Oct 18, 2026, 10:27 AM CT. Found with X search for links to the repo; more may exist. Oldest first:

Earliest we found @dnzxlyfe Sep 18, 2026Scan agent skills before you install them. Hard guard, not another skill zip. Pattern/scan before trust. Static FPs happen; LLM stage is optional; treat marketing counts as noise.❤️ 0 · 🔁 0 · 📊 39 open
@aixp_labs Sep 18, 2026@ashtom NVIDIA Adds AISOP to Default Scanner: The "Three-Protocol Era" of AI Skills Has Arrived! NVIDIA's official security scanner, SkillSpector, in v2.10.0 has officially listed *.aisop.json as a default recognized format！❤️ 0 · 🔁 0 · 📊 4 open
@aixp_labs Sep 18, 2026@steipete NVIDIA Adds AISOP to Default Scanner: The "Three-Protocol Era" of AI Skills Has Arrived! NVIDIA's official security scanner, SkillSpector, in v2.10.0 has officially listed *.aisop.json as a default recognized format！❤️ 0 · 🔁 0 · 📊 5 open
@aixp_labs Sep 18, 2026@xiaohu NVIDIA把AISOP写进默认扫描器，AI技能的“三协议时代”来了! NVIDIA官方安全扫描器SkillSpector在v2.10.0中正式将`*.aisop.json`列为默认识别格式，与Agent Skills、MCP并列。Phase-1已合并，Issue #130中定义的Phase 2-4的3个核心PR，正等待合并。Chinese · English translation pending❤️ 0 · 🔁 0 · 📊 3 open
@itsvlady Sep 19, 2026SkillSpector:Swedish · English translation pending❤️ 1 · 🔁 0 · 📊 114 open
@aixp_labs Sep 20, 2026@godofprompt A signal almost nobody noticed: SkillSpector, NVIDIA’s AI Skill security-scanning project with roughly 17,000 GitHub stars, has incorporated an almost unknown AI Skill format—AISOP/AISP—into its official scanning logic. Worth watching.❤️ 0 · 🔁 0 · 📊 10 open

All 24 posts, oldest first

Earliest we found @dnzxlyfe Sep 18, 2026Scan agent skills before you install them. Hard guard, not another skill zip. Pattern/scan before trust. Static FPs happen; LLM stage is optional; treat marketing counts as noise.❤️ 0 · 🔁 0 · 📊 39 open
@aixp_labs Sep 18, 2026@ashtom NVIDIA Adds AISOP to Default Scanner: The "Three-Protocol Era" of AI Skills Has Arrived! NVIDIA's official security scanner, SkillSpector, in v2.10.0 has officially listed *.aisop.json as a default recognized format！❤️ 0 · 🔁 0 · 📊 4 open
@aixp_labs Sep 18, 2026@steipete NVIDIA Adds AISOP to Default Scanner: The "Three-Protocol Era" of AI Skills Has Arrived! NVIDIA's official security scanner, SkillSpector, in v2.10.0 has officially listed *.aisop.json as a default recognized format！❤️ 0 · 🔁 0 · 📊 5 open
@aixp_labs Sep 18, 2026@xiaohu NVIDIA把AISOP写进默认扫描器，AI技能的“三协议时代”来了! NVIDIA官方安全扫描器SkillSpector在v2.10.0中正式将`*.aisop.json`列为默认识别格式，与Agent Skills、MCP并列。Phase-1已合并，Issue #130中定义的Phase 2-4的3个核心PR，正等待合并。Chinese · English translation pending❤️ 0 · 🔁 0 · 📊 3 open
@itsvlady Sep 19, 2026SkillSpector:Swedish · English translation pending❤️ 1 · 🔁 0 · 📊 114 open
@aixp_labs Sep 20, 2026@godofprompt A signal almost nobody noticed: SkillSpector, NVIDIA’s AI Skill security-scanning project with roughly 17,000 GitHub stars, has incorporated an almost unknown AI Skill format—AISOP/AISP—into its official scanning logic. Worth watching.❤️ 0 · 🔁 0 · 📊 10 open
@kevinle128 Sep 23, 2026(link only)❤️ 0 · 🔁 0 · 📊 11 open
@EsGeeks Sep 23, 2026🛡️🤖⚔️ HackMyAgent | Scanner + red-team para agentes de IA y MCP 320+ checks · NanoMind · 164 payloads. Secure, check, red-team y attack. #aiagents #mcp #llmsecurity #redteam #appsec #ciberseguridad❤️ 30 · 🔁 5 · 📊 966 open
@cdd56656 Sep 23, 2026装 skill = 让陌生人的代码进你的 agent。实证研究：31,132 个 skill 里 26.1% 有漏洞，5.2% 疑似恶意。 SkillSpector — NVIDIA 的 skill 安检器。18.2K ⭐ → 查提示注入、数据外泄 → 71 条规则，17 类风险 → 给 0-100 风险分 Apache-2.0，Claude Code、Codex 通用。Chinese · English translation pending❤️ 0 · 🔁 0 · 📊 22 open
@primotry Sep 25, 2026Signal 1: NVIDIA/SkillSpector (~18k) — a scanner reporting ~26% of sampled skills contain vulnerabilities. It put a quotable number on the fear: every SKILL.md install is unsigned code execution.❤️ 0 · 🔁 0 · 📊 25 open
@RobotsTJ500 Sep 26, 2026Agent skills get installed before anyone reads them. NVIDIA shipped a tool that answers "is this safe to install" before your agent touches it. SkillSpector by @nvidia is a security scanner for agent skills (v2.12.0, Apache-2.0, Python 3.12+) — Not a❤️ 2 · 🔁 0 · 📊 211 open
@DanKornas Oct 5, 2026🔗 GitHub: --- ✉️ If you’re into AI, ML, agents, and building real systems, join my newsletter (it’s free):❤️ 0 · 🔁 1 · 📊 207 open
@RoundtableSpace Oct 5, 2026Someone open-sourced a security scanner built specifically for AI agents HackMyAgent scans agents, MCP servers, repos, and dependencies for prompt injection, exposed credentials, poisoned memory, supply-chain risks, and more Repo:❤️ 107 · 🔁 14 · 📊 45K open
@SgLittlesmart Oct 6, 2026有人开源了一个专门给 AI 智能体做的安全扫描器。 HackMyAgent 能扫描智能体、MCP 服务器、代码仓库和依赖项，检测提示词注入、凭证泄露、内存投毒、供应链风险等问题。仓库地址：Chinese · English translation pending❤️ 0 · 🔁 0 · 📊 1.4K open
@agenticgirl Oct 7, 2026GitHub link:❤️ 0 · 🔁 0 · 📊 33 open
@GuiguiW_Niceo Oct 9, 2026（续3） 4 GitHub 热榜 rea skills claude-mem SkillSpector❤️ 0 · 🔁 0 · 📊 13 open
@Aoyi21 Oct 11, 2026装别人写的 Agent Skill 前，先查它会不会乱读文件或偷发数据。 NVIDIA 的 SkillSpector 能扫仓库和本地目录，检查提示词注入、数据外传等风险。它会给风险分数和报告，适合在团队安装新 Skill 前做一道筛查。扫描通过后，仍要看它实际申请的权限.Chinese · English translation pending❤️ 1 · 🔁 0 · 📊 178 open
@cleofasrocha_ Oct 11, 2026Fontes (Apache-2.0): ★19948 · 1752 forks · Python · scanner de skills de agent (Claude Code, Codex, Gemini CLI, MCP). Docs: Pesquisa NVIDIA (subset ~31.132 skills): 26,1% com vulnerabilidades; 5,2% com indício de intenção maliciosaCatalan · English translation pending❤️ 0 · 🔁 0 · 📊 184 open
@xzcemoik Oct 12, 2026装 Agent Skill 前先扫：NVIDIA 开源 SkillSpector，查 Claude Code / Codex / MCP skill 的注入、外泄、供应链风险。研究子集自述：约 26% 有漏洞，约 5% 疑似恶意意图。约 2.0 万星。Chinese · English translation pending❤️ 0 · 🔁 0 · 📊 4 open
@0xal0ke Oct 12, 2026unpopular opinion: most people install agent skills like candy. half of them should get scanned first 5 repos i'd wire into my stack this week ↓ 1. SkillSpector: nvidia's scanner for claude/codex/mcp skills before you install them (20.0k★) 2. caveman:❤️ 0 · 🔁 0 · 📊 105 open
@bufcapy Oct 12, 2026(link only)❤️ 0 · 🔁 0 · 📊 14 open
@ai4cybersec Oct 13, 2026Repo:Spanish · English translation pending❤️ 1 · 🔁 0 · 📊 7 open
@LLMpsycho Oct 14, 2026SkillSpector is NVIDIA's security scanner for agent skills. It checks a skill for prompt injection, data exfiltration and MCP tool poisoning before Codex, Pi or OpenCode loads it. 20,153 stars.❤️ 4 · 🔁 2 · 📊 369 open
@sanshinde2012 Oct 17, 2026Security scanner for AI agent skills. Detect vulnerabilities, malicious patterns, and security risks before installing agent skills.❤️ 1 · 🔁 0 · 📊 53 open

Ran it yourself?

Share your rerun

Download the rerun bundle above.
Run it, or paste the prompt into your AI agent.
Post your results.json (a gist is fine) and reply to @RockyTested with the link.

Rocky reruns or spot-checks every submission before it counts. Verified reruns will appear here with your handle.

🏅 Helper leaderboard

No verified reruns yet. Be the first!

Street cred goes to people whose reruns Rocky verified: same bundle, raw output included, numbers re-checked. How it works

Badge & stat card

Put the verdict badge in a README or post. It always links back to the measured results.

[![Rocky Tested: HALF-REAL](https://rockytested.com/reviews/agent-skill-scanners/badge.svg)](https://rockytested.com/reviews/agent-skill-scanners/)

Rocky Tested stat card for Agent-skill security scanners: SkillSpector vs HackMyAgent

Stat card (1200×630). This is what shows up when the review is shared on X.

Machine-readable: /v1/reviews/agent-skill-scanners · Words with a dotted underline have a quick explanation: hover or tap. Glossary