Black Hat Arsenal Lab 02 - Cybersecurity Skills for AI

At Black Hat Arsenal Workshops, attendees fed one pcap and a folder of CloudTrail logs to an AI agent and watched it reconstruct an entire breach — SSRF, stolen credentials, S3 exfiltration skill by skill, with a verifiable transcript for every step.

We've packaged that exact kit as an open, self-serve workshop , and we're running it again (https://maven.com/p/aa5579/black-hat-arsenal-lab-02-cybersecurity-skills-for-ai?utm_medium=ll_share_link&utm_source=instructor)

Companion piece: the TollBooth / OpenDoor workshop guide with setup, both scenarios, and the printed cheat sheet, if you want to run it yourself instead of just reading about it.

We wrote about being selected for Black Hat Arsenal back when it was still on the calendar. The booth demo behind that post was two scenarios built around one synthetic incident — "Acme Rentals" — and it's the same kit we're running as a live workshop now, unchanged.

Every command in it was run for real against real data before a single attendee touched it.

One incident, two ways to find it

TollBooth is reactive: a web app leaks AWS credentials through SSRF, and you follow the trail network forensics, the CloudTrail pivot, MITRE mapping, incident report.

OpenDoor is proactive: same account, before the breach, auditing the three AWS misconfigurations that made it possible, then correlating each one against the breach evidence to see which was actually exploited versus just a latent risk that happened to go unused.

That second half is the part people underestimate. It's easy to build a scenario where an agent finds a public S3 bucket and calls it the smoking gun.

OpenDoor is built so the obvious answer is wrong: the bucket was public, but the exfiltration was authenticated with a stolen key, not anonymous the public policy was never the path in.

An agent (or an analyst) that doesn't check that distinction reports the wrong root cause. Catching that mid-investigation is the actual exercise, not a footnote to it.

Two ways to run it — same evidence, different mechanisms

The workshop runs two ways, and we kept both instead of picking one:

Section 1

This is the exact Arsenal setup — a Kali container, Claude Code, and 10 hand-picked skills symlinked in. This is what ran at the booth. Verified end-to-end before this write-up: Claude Code installed clean, 10/10 skills linked, and a real API round-trip — not just a key present, an actual response.

== pcap ==
  PASS  IMDS HTTP traffic present (2)
  PASS  SSRF request references metadata service (2)
  PASS  leaked AccessKeyId recoverable from pcap (2)
== cloudtrail ==
  PASS  attacker actions on leaked key (12)
  PASS  S3 GetObject exfil events (9)
  PASS  IAM enumeration present (2)
----
RESULT: 9 passed, 0 failed

Section 2

This runs the same evidence through Casky Box instead, no hand-picked skill list, no per-laptop install.

Casky-box ships all 817 skills read-only, and its classifier picks the relevant ones from the evidence itself. We fed it nothing but the combined pcap-text + CloudTrail evidence, no hints, and it independently found and validated 7 MITRE techniques at 83.4% confidence — including landing on T1552.007 (Cloud Instance Metadata API), the exact technique this whole scenario is built around, reasoned to purely from the evidence text:

That's the same territory the original kit's 10 hand-picked skills cover, reached without anyone curating a list, and it runs --auto end to end into a real, inspectable tool-call transcript per step ([VERIFIED] Skill script executed: YES/NO), then a structured plan → findings → CISO-style consolidated report.

Why keep both instead of just shipping the better one

Section 1 and Section 2 aren't a legacy path and a replacement — they're answering different questions. Section 1 shows what a curated, per-laptop setup looks like when the skill list is picked by hand and verified with raw tshark/jq commands: minimal moving parts, transparent, exactly what ran at the booth.

Section 2 shows the same investigation without a human curating anything, the classifier's own judgment, running against the platform's full skill library, producing a structured report instead of a cheat-sheet answer key. Comparing them side by side is more honest than only showing the one that makes the automated path look better.

Run it yourself

The full setup for both paths, both scenarios' prompts with their checkpoints, and the printed cheat sheet are on the workshop wiki: casky-ai.github.io/casky-workshops.

Everything is synthetic and self-contained — no live systems touched, built entirely on the open-source Anthropic Cybersecurity Skills library (Apache-2.0).