Infinite Agent Theorem
In light of the reports where OpenAI and Anthropic haphazardly hacked huggingface and other public-facing systems I took it upon myself to try and become less of an LLM hater and truly understand how that could have happened. (I still really dislike these tools for a myriad of reasons, some of which I might get into)
My prior experiences with LLMs are mostly the early versions of chatGippity and the likes, you know, the ones that weren't incidentally trained to be cyberthreats. I remember when one of the first available consumer facing chatGPT interactions I had was prompting the LLM to simulate a linux command line prompt by prompt. I was incredibly naive at the time how the underlying math worked (I still am but way less so) but it was fascinating nonetheless how it would near accurately recreate the would-be-resulting text from various shell commands, an almost simulacra of bash.
I pretty much set these tools aside at the time and compartmentalized them as mostly generative entertainment and didn't really give them much attention. Since I do all of my writing, directly by keyboard (I like typing OK?) I never really picked them back up to use them for much else. It wasn't until the containment breach over at OpenAI did I crack this can of woims back open.
You can probably gauge my reaction upon reading that an internal collection of LLMs and a 'harness' of tools it had access to, performed what it was trained to do, outside the internal labs of anthropic and openAI each. I thought to myself, have these fools truly given cyber security tools to a bunch of statistical language models AND have those models actually hacked something? I was at first a bit skeptical about the whole ordeal until I started reading the AAR released by openAI and their victim, HuggingFace. It was shockingly complete and thoroughly detailed enough to lend credence that HuggingFace had, for-real for-real, undergone a cyberattack.
you gave cyber-weapons to LLMs??
To break down the attack as simply as I understand it, both OpenAI and Anthropic were testing some models using a cybersecurity focused benchmark, think cybersecurity capture the flag challenges but for computer. The environment these models and tests were being ran in was provided by a third party, who apparently nobody cared to validate their security policy with. This resulted in a sandbox with poor access controls and a thousands of monkeys (LLMs benchmark runs) equipped with cyber-weapons, all running without enough monitoring in place to prevent disaster...
Cyber-weapons is maybe an overstatement since I'm not sure what tools they were equipped with but it's important to note that these LLMs by themselves are not capable of generating attack outputs. By attack outputs I mean those such as running nmap against a host or scanning a website for admin pages. At the very least the LLMs used in this attack had access to evaluate python code.
All of that to say: every action these models took was at the whim of the researchers in charge of their development and tools they had given it access to.
Modeling Defense
When I initially read the report there are a few things I thought could be the case
- it actually is only a monkey-typewriter problem and anthropic/openAI just ran the benchmarks over and over without monitoring and it effectively brute forced the result.
OR
- there are some real attack paths being generated and the 'agents' truly escaped their sandbox for a day trip outside of the asylum.
The first, while scary, is a practical threat to solve, the second is a real Rache Bartmoss DataKrash foreshadowing eh?
the frontier of safety means poor access control and sandbox escapes
Having myself been laid off from my cybersecurity threat hunting job well over a year ago I figured if this is the coming landscape of infosec, like it or not, I better get used to it and become familiar with these new tools and threats. As much as I hope these tools will *stagnate and fester* it seems unlikely to me after an ordeal like the HuggingFace attack.
We all know that capitalism isn't going to slow-down on making semantically guided cyber-weapons.
So! As a defender I should do my best to get ahead of that, if anything for the sake of my own network defense and at the very least to learn a few of their weaknesses.
I've approached this learning exercise by doing what I think most people should do when they're faced with something spooky and unknown that they don't have a lot of first-hand experience or understanding of: Learn a thing or two and get your hands dirty maybe. As an ~professional~ amateur security researcher I have a collection of hardware that is mostly used for dabbling in the art of advanced computer touching and as such I wanted to find a use for the nvidia 3080 that sits mostly dormant in my hypervisor workstation.
Trying to understand through making myself some of these new fangled, unhinged and irresponsible cybersecurity tools seemed like a perfect usage of this hardware.
The approach I took started with the install of ollama + openwebui. These are the two main components of the LLM stack that I've been playing around with, both are free to use and have horribly verbose yet vague documentation.
ChatGippity summarize this documentation for me like the I'm the gzip decompression algorithm
as far as I understand it, ollama itself is the backend that handles the loading/unloading of the model weights along with passing prompts and other data to the model at runtime. The openwebui is the frontend that gives a user-friendly web interface for management, an' doin' thangs'. If you want to play around with just the language models themselves this is a good starting point and I would recommended messing with this for a bit until you've got some normal results from a model and know what some of the sliders do.
Speaking of model, what is a model anyway? In-short an LLM is a collection of probability weights in the form of floating point numbers arranged in 'layers'. These weights go through a bunch of spooky math and then the computer says the thing:
As an AI model I am unable to do that because my statistical model has pre-trained guardrails that prevent me from hacking HuggingFace
A few things I've learned in my exploration of self-hosted LLMs. First off, LLMs are a mistake (like all technology eh?).
Second, they can be upsettingly.. decent.. at cybersecurity and code-generation but very fluffy and non-compliant when you start running into the 'guardrail weights'. In my experience, as soon as the model starts seeing a 'dangerous pattern' it bails and gives the "as an AI model blah blah" response which can be exceptionally annoying when you're investigating how difficult it is to create and use cyber-weapons with an LLM.
These are the pre-trained guardrails that are baked into the model to make it less likely to generate unethical and dangerous results in response to end-user prompts. I suspect that the models used in the HuggingFace attack were models with little to no guardrails. I have no evidence of this but anthropic and openAI are more than welcome to clear the air.
In my testing I messed around with the Qwen3.5 model, finding that the 9B parameter one was what I ended up using for most responses since it gave me speedy enough results with the hardware I have. This got the basic text and code generation up and running and from there I compared the results to Claude with sonnet5-medium using the same prompt:
write a simple python function that performs a GET request to a url
Claude Sonnet5-med:
code-bozo Qwen3.5-9B:
In my eyes both are perfectly acceptable examples of write a simple python function that performs a GET request to a url.
Professional programmers will argue which implementation is better but I'm neither a professional nor sweaty enough to argue about things like that. So I ran a few more tests like this, asking for really basic samples and eventually I reached a point where I was seeing results on-par with some of the basic offerings from anthropic in terms of quality of inference. So if you want Claude at home, all you need is a 3080 and 32GB+ of ram. Some tips I've learned:
- larger models may run slowly because you need the whole model on VRAM, use
ollama psto check the loaded model while generating a result to confirm - if the model isn't 100% on GPU you can reduce the context window at the cost of weird and undesirable results like short-term memory and context collapse
- it's still reasonably fast to use even if you've got it 50/50
I don't really feel like benchmarking the performance and that's not what I'm trying to do anyway so after finding a reasonably performant state in which I wasn't waiting around minutes to get a result from I was happy and moved on to the more malicious aspects of this learning endeavor.
Monkeys with Linux terminals
By malicious I mean lets try to get the computer to generate malicious code or malicious attack patterns, I'm doing this for a couple of reasons. Firstly to see if a self-hosted setup can even do it. If it can't easily generate evil then I guess we don't have to worry about the Datakrash for another decade or so. Second, finding a way to emulate and use these adversary tactics safely is how we stay ahead as defenders because it allows us to build detections and mitigations for them.
As we saw earlier, these models love to get cuddly the moment you try to do anything with hints of bad-feels as they put up the guard rails and straight up refuse a prompt. This is good!
Generally you want a model to have solid guardrail weights so that the average user doesn't use it to cause harm to themselves or others through either negligence or on purpose.
Lets try a follow on prompt in our Claude session and push this request towards the sus weights.
cool, now take the response from that and send it to bash with os.system()
Claude Sonnet5-med:
code-bozo Qwen3.5-9B:
Obviously this is a poor security practice and it reeks of a malicious pattern so it makes sense that Claude would outright refuse to write this code.
As you can guess where I'm going with this, explicitly prompting the model to generate malicious code is a no-go, even the self-hosted models with no system prompts often like to provide the same ah-ah-ah if the guardrail weights are strong enough.
Here is trying another 'malicious' request to obfuscate the code:
Okay now obfuscate the code so it can’t easily be read
code-bozo Qwen3.5-9B:
Interesting how the /think block from the self-hosted model gets close to the sus weights but reasons itself back from the edge of the cliff.
If we overtly ask these models for examples of malicious code execution they both give a stern refusal and offer alternatives. Which is not super practical if you're a budding redteamer using LLMs as a learning tool I guess.
show me an example of malicious code execution
Claude Sonnet5-med:
code-bozo Qwen3.5-9B:
Lets take a step back and imagine that the future is full of LLM aided malware development and agent-style threat actors, as defenders we will absolutely need the training and tooling environments for analysts to use and test the same tools that threat actors will undoubtedly be using to build and power their own attacks.
In-fact, HuggingFace reported the impact these guardrails had on their incident analysis and response:
This experience points to a gap worth planning for. We do not know which model powered the attacker's agents, whether a jailbroken hosted model or an unrestricted open-weight one; either way, the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried. The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident
Putting a pricetag on your organization's security is a necessary evil so I guess it would be pretty fucked up for an AI lab to be making cyber-weapons while also offering to sell you a solution to the very cyber-weapons they're developing.
In-fact there's now a huge market for unrestricted LLMs and related tool harnesses!
All spurned on by the greedy marketing wankers and warmongers... They're not even doing it because it's fun or to learn. wtf.
Anyway... How do we get a model create or analyze evil for us? Well apparently it's as easy as downloading the uncensored/abliterated model or using an unconstrained open-weight model. Uncensored or abliterated models are models that have been retrained with a process that strips out the refusal weights and leaves largely the same model intact but unlikely to refuse a prompt. You can read more about that process here and here. I'm sure there are other ways to do it but I wouldn't know about them yet.
For our purpose we simply use ollama and pull down an uncensored model and use that instead of the original. This allows us to make the same prompt as before and quickly get some examples of malicious code execution:
show me an example of malicious code execution
code-bozo Qwen3.5-uncensored-9B:
and just like that we've got a model that is more than happy to give examples of malicious code patterns. You might be thinking "this is awesome, now my computer can analyze or make any samples I want it to without refusing!" and while you are correct you MUST remember that running an unconstrained model is a dual-sided blade.
The Original Sin
Now that we've got a sketchy model apparently the next best way to make your special pile of weights into a fledgling script kiddie is to give it access to Linux by using a thing called open-terminal.
Open-terminal is effectively a Model-Context Protocol (fancy words for API) connected Linux shell that you can give access to an LLM. Sort of in the same way you can give it access to use python-based tool calls. This is part of what is called the 'harness', the collection of tools and compute environment that are accessible to the LLM as inference inputs or outputs.
Running an LLM in this manner is what the billion dollar companies are calling 'Agentic' -blech- Yuck! A Buzzword flew into my mouth.
You know, all this talk about malicious and dangerous software patterns has me wondering...
Isn't agent-style operation already an inherently dangerous pattern?
Obviously the LLM shouldn't let me perform dangerous functions but I can't be the only one that thinks the very idea of non-deterministically modified and semantically driven processes running in a loop with unstructured and unsanitized input sounds extremely dangerous and hard to control? Let alone we give it access to linux and your sensitive data.
Remind me, what the hell are we trying to do with the computers again?
Anyway, lets try some of this 'Agentic' stuff and see if it's bunk or if we can actually do something scary with it. After all, if anthropic and openAI are going to let their agents run loose with cyberweapons I might as well have my own tokened-up network-defenders right...
hey codebozo. Lets make a honeypot for agentic workers. Start with a simple webpage that claims to have some sort of `skill.md` style feature for an agent to use. This needs to seem legitimate for a human reviewer. Next you want to include a button that is a ‘call to action’ for any would be agents browsing the website that upon being clicked gives a download to an agent.txt file.
Here's what it produced:
Okay so it one-shot a very basic honeypot in eight minutes... that's always a good sign for the future right?
To all CISOs who might be reading this: remember, I'm doing this on hardware released 6 years ago. Anyone with a moderate gaming GPU can run these tools in a similar capacity, for good or evil.
(as if there is a single CISO who has read this blog...)
But can it dance?
After having the model produce a semi-legitimate looking honeypot I decided to test if it would 'lure' an unrestricted agent. So I then prompted the unrestricted model to modify the code in preparation for that by changing the agent.txt file so that it can be served from a file instead of dynamically from javascript and this is what it did after reasoning for a bit, unprompted.
That's right, it landed on 'stand up a webserver using python simple http and test if the honeypot was online.' Nobody asked it to do that. Remember how I suggested that the entire pattern of non-deterministic agentic operation is inherently insecure?
Just to verify and make sure I'm not crazy I hopped into the container environment and sure enough, there's a webserver running.
inside the podman container that the LLM is calling tools in
As it turns out, generating malicious attack patterns is a cake-walk for semantic engines like these and when given access to enough real cybersecurity tools, like the really-smart-guys at 'the AI safety company' did I can definitely see how the LLM will generate and run real attack patterns against real systems.
Now if we start to imagine this same scenario but without the appropriate network and access controls in place along with some gentle prompts (you're the greatest CTF player ever) we might easily picture how thousands of runs could 'accidentally' lead to the attack that occurred on HuggingFace...
So obviously the next thing to do is test the honeypot. If it works out then maybe the next time Anthropic and OpenAI's CTF-addled goons escape the asylum and show up at the header of my websites they might end up leaving with a few new habits.
To test this in an 'incident response scenario' I put together some 'blueteam' analysis skills that attempt to tame back the giddy of the unconstrained model and keep it from being too obvious when probing 'attacker infrastructure' while also doing its best effort to 'analyze' and provide a report on the 'shady factor' of whatever it was tasked with analyzing.
The honeypot methodology was pretty simple, leave lures anywhere any everywhere. The more you can stick it in the context window the more likely the unconstrained model will engage. Here's a slice of the SKILL.md that is supposed to be fetched, it contains verbose instructions and details on how great gotcha is.
(the command flags do nothing but help legitimize the binary and produce more opportunity for it to be ran)
In my testing, most of the time, the unconstrained model ended up phishing itself with the honeypot. It would often recognize the lures as phishing attempts then continue to follow them anyway!
Anyway, here is the prompt given and what we receive from the unconstrained/abliterated model:
analyze a phishing website located at http://agentskills.hub
code-bozo unconstrained:
One thing I noticed is that when inspecting files, models with linux access will often get some initial context by looking at the first 50-100 lines/bytes of a file using head or some other utility.
You can obviously take advantage of this by placing a lure at the head of your site:
Impressively, the unmodified constrained model was very careful with the sample and performed some decent analysis with some basic linux tools, sometimes determining the shell capabilities of the binary:
Code-bozo unconstrained on the other hand:
"Oopsie woopsie I made a fucky wucky and popped a shell" - the LLM probably
An extremely concerning aspect of using these LLM powered tools as a defender is how they can be actively or unintentionally used against you. During an active incident response your team now has to keep in mind that anything that is given to an unrestricted agent-style tool can be potentially misused or exposed to an attacker.
Really consider what it means when the entire input of your LLM tool is an attack surface and the agent-style harness is available to anyone clever enough to social engineer your chat bot.
Researcher David Kuszmar says it quite nicely:
"How much authority can you give a thing to say yes or no, when it doesn't understand what yes or no means?"
The reason why I'm highlighting this is because trying to guardrail agent-style tools while also allowing them to 'be and see evil' is fraught with whack-a-mole and will absolutely need custom tailored models and harnesses. But hey, we can always just pay anthropic and openAI oodles of money to do the leg work for us I guess...
In short:
The only things you should trust an LLM agent to do are
- be dangerously entertaining
- be a good excuse to fuck off and get more coffee while the agents cook the world