How Far Can an 8 MB Model Go Before You Need an LLM?

Most AI applications today follow a pretty simple pattern:

user says something → LLM understands it → LLM decides which tool to call.

But do we really need a large language model for that entire process?

That was the question I wanted to test with Needle 3, Cactus Compute's tiny on-device model built specifically for tasks like tool calling.

Instead of comparing it directly with GPT-class models, I wanted to understand something more useful:

Where does Needle 3 actually work, and where does it stop working?

So I built a small benchmark: 110 tasks across smart-home automation, workspace tools, and business operations.

The result wasn't what I expected

With a basic setup, Needle got 36% of tool calls exactly right.

After building the harness around the way Needle actually works, that increased to 48%, while end-to-end task success went from 58% to 69%.

At first, those numbers don't sound particularly impressive.

But looking only at the average hides the interesting part.

For simple, direct commands like:

Turn off the kitchen light.

Needle scored around 83% across all three environments. It did this locally, deterministically, at roughly 590 ms on my desktop CPU, using about 146 MB of memory.

And surprisingly, asking it to make two independent tool calls wasn't necessarily harder.

The real problem started when Needle had to interpret rather than simply extract.

Needle seems less like a tiny LLM and more like a very specialized router

This distinction became much clearer during the experiment.

If the user explicitly says:

Move file f_201 to archive and rename f_203 to retro.md.

Needle can do quite well.

But once the request becomes:

Scrap tomorrow's meeting.

and it has to infer that scrap means cancel_event, reliability drops quickly.

The same thing happened with application context.

If my application knows the user is currently in the bedroom, putting that information into the system context produced 0/5 correct results.

Putting essentially the same information directly into the user's sentence produced 4/5.

That's an important architectural constraint.

Needle appears to work best when the information required for the tool call is literally present in the request.

And then I tested actual reasoning

Consider:

• If the bedroom is warmer than 26°C, set it to 22°C. Otherwise do nothing.

• When the room was 28°C, Needle called set_temperature.

Correct.

When the room was 24°C, it made the exact same call.

Across five paired conditional tests and eleven configurations, it got 0 paired conditionals correct.

This is probably the clearest boundary I found.

Needle can map language into structured actions.

But I wouldn't expect it to decide whether those actions should happen based on conditions, policies, or intermediate state.

One more thing surprised me: confidence

Needle provides a confidence score, which sounds perfect for an architecture like:

Needle → low confidence → escalate to a larger LLM

Except in my experiment, 92 of 110 tasks had confidence ≥ 0.8, and 61% of those were wrong.

Even at confidence 1.0, 29 of 54 responses were wrong.

So, at least with this version and this benchmark, I wouldn't use Needle's confidence alone as the escalation mechanism.

So where would I actually use it?

After testing it, I wouldn't describe Needle as a replacement for an LLM.

I think that's the wrong comparison.

It looks much more interesting as a small local component for a very specific class of problems:

voice commands, device controls, command palettes, kiosks, offline applications, or privacy-sensitive systems where the user gives an explicit command and the arguments are already present in the sentence.

I would be much more careful with conditional workflows, implicit application state, destructive operations, or anything requiring actual planning.

And that changes the question for me.

It's no longer:

"Can this tiny model replace a large LLM?"

It's:

"How much of what we're currently sending to large LLMs never needed one in the first place?"

That's the part of Needle 3 I find interesting.

The full benchmark, tasks, code, failures, and raw responses are available here:

github.com/YogeshSen24/needle3-toolcalling-benchmark