Why Doesn't Adding Another Mac Automatically Make an LLM Faster?

Imagine a coding assistant running beside your repository. You ask it to read a large change and explain what might break. The model is local, the files are local, and yet you are still waiting.

The obvious thought is: add another machine.

But are we adding somewhere to store the model, or somewhere to do its work? Those sound like the same upgrade. They aren't.

On October 15, 2026, Meta released Muse Spark Open, an open-weight multimodal model with 816B total parameters, roughly 42B active per token, and a one-million-token context window. Apple published a distributed MLX inference study the same day.

The interesting part for me isn't simply that a very large model can run across Macs. It's that how we divide the model changes what the extra machines actually buy us.

If the memory adds up, haven't we solved the problem?#

We have solved one problem.

Apple reports about 493 GB for its NVFP4 model representation. With inference working memory added, that configuration does not fit on one 512 GB Mac Studio.

NVFP4 is a low-precision numerical format. Quantization represents model values with fewer bits to reduce their storage requirements.

That is a statement about this representation and runtime, not a universal minimum for every quantization.

A model also needs space to work. Think of a kitchen: fitting every ingredient in the cupboards doesn't mean there is room on the counter to prepare dinner. During inference, intermediate calculations and the context's attention state need memory too.

And once everything fits, we still have to move through the computation.

What if each Mac takes half the layers?#

This is pipeline parallelism: different devices own different stages of the model.

For a simplified eight-layer model, one machine could run layers one to four, then pass the intermediate result to another machine running layers five to eight.

The important word is then.

For one token's forward pass, the second half needs the first half's output. It cannot finish its part before that input exists.

Here is a deliberately simplified example, not a benchmark. Suppose the first four layers take 30 milliseconds and the last four take another 30. Putting those halves on separate machines doesn't automatically turn the path into 30 milliseconds. It is still roughly 60 milliseconds, plus the handoff.

We distributed the storage. We didn't remove the dependency.

The Megatron-LM research explains this distinction between splitting layers and splitting operations within layers. That paper studies large-scale training; I am using it for the underlying parallelism concepts, not as evidence of Mac inference performance.

Then why use a pipeline at all?#

Because fitting a model that otherwise cannot run is already useful.

There can also be multiple pieces of work moving through the pipeline. Think of a two-stage restaurant: one person prepares a dish, another finishes it. The second person cannot finish the first dish immediately, but the first person can begin the next dish while finishing happens.

That helps explain why throughput and one-request latency are different questions.

In an autoregressive response, the next token depends on the previous one. We cannot casually treat successive output tokens as independent orders. Other requests, or suitable chunks of input processing, offer different opportunities for overlap.

This is why I would ask what a scaling chart measures before asking how impressive it looks.

What changes with tensor parallelism?#

Instead of assigning whole layers to separate machines, tensor parallelism divides calculations inside a layer.

A large matrix multiplication can be partitioned so that multiple devices calculate different pieces. Their results must then be combined. The machines are collaborating on the current stage, rather than waiting to receive the next stage's input.

The attraction is straightforward: parts of the computation can happen simultaneously.

The cost is coordination.

Splitting the arithmetic creates messages, synchronization, and opportunities for devices to wait. The Megatron-LM paper discusses these communication tradeoffs. My mental model is two engineers dividing a calculation: it helps if exchanging partial results is cheap; it helps less if every small step requires a meeting.

So what did Apple actually measure?#

Time to first token measures the wait until generation begins. Decode speed measures token generation after that.

Here is the useful comparison from Apple's study, for Muse Spark Open NVFP4 with an 8,192-token input:

These are Apple-reported measurements, using MLX, batch size one, and 128 generated tokens. The distributed setup uses Thunderbolt 5 RDMA.

The same two-machine setup produces about 1.59 times the decode rate when the partitioning changes. Doubling the tensor-parallel machine count produces about 1.47 times the rate, not twice. Those ratios are my arithmetic from the table.

There is no single-Mac Muse baseline in this comparison. It would be misleading to turn the four-Mac result into a speedup over a one-Mac run that wasn't measured.

Also notice that changing the two-machine partition barely changes time to first token here, even though decode improves substantially. “Faster” needs a workload and a metric.

Are the cables now part of the model's performance?#

Very much so.

RDMA means remote direct memory access. Here, it provides a low-latency way for machines to exchange data. It doesn't make all the memory behave like one ordinary, equally fast local allocation.

The MLX distributed documentation describes the JACCL communication backend and requires a fully connected Thunderbolt mesh: every pair of machines needs a direct connection.

Diagram source and setup requirements · MIT license.

For four machines, a full mesh has six machine-pair connections. That is simple counting, but it makes the engineering point: adding a box can also add cabling and configuration work.

One practical caution: Apple's benchmark command enables MLX_METAL_FAST_SYNCH=1. The current MLX documentation warns that this experimental option can deadlock and recommends leaving it unset. I wouldn't copy a benchmark command into a persistent service without checking that warning.

What happens outside Apple's own test?#

Hugging Face's release walkthrough reports a different experiment: an imatrix-calibrated two-bit quantization on an M3 Ultra with 512 GB, using llama.cpp, at about 18 tokens per second, with 304 GB used by the model.

That is an ecosystem developer's firsthand report, not an independent reproduction of Apple's configuration. The quantization, hardware generation, and backend differ. The paragraph does not provide the exact prompt, context length, runtime commit, or repeated-run statistics.

Its original demo is useful to inspect: at 00, the interface shows generated code and a Q2_K checkpoint label; at 00, it shows a rendered brick simulation.

Those inspected frames illustrate one coding demonstration. They do not establish elapsed inference time or general coding reliability.

Unsloth's own quantization tests add another boundary: it reports degradation on its general-task checks below UD-IQ2_S, while smaller quants fare better on chat-style checks. Unsloth supplies those quantizations and had day-zero access, so this is not a neutral lab evaluation.

A smaller checkpoint can change whether a model fits. It can also change its behavior. These reports are not evidence that two-bit and NVFP4 results are interchangeable.

I couldn't verify an outside reproduction of Apple's exact multi-Mac benchmark or a controlled independent video comparison in the sources checked.

What would I measure before adding another machine?#

I would first choose the problem I actually need to improve:

• Does the intended checkpoint fit at my working context length?
• How long before the first useful answer appears?
• How quickly are subsequent tokens generated?
• What happens when several requests arrive together?
• Does the smaller representation still complete the real task correctly?

The weights also carry Meta's Muse Model License, not an unrestricted software license; deployment terms deserve their own check.

I haven't run Muse Spark Open myself. This is a reading of primary documentation, attributed developer tests, inspected demo frames, and illustrative reasoning.

What interests me is the distinction the launch makes visible: capacity lets the computation exist; partitioning and communication determine how efficiently it moves.

So before asking whether another machine makes an LLM faster, I would ask:

Which part of the work can actually happen at the same time?