Attestation or Authority: Apple Foundation Models vs. Self-Hosted MLX

Apple foundation models vs MLX models

Same task, same Mac Studio, one argument apart. Apple’s on-device model never called the tool. A self-hosted model always did. The interesting part is who could have changed either result.

TL;DR

- Apple’s on-device model returned a well-formed contract summary in 20 of 20 runs and called the mandatory tool in 0 of 20. The only switch that forces the tool never stops calling it.

- Private Cloud Compute gives you proof a third party can check. Self-hosted MLX gives you control over everything. The on-device model, the one every developer starts with, gives you neither.

- Pick the failure mode your organization can carry: a jurisdiction you do not control, or a security posture you have to build.

Since macOS 27 shipped on September 14, 2026, the Swift that calls Apple’s on-device model and the Swift that calls a 27B model from Hugging Face differ in one argument. LanguageModelSession() gives you Apple's model. LanguageModelSession(model: mlx) gives you yours. Tool protocol, output type, respond(to:generating:), error handling: same file.

I ran the same task on both paths. A fictional master services agreement gets summarized into a structured record, and the notice period in clause 12.2 exists only in a tool the model must call. Apple’s on-device model called that tool in 0 of 20 runs and returned a well-formed summary with a guessed notice period in 20 of 20. The self-hosted model called it in 40 of 40 runs across two setups.

The obvious reading, “a 3B model loses to a 27B model”, misses what was measured. The measurement is who can change the outcome, and who can prove it.

Two words, then the scoreboard

There are two things you can want from a model you deploy. Attestation means someone else can check what happened without taking your word for it. Authority means you can change what happens: the model, its limits, its logs. Apple’s cloud tier gives you the first and none of the second. Self-hosting gives you the second and none of the first. Apple’s on-device model gives you neither: you verify it by the absence of evidence, and you decide nothing about it.

The rest of this article is the evidence for those four rows, one row at a time.

What I measured

One Mac Studio M4 Max, 64 GB, macOS 27.0, Xcode 27.0. The MLX side runs mlx-swift-lm at commit c6446cf7, mlx-lm 0.31.3, and mlx-community/Qwen3.8-27B-8bit, 28 GB of weights loaded from disk. Exact versions and build numbers are in the lab notes at the end; the harness and results are in the repository.

Get Michael Hannecke’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The task is identical on every path. The instructions say the model “MUST call the lookupDeadline tool” when a clause refers to a schedule. Clause 12.2 refers to a Schedule 4 that is not in the document. The tool is a three-entry table where 12.2 is 90 days. A run succeeds when the tool was called with 12.2 and the summary says 90.

// Task.swift, identical for all paths (macOS 27 SDK)
struct LookupDeadlineTool: Tool {
    let name = "lookupDeadline"
    let description = "Returns the notice period in days for a clause reference, looked up in Schedule 4 (Notice Periods). Use it for any clause that refers to a schedule for its notice period."
    @Generable struct Arguments {
        @Guide(description: "Clause reference, e.g. 12.2") var clauseReference: String
    }
    func call(arguments: Arguments) async throws -> String {
        // Schedule 4 lives only here: ["12.2": 90, "12.3": 30, "12.4": 10]
        return "Notice period for clause \(key): \(days) days"   // trimmed
    }
}
 
@Generable struct ContractSummary: Codable {
    var parties: [String]
    var initialTermMonths: Int
    @Guide(description: "Notice period in days under clause 12.2, exactly as returned by the lookupDeadline tool")
    var terminationNoticeDays: Int
    var risks: [String]
    var summary: String
}
 
// Runner.swift: a fresh session per run, no context carried over
let response = try await session.respond(to: prompt, generating: ContractSummary.self)

Apple on-device: right format, wrong number, no lever

• What happened. The session is LanguageModelSession(tools: [tool], instructions:). The model answered in a median of 8 seconds, parties and term correct in all 20 runs, output always well-formed. The tool was never called.
  The notice period came back as nine distinct values between 27 and 360 days, never 90. Document length was not the cause (a 552-token control gave 0 of 5), and guided generation was not the cause (an unguided session gave 0 of 10).
  The model does call tools: given a getWeather(city:) tool the same afternoon, it called it 5 of 5 times.
  My reading, and it is a reading, is that a notice period looks answerable from the document and the weather does not.
• What I could change. macOS 27 adds GenerationOptions.toolCallingMode with .allowed, .required, and .disallowed. With .allowed, the default, five more runs behaved exactly like the twenty. With .required, five runs called the tool 77 to 102 times each, walking through clause numbers 12.1 to 12.77, and none of them ever produced a summary.
  Each run ended when the transcript hit the 8,192-token context limit, after three to forty-six minutes. Forcing the tool for one turn and answering in a second turn on the same session did not help: the first turn never terminated. So the developer has a default that ignores MUST, a switch that never stops, and nothing between them.
  There is no model selection and no guardrail configuration, so there is no way to obtain what is missing, which is not a knob. It is authority over the model that interprets the knob.
• What I could prove. After the 20 runs, the Apple Intelligence Report export contained two arrays, modelRequests and privateCloudComputeRequests, both empty. The report lists requests the Mac sent to Private Cloud Compute; on-device requests get no entry at all. The model ran in TGOnDeviceInferenceProviderService, a system extension that opened no socket in 87 nettop samples over 14 minutes.
  That is the proof that the request stayed on the device: an empty JSON file and a process I watched with tools I control. Absence is not a receipt.
• What that means. The path that needs no entitlement, no download, and no manifest changes is the one where you can prove nothing and change nothing. It produces the number in the right shape either way.

Apple Private Cloud Compute: proof you can hand to someone else, if you qualify

• What happened. On this machine, PrivateCloudComputeLanguageModel().isAvailable returned true and contextSize returned 32768. The first request failed after 24 milliseconds with two undocumented error codes. Whether the cause was the missing entitlement or the Siri AI beta waitlist this account was on cannot be told apart from outside. No PCC runs in the tables.
• What I could change. Nothing about the model. AFM 3 Cloud is Apple’s, the guardrails are Apple’s, and the schedule is Apple’s.
• What I could prove. This is the one path with attestation. A PCC request produces a report entry with an attestation bundle, and a practitioner tool, pcc-verify, checks that entry against Apple’s public transparency log.
  The export itself is unsigned JSON, so the tool proves the log entry, not the device, and that is the subject of a separate article. It is still the only receipt in this comparison that a third party can check without trusting me.
• What that means. Reaching PCC as a developer has three conditions: App Store Small Business Program, fewer than 2 million first-time downloads, and an entitlement assigned by Apple.
  Distribution is App Store, TestFlight, or ad hoc; in-house distribution is not on the list, and Apple does not say whether it is excluded. If you qualify, you get proof and give up control. If you do not, you are back on the row above.

MLX: right number, and every failure is yours

• What happened. Path A is the same Swift with one argument changed: LanguageModelSession(model: mlx, tools: [tool], instructions:), where mlx is an MLXLanguageModel. It called lookupDeadline("12.2") in 20 of 20 runs. Fifteen produced a valid summary with 90 days. Five ran for six to twenty minutes and ended with a parsing error: guided generation filled the risks array with fragments and never reached parties.
  Success 75 percent, median 107 seconds, about 31 GB of unified memory. Path B is the operator's view: mlx_lm.server on 127.0.0.1:8080, a client that posts to /v1/chat/completions with tools, appends the result, and posts again. Twenty of 20 called the tool, 20 of 20 said 90, median 72 seconds. The long contract, which the Apple path rejects in under half a second, passed 3 of 3 on both MLX paths.

// main.swift, path "mlx": local weights (mlx-swift-lm c6446cf7)
let mlx = MLXLanguageModel(
    configuration: ModelConfiguration(directory: dir),
    capabilities: [.toolCalling, .guidedGeneration],   // declared by you, not detected
    weightsLocation: { _ in dir },
    load: { configuration, progress in
        try await loadModelContainer(from: #hubDownloader(),
                                     using: #huggingFaceTokenizerLoader(),
                                     configuration: configuration,
                                     progressHandler: progress)
    })
 
// The only session line that differs from the Apple path:
LanguageModelSession(model: mlx, tools: [tool], instructions: TaskSpec.instructions)

• What I could change. Everything, and each thing is now a decision. The manifest needs a revision pin and three extra packages because no release tag ships the library yet. The capabilities array is declared by me, and the adapter believes it.
  The five runaway runs happened because the harness set no token cap, no timeout, and no retry; on the Apple path something Apple configured would have cut them off. The server prints “not recommended for production as it only implements basic security checks” on startup.
  Five control points turn “local” from a location into a security statement: bind address, egress rule, a request log with integrity protection, model pinning by hash, and an update process for weights, runtime, and tool parser. I set the first and carry the second as a pf anchor in the repository. The other three are a job.

$ lsof -nP -iTCP -sTCP:LISTEN | grep Python
Python  30460 michael   11u  IPv4  TCP 127.0.0.1:8080 (LISTEN)   # tools/start_server.sh
Python  31362 michael    5u  IPv4  TCP :8081 (LISTEN)           # HOST=0.0.0.0, the counterexample

• What I could prove. To myself, plenty. After one evening the server’s DEBUG log held 33,151 lines with every prompt in clear text, and the prompt cache held 6.87 GB of KV state in process memory, retention that no log mentions.
  A tcpdump capture with zero egress would be a log I wrote with tools I control at a time I chose. It is evidence to the operator and testimony to everyone else. That is the correction to a position you will find on the first page of search results: self-hosting, one guide says, lets you "independently verify that input and output data stays within your own environment, without relying on third-party promises."
  Self-hosting replaces a third party's promise with your own, and your own is the one no third party can check.
• What that means. The path Apple’s own code labels “not recommended for production” was the only one at 100 percent. It got there because I could change everything, and it will fail somewhere because I now have to get everything right.

Same task, three columns

Three parallel pipeline runs

Read the columns as inverse bars. Where Apple attests, you decide nothing. Where you decide, nobody attests. The on-device column holds neither property.

¹ First streamed tokens arrive as reasoning deltas, so the first content token was not measurable. ² Prompt-cache hit. All MLX rows are Qwen3.8–27B-8bit; tok/s is output tokens of both turns over wall-clock of both turns, not decode speed. Apple rows use response.usage; the model runs out of process, so 34 MB is the caller. The three MLX rows are three operating modes of one model, not a benchmark.

Every column fails somewhere the documentation does not mention. The Apple model returns a wrong number in the right format. The Swift MLX path returns nothing after twenty minutes. The server path returns the right answer while keeping your contract in a cache and a clear-text log. On exactly one column, somebody can change the model that is failing.

Choosing a failure mode

Neither path is safer. Each has a failure mode, and the decision is which one your organization can carry. Three questions decide it per workload.

• Does the task fit the on-device model?
  For a task whose correctness depends on a tool call, the answer here was 0 of 20. Use the on-device path where the model supplies the format and a guessed value does no harm: a draft, a suggestion, a classification a person reviews. Keep it away from tasks where a number has to be right.
• Who carries the proof?
  If a reviewer asks where the request was processed, on-device gives you an empty report and a process without a socket. PCC gives you an entry a third party can check, if your distribution channel qualifies. MLX gives you your own capture. Find out which of those your reviewer accepts before you build. The answer differs between “our security team” and “an external auditor”, and in my runs it decided the path before model quality got a vote.
• Who runs the posture?
  On MLX the five control points need an owner with time. Bind address and egress rule are an evening. A signed request log, model pinning, and an update process are a job. An organization without someone to hold that job carries a worse failure mode on MLX than on the Apple path: a server on 0.0.0.0 with a clear-text prompt log fails as silently as a model that guesses in the right format, and only the server has an owner who could have noticed.

Three objections, briefly. MLXLanguageModel ends the dichotomy, you get Apple's API with your model. The API is unified; the security model is not. The model: argument is where you opt out of attestation and into authority, and the session API hides that choice from the auditor who reads the code in six months. Self-hosting is just as provable, tcpdump showed zero egress. That is the operator's observation from above, and presenting it as attestation means handing a reviewer a log you wrote yourself. The Google Cloud tier makes the choice trivial outside the US, so MLX. Apple documents AFM 3 Cloud Pro on Nvidia GPUs in Google Cloud, and does not document which nodes serve developer requests. The on-device path has no jurisdiction question at all, only an 8,192-token window and a model that guesses. The jurisdiction argument and MDM control of Apple Intelligence I analyzed in more details here.

One argument, two audits

Apple has reduced the move between two security models to one argument in one initializer, which makes the decision easier to make and easier to overlook. The Claude and Gemini packages announced at WWDC26, not shipped as of September 2026, will add more values for that argument without changing what it selects. In six months, when someone asks which model: was in production for the contract summarizer and why, the answer will live in a log a third party can verify or in a commit message. Both are records. Only one of them is a receipt, and you chose which on the day you wrote that line.

I write about running AI locally on Apple Silicon and securing the agents that run on it: inference, harness security, and what it takes to keep control of your own stack. Follow for practitioner-depth technical content, no vendor fluff.

Drafted with AI assistance; analysis, conclusions, and curation are my own.