Engineering notes

I measured Apple's on-device model so you don't have to

July 28, 2026 · Kaloyan Lachezarov

This is a living post. The corpus reruns on every iOS 27 beta, and the numbers below will keep evolving until the public release of iOS 27 - the run that decides the launch. Last updated August 28, 2026: it happened again - beta 7's model also arrived by download days after the OS build, and this time it was an improvement. The model changed silently twice between August 18 and 27; the repaired score never left 83-86%.

Deforget turns diary prose into reminders, calendar events, meeting notes, and an index of people - using Apple's on-device Foundation model, with no cloud fallback, because nothing leaves the device. That choice has a consequence people underestimate: you cannot pin the model. It belongs to the user's operating system. Whatever iOS ships, my users get, and every OS update can move the ground under the product.

When you depend on a model you can't version, you need instruments, not vibes. So Deforget carries its own evaluation harness: a fixed corpus of diary scenarios with typed expectations - dates, events, reminders, commitments, meeting notes, people, decisions, deduplication, and restraint (emotional prose must extract nothing). It runs on real iPad hardware against every iOS beta, and every case is scored twice: once on the raw model output, and once on the final result after a deterministic repair layer. The gap between those two numbers is the engineering, measured.

The numbers

Method notes that matter: OS comparisons use the same app binary - build once, run on the old OS, update the OS, run again without rebuilding, so nothing moves but the model. Five runs per case, because single runs lie. The corpus grows as field reports arrive, so totals shift between revisions; compare percentages, and only trust a raw-to-raw comparison within one corpus version.

The number I watch most isn't in any single row: the repair layer's contribution. It stayed close to thirty points while the model improved - 29 on iOS 26, 30 on beta 3, 31 on beta 4 - then jumped to 34 the night the model regressed, 37 on beta 6, 40 against the model that later slipped in under the same build number, and 43 on beta 7's carried-over model. Then the better model landed and the contribution fell back to 38 - the first time the number has shrunk, and it shrank because raw rose to meet it. That asymmetry is the design. When Apple ships a better model, the layer doesn't shrink; it stands on higher ground. When Apple ships a worse one, the layer carries more of the ceiling instead of falling with it.

What Apple actually fixed in iOS 27

• Restraint. The worst failure mode on iOS 26 was "I feel really good about how today went" becoming a commitment. Feelings turning into to-dos is the fastest way to make someone turn intelligence off - or stop writing honestly, which is worse. Across the first three 27 betas the restraint category went from 12 percent of checks to 84 to 100. (Beta 5 then broke it again - that story is below, and the repair layer now holds the line the model dropped.)
• Reliability. On 26, roughly one call in twenty entered a runaway loop - the model babbling until it blew its own context window. On every 27 beta so far: zero, across hundreds of calls.
• Vocabulary. The decision type effectively didn't exist in 26's output; everything flattened into commitments. 27 emits it unprompted.
• Consistency. 26's failures were flaky - one run in two. 27's failures are consistent, and a consistent failure is something you can engineer around. A flaky one isn't.

What's still broken, and what I do about it

Three failure classes survived beta 4, and each gets a different treatment - that's the point of measuring instead of guessing.

• Type confusion at the commitment/event boundary. "The offsite is on the 14th" comes back as a commitment, not a calendar event - consistently. After the third consecutive beta reproduced it, a pre-agreed deterministic retype rule went in: dated, scheduled phrasing outranks the model's label. That rule was written weeks earlier and sat disabled until the reports met its conditions. Fixes here follow a playbook, not a mood.
• Detached clock times. "Next Thursday's sync at 3:30" once resolved the time separately from the day and produced 3:30 AM. The repair layer now grafts a detached clock onto the sentence's date before it can misfire. Deterministic, tested, no model involved.
• Recall flakes. One decision case sometimes returns nothing at all. The discipline for missing output is different: wait. Across this beta cycle, recall has recovered on its own while precision held - and a silent miss is recoverable in the app, while junk output is trust-fatal. Precision gets engineering; recall gets patience. (Patience, it turned out, has a floor - beta 5 found it.)

Beta 5 took the ground away

Everything above described a model improving beta over beta. 27.0 beta 5 (24A5408d) reversed it. The first corpus run after the update posted the worst restraint since iOS 26 - hedged musings came back as decisions, past narrative ("We walked along the river...") came back as commitments, mood prose got stamped with phantom event dates - while dates, calendar events, and decisions collapsed toward emitting nothing at all. Final fell to 63 percent; beta 4 had stood at 89. Nothing in my pipeline had changed between those runs. The raw column is the receipt.

This is the scenario the regression playbook was written for, so the night ran on rails instead of adrenaline, in the pre-agreed order. Restraint first, because junk is trust-fatal and silence is merely disappointing: each new junk class got a deterministic guard - past-tense openers stop becoming plans, the hedge rule now covers decisions, an event dated by nothing but the prompt's own anchor dies. Then recall. Waiting is the usual answer for a model that says nothing, but a collapse this deep has a pre-agreed answer too: a deterministic floor that acts only when the model returned nothing actionable and the sentence itself names a future date plus an unambiguous frame - "remind me", a meeting word backed by a clock time, a first-person modal, "by Friday". No date, no entity, so the floor cannot regress restraint by construction. A past date is a memory, not a plan, and stays one.

I reran the corpus after every change and stopped when the gate passed. The two beta 5 rows in the table are the first and last runs of that night. Between them the raw score barely moved - same OS, same model, no prompt changes - which is exactly the point: every recovered point is inspectable, testable code, not a plea to the model. What the floor deliberately does not do is also on the record: sentences with no date stay silent even when recall is hurting, because a diary that invents structure is worse than a diary that misses some.

Beta 6 held still

27.0 beta 6 (24A5418b) arrived on August 17 with a tell: the update pulled down no new AI assets, and the app's own model-status check reported the Foundation model ready the moment the OS came back up. That usually means the model didn't change. But a suspicion isn't a measurement, so the corpus ran twice that night - first a fresh beta 5 anchor on the exact binary and corpus the comparison would use, then beta 6, with nothing moved but the OS.

Raw went 47 to 46 percent, noise-level agreement, and every failure landed in an already-known shape. The model is frozen at beta 5. The repaired pipeline posted 83 percent with restraint at 33 of 35 and zero call failures - both launch-gate conditions pass. A couple of decision cases even came back on their own, which is the wait-don't-patch discipline from earlier paying out: recall recovered without a line of code changing. (One meeting case technically failed because the run straddled midnight and the scorer's expected date rolled over while the model was mid-answer - the extraction itself was exactly right. Honest measurement includes catching your own instrument's artifacts.)

After three betas of the ground moving, a build where it holds still is not a non-result - it's the signal the whole series was waiting for. This late in the cycle, a frozen model is very likely the model iOS 27 ships. The launch decision is running on these numbers.

The new model came later

On August 22 the corpus ran again on the very same beta 6 build - 24A5418b, the identical OS version string, nothing updated in between. Raw came back at 40 percent. A second run that evening read 43. The beta 6 anchor had read 46, and no chain of runs on one model had ever strayed more than a couple of points. The failure shapes had changed too: decisions collapsed back toward emitting nothing, a feeling became a decision again, and mood prose got stamped as a phantom event at whatever time the extraction happened to run. So the conclusion above rewrote itself: beta 6 did bring a new model - it just didn't arrive with the update. The install kept running the beta 5 model, and the replacement arrived at some unknown point between the two measurements. No release note, no build number change, no signal at all except the corpus disagreeing with itself.

Two lessons went straight into the method. A pinned-model conclusion is only as good as its timestamp, so "the model is frozen" now always carries a date, and the only reliable tell is a re-run - not the download size, not the status check. And single runs matter less than they used to: the new model disagrees with itself run to run more than any before it, so comparisons now quote a band, not a point. The repair layer's answer to all of this is the same as ever: final held at 83 percent in both runs, and the launch gate still passes. The public-release run in September now has one more job - telling me which of these models iOS 27 actually ships.

It happened again

27.0 beta 7 (24A5424a) arrived on August 24, and that night's run read exactly the way the new method predicted: raw 42 percent, inside the late-beta-6 band, every failure a known shape. The model had carried over - the install didn't change it. But a carried-over conclusion now comes with a timestamp, so a re-run was already on the calendar.

It fired early. On August 27, on the same OS build and the same frozen binary, raw jumped to 49 percent - twelve points above the band, and up, not down. The shapes moved with it, in the right direction: the feelings-as-decisions junk that defined beta 6's model was gone from the raw output, and decision recall - the weakest category since beta 5 - roughly doubled. A second run the same day read 50; a stability run on August 28 read 48 and posted the best final score of the whole series. Two-point spreads, where the previous model disagreed with itself by six. A new model, delivered the same way as beta 6's - silently, by download, under an unchanged build string - except this one was better.

So the model has now changed silently twice between August 18 and August 27: it was still beta 5's on the 18th, beta 6's replacement was caught on the 22nd, beta 7's on the 27th. Raw read 46, then 40, then 49 percent. Final read 83, 83, 86. That pair of sentences is the whole post in miniature: the ground moved twice, once in each direction, and the repaired score moved three points. The September runs - release candidate and public build - are no longer a formality: the model is demonstrably still in motion two weeks before release, and the corpus will be waiting for it.

The corpus is alive

Every interesting field report becomes a permanent test case. A tester's iPad invented a person named "Writer" for a paragraph that named no one - now there's a sentinel case that fails if any person is attached to text that names nobody. Numbered tasks that a deduplication pass once merged ("finish the Q2 report, start the Q3 plan") are a case. This week the diary learned to index links and photos from rich-text paste, which means bare URLs now reach the model - so the corpus gained three URL cases before any guard was written, including a blog permalink with a date in its path, because a phantom calendar event minted from /2026/03/15/ would land in a real calendar. Measure first; guard only what the measurement says needs guarding. Two of those URL cases have since fired - one in the corpus, one in the field - and each bought exactly the guard it was designed to demand. The newest case came out of my own pocket: "remember to go to lidl at 7", captured from the home-screen widget one evening, produced a reminder for a 7 AM that was long gone - a bare clock with no day word is invisible to the system's date detector, so no repair ever fired. Now the repair layer resolves a bare clock on the diary page's own day, the sentence is sentinel case 41, and it has passed every run since.

The lesson, if you're building on this stack

The app's guarantees cannot live in the model, because the model isn't yours. They live in a deterministic layer that repairs dates from the source text, recognizes self-declaring types ("we decided..." is a decision no matter what the label says), titles and links meeting bullets without model involvement, kills anything that can't be grounded in the words the user actually wrote - and knows when to do nothing at all. The model contributes recall. The layer contributes truth. Sixty-ish percent raw becomes ninety-ish percent final when the model cooperates - and the night it fell to forty-eight, the layer held eighty-two. The pipeline never asks the model to be something it isn't.

All of this runs on the device - the diary never leaves it. The harness ships inside the app, and every number above comes from reports it saved on an ordinary iPad. When iOS 27 goes public I'll run the same corpus one more time, and that run decides the launch.

What all of this looks like in the app →