Posts in stories

Notes from Apple’s Siri AI Labs: Schemas, Tools, and Skills

Earlier this week, I attended Apple’s online Apple Intelligence Group Lab, much of which focused on Siri AI. Much of the ground covered is available in Apple’s documentation, but the presentation included additional detail, which is worth highlighting, although I encourage you to watch the two Apple Intelligence Group Labs for more context and details.

The big-picture change that Federico explained earlier this week is that Apple is greatly expanding the actions available to Siri. Today, Siri AI’s natural-language abilities are limited to tightly defined domains like calendar, reminders, and messages. If a developer’s app doesn’t fit a schema (or hasn’t implemented an App Shortcut), Siri won’t be able to respond to a user.

Apple isn’t dispensing with schemas, but it’s loosening the reins, giving developers far wider latitude than before. Apple noted during the lab that schemas are still developers’ best bet for getting Siri to respond because Apple’s already done the work to ensure they work.

What’s changing is that Siri will be able to directly access developers’ App Intents that have been defined as tools. That’s a big change from today where a developer’s main access to Siri is through App Shortcuts, using a defined phrase that names the app. Instead, if a developer marks their App Intents as tools, Siri will be able to access those App Intents directly when a user makes a natural-language request, much like how MCP works with tools like Codex and Claude Code.

Other interesting tidbits about tools mentioned include that they’ll use Private Cloud Compute to process the request and that apps are limited to 40 tools.

Skills are a third brand-new way to access Siri that’s designed for more complex, multistep actions that can chain up to 32 tool calls. Like the skills you may be familiar with from Codex and Claude Code, Siri’s skills are Markdown instructions with YAML front matter that explain what Siri should do with certain requests. Xcode 27.3 beta also includes coding-agent skills that help developers add Siri support to their apps and test them.

Finally, beginning with iOS 27.3, which will ship in January if history is any indicator, developers will be able to customize the look of snippets of information from their apps similar to how Apple’s own apps’ content can look in Siri. As for tools and skills, those will be available to ship to the App Store in the spring.

Siri AI’s Evolution Is Both Surprising and Obvious

Siri AI in iOS 27.3 beta.

Siri AI in iOS 27.3 beta.

As part of the release of the 27.3 developer betas yesterday, Apple announced a substantial expansion of the capabilities of Siri AI. As you recall from my iOS 27 review two months ago, Siri AI launched with support for a limited subset of app domains based on schemas that Apple specifically optimized for the first version of Siri AI. In a major reversal for a mid-cycle update to Siri AI, that limitation is entirely going away: the updated Siri AI will instead work with any kind of app that supports any custom App Intent – as long as the developers do the work of modernizing their App Intents and integrating with Siri. Effectively, though, any app can work with Siri AI now, and Apple calls this the Siri AI Developer Preview.

Read more

MacBook Pro with Touch: A Mac for the iPad Generation

Ever since I got my hands on the very first iPad Pro all the way back in November 2015, I’ve gotten used to a certain type of computer interaction that Steve Jobs once decried as “ergonomically terrible”: working with a keyboard, but sometimes also touching the screen. Of course, that interaction became even more common when Apple introduced the Magic Keyboard with trackpad for iPad Pro in 2020: by that point, I was fully alternating between trackpad control and direct touch input on a tablet that allowed me to “rip off” the display and walk away with it whenever I wanted.

Steve Jobs was right about many things, but I’ve long disagreed with his take on the “terrible” ergonomics of touchscreen laptops. Sure, you don’t want to be forced to interact with a laptop display that sits perpendicular to the keyboard at all times. But for the occasional interaction that benefits from direct manipulation of content – whether that’s zooming on a picture or rotating a 3D object or game character with your fingers – it’s nice to have a secondary input method that does not rely on a UX abstraction such as a trackpad. Think of it as motion controls on a Nintendo Switch: you primarily play games with a controller and physical buttons, but, sometimes, it is nicer – and easier – to rely on the gyroscope for motion controls.

Read more

A Day with the iPhone Duo: First Impressions

I spent Wednesday in New York, interviewing developers about their iPhone Duo apps. It was like a little slice of WWDC dropped among Manhattan’s skyscrapers, which was wonderful. There’s nothing I enjoy about what we do at MacStories more than talking to developers about their apps and the stories behind them.

I left the city with a much better appreciation for what it’s like to build apps for the Duo, which will be incredibly valuable as Federico and I work our way through the enormous list of Duo apps that developers have shared with us. I also left New York with a 512GB Night Sky iPhone Duo for testing. As I left the press event, I suddenly had a phone to set up, but hadn’t done anything to prepare and had no fast Wi-Fi to download gigabytes of apps and files. Long story short (listen to MacStories Unwind for the long version), I headed to the airport and by the time I boarded the plane, I was on the iOS 27.2 beta and had the core system apps installed.

So, suffice it to say that this is not a review of the iPhone Duo. I haven’t had it much more than 24 hours, so I haven’t fully settled into a routine with it. That said, I wanted to share my first impressions and some early surprises.

Read more

Building for iPhone Duo, in App Developers’ Own Words

I spent yesterday in New York City interviewing developers about their iPhone Duo apps and the experience of building for such a new and different device. The depth and breadth of developers Apple assembled was excellent. I spoke to indie developers like Vidit Bhargava about Deck-Man and the team behind Notewise. I also interviewed developers from big companies about their impressive Duo updates, including The New York Times for its NYT Cooking app, Netflix, and Meta’s Edits video editing app. Plus, I met Mark Otero from Azra Games, whose team poured their hearts into UnGodly, one of the best-looking games I’ve seen on iOS in a while.

A few things struck me about my conversations with developers. First, every one of them was excited about the design challenges posed by the iPhone Duo and the many orientations and ways it can be used. Second, every one of them came away impressed by how smooth Apple has made the development process. That’s not to say the design challenges weren’t real, but getting from a design idea to an app that worked on the Duo went better than anyone expected. Third, after months of building and testing, everyone was looking forward to seeing which design choices land with users.

But that’s enough of my impressions. I’ll let the developers speak for themselves in this special bonus AppStories video. I came away from New York with some great interviews, which I’ve shared on the MacStories YouTube channel. Apologies for the sometimes rough audio. There’s only so much I can do with recordings made in a room packed with press and developers, but if you’re into apps, design, and how things are built, these interviews are for you.

Also, I left New York with a Night Sky 512GB iPhone Duo that I’ve only just set up and begun testing. I’ll be writing about the Duo soon, and we’ll start covering the best of the more than 300 Duo-optimized apps that developers have sent Federico and me. Stay tuned, because the busy fall season at MacStories is just going to keep rolling.

HomeView Points to Siri AI as the Next Developer Platform

Source: Apple.

Source: Apple.

Yesterday, Wired reported that third-party apps with CarPlay functionality would be available on Apple’s new HomeView device. That struck me as odd. I was expecting, and I think many others were at least hoping, that the HomeView would be a new platform for building apps. And while strictly speaking that doesn’t seem to be the case, in an entirely different way it is. Here’s what’s going on.

Apple’s original press release caused some confusion because it doesn’t spell out the CarPlay angle. Instead, it simply says that the HomeView supports “compatible third-party audio and video apps,” and “[s]elect apps require a paired and compatible iPhone nearby,” which is also a CarPlay requirement. That press release also mentioned the HomeView’s ability to use widgets and access personal context without ever mentioning whether that applies to third-party apps.

Well, now we have some answers to most of these questions. First of all, Apple documents out-of-the-box support for audio and video apps, rather than the full range of CarPlay categories. That’s further supported by Katarina Mogus’ interview with Apple’s Cecilia Dantas, which draws a straight line between third-party video apps and CarPlay support.

Second, the Human Interface Guidelines have been updated to address which widgets can be used with the HomeView:

On Apple HomeView, people can place widgets on the Modular face and the Home Screen. Widgets on HomeView are from the small system family, and share the appearance of iPhone widgets. A person chooses widgets from among those already installed on HomeView, like Weather or World Clock, or from their individually connected iPad or iPhone.

As for whether Siri can access third-party data and actions, I suspect the answer is yes. Apple’s press release called out personal context specifically, which was confirmed by Apple’s Kaiann Drance and Lilian Rincon in an interview with Reuters. So, for example, you can presumably use the HomeView to retrieve an old message from the Messages app on your iPhone. The Q&A section on the HomeView product page also mentions taking action across apps with Siri. That strongly suggests that the HomeView’s functionality will be the same as the new HomePod mini 2’s when it gains Siri AI capabilities, even though I haven’t seen anything from Apple that addresses whether that sort of Siri integration extends to third-party apps directly.

So, as of today, Apple has documented support for CarPlay audio and video apps and small widgets. I suspect Siri AI will provide another route, too. That limited app model may disappoint some, but I think it points to where personal assistants are heading. Apps aren’t going away, but the need for a separate app on every device is.

Siri, with apps integrated through App Intents, is emerging as a new UI. Developers who have adopted that technology already give users ways to interact with their apps beyond the apps themselves. I expect HomeView will extend that opportunity even further without requiring developers to build for an entirely new device. If it does, that’s a win for developers and users alike, and perhaps just the tip of the iceberg in a world where computing extends beyond apps.

OpenAI’s Dots: A First Look

Source: OpenAI.

Source: OpenAI.

Earlier this week, I was at OpenAI’s annual DevDay conference where it launched Dots, a personal agent that lives inside the ChatGPT app. I set up my Dot and named it Puck after the mischievous sprite in Shakespeare’s A Midsummer Night’s Dream. In the days since then, I’ve found myself using Puck more and more. There are still rough edges, but I thought I’d share what’s working and what’s not, and how Dots are different from what’s already available in ChatGPT and Codex.

Dots are persistent, cloud-based agents that use the plugins and other tools you’ve connected to ChatGPT, after you allow them to do so. Your Dot has its own virtual Linux computer to assist you with your work. And at first, your Dot, which is depicted by a colorful character you can customize, doesn’t have access to anything else. However, as you give your Dot tasks that need access to other tools, it will ask for your permission to access other tools. The pacing of this initial interaction with your Dot is really well done. The requests felt natural and allowed me to get comfortable with expanding what I let Puck do without feeling like annoying interruptions.

Customizing my Dot.

Customizing my Dot.

Dots are available on mobile and desktop devices, and because they are cloud-based, your Dot’s chat is always synced between devices. But being a cloud-based agent has its downsides, too. Out of the box, your Dot can’t access your computer’s files and apps. That’s easily fixed by clicking or tapping on your Dot and granting it access to your Mac, so it can use the Mac alongside its own VM. As with Codex’s Remote feature, using a remote Mac with your Dot requires the ChatGPT app to be open on that Mac, too.

Read more

M5 Ultra Mac Studio Review: The Dream Mac for Local AI Agents

The M5 Ultra Mac Studio.

The M5 Ultra Mac Studio.

For the past few days, I’ve been testing the (currently) top-of-the-line M5 Ultra Mac Studio with 256 GB of RAM.

I’ll cut to the chase: the M5 Ultra Mac Studio is a dream machine for local AI agents. This computer makes it possible to run personal assistants powered by local models with great performance and no additional cloud costs. If you’ve been skeptical of testing OpenClaw or Hermes Agent with local models because they’d never be even remotely near the intelligence and speed of cloud ones, this Mac will change your mind about that.

Since last Thursday, I’ve been comparing this Mac Studio to its predecessor, the M3 Ultra with 512 GB of RAM, as well as my own desktop gaming PC with an RTX 5090 inside. For its size, price, thermal performance – not to mention Apple’s approach to unified memory – the M5 Ultra Mac Studio has fundamentally changed how I think about models running locally and what they can enable now. A 5090, of course, still has an edge over the M5 Ultra thanks to its higher memory bandwidth. But considering the sheer size of my PC build, as well as its heat and noise, I would prefer an M5 Ultra Mac Studio any day. It also happens to be a Mac, with an operating system that looks nice and doesn’t suck, plus a vibrant app ecosystem. (Windows fans, I’m sorry, but Microsoft software will never get my sympathy.)

As I’ll explore in this article, running the latest Qwen3.8-Flash-Next model on the M5 Ultra Mac Studio has been so nice and fast, I’ve made it my default in both Open Minis for iOS and Hermes Agent. That’s right: the personal assistants I use the most – more than Siri AI, in fact – are now entirely powered by a model running locally on a Mac Studio. Furthermore, thanks to the M5 Ultra’s faster GPU and higher memory bandwidth, these agents start responding more quickly, stay fast at larger context windows, and can run long, multi-turn loops without slowing to a crawl as the session grows. Because of this, I’ve also been using local models in the Codex app on my Mac – either as main threads or subagents orchestrated by GPT-6 Astra – and I’ve had a great experience doing so.

Local subagents running in Codex on the M5 Ultra Mac Studio.

Local subagents running in Codex on the M5 Ultra Mac Studio.

I should note upfront that I’m not an AI developer by trade: I do not train or fine-tune models. I’m a tinkerer at heart, and I’ve been playing around with local AI models for over a year at this point. This summer, I went all-in on local AI usage for a big project I was working on, which I will explain in the following section.

My goal with this article is to provide you with a mix of two things: numbers and visualizations based on the (many) tests I’ve run over the course of four days, and an explanation of my practical use cases for local AI applied to my workflow and how I get things done for MacStories.

Let’s dive in.

Read more

TestFlight Fixes and Wishes

Just as I was heading into the final stretch of review season, TestFlight was updated to reverse an update I wrote about in July. That update added search, which was one of my longstanding wishes for TestFlight. However, for some users, the update also changed the sorting of betas from reverse chronological order to alphabetical, including active betas alongside expired ones. As someone who reviews a lot of apps, that made TestFlight barely usable.

With the latest update, I’m happy to report that the sorting order has been restored to reverse chronological order. The most recently updated betas appear at the top of the list, and when you want to find something older, search is a great option. Returning to reverse chronological order may seem like a small thing, but it’s made a world of difference in keeping up with the apps I test.

Search was near the top of my list of TestFlight wishes for years, and its addition made a meaningful difference to the app’s usability. But I’ve had a lot of TestFlight wishes over the years that are scattered across multiple articles, so I thought I’d share what else I’d like Apple to tackle with TestFlight:

  • Performance. I test a lot of apps, which slows TestFlight down on every Apple platform. The issue isn’t as severe today as it was when I wrote about it in 2023, but the slowness is still bad enough to make the app hard to use. Whatever the issue is, I’d love to see the list loading time reduced and scrolling through a list of apps become as smooth as it is in the App Store.
  • Add filtering, sorting, and folders. Search is a step in the right direction, but allow me to sort by date, developer, category, and other metadata. I’d also like to be able to filter by installation status, so I can just see the betas I’ve installed or haven’t installed yet. User-defined folders for organization would be a nice bonus too.
  • Hide inactive and expired betas. I’ve thought about TestFlight’s expired betas section a lot and can’t think of a single reason it exists from a tester’s perspective. Let me hide or exclude expired and withdrawn betas.
  • Fix TestFlight emails and automatic updates. TestFlight email notifications can only be turned on or off on a per-beta basis. Adding a global switch would help a lot with managing these messages. Also, automatic updates of betas have never worked on my iPhone, and I’ve talked to others who have the same problem.
  • Restore usable betas after a device migration. Apple’s migration from one device to another has gotten much better over the years, but it still does not migrate TestFlight betas, leaving me with a bunch of broken apps on my iPhone every fall.
  • Indicate when a beta has been released on the App Store. This strikes me as easy to request, though I’m sure is difficult to implement, but I’d love to know when a beta has been superseded by an App Store version, so I can grab the stable release if I want and know I can publish a review about an app without having to check the App Store repeatedly myself.

The bottom line is that while TestFlight is once again usable, it could be much better. I recognize that TestFlight isn’t an app used by most people, but like Apple’s own OS beta testing program, TestFlight betas are an important part of developers’ workflow. They provide QA feedback and a promotional channel that the friction encountered by TestFlight users undermines.

I’m glad TestFlight continues to get meaningful updates. I just hope it gets more attention going forward.