This Week's Sponsor:

Inoreader

Build your own newsfeed and cut through the noise

Posts tagged with "Apple Intelligence"

Three Features Coming to Photos This Fall via Apple Intelligence→

Photo editing is getting some of the most interesting Apple Intelligence features this fall. At WWDC, Tyler Stalman interviewed Jon McCormack, Apple’s Vice President of Camera and Photos Software Engineering, and Della Huff, the company’s Senior Manager of Camera and Photos Product Marketing, in a video that covers Spatial Reframing, Extend, and an improved Clean Up tool, along with demos of each:

The interview goes into depth on each feature and how they’re enhanced by the integration with Apple’s improved Private Cloud Compute models, sharing nuggets like the fact that Spatial Reframing of photos is possible even if an image is taken with a single-lens iPhone and the fact that Apple includes metadata in generatively edited photos identifying the use of AI and will add SynthID watermarking, a technology developed by Google, to AI-edited photos later this year.

I’m not a fan of wholesale image generation, which is why I appreciate the thoughtful approach Apple is taking with these photo editing features. The changes are at the margins of the photo and identified as edited using AI.

Permalink

The Third Generation of Apple’s Foundation Models and AFM Core Advanced→

I just came back to my hotel room after a long day at Apple Park (I documented most of it in my Instagram stories, including a very cool shot), and, like everyone else here in Cupertino, I’m still processing the information overload from the past 12 hours. The MacStories team already covered iOS and iPadOS 27, plus Siri AI and Apple Intelligence, and we have more coming tomorrow.

Before I call it a day though, I wanted to link the first thing I read on my way back: Apple’s latest article on the Machine Learning blog about the new Apple Foundation Models that were announced today – three cloud-based models, and two on-device ones.

Read more

Siri AI and the Latest in Apple Intelligence: The MacStories Overview

Source: Apple.

Source: Apple.

The big question coming into today’s WWDC keynote was what Apple would have to offer in terms of AI – or, as the company refers to it, Apple Intelligence. Given Apple’s behind-the-scenes struggles, delayed features, and partnership with Google to re-architect its Foundation Models based on Gemini, expectations were high, and the company delivered with the announcement of a whole new Siri and a slew of Apple Intelligence features baked into its upcoming operating systems.

Siri AI, the all-new version of Apple’s virtual assistant, was the star of the show and got plenty of time in the spotlight, including several real-time demos on the iPhone and Mac. But the company also announced updates to Visual Intelligence, new photo editing tools, a Shortcuts builder, and additional Apple Intelligence features.

Source: Apple.

Source: Apple.

It all starts with the latest Foundation Models based on a new architecture developed by Apple in collaboration with Google’s Gemini models. Coming in both local and Private Cloud Compute variations, these models work across Apple devices using a combination of user input (via text, voice, or images), personal context, on-screen awareness, app actions, and world knowledge in conjunction with the new system orchestrator to power conversations and actions taken by Siri AI as well as Apple Intelligence features baked into each OS. Of course, Apple touts that these features are secure and protect user privacy whether they are run on-device or in Private Cloud Compute.

Read more

I Have Many Questions About Apple’s Updated Foundation Models and the (Great) ‘Use Model’ Action in Shortcuts

Apple's 'Use Model' action in Shortcuts.

Apple’s ‘Use Model’ action in Shortcuts.

I mentioned this on AppStories during the week of WWDC: I think Apple’s new ‘Use Model’ action in Shortcuts for iOS/iPadOS/macOS 26, which lets you prompt either the local or cloud-based Apple Foundation models, is Apple Intelligence’s best and most exciting new feature for power users this year. This blog post is a way for me to better explain why as well as publicly investigate some aspects of the updated Foundation models that I don’t fully understand yet.

Read more

Apple Intelligence Expands: Onscreen Visual Intelligence, Shortcuts, Third-Party Apps, and More

Source: Apple.

Source: Apple.

One of the big questions heading into today’s WWDC keynote was how Apple would address its AI efforts. After a splashy introduction last year followed by a staggered rollout and the eventual delay of the more personalized Siri, it was unclear how much focus the company would put on Apple Intelligence during its big announcement video.

Surprisingly, they came right out of the gate with a segment on Apple Intelligence, even going so far as to mention the fact that the more personalized Siri needed more time; it’s slated to be released “in the coming year.” But SVP of Software Craig Federighi also said that Apple Intelligence had progressed with more capable and efficient models and teased that more Apple Intelligence features would be revealed throughout the presentation. Rather than dedicating a significant portion of the keynote just to AI features, the company returned to a platform-centered structure for the rest of the video and mentioned Apple Intelligence as it related to each OS.

In its second year, Apple Intelligence is set to expand in more ways than one. Perhaps most excitingly, third-party developers will soon have access to Apple Intelligence’s on-device foundation model, enabling them to implement AI features in their apps that work offline in a privacy-respecting way. And because the framework is local, it will be available to developers at no additional cost with no API fees.

Read more

On Apple Allowing Third-Party Assistants on iOS→

This is an interesting idea by Parker Ortolani: what if Apple allowed users to change their default assistant from Siri to something else?

I do not want to harp on the Siri situation, but I do have one suggestion that I think Apple should listen to. Because I suspect it is going to take quite some time for the company to get the new Siri out the door properly, they should do what was previously unthinkable. That is, open up iOS to third-party assistants. I do not say this lightly. I am one of those folks who does not want iOS to be torn open like Android, but I am willing to sign on when it makes good common sense. Right now it does.

And:

I do not use Gemini as my primary LLM generally, I prefer to use ChatGPT and Claude most of the time for research, coding, and writing. But Gemini has proved to be the best assistant out of them all. So while we wait for Siri to get good, give us the ability to use custom assistants at the system level. It does not have to be available to everyone, heck create a special intent that Google and these companies need to apply for if you want. But these apps with proper system level overlays would be a massive improvement over the existing version of Siri. I do not want to have to launch the app every single time.

As a fan of the progressive opening up of iOS that’s been happening in Europe thanks to our laws, I can only welcome such a proposal – especially when I consider the fact that long-pressing the side button on my expensive phone defaults to an assistant that can’t even tell which month it is. If Apple truly thinks that Siri helps users “find what they need and get things done quickly”, they should create an Assistant API and allow other companies to compete with them. Let iPhone users decide which assistant they prefer in 2025.

Some people may argue that other assistants, unlike Siri, won’t be able to access key features such as sending messages or integrating with core iOS system frameworks. My reply would be: perhaps having a more prominent placement on iOS would actually push third-party companies to integrate with the iOS APIs that do exist. For instance, there is nothing stopping OpenAI from integrating ChatGPT with the Reminders app; they have done exactly that with MapKit, and if they wanted, they could plug into HomeKit, HealthKit, and the dozens of other frameworks available to developers. And for those iOS features that don’t have an API for other companies to support…well, that’s for Apple to fix.

From my perspective, it always goes back to the same idea: I should be able to freely swap out software on my Apple pocket computer just like I can thanks to a safe, established system on my Apple desktop computer. (Arguably, that is also the perspective of, you know, the law in Europe.) Even Google – a company that would have all the reasons not to let people swap the Gemini assistant for anything else – lets folks decide which assistant they want to use on Android. And, as you can imagine, competition there is producing some really interesting results.

I’m convinced that, at this point, a lot of people despise Siri and would simply prefer pressing their assistant button to talk to ChatGPT or Claude – even if that meant losing access to reminders, timers, and whatever it is that Siri can reliably accomplish these days. (I certainly wouldn’t mind putting Claude on my iPhone and leaving Siri on the Watch for timers and HomeKit.) Whether it’s because of superior world knowledge, proper multilingual abilities (something that Siri still doesn’t support!), or longer contextual conversations, hundreds of millions of people have clearly expressed their preference for new types of digital assistance and conversations that go beyond the antiquated skillset of Siri.

If a new version of Siri isn’t going to be ready for some time, and if Apple does indeed want to make the best computers for AI, maybe it’s time to open up that part of iOS in a way that goes beyond the (buggy) ChatGPT integration with Siri.

Permalink

Bloomberg Reports that Apple Is Shaking up Siri Leadership→

Less than two weeks ago, Apple announced that it was delaying the launch of a more personalized Siri. Today, Mark Gurman, reporting for Bloomberg, says the company is shuffling leadership of the project, too. According to Gurman:

Chief Executive Officer Tim Cook has lost confidence in the ability of AI head John Giannandrea to execute on product development, so he’s moving over another top executive to help: Vision Pro creator Mike Rockwell. In a new role, Rockwell will be in charge of the Siri virtual assistant, according to the people, who asked not to be identified because the moves haven’t been announced.

Giannandrea isn’t leaving Apple. Instead, Gurman says Giannandrea will continue to oversee “research, testing and technologies related to AI” including a team investigating robotics. Rockwell, who led the development of the Vision Pro, will report to Craig Federighi, Apple’s senior vice president of software.

Rockwell has had a long and successful track record at Apple, so hopefully Siri is in good hands going forward. It’s clear that there’s a lot of work to be done, but the promise of a more personalized Siri and a system for apps to communicate with each other via Apple Intelligence is something I’m glad the company isn’t giving up on. Hopefully, we’ll see some progress from Rockwell’s team soon.

Permalink

On Apple Offering an Abstraction Layer for AI on Its Platforms

Source: Apple.

Source: Apple.

I’ve been thinking about Apple’s position in AI a lot this week, and I keep coming back to this idea: if Apple is making the best consumer-grade computers for AI right now, but Apple Intelligence is failing third-party developers with a lack of AI-related APIs, should the company try something else to make it easier for developers to integrate AI into their apps?

Gus Mueller, creator of Acorn and Retrobatch, has been pondering similar thoughts:

A week or so ago I was grousing to some friends that Apple needs to open up things on the Mac so other LLMs can step in where Siri is failing. In theory we (developers) could do this today, but I would love to see a blessed system where Apple provided APIs to other LLM providers.

Are there security concerns? Yes, of course there are, there always will be. But I would like the choice.

The crux of the issue in my mind is this: Apple has a lot of good ideas, but they don’t have a monopoly on them. I would like some other folks to come in and try their ideas out. I would like things to advance at the pace of the industry, and not Apple’s. Maybe with a blessed system in place, Apple could watch and see how people use LLMs and other generative models (instead of giving us Genmoji that look like something Fisher-Price would make). And maybe open up the existing Apple-only models to developers. There are locally installed image processing models that I would love to take advantage of in my apps.

The idea is a fascinating one: if Apple Intelligence cannot compete with the likes of ChatGPT or Claude for the foreseeable future, but third-party developers are creating apps based on those APIs, is there a scenario in which Apple may regain control of the burgeoning AI app ecosystem by offering their own native bridge to those APIs?

Read more

Is Apple Shipping the Best AI Computers?→

For all the criticism (mine included) surrounding Apple’s delay of various Apple Intelligence features, I found this different perspective by Ben Thompson fascinating and worth considering:

What that means in practical terms is that Apple just shipped the best consumer-grade AI computer ever. A Mac Studio with an M3 Ultra chip and 512GB RAM can run a 4-bit quantized version of DeepSeek R1 — a state-of-the-art open-source reasoning model — right on your desktop. It’s not perfect — quantization reduces precision, and the memory bandwidth is a bottleneck that limits performance — but this is something you simply can’t do with a standalone Nvidia chip, pro or consumer. The former can, of course, be interconnected, giving you superior performance, but that costs hundreds of thousands of dollars all-in; the only real alternative for home use would be a server CPU and gobs of RAM, but that’s even slower, and you have to put it together yourself. Apple didn’t, of course, explicitly design the M3 Ultra for R1; the architectural decisions undergirding this chip were surely made years ago. In fact, if you want to include the critical decision to pursue a unified memory architecture, then your timeline has to extend back to the late 2000s, whenever the key architectural decisions were made for Apple’s first A4 chip, which debuted in the original iPad in 2010. Regardless, the fact of the matter is that you can make a strong case that Apple is the best consumer hardware company in AI, and this week affirmed that reality.

Anecdotally speaking, based on the people who cover AI that I follow these days, it seems there are largely two buckets of folks who are into local, on-device models: those who have set up pricey NVIDIA rigs at home for their CUDA cores (the vast minority); and – the undeniable majority – those who run a spectrum of local models on their Macs of different shapes and configurations (usually, MacBook Pros). If you have to run high-end, performance-intensive local models for academic or scientific workflows on a desktop, the M3 Ultra Mac Studio sounds like an absolute winner.

However, I’d point out that – again, as far as local, on-device models are concerned – Apple is not shipping the best possible hardware on smartphones.

While the entire iPhone 16 lineup is stuck on 8 GB of RAM (and we know how memory-hungry these models can be), Android phones with at least 12 GB or 16 GB of RAM are becoming pretty much the norm now, especially in flagship territory. Even better in Android land, what are being advertised as “gaming phones” with a whopping 24 GB of RAM (such as the ASUS ROG Phone 9 Pro or the RedMagic 10 Pro) may actually make for compelling pocket computers to run smaller, distilled versions of DeepSeek, LLama, or Mistral with better performance than current iPhones.

Interestingly, I keep going back to this quote from Mark Gurman’s latest report on Apple’s AI challenges:

There are also concerns internally that fixing Siri will require having more powerful AI models run on Apple’s devices. That could strain the hardware, meaning Apple either has to reduce its set of features or make the models run more slowly on current or older devices. It would also require upping the hardware capabilities of future products to make the features run at full strength.

Given Apple’s struggles, their preference for a hybrid on-device/server-based AI system, and the market’s evolution on Android, I don’t think Apple can afford to ship 8 GB on iPhones for much longer if they’re serious about AI and positioning their hardware as the best consumer-grade AI computers.

Permalink