This Week's Sponsor:

Hello Weather

Half Off for WWDC Week, Just in Time for Summer.

Posts tagged with "Apple Intelligence"

New Developer Betas Released for iOS, iPadOS, and macOS with Image Playground, ChatGPT Integration, and More Apple Intelligence Features

iOS 18.1, iPadOS 18.1, and macOS 15.1 aren’t quite out the door, but Apple has already updated its developer betas with the next round of upcoming Apple Intelligence features. Developer betas of iOS 18.2, iPadOS 18.2, and macOS 15.2 are now available for download and include the following:

  • image generation in the form of Image Playground and Image Wand;
  • Genmoji (iOS and iPadOS only)
  • Visual Intelligence (iPhone 16 line only)
  • ChatGPT integration with Siri; and
  • new text manipulation features.
Image Playground. Source: Apple.

Image Playground. Source: Apple.

Image Playground is a feature that allows you to create images in two styles using in-app themes and other tools. Image Playground is available in apps like Messages, Freeform, Pages, and Keynote, but it’s also a standalone app. Regardless of where you use it, Image Playground looks like it’s designed to make it easy to create animated and sketch-style images using a variety of tools such as suggested concepts that pull from the context the image is created in, like a Messages thread. Creations can be previewed, there’s a history feature that allows you to undo changes made to images, and images are saved to an Image Playground Library that syncs across devices via iCloud.

Image Wand. Source: Apple.

Image Wand. Source: Apple.

Image Wand, which appears in the Apple Pencil tool palette, takes a rough hand-drawn sketch, photo, or note and turns any of them into an image similar to one created by Image Playground. Image Wand can be further refined by adding text, and if you circle a blank space, it will use surrounding text to build an image.

Also, Genmoji – which is only in the iOS and iPadOS betas for now – allows you to create emoji-style images that can be used in Messages and other apps as decorative stickers. Inputs can include a text description, people in your contacts, friends and family recognized in Photos, and characters created from whole cloth.

Visual Intelligence has been added to the Camera Control on the iPhone 16 line too. The feature lets you look up details about a place and work with text, copying, reading, summarizing, and translating it.

The next betas also integrate ChatGPT into Siri. As demoed at WWDC, you can opt to pose queries to ChatGPT without disclosing you identity or IP address and without the prompts being used to train OpenAI’s large language models. The ChatGPT integration is free and does not require an account with OpenAI either.

Writing Tools lets you describe your text changes in iOS 18.2, iPadOS 18.2, and macOS 15.2.

Writing Tools lets you describe your text changes in iOS 18.2, iPadOS 18.2, and macOS 15.2.

Finally, Apple has built a new Writing Tool that provides additional flexibility when manipulating text. From the Writing Tools UI, you’ll be able to submit a prompt to alter any text you’ve written. For instance, you could have Apple Intelligence make you sound more excited in your message or rewrite it in the form of a poem, neither of which is possible with the Writing Tools found in iOS and iPadOS 18.1 or macOS 15.1.

For developers, there are also new APIs for Writing Tools, Genmoji, and Image Playground.

As we’ve covered before, Apple’s AI models have been trained on a mix of licensed data and content from the web. If you’re a publisher or a creator who doesn’t want to be part of those models, you can opt out, but it doesn’t work retroactively. In other words, opting out won’t remove any data already ingested by Apple’s web crawlers, but it will work going forward.

I’m not a fan of generative AI tools, but I am looking forward to finally going beyond tightly controlled demos of these features. I want to see how well they work in practice and compare them to other AI tools. Apple appears to have put a lot of guardrails in place to avoid some of the disasters that have befallen other tech companies, but I’m pretty good at breaking software. It will be interesting to see how well these tools hold up under pressure.

Apple’s Definition of a “Photo”→

One of my favorite parts from Nilay Patel’s review of the iPhone 16 Pro at The Verge was the answer he got from Apple’s VP of camera software engineering Jon McCormack about the company’s definition of a “photograph”:

Here’s our view of what a photograph is. The way we like to think of it is that it’s a personal celebration of something that really, actually happened.

Whether that’s a simple thing like a fancy cup of coffee that’s got some cool design on it, all the way through to my kid’s first steps, or my parents’ last breath, It’s something that really happened. It’s something that is a marker in my life, and it’s something that deserves to be celebrated.

“Something that really, actually happened” is a great baseline compared to Samsung’s nihilistic definition (nothing is real) and Google’s relativistic one (everyone has their own memories). As Jaron Schneider wrote at PetaPixel:

If you have no problem with generative AI, then what Google and Samsung said probably doesn’t bother you. However, many photographers are concerned about how AI will alter their jobs. From that perspective, those folks should be cheering on Apple for this stance. Right now, it’s the only major smartphone manufacturer that has gone on the record to steer photography away from the imagined and back to reality.

I like Apple’s realistic definition of what a photo is – right now, I feel like it comes from a place of respect and trust. But I have to wonder how malleable that definition will retroactively become to make room for Clean Up and future generative features of Apple Intelligence.

Permalink

Apple Opens Up About Language Support for Apple Intelligence→

Apple previously said Apple Intelligence will launch with support for U.S. English only, but other languages will be added over the next year. The company has gotten a little more specific than before with statements to The Verge and others that, over the next year, additional languages will include:

  • German
  • Italian
  • Korean
  • Portuguese
  • Vietnamese
  • and others

In addition, according to Allison Johnson at The Verge:

Apple’s AI feature set will expand to include localized English in the UK, Canada, Australia, South Africa, and New Zealand in December, with India and Singapore joining the mix next year. The company already announced plans to support Chinese, French, Japanese, and Spanish next year as well.

It’s good to see Apple becoming more open with information about its plans for Apple Intelligence language support. The iPhone 16 will be out this Friday, and Apple Intelligence’s language support will undoubtedly be a factor for many customers considering whether to upgrade to the latest model.

Permalink

Craig Federighi on Apple’s Private Cloud Compute Architecture→

Apple’s Craig Federighi was interviewed by Wired about the company’s Private Cloud Compute infrastructure that will handle Apple Intelligence requests that can’t be handled locally on-device. Federighi told Wired’s Lily Hay Newman:

What was really unique about the problem of doing large language model inference in the cloud was that the data had to at some level be readable by the server so it could perform the inference. And yet, we needed to make sure that that processing was hermetically sealed inside of a privacy bubble with your phone. So we had to do something new there. The technique of end-to-end encryption—where the server knows nothing—wasn’t possible here, so we had to come up with another solution to achieve a similar level of security.

Still, Apple says that it offers “end-to-end encryption from the user’s device to the validated PCC nodes, ensuring the request cannot be accessed in transit by anything outside those highly protected PCC nodes.” The system is architected so Apple Intelligence data is cryptographically unavailable to standard data center services like load balancers and logging devices. Inside a PCC cluster, data is decrypted and processed, but Apple emphasizes that once a response is encrypted and sent on its journey to the user, no data is retained or logged and none of it is ever accessible to Apple or its individual employees.

PCC is a complex system that leverages technologies that Apple has developed like the Secure Enclave, Secure Boot, and Trusted Execution Monitor to ensure customer privacy. Those technologies are backstopped by making every PCC server publicly available for inspection and verification by third parties. That said, Wired spoke to security researchers and cryptography experts who told the publication that although PCC looks promising, they hadn’t spent significant time studying it yet.

Cloud-based privacy is a tough problem to solve. I’m keen to hear what independent researchers think of Apple’s solution, once they’ve had more time to evaluate it. If PCC is as robust as Apple claims, it’s the sort of thing I could see Apple turning into a standalone product as worldwide regulation pushes tech companies to offer better privacy protections for their customers.

Permalink

Procreate Will Not Include Generative AI in Its Apps→

Today on its website, Procreate announced that it would not build generative AI tools into its apps. The company’s position is clear and unequivocal:

Generative AI is ripping the humanity out of things. Built on a foundation of theft, the technology is steering us toward a barren future. We think machine learning is a compelling technology with a lot of merit, but the path generative AI is on is wrong for us.

We’re here for the humans. We’re not chasing a technology that is a moral threat to our greatest jewel: human creativity. In this technological rush, this might make us an exception or seem at risk of being left behind. But we see this road less travelled as the more exciting and fruitful one for our community.

In a short video on X.com, Procreate CEO James Cuda said:

I don’t like what’s happening to the industry, and I don’t like what it’s doing to artists.

I couldn’t agree more or be happier to see Procreate take a stand in defense of artists. Federico and I interviewed Cuda at WWDC, and although Cuda struck a diplomatic tone having just received an Apple Design Award for Procreate Dreams, it was clear to me then that we were unlikely to see generative AI in Procreate’s apps. For everyone who wasn’t in the room with Cuda that day, though, today’s statement should set their minds at ease. I hope we see more developers whose apps support creative fields take a similar stand.

Permalink

Opting Out of AI Model Training→

Dan Moren has an excellent guide on Six Colors that explains how to exclude your website from the web crawlers used by Apple, OpenAI, and others to train large language models for their AI products. For many sites, the process simply requires a few edits to the robots.txt file on your server:

If you’re not familiar with robots.txt, it’s a text file placed at the root of a web server that can give instructions about how automated web crawlers are allowed to interact with your site. This system enables publishers to not only entirely block their sites from crawlers, but also specify just parts of the sites to allow or disallow.

The process is a little more complicated with something like a WordPress, which MacStories uses, and Dan covers that too.

Unfortunately, as Dan explains, editing robots.txt isn’t a solution for companies that ignore the file. It’s simply a convention that doesn’t carry any legal or regulatory weight. Nor does it help with Google or Microsoft’s use of your website’s copyrighted content unless you’re also willing to remove your site from the biggest search engines.

Although I’m glad there is a way to block at least some AI web crawlers prospectively, it’s cold comfort. We and many sites have years of articles that have already been crawled to train these models, and you can’t unring that bell. That said, MacStories’ robot.txt file has been updated to ban Apple and OpenAI’s crawlers, and we’re investigating additional server-level protections.

If you listen to Ruminate or follow my writing on MacStories, you know that I think what these companies are doing is wrong both in the moral and legal sense of the word. However, nothing captures it quite as well as this Mastodon post by Federico today:

If you’ve ever read the principles that guide us at MacStories, I’m sure Federico’s post came as no surprise. We care deeply about the Open Web, but ‘open’ doesn’t give tech companies free rein to appropriate our work to build their products.

Yesterday, Federico linked to Apple’s Machine Learning Research website where it was disclosed that the company has indexed the web to train its model without the consent of publishers. I was as disappointed in Apple as Federico. I also immediately thought of this 2010 clip of Steve Jobs near the end of his life, reflecting on what ‘the intersection of Technology and the Liberal Arts’ meant to Apple:

I’ve always loved that clip. It speaks to me as someone who loves technology and creates things for the web. In hindsight, I also think that Jobs was explaining what he hoped his legacy would be. It’s ironic that he spoke about ‘technology married with Liberal Arts,’ which superficially sounds like what Apple and others have done to create their AI models but couldn’t be further from what he meant. It’s hard to watch that clip now and not wonder if Apple has lost sight of what guided it in 2010.

You can follow all of our WWDC coverage through our WWDC 2024 hub or subscribe to the dedicated WWDC 2024 RSS feed.

Permalink

Apple Details Its AI Foundation Models and Applebot Web Scraping→

From Apple’s Machine Learning Research1 blog:

Our foundation models are trained on Apple’s AXLearn framework, an open-source project we released in 2023. It builds on top of JAX and XLA, and allows us to train the models with high efficiency and scalability on various training hardware and cloud platforms, including TPUs and both cloud and on-premise GPUs. We used a combination of data parallelism, tensor parallelism, sequence parallelism, and Fully Sharded Data Parallel (FSDP) to scale training along multiple dimensions such as data, model, and sequence length.

We train our foundation models on licensed data, including data selected to enhance specific features, as well as publicly available data collected by our web-crawler, AppleBot. Web publishers have the option to opt out of the use of their web content for Apple Intelligence training with a data usage control.

We never use our users’ private personal data or user interactions when training our foundation models, and we apply filters to remove personally identifiable information like social security and credit card numbers that are publicly available on the Internet. We also filter profanity and other low-quality content to prevent its inclusion in the training corpus. In addition to filtering, we perform data extraction, deduplication, and the application of a model-based classifier to identify high quality documents.

It’s a very technical read, but it shows how Apple approached building AI features in their products and how their on-device and server models compare to others in the industry (on servers, Apple claims their model is essentially neck and neck with GPT-4-Turbo, OpenAI’s older model).

This blog post, however, pretty much parallels my reaction to the WWDC keynote. Everything was fun and cool until they showed generative image creation that spits out slop “resembling” (strong word) other people; and in this post, everything was cool until they mentioned how – surprise! – Applebot had already indexed web content to train their model without publishers’ consent, who can only opt out now. (This was also confirmed by Apple executives elsewhere.)

As a creator and website owner, I guess that these things will never sit right with me. Why should we accept that certain data sets require a licensing fee but anything that is found “on the open web” can be mindlessly scraped, parsed, and regurgitated by an AI? Web publishers (and especially indie web publishers these days, who cannot afford lawsuits or hiring law firms to strike expensive deals) deserve better.

It’s disappointing to see Apple muddy an otherwise compelling set of features (some of which I really want to try) with practices that are no better than the rest of the industry.

  1. How long until this become the ‘Apple Intelligence Research’ website? ↩
Permalink

Apple Intelligence: The MacStories Overview

After months of anticipation and speculation about what Apple could be doing in the world of artificial intelligence, we now have our first glimpse at the company’s approach: Apple Intelligence. Based on generative models, Apple Intelligence uses a combination of on-device and cloud processing to offer intelligence features that are personalized, useful, and secure. In today’s WWDC keynote, Tim Cook went so far as to call it “the next big step for Apple.”

From the company’s press release on Apple Intelligence:

“We’re thrilled to introduce a new chapter in Apple innovation. Apple Intelligence will transform what users can do with our products — and what our products can do for our users,” said Tim Cook, Apple’s CEO. “Our unique approach combines generative AI with a user’s personal context to deliver truly helpful intelligence. And it can access that information in a completely private and secure way to help users do the things that matter most to them. This is AI as only Apple can deliver it, and we can’t wait for users to experience what it can do.”

It’s clear from today’s presentation that Apple is positioning itself as taking a different approach to AI than the rest of the industry. The company is putting generative models at the core of its devices while seeking to stay true to its principles. And that starts with privacy.

Read more