This Week's Sponsor:

Mindspace

A Private All-in-One Journal App Made for iPad, Offering 50% Off Your First Year.

Posts tagged with "macOS"

Apple Intelligence-Infused Accessibility Features Promise Greater Flexibility and Power

Source: Apple.

Source: Apple.

In what I expect will be an overarching theme at WWDC 2026, Apple’s Accessibility group took the wraps off an impressive collection of features coming later this year. The announcement, which is timed to lead into Global Accessibility Awareness Day on Thursday, emphasizes existing features and technologies that the company says will gain deeper capabilities thanks to Apple Intelligence.

For starters, VoiceOver will become more descriptive, allowing a device’s camera to be used to describe the user’s surroundings or a scanned document in greater detail. The feature will also make use of the Action button to trigger the camera and allow users to ask questions and make follow-up inquiries about what’s in the viewfinder. The Magnifier will gain voice controls, too, so users can simply ask it to zoom in, for example.

Source: Apple.

Source: Apple.

Voice Control will get similar enhancements. Rather than requiring a defined set of commands that need to be memorized to control a device, the feature will allow users to invoke actions with natural language, such as, “Tap the orange folder.”

Accessibility Reader will be able to handle more complex written layouts that include tables, columns, and other traditionally challenging formatting. If there’s one thing that LLMs have become extremely good at, it’s scraping the web and learning how to parse the meaningful parts of a webpage. While I’d have preferred that the web not have been pillaged as fuel for models in the first place, I’m glad at least part of that is going towards making the web and other text more readable for people who need it.

One of my favorite demos that Apple showed off during my briefing was a short video shot on an iPhone that had subtitles added to it on the fly using an on-device model. We’ve grown so accustomed to subtitles being available with the TV shows, movies, and YouTube videos we watch that they feel like they’re missing from the home movies we shoot and share with friends and family. Later this year, though, subtitles will be available for all types of video, generated privately on device.

Vision Pro wheelchair control. Source: Apple.

Vision Pro wheelchair control. Source: Apple.

The Vision Pro uses state-of-the-art eye tracking for interacting with your environment. Apple announced that it is extending that technology to motorized wheelchairs by working with partners TOLT Technologies and LUCI. The system allows a motorized wheelchair to be maneuvered by the user simply looking at controls inside the Vision Pro. The video showing off the feature was impressive and makes perfect sense if you’ve ever used the Vision Pro.

Apple also announced a new accessory with an accessibility angle. You may have seen the Hikawa Grip and Stand collection, a series of colorful accessories designed to make it easier for people to hold an iPhone more securely. Designed late last year by artist Bailey Hikawa, the Hikawa Grip and Stand is being mass-produced by PopSockets and sold in Apple retail stores in 20 markets starting today.

Finally, a bunch of other accessibility features are coming to Apple platforms later this year, including:

  • Vehicle Motion Cues, face gestures for taps, and eye-select in Dwell Control for visionOS,
  • Touch Accommodations setup customization,
  • Improvements to MFi hearing aid pairing and handoff across devices,
  • Larger Text support in tvOS,
  • Name Recognition in over 50 languages,
  • An API for adding human sign language interpreters to FaceTime, and
  • Support for Sony’s Access game controllers on iOS, iPadOS, and macOS.

With all the overblown hype surrounding artificial intelligence, it’s refreshing to see Apple putting it to practical use in ways that are meaningful to its users. One thing I’ve learned from following the work of Apple’s Accessibility team over the years is that there’s no one-size-fits-all approach to accessibility. The solutions are as unique as the people they serve. Apple has always offered a wide range of APIs and user features to make their hardware and apps available to as many people as possible, but Apple Intelligence promises to take the company’s longstanding commitment and make it more flexible and powerful for more people than ever before.

Indigo: A Clever Mashup of Bluesky and Mastodon in One Timeline

Last week, Soapbox Software (Ben McCarthy and Aaron Vegh) released Indigo, an iPhone, iPad, and Mac app that offers a unique take on social media, allowing you to log into both Bluesky and Mastodon in a single app. In the increasingly fractured social media landscape we live in, it’s a fantastic idea. Instead of bouncing back and forth between two services that have a lot of overlap for some users, why not use just one?

This isn’t Soapbox’s first collaboration. You may recall Croissant, the cross-posting utility that I covered when it released in 2024. We were so taken by the app that we gave it the Best New App award in the 2024 MacStories Selects Awards. That pedigree shows in what is a much deeper and more complex app.

Like Croissant, Indigo lets you cross-post to Bluesky and Mastodon and is beautifully designed. But unlike Croissant, Indigo is a full-blown timeline app for simultaneously catching up on your Bluesky and Mastodon feeds at the same time.

Depending on who you follow on each service, a mashup of the two has the potential to generate a timeline full of duplicates, but Soapbox took that into account with Indigo. There’s no need to change who you follow or make any other sort of adjustment yourself; instead, the app automatically detects duplicate posts and removes them from sight. However, if for some reason you want to see both, the duplicate post is always available behind the tap of a Crosspost button. It’s a great feature that alerts you to the fact that one of your timelines has been altered while also giving you the chance to check out the other post.

Indigo running on an iPad Pro.

Indigo running on an iPad Pro.

Other touches, such as the color of links, provide subtle clues to convey a post’s provenance, but the shades of blue and purple used are close enough that you might not notice the difference until you run across a Crosspost button. I also appreciate the separate character limit countdowns for each service on the New Post screen, which let you know when you’re going to have to forgo Bluesky for a chattier Mastodon post. Fortunately, the app lets you just post to one or the other service if you’d like by tapping on the character countdown.

All of the other core features you’d expect are available, too. Photos, videos, and GIFs are supported, as are @mentions and hashtags. You can filter who can see your post and who can reply to it, with some inherent differences in the underlying services’ support for those features. The app also includes search, notifications, direct messages, profile viewing, and a bunch of settings you can tweak. That said, power users of apps like Ivory may feel a little constrained in Indigo. It’s an excellent 1.0, but it doesn’t yet match the full functionality of Ivory.

Scrolling dog stories on the Mac.

Scrolling dog stories on the Mac.

Indigo strikes me as a good solution for a couple of different types of users. If you want a simple, beautifully designed way to read your Bluesky or Mastodon timeline, this is a great one. While the cross-posting and deduplication features are what will set Indigo apart for many, it works well as a standalone option for either service.

However, I expect the core audience will be people who use both Bluesky and Mastodon and follow many of the same people in both places. Especially if the people you follow cross-post a lot, Indigo greatly improves the experience.

I’ve enjoyed playing around with Indigo for the past few weeks and noticed a couple of things. Despite following roughly the same number of people on both services, the Bluesky accounts I follow are a lot chattier than those on Mastodon. I also have far fewer Crosspost buttons in my timeline than I expected. I guess I just follow very different accounts on each.

If you’ve ever felt the fatigue of jumping back and forth between a Bluesky and Mastodon timeline and found it hard to keep up with both, be sure to give Indigo a try. It makes the entire experience much nicer. You can download Indigo from the App Store on iPhone, iPad, and Mac and unlock its full feature set by purchasing the Ultraviolet tier, which costs $4.99/month, $34.99/year, or a one-time payment of $119.99.

OpenAI’s New Codex App Has the Best ‘Computer Use’ Feature I’ve Ever Tested

Computer use in Codex.

Computer use in Codex.

OpenAI rolled out their updated Codex app for Mac yesterday and, among other things, they shipped a native computer use tool for macOS that lets Codex interact with multiple Mac apps in the background using parallel cursors that do not bring apps to the foreground when agents are interacting with them. The feature that OpenAI rolled out in Codex is literally based on the Sky app that I exclusively previewed last year, and which was later acquired by OpenAI along with the team that built it.1

I feel like I’m in a pretty unique position to comment on all this since, as MacStories readers will recall, I was able to test Sky for several months last year before the team went radio-silent and joined OpenAI. Here’s the thing: I’m not exaggerating when I say that Codex now features the best computer use feature I have ever tested in any LLM or desktop agent. In fact, it’s even better than the computer use feature I used in Sky last year: Sky’s computer use was great, but it was considerably slower than Codex’s current one because it was running on Anthropic’s Claude models. With Codex for Mac today, even the (kind of slow) GPT 5.4 is faster than Sky ever was. But, using Codex with fast mode or – for simpler tasks – the Cerebras-hosted GPT-5.3-Codex-Spark model yields dramatically faster performance than Sky for Mac delivered in 2025.

But why is that? Allow me to explain. Most computer use models (such as the one in the Claude app, or even the just-released Personal Computer by Perplexity) rely on a combination of screen-recording capabilities and some AppleScript to either simulate virtual clicks on-screen and perform basic actions inside apps by calling osascript in a virtual shell. Sky was different, and Codex is different, and I can share more details today that I did not elaborate on when I wrote about Sky last year.

We all have Apple’s Accessibility team to thank for the technology that allows Codex’s computer use tool to exist. To build it, the Codex team took advantage of an advanced accessibility feature that allows third-party apps to read the “accessibility hierarchy” (also known as “AX Tree”) of any app open on macOS. My understanding is that this technology was primarily created to allow screen-readers and other assistive tools to work with Mac apps regardless of their automation/scripting features. In this case, it’s been repurposed as a way for Codex to ingest the full contents and hierarchy of any window and, essentially, load it as context for the LLM.

When I was told last year that this was how Sky worked behind the scenes, I instantly knew it reminded me of something, and I was right. We’ve seen the same technology being used before in UI Browser, the excellent (and sadly discontinued) app to inspect the visual hierarchy of any app that’s also powered by screen-reader APIs on macOS. All of this still applies to Codex’s computer use plugin today: pay attention to any chat where you’re using the plugin, and you’ll see 5.4 reason about the “accessibility tree” it wants to parse from any given application.

As someone who’s played around with GUI scripting and UI Browser many times over the years, let me tell you: this is not easy, and these frameworks were not meant for automation. For starters, they return a lot of text about any possible UI element, text field, or button inside a window. That text can be formatted in a variety of ways; it can be so deeply nested inside the XML-like structure returned by the AX framework, you often need to navigate 20 levels deep into a structure to find what you want. But this is what makes Codex’s computer use model different, why the Sky acquisition was a very clever move from OpenAI, and also why the reactions online seem overwhelmingly positive: Codex can “see” more inside apps and can control them more precisely than other models based solely on capturing screenshots, simulating clicks on certain coordinates, and running the occasional AppleScript. Codex can also do those things as fallback measures, but they’re not the primary drivers of its computer use plugin.

It also helps that computer use in Codex is exquisitely designed – not a surprise given OpenAI’s design team and the pedigree of the team behind this feature. The flow for granting permissions to the plugin is the best I’ve ever seen in a third-party Mac app – and it comes directly from Sky, which had the same onboarding experience. What Sky didn’t have is the new virtual cursor: the Codex team designed an entire system for it where the cursor can wiggle to show when the model is thinking, takes playful paths, and derives its color from the system’s wallpaper. I can only think of another company that sweats these kinds of UI details as much as the Codex team did here…and I’ll let you guess where several of Codex’s engineers and designers are, in fact, coming from.

I’ve been working with computer use in Codex all day, and while it is not as fast as a skilled human who knows a particular macOS interface well, it is very good at understanding and controlling any Mac app in the background a bit more slowly, with greater precision than competing features from Anthropic and Perplexity. That makes it ideal to automate busywork in Mac apps that do not offer an API or CLI, or which can’t be fully controlled with AppleScript. Let me give you some practical examples.

Earlier today, I asked both Perplexity’s Personal Computer and Codex to “play the latest album from the weird masked band from Quebec, I don’t remember their name”. I was referring to the exceptional Angine de Poitrine, of course. Both agents searched the web upfront and pinpointed my request, but when it came to actually controlling the Music app, Personal Computer stopped short of hitting the ‘Play’ button because its AppleScript integration couldn’t do it; Codex went ahead, opened the album with its virtual cursor, and started playing music.

Personal Computer couldn’t hit Play.

Personal Computer couldn’t hit Play.

Codex had no issues playing music in the Music app.

Codex had no issues playing music in the Music app.

I also tested Codex by asking it to look at specific channels on Slack, my Ivory timeline, and the Unread app and give me a summary of interesting updates I should know about. Codex successfully deployed parallel cursors, started scrolling and clicking around all three apps, and produced a report that included updates gathered from those apps. Could I have scrolled the apps myself, one after the other, the old fashioned way? Sure. But as an “automation” that happened in the background while I was doing my email, it was pretty good.

Codex’s report from three separate apps.

Codex’s report from three separate apps.

The other task I attempted today – which is still running, after 6 hours – was using Codex’s computer use to improve the Shortcuts Playground skill I’ve been building to create shortcuts in the Shortcuts app using coding agents in natural language. With Codex, I figured I could now ask the agent to run the skill, create shortcuts for me, but also click the resulting .shortcut files in Finder, install them, and test them for me in the Shortcuts app to spot any errors and further improve the skill. Not only was Codex’s computer use plugin able to successfully install dozens of shortcuts, but it also opened each, verified its output, and is currently evaluating what went wrong to improve some of the skill’s guidance and instructions.

Codex installed all these shortcuts via computer use.

Codex installed all these shortcuts via computer use.

The Codex cursor debugging a shortcut for me.

The Codex cursor debugging a shortcut for me.

So, long story short: Codex’s computer use plugin is the state of the art at the moment, and it’s the evolution of a strong foundation that I was able to test last year, which has been further refined and expanded by OpenAI. I’d like to see the company expand this plugin to the main ChatGPT for Mac experience (which is still stuck on the old Work with Apps integration), but, for now, I’ll take this feature inside Codex rather than the slower, and less capable, computer use models from other chatbots. More importantly, I’m happy to see that Sky ended up in good hands who can now deliver this product to the masses.

  1. I don’t use the term “literally” in a liberal sense here. When you enable the Computer Use plugin in Codex, you can head over to the app’s config.toml configuration file, open it in a text editor, and you’ll spot this line:
    /Users/username/.codex/plugins/cache/openai-bundled/computer-use/1.0.750/Codex Computer Use.app/Contents/SharedSupport/SkyComputerUseClient.app/Contents/MacOS/SkyComputerUseClient

    Open that folder and, sure enough, there’s an executable for the former Sky “app”, now loaded as a first-party 2plugin that handles the virtual computer interactions for Codex. ↩

Introducing Apple Frames 4: A Revamped Shortcut, Support for Frame Colors, Proportional Scaling, and the Apple Frames CLI for Developers

Apple Frames 4.

Apple Frames 4.

Well, it’s been a minute.

Today, I’m very happy to introduce Apple Frames 4, a major update to my shortcut for framing screenshots taken on Apple devices with official Apple product bezels. Apple Frames 4 is a complete rethinking of the shortcut that is noticeably faster, updated to support all the latest Apple devices, and designed to support even more personalization options. For the first time ever, Apple Frames supports multiple colors for each device, allowing you to mix and match different colored bezels for each framed screenshot; it also supports proportional scaling when merging screenshots from different Apple devices.

But that’s not all. In addition to an updated shortcut, I’m also releasing the Apple Frames CLI, an open source command-line utility that lets developers and tinkerers automate the process of framing screenshots directly from the Mac’s Terminal. And there’s more: the Apple Frames CLI is also designed to work with AI agents, and it comes with a Claude Code/Codex skill that lets coding agents take care of framing dozens or even hundreds of screenshots in just a few seconds, from any folder on your Mac.

Apple Frames 4 is the result of an idea I had months ago that enabled me to remove more than 500 actions from the shortcut, going from over 800 steps down to ~300. I did all that work manually, but it was worth it; the improved shortcut is faster and vastly more reliable than before thanks to a more intelligent logic that adapts to the growing ecosystem of Apple screen sizes and display resolutions.

Apple Frames 4 and the Apple Frames CLI represent a substantial step forward for screenshot automation, and I’ve been using both extensively for the past few weeks.

Let’s dive in.

Read more

macOS Tahoe’s Messy Menus→

Nikita Prokopov writing on tonsky.me about macOS Tahoe’s menu icons:

In my opinion, Apple took on an impossible task: to add an icon to every menu item. There are just not enough good metaphors to do something like that.

But even if there were, the premise itself is questionable: if everything has an icon, it doesn’t mean users will find what they are looking for faster.

And even if the premise was solid, I still wish I could say: they did the best they could, given the goal. But that’s not true either: they did a poor job consistently applying the metaphors and designing the icons themselves.

It’s a brutal assessment of the sprinkling of iconography throughout Tahoe’s menu system that had me nodding along in agreement as I read it.

There’s no denying the inconsistencies in icon choices, their lack of legibility, and the overall clutter added to menus. Yet at the same time, I can’t say I’ve been terribly bothered by them either. That’s probably because I use keyboard shortcuts and launchers so much, rarely relying on the Mac’s menu system. At the same time, though, part of me wonders whether those tiny icons are at least partially what drove me to buy a bigger monitor recently. I don’t think so, but maybe?

In any event, if you care about design, Prokopov’s detailed and well-illustrated analysis of Tahoe’s menu icons is well worth your time.

Permalink

macOS Tahoe: The MacStories Public Beta Preview

Author’s Note: Apple released the public beta of macOS 26 Tahoe last Thursday, two days after developer beta 4. Instead of rushing a preview of Tahoe to publication at the risk of missing important aspects of the release, I chose to spend the time necessary to thoroughly test Tahoe first.

A year ago, the macOS Sequoia public beta debuted with a long list of caveats. Many of the features that had been shown off at WWDC 2024 weren’t in that initial public beta release or even the initial macOS 15.0 release, and some features, like a smarter Siri, still haven’t shipped. That made Sequoia feel incomplete.

The release of macOS Tahoe 26 promises to be different. The features highlighted during WWDC 2025 are all in the public beta. Some are more polished than others, but everything is there to try today. With its surprisingly long list of new system apps, changes big and small at the macOS system level, and, of course, Liquid Glass, Tahoe’s public beta release is a fun one for users who like to explore Apple’s latest macOS innovations as early as possible.

I’m not sold on every feature, but it’s still early, and this is a beta, so I’ll reserve my final judgment for the fall. However, there’s a lot coming in macOS Tahoe, which makes it worth taking a closer look at today, so let’s dig in.

Read more

WWDC 2025: All the Small Things (Bento Box Version)

Source: Apple.

Source: Apple.

Every keynote, Apple is well-known for summarizing sections of the presentation with immaculately laid-out bento boxes containing key features. They often serve as good, easily digestible overviews of all the new features for each OS.

With that in mind, let’s take a look at all the bento boxes from today’s WWDC 2025 keynote.

Read more

macOS Tahoe: The MacStories Overview

At its WWDC 2025 keynote held earlier today, Apple officially announced the next version of macOS, macOS Tahoe. As per the company’s naming tradition over the past decade, this new release is once again named after a location in California. This year, however, to unify the version numbers across all its operating systems, Apple has decided to align the new release with the upcoming year. This is why the version number for macOS Tahoe will be macOS 26, directly up from last year’s macOS 15.

macOS 26 features the brand-new Liquid Glass design language, which Apple is also rolling out across iOS, iPadOS, visionOS, watchOS, and tvOS. But macOS Tahoe doesn’t stop there. In addition to the flashy new look, Apple has introduced many features, ranging from a supercharged new version of Spotlight and intelligent actions in Shortcuts to new Continuity and gaming-focused features for the Mac.

Here’s a recap of everything that Apple showed off today for macOS Tahoe.

Read more

From the Creators of Shortcuts, Sky Extends AI Integration and Automation to Your Entire Mac

Sky for Mac.

Sky for Mac.

Over the course of my career, I’ve had three distinct moments in which I saw a brand-new app and immediately felt it was going to change how I used my computer – and they were all about empowering people to do more with their devices.

I had that feeling the first time I tried Editorial, the scriptable Markdown text editor by Ole Zorn. I knew right away when two young developers told me about their automation app, Workflow, in 2014. And I couldn’t believe it when Apple showed that not only had they acquired Workflow, but they were going to integrate the renamed Shortcuts app system-wide on iOS and iPadOS.

Notably, the same two people – Ari Weinstein and Conrad Kramer – were involved with two of those three moments, first with Workflow, then with Shortcuts. And a couple of weeks ago, I found out that they were going to define my fourth moment, along with their co-founder Kim Beverett at Software Applications Incorporated, with the new app they’ve been working on in secret since 2023 and officially announced today.

For the past two weeks, I’ve been able to use Sky, the new app from the people behind Shortcuts who left Apple two years ago. As soon as I saw a demo, I felt the same way I did about Editorial, Workflow, and Shortcuts: I knew Sky was going to fundamentally change how I think about my macOS workflow and the role of automation in my everyday tasks.

Only this time, because of AI and LLMs, Sky is more intuitive than all those apps and requires a different approach, as I will explain in this exclusive preview story ahead of a full review of the app later this year.

Read more