Posts tagged with "OpenAI"

OpenAI Adds iCloud Calendar and Mail Plugins and Dot Texting→

OpenAI continues to add features to its ChatGPT app on a regular basis, but there are a couple of recent Apple-specific updates worth mentioning.

First, OpenAI created plugins for Calendar and Mail a couple of days ago. Both features use iCloud and require you to set up an app-specific password to use them. Here’s what OpenAI’s in-app documentation says each does:

  • iCloud Calendar: List iCloud calendars, find events, create or change events, and look up iCloud contacts.
  • iCloud Mail: Search and read iCloud Mail, manage messages and drafts, send email, and look up iCloud contacts.

I’ve been using Google Calendar more the past several months primarily because there was too much friction in getting Codex or my Dot to use my iCloud calendars. I’ve never been fond of Google Calendar, so I set up the iCloud Calendar plugin immediately. I don’t use Apple Mail as often, but it’s come a long way with the latest OS updates, so I set up the iCloud Mail plugin too and plan to give it another go.

Setting up your Dot in Messages.

Setting up your Dot in Messages.

Second, you can now text your OpenAI Dot. The setup process is very simple, which anyone who tried to set up Messages with OpenClaw in the early days will appreciate. Go to your iPhone, tap your Dot, select “Set up texting,” add your phone number, paste in the verification code it sends, and you’re all set. It took me less than a minute to do. Your Dot will even send you a contact card to add to the Contacts app, so its profile picture shows up in Messages.

You may be wondering why you’d set up texting when you have the ChatGPT app and a widget that tie directly into your Dot. It’s a fair point, but for me, the more ways I have to easily make requests of my Dot, the better, and texting gives me a low-friction way to do that that takes next to no bandwidth, ensuring that it will work whether I’ve got a strong cellular signal or not. It’s worth noting that although your Dot is assigned a phone number for texting purposes, the number can’t be used to start a voice chat, which I tried of course.

Permalink

ChatGPT Dots Gain Widget and New Shortcuts Actions

Late Friday, OpenAI updated the ChatGPT app on iOS and iPadOS with a new widget for your Dot and Shortcuts actions that let you control it three ways. The implementation is really well done and adds automation options worth considering.

First of all, the Dot widget is about as simple as it gets, but also very useful. I have the ChatGPT app set up to drop me into its Codex view because that’s typically how I use the app. I open Codex, check on a long-running project, and provide whatever input is necessary to keep things moving.

The problem I’ve run into is that I’ve been using my Dot, Puck, more and more to kick off research, ask one-off questions and check in on things like my email. Landing in Codex means tapping around more, which I can avoid now with ChatGPT’s new small-sized widget. The widget has two modes. The first simply shows your Dot’s avatar, and tapping on it opens your Dot’s message thread. However, if you edit the Dot, you can toggle on the “Show shortcut button” to display messaging and voice buttons beneath your Dot, adding the option to start a voice call with your Dot. As simple as this is, it’s been great to have the option to start in Codex via the ChatGPT app or check in with my Dot thread using the widget.

The other interesting feature is a trio of Shortcuts actions to message, dictate, or call your Dot. Each action has two parameters: the name of your Dot and how you want to communicate with it. That first parameter is an interesting one because tapping on it opens a list with your sole Dot in it because currently, you can only have one Dot. However, it also strongly suggests that the ability to set up multiple Dots is on the horizon, which OpenAI clearly telegraphed during its DevDay. After all, why have a parameter that opens a list unless there’s more than one choice?

I've set my Action button up to call my Dot.

I’ve set my Action button up to call my Dot.

The actions can be used in your shortcuts, are accessible from Spotlight Search, and most interestingly can be tied to an iPhone’s Action button, allowing you to visit your Dot’s thread in any of three ways. I’ve set my Action button up to start a Dot call. I often think of tasks and have ideas for research topics when I’m out for a walk, and I want to see if having brief calls with my Dot is an effective way to get a jump on those ideas, so they’re waiting for me in ChatGPT when I get home.

These actions and widgets are an excellent addition to OpenAI’s app and a cut above Anthropic’s more barebones approach to widgets and Shortcuts. I would, however, like to see Dots integrated even deeper into Shortcuts. The Shortcuts actions only open your Dot thread in one of the three modes. I’d love to be able to connect Dots to other apps with actions that accept input from other sources and produce outputs that can be used in other actions.

While I’m making requests, I still think multiple Dots need to be supported sooner rather than later, and OpenAI needs to provide a better mechanism for switching between Macs if you use a Dot with multiple computers. Plus, Dots are still a little buggy, with scheduled tasks failing too often and for reasons that aren’t always clear.

That said, the latest update is worth checking out if you’ve found Dots helpful. The strength of Dots is that they’re everywhere you can access ChatGPT, and now with widgets and Shortcuts actions, getting to your Dot in each of those places is easier than ever.


OpenAI’s Dots: A First Look

Source: OpenAI.

Source: OpenAI.

Earlier this week, I was at OpenAI’s annual DevDay conference where it launched Dots, a personal agent that lives inside the ChatGPT app. I set up my Dot and named it Puck after the mischievous sprite in Shakespeare’s A Midsummer Night’s Dream. In the days since then, I’ve found myself using Puck more and more. There are still rough edges, but I thought I’d share what’s working and what’s not, and how Dots are different from what’s already available in ChatGPT and Codex.

Dots are persistent, cloud-based agents that use the plugins and other tools you’ve connected to ChatGPT, after you allow them to do so. Your Dot has its own virtual Linux computer to assist you with your work. And at first, your Dot, which is depicted by a colorful character you can customize, doesn’t have access to anything else. However, as you give your Dot tasks that need access to other tools, it will ask for your permission to access other tools. The pacing of this initial interaction with your Dot is really well done. The requests felt natural and allowed me to get comfortable with expanding what I let Puck do without feeling like annoying interruptions.

Customizing my Dot.

Customizing my Dot.

Dots are available on mobile and desktop devices, and because they are cloud-based, your Dot’s chat is always synced between devices. But being a cloud-based agent has its downsides, too. Out of the box, your Dot can’t access your computer’s files and apps. That’s easily fixed by clicking or tapping on your Dot and granting it access to your Mac, so it can use the Mac alongside its own VM. As with Codex’s Remote feature, using a remote Mac with your Dot requires the ChatGPT app to be open on that Mac, too.

Read more


Inside OpenAI’s Codex with Andrew Ambrosino→

On today’s episode of AppStories, Federico and I interview Andrew Ambrosino, the project lead for Codex. It’s a great conversation that covers a lot of ground, including:

  • the long history and many iterations of Codex,
  • the merging of Codex into a new version of ChatGPT based on web technologies,
  • Computer Use and Computer History, which came along with OpenAI’s acquisition of Sky last year, and
  • what goes into designing for user trust.

Then for AppStories+ subscribers, we dug into Andrew’s Mac origin story and some of his favorite Mac apps.

AppStories is available on Apple Podcasts and all your favorite podcast players, as well as YouTube. AppStories+ is a special ad-free, extended version of AppStories that’s delivered a day early in high-bitrate audio for $5/month or $50/year by itself or as part of a Club MacStories Premier membership that adds our weekly and monthly newsletters, app discounts, and other perks for $12/month or $120/year.

Permalink


Hands-On with Computer History: OpenAI’s Take on Agent Memory

Late yesterday, OpenAI revealed the sort of automation catnip I love: Computer History. It’s a new feature for Pro, Business, and Enterprise subscribers that creates continuity for ChatGPT and Codex by converting the actions you take on your Mac into Markdown memory files that serve as context for subsequent interactions. The idea is that with the additional context, ChatGPT and Codex will better understand your requests, requiring you to provide less detail up front while getting better results. Anyone familiar with OpenClaw, Hermes Agent, and similar projects will know where this is going.

Here’s how it works.

Read more


Remodex Is the Best Codex Remote Client for iOS (Until OpenAI Releases an Official Codex Mobile App)

Remodex for iOS.

Remodex for iOS.

Various OpenAI employees and members of the Codex team have been hinting at a native Codex app for iOS lately. While I very much hope that’s in the cards – especially if the project involves connecting to a remote Mac running the full Codex app – I wanted to highlight an indie utility I’ve been using a lot lately to access my Codex setup on my Mac Studio server from my iPhone.

The app is called Remodex, and it was created by Italian indie developer Emanuele Di Pietro. Remodex, as the name suggests, acts as a remote for the Codex CLI installed on a macOS computer, and it lets you operate your existing projects and chats with a UI that is reminiscent of the official Codex app for Mac. Even better, Remodex is not based on some hack-y workaround: it’s entirely powered by OpenAI’s official (and open-source) Codex App Server.

Read more


OpenAI Targets Coding and Knowledge Work with Its New GPT-5.5 Model

OpenAI announced GPT-5.5 and GPT-5.5 Pro today, which it says are faster and able to work more autonomously than the company’s previous models. It’s a message that is sure to interest business users whether their goal is accelerating software development or increasing productivity more generally. Some of the areas that OpenAI says GPT-5.5 and GPT-5.5 Pro excel at include:

  • writing and debugging code;
  • analyzing data;
  • conducting web research;
  • creating business documents such as spreadsheets and presentations;
  • using apps; and
  • juggling multiple tools.

In its press release, OpenAI claims that:

The gains are especially strong in agentic coding, computer use, knowledge work, and early scientific research—areas where progress depends on reasoning across context and taking action over time. GPT‑5.5 delivers this step up in intelligence without compromising on speed: larger, more capable models are often slower to serve, but GPT‑5.5 matches GPT‑5.4 per-token latency in real-world serving, while performing at a much higher level of intelligence. It also uses significantly fewer tokens to complete the same Codex tasks, making it more efficient as well as more capable.

I haven’t tried either model yet, but early reactions seem to support OpenAI’s claims that GPT-5.5 understands user intent better, requiring less precise instructions. The company says it is better at using the tools at its disposal, and checking its own work, too. OpenAI says the Pro model takes that up a notch, working faster on more complex tasks, such as programming, research, and document-intensive workflows. Whether the early hype translates into real-world gains that are noticeable in everday work, remains to be seen, but we shouldn’t have long to wait though, since GPT-5.5 is rolling out to users now.

GPT-5.5 is available in ChatGPT and Codex to Plus, Pro, Business, and Enterprise subscribers, and GPT-5.5 Pro is limited to Pro, Business, and Enterprise subscribers in ChatGPT. Neither model is available through OpenAI’s API, but the company says they will be soon.


OpenAI’s New Codex App Has the Best ‘Computer Use’ Feature I’ve Ever Tested

Computer use in Codex.

Computer use in Codex.

OpenAI rolled out their updated Codex app for Mac yesterday and, among other things, they shipped a native computer use tool for macOS that lets Codex interact with multiple Mac apps in the background using parallel cursors that do not bring apps to the foreground when agents are interacting with them. The feature that OpenAI rolled out in Codex is literally based on the Sky app that I exclusively previewed last year, and which was later acquired by OpenAI along with the team that built it.1

I feel like I’m in a pretty unique position to comment on all this since, as MacStories readers will recall, I was able to test Sky for several months last year before the team went radio-silent and joined OpenAI. Here’s the thing: I’m not exaggerating when I say that Codex now features the best computer use feature I have ever tested in any LLM or desktop agent. In fact, it’s even better than the computer use feature I used in Sky last year: Sky’s computer use was great, but it was considerably slower than Codex’s current one because it was running on Anthropic’s Claude models. With Codex for Mac today, even the (kind of slow) GPT 5.4 is faster than Sky ever was. But, using Codex with fast mode or – for simpler tasks – the Cerebras-hosted GPT-5.3-Codex-Spark model yields dramatically faster performance than Sky for Mac delivered in 2025.

But why is that? Allow me to explain. Most computer use models (such as the one in the Claude app, or even the just-released Personal Computer by Perplexity) rely on a combination of screen-recording capabilities and some AppleScript to either simulate virtual clicks on-screen and perform basic actions inside apps by calling osascript in a virtual shell. Sky was different, and Codex is different, and I can share more details today that I did not elaborate on when I wrote about Sky last year.

We all have Apple’s Accessibility team to thank for the technology that allows Codex’s computer use tool to exist. To build it, the Codex team took advantage of an advanced accessibility feature that allows third-party apps to read the “accessibility hierarchy” (also known as “AX Tree”) of any app open on macOS. My understanding is that this technology was primarily created to allow screen-readers and other assistive tools to work with Mac apps regardless of their automation/scripting features. In this case, it’s been repurposed as a way for Codex to ingest the full contents and hierarchy of any window and, essentially, load it as context for the LLM.

When I was told last year that this was how Sky worked behind the scenes, I instantly knew it reminded me of something, and I was right. We’ve seen the same technology being used before in UI Browser, the excellent (and sadly discontinued) app to inspect the visual hierarchy of any app that’s also powered by screen-reader APIs on macOS. All of this still applies to Codex’s computer use plugin today: pay attention to any chat where you’re using the plugin, and you’ll see 5.4 reason about the “accessibility tree” it wants to parse from any given application.

As someone who’s played around with GUI scripting and UI Browser many times over the years, let me tell you: this is not easy, and these frameworks were not meant for automation. For starters, they return a lot of text about any possible UI element, text field, or button inside a window. That text can be formatted in a variety of ways; it can be so deeply nested inside the XML-like structure returned by the AX framework, you often need to navigate 20 levels deep into a structure to find what you want. But this is what makes Codex’s computer use model different, why the Sky acquisition was a very clever move from OpenAI, and also why the reactions online seem overwhelmingly positive: Codex can “see” more inside apps and can control them more precisely than other models based solely on capturing screenshots, simulating clicks on certain coordinates, and running the occasional AppleScript. Codex can also do those things as fallback measures, but they’re not the primary drivers of its computer use plugin.

It also helps that computer use in Codex is exquisitely designed – not a surprise given OpenAI’s design team and the pedigree of the team behind this feature. The flow for granting permissions to the plugin is the best I’ve ever seen in a third-party Mac app – and it comes directly from Sky, which had the same onboarding experience. What Sky didn’t have is the new virtual cursor: the Codex team designed an entire system for it where the cursor can wiggle to show when the model is thinking, takes playful paths, and derives its color from the system’s wallpaper. I can only think of another company that sweats these kinds of UI details as much as the Codex team did here…and I’ll let you guess where several of Codex’s engineers and designers are, in fact, coming from.

I’ve been working with computer use in Codex all day, and while it is not as fast as a skilled human who knows a particular macOS interface well, it is very good at understanding and controlling any Mac app in the background a bit more slowly, with greater precision than competing features from Anthropic and Perplexity. That makes it ideal to automate busywork in Mac apps that do not offer an API or CLI, or which can’t be fully controlled with AppleScript. Let me give you some practical examples.

Earlier today, I asked both Perplexity’s Personal Computer and Codex to “play the latest album from the weird masked band from Quebec, I don’t remember their name”. I was referring to the exceptional Angine de Poitrine, of course. Both agents searched the web upfront and pinpointed my request, but when it came to actually controlling the Music app, Personal Computer stopped short of hitting the ‘Play’ button because its AppleScript integration couldn’t do it; Codex went ahead, opened the album with its virtual cursor, and started playing music.

Personal Computer couldn’t hit Play.

Personal Computer couldn’t hit Play.

Codex had no issues playing music in the Music app.

Codex had no issues playing music in the Music app.

I also tested Codex by asking it to look at specific channels on Slack, my Ivory timeline, and the Unread app and give me a summary of interesting updates I should know about. Codex successfully deployed parallel cursors, started scrolling and clicking around all three apps, and produced a report that included updates gathered from those apps. Could I have scrolled the apps myself, one after the other, the old fashioned way? Sure. But as an “automation” that happened in the background while I was doing my email, it was pretty good.

Codex’s report from three separate apps.

Codex’s report from three separate apps.

The other task I attempted today – which is still running, after 6 hours – was using Codex’s computer use to improve the Shortcuts Playground skill I’ve been building to create shortcuts in the Shortcuts app using coding agents in natural language. With Codex, I figured I could now ask the agent to run the skill, create shortcuts for me, but also click the resulting .shortcut files in Finder, install them, and test them for me in the Shortcuts app to spot any errors and further improve the skill. Not only was Codex’s computer use plugin able to successfully install dozens of shortcuts, but it also opened each, verified its output, and is currently evaluating what went wrong to improve some of the skill’s guidance and instructions.

Codex installed all these shortcuts via computer use.

Codex installed all these shortcuts via computer use.

The Codex cursor debugging a shortcut for me.

The Codex cursor debugging a shortcut for me.

So, long story short: Codex’s computer use plugin is the state of the art at the moment, and it’s the evolution of a strong foundation that I was able to test last year, which has been further refined and expanded by OpenAI. I’d like to see the company expand this plugin to the main ChatGPT for Mac experience (which is still stuck on the old Work with Apps integration), but, for now, I’ll take this feature inside Codex rather than the slower, and less capable, computer use models from other chatbots. More importantly, I’m happy to see that Sky ended up in good hands who can now deliver this product to the masses.


  1. I don’t use the term “literally” in a liberal sense here. When you enable the Computer Use plugin in Codex, you can head over to the app’s config.toml configuration file, open it in a text editor, and you’ll spot this line:
    /Users/username/.codex/plugins/cache/openai-bundled/computer-use/1.0.750/Codex Computer Use.app/Contents/SharedSupport/SkyComputerUseClient.app/Contents/MacOS/SkyComputerUseClient

    Open that folder and, sure enough, there’s an executable for the former Sky “app”, now loaded as a first-party 2plugin that handles the virtual computer interactions for Codex. ↩