Inside Your iPhone’s Brain

Steven Levy has a fascinating inside look at Apple’s artificial intelligence and machine learning efforts on Backchannel. Levy spent most of a day with Eddy Cue, Phil Schiller, Craig Federighi, Tom Gruber, and Alex Acero in a wide-ranging discussion of the products impacted by those efforts. Perhaps the most interesting parts of the interviews revolved around what Levy refers to as the Apple Brain inside the iPhone:

How big is this brain, the dynamic cache that enables machine learning on the iPhone? Somewhat to my surprise when I asked Apple, it provided the information: about 200 megabytes, depending on how much personal information is stored (it’s always deleting older data). This includes information about app usage, interactions with other people, neural net processing, a speech modeler, and “natural language event modeling.” It also has data used for the neural nets that power object recognition, face recognition, and scene classification.

And, according to Apple, it’s all done so your preferences, predilections, and peregrinations are private.

Levy also covers the replacement of Siri’s smarts on July 30, 2014 with neural-net system. The impact according to Eddy Cue was immediate:

This was one of those things where the jump was so significant that you do the test again to make sure that somebody didn’t drop a decimal place.

Many people have commented that Siri has improved over time, but without the context beyond one’s own experience or metrics from Apple, the perceived change has been largely anecdotal. According to Acero, however:

The error rate has been cut by a factor of two in all the languages, more than a factor of two in many cases.… That’s mostly due to deep learning and the way we have optimized it — not just the algorithm itself but in the context of the whole end-to-end product.

Levy also delves into whether Apple’s stance on privacy hobbles its ability to effectively implement AI and machine learning. According to Apple, it does not. The most personal information remains on-device in the ‘Apple Brain.’ Other data, which is transmitted to Apple uses techniques like differential privacy, which is coming in iOS 10, to obfuscate a user's identity.

The entire article is worth a read to get a sense of the breadth and depth of Apple’s AI and machine learning efforts and the impact on its products. It’s also fascinating to see Apple continue to open up on its own terms as a way to rebut recent criticisms leveled against it.


Pinterest Acquires Instapaper

In a surprising move, Pinterest has acquired Instapaper, the iOS app and web service originally built by Marco Arment and sold to betaworks in 2013. According to Casey Newton of The Verge:

The goal is "to accelerate discovering and saving articles on Pinterest," the company said in a statement. It will continue to operate as a standalone app, and the Instapaper team will work on both that app and on Pinterest generally.

Pinterest has had an article saving feature since 2013, though the service is far better known as a way to bookmark images and other visual content. With the acquisition of Instapaper, which Pinterest says will be maintained as a stand-alone app, Pinterest appears to be looking for a way to expand its customer base into sectors adjacent to it’s core product.


PDF Expert 2 for Mac Adds Powerful PDF Editing Features

PDF Expert launched on the Mac last November, and in my initial review I was pretty effusive, impressed at the level of functionality, polish, and speed for an initial release. At the time I even called it "a better Preview for PDFs", and had made PDF Expert the default application for viewing PDFs on my Mac. Nine months later, and it all still rings true. Better yet, Readdle is today launching a big version 2 update for PDF Expert which makes it an even better and more powerful app. Now you can now edit text, images, and outlines in PDFs, as well as password-protect your PDFs in PDF Expert 2.

Read more

Twitter Enables Night Mode

In beta for a while now, Twitter activated night mode in its official client today. From the ‘Me’ tab, tap the gear icon to ‘Turn on night mode.’ If you don’t see the night mode option, you may need to force quit Twitter first.

Turning night mode on.

Turning night mode on.

Twitter has done a very nice job with night mode. The background is a dark blue-grey, icons and secondary text are a lighter, complimentary grey, links and certain other buttons are bright ‘Twitter blue,’ and plain text is white. The overall effect looks great. With so many apps I use everyday adding night modes and dark themes, an iOS system-level night mode feels like the natural next step.

iOS Accessibility Milestones

The impact of the accessibility features built into iOS cannot be understated. Accessibility has opened doors to computing that were previously shut to many people with disabilities. With iOS 10 launching soon, Steven Aquino takes a look at iOS accessibility milestones in a guest post on 512 Pixels that focuses on five key features: VoiceOver, Guided Access, Large Dynamic Type, Switch Control, and Magnifier.

It feels like VoiceOver has been around forever, so it was interesting to be reminded that it didn’t debut until the introduction of the iPhone 3GS and has its roots in the short-lived buttonless iPod Shuffle. Of the other accessibility feature covered, the one that will probably be the least familiar to most readers is Magnifier, which is coming in iOS 10. Aquino believes that when we look back at iOS 10 in the future, Magnifier will be viewed as one of the greatest enhancements to iOS:

The reason I’m so effusive about Magnifier is the handiness of it. So often, I’m reading a restaurant menu or looking at price tags in the grocery store, and the print in set in small font. Where previously I would strain my eyes in order to see, now all I need to do is pull out my phone and triple-press the Home button to launch Magnifier.

Steven has written a lot about the accessibility of Apple products, including here at MacStories, and does a wonderful job putting each accessibility milestone into historical context. I encourage everyone to read the full article on 512 Pixels.


Apple Acquires Health Care Start-Up Gliimpse

One of the most interesting quotes from Fast Company’s interview with Tim Cook a couple of weeks ago was his comment about healthcare:

When you look at most of the solutions, whether it’s devices, or things coming up out of Big Pharma, first and foremost, they are done to get the reimbursement [from an insurance provider]. Not thinking about what helps the patient. So if you don’t care about reimbursement, which we have the privilege of doing, that may even make the smartphone market look small.

Today, Fast Company is reporting that Apple has confirmed that it acquired start-up Gliimpse earlier this year. According to Fast Company:

Silicon Valley-based Gliimpse has built a personal health data platform that enables any American to collect, personalize, and share a picture of their health data. The company was started in 2013, and funded by serial entrepreneur Anil Sethi, who has spent the past decade working with health startups, after taking his company Sequoia Software public in 2000. He got his start as a systems engineer at Apple in the late 1980s.

Gliimpse feels like a natural fit with the Health app, HealthKit, ResearchKit, and CareKit, especially considering Apple’s focus on data privacy. It’s still very early days, but between recent keynotes featuring the health-related features of the Apple Watch and acquisitions like this, Apple’s commitment to exploring healthcare opportunities is unmistakable.


Game Day: Linia

Linia from Black Robot Games is fiendishly difficult, but strangely relaxing in its complexity. The challenge is to draw a straight line through a series of colored shapes that intersects the shapes in the order of the color sequence at the top of the screen. Here's the thing though, the shapes are moving, rotating, shrinking, growing, and changing color all at once. The movement follows a regular pattern, but it gets complicated fast.

Linia is a creature of the post-iOS 7 design aesthetic. Each of its 80 levels is composed of brightly-colored geometric flat shapes. What's ingenious about Linia and makes it particularly difficult is that it requires pattern matching, careful timing, and quick reflexes simultaneously. As you draw a line it appears white but fills in behind where you started in red, which is part of the timing element. You have to lift your finger to commit to your line before it turns completely red. If you wait to long, you'll have to start drawing your line all over, but if you release at the wrong time, you may be unable to hit the right color sequences.

I've played Linia on both my iPhone 6s Plus using my finger and on my 12.9 inch iPad Pro using the Apple Pencil. I give a slight edge to using the Pencil to play Linia on the iPad because you can draw such a precise line, but the difference between playing on the iPad and iPhone was much less than I expected.

The soundtrack that accompanies Linia plays a big role in minimizing the frustration of some of the harder puzzles. There's something inviting about the electronic vibe of the soundtrack that feels like it's encouraging you to stay a while to keep trying to beat even the hardest puzzles.

Black Robot has done a great job bringing something fresh and clever to the crowded puzzle genre on iOS. It's especially impressive given that this is Black Robot's first iOS game. With such a great start, I look forward to seeing what else Black Robot comes up with in the future.

Linia is available on the App Store for $1.99.

Emojis Explained

There's a lot of confusion about what actually constitute emojis, in no small measure due to the term's liberal use by apps like Kimoji. Owen Williams sets the record straight, dropping some emoji knowledge over on Emojipedia. Williams starts with a little history:

The term "emoji" originates from Japan, and it's a generic term there, similar to emoticon in English (though the fact they sound similar is purely coincidental). They started life as a set of pictures out of a research laboratory, and introduced nationwide after DoCoMo i-mode shipped with the first set.

Emoji evolved into the term used to describe the characters approved by the Unicode Consortium that work on any device. As Williams explains:

Put simply: [an emoji is] actually a universal code set that translates from machine speak into the pictures you see when you send a 🙃 to your friends and they know what you're talking about.

What's cool about emoji is that they work like a letter of the alphabet. Sending an emoji doesn't send an image, just the code, which each device translates into the corresponding image.

As Williams explains, Kimoji aren't emoji at all, they're image-based stickers wrapped in a custom iOS keyboard app. Twitter hashflags are also custom images that are even more restrictive because they only work on Twitter.

With iOS 10's adoption of sticker packs in Messages, the distinction between emojis and stickers is likely to get even blurrier, but remember emoji are free, built-in, and cross-platform, which makes them the most flexible way to express yourself to your friends.

(Image by


Enter the Studio Turning Deus Ex and Lara Croft Into Awesome Mobile Games

Andrew Webster, writing for The Verge:

With the Go series, Square Enix Montreal has carved out its own niche, creating something unique in the game development space. Studios often fall into one of two camps: on the one side you have the massive, 1,000-person teams that create blockbuster games, and on the other there are the tiny indie studios that build creatively ambitious games with few resources. Square Enix Montreal straddles the line between those two extremes. It has the resources of a big company, but the size and some of the creative freedom of an indie. It’s a studio that can make weird new games but attach them to hugely popular franchises.

It is great to see that Square Enix Montreal has found success in its series of Go games built on the larger franchises of Hitman, Tomb Raider and now Deus Ex. The first two Go mobile games, Hitman Go and Lara Croft Go, are genuinely great and feature a lot of creativity – so it is great to see they have continued to invest in this (critically-acclaimed) series with yesterday's launch of Deus Ex Go. This is particularly the case when so many other large mobile game publishers are instead focusing on churning out what are largely uninspired free games with in-app purchases.

To that end, Webster notes in his story that Square Enix Montreal has made some indie hires that suggests it fully intends to stay the course on its current approach to mobile games:

Outside of Deus Ex Go, Square Enix Montreal isn’t saying what it’s working on right now. But the studio has made a few recent hires that hint at desire to keep the indie-like feeling it has carefully cultivated. Those pick-ups include Teddy Dief, an artist and designer best known for his work on the crowdfunded hit Hyper Light Drifter, and Renaud Bédard, the sole programmer on seminal puzzle-platformer Fez, who most recently worked at Below developer Capy Games in Toronto. Both were tempted to join by the idea of combining the creative freedom of an indie studio with the structure and resources of a big publisher.