Posts in Linked

Inside Your iPhone’s Brain

Steven Levy has a fascinating inside look at Apple’s artificial intelligence and machine learning efforts on Backchannel. Levy spent most of a day with Eddy Cue, Phil Schiller, Craig Federighi, Tom Gruber, and Alex Acero in a wide-ranging discussion of the products impacted by those efforts. Perhaps the most interesting parts of the interviews revolved around what Levy refers to as the Apple Brain inside the iPhone:

How big is this brain, the dynamic cache that enables machine learning on the iPhone? Somewhat to my surprise when I asked Apple, it provided the information: about 200 megabytes, depending on how much personal information is stored (it’s always deleting older data). This includes information about app usage, interactions with other people, neural net processing, a speech modeler, and “natural language event modeling.” It also has data used for the neural nets that power object recognition, face recognition, and scene classification.

And, according to Apple, it’s all done so your preferences, predilections, and peregrinations are private.

Levy also covers the replacement of Siri’s smarts on July 30, 2014 with neural-net system. The impact according to Eddy Cue was immediate:

This was one of those things where the jump was so significant that you do the test again to make sure that somebody didn’t drop a decimal place.

Many people have commented that Siri has improved over time, but without the context beyond one’s own experience or metrics from Apple, the perceived change has been largely anecdotal. According to Acero, however:

The error rate has been cut by a factor of two in all the languages, more than a factor of two in many cases.… That’s mostly due to deep learning and the way we have optimized it — not just the algorithm itself but in the context of the whole end-to-end product.

Levy also delves into whether Apple’s stance on privacy hobbles its ability to effectively implement AI and machine learning. According to Apple, it does not. The most personal information remains on-device in the ‘Apple Brain.’ Other data, which is transmitted to Apple uses techniques like differential privacy, which is coming in iOS 10, to obfuscate a user's identity.

The entire article is worth a read to get a sense of the breadth and depth of Apple’s AI and machine learning efforts and the impact on its products. It’s also fascinating to see Apple continue to open up on its own terms as a way to rebut recent criticisms leveled against it.


Pinterest Acquires Instapaper

In a surprising move, Pinterest has acquired Instapaper, the iOS app and web service originally built by Marco Arment and sold to betaworks in 2013. According to Casey Newton of The Verge:

The goal is "to accelerate discovering and saving articles on Pinterest," the company said in a statement. It will continue to operate as a standalone app, and the Instapaper team will work on both that app and on Pinterest generally.

Pinterest has had an article saving feature since 2013, though the service is far better known as a way to bookmark images and other visual content. With the acquisition of Instapaper, which Pinterest says will be maintained as a stand-alone app, Pinterest appears to be looking for a way to expand its customer base into sectors adjacent to it’s core product.


iOS Accessibility Milestones

The impact of the accessibility features built into iOS cannot be understated. Accessibility has opened doors to computing that were previously shut to many people with disabilities. With iOS 10 launching soon, Steven Aquino takes a look at iOS accessibility milestones in a guest post on 512 Pixels that focuses on five key features: VoiceOver, Guided Access, Large Dynamic Type, Switch Control, and Magnifier.

It feels like VoiceOver has been around forever, so it was interesting to be reminded that it didn’t debut until the introduction of the iPhone 3GS and has its roots in the short-lived buttonless iPod Shuffle. Of the other accessibility feature covered, the one that will probably be the least familiar to most readers is Magnifier, which is coming in iOS 10. Aquino believes that when we look back at iOS 10 in the future, Magnifier will be viewed as one of the greatest enhancements to iOS:

The reason I’m so effusive about Magnifier is the handiness of it. So often, I’m reading a restaurant menu or looking at price tags in the grocery store, and the print in set in small font. Where previously I would strain my eyes in order to see, now all I need to do is pull out my phone and triple-press the Home button to launch Magnifier.

Steven has written a lot about the accessibility of Apple products, including here at MacStories, and does a wonderful job putting each accessibility milestone into historical context. I encourage everyone to read the full article on 512 Pixels.


Apple Acquires Health Care Start-Up Gliimpse

One of the most interesting quotes from Fast Company’s interview with Tim Cook a couple of weeks ago was his comment about healthcare:

When you look at most of the solutions, whether it’s devices, or things coming up out of Big Pharma, first and foremost, they are done to get the reimbursement [from an insurance provider]. Not thinking about what helps the patient. So if you don’t care about reimbursement, which we have the privilege of doing, that may even make the smartphone market look small.

Today, Fast Company is reporting that Apple has confirmed that it acquired start-up Gliimpse earlier this year. According to Fast Company:

Silicon Valley-based Gliimpse has built a personal health data platform that enables any American to collect, personalize, and share a picture of their health data. The company was started in 2013, and funded by serial entrepreneur Anil Sethi, who has spent the past decade working with health startups, after taking his company Sequoia Software public in 2000. He got his start as a systems engineer at Apple in the late 1980s.

Gliimpse feels like a natural fit with the Health app, HealthKit, ResearchKit, and CareKit, especially considering Apple’s focus on data privacy. It’s still very early days, but between recent keynotes featuring the health-related features of the Apple Watch and acquisitions like this, Apple’s commitment to exploring healthcare opportunities is unmistakable.


Emojis Explained

There's a lot of confusion about what actually constitute emojis, in no small measure due to the term's liberal use by apps like Kimoji. Owen Williams sets the record straight, dropping some emoji knowledge over on Emojipedia. Williams starts with a little history:

The term "emoji" originates from Japan, and it's a generic term there, similar to emoticon in English (though the fact they sound similar is purely coincidental). They started life as a set of pictures out of a research laboratory, and introduced nationwide after DoCoMo i-mode shipped with the first set.

Emoji evolved into the term used to describe the characters approved by the Unicode Consortium that work on any device. As Williams explains:

Put simply: [an emoji is] actually a universal code set that translates from machine speak into the pictures you see when you send a 🙃 to your friends and they know what you're talking about.

What's cool about emoji is that they work like a letter of the alphabet. Sending an emoji doesn't send an image, just the code, which each device translates into the corresponding image.

As Williams explains, Kimoji aren't emoji at all, they're image-based stickers wrapped in a custom iOS keyboard app. Twitter hashflags are also custom images that are even more restrictive because they only work on Twitter.

With iOS 10's adoption of sticker packs in Messages, the distinction between emojis and stickers is likely to get even blurrier, but remember emoji are free, built-in, and cross-platform, which makes them the most flexible way to express yourself to your friends.

(Image by


Enter the Studio Turning Deus Ex and Lara Croft Into Awesome Mobile Games

Andrew Webster, writing for The Verge:

With the Go series, Square Enix Montreal has carved out its own niche, creating something unique in the game development space. Studios often fall into one of two camps: on the one side you have the massive, 1,000-person teams that create blockbuster games, and on the other there are the tiny indie studios that build creatively ambitious games with few resources. Square Enix Montreal straddles the line between those two extremes. It has the resources of a big company, but the size and some of the creative freedom of an indie. It’s a studio that can make weird new games but attach them to hugely popular franchises.

It is great to see that Square Enix Montreal has found success in its series of Go games built on the larger franchises of Hitman, Tomb Raider and now Deus Ex. The first two Go mobile games, Hitman Go and Lara Croft Go, are genuinely great and feature a lot of creativity – so it is great to see they have continued to invest in this (critically-acclaimed) series with yesterday's launch of Deus Ex Go. This is particularly the case when so many other large mobile game publishers are instead focusing on churning out what are largely uninspired free games with in-app purchases.

To that end, Webster notes in his story that Square Enix Montreal has made some indie hires that suggests it fully intends to stay the course on its current approach to mobile games:

Outside of Deus Ex Go, Square Enix Montreal isn’t saying what it’s working on right now. But the studio has made a few recent hires that hint at desire to keep the indie-like feeling it has carefully cultivated. Those pick-ups include Teddy Dief, an artist and designer best known for his work on the crowdfunded hit Hyper Light Drifter, and Renaud Bédard, the sole programmer on seminal puzzle-platformer Fez, who most recently worked at Below developer Capy Games in Toronto. Both were tempted to join by the idea of combining the creative freedom of an indie studio with the structure and resources of a big publisher.


Apple’s Presentation at Black Hat Now Available to Watch in Full

Ivan Krstić, Apple's Head of Security Engineering and Architecture, gave a presentation at the Black Hat conference a few weeks ago, and it is now available to view in full on YouTube.

With over a billion active devices and in-depth security protections spanning every layer from silicon to software, Apple works to advance the state of the art in mobile security with every release of iOS. We will discuss three iOS security mechanisms in unprecedented technical detail, offering the first public discussion of one of them new to iOS 10.

HomeKit, Auto Unlock and iCloud Keychain are three Apple technologies that handle exceptionally sensitive user data – controlling devices (including locks) in the user's home, the ability to unlock a user's Mac from an Apple Watch, and the user's passwords and credit card information, respectively. We will discuss the cryptographic design and implementation of our novel secure synchronization fabric which moves confidential data between devices without exposing it to Apple, while affording the user the ability to recover data in case of device loss.

It was at this presentation that Apple announced that it would launch a bug bounty program for those who discover vulnerabilities in its key products. Also discussed by Krstić during his presentation is how the Secure Enclave Processor enabled Apple to adopt a new approach to data protection, as well as a new security feature in iOS 10 that makes iOS Safari JIT "a more difficult target".

(via MacRumors)


An Inside Look at the Apple Watch’s Development

Fast Company spoke to Bob Messerschmidt, who worked on the heart rate sensor for the Apple Watch after his startup was acquired by Apple in 2010, about the lessons he learned during his time at Apple. Some of the most interesting bits were Messerschmidt’s description of how designers and engineers interact on a product like the Apple Watch:

One great example is [when] I went to a meeting and said I’m going to put sensors in the watch but I’m going to put them down here (he points to the underside of the Apple Watch band he’s wearing) because I can get a more accurate reading on the bottom of the wrist than I can get on the top of the wrist. They (the Industrial Design group) said very quickly that "that’s not the design trend; that’s not the fashion trend. We want to have interchangeable bands so we don’t want to have any sensors in the band."

Like many before him, Messerschmidt was also impressed by Apple’s focus on products over technology:

At Apple I learned that design and user experience is everything when it comes to consumer products. It’s not so much the technology. It’s the design of the product that creates that sense of happiness in the user.

If you look at products like the iPhone or the iPad there aren’t too many totally new technologies included in those products. The real elegance and differentiation doesn’t have a lot to do with the technology idea itself; it’s about the packaging and the value add it gives to people. Those big (new technology) ideas generally happen elsewhere, and they happen earlier.

Messerschmidt’s interview is particularly notable because it's not often we get a perspective on the interplay between Jony Ive's design team and Apple's engineers.


Microsoft Office for iPhone Gets Drawing Support

Tom Warren, writing for The Verge:

Microsoft was quick to optimize its Office suite of apps for the iPad Pro and Apple's Pencil stylus, but the company held off on any inking support in Office for iPhone. Starting today, Microsoft is updating Word, Excel, and PowerPoint for iPhone to include a new drawing tab option. Just like the Windows and iPad variants, Office on iPhone will now let you use your finger to write, draw, and highlight documents.

Another great update for Word, Excel and PowerPoint on iOS.

Because space is limited on an iPhone screen, these drawing features are a little hidden. So in order to access these drawing features on the iPhone you'll need to tap the icon on the top navigational bar that looks like an A with a pencil cutting through it. That will trigger a pop-up on the bottom half of the screen. From there, on the top-left of the pop-up should be a drop-down menu, tap that and choose "Draw".