This week's sponsor

MacStadium

The worldwide Mac hosting authority


Beyond the Silo: How Apple Plans To Reinvent Document Management with iOS 8

With iOS 8, announced last week at WWDC, Apple is going to bring deep changes to one of the most controversial aspects of its mobile platform: document storage and management. While iCloud will play a big role with a unified iCloud Drive for iOS and OS X, third-party developers will also get a chance to add better file management functionalities to their apps.

The new features and APIs have been detailed by Apple during its opening keynote and in developer sessions throughout the week, and they follow a common thread: apps can now extend beyond their sandbox, accessing documents stored in other apps without creating unnecessary copies. To better understand the importance of these technological changes in iOS 8 and the inherent complexity that they’ll add for developers and users, I want to take a step back and contextualize how iOS currently handles file storage and management.

The Way It Was

My iPad is my primary computer, and has been since mid–2012. I’ve written about my experience with working from iOS before, and I considered the implications and consequences of my mobile-first habits when Apple released OS X Mavericks last year. Amidst Python scripting and text automation, I often mentioned one iOS feature that I was hoping Apple would eventually fix or remove completely: the Open In menu.

The Open In menu sends a document to another app, creating a copy of it.

The Open In menu sends a document to another app, creating a copy of it.

Traditionally, Apple has treated iOS apps as islands with extremely limited communication systems between them. Apple’s sandboxing model requires apps to store their files inside a proprietary container; for users of document-based apps (such as Pages or a text editor), this means they have to manually copy a document into another app’s container if they need to modify it with multiple apps.

For security reasons, iOS places each app (including its preferences and data) in a sandbox at install time. A sandbox is a set of fine-grained controls that limit the app’s access to files, preferences, network resources, hardware, and so on. As part of the sandboxing process, the system installs each app in its own sandbox directory, which acts as the home for the app and its data. – Apple

A typical example may be your average word processor: you want to write in Pages, but you also want to drop images into the document from an image editor and, to finish it up, correct everything using a dedicated syntax-checking tool. On OS X, the filesystem layer allows any compatible app to open a document created or modified by other apps, saving changes back to the original file’s location and generating versions of the file in the process. But try to do the same on iOS, and you’d end up with at least three copies of the same file, scattered across apps and consuming extra storage on your device. Not to mention that, to achieve that fairly trivial word processing workflow, you’d have to manually switch between apps every time.

The iOS sandbox.

The iOS sandbox.

It’s important to understand that Apple’s sandboxing model was intentionally designed to provide users with a secure solution meant to avoid data loss. Because an app couldn’t look into another sandbox and make modifications to a file that wasn’t its own, users could download apps from the App Store knowing that they wouldn’t have nefarious consequences on their existing documents. Developers couldn’t arbitrarily decide to make their apps capable of modifying other apps’ files. And the best part – users didn’t have to deal with the traditional filesystem: managing files with folders and sub-folders was a relic of the PC era; “app libraries” provided a much simpler, modern way of storing documents.

With time, however, Apple’s solution started to backfire as users began demanding convenience in exchange for control. Here’s what I wrote about the Open In menu in January 2013:

However, the silo model – as opposed to the central “Finder” repository of files – has one big drawback: communication between silos. Therefore “Open In”: a menu that copies a file from Location A to Location B, getting from one document to two documents, now available in two different locations. And I would argue that the second most complicated aspect of managing documents is: figuring out the “right” version of a file.

With iPhone and iPad users stuck on a file management system that was secure but getting clunky and annoying after years with no changes, developers started adopting the SDK approach: rather than waiting for Apple to add new document management and collaboration features, they’d include native support for a third-party cloud service in their apps, letting them sync and propagate changes to a single document on multiple devices at once.

Pages for iOS 7.

Pages for iOS 7.

Of all the third-party cloud services available to consumers, Dropbox is the one with the strongest foothold in the iOS productivity ecosystem. There are thousands of apps that sync documents with Dropbox, primarily because it allows users to work on the same copy of a file from multiple devices and is installed as a regular folder on desktop computers. As Dropbox’s popularity grew, native versioning and sharing features were added to the SDK for mobile apps, sweetening the deal.

Nowadays, it’s difficult to find a document-based app on the App Store that doesn’t come with some sort of syncing component for multi-device editing, be it Dropbox, Box, One Drive, or Apple’s iCloud.

An iCloud Document Library on OS X.

An iCloud Document Library on OS X.

Launched in 2011, iCloud did, in fact, provide a storage and syncing feature for documents, aptly named Documents in the Cloud. Unlike other services, though, Documents in the Cloud didn’t offer users a way to avoid creating copies of the same document if they wanted to edit it using multiple apps on an iOS device. Sandboxed documents would be pushed to iCloud and back to other devices, but, from a workflow standpoint, they’d still be isolated – incapable of communicating with other apps’ containers.

In a way, I believe the popularity of document syncing solutions for iOS apps is a byproduct of Apple’s basic sandboxing model. For many users who didn’t necessarily need sync, the advantages of sync – multi-platform, multi-device, and multi-app editing of the same file – became an indispensable tool to achieve a document management workflow that went beyond the limitations imposed by Apple. It would be fun to imagine what kind of growth popular cloud services would have seen on mobile devices had Apple shipped better document management features from day one, but that wouldn’t, ultimately, be a productive thought experiment.

With iOS 8, Apple is going to reinvent both sides of document storage and management. For users, cloud storage and sync will become more visible with iCloud Drive, and apps will be able to collaborate on the same file without creating duplicates. For third-party developers who don’t want to rely on iCloud, Apple is building a new integrated solution: storage provider extensions for any app.

iCloud Drive

For iOS 8 and OS X Yosemite, Apple has decided that a more conventional approach is what over 400 million iCloud users have been looking for to manage their documents. As with Extensibility, the new technology is comprised of user-facing features and developer APIs with an underlying theme of letting apps reach beyond the sandbox with a focus on security and familiarity. It’s a complex change that will require Apple and third-party developers to carefully consider the technological implications in their apps.

For users, the most visible new feature is iCloud Drive, a unified location for all their documents. Built on top of Apple’s new CloudKit, iCloud Drive will be available as a special folder on OS X with sub-folders for apps that are storing documents in iCloud; on iOS, iCloud Drive won’t be a dedicated app, but it’ll be built into a new document picker that every app can implement.

iCloud Drive in the iOS document picker.

iCloud Drive in the iOS document picker.

iCloud Drive marks a notable departure from the company’s Documents in the Cloud, which prevented each iCloud Document Library (contained inside its originating app) from looking into other apps’ document libraries. And while there was a way to manually browse the directories that iCloud was actually creating in the Finder on OS X, the system wasn’t officially supported by Apple and didn’t have a user-friendly interface.

iCloud Drive will display all document libraries from all iCloud apps with special folder icons for the app that created the documents. On every device, iCloud Drive will let users browse folders for documents created on iOS or OS X; desktop users will also be able to organize their iCloud Drive however they want by dropping documents and folders from the Finder into it.

So while iCloud’s principle is the same – all your documents, on all your devices, and you can pick up where you left off – the way this idea is presented to the user is different and, surprisingly, reminiscent of a traditional filesystem. For years, Apple tried to push iCloud as a document storage/persistence solution that wouldn’t require any sort of management from the user, whereas with iOS 8 and OS X Yosemite you will have the “freedom to work with the document of your choice” in “the way you like”. It’s an important change, in line with the user customization and flexibility of iOS 8.

The Document Picker

The availability of iCloud document libraries in a single iCloud Drive, though, is only one side of the document management story in iOS 8, which, alongside a new interface for iCloud storage, will give developers new APIs to store files, open them, and work with them without creating duplicates.

The iOS 8 document picker, from Apple's developer session.

The iOS 8 document picker, from Apple's developer session.

The new iOS 8 document picker will let users select files from outside an app’s sandbox by looking into other iCloud-enabled apps or storage provider extensions. The feature will be opt-in for developers, who will be able to choose whether or not their apps should offer a public scope of documents to other apps. The only way for apps to discover documents outside their sandbox will be the document picker, therefore users will always remain in control by granting their permission to access and edit documents and manually picking them (just like extensions will have to be activated by the user).

The document picker will come with four different types of file operations: Open, Move, Import, and Export. According to Apple, Open and Move will be a “significant departure” from traditional iOS document handling, as they will allow apps to access files from another app’s container, letting users edit files in place without creating unnecessary copies. This is a profound change in Apple’s sandboxing model, which prevented apps from looking into each other’s container to work on the same file.

In practical terms, it means that, if developers will support the feature, an app like Pages could edit the same document created by another word processor, or that a text editor like Byword for iOS could open and let you modify a .txt file previously created in TextEdit for Mac and stored in iCloud. "I was in awe when I saw the announcements", Jorge Pedroso, developer of Byword, told me. "One of the things on iOS I was always concerned about was data portability, so moving to a system where the user does that not have his data constrained to a single app silo is a big win for apps and specially the users", he added.

Apps in iOS 8 will also have modes for copying files into a new destination without touching the originals. The aforementioned Import and Export modes will behave like iOS’ old Open In menu but, in another improvement from iOS 7, they won’t force users to leave the app they’re currently using. Import and Export will simply create copies into or outside the app’s container, but they won’t involve any app switching. Another upside of the Import and Export mechanisms is that apps that don’t support iCloud document storage will be able to import documents from iCloud apps that have enabled access, generating an offline copy of a file.

In pre-release technical documentation, Apple stressed that developers will have to consider the file management modes they want to use in their apps, as users may not need all of them and confusion could arise over subtle differences between Import and Open modes. According to Apple, most users will want the flexibility and power offered by access to a document with Open and Move, and these operations should be emphasized by developers over “simple imports and exports”. Not to mention the fact that, even with less app switching, Import and Export modes will still create additional copies of a file, consuming extra storage space and making it difficult to work on a single document using multiple apps.

In developer sessions at WWDC, Apple engineers noted that the combination of iCloud Drive with the possibility of editing files from other apps will make users “happy” and lead to a better user experience.

It’s important to note that in spite of the changes in document handling and file operations, Apple isn’t abandoning the sandboxing model or granting indiscriminate access to app containers.

As I mentioned above, the document picker will be the only way for users to access documents outside of an app’s container and developers will have to set the scope of their app containers to public. Furthermore, apps that decide to access a document from external app containers won’t simply bypass the sandbox to access the file: rather, apps will store a reference to the file, which will include a security-scoped URL that grants access to the file. The concept of security-scoped URLs isn’t new, as it’s the same technology that Apple uses on OS X to give apps access to files outside their sandbox across relaunches and system restarts.

All these measures highlight the fact that Apple is still invested in the iOS sandboxing model, but they’re willing to make it more flexible for users by adopting technologies they first built for OS X – and all while giving the system a friendly interface. In Apple’s view, apps still can’t generally access other apps’ containers, but developers of document-based iOS 8 apps will be able to make an exception with the document picker. So while two word processors can collaborate on a single document with Open and Move operations, a Twitter client may not be able to, for instance, arbitrarily access a preference or a photo stored in another social networking app.

The rules that Apple is setting for document handling on iOS 8 share a few similarities with extensions: both will have to go through specific activation points, and both will work alongside sandboxing rather than against it. For documents, Apple is adding some extra limitations as well: apps won’t be able to write new files outside of their sandbox, and the Open and Move features will require iCloud support if an app is trying to perform those operations on another iCloud Drive-enabled app. For new files, this means that you won’t be able to open App A and create a new file directly inside App B’s container: if the two apps support the feature, App A will need to write the file to a temporary location and then request to move it. To the end user, the difference will likely be minimal, but it’s an important one for developers.

Storage Providers

iOS 8 will also give developers a way to offer their apps as storage services available as extensions in the document picker. With storage provider extensions, third-party apps will be able to present a native user interface in the document picker to communicate that they can work as document providers independently from iCloud Drive. During the WWDC keynote, Apple showed screenshots of Box and OneDrive extensions for iOS 8, suggesting that the feature is primarily intended for web services that have native iOS apps.

With the storage provider extension, apps like Box and OneDrive will be able to support the four basic file operations and handle downloads and uploads to a web service that’s not iCloud. Potentially, storage provider extensions could allow developers of apps that integrate with cloud-based file storage services to forego dedicated SDKs and rely solely on extensions.

When third-party developers implement an SDK from services like Google Drive and Dropbox, they usually need to think about sync logic and conflicts on their own, which requires lengthy development times and a solid understanding of how different web services handle sync. With storage providers, extensions will be in charge of downloads and uploads to remote servers and apps won’t need to be aware of the details of Dropbox or OneDrive – they’ll just need to support new file operations and know that they’re working with an extension.

Abstracting the UI and syncing logic from apps that work with third-party storage services will be “an enormous win for the iOS developer community”, Greg Scown, co-founder of Smile, told me. Smile is the company behind PDFpen, a popular suite of apps for managing, annotating, and creating PDF documents on OS X and iOS devices. PDFpen has built-in support for iCloud as well as Evernote, Dropbox, Google Drive, and Box, but those web integrations required Smile engineers to adapt third-party SDKs to their apps’ UIs and syncing logic. Storage providers on iOS 8, Scown added, are “likely to save most of the developer hours it takes for tens of thousands of apps to duplicate effort providing access to cloud services”, a sentiment that had been echoed by Editorial developer Ole Zorn last week.

An example of storage provider extension, from Apple's developer session.

An example of storage provider extension, from Apple's developer session.

Storage provider extensions aren’t just a solution for small independent studios looking to integrate cloud services with their apps, though. Larger companies like Box and Microsoft – and possibly others such as Dropbox, Google, and Amazon – will find tools in iOS 8 to let their storage services coexist with Apple’s iCloud in the same document picker available in every app.

“With extensibility in iOS 8, we’re excited our users will have the ability to seamlessly access all of their OneDrive files right from within their favorite apps, eliminating the need to create additional copies”, Angus Logan, Head of Product Management and Product Marketing of OneDrive at Microsoft, told me in a statement over email. OneDrive was featured by Apple on stage in a screenshot showing third-party document providers for iOS 8, and it’s been included in Apple’s Enterprise webpage for the new OS.

At this point, it’s unclear whether storage providers will cause a major shift from embedded SDKs, but developers I’ve talked to are looking forward to start testing the combination of document picker, new file operations, and storage extensions for iOS 8. “We’re processing WWDC news ourselves”, Denys Zhadanov, marketing director at Readdle, told me, noting that Apple’s announcements are going to be “big for document management and file storage apps”. In Readdle’s case, iOS 8 would make it possible for PDF Expert to continue editing the same PDF document that a user started annotating with Preview on OS X, or to work with OneDrive and Box through extensions.

Apple wants to provide a common ground for third-parties to offer extensibility features based on a consistent interface; this new approach may not have all the advanced functionalities of a full SDK and it will be subject to Apple’s design choices, but it should be more than enough for apps that want to offer storage options besides iCloud. According to Pedroso, "storage extensions are a way for document-based apps, like Byword, to support a bunch of storage/syncing service without investing resources into custom-made integrations".

“It’s great to shift storage development into the hands of those best positioned to do it”, Scown concluded.

A Better Experience with New Complexities

Document-based apps are going to face major changes with the new technologies Apple has introduced in iOS 8, and the company is advising developers to carefully consider the complexities that will arise from document sharing, extensions, and new file operations.

With the ability to open the same document across multiple apps, users will be able to process files with a better, more intuitive experience, but developers will have to account for new variables such as networking errors, background saving, and the overall challenge of working on the same file with multiple apps. Apple is telling developers to use the file coordination technology, which can coordinate access to a file from multiple locations and avoid data loss. The company believes that users will demand the more flexible Open and Move modes, which will require developers to be careful with the changes their apps make to a file.

Developers will also need to handle data migration properly: while iOS 8 users will be able to choose to continue using Documents in the Cloud or migrate to iCloud Drive, OS X Yosemite won’t support backwards compatibility and users will have to choose to migrate to iCloud Drive or documents will no longer update across devices. Pedroso noted that "the new APIs break in so many ways with what exists in iOS 7 and OS X Mavericks, that they will not be compatible."; for users, he added, this means that they "will have to make the decision to migrate on each device. Once data has been migrated on one device, all other devices must also be migrated otherwise they will stop syncing with each other. This is bound to generate some confusion and will require great communication between developers and users".

Similarly, it’ll be up to developers to include new file operations in specific parts of their apps; and, with a system capable of sharing the same document across apps, displaying the original source of a file will also be a factor at play.

Filesystem challenges and developer questions aside, Apple’s new approach to document management and storage in iOS 8 is a welcome step forward. These new technologies have likely been years in the making to ensure reliability and security while avoiding data loss, and they should put an end to the frustratingly complex (but done in good faith) system of the old iOS days.

With iCloud Drive, document sharing, a new document picker, and storage providers from third-party services, Apple is betting on a solution that combines aspects of traditional document organization with a more versatile sandboxing model designed for security and collaboration. It’s too early to tell whether the company may have finally solved inter-app document management on iOS, but – just like extensions – developers are excited, and I’m optimistic.

Unlock MacStories Extras

Club MacStories offers exclusive access to extra MacStories content, delivered every week; it’s also a way to support us directly.

Club MacStories will help you discover the best apps for your devices and get the most out of your iPhone, iPad, and Mac. Plus, it’s made in Italy.

Starting at $5/month, with an annual option available. Join the Club.

A Club MacStories membership includes:

  • MacStories Weekly newsletter, delivered every week on Friday with app collections, tips, iOS workflows, and more;
  • Monthly Log newsletter, delivered once every month with behind-the-scenes stories, app notes, personal journals, and more;
  • Access to occasional giveaways, discounts, and free downloads.