Louie Mantia (former designer at Apple and Square, among others) has a thoughtful take on designing app icons:
Consider walking down a candy aisle at a drugstore. If you have a craving for Reese’s Peanut Butter Cups, you know how to find it. Of course, it’s the bright orange wrapper with a bubbly yellow word on it. Instant. Most candy bars have very distinct brands which can be instantly recognizable at a glance of the entire aisle. This should be precisely how you approach your app icon design.
An ideal app icon reminds users of an app through shape, color, and texture. Take the Phone icon, for instance. It has evolved a tiny bit since Steve introduced it in 2007, but has remained mostly the same. Today, it is a bright green, with diagonal stripes, and a white phone symbol. These three elements together (read: by our powers combined) form an instantly recognizable mark for users. When a user needs to use the phone, it’s a no-brainer.
I may have argued in the past that the iOS Home screen needs improvements, but I would never argue against a well designed icon. I use Alfred on my Mac to launch apps, but I also rely on the dock to quickly reach out to an app and make it visible. On iOS, I rely less on Spotlight (though I use it regularly) and I value the "recognizability" of icons in the Home screen.
Read Louie's post about app icons here. His suggestions for new Zappos, Amazon, and Apple icons look great.