(
Bit nerdy - still great stuff below this, so scroll past if you have to.)Ā The depth of Appleās commitment to AI has started to manifest itself since the summer.
Stephen Levyās essay manifests those efforts in greater detail than before.Ā
We touched on some of these topics in
Exponential View #67 back in June when I wrote: āApple is going to continue its investments in improving UX through technologies in the AI stackā¦ Expect more AI in Appleās products. But I would be surprised to see large-scale open source efforts, of the kind we have seen from Google or Facebook. Open source has rarely been Appleās bag.āĀ
What is new in Levyās piece is more granular details which include the following nuggets.
- Apple runs a neural net locally on the iPhone.
- This neural net weighs 200Mb and trains itself in real-time, but especially overnight, using the GPUs in Appleās iPhone device.
- Apple cites owning the silicon design (from the far-sighted acquisition of PA Semiconductor, I guess) as a driver of improved learning performance.
- They replaced oldskoolĀ voice recognition Ā (hidden markov models) by a deep learning approach back in 2014.Ā
- Apple uses third-party sourced data to generalise training of things like photo recognition. This happens on-device.Ā
One open question raised by the Levy piece is whether Appleās mental model around privacy is a bug or feature when it comes to artificial intelligence. Apple doesnāt share user data. And Appleās global models are built not on this shared user data but on externally & expensively sourced data. And Apple doesnāt seem to send much user data back to the cloud to be learned from on super deep networks running on GPU clusters.Ā
The traditional argument would be that it is a bug. Leveraging data network effectsĀ allows you to build better, more defensible products faster. Teslaās network learning (
EV#31) is a great example of this. As is Facebookās capability in face & object detection. And keeping things on a local GPU denies your neural nets of the value of lots of GPUs (particularly for training).
The counter argument would be that user-privacy may increasingly be a differentiating feature which allows you to sell more stuff. Apple is wealthy and paying for tons of training data doesnāt make a dent in its cash pool. And, in any case, model performance often tends to a limit beyond which additional training data doesnāt help you.Ā
Hereās my fast take on this. Appleās approach to user privacy is may start to look more like a bug than a feature but it may not make a difference right now.Ā
- Externally sourced training data canāt keep up with novel use cases generated by real users. So your external training takes a long time to improve your overall performance. An Apple car training locally will generally have worse training data than a network Tesla whose models draw on edge cases from across the world. Worse performance means a worse product means worse market share meansā¦
- Their introduction of differential privacy which Levy discusses and we alluded to in EV67 suggests they see the value of data network effects and are finding a way to grab that data while staying true to the user privacy promise. What I donāt know is whether differential privacy provides sufficiently good data. Iād recommend reading this essay at High Scalability which looks in more depth at deep learning in Apple Photos and differential privacy.
- Andrew Ng, Baiduās deep learning czar, has pointed out that deep learning performance doesnāt seem to flatten out as you add more data. (EXCELLENT SHORT PIECE) You can just make the network deeper and the model continues to get more performant.Ā
- Consumers wonāt care. ForĀ better or worse they wonāt care enough especially when given the choice of products that feel more āmagicalā.
Right now (and perhaps for the next few years) this probably wonāt hinder Apple. But their approach to user privacy might start to hurt the user experience they can deliver. Now that would be an interesting tension. Ā
Elsewhere
