The web has become a hall of mirrors, filled only with reflections of our data

The “digital assistant” is proliferating, able to combine intelligent natural language processing, voice-operated control over a smartphone’s functions and access to web services. It can set calendar appointments, launch apps, and run requests. But if that sounds very clever – a computerised talking assistant, like HAL9000 from the film 2001: A Space Odyssey – it’s mostly just running search engine queries and processing the results.

Facebook has now joined Apple, Microsoft, Google and Amazon with the launch of its digital assistant M, part of its Messaging smartphone app. It’s special sauce is that M is powered not just by algorithms but by data serfs: human Facebook employees who are there to ensure that every request that it cannot parse is still fulfilled, and in doing so training M by example. That training works because every interaction with M is recorded – that’s the point, according to David Marcus, Facebook’s vice-president of messaging:

We start capturing all of your intent for the things you want to do. Intent often leads to buying something, or to a transaction, and that’s an opportunity for us to [make money] over time.

Facebook, through M, will capture and facilitate that “intent to buy” and take its cut directly from the subsequent purchase rather than as an ad middleman. It does this by leveraging messaging, which was turned into a separate app of its own so that Facebook could integrate PayPal-style peer-to-peer payments between users. This means Facebook has a log not only of your conversations but also your financial dealings. In an interview with Fortune magazine at the time, Facebook product manager, Steve Davies, said:

People talk about money all the time in Messenger but end up going somewhere else to do the transaction. With this, people can finish the conversation the same place started it.

In a somewhat creepy way, by reading your chats and knowing that you’re “talking about money all the time” – what you’re talking about buying – Facebook can build up a pretty compelling profile of interests and potential purchases. If M can capture our intent it will not be by tracking what sites we visit and targeting relevant ads, as per advert brokers such as Google and Doubleclick. Nor by targeting ads based on the links we share, as Twitter does. Instead it simply reads our messages.

‘Hello Dave. Would you like to go shopping?’summer1978/MGM/SKP, CC BY-ND

Talking about money, money talks

M is built to carry out tasks such as booking flights or restaurants or making purchases from online stores, and rather than forcing the user to leave the app in order to visit a web store to complete a purchase, M will bring the store – more specifically, the transaction – to the app.

Suddenly the 64% of smartphone purchases that happen at websites and mobile transactions outside of Facebook, are brought into Facebook. With the opportunity to make suggestions through eavesdropping on conversations, in the not too distant future our talking intelligent assistant might say:

I’m sorry Dave, I heard you talking about buying this camera. I wouldn’t do if I were you Dave: I found a much better deal elsewhere. And I know you’ve been talking about having that tattoo removed. I can recommend someone – she has an offer on right now, and three of your friends have recommended her service. Shall I book you in?

Buying a book from a known supplier may be a low risk purchase, but other services require more discernment. What kind of research about cosmetic surgery has M investigated? Did those three friends use that service, or were they paid to recommend it? Perhaps you’d rather know the follow-up statistics than have a friend’s recommendation.

Still, because of its current position as the dominant social network, Facebook knows more about us, by name, history, social circle, political interests, than any other single internet service. And it’s for this reason that Facebook wants to ensure M is more accurate and versatile than the competition, and why it’s using humans to help the AI interpret interactions and learn. The better digital assistants like M appear to us, the more trust we have in them. Simple tasks performed well builds a willingness to use that service elsewhere – say, recommending financial services, or that cosmetic treatment, which stand to offer Facebook a cut of much more costly purchase.

No such thing as a free lunch

So for Facebook, that’s more users spending more of their time using its services and generating more cash. Where’s the benefit for us?

We’ve been trained to see such services as “free”, but as the saying goes, if you don’t pay for it, then it’s you that’s the product. We’ve seen repeatedly in our Meaningful Consent Project that it’s difficult to evaluate the cost to us when we don’t know what happens to our data.

People were once nervous about how much the state knew of them, with whom they associated and what they do, for fear that if their interests and actions were not aligned with those of the state they might find ourselves detained, disappeared, or disenfranchised. Yet we give exactly this information to corporations without hesitation, because we find ourselves amplified in the exchange: that for each book, film, record or hotel we like there are others who “like” it too.

The web holds a mirror up to us, reflecting back our precise interests and behaviour. Take search, for instance. In the physical world of libraries or bookshops we glance through materials from other topics and different ideas as we hunt down our own query. Indeed we are at our creative best when we absorb the rich variety in our peripheral vision. But online, a search engine shows us only things narrowly related to what we seek. Even the edges of a web page will be filled with targeted ads related to something known to interest us. This narrowing self-reflection has grown ubiquitous online: on social networks we see ourselves relative to our self-selected peers or idols. We create reflections.

The workings of Google, Doubleclick or Facebook reveal these to be two-way mirrors: we are observed through the mirror but see only our reflection, with no way to see the machines observing us. This “free” model is so seductive – it’s all about us – yet it leads us to become absorbed in our phones-as-mirrors rather than the harder challenge of engaging with the world and those around us.

It’s said not to look too closely at how a sausage is made for fear it may put you off. If we saw behind the mirror, would we be put off by the internet? At least most menus carry the choice of more than one dish; the rise of services like M suggests that, despite the apparent wonder of less effortful interactions, the internet menu we’re offered is shrinking.

mc schraefel research research funding from the RCUK. The EPSRC project Meaningful Consent in the Digital Economy (meaningfulconsent.org) is an example of such funding. mc also holds a Royal Academy of Engineering (RAEng) Research Chair co-funded between Microsoft Research and the RAEng. She is also a visiting scientist in the Decentralised Information Group, MIT.