The human side of AI: The role of research at Expedia Group

Expedia Group Research Summit 2019

It’s hard to believe that any company involved in technology isn’t just run by robots and AI by now. It’s all we ever hear or read about anyway. So it was a very refreshing turn of events when I attended the inaugural Expedia Group Research Summit 2019. Far from just talk of big data and how some machine learning algorithm is just harvesting and churning out analytics, Expedia Group—which comprises travel brands such as Expedia, Trivago, Orbitz, Vrbo and Hotwire among others—employ human research teams that run human research and experiments. Sounds ominous, but it's not.

One of the better known examples is Expedia’s Innovation Lab where researchers, led by Senior Director of User Research, Tammy Snow, use real-time eye and emotion tracking technologies learn about how real people use their apps and websites. And that feedback helps them optimise the experience. 

Now, this particular Research Summit was something of an internal team bonding and brainstorming pow wow between 70 researchers across Expedia Group’s brands. It wasn’t exactly a press event. They weren’t launching any new products or had a particular announcement to make. The media, like myself, were invited to be privy to some insights on how human research actually help them build better products, and truthfully, the stories shared were things I never really thought about before.

Building better AI through customer research

Take a Hotels.com booking for example. Regular old machine learning algorithms might be able to learn your preferences over time and serve up images that feel more important to you, but with the addition of research data, they’ve been able to expand to providing hyper-local information that address specific customer pain points that machine learning alone wouldn’t have understood. Joss Crossick, then VP of Product Design said that their research showed it was important for Japanese travellers to know if the Japanese language was spoken at the front desk; in Germany, breakfast choices were important. This research data helps machine algorithms better determine what is the right piece of information to share with the right people at the right time to make the booking process more relevant and personal.

The idea shared here is that personalisation should not just be rule-based. You don’t tick a bunch of boxes and get a generic set of options every single time and then the experience ends there. People from different countries have different ideas and preconceptions about a location they’re visiting, people who book a second trip to the same location would be looking for different things to do, so they might tick the same boxes, but they don’t want to see the same results, and so on.

Research adds a level of understanding as to what goes on in a customer’s mind besides just floating up the cheapest room.

Sometimes, it’s the story that sells

Karl Steiner, a global research lead at Vrbo shared some of their research stories. Vrbo (pronounced “verbo”) is a vacation home rental service similar to Airbnb. Now, we’re all used to the gig economy. Why do we rent out our homes? To make extra money, of course. It’s simple, but what really drives people? Researchers found a variety of reasons from those seeking to connect with travellers to those trying to maintain a family property, and these insights help them carve the narrative they use to sell their product.

“The fact that our owners have a variety of motivations wasn't that big a surprise, but the depth of some of these particular insights that we gained, and how well our organisation took them off and incorporate that into our thinking about how do we delight these Vrbos? That was a nice surprise.”

Another story comes from Rachael Wussow, a Product Lab and Research Manager at Hotwire. Hotwire is a booking site under Brand Expedia Group that offers opaque hotel bookings, where you get guaranteed lower prices, but won’t know the exact hotel until after the booking is finalised. This might seem gimmicky right? But according to Rachael, the research they’ve done overwhelmingly shows that people are saying that there is a time saving proposition as well.

“Because there is less detail, and there's less to wade through, this is actually a more efficient way to book. And secondly, we are enabling a way for them to discover hotels they wouldn't have thought to even look into otherwise.”

There were other research stories and insights shared, such as how virtual property tours aren’t just a fancy feature upgrade to picture galleries, but actually help contribute to an increased trust factor for customers and possibly improved book-through rates. With pictures, property owners can hide things they don’t want a customer to see, which can negatively impact experience.

Lab rats are people too

After a day listening to research stories, one can’t help but start to question the validity of such research. I mean it’s all qualitative data. How often have you agreed to fill out a survey and just ticked the all the ‘good’ boxes? And if they feed that data into their AI or machine learning system, won’t that then create biased results?

According to Tammy Snow, the vast majority of researchers at Expedia Group have backgrounds in cognitive or behavioural sciences, and are well versed in understanding the different biases that might be introduced by interacting with people.

“There's a thing called the Hawthorne effect, where people behave differently when they know they're being observed. So we have to keep that in mind. And we do try to temper the way we interpret our results through the lens of understanding that we have introduced some bias just by engaging with people.”

“We actually we utilise methods that allow us for the most part to avoid introducing bias; into filtering a lot of that bias out. You would be surprised, especially with really skilled researchers, when we bring people in, they forget sometimes that they're in our lab. We also have met this when we go into the field and observe people in environments where they feel most naturally. We ask them to use the tools and sites and apps that they normally use, and there are times when they actually forget somebody’s there watching them.”

Chris Matthews, a research lead at Hotels.com added that while they are trying to elicit responses, a big part of research is also based on observed behaviour.

“It isn't about just asking, do you like this? Do you dislike that? We're interested in finding out how people use things. And then, as they encounter difficulties, or if they find it actually works quite well for them, getting a sense of the ‘why’. We're looking at the relevance of information, or the usefulness of content or tools. So it's not really about, hey, do you think you'd use this in the future? It's about how are you using it now, and how can we learn from it.”

Rachael Wussow (from Hotwire) continues, “I think we're constantly trying to link multiple insights. So we rarely take one gospel and move forward. We're adding it to a body of work, we're looking for themes, we're trying to, assess different methods to make sure we're seeing things correctly. We're sipping off of many angles to make sure that we're getting to the core behind people's intended behaviour, and it’s not just that they're telling us what we want to hear.”

I’ll end off paraphrasing from the CEO of Expedia Group, Mark Okerstrom on building AI with human oversight.

“Good AI starts with good H.I. - human intelligence, and you really need to understand what is driving your algorithms? How these algorithms interact with each other, what behaviour changes they're driving in your ecosystem, whether it's pricing recommendations, or display recommendations, etc. And you have to create feedback loops that humans can look at and say, ‘Does this make sense?’

I think that's the most the most important thing, because what AI does, is that it just creates this way to get massive amounts of information plugged into models and a model can give you action, so we still need humans to say, ‘Is that data set complete? Is that data set accurate? Do these actions that this model is directed us to do make sense for our business, for our customers or partners?’”