Big Data, and Why You Should Be My Friend

This blog post is based on a presentation given by Dr. Alicia Powers, a data scientist at Fino, at the Strata Conference in Barcelona.

I am a persistent outlier: Throughout my life, I have found myself in environments where I am different from those around me across one or two dimensions—or more. This life experience and my background as a data scientist have sparked my interest in finding connections where people often don’t think connections exist.

Information Sharing

Big Data has introduced new opportunities for people to find different ways to connect. Whether through social media activity or purchase history, people share information about their beliefs, values, preferences, and opinions every day. Companies are working to gather and use this information to meet our needs as consumers. They do this by making products, promoting messages, or presenting ads that try to build on their understanding of who we are and what we know.

The problem with these efforts is that when people are pigeonholed into general demographic models, it becomes more difficult to accurately and meaningfully understand them as individuals. The pressing question then becomes: How does Big Data deal with people on a personal level? How does it respond to people who don’t think, behave, or believe in the same way as the majority of the people around them?

Is Big Data Personal?

Big Data tends to focus on predicting how the majority of people in a given audience will behave. Not much attention, however, is paid by companies to the people who represent the minority views or opinions in a group—the people who are outliers.

Most often, companies want to know if people will vote for something, buy something, or tell a friend about something. Discovering the numbers behind these three factors fuels a lot of the predictive efforts in Big Data. But these actions—voting, buying, and sharing—are highly personal and depend on deeply individual factors like values and beliefs.

In order to build a model that accommodates outliers, businesses need to know more about the ideologies specific to individual people—about how personally willing different people are to vote, buy, or share. Businesses have to do this because predictive models based on patterns observed in a general population can rarely predict what everyone in that population will do.

Big Data’s Missed Opportunities

If businesses don’t analyze their outliers, they’re likely missing out many opportunities to improve their predictive models by ignoring the people who, by definition, are thinking outside of the box. Without looking closely at outliers, companies forfeit greater and more granular insights about what makes different people function in the marketplace and in society.

While there is a general benefit to gain from understanding the greater population, there is even deeper value in discovering what drives outliers. Companies should strive to understand outliers precisely because they are outliers: It is within the lesser-known, smaller subset of the general population that innovators, trendsetters, and early adopters are usually found. Businesses should want to appeal to this group because it can set the tone for what’s new and interesting for the rest of the population. And if businesses want to appeal to outliers, they have to invest resources in understanding why people are outliers. What sets those in the minority apart from the ‘norm’?

Being an Outlier

When you’re around people who are different from you, there’s an opportunity to create instant community if you can find a connection. One example from my personal life as an outlier occurred during a trip to Peru. I didn’t speak the language or know anyone in the country, but I discovered an unexpected commonality between me and the people of Peru: People there are shorter than the average American—and so am I. This simple commonality helped me feel a sense of community with the people around me.

This same concept of connection can also be found in the different moral and social values people hold around world. Previously, I’ve done some analysis on the World Values Survey, which is a study of over 70,000 people from across the globe to gain insight into the principles that reign supreme in different areas and cultures. By looking at these values and comparing them with my own, I discovered that on the topic of raising children, I was more likely to agree with a Saudi Arabian man than with any other person in the world. As it turns out, Saudi Arabian men and I share belief in the core values of obedience, work ethic, imagination, and tolerance.

This was a truly unexpected connection made visible because I took time to examine the world around me in a multi-dimensional way, taking a composite view of my values and others’ values instead of judging without context. When you look at people multi-dimensionally, as I did in Peru and with the World Values Survey, formerly invisible connections between people begin to appear.

Case Study: Sample Size of One

As a data scientist, I have to provide a certain big disclaimer in telling my personal story of being an outlier: I am a sample size of one. But with that said, it’s important to recognize that we are all, in our own ways, a sample size of one. All of us are individuals with individual stories, principles, and motivations.

For that reason, Big Data can absolutely help shed light on the patterns in what we do as individuals, both in our own lives and in greater society. If observed and considered appropriately, Big Data has the power to help us connect with each other on a personal level.

So, why should you be my friend? Any two people can form a connection, so long as they give enough time and energy to finding it and making it. It’s powerful when people connect, because there is always power in forging a relationship in the hearth of discovered common ground.

Three Things to Remember When Working With Outliers

1. You need to believe there is a connection. There are many preconceived notions about what works and what doesn’t work within a given demographic based on general themes or patterns that have been observed. If you don’t believe there is a connection to be found, you will not find one. An open mind is necessary.
2. You need to expect the unexpected. If you think you know the answers, act as if you don’t. That’s because you probably don’t know the answers until you dig further.
3. You need to remember that people are multi-dimensional. The more you can remember that everyone cannot be easily classified into one group or category—defined by one catch-all identifier—the more likely it is that you will find powerful insights. You must analyze your data in a way that allows people to be understood in many different ways.

There Is Power in Connection

Connection is the foundation of relationships. Whether it’s the relationship that grows between friends, the relationship that supports work with a client, or the relationship that (hopefully) forms between customers and a product—if you can find and strengthen a connection, you can create something wonderful.