Small Data Inspiration
I'm thrilled to share a critical thinking post about Big Data that really came to the forefront of my consciousness while I, with great pleasure, read Martin Lindstrom's new book "Small Data" this week (published Feb 2013 - ISBN 9781250080684). My husband bought it for me at the airport when he was coming back from a business trip - and this certainly answers the question, "What do I buy my Data Scientist wife/husband/significant other?".
Martin hits, right on the head, one of the critical oversights of the masses in today's rush toward the Mecca of "Big Data" -- touted as the cure for everything from cancer to team motivation. Big Data is certainly helping us solve some interesting modern problems and shed light on many mysteries, but it isn't all rainbows and unicorns! It doesn't instantly manifest in more money for businesses, make your customers happier, or guarantee that you'll sell more of the ugly yellow sweaters you ordered last spring. Specifically, it does not help you get a better understanding of Customer X - only of the average customer. And the average customer might not be good enough to help you achieve your business goals. The book doesn't bash Big Data, but it legitimately points out how the signal can be easily lost in the noise of Big Data, and (I think most importantly) that we have to remember how unique, valuable, and original each human's experience is. Especially when it comes to optimizing human experiences for sales and marketing goals, it is important to remember that it is individuals who make buying decisions and not averages, clusters, or segments of populations.
I found it fascinating how insightful the connections were that Martin made between his seemingly disconnected and unrelated experiences and jobs. I also have the distinct impression that Martin is an extremely keen observer as well as an astute generalizer. These sound like opposite qualities - but he has mastered the art (and ultimately the underlying science) of observing rare events and generalizing them to greater populations to make conclusions which are proven by his successes. In fact, a lot of his experiences and observations align eerily with some of my nerdier jokes about my data science experiences. I tell people at parties (etc.) when asked "what does a data scientist do" that I "predict the appearance of potholes on your street by the number of cats you own". I have a feeling if I told Martin this he would get exactly what I mean: he processes information about people and the human experience using his numerous experiences, jobs, and keen observation skills to come up with a contextual hypothesis and test it. I do the same with data, ultimately observing it from different angles, slicing and dicing it, getting a feeling for its shape, size, and behavior in order to come up with a contextual hypothesis and test it. I can't just take "data" and "find something interesting" any more than Martin can just observe people in an elevator and come up with that next great idea for a store - he goes in with an idea of what the problem is, and what his customer wants. I emphatically insist on the same when working with data.
I think that less involved people might easily insist that data is unbiased, unfeeling, absolute -- after all, it's just "numbers", right? However, someone had to collect those numbers, or program an item to collect those numbers. People are a sum of their experiences, and as a curating person was likely very close to the data, their data collection is itself skewed, colored, and otherwise biased in a myriad of ways. It might not matter or be significant to the analysis at hand, but it might also very well skew the analysis and results -- with unfortunate consequence. Martin points out how when he is too close to the situation he cannot produce a good analysis or come up with associations because he is too comfortable. This is no different than a scientist being too close to their data to see the rare signals, the skew, or other important events. It is the main reason I especially love to move from project to project: I am excellent at looking at new and unfamiliar data and taking that distanced look to find connections and patterns that might not be visible to the person already highly familiar and comfortable with the data or its collection.
Perhaps the field of Data Science would be well associated with the field of anthropology? My information scientist husband would certainly insist such.
As a final tidbit from the book I was especially intrigued by Martin's thought that as humans we will always seek balance between fast and slow, observation and ignorance, comfort and discomfort, noise and quiet, etc. I spent a lot of time thinking about what that said about myself and the people I know, and the world in general. Perhaps that is why I always see the world as the swing of a pendulum, constantly overcorrecting but always centered around the bottom of the swing. It sure is an interesting, and complex, world we live in. And I strongly believe that there is no magical algorithm or math formula that explains our existence as human ... yet. This is the myth of "Big Data": that somehow in this huge agglomeration of numbers there must be the human "pattern". Well, I don't buy it - and it seems neither does Martin.
I heartily recommend that if you want a good intellectual and thought-provoking read then you should pick up this book. Don't skim it - read it, enjoy it, and think about it. Afterwards, my challenge to you is to see where you can apply this type of observational inspiration in your life and work.