The most powerful tool you aren't using

Your intuition is your best friend

We data scientists tend to be incredibly logic-oriented. That's what makes intuition frustrating to accept, but all the more important.

You may have found yourself in a situation where a non-technical friend or stakeholder makes a bold claim about something they believe to be true.

If you disagreed with it and pulled some data to show it, you might counter with "well, numbers don't lie" or something of the sort. After all, shouldn't we back up our claims with objective facts?

This is just one example where we tend to dismiss intuition.

But, here are 2 reasons why you should give intuition a fair shake:

You don't always have the right data

Successful business owners know this to be true. There are two worlds in business:

1. The world experienced by your users

2. The world captured in your data

Ideally, the Venn diagram of these two worlds is a circle. Perfect overlap.

This is (basically) never the case, though.

Even Jeff Bezos agrees:

Morgan Housel @morganhousel

"The thing I have noticed is when the anecdotes and the data disagree, the anecdotes are usually right. There's something wrong with the way you are measuring it." -- Bezos. A quote I love and hate in equal parts.

Morgan captures the frustration we all feel.

But, Bezos is certainly right. Even when the data appears to paint a clear picture. It is vitally important to consider all potential gaps.

Maybe people stopped buying that item because a new store associate started putting it on the wrong shelf accidentally.

Maybe your advertisement only looked like it performed well because you got an unrelated shoutout from a celebrity on the same day.

Data is never complete.

Infinite problems and infinite solutions

In any reasonably complex system (i.e. most things in the real world), there are more problems to solve than there is time remaining in the universe.

And, for each of those problems, there are probably an endless number of ways you could solve it.

Great. So how do we figure out what problems we should prioritize?

Your logical brain might say: solve the problems that generate the most value. I mean, duh, I guess?

But once you think about it for a second, you realize that means you have to calculate the potential value of each of these potential problems. That is extremely time-consuming and, in some cases, not even possible.

Take one domain expert and pose them the same question, though, and they'll probably give you a pretty accurate list. They're in the trenches with the problem every day. They have a 360°, 4K high definition view of the problem. Trust them. Even if they aren't completely right, it's probably the best possible starting point you can get.

So now you picked the problem to solve, and you want to get after it.

Great! So how do you solve it?

We're right back where we started -- infinite possibilities.

This is where your intuition comes in. For an ML problem, you don't build every possible model and see what works, you pick the models that are the most likely to work and you start there.

That's all for this issue!

If you liked it, the best way you could thank me is by sharing it with one person who you think would benefit from it.

Until next time,

- Mark