Intuition and 'quit thinking and look'
Something which Dermot, Christian and I noticed last week is that on our project we’ve reached the stage where we intuitively know what the underlying problem is for any given error message in the application we’re working on.
We’re pretty much at the stage where we’re effectively pattern matching what’s going on without needing to think that much anymore.
This is a good thing because it saves a lot of time analysing every single message to try and work out what’s going on - I think this means that we’ve reached a higher level of the Dreyfus model when it comes to this particular situation.
The problem with getting too used to this approach is that eventually we’ll come across a problem that we haven’t come across before and if we don’t recognise that this is the case we’ll end up getting very frustrated.
I had the chance to work with Ade Oshineye a few years ago and he always encouraged me to "quit thinking and look" when it came to problem solving.
This idea is derived from a book titled '9 indispensable rules of debugging' and the thinking behind it is that we often go straight to the solution for a problem without spending the time to understand what the problem actually is.
We actually came across a situation like this recently while investigating a problem in production.
We were getting an exception which looked pretty similar to other problems that we’d seen previously so we immediately tried that solution without any further investigation.
It had no impact at all so we had to go back and actually look at the error message we were receiving before trying something else.
As it turned out the time we wasted picking the wrong solution was less than 30 seconds but I think we had got to the point where a bit of complacency had crept in and we believed that there weren’t any ways the application could go wrong that we hadn’t seen before.
Talking further about this with Dermot he pointed out that this situation was akin to discovering an unknown unknown i.e. we came across a problem that we didn’t know we didn’t know about since we thought we knew about all of them!
Errol Morris wrote an interesting article about this recently in the New York Times in which referenced Dunning and Kruger’s paper 'Unskilled and Unaware of It: How Difficulties of Recognizing One’s Own Incompetence Lead to Inflated Self-assessments'.
The learning for me from this is that while intuition is very useful it’s also important to at least be aware that we probably don’t know everything and that we may well come across new situations and will have to approach them as a novice again.
About the author
I'm currently working on short form content at ClickHouse. I publish short 5 minute videos showing how to solve data problems on YouTube @LearnDataWithMark. I previously worked on graph analytics at Neo4j, where I also co-authored the O'Reilly Graph Algorithms Book with Amy Hodler.