The supposed black box
On a reasonable number of the systems that I’ve worked on over the past few years there’s been a 'black box' component which the team I’ve been on has needed to integrate with.
I’ve always found it a little strange that you wouldn’t need to/want to know how that part of the system worked or that you could actually believe that it was truly a black box.
If it doesn’t work then you have no way of diagnosing the problem - did you do something wrong, was there something wrong inside the black box or was it something else.
On a project I worked on a few years ago the reason for the black box thinking was that each layer was being developed by people from a different company.
The problem we had was that we were working on the top layer, the one that was visible to the end user and therefore our progress was very visible to the stakeholders who were paying for the product to be built.
We therefore had no choice but to go into the metaphorical black box and try and gather as much information as we could to pass on to the teams working on the other layers so that they would be able to help us better.
I recently watched a talk by Artur Bergman titled 'Full Stack Awareness' where he talks about the necessity of understanding exactly what is happening when our code gets executed rather than thinking of it as magic.
Although Artur is working in a different context to most application developers who maybe don’t need to know the stack as well as he does I think the advice about treating something as magic is useful.
If we think of something as a 'black box' then effectively we are saying that it’s somewhat magic.
If the integrated component is being custom written then I think the team who needs to integrate with it should at the very least have someone who knows how it works very well so they can diagnose any problems quickly.
That person then needs to spread their knowledge amongst the rest of the team so that they don’t end up being the bottle neck.
In summary I think the term 'black box' is frequently a misnomer and we’ll rarely be able to view said black box in such an opaque way.
About the author
I'm currently working on short form content at ClickHouse. I publish short 5 minute videos showing how to solve data problems on YouTube @LearnDataWithMark. I previously worked on graph analytics at Neo4j, where I also co-authored the O'Reilly Graph Algorithms Book with Amy Hodler.