Mind the Gap
Testing is easier than ever. We still need time to understand “what” we learn.
Mind the gap between the train and the station. We hear every time a train stops at a station in the London subway system. It is repeated every single time. We would think everyone already listened to the message, they know it and they don’t need to listen to it again. But the voice is still there. Reminding us what is important.
AI and the paradigm shift
AI has burst into product teams without asking permission, and it’s now everywhere in the process. We no longer need weeks to design, prototype, build and deliver, only hours. (A few weeks ago, I saw an API migration take 16 hours instead of 9 months!) But “with great power comes great responsibility.” Or, in this case, with great speed comes a subtle risk: when errors cost almost nothing, we may also spend less time learning from them.
More and more frequently, we can notice that, with AI, laboratory research won’t be indispensable anymore. We can test in prod, compare variants and make decisions based on real data and real users.
It makes sense. And there we go. Zero time to market promise is finally here. Organizations that don’t understand this technology as a game changer, will be left behind sooner rather than later.
However, there is something that I believe we must protect: the TIME to understand what and why we are building, and what we are doing.
The gap before and the gap after
Tests aren’t the same as learning. An A/B test can tell you which variant wins but not why. Without that, we can’t translate that learning into the next move. Without that we are blind, accumulating small victories (or defeats) that don’t build a north star.
Get Mariano Iacobacci’s stories in your inbox
Join Medium for free to get updates from this writer.
When building is almost free, the temptation of building everything could affect us. There is a risk of delivering meaningless features: “use a sledgehammer to crack a nut”. Before trying, we need to understand which problem we are solving and what we expect to happen. This is not new, but still.
We can carry out tests easier than ever. But if we aren’t taking a moment to analyze what went well and what went wrong, we won’t be able to know how to improve. Launching something fast without reflection isn’t speed, it is noise.
I can give you an example: a while ago, I was able to see a new feature. Travel cancellation insurance for bus tickets. The statement was clear. Bookings will include cancellation insurance without any charge. This product was free for all in-app bookings. App acquisition cost was the metric behind this idea. Any contact with users, review of recording sessions were made.
Results don’t show any change in number of users or any other significant change.
There was no gap at all, either before or after, to understand what doesn’t work. It seemed easy just to connect the insurance system to start offering this benefit for free. There was no learning. It just didn’t work. It is not the best scenario if we want to move forward because there isn’t space to understand what doesn’t work. Was the product? Communication? The return process? Important questions weren’t answered. The team skips rapidly to the next feature without even breathing. They went into execution mode without looking where they put their feet. They didn’t mind the gap.
Mind the gap
There is no need to have a pre-settle time for that gap. We are not talking about weeks for research any more before every launching: time to understand could take a week or just a few hours. In any case, it is still necessary.
In practice, it could be as simple as writing hypotheses before launching. Make a short debrief after every test or take some time with the team to talk and share perspectives. Document what we learn and make it as a team. In a time where being an individual contributor is fashionable, meeting as a team to discuss ideas can be incredibly original.
In London’s underground “mind the gap” isn’t a warning to stop commuting. It is a reminder not to fall off in the attempt. The same happens with AI: If speed is the train, the gap is the space between it and the platform: easy to overlook, and exactly where we trip. It’s not about slowing down. It is about knowing where we are going.
