We Were Drowning in Feedback, So We Stopped Collecting It
There’s a message in one of our Slack channels from a few months back that just says “too many steps.” That’s the whole thing. Not which product, not which flow, not what they were trying to do when the steps got to be too many. And it’s internal, so this wasn’t some anonymous user I’d never track down. I knew exactly whose name was on it. I could have slacked them. Instead it just sat there, and someone had reacted to it with a thumbs up, which somehow made it worse. That message is a perfect little monument to how we used to handle feedback, which is to say we didn’t. We collected it.
And we were very good at collecting it. It came in from everywhere. A DM to whoever someone guessed owned the thing. A comment buried in a standup that never made it anywhere else. A hallway “oh, can you make it so that” that died the second the conversation moved on. A three-paragraph message in a product channel, right next to a screenshot pasted at 9pm with the word “lol.” It arrived in every possible shape. One person writes an essay about a workflow. The next writes “hate the new button.” All of it real, all of it landing in different places, none of it in a form anyone could actually do anything with.
Lots of feedback is a good sign. I still believe that. A product nobody bothers to complain about is usually a product nobody cares about, and I’ll take our firehose over silence any day. So the volume was never the problem. The problem was that having a lot of feedback and being able to do something with it are two entirely different skills, and we had exactly one of them. It came in faster than anyone could read, in a dozen shapes that didn’t line up against each other. You can’t weigh “hate the new button” against a careful paragraph about a workflow and decide which one matters more, because they aren’t even the same kind of thing. So the pile grew, and the honest question quietly shifted from “what did they say” to “where do I even start,” which is a question you can sit inside comfortably for months. The feedback was a good sign. What we were doing with it was not listening. It was storage.
So we decided to build something. And before I get into what, I should be honest that the first reasonable question anyone asked was: why are you building this. There are tools that do this. You can buy feedback platforms today that ingest, tag, and dashboard the whole thing right off the shelf, warranty included. We looked at them. Some of them are good. And we decided to build our own anyway, which I still turn over in my head at least once a week.
Here’s the analogy I keep landing on. Buying the tool is buying a car from the dealership. It has airbags, a warranty, a hundred features you’ll never touch, and it works the day you drive it off the lot. Building your own is building the car in your garage. It has exactly the three things you need and nothing you don’t, and a real, non-trivial chance that one of the boring parts you skipped turns out to be load-bearing. You find that out on the highway, not in the showroom. Every time I feel good about what we’ve built, a small voice asks whether I’m doing 70 in something missing its airbags.
But here’s what actually changed my mind on build versus buy, and it wasn’t features. The bought tools aren’t missing features. They have plenty. What you really hand over when you buy is the definitions. The vendor decides what counts as a piece of feedback, how it gets categorized, what even registers as a trend, what a “customer” is. Those assumptions feel like details right up until you’re living inside them for five years, thinking about your own users in a vocabulary someone else picked. We didn’t build to get more. We built to keep the definitions ours, and to be able to change them on a Tuesday when we learn something, instead of filing a feature request and waiting a quarter. The price of that is the airbags. That’s the actual trade, and it was never features versus features. It’s whose assumptions you’re willing to live inside.
Okay. What we built.
The first piece is the least flashy and the one I’m most convinced by. Instead of a form, the intake is a conversation, and it happens in Slack, because that’s already where everyone is. Giving feedback costs nobody a context switch, no new tab, no portal login. When someone has feedback, they talk to it right there, and it asks the boring follow-up questions a human reviewer would’ve had to ask three days later. Which product. What were you trying to do. What did you expect to happen instead. Is this blocking you or just annoying you. By the time the feedback is submitted, it’s already structured, already has the context attached, already usable.
The reason this matters is subtle, and it’s the part I’d defend hardest. A form pushes all the interpretation work downstream, onto product, days later, with none of the context the submitter had in the moment. So it never happens, and you get “this is broken” with a thumbs up. Moving to a conversation shifts that work to the one moment someone actually knows what they meant. You’re not making product do archaeology anymore. You’re catching the context while it’s still warm.
Get Steven Rai’s stories in your inbox
Join Medium for free to get updates from this writer.
Slack is just where it happens to live right now. There’s no real reason it has to stay there. The same conversation could run over SMS, or a call, or an in-app chat the day we point it at real users instead of only ourselves. The one channel that makes me laugh is email, because the whole thing runs on back and forth, on the follow-up question landing while you still remember what you meant, and email is where back and forth goes to die. You’d send “too many steps,” get a reply asking which flow, and answer it four days later from a parking lot. So, most places, eventually. Email can wait.
The second piece is the running read. All that structured feedback flows into one place, sorted by product, by category, by client, by whatever we need, and it keeps a live answer to the question we could never answer before: what is the product actually going through right now. Not the quarterly rollup. This week. And yes, you can talk to it in plain language, ask what’s trending in billing, or which clients keep raising the same thing, and it’ll tell you. That part demos well and I almost don’t want to lead with it, because the querying isn’t the point. The point is that for the first time there’s a true, current answer sitting there to be queried at all.
I want to be careful here, because a dashboard is a dangerous object. It’s one clever question away from becoming a machine for confirming whatever you already wanted to build. Ask it the right leading question and it’ll hand you a trend that justifies the roadmap you’d already drawn. The scary part was never the volume of feedback. It’s a tool that makes your existing bias feel like evidence. We’re building it, and I still keep one eye on that, because “the data says” is the most expensive lie a product team can tell itself.
Which brings me to the part we haven’t earned yet, and the real reason we’re building any of this.
Right now it organizes feedback and tells us what’s happening. The actual dream is that it tells us what to do about it. Not in a fully automated, fire-the-PMs way. In a “here’s a sorted starting point” way. It looks at the feedback coming in, at what we’ve actually shipped recently, and at what we’ve approved and rejected before, and it pre-sorts the pile into three buckets. Things we should do. Things we should do, but later. And things we should not do, with the reason attached. That last bucket is the one I want most, because “no, and here’s why we already decided that” is the answer product teams almost never have written down anywhere.
And this is the part a bought tool structurally cannot do, no matter how many airbags it has. An off-the-shelf platform doesn’t know our rejected-feedback history. It doesn’t know our roadmap, or that we said no to this exact request eight months ago for a reason that still holds. That context is the whole game, and it only lives here, in the thing we built, in the definitions we kept for ourselves. The car we’re building in the garage is the only one that can ever learn to drive our roads.
We’re not there yet. Today the AI turns “too many steps” 👍 into structured, usable feedback the moment someone types it, keeps an honest running read on what the products are going through, and gives us a real place to start instead of a pile to feel guilty about. The prioritization brain is still ahead of us. We might still find out we skipped an airbag.
But we know which car we’re in, and we chose it on purpose. It has exactly the parts we need, we can add the next one ourselves the week we understand it, and it’s learning roads no dealership model has ever driven. I’d rather finish that car than drive someone else’s and spend five years pretending their assumptions are mine.
The “too many steps” message is still sitting in that channel. If that same person typed those three words today, they’d get asked which product, which flow, what they were actually trying to do, and thirty seconds later we’d have something we could use instead of a shrug and a thumbs up. That’s the part I care about. Not the dashboard, not the dream, as much as I want both of them. Three words that used to die in a channel now go somewhere the second they land. We spent years calling that listening. It finally is.
