WebCrawler Magic
My last title at General Magic was Manager of Internet Products. By almost any measure, this was sheer absurdity.
Here’s a lesson I’ve learned in my career: It’s wholly appropriate to champion unpopular ideas, and try to push your company to do things which seem like they’re anathema. You fight the good fight, but in the end, if you fail to convince the company to pivot, you need to give up or leave. If you decide to leave, it doesn’t mean your ideas were wrong; quite the contrary. Sometimes you need to move on, and to let time go by, to have the chance to prove your ideas were right on target.
I was the Macintosh guy at Lotus. I was the IndexingKit and DBKit guy at NeXT. I fought with Steve to do CD-ROM. I became the Web guy at Magic. All of these efforts failed. Seems like we’re talking about failure a lot these days, huh?
I met Brian Pinkerton at NeXT in 1990. He was on what would become a 3-year summer internship, working initially in Avie’s OS group. Along the way, he got interested in what my team was doing with search. I managed to convince Avie to let Brian work part-time with us. I have to say, this is one of the most important things that’s ever happened to me in my career. So thanks, Avie.
Brian worked on the IndexingKit, collaborating with Jack Greenfield. They figured out how to hotwire Jack’s B-Tree into the Mach VM system. A splendid piece of work — it doesn’t get much better than that.
By late 1992, I’d become disillusioned with NeXT’s direction and prospects. Chris MacAskill and some of my other pals had already moved over to General Magic. I did likewise. One footnote on this: I was the first engineer to move over. Steve wasn’t happy, and tried to convince me to stay. Tony coined the verb “Swoo” to describe the situation. As I understand it, Steve gave Andy quite a hard time about it. Nonetheless, as they say, I persisted. I joined Magic in September, 1992.
Enter Rob Wilen. Rob had been a software evangelist at O’Connor, the Chicago-based options trading company. O’Connor was drunk on the NeXT koolaid. I’d met Rob a few times at NeXT developer events.
O’Connor was building, as Mike Slade so famously described them, Mission Critical Custom Apps. They’d figured out that NeXTstep was the most productive application platform on the planet, and that their applications were core to their company’s success.
But if their traders were going to use NeXTstep, they’d need lots of productivity tools to run on the platform. Lotus Improv fit the bill as a spreadsheet, but they needed more. O’Connor connected with Jonathan Schwartz’s company. Jonathan would go on to become CEO of Sun Microsystems, but I digress.
Anyway, all of this wound up amounting to a hill of beans, and Chris was able to lure Rob Wilen to join General Magic, right around the time I joined.
My time at Magic was, well, magical. But sometime in 1994, I realized that the Web was a tsunami that was going to overwhelm our company. That’s when I claimed the ignoble title, Manager of Internet Products.
I spent many months figuring out what I could do to help the company pivot to avoid this disaster. My team built a web browser for Magic Cap, which actually f*ing worked, I’ll have you know. More importantly, we figured out how to front-end Telescript with a web server.
Around this time, Brian (remember Brian?) reappeared. He’d left NeXT after their umpteenth pivot, and returned to the University of Washington to finish his PhD. During this time, he’d implemented WebCrawler, one of the first Web search engines, using the NeXTstep IndexingKit that we’d co-developed years before.
Get Adam Hertz’s stories in your inbox
Join Medium for free to get updates from this writer.
WebCrawler had become not just one of the first, but one of the most important search engines on the web. It was running on a short stack of servers in Brian’s room at U Dub. If he partied too hard or didn’t pay enough attention, it’d go down, and people all over the net would complain.
Brian is one of the smartest people I know. And one proof of this is that he acknowledged his limitations around WebCrawler. He knew he needed to sell it to a company that would support it.
I took it into my head that General Magic should buy WebCrawler. To prime the pump, I briefly hired Brian as a consultant, to help us figure out how to create search agents.
I started to socialize the idea around the company, and predictably ran into a lot of resistance. I remember one conversation where someone said to me, “Why would we buy WebCrawler, when it’s got nothing to do with Telescript?” I replied, “we should by WebCrawler PRECISELY because it has nothing to do with Telescript.”
It became clear to me that a Magic/WebCrawler deal was not going to happen.
Here’s a lesson I’ve learned in my career: It’s wholly appropriate to champion unpopular ideas, and try to push your company to do things which seem like they’re anathema. You fight the good fight, but in the end, if you fail to convince the company to pivot, you need to give up or leave. If you decide to leave, it doesn’t mean your ideas were wrong; quite the contrary. Sometimes you need to move on, and to let time go by, to have the chance to prove your ideas were right on target.
Around this time, John Gianandrea left Magic for Netscape. I remember meeting JG for coffee in downtown Mountain View. I told him I’d managed to front-end Telescript with an HTTP server. He applauded that, but he said it was, “long overdue, and now you have to compete with all the other tools that are being developed for this purpose.”
Indeed.
By this time, Rob Wilen had left Magic, and gone to work at AOL. They were on a buying spree, scooping up every web property they could justify buying. I hosted a New Years Day party at my house in 1996??? I invited Rob and Brian. Shortly thereafter, AOL bought WebCrawler.
Needless to say, I was simultaneously elated and crushed.
A few months later, I gave up on General Magic, along with so many others. Rob and Brian encouraged me to join them at AOL to work on WebCrawler. A few months after I joined, AOL sold WebCrawler to Excite. Rob, Brian and I went along for the ride, launching our web careers.
At the recent premier of the General Magic documentary in New York, I had a chance to reminisce about all of this with Marc Porat. I showed Marc my last Magic business card, with its silly Manager of Internet Products title. By then, I’d assembled all sorts of rationalizations for why Magic could never have bought WebCrawler. The Telescript Alliance first and foremost.
In the moment, Marc was pensive. He told me the real reason, from his point of view. Fatigue. We were all just too tired to move General Magic on to the next thing.
Here’s a lesson I’ve learned in my career: It’s wholly appropriate to champion unpopular ideas, and try to push your company to do things which seem like they’re anathema. You fight the good fight, but in the end, if you fail to convince the company to pivot, you need to give up or leave. If you decide to leave, it doesn’t mean your ideas were wrong; quite the contrary. Sometimes you need to move on, and to let time go by, to have the chance to prove your ideas were right on target.
