A few years ago I got into the habit of going on a long run each Saturday, then cooling down over breakfast to side 4 of the Frampton Comes Alive album. Maybe not the coolest music choice, but it has a live music energy that gets me going. As I began my run on this particular Saturday, I had a few thoughts on my mind.

At Grubhub we have Learning Friday, which is an afternoon per week set aside for team members to learn and innovate. Although some people make good use of this time, many of us let our day-to-day tasks consume it. I had begun wondering how we could use Learning Friday more effectively.

My thoughts about Learning Friday were spurred by something cool I’d witnessed the day before. A developer had approached a product manager with a product idea to help us make progress towards our DAGs goal. “DAGs” is an acronym for “Daily Average Grubs,” which measures how many incremental daily orders we’ll get for a given A/B test winner. DAGs are essentially our currency for tracking conversion improvement. When we implement a winner, Finance tallies the results and banks the DAGs against our annual conversion goal.

I was excited about the engineer’s initiative because we had just set an aggressive new annual conversion goal. The holy grail was to get the larger cross-functional org behind this goal, knowing how much more we can innovate with everyone brainstorming and collaborating. This engineer’s action was a sign we were on the right track, but had plenty of untapped potential.

Anyway, I was well into my run when an idea popped into my head. I was excited enough upon completing my run that I preempted my usual Frampton ritual with a quick exercise to knock out a 1 page “Dagathon” contest proposal. The rules for each entry were simple:

• Complete over 5 Learning Fridays
• Include at least 1 representative from Engineering, Design, & Product
• Launch as an A/B test
• Winners determined by our standard process of measuring DAGs

These events are not always easy to get off the ground, so I was pleasantly surprised with the enthusiastic support for the idea at Grubhub. It didn’t require a single meeting to get a green light. I received feedback from my team via Google docs, got same day approval via email from 2 very supportive executives, and had hallway conversations about it with a half dozen peers across the org and with my HR business partner. I launched the Dagathon by email on Wednesday morning, with promises of mystery prizes and eternal glory. Promotion was limited to an email per week: no posters, flyers, or t-shirts.

I’ll admit that I waited until the second week to send out a spreadsheet for tracking entries, hoping it wouldn’t look too pitiful with limited activity. To my surprise, there were already a number of ideas added to the spreadsheet by the end of that second Learning Friday.

By following our standard process instead of a one-off contest process, people were completing innovations outside of the prioritized queue, but with the normal checks and balances between Engineering, Design and Product, resulting in quality features and organized A/B test launches. Also, we were following a process that the broader org was now empowered to use for designing, developing, and launching their ideas to production at any time.

The results were nothing short of amazing. These self-formed teams brainstormed 39 ideas and launched 24 of them to production in our websites and mobile apps by the end of the 5th Learning Friday. More are underway and will be completed in the coming weeks. This was all in addition to the day to day work being accomplished in our sprints.

It was especially gratifying to hear people turn “Dagathon” into a verb.

Employee #1: Hey, it would be cool to try Idea XYZ. How can I get that prioritized?

Employee #2: Let’s just dagathon it.

People like the accolades and prizes, but what they value most is empowerment to take their best ideas and get them done. A clear path to put together a little team, and innovate without red tape. I’ll conclude with a quote that I shared in one of the Dagathon emails:

My observation, is that the doers are the major thinkers. The people that really create the things that change this industry are both the thinker and doer in one person… And a lot of people of course — it’s very easy to take credit for the thinking. The doing is more concrete. But somebody, it’s very easy to say ‘oh I thought of this three years ago’. But usually when you dig a little deeper, you find that the people that really did it were also the people that really worked through the hard intellectual problems as well.

Steve Jobs, 1990

Do you want to learn more about opportunities with our team? Visit the Grubhub careers page.

How friendly competition sparked innovation was originally published in Grubhub Bytes on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today, we are excited to announce our Reorder with Grubhub Alexa skill! Now, diners can reorder food from Grubhub’s network of more than 50,000 restaurants using only their voice.

The Grubhub skill for Alexa is available for all Grubhub diners who have a device with Amazon Alexa such as the Amazon Echo, Echo Dot and Amazon Tap. The convenient voice-ordering experience allows diners to ask Alexa to recite their past three orders. They can then reorder from one of their favorite local restaurants.

How it works:

• Diners can enable the Grubhub skill and link to their Grubhub account using the Alexa app
• Diners can then open the Grubhub skill for Alexa with a selection of prompts such as, “Alexa, open Grubhub” or, “Alexa, tell Grubhub I’m hungry”
• Alexa will respond with, “Welcome to Grubhub! Here are the latest orders available for reorder”
• Diners can then choose which meal they would like to reorder
• Alexa will place the order for the diner and provide an estimated time for delivery

To celebrate the new Alexa skill, Grubhub will be offering a sweepstakes to its customers. Diners who order three times in March using Alexa will be entered for a chance to win a $500 Grubhub gift card.

The reorder skill for Alexa will be available to all Grubhub users who have a device with Alexa. For more on the Grubhub skill for Alexa, please visit grubhub.com/alexa.

Do you want to learn more about opportunities with our team? Visit the Grubhub careers page.

Order food without lifting a finger with the Reorder with Grubhub Alexa skill was originally published in Grubhub Bytes on Medium, where people are continuing the conversation by highlighting and responding to this story.

I am often asked about Grubhub’s position on testing. “Is it always better to test more?” “How do you pick tests?” What is the difference between 10, 1,000 and 10,000 tests? “Do you like multivariate tests?” So I decided to use this platform to answer everyone’s questions at once — we process over 290,000 orders every day so efficiency is key, right?

Having an obsession with data is necessary to make product improvements. Product teams get a lot of “million dollar ideas” thrown at them. These ideas come from both inside and outside of the company. The fact of the matter is, anything that is measurable stands a much better chance of improvement, and things that improve are better for the company. Period.

Here’s a rundown of how we test new features at Grubhub:

We’re equal opportunity testers here. Every SINGLE feature that gets released on our site has to follow a process we call Daily Average Grubs (DAG) testing. DAGs are essentially the same as commerce orders, but we always think of our consumers as “Diners” and orders as “Grubs”. Our testing platforms span across our web and mobile devices, as well as our in-house restaurant technologies.

We pride ourselves on going beyond immediate conversion metrics and using Lifetime Value (LTV) as the bar that our tests have to pass. This makes the product testing hurdle a few notches higher as you have to consider the consequences of trading off immediate gratification for creating longer value. The framework for analyzing our test results are homegrown and provide for sophisticated analysis of tests by diner segments.

Testing is a science, but when done correctly, can be a work of art. Three things dictate your testing strategy the most; the maturity of your industry, the competency of your technology and the amount of traffic you have.

Maturity: A new company’s products have likely had no A/B testing. If anything, they could have some qualitative tests and feature releases. The eventual destination is a mature product where the majority of return in innovation is in small increments.

These mature products must go for scale in testing as a badge of honor. 1000 tests are better than 100, and if you had 10,000, well, I profess envy for your traffic profiles. A more open space for feature development has the ability to change the ratio to be skewed towards step function improvements versus incremental improvements. The tradeoff with step function improvements is that they typically are heavier and slower. There are mechanisms to decrease risk and create low-cost versions like painted door tests, but the test is likely still heavy on product and technology groups. The former strategy of testing is akin to the Venture Capital mindset where the sheer volume of creative innovation outweighs the risk of the return. The latter is more like Warren Buffett who has strict principles for value investing.

Remember, it’s not testing that makes you win, but the features themselves. If your industry has space for feature innovation (you’ll find many do!), spend time on strategies to pick where to test in addition to scale. Qualitative research is an excellent way to find a hypothesis to increase the return on quantitative tests.

The competency of your technology: This is an overarching variable. Is your app implementation server side or client side? Is your testing framework client or server side? Do you have continuous deployment and a multiple build framework? How componentized and patternized is your site? All of these will affect your ability to deploy tests at scale.

Grubhub has a reasonably scaled framework and deploys 200–300 tests every year. Our amazing teams get faster every day.

Traffic profile: Every test tends to have a duration that is inversely correlated to your traffic size and will get shorter or longer depending on what type of feedback you are looking for. You might want immediate feedback metrics like conversion/clicks or could be looking for delayed metrics like lifetime value. Perhaps you have tests that need traffic that has to be cordoned off (non overlapping segments) which will cause the need for more traffic.

There are no simple answers. You need to assess where your organization is in its development and imprint the right kind of testing structure. Qualitative testing is just as useful to develop hunches on where to test.

This year is every bit as packed as last year for testing. We are extending our testing frameworks to products beyond our diners and looking for new ways to create great features for our restaurant partners and delivery drivers.

We have a fantastic product team at GH that works on amazing products that our diners love to use, and we are always looking for more ways to fit into their everyday lives.

Creative teams thrive off of creative freedom, but this freedom comes with a price. For product managers, the price of freedom is usually called A/B testing.

Do you want to learn more about opportunities with our team? Visit the Grubhub careers page.

365 days of A/B testing at Grubhub was originally published in Grubhub Bytes on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are so many ways an internship program can go sideways, and we’ve all experienced the dysfunction. On paper, internships are amazingly helpful. We’ve all got valuable projects and need the resources to tackle them. We want to build a pipeline of great candidates to test the waters and make sure they are a good fit for what we are looking for. Interns pretty much want the same things. They want to gain professional work experience and deliver something valuable that they can be proud of. They would also like the opportunity to see if the company is a place they can enjoy working, and hope to get an offer. So why does it go so wrong, and how can it be better?

At Grubhub, our Product team decided to create a more effective internship program. The first step was to treat interns with the same importance as regular employees. This means that we don’t hand off to our people team and tell them to hire a batch of interns to be divvied up. We write the job descriptions, review the resumes filtered down by Recruiting to hand select the interview candidates. We then conduct the interviews and determine who is the perfect fit. Essentially, the same way we’d hire a regular team member.

After we select our interns, we then define meaningful projects before the interns arrive. We want to avoid giving interns projects that wouldn’t be a good fit for a regular team member, such as a research project with no measurable value. We choose projects that we could give to a product manager, with the same expectations: do the research, pull the data, do the planning, and execute as quickly as possible. Given the 10–12 week internship durations, there’s a limit to what they can complete end to end, but we wanted to make sure that they completed a significant, valuable deliverable.

Our third and most valuable step is to make sure that we provide a real mentorship to our interns. No fake assigned mentor who isn’t invested. Their manager, their colleagues, everyone is responsible for helping our interns grow, just as we support any new team member.

The recurring theme is that, with every intern, we’ve treated them like the permanent member of the team that we hoped they could become. Our expectations are high and the results have been outstanding. We brought in interns from Harvard, Chicago, Penn, and they delivered. One intern compared his internship to his friend at a large tech company. While his friend spent the summer creating tests to optimize a small product feature and was bored to death, our intern’s time was spent leading a multi-million dollar opportunity for us. He worked with departments across our company, gained approval of his proposal with multiple SVP’s, negotiated a contract with a 3rd party vendor, and executed the project.

After our CEO had seen the final presentations from our interns, he was impressed enough to direct that we expand the program.

Do you want to learn more about opportunities with our team? Visit the Grubhub careers page.

A more effective internship program was originally published in Grubhub Bytes on Medium, where people are continuing the conversation by highlighting and responding to this story.

As the nation’s leading online and mobile food-ordering company, Grubhub is always innovating to make it easy for our millions of diners to order from the broadest, best set of restaurants. Our motto is: Move Eating Forward and at the core of this sentiment is our technology and product. In the spirit of Move Eating Forward, we’re hoping this blog will Move Technology Forward.

We’re constantly testing new and interesting ways to meet the ever-evolving needs of our customers, persistently innovating and creating groundbreaking new products. So we decided to give our fans and fellow technology enthusiasts a closer look. What fun would it be developing these products if we couldn’t show you exactly how cool they are?

Our technology is great and all, but our favorite reason for creating the blog is to showcase the awesome people that are behind it. We’re proud to have some of the most innovative and talented people in the industry working on Grubhub’s technology teams. They’re the reason why your pad thai shows up at your doorstep, fresh and ready to eat. We know what you’re thinking, we were also shocked when we found out that this wasn’t magic or some type of sorcery, but there’s a lot of hard work that goes into it! Now we’re pulling back the curtain so you can see how it all happens.

We hope you enjoy reading this blog as much as we enjoy writing it! If you like what you see, and want to hear how you can be a part of the team, you can check out our open positions on our Careers page. We have a lot more we want to build and we need curious and innovative people like you to help.

Happy eating (and reading)!
The Grubhub Product and Technology Teams

Do you want to learn more about opportunities with our team? Visit the Grubhub careers page.

Welcome to the Grubhub Tech Blog! was originally published in Grubhub Bytes on Medium, where people are continuing the conversation by highlighting and responding to this story.