<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[doodle-tech - Medium]]></title>
        <description><![CDATA[Stories from Engineers, Product Managers and Designers - Medium]]></description>
        <link>https://medium.com/doodle-tech?source=rss----799c4b950147---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>doodle-tech - Medium</title>
            <link>https://medium.com/doodle-tech?source=rss----799c4b950147---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 05:36:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/doodle-tech" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Doodle Poll Party!]]></title>
            <link>https://medium.com/doodle-tech/doodle-poll-party-67d86d4d2db9?source=rss----799c4b950147---4</link>
            <guid isPermaLink="false">https://medium.com/p/67d86d4d2db9</guid>
            <category><![CDATA[scheduling]]></category>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[doodle]]></category>
            <dc:creator><![CDATA[Alon Shomrat]]></dc:creator>
            <pubDate>Wed, 29 May 2019 06:21:06 GMT</pubDate>
            <atom:updated>2019-05-29T06:21:41.514Z</atom:updated>
            <content:encoded><![CDATA[<p>Introducing the <a href="https://doodle.com/content/doodle-bot">Doodle Bot</a>, the next big thing for scheduling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bgHNF8Tsk8PZd2K8xTMcHg.png" /></figure><p>I’m sure you are all using Doodle polls to schedule meetings with large groups of people. Or maybe you’re using Doodle 1:1s for your quick one-on-one meetings. Whether you’re creating, participating or just viewing the results of polls, Doodling is very much an everyday thing for most of us.</p><p>What’s the other tool we use every day in our work lives to streamline communication and collaboration? Slack. Slack is the most popular business messaging platform today. Teams chat, integrate and manage daily tasks with it.</p><p>We thought — so much of our work already lives in Slack, wouldn’t it be great to create Doodle polls and schedule there too?</p><h3>So, how did we start creating the Doodle bot?</h3><p>First, we need to understand how to insert Doodle information and look into Slack platform, this will be done with the bot service. The bot service is the new service that will connect, interact and bring Doodle into Slack.</p><p>Our infrastructure at Doodle is ready to communicate with external tools and apps, we can use our API and events to get all the data and information we need. All the services needed for the new bot are using an event manager to talk to each other, so adding another service (the bot service), it can use the events from our existing services to get and add any information we need.</p><p>Second, we need to have the ability to add, modify and remove polls and information about polls in Slack, as fast as we can, as simply as we can. With the additional challenge of needing to have all the information updated in the web too. This means you can interact with a poll in any way, in Slack or on the web, and the poll will be the same everywhere. No small task!</p><h3>Connecting the Doodle Bot to Doodle</h3><p>You can create Doodle polls from the web or from the app, now we just need to throw Slack in the mix. How do we do that?</p><p>The first thing we need is to create this connection between the bot to Doodle’s DB and other services.</p><p>The bot will use Doodle’s API for creating and modifying the polls and will update the bot service according to those calls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/0*lPFOyTEeBz829NnR" /><figcaption>Communication between Doodle Bot and Doodle</figcaption></figure><p>Second, we need to update the polls created with the bot according to any changes done in the poll, via Slack or the web.</p><p>This is solved by the event manager and registration to the relevant topics. The poll’s instances will update when such an event occurs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Al0ALo5UuPDLxQPq" /><figcaption>Communication between different services</figcaption></figure><h3>Building the Doodle Bot with Slack’s Block kit</h3><p>Now, we want to start creating polls in Slack, this is being done with Slack’s new framework — the Block kit (<a href="https://api.slack.com/block-kit">https://api.slack.com/block-kit</a>).</p><p>It was extremely difficult to insert web design into Slack, but the block kit is perfect for that.</p><p>With the new block kit features, we can now have actions (buttons or dropdowns) in the same line as text, which is perfect for voting in a Doodle poll. We can pick a date easily with the new datepicker. You can get the number of voters and see who voted for what with the participant icon and we can have different sections with the new dividers.</p><ul><li>Poll title, description and location: a simple <strong>section block</strong> is the block for this one (<a href="https://api.slack.com/reference/messaging/blocks#section">https://api.slack.com/reference/messaging/blocks#section</a>).</li><li>If we want to edit those fields, we will need a button for it, an <strong>actions block</strong> with a button as an element is the right choice for it (<a href="https://api.slack.com/reference/messaging/blocks#actions">https://api.slack.com/reference/messaging/blocks#actions</a>), you can add datepicker or overflow too.</li><li>The easiest way to know who voted for what is to see the person with the voting option, an <strong>Image block</strong> will provide us the voter photo, which is what we want (<a href="https://api.slack.com/reference/messaging/blocks#image">https://api.slack.com/reference/messaging/blocks#image</a>).</li><li>Text is one of the most common things we want to send the user, we can use <strong>context block</strong> for this, and even insert photos to it (<a href="https://api.slack.com/reference/messaging/blocks#context">https://api.slack.com/reference/messaging/blocks#context</a>).</li><li>To separate each section of the poll, which inserts order, we can use the simple <strong>divider block</strong>(<a href="https://api.slack.com/reference/messaging/blocks#divider">https://api.slack.com/reference/messaging/blocks#divider</a>).</li></ul><p>With all those great tools, we can insert polls from the web to Slack with minimal changes.</p><p>Now, we can create the poll instances:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/0*iRqxexsGExS0JTr9" /><figcaption>Poll instance with a regular view, created with Block kit only</figcaption></figure><p>Each one of the poll instances gets an update when it receives a new event (for example, if one will add a new time from the web, the new time will be added to all the poll instances in Slack).</p><h3>How the Doodle Bot communicates with Slack</h3><p>The connection between Doodle Bot and Slack is being done with Slack great API:</p><ul><li><em>chat.postMessage</em> — for post a regular message with the information (<a href="https://api.slack.com/methods/chat.postMessage">https://api.slack.com/methods/chat.postMessage</a>).</li><li><em>chat.postEphemeral</em> — for posting ephemeral messages (<a href="https://api.slack.com/methods/chat.postEphemeral">https://api.slack.com/methods/chat.postEphemeral</a>).</li><li><em>chat.update</em> — for updating a message (a poll in our case) (<a href="https://api.slack.com/methods/chat.update">https://api.slack.com/methods/chat.update</a>).</li><li><em>Interactive messages</em> — many many different cases (<a href="https://api.slack.com/docs/interactive-message-field-guide">https://api.slack.com/docs/interactive-message-field-guide</a>).</li></ul><p>And the bot is triggered by slack slash command, action buttons and different events (channel events, user events…). For example, to start a new poll you just need to /doodle and hit enter (slash command trigger), you can edit the poll by clicking the `edit` button (action button trigger) and choose who to invite from the invitees dropdown, full with channels and users we have from different slack events.</p><p>So, now we can post, update and delete every instance we like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YE8ZcmskYMmxFxOI" /><figcaption>Connection between Doodle Bot and Slack</figcaption></figure><h3>The full picture</h3><p>So, we talked about the full cycle that we need in order to have polls in Slack:</p><ul><li>How to connect to Doodle and how all the services shared events with the event manager.</li><li>What do we need in order to communicate with Slack.</li><li>How we can build a Doodle poll with the new block kit.</li></ul><p>We have now everything we need, let’s connect all the parts and see the full picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AzyD2HEG0OgTrGq7" /><figcaption>Doodle Bot full architecture</figcaption></figure><p>This is a solid architecture for the bot’s needs. The bot gets from Slack or from the event manager.</p><p>The bot business logic receives the app’s payload, getting it ready and route it to the right handler.</p><p>Now the data is sync with the Doodle DB so the web poll and Slack poll will be the same properties (e.g. same participants, same time options…).</p><p>After the data is getting handled, we are update every poll instance with the new data, built with the block kit.</p><p>Alright, let’s start scheduling!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67d86d4d2db9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doodle-tech/doodle-poll-party-67d86d4d2db9">Doodle Poll Party!</a> was originally published in <a href="https://medium.com/doodle-tech">doodle-tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Integration for 1000+ Apps in 4 Months]]></title>
            <link>https://medium.com/doodle-tech/building-an-integration-for-1000-apps-in-4-months-268afd7a1164?source=rss----799c4b950147---4</link>
            <guid isPermaLink="false">https://medium.com/p/268afd7a1164</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[zapier]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Jan Carsten Lohmüller]]></dc:creator>
            <pubDate>Fri, 22 Mar 2019 15:30:48 GMT</pubDate>
            <atom:updated>2019-03-22T15:30:48.462Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pdDslhXDMlXpaR0mGCXt5w.jpeg" /></figure><p>Imagine the following: You create an awesome Doodle to invite all your friends to your birthday party. You get very excited and would like to call your mum every time someone participates in the Doodle.</p><p>That would be a lot of work, right?</p><p>Plus: You would probably get overexcited. Well, we got you covered.</p><p>Doodle now integrates with 1000+ apps to make that happen. You can simply hook up a trigger every time one of your friends participates in your Doodle and let another service call your mum — via Zapier.</p><p>Zapier is a workflow automation system that basically allows users to chain events (<em>“triggers”</em>) in one app to features (<em>“actions”</em>) in another app.</p><p>We, at Doodle, have the grand plan to “Build an ecosystem of tools that integrate Doodle”, to “Cater to large deployments at enterprises” like SSO and automation, and “Make meetings more effective by adding features and integrations to help with the full meeting lifecycle”.</p><p>While we have a general understanding that this will be needed to help Doodle grow, we have not discovered the concrete use-cases of users and the order of importance of these. Zapier shall help us (partially) answer the first question and get a user-base that is interested in integrations to talk to (to answer the second question).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*SPcxggGxffVcJvmw.png" /></figure><h3><strong>Why a blog post?</strong></h3><p>We would like to share our story of building a Zapier integration with you. Why? Because all the Stack Overflow discussions, Medium blog posts and Github example repositories helped us a lot and we would like to contribute to that.</p><p>Come join us.</p><p>Here comes a quick roundup of our Zapier integration architecture and the very exciting paradigm shift we’re currently experiencing in our engineering team.</p><h3><strong>Let’s crack things up</strong></h3><p>Doodle is currently transitioning from a traditional monolithic backend towards a more modern, sustainable and event driven microservice approach.</p><p>To get a little more detailed: We are running mostly Java 10 services with Spring Boot 2.1.2 and Apache Kafka 2.0.1. From time to time you will also find a MongoDB and a little pinch of Python here and there. However our monolith is still operating on Java 8. Infrastructure-wise we are relying on AWS with Kubernetes and everything wrapped nicely into helm charts.</p><p>The first step towards a sustainable microservice architecture for us was to raise business events to Kafka every time a domain object changes in the monolith. In order to achieve consistency we went for using <a href="https://avro.apache.org/docs/current/">AVRO schemas</a> for our events. We keep all our schemas in one repository which is carefully maintained by every backend developer because we aim for <a href="https://docs.confluent.io/current/schema-registry/docs/avro.html#full-compatibility">full schema compatibility</a>.</p><p>The first big challenge we had to tackle was our legacy authentication mechanism. Well, we abstracted from that: In order to see results quickly we chose <a href="https://www.keycloak.org/">Keycloak</a> for that purpose. The pros for Keycloak are, that it runs on standards and does what it’s supposed to do: Authentication and Authorisation, plus: You can easily implement your own so called <a href="https://www.keycloak.org/docs/latest/server_development/index.html#_providers">Service Provider Interface (SPI)</a> in order to hook up your legacy authentication endpoint to it. Keycloak will then simply use that legacy endpoint to leverage credentials validation.</p><p>After mastering the authentication flow we needed to integrate <a href="https://zapier.com/developer/documentation/v2/static-webhooks/">Zapiers webhooks</a> into our eventing ecosystem. Zapier is able to register a webhook in our app in order to point our trigger to the correct zap. The solution is yet simple and beautiful: Zapier subscribes webhooks at our hooks service like so:</p><pre>POST &lt;subscribe_endpoint&gt; <strong>\</strong><br>    -H Authenticated: Somehow <strong>\</strong><br>    -H Content-Type: application/json <strong>\</strong><br>    -d &#39;{&quot;target_url&quot;: &quot;https://hooks.zapier.com/&lt;unique_path&gt;&quot;,<br>         &quot;event&quot;: &quot;poll_created&quot;}&#39;</pre><p>This subscription gets transformed into a Kafka event and directly emitted to our hook subscription topic — that’s the easy part.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r5vgA9kHPsCP8Gm40JkcTA.png" /><figcaption>Zapier service architecture overview at Doodle</figcaption></figure><h3><strong>Fasten your seatbelts</strong></h3><p>Let’s talk about the most interesting part: Kafka Streams. We currently offer the following triggers for our users in Zapier: Trigger, when a new Doodle gets created, someone participates in your Doodle and when you close your Doodle and pick a final option. Sounds easy, right? Well, take this:</p><p>At Doodle there are 30 million active users per day, creating Doodles, participating in them and closing Doodles. That’s a lot of data!</p><p>Let’s have a look at one particular example flow: You, as a user, would like to be notified on, let’s say Slack, whenever someone participates in your Doodle. What our streams application does is:</p><ol><li>Consume the topic of subscribed hooks</li><li>Consume the topic of participations in all Doodles</li><li><a href="https://docs.confluent.io/current/streams/developer-guide/dsl-api.html#joining">Join</a> them by key user email</li><li>Trigger the correct hook as soon as a participation happens</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yiu-QeRrYK56pY7HfSEEog.png" /><figcaption>Joined streams</figcaption></figure><p>That sounds easy in theory but we had to take a long way to get there on which we learned a lot about building and operating Kafka Streams applications.</p><h3><strong>Learnings</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I9xA_Me32AdvT1X47E2k9g.jpeg" /></figure><h4><strong>Do not do interactive queries inside Kafka Streams iterators ☝️</strong></h4><p>Interactive queries are very useful when you want to query the current state of your application.</p><p>We got that wrong.</p><p>We used interactive queries inside the forEach of a kafka stream, calling another instance of our application to get a specific value if that value was not present on this instance from which we’re calling. We thought this might be a good idea because we read up about it on the <a href="https://docs.confluent.io/current/streams/developer-guide/interactive-queries.html">Confluent Blog</a>.</p><p>It was not.</p><p>This implementation ended in high latency inside of the streams threads, which then resulted in the stream transitioning into ERROR state. So we scaled out application to only one instance to make it work, but that’s not what we aimed for.</p><p>Never ever mix interactive queries with stream operations.</p><p>When you want to process corresponding values of different kafka topics in one instance of your application you have to repartition them to use the same key.</p><p>Same key, same partition.</p><p>After repartitioning the topics to the same key they can get joined and voilá: No interactive queries inside of streams needed, ergo: No crashing application.</p><p>We were able to scale up again.</p><h4><strong>Don’t let schema evolutions ruin your day</strong></h4><p>While developing our streams applications, we had to experiment with AVRO schemas. It occurred more and more often that we deployed a new version of our service to our staging environment and it would simply not work because the Schema Registry would tell us</p><p>409 : Incompatible Schema</p><p>Which ultimately ended up in a more or less painful search for the field, type, or default value we messed up. As you might remember: We aim for full compatibility of our schema. Here are the top two mistakes we made:</p><ul><li>Type “string” or “null” must have a default value null</li></ul><pre>{   <br>    &quot;name&quot;: &quot;description&quot;,   <br>    &quot;type&quot;: [     <br>        &quot;null&quot;,     <br>        &quot;string&quot;   <br>    ],   <br>    &quot;default&quot;: null  <br>    &quot;doc&quot;: &quot;Poll&#39;s description. Maximum of 512 chars&quot; <br>}</pre><ul><li>Type “array” must have a default value [] (not “null” or null)</li></ul><pre>{   <br>    &quot;name&quot;: &quot;hooks&quot;,   <br>    &quot;default&quot;: [],   <br>    &quot;type&quot;: {     <br>        &quot;type&quot;: &quot;array&quot;,<br>        &quot;items&quot;: { <br>...</pre><p>But if you already messed up your schema registry and there is no way other than deleting the schema you can do as stated in the <a href="https://docs.confluent.io/current/schema-registry/docs/schema-deletion-guidelines.html">Confluent docs</a>:</p><pre><em># Deletes all schema versions registered under the subject &quot;Kafka-value&quot;   </em></pre><pre>curl -X DELETE http://localhost:8081/subjects/Kafka-value    </pre><pre><em># Deletes version 1 of the schema registered under subject &quot;Kafka-value&quot;   </em></pre><pre>curl -X DELETE http://localhost:8081/subjects/Kafka-value/versions/1   </pre><pre><em># Deletes the most recently registered schema under subject &quot;Kafka-value&quot;</em>   </pre><pre>curl -X DELETE http://localhost:8081/subjects/Kafka-value/versions/latest   </pre><h4><strong>Keep development, staging, and production as similar as possible — in theory</strong></h4><p>As said before, at Doodle there are 30 million active users per day. Imagine you test your app in a very protected staging environment where you basically control the amount of events that get emitted to Kafka.</p><p>You feel safe.</p><p>You decide to move your application to preproduction. Huge step! This environment should be the same as production. Very exciting but still a limited amount of Kafka events. Next up:</p><p>Production.</p><p>You enter the stage and fall. Fall over. Directly. Why? Well, did I just say 30 million active users? Try to emulate that in staging!</p><p>This is still a challenge we try to figure out at the very moment. If you have an idea how to tackle this challenge, feel free to reach out to us.</p><h3>What’s next?</h3><p>Building a Zapier integration is just the first step to rapidly get rid of our monolith. Kafka streams was potentially one of the reasons why we were able to release this integration within 4 months (almost hassle free of course). So much about architectural choices. I bet there is going to be another post about that soon.</p><p><em>But what about the Zapier integration?</em></p><p>We will add a few things:</p><ul><li>Social Login, probably the most wanted feature. Currently you’re only able to log in via username and password. We want to give all users the possibility to use their Google/Facebook/Microsoft account to log in.</li><li>Actions! The first action we will add is obviously: Create a Doodle based on a trigger of another app.</li><li>…</li></ul><ul><li><a href="https://doodle.teamtailor.com/">Doodle - Join the team</a></li><li><a href="https://doodle.com/content/zapier-integration">Doodle Zapier Integration - Doodle&#39;s Content Pages</a></li><li><a href="https://zapier.com/apps/doodle/integrations">Doodle Integrations</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=268afd7a1164" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doodle-tech/building-an-integration-for-1000-apps-in-4-months-268afd7a1164">Building an Integration for 1000+ Apps in 4 Months</a> was originally published in <a href="https://medium.com/doodle-tech">doodle-tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 things I learned about A/B testing]]></title>
            <link>https://medium.com/doodle-tech/3-things-i-learned-about-a-b-testing-f4dd378cf668?source=rss----799c4b950147---4</link>
            <guid isPermaLink="false">https://medium.com/p/f4dd378cf668</guid>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[doodle]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[Matthias Suter]]></dc:creator>
            <pubDate>Thu, 07 Feb 2019 14:08:35 GMT</pubDate>
            <atom:updated>2019-02-07T14:08:34.942Z</atom:updated>
            <content:encoded><![CDATA[<p>A/B testing became a crucial tool in our process and I want to take a moment to reflect on the things I learned along the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*hDYjf0J87w8O0xjyDV6kqg.png" /></figure><p>I take it as a given that A/B testing is an excellent way to test new features. Therefore this post is not about where A/B test makes sense and where not but I will focus on the three most important learnings I made during my journey to become data informed.</p><h3>Size matters</h3><p>It is important to put some thoughts on what exactly should be tested. While about 100k polls are created each day, we only get 50 subscriptions on average per day. Traffic to test with can get a sacred good, more so as we are running more and more tests and want to move even faster (did I hear lightspeed?).</p><p>Best is to have a crisp hypothesis and a variation which addresses exactly and only the things which should be tested. This sounds logical enough but can easily go wrong. One time we tested a completely overhauled pricing page just to find out that in general, it did not perform better than the old one. The main problem was, that this was also the only real outcome. We did not really gain more knowledge with this test (except that the new page did not perform better in our test). In another experiment, we only tested where the user expects a new feature instead of understanding what the feature should be. In retrospect, it was too costly to spend 2 weeks running this experiment, more so it did not give any real insight.</p><p>It is good practice to let tests run at least two weeks to avoid selection bias. This is true even statistical significance is reached before that time. If you let your tests run shorter than one cycle (which in our case is one week) you risk having data which is not representative due to insufficient sampling. Doodle’s traffic and user base, for example, differs on the weekend from weekdays. For that reason, we usually let run tests at least two weeks to cover our cycle twice. This helps to flatten “special” days and other anomalies.</p><p>Usually, we also do not run tests much longer than two weeks. Longer tests can become non-representative due to other changes like marketing campaigns and seasonal effects. Also, the change that a user deletes his cookie, access the page and sees another version gets higher. More importantly, if there is no winner after two weeks even there is enough traffic, most likely the change in the experiment just does not lead to different behavior. In these cases, it is better to go back to the drawing board and test either another hypothesis or the same hypothesis but with different variations.</p><h3>Go for a win</h3><p>A/B testing should not be a replacement for making a decision. This was my biggest realization and is in my opinion also a common misconception. It is true that A/B testing (or testing in general) is a great way to avoid lengthy and pointless discussions where people mostly just exchange opinions. I also believe that most of the time it is easier and more efficient “just to test” something instead of making big concepts based on a lot of untested assumptions.</p><p>But I do not think that A/B testing should be used to avoid decisions in the design process and just test all possible variations one can think of. I see it more of a way to design and implement one (possibly small) change which the teams believe in to either prove or disprove an assumption. In most cases, it is faster and less work to test a well-thought variation instead of implementing a bunch of different variations to see which one will win (I call this the no-Darwinism-theorem).</p><h3>Be patient</h3><p>As already outlined in the first point, A/B tests should run at least 2 weeks. Therefore it also does not make sense to check the results every day and start drawing conclusions based on non-significant data. Otherwise, you start to sound stupid by talking about “trends” and “first insights” (been there, done that and most likely will do it again…). Best would be just to ignore the results for two weeks, but…</p><p>… that said, you should still check your A/B tests on a regular basis. Mainly to spot problems early and to get a feeling if you are likely to end up with a result at the end of the test phase or not. We always start experiments with low traffic (depends a bit, but usually 10% of the traffic) and let it run for 24 hours to see what goes wrong. If nothing goes wrong (or we do not realize that we have broken things, also happened) we go to 100% traffic.</p><p>At the moment we are a bit optimistic about running different test at the same time. We make sure not to have obviously conflicting tests but there is still room to improve. Ideally, we would make sure that every user only sees exactly one experiment. But to achieve that we need to get smarter with segmenting users and allocate the minimal needed traffic to each experiment. I hope this will be part of my next blog post (as soon we found out how to do it though…).</p><p>I hope you enjoyed the read although it did not contain any cat pics and please use the comment section to express disagreement or contribute with your insight!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4dd378cf668" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doodle-tech/3-things-i-learned-about-a-b-testing-f4dd378cf668">3 things I learned about A/B testing</a> was originally published in <a href="https://medium.com/doodle-tech">doodle-tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>