<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Comic Relief Technology - Medium]]></title>
        <description><![CDATA[This is where we showcase the ways that we are use technology to support the core aims of our organisation and how we do it. - Medium]]></description>
        <link>https://medium.com/comic-relief?source=rss----4e7e42139f79---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Comic Relief Technology - Medium</title>
            <link>https://medium.com/comic-relief?source=rss----4e7e42139f79---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 17:41:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/comic-relief" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Key Insights from Pixel Pioneers 2025: Agents, Accessibility, and the Future of Web Monetization.]]></title>
            <link>https://medium.com/comic-relief/key-insights-from-pixel-pioneers-2025-agents-accessibility-and-the-future-of-web-monetization-6a04a5f03b50?source=rss----4e7e42139f79---4</link>
            <guid isPermaLink="false">https://medium.com/p/6a04a5f03b50</guid>
            <category><![CDATA[accessibility]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[front-end-development]]></category>
            <dc:creator><![CDATA[michael o'donovan]]></dc:creator>
            <pubDate>Tue, 09 Sep 2025 16:52:37 GMT</pubDate>
            <atom:updated>2025-09-11T15:28:16.399Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d63Tq-bxzx9_pNcfFpPyBA.png" /><figcaption>Bristol Suspension Bridge</figcaption></figure><p>A few weeks back, members of our Front-End Engineering team attended <a href="https://pixelpioneers.co/events/bristol-2025">Pixel Pioneers 2025 </a>in Bristol, where a range of compelling speakers highlighted critical challenges and opportunities shaping the future of web development in 2025. From AI’s complex relationship with accessibility to emerging payment models and the potential transformation of web browsing itself, the conference offered both sobering warnings and exciting possibilities for our industry. Here are three talks that made the most impact on me.</p><p><strong>The Promise and Peril of AI in Web Accessibility — Léonie Watson</strong></p><p><a href="https://www.linkedin.com/in/lwatson">Léonie Watson</a> delivered a nuanced examination of generative AI’s impact on accessibility, highlighting both significant concerns and transformative benefits. The talk revealed troubling issues around AI training data, which often includes copyrighted material used without authorization, and the substantial environmental costs of data centers consuming electricity and water in already scarce regions.</p><p>More directly relevant to our work, Watson demonstrated AI’s current limitations in generating accessible code. When asked to create accessible HTML buttons, ChatGPT produced contradictory and incorrect markup — a reminder that these tools often reflect the poor coding practices they learned from decades of existing web content. The presentation also warned against “AI remediation” companies promising quick accessibility fixes through overlay tools, which not only fail to address fundamental accessibility issues but also raise serious privacy concerns by collecting extensive data about users with disabilities.</p><p>However, Watson also highlighted AI’s genuine benefits for people with disabilities. Modern AI tools now provide detailed image descriptions that enable blind users to explore family photos independently or make informed online purchases. Real-time description features and emerging wearable technologies offer unprecedented access to visual information. The key takeaway: while we may not control AI’s presence in our lives, we can choose how to use these tools responsibly and effectively.</p><p>YouTube recording of the talk <a href="https://www.youtube.com/watch?v=UVZPwzCQvLs">here</a>.</p><p><strong>Rethinking Web Monetization for a More Equitable Internet — Rabeb Othmani</strong></p><p><a href="https://www.linkedin.com/in/othmanirabeb">Rabeb Othmani</a> from the <a href="https://interledger.org/">Interledger Foundation</a> presented web monetization as a potential solution to the broken economics of online content. The current landscape presents a lose-lose scenario: users face paywalls blocking access to individual articles or encounter ad-cluttered pages that create poor reading experiences, while content creators — particularly smaller publishers and those in underrepresented communities — struggle to monetize their work effectively.</p><p>Web monetization proposes a “pay-as-you-browse” model where users stream micropayments directly to content creators while consuming their content. Instead of committing to full subscriptions or tolerating intrusive advertising, users can set a budget (such as 60 cents per hour) that automatically distributes small payments to websites they visit. Publishers need only add a simple link tag to their pages to receive payments.</p><p>Currently implemented as a browser extension while the team works toward native browser integration, the system uses the Interledger Protocol to facilitate payments between different wallet providers. The technology is positioned as complementary to existing monetization methods rather than a complete replacement, with the specification currently in draft stage heading toward W3C standardization.</p><p>While the concept addresses real pain points, practical challenges remain around achieving the critical mass of adoption needed for economic viability. The success depends on both widespread user adoption and sufficient participating websites to make the system worthwhile for all parties. The Inteledger Foundation has some smart solutions but I do wonder how their admiral goals will fair if they ever upset the apple cart enough to come up against the deep pockets of the existing entrenched interests of billion dollar FANG companies.</p><p><strong>Preparing for an Agentic Web</strong></p><p><a href="https://nearestnabors.com/">Rachel Lee Nabor</a>s presented perhaps the most forward-looking vision of the conference, describing an “agentic web” where AI agents, rather than human browsers, increasingly interact with web content on users’ behalf. Drawing on their extensive experience with Firefox, Edge, React, and web standards, Nabors argued that we’re approaching a fundamental shift in how people access and interact with web content.</p><p>The talk traced the evolution from the early decentralized web through the platform-dominated landscape we know today, explaining how “enshittification” has led consolidated platforms to extract increasing value from both users and content creators. With evidence like Safari usage declining for the first time due to users preferring AI-powered search tools, Nabors suggested we may be entering a post-browser era within 5–10 years.</p><p>Central to this vision is the Model Context Protocol (<a href="https://modelcontextprotocol.io/docs/getting-started/intro">MCP</a>), which enables AI agents to access tools, resources, and prompts from web services. The talk explored emerging technologies like on-device Large Language Models and “ambient UX” — interfaces that adapt automatically to user needs and contexts. These developments could potentially solve longstanding accessibility challenges by seamlessly translating content between different modalities.</p><p>For web professionals, Nabors recommends practical steps: implementing MCP servers to make content agent-accessible, creating LLMs.txt files (similar to sitemaps but for AI), and optimizing content for AI consumption through fresh, opinionated, and rich content. Most importantly, they urge the web community to actively participate in shaping these emerging standards rather than allowing others to define the future of web interaction.</p><p><strong>Key Takeaways for us as a team at Comic Relief</strong></p><p>These three talks collectively paint a picture of an industry in transition. AI presents both opportunities and risks that require careful navigation. New monetization models offer hope for more equitable content economics but face significant adoption challenges. And the potential shift toward agentic web interactions could fundamentally change how we build and design web experiences.</p><p>As web professionals, our challenge is to stay informed about these developments while maintaining focus on user needs, accessibility, and ethical practices. The future may look different from what we’re building today, but the core principles of creating inclusive, user-centered experiences remain more important than ever.</p><p>The conversations at Pixel Pioneers 2025 reminded us that while we cannot control all the forces shaping our industry, we can influence how these technologies develop and ensure they serve everyone equitably. The question isn’t whether these changes will come, but how we’ll help shape them for the better.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a04a5f03b50" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comic-relief/key-insights-from-pixel-pioneers-2025-agents-accessibility-and-the-future-of-web-monetization-6a04a5f03b50">Key Insights from Pixel Pioneers 2025: Agents, Accessibility, and the Future of Web Monetization.</a> was originally published in <a href="https://medium.com/comic-relief">Comic Relief Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating from Nightwatch and Cypress to Playwright Test Framework]]></title>
            <link>https://medium.com/comic-relief/migrating-from-nightwatch-and-cypress-to-playwright-test-framework-3c8ad0c45735?source=rss----4e7e42139f79---4</link>
            <guid isPermaLink="false">https://medium.com/p/3c8ad0c45735</guid>
            <dc:creator><![CDATA[Krupa Pammi]]></dc:creator>
            <pubDate>Fri, 02 Feb 2024 15:24:14 GMT</pubDate>
            <atom:updated>2024-02-02T15:24:14.003Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Nightwatch:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*2lV9klVNszez_0fVQWuo_g.png" /></figure><p>Our journey began as we transitioned from <a href="https://www.php.net/">PHP</a> to the <a href="https://aws.amazon.com/pm/lambda">Serverless</a> Framework, crafting our in-house Donate platform. This marked the commencement of our utilization of the Nightwatch Test Framework in May 2018. <a href="https://nightwatchjs.org/">Nightwatch.js</a>, a free automation test framework written in <a href="https://nodejs.org/en">Node.js</a>, leverages the <a href="https://www.w3.org/TR/webdriver2/">W3C WebDriver API</a>, streamlining the creation of end-to-end tests in Javascript for our web-based apps and websites. It communicates over a restful HTTP API with a WebDriver server, making it one of the most popular test automation frameworks.</p><p>Departing from PHP <a href="https://docs.behat.org/en/latest/">behat</a> tests, Nightwatch served us well by swiftly executing tests. The decision to use Nightwatch stemmed from the desire to align the programming language of our application with our entire tech stack (JavaScript and Node.js) at the backend, front end, and in automated tests. Its syntax was straightforward, utilizing <a href="https://www.selenium.dev/">Selenium</a> for DOM element operations and providing clear, easy-to-use assertions and commands.</p><p>Integration with <a href="https://www.browserstack.com/">Browserstack</a> enhanced our testing capabilities, allowing cross-browser and version compatibility checks. However, challenges arose as our application and testing suite expanded, especially when dealing with backend testing and third-party API dependencies. Nightwatch became less maintainable, prompting the search for a more efficient testing framework after Red Nose Day 2022. In case of a failure, reviewing the videos to identify issues was a straightforward process. The syntax was easily comprehensible. However, as our application expanded, so did the testing suite, especially concerning backend testing to ensure frontend data ingestion into the database. To accommodate this, I introduced numerous waits. Testing across multiple providers and reliance on third-party APIs led to prolonged test durations, occasional flakiness, and the need to either disable or fix tests via pull requests.</p><p>While the tests remained intermittently flaky yet manageable, challenges emerged upon implementing a Serverless Service for our ERP system. This service, which is responsible for executing actions from our Supporter Event Service to create and update records in ERPNext, required thorough checks to ensure accurate data storage across various applications. We implemented lengthy timeouts to counteract delays caused by extended data processing times.</p><p><strong>Cypress:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*BygcoyKIcv2r9xdHkVMfsw.png" /></figure><p>Running Nightwatch tests on staging alongside <a href="https://concourse-ci.org/">Concourse</a> pipeline-triggered end-to-end tests, we incorporated <a href="https://www.cypress.io/">Cypress</a> tests into pull requests with Github actions. Cypress offered advantages like automatic screenshots of failures and commendable documentation, aiding us during installation and initial test creation. However, its limitations included inflexible backend testing, limited iFrame support, and constraints on driving two browsers simultaneously. The lack of parallel testing support also posed challenges.</p><p><strong>Playwright:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*KzzHIyfIsosvUJ713I-qug.png" /></figure><p>After the challenges faced, particularly after Red Nose Day 2022, I dedicated time to investigating a testing framework. The goal was to find a fast, reliable end-to-end testing framework that could effectively address the issues encountered with Nightwatch and Cypress.</p><p>Addressing the challenges encountered with both Nightwatch and Cypress, we discovered <a href="https://playwright.dev/">Playwright</a>. After thorough research and experimentation, we found Playwright to be robust and faster than its counterparts. Implementing Playwright application by application proved seamless, with straightforward setup, configuration, and test examples. Integration with Browserstack for staging end-to-end tests allowed easy tracking and video viewing on test failures, offering flexibility to switch browsers. Playwright’s device emulation is another feather in its cap, allowing us to emulate various devices and screen sizes. This capability ensures thorough testing for responsive design, a critical aspect of our applications.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*8sRCcj6ffdSdvzaQuVkqCw.png" /></figure><p><strong>Advantages of Playwright:</strong></p><p>One noteworthy advantage Playwright realized after transitioning from Nightwatch and Cypress is the significant reduction in test duration — from 1 hour to under 10 minutes for some applications. This efficiency has streamlined our code delivery into production, ensuring reliable tests even with frequent deployments. Our CI/CD pipelines consistently remain green, showcasing the reliability and effectiveness of Playwright in our testing workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*gx_IB305f0wYVKBLUxrSCQ.png" /></figure><p>Opting for Playwright over Nightwatch has proven to be a strategic decision, significantly elevating our capabilities in web testing and browser automation. This robust automation framework brings forth a multitude of advantages that have reshaped our testing approach.</p><p>One of Playwright’s standout features is its automatic wait handling, as it intelligently waits for elements to be ready before interaction, reducing the need for explicit waits in our test scripts. This contributes significantly to the stability and reliability of our tests.</p><p>Playwright offers impeccable cross-browser support. Whether it’s Chrome, Firefox, or WebKit, Playwright ensures a seamless testing experience across various browsers. What sets Playwright apart is its utilization of a unified API, simplifying automation efforts by providing a single interface for multiple browsers. This not only streamlines our code but also enhances maintenance efficiency.</p><p>Playwright’s flexibility is evident in its support for both headless and headful test executions. This adaptability allows us to cater to different testing environments based on our needs. Moreover, the framework introduces a paradigm shift in test execution speed by enabling parallel execution of multiple tests. This not only expedites our testing processes but also contributes to a more efficient testing workflow.</p><p>The framework’s support for creating and managing multiple browser contexts adds a layer of isolation for different sets of tests. This not only enhances modularity in our testing approach but also fosters test independence.</p><p>Playwright’s commitment to comprehensive documentation and active community support further solidifies its position as a go-to automation solution. Access to resources, support, and updates on best practices and new features is readily available, making our journey with Playwright both seamless and enriching.</p><p>Integration with popular test runners like <a href="https://jestjs.io/">Jest</a> and <a href="https://mochajs.org/">Mocha</a> is the icing on the cake, ensuring Playwright seamlessly fits into our existing testing workflows.</p><p>In conclusion, by embracing Playwright, we have not only embraced a powerful automation framework but also revolutionized our testing landscape. Playwright’s array of features has not just streamlined our testing processes but has also empowered us to navigate the complexities of web testing with confidence and efficiency.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c8ad0c45735" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comic-relief/migrating-from-nightwatch-and-cypress-to-playwright-test-framework-3c8ad0c45735">Migrating from Nightwatch and Cypress to Playwright Test Framework</a> was originally published in <a href="https://medium.com/comic-relief">Comic Relief Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Starting out with data puddles, then we’ll think about data lakes]]></title>
            <link>https://medium.com/comic-relief/starting-out-with-data-puddles-then-well-think-about-data-lakes-f103111946db?source=rss----4e7e42139f79---4</link>
            <guid isPermaLink="false">https://medium.com/p/f103111946db</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[athena]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Adam Clark]]></dc:creator>
            <pubDate>Fri, 31 Jan 2020 11:41:44 GMT</pubDate>
            <atom:updated>2020-02-01T00:27:26.799Z</atom:updated>
            <content:encoded><![CDATA[<h4>Comic Relief is re-thinking its data ingestion, storage and query stack with Lambda, S3 &amp; Athena. Here is a quick intro to how we are going about it and why.</h4><p>Doing data right is time-consuming and hard! There you go the secret is out. But can we make it easier? Surely that is just part of engineering 101 and we should just accept it, right?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3oEhn4A5PQmhnYD3t6%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F3oEhn4A5PQmhnYD3t6%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F3oEhn4A5PQmhnYD3t6%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/77243b3d91d7b272c0a70735806df7b3/href">https://medium.com/media/77243b3d91d7b272c0a70735806df7b3/href</a></iframe><h3>So what’s the problem</h3><p>The issue for Comic Relief stems partly from the fact that we are now very comfortable with creating lightweight Serverless applications using commodity services (whenever possible), reducing code to the point where very little can go wrong as we are using off the shelf, battle-tested services and applications with the underlying code being basically just lightweight glue. This has given us the drive to reduce our code footprint as much as possible, surely if we write less code then we have fewer bugs :-)</p><p>Say we do all of the data integration work from the backend to our traditional data warehouse to satisfy our stakeholder’s requests, a month later the business needs change and we have to bring new data into the warehouse and backfill. Well, we have to,</p><ul><li>Alter the data contracts, to ensure data continuity and integrity.</li><li>Update the producers to ingest the new data to our staging databases.</li><li>Update the consumer to consume the new data.</li><li>Write the new data to our staging databases.</li><li>Figure out how to shoehorn the data into the warehouse, figuring out across our abstractions where the new data can fit.</li><li>Consume from the staging data and send off to our user matching service.</li><li>Consume from user matching service into the warehouse.</li><li>Backfill the data in a one-off job</li><li>Not forget to write tests</li></ul><h3>Defining our strategy</h3><p>Organisations need to grow and evolve at speed, we need the tooling to be able to stay abreast of those requests and not impose our current belief on what the data we are storing should look like, but rather let the data dictate and us be able to easily satisfy whatever the business needs with ease. So there are two key approach changes that we have made or are in the process of making, these are.</p><h4><strong>1. Commodity Solutions where possible</strong></h4><p>Most of the business problems that we face have been solved. The problem we face is the belief that we can find one amazing solution to solve them all. Breaking down our problems into single use-case applications allows us to focus on our stakeholder&#39;s needs and generally pick an off the shelf solution, unless it is something so specific to our needs that we need to build it, like a <a href="https://medium.com/comic-relief/the-journey-to-90-serverless-at-comic-relief-a79356fb76cd">high volume donation platform</a>.</p><p>The simple summation of this philosophy is, “<em>buy it if every business has the same issue, build if it is a specific problem to just us or gives us a competitive advantage”</em>.</p><p>Taking this approach also means that we can utilise modern integration techniques to remove some of the data shipping work and focus on the bigger issues. Stitch is an awesome tool for this and allows us to ship data from Braintree, Stripe, Salesforce, Freshdesk and many other platforms into S3 in around about 3 clicks. Engineering time is the expensive thing, engineering time can be better spent and I can promise you it costs a lot more than Stitch a month.</p><p>As a team, we need to find our way up the value chain. Automate tasks, level up and don’t do them again. Share components across our frontends, level up. It’s all about automating away the monotony.</p><p>It also means that if one of our solutions aren’t fulfilling the needs of its users, then we just need to satisfy one target stakeholder groups needs to replace the failing part, rather than a massive multi-department consultation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/0*mO_RCNqVh8W4-otW" /></figure><h4><strong>2. Flexible data puddles for the win</strong></h4><p>We have created a producer service written in nodeJS with <a href="https://serverless.com/">Serverless</a> Framework that exposes a private rest endpoint for data consumption. This matches the data to a data contract and then fires the data off into a queue that triggers consumption lambda’s every second.</p><p>Data Stores (Puddle Consumers) then subscribe to this queue and consume based on the individual data stores interests, whether that be transactional data, customer data or any other data.</p><p>Each store defines an <a href="https://aws.amazon.com/glue/">AWS Glue</a> Database and crawlers. The individual store takes the data from the queue and stores it in S3 in parquet format. We then have some CTAS queries running on lambda via events to concatenate and compress the files. Realistically we could have picked Glue Jobs for this use case and might still do, just didn’t feel very Serverless at the time of build.</p><p>We are then able to query the data using <a href="https://aws.amazon.com/athena/">AWS Athena</a>, either in code or via Tableau.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3osxYcKIttf3N5GTsY%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F3osxYcKIttf3N5GTsY%2Fsource.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F3osxYcKIttf3N5GTsY%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="326" frameborder="0" scrolling="no"><a href="https://medium.com/media/d638877262c8a9738e3fe6f097b29b22/href">https://medium.com/media/d638877262c8a9738e3fe6f097b29b22/href</a></iframe><h3>The first failure</h3><p>So, was probably not going to mention the first attempt, but I sometimes forget that you need to fail sometimes in order to learn how to do something properly, it also makes a good story when everything isn’t awesome.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*UBDYdS4rcxEzeFhy" /></figure><p>The first try was to replicate our entire data warehouse in something that looked kind of <a href="https://serverless.com/">Serverless</a> (but 100% wasn’t) which we called the data ingestion pipeline which was using <a href="https://aws.amazon.com/lambda/">Lambda</a> and <a href="https://aws.amazon.com/rds/aurora/">Aurora</a>. The Alarm bells that this wasn’t the serverless masterpiece we had envisaged started to ring when we saw our AWS spend shoot up rapidly during the POC phase.</p><p>I won&#39;t go into to much detail for the sake of your sanity and this article. To sum it up though, the real failure was trying to eat an entire elephant in one go. The end result was getting squashed by the elephant, whereas I have been reliably told the only way to eat an elephant is in small chunks with lots of breaks to digest.</p><p>The above is probably a bad analogy for something I should have known. Iterative cycles, small single-purpose, single responsibility services and all of the rest of the stuff that has been banged into my head for the previous 15 years. But hey, if you aren’t failing, you aren’t learning, right?</p><h3><strong>The first real use case</strong></h3><p>So now for the first real use case, as with our initial foray into Serverless, it wasn’t a big mission-critical requirement. We had an internal request to make a Giro Request form for Schools. The data needed to be stored from the form and an endpoint needed to be exposed to one of our suppliers to be able to fetch a batch of the data, create the forms and send off to the schools.</p><p>We created a simple <a href="https://serverless.com/">Serverless</a> project that took the data, stored that in parquet to S3 and defined the necessary resources in cloud formation to define the database in <a href="https://aws.amazon.com/glue/">AWS Glue</a>.</p><p>We then used an <a href="https://www.npmjs.com/package/athena-client">Athena NPM module</a> to create a private endpoint for the data to be fetched. All in all the approach was super simple and proved that the approach would have some legs.</p><h3>The marginally scary should have picked something smaller second use case</h3><p>So Comic Relief gets a lot of transactions in a short space of time, probably around 4 hours, with the peak coming in around a 45-minute slot at around 9 in the evening. We need to bring totals from our Online Donation platform, SMS providers, IVR and Call Centres and mash them into one massive total.</p><p>Serverless has allowed us to <a href="https://medium.com/comic-relief/the-journey-to-90-serverless-at-comic-relief-a79356fb76cd">confidently take up to 350 donations per second.</a> The problem is we also need to report that to the BBC periodically so that they can show a nice big total on the TV. It is also hugely important to have an auditable trail of how we got to this total, so doing it at a transactional &amp; replicable and thus immutable level makes a lot of sense (and makes finance &amp; compliance happy).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*ggxyKTVEIBSMsLQy.jpg" /></figure><p>This used to be fine as we would just take a running tally. This year we decided that we wanted to be better and be able to provide faster real-time analysis of transaction attribution to call to actions and time-boxing. We also wanted to speed up the financial reconciliation process and allow for transactions to be tracked from the first click all the way through to money in our bank account. We also need to be able to report on this in very near realtime.</p><p>In previous years, we ran multiple systems, with speed layers and analytical layers with duplicated logic to ensure that we could fulfil the business needs. However hard we tried there was always some variance, whereas organisations want absolutes, as divergence suggests inaccuracy, which isn’t what anyone wants.</p><p>So we used the same consumer as in the previous use case and created a transactional store that read messages off of it. Created some Glue Databases and some Athena queries and bam, we had our new way to get fast and very accurate totals. Realistically there are some massive engineering hurdles in the background here and we are doing some caching and aggregation to make stuff work as it should.</p><p>End result though is no servers, no databases and a system that scales with zero effort from us, which we can use and rely on at peak second and then 5 months after with no cost in the middle except for storage.</p><h3>The lessons learned &amp; what’s next</h3><p>The next steps for us are to rinse and repeat the process wherever possible and begin the process of lifting and shifting. So here are the lessons learned,</p><ul><li><strong>You have to get lost in the woods to know how to find your way out of them</strong>. The first attempts didn’t satisfy our needs, however, the underlying learnings took us through to a point where we could satisfy our business needs. This, unfortunately, takes perseverance and that can be hard to justify within an org for business-critical systems.</li><li><strong>We aren’t big data, we get lots of data fast.</strong> Listen to your own needs and not the general use case patterns of the systems that you are implementing.</li><li><strong>Speed to implement, Speed to run &amp; Cost to run. </strong>If the speed to implement is fast, the speed to run is fast enough and the cost to run is low, then why are you not doing it. Fast enough is a key point here and is why I enjoy working with a solid product team so much, the ability to get down to base stakeholder needs vs requests is a rare art.</li><li><strong>Commodity solutions for the win</strong>. Your organisation has needs that make it special, so spend your time on problems that are individual to it. Sure, knitting’s fun for some people, but you’re on the company’s time, go buy the jumper from the store and don’t waste your time knitting a shitty one.</li><li><strong>Data tooling has changed. </strong>Your TV can link up to your thermostat, the same tooling is happening in the systems integration world. Focus on the queries and generating actionable business insights and not spend all your time landing the data in a warehouse.</li><li><strong>Focus on value creation.</strong> Try to automate away the most arduous of your teams&#39; tasks, rip apart the process book and let them focus on the fun stuff that can challenge them.</li><li><strong>Take yourself to your end-user</strong>. The first part of getting in front of a bad process is taking yourself to the process and then finding efficiencies. Iteration takes people on a journey, wholesale replacement is a dictatorship.</li></ul><p>To learn more about our journey to serverless, check out these articles from the <a href="https://medium.com/comic-relief">Comic Relief Digital &amp; Innovation Blog</a> about <a href="https://medium.com/comic-relief/the-journey-to-90-serverless-at-comic-relief-a79356fb76cd?source=collection_home---2------3-----------------------">our journey to serverless</a> and the <a href="https://medium.com/comic-relief/aws-summit-london-case-study-going-serverless-at-comic-relief-51a22c15e398">business value it has created</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f103111946db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comic-relief/starting-out-with-data-puddles-then-well-think-about-data-lakes-f103111946db">Starting out with data puddles, then we’ll think about data lakes</a> was originally published in <a href="https://medium.com/comic-relief">Comic Relief Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating from a conventional CMS to a decoupled technology stack!]]></title>
            <link>https://medium.com/comic-relief/migrating-from-a-conventional-cms-to-a-decoupled-technology-stack-6114cb97aaf5?source=rss----4e7e42139f79---4</link>
            <guid isPermaLink="false">https://medium.com/p/6114cb97aaf5</guid>
            <category><![CDATA[gatsbyjs]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[jamstack]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[contentful]]></category>
            <dc:creator><![CDATA[Gustavo Liedke]]></dc:creator>
            <pubDate>Tue, 28 Jan 2020 14:35:52 GMT</pubDate>
            <atom:updated>2020-01-28T17:28:48.621Z</atom:updated>
            <content:encoded><![CDATA[<h4>How we migrated the main Comic Relief site over a 5 month period from Drupal 8 to a headless CMS and a more modern software stack (aka, JAM stack).</h4><h4>From “serverness” to serverless 🚀</h4><h4>Why move away from Drupal?👋</h4><p>Initially, we thought we could iterate on our Drupal platform to move to the headless model. From Drupal 7 to Drupal 8, we have always been big advocates of Drupal and avid contributors to its community. After a long and happy relationship, we have decided to let it go and move on, despite having a very strong and capable platform, some of the constraints were becoming too much for us to keep the old stack.</p><blockquote>“Regular core updates for security fixes, sometimes vital modules are no longer supported in the new version, or no longer maintained at all, requiring some bespoke patching.”</blockquote><blockquote>“Tied into the supplied template engine (Twig): flexible but often requires a lot of pre-processing and digging into render arrays to get exactly what is required for any one piece of work/design. Even simple fields provide a lot of stuff to the FE, which is customisable to some extent but still time consuming”</blockquote><blockquote>- Andy Phipps, Senior Eng.</blockquote><h3>What’s JAM stack?</h3><blockquote>“A modern web development architecture based on client-side JavaScript, reusable APIs, and prebuilt Markup”<br>- Mathias Biilmann (CEO &amp; Co-founder of Netlify).</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*KtqbRFyyRx5FaCOe9dFHXQ.gif" /><figcaption>To learn more about it check out <a href="https://jamstack.org/">https://jamstack.org/</a> .</figcaption></figure><h3>Why did we decide to go JAM?</h3><p><strong>Firstly</strong>, Part of our core strategy is to use Serverless technology wherever possible to reduce stress and the potential scaling issues that <a href="https://en.wikipedia.org/wiki/Red_Nose_Day_2019">our night on TV creates</a>. Read Caroline’s article (our Product Lead), to understand more about our approach “<a href="https://medium.com/comic-relief/the-real-business-value-comic-relief-has-unlocked-by-going-serverless-6c68991ab030">The real business value unlocked by going Serverless</a>”.</p><p><strong>Secondly</strong>, we wanted to focus on the core foundations of the web experience. We want to deliver a fast, reliable and accessible website to our users. Security, Performance, best practice and A11y are the attributes we value the most on every application we build. JAM stack provides us with all the tools we need to achieve that.</p><figure><img alt="lighthouse report" src="https://cdn-images-1.medium.com/max/756/1*SjcgOXoZZThLNqxhCYD9iw.png" /><figcaption>From very early on the project we kept those reports 100% as our benchmark.</figcaption></figure><p><strong>The cost was a big factor too. </strong>Our server cost was very high.<strong> </strong>To host our Drupal site, we were spending around £ 70k a year. Hosting static files in CloudFront, we are on track to reduce our costs by around 90% while improving reliability, scalability and performance.</p><p><strong>Engineer Experience. </strong>The experience developing in a JAM stack is quite delightful. Our new setup is very attractive. We believe that to bring the best talent is important to stay relevant and updated with the latest technologies such as <strong>React</strong> and <strong>GraphQL</strong>. ( <a href="https://app.beapplied.com/org/comic-relief">we are hiring</a> 😃)</p><blockquote>“Contentful is very user-friendly, without much overhead on development.”</blockquote><blockquote>“User experience improved a lot through serving less demanding static content.</blockquote><blockquote>- Mohamed Labib, Senior Eng.</blockquote><p><strong>Faster development time. </strong>Developing in headless CMS setup makes things easier and faster. Engineers do their work very independently and were able to add more components into the CMS than we previously achieved on the Drupal platform.</p><blockquote>“Aligning the front end across our products and using Contentful has allowed us to give much greater flexibility for content management — tasks which previously had to be developed in code are now achievable for our content management team. This frees up engineering team time to work on improving accessibility and iterating on components to continually improve the site.”</blockquote><blockquote>- Caroline Rennie, Product Lead</blockquote><h3>New comicrelief.com 🎉</h3><p>To manage our content we use <a href="https://www.contentful.com/"><strong>Contentful</strong></a> as our headless CMS. To generate our frontend app we use <a href="https://www.gatsbyjs.com/"><strong>Gatsby</strong></a>. To style our react component we use <a href="https://styled-components.com/"><strong>styled-components</strong></a>. To build our Component Library we use <a href="https://react-styleguidist.js.org/">Styleguidist</a>. To deploy our app <a href="https://concourse-ci.org/"><strong>Concourse CI / CD</strong></a> and to host our files we use <a href="https://aws.amazon.com/cloudfront/"><strong>AWS CloudFront</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*HEJ4L_aS9FD4CiCXod4K-A.gif" /><figcaption>Image inspired by <a href="https://www.gatsbyjs.org/">Gatsbyjs.org</a></figcaption></figure><p><strong>Contentful</strong>. What once was different Drupal backends to manage content of our campaign websites. Now we have one content hub, very user-friendly, to deliver content to different applications across our organisation.</p><p><strong>Gatsby</strong>. We wanted a framework to help us to focus on what matters the most to us, which is <strong>delivering content to our users</strong> in an interactive, clean and transparent way. And that’s what Gatsby provide us with.</p><blockquote>Gatsby gives you easy access to features like modern Javascript syntax, code bundling and hot reloading, without having to maintain custom tooling. Build app-like experiences faster — with Gatsby.</blockquote><p><strong>Styled-components. </strong>Our CSS-in-JS library of choice made building our react components very fun. Once we made the transition from “modifiers CSS classes” to <strong>props</strong> we could be very creative and flexible with our components.</p><blockquote>It’s very interesting to use props instead of classes to customise the behaviour of a component. All your styles are contained within your component so there is less risk to overwrite the style of other components. I really like the Jest integration and features like css nesting.</blockquote><blockquote>- Louis Tchamegni, Eng.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*Lu3CvqsfESkcVjBDEqFdAQ.png" /></figure><p><strong>Styleguidist</strong>. To develop our react components in isolation we chose Styleguidist. It helped us to keep our code documented and tested. Easy way to share with the team and show our components variations and interactivity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*vt65MOuKn4f76Lk8s-LY6Q.gif" /></figure><p><strong>ConcourseCI</strong>. We have been using concourse CI for quite some time. We love to see our pipeline executing all our jobs, tasks, tests and notifications making sure our code is reliable and our site doesn’t degrade.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*pmwbeohZw99ZoJQ7LmvYWA.gif" /></figure><p><strong>Tests. </strong>We use <a href="https://github.com/styled-components/jest-styled-components">jest-styled-components</a> to snapshot our component once it’s been tested. And in the app level, we use <a href="https://www.cypress.io/">Cypress</a> for our end-to-end testing.</p><p><strong>CloudFront</strong>. Is our CDN (it speaks for itself).</p><h3>Final thoughts</h3><p>My experience in this project was great. I’d say if we were not using this selection of tools, frameworks and libraries we would not achieve what we did in the timeframe we had. Building our react styled-components in isolation on Styleguidist was a great way to keep us working in the frontend while we were still working out our backend integration.</p><p>Using GraphQL in Gatsby was so effective to pull the data we need and how we need it. Contentful and Gatsby work really well together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*XYlORKpTg6Y_iL7vy-uEhg.png" /></figure><p>Contentful <a href="https://github.com/contentful/rich-text">richText</a> field made it possible for us to use our “atoms” in our content models body fields.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*lcr_rp3N-MpDRlt5DcfPfQ.png" /></figure><p>And Gatsby just pulls it all together very nicely.</p><p>Our concerns were rapidly resolved. At first, we were worried about having to <strong>build our site for every content update</strong> but in the end, it meant we test every content that goes to production.</p><p>How did we <strong>migrate our content from Drupal to Contentful</strong>? We managed to migrate the majority of our content using <a href="https://github.com/contentful/contentful-migration/blob/master/README.md#reference-documentation">contentful-migration</a>. Having more them 3k pages, our initial build time was around 10mins. But with the latest Gatsby updates and some performance improvements that we made in our build process, we went down to <strong>3mins</strong>.</p><p><strong>Previewing content</strong>? <a href="https://www.gatsbyjs.com/preview/">Gatsby preview</a>! It’s a quite recent feature, Gatsby team is doing an awesome job improving it almost day by day. Gatsby preview provides us with a temporary URL allowing us to preview our content changes almost immediately.</p><p>What’s even better now is how fast and smart we can deliver value to our product and it’s end-users. The team and our stakeholder are happy :)</p><p>That’s an overall of our new setup. I will be going in-depth on the use of each technology in future articles. Meanwhile, if you have any question about our tools and processes let me know! I will also be attending to <a href="https://reactsummit.com/">React Summit</a> in Amsterdam in case you wanna have a chat! Our friends can get 15% discount <a href="https://www.eventbrite.com/e/react-summit-amsterdam-2020-tickets-68861704211?discount=ComicRelief">here</a> 😃. Or to apply for Diversity scholarships for React Summit 2020 <a href="https://medium.com/@ReactAmsterdam/diversity-scholarships-for-react-summit-2020-are-now-available-468e6bb15f23">visit</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6114cb97aaf5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comic-relief/migrating-from-a-conventional-cms-to-a-decoupled-technology-stack-6114cb97aaf5">Migrating from a conventional CMS to a decoupled technology stack!</a> was originally published in <a href="https://medium.com/comic-relief">Comic Relief Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bringing our Comic Relief Values to life with IoT and Serverless]]></title>
            <link>https://medium.com/comic-relief/bringing-our-comic-relief-values-to-life-with-iot-and-serverless-782fb5f9ae5e?source=rss----4e7e42139f79---4</link>
            <guid isPermaLink="false">https://medium.com/p/782fb5f9ae5e</guid>
            <category><![CDATA[product]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Peter Vanhee]]></dc:creator>
            <pubDate>Fri, 06 Dec 2019 16:40:32 GMT</pubDate>
            <atom:updated>2019-12-10T11:36:13.751Z</atom:updated>
            <content:encoded><![CDATA[<h3>Bringing our Comic Relief Values to life with IoT, Amplify and Serverless</h3><p>Seven values, from “Break down Barriers”, to “Have a sense of humour, always”, together with a wish from our director Charlotte to bring our values to life, proved to be enough inspiration to spend a fun afternoon hacking together a live values dashboard.</p><p>The idea? Put an <a href="https://www.amazon.co.uk/All-New-AWS-IoT-Enterprise-Button/dp/B075FPHHGG">Amazon Dash</a> button on each card, make people vote for the value they’ll carry out throughout the day by pushing the button on the card when they enter the office, and display a live dashboard of “most voted values”.</p><p>While our dashboard certainly piqued the interest of colleagues, ranging from “this is cool!”, to mild curiosity, to “what the hell is this?”, we really did this as a fun exercise in turning a simple idea into an actual scalable product in a short time frame and using our <a href="https://medium.com/comic-relief/serverless/home">familiar serverless stack</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AFtLM7rknEEpDYh2n0FT4w.jpeg" /><figcaption>Our seven Comic Relief Values.</figcaption></figure><h3>Our tools</h3><p>This is how we connected it all up:</p><ul><li>Seven <a href="https://www.amazon.co.uk/All-New-AWS-IoT-Enterprise-Button/dp/B075FPHHGG">AWS IoT enterprise buttons</a> we purchased and registered to our AWS account, with each button associated to one of our seven values.</li><li>A DynamoDB table to store all the votes</li><li><a href="https://aws.amazon.com/appsync/">AWS AppSync</a>, so we can use GraphQL to update our DynamoDB table and streams updates to the front-end using subscriptions</li><li>A Lambda function which connects the IoT button click and runs a GraphQL mutation update DynamoDB</li><li>A ReactJS front-end bootstrapped using create-react-app and using our Comic Relief component library.</li></ul><h3>Real-time values dashboard</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jlgviAf1zQIqyGOMsjAKBA.gif" /><figcaption>The dashboard in action.</figcaption></figure><p>It takes approximate 3–4 seconds from button push to dashboard update, and most of that time is spent by the dash button connecting to the WiFi and sending the payload to the Lambda function, with &lt;1sec to update DynamoDB and stream the updates to the front-end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*wysN5cCL0dNAiGgI2KhpWQ.gif" /><figcaption>In real life.</figcaption></figure><p>To get some insight into usage, we hooked it up to <a href="http://www.serverless.com/">Serverless dashboard</a> which will send out alerts to Slack when there is lots of activity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*IZzOTBotsFBzmeK8LhFMTw.png" /><figcaption>Using <a href="https://serverless.com/dashboard/">Serverless dashboard</a> to get a bit of insight into button pushes.</figcaption></figure><h3>How we built it</h3><p>We used <a href="https://docs.aws.amazon.com/amplify/">AWS Amplify</a>, and one of the things that surprised me most is the speed at which we got this all running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yug2bdNZNt9ed72fFH_9Ag.png" /><figcaption>Architecture diagam.</figcaption></figure><p>First of all, let’s initialise our app</p><pre># Create a new react app<br>npx create-react-app cr-values<br>cd cr-values</pre><pre># Initialize amplify<br>amplify init</pre><p>Now, adding a GraphQL API is as simple as</p><pre>amplify add api</pre><p>The only input you need to give is the GraphQL schema, which in our case was really simple. We associate a random ID with a buttonType (any of the seven values), a clickType (single click, double click or long click), and a timestamp.</p><pre>type ButtonPush @model {<br>  id: ID!<br>  buttonType: String!<br>  clickType: String<br>  created: AWSTimestamp<br>}</pre><p>Amplify automatically handles the entire API creation, and generates GraphQL queries, mutations and subscriptions you can use in your ReactJs front-end.</p><p>Now, we need to connect button pushes to a GraphQL mutation using Lambda. The event contains which button was pushed, and the clickType. We then send this to GraphQL using the aws-appsync SDK.</p><pre>const push = async (event, context) =&gt; {<br>  const { placementInfo, deviceEvent } = event;<br><br>  const mutation = gql`<br>    mutation createButtonPush {<br>      createButtonPush(input: { <br>        id: &quot;${uuid()}&quot;, <br>        buttonType: &quot;${placementInfo.attributes.buttonType}&quot;,<br>        clickType: &quot;${deviceEvent.buttonClicked.clickType}&quot;,<br>        created: ${new Date().getTime()} <br>      }) {<br>          id<br>          buttonType<br>          clickType<br>          created<br>      }<br>    }<br>  `;<br><br>  await graphqlClient.mutate({ mutation });</pre><pre>  return {<br>    statusCode: 200,<br>    body: &#39;Button clicked&#39;,<br>  }),<br>};</pre><pre>export default push;</pre><p>For our front-end, we have a simple React component which uses the useEffect hook to run a subscription operation and store button pushes using useState so we can re-render our component.</p><pre>const [newPushes, setNewPushes] = useState([]);</pre><pre>useEffect(() =&gt; {<br>  <strong><em>API</em></strong>.graphql(<br>    <strong><em>graphqlOperation</em></strong>(subscriptions.<strong><em>onCreateButtonPush</em></strong>)<br>  ).subscribe({<br>    next: (todoData) =&gt; setNewPushes(todoData.value.data.onCreateButtonPush)<br>  });<br>}, []);</pre><p>The actual code is a little more complicated, as we use another effect to run GraphQL list operation to populate initial state, and re-render upon changes of the newPushes.</p><p>And that’s all — we proved that with the right tooling it’s incredibly quick to build something useable. We’re excited about this tech stack (AWS Amplify is something I’ll personally explore a lot more going forward) and will certainly do a few more iterations with our colleagues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V0_vHId6qSJjmtzYPz2Rew.png" /><figcaption>Dashboard snapshot.</figcaption></figure><p><em>Massive thanks to </em><a href="https://twitter.com/adamclark_dev"><em>Adam Clark</em></a><em> and </em><a href="https://twitter.com/gusliedke"><em>Gustavo Liedke</em></a><em> for building this together in an afternoon and learning from each other, and </em><a href="https://twitter.com/crashtherocks"><em>Charlotte Hillenbrand</em></a><em> for launching our Comic Relief values and providing us with the inspiration to build this.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=782fb5f9ae5e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comic-relief/bringing-our-comic-relief-values-to-life-with-iot-and-serverless-782fb5f9ae5e">Bringing our Comic Relief Values to life with IoT and Serverless</a> was originally published in <a href="https://medium.com/comic-relief">Comic Relief Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we made our Serverless payments platform CMS driven]]></title>
            <link>https://medium.com/comic-relief/how-we-made-our-serverless-payments-platform-cms-driven-ec793ba6a191?source=rss----4e7e42139f79---4</link>
            <guid isPermaLink="false">https://medium.com/p/ec793ba6a191</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Adam Clark]]></dc:creator>
            <pubDate>Mon, 29 Jul 2019 09:06:25 GMT</pubDate>
            <atom:updated>2019-07-29T10:17:38.589Z</atom:updated>
            <content:encoded><![CDATA[<p>Diving into how we made our payments platform CMS driven while maintaining the ability to happily scale to 300 donations per second using <a href="https://www.contentful.com/">Contentful</a> and <a href="https://webpack.js.org/">Webpack</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R989Tf1YTMNT9kFIiScpqQ.png" /><figcaption>The donation platform homepage</figcaption></figure><h3>The Problem</h3><p>During a cross directorate away day with our fundraising team, one of the frustrations that came out of our group was the process for creating and managing carts within our donation system.</p><p>For all of our campaigns, we create different carts to,</p><ul><li>Change imagery and money-buys (amounts and their descriptions) to reflect the campaign asks</li><li>Track the activity and performance of the campaign</li></ul><p>The old process was for a member of the fundraising team to submit a form to our team, we would then manually edit a JSON file which would then be put into a pull request, reviewed and would then go through our CI/CD pipeline. This was a massive pain for developers as usually required context switching, lots of back and forth between teams and always felt like the process could be simplified for all parties involved.</p><h3>The Requirements</h3><p>We had the following requirements from whatever solution we put in,</p><ul><li>Due to the scalability needs of our platform, the solution would need to be able to be statically compiled and not add extra dependencies.</li><li>Editors would need to be able to preview their changes in near-real-time before being published into production.</li><li>It would need to work in a local development environment.</li><li>All changes would need to be automated and tested before hitting production.</li></ul><h3>How we achieved it</h3><p>We use contentful to handle a load of our content requirements and are currently in the process of moving our main website over to it (with <a href="https://www.gatsbyjs.org/">Gatbsy</a>) from Drupal 8. It was a fairly obvious route to just set up a new space in contentful and plumb it in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LMiMrr0Pkmmka-SSUJacpA.png" /><figcaption>Managing Carts on The Contentful CMS</figcaption></figure><p>The first step was to define the content model and then migrate over the current cart ids from the JSON file. <a href="https://www.contentful.com/">Contentful</a> allows you to do this very quickly and is very intuitive for developers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o8AjHBcEYj3zj6_GAtwMqA.png" /><figcaption>Managing a cart in Contentful</figcaption></figure><p>The next step was to alter our React <a href="https://www.contentful.com/">Webpack</a> configuration to pull the configuration from <a href="https://www.contentful.com/">Contentful</a> and build this into a replacement for the config.json file (We are huge fans of the way that <a href="https://www.gatsbyjs.org/">Gatsby</a> statically compiles, as is a perfect fit for our traffic profile).</p><p>We created a script to get the configuration from <a href="https://www.contentful.com/">Contentful</a> and then write the cart.json file ready for Webpack to build it. We then edited our applications start and build scripts to run the fetch script before a build or develop command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*fOzlf4sLf6EA6Raa2bv1bg.png" /><figcaption>create react app build script with the fetch cart configuration command now at the top</figcaption></figure><p>The next step was to add some configuration to automatically re-fetch the carts and look for differences every 10 seconds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vqj8xpT6STxqiTVX2MA5rA.png" /><figcaption>Automatic reloading of carts</figcaption></figure><p>We could now change a cart in <a href="https://www.contentful.com/">Contentful</a> and see it change automatically within the 10 seconds that we defined. The big question now was how could we do this in a preview environment that our content editors could easily use, seeing their changes immediately.</p><p>So, first thought was we could use EC2 running a node server. However, this goes against our philosophy of managing as few a servers as possible if we really don’t need to. We settled on a really simple (and free) solution in <a href="https://www.heroku.com/">Heroku</a>.</p><p>We created a Procfile in the base of our react application and then configured Heroku to automatically deploy off of our master branch from the <a href="https://www.contentful.com/">Contentful</a> preview API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*4q5h7UoYMEupbwnq1Xny-Q.png" /><figcaption>Our super simple application Procfile</figcaption></figure><p>The end result was that we were able to edit our frontend in near-realtime prior to publishing content, giving editors immediate feedback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*oUv57FqDxBuDE1F7.gif" /><figcaption>Editing content in realtime on our donation platform</figcaption></figure><p>After this, we updated our CI/CD pipeline to include the new environment variables and fixed all of our unit and integration tests.</p><p>We then needed to find a way to trigger our pipeline whenever content edits happened. We use <a href="https://concourse-ci.org/">Concourse</a> and found <a href="https://github.com/bigorangemachine/concourse-contentful-webhook">this repository</a>, that would check <a href="https://www.contentful.com/">Contentful</a> constantly and trigger the pipeline whenever there was an update. Unfortunately, it was looking at revisions and did not seem to be getting the latest updated content, so we forked it and made the changes that we needed in <a href="https://github.com/comicrelief/concourse-contentful-webhook">this repository</a> to look at the latest updated data and use the timestamp to trigger to a build.</p><p>All we needed to do then was to add a webhook to <a href="https://www.contentful.com/">Contentful</a> to trigger the resource change and we were good to go.</p><p>We also needed to add another pipeline to backup our <a href="https://www.contentful.com/">Contentful</a> configuration and also to move content type configuration from a <a href="https://www.contentful.com/">Contentful</a> Staging Environment to its master environment. We did all this with a couple of <a href="https://concourse-ci.org/">Concourse</a> tasks and the awesome <a href="https://github.com/contentful/contentful-cli">Contentful CLI tool</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0znSx39QINEyWHVKRa_mmA.png" /><figcaption>The Donation frontend deployment pipeline</figcaption></figure><h3>The Learnings</h3><ul><li>Once upon a time, making something content managed was a big piece of work. It isn’t anymore.</li><li>Content management as a service can easily be part of a Serverless application. It only took near enough a day and a half to implement this, much of which was pipeline integration.</li><li>5 minutes to production can be fine for CMS edits, as long as your stakeholders can see their changes live in a preview environment.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec793ba6a191" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comic-relief/how-we-made-our-serverless-payments-platform-cms-driven-ec793ba6a191">How we made our Serverless payments platform CMS driven</a> was originally published in <a href="https://medium.com/comic-relief">Comic Relief Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Philanthropy: My experience at Nectar’s Data Swarm Event]]></title>
            <link>https://medium.com/comic-relief/data-philanthropy-my-experience-at-nectars-data-swarm-event-e0a620e2714?source=rss----4e7e42139f79---4</link>
            <guid isPermaLink="false">https://medium.com/p/e0a620e2714</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[philanthropy]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[tech-for-good]]></category>
            <dc:creator><![CDATA[Florence Hope]]></dc:creator>
            <pubDate>Mon, 22 Jul 2019 21:01:04 GMT</pubDate>
            <atom:updated>2019-07-22T21:01:04.546Z</atom:updated>
            <content:encoded><![CDATA[<p>Of the many ways to support charities and the causes we believe in, financial donation is both the most straightforward and the most prevalent.<a href="https://www.cafonline.org/docs/default-source/about-us-publications/caf-uk-giving-2019-report-an-overview-of-charitable-giving-in-the-uk.pdf"> 57% of the adult UK population gave directly to a charity in 2018</a>, compared to 16% of us volunteered in some capacity. While harder to quantify, the value of donated time can be equally high, with many organisations relying on volunteers to deliver their core programmes of work. When we think of charitable volunteering, we tend to imagine event orchestration and front-line work. Consultancy also constitutes a significant part of the volunteering landscape, with many charities leveraging corporate partnerships to plug internal skills gaps and <a href="https://reachvolunteering.org.uk/">Skills-based initiatives such as Reach</a> exist to connect specific individuals with the organisations that need them.</p><p>Nectar’s Data Swarm, facilitated in tandem with CSR &amp; Philanthropy agency <a href="https://www.thegivingdepartment.com/">The Giving Department</a>, is an interesting example of highly skilled volunteering in which more than 80 analysts and data scientists from Nectar and Sainsbury’s assemble for a Hackathon designed to solve the data problems of small to medium sized charities. At Comic Relief we have a comprehensive data warehouse and three in-house analysts to report and advise on a national television, media, digital, and analogue community fundraising campaign, as well as tailor communications to email and postal audiences. While in the private sector our data setup would likely be considered prohibitively lean for an organisation of our size, revenue, and diverse activity portfolio, we are comparatively fortunate; many charitable organisations lack in-house data professionals entirely, and as such are throttled in their ability to fully leverage the data at their fingertips. As a firm believer in the power of effective data analysis to drive better decisions and de-risk innovation and change I was keen to see how a charitable donation of data skills at mass scale would play out in practise. I volunteered myself as a participant for the event as soon as I heard about it and a few weeks later I was in the basement of Sainsbury’s London Headquarters with a borrowed laptop, part of a small unit attempting to build a predictive model from an open data set in six hours in benefit of Missing People UK.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9yO0LjSL6tEjTchI-wom7Q.png" /><figcaption>A mid-hack standup</figcaption></figure><p><strong>The Data</strong></p><p>Our task — carefully and impressively scoped in advance of the event by the team at <a href="https://www.missingpeople.org.uk/">Missing People</a> — focused on the <a href="https://cls.ucl.ac.uk/cls-studies/millennium-cohort-study/">Millennial Cohort Study</a>, a three-yearly survey of 18k+ consistent participants who were born in the UK in the year 2000 or 2001. It’s a very comprehensive piece of data collection; thousands of questions touching several areas of a child’s lived experience including education, home environment are asked in face-to-face interviews with both children and parents every time a sweep is conducted. In the 2015 study, Missing People were able to include a question to ask whether a teenager had stayed out overnight without parents knowing where you are in the last 12 months. Of the participants who were asked this question, 842 of 11,593 answered positively.</p><p>This formed the basis of Missing People’s ask to Nectar and Sainsbury’s: if we take staying out overnight without parental consent as an instance of going missing, we have a wealth of variables within the 2015 study and all prior and subsequent sweeps against which to correlate this target and provide a richer portrait of those teenagers who do go missing, and the factors that precipitate the event. What can we learn from this rich and varied dataset?</p><p>Before the day, engineers from Sainsbury’s loaded the data from the 2012 and 2015 surveys into a Snowflake data warehouse. We were given burner Snowflake accounts, data dictionaries, and Jupyter notebooks with which to access the database. This excellent level of preparation by hack day facilitators allowed participants on the morning to get right into the meat of the analysis as soon as the hackathon began.</p><p><strong>The Task</strong></p><p>Six small to medium charities took part in the event, with analysts being assigned to solving the data problems of <a href="https://www.onsideyouthzones.org/">Onside</a>, <a href="https://www.bouncebackproject.com/">Bounce Back</a>, <a href="https://www.togetherforshortlives.org.uk/">Together for short lives</a>, <a href="https://piponline.org.uk/">Pursuing Independent Paths</a>, <a href="https://www.missingpeople.org.uk/">Missing People</a>, or <a href="https://annefrank.org.uk/">The Anne Frank Trust</a> prior to the event. On the morning, the large team assigned to Missing People’s questions was sub-divided into smaller units, most looking at correlations describing the likelihood to go missing within different modules of the 2015 survey itself; i.e. how does a child’s experience with crime affect their likelihood to go missing? What about their personal wellbeing, education, happiness, propensity to take risks?</p><p>My sub-team’s task was a little bit different. We decided to look at the 2012 study, with the question; ‘given the data collected when a child was 11, can we predict whether they will go missing at 14?’ and attempt to build a predictive model. The practical implications of such a model are tangible. If such a predictive link between the two datasets exists, we may be able to use it as the basis for a set of questions to ask 11 year olds and understand their level of risk and facilitate an early intervention approach.</p><p><strong>The Outcome</strong></p><p>In Data Science, the lion’s share of the effort is often in iteration and feature engineering. You have a target outcome and some number of variables which potentially predict that outcome; assuming the problem is predictable, there exists some secret recipe which best combines variables and combinations of variables, model types, and parameters that will provide the optimal estimative relationship between your input variables and your target field. It’s unlikely you will ever find the absolute optimal solution to the problem space. In commercial Data Science, the decision to put a model into production is usually made when the modelling project has reached such a point of diminishing returns that the Data Scientist’s time would be more valuable focused elsewhere. Within six hours you are unlikely arrive at that inflection point unless your problem space is tiny. My goal therefore was to establish a baseline for the problem — <em>is </em>the target predictable given the inputs, and what features might contribute to a strong estimation?</p><p>Given the time constraints, my approach was a crude one: join all the tables together and use a Random Forest model<strong> </strong>(known for strong, out-the-box performance on poorly understood datasets) to understand the overall predictability and isolate some of the most predictive variables before moving on to a more targeted modelling phase. Due in part to spending the morning chasing a red herring and significant time outage due to getting locked out of the Sainsbury’s network I wasn’t able to move to the next step, which would have been honing in on features and playing with different model types to achieve a better outcome. At the end of the hack day, I had produced a model with 94% accuracy, but low possibility for utilisation: it achieved its strong results by predicting that almost no one was going to run away, a common problem when trying to predict a binary target with unbalanced results. This would make it unsuitable for identifying risk in target populations. Fortunately a member of my subteam took a smarter approach to her process, and was able to construct a decent logistic regression model and provide Missing People with some key inferences and factors.</p><p>Despite the lack of a strong model, I was able to infer some valuable points from the exercise:</p><p>- It appears it <em>is</em> overall possible to predict with reasonable accuracy whether a child will go missing at age 14 from questions they are asked at age 11. Given that reasonable models were produced in a few hours, it would be worth investing further time in building a model to inform an early intervention approach using the Millenial Cohort questions.</p><p>- The survey question with the highest weight in both models we created was ‘how long did the interviewer spend in the house?’. At first this seemed bizarre, but the fact that two completely different models reached the same conclusion lent credence to the finding. Our working hypothesis is that the time it takes to ask a family thousands of questions provides a reflection of the level of chaos within that household. This hypothesis could be carefully examined under less time-constricted conditions.</p><p>- Finally, the areas of the Millennial Cohort study from which my model picked its most predictive features were related to the attitudes, cognitive and behavioural issues present in the parents, and not the child. This finding made me a bit sad, illuminating the well-known notion that problem behaviours in children are often a reflection of their home environments.</p><p><strong>Overall Impressions</strong></p><p>While my model didn’t work as well as I would have liked it to, I left the day feeling hugely exhilarated by the energy, organisation and, most importantly, tangible impact of the day. Among other deliverables, Missing People had a treasure trove of information to work with, Onside had data-driven backing for their current strategy in increasing gender diversity, and Pursuing Independent Paths walked away with a dashboard that proved the value of the work that they were doing, automated by the Swarm analysts to provide ongoing measurement. The day left me with a conviction that skills should be part of the charity donation fabric, and I’m excited to see how Data Philanthropy can be integrated into the existing partnership between Nectar, Sainsbury’s, and Comic Relief.</p><figure><img alt="Data Swarm subteams at work" src="https://cdn-images-1.medium.com/max/1024/1*L0FZ-Tf-so24UxKSi10DJw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0a620e2714" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comic-relief/data-philanthropy-my-experience-at-nectars-data-swarm-event-e0a620e2714">Data Philanthropy: My experience at Nectar’s Data Swarm Event</a> was originally published in <a href="https://medium.com/comic-relief">Comic Relief Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The real business value Comic Relief has unlocked by going Serverless]]></title>
            <link>https://medium.com/comic-relief/the-real-business-value-comic-relief-has-unlocked-by-going-serverless-6c68991ab030?source=rss----4e7e42139f79---4</link>
            <guid isPermaLink="false">https://medium.com/p/6c68991ab030</guid>
            <category><![CDATA[product]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Caroline Rennie]]></dc:creator>
            <pubDate>Tue, 09 Jul 2019 17:59:53 GMT</pubDate>
            <atom:updated>2019-07-09T17:57:40.630Z</atom:updated>
            <content:encoded><![CDATA[<h3>The real business value unlocked by going Serverless</h3><p>I recently spoke at the AWS Summit London about Comic Relief’s journey into Serverless (<a href="https://medium.com/comic-relief/aws-summit-london-case-study-going-serverless-at-comic-relief-51a22c15e398">full video of Serverless Deep Dive session here</a>). During some press interviews at the conference, myself and Adam Clark (Senior Engineer) got to speak in a bit more detail about some of the more nuanced advantages that going Serverless has unlocked for our organisation. This article will address how, by taking advantage of commodity services from AWS, we’re now able to focus our team’s energy and output on problems which are unique to our organisation.</p><p>Moving the majority of our user-facing tech to Serverless has led to some incredible cost savings, but that’s not where the real business value has been unlocked for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yLGVyOr3V1HQiarnzdVLDw.png" /><figcaption>Peak AWS bill in March 2015 VS March 2019 costs — 93% saving</figcaption></figure><p>For many years, the digital team at Comic Relief were stuck in what we call the “campaign-cycle”; we have a huge national event every year and the approach to delivering sites and services was to build — almost from scratch — the new campaign’s digital products. In 2016, an effective strategy was set so that we could focus on reducing repetition and reducing costs. Over a 3 year period, this led us to adopting Serverless for more and more of our tech stack and resulted in a 93% reduction in our AWS bill (March 2015 VS March 2019). Head of Technology, Peter Vanhee, describes this process as <em>“reducing the cost of ownership”</em> which is having significant impact on the total cost of operations within our technology department. The savings are incredible but not, in my opinion, as valuable as the opportunities that increased velocity and speed of delivery has opened up.</p><h3>Moving from reactive to proactive</h3><p>It may seem obvious, but automation and reusability are key to giving your team and organisation head space to look up from the daily grind, identify high value opportunities and chase after them. At Comic Relief, there’s always been -and will always be- an attitude that you’ve got to pounce on an opportunity. In other departments this may mean grabbing some time with a celebrity on a press tour or pulling together creative social content last minute, but our technology team can’t always be that adhoc. For us to move from reactive and defensive development patterns — the kinds that filled our products with levels of tech debt which made them non-viable for another 12 months — we had to get smarter about when to build and when to buy.</p><p>Commodity tech services have allowed our team to focus on business value. Utilising technology which handles basic operational needs enables us to tackle problems which are unique to our organisation and allows for a lighter touch ops function which has been democratised across the team. Where we previously had 2–3 person webops team, serverless has enabled us to repurpose those roles into full-stack engineers who are helping us build solutions to our unique business problems.</p><h3>What does that mean in practice?</h3><p>Where previously our team’s remit was almost entirely supporter-facing tech, we are now able to flex our full <em>-marginally pretentious-</em> directorate title of Digital <strong>&amp; Innovation</strong>. As an organisation, we have many years of what I would describe as “Process debt” — the kind of problems that arise from adhoc and reactive processes which have not been effectively refactored. We successfully utilised the latest tech for our donation platform, but when it came to other business processes, we were stuck with systems which relied far too much on human-powered data modification rather than automation.</p><p>By making smarter choices around what to build vs what to buy — whether that’s utilising services on offer from AWS over our own infrastructure or choosing Just Giving over developing our own fundraising pages — we have opened up more problem spaces within our organisation for us to innovate. While “process debt” may not sound like the most appealing area to innovate, the opportunities which will be unlocked once we have inter-operable systems are huge. Previously, our team were only focused on the public fundraising activity of our organisation, now we’re able to look at the bigger picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vRw-e1NZHneo0xjplYqWug.png" /><figcaption>Comic Relief is best known for TV appeal show and celebrity engagement — we’re now able to use tech to better represent our innovative grant making, impact monitoring, influencing work and storytelling.</figcaption></figure><p>One example of these new opportunity spaces would be the disconnect between our supporters and the projects we fund. This stems from our biggest communications format (our Night of TV), which historically lends itself to over-simplification of how funding projects works. In the past few years, our broadcast team have worked hard to change the narrative and try to convey the nuances of the way the public’s money is helping create a just world free from poverty. Without easy access to data and case studies, our storytelling team can’t possibly convey the true breadth of impact that Comic Relief’s funding has. We believe it is these problems where innovation and digital transformation will have the greatest effect on our organisation.</p><h3>So, what’s next?</h3><p>We’ve made changes to the areas of focus for our product team — where we previously were focusing across the different areas of public fundraising (website, donation platform, shop, etc.), many of these products are now at a level of maturity where we can be slightly lighter touch. As such, I have been able to rebalance my team to focus on some of those new areas of innovation:</p><ol><li>Developing fit-for-purpose tools and processes for our impact investments</li><li>Building trust through data in our storytelling</li><li>Creating new experiences and products to raise money</li></ol><p>For each of these areas, we’ll be taking the commodity-first approach which can lead to high speed delivery and lower costs, especially as we continue to develop our technology ecosystem into a network of decoupled but integratable systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c68991ab030" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comic-relief/the-real-business-value-comic-relief-has-unlocked-by-going-serverless-6c68991ab030">The real business value Comic Relief has unlocked by going Serverless</a> was originally published in <a href="https://medium.com/comic-relief">Comic Relief Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Summit London case study: Going Serverless at Comic Relief]]></title>
            <link>https://medium.com/comic-relief/aws-summit-london-case-study-going-serverless-at-comic-relief-51a22c15e398?source=rss----4e7e42139f79---4</link>
            <guid isPermaLink="false">https://medium.com/p/51a22c15e398</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[product]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[digital-transformation]]></category>
            <dc:creator><![CDATA[Caroline Rennie]]></dc:creator>
            <pubDate>Thu, 06 Jun 2019 17:38:54 GMT</pubDate>
            <atom:updated>2019-06-13T09:52:39.600Z</atom:updated>
            <content:encoded><![CDATA[<p>This Serverless deep dive talk from the AWS Summit London was led by <a href="https://twitter.com/danilop">Danilo Poccia</a>, an AWS evangelist for Serverless, followed by the Comic Relief case study of our experience implementing Serverless. I wanted to share the video along with some additional notes which I either forgot to say at the time or Twitter questions have revealed weren’t clear enough — hope you enjoy!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FKb2Qk3vAkJI%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKb2Qk3vAkJI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FKb2Qk3vAkJI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d32ab406510515c52c5310e341811c5/href">https://medium.com/media/8d32ab406510515c52c5310e341811c5/href</a></iframe><p>This talk will be covering the journey that Comic Relief has undertaken to adopt Serverless technologies as our default.</p><p>First, some context. Comic Relief is a charity with the aim to create a Just world free from poverty. Our flagship event is Red Nose Day — which is sort of a national takeover to come together in March to raise money and awareness for the projects we fund which are tackling some of the biggest issues of our time.</p><p>The enthusiasm of the public for our campaigns each year is incredible, but it also comes with some challenges — our traffic is predominantly driven by broadcast moments on the BBC — and as a team, we can never be sure what the response will be to an on-air calls to action. Where other e-commerce products have to manage considerations like stock availability, donations have no limit to the number we can sell — conversely, where a festival ticket sales site can keep users refreshing for hours to pay for their product, user’s who wish to donate do so in a highly emotive and impulsive moment — if their donation experience is disrupted, that impulsive moment may be lost.</p><p>Our annual campaign culminates in a 7 hour TV show which is the main driver for donations from the public, requests to our platforms can go from zero to tens of thousands in a minute — this year we peaked at 350 donations per second but our systems are designed to handle significantly more than that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RQsWPYf9XAKV_VTG3ZHMYg.png" /><figcaption>Scaling challenges — this graph shows our spikes in transactions over 4 hours of the TV show. Across our 3 donations mechanisms (online, SMS and call centres) we saw a peak donation second of 350 donations.</figcaption></figure><p>So, when asked “why serverless” — there’s an obvious answer around our inconsistent and unpredictable traffic, scaling when we need to scale and only ever paying for the hosting we use. But that wasn’t actually our motivation for trying out serverless.</p><p>To set the scene, let’s start with our 2016 campaign legacy tech stack — Before Serverless.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DydD7_Yj9kKdyYsPHsjc2Q.png" /><figcaption>2016 consumer facing tech stack — one monolith and two independent products.</figcaption></figure><p>Our products could be relatively neatly divided into “the website” — a Drupal 7 monolith which handled almost everything an end user could want to do with Comic Relief — static web content, taking payments, SMS gift aid declarations, collecting images of fundraisers and a bunch of other stuff. The other 2 products which escaped the Drupal 7 platform were our Giving Pages — where users could set up accounts and fundraising pages — and our “Night of TV” donation platform which was a separate application due to its high availability and resilience requirements which you’ll hear a bit more about later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ScKexkAR1yfXIUkpI5ktVw.png" /><figcaption>2017 consumer facing tech stack — introduction of the Drupal 8 platform for CMS and microservices for pay-in fundraising and SMS giftaid. First serverless application the fundraiser gallery.</figcaption></figure><p>After the 2016 campaign, our team started on a Drupal 8 platform — <em>I know, you’re at a serverless talk, why do I keep saying Drupal, please bear with</em> — this platform was designed to be a flexible and modular CMS. One critical point to make here is that the decision was made to decouple products which collect user data from the Drupal platform. We built separate applications for paying in fundraising and gift aid declarations — both of which were built in Symphony — so <em>still</em> not serverless.</p><p>In the 2017 campaign we delivered our first serverless application to end users. It came about due to the requirement for a fundraiser gallery which had been continually de-prioritised and suddenly we had a very real requirement to deliver — fast. Luckily, we had a frontend engineer who had been playing around with React on the weekends and an engineering lead hankering to try out this new serverless thing. The application was simple, a form which users can add up to 5 images to, enter their email address and a checkbox with permission for us to use the image in a gallery on our website. We used Cloudinary as our quickest way to get a product spun up and within 2 weeks we had a viable application. We popped it in an iFrame on a content page of the Drupal site and it had thousands of users sharing their photos through it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jMltple-7NQ1hXNEv3GbrQ.png" /><figcaption>2018 consumer facing tech stack — serverless applications for discrete one-off product requirements.</figcaption></figure><p>The next step for us was to fully retire the Drupal 7 platform and, in doing so, needed a new way to deal with the contact us form, another piece of our legacy monolith that needed upgrading. It also meant getting more of our team adopting the serverless frameworks, React for our frontend and NodeJS for the backend. The speed of delivery was great and this combination became our go-to for delivering one-off discrete products that we didn’t feel too precious about. So, by the 2018 campaign, we had a fundraising gallery, a competition to design the next red nose, the contact form and a calculator for teachers to add their school’s steps, all running serverless.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U417tdaDxcN4Alcgnmnqag.png" /><figcaption>Shared services across products allowed for rapid reusability.</figcaption></figure><p>By this point, we’d managed to identify quick wins in reusability, so functions like the serverless mailer for triggering transactional emails and our postcode lookup service were already being shared across products, even products which weren’t built on the serverless framework. Our frontend components were all being developed in Storybook — another time-saving method to save your engineering team repeating themselves.</p><p>Identifying elements of ecosystem which never interact with each other is a good place to start when wanting to move to serverless. Understanding what can function independently is as important as understanding your dependancies.</p><p>You may have noticed that what we’d built at this point weren’t really our business-critical products. I think that’s one of the most important lessons we’d hope to pass on — these weren’t our flagship products. They weren’t the products that if something went wrong we’d risk losing millions of pounds for our charity. But they were the right products for us to learn and test out different tooling before setting our sights on the higher profile products.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ea6hpzfAbwq1rti-k4BeQ.png" /><figcaption>2019 consumer facing tech stack — Donate and SMS Gift Aid moved into serverless and more reusable services such as the marketing preferences and payment service layer.</figcaption></figure><p>By Red Nose Day 2019, we brought the big money products into our serverless stack too — the gift aid form and online donation application. While I wish I could say that in 2016 we had what now looks like a well-thought-through strategy to move our tech stack to be serverless — if you look back at our technology strategy from that time, the terms you’ll actually see are “Reusability” and “Reduced Costs” — and the best technology solution we’ve found to meet those strategic goals has been serverless.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TYpqAnpiuwG4-j5OQwoU_A.png" /><figcaption>Architecture diagram of the new donation platform (link for Cloudcraft diagram<a href="https://cloudcraft.co/view/fe598a19-ef3b-475e-b778-fa25eb66f61b?key=54ozoymTe-p4AJYR4d1LfQ"> here</a>)</figcaption></figure><p>To give an idea of just how fast serverless development is, the first code for our new serverless payments system was committed to Github April 29th 2018 and we had the new donation platform live in production week commencing September 4th 2018. With a team of 2–3 engineers we were able to take the learnings from the previous platform and build parity with it in terms of donations per second with a much leaner, cleaner, codebase.</p><p>Getting the product released early meant we could extensively test and iterate on the solution. When working with abstracted functions and services you’re also able to refactor your code as quickly as you write it, so no one’s getting precious about their masterpiece. We worked with the Armakuni team, NCC and AWS Well-Architected to challenge our approach and identify if there were any vulnerabilities which weren’t covered in our new system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9tUcHgzQZSAtBNAx2M6oTQ.png" /><figcaption>Old vs new architectures for our donation platform</figcaption></figure><p>Now, onto the donate product. To understand our platform, it’s good to understand the system it was replacing. Our previous donation platform was built by Armakuni, a third party that specialises in high resilience and high redundancy systems. To ensure 100% uptime, Armakuni built an application that was hosted multi-cloud and multi-region and could take up to 300 donations per second. It had also been one of our largest tech investments and never failed us in 6 years of Night of TV. So, needless to say, we had some big shoes to fill.</p><p>We knew we wanted to strive for functional parity with the Armakuni platform. Resilience and redundancy were areas which we really needed to focus on — it was unchartered water for us. To handle resilience in the past, the Armakuni platform had an always-on suite of Redis Datastore across multi-cloud to handle high volume moments, but this wouldn’t be an option for us in the serverless world.</p><p>Instead, our resilience was built in through redundancies across multiple services — if the RDS failed then we had SQS, if SQS failed then we had S3 backups. We built circuit breakers into every step of the application so if a function was to fail, then the transaction isn’t lost and the function can automatically retry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NCYqb0h9C0z6pKfIlI-R_w.png" /><figcaption>AWS costs from 2013–2019 — comparing March 2015 (peak cost) with March 2019 (serverless cost)</figcaption></figure><p>One of the biggest wins for us was reducing our AWS hosting costs compared 2015 by 93% — this is including our remaining non-serverless architecture.</p><p><strong>Something I didn’t get to say on the day:</strong> <em>It’s worth pointing out here that these aren’t necessarily apples for apples comparisons — every campaign will have different requirements for hosting. One Twitter comment I enjoyed after the talk was about how we couldn’t have been optimized in the first place. That’s probably true, but also clearly shows I didn’t emphasise the requirement for resilience well enough. Comic Relief makes almost all public donations within a 7 hour window once a year — downtime for our applications during that period would cost us millions of pounds in donations. What’s changed is our attitude and trust in technology partners like AWS for our big events!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vs2wqVgbCY5U4BZZ3ly0lQ.png" /><figcaption>Actual costs for our donation serverless platform during the TV show for 2019 — $92</figcaption></figure><p>The best thing about it, in my opinion, is that we’ve got a system that could be slammed with those levels of traffic any day of the week — not just for a one-off event each year.</p><h3><strong>What you can take away from this talk</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2KBihSkP9xuHIkGaw_IR_w.png" /><figcaption>4 things we learnt along the way</figcaption></figure><p><strong>#1: Identify your limiting factors</strong>.</p><p>When you move to serverless infrastructure, you’ll be pushing problems downstream. An example of this for us is with Payment Service Providers (PSP) — we integrated with multiple PSP for resilience in our platform. Each provider has limits which peak broadcast moments can exceed. We built simple functions for each PSP and if a PSP were to struggle under load — then we can limit the number of transactions that will go through them. Your serverless products will inevitably live in a hybrid ecosystem — recognise where your limits are and find solutions that mitigate those risks.</p><p><strong>#2 Take advantage of rapid reusability</strong>.</p><p>When you change your mindset to functions-first, you can create a micro-service out of anything. I’ve already covered some of the reusability of our services and Storybook but I also want to call out the Lambda Wrapper which allows for boring repetitive code to be automatically and consistently applied to all functions so you can deliver new functions faster. Our movement towards services was really born out of engineers identifying when work was duplicated and taking responsibility to isolate and abstract reusable code.</p><p><strong>#3 Use monitoring for optimization, not just catching problems.</strong></p><p>Just a quick point on observability — we chose iOpipe and Sentry which enable us to identify and quantify impact of issues really quickly. We know down to the function what the problem is and who it’s impacting. This combined with Grafana dashboards allows us to have a really clear view of performance of the application and we can make changes to optimize experiences in real time — for example pushing more traffic through the highest converting PSP.</p><p><strong>#4 Use serverless for load testing.</strong></p><p>This can be done no matter which stage you’re at with your serverless discovery.<strong> </strong>Serverless has been a really cost effective thing for us in general but I think one of the best places where those savings are realised is in our load testing. Previously, we would spend a couple of thousand pounds each campaign just on load testing — now it’s a totally negligible cost. We’re able to run Drip, Slam and Ramp tests using the Serverless Artillery and load testing is democratized throughout the team so anyone can easily test out their work under load. It’s also a tool you could use without any of your infrastructure in serverless so why not give it a go?</p><h3>We couldn’t have done it without…</h3><p>I’m just going to finish up on a couple of things that we really wouldn’t have succeeded without.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SqLfwKw1qWegdhyRPvOBmA.png" /></figure><p><strong>Inquisitive engineers. </strong>Working with a team who are happy to learn new skills and try out new tools has been fundamental to our success. Not everyone has this inquisitive nature but we’re really lucky that our relatively compact team were willing to work out of their comfort zones.</p><p><strong>Great tech partners. </strong>AWS were a natural choice for us moving into the serverless world — not only because they were market leaders but also because we’re able to pick and choose which parts of their offering work for us. Jason Thoday who is always able to open conversations for us in what’s new at AWS and how it can fit with Comic Relief’s goals and direction.</p><p><strong>Trust from your organisation. </strong>This is something which our team has built up over the past few years. Building trust from senior management and capability within our team in the smaller products first, allowed us to make pretty drastic changes over a relatively short period of time. Sounds simple but the ability to experiment and explore is actually one of the most valuable things that you can give to your product teams.</p><p>Thanks so much for listening (well, reading), if you want to learn more details about what we’ve done and how we’ve done it, check out the <a href="https://medium.com/comic-relief">Comic Relief tech blog</a> and if you want to see our much-boasted about donation platform, please head straight to <a href="https://donation.comicrelief.com/">comicrelief.com/donate</a> and test us out! Thank you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51a22c15e398" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comic-relief/aws-summit-london-case-study-going-serverless-at-comic-relief-51a22c15e398">AWS Summit London case study: Going Serverless at Comic Relief</a> was originally published in <a href="https://medium.com/comic-relief">Comic Relief Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimistic about our future optimization]]></title>
            <link>https://medium.com/comic-relief/optimistic-about-our-future-optimization-cf97731dc7cc?source=rss----4e7e42139f79---4</link>
            <guid isPermaLink="false">https://medium.com/p/cf97731dc7cc</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[product]]></category>
            <dc:creator><![CDATA[Caroline Rennie]]></dc:creator>
            <pubDate>Tue, 30 Apr 2019 16:37:10 GMT</pubDate>
            <atom:updated>2019-04-30T16:37:10.441Z</atom:updated>
            <content:encoded><![CDATA[<p>At Comic Relief, we have 5 core values that we aim to achieve in everything we do; Bold, Creative, Fun, Trustworthy and Engaging. These values can be seen front-and-centre in our campaign activity, but we also embrace these values in how we optimize our digital product offering to ensure that, as well as hitting our campaign targets, we can continue to innovate.</p><p>On 9th November, Hiral Patel (Comic Relief’s Analytics Manager) and I attended Opticon London; the annual conference hosted by Optimizely. It was a great event with speakers from multiple industries addressing the unpredictable wins optimization can bring. The day was a good mix of talks and Q&amp;A sessions which allowed us to consider how we may shape our long term strategy and also some immediate best practice learnings we could implement on our return to the office. Below are a couple of those learnings, along with some other observations from the day.</p><h3>A/B testing shouldn’t always be iterative</h3><p>It may grate the nerves of the scientific mind to say it, but Secret Escapes’ Rohit Gupta’s talk “Don’t be a scientist” was really helpful to remind us all that the purpose of testing is to optimize a product to achieve its goal, not necessarily to understand why. Is it better to get a 5% increase in conversion and know it was because of a specific button CTA change or a 10% increase where the entire page was different? The scientist in us all wants to know what it was that changed user behaviour, but that doesn’t actually matter — what matters is that the new version worked (or didn’t work) — because you can always find out the ‘why’ later!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*3Adc4h0NK7Km_1ND" /></figure><h3>Use Minimum Viable Product principles when you test</h3><p>Stephen from conversion.com gave a fantastic talk regarding the effort/resource levels which should go into testing — and the importance of testing broadly in order to make sure that you don’t waste resource pursuing a ‘favourite’ hypothesis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*BPAkoCSvNjp-I5gV" /></figure><p>His example of the Guardian’s ‘Save for later’ button was one which really stuck in my mind; rather than building in the functionality which already existed on mobile for the desktop product, the Guardian’s desktop product team simply added the button without the backend functionality to back it up. This meant the test could be executed quickly and cheaply to provide insight into whether the feature would be used by their desktop users. I’m now building out some similar tests to run on our Comic Relief websites — results to follow soon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*OYT3JFmHgl95McaA" /></figure><h3>Every business has constraints on testing</h3><p><em>“Experiment Everywhere”</em> was the theme of the day but it was clear that all the attendees found that there were limits to their ability to test — the three most common were:</p><ul><li><strong>Resource</strong> — whilst tools like Optimizely are making it easier for us to test without relying on development time, any test will still require time to implement and, naturally, nobody has unlimited resource. Prioritizing resource for testing can sometimes feel like a ‘nice to have’ (otherwise known as bottom-of-the-backlog) — but companies who use testing in early proof of concept work can save immeasurable amounts of time and money as testing will invalidate hypotheses as well as validate them.</li><li><strong>Company culture</strong> — the speaker from Orange Europe found that the company culture didn’t really have experimentation and analytics as a regular way of working. Even though there was an organisation-wide training programme, only 5% of staff where applying the training in their day-to-day work.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*x0JRSNcOsu6vNAub" /></figure><ul><li>The way Orange repositioned experimentation was by allowing anyone in the company to propose tests; initially they wouldn’t need to bring anything much more than the barebones concept but once teams saw the benefits and insights which testing produced, more and more of the company got involved and made use of their analytics knowledge to propose other tests. Now Orange Europe has 50% of those who were trained using analytic tools (how’s <em>that</em> for a nice ROI stat?)</li><li><strong>Traffic levels</strong> — this is a big one for us at Comic Relief (but Sarah Stellini, Head of Growth Optimization at Betssen also feels our pain!) — whilst we experience huge spikes in traffic around campaign time (Red Nose Day or Sport Relief), achieving traffic levels to gather significant results outside campaign time can be an issue. We also struggle as different audiences typically use our sites at different times of year; results from a test run in September — December would not necessarily be relevant to our Night Of TV traffic for example.</li></ul><h3>So what’s next for experimentation at Comic Relief?</h3><p>First up, we’re going to be doing more of it! With Red Nose Day 2017 sneaking up on us at speed, we’re about to hit the best traffic levels for us to test. We’re also going to embody our company value of being bold with our tests and encourage the company as a whole to make use of the tools we have at our fingertips. And lastly, we’re going to talk about the results more — not just the tests which worked; acknowledging and being open about when our assumptions are proven wrong is potentially more important that celebrating when they were right.</p><p>To finish, I’d like to thank Optimizely for hosting us and the other attendees for sharing insights into their own challenges and successes with testing. All the speakers and further information can be found on the <a href="https://www.optimizely.com/opticon/london/">Opticon London website</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf97731dc7cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/comic-relief/optimistic-about-our-future-optimization-cf97731dc7cc">Optimistic about our future optimization</a> was originally published in <a href="https://medium.com/comic-relief">Comic Relief Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>