TL:DR We are hiring | Social journalist | SEO journalist | Audience strategist

When starting an audience role, expect your family and friends to give you some puzzled looks. “What does it mean?” will be one of the most frequently asked questions. Throw a few words like “SEO”, “engagement” or “CTR” their way and you’re likely to get some more blank stares.

Some colleagues might not understand what your day-to-day looks like either. The roles of editors, reporters and sub-editors have been in place for yonks so they’re clearly defined. Meanwhile, “growth editor”, “SEO journalist” and “audience strategist” are relatively new additions and can seem rather vague.

Publishing used to be much more straightforward too — finish your pages for the day and let the distributors do the rest. The next day your reader can pick up their paper of choice from the newsagents and you start the whole process again.

Now, media companies are faced with a digital environment that’s constantly changing. New distribution platforms pop up all the time, each with their own readership. Getting your journalism seen — and most importantly read — by a global audience requires a different skillset.

This is where the audience team comes in.

The traditional view that we tell the readers what they should care about no longer rings true. With the internet, readers have the whole world at their fingertips. With that comes choice. Publishers no longer dictate the biggest stories of the day. Instead, we can see what people are interested in, both in real-time and over longer periods, and cater to their needs.

The SEO team plays an important role in communicating those needs to the rest of the newsroom, making sure we cover the most relevant topics at the most opportune moment in order to reach the right audience, as well as providing advice on how to frame stories to increase visibility. For us, this does not mean publishing clickbait articles to appear in search results, but wading through data to identify our readers’ needs. Then we choose the topics in need of expert analysis and give our subscribers and new audiences the answers they’re looking for.

We also need to be where our readers are; whether that’s on stalwarts like Facebook and Twitter, or comparatively newer platforms like Pinterest, Instagram or TikTok. By combining our knowledge of who our followers are with in-house data on what engages particular audience groups, we can make increasingly targeted decisions on who to reach and where.

As well as reaching new people, we need to be smart about how we publish. Our audience strategists spend their time figuring out how our journalism can make the biggest impact. How can we split out a package of stories to satisfy different reader groups, while avoiding fatigue and keeping engagement high? When is the right time to publish, and which platforms are best suited for getting the word out? Are we including the right elements on the page to keep readers informed? The answers to these questions can be complex, but often have data at their heart.

When we gain subscribers we need them to use our product as much as possible in order to build a habit. Our engagement team aims to achieve just that. They cover everything from curating conversation below the line to producing polls for readers to have their say. They also focus on creating content directly targeted to specific groups such as our female or under-40 audiences, giving them a reason to remain a subscriber long into the future.

We also produce a raft of newsletters to build relationships with our readers through their email inbox. They are carefully designed to reach specific groups, ranging from politics buffs and book lovers to fitness fanatics. The audience team works closely with the marketing department to reach subscribers through tailored push notifications too.

Make no mistake: a huge amount of planning and cross-team collaboration goes into making all of this happen. Apart from team-specific skills such as technical SEO know-how or selling a story on different social platforms, there are some broader traits you need to have regardless of audience discipline.

Patience, resilience and optimism are paramount to succeeding within an audience role (Netflix founder Reed Hastings famously said that he only wants to hire “irrational optimists”).

Changing the way newsrooms have worked for hundreds of years is a slow and sometimes painful process, which is why every audience team needs individuals with enthusiasm and a dogged persistence to help digital transformation happen.

What to expect from a role in audience development was originally published in Digital Times on Medium, where people are continuing the conversation by highlighting and responding to this story.

How to reflect on seven years as head of digital at The Times and The Sunday Times? Start at the beginning is always good advice, but also acknowledges the distance we — and I — have travelled.

The instructions when I was given the job in November 2013 were very clear. “Fix the tablet,” I was told.

Our tablet app, which had been launched to great acclaim in 2010 when the iPad was released, had recently been rebuilt and it had not been a great success.

That’s an understatement. On the day it was relaunched in February 2013, the problems being experienced by users — the worst of which was an extreme overheating of their device — led to more than 30,000 calls to our customer helpline in a single day.

By the time I took over as head of digital nine months later, the initial crisis had passed but there were still problems with the app. Headlines and pictures looked out of kilter; we would often be left with a single lonely word on a page; picture credits had absurdly large capital letters.

With all the confidence of a print editor with more than 15 years of experience in newspapers — but, crucially, almost none in digital — I thought, “This shouldn’t be too difficult. I’ll just sit with the production team and sort it out.”

I spent a night with the production editor, who not only worked a godforsaken 6pm until 2am shift, but now had a boss at her shoulder who was full of ideas. We sat and went through the changes that I, as an editor and regular consumer of our tablet edition, wanted to make.

It soon became clear that I was dealing with something very different from print. The headlines were written to work on three different device sizes, so of course they didn’t always look perfect on the one I was looking at; the pictures were autosized by an algorithm; the picture credits were hardcoded, and frankly redoing that work was not a priority; the templates only allowed us to do this, not that. Everything was more complicated than it at first appeared.

I certainly didn’t give up trying to improve the tablet after that first experience but I realised very quickly that it wouldn’t happen quickly and that many different requirements would be competing for our developers’ time.

And so began the learning experience that has characterised the seven years I’ve been in this job, which have been by far the most challenging and stimulating of my career. That experience with the tablet was salutary: I knew then I had to learn a new way of doing things, to reprogramme myself out of my print ways, to start to think digitally.

What an uplifting journey it has been and along the way I hope I’ve learnt some things that will be interesting, maybe even useful, to others.

The main lesson is listen to your customers — because they really do know what they want, especially in as mature a market as news. I realise that this is not an original insight but too often internal company dynamics and the ideas of senior people get in the way. The times we have truly listened to customers is when we have succeeded.

A great example was our launch of a new smartphone app and website in 2016. Our existing sites and apps weren’t considered to be a priority at that time — we were all about the tablet — but I’d looked at the customer satisfaction data and our readers hated them. They weren’t just indifferent to them — they actively disliked them.

It’s not surprising: they were a jumble of poorly done breaking news mixed up with the content from the paper. Oh, and there were two websites, one each for The Times and The Sunday Times, which in most readers’ minds were one and the same publication.

Happily, it was eventually accepted that this was a crazy situation and we worked with Ideo, a design thinking agency who had grown out of the famous d school at Stanford University (I later attended an excellent design thinking bootcamp there), to reimagine them.

The idea we came up after an intensive period of listening to users was familiar and yet radical. We proposed that we leave breaking news to others and instead focus on providing finite and finishable editions for readers at set times of the day. We wouldn’t tell them everything that was happening but we would tell them what was important.

A lot of observers thought we were just copying a print model. But no, this was driven by our digital readers.

They told us they didn’t check the news as often as we as journalists do. I guess that shouldn’t be a surprise: unlike journalists most people don’t have screens around their home or office permanently tuned to TV news channels, nor do they opt for notifications from gazillions of news providers (they actually said they found the ones they did get a bit annoying), or sign up for many dozens of news emails. Mostly they checked the news once or twice a day and then got on with their lives.

They also told us that it’s the authority of our reporting that people come for, not the speed at which we can get it out. We asked readers if they’d like us to be quick or right? They all said right.

The results after the launch of the resulting products were an increasing frequency of visits — even though we were updating less often — and increased engagement. Visitors to the smartphone app, whose daily usage tripled, read four times as many articles as before.

There are many ways of listening to your readers and the one that I think is still yet to be fully utilised is data. I find it astonishing that some journalists remain reluctant to embrace all it can offer because, after all, the data is your readers. We know in far, far greater detail and perhaps really for the first time what our subscribers and potential subscribers want from us.

I can see why this is unsettling. In print, there were three data points. Daily or weekly sales; ad revenue; and, most important of all, is the editor happy? I’m joking but really, what happy days those were when these were the only things you had to think about.

Now, for every one of the 200 articles we publish a day we measure unique visitors, page views, dwell time, comments, shares, saves, referrers, and the demographic details of readers. And at The Times my team worked with our data scientists to introduce bespoke proprietary indices within our award-winning Inca analytics platform that judge whether a particular article has succeeded for engagement in terms of subscribers, guests, women, international readers, young readers, less engaged subscribers and so on.

Now these have evolved into algorithms that suggest whether to move an article up or down the edition, whether to think about changing its headline, whether the content doesn’t match the promise of the sell.

This real-time data, coupled with the detailed (and award-winning) Content Review we conducted a couple of years ago, means we can now say with a high level of confidence that we know what type and amount of content works best for digital news subscriptions. This has the potential rapidly to accelerate the growth of The Times and The Sunday Times in the next few years.

For this, further cultural change is required and I think this remains the most difficult thing for legacy news brands.

In our case we had 200 years of history pulling us towards print. And compared with digital, print is a very simple business indeed, and one whose basic structures haven’t changed for decades, centuries even.

Digital news is different and challenging to those structures. Not least because in very hierarchical newsrooms often the youngest, most junior people have the best ideas. It is a feature of our age that not only do junior staff often think they know more than their superiors — that’s not new — this time they really do.

This is a huge challenge, especially when newsrooms have typically adopted very strong command-and-control models.

I often saw my job as explaining to senior people why their idea was perhaps, er, suboptimal, why they should trust the 25-year-old in the social team more than what their contact told them at some high-powered lunch, why their brilliant new product feature was actually technically impossible. Unsurprisingly this wasn’t always easy.

But we made a lot of progress along the way. And a lot of it was down to my team. When I look back at my career at The Times the thing I am most proud of is the digital strategy and development team I built up.

Its aim was to provide the specialist skills — in social, newsletters, community journalism, video, audio, digital storytelling, data journalism, digital strategy — that the newsroom didn’t have.

We took the view that it is unrealistic to expect all our outstanding story-getters to switch quickly to a digital mindset and to have the skills to tell stories differently. But pair them with a specialist and you can work wonders.

It produced great work — such as when our environment editor worked with our data team to expose the toxic levels of air pollution around Britain’s primary schools. This led directly to a change in UK government policy.

We have also brought new skills into the newsroom: among my team are developers, a post-doctoral physicist, a former civil servant with a degree in robotics and a former employee of a trade union.

Many of them are not, as I have been told before, “journalists” in the traditional sense. But they bring diverse skills that legacy newsrooms must embrace in order to survive.

Managing them was easy once I realised I would have to discard almost everything I’d learnt about management in the print newsroom. The command-and-control structure just doesn’t work across such a wide range of specialist skills and with such a huge array of output. One person cannot possibly keep on top of every video, every social post, every newsletter, every podcast, every story on the website and apps, every data investigation, every interactive, every bit of audience work in the same way as they can in a print newspaper.

Instead I opted to pursue a policy of radical delegation with my team. I told them that we put a lot of effort into our hiring process so I knew we’d always employ good people. And now — within a framework of team goals — I was leaving them to get on with it. Obviously I’d get periodic reports of how they were tracking against targets but mostly my message was: “Do what you need to do, but rest assured that if you get into trouble you can come to me and I’ll sort it out.”

The good thing is that mostly I had few issues to sort out and this approach produced great work from an empowered team. I heartily recommend this course of action. It’s one of the best things I’ve learnt in the past decade.

Between us we launched new products — some that worked brilliantly, like Red Box, our politics strand based on a daily newsletter; some that didn’t, like Times Weekly, an international edition that attracted a lot of trialists but not enough paying subscribers.

We built not one but two CMSs — Deck for producing newsletters and interactives, and Edition Builder for web and app curation. We devised and ran 24 different training sessions for the newsrooms whose cumulative attendance ran into the thousands (my favourite was “Why everything is not a podcast” but that’s another story).

We set up an audience team, who found that the less we published the more engagement we got from readers, and a data and digital storytelling team, who revolutionised the way we approach our stories on web and apps. We ran hackdays to unearth new talent, as well as winning “the World Cup of news hackdays”.

We pioneered using Slack in the newsroom … “That’s no big deal because everyone uses it,” I hear you say. Well, it was in 2013.

We worked with machine learning specialists to develop fact-checking and style book checking tools.

But most of all I’ve learnt that thinking digitally is about being humble in the face of your users and your data; it’s about collaboration; it’s about challenging the status quo; it’s about embracing “I wonder if…” even if that takes you somewhere you’d never thought of; it’s about listening and learning.

And the tablet? It’s no longer our “hero product”, having long ago been eclipsed by the smartphone app. And it’s soon to have a significant upgrade as we move to a universal app, which will meet the customers’ desire to find “the same Times everywhere I look at it” and give them faster access to all the work the teams across the business are working on.

But that’s a story that someone else will have to tell.

To everyone who has helped me along the way — and there’s too many to mention by name for fear of leaving someone out — thank you for the inspiration, the ideas and the camaraderie.

Digital transformation at The Times of London was originally published in Digital Times on Medium, where people are continuing the conversation by highlighting and responding to this story.

A selection of our favourite projects which will be updated throughout 2021. You can see the best of our work from 2020 here and some of the best of our other past work here.

How has lockdown affected your mood?

On the anniversary of the first national lockdown, we asked our readers a simple question: how have you been feeling? With a debut for our new interactive tool, we asked Times readers to plot how their happiness, loneliness, anxiety and life satisfaction have fluctuated over the past 12 months. The results were charted alongside the national average courtesy of the weekly Office for National Statistics Opinions and Lifestyle survey.

The 85 lords born to rule

Britain is the only country in the world beside Lesotho to reserve seats in parliament for hereditary peers. In this front page Sunday Times investigation we exposed the truth about the 85 lords entitled to make and vote on legislation because of their birthright.

The findings were stark: they are all male, just under half went to Eton and they own an estimated 175,000 acres of land between them — nearly half the size of Greater London.

They claim more in expenses, but contribute less than the average life peer. When they do speak, they are 60% more likely to talk about their own business and personal interests, from land to tobacco.

We built an interactive that ran front and centre, with readers able to explore the individual histories of the 85 peers and sort them according to five key metrics: age, school, expenses, political party and when their peerage began (with some dating back to the 13th century).

Census 2021: how recording the UK population has changed

In anticipation of the 2021 census, we analysed 220 years of national data collection to bring you a visual guide to Britain’s history. By delving into and categorising 433 questions we were able to depict a country with evolving priorities, social mores and technological capabilities. We combined an interactive scrollytelling template with video animation to bring the forms to life.

Britain set to jab all adults by early June

Working with science correspondent Rhys Blakely we analysed the rate of vaccinations in the UK and across Europe to reveal that all adults would be offered their first jab by early June. Why does it matter? A separate estimate by Dr Katharina Hauck, of Imperial College London, suggests that lockdown costs the UK at least £18 billion a month in GDP. An earlier than expected immunisation date means that the UK vaccination campaign, thought to cost £12 billion, will pay for itself by the end of the summer.

Uni money laundering

We used freedom of information disclosures to establish that at least 49 British universities let students use cash to pay £52 million in student fees over the past five years, including millions from China, India, Russia and Nigeria, sparking money laundering concerns. MPs have subsequently called for these payments to be banned.

Problem solvers

In the first of our new problem solvers series, which meets the pioneers seeking answers to society’s biggest problems, we spoke to the doctors giving new hope to transplant patients by turning previously unusable hearts, kidneys and livers into donor organs that work again.

In the second article in the series, we looked at how technology that sees through the earth could help build cities of the future.

Covid: How busy are hospitals near me?

We analysed NHS data to show the impact the pandemic is having on our hospitals and give readers the chance to see how their trust is coping, from bed occupancy to waiting times.

When will lockdown end?

The speed at which lockdown is eased will be determined by a number of factors, including how effective the vaccine rollout is and how well jabs stop the spread of the virus. We built an interactive slider to allow readers to see how changes in these factors would affect cases, bringing them down to a point where restrictions can be lifted.

Tory rebels

Through analysing thousands of votes from the House of Commons we revealed that the most rebellious Conservative MP of this government is….David Davis. We uncovered that the majority of Tories have voted against their party at least once, and that 23 have done so at least ten times.

100,000 deaths

As Britain reached a toll that seemed unimaginable at the start of the crisis, we analysed the 100,000 Covid deaths and spoke to clerics, health leaders and the bereaved who warned that the staggering human cost must not be forgotten.

Where can you get the vaccine?

To help readers locate their closest vaccination centre we built a tool that showed them the sites near them and how long it would take to drive to the nearest one.

German aristocrat and cannabis firms given PPE contracts

Contracts disclosed by 40 hospital authorities after freedom of information requests exposed the businesses with little experience of providing medical supplies paid millions by NHS trusts to supply personal protective equipment to staff.

One Midlands hospital trust bought masks and a disinfecting machine from a German aristocrat who spread a Covid-19 conspiracy theory on Facebook. A lingerie firm who donated money to Boris Johnson’s leadership campaign also received a contract.

The best of the Times and the Sunday Times data and digital storytelling team — 2021 was originally published in Digital Times on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the last post I took you through a short summary of what it took to get Edition Builder live at The Times and Sunday Times. If you haven’t read it, you should, it’s pretty good background.

(In the last post I also mentioned I’d be writing this in a few weeks. That was pre-coronavirus, so these posts have taken a seat at the very back of a very long series of seats. Apologies for that!)

As one of the larger engineering projects we have undertaken, Edition Builder broke new ground in a bunch of areas. We tried things that we hadn’t attempted before, and pushed systems to their limits (and well past them in one incident).

There were a lot of challenges we encountered as a team that we had to work out how to navigate. Though its not likely that you’ll learn a lot about the specific issues we encountered, I hope there are patterns you can recognise that are best avoided, and things you can think about should you work on anything similar.

Searching for articles

Search in Edition Builder was, for most of its life, unworkable. I don’t know why I persisted so long convincing myself that it would work, but it was only ever going to get worse from the moment we realised this problem. Not raising this earlier in the lifecycle of the project was a big mistake from me, I think because I thought it was unsolvable, but as with most things online, someone else has usually solved the problem you are facing, and often they’ve built a SaaS solution that will cost substantially less than your engineering time building and maintaining it.

On to how search was terrible. Our content API was never built for search. This is because of the link between our CMS and our content api was only one way: via publishing. Our readers used a managed SOLR cluster for site search. Editorial users only ever needed to work with content inside the CMS, which had reasonable search and categorisation functionality. In the world of Edition Builder existing, suddenly there was a need to find articles before they had gone through the publishing process.

“No problem”, we thought, “we’ll set up another version of that API” (because building versioning in to the API was not something we wanted to be responsible for). (and it has ended up coming anyway as part of a later rewrite).

Except of course there was no reasonable way to perform a scan or search as the endpoints just didn’t exist. So in the end we settled for fetching a range of data between two dates, and storing that in the application state to then sift through locally in the browser.

Constructing that index took as long as three seconds per 24hr period searched, so we limited it to three days at a time. Worse: to get new articles, you had to refresh your local index, causing another delay of however long you had selected.

The moment this hit user testing it became clear to us it was a big problem. The search was just too slow. But was it even within our power to fix? We sat on the problem, hoping a solution would rear its head, as every fix we could think of involved work well outside the scope of the project (and adding months to its delivery).

Step forward a couple of months and a couple of members of the team had been tasked with something new. We’d just signed up for Algolia, a SaaS search provider, and needed to get it working for our readers as soon as possible, decommissioning the SOLR cluster as we went.

Algolia is a breeze to work with, and we quickly saw significant improvements on what SOLR had been returning. We feature flagged one engine over the other, and, with journalists being some of the world’s best critics on search engines to surface their own writing, I jerry-rigged a glitch.com app that we sent round to find out which performed better.

A few tweaks to the ranking algorithm later, we were satisfied. And then it dawned on us that we could use exactly the same thing in Edition Builder.

Tearing out the old system and replacing it with a prototype took just a few days. Again we played around with ranking and faceting to get what we needed, and then we threw it in front of users to test.

Immediately we saw improvements. A local in-browser index was no-longer necessary. Every query went fresh to the remote index, resulting in more recent results. We could return results almost instantly, just as you typed into the box.

Even better, when users talked about issues they were having with search, they were relatively easy to diagnose. We could replicate the same searches, see how the ranking affected the results, and adjust accordingly, even on a per-query basis.

How we structure edition updates

A second issue we looked to solve focused on how we structured editions. On reflection, I think we could have gone further, and we shouldn’t have shut conversations down so early without seeing them out. Again here there was too much of an assumption we had the best solution at the time, and not enough thought to restructure things early on.

In the previous system, editions over the course of the day were overwritten when the latest update was published. This meant throwing a lot of useful data away, a lot of which we were technically paying for in terms of journalistic decision making. Wouldn’t it potentially be useful to be able to learn about how stories move around the edition, and save different updates separately.

We initially explored a git-style branching system where root editions would have branches with changes on them that then potentially were merged back over time. This had the advantages of immediately making sense to developers, and the disadvantage of it not making any sense to any of our users 😭. Can you imagine playing with git-editions on deadline?!

The ‘git’ style feature would have brought us some level of historical and history data, which we could have used for scrubbing back and forth, but the complexity add at the time, along with the UI challenges of having this work seamlessly for users was problematic.

We set out to solve the problem actually in front of us, and built the simplest solution we could see that did that.

Simply enabling users to copy past published editions into future timeslots solved the data problem, and enabled users to work on future editions whilst still being able to make tweaks and changes to past ones.

In hindsight though, it didn’t solve enough of the problem. For example, in a short timeframe you might want to urgently prepare two options for a future version. A direct copy gives you very little flexibility as other stories that might change need to then be made in two places. We thought at the time about creating overlays you could flatten into an edition update, but again it was difficult to work out if the value would ever be delivered.

In hindsight I wish we’d thought harder about how sections interacted with editions and made that less fixed. In reality an edition is just a bundle of sections, and making them updatable independently and then bundleable for an update might have both provided an easier structure to manage, and an easier product to break up should we plan to in the future.

The saving grace at least, given the changes made, is that it will be significantly easier to integrate our bespoke analytics service (INCA) into this system, so as to show journalists when they place stories what the story is expected to do based on its position in terms of our key metrics. This will help journalists understand the power of their decision making better (as stories towards the top of the edition have generally higher numbers than those lower, stories in larger slots likewise etc).

Publishing and the initial breakdown

Finally we have the bug that held us up for four months. The lesson: never trust orphaned systems to do anything.

Edition Builder has a feature that shows the entire edition to you at once in a single screen. In order to do this we have to fetch all the articles that exist in this view from the API.

In addition, Edition Builder, for the first time, had to handle published and unpublished content (as mentioned in the search feature). This meant that two versions of the same article could exist at the same time, given editorially you might want to feature either (the unadjusted published version, or the perhaps more-up-to-date unpublished version).

The consequence being this view needed to switch from two sourcing content APIs, the in-production API, and our preview API, depending on the article.

And when we published articles, we switched any reference for this lookup from the preview version to the production version. Simple!

When it came to running our first live editions everything went really smoothly. We worked our way back from the 5pm edition to the 9am, in order to have the least impact in the most supportable working hours, and finally had the midnight edition in our sights.

Where the 9am, 12pm, and 5pm editions are relatively small updates on the main bulk, the midnight edition publishes several hundred articles all at once. For architecture that was not necessarily built for this in mind, bursts like this can be fatal… which we learned to our peril.

When Edition Builder publishes the midnight edition, all of these articles get posted into the content API in the same queue. This usually causes a little bit of a spike.

But Edition Builders incredibly smart and well-meaning feature that showed you the entire edition after publish, then immediately tried to load these hundreds of articles from the same system that was trying desperately to ingest them.

And what’s more, in order to check if there were any changes, it reloaded them every thirty seconds.

Cue heavy breathing from our operations team as our apps started to intermittently return 500 errors. And by intermittently, I mean all the time for about fifteen minutes. 😬

…

Fixing this was no mean feat. The content API at this stage was practically orphaned, with a replacement on the way, but with no clear end in sight.

What’s more, this problem only occurred in production, and we couldn’t work out how to replicate it on our pre-production instances, which all handled the load fine (perhaps because it was the only load they were seeing 👀).

We, sensibly, were not allowed to continue our production rollout whilst this error still occurred, but we couldn’t replicate the error to show it was solved, so our catch-22 just kept enumerating.

After far too long, our solution came in the form of new talent. Stefano Berdusco joined me on the long road to salvation and together we wrote load tests (he wrote them), we build a perfect replica of production (he built it), and we flipped the sourcing to the new API, so as not to cripple the old one (maybe my idea, but given my track record probably also his).

😍 Our tests, with load applied, showed the new API (built with this burst in mind) was able to easily handle the preview state. And we were back on track.

So that’s that. Thanks for reading, I hope we all learned something.

(Also in going through our Slack logs for this post, perhaps my favourite short-lived bug below)

What we learned from rebuilding our editions CMS for The Times was originally published in Digital Times on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Ryan Watts and Sam Joiner,
Times and Sunday Times Data and Digital Storytelling team

Covid-19 did not shoot to the top of the news agenda overnight and our first charts and maps on the outbreak appeared in our digital editions in January.

As the virus spread and a national disaster became a global pandemic, presenting the latest information to our readers became an important service. Almost overnight, we had a whole host of visuals that needed daily updates.

Principally there were three processes taking up our time: adding the latest data to our charts, maps and tables; recreating these for the print edition and updating the words in our global explainer and UK tracker.

The unprecedented nature of the outbreak meant this was undoubtedly a good use of resources, but with updates required at regular intervals, live feeds of data from reputable sources and a powerful charting tool at our disposal could much of it be automated?

The answer was yes, and this is how we did it.

Automating our charts

Using R, a programming language for working with data, we wrote a script that pulls statistics from the Johns Hopkins dashboard and used Windows Task Scheduler to set it to publish every hour.

There is an R package which works with the API of Datawrapper, our chart building tool, which allows the user to update a chart’s data, add a timestamp as a note — e.g. last updated at 3pm, March 19 — and republish all by running the same script.

Our UK coronavirus tracker also includes two animated maps built in D3 that show how the number of cases has grown at local authority level across the UK.

The data that informs them is generated in the same way: an R script pulls the latest update, formats it and joins latitudes and longitudes. It is then pushed to an S3 bucket on AWS from R.

The data we’ve used for maps and charts of the UK comes from Public Health England. New data is usually published between 4 and 7pm, but the time has varied considerably and been as late as 10pm.

To save us from the joys of constantly refreshing the page we wrote a script that downloads the daily summary data, checks that the date in the file matches today’s and, if it does, sends a notification in Slack to tell us new data is available.

Automating the words

Once the data comes through from Public Health England we use it to update the copy in our UK tracker, using R to generate the figures in the article at the touch of a button.

The process runs a script which updates every variable within the page and then stitches it together before exporting a .txt file.

We then share that file with our online news team who can confidently make updates to the page whenever they are needed.

Automating graphics for the paper

A map that works online might need to be rethought for print given space and design requirements, so we set up a separate script that generates two more maps for the graphics team:

1. A world choropleth map showing cases per million people
2. A proportional circle or ridge map of cases in the UK

The aforementioned R script pushes the data to Datawrapper, allowing our graphics desk to download a PDF which can be uploaded to Adobe Illustrator and prepared for the page.

The ridge map was too complex for Datawrapper meaning our graphics desk is unable to download and reuse the map in the same way.

To solve this, we built a page to host the ridge map which our graphics team now use to download the most recent version.

The raw data behind the map is published in the same way allowing graphics to use the totals for regions when needed.

This page automatically updates when we change the version of the map used on the website.

What have we learned?

You still need humans

No matter how much a process can be automated you still need someone to make decisions and review what is being produced.

For our coronavirus coverage, this means feeds are checked for errors on a daily basis and editors continue to do a manual review of all the figures before anything is published.

This goes for the graphics as well. For our UK map, for example, the volume of cases meant the circles on our symbol map were beginning to overlap and individual areas were becoming difficult to see.

Reviewing it as a team we decided to switch to a ridge map, which clearly shows the peaks and has the added benefit of allowing us to label key areas.

But overall, automating the process of updating both pages has saved countless hours and allowed the team to focus on other ways we can report on both the coronavirus outbreak and stories unrelated to the pandemic.

Stick to the script

Harmless and well-intentioned amendments to the data — like changing the UK to the United Kingdom or the clocks going forward — can throw a spanner in the works.

A little bit of error-handling helps, but it is still important to check formatting and feeds for errors on a regular basis.

Keeping everything together saves on duplication

In the rush to tell a story lots of people often have the same idea. That can also be the case within the same newsroom and avoiding the commissioning of charts which already exist is a daily challenge.

Our Datawrapper Slack integration — which automatically posts published charts into a channel — plays an important role here, but talking to colleagues is key when working remotely.

On a much more manual level, a public shout of “have any of you built this already?” can save a lot of time.

We’re updating our system as requirements change and the story shifts and we’d love to hear from you if you think you can make our system more efficient. Send me a message on Twitter: @ryanleewatts

How we automated our coronavirus coverage was originally published in Digital Times on Medium, where people are continuing the conversation by highlighting and responding to this story.

A selection of our favourite projects throughout 2020. You can see the best of our past work here.

Where are you in the vaccine queue?

With the UK plunged into a third national lockdown it became clear we were in a straight race against covid: inject or infect. Using data from the Office for National Statistics we gave readers an estimate of their place in the vaccine queue at both a local and national level.

Perceptions quiz

Studies show most people misunderstand societal trends. They think rare events happen often and declining occurrences are ubiquitous. Our interactive quiz challenged those perceptions, plotting the results alongside predictions from the general public and fellow Times readers.

Relaxing lockdown rules

Working with Sunday Times political editor Tim Shipman and health editor Andrew Gregory we produced a front page story which outlined the concerns NHS chiefs had about the easing of coronavirus restrictions in December and the impact it would have in January. Sadly those concerns were proved right.

HMRC debt collecting

Working alongside Paul Morgan-Bentley, head of investigations for the Times, we reported that revenue officials had written to families struggling to settle bills during the pandemic and threatened to “take things you own and sell them”.

The months-long investigation found that HMRC had passed on 4.5 million personal records to private debt collectors since 2014 without taxpayers’ specific consent and how staff were incentivised to maximise takings.

The state of England’s roads

FOI-guru George Greenwood obtained figures from Highways England which showed that nearly half of the bridges on England’s busiest roads were in a state of disrepair. Of the 550 that had not been reinspected in two years 30 per cent were rated poor or very poor.

Do we want London to reach for the stars?

Do we want London to reach for the stars? In one of the most ambitious visual pieces attempted by the Times we teamed up with architecture correspondent Jonathan Morrison and 3D illustration specialists Mental Canvas to explain how the capital’s skyline has evolved since the 1960s — and what’s next for skyscrapers in the city.

The great escape of Carlos Ghosn

How did Carlos Ghosn flee Japan to become the world’s most famous fugitive? John Arlidge uncovered fascinating new detail about his escape when he met the former Nissan boss in Beirut — which we used to create a Sin City inspired graphic.

US Election

The data and digital storytelling team was heavily involved in the Times’ and the Sunday Times’ political coverage before, during and after the US election. Below is a snapshot of some of the content we are most proud of.

How coronavirus shaped the US election

Washington Correspondent Henry Zeffman and Times cartoonist Morten Morland guide us through a race for the White House like no other. We used our stories template to provide readers with a visual feast in the run up to the election.

Possible scenarios

What happens if the US election polls are wrong? We teamed up with Henry Zeffman once again to look at eight possible election night outcomes.

Homepage results

Home page data viz alert! Our election banner, updated every 30 seconds, sat at the top of the Times and Sunday Times homepage from Tuesday night until Biden’s victory was confirmed. A Times towers record.

Maps and charts

In a real team effort our maps and charts extravaganza was launched at 9am the morning after the election. We kept the piece up-to-date over the next few days as the counting process played out.

How did Joe Biden win?

Once Joe Biden’s victory was confirmed as the next President of the United States we looked at how the 77-year-old had secured victory. The margins by which the Democratic candidate won key states were some of the smallest in history, with postal votes securing a nail-biting win.

Trees in our cities

It began with a huge dataset noting every tree in London. Working with Rosie Kinchen we analysed other areas in the UK and created a long read exploring the future of our urban forests. We produced maps in QGIS, finessed them in illustrator and added illustrations on five of the most populous trees by Julian Osbaldstone.

‘Cells’ for rent

This piece explored how the conversion of disused shops into tiny flats is a booming business, reviving the spectre of slum housing. Scarcely bigger than a parking space and starved of natural light, the flats are becoming increasingly popular for developers who are taking advantage of relaxed planning laws and the impact of coronavirus.

Virus shifts to young as lockdown fears ease

Working with whitehall editor Chris Smyth we analysed the latest Public Health England figures on coronavirus cases by age group. It revealed that most confirmed cases are now in younger people in an “extraordinary” shift. Following the front page story Health Secretary Matt Hancock admitted it was affluent young people who are catching the virus.

How safe is it for you to cycle to work?

Through combining the national accidents database with maps of Britain’s road network we worked out how many incidents involving cyclists have taken place on every road in the country.We identified the most dangerous roads for cyclists and built an interactive to let readers know how dangerous their cycle route is.

Short thread on the technical process that went into this ⬇️ https://t.co/EJz6Y8akvQ

Insight investigation

Were the Ainsworths murdered by a serial killer? Working with Mental Canvas we recreated one of the crime scenes for our Insight investigation. Readers were able to explore the clues that cast doubt on the conclusion Howard Ainsworth murdered his wife Bea before killing himself.

Gangs in schools

We spoke to 1,300 schools — believed to be the largest survey of its kind — covering half a million pupils. They told us how gangs, violence and county lines drug dealing are affecting pupils as young as eight.

More than 60% of secondary schools had searched pupils, some more than once a week, using airport-style security, metal wands and sniffer dogs to find drugs and weapons in the last full academic year.

The four-story package covered reaction from the children themselves and investigated how schools are educating pupils on gang culture.

Lottery lets children gamble £350 a week

This front page investigation lifted the lid on goings on at the National Lottery. Since 2009 the percentage of sales from highly addictive scratchcards and instant-win games has soared from 24% to 43%, fuelling a rise in profits and salaries.

We also revealed that Camelot, who run the lottery, allow children to gamble up to £350 a week online. The stories piled pressure on the government, which is expected to finally raise the minimum age of playing the lottery to 18 in the coming months.

Aquind investigation

The Aquind investigation revealed for the first time the extent of Russia-linked donations to Conservative ministers, and the access that these donors have gained to the Tory party.

The investigation also revealed how a former Russian oil company tycoon who has given £100,000s to the Conservative party was able to obscure his ownership of a billion pound energy infrastructure project.

Local lockdowns

Using data from NHS England we identified areas that could be hit with lockdowns by comparing each local authority’s daily figures to their peak day. Many of the areas we highlighted went on to experience a local lockdown — including Manchester, Rochdale and Sandwell — highlighting the effectiveness of this technique.

Perceptions quiz

Inspired by studies which show that people frequently misconceive societal trends, we set 13 questions on life in Britain and put them to the public via polling company YouGov. We then put the same questions to Times readers via a slider quiz.

The resulting visualisations showed readers how their answers compared to the average response from both the public and Times readers, which updated live as more people completed the quiz.

Diamond Princess

When the Diamond Princess was quarantined back in February it had the largest Covid-19 outbreak of any location outside China. Working with Katie Glass for the Sunday Times magazine, we created a series of detailed illustrations to help add context to the story and let the reader see what life was like on board the infected ship.

Staying safe while exercising

We teamed up with science editor Tom Whipple to look at covid transmission rates during different types of exercise. The findings, based on an aerodynamics study from the Eindhoven university of technology, found that it is a lot more complex than just keeping two metres apart.

Lifting lockdown: what Britain can learn from the rest of the world

Using data from the University of Oxford we looked at how quickly governments had reacted to the pandemic, assessing the impact greater stringency had on death rates.

The data gave us the opportunity to create a scrolly chart that highlighted how many deaths each nation had at the point of lockdown and how some countries were beginning to ease restrictions.

Behind the numbers

This year we launched a new digital series called Behind the Numbers which uses statistics to explain a trend or pattern within a newsworthy topic.

In the first piece in the series we investigated why more men are dying of coronavirus than women.

In the second, we explained the complexities that come with recording deaths in the UK and why it means using the daily mortality count for Covid-19 can be misleading.

When will a coronavirus vaccine be ready?

Working with science correspondent Rhys Blakely we produced a set of visuals to help explain the inside story of the scientific race to protect us from Covid-19.

We wanted to show readers — in layman’s terms — how vaccines work, how they are created and some of the different types currently in development.

How many coronavirus cases are being recorded in the UK?

We launched our UK tracker page at the beginning of March and have been updating it on a daily basis.

Using feedback in the comments section and on Twitter we have tweaked the visuals on numerous occasions to make sure we’re telling the story in a clear and accurate way.

The page is the most viewed page published on the Times website so far this year.

Lockdown is breath of fresh air for British cities

We analysed levels of nitrogen dioxide, a harmful gas produced by fossil fuels, and compared it with readings from last year to show one benefit of the lockdown: pollution levels have halved in British cities.

As well as small multiple charts showing how levels of NO2 have changed in the UK, our show-stopping visual for this was an animated map that revealed the impact of lockdown across the continent.

The race to be the Democrat candidate

Working with Washington Correspondent Henry Zeffman we created a digital guide to the US primaries, with visuals helping tell the story as candidates bid to become the Democrat nominee to go up against Donald Trump in November.

The coronavirus outbreak in maps and charts

The coronavirus outbreak has dominated the headlines so far this year, with the visual representation of the spread of the virus a key part of the coverage for a number of major news organisations.

This has led to much discussion around how to responsibly visualise the outbreak and after a number of iterations we have settled on a series of charts and maps which we believe accurately and responsibly tell the story.

We have streamlined our data processes to ensure the graphics are kept up to date and they are being used across our coverage to contextualise the global epidemics.

Inside the Times

From Anthony Loyd’s epic search for Shamima Begum to Christina Lamb’s exposé of cobalt production in the Congo, listen the inside story of our biggest scoops in this 10 episode audio series.

Houses of Lords expenses

The team’s first front page story of 2020 came in the Sunday Times as we revealed that the cost of Lords’ expenses and daily attendance allowance rose by 29% to £23m in just one year.

The average tax-free payment was £30,827, higher than the median salary of a UK worker, while 31 lords claimed more in expenses than the standard take-home pay of an MP.

Our analysis revealed that the latest round of peerages is expected to bring the total number of lords to 834, the highest since Tony Blair axed the majority of hereditary peers in 1999.

More than 110 peers did not make any spoken or written contribution to the House during the period, but claimed a total of more than £1m in expenses.

We produced a searchable table so our digital readers could see which Lords had claimed the most.

Innocent are left with enormous bills after cutbacks in legal aid

Our investigation revealed how tens of thousands of people have been left out of pocket after being acquitted of serious crimes over the past four years because the government ended the reimbursement of legal fees.

The investigation saw the launch of the first article in a new series, which will look at the numbers behind the story to provide context to our data-driven investigations.

Ten years of Tory rule: the long and short of cabinet careers

Ahead of Boris Johnson’s cabinet reshuffle we created our biggest ever* interactive, showing every cabinet from a decade of Tory rule.

Working alongside Red Box editor Matt Chorley this revealed how an early commitment by David Cameron to keep ministers in their roles was followed by huge churn under Theresa May as Brexit led to wave after wave of resignations.

The chart was made even more striking by Boris Johnson’s decision to promote a series of new faces to the top table.

*the longest we can remember, at least!

The Thousand Mile Trial: Confessions of an electric car virgin

The car industry is bracing itself for 2020 to be a breakthrough year for pure electric cars, with the number of models in British showrooms set to more than double and annual sales of zero-emission passenger cars potentially breaking through the 100,000 mark.

This led to the business desk launching a six-part series on the future of motoring in the UK and what part the electric car might play.

As part of the series Robert Lea considers what pure models are available, how far they go on a charge and how much they cost. This culminated in his attempt to recreate the famous London — Edinburgh — London road trip of 1900.

His road trip gave us the opportunity to build a scrolly-telling special, showing the details of his thousand mile journey around the UK in a Nissan Leaf, complete with thrills, chills and just a little emotional trauma.

The best of the Times and the Sunday Times data and digital storytelling team was originally published in Digital Times on Medium, where people are continuing the conversation by highlighting and responding to this story.

As the data and interactive teams at The Times and The Sunday Times have become more technically proficient a huge crossover of skills between the two has taken place.

A decision was recently taken to combine them, creating one multi-skilled group focused on bringing stories to life for our digital readers. Here’s why.

It better reflects what we do, but also why we do it

Interactivity is just one way that you can visually enhance an article and the team has the ability to tell stories in a number of ways. Digital storytelling better reflects this. We don’t want to be defined by a medium, but by what we can offer a project, from charts to maps or bespoke, personalised experiences.

However for The Times and The Sunday Times, creating a team focused solely on digital storytelling is more significant than that: it is another step towards implementing a reader-first approach and an important development for two newspapers with a 400-year history.

Working closely with the audience development team, this new desk will shape ideas using information we know about our readers and the wider goals the papers want to achieve. The aims for an article in sport are different to news, arts, business or travel, for example, and from length to tone this should be reflected in the way the story is told.

We want to ensure our digital subscribers are engaging with our journalism in the best possible format, whether that be reading, watching or interacting with a story. The data and digital storytelling team will be given the time and space to make this happen.

We can be more ambitious with projects we take on

Combining the teams has brought together not just a wide range of skills (scraping, statistical modelling, D3, spatial analysis) but a knowledge of subject areas (politics, sport, consumer journalism) to make our work as insightful and impactful as possible across different sections of the website.

We now also have consistent standards for data analysis. Everyone is using the same programming language (R), which means we can make sure our findings are reproducible before publishing a story.

The journey from data to visuals is a continuous one

Graphic representation is an important step in our data analysis, and the interpretation of numbers underpins the majority of our visuals.

Having the same people working on both means we’re able to consider the visual approach from the start of a story or investigation, which gives us more time to choose the best format for readers.

One team also allows those who specialise in both fields to work in close proximity and learn from each other. This means improved stories: our investigations are better presented, our visual projects use better data and are more agenda-driven.

Things are simpler for the newsroom

In the past few years reporters have been unsure as to which team they should turn to when they have a data-led or visual idea, with the fact that one can often lead to the other only heightening the confusion.

Now it’s clear who to talk to, with the broad range of skills we have in the team allowing us to consider the diversity of mediums available before we begin a project.

Why we’re combining our data and interactive teams to form a single data and digital storytelling… was originally published in Digital Times on Medium, where people are continuing the conversation by highlighting and responding to this story.

Scrollytelling mission reports, rockets whizzing through star-filled backgrounds and augmented reality moonwalks: the Apollo 11 anniversary spawned a flurry of digital projects that were almost as complex as the mission itself.

On the Times interactive team we’re always looking for an excuse to show off what we’re capable of. But six weeks out from the 50th anniversary on 20 July, we faced a dilemma.

Our goal was to show what could have gone wrong using the near misses and moments of panic to emphasise how close the Apollo 11 mission came to failure.

But we were yet to decide the treatment: did we go down the complicated, all singing all dancing interactive approach and build an immersive, development-intensive interactive page? Or did we simplify our visual style and lean on a few old tricks to bring the story to life and allow ourselves more time to perfect the content?

As visual journalists a lot of what we do is abstraction. The charts we build are not realistic depictions of life but present a useful angle in a simple, numeric format. Crime doesn’t actually look like a bar chart and inflation isn’t a moving line, but both those visualisations are useful, interpretable ways of showing those things. What if we applied the same philosophy to the moon landings?

Space travel graphics can be encyclopaedic — there is so much information to convey that they become crowded. Stripping them back to the key points can focus the reader on information relevant to the copy. This allows the graphics to communicate a specific message, rather than overloading the reader with information.

Out went the twinkling black backgrounds, hyper-realistic rockets and photographs of astronauts. Instead, we developed a minimalist style which worked for charts and diagrams alike.

This helped our readers see parts of the mission not as they recognised them in grainy 1960s footage, but in an everyday context: how the Saturn V was taller than the Elizabeth tower, how the moonwalk covered an area the size of a football pitch and how Armstrong and Aldrin could barely stand inside the cramped lunar module.

Five weeks out we had decided on our style but it took a three-hour storyboarding session for the piece to really take shape. We went through the draft copy line by line, discussing with our brilliant designers the best way of presenting each step. Being there in person, focused on the project, allowed us to efficiently debate ideas as we thought of them and make sure everyone understood the aim of the project.

The most challenging graphics explained the complex mathematics required to send astronauts 200,000 miles from earth. We wanted to show how a few minutes difference in rocket engine burn time could produce catastrophic results at two points in the mission: leaving earth’s orbit and approaching the moon.

For these two graphics we sketched out rocket paths for three potential routes: what would happen if they burned too early, or too short; what happened if they burned too late, or for too long; and the “goldilocks” route. We decided to use animation to make them clearer, having each path trace one by one.

We could have used GIFs or videos but instead went for a solution inspired by a previous project. Back in 2015, we produced an explainer on GCHQ, where the user scrolled through the article to trace the paths of SVG line drawings. This technique looked like the neatest way of tracing the rocket’s path through space. It’s also extremely efficient for a reader’s bandwidth and load-time.

To complement the animations, the copy described the three paths of the rocket in the order that they appeared, leading the reader through the visuals step by step.

The other graphics were done with AI2html artboards, a Times interactive team staple. This gave us the greatest control over what our readers saw and allowed us to prioritise clarity and resist the temptation to pack each graphic with unnecessary annotation.

Ultimately our decision was vindicated. We were able to spend our time focusing on the graphics and the words, rather than the technology that underpinned them.

But above all, our aim was reader enjoyment. The evidence suggests we were successful: we received hundreds of positive comments, while the average reader spent six minutes on the article. It also had more readers than any other story we published on the moon landing.

There will always be a drive in interactive journalism to produce bigger, better and more complicated pieces to wow your readers (and impress your peers). But in the end, just like Buzz Aldrin fixing a lunar module circuit breaker with a felt-tip pen, sometimes the simple solution is the best one.

You can read our moon landing piece here.

Apollo 11: why we chose the simple over the complex was originally published in Digital Times on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Ryan Watts and Daniel Clark of The Times and Sunday Times interactive team

One of the reasons we add interactives to a story at The Times and The Sunday Times is to help guide readers through important but complex data. The data around air pollution fits this description to a tee.

The units of measurement need explaining, the weight given to different pollutants requires thought and determining whether the results are good or bad when the full health implications are unknown is complicated.

On May 9 The Times launched the Clean Air for All campaign, which examined the scale of Britain’s dirty air, accused the government of inaction in tackling pollution and called for the introduction of a new Clean Air Act.

The campaign has featured on the front page of the newspaper four times with more than 30 pieces published on the subject.

The interactive team were excited from the off. Air pollution is the perfect topic for personalisation — readers want to know how it affects them personally — and there is scope for producing visualisations which both wow and inform our readers.

We decided to focus on three projects:

An alternative league table of air quality at London’s schools

This showed readers how air quality at any particular school compared to others in the borough, how it was predicted to change in the coming decade and whether a school near by offered cleaner air.

How bad is the air you breathe? National postcode lookup

Using government modelling we compared the air quality in a chosen postcode with that in the rest of the country. We also mapped how far a reader needed to move to live in an area where air quality would meet World Health Organisation guidelines.

How do we make the air in our cities breathable? A visual explainer

We wanted to be constructive and look at how we might fix the air pollution problem. This project used a bespoke storytelling template, similar to the stories feature on Instagram, and aimed to redesign a London street to promote healthier air.

All three projects were collaborative efforts with unique challenges. Below we explore the obstacles we faced in researching, designing and developing our school league table project and how we overcame them.

Challenges in data journalism

Finding the data was our first hurdle. Britain is not set up to measure air pollution across the country, although installing a monitor in every postcode is one of our campaign demands, so we knew that we would need to rely on modelling.

This modelled data, where pollution levels are estimated based on real data, needed to be of a high enough density to return accurate readings for every school in the capital. Thankfully the Greater London Authority publish ? 20-metre by 20-metre data, which gave us thelevel of detail required.

Outside London the data is only available in 1km by 1km squares, which was not detailed enough to create a score for specific schools. Air pollution outside the capital is significantly worse than elsewhere, however, so we were satisfied that a London-only tool was justifiable.

We initially planned to use school postcodes to match them with the air pollution data, joining each school to the nearest point. But when we overlaid our geo-referenced postcodes on to an Ordnance Survey basemap, we found that these postcodes didn’t always fall within the boundaries of the schools. When we added the air pollution data we also found that there were multiple points within the grounds of any one school.

Using the closest point felt a little arbitrary and it wouldn’t capture the best and worst air quality to which a single school site was exposed. We decided to use shapefiles of school boundaries instead, and if a point of air pollution data fell within the shape we added it to the school’s file. Averaging these points gave us a much more accurate estimation.

Working our way through this process required a significant amount of experimentation but ultimately we were able to produce a script that took in five million rows of air pollution data and accurately matched it with the shapefiles of 3,000 London schools.

We found that an element of data wrangling was required for each of our three projects (and for other stories in the campaign), so keeping a single, well-documented repository of scripts made the process much easier to manage.

Challenges in development

The main challenge we faced from a development point of view was the volume of data we needed to host and display. To put this in context, we had detailed data for more than 3,000 schools in London, each with 20 to 25 data fields, and 1.7 million postcode records across the UK.

For projects with less data we may load this information from a file within the interactive, but the data for the national dashboard was 434MB, which would have drastically slowed down the reader experience, so for us this wasn’t a palatable option.

To overcome this we decided to create an application programming interface (API) that would read information from a database and return it to us as and when we needed it. We started by importing the data into an SQL database using Sequelize — a tool used in Node.js for mapping data into a database.

The next stage was setting up a simple API with two different end points for schools and the national picture. For the schools version, we needed a way of returning the selected school as well as all of the schools within a borough. The code below shows how we utilised the lambda event object to handle this, meaning we could pass in ?school_id=… or ?borough=… when calling the API.

const { urn, borough } = event.queryStringParameters || {};

We could then use these variables when querying the database to make sure that the correct information was returned.

Finally, we deployed the API to AWS Lambda via serverless, a framework that we’ve written about in depth before. We learnt a lot throughout this process, and we’ll definitely be using this method of data storage again for future projects.

Challenges in design and data visualisation

The biggest challenge from a design point of view was getting colours that worked. We wanted to use a consistent palette that readers could use as a shortcut for working out whether air pollution was high or low. This would be difficult with two different measurements so we opted for one: particulate pollution.

We discovered that every school in London has an estimated particulate pollution level higher than the WHO limit. Using a green-yellow-orange-red colour scale — commonly used when presenting air pollution data — was therefore misleading, as green is too often associated with a healthy environment, or clean air in this case. Only having red and yellow helped to communicate the seriousness of London’s air pollution within every search result.

In reusing the same colours throughout our work, the same scale was present in each section of the interactive: two charts, a map and table. This allowed us to use a contrasting blue to represent a pollution level lower than the WHO recommends, which made a striking point in itself.

What’s next?

Clean Air for All is a live campaign and our attention has turned to where we can contribute next. Keeping journalists, designers and developers involved from the start led to one of the most striking projects we’ve produced this year and when we do build a new air pollution project we’ll be sure to keep the process collaborative from the off.

Try the interactives for yourself

• London air pollution mapped: how does your child's school compare?
• Air pollution postcode checker: What is the level near me?
• How to solve the UK's air pollution crisis

How we used interactives to bring The Times’ clean air campaign to life was originally published in Digital Times on Medium, where people are continuing the conversation by highlighting and responding to this story.