We have moved was originally published in BBC News Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

BBC Design + Engineering

We have moved was originally published in BBC iPlayer Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

There is a lot of collaboration across the BBC’s Design & Engineering teams to bring the World Cup and Wimbledon to the BBC. From the actual application, over scheduling behind the scenes, how the actual video transmission makes it way into iPlayer and BBC Sport and how this lets users actually watch a live broadcast.

This year we are also broadcasting for the first time in UHD (Ultra-High Definition) resolution for the World Cup and selected Wimbledon matches.

Let’s start with how most people watch a match — they tune in into the match. BBC One is a pretty good bet to try if you’re in the UK, and if you’re lucky might see one of our ‘UHD Call To Actions’ on the screen and one of the commentators tell you this:

You can watch this match in Ultra HD via the BBC iPlayer

Someone decided that some of the World Cup matches are broadcast on the BBC in HD and UHD — more on the production side, and how our UHD trials worked this summer in this blog post from BBC R&D (Research and Development).

Once we know the schedule, we co-ordinate the schedule for our Audio/Video encoders to deliver the correct streams for the correct events at the correct times. For the World Cup, the UHD streams are actually completely separate episodes in the BBC programme structure, but embargoed in the system so they do not show up anywhere, including programme guides.

For Wimbledon, we also can’t use the same episodes as on broadcast TV since the broadcasts sometimes switch between channels, and are split up, so the UHD broadcast is attached to the equivalent Webcasts.

For the UHD streams, the encoder, packagers and backups, as well as the CDN (Content Delivery Network) configuration are controlled manually. This means, that prior to the match BBC Sport staff in Salford co-ordinate with their Media Services and OTG (Online Technology Group) colleagues in London, who are responsible for the stream delivery, and the broadcast staff in Moscow responsible for the contribution feed coming into London to turn everything on in the right order and start the stream.

This needs to happen before the broadcast begins because as soon as the broadcast episode becomes ‘available’ on iPlayer, the TV app starts to offer the UHD version — and if the encoder isn’t turned on, that does not look pretty.

Once everything is running, our CDNs are warmed up and we see no issues, we are getting closer to the match. The ‘Call to Action’ (CTA) for users to switch to iPlayer for UHD actually comes from the Outside Broadcast truck in Russia.

When everything is prepared, we are ready to serve the live streams to the audience, and that is co-ordinated across the different iPlayer teams.

For the app on smart TVs and set-top boxes, the basic architecture is a pretty typical client-server one. We have a common client application which is used for all of the BBC’s Connected TV products and numerous backend services. These services often take the form of a Node.js application deployed to Amazon EC2.

Our usual scaling process is to scale in response to AWS CloudWatch alarms which are hooked up to appropriate metrics (e.g. memory utilisation, CPU utilisation, network throughput) for the service in question. This approach works well with our usual traffic patterns as they are reasonably gradual increases. When an on-air CTA is made, however, our traffic increases at a rate far too great for AWS AutoScaling to have an effect. We have seen the network traffic through our NAT Gateways increase by 150% in the space of 60 seconds and traffic we send to the API that powers the iPlayer homepage tripled in less than 20 seconds.

Nik Rahmel on Twitter

I wonder if we can thank the @EveningStandard for a strong peak in requests to iPlayer at 7pm?

As soon as we find out which games are due to be broadcast on BBC One (and therefore in UHD) we start analysing the times to determine when the peak traffic points are expected to be. We liaise with our colleagues from across the whole of the BBC to determine if there are any on-air CTAs planned and if so, when they are likely to occur.

Once we’ve got all the necessary information we begin planning our scaling rules. Unfortunately this process is (for now) rather manual as we make use of AWS’ Scheduled Scaling. Internally we’ve started looking at various other solutions that would allow automated scaling to be effective in these situations — hopefully we can write another blog post about this in the future.

To enforce a limit on the number of UHD streams, we couldn’t rely on previously used measurement systems, as the latency is too high. At kick off and on-screen CTAs, we have seen the number of viewers increase by a third within 60 seconds. We need to ensure that we can “close the gate” reliably when we reach the limit, as there is a real possibility to reach the limits of the UK internet infrastructure due to the bandwidth requirements of UHD — roughly 8x that of HD.

There is a Counting Service built by our Live Sport team, which uses AWS Kinesis Firehose to collect the viewer-count statistics, which was linked to the “gate”, and heavily used in the Live Pages — Mark Woosey has blogged about the details separately here.

BBC Account

In order to allow users to see iPlayer or Sport content, we ask you to sign in to your BBC account. This is a collection of applications which open the gateway to user personalisation for the whole of the BBC, and not just for iPlayer and our live streams.

As with many of the modern BBC services, our applications are a set of Node.js microservices, which are deployed using AWS cloud services, and communicate with other BBC services via HTTP requests. We have an isomorphic React application which displays the sign in and registration forms and then securely handles your credentials to confirm your identity. We’ve spent a lot of time ensuring that the applications can be scaled effectively for large events and we do so using AWS autoscaling. Our amazing Quality Assurance (QA) team had already spent a lot of time performance testing our systems in the past in order to understand the expected traffic patterns and server load for our usual “big” events, when we authenticate users during the live online votes for Strictly Come Dancing and Sports Personality Of The Year. That said, we knew that our audience would be keen to watch both the World Cup and Wimbledon, so we performance tested some different traffic profiles, and then added contingency in the shape of additional virtual server instances to support potentially higher loads.

Needless to say you didn’t disappoint us!

Marc Burrows on Twitter

@MarcLittlemore Can you tell when england scored

As you can see from the tweet of one of our graphs during the England vs Sweden game, we had huge peaks of traffic as you signed into, or registered for, your BBC accounts at the start of the game. In fact, we had the most registrations we’ve ever had during a single day! You can also see where the two England goals occurred from the second and third peaks. People were quick to sign in and watch the goals again on our iPlayer/BBC Sport streams.

While you just want to watch the World Cup goals on our live streams, our goal is to verify that you have a valid BBC account which allows you to watch the BBC content that you love as quickly as possible. During the peak periods of the England vs Sweden game, we had around 24 times the amount of users signing in that we would normally expect during weekend usage, so our additional application resources help us to serve requests as quickly as we can. Our applications also help to support the BBC account mobile team, and their framework is used by all of the BBC mobile applications, such as iPlayer on your phone or tablet.

Once signed in, we authorise users to access various BBC services by means of an access token which, to ensure our and your security, refreshes on a regular basis. During the peak traffic of this game, we saw approximately 5 times the normal amount of traffic as people turned on their mobile devices, we refresh these access tokens and then you watch the World Cup goals quickly after they went in.

We try to be as unintrusive as possible so once we’ve signed you into your BBC account, we send you back to the page you clicked “sign in” from. For the World Cup and Wimbledon this was probably the BBC iPlayer or BBC Sport Live pages.

Playback

Every press of a play button triggers a call to our Media Selector API. This service uses a range of factors including the following to decide whether your device is authorised to play a stream, and if so, where it should play it from.

• Device type
• Your location
• The rights of the asset being requested
• Current business rules for load balancing across CDNs based on capacity and performance

You can find out more about how we built Media Selector 6, our current iteration in Henry’s post here.

Media Selector generally serves between a few hundred and a few thousand requests every second, with regular but often unpredictable spikes in load. Before the World Cup, its traffic peak was when BBC News published a push notification after the stage invasion at Eurovision.

With the Media Selector response in hand, the device then selects a playback URL from the options provided to it by Media Selector, and starts streaming from a CDN.

In addition to our own CDN, BIDI, the BBC engages with two commercial CDNs to be able to handle the users and bitrates our streams require. As well as providing huge amounts of network capacity and substantial cache offload, the CDNs also have a lot of intelligent logic that try to protect users from transient issues such as slow network paths to the content origin.

Each CDN has its own set of rules around retrying requests to our origin servers, timing them out, or trying to route around specific parts of the network. Tuning these rules is a very complex and nuanced process, where every timeout value changed needs to be considered in the form of what impact it will have on every other part of the distribution chain. In the event of a CDN having issues providing a stream to a user, our Media players Standard Media Player (SMP) and TAP will automatically attempt to use another CDN rather than showing an error to the user.

Each CDN has a quota of UK distribution capacity that they allocate to the BBC for use — how many bits they think they can push before performance will begin to degrade. Throughout the World Cup performance and capacity were reviewed across different platforms and services as we broke and re-broke our peak throughput records, regularly re-routing traffic whilst tweaking configurations as they hit new barriers that if not addressed would cause quality to degrade before we hit our total caps on the CDNs.

There is good news with capacity planning though: football fans are predictable. The number of users and amount of traffic delivered at kick-off for a football match is usually around a factor of √2 out from the amount of traffic we’ll be handling at full time. This essentially means that as soon as the timer starts, we know what, if any, traffic engineering will need to have taken place by half time to keep things running smoothly.

One exception to this rule has been UHD. We initially expected it to follow the same pattern but the traffic levels have been far flatter — it looks like the CTA which stated that there were a limited number of viewers allowed caused everyone to try and join early and stay on. Whilst for HD there’s a dip in concurrent viewers at half time whilst they pop the kettle on, UHD stays stable (perhaps because viewers know that if they close their UHD stream, they might not get back on after half time due to the limit on concurrent viewers).

Digital 24/7 Operations

What it’s like to be on shift — a personal insight

Operations, I like to think, is the glue that holds this whole process together. We are the eyes and ears of the BBC.

Prior to one of the matches starting, we prepare. And once, we’ve prepared, we prepare some more!

This includes checking the UHD encoders have started, Wimbledon & World Cup promos have been scheduled ready to be displayed on iPlayer to chairing our tier 1 event stand-ups, ensuring our RAG status is green across our broadcast, online & network services & sending out comms to all our stakeholders.

We constantly monitor.

Now, we love our monitoring in 24/7, as you can see from our monitoring wall image below!

We monitor everything from our content delivery network, live video & audio output for iPlayer & iPlayer Radio, live sport pages, RedButton and whole heap of other alarms, products & services.

I work with very talented people in 24/7, some of whom have built bespoke monitoring tools to solve problems we have had here.

For example, for AWS Alarms we have LookOut. LookOut has a UI that utilises colours to help identify the state of alerts at a glance and exposes key elements to members of the Ops team, e.g. Wormhole links, SQS queue fetcher and it gives us the flexibility to customise visibility of alarms in the UI.

For monitoring our content delivery network we have Sawyer. A script that consumes the output of an API and performs calculations to determine the difference between the CDNs, compared against a customisable threshold. A UI that displays the desired metrics in a quick-to-digest dashboard view, with changing colours to draw attention to deviations between our third-party CDN providers — we like to avoid the buffer faces for our Audience Members!

Our monitoring tools ensure that we are able to identify & resolve incidents fast!

And when the match is over?

Did I not mention we have another ~1,400 components that we support on top of the components being utilised for the World Cup and Wimbledon?

Headline Stats

Now that both World Cup and Wimbledon are over, and with that our special events UHD live trial, it’s probably good to reflect on the month. There were some initial hiccoughs — co-ordinating the CTAs with the Moscow team the impact of unprecedented traffic spikes on a new iPlayer homepage architecture and event based scaling of the underlying infrastructure.

But by the end of it, we have broken several records multiple times:

• Over 5.5 Terabit/s in live video streams
• 3.8m concurrent streaming requests for the England vs Sweden match
• 15,000 requests for the VR version of the match stream
• 1.6 million requests for the UHD version over the course of the tournaments

Of course there were some other problems — A large proportion of non-UHD viewers will have noticed the stream cutting out just before the end of the England vs Sweden, which was swiftly dealt with. Simon Thompson wrote another post summarising the teething problems we had and learned from with UHD.

Thanks

Thanks to everyone involved in writing this blog post and to Jenny Wong on Twitter for suggesting that this would be an interesting topic. Thanks also to Quinn Cowper the provide us with some of the picture

Ask the teams any questions and we’ll be happy to answer as many as we can.

Bringing the BBC’s World Cup & Wimbledon online coverage to you was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Anybody who uses AWS Lambda might have come across something like in the screenshot below when trying to access any kind of log, and felt emotions such as confusion, anger, and annoyance.

Those emotions were probably even stronger if you have been spoilt by having an ELK stack for your all your centralised logging needs for all of your microservices — maybe even with some neat Kibana dashboards that allow you to dig into the data, filter it, and visualise it.

That’s how we felt anyway on the iBL team (the iPlayer API team) — for the past 3.5 years all of our microservices sent their application and access logs into our Elasticsearch cluster, making monitoring and operational investigations a breeze. Since then we’ve started utilising AWS Lambda a lot more for various tasks — be it for serverless, decoupled integration with other AWS services, or to handle certain traffic requirements.

But with serverless environments, there comes one issue: Not having access to the server environment. No journalbeat or filebeat for us to ship off logs to logstash, instead AWS just puts everything into CloudWatch logs, and by default not in the most accessible way. There are some teams in iPlayer who are fully utilising CloudWatch for their logs, and I’m sure that they enjoy just having Lambda logs delivered without much work, but after reviewing our requirements for iBL, we decided to stick with ELK for our main logging tool — among other things it is a lot cheaper to run our own ElasticSearch cluster.

In order to tie up Lambda logs into that, we have.. created another Lambda!

The Pipeline

CloudWatch Logs has a feature called Subscriptions for “Real-time Processing of Log Data”, which is just what we want. It allows sending log events to a Kinesis Stream, Kinesis Firehose Stream, or a Lambda. It is created on a log group. Each log group contains a number of log streams. The screenshot above shows a number of log streams for a log group — and each Lambda gets its own log group created.

So what do we do now with this feature? For each of our existing Lambdas’ log groups, we create a subscription to send the Logs to the new Lambda, the iBL CloudWatch Logs Processor. As per the AWS Example Guide, the event the Lambda gets invoked with looks something like this after parsing:

{
  "owner": "123456789012",
  "logGroup": "/aws/lambda/ibl-example-lambda",
  "logStream": "2019/01/11/[28]cde6dd2161e44c61823b6b73d5e8471e",
  "subscriptionFilters": [
    "cloudwatch-logs-processor"
  ],
  "messageType": "DATA_MESSAGE",
  "logEvents": [
    {
      "id": "3195310660696698337880",
      "timestamp": 1432826855000,
      "message": "{\"eventVersion\":\"1.03\",\"userIdentity\":{\"type\":\"Root\"}"
    },
  ]
}

From the logGroup key, we extract the function name, and create our payload:

const { logGroup, logEvents } = logStream;

const payload = logEvents.map((event) => {
  return {
    function: getFunctionNameFromLogGroup(logGroup),
    timestamp: event.timestamp,
    message: event.message
  }
}

On our logstash server, we have created a dedicated input on port 5090 to add a lambda type to the events. They get stored in our ElasticSearch cluster, alongside all other application logs, and are easily accessible through Kibana, so we end up with the following architecture:

Automating Subscriptions

So far so good — but what when we create a new Lambda? We could add the Subscription to our CloudFormation templates that our yeoman generator creates when we start a new lambda project.. or we can go a bit meta and have the CloudWatch Processor create those by itself!

Of course, we go meta.

We can use CloudWatch Events to invoke the Logs Processor every 30 minutes to check for any new Log Groups that need subscriptions created for. Using the AWS SDK, we get a list of all Log Groups and their existing subscriptions, filter out all those that already have subscriptions, and create subscriptions for the remaining ones. Our new architecture diagram does not look too different now:

One thing to be noted is, that we shall not create a subscription for the Log Group of the Logs Processor Lambda itself — every invocation would generate a log that it has been invoked, and in turn invoke it again. A neverending cataclysm of Lambda invocation would not make for a pleasant experience!

The End Result

Getting all log events pumped straight into ElasticSearch allows us to create dashboards in Kibana like this:

Or we can find all log messages from a select Lambda containing the name of a service we’re investigating:

Of course, there are various alternatives to this — AWS CloudWatch now supports Logs Insights for more effective querying, and Grafana is about to release Loki, which is much simpler and does not require an ElasticSearch cluster.

Let us know if you have any thoughts about the above, or if you fancy using or improving it yourself, why not join us?

Lambda Logs in ELK was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

API Engineers as Consultants in collaboration with user-facing product teams

iBL stands for iPlayer Business Layer — yes, we are ‘the API team’, but we are also the team that implements any shared business logic that affects iPlayer on any of the 3 key platforms (TV, Mobile and Web).

As an engineer in the iBL team, we’ve got great responsibility in maintaining a robust and resilient API. We are also very closely integrated with the audience facing teams to drive the product forward. We consult the teams on how the data works, and what impacts their decisions will have on the rest of the product.

Recently I have been consulting one of our audience facing teams on a new feature — from discovery to implementation, I wrote very little code and helped deliver a major milestone for iPlayer.

This is the story of how we go about embedding engineers as consultants in other teams.

iPlayer Product Teams

iPlayer runs on many platforms — we have the website, the mobile App on the Google Play Store, Apple AppStore (which includes Apple TV) and Amazon Store, and of course connected TV devices, set-top boxes and other syndicated platforms such as Virgin and SkyQ.

We have many product teams across all the major platforms from connected TV to Mobile to Web. All the teams are cross-discipline teams with engineers, testers, UX Designers, Project Managers. Each product teams are all consumer/clients of the API, the team I work in, called iBL. Kit wrote a nice post detailing the tech stack behind all of this.

iBL — The product API

A lot of user-experience defining features on iPlayer can be categorised as data-driven or UI-driven.

To give an example, guidance warnings on certain episodes. Some episodes have warnings about strong language or similar. The way this message is communicated to the user is primarily a UI feature, but the actual message is data feature — and data features like these are decided in iBL. Which episodes have guidance warnings, what the message is, all this comes from iBL, and we can change it. But any change we make to the data need to be understood by the clients.

In this post I will go into some detail about how iBL works with the platform teams, and other supporting engineering functions within the BBC.

The product question

The specific question we want to answer is: What impact does changing the promotions on the top have for a certain age range?

Specifically, we want to explore, deliver and measure the impact of a personalised experience for the ‘youth segment’ of 16–34 year old users who have opted-in to personalisation. This requires teamwork, not just from the product teams above, but also the editorial curation and strategy teams.

Why does iBL need to be involved in this to consult? Users sign on in the clients, not in iBL, so the clients can probably figure out the users' profile without help from iBL. But this is where it stops — the clients would need to know what a different age group means. They need to know what episodes are promoted, where the promotion comes from, how the promotions are scheduled. All of this is data. Data that is in iBL, data that is part of the cross-platform business logic.

However, iBL can’t just make the decision to serve another feed to the youth audience — we do not have the resources to analyse the best ways this feature works with the rest of the user experience. Getting a product team to look after this means there are UX, product, and data analysis resources to prove the hypothesis.

The Data and the product

As first step to a more personalised experience in iPlayer, we have spent the past year revamping how the homepage is assembled. We now have one GraphQL API that returns a highly personalised view.

This really is a change to be implemented in iBL. However, iBL is a team consisting of four engineers — the product managers are embedded in and managing the product teams. This is a product feature, and needs input from disciplines other than engineering — product, UX, editorial and our data analysts.

This is why we decided to ‘open up’ iBL to contributions from client teams — they know the product, the audience and the user needs, and we can facilitate this collaboration. The ‘iBL Consultant’ is born!

Consulting

As a consultant, our responsibilities are beyond code, and sometimes do not even include any coding at all. There are three main pillars:

• Supporting Discovery
  This includes advising on what’s possible — client teams, including engineers, are not usually aware of the capabilities of the platform built by iBL. Working with the client team in workshops to define the feature and elicit requirements.
• Technical Delivery
  Providing technical support to the client teams — if it is a feature for which iBL provides a ‘starter kit’, say a new bundle based on existing feeds, the client team may decide to have it implemented by their engineers. Alternatively, we may decide to run a developer exchange of either an iBL engineer into a client team, or a client team’s engineer into iBL to pair on aspects of the feature.
• Delivery/Project Management
  The iBL roadmap is rarely empty, and iPlayer project managers need to keep informed about what is happening. As consultant, we manage prioritisation of any work relating to the feature within iBL and ensure deliverables are aligned across all platforms with the lead project manager for iPlayer.

It should be noted that, once delivered, ownership of the code lies with iBL.

The Youth Bundle

At the beginning of this project, the objective was to create a personalised homepage experience for 16–34 year olds, which can be A/B tested. The scope of the ‘experience’ was not yet defined. That’s where the discovery session kicks in! The Editor of iPlayer, the Editorial Manager, and me as a consultant make their way up North to create that definition with the Mobile product team, and their ‘Ada’ crew of engineers.

Since this was going to be the first time we are segmenting the audience in iBL, we wanted to keep it fairly basic. Some of the main questions we tried to answer:

• What is the experience going to be like?
  Change the whole page? Add a new bundle? Modify an existing bundle?
• How do we curate this new experience?
  Is it all manually selected? Can we use existing data (e.g. channel, genre, category) to select the content automatically? Can it be semi-automatic where the editorial team can define weightings or priorities? How much effort will this be, both upfront in engineering cost, and to keep it running editorially?
• What timeline are we operating on?
  Are there any big events coming up? Are there any big releases coming up?
• How do we build this thing?
  Are we re-using existing feeds? Do we need new feeds? Who else do we need to collaborate with in order to deliver this? Does iBL need to do any work, or can the client team deliver this themselves?
• How do we measure the impact?
  Whilst we have A/B testing on certain platforms as Andy explained in a previous post, extending this to measure the same thing on all platforms equally is a bit more difficult.

There were introductions, hypotheses, post-its, a big whiteboard matrix, KPI definitions, edge cases and much more. The product team’s Business Analyst, Project Manager, Product Owner and Engineers, who are used to defining mobile app features and not data feeds, had to wrap their heads around a whole new world of workflows — and how it all works when you translate it to the website or TV devices. This is where I come in as a consultant from iBL, with experience of all that, and knowledge of the respective implications of the above questions on it.

Eventually, it was decided that we will modify the existing ‘Featured’ bundle with a specifically curated feed for the target audience segment. We also decided that we will run an A/B test on the new feature to assess any impact on key metrics such as consumption.

We also had to liaise with another team in the Online Production Systems group to create the new promotion destination in our editorial tools.

Since this was the first team we make use of segmentation in production, we also had to define an architecture for incorporating that in iBL — this was a discussion I had with the rest of iBL team.

Once all of that was ironed out, we could get started on modifying the existing editorial bundle. For that one of the mobile engineers, who works usually on Android in Java and Kotlin, joined us to implement this in our Typescript codebase.

We enabled the new feed earlier this year, and luckily I fall into the variant segment, so for me the Editorial bundle looks like this now:

Results and next steps

As for the feature, this was kept split for a couple of weeks, to observe the impact this has on audience behaviour.

Our data analysts then have some fun crunching numbers, and as outcome we have now rolled this out to the whole audience in the 16–24 age group only as this saw the highest positive impact.

We are also working on our technical capabilities to make testing a feature like this easier in the future. In the future, segments may not be as easily defined, or a lot more dynamic, so we are working to establish one common A/B (or even more variants) workflow for testing the impact of different bundles and promotions across all platforms which can be used as easily as the feature A/B tests we run on individual platforms already. This is a different challenge, and I’m sure there will be a different post about that at some point.

Engineers as Consultants?

Does it work?

For this feature, it worked. Why?

First of all, the product team didn’t just ‘look after it’, they made it their priority for the quarter. The product manager, BA, and project manager wanted to get results, and understood the need for iBL consultation and cross-platform approach.

The engineers in the client team were keen to get involved and work with iBL, and I was also motivated by the impact of consulting on this. In terms of UX, the team could design and effectively conclude their hypotheses of how it should work and feel.

Did anything not work?

We had some issues with the data analysis — the client teams do not have their own embedded data analysts beyond the BAs, so it wasn’t clear who was responsible for this. The teams are also split between MediaCity near Manchester, and White City in London.

Once this was clear, we could analyse the data and had a conclusion. But to improve this in the future, we are starting an iPlayer-specific data team based in London, ‘Data Force’, which you’ll probably hear about in another blog post. They have a logo and latin motto and everything..

I also believe that team and people dynamics play a big role — it needs to be the right feature, and it needs to have buy-in from the product team and the consultant. In this role, the consultant is working with people they may not usually work with, so it’s even more important to have something interesting to work towards.

Conclusion

I hope this gives an insight into how we tackle cross-platform collaboration at iPlayer, and that the life of a “backend engineer” in iBL can be a lot more exciting than one might think — Educating and Informing Discovery is very much in the BBC spirit as well.

If this environment sounds like something you want to be a part of, maybe there is a role open that suits you.

How an iPlayer API engineer delivers platform features across platforms was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Over the past year, we have started personalising the iPlayer homepage a lot more than we did previously — with this came a re-architecture of the API.

All the data any of the iPlayer clients require to show the complete homepage, be it in the browser, on the TV or in the mobile app, can be requested with one request to the underlying GraphQL API.

Each response is personalized to the user — it includes their currently watched shows, their ‘added’ list and recommendations tailored to their watching behaviour.

All this personalisation makes each response unique and difficult to cache on the edge caching layer.

Where previously we could utilise a simple cache layer built using Varnish for a few hundred different homepage variations, this has become impossible with the new architecture.

The computational overheads for each request have increased massively, which requires us to make our code as efficient as possible and optimise any CPU intensive tasks. During initial development, we didn’t meet our prototype performance expectation to handle 3,000 requests per second without breaking the bank, so we started to investigate where our bottlenecks lie.

Thanks to the excellent debug functionalities provided in current versions of Node.js (we are running a TypeScript project on Node.js 8), and the Javascript debug tools in Chrome, simple profiling identified the task that takes the most CPU time in our average requests: Translations.

Translations have been a part of iPlayer for a long time — the website is localised to English, Welsh, Scots Gaelic and Irish Gaelic, with hundreds of pre-set translation strings, and all of these get applied depending on the user in the translations module of this API.

Within the iPlayer API, we use a translation module to convert text (or parts of the text) into a specified language. This is achieved with templated strings. For example, a templated string like “#{available-for} 26 #{days}”, would become “Available for 26 days” in English or “Ar gael am 26 o ddyddiau” in Welsh.

The module to do this is relatively straightforward. It exports a translate function which accepts a templated string and locale code. Translate parses the string for translation keys and looks up each key in the corresponding translation lookup. For example, a Welsh translation (cy), `available-for` maps to `Ar gael am`. There are approximately 100 translations per language.

Translation keys in a string are represented using the following syntax: ${example-translation-key}.

Here is an example of an English and Welsh :

const templatedString = '#{available-for} 26 #{days}';

const toEnglish = translate(template, 'en');
console.log(toEnglish) // "Available for 26 days"

const toWelsh = translate(template, 'cy');
console.log(toWelsh) // "Ar gael am 26 o ddyddiau"

Given the levels of caching within the iPlayer architecture, modules like this have often been written to favour simplicity, testability and accuracy rather than performance.

Here is the original translate function:

const TRANSLATIONS = loadTranslations();
const KEY_REGEX = /#{(.*?)}/g;

function translate(value, language = 'en') {
 let translated = value;
 Object.keys(TRANSLATIONS[language]).forEach((key) => {
 const translation = TRANSLATIONS[language][key];
 const re = new RegExp(`#{${key}}`, 'g');
 translated = translated.replace(re, translation);
 });

return translated;
}

This function will translate zero or more translations templates in a given string. It works, and it’s very simple but it’s inefficient. Any ideas? Well, for the specified language it attempts to translate every key in the translation lookup. Based on the way this module is currently used within iPlayer, each call will translate between 0 and 3 translations. In a worst-case scenario, it will attempt to make approximately 100 template translations even if a string doesn’t have any.

A potential optimization is to only perform the required number of translations for a given string:

const TRANSLATIONS = loadTranslations();
const KEY_REGEX = /#{(.*?)}/g;

function translate(value, language = 'en') {
 if (!value) {
 return value;
 }

let translated = value;
 let match;
 while ((match = (KEY_REGEX.exec(value)))) {
 if (match) {
 const translation = TRANSLATIONS[language][match[1]];
 if (translation === undefined) {
 continue;
 }
 const key = "#{" + match[1] + "}";
 translated = translated.replace(key, translation);
 }
 }
 return translated;
}

From strings containing between 0 and 3 translations, benchmarks indicated that selectively replacing translations keys proved significantly quicker. Of course, it has a couple of drawbacks. First, although not rocket science it does add further complexity. Secondly, the performance plummets when a string has 4 or more keys due to the while loop.

Further inspection of the Javascript string replace method signature suggested that a replace function can be supplied as a second parameter. This replacement function will be invoked after each translation is found.

Here is a variant of the translation function using “string.replace” with a supplied replacement function:

const TRANSLATIONS = loadTranslations();
const KEY_REGEX = /#{(.*?)}/g;

function translate(value, language = 'en') {
 if (!value) {
 return value;
 }

return value.replace(KEY_REGEX, (a, b) => {
 const translation = TRANSLATIONS[b];
 if (translation === undefined) {
 return a;
 }

return translation;
 });
}

Arguably a more elegant solution. As a rule of thumb, it often pays dividends to reuse standard libraries or trusted third party solutions rather than rolling out your own solutions. Time has already been invested by talented engineers to optimize and make the code robust. Furthermore, standard libraries are officially supported and will be well tested.

This was evident is the benchmarks; this solution proved to be as fast (approximately) as the first optimisation for 0–3 translations but dramatically quicker for 4 or more.

One of the main objectives when it comes to optimisations like these is the user experience, and serving the majority of requests as quickly as possible.

In order to validate our changes, we run load tests on a controlled test environment. Alongside the profiling detailed above, this informs our prioritisation of potential optimisations.

For our initial validation, we had to find the current breaking point. We use Gatling as a load testing tool, with a script that can make personalised requests for any number of user profiles and client configurations. We can pick to test with a representative split of the audience, majorly favouring the English language, or an even split of all our supported languages.

Since our objective was to improve the CPU cost of translations, and not the way we store or cache the results, this is largely irrelevant, and we tested with an even language split. We picked 2 instances of a relatively small machine type, and a relatively low number of users, 30; each making up to 1 request per second.

Looking at the response times over time, we get the following graph:

Here we see that response times start out stable and with low variations in response times, but by the time we reach 20 or so users they reach over one second, which results in us not reaching our goal of 30rps as each user has to wait for their request to finish before they can make a new one. However, the API does not fall over completely — there is a much more noticeable slow down at around 12:00:20, but it recovers itself.

With our first iteration of algorithm improvements, we get a much more stable graph with the same test setup:

There are minutely spikes in response times for the upper percentiles, which are due to some cache refreshes, and irrelevant to this. We have made improvements! But what is our new limit?

For our second round of testing, we change the test setup: We’re only using one machine to run the API now, and we increase the number of users iteratively until we find the new breaking point. This is the graph for 50 users:

We can ignore the minute spikes again as before.

As before, when we reach around 40 users, the slow down becomes very pronounced, and looking at the profiling, it seems like we can make some more improvements — translate still takes up a majority of CPU time for our responses, and we’d like to squeeze every little bit of performance out of this.

After the second round of improvements, with the same test set up, our graph looks like this:

It looks like it starts to slow down again at 20 users, but if we look at the left y-axis, we can see that the range of slowdowns is much lower, and within the acceptable range for this load test.

Profiling now shows us that translate is not the biggest CPU expense now, and in combination with the graph above, it is time to look at where some of the patterns in spikes come from. But this is for another post!

Originally published at iplayer.engineering on December 7, 2018.

Using Node.JS to build a web scraper from scratch

My time at university was defined by a very large part by my involvement in Student Media — the student newspaper, the student TV station, radio station. I still work with the Student Radio Association (see you at the awards in November?) and the National Student Television Association to this day.

Last week, my student paper announced that they will delete all posts from their website from 2015 and older, so I thought I’d see if I can create my own little backup. Obviously, having graduated some years ago, I do not have a valid login anymore, so I had to figure out my own way to scrape the data.

The Headlines

Before I come to the technicalities, here is what some basic data analysis of the scraping shows:

How many posts will be deleted?
Analysing the posts accessible, we’re looking at around 58% of all posts to be deleted.
This is the number of posts published per year:
2014: 1883
2015: 1903
2016: 1388
2017: 900
2018: 471

Interestingly, the website was set up around 2011/2012. There was a provider migration from wordpress.com to a 3rd party around 2014. This analysis and backup is only based on the current website.

Note that during my time it was practice to publish all posts from the print edition online, as well. I do not know if this is still current practice, but it would explain the higher numbers for 2014/2015.

How much data will be freed?
For a lot of the older posts, the related images are already removed or not accessible in one way or another, so the majority of images is from more recent years.
Removing all from 2015 and earlier will result in around 13% less space taken up by locally hosted media:
2014: 7.08 MB
2015: 740.75 MB
2016: 3.39 GB
2017: 1.2 GB
2018: 542.67 MB
Total Size: 5.84 GB

So all in all, this will remove <1 GB of publicly accessible data — obviously, I do not know how much data there is, that isn’t exposed in the post content or part of other pages in the system.

The cost
All my analysed data is taking up in zipped form around 4.8 GB. I have uploaded a copy to an S3 bucket which is going to cost me around £1.02/year at current exchange rates.

I am pretty sure that the storage class they have to host the site allows for 20 GB, so I am a bit surprised that there seem to be less than 5 GB of content accessible whilst reaching the limit. I don’t know if the actual posts count towards the allowance, but my raw JSON file containing the post content and basic metadata weighs in at around 50 MB for the whole site.

I know Student Unions don’t have much money, and also not the resources to support custom-built solutions, but I am still disappointed that it has come to this — I am very interested to see what impact tomorrow’s planned deletion actually has, but the communication of the actual issue at heart has not been very thorough.

The Technicals

I am just interested in the actual post data and media content related to the posts, not the whole rendered output. I am also not a fan of HTML scraping, it always seems to end up messy even if you have a well-structured site — and with all the changes the new editorial teams have made throughout the years I have no idea how structured the site was anymore.

Searching for Wordpress Backup or Wordpress Scraping didn’t really yield any valuable results without having access to the Admin interface or database, so I took it as a bit of a challenge to write some code.

RSS Lives

I had a hunch as to how I can get standardised post data — good ol’ trusty RSS. I still use RSS daily; Feedly is my second most used website, and trying out /feed on the site gave me a nice bit of XML. It contained 10 posts. Great. At least they seemed to be full versions and not abridged. Yay!

I was expecting there to be thousands though, and there were no pagination indicators as per RFC 5005. Back to the drawing board? No!/feed/?page=2 showed me the second page. Thanks to user2611 on the Wordpress StackExchange for that tip!

The feed

Through some trial and error, I found out that there are 655 pages of RSS to paginate through, resulting in around 6,500 posts.

With this knowledge it’s fairly trivial to get the content of the complete feed:

const request = require('request-promise');
const bluebird = require('bluebird');
const fs = require('fs');

const PAGES = 655
const HOST = 'http://example.com'

async function getPage(i) {
  return request(`${HOST}/feed/?paged=${i}`);
};

const pages = bluebird.map(Array(PAGES), (v, i) => {
  console.log(`Getting Page ${i}`);
  return getPage(i);
}, {concurrency: 5});

pages.then((res) => {
  fs.writeFileSync('rss.json', JSON.stringify(res));
});

This gives me a JSON file containing an array of strings which represent the pages of the RSS feed

The posts

In the RSS pages there is a lot of data I don’t need — XML is quite verbose, and a lot of the metadata is not related to the posts. I used a nice little library called rss-parser to extract the actual posts from the pages:

const _ = require('lodash');
const fs = require('fs');
const Parser = require('rss-parser');
const parser = new Parser();

const rsss = require('./rss.json');

const itemsP = rsss.map(async (rss) => {
    const feed = await parser.parseString(rss);
    return feed.items;
});

Promise.all(itemsP).then((itemsA) => {
    const items = _.flatten(itemsA); // without this, we'd have an array of pages containg an array of posts
    console.log(`Saving ${items.length} posts`);
    fs.writeFileSync('items.json', JSON.stringify(items));
})

This gives me a JSON file containing an array of post objects — including relevant metadata for content , creator , title , link ,date , categories and guid.

To make them more easily browsable I’ve split them up into their own files each; using the guid , padded to a common length, as filenames:

const fs = require('fs');
const items = require('./items');

function pad(num, size) {
    let s = num.toString());
    while (s.length < size) s = `0${s}`;
    return s;
}

items.forEach(item => {
    const id = item.guid.split('=')[1]; // extract ID from full URL
    const idNumber = parseInt(id);
    const idFixed = pad(idNumber, 8);
    fs.writeFileSync(`./items/${idFixed}.json`, JSON.stringify(item, null, 2));
});

Let’s analyse some of this — how many posts are there per year?
I used the infamous moment.js to find the year for each item with the following code:

const posts = require('./items');
const moment = require('moment');

const years = {}

function getYear(post) {
    if (!post.isoDate) throw new Error('No Date found');
    const postDate = moment(post.isoDate);
    return postDate.year();
}

posts.forEach(post => {
    const year = getYear(post);
    if (!years[year]) years[year] = 0;
    years[year]++;
});

Object.keys(years).map(year => {
    console.log(`${year}: ${years[year]}`);
});

And these are the stats we find if you skipped the intro:

2014: 1883
2015: 1903
2016: 1388
2017: 900
2018: 471

The media

Photos and other imagery make out a large part of the posts, so I wanted to find all media links the posts mentioned.

Let’s start with all URL’s mentioned in the post content. I used another library that can comb through any string and identifies URLs, get-urls:

const posts = require('./items.json');
const fs = require('fs');
const _ = require('lodash');
const getUrls = require('get-urls');

const allUrls = new Set();
posts.forEach((post) => {
    const values = [post['content:encoded'], post.content, post.contentSnippet];
    const strings = _.compact(values).join(' ');
    const urls = getUrls(strings);
    urls.forEach(url => {
        allUrls.add(url);
    });
});

fs.writeFileSync('allUrls.json', JSON.stringify([...allUrls], null, 2));

I also noticed that quite of them didn’t resolve to valid files, and a status code 404 was returned from the host, which I also wanted to filter out.
Since I have to do requests for this anyway, I thought it might be nice to figure out how much storage these files actually take up, so for each valid URL I also recorded the reported file size:

const allUrls = require('./allUrls');
const _ = require('lodash');
const bluebird = require('bluebird');
const fs = require('fs');
const request = require('request-promise');

const storedUrls = allUrls.filter((url) => {
    const isCorrectHost = url.includes('example1.com') || url.includes('cdnbucket.com');
    const isUpload = url.includes('/wp-content/uploads')
    return isCorrectHost && isUpload;
});

let counter = 0; // To store and display progress
const urlsAndSizes = bluebird.map(storedUrls, async (url) => {
    let res;
    try {
        res = await request(url, {
            method: 'HEAD',
            resolveWithFullResponse: true
        });
    } catch (error) {
        if (error.statusCode === 404) return;
    };
    counter++;
    const percentage = ((counter/storedUrls.length) * 100).toFixed(2)
    console.log(`Finished ${percentage}%`);
    return {
        url,
        size: res.headers['content-length']
    };
}, {concurrency: 5});

urlsAndSizes.then((urls) => {
    const validurlsAndSizes = _.compact(urls); // remove 404s
    const fileContent = JSON.stringify(validurlsAndSizes, null, 2);
    fs.writeFileSync('validUrlsAndSizes.json', fileContent);
});

This gives me a JSON file containing an array of objects with urls and their reported file size in bytes. Let’s analyse the data!

Again, I am interested in a storage cost breakdown by year — does removing content from 2015 and before actually result in substantial savings?
Similar to the post analyses above, we can make use of the previously extracted media URLs:

const filesize = require('filesize');
const objects = require('./validUrlsAndSizes');

const years = {}
let totalSize = 0;

function getYear(url) {
    const split = url.split('uploads/');
    if (split.length != 2) throw new Error('Invalid URL');
    return split[1].slice(0, 4); //the year in the upload path
}

objects.forEach(medium => {
    const year = getYear(medium.url);
    if (!years[year]) years[year] = 0;
    totalSize += parseInt(medium.size);
    years[year] += parseInt(medium.size);
});

Object.keys(years).map(year => {
    console.log(`${year}: ${filesize(years[year])}`);
});

console.log(`Total Size: ${filesize(totalSize)}`)

This is what we get for this:

2014: 7.08 MB
2015: 740.75 MB
2016: 3.39 GB
2017: 1.2 GB
2018: 542.67 MB

Total Size: 5.84 GB

Archiving

I have downloaded all the media files from my validUrlsAndSizes.json file, preserving their relative paths using wget:

wget --no-host-directories --force-directories --input-file=urls.txt

In the BBC, Artificial Intelligence and Machine Learning are currently being explored heavily by pretty much all product teams — Why? For the answer to this, I am going to point at this blog post in which Ali, our Head of Emerging Technology, goes into a bit more detail about the why and covering not just BBC iPlayer.

While exploration is very important, in iPlayer we’ve already gone a step further: A couple of months ago we’ve trained our first machine learning model using Google’s TensorFlow. Its job is to classify video frames by certain characteristics and give us a score back. This is has been running in production for a while now. And we’re iPlayer — we have a lot of videos, and therefore a lot of video frames to be processed and classified.

How many you ask? Well, most of our catalogue is available for 30 days. Every time we broadcast an episode that is not currently available, it becomes ‘available’, and thousands of episodes of iPlayer content become available each month.

Every time an episode becomes ‘available’, we need between 80–200 video frames classified. Ideally as quickly as possible.

How? The obvious step would be to look at Google, who make TensorFlow, and see that they have a service called the Cloud Machine Learning Engine. And whilst we’re experimenting with Google Cloud Platform (and yes, some BBC teams are looking at Azure), AWS is our production platform, so we had to work with that as a constraint.

I did spoiler it in the title for this blog post what our solution is: AWS Lambda. In theory, we should be able to invoke a classification function 150 times in parallel, pay for that compute time, and get results back pretty quickly.

In practice — it works! However, it was a bit of a journey:

The Problem

AWS does place certain limitations on Lambda functions. Of interest here is the Lambda function deployment package size at a neat 50 MB (N.B. when using the console to upload the function this is 10 MB).

Let’s have a look at what we need to provide to classify our video frames:

• A trained model necessary for classification takes up space — in our case around 85 MB.
• The TensorFlow library itself to run a video frame against the model takes up space — uncompressed we’re looking at 280 MB for that and the Python dependencies it requires that don’t come with Amazon Linux (the environment in which Lambda functions get executed).
• Some code to process the data, and return something — this weighs in at around 15 kB in this case.

Doing some maths this adds up to 365 MB. Just a bit more than the 50 MB AWS allows us to upload.

How could we get around that? Compression I hear you shout. We use TensorFlow 0.11 — the download size for that alone in a Lambda-like environment is 39.8MB. Add all the dependencies to that and we’re way over our limit.

Now you could download and install the dependencies and TensorFlow at runtime — but for several 10s of thousands of invocations a day that’ll add up in computation time and bandwidth. It’s probably not a secret that iPlayer doesn’t quite have the $500M R&D budget of a certain competitor to even attempt justifying that. Thankfully there are some work arounds for that to which we’ll get to in due time.

How about the model? Can we compress that? Well, there’s some research, Caffe models can do it. TensorFlow? Nothing production-ready. And regardless of that, when the model gets trained further or becomes more complex, its size will increase, so this will not help us much in the long term.

The Solution

The Hack: --exclude

Let’s begin with the hackiest part: --exclude. A TensorFlow 0.11 installation gives us a lot of tools included for visualisation: TensorBoard, Eigen, and libraries for making use of other tools such as Numpy and ffmpeg. Additionally, when you run Python, the interpreter will save the compiled byte code with the source files — we do not need to include all of when packaging up our zip. So we’re not including any of that when packaging it up. To be precise, we exclude *.DS_Store *.pyc /external/* /tensorflow/contrib/* /tensorflow/include/unsupported/*. This saves us 83 MB or so. We can probably save a lot more on that front by excluding more unnecessary libraries or using a different version of TensorFlow to evaluate models, but it turns out this is enough for us for — alongside the other tweaks.

There is only one more thing to be removed before our zipped package weighs in at under 50 MB and is ready to be deployed: The model.

The storage: S3 and /tmp

We store the model in S3. Sounds simple, right? The benefits are clear: We can decouple that, train and update the model separately and we don’t need to include some extra 85 MB of data in the lambda.
On the other hand, we have to pay for S3: storage of the model and requests from the lambda. Also, it takes extra compute time we’d have wait for and pay for, which isn’t ideal. If we go back to the Lambda limits I mentioned earlier, it states that there are 512 MB of Ephemeral disk capacity (“/tmp” space) available. And that’s where we are going to keep our model:

Because we run 200 invocations or so in parallel we’ll only need to download the model once and save it there — the other invocations should theoretically have access to it if it’s already saved. So one of the first things we do when invoking the function is to check whether the model exists in that location and download it using Boto3 (the AWS SDK for Python) if it doesn’t:

import boto3
import os

if os.path.isfile('/tmp/graph.pb') != True:
    s3 = boto3.client('s3')
    s3.download_file('modelbucket', 'graph.pb', '/tmp/graph.pb')

Dandy right? Except there’s one more item on my solutions list above:

The warm-up: CloudWatch Scheduled Events

Some of you might have got some questions in regard to the above — if we’re kicking off 200 invocations in parallel, wouldn’t they all have to download the model as at that point it won’t exist? How long does /tmp persist anyway? We’ve done some digging.

Exhibit 1: Understanding Container Reuse in AWS Lambda. This blog post from 2014 when Lambda was still in preview mentions the following:

Files that you wrote to /tmp last time around will still be there if the sandbox gets reused.

Exhibit 2: AWS Lambda: How It Works:

Each container provides some disk space in the /tmp directory. The directory content remains when the container is frozen, providing transient cache that can be used for multiple invocations. You can add extra code to check if the cache has the data that you stored. For disk space size, see AWS Lambda Limits.

[…] Do not assume that AWS Lambda always reuses the container because AWS Lambda may choose not to reuse the container. Depending on various other factors, AWS Lambda may simply create a new container instead of reusing an existing container.

So in theory, it should just stay there, except it might not. So we have set up a scheduled event to keep the container ‘warm’. We invoke the function every couple of minutes not to do any processing, but to make sure we’ve got the model ready. This accounts for a very small percentage of invocations but helps us mitigate race conditions in the model download.(We also limit artificially how many lambdas we invoke in parallel as an additional measure).

So how big is our function now? The zip measures in at 40MB at the moment — turns out python source files compress quite nicely.
This should remain at this level for this specific one as we don’t need to update the lambda to train the model further.

This brings us to the final question: How do we invoke the function? — API Gateway is the answer to that one.

The access: API Gateway

We have an API Gateway POST endpoint that takes the base64-encoded jpeg of the video frame, and returns JSON 10 seconds later. This also allows quick and effective testing of changes to the model without having to write any code to invoke the function — curl and base64 is all that’s needed to evaluate a frame.

API Gateway does actually support binary data now, so we could scrap the need for base64 as it does the base64 encoding for you, but when we first deployed this, that feature was not available using CloudFormation.

All in all we are quite happy with how this is running in production at the moment , in terms of reliability and efficiency, but we are looking forward to improving on this pipeline, and putting it to use for many more applications to come.

We are gathering usage data of this classifier and with help from the editorial staff we are analysing the accuracy of the model, but the training of the model is currently a manual process which we are hoping to automate in the future.

We are also looking at other metadata which might be available to us elsewhere in the broadcast chain to enrich our content, and how we might use it to improve the audience experience — stay tuned for more on what we are actually classifying if the gif above doesn’t give away too much.

Join us

If you’re interested and excited by the technical challenges we’re facing, we’re always looking for driven engineers to join us.

You can see all our job listings for BBC iPlayer here by searching for iPlayer.

Evaluating TensorFlow Models in AWS Lambda was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Anyway, for this feature our UX people really wanted to blur images, because that’s up and coming these days. Now that doesn’t exactly sound like a challenge, what was CSS3 invented for anyway? Well, sadly we still have to give some consideration Browsers from the pre-CSS3 era, and whilst the funky old Microsoft DXImageTransform filters do have a nice blur, it’s not really a route we wanted to settle on (and also would only be supported in IE 5.5–8 and 10+ I believe, because who likes logic anyway)

So we looked at alternatives. Client-side we looked at drawing in a HTML canvas and putting our image inside of an SVG with a filter on. The drawback of the HTML canvas is that there’s no simple way to blur without much code — we’d have to package our own blur algorithm (this one seems to pretty popular, coming in at 6kB minified), we don’t really want to do that for reasons.

So SVG blur is the other client-side solution, which is also used by other teams within the organisation, but they have quite a different set of requirements. And it’s pretty simple, look at that:

<svg> <defs> <filter id="svgBlur"> <feGaussianBlur in="SourceGraphic" stdDeviation="25"/> </filter> </defs> <image xlink:href="http://7nr.de/somepicjpg" width="100%" height="100%" filter="url(#svgBlur)" /> </svg>

Pretty straightforward, hm? It also looks beautiful, in Safari at least. It’s a very different situation in Chrome or Firefox, however:

• There is some serious banding on the graphic. No idea where it comes from or why, but it just looks a bit ugly.
• Due to the nature of the Gaussian Blurs and SVGs, the edges of the image get blurred with the surroundings, in this case, nothing — i.e. it just fades out and becomes semi-transparent towards the edges. This is something that can probably be fixed easily by setting the width/height of the filter, or just zooming in the image or.. there are probably a couple other methods.
• There was a third point, but I don’t remember it now.

This made us think about alternatives, and our infrastructure — and our infrastructure is quite clever! For any raw source image, our image server can generate different resolutions and caches them and whatnot. Not only can you specify a resolution, but these ‘recipes’ that we use can contain a whole range of commands to be applied (it uses GraphicsMagick under the hood).

One of those commands is ‘blur’, and it looks beautiful. And because there’s no client-side operation, it’s supported in any browser that can display JPEGs — which is definitely a superset of the browsers we care about so yay :)

To just illustrate the differences between a GraphicsMagick blur and an SVG blur in our case and in this case in my specific configuration on my machine, here two screenshots (you might want to open them in a new tab/view them full sized):

SVG Gaussian Blur:

GraphicsMagick Blur:

Originally published at niklasr.github.io on July 13, 2016.

A couple of weeks ago I started a Backup task for all my photos on my NAS. So my DS410 has eagerly been uploading hundreds of gigabytes of DNGs to S3 24/7, and was about 25% finished (yay under 2 months left to be done!). And I just pulled the plug. Smart move, Nik, smart move.

Plugged it back in, logged in and luckily saw no drive errors, which is a wonder seeing that the oldest drive in the bunch has 41620 hours “Power-On Time” (the youngest has 5862 hours).

Restarted the Backup (apparently it was ‘Never backed up’ before), and it started uploading.. 10MB.. 20MB.. 0%. I logged into my AWS account to check it hadn’t deleted the 250GB it had already uploaded.. it didn’t.. Mysterious, what was it doing? We got to ~ 60MB uploaded, and suddenly.. 26% done! Thank the lord it picked it back up, and I feel a bit safer about this back up now.

Next step will be downloading it onto an RS815+ at home from the S3 backup. Redundancy, accessibility and cost-efficiency. Yada-yada..

Oh, and the Raspberry Pi? Well, I realised I don’t actually have a USB keyboard at home, so that’ll have to wait until some other time, too.

Originally published at niklasr.github.io on April 18, 2016.