Leadership does not mean telling people what to do.

Before working at the BBC, I thought a tech lead’s job was to think of some work to do and command others to do it.

It was quite a damaging misconception, because it stopped me considering moving up from a software engineer into management. I don’t like telling people what to do. This is because I also don’t like being told what to do. As the youngest of four siblings, I have perhaps been overexposed to being bossed around. If someone ever talks to me in the imperative, I have an immediate, instinctive visceral response which vastly reduces the likelihood of my compliance with the request. I suspect I am not unique in this. So I never wanted to be That Guy, who makes other people feel like that.

But the excellent news is — it turns out you don’t need to be! The great thing is, grown up people are entirely capable of managing their own workloads. Assigning people specific tasks can often feel like micromanagement to them. If it is clear what tasks need to be done, people will prioritise the tasks into their own work streams and pick them up when they have capacity to do so.

Collective decision making is good

Another thing I used to think leaders had to do, was go and make up bold, dazzling strategies all by themselves and then tell everyone about their vision. Dictatorially make a swathe of tickets and dump them onto the board, whilst loudly trumpeting about their own self-worth in some public channel. In real life this will get you a lot of side-eye of course.

I vastly prefer the approach of talking through what needs to be done with the team and making sure everyone understands why it is important. We can work together to outline the high level goals and then discuss breaking the work down into tickets. Working together towards a shared vision is much more motivating for the team than being told exactly what to do and when. You want to get plenty of input from everyone when defining a scope of work — a team are likely to come up with better ideas for implementing stuff than a single person.

Not every hill is worth dying on

Like many things in people management, this is reminiscent of a common parenting strategy — often summarised as “pick your battles”.
Influence in the workplace is like a currency which you can choose to spend in a particular argument or not. You can only really build this up to a high level in a slow, laborious way by gaining people’s trust over time. You get people’s trust by listening to their problems, seeing things from their side and helping them to fix things.

There aren’t really any shortcuts here unfortunately. If you try and wield influence without having enough trust behind it, people will generally politely ignore you. And even if you have plenty of trust, trying to influence too many things is not a good idea. People’s goodwill gets depleted the more you mither them. They will eventually stop listening to you if you are constantly sticking your oar in where it’s not required. You should also trust your team to make their own decisions.

Influence is most powerful when it’s used sparingly on something that is genuinely important — eg. you have an idea which will have a significant observable impact on your users, and you can clearly explain how it works and how it could be measured. If someone has used tabs instead of spaces in a PR, whilst it may superficially irritate you on a stylistic level, you might want to consider saving up that influence for another day.

Rumpelstiltskin is a role model

To be 100% clear, not in terms of theft of babies or weird name-guessing antics. For metaphorical purposes I am specifically thinking about this fairytale character’s ability to spin a roomful of straw into gold.

Sometimes at work, you will be given a task that you do not agree with but have little influence over. In this case, I think it’s useful to flag any concerns, with a clear, objective description of where you can foresee problems. But sometimes, the needs of other stakeholders outweigh other issues that you may have identified. I don’t generally expect my concerns to override everyone else’s and do not seek to veto every decision according to my personal whims.

So maybe a decision didn’t go your way. Congratulations! You are now the proud owner of a roomful of straw and you need to give this to your team to build meaningful things from. They are not going to like being given straw. So I recommend spinning it into gold first before presenting it to the team. Yes, this is basically politics, a word which is often used pejoratively at work. But it is an incredibly useful skill for keeping teams engaged.

Nobody is going to feel inspired and motivated to build something that is presented to them as a bad idea from the outset.
Your goal is to try and build team enthusiasm in the project. How do you do this if you don’t feel 100% behind the idea? It is hard, when your ego has been bruised. But at work in general, your ego is best left at the door. You could present the motivations that other people have given for it — acknowledge concerns but focus on the positives. If you are not in a situation where you can influence the decision, there is no benefit from stating that you don’t agree with it. And actually this can end up being more frustrating to the team than if you commit to an unpalatable outcome.
It’s good to document difficult decisions like this, as it can be painful to keep re-hashing conversations. You could use the document to finesse the framing of the approach — focussing on a long term preferred vision which addresses all concerns and keeps all stakeholders happy.

What is the most helpful thing I can do for the team right now?

This is a question I like to ask myself a lot.
As a leader, it can be hard to know what to focus on. Gone are the days of all your work being nicely encapsulated for you in tickets and your days punctuated by meetings that other people organise.
Instead you often veer between having a deluge of messages clamouring for your attention, to periods of complete radio silence where you suddenly need to decide what is the best use of your time.

I like the idea of the ‘servant leader’ — who keeps themselves fully aware of the challenges the team is facing, and constantly works to remove blockers and resolve problems. A ‘fixer’ if you will. Like Mike Ehrmantraut in Breaking Bad or The Wolf in Pulp Fiction.

Sometimes this could be putting in a PR for some dull but essential code that is important but would disrupt the team’s current focus.

Sometimes this could be a difficult conversation where people are arguing and you can help mediate.

Sometimes this could be a boring conversation about who owns that repo and how to get access.

Sometimes, after spending too long in Rumpelstiltskin mode, this could be simply to go into a quiet room and scream into the void for a while until you feel a bit better. If you are feeling stressed and emotional, getting your own head straight is often genuinely the best thing you can do for your team right now.

Burnout

I should put in a disclaimer before starting this bit, that I haven’t done much reading about burnout, because I have been too busy.
So I have an idea in my head of what this word means, which may or may not have any overlap with accepted clinical definitions.

I do know that I love my job, and I love being busy. But occasionally, one day of continuous consecutive context-switching too many has tipped me over into a state where I am unable to sleep because my brain is constantly buzzing with work related problems. This is not a good place for a person to be. The fatigue carries over into their working day and feeds a positive feedback loop of increased stress which further impedes their ability to sleep. At this point, they really just need a bit of a break to regain their mental and physical composure. Everyone has a limit and I feel there’s little to be gained in the workplace by stumbling over it into a self-destructive realm of vastly reduced productivity.

The good thing about having had this experience is, I can now recognise the early stages of this process.
If I notice this happening to me, I know it’s really important to say no to things for a bit until my concurrent workload subsides.
I also think it’s a great idea to look out for this in other people. I have seen people at work actively defend other people’s diaries. e.g. In the classic case of a meeting that only exists to spawn another meeting. When someone comes up with a knee-jerk “let’s have another call and invite X” I’ve heard some people reply “Is this really relevant to X? X is pretty busy — I don’t think they really need to go to that call”. These people are legit heroes and we should carry them round on our shoulders cheering.

Relationships

Interpersonal relationships, to me, are the most important part of a job.
I like to invest as much time as possible into building good relationships with my colleagues. Small talk may feel like an indulgence, but it makes the big talk easier to approach when it becomes necessary.

If you have fallen out with someone at work, it is important to try and resolve it, however much you may want to skip around the edges of a conflict. If communication is not flowing, it can severely impact the work you are doing.

Conversely, if you have good relationships with your team, literally everything is easier. Understanding a task, asking all the stupid little questions that are in your head. It is much easier to reach out to people that you get on with. And if you genuinely value your colleagues, it makes it a much happier place to work.

The only slight disadvantage of this approach is that there is a risk that when you leave, you will have built up relationships that are very dear to you, and the leaving process may end up accidentally breaking your heart.

Are you a leader?

Tech leadership may have its tribulations but it is a wonderful and rewarding path to follow. I have learned a lot during my tenure as team lead from the various amazing people that I have had the privilege of working with. I would recommend this path to anyone who hasn’t been put off by my ramblings above. Thank you for reading and good luck upon your quest x

Things I have learned about Tech Leadership was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Recent times have tested the resilience of our teams and our services but we’ve delivered at every moment. Overcoming significant technical challenges and enabling the BBC to serve millions of users with great content, across different products and devices.

And so we thought it was about time that the team had a chance to get together, connect with each other face to face, and get inspired by how much we do for our audiences.

In October, over 600 colleagues from more than 150 technical teams from across the BBC got together in Manchester for the first BBC Engineering Conference. The day covered a range technical areas, knowledge sharing and networking. We had both internal and external speakers, as well as a leadership panel.

Sometimes we set out to inspire our teams and they inspire us even more in return, this was one of those moments. It sets up perfectly for a great end of 2022 and also for a future Digital First BBC.

The video is a snapshot of what we got up to!

Getting inspired at the BBC Engineering Conference was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

It’s my job to help the BBC get the most out of modern digital agile ways of working. Bringing more product thinking to how we work by focussing on outcomes and putting user needs at the heart of what we do.

Much of my role is helping teams collaborate effectively together at scale. Product Group is over 100 teams and we work with many more teams in editorial, marketing and across the BBC. We are responsible for products like iPlayer, Sounds, Bitesize, BBC News and Sport. We’ve got huge ambitions at the BBC to deliver more value to users through our digital products and services by taking a more user led iterative approach to what we do.

Creating our Delivery Profession

Over the last few months one of my priorities as Head of Delivery has been to create and build a new Delivery Profession at the BBC for agile delivery practitioners. I’m thrilled that this work is now complete and we have a dedicated Delivery Profession with a new career path framework, new roles and new job descriptions at the BBC.

When designing what we needed I was keen we also introduced an individual contributor track so that people could develop their careers here by deepening their skill and practice as well as through the traditional line management route.

We are building a thriving Community of Practice who meet weekly to network, share and learn together. More on that to come!

Our Vision

We recently wrote down our vision for Delivery Management at the BBC and we think this encapsulates pretty well who we are and what we’re here to to do:

“We inspire ourselves and our teams to raise the standard of agile collaborative delivery at the BBC.

We help make teams more effective and maximise their impact so we deliver value for users often, iteratively and repeatedly.

We focus on outcomes and making it happen.”

We’re looking for Delivery Managers

Delivery Managers play a huge part in our multi-disciplinary teams driving forward our change to user centred agile ways of working at the BBC. It is an exciting time to join us and we’ve got plenty of new roles in locations across Salford, London, Glasgow and Newcastle.

Current vacancies can be found on our careers hub page and we’ve got lots of opportunities across multiple locations including:
Associate Delivery Manager, Delivery Manager, Agile Coach, Senior Delivery Manager and Lead Delivery Manager.

Just type the role and location into the careers hub search.

Come and help us build the BBC.

Sara Bowley

Hello from Delivery at the BBC was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

A bit of context first

I started a self-study path to learn the theoretical fundamentals of Data Science and Machine Learning. I have also been playing with Python ever since, with coding exercises during the “Friday 10% time” here at the BBC and on my own time when I have some to spare.

I created this playground repository to refresh my Pythonic knowledge that was buried deep inside the funny head of mine. This repo is a mix of material and exercises to keep my Python coding skills sharp.

GitHub - simonespa/python-playground: A playground repo for Python

I also created a specific playground focused on Data Science and Machine learning. In this playground I practice with libraries such as NumPy, Pandas, Scikit-Learn, TensorFlow (et. all) with a Jupyter Notebook. It also contains notes and summaries of what I learn in forms of README or notebooks.

GitHub - simonespa/machine-learning-playground: A playground to practice machine learning, data modelling and visualisation

Nothing fancy or ground-breaking, just a couple of playground repos for me to practice and learn at my own pace.

The 10% time at the BBC

This happens every other Friday and it is about learning. Engineers use this time to learn new technologies and/or languages, discover new approaches and techniques.

We can organise the day as we want. It could be a self-paced learning session (reading, watching tutorials, etc.), a day of study for an apprenticeship or a university degree etc., an actual hands-on prototyping session to work on a long-running personal project, or a group collaboration with other like-minded people.

There is no limitation to how we plan our day, as long as we keep in mind the end goal. In simple terms, satisfy our own curiosity and creativity by strengthening our technical knowledge and acquire new skills to widen our horizon towards new engineering areas.

Unlike the hack day, we don’t have to present a demo to a panel. Nonetheless, we like to share with the other members of the wider team what we’ve learned during the day if we want to, discussing about different topics, Q&A and why not, showcasing a prototype if someone has built one. No prizes but praises.

The UK Bank Holidays data

I needed to find a dataset that I could use to practice with Pandas on a Jupyter Notebook.

You may be familiar with the Gov.uk page that shows the list of past and upcoming UK bank holidays (i.e. https://www.gov.uk/bank-holidays) and maybe less familiar with https://www.gov.uk/bank-holidays.json which is the same URL plus the “.json” part appended at the end.

The JSON URL returns the same information of the HTML page but in a machine-readable format (i.e. JSON response) and it’s that URL that I used to retrieve the information i needed to manipulate.

UK bank holidays

The data is released under MIT License and if you are curious about its implementation, you can find more details at the following link

calendars/README.md at master · alphagov/calendars

This is my first Notebook by the way, which I wrote using plain Jupyter on my localhost. Once I mastered the craft, I then decided to publish it on Kaggle which is way better if you then want to practice on more interesting datasets and tasks: from simple data manipulation and visualisation to fully fledged machine learning competitions. You can find my published UK Bank Holidays Kaggle at the following link.

UK Bank Holidays

In this project I use Pandas to fetch, manipulate and query the JSON data. The main trick to work with JSON datasets and manipulate them as Pandas DataFrame is to flatten the portion of properties you need with the “json_normalize” Pandas function, no matter how deep the props are.

N.B. In this post I’m assuming that you are familiar with the tools and libraries I’m referring to. If not, it would be presumptuous of me to tell you where and how to find material on the Internet. But if I were asked, I’d recommend https://towardsdatascience.com, a publication of independent authors and volunteer editors of dedicated and curated Medium blog posts specialised in Data Science and Machine learning. I read it myself to support the material I am currently studying and it’s being really useful so far.

The Notebook

The JSON structure of the response returned by https://www.gov.uk/bank-holidays.json is the following

{
  "england-and-wales": {
    "division": "england-and-wales",
    "events": [
      {
        "title": "New Year’s Day",
        "date": "2017-01-02",
        "notes": "Substitute day",
        "bunting": true
      },
      { ... },
      ...
    ]
  },
  "scotland": {
    "division": "scotland",
    "events": []
  },
  "northern-ireland": {
    "division": "northern-ireland",
    "events": []
  }
}

Pandas plays well with tabular data and it is able to read JSON too. The challenge with JSON though is hat it is hierarchical by nature, and Pandas can only go as far as 2 levels deep to cover the two dimensions (columns and rows). In order to get nested properties, we need to specify their path in order to flatten the data in tabular format.

Data collection

First of all, we need to import the Pandas library

import pandas as pd

and fetch the data from https://www.gov.uk/bank-holidays.json

bank_holidays = pd.read_json(path_or_buf = 'https://www.gov.uk/bank-holidays.json')

Data exploration and transformation

The data frame has a row called events which contains data in JSON format. We need to deconstruct this data in tabular format, so that we are able to query it with Pandas.

With this approach we transform the bank_holidays DataFrame to a Python dictionary. The dictionary can then be manipulated with the json_normalize Pandas function, by selecting the nested property to flatten into a table.

The data frame is then returned with the appropriate data types set for each column.

The following examples refer to the bank holidays in England and Wales.

def get_england_and_wales(data_frame):
    return (
        pd
        .json_normalize(
            data_frame.to_dict(),
            record_path=[['england-and-wales', 'events']]
        )
        .astype({
            'title': 'string',
            'date': 'datetime64[ns]',
            'notes': 'string',
            'bunting': 'bool'
        })
    )

bh = get_england_and_wales(bank_holidays)

NOTE: This approach can be used with any JSON data no matter the complexity of its structure and how nested a property is.

Get all bank holidays within a specified time interval

The following examples return the 2022 holidays in England and Wales. Let’s declare the “start” and “end” dates.

start_date = '2022-1-1'
end_date = '2022-12-31'

Create a boolean mask

mask = (bh['date'] > start_date) & (bh['date'] <= end_date)

Query the data frame to select the subset rows

bh[mask]

Another way to query the data frame is by using the query function. It gives you the same result as before but the syntax is powerful and more readable. Note that the @ syntax is used to reference the global variables previously defined.

bh.query('date > @start_date and date <= @end_date')

If you are going to do a lot of selections by date, you may want to set the date column as the index for the data frame.

bh = bh.set_index(['date'])

Now you can select the rows by date using pd.loc[start_date:end_date].

bh.loc[start_date:end_date]

Upcoming bank holidays in England and Wales

The following example returns the remaining holidays between “now” until the end of the year.

To do so, we need to import the date object from the datetime module.

from datetime import date

Calculate the date range from today to the end of the year

today = date.today().strftime('%Y-%m-%d')
end_of_the_year = f'{date.today().year}-12-31'

Now you can query the data frame

bh.loc[today:end_of_the_year]

Conclusion

This for me was a pretty good learning on how to use Pandas and Jupyter with a real example and how to manipulate and query data. Don’t get me wrong, I just scratched the surface of the iceberg by using just a few of the many functions that this library provides. It’s a powerful tool and it takes a bit more practice on many more projects to really get the hang of it. But the good news is that the documentation is comprehensive and well written plus there is plenty of examples and informations out there for you to find and support your work.

If you are already familiar with the tool and are looking for interesting datasets, I’d suggest to signup on Kaggle as I did, search for the right exercise that fit your level of expertise, and practice with it. If that is not enough, just remember that many big organisations, government agencies, public bodies etc. provide open data for the public to use: e.g https://data.nasa.gov and https://data.police.uk/data just to name two.

In conclusion, if you are in the UK and you want to check how long until the next holiday to have some time off, please just consult the Gov.uk website, it’s faster 😃. But if like me you want to learn something new while you do it, well you’ve found the right post to read.

UK Bank Holidays with Pandas on Jupyter was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Modern JavaScript websites are the result of complex code bases, and the code base which powers BBC Online is no exception. There are many dependencies and frameworks that we rely upon in order to provide the functionality that our end-users expect, as well as code written in-house to cater to the needs of our website and its users; being able to click a button to follow a topic, search for content and play videos all require JavaScript code to be run by the browser.

For a website as large and with as many seemingly disparate needs as the BBC’s various pages, the amount of code required can grow exponentially. The code needed to render a news article - with images, social media embeds, videos and onward journeys - has only small elements of commonality with the code required to power complex sports data tables - with team badges, functionality to hide or show columns based on the screen width of the user’s device, display a team’s form, or provide live in-page updates for events that are happening live.

As such, it has long been the practice of the JavaScript community to implement code splitting. Rather than providing all of our JavaScript and dependencies into a single large file to be downloaded by the user, we can split our code into separate files. This can provide advantages on two fronts; downloading multiple smaller files can be more efficient when devices can start those downloads in parallel (though going too far and downloading too many small files can cause problems itself!), and the way that we choose to split our files can mean that some only need to be downloaded on certain pages. It is this second benefit which can be most powerful; if you have a large piece of code on your site that is only needed for a single page, then forcing users to download that code on every page is wasteful, and can cause problems for users who might have limits on the amount of data they can download on their network.

We have been using code splitting within the Presentation Layer, which powers large portions of BBC Online, for a number of years now. However, we had realised that the platform had outgrown the existing strategy we had in place for code splitting. So first, what was that strategy?

We had decided to create a small number of core bundles, which would be shared across the website and required for every page. This strategy meant that although the user would need to download around 5 relatively large files on an initial page load, those files would then be cached for future visits, or for when the user visited another page on the site. We make use of contenthashes in our bundle file names, which are a way of identifying whether the content of a file is different at build time — that means that when a code change is deployed, only the bundle affected needs to be re-downloaded, and the cache can be relied upon for the unchanged bundles. These core bundles consisted of:

• our third-party dependencies (commonly referred to as the vendor bundle)
• the code for our rendering framework
• our GEL packages — components that are centrally owned and designed to be highly reusable and adhering to our Global Experience Language guidelines
• additional components and utilities, that have either more specific use-cases or are not yet fully reusable — referred to as our website bundle
• common utilities and libraries used across the website — such as our compliance library, used to ensure that we are not setting cookies that the user has not agreed to — referred to as our base-website bundle

In addition, the Presentation Layer has the concept of Containers. Containers are isolated slices of the experience that are data-aware, and combine together multiple components into a single experience. Our navigation bar is a container, as is the core view of a news article, whilst the lists of onward journeys are a separate container again. Each of these containers has a high level of separation of concerns, and are not allowed to directly depend on one-another, and so each one of these containers gets is own separate bundle file. This is what allowed us to not deliver too much unused code to the browser, and only provide the container-level code for pages that use that container.

So what went wrong with this approach?

Firstly, this approach was not wrong, in any sense. At the time we implemented it, it was the most suitable solution. However, in the time since, the number of pages being powered by the Presentation Layer has grown; the number of new experiences being developed for the audience has increased, and the resulting complexity of our code base has increased with them. This resulted in our core bundles growing to the point where our bundling library was reporting their combined size as over 1MB. Whilst these files were compressed before being sent to users, we were in essence requiring users to download over 1MB worth of JavaScript for every page. Much of this included code that was only used on a small number of pages — as portions of the Bitesize and Children’s websites began to move over to the new platform, the games library was added, which was only needed on games pages; as Sport pages began to move over, components specifically tailored to showing match data between teams or league tables were added, which were only needed on those pages.

If we take a look at the visual representation of our code-split bundles, provided by our build tool Webpack, we can readily see the effect that this has had on our core bundles:

In short, we were no longer optimised for reducing unused JavaScript being downloaded by end users, and we needed to take a second look at our strategy.

If you search around the internet for solutions to this problem, you will quickly find that there is no general guideline; every website is different, and the ideal strategy relies on the specific needs of the page and the dependencies being loaded. So we had to do some investigation of our own in order to decide how to proceed. Our first target however, was in reducing the size of the largest bundle; defaultVendors.

If we take a look into the visual representation above, we can easily see that there are three very large categories dominating this bundle; @bbc with the games-atlas package, react-dom and @optimizely are very clearly the largest components here. Our website is React-based, which means that react-dom is necessary on all of our pages, and so wouldn’t make sense to remove from this bundle, but games-atlas and @optimizely are not used on every page, and so they could easily be removed from that bundle.

Initially, we tried moving both of these into another shared bundle; optionalVendors. If you take a look at the visual representation of that however, you may be able to see straight away what the problem with that approach was:

If any page needs to make use of the @optimizely packages, it must still download games-atlas, whether it needs it or not. So whilst pages that use neither of these packages would be aided by this change, it wasn’t ideal.

Our next step was to create separate bundles for each third party dependency that we had determined could be removed from the defaultVendors bundle. This would achieve the aim of only downloading this third-party code when it was needed, and give a benefit to the largest number of pages. In addition to creating separate bundles for games-atlas and @optimizely, we decided to also create a separate bundle for date-fns, and its related date-fns-tz — these libraries are known to be quite large, and whilst we use them on most pages server-side for formatting date strings, their use client-side is on a fairly limited number of pages.

This ultimately allowed us to trim around 80KB (gzipped) from the download size of our defaultVendors bundle — a saving of around 44% of its original size.

There are some small oddities with this approach, which you will see in the images later in this post. By defining these separate bundles and marking them as async bundles, that indicates to Webpack to fill them with code that is not needed on initial page load, and can be downloaded as-needed. However, some code — particularly in the @optimizely packages — is needed on initial page load. As such, Webpack has split these packages into two — the code that can by downloaded asynchronously is in its own bundle, whilst the parts of the code that are needed on initial page load have become part of another bundle. We still have an overall saving in the amount of JS being sent to users however, so this was an acceptable compromise to us.

The next target was our second-largest bundle; website. As noted previously, this bundle contains the majority of our components that are either very focused in their use-cases or aren’t quite shareable enough yet to be moved into the GEL bundle. It also includes some utilities used within those components and across some of our containers, page-level layouts, and non-visual parts of the page experience such as setting dynamic page metadata. The single largest category within it however are our UI components.

Our initial approach was to just exclude these components from the website bundle group, and allow Webpack to do the heavy lifting of deciding where to perform grouping and splitting of the code. However we found that this approach wasn’t ideal — other core bundles started to absorb some of these components, and not ones that were necessarily that commonly reused either. Other times, components ended up being placed into the container bundles. Overall, it felt like we needed to exert a bit more deliberate control over this approach.

The next step was to give each component its own bundle, in the same manner as our containers. Whilst this was easy enough to implement, it went a bit too far in the other direction; some of our UI components are very small as far as their JavaScript code goes, and it felt unnecessary to have a separate bundle for such a small amount of code. This approach worked better than the previous one, but it needed a bit of extra nuance.

Webpack provides the ability to set additional rules to control when to actually create a new bundle. Up until now, we had been setting rules entirely on a path-based approach; now however, we started to make use of the size-based rules, which also required making use of its priority rules. By setting a minimum size requirement for a new bundle to be created, we could control when to create a new component bundle and when to not. And by setting the path-based rule on our website bundle to be permissive enough to allow components to pass the rule, but with a lower priority than the individual component bundles, that allowed those components that fell short of the minimum size rule to fall back into the website bundle. This approach gave us the best of both worlds; those components which were comparatively large in size could get their own bundle (along with the contenthash-based caching benefits mentioned earlier), and would only be downloaded on pages that needed them, whilst the smaller bundles could remain a part of the core website bundle, downloaded and ready to use on each page. We felt this struck the best balance between not wanting too few, excessively large bundle files to download, versus having too many very small bundle files.

All of these changes have clearly had an impact on the raw size of the core JavaScript bundles we’re delivering to users, but how do we know that will have a positive effect on our end users? We did attempt to run these changes on a non-production environment for a time and compare some performance metrics with our live website, but we quickly realised that we were not comparing like-for-like; our live website benefits from a robust caching infrastructure, which can have a knock-on effect to performance metrics and led to our non-production environment showing significantly worse metrics than we expected.

Instead, we decided to merge in our bundling changes and deploy them to the live website, and monitor the relative performance from before the change.

This change has been in place for 3 weeks now, and has seen the following changes;

• The number of requests for JavaScript files has risen by around 21%, but the total size of the JavaScript downloaded has fallen for all page types — down 14% on average
• JavaScript Total Blocking Time has fallen by 27% on average, and Time to Interactive has fallen by 6% on average
• The measure of how long the browser spends on the single longest JavaScript task has fallen by 16%, and the time on all JavaScript ‘long tasks’ as a whole has fallen by 20%

One of our pages that makes use of more custom components than others has actually seen the largest improvements in some of these metrics — whilst the number of JavaScript files being downloaded has gone up by 43%, the total size of those files has still fallen by 15%; Total Blocking Time has fallen by 31% and Time to Interactive has fallen by 14%. Given that Sports pages are some of those with the most custom components and are also some of the pages which need to feel the most responsive to their users, this is quite a significant benefit.

This still isn’t likely the end of our code-splitting journey; whilst we have arrived at a more optimal code splitting strategy for our current use-cases, we are working on an ever-evolving and growing platform. We fully expect to need to review this again in future as even more pages join our platform, and as new use-cases evolve. There will likely never be a perfect code splitting strategy, but by reviewing ours on a regular basis, and getting it as good as it can be with the information we have at the time, we can keep our focus on providing the most performant experience for our audience.

Optimising BBC Online’s Code Splitting Strategy was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Its been a little over a year since I published my last two blog posts, in which I outlined the process we went through to choose the technology for BBC online and the steps we took to optimise serverless for our use. Recently my colleague Graeme has published a blog post on the organisational challenges we’ve faced in delivering this project.

In this post I reflect on our progress so far, and some of the interesting challenges we are facing while building the BBC’s critical digital services. If you’re a technology builder, interested in the biggest of challenges, we’re always looking for people like you to join our adventure.

Throughout the last twelve months our engineering teams have been reimagining and innovating on our products. In doing so they have moved from our older technology platforms on to our new shared serverless platform called ‘WebCore’. We’re about 30% of our way through moving to this new platform. Despite this, the benefits of using a serverless platform are starting to show.

During this time we’ve not had any operational incidents (lack of audience service, or inability to update our pages) caused by the underlying technology. This isn’t by luck, serverless has handled many of the performance and operational challenges for us, letting us focus on other parts of our platform resulting in a higher quality platform that is more dependable

Using serverless takes away the need to spend time engineering the fundamentals of your platform, instead allowing you to focus on the value you deliver to your customers. So what have we been doing with that time?

Personalisation / Relevancy

The BBC achieves its high levels of reliability and performance by relying upon caching, both internally within our systems, at the edge of our networks and via CDNs. The result is fast loading pages that can be stored at the edge. Yet, in a world where relevant content is key, we need to adapt our services to deliver more tailored and personalised content to our audiences. This means more variation and more load.

We spent a number of months preparing the platform to handle personalised requests end-to-end across the stack. This has now launched for some pages. For example, if you’re signed in, are located within UK territory and visit the BBC homepage you’ll get a page customised for you.

The page takes around 500ms to render and be delivered to the audience. In that timeframe we invoke around 30 functions. Around 150ms is spent running React to render the content to HTML, the rest is spent fetching or processing data.

In February 2022, 145 million requests were personalised, representing 6% of WebCore’s traffic.

Our goal: over the next 12 months we aim to personalise almost every page in some way — making it relevant for every user on every request.

Experimentation

Delivering value and improving audience experience is at the heart of what we do, previous platforms that have relied heavily on caching for performance have meant we’ve been unable to perform experiments on our pages. A/B/n testing allows us to improve the audience experience by understanding what users want to interact with on a page.

Experiments means more variations of a page, consider if we ran an experiment with 6 variations on the global navigation for the website (the bar listing Home, News, Sport, Weather etc.). If we did this, we’d increase the number of pages we’d render by a factor of 6, instantly as soon as the experiment was activated.

The team that builds the global navigation is separate to the team building the news story pages or the news onwards journeys (right hand side bar on story pages). Imagine if all three teams started experimenting at the same time, there would be N * N * N more variations of the page! As builders of the platform running this, we have no control over who runs an experiment with how many variations. We just have to be ready!

We’re in our late stages of development with this piece of work — but when it is active, we’ll enable experimentation across the platform at any scale.

Our goal: experiment on every new feature we release, to ensure it’s valuable to our audience.

Speed of updates

During the working day, the BBC News and Sport websites have editorial changes around ~3 times per second. This includes changes to our news and sport articles, indexes, videos and short form posts (excluding non-editorial updates such as sport events scores).

For those pages that aren’t personalised (and so rendered for you), we want the content to be as fresh as possible. Previous platforms have relied on long (30 second+) caches times and multiple caches to handle the load and provide the resilience required.

Serverless has unlocked the ability for us to process more, reliably. This not only includes the execution environment where we transform our data and render our web pages, but also using serverless Redis that can scale to meet the demands of our workload.

Throughout the last 8 months we’ve slowly reduced our cache times on the stack, such that editorial updates for content have decreased from 2 ½ minutes to 30 seconds.

We think we can improve this further!

Our goal: In the next 12 months, get editorial updates to under 5 seconds.

Build times

Enabling our engineering teams to deploy often and safely is key to our success. So we’ve taken the time to move away from our traditional CI/CD pipeline. Historically we were based in Jenkins on a small number of compute instances. Now we’re making use of serverless CI/CD pipelines to enable us to scale our build and deployment process based on the load and demand on the system.

Its been a change in mindset, rather than running sequential builds, why not run every build step and test as concurrently as possible.

Currently our build times are around 5 minutes, down from 24 minutes at the start of the project due to using serverless for our CI/CD process.

Our goal: Reduce our build times below two minutes!

Only a few minor issues

While our experience of using serverless has been excellent, there have been a few minor issues with the underlying technology along the way.

One of the primary issues we’ve faced, was that of noisy neighbours. Most serverless platforms are shared, in that the provider’s resources are allocated to multiple customers, with all data being securely sandboxed and isolated from one another.

This is optimal from the providers point of view as they can maximise the efficiency of their service. Keeping carbon footprints and costs down.

Our serverless execution is time sensitive, as a user asking for a web page will have to wait for the Lambda function to respond. So we keep a close eye on the response times. We started to notice an increase in our function duration on the hour and quarter hour.

After ruling out any internal issues, e.g. cache clean ups, delays in services providing data, increased traffic from our content systems. We determined this was an underlying platform issue.

The graph shows increased latency on the hour and quarter hour, caused by the underlying serverless infrastructure.

Working with our service provider, we were able to identify the issue as being a noisy neighbour problem, caused by other customers having ‘on the hour’ workloads.

These improvements delivered by the serverless team benefit everyone using their serverless product.

Summing Up

As the BBC’s audience continues to grow globally, we continue to push the boundaries of what we can achieve with the technology. Serverless has proven to be a powerful tool to help grow the digital offering of one of the worlds largest websites, both from content delivery through to powering our build tooling.

Personally I’m very proud of the work which goes on within the BBC. There is an amazing software engineering community, filled with amazingly talented people. Recent events have shown the critical nature of the service we provide, to be part of a team that helps keep the world informed, educated and entertained is a privilege.

We’re always looking for talented people to take on the goals we’ve set ourselves. If you want to work as part of an amazing team, on a world class product take a look at our roles.

BBC Online — A year with serverless was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

In earlier blog posts, we explored how we moved BBC online to the cloud. This was part of a wider initiative called WebCore, which used a range of serverless technologies to reimagine the BBC website. There were technical challenges we faced integrating WebCore with our architecture, but this initiative has been as much a refactor of our organisation, with profound implications on the people within it.

In this blog post, rather than focus just on the technical challenges, I want to share:

• Some of the organisational history that led to WebCore
• How we went about re-organising for success
• How we viewed the team topologies and interaction modes between teams
• How we view the role of the platform (and the Core teams who build it) in enabling many teams to interact with one another

First — some history…

The BBC website is made up of several digital services, including iPlayer, Sounds, News and Sport. Each is a major service in its own right — with millions of visits every week — and they have grown independently of one another over several years. This was reflected in the way the organisation was shaped — with separate departments owning each digital service.

This shape meant each individual product achieved its own aims well, but it also meant the user experience across multiple services wasn’t as seamless and consistent as it could have been. There were mismatched technologies and duplication between the services. Lots of features, despite being conceptually similar, were implemented multiple times — with the total cost of maintenance also paid multiple times. Cross-cutting capabilities — e.g., personalisation and analytics — were major tasks that each product development team had to tackle themselves.

WebCore is part of a strategy to re-imagine BBC Online and create a single website for all the digital services. Standardising onto a single technology set, building shared ‘horizontal’ capabilities, and sharing commodity components between teams — will allow the BBC to better leverage the technology and build more innovative features for the audience.

There are real-world implications to the autonomy and empowerment of individuals and teams. The reality is that WebCore was as much a refactor of the organisation as it was a reimplementation of technology.

High autonomy is vital to high-performing product development teams. The previously decoupled technology stacks made autonomy easier to achieve, so the WebCore strategy forced us to think about how we could balance autonomy with standardised technology.

Organise for success

Humans will rally behind a common cause and form close-knit relationships within their group. Cohesion is a powerful thing — the sense of belonging brings people together to achieve something greater than the sum of the parts. We leverage this within any organisation to significant effect.

Missions are typically multi-team initiatives within larger organisations, such as the BBC. We can group teams together very deliberately with our organisation shape (i.e., departments) or through more fluid constructs that cross departmental boundaries. Departments are important as a primary way to create purpose and alignment amongst teams, and there are multiple ways they can be shaped. Interactions will inevitably be stronger between teams within the same department as opposed to ones that cross departmental boundaries

The evolution of the BBC’s digital products was reflected in the internal department structures, which were aligned with digital service ownership. Operating across these traditional departmental boundaries — e.g., improvements to the time it takes for newly published content to be rendered to the audience — was more difficult.

One early WebCore decision was to create a new internal department called Core Services — a department that is not orientated towards delivering one of the audience-facing services.

Instead, this is a set of teams with a mission centred around platforming and enablement for all the services. Rather than each department owning and running their own platform services, there is a specialised department for this.

The WebCore platform is an internal product, made of a generic set of components (systems, services, and frameworks) and the customers are the internal product development teams. Rather than basing their success on audience engagement, these teams strive to elevate across the organisation and enable the tenant teams who depend on the platform.

Optimise rather than maximise

In the first stages of migrating content onto the WebCore platform, there were a handful of teams interacting with one another — the core platform teams along with the first tenant teams to contribute components. This meant it was workable for teams to be aware of the components other teams were creating. There was a desire to maximise re-use and avoid any duplication; after all, this was the fundamental premise of WebCore.

Over time, the number of teams using the platform grew. Before long, over 100 engineers were creating components — and keeping track of what was being made became much harder. The early high-bar of ‘zero duplication’ was creating friction — how could teams move fast, iterate, and be inquisitive without causing duplication?

The answer has been to develop a more pragmatic mindset towards technical debt — developing tools to manage it while accepting that aiming for zero debt is not cost-effective. Just like financial debt, technical debt is not always a bad thing. Debt becomes a problem if it is not tracked, becomes too large or if debtors cannot be held accountable.

For example, Component Health tooling allowed us to create a sense of ownership of various parts of the codebase. By instrumenting various key health metrics, teams were allowed some latitude to make their own decisions and be accountable for them over the component’s lifecycle.

This highlights an important principle — inverting control back towards tenant teams.

Inversion of Control

Using shared technology can create a sense of disempowerment within teams. When a product development team is used to owning and controlling its entire technology stack, moving to a situation where they are dependent on others can be uncomfortable. However, from operating systems to frameworks, no team writes everything from scratch.

Bringing back the sense of control is important — and this is where maturing the platform to provide tenants with self-service interfaces and tooling is central. While earlier phases had focussed on building the systems that can handle traffic (scale, resilience, performance, observability etc.), we have increasingly looked at how we scale up the enablement of engineers and teams. To do this, we looked at team topologies, and especially interaction modes between teams.

Team Topologies — interaction modes

In the early phases, we tended to view enablement as “collaboration between core teams and contributing teams”. This was the right fit in those early days — everything was new to everyone, and the number of interacting teams was manageable. However, as the number of teams contributing components has scaled up, we now have a more nuanced view of a spectrum of interaction modes between teams that are available to us.

Collaboration — this is when a group shares the same mission. When multiple teams interact in this mode, it might look like a cross-team squad who work all day, every day, for an extended period on the same mission. Perfect for pioneering.

Co-operation/Facilitation — This can look like a support model; we have mechanisms where teams can ask for help when they need it. It can also be about encouraging cross-community support, either from subject matter experts or from the wider product development community. Perfect for settling.

Systemisation — this is about removing manual intervention. It could be documentation, processes, tools, or user interfaces. Perfect for town planning.

Collaboration is expensive but is ideal when new ground is being broken. It accelerates the feedback between the tenant and core platform teams, allowing gaps in the platform to be addressed quickly. However, it is essentially a manual intervention and does not scale.

Systemisation is about removing manual intervention and is the horizon goal for the platform. When the patterns are proven then the teams that follow can tread the beaten path. Golden path tooling can be used to accelerate teams and keep them within the guardrails of the known-good path. Within the limits of these guardrails, the tenant teams have a high degree of autonomy and can deliver to their audience-facing services with minimal impediments.

Co-operation and facilitation are the grey middle ground between the two. While tools and processes can alleviate some of the burden, this is always likely to be needed. Individual people looking for help, staff turnover etc. will always create a need for this.

Lessons learned

It’s not always been pretty — but it’s been a fascinating journey and one which I think a lot of organisations will tread.

Organise around the right mission — whether it’s the team or department structure, this is the primary way to create alignment and group cohesion. If it’s not right, the result won’t be either.

Optimise, don’t idealise — with lots of humans sharing and contributing to the same codebases, there will need to be some rules to keep everyone on the straight-and-narrow. Between humans, try to keep these as rules-of-thumb. Over time, use dashboards and tools to guide the humans and let the robots create enforcement.

Work together but be deliberate about it — sharing technology will require engineers working with other engineers across traditional organisational boundaries. It can easily become overwhelming, so pick the right interaction mode for the situation. There may not be one single optimal interaction mode, so a blended approach will be needed.

The key from a platform perspective is to try and get out of the way and let the tenant teams deliver unimpeded. This means relentlessly finding and removing any parts of the development lifecycle where the platform is an obstacle. This may mean providing tenant teams with controls — configuration, tooling, user interfaces etc. — but it may also mean agreeing and enforcing certain standards.

Autonomy is not a silver bullet — without alignment between teams, autonomy can breed inefficiency and low-value variation. Alignment can be created by organising humans and teams with a clear purpose. It can also be created with an opinionated platform — one which makes it easier to do the right thing, and harder to do the wrong thing. Within these constraints, there is plenty of scope of high team autonomy.

The ultimate success of WebCore will be measured over time — in excellent and innovative digital products that will inform, educate and entertain the BBC’s audiences for the next decade. This organisational refactor is a trade-off between optimising for standardised commodity components and the autonomy needed for innovative features.

Change is hard. Substantial change especially so — and where humans are involved in an endeavour that is as complex, creative, and technical as product development it can be challenging to keep sight of the big picture. But by realising that an organisation is a system — albeit a hugely complex one — then we can apply leverage to steer effective change.

Refactor → Organisation was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Imagine working in a team that develops and runs digital products that are used by millions of users every day.

A team that is not only responsible for some of the biggest websites in the world but on top of that has one of the pioneering and most advanced video streaming services there is, available on hundreds of different devices, alongside an audio streaming service used by millions on a weekly basis.

But that’s not it, there are more services that are truly positioned to do good in the world like helping parents, teachers and kids get all the fun and information they need. Digital products which deliver unbiased news across the world, which join up communities across the UK, and which appear in multiple different languages for audiences at home and abroad. Digital products that are equally used to help you understand what’s happening globally, the latest from the world of sport, to find out the weather outside your front door, and to provide a recipe for you to make this evening.

That’s what we do. That’s our BBC.

I won’t lie to you, this is quite a complex operation but with our mission to inform, educate and entertain, we wouldn’t have it any other way. We pride ourselves in our public service mission: delivering value to all audiences.

The truth is that in the middle of this complexity there is huge opportunity and we continuously strive to improve, not only the quality of our audience experiences but also the quality of all the tech that powers it. We run pretty much all these services and products in the cloud and make the most of that, whilst improving our practices to ensure that the BBC’s digital services are as reliable as its broadcast ones, the best there is.

It’s tricky though, there are hundreds of micro services across our estate so we have cross discipline teams that are responsible for each of these. We do our best to strike that balance between empowering each of these teams and ensuring that we have high collaboration across the board, there is always a lot we can learn from each other and a lot we can do to minimise waste, we are dealing with public money here and never forget that.

Recently, we decided to write down our vision for Software Engineering and we think this gives a pretty good idea of what we are about:

We want the BBC to be as renowned for the quality of its engineering as it is for the quality of its content.

We aspire to be the best, so that we can engineer outstanding digital products, at scale.

Working here in an engineering role means being part of a world-class team — and a chance to do the most meaningful work of your career.

And to help explain more of what this means, we came up with five guiding stars that we use not only to aid our decision making but also our day to day behaviour & focus

• The place to work: We want people in the tech industry to choose the BBC as the best place to grow their career. It’s a place to continuously learn, to be collaborative, to be creative, and to deliver challenging yet rewarding projects
• Building the best: We aim to create the very best digital products, at scale — backed by engineering rigour that ensures audiences can always depend on them
• Strong engineering leadership: At all levels, our engineering leaders work together to deliver on our goals, to build strong, open and diverse teams where people truly belong
• Operationally excellent: The quality of service of our digital products is at the same level as our broadcast services — the best there is
• Clear technical strategy: We have a clear, pragmatic tech strategy, that aligns with our business and product goals, and that we execute on, together

There is lots to do of course and we have some ambitious plans for 2022 and beyond, if you think this is your kind of place, we would love to hear from you. Our team is growing and we want to continue to make it as diverse and inclusive as it can be so do get in touch!

It’s time we all started thinking of BBC’s Engineering as one of the best in the world was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

On my first day all the team were very welcoming and friendly, and have explained (and continue to explain!) every BBC acronym very patiently. As a new Product person, it’s often intimidating to join a new organisation, not to mention within a new domain. As I have joined during the COVID-19 Omicron wave, everyone is working from home. I am keen to meet the team in person as much as possible — though we are quite geographically distributed so we will have to plan quite carefully.

Product

On an organisational level — the first thing that was clear to me was that the BBC Product Culture follows the principles and values from good Product Management practice such as that advocated by product experts such as Roman Pilcher or Melissa Perri. Having read many books and blogs over the years about say — having a roadmap that indicates what you are working on now, next and later to offer transparency to stakeholders rather than a delivery plan — has always been something I’ve desired to do, and here it is being used in practice!

Agile Values

Another thing I noticed was the enthusiasm for teamwork and team values — as one of the principles of the Agile Manifesto that I am passionate about “The best architectures, requirements, and designs emerge from self-organizing teams” — again in many organisations this may start as an ambition but be waylaid by older management practices and a blindness to what is a creative process.

My own reaction to this approach is that I feel there is the right focus on thinking about why we are building, and even if we should be building — this makes me feel that people think the same way that I do, and that experimentation and exploration in the problem and discovery space is supported

Expert Voices

Finally, there are many truly expert specialists at the top of their game to support you — from world class UX through willingness to focus on the data for decisions, to teams dedicated to experimentation and testing, it’s obvious quite quickly that the BBC has many very skilled people to support product initiatives. I’m excited to move on with my next 30 days and see what they hold!

My First 30 days as a Product Manager at the BBC was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

The premise is simple, given a theme or problem space, the hackathon participants must self-organise into teams to explore ideas, write some code, and deliver a working prototype or demo in a ridiculously short period of time.

In my time at the BBC I’ve taken part in many different flavours of hack — from large multi-department events, collaborations with other organisations, 24-hour hacks, week long hacks … no two are ever the same. It’s also been my privilege to organise many of these events — in particular our annual end-of-year hack week.

The tradition of the end-of-year hack predates my 9 years at the BBC. It’s origins are back in our pre-devops days, where our operations team managed our live systems for us, and the run up to Christmas was accompanied by a strict change freeze that brought all development to a crawl. Using this downtime for hacks and ideation was a great change of pace and a chance to connect with colleagues.

The days of strict change freezes are long gone, but the end-of-year hack has just grown and grown over the years, and has firmly established itself as a fixture of software engineering at the BBC. Our most recent hack was the biggest ever, a week-long event in the middle of December, bringing together all our development teams. Everyone was invited to participate; Omicron had just cast a new shadow of uncertainty, and forced us all back into Zoom again. A week focused on innovation and collaboration was very much the distraction we needed.

If you ever have the good fortune to organise an event of this size (we had 68 teams and nearly enough participants to smash the 500-person Zoom limit), there are some things we’ve learnt over the years to make these events the success…

Allow teams to self-organise

Resist the urge to over-manage the hack. Teams tend to pull together the people they need, and the good ideas float to the top. As an organiser your job is to make those teams and ideas visible, and join up teams where there are similar or complimentary ideas.

Identify your goals up front

Why are you running the hack? Is it for ideation? Is a development opportunity? Maybe its only purpose is to encourage collaboration — breaking down team boundaries and encouraging hacking across your organisation with people you wouldn’t normally work with is a fine goal in and of itself.

Target all disciplines

Hacks by their nature can seem a bit intimidating to the non-programmers amongst us, but that’s a real missed opportunity. A good hack idea can really be elevated by a multi-disciplinary approach, but also gives your contributors a safe and temporary opportunity to try other roles and exercise underutilised skills.

Get management buy-in

Whatever you do make sure you have the support of your senior management. You’ll need them to help publicise your event, but also to buy you the time away from the day-to-day of your regular work. You’ll also need to plan way in advance to make sure your contributors can step away from their regular responsibilities without it causing panic.

Work out how you’re going to showcase the ideas

With a small event, you can easily get all the teams to demo at the end of the event, but when you have 60+ teams this quickly becomes unfeasible. We opt for getting teams to produce a short (2 minute) video and compiling a showcase of all the hacks — or a selection of the best ones. If your event is massive you might want to consider a competition scenario to drive interest.

Remove the pressure to succeed

There can be too much pressure for teams to deliver a new and brilliant feature to your users. It needs to be absolutely fine for a team to say: “we tried it and it didn’t work”. Some of the best hacks I’ve seen focussed on clarrifying a technical direction, exploring some new design concepts, or shaving minutes off the build process. Even just an outcome of “we learned some stuff” should be applauded.

Whatever you’re planning, remember you’re trying to facilitate creativity — so you need to give everyone the freedom to think big, but the right constraints to focus that creativity (just google “constraint + innovation”) if you want to dwell on that some more. In the meantime, we’ve made a short video that captures some of the highlights of our most recent event, and some thoughts from our contributors…

BBC Hack Week was originally published in BBC Product & Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.