Bloated JavaScript causes a page to load slowly as your customer’s device has to download, parse and execute every byte, so every byte counts. As developers, we often don’t feel this performance pain because we write and test our code on high powered devices, but our customers may use older, slower devices with unreliable network connections. Some performance improvements like compression help by reducing the file download time, but uncompressed, the file is just as large, and the device still has to deal with these large files. Therefore, sending less JavaScript makes our pages load faster.

In this blog post, I’ll present a performance problem, and how I improved it. One of our pages allows people to create an issue by scanning a QR code. Except, we found that less than half of the people who scanned the QR code completed the flow. Our team thought that due to the poor performance of this page, our customers didn’t want to wait for the page to load. To fix this problem, I identified and made some small changes to improve the performance, which significantly increased the retention rate. When I work through performance improvements I think of it as a series of experiments (like in this previous blog post) so that I can measure the impact of each change, and that’s how this blog post is set out as well.

Starting Point

When I first started looking at this page it had a Lighthouse score of 48, so there was a lot of room to improve! My primary focus was to reduce the JavaScript we sent, and to improve the way the page loaded. I felt confident we could get this much higher. Along with Lighthouse score, I focused on improving First Contentful Paint (FCP) and Time To Interactive (TTI).

Experiment 1 — Proper bundle splitting

This page was designed to be used by unauthenticated users. However, when this page was first created, it was not split from the normal authenticated SPA due to a deadline. The side effect of rushing this work out was that we were making our customers download a lot of code that they didn’t need. Whenever someone scanned the QR code, their browser downloaded a lot of code, parsed it, and then discarded it. This impacts performance by slowing down the network request and giving the device more work to do.

Hypothesis: If this standalone page is split from the main application, a lot of this unnecessary code won’t be downloaded. This will decrease download, parse and execution time.

Method: This set of changes ended up being fairly simple. I moved this page from the main bundle into it’s own package, untangled some internal dependencies and then set it up to be deployed as part of a separate pipeline.

Result: This simple change led to a significant improvement by improving the Lighthouse score from 48 to 78. More importantly, these results validated the approach of sending less JavaScript to improve performance. By splitting the page out from the main SPA we removed unnecessary API calls, reduced the size of downloaded code bundles, and simplified the network waterfall. By doing so, this change improved all of the core metrics that we measured.

Bundle Size: 2642kb => 2012kb
Lighthouse: 48 => 78
FCP: 2.6s => 1.1s
TTI: 3.7s => 2.3s

Experiment 2 — Improving the waterfall

In Experiment 1, I observed that the network waterfall became simpler by reducing the requested JavaScript. I wanted to pull on this thread to see if we could simplify the waterfall even more.

Hypothesis: Below is a screenshot of the page’s network waterfall from WebPageTest. The network waterfall is a representation of the network events that occur to load a page. It generally consists of downloaded JavaScript bundles, as well as other network calls that are used to fetch data. In the below image, the orange bars represent JavaScript, and the pink bar after them is the execution of that JavaScript.

The flow of this page works like this.

1. The initial index.html is downloaded (position 1).
2. It downloads the vendors~boot.js bundle (position 2) and the boot.js bundle (position 3) because they’re included in the <head> tag of index.html.
3. Once these two bundles are downloaded, parsed, and executed, boot.js executes a request for some more required JavaScript and parallel network requests to fetch this are instantiated (lines 6–9).
4. Once these bundles have been processed, one of the bundles initiates an API call to fetch some data to display on the page (position 11). This API call happens 8 whole seconds after the page is first loaded.

If these separate bundles can be combined, the application will download the same amount of code, but the amount of time spent on network and execution calls will be reduced. Therefore, the API call that fetches data will occur sooner, and the page should load quicker.

Experiment: Because this page now has a separate pipeline, we don’t need a generic boot.js file for the SPA and the required code can be sent to the client in a single bundle. To do this, I changed the Webpack configuration for this page to include all of the required code in one file instead of splitting this into multiple files.

Result: As expected, the overall bundle size didn’t change much. However, I was hoping that the network waterfall would be simplified. In the below screenshot, a similar amount of JavaScript gets downloaded, but the API call happens a lot earlier. Now, it’s moved up to position 4, and is initiated at around 5s, instead of 8s in the previous waterfall. Additionally, the network waterfall is much simpler; only two JS bundles are downloaded instead of 6.

Even though the bundle size only shrank by about 50kb, simplifying the network waterfall improved the Lighthouse score from 78 => 89 which is a big win.

Bundle Size: 2012kb => 1960kb
Lighthouse: 78 => 89
FCP: 1.1s => 1.4s
TTI: 2.3s => 1.4s

Experiment 3 — Adding a skeleton loader

Hypothesis: We thought by that adding a hardcoded skeleton to the index.html file, our customers will see something straight away. We weren’t sure what impact this would have on page load, but we hoped that this change would improve perceived performance. One of our reasons for people not using the page was that it was slow to load. This can be addressed by improving the pages performance, but by showing something to the customer we improve the perceived performance and enhance that impact even further.

Experiment: This experiment involved replacing the existing generic index.html page with one that was based on the layout of this page. Most SPAs have a generic index.html page that is used as a base for any page that is loaded. This works well for a multi-page SPA, but forces the browser to redraw the page when new elements are rendered. By having a skeleton that looks similar to what the final layout will look like, the browser has less work to do when the actual content is rendered.

Result: This change actually increased most of the core metrics by a small amount, however FCP dropped from 1.4s to 0.2s. This is still a useful change, because perceived performance will improve, even if some of the metrics have degraded.

Bundle Size: 1960kb => 1960kb
Lighthouse Score: 89 => 86
FCP: 1.4s => 0.2s
TTI: 1.4s => 1.4s

Experiment 4 — Manual tree shaking

At this point in time, this application was on Webpack 4 in a Lerna monorepo. Throughout the course of this work, I spent a lot of time staring at JavaScript bundles and trying to understand where certain imports came from. Whilst staring, I came across a number of anomalies that didn’t need to be there like constants from other domains, random images, unnecessary client code etc. Analysing these imports made me realise that Webpack couldn’t always figure out what was needed from a module and what wasn’t. So to be safe, Webpack included everything from that module. This was good for correctness, but bad for performance and indicated that tree shaking wasn’t working correctly in our environment. This particular problem was fixed in Webpack 5, but we were blocked from upgrading due to other reasons.

Hypothesis: The workaround in the short term was to manually tree shake by being more specific with our imports to make the bundle smaller. Importing something from the index.js at the top level of the package like this:

import { someFunction } from "@safetyculture/top-level-package"

ended up including everything from that package in the bundle 😳. This led to some pretty extreme blowouts in package sizes. Updating the import to look like the below should shrink the bundles by giving Webpack better cues on what it can eliminate, and letting it discard everything that it didn’t need.

import { someFunction } from "@safetyculture/top-level-package/a/much/more/specific/directory";

Experiment: I wouldn’t recommend manual tree shaking, but fortunately this page was fairly small and I could trace the imports quite easily. I went through and updated the imports to be more specific, and got some good results.

Result: After this experiment, I understood why getting automatic tree shaking to work was a big deal. By updating a few imports to be more specific, about 300kb was removed from the overall bundle size and the Lighthouse score increased from 86 to 93.

Bundle Size: 1960kb => 1664kb
Lighthouse Score: 86 => 93
FCP: 0.2s => 0.3s
TTI: 1.4s => 1.2s

Conclusion

This series of experiments led to a huge improvement on this page. We saw a large increase in user engagement which proved our overall thesis that bad performance was driving our customers away from this page. There are a number of ways you can improve the performance of your page, from decreasing the bundle size to simplifying the network waterfall, and your customers (and the rest of the business) will be grateful you took the time to make these changes.

If you have some performance improvements that you’re interested in making, I’d encourage you to break it up into small experiments like this. It’s easier to measure, plus you can use the gradual momentum and improvements to convince others that this work is important and worthwhile.

Overall improvement

Bundle Size: 2642kb => 1664kb
Lighthouse Score: 48 => 93
FCP: 2.6s => 0.3s
TTI: 3.7s => 1.2s

Make pages load lightning fast by shipping less JavaScript was originally published in Mitti Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

A customer service mindset means putting the needs of your customer first and working to improve their experience. Engineers think about their customers, but they’re often forgotten when we start thinking about technical details. Technical details matter, but your business lives and dies on its customers, and having them front of mind will lead to a better product.

Shifting your mindset

How you define a customer differs depending on your role, for example, on a platform team, your customers are the engineers who use your products. We’ll be focusing on customer facing team where your customers are the end users of the product. In engineering we usually don’t interact with our customers too much. Depending on where you work, you may never interact with your customers. However, there are others in your business, such as sales and customer support, who do. Speaking to these people gives you valuable insights into how your customers are using the product, and give you opportunities to improve it.

Communicating with more customer facing people in the business helps to shift your thinking to put your customers first. This helps you solve urgent problems sooner, but is a valuable skill to allow you to increase ownership and impact. Doing this opens your mind up to asking questions and trying to do what is best for your customer. I’ve found that work is much more engaging when you can think deeper about the impact you’re having, rather than just working on the next ticket that comes your way. This helps you care more about your work and become a better engineer by really understanding the problems your customers are facing, and influence how to solve them.

Technical tasks through a customer lens

This also allows you to look at technical work with a different lens and think about the customer impact your changes can have. Being able to express this well helps convince non-technical folk about why a piece of work matters and can help to prioritise it.

For example, there’s part of your codebase with poor test coverage. The engineers on your team hate working on this area, but nobody has the time to improve the tests. Tying these tests to their customer impact makes it easier to prioritise, and will allow you to think about the kinds of tests you should write. Compare these two approaches:

I want to write some tests for this component.

vs

I want to write some tests for this component because improving the code quality will give the team greater confidence in the code, make it easier to make changes in the future and will provide an opportunity to fix any bugs that customers may be experiencing at the moment

The second statement is a bit wordier, but explaining the customer impact of this work makes it much easier to get others on board, and get it done.

How to adopt a customer mindset

Developing a customer service mindset comes from practice like any other skill. However, it’s different from a skill like programming that develops through work; you need to re-train your brain. There’s a number of ways you can do this:

1. Answering questions about your product area from around the business. We have internal Slack channels for each product team, and there’s often questions about our team’s product. Answering these questions helps you understand different issues, and will make you feel a sense of ownership over your product. Plus, your teammates will appreciate your help.
2. Reviewing customer facing documentation. Depending on the size of your company, there may be a team who write customer support documentation. When your team releases a new feature, read the documentation for it. This helps build empathy for those writing the documentation, helps you understand how the product solution is presented to customers, and gives you an opportunity to improve the documentation as well.
3. Writing customer facing documentation. Conversely, you may have opportunities to write customer facing documentation, such as for API endpoints. These opportunities are valuable because they give you a great opportunity to think about how your customers will use the endpoints, and allows you to think about how to best present this information.
4. Site visits and customer calls. Speaking to customers is a great way to build empathy for them, as well as understand how they use the product. These understandings help engineers see the impact of their work, and can be used to make the product even better.
5. Investigate metrics for features you’ve worked on. Once you’ve built something, work with your product manager to look at how it’s performing. This is great validation for the work you’ve completed and can give you some insights into if it was the right solution.
6. Read feedback about your product. Your product team talk to your customers a lot and they may collate this feedback somewhere. Read through this, and see what your customers care about. Some of these may be easy to implement, if so, see the next point.
7. Implement quick wins. This is the best way to do this, and is often the result from the previous points. If you hear about a problem that seems easy to fix like poor scrolling or bad keyboard controls, fix it. Nothing will make people feel heard more than a quick resolution to a problem they’ve raised. Beloved products are built through many, many small changes and this is a great way to improve your product over time.

To summarise: show up, care, make things better. A customer service mindset allows you to see the opportunities in your product, and quickly addressing these will elevate your product from good to great.

Developing a customer service mindset as a software engineer was originally published in Mitti Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

First steps

Think

Feature leading has two goals. The first, and most obvious, being to get the feature over the line. The second being to develop a really deep understanding of the feature. Ask lots of questions, write everything down, go for a walk, and then come back to ask more questions. Try to understand the big picture goals and low level minutia of the feature. Think about questions that your teammates might ask to be better prepared to answer them. Try and understand the feature deeply, and cover as many scenarios as possible.

Plan

When I first started programming, I’d always jump right into a problem; I’d start hacking something together, only to realise that my initial attempt wasn’t quite right. As I gained more experience, I learned to stop and form a plan before making any large code changes. It’s the same with feature leading. Jumping in without a plan will lead to edge cases, and make completing the work much harder. Time spent in planning is never wasted, and leads to better outcomes.

Find the why

An important question to figure out for this feature is ‘why are we building this?’. Growing as an engineer involves lifting your focus from only code, to thinking about what you’re building and why. Code is a tool used to solve a problem, and that problem is the one solved by the product. Thinking about the why ensures that the solution solves the problem. Finding the why, and lifting your focus beyond code makes a better engineer, and will make you a more effective feature lead.

What do you need?

Before you start planning, do you have everything you need? Do you have all the required designs? What do the different user flows look like, including error states? What are the required changes? When is the projected release date? Make sure you have everything you need before you start planning. This ensures you haven’t missed something critical that could put the feature at risk.

Who needs to know?

This feature may interest different people or departments; find out who they are. Think about the different people usually involved in new product releases. Does the customer support team have enough lead time to write customer documentation? Are marketing running a new campaign? Will any of the code changes have an impact on other parts of the platform? Part of the role of a feature lead keeping everybody in the loop about the feature. When you’ve thought about who needs to know, this becomes much easier.

Measuring success and determining scope

What’s the goal of this feature, and how will the team know when they’ve achieved it? If you’ve set out to do one thing but do another, the feature may not have achieved its goal. With a goal in mind, it’s the priority of a requirement can be identified. Knowing the goal helps to determine the scope of the feature.

The requirements of the feature make up its scope. If a requirement is “In scope”, it must be completed. Similarly, “out of scope”, means the requirement won’t be completed. Some requirements will be neither in or out of scope, and are “nice to have”. Scope is important because the team will only have limited time to deliver this feature.

Time, scope and cost are the three parts of the project management triangle. This says that the competing demands of scope, time and cost determine the total quality of a project. For example, cutting scope could lead to an earlier completion date, but may lead to an incomplete feature that doesn’t achieve the set goals.

Therefore, a feature lead’s job is to maximise quality, while managing scope, time and cost. One of the most powerful ways to do this is by changing and negotiating which requirements are in scope. This also means that some requirements will be left out. This is necessary to ensure the product is of high quality, and completed within a reasonable timeframe. Cutting scope can help to focus a feature. If a feature is too large and there isn’t enough time to complete it, this will often lead to a rushed and buggy product.

For example, these hypothetical requirements are to add some new functionality to a form.

Requirement: Update the form with a new redesign.
Response: This will add an extra two weeks, so this is out of scope. Maybe we can do it as a separate feature, because the work is complex.

Requirement: Update the colours to fit the new brand theme.
Response: This will only take an extra two days, so this should be possible. Because it’s not completely necessary, this work is nice to have. We’ll try and fit this in if we have time.

Requirement: Add the new form field and connect it to the back end. Response: This is in scope, because without it the feature doesn’t achieve its goals.

If a feature is too large and there isn’t enough time to complete it, this can lead to a rushed and buggy product. Therefore, managing what will and will not get completed in a given timeframe is critical. Some requirements can be rejected during planning, and your team can help reject others later on when your estimates are firmer. It sounds scary to push back during planning, but this is expected of a feature lead. The feature lead has the most context on the technical side of things, and is in charge of the feature. They are therefore best positioned to help steer the feature in the right direction. If the feature lead doesn’t push back where appropriate, they’ll be under pressure to get the feature back on track if it starts to fall behind. It’s easy to add extra requirements if there’s spare time, but much harder to remove them if the timeline gets tight and promises have been made. The team will be grateful if unnecessary requirements can be trimmed early. Once this information has been finalised, add it to a product page in Confluence or somewhere similar. This will be the source of truth going forward, and will be often referred to during the course of the feature.

Communicate to the team

Once the engineering manager and product manager agree on the requirements, bring in the team who will be doing the work.

At SafetyCulture, we do this in a Product Kickoff meeting. This meeting brings in product, design and engineering to align everybody on the goals and outcome of the feature. During this session, the team will ask questions that might not have been covered yet. This is a great chance to make sure that any questions are answered and all requirements are documented. Make sure to document anything new in your product page.

Design the technical implementation

Next, you’ll need to plan the system design for the implementation. It’s important that the implementation is sensible, performant and scalable. This is important, because changing the implementation before anybody has begun coding is much easier than changing it after.

System design is a large topic that’s out of scope (get it?) of this article. There’s many resources out there that focus on it, so check those out to learn more. This article will instead discuss how to present your design.

At SafetyCulture we write about the design in a Confluence document. This way, your team and other teams can reference this page as required. It’s also useful as a historical document for the engineering team’s reference in the future. When you’re writing a design out, think about these points:

1. Outline the problem. Your technical solution exists to solve a problem. Ensure that whoever is reading this document knows what that problem is. Write this for an audience that’s broader than your team. Your team may know the problem in depth, but others may not.
2. Write for a particular audience. I try to keep my CTO in mind when I’m writing up a design. They understand technology and the business, but may not have a deep understanding of implementation specific details. This helps me think about what’s relevant, what to expand on and what doesn’t need as many details.
3. Start broad and narrow down. Start with the big picture, and then separate out larger portions into their own section. Don’t start from a narrow context, as this makes it harder for you to communicate the complete picture.
4. Keep details relevant. Only include relevant details. Specifying that the feature is using Postgres and then doing a deep dive into Postgres itself isn’t helpful. However, outlining a new database table and its columns and indexes would be very helpful.
5. Discuss any alternatives. During the design process, there may have been some alternatives that weren’t chosen. Mention these, as someone else may think about this solution too. It’s helpful for them to know that these possibilities have been thought about and disqualified. This also gives the team confidence that the proposed solution is the best way forward, and not the first idea you came up with.

After you’ve done this, get feedback on your design. Start with your team, and then open it up to a wider audience. SafetyCulture runs a design review session where teams present their designs to collect feedback from engineers outside of their immediate team. This session helps to refine a design, and helps shape them into better solutions. If your organisation doesn’t do this yet, organise it for your next design. Others within engineering will benefit from this as well.

Figure out and plan tasks

After you’ve incorporated any feedback into the design, break the requirements down into stories. At the outset, be super granular with the aim of combining some tickets later on. I do this in a rough document before moving the tickets to Jira. It’s easier to get feedback on the tasks, as well as consolidating them if needed. Doing this in Jira means a lot more clicks to change or combine a task.

Think about this in broad tasks like, “Set up new microservice”, or “Show new information to customers”. These broad tasks can now be broken down into the specific changes that they need. “Display new information to customers” might include updating your web, iOS and Android clients, updating the endpoint to return new data and adding end-to-end tests. These five tasks are easier to communicate, assign and parallelize between the right people, as opposed to the one big task.

Work vertically instead of horizontally. It’s easy to think about tasks as “backend” tasks, or “mobile” tasks. A better approach is to split the work up so that smaller pieces of work can be tested and verified early. Let’s say we are creating a new endpoint and modifying an existing one. A horizontal approach would be to work on all of the backend tickets for each endpoint, then mobile tickets to support these changes, and so on. The problem is that it becomes harder to test a set of changes until the feature is close to completion. By working to finish one endpoint before the other, it’s easier to de-risk the feature and build confidence in the testing plan as early as possible. This means less testing at the end because large portions have already been tested

Sequence the work to make things smoother. When looking at the work involved, the order that functionality is completed can make a huge difference. For example, part of a new feature work involves adding a new data type to an existing data structure. If the vertical slice that allows that data to be added via the UI is done first, it’s much easier to work on the slice that focuses on fetching and displaying that data because the team can add the data in the same way a real customer would. If this was done the other way around, the team will probably be adding data manually to the database. This works, but is slow, error prone, and much harder to find edge cases. By figuring out the correct sequence of work the implementation of the entire feature is much simpler.

Next, go through the tasks and make sure that the implementation works if the tasks are executed as outlined. Try to find any gaps, missed areas or flaws. Do this with a teammate to ensure it makes sense, and make sure that nothing has been missed. Add any missed tasks in, or tweak the design if needed.

Add some estimates to the tickets to get a rough feel for how much time this work will take. Bring this to your engineering manager and get their feedback. If the estimates are higher than the allotted time, you might need to work together to cut some scope or revise the target delivery date. With these estimates in hand, I’d recommend visually planing out the tasks. I’m a big fan of Jira’s planner functionality. It includes functionality to assign tasks to people, add blockers and visualise the flow of the feature. This way the work can be spread evenly amongst the team so that nobody is stretched too thin. Visualising the tasks is optional, but it’s a helpful way to see how the feature is going, and correct course if it begins to to slip behind schedule.

Align your team and start

With your plan and tickets, it’s time to bring your team together and get the feature underway. At SafetyCulture we do this as part of an engineering kickoff. The purpose of this session is to make sure the team is ready to start working on the feature. The team should have already read through the implementation, but go through it with them to ensure that everyone is on the same page. When everybody understands the big picture, go through the tasks and get the team’s estimates. Hopefully the estimates are similar, but if your teammates have different estimates, add theirs to the ticket. Their estimates should give good insight into how long this feature will take.

From here, you and your team can start working on this feature. During the course of the feature you may find some problems or gaps. This is okay! Your job is to facilitate fixing these problems, and finding ways to plug any gaps. Provide solutions to these problems and be open and communicative to your stakeholders.

The bottom line is to be honest and as communicative as possible. If the feature is slipping behind schedule, talk about it. Your manager and team want to know when things are going off course so that they can help you fix it. Hiding information only make things worse, even if being outspoken feels difficult.

Test and deploy

Once the work is complete, run a testing session to ensure that everything works as expected. Come up with scenarios that cover happy paths, and various edge cases. This will give you confidence that when your customers are using it, you won’t get any nasty surprises. You’ll only be able to test the whole thing when the work is complete, but you can still test the feature while your team is working on it, especially if you’ve broken the work into testable vertical slices. This will help you find bugs and missed edge cases before your customers do.

Once you’re confident with your testing, switch on the feature flags and progressively roll it out. Keep an eye on alarms, logs and performance metrics, and don’t be afraid to halt deployment if you find something.

Reflect and retro

Congratulations! That’s your first successful feature completed. Schedule a feature retro, and find out how your team felt about it. Take any feedback on board and work on their suggestions next time. Find some time to self reflect and give yourself some feedback too. You’ll have learnt a lot, and now is a great time to consolidate these learnings.

Make sure to document any cut scope. Your team may be able to work on this as a quick win sometime now that most of the hard work is out of the way. If the cut scope was a bit larger, it might be an additional feature that needs more work and planning.

Finally, give yourself a huge pat on the back. Leading your first feature is incredibly challenging, but very rewarding. You’ll have levelled up a lot in your product knowledge, technical expertise and understanding of the software lifecycle. Plus, the greatest reward of all, you’ll be a better engineer afterwards.

TL;DR

1. Think and determine scope
2. Product kickoff
3. Technical design/implementation and feedback
4. Task breakdown
5. Engineering kickoff with the team
6. Plan things out visually
7. Test, deploy, retro
8. Congratulate self, rest

An introduction to feature leading was originally published in Mitti Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

When exploring this space, I found that a lot of articles focused on contrived examples, which made it difficult to apply this to production code. This article uses a real-world application that I work on every day, and explores some improvements I made to it. By the end of this article, I hope you’ll have some new knowledge about how to identify and tune performance bottlenecks.

The application that we’ll focus on powers the Issues product at SafetyCulture. It, and all of our other services, report application stats to Grafana, including GC pause time. Before any improvements were made, the average GC pause time for this service was about 16ms, with the max GC pause time spiking from around 500ms — 1s.

As Go can do sub-millisecond GC pauses, this seemed high. However, considering this is a constantly running webserver handling a large volume of HTTP and gRPC requests, I wouldn’t expect numbers that low. In any case, I wanted to see how I could reduce this max GC pause time.

Analysis with pprof

After finding an area to investigate, I ran a pprof analysis. pprof is a tool that’s baked into the Go language that allows for analysis and visualisation of profiling data collected from a running application. It’s a very helpful tool that collects data from a running Go application and is a great starting point for performance analysis. I’d recommend running pprof in production so you get a realistic sample of what your customers are doing.

When you run pprof you’ll get some files that focus on goroutines, CPU, memory usage and some other things according to your configuration. We’re going to focus on the heap file to dig into memory and GC stats. I like to view pprof in the browser because I find it easier to find actionable data points. You can do that with the below command.

go tool pprof -http=:8080 profile_name-heap.pb.gz

pprof has a CLI tool as well, but I prefer the browser option because I find it easier to navigate. My personal recommendation is to use the flame graph. I find that it’s the easiest visualiser to make sense of, so I use that view most of the time. The flame graph is a visual version of a function’s stack trace. The function at the top is the called function, and everything underneath it is called during the execution of that function. You can click on individual function calls to zoom in on them which changes the view. This lets you dig deeper into the execution of a specific function, which is really helpful. Note that the flame graph shows the functions that consume the most resources so some functions won’t be there. This makes it easier to figure out where the biggest bottlenecks are.

When you’re selecting the sample you want to view, these are your options:

• inuse_space: See memory size in use at time of profiling that hasn't been released.
• inuse_objects: See object count in use at time of profiling and that hasn't been released.
• alloc_space: See allocated memory size across the lifetime of the application.
• alloc_objects: See allocated object count across the lifetime of the application.

To analyze GC, we want to look at the alloc_objects option. This will help us understand how memory objects are allocated over the lifetime of the application. When analyzing the flame graph, there was one function that was responsible for a third of all memory allocation, which we’ll call fetchData. This function seemed like a great place to begin investigating.

When I’m investigating performance improvements I frame it as an experiment, and pprof gave me enough data for my first experiment.

Experiment 1

Objective: Reduce the amount of memory being allocated by fetchData.

Hypothesis: After reading through the code that calls fetchData, I don’t think we need every field that is requested. If we remove this from the request, the amount of allocated memory might be reduced.

Investigation: At SafetyCulture, inspections are a big part of how our customers use our product. Because of this, bringing inspections into other parts of the product is really valuable. fetchData gets inspections data from an API in JSON format. The response size varies, but the largest responses can be quite big, at around 20MB. Unmarshaling and parsing a 20MB JSON object uses up a lot of memory. This is correlated by the stack trace of fetchData mostly consisting of a large number of encoding/json functions. As mentioned in the hypothesis, it looks like a requested field is no longer used. It’s still referenced in the code, but in a path that looks like it will never be executed. Can we remove this field, and will it improve memory usage?

We now have a few questions to answer:

• Is the suspected code actually dead?
• Are we able to remove the suspect field from the request without breaking anything?
• Will removing this field from the request improve memory usage?

Method: Since this field looks unused, I wanted to find out why it was unused. I went back to the PR that first added this field and traced the history of the file. I found a merged PR from a few months ago (2 years after the initial commit!) that made this field redundant. I now felt confident that removing this field would be safe. I deleted the field from the request, deleted the code referring to it, did some testing and merged it.

I monitored the application in production and everything looked ok. It seemed that the part of my hypothesis about the unneeded field was correct. Now, we need to see if the amount of memory allocations is reduced.

Results:

• Memory utilisation had reduced. On the left, you can see memory use before the changes, and afterwards on the right. The constant blue line is what’s the Go runtime uses, and the coloured spikes are from our application. As you can see, the memory use on average is lower, and the spikes in memory usage are far smaller as well. Before, spikes in memory usage were about ~1018MB, whereas afterwards, the biggest spikes were about ~649MB. On average, memory use before was between 100MB — 250MB, whereas afterwards, it was between 80MB — 120MB. These changes caused less memory to be used and reduced the variance significantly.

• Allocated objects for fetchData had reduced. Before, 33% of allocated objects were going towards this function. After this had dropped to around 28%.
• The 99th percentile latency for the endpoint. The changes were deployed on Wednesday 26 (which isn’t displayed on the graph). The bigger spikes in the 99th percentile have dropped (except for the one weird spike on Thursday 27). The 99th percentile latency is now around 800ms, down from 1.2s
• The max GC pause time has dropped (a bit). It’s not a huge drop, but it’s still a drop. Since deploying on the 26th, the biggest spikes in max GC pause time have dropped, and the volume of large spikes have reduced as well.

Overall, I was really happy with these results! Memory usage was reduced, and the customers who experience the 99th percentile response time will now receive a better experience.

But, can we do more?

Experiment 2

The fetchData function is still responsible for 28% of all memory allocations. This is because the big JSON object is put into memory and unmarshaled into a Go struct. This takes us to the second experiment.

Objective: Reduce the amount of memory allocations from unmarshaling the JSON object.

Hypothesis: If the process of extracting data from JSON can be improved, the number of memory allocations might be reduced.

Investigation: I wanted to understand why this function uses so much memory. First, a POST request fetches the JSON and then the calling function receives the decoded body via json.NewDecoder(response.body).Decode(). Then, json.Unmarshal converts the JSON into a suitable struct.

This is fine for handling most JSON responses, but this JSON response can be massive (up to 20MB). This means that this is a bottleneck for this particular code path. Under the hood, json.Decode writes the entire JSON object to memory, where it is immediately unmarshaled. Because this JSON object can be so big, it’s no surprise that this function uses up so much memory.

Method: I was reading through the docs for the encoding/json package to see what other approaches were available. I came across the Token function that decodes the JSON object as a stream. This means the application can decode the JSON on the fly, instead of writing the entire JSON object to memory. The Token() function lets you manually step through the body of the JSON object to extract the data you need. This method has some tradeoffs. If the JSON response schema changes, your parsing code won’t work anymore. It’s also more verbose and complex when compared to json.Unmarshal. However, if this approach is used sparingly, it can lead to some performance improvements. There’s a number of examples available online, such as this post. As you can see, the code becomes much more complex than json.Unmarshal. If you take this approach, I’d recommend annotating the code so that your colleagues can understand it in the future.

Results:

• Memory utilisation has reduced even further: As before, left is the memory use before the changes were deployed, and afterwards is on the right. Experiment 1 resulted in memory usage around 80–120MB and this has now dropped to 70MB — 90MB. The big spikes are much less severe as well.

• The 99th percentile latency for the calling function has dropped even further. In the below screenshot, my changes went out just before 12PM in between Tue 22 and Wed 23, corresponding to the dip in the 99th percentile. Experiment 1 brought the 99th percentile down to around 800ms, but after Experiment 2 this has come dropped even further to 500ms.
• The max GC pause time has dropped. The code changes went live in the below screenshot at around 8:30 AM. After this time, the max GC pause time is lower for all garbage collections, and many of the GC pauses take around 100ms. This brings the max GC pause time from around 300ms-400ms to around 100–300ms. There are still some high spikes here, but the max GC pause time is much lower.

Overall Results

• Average memory usage has dropped from around 100MB-250MB to 70–90MB, and spikes in memory usage have dropped from around 1018MB to 120MB.
• Max GC pause time has dropped. When we first started investigating, the max GC pause time was around 400ms. It’s now around 300ms but is often much quicker. We’re now seeing GC pause time events of 100ms, which never happened before these experiments. There were also occasional spikes in GC pause time that would take a second, and these no longer appear.
• Most importantly, the endpoint using fetchData has had a big drop in 99th percentile response time. At the start, the 99th percentile was around 1.2s but now sits around 500ms. As far as I’m concerned, this is the most impactful change. It’s easy to lose sight of why performance improvements are important. If you focus on your customer and their experience of using your product, you’ll always head in the right direction.

I hope that after reading this you’ll have more confidence exploring your application’s performance. First, figure out which pieces of your application take up the most resources. Then you can start asking questions about why they use up these resources. You can then take measurable steps towards improvements and improving your customers’ experience.

Analyzing and improving memory usage in Go was originally published in Mitti Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Manual Memory Management

In a language like C the programmer will call a function such as malloc or calloc to write an object to memory. These functions return a pointer to the location of that object in heap memory. When this object is not needed anymore, the programmer calls the free function to use this chunk of memory again. This method of memory management is called explicit deallocation and is quite powerful. It gives the programmer greater control over the memory in use, which allows for some types of easier optimisation, particularly in low memory environments. However, it leads to two types of programming errors.

One, calling free prematurely which creates a dangling pointer. Dangling pointers are pointers that no longer point to valid objects in memory. This is bad because the program expects a defined value to live at a pointer. When this pointer is later accessed there’s no guarantee of what value exists at that location in memory. There may be nothing there, or some other value entirely. Two, failing to free memory at all. If the programmer forgets to free an object they may face a memory leak as memory fills up with more and more objects. This can lead to the program slowing down or crashing if it runs out of memory. Unpredictable bugs can be introduced into a program when memory has to be explicitly managed.

Automatic Memory Management

This is why languages like Go offer automatic dynamic memory management, or more simply, garbage collection. Languages with garbage collection offer benefits like:

- increased security

- better portability across operating systems

- less code to write

- runtime verification of code

- bounds checking of arrays

Garbage collection has a performance overhead, but it isn’t as much as is often assumed. The tradeoff is that a programmer can focus on the business logic of their program and ensure it is fit for purpose, instead of worrying about managing memory.

A running program will store objects in two memory locations, the heap and the stack. Garbage collection operates on the heap, not the stack. The stack is a LIFO data structure that stores function values. Calling another function from within a function pushes a new frame onto the stack, which will contain the values of that function and so on. When the called function returns, its stack frame is popped off the stack. You might be familiar with the stack from when you’re debugging a crashed program. Most language compilers will return a stack trace to aid in debugging, which displays the functions that have been called leading up to that point.

In contrast, the heap contains values that are referenced outside of a function. For example, statically defined constants at the start of a program, or more complex objects, like Go structs. When the programmer defines an object that gets placed on the heap, the needed amount of memory is allocated and a pointer to it is returned. The heap is a graph where objects are represented as nodes which are referred to in code or by other objects in the heap. As a program runs, the heap will continue to grow as objects are added unless the heap is cleaned up.

Garbage Collection in Go

Go prefers to allocate memory on the stack, so most memory allocations will end up there. This means that Go has a stack per goroutine and when possible Go will allocate variables to this stack. The Go compiler attempts to prove that a variable is not needed outside of the function by performing escape analysis to see if an object “escapes” the function. If the compiler can determine a variables lifetime, it will be allocated to a stack. However, if the variable’s lifetime is unclear it will be allocated on the heap. Generally if a Go program has a pointer to an object then that object is stored on the heap. Take a look at this sample code:

type myStruct struct {

  value int

}

var testStruct = myStruct{value: 0}

func addTwoNumbers(a int, b int) int {

  return a + b

}

func myFunction() {

  testVar1 := 123

  testVar2 := 456

  testStruct.value = addTwoNumbers(testVar1, testVar2)

}

func someOtherFunction() {

  // some other code

  myFunction()

  // some more code

}

For the purposes of this example, let’s imagine this is part of a running program because if this was the whole program the Go compiler would optimise this by allocating the variables into stacks. When the program runs:

1. testStruct is defined and placed on the heap in an available block of memory.

2. myFunction is executed and allocated a stack while the function is being executed. testVar1 and testVar2 are both stored on this stack.

3. When addTwoNumbers is called a new stack frame is pushed onto the stack with the two function arguments.

4. When addTwoNumbers finishes execution, it’s result is returned to myFunction and the stack frame for addTwoNumbers is popped off the stack as it’s no longer needed.

5. The pointer to testStruct is followed to the location on the heap containing it and the value field is updated.

6. myFunction exits and the stack created for it is cleaned up. The value for testStruct stays on the heap until garbage collection occurs.

testStruct is now on the heap and without analysis, the Go runtime doesn’t know if it’s still needed. To do this, Go relies on a garbage collector. Garbage collectors have two key parts, a mutator and a collector. The collector executes garbage collection logic and finds objects that should have their memory freed. The mutator executes application code and allocates new objects to the heap. It also updates existing objects on the heap as the program runs, which includes making some objects unreachable when they’re no longer needed.

The implementation of Go’s garbage collector

Go’s garbage collector is a non-generational concurrent, tri-color mark and sweep garbage collector. Let’s break these terms down.

The generational hypothesis assumes that short lived objects, like temporary variables, are reclaimed most often. Thus, a generational garbage collector focuses on recently allocated objects. However, as mentioned before, compiler optimisations allow the Go compiler to allocate objects with a known lifetime to the stack. This means fewer objects will be on the heap, so fewer objects will be garbage collected. This means that a generational garbage collector is not necessary in Go. So, Go uses a non-generational garbage collector. Concurrent means that the collector runs at the same time as mutator threads. Therefore, Go uses a non-generational concurrent garbage collector. Mark and sweep is the type of garbage collector and tri-color is the algorithm used to implement this

A mark and sweep garbage collector has two phases, unsurprisingly named mark and sweep. In the mark phase the collector traverses the heap and marks objects that are no longer needed. The follow-up sweep phase removes these objects. Mark and sweep is an indirect algorithm, as it marks live objects, and removes everything else.

Go implements this in a few steps:

Go has all goroutines reach a garbage collection safe point with a process called stop the world. This temporarily stops the program from running and turns a write barrier on to maintain data integrity on the heap. This allows for concurrency by allowing goroutines and the collector to run simultaneously.

Once all goroutines have the write barrier turned on, the Go runtime starts the world and has workers perform the garbage collection work.

Marking is implemented by using a tri-color algorithm. When marking begins, all objects are white except for the root objects which are grey. Roots are an object that all other heap objects come from, and are instantiated as part of running the program. The garbage collector begins marking by scanning stacks, globals and heap pointers to understand what is in use. When scanning a stack, the worker stops the goroutine and marks all found objects grey by traversing downwards from the roots. It then resumes the goroutine.

The grey objects are then enqueued to be turned black, which indicates that they’re still in use. Once all grey objects have been turned black, the collector will stop the world again and clean up all the white nodes that are no longer needed. The program can now continue running until it needs to clean up more memory again.

This process is initiated again once the program has allocated extra memory proportional to the memory in use. The `GOGC` environment variable determines this, and is set to 100 by default. The Go source code describes this as:

If GOGC=100 and we’re using 4M, we’ll GC again when we get to 8M (this mark is tracked in next_gc variable). This keeps the GC cost in linear proportion to the allocation cost. Adjusting GOGC just changes the linear constant (and also the amount of extra memory used).

Go’s garbage collector improves your efficiency by abstracting memory management into the runtime and is one part of what enables Go to be so performant. Go has built in tooling to allow you to optimise how garbage collection occurs in your program that you can investigate if you’re interested. For now, I hope you learnt a bit more about how garbage collection works and how it’s implemented in Go.

References

Garbage Collection in Go: Part 1

Getting to Go: The Journey of Go’s Garbage Collector

Go: How Does the Garbage Collector Mark the Memory?

Golang: Cost of using the heap

Golang FAQ

Google Groups discussion, comment by Ian Lance Taylor

Implementing memory management with Golang’s garbage collector

Memory Management Reference

Stack (abstract data type)

The Garbage Collection Handbook

Tracing garbage collection: Tri-color marking

An overview of memory management in Go was originally published in Mitti Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

After reading a couple of chapters, I decided to write a trivial search engine to understand what I was reading. You can find it here. It indexes three Shakespeare texts, and you can search through those 🎉. It’s a bit buggy (try searching for more than two terms 🤷🏻‍♂️), but I’m pretty happy with it for a week’s worth of evenings. My goals in building this were to:

• Read some documents
• Index these documents
• Figure out the term document frequencies
• Develop an inverted index of these documents
• Allow a user to search for multiple search terms and get back some results.

Let’s break some of this down:

The material that you can search for within a particular search engine is a document. So my search engine has Shakespeare plays, as Google allows you to search through webpages. Indexing a document involves saving this into a form that you can access later on.

After you’ve indexed a document, you’ll figure out the term document frequency of each word. This is a (long) list of how many times a particular term will appear in a document. This is accomplished by breaking down the text through tokenisation and stemming. Tokenisation is separating the words in a document into their individual words. Stemming is the process of bringing a word down to it’s “stem” word so that it can be compared to other similar words. For example, the words ‘rebellious’, ‘rebelling’, ‘rebels’, and ‘rebellion’ all come back to the stem word ‘rebel’. You’ll see this in the below examples where some words look a little bit weird. There many stemming algorithms, such as the Snowball and the Porter stemmer. As part of this project I did a port of the Porter algorithm, which you can find here if you’re interested. A stemmer basically drills down to the stem of a word through a series of more and more pointed regex trimmings. A snippet of this from Hamlet looks like this:

"mind": 16,

"mine": 33,

"miner": 1,

"minist": 2,

"ministr": 1,

"minut": 2,

"miracul": 1,

"mirror": 2,

"mirth": 2,

"mischanc": 2,

"mischief": 1,

"miss": 1,

After the term document frequencies for each document is calculated , the search engine will build an inverted index. This sounds a lot scarier than it is. An inverted index is a combined term document frequency for your documents, as well as a record of which documents contain those words. Below, the key “doc_freq” is the combined document frequency of these terms, and the “postings_list” is the list of document ID’s that these terms have appeared in. This inverted index looks something like this:

"deep": {

  "doc_freq": 13,

  "postings_list": [

    1,

    2,

    3

  ]

},

"deer": {

  "doc_freq": 8,

  "postings_list": [

    1,

    2,

    3

  ]

},

"defenc": {

  "doc_freq": 3,

  "postings_list": [

    1,

    2
  ]

},

After you’ve done this you can start searching. My search engine has a very basic search parser. When your user types in a query such as ‘Hamlet sword’, this gets converted to ‘Hamlet AND sword’ so both of these terms are searched for. The upgrade to this is a Boolean search parser. This allows your user to use some Boolean logic in their query, such as: ‘Hamlet AND sword OR cat AND been’. This can be used with a phrase query, which is many words surrounded by double quotes, which will match everything inside those quotes. With a Boolean query, this might look like: ‘Hamlet AND sword OR cat AND “who was that”’.

At this point, a user will see some results come back and our job. Reading this information was interesting, but confusing. Implementing it, even simplistically, has been a great way of understanding how this works.

There’s many ways that this could be improved. For example, better performance, indexing more documents, and implementing a ranked search system. Feel free to check out the code for the porter here, the search engine here, or view the final product here.

Thanks for reading!

On Saturday 7 May 2018, Jesica Saussay, Erin Campbell and I ran a workshop at NodeGirls Sydney teaching React to a classroom full of aspiring programmers. (Shoutout to Jes who did most of the work in coming up with the app to build, and how we’ll teach it. 📣)

NodeGirls is a non-profit that runs Javascript and Node workshops for women. This is an awesome initiative to improve diversity in software development roles, and being able to help out with teaching is really great. 🚀

The two other workshops, Intro to Javascript, and an Express/Node were self-guided where you work through the course at your own pace and ask for help when you need it. Conversely, we ran our React workshop as a code-along, building an application at the same time as our students did. This was a really fun style of learning, and it felt like our classroom were really involved with our workshop and with the learning of those around them. Not to mention Jes was making everyone answer questions every 20 seconds. ⏰

The main goal 🥅 of our workshop was to teach React, and make sure that everyone walked away with a working app. The attendees felt like they had accomplished something and can further iterate on what they had built in their own time if they feel like it. The app we built together was a weather app, which is a really good example of React because of how dynamic it is. And if you’re interested, you can check it out here:

https://github.com/node-girls-australia/react-workshop

🌞🌦️🌀

Quick side note about how skills develop. When I used to work in Apple Retail 🍏 I ran heaps of workshops teaching people how to use their computers, and really worked on developing patience with people when they didn’t understand things. Then when I was at uni I ended up performing in a number of theatre shows (musicals, plays, revues etc) 💃 and doing this helped me get skills in how to project my voice, have confidence in front of a room full of people and probably other things I’m not aware of. When I left uni and part time retail and moved into full time development, I didn’t think that the skills I’d developed as a student would be useful again. After we ran this workshop and I’d had a bit of time to reflect on it, I realised that the skills I’d worked on had helped me in ways that I hadn’t really thought about. Anyway, never dismiss the things you’ve learned in the past, you never know when you’ll be able to utilise them.

Now, for some photos: 📷

Teaching React at NodeGirls was originally published in Tech @ Domain on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of their biggest revenue drivers was sending out EDMs about sales. These always caused revenues to spike, and everyone to feel great that we were making money. FYI, An EDM is a promotional email companies send out for marketing purposes.

Being the junior developer, I had to build these EDMs, and it was my least favourite part of the job.

For anyone who hasn’t had to build HTML emails, it’s a painful process. First, imagine a marketing email you’ve received lately. A lot of these are big poster-like images that get divided into pieces; buttons, photos, headers etc . This image gets sliced in Photoshop, exported as HTML and then fleshed out so it will work everywhere.

This is difficult because HTML support in email clients is very outdated. You have to use antiquated standards to make everything work, like wrapping everything in millions of <table> tags. On top of this, different email clients implement these standards in their own way. Painful, right?

Like any good engineer, I sought to automate this process. We built a Grunt tool that reduced the time it took to build a simple EDM from ~20 minutes, to 2 minutes. This also enabled us to build complicated emails much easier.

But, like any good lazy person, my real goal was to make a tool that would mean someone else could do this job that I hated. This would have involved making a technical task accessible to non-technical people.

At the beginning of my time at this job (and at the beginning of my career), I tried to make something like this, but didn’t get very far. Last year I gave it another shot. I got a bit further, but never completed it. Although I never completed it, there were a few cool things about this project. For example, custom login and user systems, and the ability to send out emails to reset passwords.

The reason I never got to finish was that I didn’t prioritise well. EDM creation relies on image slicing, which I couldn’t get to completely work. For an app that boils down to image slicing, this is a dealbreaker.

Because I found this hard, I kept on procrastinating fixing this. This lead to building other features, but got me away from the core of the application. This was pretty dumb, because all the stuff I built would never get used until I fixed the image slicing.

One of the most important lessons that I got out of this process was to build the core features first. If I had finished the image slicing, I would have gotten this app out, or realised sooner that I wouldn’t finish it.

When I was studying Product Design at uni, one of the things that I learnt was to make prototypes. Lots and lots of prototypes. By making prototypes you’re able to figure out what works and what doesn’t. You then get the privilege of using these lessons to inform your decisions going forward.

I should have narrowed my focus to make a prototype to “slice images and convert them into HTML”. Instead I created an unfocused prototype where users could do everything but that.

It’s important to prioritise the most important things for your user and do those things first. Nobody would care that I had all these other systems built out if the core purpose of my app sucked.

My users would have come back to create emails, and creating emails should have been the thing that I focused on making awesome.

When I was building this, I focused on getting this out the door. This means the code sucks. Sorry about that. There aren’t any tests either, which all leads to a lot of mess across the whole repo. I planned on coming back to develop this at some point, but the above meant I didn’t bother. I’m not sure where the line between code quality and a speedy release is, but I should have considered it some more.

This app isn’t pretty in a lot of places, but I’m still proud of it. It represents a lot of hard work and ambition that would have been great to finish. At least getting it out into the world is better than letting it languish as a private repo.

Check out the code below. It’s not great, but I definitely learned a lot from writing it. I promise my ability to pull together an application has improved since 😅.

I never would have learned all this without building something. I definitely encourage you to go and build something yourself; it’s the best way to learn anything.

https://github.com/Gangezilla/edm-release