Part III: Setting Tech Team Objectives and Deliverables from a Vice President of Technology

This is the third and final instalment of the SSENSE-TECH series on the what and why behind performance metrics. Read Part I on squad metrics and Part II’s dive into technical metrics, for more.

Many people know the famous quote from the management guru, Peter Drucker: “What gets measured gets managed.” Several organizations have adopted this mantra to measure as many indicators as they can.

This quote was actually misattributed to Drucker, according to the Drucker Institute itself, and confirmed here as well. The full quote dates back to 1956, from an academic named V.F. Ridgway: “What gets measured gets managed — even when it’s pointless to measure and manage it, and even if it harms the purpose of the organization to do so.”

As we set our goals with the Tech leadership team for the department, we intentionally avoided measurement for the sake of measurement. Metrics need to be clearly linked towards our company objectives. We should measure only what matters, recognizing that many things done by knowledge workers such as developers, product managers, etc., are not necessarily simple to measure but are nonetheless essential (e.g. impact of a developer investing in coaching a more junior team member).

At a department level, Technology tracks five main objectives:

1. Ensure the reliability of our services and the quality of our code
2. Deliver the technical foundation to support the internationalization of our operations
3. Increase conversion on the website by X%
4. Grow the Mobile app share of revenue to X%
5. Retain, develop, and attract an engaged and high performing team

These objectives are directly derived from our company objectives, and you can notice that aside from the first one, their outcome is not necessarily technology-focused. These objectives speak to what SSENSE wants to achieve by investing in technology. With that as the foundation of the objectives, let’s now examine the more technology-focused goal while briefly touching upon the others.

Ensure the reliability of our services and the quality of our code

To best serve our customers, the reliability of our services is key.

I initially thought about setting a target on the maximum number of incidents we can have in a year. The fewer incidents we have, the more reliable our systems are. It is a measure that would be easy to understand and communicate to an executive audience. But when you think about it, by setting such a target, you run the risk of people trying to hide incidents as we come close or go over the target. We could end up in long and worthless debates about whether an issue is an incident or not. To illustrate this, if we created a bug that caused our packing station to run slower for 15 minutes before we fixed it, and given no customer would notice, would that qualify as an incident? How about a product not appearing on the website because an optional field was not properly completed when we created the product? Should technology have anticipated such a case?

We ended up dropping the number of incidents as a metric and requested that each incident come with an incident report, to learn from them instead of hiding them.

Next, we considered setting a maximum for allowed rollbacks. However, this would create a similar risk. Teams could be tempted to fix forward, which takes longer and prolongs a negative customer experience. In prioritizing team efficiencies and the customer journey, we ultimately found setting a maximum to be counterintuitive.

Measurability as a tool to drive organizational alignment towards quality

So what do we measure instead? If you have read Part II of this series, you will know that we put significant effort into measuring the yield of each service to evaluate reliability. The SSENSE Tech leadership team agreed to setting a yield objective for each service and for the critical aspects of the purchase funnel. I find yield objectives to be useful for many reasons:

• It is simple to communicate the relevancy of this metric to my executive peers.
• It drives higher accountability. I had my eye on the yield of the critical services of our purchase flow and regularly sent questions to the team when it observably deteriorated.
• It is actionable and unforgiving. It created visibility on defects that existed without us knowing. It drove investigation and code changes such as tweaks to our retry policy, or keepalive settings.
• It can be shared between Engineering and Product. While the yield can be seen as a technical metric initially, it is also in fact a product-level metric that both the developers and product managers should be incentivized on. While teams can deliver new customer-facing features, if the service powering these experiences is down frequently, the impact will not fully materialize. To ensure quality is a shared incentive, some organizations do not allow feature deployment until the yield is above its objective for a certain number of days. If developers can’t deploy their code, it will impact the team’s ability to deliver their roadmap. It ensures that technical sustainability is seriously taken into account and debated when establishing roadmaps.

While setting the exact target for yield, it is important to guide your executive team. The target needs to be high yet achievable, otherwise it will be counterproductive and not taken seriously.

For example, targets such as 99.999% yield could be set aspirationally without understanding the complexity it represents to reach such high availability, and the costs associated with such a decision — such as major architecture redesign of key components, multi-AZ strategy, drastic slow down of team velocity, etc.

While this objective is kept simple yet powerful for executive communication, there are many detailed sub-objectives that Engineering Directors are held accountable to. For example, implementing planned and unplanned degraded experiences, implementing circuit breakers, ensuring proper classification of errors, increasing availability zones within AWS, etc.

Deliver technical foundation to support the internationalization of our operations

The challenge in this objective is that no software is live, and we can’t focus on the business impact of the Tech team’s work prior to its delivery. Most of our tech efforts to enable our international growth fall under this objective. To adequately measure our performance, we decided to set objectives based on meeting specific deadlines for product launches, while meeting the business requirements and staying within budget. This approach is more waterfall than I would like, but we did not find a better way to measure the success of these kinds of projects. ‘Launching a new ERP system by August 31’ is an example of such a project we had to achieve. Once these technical components are live, we then can transition to metrics like yield and business outcomes.

Increase conversion by X%

For website features, we originally had a double digit conversion increase objective. However, we rapidly moved to a revenue per session objective, as features can decrease purchase propensity while having an outsized impact on the financial productivity of a user’s session (e.g. personalization, search result relevance, or app performance optimization).

We rigorously track our target of percentage increase of revenue per session by A/B testing all the features we launch. Business stakeholders contribute to experiment design, to ensure we have consensus on the assumptions made about the customer journey. Once experiments conclude, we then validate the impact with the Finance team.

Grow Mobile app share of revenue to X%

Having launched the SSENSE iOS app in October of 2019, we set a specific objective for Mobile to reach a certain share of total company revenue. We first contemplated whether we had the confidence in the app to actively shift traffic volume to the channel. Ultimately, this objective drives interesting discussions and is ensuring alignment with our Marketing and Advertising teams.

Retain, develop, and attract an engaged and high performing team

None of the above would be delivered without a dedicated and high performing Tech team. So we naturally have an objective around our talent. We track our engagement score, our employee Net Promoter Score, and our attrition.

For the first two, we use a software that sends regular anonymous survey questions to everyone in the company on a regular basis. With the help of our HR Business Partner (HRBP), I regularly review these results for the department, as well as aggregate findings from exit interviews. It allows us to detect any issues and react quickly. It also provides invaluable insights to influence our decisions. Our HRBP also uses these metrics and surveys to prepare monthly HR reviews with all managers in the department to address the most important topics brought forward by our employees.

CONCLUSION

These 5 objectives are converted into a scorecard that I am accountable for with the CEO and with the other members of the executive team. It is high-level enough to be understood well, communicated, and remembered.

To make these objectives meaningful, it is important to keep them fresh in our team’s mind. Every 6 weeks we have a Tech all-hands and I report transparently on each metric to the entire department. Every quarter we have a Quarterly Tech Review with the executive team where I give a detailed update on each of them.

Finally, within the tech department, these 5 elements are then broken down into 100+ smaller objectives that are distributed across all directors in the department and trickled down to each team. This ensures a clear alignment with the Tech leadership team.

Editorial reviews by Deanna Chow, Liela Touré, & Gregory Belhumeur.

The “What” and “Why” Behind Performance Metrics Part III: Setting Tech Team Objectives and… was originally published in SSENSE-TECH on Medium, where people are continuing the conversation by highlighting and responding to this story.

When I first joined SSENSE in January 2017 as a Director of Application Development, I started by conducting roughly 20 in-depth discussions to get a sense of the performance and health of the Engineering department. At the time, we were 45 engineers; we have since grown to 100.

In every single discussion we had, engineers mentioned career tracks as a source of uncertainty. They raised questions that will sound familiar to many readers working in relatively young (SSENSE was founded in 2003), fast-paced, and high growth companies:

“What is expected of me? Under what criteria will my performance be evaluated?”

“What can I do to get promoted?”

“Why was person X promoted?”

“Do I need to become a manager to progress in my career?”

When the department is still small, you can manage careers without a clearly defined ladder. In fact, if your employees are willing to accept some inconsistent titles, the system can be sustained for surprisingly long periods of time. In our case, we had ‘Frontend Developers’, ‘Software Developers’, and ‘Web Developers’ — all of whom were performing similar, if not identical tasks.

Given our rate of growth and the size of the department at the end of 2017, we had reached a point where these titles became disconnected from the reality of our day to day and our actual career paths were no longer satisfactory. This led not only to inconsistencies in titles, but shortcomings such as:

• Career progression was not being recognized often enough due to the ladder being only four titles deep (Junior, Intermediate, Senior, Tech Lead).
• The ‘Tech Lead’ title became a bottleneck for technical progression. There are only as many Tech Leads as we have squads. This imposed an artificial limitation that prevented us from promoting deserving engineers.
• There was no way to progress in a technical capacity beyond the ‘Tech Lead’ position, leaving engineers with no option but to move into management if they wanted to continue growing at SSENSE.
• Outdated job descriptions made expectations for each title unclear.

It was time to revamp the career path for our engineers. Changing career tracks in a tech organization is significant. It impacts how you hire people, how you evaluate their performance, and how people progress in their career. It can drive up the rate of attrition if the ladder is not clear or motivating. Moreover, career tracks and associated job descriptions are an opportunity to influence the culture you want to put in place.

So how do you define the right titles? Following is the approach we took to defining a more robust and clear career progression at SSENSE for individual contributors in Engineering who wanted to progress without entering management.

Approach

We chose a task force approach to define the best solution and facilitate its adoption. We gathered more than 10 engineers (developers, QA, DevOps) interested in dedicating some time to this topic.

Over the course of a few months, we conducted roughly eight workshops. While the task force approach has many benefits, it comes with challenges such as receiving suggestions for individual circumstances and not the department, low engagement during the session, etc. Three factors successfully drove the group forward:

• We began every workshop with a proposal for the task force to reflect upon. This is a mix between a top-down and a bottom-up approach. I found starting with a blank board to be extremely time consuming as the discussions tended to wander in many directions.
• I would list all topical questions for the group to think on, after which the group would agree on and document the topics we considered to be within scope. For example, the group raised questions about compensation and its correlation to titles. We wrote down all questions and sent them to our HR department as this topic was out of the group’s scope.
• We agreed on six design principles to guide our decisions. Our first two workshops were held without these principles, which led to us re-visiting the same topics and challenges multiple times.

For a few sessions, we even invited our Chief Operating Officer to join our discussions and challenge us. This helped make the importance of this initiative clear to all participants.

In parallel, we worked with our HR team to understand internal requirements to ensure we remained consistent across all SSENSE departments. Finally, we benchmarked a few tech companies to identify best practices (see the source section for some good reads).

Result

We confirmed three possible career paths: Individual Contributor, Manager, and Subject Matter Expert. We decided to first tackle the Individual Contributor career paths as this represented the majority of our headcount.

Our third design principle helped us structure the progression around the breadth of impact expected at any given level of seniority — ranging from recent graduates whose focus would be on ramping up as individuals, to our most senior developers who would be expected to lead initiatives that impact the department, or the company as a whole.

When combining the two previous charts the result is the ladder structure below:

As you might notice from the diagram, we decided to change the ‘Tech Lead’ title to a role that can be taken at the senior and staff level. It avoids creating an unnecessary bottleneck in our ladder and gives our developers and QA testers the opportunity to try this role more easily. However, it also creates its own set of challenges such as finding a strategy to compensate people taking on this role.

Implementation

After finalizing the ladder, we worked with the department’s managers to update all corresponding job descriptions. Once completed, it was time to roll out the new career track. While the new track is imposed on everyone in the Engineering department, ensuring smooth adoption and support remains critical to keeping motivation high and attrition low.

Providing as much visibility as possible during the process is imperative to the implementation. We regularly communicated updates in our department’s ‘All-Hands’ meetings, and the final ladder and job descriptions were shared months before its implementation. Additionally, town halls were held to present and explain the new career track in more detail, answer any questions, and distribute job descriptions. The fact that more than ten people were involved in making the ladder helped with its positive reception.

The last and most sensitive step was to map current titles to new ones. We did not always find a one to one relationship, causing some employees who shared the same title previously, to end up with different titles following the implementation. We used our regular performance evaluation cycle to review this mapping with employees. As part of this cycle, all managers, directors, as well as our Chief Technology Officer gathered for calibration meetings for two days to review the individual performance of all employees in the department. We also evaluated their performance in relation to their previous job descriptions to ensure fairness. We can’t hold employees accountable for expectations that were not set before the start of the quarter. However, we also mapped each person to the new title and gave feedback based on their new job descriptions to highlight what they should work on for the next evaluation cycle in six months.

Conclusion

The SSENSE Engineering department has now successfully integrated the new job titles and ladder for the past nine months. Using a formal career ladder with robust job descriptions has yielded tangible benefits. This framework helps our managers set clear expectations for the engineers they manage, give more specific and fact-based feedback, and guide them in preparing for calibration meetings. In a growing organization with more than 100 engineers and several managers, it also ensures a standardized and fair approach to evaluation. As for engineers, it helps them better understand what is expected of them at their current level and motivates them to grow towards the next one.

One risk we need to mitigate is that a job’s description might be misconstrued as an exhaustive checklist. This could lead to situations wherein employees are left wondering why they did not get promoted despite checking all the boxes. It is management’s responsibility to clearly communicate that the decision to promote accounts for factors that cannot be objectively defined as a checklist, such as attitude towards collaboration, accountability etc. Another trap to avoid is directly copying and pasting the position description over to the job posting. Instead, pull the key tasks and responsibilities you’ve defined, communicate them in an engaging manner, and showcase your company’s culture and history.

If you are considering restructuring career tracks in your organization, I advise against duplicating another company’s ladder and/or job descriptions. Career tracks can deeply influence your company’s culture and they must be uniquely adjusted to the existing ecosystem. I recommend researching the subject thoroughly (see ‘Further Readings’ as a starting point), then taking a step back to organize your thoughts and identify the mindset, behaviors and skills your organization needs in order to be successful. Your career tracks should align with these needs. Finally, the implementation needs to be very carefully planned. Poor execution can create frustration, undermine your culture, and ultimately lead to unwanted attrition. If implemented well, your career tracks will prove to be a critical asset in structuring and growing your company.

Further Readings

• Building a technical career path at Spotify
• What does a director of engineering do?
• I am a technical lead, but am I a manager?
• Sharing our engineering ladder
• What truly makes a Senior Developer
• How Individuals Advance at Buffer, Without Becoming Managers

Editorial reviews by Deanna Chow, Liela Touré & Prateek Sanyal

Want to work with us? Click here to see all open positions at SSENSE!

Revamping the Engineering Career Track at SSENSE was originally published in SSENSE-TECH on Medium, where people are continuing the conversation by highlighting and responding to this story.