If you’ve been in any professional development setting in the last 30 years, you’ve probably heard of SMART goals. They were first pitched by George Doran in 1981 in an article titled, “There’s a S.M.A.R.T. way to write management’s goals and objectives”, which was focused on helping businesses give clear objectives to the managers of individual teams.

When used in the right context, SMART goals are a great framework for understandable, reasonable goal setting. However, since its introduction, the SMART goal acronym has been co-opted and poorly copy-pasted into the personal development space, where it is not helpful, leading to much headache and heartache for all involved.

In this post, I’ll talk through why SMART goals fail in this domain, and offer an alternative that works better.

Where SMART Goals Fail

In general, SMART goals start to become less useful the more abstract an objective is, because it becomes harder to fit each of the letters — Specific, Measurable, Achievable, Relevant, Time-bound — into this abstract idea. The place I see this commonly is in company-mandated Personal or Team Goal Setting.

Here’s a representative story, which I’m sure many of you will relate to (I certainly have):

• Your company requires you to enter personal goals into your HR system every year
• You come up with a goal related to your job — let’s say something like “Get better at public speaking”.
• You plugin the SMART formula, and immediately hit some issues…

This goal is definitely “Relevant” to your job, and it’s probably “Achievable”, but “Measurable”? You suppose you could measure public-speaking prowess by handing out surveys to your team members after you give a presentation — but that’s not very practical. Also, you would need to define some KPI, then set a target to make this “Specific”. And we haven’t even started with “Time-Bound”…

Now, you have 2 choices: awkwardly make up numbers to fulfill the SMART criteria — or change your goal to make one that is SMART, but not really the goal you had in mind. After a ton of back-and-forth, you might end up wiht a truly silly personal development goal like:

“Give 4 brown-bag lunch presentations to an audience beyond my team in 2027, to increase my public speaking abilities”

This goal is specific, measurable, achievable, relevant, and time-bound. But it is not motivating or inspiring, and it is so specific that is misses the point completely.

The most likely outcome of this goal is that you forget about it completely until the next performance review cycle. Or you just go through the motions and give 4 talks just to say you “met the goal”, like it’s another item on a checklist. Unfortunately, this story is not just made-up, I’ve seen it play out dozens of times in my career.

A Better Alternative: Goal + Signals

Here’s the system that I use for myself, and for all the teams that I manage. This system takes what works from SMART goals, and borrows a bit from the “Objective and Key Results” framework often used in businesses.

First, define your abstract goal in a sentence or two, making sure to hit at least “Specific” and “Relevant”, and maybe “Achievable” if you can. It’s best to focus on a desired outcome you want to achieve, not just a statement of “be better at X”.

I want to get better at public speaking to non-technical audiences. I’d like to be able to consistently present engineering materials to non-engineers and be understood.

Now, we need a way to track progress on this goal. Define 3–4 relevant signals which you will use as indicators for progress. Signals should be things that you can measure or observe, which either tell you that you’re doing well or poorly. They are not the goal itself — they are proxy metrics for the true goal. Some example signals for this public-speaking example could be:

signal: Number of positive audience interactions during my presentations — productive discussion, “thumbs ups”, etc.

signal: Number of clarifying questions audiences ask during my presentations

signal: Personal evaluation of “audience engagement” (1–5)

signal: Time spent preparing for each presentation (less is better)

Then, set regular checkpoints with yourself (and your manager if possible) to do a holistic evaluation of your progress. Review the goal, discuss each of the signals, then honestly decide for yourself where you are, and what you can do next to keep the goal progressing. Make a write-up like below:

Goal: I want to get better at public speaking to non-technical audiences. I’d like to be able to consistently present engineering materials to non-engineers and be understood.

Signal: audience interactions — started okay, but got slightly better, especially in my last presentation on Thursday

Signal: clarifying questions — most presentations recently have 1–2 clarifying questions, usually someone asking for the context behind a decision

Signal: self-assessed audience engagement — my own notes show some presentations in the 2/5, but also a few in 4/5.

Signal: time spent — I spend on average 2 days prepping for a big presentation, which is much better than it used to be.

Evaluation: I’m definitely getting better at making technical presentations, especially in the time I spend — but consistency and impact is still not there. I have trouble anticipating what people will ask, and frequently leave out some business context that makes the audience care about what I’m speaking about.

Next Steps: In future presentations, focus more on background, context, and business impact — instead of technical details.

Why this works better

This system has worked much better for me for several reasons:

• It uses several measurable / observable signals holistically, instead of relying on a single arbitrarily-chosen metric.
• It comfortably accommodates abstract goals common in personal development objectives.
• It recognizing continually-improving goals, focusing more on “where are we now” and “how can we get better”, instead of a single binary “you hit the goal” vs “you didn’t hit the goal”.

If you’re in performance-review season with your team or manager, I’d consider giving this system a shot. It’s been a game-changer for me, turning yearly goals from an HR chore into something that is truly helpful and motivating for people. Thanks for reading!

Intro

This article is my standard playbook for building and designing a process for business teams at work. For context, I work in Software Engineering at a medium-sized Manufacturing company (3k people), so there are plenty of processes we use every day. But of course, new processes are being created and adjusted every day, and it’s nice to have some playbook for process design.

Steps

The overall steps for designing any business process are as follows:

1. Define the goal of the process
2. Work backwards to define tasks
3. Order your tasks and assign owners
4. Define formal interfaces between steps
5. Choose a driving mechanism
6. Document

Step 1 — Define the requirements and context of the process

The first step might seem obvious, but it is important to not skip it. The whole reason you’re reading this is because there is some issue within your company that needs to be solved. Usually, processes are designed to solve one of 2 problems:

• Some important task is not getting executed
• Some important task is getting executed inconsistently

And when any of these 2 things happen, someone always suggests, “Let’s just create a process for this, so that this important task will never get missed / messed-up again”. Then they assign it to you and you find this article.

So before you actually define the process, define some of the issues your company is seeing today. Then, define what end-results the process must deliver for the company. This can take the form of a requirements list.

Example:
The IT team at Company XYZ is having issues with inconsistent resolution of requests. Some of our team members are good about addressing customer issues, but not all. And we’re seeing that ocassionaly requests just get forgotten, and customers have to follow up themselves to get movement on them. We need a process that:

• Makes sure no IT request ever gets forgotten
• Makes sure that no IT request will take longer than 7 days
• Over time, builds up internal knowledge so that common issues get resolved quicker

Step 2 — Work backwards to define tasks

Now that you know what the end-result of your process should be, work backwards from that to define the steps that must happen to achieve that result. Keep your steps high-level at this point, so you don’t get lost in the details.

Continuing the example from earlier:

• If we want to make sure no IT service requests are missed, then we’d need each request to be logged in a centralized ticketing system.
• If we want to make sure no IT service requests last longer than 7 days, then we’d need a regular review of all tickets to identify any that are close to that point.
• If we want to build up internal knowledge over time, then we’d need a searchable knowledge base for our IT staff.
• If we want this knowledge base to grow over time, then we’d need a way to identify common issues, and write standard resolution steps for those issues.

Step 3 — Order your tasks, and assign owners

You now have a list of tasks that must be met, so you can start to build the workflow, the sequence of steps from beginning to end. Once you have this, clearly label who is responsible for each step in the process.

Example:

1. customer: sends a request to an IT group mailbox
2. automated system: reads the email and creates a ticket in the IT team’s request backlog
3. IT manager: reviews the request and assigns it to the most appropriate team member
4. IT team member: works with the customer to resolve the issue, using the knowledge-base as appropriate
5. IT team member: documents how they resolved the issue, and closes the ticket
6. IT manager: reviews status of all tickets daily to see if any are at risk of breaking the 7 day SLA.

Step 4 — Define formal deliverables, artifacts

You may think you have a process now, but you don’t. All you have is the idea of a process. In order to get a process that is delivered consistently, you need to clearly define the deliverable at each step — especially if there are different personas between them. These artifacts serve as both proof that the step was completed, and as a goal for people to work towards.

Here are the deliverables / artifacts we’d expect in our ITSM process:

1. After a customer request, we have an email
2. After automation kicks in, we have a ticket in our ticketing system
3. After assigning the ticket to someone, the ticket has an “assignee”
4. After resolution, the ticket is marked as “closed”, and resolution notes are left in the ticket comments

By clearly defining the deliverable at the end of each step, we make it obvious when a step has been completed. For any given work item going through the process, you can look at these deliverables and know immediately which steps have been done, and which step is next.

Step 5 — Choose a driving mechanism

Every process needs a driving force that ensures it moves forward. I have seen too many processes that are well designed on paper, but lack a driver, and therefore die because nothing is forcing the process to move.

Some examples of common driving mechanisms include:

• A regular meeting where all work items are reviewed — like a standup or project status meeting
• Automated email notifications tied to a ticketing system. Especially common in ITSM, Customer-Service, or approval processes, where an assigned team member will get a daily reminder when something is waiting for them.
• A project manager, or process owner. Sometimes there’s someone in the business who has enough stake in this process to drive it themselves. They may elect not to hold meetings, but instead personally drive each work item forward because of their relationship with the process or team itself.
• Customer-driven process. In some cases, if the process itself provides some incentive to the customers, and it is well-documented, then they can drive it themselves. An example here would be filing taxes. Everyone in the US is forced to learn and follow the complicated tax processes, because of the incredible incentive at the end (not going to jail).

Step 6 — Document

Now you have a process, but if you expect everyone to follow it consistently, you need to document everything. At a minimum, you should provide these documents to all involved parties:

• An overview of the whole process, including: steps, owners, and desired outcome (for leadership and stakeholders)
• Guidelines for team members working the process, including the expected outputs / artifacts at each step, and the driving mechanism that will be used.
• Standard Operating Procedures for any steps that need both consistency and scale

Follow Up Topics

This article truly is just “101” of process engineering. It gets you in the door for designing processes, but if you find this interesting, here are some other topics you might want to look at (and if there’s interest, I can write articles on them too).

• Change Management 101
• Internal marketing within a large enterprise
• Avoiding bottlenecks in a process using Lean principles
• Standard processes of work management in engineering teams (Scrum, Kanban, Waterfall, etc.)

Introduction

If you’re an engineering pod lead, manager, or even a senior engineer, you’ve probably dealt with a situation like this. Something about your team’s structure, environment, or processes is wrong, but you can’t pin it down exactly. You hear complaints frequently from your own engineers, and maybe even from other engineers on adjacent teams. Occasionally, someone will offer a solution too, but often they are narrow solutions, just band-aids, not cures.

How do you, as an Engineering manager, sift through all the noise, and identify the core, underlying issue? This article is a breakdown of how I’ve done this in Engineering teams I’ve managed, and hopefully it will be useful to you as well.

Step 1: Collect Datapoints

The first step of the diagnosis process is to collect as much information as you can. Create a single document which lists all of the individual complaints or issues that you’ve heard. This can be a simple markdown or word doc, what matters is that you capture everything.

Here’s an example, taken from my day-job as an Engineering manager of 20+ engineers. All the names are changed, and details about the specific issues have been altered, but the core of the idea is the same:

At this phase, you’re just consolidating all your information into 1 place, so you can think clearly about it. I find it’s best to collect these issues from 1:1’s with Engineers, Incident reports, Team group-chats, and from your own direct observation. You can also ask Managers of adjacent teams to your own, because many of these issues may manifest to them as well.

Step 2: Find common patterns, summarize and condense

Unless you have uniquely introspective engineers, the issues that get reported to you are likely not true systemic issues. They are just symptoms of the underlying problem. Further complicating the task, people are messy, and management is messy. Some issues are just isolated instances, not related to any structural gap in the team at all.

However, there is a simple way to separate structural issues from isolated incidents.

• If, when you review your list of complaints, you see a common thread across many people, across many processes, across time. There is likely a structural issue in your team.
• If a specific issue has only appeared once or twice, or is isolated to only a few individuals on the team — it is likely an isolated issue with those processes or persons, and not a structural issue on the team.

But how do you narrow down the specific structural issue? Here, your best tool is to brainstorm and test. Use your engineering brain (which probably got you into management in the first place) and try to come up with a few possible descriptions of the issue. The closer your description gets to the real issue, the more it will resonate. Here are a few heuristics to help with this process:

• Engineered systems are complex, but engineering teams are simple. Structural issues in engineering teams are almost always simple and easy to articulate.
• A clear articulation of your structural issue will explain a great number of the complaints you collected before.
• When you articulate the structural issue, it will resonate with your engineering team. Many times, these are “I know it when I see it” situations.

In the example above (which again, is a real example from one of my teams), the issue ended up being this:

Notice how simple it is? However, despite its simplicity, these 2 sentences resonated with almost everyone on the team, and they also explained many of the complaints we were seeing:

• Senior members of the team (who often handle ad-hoc requests) were getting burnt out because of the sheer volume of requests, and our inability to distribute them in an organized way.
• Because we had no way to track these requests, some of them just never got completed.
• Because these requests are all ad-hoc work, they were not planned for in the sprint, leading to our mystery “ghost” story-points in our Scrum board.

Once we gave this structural issue a short description, and a name, we could start talking about it, and making plans to solve it. But how do we do that?

Step 3: Imagine a perfect world

First, I do a simple thought experiment:

Imagine a perfect engineering team that doesn’t experience the issues your team sees, but exists in the same environment. Let this be your “End goal”, or your dream state.

Once you have this “perfect” team in your head, start to ask yourself, what processes or structures could make this perfect team possible? For each point, brainstorm some ways that this imagined engineering team could achieve this end goal.

You can also use concrete examples to help imagine this perfect world. Many high-functioning engineering teams love to publish their inner mechanization to the world (looking at you Netflix Engineering Blog), and many of them have already solved the problems your team is likely facing.

Regardless, working backwards from this dream-state team is your best bet to finding a solution to the problems on your team. By continually asking yourself “How can we get there?” you will eventually land on some next-steps that are actually achievable for your Engineering team.

Step 4: Tell a story

This is the last step, but also the most important. To get any change to happen in your organization, even one that you control, you will have to get everyone on the same page. Luckily, you have already written the story you need to tell, in steps 1,2, and 3.

Lets piece the examples in this article into a single story:

1. The symptoms: Many of our team members are experiencing burnout, and we’re dropping the ball with our support of other engineering teams.
2. The diagnosis: This is all because we don’t have a unified support process. Instead of support requests coming through via a clear, organized ticketing system, they’re coming to us through emails, DMs, and hallway conversations. Of course we can’t keep up, it’s chaos.
3. The cure: To solve this, we’re going to create a simple “Service Desk” form, with a backing Jira board. This will consolidate all service requests into a single place, and let us tackle them in a systematic manner, not randomly.
4. The dream state: When we’ve implemented this, all our support requests will be tracked visible in a single place, we’ll be able to take shifts on working support requests, instead of interrupting all of our engineers, and lastly, we’ll be able to negotiate better with other teams to see why they’re making so many requests in the first place. None of you will have to spend more than 25% of your time working these.

Conclusion

Hope this was useful to you! Like, subscribe, click the linky in the description, all that jazz. If you like this content, follow me on medium (https://medium.com/@bensivo) or linkedin (https://www.linkedin.com/in/benjaminsivoravong), and if you do, I’ll get a little hit of dopamine, and love you forever. Kbye!

Background / Overview

My career in software engineering started like many do, in full-stack web development. The first few years of my career were spent learning how to build enterprise applications, taking into account testability, observability, resilience, scalability, and user-experience.

However, in the last year, my career has shifted from product development to Platform Engineering and Data Analytics. It’s been a very fun and interesting change of pace. I’ve learned a lot, but the shift has not been seamless. In many cases, I had to break down the way I think about applications, and learn entirely new paradigms of design.

This article focuses on my learnings from the world of Data Analytics. Here, I’ll outline the basic components of a Data Analytics Platform, starting from the core concepts and ideas.

The Goal of Data Analytics

Before we can dive into how data analytics pipelines work, we need to understand why they exist. I’ll explain with an example.

Imagine you are a Product Owner for a car company, and you’re faced with a relatively simple question, “How much should we sell our cars for?”. If you choose a very low price, you’ll probably sell tons of cars, but you might not make a profit. On the other hand, if you set the price too high, you’ll make more profit per-car, but won’t sell as many. For simplicity’s sake, we’ll assume your goal is just to maximize profits.

You can imagine a graph like below exists, plotting the price of the car against the total profit you’d make. If you were able to accurately create a graph like this, then the decision would be easy — set the price to whatever value maximizes profits. This is where you turn to Data Analytics.

Data Engineering and Analytics exist to support decision making — often by surfacing key relationships in data, or by predicting future events based on past trends. In this hypothetical example, we would’ve used our analytics platform to produce this graph, total profits by car price. Then, as the Product owner, we could use that graph to support the decision on the sale price of the car.

It’s important to call out, the graph is not the goal, the graph only exists to support a decision. Many people, experienced or new to analytics, think that creating pretty graphs is an end unto itself, but it is not. Graphs and visualizations are only useful in their ability to grant insights to decision makers. If they do not, they are simply technical debt.

Building a system that can consistently and efficiently derive these insights is the challenge of Data Platform architecture. Next, we’ll discuss the key components that make up the traditional Data Platform:

• Data Lakes
• Data Warehouses (and Lakehouses)
• Distributed Compute Engines
• Data Visualization Platforms

1. Data Lakes

Before you can start doing calculations on data, you need to collect it in the first place. In modern software systems, data often lives in many different locations, so the first step is to collect it all into a single place. That centralized location is called a Data Lake.

Data lakes have a few key factors that influence their design:

• Variety of producers — Data comes from many different places, so the Data Lake must use an interface / API that is relatively universal.
• Volume / Cost — Data costs money to store, and data lakes tend to store large amounts of data. In data storage, there’s generally a 3-way trade-off between cost, performance, and accessibility.
• Lifecycle — Data tends to become less relevant to a business the older it is. In some cases, you may want to delete data that’s too old. In other cases, you may just move it to a cheaper, less-frequently-accessed storage option.

All of these factors combined have led the industry to settle on cloud object-storage as the key storage solution for building Data Lakes. Objects in object-storage are kind of like files on a file-system, which turns out to be an abstraction that is simple and universal enough for most data producers. Perhaps more importantly, object storage is very cheap.

If you were to just start throwing files into an folder with no organization at all, you’d have a difficult time actually finding the data you need later. Therefore, most data-lakes encode some basic information about a file into the file path. A good starting point is to use the name of the record, and the timestamp. The image below shows an example folder-structure you could use for a data lake.

In the image above, we’ve used the ‘.jsonl’ file format. In a Data Lake, the file format can change based on the application that produced the record. Some applications will natively pass data around in JSON format — others will use CSV, Parquet, or Avro. In the “raw” folder of your data-lake, it’s best to just save exactly what your applications produce, with no transformation at all. This gives you replay-ability in your data pipeline. If something in your data pipeline breaks, you don’t have to ask your applications to re-produce records, you can replay them directly from your Data Lake.

2. Data Warehouses / Lakehouses

After your data has been collected into the Data Lake, you can move on to the next step — making it queryable. This is where the Data Warehouse comes in.

A Data Warehouse, just like a Data Lake, is also a system for storing large amounts of data, but with predefined schemas and file-formats, to enable efficient querying. Traditional Data Warehouses are pretty much just large, specialized Database servers. I personally haven’t used many of these kinds of Data Warehouses, so I won’t focus on them here. At very large scales, it’s common to build a Data Warehouse also using object-storage, just like our Data Lake. This has given rise to the term “Lakehouse”.

Data Lakehouses are built around the concept of an Open Table Format (OTF), first popularized by Apache Hive in 2008. The idea of OTF is simple: create efficient queryable SQL tables using object-storage, instead of large expensive database servers. Hive achieved this by separating its tables’ metadata (schemas, indexes, filepaths) from the data itself. The metadata was stored in a traditional relational database, called a metastore, and the data was stored in object-storage. To execute a query, a client would connect to the metastore to generate a query-plan, then follow that query-plan to access the files containing the desired records.

Today, Apache Hive’s Metastore architecture is somewhat dated. A second generation of OTFs has sprung up, offering the same core concept, queryable tables built on low-cost object storage, but with better performance, and more features. The three most common OTFs used today are Apache Hudi, Apache Iceberg, and Delta Lake. Each one specializes in a slightly different kind of data, and each has a different ecosystem of tooling, but all have the basic features mentioned above. The image below (taken from a great blog post from Dremio) is a diagram of Iceberg’s data format.

Using these OTF tools is pretty straight forward. In the case of Apache Iceberg, you just need to install the iceberg client package, configure access to your object-storage location and catalog, then start reading and writing tables. This can be done with a programmatic API, or with good-old-fashioned SQL.

The most common way to interact with OTFs (including Iceberg) is through a distributed-compute engine like Apache Spark, which is the next core building-block.

3. Distributed Compute Engines

With our Data Lake, we have a place to store large amounts of raw application data. With the Data Warehouse, we have a place to store large amounts of structured, queryable data. The next step is to have a compute engine that can handle large amounts of data processing at scale.

In the world of big data, scalability is achieved horizontally, by partitioning a dataset into slices, performing a processing task on each of those slices in parallel, then combining the results. The most popular distributed compute engine used today is Apache Spark, replacing the older Hadoop MapReduce.

The Spark system is made up of a few components:

• A single “master” node, which accepts jobs, and distributes them to workers
• One or more “worker” nodes, which execute the jobs given to them by the master
• A client-side API for defining jobs (currently there are APIs for java, Scala, Python, R, and SQL)
• A client-side tool for submitting jobs to the master node (done through the command-line executable spark-submit)

Running a Spark Cluster is no trivial task, but luckily, most cloud providers offer managed Spark clusters as a service (Azure Databricks, AWS EMR, GCP DataProc). Once the cluster is up and running, you can write your data-processing jobs using one of the supported programming languages.

In the screenshot below, I use Spark’s Python API (PySpark) to read a directory of JSON files from object-storage (MinIO), then write them into an Apache Iceberg table.

This script is relatively simple, it doesn’t do any transformation or aggregation at all, but the Spark API is robust, and contains operations for all of the aggregations and calculations you could dream of. And for those with SQL experience, the spark.sql() API exposes the same operations using SQL instead of procedural function calls.

This spark job is saved as a python file, then submitted to the spark cluster using the “spark-submit” command-line utility, which I run using a podman (Docker) container. The source code for this can be found on github here: https://github.com/bensivo/data-platform-study.

It’s worth mentioning, many Data Platforms manage write and submit Spark jobs using Jupyter notebooks hosted on a shared Jupyter server. This setup makes it easier for Data engineers to submit jobs (by just running cells in a notebook), and makes dependency management and data visualization easier as well.

There are generally 3 kinds of jobs that one writes as a Data Engineer. These correspond to 3 different forms data takes while being transformed from raw records into analytical insights.

1. Ingest / Load — Read raw data from the Data Lake, then insert it into the Data Warehouse in a “bronze” layer. Sometimes called “EL” for “extract” and “load”.
2. Clean — Take the data in the “bronze” layer, and apply filters to clean it. Remove duplicates, normalize naming conventions, and fix outliers. Write this cleaned dataset to a separate “silver” layer.
3. Aggregate — Now that you have a bunch of cleaned / normalized datasets, you can run aggregations to answer your business questions. This step’s logic is purely use-case dependent, and can involve joins, transformations, and any business-specific logic you need. The output of this transformation is the “gold” layer.

There are many implementations of this general pattern. The names “bronze” “silver” and “gold”, I borrowed from the Databricks ecosystem’s “Medallion Architecture”. However, no matter where you go in Data engineering, you’ll find these 3 layers of data. They may be named something different, but they will exist.

Finally, we can go on to the last step of our Data analytics process, sharing data with users.

4. Data Visualization Platforms

The last step to a Data Analytics platform is to take the data aggregates you’ve computed, and communicate them to people in your business. This is where our data visualization platforms come in.

The most common method for communicating about data is to draw graph or a chart, so this is what Data Visualization Platforms do. If you’re reading this article, you’ve probably heard of one Data Visualization Platform — PowerBI. Owned by Microsoft, this product is by far the industry standard in this space. Through PowerBI, you can connect to a variety of data-sources, then use the data to make any kinds of graphs and dashboards supported by the platform.

I personally have less experience working in Data Visualization, as my work experience has been more on the Data Engineering side, but I would be misguided to try to explain all of it here anyways. The internet is full of articles and tutorials on how to use Data Visualization Platforms, especially the popular ones like PowerBI, Tableau, and Apache Superset. However, whichever of these tools you use, the basic steps will be the same:

1. Load your data into the visualization platform
2. Create graphs, charts, and tables
3. Share your graphs with your business users, to guide their decision making

5. Conclusion

Thank you all for reading this far! I hope this article was useful in some way to some of you, and if it was, it’s purpose has been achieved. I have a lot more to learn about Data and Data Analytics, and in the future I hope to write more articles diving deeper into specific technologies mentioned in this article. If you like these articles feel free to clap, follow me here on Medium, or connect with me on LinkedIn. Thanks!

Background

When I was a new Software Engineer, I looked in awe at the so-called “Software Architects” in the companies that I worked in, and imagined what it would be like to know everything they knew. To my younger self, software architects were like beings from another world, filled with knowledge, masterfully creating complex systems too far-reaching for any mortal to understand. They were the omnipotent designers of our digital world, and I was a just cog in the machine.

Now, with probably a few dozen projects under my belt, I know better. Software Architecture is not an art, and architects are not omnipotent. Software architecture, at its core, is just a process of collecting and organizing information, then making decisions using that information. Even more, the process is not even particularly unique. Although every project is special in its own way, the process of discovery and information gathering is relatively consistent for all the projects I’ve been involved in.

This article is the current state of my personal Software Architecture process. This process is always evolving, but as of writing this, it’s generally proven to be consistent for the last few years, and has proved useful in projects from unit-testing POCs to full-blown Enterprise Data Platforms.

Steps

At a high-level, the process for designing a software system is something like this:

1. Problem Discovery — Define the problem you’re trying to solve in the first place. Who does it affect? Is there an existing system (and if so, what are its issues)?
2. Scoping / Prioritization — Decide how much time / resourcing do you have (or how much you’re willing to spend on this effort). Based on that, define the must-have and the nice-to-have features.
3. Requirements — Formally define functional and non-functional requirements for your system.
4. Solution Discovery — Research potential solutions, find things other people have done, and note the pros and cons of each approach that you find.
5. Solution Design — Take all of your requirements, all of your research, and actually make design decisions. Piece together a system that accommodates the needs of your users, your developers, and your timelines.
6. Design Review — Bring all the project’s stakeholders into a room and review the solution design. Collect feedback, and if needed, go back to the drawing board. But don’t proceed until you have alignment on what’s going to be built.
7. Implementation — You have requirements, you have design, you have buy-in from stakeholders. Go out and build it.

It’s important to call out that these are guidelines, not a strict process. Feel free to skip some steps, add steps, whatever is necessary for your specific project.

For example, if your system involves user-facing components, you may want to add a “Wireframes / Mockups” step, along with a subsequent “UX review”. On the other hand, if you are investigating a new technology, you may not need a “Scoping / Prioritization” step.

Documentation

In my experience, I have found 2 things to be true:

• If you have all the information necessary to make a design decision, the decision is obvious. Design is only hard when you have incomplete information.
• Disagreement arises when 2 people are working with different underlying assumptions about a system. This is usually from a lack or mismatch of information.

To combat both of these issues, it is critical that you produce documentation for each of the steps listed above. These documents will give you the information you need to make design decisions, and will help communicate your thought process to all the Stakeholders involved in this project who are not you.

Documenting these steps can take many forms, but it does not have to be a tedious process. There are many pretty (and verbose) templates for software-architecture docs online, but I find that nobody really cares if the document is pretty. They just want the information.

For myself, I like writing my documentation in simple Markdown files, which are checked in to a Git repo, and also published to a company wiki (like Confluence).

Here are some example documents I write for myself. You will notice that they align pretty closely with the steps outlined in the previous section.

• problem-statement.md — Contains a simple bulleted list of problem statements, organized by user-persona. Usually, these are direct quotes from user-interviews that I do during the discovery process.
• scoping.md — A short document that outlines my project’s rough timeline, resource allocation (i.e. how much money am I willing to spend on it), and also identifies which features my users have identified as must-haves, and which are not necessary.
• requirements.md — A more formal BRD-style document. Lists functional requirements (written as user-stories, organized by user-persona) and non-functional requirements.
• solution-discovery.md — Lists all of the potential technologies and components that could go into this project. For each component, I take notes on what it is, pros, cons, potential consequences, and links to where I got all of this information. If this doc gets too long, I may create a separate document for each technology that I researched.
• adrs.md — Architectural Decision Records. After you create your list of potential options, you will need to choose some. Each decision and the reasoning behind it should be documented in an ADR. This is mostly for future stakeholders (or yourself) asking “Why was it done this way?”.
• solution-design.md — The final solution design. I find the C4 model (https://c4model.com/) to be a very useful tool for visualizing system design, but I usually only use the first 2 levels of the model (system-context, container). I may also produce a deployment diagram and dynamic diagrams (also from the C4 model), if I believe they will be useful for communicating the design.
• meeting-notes-YYYY-MM-DD-NAME.md — All meetings should produce notes documenting what was discussed and decided. This includes any scoping / prioritization meetings, the design review, and user-interviews.

By writing these documents, you will have artifacts which you can use to bring stakeholders up to speed on what’s been done. You will also be able to reference them later, if anyone has questions about the design, or need to make further changes to the system architecture.

I have templates and examples of most of these documents in my Github profile here: https://github.com/bensivo/software-architecture-doc-templates.

Conclusion

If you made it here, thank you for taking the time to read this article. If you are new to Software Architecture, I hope this gives you a decent starting-off point for creating your first projects in a well-thought-out and considered manner. This framework has been super helpful for myself, especially as the number of projects I manage grows and grows, and I hope it can be of assistance to you as well.

Background

I’m writing this article from my perspective as Staff Engineer turned Software Engineering Manager. I had been operating as an individual-contributor for my whole career, but in the last few years I’ve been transitioning into more of a management role. With this shift into management, my day-to-day work has switched from developing my own technical skills, to developing the skills of others.

Over the last few years, I noticed many of folks on my team kept asking me for essentially the same advice every-time we met in a one-on-one. After a few years performing solidly as a mid-level software engineer, they want to know what it takes to get to the Senior Level. Here’s one thing that I tell them.

The Role of Side Projects in Technical Growth

One of the harsh facts of life in corporate America’s tech industry is this: being good at your job is not enough to get promoted to the next level, you must prove that you are already capable of performing the next job up. So if you are aspiring to be a Senior Software Engineer, you need to develop technical skills of a Senior, while still working as a Mid-level.

The main differentiator between a mid-level Software Engineer and a Senior Software Engineer is depth of knowledge in their technical domain. If every line of code ever written executed perfectly, and nothing ever broke in unexpected ways, we could get ChatGPT to build products. However, in real life, applications break, contracts get broken, and abstractions that support entire code-bases fall apart in weird, unexpected ways. This is when you bring in your Senior Engineers, as they have enough experience and deep technical knowledge to debug these issues and restore order to the chaos.

This situation is why I advocate for side-projects as a tool for professional growth. We gain deep technical knowledge of systems by stretching them to their limits, seeing where they break, and patching over those breaks to create products that work regardless. But in day-to-day work in product development, it is in a team’s best interest to NOT stretch systems to their breaking points. Additionally, product-teams never want to waste time re-inventing the wheel. When building products, a team will build something once, create an abstraction around it, then move on to bigger and better things. As a mid-level engineer on one of these teams, you may never get an opportunity to see how a system is created from the ground-up, or see where its technical limitations are, because that is dangerous and wasteful.

Side projects give you an opportunity to go back to the core principles of a system, built it from the ground-up, and learn exactly how it’s put together, without the risk of setting fire to production. In the process, you’ll inevitably do something wrong, break contracts, and overall just make a mess of everything. But after you’ve looked at all the pieces, studied each of them in depth, and restored order and functionality to your system, you’ll finally understand on a deep level how it all works together. You’ll be well on your way to Senior Engineer.

Project 1: DIY Web Application

This project is best for full-stack engineers who want to gain a full understanding of how internet-facing applications work. The core project is this to deploy a web application using as few managed services as possible. The only managed services involved are a VM, and a Domain name.

The steps involved in this project are:

• Create a basic full-stack web application in your framework of choice. It should have a front-end web page, a REST API, and a database of some kind. All 3 should run in docker.
• Provision a VM from any VPS Provider, like Linode or Digital Ocean. Don’t use a major cloud provider like AWS (too many abstractions, the point here is learn core concepts and fundamentals)
• Install and configure UFW, use it to allow only the ports you intent to expose publicly (22 for SSH, 80 for HTTP, 443 for HTTPS)
• Deploy the 3 docker containers to your VM, running on 3 different local ports (e.g. 8080 for the web app, 3000 for the api, 5432 for the db if you’re using postgres)
• Install NGINX on the VM, and configure it as a reverse proxy, serving your web-app at /app, and your api at /api
• Buy a domain (I used google domains, now Squarespace), and create an A record pointing to your VM’s public IP address
• Install and configure Certbot on the VM, to enable HTTPS traffic on your website

When it’s all said and done, you should have a fully functional web application accessible over the public internet with HTTPS. You will have something like the diagram below.

This project gives you hands-on experience with some of the core concepts that build the modern internet, including:

• Firewalls
• Reverse proxies and Load Balancers
• TLS certificates
• DNS records
• Basic Linux systems administration

The first time I did this project, it took me 3 months of hacking to get it to work. But in that process, I learned so more much about the fundamentals of the internet than I would have in an entire year of regular work using managed services. Now, I can provision this entire stack in less than an hour, and I have deep experience-based knowledge of each of these software systems.

However, we’re definitely not done after getting the basic application running. After the basic web application is built out, you can go deeper into one of these topics with the following extensions:

• IaC and Device Configuration: Use Terraform (or OpenTofu) to provision the DNS records and the VM itself. Then, use Ansible playbooks to install Nginx, Certbot, and Docker. Finally, use Ansible again to configure all these services and deploy the Docker images.
• Network and Application Load Balancers: Deploy multiple copies of your web application and REST API. Split the traffic between them in a round-robin fashion. Then split the traffic between them based on HTTP session cookies (NOTE: nginx can do all of these things)
• Networking: In your VM provider, create a VPC with 2 subnets. A “public” subnet, which can receive traffic from the internet, and a “private” subnet, which is only accessible from itself and the public subnet. Deploy NGINX into a VM in the public subnet, then deploy the Web app and REST API into VM’s in the private subnet.

Project 2: IoT Device Manager

This project is inspired by my previous job at an IoT + AI startup based in Dallas, TX called Spacee. Transitioning from full-stack cloud applications to Edge and IoT systems exposed me to tons of new software architecture paradigms, and really stretched my ability to design robust, bullet-proof code.

The IoT domain offers a few challenges in the realm of deployments and deployment orchestration, which are not present in cloud-native applications:

• Devices are usually installed in private networks, with no inbound connectivity. The device itself must be the one initiating any communication.
• Devices are not always online, so deployments must be orchestrated using eventual consistency patterns, not all-at-once updates.
• Most communication follows event-driven architecture, not request-response paradigms.

In this project, you deploy a metrics-collector service to IoT Edge devices, and use the MQTT protocol to send data to the cloud. A separate backend service reads this data and exposes it as a REST API. To finish it out, you implement eventually-consistent device updates using another MQTT topic and a second “controller” service on the edge device.

The steps involved in this project are:

1. Deploy an instance of the Mosquitto MQTT broker in a VPS / VM. Make sure you can connect to it from your home network over port 8883 using MQTTS.
2. Provision either a raspberry pi or another VPS to act as your “edge” device. This is the IoT device that we will pretend is installed in an inaccessible location.
3. Write a systemd service for the edge device, which periodically reads the device’s hardware metrics and publishes them to MQTT . Call this the “metrics-reporter” (NOTE: the actual functionality of this application doesn’t matter, it could do anything).

mqtts://<device_id>/metrics
{
    "cpuPercent": 12,
    "memoryPercent": 50
}

4. Write a backend service, deployed in the same VPS as the mosquitto broker. This service will subscribe to all of the metrics topics, and expose a REST API for fetching the latest reported metrics for all devices.

HTTP GET /devices
[
  {
    "deviceId": "my-device-01",
    "cpuPercent": 12,
    "memoryPercent": 50,
    "reportedAt": "1970-01-01T00:00:00Z"
  },
  {
    "deviceId": "my-device-02",
    "cpuPercent": 60,
    "memoryPercent": 55,
    "reportedAt": "1970-01-01T00:00:00Z"
  }
]

5. Write another systemd service, which will be used to control the metrics-reporter. This will subscribe to a “desired_configuration” topic, and update the version of metrics-reporter as appropriate. It should also publish to a topic called “reported_configuration” on bootup, or after an update.

NOTE: both of these topics should use the “retained message” feature of MQTT to ensure new consumers will receive the latest published value

mqtts://<device_id>/desired_configuration
{
  "version":"1.0.0",
  "requestedAt": "1970-01-01T00:00:00Z"
}


mqtts://<device_id>/reported_configuration
{
  "version":"1.0.0",
  "updatedAt": "1970-01-01T00:00:00Z"
}

6. Finally, add an endpoint to the REST API for setting the desired configuration, and fetching the reported version of each device

HTTP POST /devices/<device_id>/configuration
{
  "version":"1.0.0"
}

HTTP GET /devices/<device_id>/configuration
{
  "desiredVersion":"1.0.0",
  "requestedAt":"1970-01-01T00:00:00Z",
  "reportedVersion":"0.0.9",
  "updatedAt":"1970-01-01T00:00:00Z"
}

The final solution will look like this:

This project is more complex than a standard full-stack web application, but it serves as a great introduction to lower-level Linux systems. It is also a great introduction to event-driven systems with MQTT, and builds on all the networking fundamentals that were developed in Project 1.

Similar to Project 1, once the POC is built out, you can dive deep into several topics with these extensions to the system:

• x509 Certificates: Create your own certificate-authority, and use it to provision a separate x509 certificate for each Edge Device. Then, swap out user-password authentication for MQTT with mutual TLS authentication.
• Data Analytics: Add another consumer to the metrics MQTT topic, and sink all of your device messages into an object-storage service like Amazon S3 or Azure Blob storage. Then use a python script to read these files, and create a graph of “Device CPU utilization over time”.
• Observability: Create a free account on NewRelic Cloud, and use the OpenTelemetry protocol to ship your device metrics to NewRelic’s OTLP endpoint. Then, create a NewRelic dashboard tracking CPU utilization over time, and setup email alerts to notify you if CPU goes over a specified threshold. For bonus points, also ship your edge-device’s application logs to NewRelic using OpenTelemetry.
• Embedded Systems / Electrical Engineering: Instead of just reporting your device’s CPU and Memory metrics, buy temperature and humidity sensors and connect them to your edge device. Then use the appropriate serial communication protocol (I2C, GPIO, etc) to read the values from those sensors and turn your metrics-reporter into a climate monitor. This is only possible if you used a raspberry pi as your edge device, not a VPS.

Introduction

This article demonstrates how I use Taskfile.dev to manage small monorepos for my personal projects.

All the code for this article can be found on my Github here: https://github.com/bensivo/monorepo-example-taskfile

Requirements for a Monorepo

Before we start, we need to define set of features required for this “minimalist monorepo”. Monorepos have been around for a long time, so we can draw on some existing tool sets to get a list of features that seem to be common:

• Ability to define packages, with “tasks” in each package (e.g. lint, build, test). These are often called “targets” in many build tools
• Ability to run all tasks from a single executable
• Ability to define task dependencies, within 1 package, and between 2 packages
• Ability to cache task inputs and outputs, to avoid re-running tasks if inputs have not changed

Why Taskfile.dev?

At work, my team uses NX for our monorepo tool chain. NX is a great tool, if you are part of it’s target market — highly structured, multi-project teams, using Javascript. When working on personal projects, I don’t fit these criteria very frequently, so I have turned to Taskfile.dev instead. Compared to NX, Taskfile.dev has:

• Less bloat — NX is based on javascript, therefore adding package.json and node_modules to your repo. This is fine if you’re working in javascript, but feels like bloat if you’re using any other language.
• Cleaner configuration — NX uses very large, clunky JSON files for configuration, but Taskfile.dev is configured in yaml, and has much less configuration. Much better for readability
• No opinions — NX (by default) is very opinionated. On the whole, their opinions are good ones, especially for larger organizations, but for personal projects, the freedom to pave your own way can be beneficial.
• More transparency — NX is built on a plugin architecture. Plugin authors maintain wrappers for underlying technologies, and expose their functionality through JSON configuration. For example, instead of writing a jest command npx jest --config ./jest.config.js ./tests/**/*.spec.ts , you would install the Jest NX plugin, and set the appropriate configuration options to get the same setup. This hides the underlying details of Jest from the user. For large teams, this is a very useful feature. For personal projects, it’s overkill.

Defining Packages

To define packages, we will create a root-level packages/ folder. Each sub-directory in this folder will contain either an application, library, or some other unit of code. For example, I can setup a basic full-stack web application with the 4 packages below

packages/
	helloworld/
	helloworld-api/
	helloworld-app/
	helloworld-core/

The packages themselves are explained below:

• helloworld: Contains a docker-compose file, for running both the api and app in a single stack
• helloworld-api: REST Api exposing a single GET /greetings/:name endpoint
• helloworld-app: React SPA. Prompts for a name, and calls the API
• helloworld-core: Common libs used by helloworld-api

Defining a Task

Each package will have a set of Tasks that it can run, like “build”, “lint”, “test”, and “run”.

To define tasks for a package, create a file called Taskfile.yml in the package directory. Below is a basic Taskfile for helloworld-api, defining the “build” task.

# ./packages/helloworld-api/Taskfile.yml
version: 3

tasks:
  build:
    cmds:
      - env GOOS=linux GOARCH=arm64 go build -o dist/helloworld-api ./main.go

You can execute the task with the task command, as long as you are in the package directory (or a subdirectory).

When working across multiple packages, it will get tedious to constantly have to cd into each package folder. To make life a little easier for us, we can put another Taskfile in our root directory, which “includes” all the other Taskfiles.

# ./Taskfile.yml
version: '3'

includes:
  helloworld-api: 
    taskfile: packages/helloworld-api/Taskfile.yml
    dir: packages/helloworld-api

Now, we can execute the build task from the project root, by prefixing the command with the package name:

Defining Tasks with Dependencies

The true power of Taskfile is unlocked when we start defining task dependencies. Lets imagine we want to build our helloworld-api, then package it into a Docker image, then run that Docker image. We could do that like this:

go build -o dist/helloworld-api ./main.go
docker build -t helloworld-api ./
docker run -p 3000:3000 helloworld-api

We can break this up into 3 distinct tasks: build, docker, and run, with each task invoking the task before it. You can see in the example below, we can use the task: <name> block to call one task from another.

# ./packages/helloworld-api/Taskfile.yml
version: 3

tasks:
  build:
    cmds:
      - env GOOS=linux GOARCH=arm64 go build -o dist/helloworld-api ./main.go
  docker:
    cmds:
      - task: build
      - docker build -t helloworld-api .
  run:
    cmds:
      - task: docker
      - docker run -p 3000:3000 helloworld-api

Now, we can run the entire pipeline build → docker → run, by just calling the final task: task helloworld-api:run

Defining cross-package dependencies

This works great when all the tasks are in the same package, but one of the great features of monorepos is the ability to define a shared library, and reuse it between several applications.

Lets image we have another package called helloworld-core, which contains some dependencies for helloworld-api. And we want to build helloworld-core, before building the api.

We can do that with the setup below:

# ./packages/helloworld-core/Taskfile.yml
version: 3

tasks:
  build:
    cmds:
      - echo "Building helloworld-core"

# ./packages/helloworld-api/Taskfile.yml
version: 3

includes:
  helloworld-core: 
    taskfile: ../helloworld-core/Taskfile.yml
    dir: ../helloworld-core
    internal: true

tasks:
  build:
    cmds:
      - task: helloworld-core:build
      - env GOOS=linux GOARCH=arm64 go build -o dist/helloworld-api ./main.go

...

You can see we used the “includes” block again, but this time between Taskfiles in 2 different packages. We also set internal:true, to prevent helloworld-api from re-exporting all of the tasks in helloworld-core.

Now, we can run task helloworld-api:run from the project root, and it will execute the entire pipeline: helloworld-core:build → helloworld-api:build → helloworld-api:docker → helloworld-api:run

Preventing re-execution (task caching)

There is one issue with our current setup — every time I execute helloworld-api:run, it does a full build of helloworld-core, and helloworld-api, even if nothing has changed.

We can remediate this with Taskfile’s sources and generates blocks. These are used to define the inputs and outputs of each Task.

• Before running each Task, Taskfile will calculate the hash of all files defined in sources and generates. If that hash matches the saved version found in the .task/ folder, the task will be skipped
• After running a Task (assuming it was not skipped), Taskfile will re-calculate the hash, and save that in the .task/ folder

Here is an example defining these blocks for our “build” task:

# ./packages/helloworld-api/Taskfile.yml
version: 3

tasks:
  build:
    sources:
      - ./**/*
      - exclude: ./dist/**/*
    generates:
      - ./dist/**/*
    cmds:
      - env GOOS=linux GOARCH=arm64 go build -o dist/helloworld-api ./main.go

You can see in the output below, we run helloworld-api:build twice, the second time is skipped.

Then, we change one of the input files (adding a README file), and the task is no longer skipped.