Disclaimer: This article is an exclusive content based on the book, Effective Platform Engineering. Find more about the book here.

Recent research from the State of AI in Platform Engineering [1], as well as the AI Capabilities Model introduced in DORA 2025 report [2], proves definitively that GenAI only accelerates platform delivery when coupled with strong version control, small-batch working, and quality internal platforms. Without these foundations, AI amplifies chaos rather than productivity, introducing persistent instability risks that can undermine the very velocity gains organizations seek.

AI’s primary role is that of an amplifier. It magnifies the strengths of high-performing organizations and the dysfunctions of struggling ones in equal measures. Successfully embedding GenAI into the Software Development Lifecycle (SDLC) to optimize internal platform delivery is fundamentally dependent on cultivating foundational engineering practices, specifically, fast feedback loops, robust version control practices, and the discipline of working in small batches. These foundational practices have nothing to do with AI in the first place, yet they determine whether AI becomes an accelerant or an anchor. The illustration below shows where most organizations are today and where they need to be, by transforming their AI efforts from a road block to an booster.

Figure 1: The reality and the required transformation for the AI journey

The Universal Adoption and the Hidden Paradox

The adoption of AI in software development has become nearly universal, with 90% of technology professionals surveyed reporting its use as part of their work. This suggests AI use has become the industry standard rather than something that gives a company a competitive advantage.

Yet beneath this enthusiasm lies a paradox that platform engineering leaders must confront. While the research demonstrates that AI adoption consistently correlates with positive outcomes, higher individual effectiveness, improved code quality, enhanced organizational and team performance, and increased time spent on valuable work, it simultaneously correlates with higher levels of software delivery instability. This persistent instability captures the quality and reliability of the software delivery process, measured partly by change failure rate and rework rate.

The duality reveals AI’s nature as an amplifier. While AI excels at accelerating individual tasks, the organization’s existing governance, verification, and deployment systems often struggle to manage the resulting increased volume and speed of changes. For platform engineering teams, whose job centers on system reliability and reducing friction, mitigating this amplified instability becomes paramount. If AI increases the volume of changes without a corresponding evolution of guardrails, roles, and golden paths, it increases verification and coordination costs, ultimately leading to downstream chaos.

Strategic GenAI Application Across the Platform SDLC

As explored in depth in our unpublished Chapter 11 of Effective Platform Engineering[3], the platform engineering team must strategically apply GenAI capabilities across the entire SDLC to minimize friction and cognitive load for internal developers. This requires conducting a detailed path-to-production analysis to pinpoint where inefficiencies occur within the development lifecycle. Not surprisingly, these techniques around VSM[4] and Flow Engineering[5], coupled with Team Topologies[6], are neither new nor different from how platform engineering has historically been done right.

Planning and Design Acceleration

GenAI proves highly effective at handling boilerplate and basic scaffolding, freeing product owners and engineers to concentrate on more complex tasks like strategic design and requirement analysis. In the planning phase, GenAI can accelerate the work of business analysts and product owners by translating raw customer feedback and high-level business objectives into comprehensive user stories and acceptance criteria.

The case of Epetech, the case study we have detailed throughout the effective platform engineering book, illustrates this capability. By inputting high-level business objectives into their GenAI system, Epetech generated comprehensive user stories that drastically reduced time spent on manual documentation while improving alignment with stakeholder needs. GenAI can also analyze planning goals to identify priorities and potential conflicts arising from inconsistent requirements, turning what was once a manual process into an automated consistency check.

In the design phase, GenAI enhances both quality and speed. LLMs trained on design patterns can generate architectural diagrams, data models, and descriptive visualizations. Tools automatically create Entity-Relationship Diagrams (ERDs) and provide options driven by design criteria, enabling an iterative design process. These models can estimate potential implementation costs and performance considerations, making decision-making more solid. For a microservices project, Epetech used their GenAI model to analyze desired functionality and automatically propose an architecture including service boundaries, API endpoints, data storage solutions, and security considerations. This capability transforms architectural design from a weeks-long exercise requiring deep expertise into a collaborative process where GenAI serves as an intelligent design partner, surfacing options that human architects can evaluate and refine.

Development and Modernization Excellence

71% of respondents whose jobs involve writing code using AI to assist them. The use of AI is transforming coding toward an iterative conversation, sometimes called “vibe coding,” where developers tell an AI system what they want and AI translates that into implementation.

AI-augmented coding tools, such as GitHub Copilot, provide real-time insights and suggestions, reducing the need for manual boilerplate code. The development process becomes more efficient and enjoyable with clear outcomes.GenAI systems also conduct smart code reviews, offering immediate feedback on code quality, potential bugs, and adherence to coding standards. This automated review process accelerates code quality assurance for organizations, providing real improvements.

For organizations not starting as startups or scaleups, dealing with legacy codebases which haven’t always been built well remains common. LLMs trained on refactoring patterns can analyze tightly tied together systems and recommend how to break them apart. AI-driven tools are emerging to assist in migrating mainframes and codebases that are difficult to find developers for what the book terms as “managing legacy code” through intelligent pattern recognition and automated refactoring suggestions.

Incorporating these migration tools becomes a key part of the platform strategy so maintainability issues can be discovered sooner.

Testing: The High-Potential Frontier

Testing represents the highest-potential area for GenAI to improve quality and mitigate the system instability risk that accompanies increased development velocity. GenAI is used for tasks like creating test cases, making it one of the most widely adopted AI applications in the SDLC.

GenAI techniques help perform efficient root cause analysis (RCA) by quickly evaluating data such as metrics, logs, and traces. The strategic focus should be on test case generation and increasing automation. GenAI-augmented tools like SonarQube, CodeClimate, and Coverity automatically generate unit and integration tests based on the existing codebase and identify gaps in test coverage.

Early quality feedback represents another important ability. GenAI detects code complexity issues early, recommending refactoring where necessary to enhance code maintainability. By integrating GenAI with static code analysis, complexity and maintainability issues can be discovered sooner, accelerating the implementation of possible solutions. Continuous monitoring of metrics like cyclomatic complexity and code duplication helps platform teams maintain a robust and reliable SDLC.

The evolution from product requirements through test execution to metrics against goals shows how GenAI can inform each stage. Test goals and automation coding standards inform the generation of test cases, datasets, and automation. These feed into root cause analysis and anomaly detection, which produce test metrics against established goals, creating a continuous improvement loop.

DORA Capabilities: The Essential Guardrails Against Instability

The DORA AI Capabilities Model identifies seven foundational practices that significantly amplify AI’s benefits. Platform teams must enforce these capabilities to manage the high speed of change introduced by GenAI. These capabilities represent the difference between AI as an accelerant and AI as a source of chaos. We will now attempt to map these foundational practices in the context of platform engineering.

Figure 2: What can GenAI realistically amplify today

Strong Version Control Practices and the Rollback Safety Net

As AI increases the volume and speed of code, strong version control practices become even more critical. The ability to undo changes acts as a psychological safety net, encouraging teams to adopt high-speed tools with confidence. Developers who commit frequently can isolate changes, making it easier to identify what went wrong when issues arise. More importantly, the ability to use rollback features to undo or revert changes amplifies AI’s positive influence on team performance.

Since AI use is associated with a higher degree of software delivery instability, the ability to rapidly undo changes acts as a critical safety net when working with the potentially larger batches of code AI produces. Platform teams must embrace and fortify these safety nets, making rollback not just possible but frictionless. This means investing in version control workflows that support rapid reverting, clear commit histories that enable surgical rollbacks, and deployment systems that can quickly restore previous states.

Working in Small Batches: Reducing Systemic Friction

Working in small batches involves committing fewer lines of code per change, fewer changes per release, and assigning work that can be completed in a shorter amount of time. This discipline is critical for integrating the output of AI tools, which otherwise produces larger, harder-to-review batches of code, which, in our experience, is one of the biggest challenges when it comes to maintaining AI-generated code in the long run. Working in small batches turns AI’s neutral effect on friction into a beneficial one, showing a measurable decrease in friction. This capability addresses the systemic friction that can negate individual productivity gains. For individuals, friction doesn’t vanish entirely; rather, it shifts from “manual grind” to new forms like prompt iteration, result vetting, and assessing AI-generated code. Small batch work helps smooth this transition.

When AI-assisted teams work in small batches, AI’s positive influence on product performance is amplified. This enforces the discipline necessary for high quality and ensures that individual effectiveness translates into product outcomes. Interestingly, working in small batches decreases individual effectiveness from AI use. This is theorized because AI predominantly increases effectiveness by helping developers quickly generate large amounts of code. However, the organizational benefits, in our opinion, reduced friction and amplified product performance, outweigh any potential harm to individual effectivenesss.

The small-batch discipline forces a healthy constraint on AI-generated output. Rather than accepting a 5000-line AI-generated function wholesale, developers working in small batches will break that into logical, reviewable commits. This practice makes code reviews more effective, reduces the cognitive load on reviewers, and ensures that when problems do arise, they can be isolated and addressed quickly.

Quality Internal Platforms and Fast Feedback Loops

The success of AI adoption is fundamentally nested within the quality of the organizational platform. A Quality Internal Platform (QIP) is a set of capabilities shared across multiple applications, aiming to make these capabilities widely available. Investing in QIPs emerges as a powerful strategic lever in the research. The results are clear: AI adoption has a negligible effect on organizational performance when platform quality is low, but the effect becomes strong and positive when platform quality is high. This means that a robust platform, which provides “paved roads” or “golden paths,” is necessary for AI benefits to scale beyond individual productivity to the organization overall.

Platform quality encompasses several aspects: self-service capabilities that reduce dependencies, standardized workflows that reduce effort, automated governance that ensures compliance without manual checks, and comprehensive observability that surfaces issues before they become incidents. Without these foundations, AI-generated code may work locally but fail when put into place, AI-suggested architectures may violate organizational constraints, and AI-accelerated development may overwhelm manual verification processes. Fast feedback loops complement platform quality by ensuring that the acceleration AI provides doesn’t outpace the organization’s ability to verify and validate changes. Code reviews often represent a major bottleneck in development workflows. GenAI can augment tools to accelerate these reviews and clarify workflow steps, enabling teams to adapt to the higher volume of code produced by AI.

The feedback loop must operate at multiple levels: immediate feedback from AI tools on code quality and potential issues, rapid automated testing that validates functionality, swift code reviews that catch design and architectural concerns, and quick deployment cycles that surface integration issues. When these feedback mechanisms operate efficiently, the organization can safely absorb the increased velocity that AI provides.

Measuring Impact Through Modern AI Methodologies

GenAI in platform engineering must be accurately measured using industry-standard AI tools and techniques, focusing on key performance indicators like the four key DORA metrics: deployment frequency, lead time for changes, mean time to recovery, and change failure rate.

Advanced analytics platforms like Datadog, New Relic, and Dynatrace now incorporate AI and machine learning capabilities to process vast amounts of data from CI/CD pipelines, monitoring systems, and code repositories. These tools, often enhanced by LLMs, can analyze historical deployment data, source code changes, and operational metrics to identify inefficiencies and recommend targeted improvements. Organizations can leverage AI-driven observability tools to measure the tangible impact of GenAI on platform engineering efforts. These observability platforms, powered by LLMs, process logs, metrics, traces, and user interactions to gain deep insights into system behavior. By integrating tools like Splunk and the ELK stack with GenAI models, organizations can automatically parse logs and identify patterns correlating with deployment failures or performance bottlenecks.

Through this data-driven approach, teams can pinpoint where GenAI-driven automation has the most significant impact. Analysis may reveal that automated code generation features lead to a 25% increase in developer productivity by reducing manual coding efforts. Similarly, implementing smart code reviews using GenAI might lead to a 30% reduction in production defects by providing developers with immediate, context-aware feedback during development. Needless to say, every organization should make a determination of the operational and social cost of achieving these improvements and if they are worth it for them.

The Role of Software Engineering Intelligence Platforms

Tools like Jellyfish, LinearB, and Faros.ai represent the forefront of integrating GenAI into developer environments to enhance software engineering quality and boost organizational productivity. These platforms serve as engineering operations systems, aggregating data across various stages of the SDLC and incorporating GenAI-driven insights to provide a unified view of engineering activities. These tools leverage data from diverse sources such as CI/CD pipelines, version control systems, issue trackers, and observability platforms to create a comprehensive picture of the development process. When augmented with GenAI, this data is not only aggregated but also analyzed to detect patterns, identify bottlenecks, and recommend improvements. For example, the platform can use LLMs to analyze commit histories, code review comments, and deployment logs to pinpoint areas where developer friction occurs, such as prolonged code review cycles or frequent deployment failures.

One key benefit of incorporating GenAI into these platforms is their ability to measure and report on key performance indicators (KPIs) more intelligently. Traditional metrics like deployment frequency, lead time for changes, and mean time to recovery are enhanced with AI-generated context. These tools can analyze deployment frequency alongside code complexity metrics to understand whether high deployment rates correlate with increased complexity and potential technical debt, then use GenAI to recommend specific interventions.

The Future Evolution: From Copilots to Autonomous Agents

Our research and the ensuing exploration in our book Effective Platform Engineering of future frontiers, reveal a path from today’s copilot tools toward increasingly autonomous agents that not only assist but actively manage aspects of the platform engineering lifecycle. Current copilot tools like GitHub Copilot, Amazon CodeWhisperer, Tabnine, and Replit Ghostwriter embody GenAI’s transformative potential by automating coding tasks, enhancing code quality, and personalizing the development environment. They align with core principles of improving developer experience and reducing friction. As intelligent assistants, these tools accelerate the SDLC and empower developers to focus on innovation, creativity, and building high-quality software.

The next evolution involves conversational interfaces that simplify infrastructure interaction. Tools like Kubiya.ai represent a new generation where developers and operations teams can manage cloud infrastructure, Kubernetes environments, and DevOps workflows using natural language commands. Instead of memorizing complex scripts, engineers can simply say, “Scale the payment service deployment” or “Restart the CI pipeline,” and the platform executes the command accurately.

Beyond conversational interfaces lies the frontier of Agentic AI, autonomous agents capable of reasoning, adapting, and making complex decisions. These AI-powered agents move beyond simple automation toward goal-driven systems that dynamically respond to platform challenges. Unlike traditional automation that follows predefined scripts, Agentic AI learns, adapts, and optimizes processes in real time. There are potentially at least four use cases organizations can adopt in this scenario

1. Fixing production issues through AI-driven remediation that proactively analyzes logs and applies fixes
2. Autonomous pipelines that self-optimize for faster releases
3. Smarter observability with predictive anomaly detection
4. Automated governance and cost optimization that continuously adjusts resource allocation.

Each use case demonstrates how Agentic AI builds upon a solid engineering platform foundation to achieve outcomes that would be impossible through manual effort alone.

The Importance of Foundation Before Acceleration

The central message emerging from both the State of AI report, DORA research, and our work on the platform engineering book is clear: Generative AI is not a stand-alone solution for platform engineering. It is a technological amplifier that demands organizational maturity. While AI is associated with significant boosts in individual effectiveness and code quality, its integration must be managed carefully to counter the persistent rise in software delivery instability.

Platform engineering teams achieve sustainable speed and reliability by embedding AI across the SDLC, from requirements generation to automated testing, while simultaneously reinforcing DORA capabilities. This means investing in strong version control practices that make rollback effortless, disciplined small-batch working that keeps changes reviewable and testable, and Quality Internal Platforms that abstract complexity and provide golden paths.

This strategic, systems-level approach ensures that the speed gained from GenAI translates into superior organizational outcomes. Organizations that neglect these foundations will find that AI amplifies their problems: poor version control becomes chaotic branching, large batches become massive hard-to-review changes, and weak platforms become bottlenecks that negate individual productivity gains. The path forward requires platform engineering leaders to resist the temptation of viewing AI as a shortcut past foundational work. Instead, they must recognize that AI’s power is unleashed only when applied to well-built systems with mature practices. The organizations that will derive the greatest strategic returns from GenAI investment are those that have already built — or are committed to building — the foundational capabilities that allow AI to amplify strengths rather than weaknesses.

Implementing the Vision: Practical Steps for Platform Teams

For platform engineering teams seeking to realize these benefits, several concrete actions emerge from the research and practice. Here is a 5-point checklist that can be followed as you rejigger your platform strategy with appropriate doses of AI.

1. Conduct a foundational capability audit. Before investing heavily in AI tools, assess your organization’s current state across the DORA capabilities. How strong are your version control practices? How consistently do teams work in small batches? What is the quality of your internal platform? These assessments reveal where investment will yield the greatest returns.
2. Prioritize platform quality investments. Given the research showing that AI’s impact scales with platform quality, investments in platform capabilities should precede or accompany AI tool adoption. This means building self-service capabilities, standardizing workflows, automating governance, and instrumenting comprehensive observability.
3. Establish fast feedback loops. Accelerate every verification mechanism in your SDLC. This includes automated testing that runs in minutes, not hours, code review processes that provide feedback within hours, not days, and deployment pipelines that surface integration issues immediately. Fast feedback loops create the safety net that allows teams to move quickly with confidence.
4. Embed AI strategically across the SDLC. Rather than adopting AI tools opportunistically, conduct a path-to-production analysis to identify high-friction points. Deploy AI capabilities where they will have the greatest impact on reducing toil and accelerating feedback, whether that’s in requirements generation, design, development, testing, or operations.
5. Measure and iterate. Use software engineering intelligence platforms to measure the actual impact of AI adoption on both individual and organizational metrics. Track not just productivity gains but also quality indicators and stability metrics. Use this data to refine your approach, doubling down on what works and adjusting what doesn’t.
6. Organizations must plan and build for the inevitable future of agency. While current copilot tools deliver immediate value, architect your platform with an eye toward more autonomous agents. This involves creating clear interfaces, establishing programmatically operable governance frameworks, and building observability that machines can interpret and act upon.

Conclusion: Amplifying Excellence, Not Compensating for Weakness

The promise of Generative AI in platform engineering is extraordinary, but its realization depends entirely on the foundation upon which it’s built. AI acts as an amplifier; it magnifies what already exists. For high-performing organizations with strong version control, disciplined small-batch working, and quality internal platforms, AI accelerates their excellence, turning good practices into exceptional outcomes. For struggling organizations lacking these foundations, AI amplifies dysfunction, turning manageable problems into crises.

Foundational engineering excellence is not optional when adopting AI. It determines whether AI becomes a competitive advantage or a liability. Platform engineering teams must resist the allure of AI as a shortcut and instead embrace it as an accelerant applied to solid foundations. By embedding GenAI across the SDLC while simultaneously reinforcing DORA capabilities, platform teams create an environment where localized productivity boosts compound into organizational throughput gains, where increased speed is matched by maintained or improved stability, and where the platform truly becomes helpful. This is the path to sustainable competitive advantage in an AI-augmented world. This is not achieved through the new fancy tools alone, but through the strategic combination of advanced technology and fundamental engineering discipline. The organizations that will thrive in the era of AI-accelerated development are those that recognize this truth and act on it: GenAI is not a replacement for excellence but an amplifier of it. Build the foundation first, then let AI magnify your strengths. This is how platform engineering teams will deliver extraordinary outcomes in the age of Generative AI.

About the Authors

Ajay Chankramath is the Founder and CEO of Platformetrics. Nic Cheneweth and Bryan Oliver are Principal Consultants at Thoughtworks. Sean Alvarez is the CTO of Life Sciences at Brillio.

References

[1] https://platformengineering.org/reports/state-of-ai-in-platform-engineering-2025

[2] https://dora.dev/research/2025/

[3] https://effectiveplatformengineering.com

[4] https://tkmg.com/books/value-stream-mapping/

[5] https://flowengineering.org/

[6] https://teamtopologies.com/

Every pixel you place matters. Every button you code shapes behavior. That might sound dramatic, but it’s true — as designers, developers, and product teams, we’re not just building interfaces. We’re shaping human experiences.

This means our work has a real impact. Design choices can empower, include, and delight, but they can also manipulate, frustrate, and harm. The line between persuasive design and digital manipulation is thinner than we’d like to admit.

So how do we build beautiful, effective products without compromising our values? One word: ethics.

Does Design Really Need Ethics?

Design isn’t just about aesthetics or usability; it’s also deeply ethical. By understanding a few principles of psychology and visual hierarchy, you can influence how users behave. You know how to keep someone scrolling, clicking, and coming back for more. Many of the engagement metrics your company celebrates stem from habits you helped design.

That’s a lot of power. Ethics, our internal compass for right and wrong, is what keeps that power from being abused. Ethics help us make decisions when the business goal is to maximize engagement, but our gut says, “This feels wrong.”

Who knows if humanity would have had an atomic bomb today if Oppenheimer had thought seriously about ethics? You might think, “Come on, we’re not launching nukes, we’re just designing apps.”

But think about online gambling platforms. Their interfaces are designed to lure people into endless loops with flashing lights, near-win animations, and delayed rewards. It’s all psychological manipulation. What about when someone loses their savings, their family, or their hope? That’s not harmless. It’s the result of unethical design. family, their hope? That’s not harmless. That’s the result of design without ethics.

Okay, most cases aren’t that tragic. Usually, victims of poor design are just left feeling frustrated and angry. For example, I thought I had canceled my subscription to a streaming platform. The next month, another charge appeared. It turns out that clicking “unsubscribe” wasn’t enough. I had to navigate through what felt like nine circles of subscription hell to cancel. There were different pages, confusing language, and hidden buttons.

That’s not just poor design, but deliberate disrespect. I wasn’t a valued customer; I was a revenue stream they wanted to keep flowing at any cost. Needless to say, I’ll never go back to that platform. One dark pattern cost them a customer for life.

These aren’t accidents. They’re designed and developed. And they have names:

• Obstruction: Making simple tasks, like deleting an account, deliberately difficult.
• Forced Continuity: Starting with a free trial that morphs into paid billing with no clear opt-out.
• Confirmshaming: Guilt-tripping users into choices. “No thanks” becomes “No, I don’t want to save money.”
• Bait and Switch: Promising one thing, delivering another. Download a “free app,” end up with a paid subscription.

These patterns work because they exploit how our brains are wired — we avoid conflict, take shortcuts, trust the interface.

Even worse, vulnerable populations, such as elderly users, people with cognitive disabilities, and those with limited tech literacy, suffer the most. What might be a minor inconvenience for you could be a devastating financial trap for someone’s grandmother.

The long-term damage goes beyond mere frustration. Users may lose trust in digital products entirely. They become defensive, suspicious, and exhausted by every interaction.

And all of that happens when ethics never enter the process.

“Just Be Good” Isn’t Enough

You might think the solution is simple: don’t lie, don’t manipulate, don’t trick users. But ethics isn’t that black and white.

I devoted nearly an entire chapter to ethics in my book ‘Emotional UX’, and even that barely scratched the surface. People have debated morality for thousands of years. There are no easy answers, just better questions.

Think about it: Is it more ethical to let someone doom-scroll for hours because it’s their choice, or to build features that gently discourage excessive use? Should we design addictive games because people enjoy them or refuse to because we know they’re harmful?

These are real design decisions, and they live in gray areas. The more influence we have over user behavior, the more responsibility we bear.

If you’ve asked yourself questions like these — congratulations! You care. But caring isn’t the same as having answers. Our instincts are helpful, but they’re not enough. Some people feel guilty doing the wrong thing. Others sleep just fine. And some feel righteous even when they’re doing harm.

We need something stronger than instinct. We need a framework.

The Ethical Framework That Helps Us Decide

In 2021, researchers Jonathan Kwan, Margaret McLean, and Irina Raicu from the Markkula Center for Applied Ethics created a powerful decision-making tool. It’s simple, flexible, and designed for exactly what we deal with in product and design work.

The framework works through six different ethical “lenses”, or different ways of asking, “What’s the right thing to do?”

1. Rights — Does this respect users’ dignity and autonomy?
2. Justice — Are we being fair? Who benefits, and who pays the cost?
3. Utilitarianism — Are we creating the greatest good for the greatest number?
4. Common Good — What does this mean for society, especially for those most vulnerable?
5. Virtue — Who are we becoming by making this choice? Does it align with values like honesty and compassion?
6. Care — How does this decision affect human relationships and communities?

These lenses offer valuable insights, but don’t expect them to give you easy answers. You and your team might apply the same ethical lens to a design decision and reach completely different conclusions about what’s right. Ethics is challenging.

Turning Reflection into Action

So how do you actually use this in your work?

Bookmark this process. Believe me, you’ll need it more often than you think. Whether you’re debating a notification strategy or questioning a signup flow, these five steps will guide you toward the right choice.

Here’s a quick 5-step roadmap to walk through when facing a questionable decision — from a sketch in Figma to a company-wide feature rollout:

1. Recognize there’s a dilemma. Trust that little “hmm…” feeling.
2. Frame the situation clearly — who’s impacted and how?
3. Evaluate using the six lenses above.
4. Test the solution with real users, not just your team.
5. Reflect and document what you learned for the next time.

I wish I could tell you this framework will solve all ethical dilemmas. But that would be lying. You’ll still run into situations where every choice feels wrong in some way. The difference is that now you have a way to work through those tough calls systematically instead of just hoping for the best.

I’ve found that using this process gives me more confidence when I need to push back on questionable ideas. Rather than saying, “This feels wrong,” I can explain the reasoning to my team. Most people aren’t trying to be malicious, they just haven’t considered all the implications. This framework provides an opportunity to think through those implications, which usually results in better decisions.

Plus, I sleep better knowing I did the work to consider the consequences. Because at the end of the day, the digital world I create is the one my kids will grow up in.

Emotional UX, by Vadym Grin

If you want a deeper dive into the topic of Ethics in tech and design, I devoted a chapter to it in my book Emotional UX. It’s packed with everything I wish someone had told me years ago.

When Pixels Meet Psychology

When I started my career as a digital designer, no one mentioned that I would be enrolled in a Psychology 101 course. Nobody warned me that behind every tap, swipe, and click lives a complex human being with feelings that can make or break my carefully crafted interface.

And then I realized the hard way: users don’t just see your design with their eyes or test it with their fingertips. The real judgment happens somewhere deeper. In that messy, unpredictable place where emotions live.

That’s why pretty layouts and smooth interactions aren’t enough. Your design might be pixel-perfect and your user flow flawless, but if you haven’t considered what your users feel, you’re designing with one hand tied behind your back.

Why Your Users’ Hearts Matter More Than You Think

Early in my career, I clung to the common idea that “Beauty is in the eye of the beholder.” It felt safe. If everyone had different tastes, then I could never really be wrong, right?

Not really.

But how do you create something beautiful when beauty supposedly doesn’t exist? How do you design for everyone when everyone wants something different?

Then I discovered research by Stefan Sagmeister and Jessica Walsh that changed everything. They found that certain aspects of beauty resonate universally. As human-beings we share deep-rooted patterns in how we perceive and respond to visual information. Beauty isn’t entirely random — it follows psychological principles we can understand and apply.

This revelation opened a door I didn’t know existed. If there are universal patterns in how we experience beauty, what about emotions? Fear, joy, trust, frustration — do these follow patterns too?

They do!

The real question isn’t whether digital design is subjective or objective; it’s whether you’re designing for the whole human experience. Can you create something that connects emotionally, builds loyalty, and simply works for everyone?

Designing for Humans, Not Users

The design that creates this connection is called human-centered design. But what does “human-centered” actually mean?

It starts with a simple shift in perspective. Stop seeing people as “users” and start seeing them as what they really are: complex beings with emotions, memories, bad days, and personal quirks.

Your design should feel like a conversation with a thoughtful friend. Someone who notices when you’re in a hurry. Someone who remembers what you care about. Someone who doesn’t make you feel stupid when you make a mistake.

When you design with emotions in mind, something magical happens. Casual users become loyal advocates. One-time visitors turn into die-hard fans. Your product doesn’t just solve problems — it creates connections.

To design products that resonate emotionally, we need to understand the psychology behind our design choices. How do colors affect mood? What makes an interface feel trustworthy? Why do some interactions feel satisfying while others feel frustrating?

The 50-Millisecond Challenge

As designers, we live and die by the emotional responses our work triggers. And the timeline is brutal.

Research by Gitte Lindgaard and her team revealed something that should keep every designer up at night: people form their first impression of a website in just fifty milliseconds. That’s it. Blink and you’ve missed your chance.

Fifty milliseconds to spark trust. Fifty milliseconds to communicate value. Fifty milliseconds to make someone fall in love with your product.

That’s faster than conscious thought. It’s pure emotional reflex — your user’s brain making snap judgments before they even realize they’re judging.

So how do you win someone’s heart in the time it takes a hummingbird to flap its wings twice? You can’t rely on clever copy or detailed explanations. You need to work on every level of the emotional experience.

The Three Levels of Emotional Design

Don Norman, the godfather of user experience design, figured out a framework that breaks this challenge into manageable pieces. He identified three distinct levels where emotional connection happens: visceral, behavioral, and reflective.

The Three Levels of Emotional Design by Don Norman

You may think of these as layers of a relationship. First comes attraction, then compatibility, then meaning.

Visceral Level: Love at First Sight

This is your 50-millisecond moment. Visceral design is a pure, immediate emotional reaction. It’s the part of your brain that gasps “wow” before you even know what you’re looking at.

Visceral design lives in the realm of first impressions. It’s about: color palettes, typography, visual hierarchy, and imagery.

Your visceral design asks: “Am I attractive? Do I feel safe? Do I spark curiosity?”

This level operates completely below conscious awareness. Users can’t tell you why they trust your interface — they just do.

Behavioral Level: The Daily Dance

Once you’ve caught someone’s attention, the behavioral level takes over. This is where usability lives. It’s not about how something looks — it’s about how it feels to use.

The behavioral level covers: intuitive navigation, responsive interactions, error prevention, and performance.

The behavioral design asks: “Am I useful? Am I easy to use? Do I make you feel competent?”

This is where most designers spend their time, and for good reason. Poor behavioral design can kill even the most beautiful interface. But great behavioral design? It builds trust and creates habits.

Reflective Level: The Loyalty Bond

Here’s where things get deeply personal. Reflective design isn’t about the moment someone uses your product — it’s about how they feel about themselves when they use it.

Reflective level in design shows up in: brand values, social features, customization options, stories, and messaging.

The reflective level asks: “What does using this say about me? How does this fit into my life story? What do I tell others about this product?”

Apple masters this level. Using an iPhone isn’t just about having a phone, it’s about being part of a community that values design and innovation. Netflix doesn’t just stream movies, it sells access to a shared cultural moment.

Think about why people choose certain brands even when cheaper alternatives exist. It’s not just about function — it’s about what that choice says about them. Are they creative? Professional? Environmentally conscious? Rebellious?

And here’s why every product desperately wants to reach this level: it’s where true loyalty lives. When users identify with your product on a reflective level, they don’t just use it — they defend it. They recommend it. They become unpaid evangelists who can’t imagine switching to something else.

This is where customers become fans, and fans become part of your extended team.

Visceral Design in Action: The First Impression Makers

Let’s see how this plays out in real apps. Take a look at these three mobile interfaces — each one nails the visceral level, but in completely different ways.

Mobile Apps With a Wow Factor: Stoic, DailyArt and Headspace

The Stoic app (left) whispers calm. Clean white space, gentle curves, and minimal icons create an immediate sense of peace. Before you read a single word, your nervous system relaxes. The design promises serenity. Isn’t this a conducive digital atmosphere to start journaling?

The second one is a daily art discovery app — DailyArt. You’ve probably encountered many educational or cultural apps that promise inspiration but ultimately feel dry or mechanical. Not DailyArt. DailyArt stands out with its elegant simplicity and quiet emotional pull. Each day, it showcases a single piece of artwork, beautifully displayed and paired with a short, captivating story. Its minimal yet refined design allows the art to shine, and its soft colors and typography set a contemplative tone. It’s not just about learning art history; it’s about experiencing emotions such as awe, curiosity, and calm, even if only for a moment. What better way to start your morning than with a masterpiece?

The home screen of the mental health app Headspace (right) radiates warmth and approachability. Those soft, rounded cards with friendly illustrations feel like a hug from a friend. The personalized greeting and cheerful colors signal that this experience will be gentle and supportive.

Same emotional level, three completely different approaches. But they all succeed because they understand something crucial about human psychology.

Building Aww-Inducing Design Solutions

Creating designs that make people go “aww” isn’t about luck — it’s about understanding how human perception works and using that knowledge strategically.

Your brain is a pattern-making machine. Thanks to Gestalt psychology principles, we know that people automatically group visual elements, fill in missing information, and see relationships even where none exist. Elements placed close together feel related. Similar colors create visual families. Empty space builds natural boundaries.

Color triggers instant emotional responses. But remember, these responses aren’t universal. What feels trustworthy in one culture might signal danger in another. The key is knowing your audience and choosing colors that align with the emotions you want to evoke.

Visual rhythm guides emotional flow. Smooth curves suggest gentleness, sharp angles imply precision or urgency, and rounded corners feel friendly and safe. Typography weight and spacing can make content feel either overwhelming or breathable.

Micro-details create macro-feelings. A subtle animation, a perfectly rounded corner, or a carefully chosen shadow can be the difference between a design that feels mechanical and one that feels human.

Behavioral Design: Becoming a Digital Detective

Remember Dr. House? His golden rule was simple: “Everybody lies.” Patients would tell him what they thought he wanted to hear, what made them look better, or what they genuinely believed was true. But House knew better. He watched what they did, not what they said.

As designers, we need to channel our inner Dr. House.

Users will tell you they want more features, faster loading times, or prettier interfaces. But watch what they actually do, and you’ll discover something different. They abandon forms halfway through, ignore your carefully crafted navigation and search instead, and use your product in ways you never imagined.

According to the Interaction Design Foundation, behavioral design applies principles from psychology and economics to influence human behavior. But here’s what that really means: it’s about understanding the gap between what people say they want and what they actually need.

People are wonderfully irrational. We procrastinate on important tasks but spend hours organizing our photos. We claim we value privacy, but share everything on social media. And while we want to be healthier, choose convenience over nutrition.

Great behavioral design works with these contradictions, not against them. It creates products that align with how people actually think and feel, not how we think they should think and feel.

This isn’t manipulation — it’s empathy in action. When you truly understand someone’s behavioral patterns, you can design experiences that feel almost magical. The app that reminds you to drink water right when you’re getting distracted. The interface that makes complex tasks feel simple. The product that somehow knows what you need before you do.

Reflective Design: From Nice-to-Have to Can’t-Live-Without

The reflective level is where products stop being tools and start being extensions of who you are. But here’s the catch: you can’t force your way here. Just like you wouldn’t propose marriage on a first date, you can’t expect users to form deep connections with a broken product. Your design needs to nail the basics first.

Your product must look stunning and work flawlessly before users will let it into their identity. Skip those steps, and reflective design becomes wishful thinking.

And this is where cognitive psychology becomes your backstage pass to the human mind. It shows you how people’s personal realities shape their behavior, how memories influence decisions, and why we sometimes make wonderfully irrational choices that make perfect emotional sense.

Cognitive psychology gives us tools to understand these patterns. How memory works, why certain mental shortcuts lead us astray, and what makes experiences feel enriching rather than just stimulating.

At this level, you’re not just designing interfaces, but the whole identity. You’re creating products that make people feel like they’re leveling up their lives, becoming better versions of themselves, or contributing to something meaningful.

This is where dopamine, your brain’s reward system, comes into action. It’s that warm feeling when you complete a task, discover something new, or connect with others. Dopamine drives motivation and learning, making experiences feel satisfying and memorable.

But here’s where things get tricky. Too much easy dopamine turns products into digital junk food. You know those apps people can’t put down even when they want to? That’s dopamine hijacking, not good design.

So our goal isn’t to create addiction, but to create genuine value that happens to feel rewarding. When someone uses your meditation app and genuinely feels calmer, that dopamine hit reinforces a positive behavior. When your productivity tool actually helps someone accomplish their goals, the satisfaction is earned.

Reflective design succeeds when users can’t imagine reverting to their old way of doing things. Not because they’re hooked, but because your product genuinely supports who they’ve become.

That’s when you’ve moved from nice-to-have to can’t-live-without, and that’s the space where true loyalty lives.

So here’s what I learned from accidentally becoming a psychologist: understanding human emotions, behaviors, and motivations is what separates pretty interfaces from products people genuinely love. Master empathy, behavioral patterns, and emotional design, and you’ll create experiences that don’t just work — they connect.

Emotional UX, by Vadym Grin

If you’re interested in delving deeper into the topic of ethics in tech and design, I devoted a chapter to it in my book, Emotional UX. It’s full of information that I wish I had known years ago.

If you want to start writing technical blogs but are desperately looking for a compelling motive to finally take action, you’ve come to the right place. And if you’re an experienced writer who has fallen out of practice, perhaps a few moments reflecting on the benefits will inspire you to re-prioritize writing.

Here are some of the many reasons to write engineering blog posts…

Leaving your comfort zone

Enjoying small personal challenges is often “a thing” for programmers. It may manifest as:

• Coding daily HackerRank challenges
• Participating in the Advent of Code
• Enduring personal all-night hackathons
• Finishing the 100 pushups program
• Catching up on those long-abandoned piano lessons

… so why not write a blog post or two, just to show yourself that you can do it? Getting started requires an act of willpower, and the outcome is just as satisfying as finally figuring out the asymptotically optimal solution for an algorithmic problem you’ve been trying to solve for hours.

Really understanding your code

Want to see if you really understand your code? Try explaining it to someone else. Writing a blog post can help you with that.

First of all, you need to articulate the problem and solution in natural language. That’s proven to be a helpful debugging mechanism, and even earned an adorable name: rubber duck debugging. Instead of talking to a bathtub toy, you can write your thoughts down for other people to read and learn from.

Writing a blog post sometimes exposes that your code has a terrible logical fallacy, missed a corner case, or was designed incorrectly. Quite often, blogging becomes yet another part of the iterative programming technique. Describing the design and implementation becomes just a phase of development, followed by returning to the code to fix the mistakes and rethink incorrect assumptions.

All those blog posts also improve knowledge retention. If you can just redirect somebody to your blog post instead of explaining your design decisions over and over one-on-one, it saves everyone’s time. And if you move on, it’s nice to know that people who take over the code can onboard themselves with your articles describing how you found and fixed the issues.

You also tend to get your work vetted by the community, as described in the next section.

Free peer review

Code review is a vital part of most successful programming projects. Very few people have the superhero brains capable of hacking a complicated system on their own and maintaining it later. The more people review the code, the better. A small number of reviewers can lead to bias, especially in small startup companies where like-minded people collaborate on a common problem.

Writing a public blog post is an implicit call for (unsolicited) help. One certain thing about the online community of programmers is that they love pointing out other people’s mistakes and suggesting their own ideas as ultimate sources of truth.

Our dear colleague Pekka Enberg often jokingly says: “Don’t read the comment sections on the orange site” [Referring to the distinctive coloring theme of the Hacker News web page]. Although the online commentary is negative more often than not, it’s still just as valuable, maybe even more so. Comment sections on various web pages are accurately called “cesspits” and other derogatory terms. Yet, among all the vitriol, you can often find genuine suggestions on how to improve. It’s quite common for authors to publish an errata for both the blog post and the corresponding code because somebody was kind enough to point out a mistake in the comment section.

Personal brand boost

Thousands of self-proclaimed life coaches infiltrating social media sites try to convince you that improving your personal brand is very important — especially if you want to earn tens of thousands of dollars a day, just like them. Writing blog posts isn’t a direct line to wealth (see “Writing ≠ Riches” below), but it certainly is a good start for earning recognition and respect across the tech community.

Publishing a blog post is usually naturally followed by promoting it on social media. You’ll want to share your own blog post in a non-cringey way (we’ll talk more about this later) and the publisher (often your company) will most likely promote it too. That’s a prime opportunity to slowly build your audience and get noticed. A good blog post can earn you the label of “that person who wrote an interesting article on XYZ.” This builds the foundation for a personal brand.

That “seed” audience helps expand your reach. If you write additional blog posts or comments on topics that resonate with this group, they’re likely to share your posts across their own network. Rinse, repeat, and you’ll be an influencer writing viral content in no time (or, in reality, you’ll gain a reputation for writing blogs worth reading and sharing). Having a large group of regular readers also increases the chances of your blog post landing on the trending list of an article aggregator, like Hacker News or Reddit.

Once you’ve reached some critical mass of exposure, interesting people start approaching you and leaders in that space begin to recognize you as an expert. Blogs are a prime hunting ground for publishers, conference organizers, and other talent scouts. It’s not uncommon for a single successful blog post to spark invitations to meetups, conferences, discussion panels, podcasts, etc. Others might just contact you with a congratulatory note. Regardless of the reason, each such contact is an opportunity to stay in touch with a community peer who cares about what you’ve been working on.

If you have a personal web page, a list of your published blog posts is a great addition to it. And if you don’t have a personal web page, the fact that you can list your published blog posts is a great excuse to go and build one!

Career development

Once you’ve published at least one blog post, there’s one thing you can be certain of: armies of LinkedIn talent sourcers have already found it and are ready to put the title in that first (and only) customized sentence of an otherwise templated message: “Hi [your slightly misspelled name here], I read your article on [the blog post topic], it was such a treat! I am also impressed beyond belief by your experience at (…)”.

But the benefits of blogging go beyond slightly more customized poaching attempts. You’re also standing out for CTOs and founders of startups that work in the same industry, who might have genuinely enjoyed that technical take of yours. When the people reaching out to you have taken the time to read your work and track you down, it’s fair to expect that 1) the position will be well-aligned with your interests and 2) you’ll skip to the front of the hiring line.

Even if your blog doesn’t directly land you new job opportunities, listing compelling blog posts on your resume helps you stand out in a crowded applicant pool. And if one of those blogs links to a Hacker News discussion with an impressive amount of points — or shows some nice engagement via Medium claps, social media reactions, etc. — it’s even more prestigious. Put yourself in the hiring manager’s shoes. Would you rather spend time on someone who has authored an intelligible blog post on a technology that the team uses, or someone who simply tosses the technology name into a long laundry list of “competencies.”

Blogs are also helpful for internal advancement. Wish leaders who don’t read code had some idea what you’ve accomplished and the challenges you faced? This is your chance. Moreover, publishing an impactful blog could help make the case for a promotion. For example, completing and communicating a significant technical achievement is often a key factor for promotion to Staff Engineer. And remember the pain of completing the “what you did this year” section of the dreaded annual performance review forms? Having blog post URLs is quite handy when it’s time to remember what you did and explain complex projects in n characters or less.

Moreover, the writing skills that you develop and refine while working on blogs help your career in other ways too. For example, if you want your design decisions to be adopted, you will need to communicate their merit in writing so your ideas can be reviewed asynchronously by multiple stakeholders. Being able to produce correct code is one thing. If you want to have a greater influence, you often need to convince others (and sometimes yourself as well) that your proposed specification, architecture, and design all make sense.

Staying on top of the latest technologies

Readers turn to tech blogs to learn about modern technology, bleeding edge software, and the latest groundbreaking hardware. When a new CPU architecture gets released, you can expect to see a surge of articles describing how it works (or doesn’t), if it’s a good fit for some specific use cases, if it was a life-changing performance boost for somebody, or if it was rather disappointing after running some real-life benchmarks.

Given that there’s a sizable audience hungry for new content about new technologies, why not volunteer to learn and share something interesting? That’s a very healthy environment for an engineer — staying up to date with modern technology quickly pays off. It not only helps you with your daily duties, but also increases the chances you’re still a valuable asset on the job market in case you ever want (or require) a career change. We all know firsthand that it’s quite tempting to become lazy in a cushy programming job (please don’t tell my boss that I typed that), and regular blogging about the latest technologies is a perfect remedy for this

Weekend programming projects also help — twofold. Of course, they help you stay up-to-date with the latest tech. But what’s more important here, they’re great material for a blog post. A fair number of popular technical blog posts are just descriptions of unusual (e.g. reverse-engineering a Christmas lights controller), nerdy (e.g. a new toy Raft implementation), melancholic (e.g. one about your good ol’ gameboy™), or otherwise impressive weekend projects.

Improving your skills

Blog posts are a perfect training ground for all kinds of skills, not just technical ones.

Writing

Writing proficiency comes with practice. The more you write, the better you become at expressing ideas, structuring content, and crafting clear sentences. Consistent practice (and feedback) contributes significantly to your writing skills. It also helps you find and polish your writing style.

With each post you write, the next one becomes easier and more natural, like another entry in your diary. Every time you write, you’re essentially considering and navigating countless decisions with every word, sentence, and revision. Each negotiation builds up a personal knowledge base that you can subconsciously tap the next time you’re stuck. And confronting your own writing challenges prepares you to provide constructive feedback for other people’s blog posts. That, in turn, creates a healthy writing culture among your colleagues and peers.

Language

Regular writing sessions provide an opportunity to enhance language skills in the context of your work. This is especially valuable for non-native English speakers. You’ll be forced to think hard about vocabulary (both technical and non-technical), grammar, and overall proficiency. It’s a great exercise, especially in the age of automatic grammar-checking software and all kinds of tools for finding synonyms, antonyms, and all the other -nyms. Rather than mechanically accepting suggestions, you learn as you go!

Engineering

Writing a blog post creates internal pressure to verify all the information you’re about to publish under your name. It compels you to research the topic, double-check that the code snippets are sensible and correct, and browse the Internet for similar articles. And that’s great! It is motivation to learn in its purest form — because if you don’t do it, you may make a fool of yourself online. You usually won’t, of course, but try and explain that to your impostor syndrome.

If this post has inspired you to start writing (or write better), you’re not alone. The benefits are clear, but knowing how to craft engineering blog posts that truly resonate with readers and hiring managers is another challenge entirely.

That’s exactly why we published Writing for Developers by Piotr Sarna and Cynthia Dunlop, a practical guide that takes you beyond the “why” and deep into the “how” of engineering blog writing.

Whether you’re writing your first post or looking to make your existing blogs more impactful, this guide gives you the tools to publish engineering content that gets noticed, shared, and remembered.

Stop wondering if you should write that blog post, and learn how to make it count.

You’ve built something beautiful. Your code is elegant, your design is sleek, and your pitch deck has investors nodding appreciatively. You’re convinced you’re the next unicorn.

Then reality hits.
Nobody wants what you’ve built.

Welcome to the brutal truth about product-market fit (PMF) — the single most important milestone in your startup’s journey, and the graveyard where 90% of startups meet their end. Proving PMF is what defines the success of a startup. There are things in a startup you can work on and improve, and there are things you can’t. The most common thing a startup can’t fix later is PMF.

As one battle-tested entrepreneur, Jothy Rosenberg, who’s founded nine
companies puts it: you need to make sure the dogs (i.e. customers) will actually eat (i.e. buy) your dog food (i.e. product).

The Mirage That Kills Dreams

Here’s the uncomfortable truth most founders refuse to acknowledge: having a few customers doesn’t mean you have product-market fit. Getting positive feedback from beta users doesn’t mean you have product-market fit. Even raising a Series A doesn’t mean you have product-market fit.

Product-market fit is that magical moment when your customers are pulling your product out of your hands faster than you can build it. It’s when your biggest problem shifts from, “How do we get customers?” to “How do we serve all these customers banging down our door?”

Marc Andreessen, who coined the term, described it perfectly: “You can always feel when product-market fit isn’t happening. The customers aren’t quite getting value out of the product, word of mouth isn’t spreading, usage isn’t growing that fast, press reviews are kind of ‘blah,’ the sales cycle takes too long, and lots of deals never close.”

Sound familiar? That’s because most startups spend months or even years in this twilight zone, burning through cash while convincing themselves they’re “almost there.”

The Three Deadly Delusions

Delusion #1: “We just need more features”
When customers aren’t biting, the natural instinct is to add more bells and
whistles. More features mean more value, right? Wrong. If your core value
proposition isn’t compelling, adding complexity only makes things worse. You’re not building a Swiss Army knife, you’re building a painkiller for a specific problem.

Delusion #2: “We just need better marketing”
No amount of marketing wizardry can force people to want something they don’t actually need. Marketing amplifies demand; it doesn’t create it from thin air. If you’re struggling to get customers, the problem isn’t your messaging, it’s your product.

Delusion #3: “We just need to find the right customers”
This is perhaps the most dangerous delusion because it contains a grain of
truth. Yes, you need to find your ideal customer profile. But if you’ve been
searching for months and still haven’t found a substantial group of people who desperately want what you’re building, the problem isn’t your customer research, it’s your product.

Signals of Product-Market Fit

Forget the vanity metrics. Here are the signals that actually matter:

Organic Growth That Scares You
When you have true product-market fit, growth becomes your biggest
operational challenge. Customers are referring other customers without being asked. Your servers are crashing from demand, not from bugs. You’re scrambling to hire support staff because you can’t keep up with inbound
requests.

Retention That Defies Logic
Your customers aren’t just using your product — they’re dependent on it. Churn rates approach zero not because customers are satisfied, but because switching would be painful. They’ve integrated your solution into their daily workflow, and removing it would create genuine hardship.

The “Hair on Fire” Test
Your customers should feel genuine pain when your product isn’t available. If your service goes down and customers shrug and find alternatives, you don’t have product-market fit. If they’re calling, emailing, and tweeting frantically until you’re back online, you might be onto something.

The Pivot Moment: When to Hold and When to Fold

Here’s the hardest decision every founder faces: when to persist and when to pivot. Persistence built companies like Twitter (originally a podcasting platform) and Instagram (originally a location-based check-in app). But persistence also kills companies that refuse to acknowledge market reality.

The key is honest measurement. Set clear metrics and deadlines. If you can’t achieve meaningful traction within a reasonable timeframe — usually 6–12 months of focused effort — it’s time to seriously consider a pivot.
Don’t fall into the sunk cost fallacy. The months you’ve spent building your
current product are gone whether you pivot or not. The question is: what’s the best use of your remaining runway?

The Anti-Pattern That Fools Everyone

Perhaps the most dangerous situation is “polite market fit,” when customers kind of like your product, sort of use it occasionally, and give you generally positive feedback. This lukewarm reception feels like progress, but it’s actually quicksand.

Polite market fit generates just enough positive signals to keep you building in the wrong direction. You’ll get renewal rates that aren’t terrible, usage that isn’t great, and growth that isn’t exciting. You’ll convince yourself you’re on the right track because the signals aren’t screaming “failure.”

But mediocre products rarely become great products. They become zombie
companies, neither thriving nor dying, just consuming resources while founders slowly realize their dreams are slipping away.

The Brutal Path Forward

Achieving true product-market fit requires brutal honesty about your current situation. Start by asking yourself these uncomfortable questions:

• If your product disappeared tomorrow, how many customers would be
genuinely distressed?
• Are customers actively recommending your product to others without
incentives?
• When you release new features, do customers actually use them?
• Are you constantly explaining why your product is valuable, or do
customers immediately understand?

If your answers make you uncomfortable, congratulations, you’re ready to do the hard work of finding product-market fit.

The Methodology That Actually Works

Stop building and start listening. Spend the next month having in-depth
conversations with your current customers, lost prospects, and churned users. Don’t ask what features they wan, ask about their problems, their workflows, and their pain points.

Look for patterns in their responses. What problems come up repeatedly? What solutions are they already using? Where are the gaps that cause genuine frustration?

Then — and this is crucial — resist the urge to build immediately. Instead, craft a hypothesis about the core problem you can solve better than anyone else. Define exactly who has this problem and why existing solutions don’t work. Only then should you build, but build the minimum viable solution, not the comprehensive platform you’ve been dreaming about.

The Moment Everything Changes

When you achieve true product-market fit, you’ll know. The conversations with prospects will change from, “Let me explain why you need this” to “When can we get started?” Customer acquisition cost will plummet while lifetime value soars. Growth will shift from a constant struggle to an operational challenge. Most importantly, you’ll stop worrying about whether people want what you’re building. Instead, you’ll be worried about whether you can build fast enough to meet the demand.

That’s when you know the dogs are finally eating your dog food, and coming back for more.

Ready to find your product-market fit but not sure where to start? You don’t
have to navigate it alone. Discover the proven frameworks and battle-tested
strategies that have helped hundreds of startups find their footing in “Tech
Startup Toolkit: How to Launch Strong and Exit Big” — the comprehensive guide for anyone aspiring to start up a technology company, based on decades of business experience and written by Jothy Rosenberg, an entrepreneur who’s been through this journey nine times.

Why Most AI Product Initiatives Fail

A fundamental challenge in AI product development is knowing where AI can create real value. Too many teams build AI “for the sake of AI,” lacking alignment with customer needs or business outcomes. To avoid this trap, The Art of AI Product Development introduces the AI Opportunity Tree, a practical framework to help you generate, evaluate, and prioritize high-impact AI product ideas.

What Is the AI Opportunity Tree?

The AI Opportunity Tree helps product teams map AI benefits to real-world problems. Instead of starting with technology, this approach begins with the unique strengths of AI and matches them to user needs.

Your users don’t care about AI per se, they care about better experiences. Often, they associate AI with risks: job loss, privacy issues, hallucinations. To succeed, you must solve meaningful problems using AI in responsible, impactful ways. As an example, imagine that you are managing a music streaming app. As you explore AI opportunities along these types of benefits, you end up with the following opportunity tree:

Six Categories Where AI Adds Real Value

Each branch of the opportunity tree represents a core benefit of AI:

1. Automation & Productivity: Free up time with AI that handles repetitive tasks (e.g., invoice processing, chatbots).

2. Improvement & Augmentation: Empower users with tools like AI-assisted writing or AI-enhanced diagnostics.

3. Personalization: Deliver real-time, dynamic experiences based on user behavior and preferences.

4. Inspiration & Innovation: Use AI for creative breakthroughs — generating ideas, music, designs, or research insights.

5. Convenience: Simplify interactions with voice interfaces, smart defaults, or predictive features.

6. Emotional Benefits: Create deeper connections with users through empathetic and emotionally aware experiences.

Sourcing Ideas for AI Features

AI innovation doesn’t happen in a vacuum. Here’s how top teams source, validate, and evolve high-potential ideas:

• Customer Insights — Surveys, interviews, support logs
• Internal Brainstorming — Cross-functional collaboration
• Domain Experts — Tap into deep user understanding
• Data Signals — Behavioral analytics, prompt testing
• Internal Crowdsourcing — Source ideas from across your organization
• Dogfooding — Build tools for your own team and learn from usage

Horizontal vs. Vertical AI Opportunities

Understanding your product’s market scope helps guide opportunity selection:

Horizontal AI Products: Solve common problems across industries (e.g., Grammarly).

Vertical AI Products: Address niche needs within a specific sector (e.g., AI for financial services).

Depending on whether your product has a horizontal or a vertical focus, your job and the skillset in your team will look differently. Horizontal products require you to understand a broad market of potentially diverse users and identify common needs shared by all users. By contrast, the market for vertical products is narrower but more demanding in terms of domain knowledge. Thus, you will need to bring rich domain knowledge to the table and likely also fine-tune AI models to the domain.

For an excellent industry breakdown, refer to McKinsey’s 2023 report on generative AI.

Identifying & Prioritizing AI Use Cases

When working with AI, it’s easy to get excited about the numerous automation, augmentation, and personalization opportunities. How can you achieve focus and quickly decide what to tackle first?

To start, once you’ve identified multiple potential use cases, it’s critical to prioritize them in a structured way. Here are four key criteria you can use:

1. User Value: How strongly does the use case address a real user pain point or unlock a desired outcome? Higher value to users means higher impact on product adoption and satisfaction.

2. Business Impact: Does the use case support business goals such as increasing revenue, reducing churn, or improving operational efficiency?

3. Feasibility: Do you have the necessary data, technical resources, and domain knowledge to build this feature effectively? How mature are the AI tools/models required?

4. Strategic Fit: Is this use case aligned with your product vision and long-term roadmap? Does it differentiate your product from competitors?

You can score each use case across these dimensions (e.g., on a scale of 1 to 5), then rank or plot them on a matrix to identify quick wins, long-term bets, and ideas to discard or revisit later. Here is what a matrix might look like using the example music streaming app and its potential for personalized music recommendations:

While the result looks simple and convincing on the surface, this method should be used with a certain caution. Prioritization is a highly context-dependent and fuzzy activity, and molding it into a numeric matrix can hide many relevant details and nuances. Rather, you and your team should acknowledge and respect the complexity and uncertainty of the task. Leaving enough space for doubt will encourage you to stay open to other choices on your opportunity map if your initial decision doesn’t lead to the expected outcome.

It is more important to use the exercise to collect more intelligence and data about each opportunity as it is on the verge of reality. So far, your ideation efforts have lived on paper. Now, things are getting concrete, and this, hopefully positive, pressure can dramatically change the dynamics in your team. Suddenly, assessments and views start to diverge, feasibility doubts are raised, and for some reason, your data scientist needs to double-check the customer feedback on personalization. Let your data scientist be, embrace the rest of the messiness, and encourage everyone to explain and challenge their individual views. At some point, you need to decide which opportunity to tackle next. Even if you don’t reach complete agreement, your team should be clear that you will pivot to the next option if the decision is wrong. Leaving a prioritization session with notes and details about the challenges, technical options, and open questions for each opportunity is more important than creating a sleek but simple prioritization matrix.

Balancing Quick Wins and Long-Term Value

When prioritizing your AI opportunities, you should aim for a healthy mix of short-term wins and long-term growth. The optimal distribution depends on the role of AI in your overall product strategy. If you use AI to stay on track and not fall behind your competitors, you can focus on short-term wins and table stakes features. By contrast, if you want to turn AI into a key differentiator and build a strategic advantage, you should leave enough time and space for more defensible, long-term opportunities.

Two Approaches to Execution: Careful vs. Fast

Most ideas and opportunities are born vague. They indicate a potential direction, but the road ahead is foggy and uncertain. If you completed your prioritization exercise, you probably already added some missing pieces to the puzzle. Now, you need to collect further evidence to explore potential solutions, increase confidence, and shape a vision that motivates your team and your users. Here, you have a choice between two approaches:

“Ready, Aim, Fire” — Ideal for regulated or high-risk environments (e.g., healthcare)

“Ready, Fire, Aim” — Suited for fast-paced, iterative experimentation in consumer apps

These are two different ways to manage risk, speed, and adaptability. Understanding when to use which approach can be key to building successful AI products and features.

Take Action: Build the Right AI Product at the Right Time

The AI Opportunity Tree is more than just an ideation tool, it’s a strategic framework that guides smarter product decisions in the age of AI. By using structured prioritization, you ensure you’re working on the right problems at the right time.

To dive deeper into learning how to build successful AI products in the real world, including efficiently communicating with data scientists and ML engineers, designing user-friendly AI interfaces that emphasize trust and transparency, and implementing safe, ethical AI with proper governance processes, read Dr. Janna Lipenkova’s The Art of AI Product Development.

About the Author

Dr. Janna Lipenkova is the founder of an AI and analytics business where she has successfully managed AI projects for world-class companies like BMW, Lufthansa, and Volkswagen.

Thank you for being a part of the community

Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: X | LinkedIn | YouTube | Newsletter | Podcast | Twitch
• Start your own free AI-powered blog on Differ 🚀
• Join our content creators community on Discord 🧑🏻‍💻
• For more content, visit plainenglish.io + stackademic.com

How to Identify High-Value AI Product Opportunities was originally published in Artificial Intelligence in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

This book is intended to get you up to speed with Go by using it in a series of immediately useful, small-scale projects. If you’re a developer who’s interested in learning Go and doing in a fast and practical way, read on!

Learning Go on the Go

This book aims to get the reader up and running using the Go language on the job, in the context of modern engineering, be it for personal curiosity, as part of a research exercise, or in the context of an industrial project.
We will not cover everything there is to know about the language, but focus instead on the main things we need, as developers, in order to be productive and efficient.
This book provides you with a set of fun projects to progressively explore the features of the Go language. Each pocket-sized project is written in a reasonable number of lines. Our goal is to provide various exercises so any developer who wants to begin with Go or to explore the language can follow the steps described in each chapter.
We want to help the reader become a good modern software developer by using the Go language. We will use our experience as software engineers to provide meaningful advice for newcomers and seasoned developers.
This book also contains tutorials for implementing APIs with microservices, demonstrating how the language is great for cloud computing. It finishes with a project that uses TinyGo, the compiler for embedded systems.
If you are a beginner at programming, we wholeheartedly suggest starting your Go experience with Get Programming with Go.
Let’s see why Go is a great investment of your learning time as a developer.

How and where can Go help you?

Go is a versatile language designed for maintainability and readability. It is optimal for backend software development and has great integration with modern cloud technologies.
Considering the average turnover in tech companies is getting lower every year, a little over a year today, it is important that code written by one person can be read by another after they leave the company. It is therefore crucial that the language chosen by the company aims for readability.
Go’s key features make it a reliable and secure language, with a fast build time.
Some applications leverage goroutines which are a safer and less costly way of dealing with parallel computing than threads. Threads rely on the OS which has a limit related to the size and power of the CPU, whereas goroutines happen at the application’s level. To make the stacks small, Go uses resizable, bounded stacks. A newly minted goroutine is given a few kilobytes, which is almost always enough, and can grow and shrink, allowing many goroutines to live in a modest amount of memory. It is practical to create hundreds of thousands of goroutines in the same address space.
Even though it is not an object-oriented language and has no inheritance system, Go supports most of the features via composition and implicit interfaces. The old argument of Go missing generics, making it verbose and requiring a lot of boilerplate copy paste, has come a long way towards becoming invalid since the famous 1.18 version. As we write, discussions are still in progress to make it as versatile as it is simple.
Go is a compiled language, meaning that all syntax errors will be found during compilation rather than at runtime. We all prefer to know about mistakes in the safety of our own computer rather than discovering them, say, in production.
It is easier to run applications at scale with Go than with many other languages. Go was built by Google to solve problems at a Google-sized scale. It is ideal for large concurrent applications.
Cloud platforms love Go. They provide support for Go as a major language. For example, cloud functions and lambdas support Go in all the most-used providers. Major cloud tools, such as Kubernetes or Docker, are written in Go.

Where can Go NOT help you?

Despite its high versatility, there are few use cases that Go is not made to cover.
Go relies on a garbage collector to release the memory it uses. If your application requires full control over memory, pick a lower-level language like the C family can provide. Go can wrap libraries written in C/C++ with CGo, a translation layer created to ease the transition between the 2 languages. With this CGo twist, you can wrap dynamically-linkable libraries.
The Go toolchain mostly produces executables — generating a Go-compiled library is painfully achievable. We won’t cover it in this book. In most common cases, updating the version of a Go dependency implies rebuilding the binary with that new version. This also means that, in order to use a Go library, you need to have access to its source code.
The Go compiler supports an interesting list of different platforms and operating systems, but we wouldn’t recommend writing an operating system with Go, although many brave souls have done it. The main reason for this is the handling of the memory as it is done in Go: the garbage collector regularly discards bits that are no longer used. As with all garbage collectors, it is adjustable, but it won’t release memory exactly how you want or when you want.
Go binary files are known to be bigger than average. It is usually not a problem in a Cloud environment, but if you require light binaries, consider using the TinyGo compiler.
Last, the difficulty to google up answers — seriously, who names their language with such a common word? Apparently, Google themselves. Here’s a pro tip: when trying to find an answer, use “golang”, which is not the real name but is what search engines will recognize. Sometimes it’s like trying to find documentation in C on strings — you don’t get what you were expecting.
We can also mention the difficulty in hiring Go developers, which is, actually, good for us developers.

Why Pocket Projects?

At the end of the XIXth century, scientists started to theorize learning. Among them, John Dewey wrote in 1897 a long list of good reasons why doing is the best way to learn. Since then, experience has proven his claims in many education systems and learning situations.
The projects that we propose here are timed for busy people. We made sure to keep them as small as possible while still making them rewarding. We admit that some of them are not particularly fun, but these are the most useful in a real-world project.

Whom this book is for

First and foremost, we are writing for developers who know and use another language and would like to extend their professional skills. We want to get you up and running by sharing practical usage of the language.
We focus on industry standards, including long-term considerations and not just a tiny project’s throwaway code that you don’t bother to test.
We are also thinking about teams who are evaluating Go for their next project. Diving into Go’s main features will help you decide if it’s the best language you can possibly invest in.

What you will know after reading the book (and writing the code)

First, we want to make sure that you understand what the book explains. For this matter, we will guide you through our journey of chapters describing the implementation of the current code iteratively, on a commit-by-commit basis, as we consider it important to understand what’s happening bit by bit.
Second, we aim at providing good and clear examples for writing industry-level Go code — recommendations that apply outside our examples, and that will help you venture into the real world of development. All of our examples contain functions that are reusable in a company.
Last, our goal is to make you realize that you can write excellent Go code by yourself once you’ve understood the basics.
We start at the Hello-World level, discovering the syntax of the language, and progress all the way to a service ready to be deployed in the Cloud, walking you through architectural decisions.

Grammar and syntax

The first chapters focus on the grammar specific to Go. For example, how all loops start with the same keyword, breaks are implicit in switches, but also how to expose (and not) some of your constants and methods (what Java calls public or private).
What Go chose in its code design was to make its interfaces implicit. In most of the other big languages, in order to have one entity (or class) be considered as implementing an interface, it needs to explicitly state so in its definition. In Go, implementing the methods is enough. You can therefore unknowingly implement an interface that you don’t know yet. This opens worlds of new possibilities in how we envision mocking and stubbing, dependency injection, and interoperability.
Even though goroutines are a great feature of Go, we won’t dwell on them. In our experience, you can program efficiently in Go without them. Only one project uses them.
Finally, as you will learn throughout the book, Go does not use exceptions. It prefers to consider errors as values. This changes the way we deal with flows that don’t follow the happy path, where nothing ever fails. Every program has to deal with errors at some point, and we’ll cover this throughout the projects.

Testing your code

Each and every chapter includes unit testing.
All of them.
No developer today would dream about delivering code in production that is not covered by at least some tests, whatever their level. They are indispensable for any evolution the software grows through. That is why we include unit tests everywhere.
Go is also great at benchmarking different algorithms with a built-in bench command. It allows developers to compare versions too, which means you can use it to check on every commit that your code-level performance does not decrease. You will see a few examples throughout the book.
One last recent feature of the Go tooling test chain is fuzzing. Fuzzing is a way to test a system by throwing random values at it and seeing how it behaves. It is a great help in checking for vulnerabilities.

Clean code best practices

While the first few projects fit in one file, we’ll quickly need to organize the code in a way that makes it easy to maintain. By maintain, we mean that it should allow a newcomer to find their way through your code in order to fix a bug or add a feature. This fictional newcomer could be you in a very short time.
We suggest and explain some code organization practices. We believe that Go is great for domain-driven design and organize our code accordingly. There is of course no single folder organization for a Go project, but we aim for what makes more sense.
What to expose and what to keep for yourself has tormented humanity for millennia, and software developers for decades. This question is covered as soon as we create something that goes beyond one package.

Architectural decisions

As Go is mostly used for writing services deployed in cloud environments, we added two projects to help you pick your favorite protocol: one serves HTML over HTTP, the other uses protobuf over gRPC. You will write fully-functioning services that you can easily deploy to play around and see what you prefer and what best fits your needs.
Once they are running, you need to monitor what happens in your program. One of the early and easy projects is a logger that goes beyond what the default standard library does. Another one reads an API and acts as an anti-corruption layer to insert the data from that API into your domain. A third one is a simple load balancer for your system’s traffic, that you can make more complex according to your needs.

IoT is fun

The last project is designed to run on an Arduino microcontroller, using a different compiler. It is not enough to make you an embedded systems expert. It will just play as an intro to some of the lesser-known features of Go. We hope to tickle your creativity and hope you will enjoy it all.

Summary

• Go is a modern, industry-oriented, simple and versatile language, best for backend development, widely used for cloud-oriented tools, great for CLI and even adapted to embedded systems.
• Easy to learn, teams are quickly efficient with it, writing complicated frustrating code is possible but not easy.
• This book: learn by doing pocket projects that only take a few hours, gets you up and running.

Learn more about the book here.

Learning Go via Pocket-Sized Projects was originally published in CodeX on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you want to excel in ML and deep learning, you need to know more than how to implement the algorithms — you need to know them inside-out. This book delves into selected algorithms and teaches you how to build your own from scratch.

Read on to learn more.

Why Learn Algorithms from Scratch?

This book dives into the design of machine learning (ML) algorithms from scratch. It builds up algorithms from first principles and takes the reader on a journey from mathematical derivation to software implementation of some of the most exciting algorithms in the field of ML. Understanding ML algorithms from scratch has a number of advantages.
First, it builds the reader’s depth of knowledge of the field. This can be useful when you need to modify an existing algorithm to meet the needs of your application. Second, it allows the reader to choose the right algorithm for the task. By knowing the inner workings of an algorithm, it’s easier to understand the design tradeoffs and choices made in developing the algorithm. Third, the reader will be able to interpret and explain the results of the algorithm to the stakeholders in an industrial or an academic setting. Fourth, the reader will be able to use intuition developed by reading this book to troubleshoot advanced ML problems.
Breaking down a complex problem into smaller pieces and understanding where things went wrong often requires a strong sense of fundamentals and algorithmic intuition. This book will allow the reader to construct minimum working examples and build upon existing algorithms to develop and be able to debug more complex models. Finally, we are often interested in improving the performance of existing models. The principles discussed in this book will enable the reader to accomplish that. To summarize, understanding ML algorithms from scratch will help you choose the right algorithm for the task, explain the results, troubleshoot advanced problems, extend an algorithm to a new situation and improve performance of existing algorithms.

Implementing Algorithms

A key part of learning algorithms from scratch is software implementation. It’s important to write good code that is both efficient in terms of its use of data structures and low algorithmic complexity. While the intention of this book is to write all code from scratch (without reliance on third party libraries), we can still use ML libraries (such as scikit-learn: https://scikit-learn.org/stable/) to check the results of our implementation if available. We’ll be using Python language throughout this book.

Data Structures

A data structure is a way of storing and organizing data. Each data structure offers different performance trade-offs and some are more suitable for the task than others. We’ll be using a lot of linear data structures such as fixed size arrays in our implementation of ML algorithms since the time to access an element in the array is constant O(1). We’ll also frequently use dynamically resizable arrays (such as lists in Python) to keep track of data over multiple iterations.
Throughout the book, we’ll be using non-linear data structures, such as map (dictionary) and set. The reason is that ordered dictionary and ordered set are built upon self-balanced binary search trees (BSTs) that guarantee O(nlogn) insertion, search and deletion operations. Finally, a hash table or unordered map is another commonly used, efficient data structure with O(1) access time assuming no collisions.

Problem-Solving Paradigms

Many ML algorithms that are explored in this book can be grouped into four main problem-solving paradigms: complete search, greedy, divide and conquer, and dynamic programming. Complete search is a method for solving a problem by traversing the entire search space in search of a solution. A machine learning example where complete search takes place is an exact inference by complete enumeration. During exact inference, we must completely specify a number of probability tables to carry out our calculations. A greedy algorithm takes a locally optimum choice at each step with the hope of eventually reaching a globally optimum solution. Greedy algorithms often rely on a greedy heuristic. A machine learning example of a greedy algorithm consists of sensor placement. For example, given a room and several temperature sensors, we would like to place the sensors in a way that maximizes room coverage. Divide and conquer is a technique that divides the problem into smaller, independent sub-problems and then combines the solutions to each of the sub-problems. A machine learning example that uses divide and conquer paradigm can be found in CART decision tree algorithm. As we’ll see in a future chapter, in CART algorithm an optimum threshold for splitting a decision tree is found by optimizing a classification objective (such as Gini index). The same procedure is applied to a tree of depth one greater resulting in a recursive algorithm. Finally, Dynamic Programming (DP) is a technique that divides a problem into smaller, overlapping sub-problems, computes a solution for each sub-problem and stores it in a DP table. A machine learning example that uses dynamic programming occurs in Reinforcement Learning (RL) in finding a solution to Bellman equations. For a small number of states, we can compute the Q-function in tabular way using dynamic programming.

What you will learn in this book

This book is about learning algorithms from the inside out by building them from scratch. You can expect to learn the following:

1. Develop mathematical intuition for classic and modern ML algorithms
2. Learn the fundamentals of Bayesian inference and deep learning
3. Learn about data structures and algorithmic paradigms in ML
4. Learn about supervised, unsupervised and deep learning algorithms
5. Apply algorithms developed in this book to the areas of core ML, natural language processing, computer vision, optimization, computational biology and finance
6. and much more!

Who this book is for

This book is for anyone who wants to learn the inner workings of machine learning and deep learning algorithms and those who want to learn how to design their own algorithms, e.g. data scientists, and ML and deep learning engineers. Knowledge of Python is assumed and experience with ML and deep learning algorithms, tools, and libraries is helpful.

Learn more about the book here.

This article introduces the concept and definition of differential privacy.

Read it if you’re a machine learning engineer, or a developer building around machine learning.

There are many privacy-related threats and vulnerabilities in machine learning, and numerous concepts of privacy-enhancing technologies. In this article, we will introduce a very important privacy protection concept: differential privacy. Differential privacy is one of the most popular privacy protection schemes in use today. It introduces a very interesting concept on how to make a dataset robust enough to changes of any single sample in the dataset, while computing the data statistics (e.g. machine learning models could be considered as certain very complex statistics to describe the distribution of its training data).

What is Differential Privacy?

Data explosion in the Big Data Era results in tons of data that are generated and held by the individual (e.g. a person) and entities (e.g. organizations, companies, and governments), such as the personal images, financial records, medical records, transaction logs and census data. Meanwhile, there is a serious privacy concern when the data leaves the hands of the data owners and participates in certain applications. The AOL search engine log data leak (Shen, Tan, & Zhai, 2007) and Netflix recommendation contest privacy lawsuit (Narayanan & Shmatikov, 2008) show the threats and the need of rigorous privacy-enhancing techniques for machine learning algorithms. Differential privacy (DP) (Cynthia, 2006; Dwork & Roth, 2014), as a promising privacy-enhancing technique, provides a rigorous definition of the privacy and how it can be protected.

The Concept of Differential Privacy

Before introducing the definition of deferential privacy, let us glance through an example.

As shown in Figure 1, Alice and Bob are two news reporters from two different news agencies, that both would like to report the average salary of a private company. This company has a database that contains the personal information (e.g. position, salaries, contacts, etc.) of all its employees. Since the database contains the personal information subject to privacy concerns, any direct access to the database is restricted. To obtain the access, Alice and Bob were required to demonstrate that they planned to follow the company’s protocols for handling personal data, by undergoing confidentiality training and signing data use agreements proscribing their use and disclosure of personal information obtained from the database.
To accommodate the requests from Alice and Bob to report salary statistics of the company on their articles, the company agrees to grant the access to query certain aggregated statistics from the company’s database, including the total number of employees and the average salaries; but not having the permission to access any of the personal information such as Name or Age.
Now let us consider the following scenario. In January 2020, Alice writes an article according to the information obtained from that database, reporting that the private company has 100 employees, and the average salaries is $55,000. Meantime, in February 2020, Bob also writes another article according to the information obtained from that database in the same way, reporting that the private company has 101 employees, and the average salaries is $56,000. The only difference is Alice accessed the database in January whereas Bob accessed it in the February.
Eve is another news reporter working at the third news agency. After reading both articles from Alice and Bob, Eve concludes that between January and February, one new employee has joined that company, and his/her salary is $156,000 (i.e. $56,000 X 101 — $55,000 X 100). Eve then interviews certain employees in that company incognito, and someone told Eve that Mallory joins the company during that time period. Eve then writes another article reporting that Mallory joins the private company and earns $156,000 a year, which is much higher than the average salary of that company.
Now at this point, Mallory’s private information (i.e. salary) has been compromised. Thus, Mallory lodged a complaint with the relevant authorities and planning to sue the company and the reporters. However, on the contrary both Alice and Bob did nothing against the policies, where they just reported the aggregated information which does not contain any personal records.

This story shows a typical privacy leakage scenario, where even though the study, analysis or computation only release the aggregated/statistical information of a database or a dataset, it could lead to certain meaningful but sensitive conclusions about the individuals. So, how can we take care of these kind of issues in a straight but rigorous way? Here comes the differential privacy.
Differential Privacy (DP) attracts lots of attention in the privacy research community in the last decades, which provides a measurement of the information leakage of individual sample from the computation over an underlying dataset. Let us consider the following Figure 2. In DP there is a trusted data curator who gathers the data from multiple data owners to form a dataset. As we discussed in our previous example, the private company is the data curator, and the data owners are the employees of that company. The idea is to perform some computation or an analysis on this collected dataset, such as finding the mean value (e.g. the average salary) so that the data users can access this information without disclosing privacy of the data owners.

Informally, DP aims to strengthen the protection by releasing the aggregated or statistical information of a database or dataset without revealing the presence of an individual in that database. As discussed in the story above, we can consider that the databases in January and February are identical to each other, other than the February database carries Mallory’s information. DP ensures that, no matter on which database the analysis is conducted, the probabilities to get the same aggregated or statistical information or reach to a specific conclusion would be the same (as shown in Figure 3).
In other words, if the data owner’s private data won’t significantly affect the calculation of certain aggregated or statistical information or reach to a specific conclusion from the database, he/she should be less worried about sharing his/her data in the database, since the analysis results could not distinguish whether he/she participates or not (i.e., nearly the same probabilities). This also well explains the “differential” word in the name of DP.
By far, we discussed a general idea about DP. Next, we will introduce how does DP works in real-world scenarios.

How Differential Privacy Works

As shown in Figure 2, to ensure that no one can reliably infer any individual’s information from the computation result, the curator adds random noise (i.e. DP sanitizer) to the result such that the released result would not change if any individual’s information of the underlying data changes. Since no single individual’s information can significantly affect the distribution, adversaries cannot infer such information corresponding to any individual confidently.
Let us continue our previous scenario and see what would happen if the private company had added random noise to the query results (i.e. the total number of employees and the average salaries), before sending them to the news reporters Alice and Bob.

Suppose in our previous scenario, the private company had utilized another advanced private information sharing protocol, as shown in Figure 3, where certain random noises would be added to the aggregated statistics calculated from the company’s database before sharing them to any news reporters.
As such, in January, Alice would write an article according to the information obtained from that database (accessed in January), reporting that the private company has 103 employees (where 100 is the real number and 3 is the added noise), and the average salary is $55,500 (where $55,000 is the real value and $500 is the added noise).
In February, Bob would write another article according to the information obtained from that database in the same way (but accessed in February), reporting that the private company has 99 employees (where 101 is the real number and -2 is the added noise), and the average salary is $55,600 (where $56,000 is the real value and -$400 is the added noise).
At that point, the noisy version of the total number of employees and average salary doesn’t have much effect on the fact of the private company’s information appearing in the public news reports (e.g. the number of employees is around 100, and the average salary is around $55,000-$56,000). However, this could effectively prevent Eve from concluding that between January and February, one new employee joins that private company (since 99–104=-5) and figuring out his/her salary, thus reducing the risk that Mallory’s personal information being revealed by such news reports.

This example enlightens us the idea about how differential privacy works by adding random noise to the aggregated data before publishing it. The next question would be how much noise we should add for each DP application. To answer this question, we would like to introduce the “sensitivity” and “privacy budget” within the DP applications.

THE “SENSITIVITY” OF DP APPLICATIONS

One of the core technical problems in DP is to determine the amount of random noise to be added to the aggregated data before publishing it. The random noise cannot come from an arbitrary random variable.
If the random noise is too small, it cannot provide enough protection to each individual’s private information. For example, if Alice has reported that the company is having 100.1 employees (i.e. +0.1 noise) and the average salary is $55,000.1 (i.e. +$0.1 noise), while in February, if Bob has reported the private company is having 101.2 employees (i.e. +0.2 noise) and the average salary is $55,999.9 (i.e. -$0.1 noise), still Eve could infer that most likely one new employee had joined that company, and his/her salary is around $155,979.9, which is identical to $156,000, the true value.
On the contrary, if the random noise is too large, the published aggregated data would be distorted and meaningless, thus providing no utility. For instance, if Alice reported the private company has 200 employees (i.e. +100 noise) and the average salary is $65,000 (i.e. +$10,000 noise), while Bob reported the private company has 51 employees (i.e. -50 noise) and the average salary is $50,000 (i.e. -$6,000 noise), even though nearly none of the employee’s private information would be revealed, such reports do not carry any meaningful information of the real situation.
So, how could we decide the amount of random noise to be added to the aggregated data before publishing it in a meaningful and scientific way? Roughly speaking, given an application or analysis that need to publish aggregated data on a dataset, the amount of random noise to be added to such aggregated data should be proportional to the largest possible difference that one individual’s private information (e.g. one row within a database table) could make to that aggregated data.
In DP, we call “the largest possible difference that one individual’s private information could make” as the “sensitivity” of the DP application. The “sensitivity” usually measures the maximum possible effect of each individual’s information on the output of the analysis.
For example, in our “private company” scenario, we have two aggregated data, the total number of employees and the average salary, to be published. Since an old employee leaving or a new employee joining the company could at most make “+1” or “-1” difference to the total number of employees, thus, its sensitivity is “1”. For the average salary, since different employees (having different salaries) leaving/joining the company could make different influence to the average salary, and “the largest possible difference” should come from the employee that has the highest possible salary, thus, its sensitivity should be proportional to the “highest salary”.
Usually, it would be difficult to calculate the exact sensitivity for an arbitrary application, and sometimes we need to estimate the approximation of certain sensitivities.

THERE’S NO FREE LUNCH FOR DP — INTRODUCING THE “PRIVACY BUDGET”

As we just discussed, it is important to determine the appropriate amount of random noise to be added while applying DP and such random noise should be proportional to the sensitivity of the applications (e.g. mean estimation, frequency estimation, regression, classification, etc.). But “proportional” is a fuzzy word, that could be a small proportion or a large proportion. So, is there anything else we should consider determining the actual amount of random noise to be added?
Let us first revisit our “private company” scenario that applies DP, where a random noise has been added to the query results (before and after Mallory joining the company) of the database, and the news reporters (i.e. Alice and Bob) would like to estimate the average salary of that company by looking at the “noisy” query results they retrieved. Ideally, in the context of applying DP, such estimation of the average salary should remain the same whether or not an employee, say Mallory, is leaving or joining the company. However, to ensure such property “exactly the same”, it requires the total exclusion of Mallory’s information from this study. If so, we would continue with the same argument and exclude the personal information of every employee in that company’s database. In such case, the estimated average salary cannot rely on any employee’s information, and hence it would be meaningless.
To avoid such dilemma, DP requires that the output of the analysis remain “approximately the same” but not “exactly the same”, with or without Mallory’s information. In other words, DP permits a slight deviation between the output of the analysis with or without any individual’s information. The “permitted deviation” is also considered as the “privacy budget” for DP. If one could tolerate more permitted deviation with or without his/her data, he/she could tolerate more privacy leakage, thus has more privacy budget to spend.
The Greek letter 𝜖 (epsilon) has been utilized to represent such “privacy budget” to quantify the extent of the permitted deviation. As discussed, the privacy budget 𝜖 is usually set by the data owners to tune the level of privacy protection required. A smaller value of 𝜖 results in a smaller permitted deviation (i.e., less privacy budget), and thus is associated with stronger privacy protection but less accuracy for utility. There is no free lunch for DP. For instance, we can set 𝜖 to zero, which permits zero privacy budget and provides perfect privacy protection meaning zero privacy leakage under the definition of DP. The output of the analysis would always be the same no matter whose information has been added or removed. However, as we discussed previously, this also requires ignoring all the information available, thus would not provide any meaningful results. What if we set 𝜖 to 0.1, a small number but larger than zero? The permitted deviation with or without any individual’s information would be small, providing stronger privacy protection while also enabling the data users (e.g., news reporters) to learn something meaningful.
In practice, 𝜖 is usually a small number. For statistical analysis tasks (e.g. mean/frequency estimation), 𝜖 is usually set between approximately 0.001 and 1.0. For machine learning or deep learning tasks, 𝜖 is usually set between somewhere around 0.1 and 10.0.

FORMULATING THE DP SOLUTION FOR THE “PRIVATE COMPANY” SCENARIO

Now that, you probably have a general idea about the concept of DP and how to derive the random noise based on the sensitivity of the application and the data owner determined privacy budget. Let us see how we can apply such techniques in our previous “private company” scenario mathematically.

Go is an ideal language to use to learn about concurrent programming because its creators designed it with high-performance concurrency in mind. The aim was to produce a language that was efficient at runtime, readable, and easy to use. This means that Go has many tools for concurrent programming. Let’s take a look at some of the advantages to using Go for concurrent programs.

Goroutines at a glance

As illustrated in Figure 4, on the left-hand side is the private company’s January database (i.e. before Mallory joining the company) that is queried by Alice, and on the right-hand side is the private company’s February database (i.e. after Mallory joining the company) that is queried by Bob. Now, we would like to derive the differentially private solutions for the private company to share two aggregated data to the public (e.g. Alice and Bob): 1) the total number of employees and 2) the average salary.
Let us start from the easier task, publishing the total number of employees. First, let us, derive the sensitivity. As discussed above, since whether an employee leaving or joining the company, the influence on the total number of employees is 1, thus the sensitivity of this task is 1. Second, we should design the random noise to be added to the total number of employees before publishing it. As we discussed above, the amount of noise should be positively correlated to the sensitivity, and negatively correlated to the privacy budget (since less privacy budget means stronger privacy protection, thus requires more noise). As such, the random noise should be proportional to △f/𝜖, where △f is the sensitivity and 𝜖 is the privacy budget.
In this example, we could draw our random noise from the zero mean Laplace distribution, which is “double-sided” exponential distribution, defined as below.

The Laplace Distribution (centered at μ, usually we use μ = 0) with scale b is defined by the probability density function as:

where the variance of such Laplace Distribution is σ2 = 2b2. We can also consider Laplace distribution as a symmetric version of the exponential distribution.

With that, we can calculate b = △f/𝜖, and draw random noise from Lap(x|△f/𝜖) to be added to the total number of employees. Figure 6 illustrates the published total number of employees of January and February, respectively, while applying different privacy budgets (i.e. 𝜖). In this case, based on the definition of DP, better privacy protection usually means the two published total number of employees of January and February are closer to each other with a higher probability. In other words, it has lower probability to reveal Mallory’s information, by just seeing those two published values. The utility performance usually means the accuracy of the computation of certain aggregated data or statistics from the database or dataset. In this case, it is the difference between the published perturbed value and its real value, and lower difference means better utility performance. In DP, it is usually a trade-off between privacy protection and utility performance. For instance, as 𝜖 decreasing, more noise would be added, and the difference between the two published values would be closer to each other, thus leading to stronger privacy protection. On the contrary, specifying larger 𝜖 could provide better utility but less privacy protection.
Now, let us consider the case of publishing the differentially private average salary. As we discussed above, while any individual employee leaving or joining the company, “the largest possible difference” should come from the employee that has the highest possible salary (i.e. CEO — Jack, $290,000). For instance, in the extreme situation, there is only one employee in the database, and that happens to be the CEO — Jack, who has the highest possible salary, $290,000, and his leaving would result in the “the largest possible difference” of the company’s average salary. Hence, the sensitivity should be $290,000. Then, we could follow the same procedure as dealing with the total number of employees, by adding random noise drawn from lap(x|△f/𝜖).

Learn more about the book here.

References

Cynthia, D. (2006). Differential privacy. Automata, languages and programming, 1–12.
Dwork, C., & Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4), 211–407.
Narayanan, A., & Shmatikov, V. (2008). Robust de-anonymization of large sparse datasets. Paper presented at the 2008 IEEE Symposium on Security and Privacy (sp 2008).
Shen, X., Tan, B., & Zhai, C. (2007). Privacy protection in personalized search. Paper presented at the ACM SIGIR Forum.

Differential Privacy in Action was originally published in Dev Genius on Medium, where people are continuing the conversation by highlighting and responding to this story.