By George Mujuru

1. The problem: why streamed AI still feels slow

Latency is usually the first thing Android teams notice when they add AI features. Five seconds for a chat reply feels slow. Fifteen seconds feels unbearable. That is why streaming, through Server-Sent Events (SSE) and token-by-token updates, has become the default pattern, popularised by ChatGPT.

But streamed text does not necessarily show meaningful progress. Users are not waiting for words to appear; they are waiting for a useful result, whether that is a four-day trip itinerary, a contract summary or a workout draft.

Raw token streaming shows the model “typing”, but not the artefact taking shape. Instead of seeing a plan assemble itself, users watch JSON fragments and formatting noise scroll past — and then suddenly get a complete result. The transition is jarring and breaks immersion. The system still feels slow, even when it is not.

The question, then, is not just how to stream faster. It is how to make users feel that something useful is being built in front of them.

2. The solution: parser-first streaming with WayPoint

Parser-first streaming reverses the usual approach. Instead of accumulating all incoming data and parsing after the stream ends, the parser processes each incoming chunk incrementally and emits structural events as soon as a recognisable unit is ready. The UI updates progressively, titles appear, days fill in, activities slot into place, as the model generates them.

This approach is demonstrated in WayPoint, an open-source Android travel planner built to explore exactly this problem. The project is available at:

github.com/tungamiraimujuru/Waypoint

WayPoint integrates with Anthropic’s Claude API to generate structured travel itineraries as JSON, streams the response via SSE, and renders each piece of the itinerary as it arrives — without ever showing the user a raw JSON fragment.

3. Architecture overview

WayPoint’s architecture started with a simple question: where should the work live? The easiest answer is to do everything inside a single Composable, with networking, streaming state, parsing and rendering all handled in one place. That works for a demo, but it breaks down quickly when you need to test, swap AI providers or reuse logic across screens.

The work is instead split into five focused components:

Orchestrator

Exposes a simple interface: take a request, return a stream of AI events. This shields the rest of the app from which model or provider is in use — whether Anthropic, Gemini, an on-device model or a test fixture.

SSE client

Handles HTTP streaming and is unaware of prompts or domain logic. It streams text deltas and cleans up on cancellation. This keeps business logic out of networking code.

Parser

The heart of the system. A pure, dependency-free component that processes incoming bytes incrementally and emits structural events. It has no Android dependencies and no coroutines, which keeps it reusable and easy to test in isolation.

Reducer

A pure function that takes the current UI state and an event, and returns a new UI state. It lives outside the ViewModel, making logic straightforward to test without any Android frameworks.

ViewModel

Glue code that collects events from the orchestrator, runs them through the reducer and exposes a StateFlow to the UI. It stays small because the heavy lifting is elsewhere.

A few design choices here diverge from typical Android practice. There is no separate use-cases layer — it was an unnecessary ceremony for this scope. A single sealed event stream preserves event ordering. And the parser is kept separate from the network client so that parsing logic stays reusable and isolated.

4. How the parser works

Most streaming implementations buffer all incoming data and parse it once the stream ends. This is simple to implement, but the experience it produces, raw JSON appearing character by character, followed by an instant render, is exactly the problem we are trying to solve.

The WayPoint parser takes a different approach. It is a compact state machine, around 200 lines of Kotlin, that:

• Tracks opening and closing braces to identify when a structural unit (an activity, a day, a title) is complete
• Ignores braces inside strings, to avoid false positives from quoted content
• Strips Markdown code fences that Claude occasionally inserts despite explicit instructions not to — this happened roughly 5% of the time in testing
• Handles truncated buffers mid-string and stays re-entrant by consuming the buffer as it processes

The parser runs in microseconds, has no Android dependencies and is the most defensively coded piece in the project. Here is a simplified version of the core parsing loop:

fun processChunk(chunk: String) { 
    buffer += chunk 
    while (buffer.isNotEmpty()) { 
        when (state) { 
            OUTSIDE_OBJECT -> { 
                val start = buffer.indexOf('{') 
                if (start == -1) {
                 buffer = "";
                 return 
                } 
                buffer = buffer.substring(start) 
                state = INSIDE_OBJECT 
                depth = 0 
                inString = false 
                escaped = false 
            } 
            INSIDE_OBJECT -> { 
                loop@ for (i in buffer.indices) { 
                    val c = buffer[i] 
                    when { 
                        escaped -> escaped = false 
                        c == '\\' && inString -> escaped = true 
                        c == '"' -> inString = !inString 
                        !inString && c == '{' -> depth++ 
                        !inString && c == '}' -> { 
                            depth-- 
                            if (depth == 0) { 
                                emit(buffer.substring(0, i + 1)) 
                                buffer = buffer.substring(i + 1) 
                                state = OUTSIDE_OBJECT 
                                break@loop 
                            } 
                        } 
                    } 
                } 
            } 
        } 
    } 
} 

Each time a complete JSON object is identified, it is emitted as a structural event. The UI reacts to that event immediately, rendering the new element without waiting for the rest of the stream.

5. Compose UI rendering

The parser emits structural events, and the UI has to render them efficiently, sometimes dozens of times per second under heavy streaming load.

The standard Compose pattern of mutating state directly inside the ViewModel became cumbersome at this frequency. The fix was to move state transformation into the pure reducer described above. The ViewModel collects events, applies the reducer and updates a StateFlow. That keeps the ViewModel thin and makes the state transformation logic easy to test independently.

Two Compose-specific optimisations made a significant difference:

• Stable keys on list items prevent unnecessary recomposition when the list grows incrementally
• Separating the itinerary preview from the streaming message panel means only the component that changes gets recomposed

Here is a simplified example of how the ViewModel wires events to state:

viewModelScope.launch { 
    orchestrator.stream(request).collect { event -> 
        _uiState.update { currentState -> 
            reducer(currentState, event) 
        } 
    } 
} 

This pattern kept UI performance smooth under streaming loads, with no dropped frames observed during testing on a mid-range Android device.

6. Claude integration: what worked and what did not

Using Anthropic’s Claude was a straightforward choice given its stable API and reliable structured JSON output. But integration revealed a few practical challenges worth documenting.

Default serialisation omits default values

Kotlin’s default serialisation behaviour omits fields that match their default values. This caused 400 errors because the Claude API expected explicit field names — including the model name — on every request. The fix was to enable encoding of defaults explicitly in the serialisation configuration, ensuring all required fields are always sent.

Retry logic for streaming failures

Streaming can fail mid-response: network drops, 5xx errors and rate limits all occur in practice. WayPoint implements bounded retries on these transient errors. Parse errors and client-side 4xx errors do not trigger retries, to avoid wasting API credits on requests that will not succeed.

Markdown fencing

Despite explicit instructions in the system prompt to return only raw JSON, Claude wrapped its output in Markdown code fences (```json … ```) approximately 5% of the time during testing. The parser strips these defensively. This is a good example of why production LLM integration requires forgiving parsers — even well-instructed models are not perfectly consistent.

7. Try it yourself

Get the code

WayPoint is open-source and available on GitHub:

github.com/tungamiraimujuru/Waypoint

Prerequisites

Before you begin, you will need:

• Android Studio Hedgehog or later
• Kotlin 1.9+
• An Anthropic API key (available at console.anthropic.com)
• Gradle 8.0+

Key dependencies

Add the following to your build.gradle.kts:

// HTTP streaming 
implementation("com.squareup.okhttp3:okhttp:4.12.0") 
// JSON serialisation 
implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.6.3") 
// Coroutines 
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.7.3") 

Getting started

Clone the repository and open it in Android Studio:

git clone https://github.com/tungamiraimujuru/Waypoint.git 
cd Waypoint 

Add your Anthropic API key to local.properties:

ANTHROPIC_API_KEY=your_key_here 

Build and run on a device or emulator running Android API 26 or above. The main entry point for the streaming logic is in StreamingOrchestrator.kt. The parser lives in IncrementalJsonParser.kt — this is the best place to start if you want to adapt the pattern for a different structured output format.

The architecture is intentionally modular. Swapping Claude for a different provider means replacing the SSE client and adjusting the prompt — the parser, reducer and UI layer remain unchanged.

8. Conclusion

The core lesson is straightforward: streaming structured AI output is a user experience problem before it is a networking problem. If users cannot see the result taking shape, the system still feels slow, regardless of actual response times.

A parser-first approach solves this by turning incoming tokens into visible structure. Users do not watch syntax arrive; they watch a plan, a summary or a draft assemble itself step by step.

WayPoint is a prototype, not a production system. It is missing observability tooling, remote configuration for prompts and model settings, provider fallback chains and performance instrumentation. But the architecture already makes room for these without major rework, and the pattern itself is solid enough to build on.

Android still lacks much of the ready-made streaming tooling that exists in the JavaScript ecosystem. That leaves room to experiment, and hopefully WayPoint is a useful starting point for teams trying to make AI on Android feel less clunky and more useful.

How to stream structured AI on Android without sacrificing UX was originally published in DVT Software Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building a Computer Vision AI Bot for Subway Surfers Using ML.NET and OpenCVSharp (in C#)

This walkthrough shows how to build a real-time computer vision pipeline in C# using:

• A .NET 10 console app
• ML.NET object detection model trained with Visual Studio Model Builder
• OpenCvSharp for debug visualization (drawing boxes, saving frames)
• Win32 SendInput for swipe-like mouse gestures

In this article we will create a real-time computer vision pipeline that captures live screen frames, runs them through an ML.NET object detection model, and reacts to predictions in milliseconds — all without leaving the .NET ecosystem

The goal is to show how little code you actually need to go from raw screen pixels to real-time object detection once you have a trained ML.NET model. (excluding the debugging code to help us debug what the model sees and outputs)

Disclaimer

This example is shared for educational purposes to demonstrate real-time computer vision, ML.NET object detection, and input automation techniques in c#. Automating gameplay may violate a game’s terms of service, so please ensure you understand and comply with the relevant policies before experimenting with similar approaches.

What We’re Building

By the end we’ll have:

• A program that captures a region of the screen (your emulator window)
• A trained object detection model that detects obstacles like trains, hurdles, barriers, etc.
• Logic that decides an action (jump/roll/left/right)
• Input automation that performs the action
• Debug images saved to disk showing what the model saw and what the bot decided

Prerequisites

• Windows
• Visual Studio 2022+ with the ML.NET Model Builder extension installed
• An Android emulator running Subway Surfers
• Basic C# knowledge

Step 1: Create a .NET 10 console project and install packages

Run the following in your favourite terminal.

Create the project:

dotnet new console -n SubwaySurferPlayer -f net10.0
cd SubwaySurferPlayer

Add the packages:

dotnet add package Microsoft.ML
dotnet add package Microsoft.ML.ImageAnalytics
dotnet add package OpenCvSharp4
dotnet add package OpenCvSharp4.runtime.win
dotnet add package OpenCvSharp4.Extensions
dotnet add package System.Drawing.Common

Step 2: Set up your emulator

For the emulation, I’ll be using Memu Play 9 but feel free to use whatever emulator you are comfortable with. After setting up Google Play services, install Subway Surfers.

Now we need our program to see the screen.

Step 3: Capture the emulator window

Create ScreenCapture.cs. It provides two capture methods: CaptureScreen returns an OpenCV Mat (handy for direct image processing), and CaptureScreenToStream returns a MemoryStream (what ML.NET's MLImage expects). SaveImage is a thin wrapper around OpenCV's ImWrite.

using OpenCvSharp;
using OpenCvSharp.Extensions;
using System.Drawing;
using System.Drawing.Imaging;

namespace SubwaySurferPlayer
{
    public static class ScreenCapture
    {
        public static Mat CaptureScreen(int x, int y, int width, int height)
        {
            using var bitmap = new Bitmap(width, height, PixelFormat.Format24bppRgb);
            using var graphics = Graphics.FromImage(bitmap);
            graphics.CopyFromScreen(x, y, 0, 0, new System.Drawing.Size(width, height), CopyPixelOperation.SourceCopy);
            return BitmapConverter.ToMat(bitmap);
        }
        public static MemoryStream CaptureScreenToStream(int x, int y, int width, int height)
        {
            using var bitmap = new Bitmap(width, height, PixelFormat.Format24bppRgb);
            using var graphics = Graphics.FromImage(bitmap);
            graphics.CopyFromScreen(x, y, 0, 0, new System.Drawing.Size(width, height), CopyPixelOperation.SourceCopy);
            var stream = new MemoryStream();
            bitmap.Save(stream, ImageFormat.Png);
            stream.Position = 0;
            return stream;
        }
        public static bool SaveImage(Mat image, string filename)
        {
            return Cv2.ImWrite(filename, image);
        }
    }
}

Think of x and y as the top-left corner of your capture rectangle, and width/height as the distance to travel from that point. The top-left corner of your screen is x: 0, y: 0; the bottom-right is your screen resolution, e.g. 1920 x 1080.

Finding your capture coordinates

Figuring out the right capture rectangle can be tricky. Create CaptureHelper.cs as a one-shot diagnostic tool. Run it once and it saves test screenshots of several candidate regions to disk so you can see which one frames your emulator correctly.

using OpenCvSharp;

namespace SubwaySurferPlayer
{
    internal static class CaptureHelper
    {
        public static void RunDiagnostic()
        {
            Console.WriteLine("=== Screen Capture Helper ===");
            Console.WriteLine("This tool helps you find the correct coordinates for your game window.\n");
            var testAreas = new[]
            {
                new { Name = "Default",        X = 900, Y =   0, W = 800, H = 1368 },
                new { Name = "Top-Left",       X =   0, Y =   0, W = 800, H = 1368 },
                new { Name = "Center",         X = 560, Y =   0, W = 800, H = 1368 },
                new { Name = "Full-HD-Center", X = 560, Y = 140, W = 800, H = 1000 },
            };
            Directory.CreateDirectory("capture_tests");
            Console.WriteLine("Capturing test images in 3 seconds...\n");
            Thread.Sleep(3000);
            foreach (var area in testAreas)
            {
                try
                {
                    Console.WriteLine($"Capturing: {area.Name} (X={area.X}, Y={area.Y}, W={area.W}, H={area.H})");
                    using var image = ScreenCapture.CaptureScreen(area.X, area.Y, area.W, area.H);
                    Cv2.PutText(image,
                        $"{area.Name}: X={area.X}, Y={area.Y}, {area.W}x{area.H}",
                        new Point(10, 30),
                        HersheyFonts.HersheySimplex, 1.0, new Scalar(0, 255, 0), 2);
                    string filename = $"capture_tests/{area.Name}.png";
                    ScreenCapture.SaveImage(image, filename);
                    Console.WriteLine($"  -> Saved: {filename}\n");
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"  -> ERROR: {ex.Message}\n");
                }
            }
            Console.WriteLine("Check the images in 'capture_tests/' and update GAME_X/GAME_Y/GAME_WIDTH/GAME_HEIGHT in Program.cs.");
            Console.WriteLine("\nPress any key to continue...");
            Console.ReadKey();
        }
    }
}

To invoke the diagnostic, add this check at the top of Main (shown in full in Step 7):

Log("INFO", "Press 'D' to run screen capture diagnostic, or any other key to start bot...");
var key = Console.ReadKey(true);
if (key.Key == ConsoleKey.D)
{
    CaptureHelper.RunDiagnostic();
    return;
}

Alternatively, take a screenshot with Win + Shift + S, open it in Paint, hover over the top-left corner of the emulator window, and note the X, Y values. Then measure the window's width and height.

Step 4: Train an object detection model using ML.NET Model Builder

This is the “magic sauce” that makes ML.NET feel easy. If you haven’t set up Model Builder yet, follow the Getting Started docs to install the Visual Studio extension.

In Visual Studio:

1. Right-click your project → Add → Machine Learning Model…
2. Choose Object Detection

3. Point it at your dataset (images + labels). I used VoTT for labelling — screenshots of the game with each obstacle type annotated.

4. Choose your label format (I used VoTT)

5. Train. My training took about 5 minutes on a GTX 1060 6 GB. Time will vary based on dataset size and hardware.

6. Evaluate. Upload a test image to verify the model looks reasonable. If you’re happy, move on.

7. Generate code. A code snippet is generated for you, or you can add a console app or web API to the solution with full boilerplate.

After training you’ll have:

• An ObjectDetection.mbconfig file
• A generated ObjectDetection class with:
• ModelInput containing an MLImage Image property
• A Predict(ModelInput) method
• ModelOutput containing PredictedBoundingBoxes, Score, and PredictedLabel

That generated ObjectDetection class is what Program.cs calls.

Step 5: Draw detections and save debug frames

This step is optional but strongly recommended. Being able to see exactly what the model detected and what action the bot decided makes tuning much faster.

Calling Predict on our ObjectDection class calls ML.NET’s PredictionEngine Predict method. This will return a ModelOuput object with the following properies:

• PredictedLabel: A list of names for each thing the model spotted, e.g. “train red” or “hurdle high”.
• Score: A confidence level between 0 and 1 for each of the detections. 0.9 means very sure while 0.3 would be a guess.
• PredictedBoundingBoxes: The location of each detected thing on screen, stored as a flat list of numbers. Every four numbers describes the box like so: [x_topleft, y_topleft, x_bottomright, y_bottomright]
• Labels: (Not useful at runtime, perhaps for a fallback) The index of a label from the list of potential labels, e.g. Labels[i] = 2, PredictedLabel[i] = the string mapped from that index
• Box: (Not useful at runtime) Used in training for holding the VoTT labels we define when creating training data.
• Image: (Not useful at runtime, used in retraining) The input image after it’s been resized to the training dimensions in ObjectDetection.BuildPipeline

You could iterate the model output like below.

0 ≥ i < PredictedLabelLength.

PredictedLabel[i] → the label
Score[i] → the confidence
PredictedBoundingBoxes[i * 4] → x top-left
PredictedBoundingBoxes[i * 4 + 1] → y top-left
PredictedBoundingBoxes[i * 4 + 2] → x bottom-right
PredictedBoundingBoxes[i * 4 + 3] → y bottom-right

Debugging

SaveDetectionsDebugImage decodes the captured frame bytes into an OpenCV Mat, maps bounding boxes into pixel space via NormalizeRect, draws labelled rectangles with a deterministic colour per label, then writes the annotated frame to disk. It lives in Program.cs alongside the rest of the bot logic.

The bounding-box normalisation (NormalizeRect) handles two coordinate systems the model might produce: 0..1 normalised values, or larger model-space coordinates that need to be scaled into capture-space. You'll see its full implementation in Step 7.

SummarizeDetections builds a compact filename-safe string of the top detections so each saved frame self-describes what the model saw. SanitizeFileName keeps the resulting path legal on Windows.

To keep debug saves from stalling the prediction loop, all three methods are called from a background BlockingCollection writer that is set up in Main:

using var debugCts = new CancellationTokenSource();
var debugQueue = new BlockingCollection<Action>(boundedCapacity: maxQueuedDebugFrames);
var debugWriter = Task.Run(() =>
{
    try
    {
        foreach (var work in debugQueue.GetConsumingEnumerable(debugCts.Token))
        {
            try { work(); }
            catch { }
        }
    }
    catch (OperationCanceledException) { }
});

Then instead of calling SaveDetectionsDebugImage directly on the hot path, the frame is captured into local variables and enqueued:

var bytesCopy  = frameBytes;
var boxesCopy  = predictionResult.PredictedBoundingBoxes;
var scoresCopy = predictionResult.Score;
var labelsCopy = predictionResult.PredictedLabel;
var tagCopy    = frameDecisionTag;
var reasonCopy = frameDecisionReason;

if (!debugQueue.IsAddingCompleted)
{
    debugQueue.TryAdd(() =>
    {
        SaveDetectionsDebugImage(
            bytesCopy, detectionsFolder, frameCount,
            boxesCopy, scoresCopy, labelsCopy,
            GAME_WIDTH, GAME_HEIGHT, tagCopy, reasonCopy);
    });
}

The locals are captured into the lambda so the main loop can continue to the next frame immediately. The full implementations of these methods are included in the complete Program.cs in Step 7.

Step 6: Input automation

Because Subway Surfers runs inside an emulator window, the simplest reliable approach is to simulate what your finger would do on a touchscreen: press, drag, release. On Windows we can generate low-level mouse input using the Win32 API SendInput(uint, Input[], int).

SendInput lives in user32.dll, which is unmanaged (native) Windows code. C# runs in managed .NET, so we use P/Invoke (Platform Invocation Services) to call it.

P/Invoke declarations

These go inside the Program class:

[DllImport("user32.dll", EntryPoint = "SetCursorPos")]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool SetCursorPos(int x, int y);

[DllImport("user32.dll")]
public static extern bool GetCursorPos(out POINT lpPoint);

[DllImport("user32.dll", SetLastError = true)]
private static extern uint SendInput(uint nInputs, Input[] pInputs, int cbSize);

[DllImport("user32.dll")]
private static extern IntPtr GetMessageExtraInfo();

[DllImport("User32.dll", ExactSpelling = true, CharSet = CharSet.Auto)]
public static extern int GetSystemMetrics(int nIndex);

Interop structs and enums

These are defined as top-level types outside the namespace so they are available everywhere in the file. The [StructLayout] attributes tell .NET how to lay out the structs in memory so they match what Win32 expects.

[StructLayout(LayoutKind.Sequential)]
public struct POINT
{
    public int X;
    public int Y;
}

[StructLayout(LayoutKind.Sequential)]
public struct KeyboardInput
{
    public ushort wVk;
    public ushort wScan;
    public uint dwFlags;
    public uint time;
    public IntPtr dwExtraInfo;
}
[StructLayout(LayoutKind.Sequential)]
public struct MouseInput
{
    public int dx;           // delta x
    public int dy;           // delta y
    public uint mouseData;   // wheel info when MOUSEEVENTF_WHEEL is set
    public uint dwFlags;     // bit flags for mouse buttons and movement
    public uint time;
    public IntPtr dwExtraInfo;
}
[StructLayout(LayoutKind.Sequential)]
public struct HardwareInput
{
    public uint uMsg;
    public ushort wParamL;
    public ushort wParamH;
}
[StructLayout(LayoutKind.Explicit)]
public struct InputUnion
{
    [FieldOffset(0)] public MouseInput mi;
    [FieldOffset(0)] public KeyboardInput ki;
    [FieldOffset(0)] public HardwareInput hi;
}
public struct Input
{
    public int type;
    public InputUnion u;
}
[Flags]
public enum InputType
{
    Mouse    = 0,
    Keyboard = 1,
    Hardware = 2
}
[Flags]
public enum KeyEventF
{
    KeyDown    = 0x0000,
    ExtendedKey = 0x0001,
    KeyUp      = 0x0002,
    Unicode    = 0x0004,
    Scancode   = 0x0008
}
[Flags]
public enum MouseEventF
{
    Absolute       = 0x8000,
    HWheel         = 0x01000,
    Move           = 0x0001,
    MoveNoCoalesce = 0x2000,
    LeftDown       = 0x0002,
    LeftUp         = 0x0004,
    RightDown      = 0x0008,
    RightUp        = 0x0010,
    MiddleDown     = 0x0020,
    MiddleUp       = 0x0040,
    VirtualDesk    = 0x4000,
    Wheel          = 0x0800,
    XDown          = 0x0080,
    XUp            = 0x0100
}

SendMouseEvent helper

A convenience wrapper that builds a single-element Input array and calls SendInput. It logs a warning if Windows reports that fewer inputs were sent than expected (which can happen if another application is blocking input).

private static void SendMouseEvent(MouseEventF flags, int dx = 0, int dy = 0)
{
    Input[] inputs = new Input[]
    {
        new Input
        {
            type = (int)InputType.Mouse,
            u = new InputUnion
            {
                mi = new MouseInput
                {
                    dx = dx,
                    dy = dy,
                    dwFlags = (uint)flags,
                    dwExtraInfo = GetMessageExtraInfo()
                }
            }
        }
    };
    var sent = SendInput((uint)inputs.Length, inputs, Marshal.SizeOf(typeof(Input)));
    if (sent != inputs.Length)
    {
        var err = Marshal.GetLastWin32Error();
        Log("WARN", $"SendInput(mouse) sent={sent}/{inputs.Length} err={err} flags={flags} dx={dx} dy={dy}");
    }
}

SendKeyboardInput helper

Used at startup to send Alt+Tab to bring the emulator window to the foreground before the bot begins. It accepts an optional second key so the two keys are pressed together as a chord and then released in the correct order.

private static void SendKeyboardInput(ushort key1, ushort? key2 = null)
{
    List<Input> inputs =
    [
        new Input
        {
            type = (int)InputType.Keyboard,
            u = new InputUnion
            {
                ki = new KeyboardInput
                {
                    wVk = key1,
                    dwFlags = (uint)KeyEventF.KeyDown,
                    dwExtraInfo = GetMessageExtraInfo()
                }
            }
        },
    ];

    if (key2.HasValue)
    {
        inputs.Add(new Input
        {
            type = (int)InputType.Keyboard,
            u = new InputUnion
            {
                ki = new KeyboardInput
                {
                    wVk = key2.Value,
                    dwFlags = (uint)KeyEventF.KeyDown,
                    dwExtraInfo = GetMessageExtraInfo()
                }
            }
        });
    }
    if (key2.HasValue)
    {
        inputs.Add(new Input
        {
            type = (int)InputType.Keyboard,
            u = new InputUnion
            {
                ki = new KeyboardInput
                {
                    wVk = key2.Value,
                    dwFlags = (uint)KeyEventF.KeyUp,
                    dwExtraInfo = GetMessageExtraInfo()
                }
            }
        });
    }
    inputs.Add(new Input
    {
        type = (int)InputType.Keyboard,
        u = new InputUnion
        {
            ki = new KeyboardInput
            {
                wVk = key1,
                dwFlags = (uint)KeyEventF.KeyUp,
                dwExtraInfo = GetMessageExtraInfo()
            }
        }
    });
    SendInput((uint)inputs.Count, inputs.ToArray(), Marshal.SizeOf(typeof(Input)));
}

GameAction and PerformAction

GameAction names the four possible swipe directions. PerformAction resets the cursor to screen centre (so every swipe starts from a consistent position), then performs a press-move-release sequence. The preMoveHoldMs delay between mouse down and the drag is important - without it the emulator can interpret the gesture as a tap rather than a swipe.

public enum GameAction
{
    SwipeUp,
    SwipeDown,
    SwipeLeft,
    SwipeRight
}

private static bool PerformAction(GameAction action, int swipeDistance, int swipeDurationMs, out string? error)
{
    error = null;
    var screenWidth  = GetSystemMetrics(0);
    var screenHeight = GetSystemMetrics(1);
    int centerX = screenWidth  / 2;
    int centerY = screenHeight / 2;
    SetCursorPos(centerX, centerY);
    Thread.Sleep(5);
    int preMoveHoldMs = 8;
    int postReleaseMs = 10;
    try
    {
        switch (action)
        {
            case GameAction.SwipeUp:
                SendMouseEvent(MouseEventF.LeftDown);
                Thread.Sleep(preMoveHoldMs);
                SendMouseEvent(MouseEventF.Move, 0, -swipeDistance);
                Thread.Sleep(Math.Max(1, swipeDurationMs));
                SendMouseEvent(MouseEventF.LeftUp);
                Thread.Sleep(postReleaseMs);
                break;
            case GameAction.SwipeDown:
                SendMouseEvent(MouseEventF.LeftDown);
                Thread.Sleep(preMoveHoldMs);
                SendMouseEvent(MouseEventF.Move, 0, swipeDistance);
                Thread.Sleep(Math.Max(1, swipeDurationMs));
                SendMouseEvent(MouseEventF.LeftUp);
                Thread.Sleep(postReleaseMs);
                break;
            case GameAction.SwipeLeft:
                SendMouseEvent(MouseEventF.LeftDown);
                Thread.Sleep(preMoveHoldMs);
                SendMouseEvent(MouseEventF.Move, -swipeDistance, 0);
                Thread.Sleep(Math.Max(1, swipeDurationMs));
                SendMouseEvent(MouseEventF.LeftUp);
                Thread.Sleep(postReleaseMs);
                break;
            case GameAction.SwipeRight:
                SendMouseEvent(MouseEventF.LeftDown);
                Thread.Sleep(preMoveHoldMs);
                SendMouseEvent(MouseEventF.Move, swipeDistance, 0);
                Thread.Sleep(Math.Max(1, swipeDurationMs));
                SendMouseEvent(MouseEventF.LeftUp);
                Thread.Sleep(postReleaseMs);
                break;
            default:
                error = "No action";
                return false;
        }
    }
    catch (Exception ex)
    {
        error = ex.Message;
        return false;
    }
    Thread.Sleep(5);
    SetCursorPos(centerX, centerY);
    return true;
}

Tuning tips

• If swipes don’t register: increase swipeDistance or swipeDurationMs
• If movement is too slow: reduce swipeDurationMs
• If it sometimes taps instead of swiping: increase preMoveHoldMs slightly

Step 7: The complete Program.cs

The main loop performs the following:

1. Capture screenshot of the screen
2. Create instance of ModelInput using the byte array of the screenshot
3. Call predict on ObjectDetection class and passes the output to TryDecideAndPerformAction

TryDecideAndPerformAction

1. Normalize the predicted bounding boxes
2. Determine which lane the player is in by looking at the ‘boy’ detection’s bounding box if present, otherwise assume center.
3. Get player’s Y position, filter detections that overlapp with player and determine the closest obect to the player.
4. Perform the appropriate action based on the closest detection, e.g. if the label is “hurdle high” then perform GameAction.SwipeDown action to roll under the hurdle.

After performing the action, we take care of any needed debugging and logging steps.

With ScreenCapture.cs, CaptureHelper.cs, and the ML.NET generated ObjectDetection class in place, Program.cs is the only remaining file. It ties everything together: setup, the capture/predict/decide loop, the decision logic, the debug image helpers, and all the input automation from Step 6.

using Microsoft.ML.Data;
using OpenCvSharp;
using SubwaySurferPlayer;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Collections.Concurrent;

namespace SubwaySurferPlayer
{
    internal static class Program
    {
        private const float scoreThreshold = 0.43f;
        private const float obstacleTriggerFraction = 0.35f;
        private const bool enableDebugImages = true;
        private const bool enableDebugLogs = false;
        private const int maxQueuedDebugFrames = 3;
        private static readonly object _logLock = new();
        private readonly record struct BotConfig(
            Rect CaptureRegion,
            int SwipeDistancePx,
            int SwipeDurationMs,
            TimeSpan ActionCooldown,
            TimeSpan NoActionDebugImageCooldown,
            string DetectionsFolder);
        private static void Log(string level, string message)
        {
            lock (_logLock)
            {
                Console.WriteLine($"[{DateTime.Now:HH:mm:ss.fff}] [{level}] {message}");
            }
        }
        [DllImport("user32.dll", EntryPoint = "SetCursorPos")]
        [return: MarshalAs(UnmanagedType.Bool)]
        public static extern bool SetCursorPos(int x, int y);
        [DllImport("user32.dll")]
        public static extern bool GetCursorPos(out POINT lpPoint);
        [DllImport("user32.dll", SetLastError = true)]
        private static extern uint SendInput(uint nInputs, Input[] pInputs, int cbSize);
        [DllImport("user32.dll")]
        private static extern IntPtr GetMessageExtraInfo();
        [DllImport("User32.dll", ExactSpelling = true, CharSet = CharSet.Auto)]
        public static extern int GetSystemMetrics(int nIndex);

        static void Main(string[] args)
        {
            Log("INFO", "=== Subway Surfers AI Bot with ML.Net ===");
            Log("INFO", "======================================================");
            // Option to run capture diagnostic
            Log("INFO", "Press 'D' to run screen capture diagnostic, or any other key to start bot...");
            var key = Console.ReadKey(true);
            if (key.Key == ConsoleKey.D)
            {
                CaptureHelper.RunDiagnostic();
                return;
            }
            // Game window configuration
            const int GAME_X = 900;
            const int GAME_Y = 32;
            const int GAME_WIDTH = 800;
            const int GAME_HEIGHT = 1368;
            // Swipe configuration
            const int swipeDistance = 170;   // in pixels
            const int swipeDurationMs = 100; // duration of the swipe
            Log("INFO", "=== Loading AI Model ===");
            // The class labels your model was trained on
            var labels = new[]
            {
                "barrier",
                "boy",
                "crate",
                "crate left",
                "crate right",
                "crate red left",
                "crate red right",
                "hurdle high",
                "hurdle low",
                "pillar",
                "platform",
                "platform left",
                "platform right",
                "ramp",
                "train green",
                "train green left",
                "train green right",
                "train red",
                "train red left",
                "train red right",
                "wall"
            };

            string detectionsFolder = Path.Combine(AppContext.BaseDirectory, "detections");
            Directory.CreateDirectory(detectionsFolder);
            Log("INFO", "✓ AI Bot initialized successfully!");
            Log("INFO", "Instructions:");
            Log("INFO", "1. Start Subway Surfers");
            Log("INFO", "2. Position game window where the bot will capture");
            Log("INFO", "3. Press ENTER to start the bot");
            Log("INFO", "4. Press Ctrl+C to stop");
            Log("INFO", "Press 'R' at any time to restart");
            Console.ReadLine();

            // Background worker to write debug images without stalling prediction/action
            using var debugCts = new CancellationTokenSource();
            var debugQueue = new BlockingCollection<Action>(boundedCapacity: maxQueuedDebugFrames);
            var debugWriter = Task.Run(() =>
            {
                try
                {
                    foreach (var work in debugQueue.GetConsumingEnumerable(debugCts.Token))
                    {
                        try { work(); }
                        catch { }
                    }
                }
                catch (OperationCanceledException) { }
            });

            // Warm up ML.NET prediction engine to avoid first-predict latency spike
            try
            {
                Log("INFO", "Warming up model...");
                using var warmupStream = ScreenCapture.CaptureScreenToStream(GAME_X, GAME_Y, GAME_WIDTH, GAME_HEIGHT);
                var warmupImage = MLImage.CreateFromStream(warmupStream);
                _ = ObjectDetection.Predict(new ObjectDetection.ModelInput { Image = warmupImage });
                Log("INFO", "Model warm-up complete.");
            }
            catch (Exception ex)
            {
                Log("WARN", $"Model warm-up failed (continuing): {ex.Message}");
            }

            // Focus the emulator window
            var screenWidth = GetSystemMetrics(0);
            var screenHeight = GetSystemMetrics(1);
            const ushort ALT_KEY = 0x12;
            const ushort TAB_KEY = 0x09;
            SendKeyboardInput(ALT_KEY, TAB_KEY);
            Thread.Sleep(500);
            SetCursorPos(screenWidth / 2, screenHeight / 2);
            Thread.Sleep(500);

            // Tap to start the game
            SendMouseEvent(MouseEventF.LeftDown);
            SendMouseEvent(MouseEventF.LeftUp);
            Thread.Sleep(1000);
            Log("INFO", "Bot is now running!");
            Log("INFO", $"Capturing area: X={GAME_X}, Y={GAME_Y}, Width={GAME_WIDTH}, Height={GAME_HEIGHT}");
            int frameCount = 0;
            var lastActionTime = DateTime.UtcNow;
            var actionCooldown = TimeSpan.FromMilliseconds(120);
            var lastDebugImageTime = DateTime.UtcNow;
            var debugImageCooldown = TimeSpan.FromMilliseconds(250);

            // Clear old debug images before starting
            foreach (var img in Directory.GetFiles(detectionsFolder, "frame_*.png"))
            {
                try { File.Delete(img); }
                catch { }
            }
            try
            {
                while (true)
                {
                    frameCount++;
                    Stopwatch? stopWatch = null;
                    if (enableDebugLogs)
                    {
                        stopWatch = Stopwatch.StartNew();
                    }
                    using var gameScreen = ScreenCapture.CaptureScreenToStream(GAME_X, GAME_Y, GAME_WIDTH, GAME_HEIGHT);
                    using var capturedBytes = new MemoryStream();
                    gameScreen.CopyTo(capturedBytes);
                    var frameBytes = capturedBytes.ToArray();
                    var image = MLImage.CreateFromStream(new MemoryStream(frameBytes));
                    ObjectDetection.ModelInput sampleData = new ObjectDetection.ModelInput()
                    {
                        Image = image,
                    };
                    if (stopWatch is not null)
                    {
                        stopWatch.Stop();
                        Console.WriteLine($"capture {stopWatch.ElapsedMilliseconds}");
                        stopWatch.Restart();
                    }
                    var predictionResult = ObjectDetection.Predict(sampleData);
                    if (stopWatch is not null)
                    {
                        stopWatch.Stop();
                        Console.WriteLine($"predict {stopWatch.ElapsedMilliseconds}");
                    }
                    if (enableDebugLogs)
                    {
                        Console.WriteLine("\n\nPredicted Boxes:\n");
                    }
                    if (predictionResult.PredictedBoundingBoxes == null)
                    {
                        if (enableDebugLogs)
                        {
                            Console.WriteLine("No Predicted Bounding Boxes");
                        }
                        // Some models can intermittently return null outputs; treat as no detections and continue.
                        continue;
                    }
                    if (enableDebugLogs)
                    {
                        var boxes =
                            predictionResult.PredictedBoundingBoxes.Chunk(4)
                                .Select(x => new { XTop = x[0], YTop = x[1], XBottom = x[2], YBottom = x[3] })
                                .Zip(predictionResult.Score, (a, b) => new { Box = a, Score = b });
                        foreach (var item in boxes)
                        {
                            Console.WriteLine($"XTop: {item.Box.XTop},YTop: {item.Box.YTop},XBottom: {item.Box.XBottom},YBottom: {item.Box.YBottom}, Score: {item.Score}");
                        }
                    }
                    string frameDecisionTag = "";
                    string frameDecisionReason = "";
                    try
                    {
                        if (DateTime.UtcNow - lastActionTime >= actionCooldown)
                        {
                            if (TryDecideAndPerformAction(
                                predictionResult.PredictedBoundingBoxes,
                                predictionResult.Score,
                                predictionResult.PredictedLabel,
                                GAME_WIDTH,
                                GAME_HEIGHT,
                                swipeDistance,
                                swipeDurationMs,
                                out var decidedAction,
                                out var decisionReason,
                                out var error))
                            {
                                lastActionTime = DateTime.UtcNow;
                                Log("INFO", $"Action={decidedAction} reason={decisionReason}");
                                frameDecisionTag = decidedAction.ToString();
                                frameDecisionReason = decisionReason;
                                if (enableDebugImages)
                                {
                                    var bytesCopy  = frameBytes;
                                    var boxesCopy  = predictionResult.PredictedBoundingBoxes;
                                    var scoresCopy = predictionResult.Score;
                                    var labelsCopy = predictionResult.PredictedLabel;
                                    var tagCopy    = frameDecisionTag;
                                    var reasonCopy = frameDecisionReason;
                                    if (!debugQueue.IsAddingCompleted)
                                    {
                                        debugQueue.TryAdd(() =>
                                        {
                                            SaveDetectionsDebugImage(
                                                bytesCopy, detectionsFolder, frameCount,
                                                boxesCopy, scoresCopy, labelsCopy,
                                                GAME_WIDTH, GAME_HEIGHT, tagCopy, reasonCopy);
                                        });
                                    }
                                }
                            }
                            else
                            {
                                if (enableDebugLogs && !string.IsNullOrWhiteSpace(decisionReason))
                                {
                                    Log("DEBUG", $"NoAction reason={decisionReason}");
                                }
                                frameDecisionTag = "NoAction";
                                frameDecisionReason = decisionReason;
                                // Save a throttled debug image even when no action is taken
                                if (enableDebugImages && DateTime.UtcNow - lastDebugImageTime >= debugImageCooldown)
                                {
                                    lastDebugImageTime = DateTime.UtcNow;
                                    var bytesCopy  = frameBytes;
                                    var boxesCopy  = predictionResult.PredictedBoundingBoxes;
                                    var scoresCopy = predictionResult.Score;
                                    var labelsCopy = predictionResult.PredictedLabel;
                                    var tagCopy    = frameDecisionTag;
                                    var reasonCopy = frameDecisionReason;
                                    if (!debugQueue.IsAddingCompleted)
                                    {
                                        debugQueue.TryAdd(() =>
                                        {
                                            SaveDetectionsDebugImage(
                                                bytesCopy, detectionsFolder, frameCount,
                                                boxesCopy, scoresCopy, labelsCopy,
                                                GAME_WIDTH, GAME_HEIGHT, tagCopy, reasonCopy);
                                        });
                                    }
                                }
                                if (!string.IsNullOrWhiteSpace(error))
                                {
                                    Log("WARN", $"Action failed: {error}");
                                    if (enableDebugImages)
                                    {
                                        var bytesCopy  = frameBytes;
                                        var boxesCopy  = predictionResult.PredictedBoundingBoxes;
                                        var scoresCopy = predictionResult.Score;
                                        var labelsCopy = predictionResult.PredictedLabel;
                                        var tagCopy    = frameDecisionTag;
                                        var reasonCopy = $"{frameDecisionReason}__err_{error}";
                                        if (!debugQueue.IsAddingCompleted)
                                        {
                                            debugQueue.TryAdd(() =>
                                            {
                                                SaveDetectionsDebugImage(
                                                    bytesCopy, detectionsFolder, frameCount,
                                                    boxesCopy, scoresCopy, labelsCopy,
                                                    GAME_WIDTH, GAME_HEIGHT, tagCopy, reasonCopy);
                                            });
                                        }
                                    }
                                }
                            }
                        }
                        else
                        {
                            frameDecisionTag = "Cooldown";
                            frameDecisionReason = $"cooldown {(actionCooldown - (DateTime.UtcNow - lastActionTime)).TotalMilliseconds:0}ms";
                        }
                    }
                    catch (Exception ex)
                    {
                        Log("WARN", $"Decision loop error: {ex.Message}");
                        frameDecisionTag = "DecisionError";
                        frameDecisionReason = ex.Message;
                    }
                    if (enableDebugLogs)
                    {
                        try
                        {
                            var summary = SummarizeDetections(predictionResult.PredictedLabel, predictionResult.Score);
                            Log("DEBUG", $"Frame={frameCount} det=[{summary}] decision={frameDecisionTag} reason={frameDecisionReason}");
                        }
                        catch { }
                    }
                }
            }
            finally
            {
                // Graceful shutdown of the debug writer
                try
                {
                    debugQueue.CompleteAdding();
                    debugCts.Cancel();
                    debugWriter.Wait(TimeSpan.FromSeconds(2));
                }
                catch { }
            }
        }
        // -----------------------------------------------------------------------
        // Debug image helpers
        // -----------------------------------------------------------------------
        private static void SaveDetectionsDebugImage(
            byte[] frameBytes,
            string detectionsFolder,
            int frameCount,
            float[]? predictedBoundingBoxes,
            float[]? scores,
            string[]? predictedLabels,
            int imageWidth,
            int imageHeight,
            string decisionTag,
            string decisionReason)
        {
            if (predictedBoundingBoxes is null || scores is null)
            {
                return;
            }
            using var mat = Cv2.ImDecode(frameBytes, ImreadModes.Color);
            if (mat.Empty())
            {
                Log("WARN", "OpenCV failed to decode captured frame; no debug image saved.");
                return;
            }
            var boxCount = predictedBoundingBoxes.Length / 4;
            for (int i = 0; i < boxCount && i < scores.Length; i++)
            {
                var score = scores[i];
                if (score <= 0)
                {
                    continue;
                }
                var rect = NormalizeRect(predictedBoundingBoxes, i, imageWidth, imageHeight, boxCount);
                if (rect.Width <= 0 || rect.Height <= 0)
                {
                    continue;
                }
                string label = (predictedLabels is not null && i < predictedLabels.Length)
                    ? predictedLabels[i]
                    : $"#{i}";
                string caption = $"{label} {score:0.00}";
                // Deterministic colour per label
                var color = new Scalar(
                    (Math.Abs(label.GetHashCode()) & 0xFF),
                    ((Math.Abs(label.GetHashCode()) >> 8) & 0xFF),
                    ((Math.Abs(label.GetHashCode()) >> 16) & 0xFF));
                Cv2.Rectangle(mat, rect, color, thickness: 2);
                var textOrg = new OpenCvSharp.Point(rect.X, Math.Max(0, rect.Y - 5));
                Cv2.PutText(mat, caption, textOrg, HersheyFonts.HersheySimplex, 0.6, Scalar.Black, thickness: 3);
                Cv2.PutText(mat, caption, textOrg, HersheyFonts.HersheySimplex, 0.6, color, thickness: 1);
            }
            var detSummary = SummarizeDetections(predictedLabels, scores);
            var filename = $"frame_{frameCount:000000}__det_{detSummary}__dec_{decisionTag}_{decisionReason}.png";
            filename = SanitizeFileName(filename, maxLength: 180);
            string outputPath = Path.Combine(detectionsFolder, filename);
            if (!Cv2.ImWrite(outputPath, mat))
            {
                Log("WARN", $"OpenCV failed to write debug image: {outputPath}");
            }
        }
        private static string SummarizeDetections(string[]? labels, float[]? scores)
        {
            if (labels is null || scores is null)
            {
                return "none";
            }
            var n = Math.Min(labels.Length, scores.Length);
            if (n == 0)
            {
                return "none";
            }
            bool IsRelevant(string l) => !string.Equals(l, "boy", StringComparison.OrdinalIgnoreCase);
            var top = Enumerable.Range(0, n)
                .Where(i => scores[i] > 0)
                .Select(i => new { Label = labels[i], Score = scores[i] })
                .Where(x => !string.IsNullOrWhiteSpace(x.Label) && IsRelevant(x.Label))
                .OrderByDescending(x => x.Score)
                .Take(3)
                .Select(x => $"{x.Label.Replace(' ', '-')}-{x.Score:0.00}");
            var s = string.Join("_", top);
            return string.IsNullOrWhiteSpace(s) ? "none" : s;
        }
        private static string SanitizeFileName(string name, int maxLength)
        {
            var invalid = Path.GetInvalidFileNameChars();
            var cleaned = new string(name.Select(ch => invalid.Contains(ch) ? '_' : ch).ToArray());
            cleaned = cleaned.Replace("..", ".");
            if (cleaned.Length > maxLength)
            {
                cleaned = cleaned[..maxLength];
            }
            return cleaned;
        }
        // -----------------------------------------------------------------------
        // Detection helpers
        // -----------------------------------------------------------------------
        private readonly record struct Det(int Index, string Label, float Score, Rect Rect);
        private static float IoU(Rect a, Rect b)
        {
            int x1 = Math.Max(a.Left, b.Left);
            int y1 = Math.Max(a.Top, b.Top);
            int x2 = Math.Min(a.Right, b.Right);
            int y2 = Math.Min(a.Bottom, b.Bottom);
            int iw = x2 - x1;
            int ih = y2 - y1;
            if (iw <= 0 || ih <= 0)
            {
                return 0f;
            }
            float inter = iw * ih;
            float union = (a.Width * a.Height) + (b.Width * b.Height) - inter;
            return union <= 0 ? 0f : (inter / union);
        }
        private static Rect NormalizeRect(float[] boxes, int index, int imageWidth, int imageHeight, int boxCount = -1)
        {
            if (boxCount <= 0)
            {
                boxCount = boxes.Length / 4;
            }
            int baseIdx = index * 4;
            float xTop    = boxes[baseIdx + 0];
            float yTop    = boxes[baseIdx + 1];
            float xBottom = boxes[baseIdx + 2];
            float yBottom = boxes[baseIdx + 3];
            // Case 1: normalised 0..1 coordinates
            bool isNormalized01 = xTop >= 0 && xTop <= 1.5f && yTop >= 0 && yTop <= 1.5f
                                && xBottom >= 0 && xBottom <= 1.5f && yBottom >= 0 && yBottom <= 1.5f;
            if (isNormalized01)
            {
                xTop    *= imageWidth;
                xBottom *= imageWidth;
                yTop    *= imageHeight;
                yBottom *= imageHeight;
            }
            else
            {
                // Case 2: model-space coordinates (e.g. up to ~1024). Scale into capture size
                // using the max extents across the whole frame for consistent lane detection.
                float maxX = 0;
                float maxY = 0;
                for (int i = 0; i < boxCount; i++)
                {
                    int bi = i * 4;
                    maxX = MathF.Max(maxX, MathF.Abs(boxes[bi + 0]));
                    maxX = MathF.Max(maxX, MathF.Abs(boxes[bi + 2]));
                    maxY = MathF.Max(maxY, MathF.Abs(boxes[bi + 1]));
                    maxY = MathF.Max(maxY, MathF.Abs(boxes[bi + 3]));
                }
                if (maxX > 0 && maxX > imageWidth * 1.1f)
                {
                    float sx = imageWidth / maxX;
                    xTop    *= sx;
                    xBottom *= sx;
                }
                if (maxY > 0 && maxY > imageHeight * 1.1f)
                {
                    float sy = imageHeight / maxY;
                    yTop    *= sy;
                    yBottom *= sy;
                }
            }
            int left   = (int)MathF.Round(MathF.Min(xTop, xBottom));
            int top    = (int)MathF.Round(MathF.Min(yTop, yBottom));
            int right  = (int)MathF.Round(MathF.Max(xTop, xBottom));
            int bottom = (int)MathF.Round(MathF.Max(yTop, yBottom));
            left   = Math.Clamp(left,   0, imageWidth  - 1);
            top    = Math.Clamp(top,    0, imageHeight - 1);
            right  = Math.Clamp(right,  0, imageWidth  - 1);
            bottom = Math.Clamp(bottom, 0, imageHeight - 1);
            return new Rect(left, top, Math.Max(1, right - left), Math.Max(1, bottom - top));
        }
        private static int LaneFromX(float x, int imageWidth)
        {
            var third = imageWidth / 3f;
            if (x < third) return 0;
            if (x < third * 2) return 1;
            return 2;
        }
        private static int LaneFromX(float x, float laneOriginX, float laneWidth)
        {
            if (laneWidth <= 1)
            {
                return 1;
            }
            var local = x - laneOriginX;
            var idx = (int)MathF.Floor(local / laneWidth);
            return Math.Clamp(idx, 0, 2);
        }
        // -----------------------------------------------------------------------
        // Decision logic
        // -----------------------------------------------------------------------
        private static bool TryDecideAndPerformAction(
            float[]? predictedBoundingBoxes,
            float[]? scores,
            string[]? predictedLabels,
            int imageWidth,
            int imageHeight,
            int swipeDistance,
            int swipeDurationMs,
            out GameAction decidedAction,
            out string decisionReason,
            out string? error)
        {
            decidedAction = default;
            decisionReason = string.Empty;
            error = null;
            if (predictedBoundingBoxes is null || scores is null || predictedLabels is null)
            {
                return false;
            }
            var count = Math.Min(scores.Length, predictedLabels.Length);
            count = Math.Min(count, predictedBoundingBoxes.Length / 4);
            if (count <= 0)
            {
                return false;
            }
            var dets = Enumerable.Range(0, count)
                .Select(i => new Det(i, predictedLabels[i], scores[i], default))
                .Where(d => d.Score >= scoreThreshold)
                .ToArray();
            // Normalise all rectangles in one pass using a consistent scale for the frame
            dets = dets
                .Select(d => d with { Rect = NormalizeRect(predictedBoundingBoxes, d.Index, imageWidth, imageHeight, count) })
                .Where(d => d.Rect.Width > 0 && d.Rect.Height > 0)
                .ToArray();
            // If a ramp overlaps a crate, relabel it as a crate (model sometimes double-detects).
            // Also treat a stand-alone ramp with crate-like proportions as a crate.
            const float rampCrateOverlapIou           = 0.25f;
            const float rampLooksLikeCrateAspectMin   = 0.8f;
            const float rampLooksLikeCrateAspectMax   = 1.6f;
            const float rampLooksLikeCrateAreaFracMin = 0.02f;
            {
                var crates = dets
                    .Where(d => d.Label.StartsWith("crate", StringComparison.OrdinalIgnoreCase))
                    .ToArray();
                dets = dets
                    .Select(d =>
                    {
                        if (!string.Equals(d.Label, "ramp", StringComparison.OrdinalIgnoreCase))
                        {
                            return d;
                        }
                        if (crates.Length > 0)
                        {
                            var best = crates.Max(c => IoU(d.Rect, c.Rect));
                            if (best >= rampCrateOverlapIou)
                            {
                                return d with { Label = "crate" };
                            }
                        }
                        float aspect   = d.Rect.Height <= 0 ? 0 : (d.Rect.Width / (float)d.Rect.Height);
                        float areaFrac = (d.Rect.Width * d.Rect.Height) / (float)(imageWidth * imageHeight);
                        if (areaFrac >= rampLooksLikeCrateAreaFracMin
                            && aspect >= rampLooksLikeCrateAspectMin
                            && aspect <= rampLooksLikeCrateAspectMax)
                        {
                            return d with { Label = "crate" };
                        }
                        return d;
                    })
                    .ToArray();
            }
            if (dets.Length == 0)
            {
                decisionReason = "no detections";
                return false;
            }
            // Determine player lane from the "boy" detection; fall back to centre
            var boy = dets
                .Where(d => string.Equals(d.Label, "boy", StringComparison.OrdinalIgnoreCase))
                .OrderByDescending(d => d.Score)
                .FirstOrDefault();
            // Infer the road boundary from obstacle x extents to reduce UI/border influence
            float roadLeft  = 0;
            float roadRight = imageWidth;
            {
                var xs = dets
                    .Where(d => !string.IsNullOrWhiteSpace(d.Label)
                             && !string.Equals(d.Label, "boy", StringComparison.OrdinalIgnoreCase))
                    .Select(d => (Left: d.Rect.X, Right: d.Rect.X + d.Rect.Width))
                    .ToArray();
                if (xs.Length > 0)
                {
                    roadLeft  = Math.Clamp(xs.Min(v => (float)v.Left), 0, imageWidth - 1);
                    roadRight = Math.Clamp(xs.Max(v => (float)v.Right), roadLeft + 1, imageWidth);
                }
            }
            var laneWidth = (roadRight - roadLeft) / 3f;
            int playerLane = !string.IsNullOrWhiteSpace(boy.Label)
                ? LaneFromX(boy.Rect.X + (boy.Rect.Width / 2f), roadLeft, laneWidth)
                : 1;
            float playerYBottom = string.IsNullOrWhiteSpace(boy.Label)
                ? imageHeight * 0.85f
                : (boy.Rect.Y + boy.Rect.Height);
            // Discard detections that overlap or are behind the player
            if (!string.IsNullOrWhiteSpace(boy.Label))
            {
                const float playerOverlapIouThreshold = 0.15f;
                const float behindPlayerSlackPx       = 10f;
                dets = dets
                    .Where(d =>
                    {
                        if (string.Equals(d.Label, "boy", StringComparison.OrdinalIgnoreCase))
                        {
                            return true;
                        }
                        var dBottom = d.Rect.Y + d.Rect.Height;
                        if (dBottom >= playerYBottom - behindPlayerSlackPx)
                        {
                            return false;
                        }
                        if (IoU(d.Rect, boy.Rect) >= playerOverlapIouThreshold)
                        {
                            return false;
                        }
                        return true;
                    })
                    .ToArray();
            }
            bool IsObstacle(string label) =>
                label.StartsWith("hurdle", StringComparison.OrdinalIgnoreCase)
                || label.StartsWith("train",  StringComparison.OrdinalIgnoreCase)
                || string.Equals(label, "barrier", StringComparison.OrdinalIgnoreCase)
                || string.Equals(label, "wall",    StringComparison.OrdinalIgnoreCase)
                || string.Equals(label, "pillar",  StringComparison.OrdinalIgnoreCase)
                || label.StartsWith("crate", StringComparison.OrdinalIgnoreCase);
            var obstacles = dets
                .Where(d => IsObstacle(d.Label))
                .Select(d => (Det: d, Lane: LaneFromX(d.Rect.X + (d.Rect.Width / 2f), roadLeft, laneWidth)))
                .ToArray();
            if (obstacles.Length == 0)
            {
                decisionReason = "no obstacles";
                return false;
            }
            // Nearest obstacle per lane by yBottom (larger = lower on screen = closer)
            var nearestPerLane = obstacles
                .GroupBy(o => o.Lane)
                .Select(g => g.OrderByDescending(x => x.Det.Rect.Y + x.Det.Rect.Height).First())
                .ToDictionary(x => x.Lane, x => x.Det);
            if (!nearestPerLane.TryGetValue(playerLane, out var nearest))
            {
                var lanes = string.Join(",", nearestPerLane.OrderBy(k => k.Key).Select(k => $"{k.Key}:{k.Value.Label}"));
                decisionReason = $"no obstacle in player lane {playerLane} (roadLeft={roadLeft:0} roadRight={roadRight:0} lanes=[{lanes}])";
                return false;
            }
            var nearestYBottom = nearest.Rect.Y + nearest.Rect.Height;
            // Only react once the obstacle is in the lower portion of the screen
            if (nearestYBottom < imageHeight * obstacleTriggerFraction)
            {
                decisionReason = $"nearest {nearest.Label} too far yBottom={nearestYBottom:0} lane={playerLane} trigger={obstacleTriggerFraction:0.00}";
                return false;
            }
            // Policy: specific obstacles get specific moves
            if (nearest.Label.Equals("hurdle high", StringComparison.OrdinalIgnoreCase))
            {
                decidedAction = GameAction.SwipeDown;
                decisionReason = $"hurdle high in lane {playerLane} yBottom={nearestYBottom}";
                return PerformAction(decidedAction, swipeDistance, swipeDurationMs, out error);
            }
            if (nearest.Label.Equals("hurdle low", StringComparison.OrdinalIgnoreCase)
                || nearest.Label.Equals("barrier", StringComparison.OrdinalIgnoreCase))
            {
                decidedAction = GameAction.SwipeUp;
                decisionReason = $"{nearest.Label} in lane {playerLane} yBottom={nearestYBottom}";
                return PerformAction(decidedAction, swipeDistance, swipeDurationMs, out error);
            }
            // Default: solid obstacle in our lane - try to switch to a free lane
            var close = nearestPerLane
                .Where(kvp => (kvp.Value.Rect.Y + kvp.Value.Rect.Height) >= imageHeight * 0.35f)
                .ToDictionary(kvp => kvp.Key, kvp => kvp.Value.Score);
            int? targetLane = null;
            if (!close.ContainsKey(1)) targetLane = 1;
            if (!close.ContainsKey(0)) targetLane ??= 0;
            if (!close.ContainsKey(2)) targetLane ??= 2;
            if (targetLane.HasValue && targetLane.Value != playerLane)
            {
                decidedAction = targetLane.Value < playerLane ? GameAction.SwipeLeft : GameAction.SwipeRight;
                decisionReason = $"lane change {playerLane}->{targetLane.Value} due to {nearest.Label}";
                return PerformAction(decidedAction, swipeDistance, swipeDurationMs, out error);
            }
            decisionReason = $"blocked; no free lane (playerLane={playerLane}, nearest={nearest.Label} yBottom={nearestYBottom:0})";
            return false;
        }
        // -----------------------------------------------------------------------
        // Input
        // -----------------------------------------------------------------------
        public enum GameAction
        {
            SwipeUp,
            SwipeDown,
            SwipeLeft,
            SwipeRight
        }
        private static bool PerformAction(GameAction action, int swipeDistance, int swipeDurationMs, out string? error)
        {
            error = null;
            var screenWidth  = GetSystemMetrics(0);
            var screenHeight = GetSystemMetrics(1);
            int centerX = screenWidth  / 2;
            int centerY = screenHeight / 2;
            SetCursorPos(centerX, centerY);
            Thread.Sleep(5);
            int preMoveHoldMs = 8;
            int postReleaseMs = 10;
            try
            {
                switch (action)
                {
                    case GameAction.SwipeUp:
                        SendMouseEvent(MouseEventF.LeftDown);
                        Thread.Sleep(preMoveHoldMs);
                        SendMouseEvent(MouseEventF.Move, 0, -swipeDistance);
                        Thread.Sleep(Math.Max(1, swipeDurationMs));
                        SendMouseEvent(MouseEventF.LeftUp);
                        Thread.Sleep(postReleaseMs);
                        break;
                    case GameAction.SwipeDown:
                        SendMouseEvent(MouseEventF.LeftDown);
                        Thread.Sleep(preMoveHoldMs);
                        SendMouseEvent(MouseEventF.Move, 0, swipeDistance);
                        Thread.Sleep(Math.Max(1, swipeDurationMs));
                        SendMouseEvent(MouseEventF.LeftUp);
                        Thread.Sleep(postReleaseMs);
                        break;
                    case GameAction.SwipeLeft:
                        SendMouseEvent(MouseEventF.LeftDown);
                        Thread.Sleep(preMoveHoldMs);
                        SendMouseEvent(MouseEventF.Move, -swipeDistance, 0);
                        Thread.Sleep(Math.Max(1, swipeDurationMs));
                        SendMouseEvent(MouseEventF.LeftUp);
                        Thread.Sleep(postReleaseMs);
                        break;
                    case GameAction.SwipeRight:
                        SendMouseEvent(MouseEventF.LeftDown);
                        Thread.Sleep(preMoveHoldMs);
                        SendMouseEvent(MouseEventF.Move, swipeDistance, 0);
                        Thread.Sleep(Math.Max(1, swipeDurationMs));
                        SendMouseEvent(MouseEventF.LeftUp);
                        Thread.Sleep(postReleaseMs);
                        break;
                    default:
                        error = "No action";
                        return false;
                }
            }
            catch (Exception ex)
            {
                error = ex.Message;
                return false;
            }
            Thread.Sleep(5);
            SetCursorPos(centerX, centerY);
            return true;
        }
        private static void SendMouseEvent(MouseEventF flags, int dx = 0, int dy = 0)
        {
            Input[] inputs = new Input[]
            {
                new Input
                {
                    type = (int)InputType.Mouse,
                    u = new InputUnion
                    {
                        mi = new MouseInput
                        {
                            dx = dx,
                            dy = dy,
                            dwFlags = (uint)flags,
                            dwExtraInfo = GetMessageExtraInfo()
                        }
                    }
                }
            };
            var sent = SendInput((uint)inputs.Length, inputs, Marshal.SizeOf(typeof(Input)));
            if (sent != inputs.Length)
            {
                var err = Marshal.GetLastWin32Error();
                Log("WARN", $"SendInput(mouse) sent={sent}/{inputs.Length} err={err} flags={flags} dx={dx} dy={dy}");
            }
        }
        private static void SendKeyboardInput(ushort key1, ushort? key2 = null)
        {
            List<Input> inputs =
            [
                new Input
                {
                    type = (int)InputType.Keyboard,
                    u = new InputUnion
                    {
                        ki = new KeyboardInput
                        {
                            wVk = key1,
                            dwFlags = (uint)KeyEventF.KeyDown,
                            dwExtraInfo = GetMessageExtraInfo()
                        }
                    }
                },
            ];
            if (key2.HasValue)
            {
                inputs.Add(new Input
                {
                    type = (int)InputType.Keyboard,
                    u = new InputUnion
                    {
                        ki = new KeyboardInput
                        {
                            wVk = key2.Value,
                            dwFlags = (uint)KeyEventF.KeyDown,
                            dwExtraInfo = GetMessageExtraInfo()
                        }
                    }
                });
            }
            if (key2.HasValue)
            {
                inputs.Add(new Input
                {
                    type = (int)InputType.Keyboard,
                    u = new InputUnion
                    {
                        ki = new KeyboardInput
                        {
                            wVk = key2.Value,
                            dwFlags = (uint)KeyEventF.KeyUp,
                            dwExtraInfo = GetMessageExtraInfo()
                        }
                    }
                });
            }
            inputs.Add(new Input
            {
                type = (int)InputType.Keyboard,
                u = new InputUnion
                {
                    ki = new KeyboardInput
                    {
                        wVk = key1,
                        dwFlags = (uint)KeyEventF.KeyUp,
                        dwExtraInfo = GetMessageExtraInfo()
                    }
                }
            });
            SendInput((uint)inputs.Count, inputs.ToArray(), Marshal.SizeOf(typeof(Input)));
        }
    }
}
// -----------------------------------------------------------------------
// Top-level interop types (outside namespace so they are globally visible)
// -----------------------------------------------------------------------
[StructLayout(LayoutKind.Sequential)]
public struct POINT
{
    public int X;
    public int Y;
}
[StructLayout(LayoutKind.Sequential)]
public struct KeyboardInput
{
    public ushort wVk;
    public ushort wScan;
    public uint dwFlags;
    public uint time;
    public IntPtr dwExtraInfo;
}
[StructLayout(LayoutKind.Sequential)]
public struct MouseInput
{
    public int dx;           // delta x
    public int dy;           // delta y
    public uint mouseData;   // wheel info when MOUSEEVENTF_WHEEL is set
    public uint dwFlags;     // bit flags for mouse buttons and movement
    public uint time;
    public IntPtr dwExtraInfo;
}
[StructLayout(LayoutKind.Sequential)]
public struct HardwareInput
{
    public uint uMsg;
    public ushort wParamL;
    public ushort wParamH;
}
[StructLayout(LayoutKind.Explicit)]
public struct InputUnion
{
    [FieldOffset(0)] public MouseInput mi;
    [FieldOffset(0)] public KeyboardInput ki;
    [FieldOffset(0)] public HardwareInput hi;
}
public struct Input
{
    public int type;
    public InputUnion u;
}
[Flags]
public enum InputType
{
    Mouse    = 0,
    Keyboard = 1,
    Hardware = 2
}
[Flags]
public enum KeyEventF
{
    KeyDown     = 0x0000,
    ExtendedKey = 0x0001,
    KeyUp       = 0x0002,
    Unicode     = 0x0004,
    Scancode    = 0x0008
}
[Flags]
public enum MouseEventF
{
    Absolute       = 0x8000,
    HWheel         = 0x01000,
    Move           = 0x0001,
    MoveNoCoalesce = 0x2000,
    LeftDown       = 0x0002,
    LeftUp         = 0x0004,
    RightDown      = 0x0008,
    RightUp        = 0x0010,
    MiddleDown     = 0x0020,
    MiddleUp       = 0x0040,
    VirtualDesk    = 0x4000,
    Wheel          = 0x0800,
    XDown          = 0x0080,
    XUp            = 0x0100
}

At this point your solution should contain three files (ScreenCapture.cs, CaptureHelper.cs, Program.cs) and the ML.NET generated ObjectDetection class. That's all you need to run the bot.

How it all fits together

At runtime the loop does the following on every frame:

1. Capture a PNG of the emulator window region into a byte array
2. Wrap it in an MLImage and call ObjectDetection.Predict(...)
3. Read PredictedBoundingBoxes, Score, and PredictedLabel from the result
4. Pass the predictions to TryDecideAndPerformAction, which:

• Normalises bounding boxes into pixel space
• Infers lane boundaries from obstacle extents
• Determines which lane the player is in from the boy detection
• Filters out detections that overlap or sit below the player
• Picks the nearest obstacle in the player’s lane
• Chooses a swipe direction based on obstacle type and free lanes

5. If an action was chosen, PerformAction executes a press-drag-release via SendInput

6. The annotated debug frame is enqueued for the background writer

The performance of your bot will depend on the size and accuracy of your training data.

The below gif shows the best run my bot could do on the small data set I gave it.

Conclusion

Now we have a working computer vision bot in a few hundred lines of C# — no Python, no cloud API, no game mods. The whole thing runs locally, trained and deployed entirely inside Visual Studio.

The performance of your bot will depend on the size and accuracy of your training data. If the bot is making bad decisions, open the detections folder first. The annotated frames will tell you whether the problem is in the model or the logic.

A few things worth trying if you want to push further:

• More training data for the obstacles the model struggles with most — red crates and partially occluded trains were the worst offenders in my runs
• Lower obstacleTriggerFraction if the bot is reacting too late at higher game speeds
• Detect the game-over screen so the bot can tap to restart automatically and run unattended

Thanks for reading and happy coding.

Building a Computer Vision AI Bot for Subway Surfers Using ML.NET, and OpenCVSharp (in C#) was originally published in DVT Software Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is a homelab?

A homelab is a small-scale, personal IT environment you operate at home for learning, testing ideas, and experimenting with different technologies. It can be as simple as a single low-power device or as complex as a miniature datacenter with multiple servers, switches, and services. This setup allows you to explore virtualisation, networking, automation, and self-hosting in a hands-on way without risking production systems or relying solely on theory.

The setup can be as simple or as complex as you would like.

In this example, the homelab runs as a node within a Tailscale tailnet. Remote devices connect securely through the mesh VPN without port forwarding. Subnet routing can also be enabled to expose the broader home network, allowing access to printers, NAS devices and IoT systems without installing Tailscale on each device.As you expand with new features, hardware, and security layers, your requirements will naturally evolve.

For example, if we wanted to illustrate the Tailscale side of this design, we’d show how remote devices connect through the VPN mesh and interact with the homelab.

Devices outside the home network connect via the Tailscale VPN mesh. Because the homelab is itself a node within the tailnet, remote clients can access it directly. Tailscale can also be configured to advertise the entire home network using subnet routing, effectively exposing all local devices to the mesh and expanding the architecture shown above. This enables Tailscale clients to reach any device within the advertised subnet through the subnet router, without requiring each home device to run Tailscale or become an individual mesh node. This is useful for devices such as printers, NAS, or even IoT devices.

Why you should set up your own homelab

Technology changes quickly, and the best way to keep up is by working directly with the tools you want to master. A homelab gives you a safe place to explore, build, and break things without consequences (mostly).

Whether you’re just getting into IT or already work in the field, a home environment can dramatically accelerate your learning. A homelab is also a good way to make use of old hardware and breathe new life into systems and components that have been lying around, especially with the increasing prices of new hardware. If you don’t have anything lying around, not to worry. You can get up and running with a Raspberry Pi, or you can purchase a refurbished computer for similar pricing to the Raspberry Pi 5.

1. Learn through real experience

Instead of reading explanations or watching demos, you can interact with real systems. A homelab lets you deploy VMs and containers, configure networks, automate tasks, and troubleshoot issues, mirroring what you would encounter in professional environments.

2. Strengthen your career prospects

Practical skills stand out. Having your own lab gives you a place to practice for certifications, experiment with new technologies, and demonstrate your understanding of modern platforms. It also shows initiative and curiosity, qualities employers value.

3. Gain control through self-hosting

If you’d rather avoid depending on third‑party cloud services, a homelab gives you the freedom to host your own. With even a modest setup, you can run a media server like Plex or Jellyfin to stream movies and music, deploy DNS filtering with Pi‑hole to block ads and trackers, manage home automation through Home Assistant, or spin up development environments for coding projects. You can also add monitoring tools such as Grafana and Prometheus to keep an eye on your systems.

By self‑hosting, you reduce monthly subscription costs, gain more control over your data, and tailor your digital ecosystem exactly to your needs. For example, instead of paying for Spotify, Netflix, or cloud backup services, you could cancel those subscriptions and replace them with self‑hosted alternatives like Jellyfin, Nextcloud, or Syncthing to access media and files that you already own.

4. Experiment freely

Curious about a new operating system? Want to try clustering, build a new service, or test backup strategies? Your homelab gives you a safe playground to explore unfamiliar territory and simulate real-world setups, all without risking important data.

5. Start with what you have

You don’t need enterprise gear to begin. Many people start with a Raspberry Pi, an old laptop, or a budget-friendly mini PC and expand gradually. As your interests and knowledge grow, your homelab can grow with you.

6. Stay technically sharp

Running your own systems keeps you in tune with trends in virtualization, automation, networking, containerization, and more. This continuous exposure helps you stay relevant in a rapidly evolving tech landscape.

What can you self-host?

There is a vast ecosystem of self-hosted software available, often as Docker containers or installable directly on Linux. Tools like Portainer simplify managing containers and templates, making it easy to get up and running, especially if you’re new to self-hosting or working with small devices such as Raspberry Pi boards.

I recommend following the pi-hosted series if you are doing this for the first time: https://github.com/novaspirit/pi-hosted. There is also a community that contributes to these Portainer templates to allow beginners to get up to speed quickly.

If you want something more advanced, try setting up Proxmox instead of Portainer.

Below are some of my favourite services to use in a self-host setup:

🎬 Media and entertainment

• Jellyfin — Open‑source media server for movies, shows, photos, and music.
  Highlights: multi‑user streaming, metadata fetching, plugin ecosystem, cross‑device compatibility.
• mStream — Personal music server for streaming your audio collection anywhere.
  Highlights: fast interface, playlists, browser/mobile playback, and optional transcoding.
• Calibre‑Web — Web portal for managing and reading your Calibre eBook library.
  Highlights: browser‑based reading, metadata tools, uploads/downloads, and access control.

🏠 Home automation and networking

• Home Assistant — Central hub for smart home automation.
  Highlights: large integration library, local‑first design, customizable dashboards, automation engine.
• Tailscale — Secure VPN mesh built on WireGuard.
  Highlights: seamless connectivity, encrypted peer‑to‑peer links, identity‑based access controls.
• Pi‑hole — DNS‑level ad and tracker blocker.
  Highlights: detailed analytics, custom lists, optional DHCP server, lightweight resource usage.

🔒 Monitoring and security

• Uptime Kuma — Service monitoring and uptime tracking.
  Highlights: multiple check types, clean dashboards, notifications, and incident tracking.
• Frigate — Lightweight NVR with real‑time object detection.
  Highlights: motion/continuous recording, snapshots, alerts, Home Assistant integration, Docker‑friendly deployment.

📚 Productivity and knowledge management

• Wiki.js — Modern documentation platform for notes or knowledge bases.
  Highlights: Markdown editing, revision tracking, Git integration, theming.
• Paperless‑NGX — Document archiving and OCR search.
  Highlights: tagging, metadata editing, powerful search, automation API.
• Mealie — Recipe organizer and meal‑planning tool.
  Highlights: import recipes, weekly planning, grocery lists, and nutrition data.
• n8n — Workflow automation engine.
  Highlights: visual editor, integrations, event triggers, and custom scripting.
• Whoogle — Privacy‑friendly search interface.
  Highlights: no tracking, clean interface, customizable filters.

💻 System and file management

• Filebrowser — Web‑based file manager.
  Highlights: uploads, edits, permissions, sharing links, API integration.
• Webmin — Browser‑based Linux administration.
  Highlights: manage services, edit configs, monitor metrics.
• Snippetbox — Save and share text/code snippets.
  Highlights: syntax highlighting, expiration options, and simple organization.
• Homarr — Dashboard for organizing homelab services.
  Highlights: drag‑and‑drop tiles, real‑time widgets, and theme support.

⚡ Practical considerations

• Power & UPS: As your lab grows, consider a UPS to protect against outages and calculate electricity costs to balance efficiency with uptime.
• Backup strategies: Follow the 3‑2‑1 rule: three copies of your data, two different media, one offsite. Tools like Syncthing or external drives can help. But also remember that if you can not cover your backup needs fully, any backup is better than no backup.
• Security: Exposing services online requires caution. Use VPNs, reverse proxies with SSL, strong authentication, and regular updates.

Security and remote access

Exposing services directly to the internet carries inherent risks. Rather than opening ports on your router, use a secure access layer such as Tailscale or a properly configured reverse proxy with SSL and strong authentication.

The goal is simple: minimise your attack surface. Keep systems patched, restrict access to authenticated devices, and monitor what you expose.

If you’d like a deeper comparison between traditional VPN setups and Tailscale, you can read our breakdown here: https://medium.com/dvt-engineering/traditional-vpn-vs-tailscale-fd077d943e1f

Pitfalls to avoid

When setting up a homelab, these are the common pitfalls users usually fall into:

• Expanding too quickly without planning
  Avoid rushing to build your entire vision at once. Take it step by step, learn from each stage, and make sure you understand your setup before tackling the next challenge.
• Ignoring backups
  Imagine losing all your data and progress to a sudden hardware or software failure; that’s why backups are essential. Always implement a backup strategy to protect what you’ve built. I recommend a service such as Zerobyte. Do not store your backups of your homelab on your homelab (Thank me later!)
• Poor documentation of ports and configurations
  Keep clear notes on what you’ve configured and how. Good documentation saves time when troubleshooting or rebuilding after a failure.
• Exposing services insecurely
  There are many ways to make your services accessible outside your home network, but not all are safe. Stick to secure options like Tailscale or similar VPN solutions rather than opening raw ports.
• Overlooking electricity and hardware costs
  While power usage may not be a major concern for a small homelab, it’s important to match your hardware to the tasks you expect it to handle. For example, running Pi‑hole on a high‑end gaming PC with a 1000W power supply left on all day is overkill and could become an unnecessarily expensive exercise.

My Homelab Setup

I’ve been building and expanding my homelab for several years, running a variety of services ranging from home automation and backups to camera management and service monitoring.

My primary motivation for running a homelab is to regain control and improve privacy. In fact, I run two homelabs to provide a level of redundancy.

My hardware.

• Main Server:
  Refurbished HP ProDesk 600 G3 SFF
  Intel i5–6500 (6th Gen)
  8GB DDR4 RAM
  256GB NVMe SSD
• Secondary Server:
  Raspberry Pi 5 (8GB)

What were my key requirements and solutions?

Remote access to my servers and services

I need secure access to my servers when I’m away from home. To achieve this, both systems are connected to my Tailscale tailnet, allowing encrypted remote access without port forwarding.

For full desktop access, I pair this with RustDesk. This gives me remote control capabilities without having to expose additional services publicly. Combined with the Homarr dashboard, I can manage and monitor my environment from anywhere.

Privacy and ad blocking (Network-wide & remote)

Privacy and ad-free browsing are a priority for me, so I run Pi-hole on both my main and secondary servers. This not only enhances my security and privacy, but also improves overall network performance, as browsers and apps no longer need to load advertisements or allow tracking services to “phone home.”

- My home router is configured to use both servers as DNS gateways, ensuring Pi-hole is always active across my entire home network.

- To extend ad blocking beyond my home network, I use Tailscale’s DNS configuration so that connected devices resolve traffic through my Pi-hole instances.

As a result, any device connected to Tailscale automatically benefits from network-level ad blocking, helping me maintain control and protect my privacy wherever I am.

Centralised camera management

I wanted a single dashboard to manage all my network cameras. I use Frigate to handle RTSP streams for all cameras and enabled object detection for people and cars, along with motion detection.

My setup is optimised so that:

- Recording only occurs when a person or a car is detected, and there is motion.

- This prevents unnecessary recordings, such as continuous footage of my own car in the garage.

I have a version of the config that I updated in my GitHub gist section, which you can find here.

Personal media access

Like many people, I wanted easy access to my own movies, music, photos, and books without relying on external drives or physical media.

- Photos: PhotoShow (basic) and PhotoPrism (advanced with face/object recognition and smart albums)

- Movies & series: Jellyfin, streaming from external storage connected to my server

- Music: mStream for streaming my personal music collection

- Books: calibre-web for organizing and reading books online (a requirement from my wife)

In my case, I ripped my own CDs and DVDs so all media could be stored and accessed digitally from my server.

Server configuration backups

To protect my server configurations, I use Zerobyte to back up configs to:

- An external drive

- A Samba network share

This ensures I can quickly restore or recover in case of failure and can easily be extended to more storage solutions if needed.

Smart home & automation

For managing smart devices and automation, I use Home Assistant. It allows me to control and automate various parts of my home, including:

- Swimming pool filter timer

- See my solar system stats and have automation set up around it. I use Solar Assistant running on a Raspberry Pi connected to the inverter. The stats are then fed into my Home Assistant through integration options.

- Geyser timer

- Various other smart devices and automations

Home Assistant acts as the central hub for my smart home and ties everything together.

There are many more services that I self-host to meet other needs, although these were my main requirements up to this point.

Conclusion

Setting up a homelab is one of the most effective ways to expand your technical skills while gaining more control over your digital environment. With countless services to self‑host, it gives you a private space to experiment, replicate real‑world setups, and grow at your own pace.

But success isn’t just about what you build, it’s also about how you manage it. Protect your investment with reliable backups, safeguard your systems by exposing services securely, and plan for practical considerations like electricity usage, hardware costs, and power protection with a UPS. Avoid common pitfalls such as expanding too quickly or neglecting documentation, and you’ll create a lab that’s not only powerful but sustainable.

Your homelab becomes what you make it. Start small, learn through each step, plan carefully, and protect what you’ve built. Over time, you’ll gain both confidence and resilience, turning your homelab into a lasting foundation for curiosity, growth, and technical mastery.

Building a Homelab: Elevate your Home Setup and Tech Skills was originally published in DVT Software Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Joining a new organisation as a business analyst, particularly as a consultant, often comes with the expectation that you are “here to make things happen”. You may find yourself starting a new project or taking over one that has stalled. Either way, the pressure to deliver quickly is familiar to many consultants.

To support business analysts who are beginning their journey with cloud data migration, I have outlined several practical guidelines and considerations that can make the work clearer, more structured, and more collaborative.

This guide covers the key steps and best practices to follow when preparing for and executing a cloud data migration project.

Understand your data sources

A successful migration begins with knowing your data in detail:

· Where the data currently resides (on-prem environment, system, or platform)

· What format it is in (structured, semi-structured, or unstructured)

· How it will be accessed (manually, via APIs or services, or through automated pipelines)

Common types of data sources:

Understanding these fundamentals early helps you identify potential risks, constraints, and transformation needs.

Define the ingestion and transformation flow

Once the data sources are known, the next step is to map out exactly how the data will move into the cloud and what needs to happen along the way.

Consider the following:
• How will the data be received? Pushed, pulled, or manually uploaded?
• What transformation logic is required? i.e. Fields to map, normalisation rules, enrichment steps, or data cleansing?
• What should the final state look like? i.e. The expected structure, schema, or model in the cloud environment?

A complete end-to-end flow typically includes:

· Raw ingestion layer

· Transformation or staging layer

· Final curated layer for business consumption

Documenting this flow early reduces confusion and supports smoother hand-offs between teams.

Visualise the data flow

A visual representation makes the ingestion process easier to understand, maintain, and scale as new data sources are added.

Useful tools include:

· Lucidchart

· Microsoft Visio

A well-designed diagram becomes a reliable reference point for developers, analysts, new team members, and business stakeholders

Prioritise strong documentation

Clear documentation is vital in any data migration. Each data source should include:

· A description of the processes involved

· A mapping document showing every attribute from source to destination

· Notes on business rules, exceptions, and dependencies

Good documentation becomes invaluable during testing, sign-off, handover, and future changes.

Where the Business Analyst adds real value

Data migration can be uncomfortable for teams, particularly when long-established processes begin to change. People may feel uncertain or reluctant as they move from known routines to new ways of working. As a business analyst, your role is to help make this transition smoother by documenting both the current state (AS-IS) and the future state (TO-BE) clearly and communicating these effectively.

In some organisations, AS-IS documentation may be outdated or missing entirely. This presents an opportunity to document existing processes accurately before defining the future state.

You will often work closely with external delivery partners responsible for the technical execution of the migration. Your role is to engage confidently, ask the right questions, build strong working relationships, and ensure that internal teams are fully equipped to operate the new processes end-to-end once the migration is complete.

Always refer back to the existing processes and use cases to ensure that all roles, touchpoints, and hand-offs have been considered. In some instances, tasks previously performed by multiple individuals may be consolidated into a single role in the new environment. This must be identified and clearly documented.

Example use cases

· As a developer, I want to ingest data from a database extract so that I can stage it for transformation and loading into the target system.

· As a development manager, I want to manage access permissions and monitor storage requirements so that I can approve resource usage and associated costs.

What went wrong?

In a previous migration project, a key lookup table transformation was overlooked during the ingestion process. The reason? The transformation requirement was not documented clearly, a crucial measure was missing from a final business report.

This reinforces the importance of thorough, validated documentation across every stage of the process: source, transformation, landing, and consumption.

What could have been done better?

· Identify and document all touchpoints and dependencies

· Define exception and validation rules for each stage

· Allow space to identify issues early and correct them quickly

· Introduce sign-off gates at each stage of the migration lifecycle

Learnings for the BA role

· Be willing to learn and deepen your technical understanding. Interpreting data is essential.

· Ensure that every stage of the migration has clear reviewers and approvers.

· Document all data attributes at source, transformation, and destination layers.

· Remember that these guidelines represent only some of the key considerations in a data migration project.

Data migration is complex, but with structured processes, clear communication, and strong documentation, the business analyst plays a crucial role in ensuring a successful transition.

Data migration from a business analyst’s perspective was originally published in DVT Software Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

It has been a few years since Google’s Jetpack Compose made its stable debut, and in that time, it has completely changed how Android apps are built. What began as a modern alternative to the traditional View system has now become the go-to toolkit for Android developers around the world.

In this blog, we will take a closer look at how Compose has grown and where it stands today. We will look at how widely it has been adopted across the industry, with examples from major apps and development teams that have embraced it fully.

So, whether you are already using Compose or thinking about giving it a try, this post will give you a holistic view of where things stand.

Goodbye XML, Hello Compose

Jetpack Compose was designed to simplify and accelerate UI development on Android. The old view system used XML layouts combined with imperative logic in activities and fragments. This resulted in a lot of boilerplate code, difficult state management and UI inconsistencies as the applications grew more complex.

In response to these limitations, Compose was developed. The key reasons were:
• Declarative UI: Build UI with functions that react to state changes.
• Less Boilerplate: No more XML, findViewById, the chaos of adapters.
• Unified Toolkit: Everything from layouts to animations is handled in Kotlin.
• Faster Iteration: Live previews, hot reload, and simpler testing.

In 2025, Jetpack Compose has improved in several core areas:
• Live Edit 2.0 brings real-time updates to stateful composables and animations, significantly improving developer productivity.
• Compose Multiplatform is now stable and production-ready for Android and desktop, with web support rapidly evolving.
• Material 3 feels complete, offering robust theming and accessibility out of the box.
• Animation APIs enable smoother transitions and improved control, helping developers detect and resolve jank earlier in the development process.
• UI Testing Tools have matured with visual diffing, semantic matchers, and flaky test detection, making test coverage more reliable.

Today, compose has matured into a powerful and stable solution used to build many applications. What started as a newcomer in the Android development world has now become the norm for modern UI development.

A Good Fit for Modern Android

While improvements are still being made, Jetpack Compose has kept up with the needs of modern Android development. It works well for building modern interfaces, including smart features powered by AI, apps for smartwatches using Wear Compose, and flexible layouts for foldable phones and large screens. It’s also used in embedded and IoT environments, like car displays and smart dashboards, and performs surprisingly well.

Compose Multiplatform goes beyond mobile. Support for desktop, iOS and Android is stable, while web support is still in alpha. However, it’s already useful for trying out internal tools. At the time of writing, version 1.8.0 is the latest stable release for the Android libraries.

Global multinational companies such as Airbnb, Disney+, Meta (notably Threads), Spotify, Chat GPT and even Google Play have adopted Compose for real-world, user-facing applications. The benefits reported include faster delivery cycles, better performance, and reduced defect rates. In the local and African context, uptake has been slow, with only Standard Bank and Discovery being the most notable users of Jetpack Compose, indicating growth potential.

Developer Experience over the Past Two Years: A Mixed Bag

Developers praise the framework for its rapid iteration, composable architecture, and reusability. However, some challenges remain. Understanding recomposition, managing state effectively, and integrating with legacy View-based systems can still be tricky. For teams working with legacy codebases, ComposeView offers a practical way to bring Compose into existing XML-based projects. It lets developers embed composables directly into traditional views, making the migration process less daunting and more incremental. Tooling has improved significantly, though debugging and UI inspection tools still have room to mature.

Performance has also improved significantly. Smarter recomposition, more efficient lazy layouts, a smoother Skia renderer, and Kotlin 2.x have led to faster builds and improved runtime performance. Development tools such as modifier tracing, recomposition highlights, and memory leak detection further enhance reliability.

Over the past few years, I have been using Jetpack Compose as my main tool for Android development, and I have learned a lot through hands-on experience:

• Building UIs from designs is quicker and easier. It is much faster to go from a design to a working screen. I can create reusable components which help speed things up. The live preview in Android Studio also saves time since I don’t have to rebuild the whole app just to see small changes.
• I used to struggle with recomposition, but I’ve improved over time and now understand how to manage it better.

•State hoisting has helped me write cleaner code, especially when working with shared or lifted state across components.
• The Android developer community has been helpful; for example, tools like custom lint checks (such as Compose Lints by Slack) help catch potential issues early, making it easier to write better Compose code.

The Ecosystem: Mature and Expanding

Google continues to invest in its Jetpack Compose through key libraries like Navigation, Paging, Glance, and Material, covering everything from app architecture to home screen widgets. On top of this, community-driven tools such as Accompanist, Showkase, Voyager, and Lottie make it easier to build polished, responsive UIs.

The tooling is just as strong. Android Studio offers features like Live Edit, Compose Metrics, and helpful lint checks that support quick iteration and code quality.

Together, this growing ecosystem helps developers build high-quality apps faster, whether for Android alone or across platforms with Compose Multiplatform. Despite the progress, there are still a few areas where Google could improve Jetpack Compose:
• Working with XML and older code, in larger apps, switching from XML to Compose can still be tricky. Having better migration tools built in would make it easier to deal with older codebases.
• Simpler architectural guidance, better official guidance could assist in removing confusion for newcomers.
• Some error messages, especially when a composable can’t be shown, could be clearer and more helpful.

Compose is also expected to expand further into system-level apps and embedded platforms.

Conclusion

In 2025, Jetpack Compose is no longer the future; it is the present. It offers faster development, cleaner architectures, and cross-platform potential. While challenges around learning and tooling remain, the benefits far outweigh the drawbacks. For any developer building Android apps today, sticking with the legacy View system now feels like an avoidable limitation.

If you haven’t tried Compose yet, now’s the time to explore it. The ecosystem is mature, the community is active, and the future is clearly composable.

References

1. https://developer.android.com/jetpack/androidx/releases/compose
2. https://slackhq.github.io/compose-lints/

3. https://betulnecanli.medium.com/why-you-should-learn-jetpack-compose-in-2025-and-how-to-do-it-for-free-a9e1b1db48e8
4. https://android-developers.googleblog.com/2022/05/airbnb-uses-jetpack-
compose.html
5. https://github.com/airbnb/Showkase
6. https://www.linkedin.com/posts/jorgecastillopr_guiding-the-jetpack-compose-migration-at-activity-7308205675501789185-LLdj?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAznNGgBl_8Tm0rhhda-rOFjm37wD5qe5V8
7. https://www.jetbrains.com/lp/devecosystem-2024/
8. https://x.com/JorgeCastilloPr/status/1902439932129390808
9. https://www.youtube.com/watch?v=23sNq-N06xU

The State of Jetpack Compose in 2025 was originally published in DVT Software Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the things that bothers me is when young designers define ‘Graphic Design’ as “design for print” while ‘Digital Design’ is “real, modern design on computers”.

While the latter isn’t necessarily wrong, something they are missing is that the “Graphic” in graphic design merely emphasises that the design is visual; ie, you can see it.

What seems to get lost on the next generation of designers is that the core principles of design work across mediums and across generations, which is why we have had graphic designers in movies, music, video games… You name it, decades before computers and software became a thing.

A core principle in the design world is “Design Thinking”, which is a problem-solving approach that emphasises empathy, creativity, and iterative testing to develop user-centered solutions. It involves five stages: empathise, define, ideate, prototype, and test, where teams understand user needs, brainstorm ideas, create prototypes, and refine solutions based on feedback. This method fosters innovation by prioritising human experiences and collaboration.

Now that the boring stuff is out of the way, let me explain what this has to do with ‘McDonald's’, a fast food chain that has been running for nearly a century.

Back in the 1940’s, Richard and Maurice (Dick and Mac) McDonald wanted to reinvent their restaurant, allowing them to lower costs while maintaining a premium product. Among many of the decisions they made was the choice by Dick to go out with their staff to a tennis court and draw out the layout of the kitchen in chalk, illustrated here in the movie “The Founder” (2016)

They then asked their staff to walk around the “kitchen”, manning their posts and pretending to go about their day-to-day business. Here we see the McDonald’s brothers incorporating many of the Design Thinking methodologies at once. They ideated, created a ‘low fidelity’ prototype and tested.

Viewing from the sidelines, they witnessed staff bumping into one another and being generally inefficient.

So they erased the chalk and started again:

And again they stood on the sidelines discussing, analysing, refining as the kitchen staff moved back and forth through the new layout until finally they iterated on their design for a 3rd time:

6 hours.
3 Design iterations.
Zero cost.

While modern designers are often eager to dive into software, spending hours, days, or even weeks on high-fidelity designs at the client’s expense, there’s value in stepping back to create quick, low-cost iterations. When software crashes or tools are unavailable, the instinct to wait can stall progress. But Design Thinking teaches adaptability — like grabbing a pen and paper to sketch ideas during a power outage, ensuring the creative process continues uninterrupted.

As we can see, there are definite lessons to be learned from the McDonald’s brothers.

Even if their staff did burn my last burger.

From Tennis Court to Table: McDonald’s Design Thinking Legacy was originally published in DVT Software Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

I recently took one of those hyped-up AI design tools for a spin, the kind that promise to churn out polished UI screens faster than you can say “deliverable.” It delivered, alright — crisp layouts, neatly aligned components, colours that didn’t make my eyes bleed. But after testing it on a real project, I was left with one gut-punch realisation: it’s all polish, no soul. Like a stock photo of a sunset, it’s technically fine but feels like it’s trying to sell you something instead of telling a story.

The Generic Trap
The output was, frankly, generic as hell. It looked like every other trendy UI kit you’ve seen on Dribbble — sleek buttons, safe typography, and a colour palette that screams “I read a blog post about 2025 design trends.” There’s no depth, no narrative. It’s decoration, not design.

As a designer, I could feel the absence of intent. This wasn’t a solution crafted for a specific user or problem; it was a one-size-fits-all template dressed up as innovation. And that’s a problem when your job is to create something that connects, not just something that looks “nice.”

Skipping the Soul of Design
Here’s where it gets worse: tools like this don’t just produce bland results; they encourage you to skip the actual design process. No wireframing, no sketching, no wrestling with ideas in that messy, beautiful way that leads to breakthroughs. Those early stages aren’t just prep work — they’re where the magic happens. It’s where you ask, “Who is this for? What are we solving? How do we make it feel human?”

When you jump straight to a pre-baked layout, you’re not designing; you’re decorating someone else’s idea. My test run with the tool proved it. The AI spat out a screen in seconds, but it was like being handed a puzzle with half the pieces already placed — except they didn’t quite fit. I spent more time tweaking redundant components than I would have sketching from scratch. And because I didn’t build the logic myself, I couldn’t even spot what was missing, or even unnecessary, until I was knee-deep in revisions. It felt like un-designing a completed design, with blinders on, following a script I didn’t write.

Redundancy Without Insight
Speaking of redundant components, let’s talk about the mess these tools create. The layouts generated tend to be bloated with repetitive elements — think three variations of the same button style, none of which served a clear purpose. It was as if the AI was throwing in extras to look busy, like a kid padding an essay to hit the word count. Worse, it obscured what was actually needed. Because I hadn’t mapped out the user flow or problem space myself, I was stuck playing catch-up, trying to reverse-engineer the AI’s choices. That’s not efficiency; it’s chaos masquerading as a shortcut.

This is why we wireframe and sketch first. Those steps force you to think through the problem, to question every element’s purpose. Without them, you risk ending up with a design that’s busy but empty, like a restaurant menu with 50 items but nothing you want to eat.

The “Pixel-Pusher” Stereotype
Maybe the most frustrating part is how these tools feed into a narrative that designers have been fighting forever: the idea that we’re just here to “make things pretty.” You know the stereotype — “Oh, you’re a designer? So you push pixels around and pick nice colours?” It’s infuriating, and tools like this make it worse. By reducing design to a button press, they strip away the strategy, the problem-solving, the human insight that’s at the core of what we do.While chatting my colleague Roberto and I hashed it out: these tools might dazzle stakeholders in a sprint review, but they risk wrapping the wrong solution in a shiny bow. Dead right. Clients love quick turnarounds — time is money, after all — but speed without context is a disaster waiting to happen. A beautiful UI that doesn’t serve the user is just lipstick on a pig. We’re not decorators; we’re translators, turning needs into experiences. That takes thought, empathy, and iteration — not an algorithm spitting out a template.

What AI Should Be
Don’t get me wrong — I’m not anti-AI. There’s huge potential for tools that support designers, especially for the grunt work. Automate repetitive tasks like resizing assets, generating style guides, or cleaning up file exports? Hell yes, sign me up. That’s the kind of AI I’d welcome with open arms, like one that could tackle my laundry pile while I’m at it. But when AI tries to replace the creative process entirely, it’s not just unhelpful — it’s irresponsible.

Design isn’t just about what something looks like; it’s about why it exists, who it serves, and how it feels. AI can’t replicate the messy, human spark of ideation or the gut-check moments when you realise a solution isn’t quite right. It can’t sit in a room with a client and sense their unspoken needs. And it sure as hell can’t replace the satisfaction of crafting something that feels alive.

A Call for Better Tools
If AI is going to be part of our workflow, it needs to elevate our thinking, not bypass it. Imagine a tool that acts like a collaborator — suggesting ideas, flagging inconsistencies, or speeding up iterations while leaving the big decisions to us. That’s the kind of AI that respects design as a craft, not a commodity. Until then, I’ll stick to my sketches, my wireframes, and my tea-fuelled brainstorming sessions. Because that’s where the real work happens — and no algorithm can touch that.

Fast But Soulless: Why AI Design Tools Miss the Mark was originally published in DVT Software Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Lo-fi beats, late nights, and no JIRA tickets in sight ; vibe coding feels like freedom. But when passion projects turn into production code, things can go sideways fast. This love letter celebrates the joy of coding in flow and gently reminds you when to put the vibes away.

Let’s face it; sometimes, you’re not writing code. You’re vibing code. Isn’t it?

You know the feeling. It’s 2 AM, and you’ve got lo-fi beats in your ears, caffeine (or Red Bull and Monster) in your veins, and an idea that might change the world or at least get a few GitHub stars. No stakeholders. No JIRA tickets. Just raw creative energy and a slightly unstable prototype named my_final_version_really_this_one_last_time_v7.

Welcome to vibe coding. It’s fun. It’s fast. It’s how pet projects are born, how AI tools get test-driven, and how developers accidentally invent the future (or break their laptops in the process).

When is vibe coding a vibe? 😏

1. Pet projects, hackathons, and rapid prototypes: These are playgrounds. There’s no legacy code, no stakeholders breathing down your neck, and no tight deadlines to meet. It’s just you and your imagination. Use that fancy new AI framework. Try a weird new layout strategy. Let the vibes flow.
2. Learning and experimentation: Want to wrap your head around Claude Desktop, Cursor, Devin AI, Spring AI, GitHub’s MCP tools or Cohere APIs? Vibe coding is a great way to get your hands dirty without getting tied down. It’s spontaneous. It’s messy and that’s the point. You learn by building, however weird and wonderful it turns out.
3. Starting something from scratch: No tech debt. No rigid architecture. Just ideas and momentum. It’s easier to fix mistakes early, without dealing with legacy decisions, especially when it’s not yet a client’s production environment.

When is vibe coding a recipe for chaos? 😦

1. Enterprise Applications: You’re not just coding; you’re contributing to a carefully orchestrated system. Teams depend on structure, patterns, consistency, scalability, security and documentation. Vibe coding here is like trying to improv jazz in a symphony orchestra. Attempting it in a production environment is likely to result in someone being fired.
2. Shared Codebases: When multiple devs touch the same repositories, standards matter more than vibes. Vibe coding can lead to inconsistent patterns, mysterious frameworks, and everyone’s favourite game: “Who wrote this and what were they thinking?” followed closely by “What is this for?” moments.
3. High-Stakes Projects: Deadlines. Compliance. Security audits. Your code doesn’t need to feel right, it needs to be right. Vibes don’t pass a PEN test. Run one on your freestyle masterpiece and watch it light up like a Christmas tree.

AI Assistants and Tools: A humble request to use them wisely 🙏

Sure, AI can autocomplete your thoughts and generate code like a caffeinated programmer. But should you always trust it?

1. AI can enhance your efficiency, but don’t hand over your judgment.
2. Be selective. Use tools that solve real problems, not just ones trending on X or paid news platforms, even if you’re not 100% sure how to test them (yet).
3. Evaluate tools based on your team’s maturity, the project’s lifecycle, and your tech stack not just the hype cycle.

Bottom line: Use vibes for flow, not for foundations 💡

Vibe coding is brilliant for sparking creativity. But enterprise apps need architecture, not astrology. The key is balance. Experiment with enthusiasm, but adopt with intention and discipline.

Think of it like seasoning: a sprinkle of vibe coding can make your project flavourful, but pouring in the whole bottle? Now, that’s just a mess.

Now, go forth, code hard, and vibe responsibly. May the Force be with you!

By Saurabh Agrawal, Practice and Engineering Lead: Java, Backend Open-Source Technologies and AWS Cloud Solutions at DVT.

Vibe Coding vs. Reality: A Love Letter (and Warning) to Developers was originally published in DVT Software Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Many developers likely share my perspective on the evolution — perhaps over-complication — of the tech stack over the years. Not too long ago, we would configure and deploy an application server. While setting up the server required some configuration, it was not an overly complex process. However, this approach had its limitations, particularly in terms of scalability and fail over recovery.

Modern tools like Docker, cloud platforms and Kubernetes have since addressed these challenges. Yet, these solutions, often embraced by corporate environments, introduce their own drawbacks — primarily the complexity of orchestration and the necessity of upskilling in numerous additional tools.

Let’s break this down and explore whether the resulting hype and complexity were truly justified.

The Evolution: Simplicity to Complexity

Traditional Application Servers:

• Early application servers like Tomcat, WebLogic, and JBoss provided a centralised runtime environment for deploying applications.
• Setup and Deployment: Configuring these servers for a single-node environment was relatively simple and well-documented.

Challenges:

• Scalability: Scaling typically involved manual efforts, such as setting up multiple server nodes and load balancers.
• Fail over and Recovery: Required proprietary solutions or additional tools, which were often costly or platform-specific.
• Vendor Lock-In: Organisations were often tied to a specific vendor’s ecosystem, limiting flexibility and innovation.

The arrival of Docker and Cloud Services:

• Docker: Revolutionised development by introducing lightweight containers that package applications and their dependencies into portable units.
• Impact: This “write once, run anywhere” model ensured consistent behaviour across environments.
• Development Agility: Enabled micro services architecture by allowing applications to be split into modular, independently deployable units.
• Cloud Services: Platforms like AWS Elastic Beanstalk, Azure App Services, and Google App Engine abstracted the complexities of infrastructure management.
• Impact: Simplified deployment and automated scaling, making enterprise-grade capabilities accessible to smaller teams.
• Integration: Tools like CI/CD pipelines integrated seamlessly, reducing the time to market.

Challenges:

• Dependency on Cloud Vendors: Created a new form of vendor lock-in.
• Cost Management: Scaling services dynamically often led to unpredictable expenses.

Kubernetes and the Rise of Orchestration Tools:

• Kubernetes: Evolved to address the complexities of managing containerized applications at scale.
• Capabilities: Automated deployment, scaling, and management of containerized workloads in distributed systems.
• Ecosystem: Grew to include supporting tools like Helm (for package management), Kustomize (for configuration management), and Istio (for service mesh).

Challenges:

• Complexity: Kubernetes itself has a steep learning curve, requiring significant expertise to manage.
• Operational Overhead: Monitoring, debugging, and securing Kubernetes clusters require additional layers of tooling, such as Prometheus, Grafana, and Kubescape.
• Over-Engineering Risks: Small to mid-sized teams often find Kubernetes overkill for simpler workloads, leading to inefficiencies.

Modern Landscape and Emerging Trends:

• Serverless Architectures: Solutions like AWS Lambda, Azure Functions, and Google Cloud Functions aim to further abstract infrastructure management by allowing developers to focus solely on code.

Benefits: Scalability and fail over are handled automatically by the provider.

Challenges: Observability and cold-start latency can be significant drawbacks for some use cases.

• Edge Computing: Platforms like Cloudflare Workers and AWS Lambda@Edge enable applications to run closer to users for low-latency performance.
• Low-Code/No-Code Solutions: Tools like OutSystems and Bubble offer simplified environments for building applications without extensive coding, catering to non-technical teams.

By tracing the evolution from traditional servers to modern orchestration and serverless solutions, we can see how the development world has shifted from simplicity to increasingly powerful but complex ecosystems. This journey highlights the balance between solving legacy challenges and introducing new ones, ultimately shaping the way we build and deploy software today.

The Trade-Offs: Why the Complexity?

Scalability and Resilience at Scale

Global Scale: Modern applications often serve millions of users across multiple regions, necessitating robust infrastructure capable of:

• Load Balancing: Distributing traffic efficiently across nodes to prevent overload.
• Data Replication: Ensuring consistent and fast access to data worldwide.

Advanced Resilience: Beyond self-healing, modern systems demand:

• Disaster Recovery: Automating fail over to backup systems in case of catastrophic failures.
• Zero Downtime Deployments: Strategies like blue-green and canary deployments to update services without disrupting users.

Cost of Complexity: Achieving these capabilities with Kubernetes and related tools requires:

• Significant expertise in configuration and tuning.
• Investments in monitoring and logging systems to maintain visibility.

DevOps Culture and Automation

Shift in Responsibilities: Developers are now expected to understand and manage operational aspects traditionally handled by sysadmins.

• Infrastructure as Code (IaC): Tools like Terraform and Ansible are now essential for managing infrastructure declaratively.
• Build Pipelines: Automated testing, building, and deployment are table stakes for modern development.

Explosion of Tools: The demand for automation has created a crowded ecosystem, including:

• Monitoring and Observability: Prometheus, Grafana, and New Relic to ensure systems remain performant.
• Secret Management: Tools like Vault to securely manage sensitive configuration.
• Security Scanning: Solutions like Aqua Security or Trivy to secure containers and their dependencies.

Cloud-Native Complexity

Portability vs. Complexity:

• Portability: Kubernetes provides a consistent API and deployment model across cloud providers and on-premises systems, reducing vendor lock-in.
• Complexity: Managing cloud-native workloads introduces abstractions like:

Operators: Custom controllers for automating application-specific tasks.

Helm Charts: Simplifying Kubernetes configurations but adding another layer of templating complexity.

Security and Compliance:

• Multi-Tenant Systems: Ensuring isolation and security in shared environments adds overhead.
• Compliance Requirements: Adhering to standards like GDPR or HIPAA often requires custom implementations for data encryption, audit logging, and access control.

Developer Experience vs. Operational Burden

• Learning Curve: The number of tools and the depth of knowledge required to operate modern systems can be overwhelming for developers.
• Trade-Offs in Speed: While DevOps practices and cloud-native tools promise faster delivery, the initial setup and ongoing maintenance can slow teams down if not well-managed.

Emerging Efforts to Reduce Complexity

• Platform Engineering: Organisations are increasingly building internal developer platforms (IDPs) to abstract away complexity, enabling developers to focus on application code instead of infrastructure.
• Managed Kubernetes Services: Offerings like Amazon EKS, Google GKE, and Azure AKS offload operational burdens while retaining Kubernetes’ benefits.
• Serverless as an Alternative: For simpler use cases, serverless computing eliminates the need to manage infrastructure altogether, shifting focus entirely to the application logic.

Was It Worth It?

The Good:

Unprecedented Agility and Innovation: The shift to containerization, orchestration, and cloud-native tools has empowered organisations to:

• Release features faster with CI/CD pipelines.
• Rapidly adapt to changing business needs and customer demands.

Resilience at Scale: For companies with global reach, like Google, Netflix, and AWS, these technologies ensure:

• High availability with near-zero downtime.
• Robust disaster recovery strategies.

Developer Empowerment: DevOps practices and tools enable teams to own the entire lifecycle of their applications, fostering collaboration and innovation.

Standardisation: Kubernetes and its ecosystem have created a common language for infrastructure, reducing fragmentation across platforms and providers.

The Bad:

Over engineering Risks:

• For smaller applications or simpler workloads, Kubernetes often introduces more complexity than value.
• Organisations may find themselves spending significant time and resources on managing tools rather than delivering features.

Steep Learning Curve:

• Teams face a daunting task in learning and adopting a sprawling ecosystem of tools (Kubernetes, Helm, Istio, Prometheus, etc.). These days many job adverts for developers are not even considering the candidate for a developer role unless they have knowledge of DevOps or some of the DevOps tools.
• The learning curve can lead to productivity dips and delayed project timelines.
• Operational Overhead: Even with managed services, operational expertise is required to troubleshoot issues, optimise performance, and secure systems effectively.

Missed Opportunities with Simpler Solutions:

• Many organisations overlook managed services (e.g., Heroku, AWS App Runner, Elastic Beanstalk) or serverless architectures that could achieve similar results with far less effort.

A Balanced Perspective:

When Complexity Makes Sense:

• If your organisation operates in a multi-cloud or hybrid cloud environment, Kubernetes’ portability can offer significant advantages.
• When running highly distributed systems or microservices architectures with demanding scalability needs, the investment pays off.

When It Doesn’t:

• Monolithic applications or workloads with predictable traffic often don’t benefit from Kubernetes’ features.
• Startups and smaller teams may be better served by simpler solutions that allow them to focus on product development rather than infrastructure management.

Emerging Trends and Alternatives:

• Platform Engineering: Many organisations are moving towards creating Internal Developer Platforms (IDPs) to abstract away the complexity of Kubernetes and provide developers with self-service tools.
• Serverless and Edge Computing: Serverless platforms like AWS Lambda or edge services like Cloudflare Workers provide scalable solutions without requiring teams to manage infrastructure.
• Kubernetes Alternatives: Lightweight orchestration tools like Nomad or frameworks like Docker Swarm might be a better fit for less complex use cases.

Cloud App Services

Managed cloud app services like AWS Elastic Beanstalk, Azure App Services, or Google Cloud Run solutions abstract away much of the complexity:

• Automatic scaling.
• Built-in failover.
• Simplified deployments.

For many use cases, these services hit the sweet spot of simplicity and functionality without needing to manage Kubernetes clusters. These could serve as the ideal solution for smaller teams, or teams that are lacking the skillset to manage and orchestrate entire Kubernetes clusters.

What Can We Learn?

Pick the Right Tool for the Job:

• Understand Your Needs: Before jumping into complex solutions, assess your application’s requirements in terms of traffic, scalability, and deployment frequency.
• Ask Questions: Does your application truly need multi-region resilience? Can managed services or simpler orchestration tools meet your needs?
• Avoid Hype-Driven Decisions: Evaluate tools based on your team’s skillset and the problem you’re solving, not on industry trends.

Embrace Managed Kubernetes:

• Leverage Ecosystem Tools: Pair managed Kubernetes with tools like Helm, ArgoCD, or Crossplane to simplify management.
• Cost Optimisation: Managed services often include features like auto-scaling and resource monitoring, which can help optimize cloud spend.
• Start Small: Begin with a single managed cluster for non-critical workloads to build confidence before scaling up.

Push for Developer Experience:

• Adopt Internal Platforms: Many organisations are building Internal Developer Platforms (IDPs) to abstract complexity, allowing developers to focus on code rather than infrastructure.
• Explore Low-Code and No-Code Tools: Depending on the use case, platforms like OutSystems, Bubble, or Retool can dramatically simplify development and deployment.
• Prioritise Usability: Invest in tools and practices that enhance usability, such as simplified deployment pipelines, integrated monitoring, and clear documentation.

Balance Infrastructure with Innovation:

• Shift Complexity Away from Developers: Consider adopting platform engineering practices to centralise operational complexity within a dedicated team, freeing developers to focus on features and functionality.
• Encourage Experimentation: Create environments where teams can test and deploy using simpler solutions without penalty. Not every project needs the full complexity of Kubernetes or cloud-native stacks.

Future-Proof Your Choices:

• Stay Agile: The tech landscape evolves rapidly. Adopt tools and practices that allow flexibility and adaptability, avoiding rigid vendor lock-in or overly customised implementations.
• Invest in Training: Equip teams with the knowledge to make informed decisions about when and how to adopt new tools. Training reduces the risk of failed implementations and promotes better long-term adoption.

Final Thoughts

The evolution of development and infrastructure tools has undeniably brought immense value to organisations that require scale, resilience, and innovation. Kubernetes and its ecosystem have enabled industry giants to operate globally with high availability and fault tolerance. However, for many smaller organisations or less complex use cases, the trade-offs in complexity, cost, and operational burden make simpler solutions more practical and equally effective. Whether it was worth it largely depends on the organisation’s scale, use case, and its ability to manage the accompanying complexity.

The developer world has historically oscillated between simplicity and complexity, and the Kubernetes era represents a period of heavy investment in complexity to meet the demands of large-scale, distributed systems. However, the rise of managed services, serverless offerings, and even the evolving field of platform engineering signals a shift towards more developer-friendly solutions. These tools promise to strike a balance between scalability and ease of use, potentially providing a future where we can achieve the benefits of cloud-native systems without drowning in a sea of orchestration tooling.

As we look forward, the key lesson is that there is no one-size-fits-all solution. Each organisation must evaluate its needs, its capabilities, and its long-term goals to determine the right mix of tools. Simpler solutions may often be overlooked in the rush to adopt “cutting-edge” technology, but they can provide just as much value — if not more — when aligned with the appropriate context.

While developers face significant pressure to master the multitude of tools necessary for managing and orchestrating these new deployment and setup strategies, the shift towards complexity is not inherently negative. It mirrors the increasing demands placed on contemporary systems. The true challenge lies in adeptly navigating these evolving landscapes, striking a balance between the need for flexibility and scalability, and maintaining the simplicity and productivity that are the cornerstones of exceptional software development.

The evolution of software development was originally published in DVT Software Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

In today’s tech landscape, tech enthusiasts and small businesses alike increasingly value the ability to connect to their networks remotely. Traditionally, this has been achieved using tools like WireGuard and OpenVPN, which — while effective — require a degree of networking knowledge for configuration and ongoing maintenance, as keys or user accounts need to be managed individually. However, software-defined networking has simplified this process considerably, reducing the need for extensive configuration and maintenance.

In this overview, we’ll examine the benefits of Tailscale as a networking solution well-suited for home and enterprise environments. Configuring a home VPN typically requires exposing a port on the router, which can raise security concerns. Tailscale, on the other hand, eliminates this need by establishing secure peer-to-peer connections between devices without requiring open ports. It leverages WireGuard as its encryption protocol, a familiar technology for those experienced with traditional VPNs. Tailscale is also available as a free VPN solution for basic home use.

Tailscale is a mesh VPN solution that uses the modern WireGuard protocol to secure traffic within a peer-to-peer network. Its design prioritizes ease of use, scalability, and secure device-to-device communication without requiring centralized infrastructure. Unlike traditional VPNs, Tailscale forms directly encrypted connections between devices, avoiding the need for a central VPN server to route traffic. This direct communication reduces latency, enhances speed, and benefits from WireGuard’s lightweight, high-performance encryption.

Tailscale streamlines network management by integrating single sign-on (SSO) services and identity providers like Google and GitHub for authentication. It automates IP allocation, firewall traversal, and device registration, creating a straightforward set-up experience. With automatic firewall and NAT bypassing, Tailscale enables devices to connect from nearly any network without manual intervention, making it ideal for remote access across distributed teams or dynamic environments. Its fine-grained access control using ACLs enables tailored access to specific devices or users.

By removing the need for centralized infrastructure, Tailscale also reduces operational costs for small-to-medium businesses, eliminating the expenses associated with dedicated VPN servers and additional resources.

How to get started

Getting started with Tailscale is relatively simple. First, you will have to create a Tailscale account. There are multiple methods for logging in.

Choose the option that you are most comfortable with.

Once you have an account and are logged in, you will see the admin console. This is where you can manage your Tailscale account, devices, users, etc.

Note: Your admin console will look slightly different from the above as the devices linked in the images displayed are already linked. Continue to link your first device.

How to link a device to your Tailscale accounts

Now that you have a Tailscale account, you can link the devices you want to connect to through your Tailscale VPN. To start, visit the official download page to view all supported devices and platforms. You can also click on the “Add device” button located on your admin console and follow the steps for your type of device.

To link your device to Tailscale, first identify the type of device you wish to connect and follow the installation instructions specific to that platform. After successfully installing the application, launch it and sign in with the Tailscale account you created. Once logged in, the device will be added to your Tailscale account, and you can connect to your VPN, making it accessible to other devices on your Tailscale network (also referred to as a tailnet).

In the admin console, you will notice an IP address associated with your device, representing its address within the Tailscale network. If MagicDNS is enabled (typically the default setting), you can access your device by its hostname rather than its IP address. This feature simplifies device management, eliminating the need to remember multiple IP addresses.

For example, if you have a home server hosting Pi-hole with an internal IP address of 192.168.0.10, you would typically use this address to access it within your local network. However, when you connect this device to Tailscale, it will receive an additional IP address that can be viewed in the admin console. If you attempt to access your Pi-hole using the internal IP address (192.168.0.10) while away from home, the connection will fail, as Tailscale is unaware of this internal address.

Instead, if you utilise the IP address provided in the admin console or the domain name generated by Tailscale (typically ends with .ts.net), the connection will succeed. To facilitate seamless transitions between being connected to the VPN and being disconnected, you can take advantage of MagicDNS. For instance, if your Pi-hole has a host name of “pi-hole-server,” you can access it using this host name instead of its internal IP address. Thus, when connected to the Tailscale network while away from home, you can still reach the server using the same host name you would at home, eliminating the need to remember its IP address.

To confirm that MagicDNS is enabled, navigate to the DNS tab within your admin console.

Once you have clicked on the DNS tab you can scroll down until you see the MagicDNS config.

Ensure that this is enabled, and you should be able to access your devices easily via your host name.

Once the application is installed on the device that you want to link and you have logged in with your account details, you should be able to manage your Tailscale connection from your device:

• On Windows and macOS: Tailscale is running in the system tray. Click the tray icon to view your connection status and connect or disconnect as needed.
• On Android and iOS: Open the Tailscale app and use the toggle at the top to control the connection.
• On Linux: Since there’s no graphical interface, connect by running these commands:

To connect to the Tailscale network:

sudo tailscale up

To disconnect, use:

sudo tailscale down

And to check the current status:

tailscale status

The device will now be connected to your Tailscale network if connected correctly. You should be able to see your device and its info on your admin console. Each device added to your Tailscale account is managed by a unique key with an expiration date. When a key expires, you must log back into Tailscale on that device to generate a new one. In traditional VPNs, these keys would have been managed by someone. Tailscale manages these keys for you. For critical devices, like servers or home labs, that require uninterrupted access, you may want to disable key expiry.

Disabling Key Expiry for Specific Devices

To disable key expiry for select devices, follow these steps:

1. Open the Machines page in the admin console.
2. Locate the row for the device you want to modify.
3. Click the ellipsis menu (three dots) on the far right and select Disable Key Expiry.

Once this setting is applied, the keys for that device will no longer expire, which is particularly useful for remote or critical devices that need continuous access.

Expanding Your Secure Network

With your secure VPN setup, you can now access all connected devices from anywhere. You may want to explore further features to enhance your network:

• Ad-blocking with Pi-hole: Connect a Pi-hole device to your Tailscale network and set its IP as your Tailscale DNS. This allows all connected devices to benefit from ad-blocking, even when away from home. You can follow the steps here.
• Other Advanced Tailscale Features: Tailscale provides various advanced features, including the ability to designate a device as an exit node, integrate with cloud providers and containers, expose entire subnets, and share devices between other Tailscale accounts that you trust. Please refer to the official Tailscale documentation for further details on these features.
• Home Assistant: Home Assistant offers a Tailscale integration, allowing you to create a dashboard that displays device statuses and network statistics. These insights can be leveraged for additional home automation tasks, enabling seamless management of your smart home devices.

Conclusion

Setting up a home VPN with Tailscale creates a secure, software-defined network that provides flexibility, scalability, and simplified management in less than a few minutes. This simplicity makes it attractive for personal use, such as a home lab. This setup can be a compelling alternative to traditional VPNs and offers substantial advantages for managing your devices from anywhere. I hope you found this guide informative and helpful in understanding the benefits of software-defined networking with Tailscale.

Traditional VPN vs Tailscale was originally published in DVT Software Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.