<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Viachaslau Kitsun on Medium]]></title>
        <description><![CDATA[Stories by Viachaslau Kitsun on Medium]]></description>
        <link>https://medium.com/@vkitsun?source=rss-be0afd09aaa1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3WttPEBJHrkcEBlD6u5Pog.jpeg</url>
            <title>Stories by Viachaslau Kitsun on Medium</title>
            <link>https://medium.com/@vkitsun?source=rss-be0afd09aaa1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:58:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vkitsun/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Which SDD Tool Should You Actually Use? GSD, OpenSpec, BMad, and Kiro Compared]]></title>
            <link>https://medium.com/emerline-tech-talk/which-sdd-tool-should-you-actually-use-gsd-openspec-bmad-and-kiro-compared-b487807243cb?source=rss-be0afd09aaa1------2</link>
            <guid isPermaLink="false">https://medium.com/p/b487807243cb</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[spec-driven-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Viachaslau Kitsun]]></dc:creator>
            <pubDate>Wed, 28 Oct 2026 14:02:02 GMT</pubDate>
            <atom:updated>2026-10-28T14:02:02.062Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fgeZrWAwr34cBT0U8nB1Cw.png" /></figure><p><em>Planning, context, execution, and verification in greenfield and brownfield work</em></p><p>I’ve tried GSD, OpenSpec, BMad, and Kiro, and what became clear is that they put structure around different parts of development. BMad helps turn an unclear request into buildable work. GSD organizes execution across phases and sessions. OpenSpec keeps changes to product behavior explicit, while Kiro connects requirements, design, and implementation tasks in one environment. Their feature lists overlap, but working with them involves different decisions: what to review before coding starts, how much context to give the agent, and what evidence to check before accepting its work.</p><p>To make those differences concrete, I’ll use an internal resource-management application for an outsourcing company as a shared example. It covers people, projects, staffing, utilization reports, and hiring demand when a need cannot be filled internally. The example gives each tool the same scope, so the comparison stays focused on the development workflow rather than four different application designs.</p><p>The first increment starts from an empty repository and covers people, projects, confirmed allocations, and a basic utilization view. Then comes a change: tentative bookings for potential projects, which must remain separate from confirmed allocations in staffing views, reports, and hiring demand. This is where the differences become more useful than a feature checklist. Each workflow has to carry a revised requirement through existing code, update the relevant artifacts, and provide evidence that the change works without breaking what was already there.</p><p>The examples below focus on how each tool fits into the development process, from planning the first increment to changing an existing application. For setup instructions, available commands, and configuration options, see the documentation for<a href="https://docs.bmad-method.org/reference/skills-and-agents/"> BMad</a>,<a href="https://github.com/open-gsd/gsd-core/blob/next/docs/COMMANDS.md"> GSD</a>,<a href="https://openspec.dev/docs/supported-tools"> OpenSpec</a>, and<a href="https://kiro.dev/docs/specs/feature-specs/"> Kiro</a>. These references also cover workflows and integrations beyond the ones used in this comparison.</p><h3>BMad: Settle the Scope, Then Build One Ticket at a Time</h3><p>BMad makes most sense when the work is not yet ready to hand to a coding agent. In our example, “build resource management” is not a useful implementation request. The first useful interaction is to turn it into an agreed increment and ordered work, without generating documents for every future feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CgyDoZWAelCKNn7L2kc-QQ.png" /></figure><p>Start with the Product Owner agent, Sofia, through /bmad-agent-po, and give it a bounded request: “Plan the first increment: people, projects, confirmed allocations, and a utilization view. Leave tentative bookings and hiring-demand tracking for later. Order the work so the first completed slice can be exercised through the UI.” The worked examples here are illustrative; they are not implementation results or a head-to-head benchmark.</p><p>The current <a href="https://docs.bmad-method.org/plan/choose-a-planning-path/">planning workflow</a> maintains requirements at initiative, epic, or story level and orders a ticket tree through tickets/tickets.toml. Inspect that order. A backlog that builds every database table, then every endpoint, then every screen postpones feedback until most of the application exists. A ticket that connects one allocation to its visible utilization result gives the implementation and review a smaller target.</p><p>Use a PRD or an architecture document when a decision needs one. For example, separate staffing and reporting tickets need an agreed source of allocation data. They do not need two agents independently choosing what that source should be. BMad’s current planning tools are optional entry points, not a mandatory ceremony before every build.</p><p>Once a ticket is ready, open a fresh chat and invoke /bmad-build 1.2, using the actual reference from your tree. The <a href="https://docs.bmad-method.org/build/build-a-change/">Build workflow</a> investigates the repository and upstream material before implementing. A clear, contained change can follow a minimal plan; unclear intent, irreversible actions, or a larger footprint lead to a written plan and approval. At that checkpoint, inspect the proposed changes and test strategy, not just whether the plan repeats the ticket’s wording.</p><p>Build then implements, sends the change through independent review, addresses relevant findings, and commits locally. The useful output is a diff, verification evidence, and a summary of what remains, not merely the commit. Check that the acceptance path was exercised and that deferred findings really are unrelated. Pushing or opening a PR remains a separate action.</p><p>The ticket’s story-&lt;slug&gt;-plan.md records the pre-change baseline and owns build status. <strong>Build finishes at built, not done</strong>; closure and the external tracker’s status are separate. Keep that plan after completion, because later review and replanning use its evidence. This is a concrete limitation of treating “the agent finished” as a delivery status. <a href="https://docs.bmad-method.org/plan/break-work-into-stories-and-track-it/">Tracking and closure</a>.</p><p>For another review pass, give Build its previous plan while its status remains built. For a broader review, use /bmad-code-review with the PR and the intended behavior. Without the plan, review has only the diff and commit messages to infer intent. Unrelated findings go to deferred-work.md, so a clean current-change review is not a clean bill of health for the repository. <a href="https://docs.bmad-method.org/build/review-a-change/">Review workflow</a>.</p><p>When tentative bookings arrive later, tell Sofia which accepted requirements change and have the affected tickets re-sliced. If the implementation is wrong because the plan encoded the wrong rule, fix that upstream rule before rerunning Build. For a small, unambiguous report correction, go directly to Build instead of reopening product discovery. In an existing repository, bmad-project-context can maintain a reviewed block of constraints, commands, and pitfalls in root AGENTS.md; inspect that block before letting future sessions rely on it. <a href="https://docs.bmad-method.org/existing-codebases/set-and-maintain-project-context/">Project-context workflow</a>.</p><h3>GSD: Control the Execution Across Phases and Sessions</h3><p>GSD is useful when the scope is reasonably clear, but the build is too large to trust to one continuously growing conversation. Its main contribution is the execution structure: persistent planning state, bounded plans, fresh workers, and explicit recovery points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UtWHxwhFcnZ94wKfg5ST6g.png" /></figure><p>For the empty repository, start with /gsd-new-project. Give it the same first increment and exclusions. Before continuing, inspect .planning/PROJECT.md, REQUIREMENTS.md, and ROADMAP.md. The first phase should let someone create an allocation and see its effect in the application; subsequent phases can add hiring demand and broader reporting. STATE.md records where the work stands. <a href="https://github.com/open-gsd/gsd-core/blob/next/docs/reference/planning-artifacts.md">Planning artifacts</a>.</p><p>The working sequence is /gsd-discuss-phase 1, /gsd-plan-phase 1, and /gsd-execute-phase 1. Discussion records decisions in the phase’s 01-CONTEXT.md; planning produces files such as 01–01-PLAN.md. Current planning favors an end-to-end tracer slice before expansion. Still inspect the generated plan: does the reporting task receive the allocation contract? Are migrations ordered before dependent code? Can the verification demonstrate the visible result? <a href="https://github.com/open-gsd/gsd-core/blob/next/commands/gsd/plan-phase.md">Planning command</a>.</p><p>Execution can distribute plans across dependency waves. Fresh workers get context from repository artifacts rather than inheriting the entire conversation. That reduces dependence on a long chat, but also creates a specific failure mode: an important decision mentioned only in conversation may never reach the worker. Put it in the phase context or the relevant plan. The <a href="https://github.com/open-gsd/gsd-core/blob/next/docs/explanation/the-phase-loop.md">phase loop</a> is a context-management mechanism, not a guarantee that every worker received the right information.</p><p>After execution, compare the plan’s 01–01-SUMMARY.md with 01-VERIFICATION.md. Then use /gsd-verify-work 1 for acceptance, recorded in 01-UAT.md. Current GSD can accept some deliverables from coverage evidence and reserve judgment-dependent checks for human sign-off. Do not treat that as evidence that a person inspected every screen. For this phase, the application should show the allocation and the expected utilization result; a component test alone may not establish that the integrated path works.</p><p>If acceptance fails, verify-work diagnoses the gaps and prepares fix plans. /gsd-execute-phase 1 — gaps-only runs the plans marked for gap closure; repeat verification afterward. If the gaps came from the automated verification report, /gsd-plan-phase 1 — gaps plans corrections from that report. This is a more controlled recovery than starting an unrelated “fix everything” conversation. /gsd-ship 1 opens a PR; it does not mean merged or deployed. <a href="https://github.com/open-gsd/gsd-core/blob/next/commands/gsd/execute-phase.md">Execution command</a>, <a href="https://github.com/open-gsd/gsd-core/blob/next/docs/tutorials/your-first-project.md">first-project walkthrough</a>.</p><p>For a repository not yet onboarded, /gsd-onboard starts brownfield setup. For the later change in this already-tracked project, refresh stale codebase maps and use /gsd-phase “Tentative bookings” to append a phase, or — edit to revise an existing one. A roadmap edit does not update every existing plan. Replan affected work, or a fresh worker will faithfully implement the old requirement. /gsd-pause-work and /gsd-resume-work provide explicit session handoffs. <a href="https://github.com/open-gsd/gsd-core/blob/next/docs/COMMANDS.md">Command reference</a>.</p><p>GSD’s overhead is visible: discussion, research, planning, checking, execution, and verification consume time and model calls. Use that structure for a substantial increment, not a label rename. <strong>Default /gsd-quick skips discussion, research, plan checking, and verification.</strong> — validate restores the latter two; — full adds discussion and research too. Quick work lives under .planning/quick/ and updates state rather than the roadmap. It is not the full phase pipeline running faster. <a href="https://github.com/open-gsd/gsd-core/blob/next/commands/gsd/quick.md">Quick command</a>.</p><h3>OpenSpec: Make a Change Reviewable Before the Agent Implements It</h3><p>OpenSpec is the most direct fit here once the product direction is settled and the team wants behavior to remain documented as the application changes. It distinguishes the maintained baseline in openspec/specs/ from active changes in openspec/changes/. That is a different concern from GSD’s execution state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m6dQ0wHzHFC6RH26Caqz-A.png" /></figure><p>Run openspec init in the terminal and choose the coding tool that will host the workflow. In the agent chat, ask /opsx:propose add-resource-core to create the first increment, with the same exclusions. Before /opsx:apply add-resource-core, review proposal.md, capability delta specs, and tasks.md. The default schema also uses design.md when technical decisions warrant it; it is not compulsory for every tiny change. A data-model change spanning allocations and reports does warrant that design review. <a href="https://openspec.dev/docs/schemas/spec-driven">Default schema</a>.</p><p>For a terminal-level inspection, openspec status — change add-resource-core shows the artifact state, and openspec instructions tasks — change add-resource-core returns the instructions and dependencies for the tasks artifact. The change name is illustrative. openspec new change add-resource-core creates the change metadata; it does not, by itself, write all those documents for you. The agent workflow or the developer still has to author them. <a href="https://openspec.dev/docs/cli">CLI reference</a>.</p><p>The brownfield example shows the advantage more clearly. Propose add-tentative-bookings and ask for deltas covering staffing, utilization reports, and hiring demand, without changing the existing meaning of a confirmed allocation. Inspect the modified requirements and their scenarios before applying tasks. The review should show exactly which baseline behavior changes. It should not make the reviewer reconstruct the change by comparing two complete product documents.</p><p>Run openspec validate add-tentative-bookings — strict, but read its findings rather than trusting the exit code. Validation checks the spec’s structure and rules; it does not run the application. Current documentation also treats some merge conflicts as informational, even under strict validation. A green result therefore establishes neither correct reporting nor a conflict-free baseline update.</p><p><strong>Verification is optional in the current workflow set.</strong> Enable it through the openspec config profile and refresh the project’s installed workflows with openspec update before expecting /opsx:verify to exist. That workflow reports on artifacts versus implementation; it neither repairs the code nor archives the change. It also does not replace executable tests. <a href="https://openspec.dev/docs/profiles">Profiles</a>, <a href="https://openspec.dev/docs/skills">workflow reference</a>.</p><p>After an interrupted session, apply the same change again; task checkboxes carry progress. If the plan is wrong, use /opsx:update add-tentative-bookings to revise its artifacts before continuing. That updates the plan, not the code. <a href="https://openspec.dev/docs/quickstart">Quickstart</a>, <a href="https://openspec.dev/docs/skills">workflow reference</a>.</p><p>A MODIFIED delta replaces the entire requirement block. Preserve surviving scenarios rather than submitting only the changed sentence. Reconcile concurrent edits against the current baseline before sync or archive. Archiving with — skip-specs leaves that baseline unchanged, and incomplete tasks warn rather than universally block archiving. An archive folder is not proof that the specs match shipped behavior. <a href="https://github.com/Fission-AI/OpenSpec/blob/main/docs/troubleshooting.md">Delta troubleshooting</a>, <a href="https://openspec.dev/docs/cli">CLI reference</a>.</p><p>The practical limit is coordination across changes. Separate folders do not make simultaneous edits to the same capability independently. Decide their order, resolve the combined semantics, and review the merged baseline. OpenSpec gives that review a useful representation; it is not a product roadmap or a scheduler for a fleet of implementation agents.</p><h3>Kiro: Execute the Feature Inside the Spec Workflow</h3><p>Kiro is the simplest choice when the team wants to review requirements, inspect design, and run implementation tasks in the same environment. Its native Specs are their own workflow, not an implementation of OpenSpec. OpenSpec can use Kiro as a host, which is a separate choice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4lwArJOr99Bleo1REdpdVw.png" /></figure><p>In the IDE, use Specs → + → Feature → Requirements-First. Submit the first-increment request, then review requirements.md before moving to design.md and tasks.md. Design-first is available when an existing technical approach is the starting point. For this example, requirements-first exposes a weak requirement such as “show resource availability” before the agent embeds its interpretation in code. <a href="https://kiro.dev/docs/specs/feature-specs/">Feature Specs</a>.</p><p>Use Analyze Requirements at that point. It identifies ambiguity, contradictions, and missing assumptions, then asks questions whose answers can update the requirements. For example, “Should tentative bookings reduce confirmed utilization?” is a decision to make explicitly, not a detail for the reporting implementer to guess. Run the analysis again after a substantial manual edit. The analysis can expose a missing decision; it cannot choose the company’s policy. <a href="https://kiro.dev/docs/specs/analyze-requirements/">Requirements analysis</a>.</p><p>In the design, inspect how each requirement reaches components and interfaces. In tasks.md, select a task to execute, or use Run all Tasks. <strong>That button runs incomplete required tasks; optional tasks can remain undone.</strong> For a focused request, #spec:resource-core implement task 2.3 targets a particular spec and task. Check which tests are required before starting, rather than using the button label as the definition of done. <a href="https://kiro.dev/docs/specs/best-practices/">Execution and task handling</a>.</p><p>Kiro’s correctness workflow can derive properties from requirements and generate property-based tests. The current documentation lists this for the IDE, with property tests optional by default. If a property matters to acceptance, make sure its task is included and executed. A property such as “adding a tentative booking leaves confirmed utilization unchanged” is useful only after that behavior has been agreed. More generated cases cannot rescue the wrong property. <a href="https://kiro.dev/docs/specs/correctness/">Correctness support</a>.</p><p>Shared guidance belongs in steering, not in every prompt. The foundation files cover product, technology, and structure. Use focused inclusion rules for specialized guidance so unrelated tasks do not receive the entire project’s documentation. There is a host-specific trap here: custom CLI agents need steering resources configured explicitly; do not assume they receive every file merely because it exists under .kiro/steering/. <a href="https://kiro.dev/docs/steering/">Steering</a>.</p><p>For tentative bookings, edit requirements.md, ask Kiro to update design.md, then use Sync Files in tasks.md. Review the reconciled task list before resuming. If a task fails, inspect partial edits and failure output, correct the code or upstream assumption, and rerun the affected work; do not assume automatic rollback. Earlier staffing, reporting, and hiring-demand specs still need a conflict check. Native feature specs do not give you OpenSpec’s explicit baseline-and-delta model across the product. <a href="https://kiro.dev/docs/specs/best-practices/">Change handling</a>.</p><p>If execution instead exposes a pre-existing utilization defect, use a Bugfix Spec for that defect. bugfix.md separates current, expected, and unchanged behavior; design and tasks then address the correction and preservation checks. That is more precise than quietly expanding the feature into a reporting rewrite. Kiro’s limitation is not an absence of structure, but the need to keep separate feature documents, shared steering, and optional verification aligned. <a href="https://kiro.dev/docs/specs/bugfix-specs/">Bugfix Specs</a>.</p><h3>My Choice for This Application</h3><p>For a team that already agrees on the first increment and is comfortable with its existing coding agent, <strong>OpenSpec is my starting point.</strong> The ongoing work in this example is a sequence of behavioral changes: add allocations, introduce tentative bookings, alter a report, connect hiring demand. A maintained baseline and explicit deltas give reviewers something useful after implementation, without introducing a second product-planning process. Keep each change small enough that its code, scenarios, and baseline update can be reviewed together.</p><p>That recommendation changes when the problem changes. Choose BMad first when stakeholders still disagree about the release boundary or several work streams need shared decisions before they can build. Its planning and ticket structure address that uncertainty, and Build can carry the work through implementation. Do not export the backlog to GSD just because both tools are available.</p><p>Choose GSD when scope is agreed but a substantial build spans many sessions, dependencies, and execution handoffs. The reason to accept its extra artifacts is to recover the state of that build and inspect what each worker will receive. For a single bounded report change, that full phase structure is unnecessary; use a lighter path or stay with the existing change workflow.</p><p>Choose Kiro when the team wants the native requirements-to-tasks experience and will actually review its intermediate documents. For that team, adding OpenSpec on day one mostly creates another place to maintain the same requirement. Kiro becomes a weaker fit for a proposed process when the team expects every worker to consume identical steering automatically or assumes that all generated tests run by default.</p><p>The combination most worth evaluating here is OpenSpec for maintained behavior and GSD for a particularly large implementation. Keep the accepted OpenSpec change authoritative; give GSD its revision, relevant baseline specs, acceptance scenarios, and repository context. Check the generated GSD requirements and plans against that input. If the requirement changes, update the OpenSpec change first, then regenerate or amend the affected plans and tests. This is a proposed integration, not a built-in conversion or a combination I am claiming to have validated end to end.</p><p>BMad-to-GSD is also possible, but only if BMad’s own Build workflow leaves an actual execution problem unresolved. Otherwise, the handoff adds translation and duplicate state. Running all four would be harder to justify than choosing one and learning where it fails.</p><p>Before adopting any of them, put the same first increment and the same tentative-booking change through the chosen workflow. Inspect four concrete results: the artifact that owns the requirement, the context passed to implementation, the executed evidence for acceptance, and the updated documentation after the change. For the brownfield case, verify that confirmed totals stay unchanged while tentative bookings appear separately in staffing, reporting, and hiring demand. Reject a workflow that can mark the work complete without making those results inspectable. That is a more useful selection criterion than the number of agents, templates, or files it generates.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b487807243cb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/emerline-tech-talk/which-sdd-tool-should-you-actually-use-gsd-openspec-bmad-and-kiro-compared-b487807243cb">Which SDD Tool Should You Actually Use? GSD, OpenSpec, BMad, and Kiro Compared</a> was originally published in <a href="https://medium.com/emerline-tech-talk">Emerline Tech Talk</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dark Factory, Part 2: From a Greenfield Brief to Verified Delivery]]></title>
            <link>https://medium.com/emerline-tech-talk/dark-factory-part-2-from-a-greenfield-brief-to-verified-delivery-b925e1709f68?source=rss-be0afd09aaa1------2</link>
            <guid isPermaLink="false">https://medium.com/p/b925e1709f68</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Viachaslau Kitsun]]></dc:creator>
            <pubDate>Mon, 12 Oct 2026 08:53:08 GMT</pubDate>
            <atom:updated>2026-10-12T17:47:25.830Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bVmGP713VASLCiPJljo8UA.png" /></figure><p><em>How we embedded GSD into dependent Kanban tasks, with plan review, component-aware deployment, and recovery inside the factory.</em></p><p>We started by<a href="https://medium.com/emerline-tech-talk/from-no-code-to-ai-engineering-with-sdd-ab0d159b2196"> using specification-driven development and GSD to move from no-code to custom applications</a>. That gave us a structured way to turn requirements into plans, implement them, and verify them.</p><p>Then we<a href="https://medium.com/emerline-tech-talk/dark-factory-how-we-turned-google-chat-into-a-code-factory-1224fdd75bd2"> built a code factory around Google Chat</a>, so stakeholders could request changes to existing applications while the factory coordinated the engineering work through review and delivery.</p><p>We still had a gap between those workflows. Starting a project meant coordinating the greenfield process outside the factory, then bringing the application into the factory once it was ready for ongoing changes.</p><p>We have integrated GSD’s greenfield workflow into Dark Factory.</p><p>A stakeholder can start in chat with an idea, mockups, and supporting documents. We use those materials to establish the project context, generate a roadmap, and create the Kanban tasks needed to deliver its phases. We assign responsibilities, define dependencies, check plans before implementation, and require deployment and verification before the next phase proceeds.</p><p>The existing brownfield pipeline stays separate. We have added a path for building a new application without turning ordinary change requests into project-initialization exercises.</p><p>That extension required work beyond launching GSD from a message. We had to coordinate its stages, recover interrupted handoffs, and expose the revision and repair cycles between the initial brief and a verified result.</p><p><em>The access-request application below is a running example. Its roadmap, review findings, and QA failure illustrate the workflow rather than document a specific production execution.</em></p><h3>Starting Before There Is a Codebase</h3><p>Consider a stakeholder who sends this message in Google Chat:</p><p>Build an internal access-request application. Employees submit requests, managers approve or reject them, and IT records provisioning. Keep a history of status changes. I’ve attached mockups for the request form and approval queue, plus a requirements document.</p><p>For an existing application, we can inspect the repository and connect a request to code that already implements part of the product. A greenfield project starts with a different set of inputs.</p><p>We have an idea, supporting documents, and visual references. We still need to establish the project context that downstream agents will use.</p><p>The factory collects those materials into an intake and creates the repository scaffold. GSD then prepares PROJECT.md, the requirements, and ROADMAP.md. These artifacts give later stages a shared reference outside the chat history.</p><p>For our access-request example, that reference needs to capture the actors, the requested workflow, known constraints, and the boundaries of the first version. The mockups explain the intended interaction. The requirements document supplies behavior that a picture cannot express, such as who may approve a request.</p><p>We also added a guard before the expensive part of this process: an empty intake does not consume the resources required for a full planning run. The factory should not spend a planning budget interpreting the absence of a product brief.</p><p>For applications that need more discovery, we can enable an upfront interview, or grill, at the application level. That gives us a place to develop an incomplete idea before committing to a delivery cycle.</p><p>We keep this greenfield path separate from the existing brownfield process. A request to change a field in a working application should continue through the change workflow. It should not trigger repository initialization or a new project-definition exercise.</p><h3>The Roadmap Determines the Work We Create</h3><p>After GSD produces ROADMAP.md, the factory creates the tasks for its phases.</p><p>We do not hardcode a fixed phase count or assume that a new application must follow a predefined sequence of feature names. We derive the work graph from the roadmap for that project.</p><p>For the access-request application, one possible breakdown would be:</p><p><strong>Phase</strong></p><p><strong>Capability</strong></p><p>1. Application foundation</p><p>Sign-in, user roles, application structure, and persistence foundations.</p><p>2. Request and approval flow</p><p>Employees submit requests; managers approve or reject them; the application records status changes.</p><p>3. Provisioning and history</p><p>IT records provisioning, and users can inspect the relevant request history.</p><p>This is an example roadmap, not a factory template. A different brief could require different boundaries and a different number of phases.</p><p>Each phase receives its own engineering tasks and dependency chain. For a project with remote delivery configured, the sequence looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zyV_hT9rgEKo8TtmpNFvXw.png" /></figure><p>These are Kanban tasks with assigned responsibilities, prerequisites, and completion checks. We do not depend on one agent remembering the entire sequence and deciding, within a long session, that it is time to move on.</p><p>The next phase must wait for the preceding phase to clear its verification gates. Implementation alone does not satisfy the phase-completion dependency.</p><p>We also support starting a new milestone for a project that already exists. We use that entry point to establish the next scope and roadmap, then create the corresponding phase tasks without treating the repository as a new project.</p><p>That gives us separate paths for starting an application, delivering another milestone, and handling an individual change.</p><h3>Product Questions Belong Before Implementation</h3><p>Assume the foundation phase has passed verification. We can now focus on the request-and-approval phase.</p><p>The mockups show an approval queue and an Approve button. They may leave an important question unanswered: can someone who has the manager role approve their own request?</p><p>That is a product decision. We should ask the stakeholder during the discussion and record the answer before planning the implementation.</p><p>For this example, assume the answer is no.</p><p>The phase context now contains a rule that the planning, implementation, and verification stages can use. We should not ask the stakeholder to make that decision again during implementation because a later agent did not receive the earlier answer.</p><p>For a phase with frontend work, we can also create a UI specification. In this example, it would cover the request form, approval queue, available actions, and the loading, empty, and error states that the initial mockups may omit.</p><p>This is where we separate questions that require human judgment from questions an engineering agent can resolve from the contract.</p><p>Whether managers may approve their own requests requires a business decision. How to enforce an established prohibition through the application’s authorization structure is engineering work.</p><p>That distinction prevents the factory from turning routine implementation problems into an ongoing interview with the customer.</p><h3>Planning and Execution Have Different Responsibilities</h3><p>We assign planning and execution to separate roles.</p><p>The architect prepares the plan. The plan checker evaluates whether that plan can satisfy the phase requirements. The engineer implements the approved plan within its constraints.</p><p>Consider a flaw in the first plan for our approval flow: it includes hiding the Approve button for a requester’s own request but omits the corresponding backend check.</p><p>The plan checker should treat that as a blocking finding. The product rule concerns the ability to approve, so hiding one control does not cover the requirement.</p><p>The architect revises the plan to include backend enforcement and the relevant tests. The revised plan then returns for another check before implementation starts.</p><p>That gives us an explicit revision cycle:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l6oBuzA4_kB_PQTjVYPeNQ.png" /></figure><p>The same separation matters after execution begins.</p><p>An engineer may discover that a planned step depends on something the plan never establishes. Perhaps the plan expects an API field that the earlier phase never defined or implemented. The engineer should report that discrepancy and return the defective plan to the architect.</p><p>We do not want the executor to redesign the phase without recording the change. We also do not want it to disguise the defect as “we need to ask the user.”</p><p>The customer may have supplied enough information. In that situation, we need to repair our engineering plan.</p><p>The engineer still has discretion over implementation details that fit the checked contract. Changes to the contract itself go through revision and re-checking so that later verification targets the work we agreed to build.</p><h3>Browser QA Waits for the Relevant Deployments</h3><p>After implementation, the phase goes through code review and merge. The factory then creates deployment work for the components that the phase affects.</p><p>Our approval phase touches both frontend and backend behavior. The frontend introduces the form and approval controls; the backend handles the request lifecycle and authorization.</p><p>We mark both components as affected, and QA depends on their deployments.</p><p>If the frontend deployment finishes while the backend deployment is still running, browser QA waits. Starting the test at that point could exercise the new interface against the previous version of the API.</p><p>That would produce evidence about an intermediate environment rather than the completed phase.</p><p>This dependency also matters during repair. A frontend-only fix should require the relevant frontend deployment. A repair that changes the API must include the backend deployment before the next browser run.</p><p>We derive that work from the affected components instead of treating deployment as one generic checkbox.</p><p>For this walkthrough, assume the project has a configured Git remote and staging environment. Without a remote, the factory does not claim that it created a pull request, completed a remote merge, or deployed a build.</p><p>We keep the result at the level we can substantiate: local implementation and locally executable verification checks. Missing delivery infrastructure remains visible in the outcome.</p><h3>A Failed Test Creates a Dependent Repair Cycle</h3><p>Suppose Playwright exercises the following scenario in our approval phase: an employee submits a request, a manager opens the queue and approves it, and the interface should display the updated status.</p><p>The backend accepts the approval, but the queue continues to show <strong>Pending</strong> until the page reloads.</p><p>This failure identifies an observable mismatch between the required interaction and the running application.</p><p>The repair task needs that evidence: the action the tester performed, the expected result, the actual result, and the available screenshots and diagnostic output. “The approval screen is broken” gives an engineer much less to work with.</p><p>For this example, useful QA artifacts would include the failed browser assertion and a screenshot of the stale queue, alongside any available trace or logs. Those artifacts should let the engineer reproduce the problem and help the reviewer understand what the repair must address.</p><p>The factory creates a FIX task from the finding. In this example, the repair could address how the frontend refreshes the request state after approval.</p><p>The next issue is dependency sequencing.</p><p>We serialize FIX and RE-TEST. The retest must wait until the fix has passed the required review, has been merged, and has been deployed to the environment that QA will test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HW_9ExWLJ2ccFobNfkAmLQ.png" /></figure><p>Starting RE-TEST while the fix exists only on a branch would test the old behavior again. A second failure could then create another repair cycle for an issue that an engineer has already addressed.</p><p>We prevent that sequence through the task dependencies.</p><p>A successful retest allows the phase to continue to verification and user acceptance testing (UAT). An unresolved failure keeps the phase open, along with the dependencies that prevent later phases from building on an unverified result.</p><h3>Recovery Covers the Handoffs Between Stages</h3><p>Some interruptions happen outside the implementation itself.</p><p>A worker can finish generating the roadmap but fail to launch the continuation that creates the phase work. Without another recovery path, the project would have valid planning artifacts and no active delivery process.</p><p>We add a gateway fallback for that handoff. If the worker cannot start the continuation, the gateway can pick it up after the roadmap step.</p><p>We also restore missing QA gates. If a required QA task is missing, the factory must not mark the phase as complete. We need to preserve the intended verification path.</p><p>These mechanisms address specific coordination failures: continuing after roadmap creation, restoring required checks, and sequencing repairs against retests. They let us recover useful completed work without treating an interrupted handoff as a reason to restart the entire project.</p><p>They also change what we need to show in Google Chat.</p><p>The progress card includes the revision and repair cycles created during execution. For our example, the stakeholder should see that the approval phase required a plan revision, then entered browser QA, then needed a fix and another test.</p><p>A card that shows only the original stages would conceal much of the work required to deliver the phase. It could also leave the user wondering why a completed implementation still has no verified result.</p><p>We want the visible progress to match the work that remains, including the loops we did not know about when the run started.</p><h3>Jev Handles Narrow Decisions Along the Way</h3><p>This workflow contains decisions that do not need the same depth of reasoning as a phase plan.</p><p>We added <a href="https://typesafe.ai/blog/introducing-system-one-models-and-jev">Jev</a> as a fast decision layer for those narrower questions.</p><p>Inside the factory, we use it to help decide whether a message should route to FACTORY or receive an ordinary conversational answer. It also helps recognize test-and-fix requests and low-complexity tasks, assess the need for clarification, and identify questions that require more care because of their risk.</p><p>During plan review, Jev can help distinguish a blocking finding from a cosmetic comment.</p><p>The missing backend authorization check in our example affects a stated requirement. A suggestion to rename a heading in the plan has a different consequence. Treating both findings as equivalent would add revision work without improving the same kind of correctness.</p><p>Jev supports that classification. The architect and checker still own the substantive planning work, and the engineer still executes the resulting plan.</p><p>We retain an LLM fallback. Low confidence or an unavailable Jev service sends the decision to that path. When confidence is insufficient, the system must not choose a branch that starts implementation or allows the phase to continue.</p><p>The intended optimization is to avoid a full LLM run for a short decision that a narrower decision layer can make with sufficient confidence.</p><p>We are not attaching a measured savings percentage to this integration. That would require data on routing accuracy, fallback frequency, latency, and total cost through an accepted result. A cheaper classification step would offer little benefit if incorrect routing created more planning or repair work downstream.</p><h3>Completing the Phase Means Being Able to Inspect the Result</h3><p>Return to the access-request application.</p><p>A successful result for the approval phase would let an employee submit a request and an authorized manager approve or reject it. The interface would reflect the resulting state, and the application would preserve that state after a reload.</p><p>Verification would also need to cover the less visible requirements: preventing self-approval and recording the required status changes.</p><p>A screenshot of an approved request helps demonstrate the interaction. It does not, on its own, establish that the backend rejects prohibited actions or that the application records the required history. We need evidence appropriate to each requirement.</p><p>For this example, the phase handoff should connect the original requirements with the validated plan, implementation, review results, deployment status, and QA results. The plan revision and fix/re-test cycle belong in that history because they explain how we reached the final result.</p><p>During verification, we check that evidence against the phase goals. UAT gives the stakeholder a structured opportunity to confirm the behavior, with any remaining human checks visible.</p><p>After those gates pass, the factory can start the provisioning-and-history phase on top of a verified approval flow.</p><p>The stakeholder can inspect both the application and the work that produced it. They can see which product question needed their input, which plan defect we corrected within the engineering workflow, and which browser failure led to another implementation pass.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gYuSX5nP-ntCEgsSz29GHw.png" /></figure><h3>Bringing Greenfield Delivery Into Dark Factory</h3><p>We have extended Dark Factory to cover the work that happens before an application exists.</p><p>Starting with a brief and supporting materials in chat, we can now create the project foundation, build a roadmap, and coordinate delivery through its phases. We also support new milestones for established projects while keeping the brownfield workflow for ongoing changes intact.</p><p>The engineering improvement lies in how we control progression. We generate phase tasks from the roadmap instead of forcing projects through a fixed sequence. We check plans before execution, return plan defects to the architect, and require the relevant deployments before browser QA. We keep the next phase blocked until the preceding work clears verification.</p><p>We have also made revision and recovery part of the delivery process. A blocking plan finding leads to revision and another check. A failed browser test leads to a fix and a dependent retest. We can recover the continuation after roadmap generation and restore missing QA gates. Through the progress card, stakeholders can see those cycles alongside the original phase work.</p><p>Jev gives us a narrower decision path for routing and classification, with an LLM fallback when confidence is insufficient. We reserve human input for unresolved product choices and questions that warrant it, while keeping repairable engineering defects within the engineering workflow.</p><p>Together, these changes give us more control over how a project moves from intention to implementation. We can inspect the work in progress, identify the dependency that prevents it from advancing, and distinguish a local result from a merged, deployed, and verified one. We have established those controls; any claim about speed or cost savings still needs measurements.</p><p>For the stakeholder, the starting point remains a conversation about an application they need. Behind that conversation, we now coordinate the project definition, engineering decisions, and checks required to deliver it. At the end of a phase, they can inspect the working behavior and the evidence behind its completion, with unresolved issues still visible.</p><p>That is the next step for Dark Factory: coordinating the delivery process from the first project brief, including the revisions and failed attempts required to reach a result we can verify.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b925e1709f68" width="1" height="1" alt=""><hr><p><a href="https://medium.com/emerline-tech-talk/dark-factory-part-2-from-a-greenfield-brief-to-verified-delivery-b925e1709f68">Dark Factory, Part 2: From a Greenfield Brief to Verified Delivery</a> was originally published in <a href="https://medium.com/emerline-tech-talk">Emerline Tech Talk</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dark Factory: How We Turned Google Chat into a Code Factory]]></title>
            <link>https://medium.com/emerline-tech-talk/dark-factory-how-we-turned-google-chat-into-a-code-factory-1224fdd75bd2?source=rss-be0afd09aaa1------2</link>
            <guid isPermaLink="false">https://medium.com/p/1224fdd75bd2</guid>
            <category><![CDATA[hermes-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Viachaslau Kitsun]]></dc:creator>
            <pubDate>Wed, 26 Aug 2026 09:11:42 GMT</pubDate>
            <atom:updated>2026-08-26T09:11:42.109Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dhI7j0zwqKzfGsifMXbGOQ.png" /></figure><p><em>How we connected Hermes, an orchestration gateway, and a custom Factory plugin so that a stakeholder request can become tested code on staging without an engineer coordinating every step.</em></p><h3>From UAT Requests to a Development Interface</h3><p>We built the first version of our application with spec-driven development using GSD. At that stage, the workflow fit our needs well: we defined the product through specifications, used those specifications to guide implementation, and moved through development with a relatively clear connection between what we intended to build and what was ultimately implemented.</p><p>UAT changed the nature of the work.</p><p>Once stakeholders had a real application in front of them, they stopped thinking in specifications and started reacting to what they saw. Requests came in the form you would expect from somebody using a product rather than designing one from scratch: move this button, rename that field, add another column here, change this part of the workflow, support Google sign-in.</p><p>We continued handling those requests through GSD. The stakeholder would describe the change; someone from our team would understand what they meant, connect the request to the existing specification and code, prepare the task, launch the development cycle, inspect the result, and put another version on staging.</p><p>Nothing was fundamentally wrong with that process, but after enough UAT requests, it became clear that we were spending a surprising amount of time on the part between the stakeholder expressing an intention and the development machinery receiving a usable task.</p><p>For many requests, that part did not require much engineering judgment. The stakeholder already knew what they wanted. The specification already contained the product context. The repository already contained the implementation context. GSD and the LLM-based tooling already knew how to work with both.</p><p>We started asking why an engineer still needed to sit between those pieces for every change.</p><p>Instead of taking a request from Google Chat, translating it into an engineering prompt, feeding it into the development process, and reporting the result back to the same stakeholder, we could give the stakeholder access to that process through the chat itself.</p><p>They would describe what they wanted in the same language they already used with us. An LLM could determine whether the message required development work, inspect the current product before preparing the task, and send the qualified request into an automated engineering workflow. The stakeholder would approve the run, watch what happened, intervene when necessary, and receive the resulting build on staging.</p><p>That idea became Dark Factory.</p><p>We built it around Hermes, an orchestration gateway, and a custom version of our Factory plugin. Together, they turned Google Chat from a place where development requests arrived into an interface for running the development process itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pV99knztPD3mgKrazfiBng.png" /></figure><h3>Before We Change Code, We Need To Understand What the User Actually Wants</h3><p>Giving stakeholders a chat connected to the development environment created an immediate ambiguity: not every message about the application asks us to change it.</p><p>A user can write:</p><p><em>How do I restart the approval flow?</em></p><p>and then, in the same conversation:</p><p><em>Add a button to restart the approval flow from this screen.</em></p><p>The first message asks about the current product. The second asks us to make a different product.</p><p>That distinction matters because starting a development workflow for an ordinary question would be wasteful. Even a small factory run involves context collection, model calls, environments, agents, testing, and review. At the same time, forcing the user to select a special “development mode” before every request would make the chat less useful.</p><p>We therefore make qualification the first step of every interaction.</p><p>A dedicated classification agent receives the user message together with the relevant conversation and application context. It uses an LLM to determine whether the user is asking a question about the existing solution or requesting a change to that solution.</p><p>If the user wants to know how something works, Hermes answers in the same conversation. The user can ask where a feature lives, what a particular state means, how to start a process, or why the current application behaves in a certain way. We do not start the factory for those questions.</p><p>If the classifier recognizes a development request, the plugin starts a different flow. At that point, we know that the stakeholder wants the application to change, but we still do not send their raw message to a developer agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xzpMb4yawq5gppjcpp5cQQ.png" /></figure><h3>A Stakeholder Request Is Not the Same Thing as an Engineering Prompt</h3><p>A stakeholder should be able to write:</p><pre>Add Google sign-in.</pre><p>That is a valid product request. It tells us what outcome they want.</p><p>It does not provide an engineering agent with enough information about the application to implement the feature effectively. The agent still needs to understand how authentication works today, how the project represents users, which frontend and backend components participate in login, how configuration works, whether an existing account should link to Google by email, and which failure scenarios the implementation needs to handle.</p><p>One way to address this would be to ask the stakeholder a series of technical questions. We would end up rebuilding a ticket form inside a chatbot.</p><p>Instead, Dark Factory assigns the qualified request to a prompt-augmentation agent.</p><p>This agent works against the actual project. It receives the original request and conversation context, but it can also inspect the specification and source code. Its job is to understand how the requested change fits into the application before anybody starts writing code.</p><p>For a small UI request, it may find the relevant component, inspect the current styles and states, identify nearby patterns, and turn “make this button smaller” into a prompt that points the developer to the right place and defines what should remain unchanged.</p><p>For a larger feature, it can inspect several parts of the system and prepare a task that includes the current architecture, affected flows, data structures, dependencies, technical constraints, and acceptance criteria.</p><p>The important part is that the augmentation agent does not invent this context from the user’s one-line request. It derives as much as possible from the existing specification and codebase.</p><p>This changes the quality of the input we give downstream agents. A developer does not have to spend the first part of every run guessing which interpretation of the user’s request makes sense. The factory has already connected that request to the current implementation.</p><p>It also preserves the reason we wanted the chat interface in the first place. Stakeholders can describe intent in product terms rather than learning to write prompts for coding agents.</p><h3>The Request Determines Which Agents Participate</h3><p>Once the plugin has a grounded implementation prompt, it can decide what kind of engineering process the task needs.</p><p>We learned early that putting every request through a universal agent chain makes little sense. If somebody asks us to move a button three pixels to the left, adding an analyst and an architect does not improve the result enough to justify their cost. If someone asks us to introduce a new authentication mechanism, skipping the architecture and security work can lead to much more expensive problems later.</p><p>The Factory plugin, therefore, builds an execution chain for each request.</p><p>A narrow frontend task may require a developer, a UI autotester, a reviewer, and a deployment stage. A change that touches authentication, persistence, or several parts of the application can involve an analyst, architect, developer, security agent, UI autotester, reviewer, and deployment engineer.</p><p>The names of the roles matter less than the principle behind them: the task defines the workflow.</p><p>Each selected agent receives a bounded responsibility. An analyst resolves gaps in functional requirements when they matter. An architect deals with structural decisions that should not be left to a developer in the middle of implementation. A security agent reviews the parts of the change that warrant a security review. The reviewer inspects the resulting implementation rather than participating in its creation.</p><p>This also gives us one of the first opportunities to control the Factory’s economics. Every agent adds model calls, context, execution time, and other dependencies that can fail. We only want that cost when the role provides enough value for the task.</p><p>Before any of those agents start working, the stakeholder sees the proposed run in Google Chat.</p><p>The custom plugin creates a launch card that contains the interpreted request and the planned stages. The user reviews it and presses <strong>Start</strong>.</p><p>That approval matters because the system may have produced a technically sophisticated interpretation of a request that the stakeholder never intended. The classification and augmentation stages can reduce ambiguity, but they cannot replace the person who knows what outcome they wanted in the first place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G6CGVNxR9rBeyw1yNQoaCA.png" /></figure><h3>Execution Is a Loop, Not a Conveyor Belt</h3><p>After approval, the Factory plugin starts the selected chain and coordinates the handoffs between agents.</p><p>If the task needs analysis or architecture, those agents produce context that the developer receives together with the augmented prompt. The developer works on the current codebase and implements the feature. From there, the Factory moves into verification.</p><p>We wanted verification to include the application that the stakeholder would use, not only static analysis of the code.</p><p>For that reason, the Factory has a dedicated UI autotester built around Playwright. The testing agent opens the running application in a browser, navigates to the affected area, clicks controls, enters data, follows the relevant flow, and checks the result against the expected behavior.</p><p>While doing that, it takes screenshots.</p><p>The plugin sends links to those screenshots back into the Google Chat thread. The stakeholder can see what the test agent saw without opening another system. For visual changes, the screenshots provide a quick way to inspect the result. For functional failures, they provide evidence of the state in which the test broke.</p><p>The testing stage became much more useful once we connected it to an automated repair cycle.</p><p>If the Playwright agent finds a bug, we do not just mark the run as failed and hand the task back to the development team. The plugin spawns an additional engineer and gives that agent the original task, current implementation, test output, available logs, and screenshots from the failing scenario.</p><p>The engineer fixes the issue and sends the result back to verification.</p><p>The reviewer works in a similar loop. If the review uncovers a broken edge case, a poor implementation decision, or a mismatch with the requested behavior, the plugin can create another engineering stage with that feedback. The new implementation returns to the checks that need to run again.</p><p>A real Factory run can therefore contain several development and verification passes. The initial developer produces the first candidate implementation, tests it, and reviews it, trying to break it. Additional engineers address the findings until the selected checks pass or the run reaches a condition where we want a human to decide what happens next.</p><p>That feedback loop is more important to us than simply having several agents with different job titles. Without it, a multi-agent pipeline can still amount to several models taking turns producing text. With it, the outputs from one stage create concrete work for another, and the system can recover from a meaningful class of implementation mistakes without restarting the entire request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_mITF8Lymc3TnTK632JJRg.png" /></figure><h3>Google Chat Shows the State of the Factory While It Runs</h3><p>A task that involves several agents and repair cycles can run long enough that the stakeholder needs to understand where it is.</p><p>We keep that state inside the conversation where the request started.</p><p>The Factory plugin maintains one live Google Chat card for the run. It shows the stages selected for that request that have been completed, who owns the task now, and how long each completed stage took. When the Factory starts a repair stage, the card reflects that work as part of the same run.</p><p>The plugin updates the existing card instead of posting a stream of status messages. Test screenshots appear as links in the thread, while the card remains the place where the stakeholder can see the overall state.</p><p>The same interface provides control over the execution. The user can stop the Factory. When a stage becomes blocked, the plugin can expose <strong>Retry</strong> or <strong>Continue</strong>, depending on the situation. Retry gives the stage another attempt; Continue accepts the available output, allowing the chain to move forward.</p><p>Stopping a run also means stopping the work behind the card. The plugin terminates the active execution, records which stages completed, and leaves the final state visible in the conversation.</p><p>This part sounds like interface work, but it changes how usable the system feels. Once development runs without somebody manually coordinating it, the stakeholder needs enough visibility to distinguish “the Factory is working” from “something disappeared into an LLM call fifteen minutes ago.”</p><p>The chat thread ends up containing the original request, the Factory’s interpretation, launch approval, current progress, screenshots, repair history, and the final result. We do not ask the stakeholder to assemble that story from a ticket tracker, an agent console, a CI system, and a deployment dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LEWY3piTSZ8tZ1FkSnIbSw.png" /></figure><h3>The Current End Point Is Staging</h3><p>Once the selected tests and reviews pass, the deployment stage publishes the result to staging.</p><p>The stakeholder receives the outcome in the same thread where the request started and can open the application to inspect the change in context. At that point, they can decide whether the Factory understood the request and whether the implementation is sufficient to move forward.</p><p>We stop the automated flow there.</p><p>For now, we consider staging the right boundary for this level of autonomy. Dark Factory can modify a real application, execute code, interact with the UI, repair problems, and deploy a build, but a successful agent chain still does not give us enough confidence to turn an arbitrary natural-language request into a production release without another control point.</p><p>That boundary may move as we improve the system. We have no reason to move it just to make the architecture look more autonomous.</p><h3>The Expensive Part Is Not Only the Coding Agent</h3><p>The most obvious weakness of the current approach is cost.</p><p>Looking at the price of the developer model alone gives a misleading picture of what a Factory run consumes. Before development starts, we may already have paid for classification and prompt augmentation. A larger task can then add analysis and architecture. After implementation comes security work, browser testing, review, and deployment. If testing or review finds an issue, another engineer joins the run, and parts of the verification process are repeated.</p><p>Several agents also need an overlapping project context. The specification may be large. The repository may contain enough code that selecting the right context becomes a problem in itself. Repair engineers need information from previous stages in addition to the original task.</p><p>A single stakeholder request can therefore consume far more tokens than the coding step suggests.</p><p>We are still working on model assignment across the Factory.</p><p>Using the strongest available model for every role wastes money. A classification agent makes a narrow decision and does not need the same capabilities as an architect working across several subsystems. At the same time, choosing the cheapest model for a developer or reviewer can create a false economy. If a weak developer produces a poor implementation, the Factory pays for testing, repair, and another round of verification. If a weak reviewer misses a defect, the cost appears later in a different form.</p><p>For that reason, we have started thinking in terms of the cost of an accepted result on staging rather than the cost of an individual LLM call.</p><p>The model is only one variable. Context size matters. Agent selection matters. The number of allowed repair cycles matters. Even the order of verification stages can affect how much work the Factory repeats after a finding.</p><p>Finding the right combination is still an open part of the project.</p><h3>Some Requests Are Cheaper To Do by Hand</h3><p>The Factory also introduces latency that becomes hard to justify for very small tasks.</p><p>An experienced developer can sometimes make a tiny UI change, open the page, verify it, and commit the result faster than Dark Factory can classify the request, augment the prompt, prepare an environment, start an agent, run the browser test, and complete review.</p><p>Dynamic team selection helps because we do not send those tasks through the architecture and security work. It cannot remove the fixed cost of orchestration.</p><p>This creates a boundary we have not fully mapped yet. For some classes of work, automation simplifies coordination and gives the stakeholder a much shorter path to results. For others, especially tiny low-risk changes, the Factory can spend more resources than the implementation deserves.</p><p>The obvious answer is not to remove those requests from the system. We need better routing that takes expected effort and risk into account, then chooses how much of the Factory a request should use.</p><h3>More Agents Do Not Guarantee the Right Interpretation</h3><p>The system can also produce a technically correct solution to the wrong problem.</p><p>Imagine that the augmentation agent interprets an ambiguous request in a way the stakeholder did not intend. An architect can create a coherent design based on that interpretation. The developer can implement it well. The tester can validate the behavior against acceptance criteria derived from the same prompt, and the reviewer can confirm that the implementation matches the prepared task.</p><p>Every stage can do its job correctly, while the final result still misses the original intent.</p><p>The launch card gives the stakeholder a chance to catch this before execution, and grounding the prompt in the specification and source code reduces the number of arbitrary assumptions. Neither measure solves the problem in full.</p><p>This is one reason we do not consider the Factory a replacement for product judgment. It automates much of the development work after someone expresses intent, but intent itself remains ambiguous, and another LLM cannot always resolve it by adding more reasoning.</p><h3>The Factory Inherits the Inconsistencies of the Project</h3><p>Dark Factory works best when the project gives agents a reliable picture of the system.</p><p>Spec-driven development helps because we already have structured specifications that explain much of the expected behavior. The codebase provides a second source that shows the current implementation.</p><p>Those sources can disagree.</p><p>A team can change behavior without updating a specification. An old flow can remain in documentation after the product moves on. The source code can contain two competing patterns after a partial refactoring.</p><p>Human engineers often resolve those situations through knowledge that never made it into either artifact. They remember why something changed, which implementation is transitional, or which paragraph of a specification nobody should trust anymore.</p><p>An agent does not have that institutional memory unless we give it access to another source that contains it.</p><p>When the project context contradicts itself, the Factory can make the wrong choice with high confidence. Improving retrieval and context selection helps, but the deeper problem is that autonomous development amplifies the value of keeping engineering artifacts accurate.</p><h3>Automated UI Testing Can Create False Repair Work</h3><p>Playwright testing catches problems that code review cannot, but it brings the normal weaknesses of browser automation into an autonomous workflow.</p><p>A page can take longer to load. Test data can change. A selector can become stale. An external service can fail. A modal can appear at a different point because of timing.</p><p>The testing agent then has to decide whether it found a defect in the application or a defect in its own test environment.</p><p>That distinction matters much more when a failed test can spawn another engineer.</p><p>In a conventional pipeline, a flaky test produces a red build and annoys a developer. In Dark Factory, a false positive can cause another LLM to inspect correct code and modify it in an attempt to fix a nonexistent problem.</p><p>We therefore need strong evidence around failures: screenshots, browser state, logs, controlled retries, and enough context for the Factory to distinguish a transient execution problem from a product defect before it starts rewriting code.</p><p>We have improved this, but we do not consider it solved.</p><h3>Repair Loops Need Budgets</h3><p>The repair mechanism enables many runs to recover from defects without human intervention, making it one of the most valuable parts of the system.</p><p>It can also become one of the most expensive.</p><p>A tester finds an issue and spawns an engineer. The engineer fixes it, but the reviewer finds another problem. The next change affects the original browser scenario, so the tester fails again. The Factory can keep making plausible progress while the cost of the run grows far beyond the value of the stakeholder request.</p><p>An autonomous workflow, therefore, needs limits on its own persistence.</p><p>We need to decide how many repairs we allow, how much of the token budget a single request can consume, how long a run can continue, and which repeated failure patterns should stop the chain and prompt a human engineer to look at the problem.</p><p>Those thresholds involve tradeoffs. Stopping too soon interrupts tasks that the Factory could have recovered on the next pass. Wait too long, and a trivial request can consume the budget of a much larger feature.</p><p>We are still tuning that part of the system from real runs rather than treating it as a solved configuration problem.</p><h3>LLM Providers Have Become Infrastructure Dependencies</h3><p>We also learned that model providers need the same reliability thinking as any other critical dependency in the pipeline.</p><p>We have seen providers reject requests that some agents produced. We have hit usage limits while active work is underway. Either problem can leave a task stuck after earlier agents have already consumed tokens and modified the working state.</p><p>A failure in a normal chatbot costs one response. A failure in the middle of a development chain can waste work from several earlier stages and leave us in a state that the orchestrator needs to recover from.</p><p>Rate limits, context limits, quotas, provider availability, and model changes therefore affect Dark Factory’s architecture. We need fallback behavior and enough execution state to resume or fail a run without losing track of what has already happened.</p><p>This is also part of why we are cautious about optimizing the whole Factory around a single model provider or model family.</p><h3>Debugging an LLM Workflow Requires Much More History</h3><p>Dark Factory also lacks the determinism we expect from conventional automation.</p><p>Two developer agents can receive the same task and choose different implementations. A repair engineer can address a review comment in the same way as the original developer. A provider can update a model and change agent behavior while our own code remains untouched.</p><p>When one of those runs goes wrong, the final Git diff does not tell us enough.</p><p>To understand what happened, we need the original user message, the augmented prompt, the models assigned to each stage, the agents’ outputs, tool activity, test evidence, review findings, repair history, and the state transitions managed by the plugin.</p><p>That execution history is becoming part of the product rather than diagnostic data we can add later. Without it, debugging the Factory means trying to reconstruct why a sequence of probabilistic components made the decisions they did.</p><h3>Useful Agents Also Need Dangerous Permissions</h3><p>The Factory only provides value because its agents can work with real systems.</p><p>A developer needs access to source code and build tools. The Playwright agent needs a running application, a browser, and test credentials. The deployment stage needs permissions that neither of those roles should receive.</p><p>Giving every agent access to everything would make orchestration much simpler, but the resulting system would be much harder to trust.</p><p>We isolate agents and implement role-specific access so that each stage receives the files, tools, credentials, and network permissions it needs, rather than inheriting everything available to the Factory.</p><p>The problem becomes more important as we give agents more autonomy. Once an LLM can edit code, execute commands, interact with services, and trigger deployments, permission boundaries stop being an implementation detail.</p><p>This is another reason staging remains our endpoint today.</p><h3>Where Dark Factory Is Today</h3><p>The stable version we run now starts with a normal message in Google Chat.</p><p>A classification agent determines whether the stakeholder wants information about the existing application or wants to change it. Hermes handles ordinary product questions in the conversation. Development requests go to an augmentation agent, which combines the stakeholder’s intent with the specification and current source code to prepare a grounded engineering prompt.</p><p>The custom Factory plugin uses that prompt to select the agents required for the task and shows the planned run to the stakeholder for approval.</p><p>After approval, the selected agents execute the task. A Playwright-based autotester interacts with the running application and sends screenshot links into the chat. Reviewers inspect the implementation. Test or review findings can spawn additional engineers, whose fixes return to verification without forcing the stakeholder to create another request.</p><p>Throughout the run, the Google Chat card shows the current state and exposes the controls we want a person to retain. Once the selected checks pass, the deployment stage publishes the result to staging.</p><p>The system still has rough edges. It consumes a lot of tokens, and we have not finished finding the right model for each role. Small tasks can suffer from orchestration overhead. Agents can share the same misunderstanding. Project context can drift. Browser automation can create false failures. Repair loops need limits. Model providers can fail mid-run. Debugging requires a detailed execution history, and useful autonomy creates a real permissions problem.</p><p>Those issues have changed the way we think about the project.</p><p>At the beginning, the question was whether we could let a stakeholder describe a change in Google Chat and automate the path from that request to working code.</p><p>We can.</p><p>The harder work now lies in deciding how to make that path economical, observable, recoverable, and safe enough for us to trust it with a growing share of day-to-day development. Dark Factory has reduced the amount of manual coordination between a stakeholder request and a build on staging, but it has also turned that coordination into an engineering system of its own, with all the cost, reliability, and control problems that come with one.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1224fdd75bd2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/emerline-tech-talk/dark-factory-how-we-turned-google-chat-into-a-code-factory-1224fdd75bd2">Dark Factory: How We Turned Google Chat into a Code Factory</a> was originally published in <a href="https://medium.com/emerline-tech-talk">Emerline Tech Talk</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From No-Code to AI Engineering with SDD]]></title>
            <link>https://medium.com/emerline-tech-talk/from-no-code-to-ai-engineering-with-sdd-ab0d159b2196?source=rss-be0afd09aaa1------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab0d159b2196</guid>
            <category><![CDATA[react]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[spec-driven-development]]></category>
            <category><![CDATA[openai-codex]]></category>
            <dc:creator><![CDATA[Viachaslau Kitsun]]></dc:creator>
            <pubDate>Tue, 21 Jul 2026 08:20:38 GMT</pubDate>
            <atom:updated>2026-07-21T08:20:38.709Z</atom:updated>
            <content:encoded><![CDATA[<h3>From No-Code to AI Engineering. Evolving from AppSheet to Spec-Driven Development with Get.Ship.Done</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TVdfnfiykdK6HL43nbVXQg.png" /></figure><p>Most internal automation projects don’t start as formal software products. They usually begin with a practical, immediate need: eliminate manual repetition, smooth out operational friction, and give teams a better way to handle daily tasks.</p><p>It usually starts small. Someone manages a spreadsheet; another sends status updates via chat; approvals happen over email; data is manually copied between systems. Individually, these workflows seem harmless, but collectively they evolve into a hidden tax on the entire organization.</p><h3>From No-Code Automation to Custom Development</h3><p>We started internal automation with Google AppSheet because it solved the immediate problem: we could quickly turn spreadsheets into working apps, add forms, set up approvals and simple workflows, and give business users tools they could use almost immediately.</p><p>At that stage, no-code was the right choice. It helped us move faster, test ideas with real users, and understand which processes were stable enough to automate.</p><p>But as the system grew, the limitations became clear. We hit constraints around UI flexibility, performance, database structure, permissions, and more complex business logic. Some requirements could be implemented only through workarounds; others could not be implemented at the quality level we needed.</p><p>The system was no longer a quick internal tool. It had become part of daily operations, and the no-code layer was slowing us down rather than helping us move faster.</p><p>That is why we moved to full custom development. With React, .NET, PostgreSQL, Google SSO, and Azure, we regained control over the user experience, data model, performance, integrations, security, and deployment process.</p><h3>From AI Coding to Spec-Driven Development</h3><p>Moving to custom development did not mean returning to slow, fully manual engineering. AI became a natural part of development, helping with boilerplate, refactoring, debugging, documentation, and faster feature implementation.</p><p>But using AI directly is not enough. Asking an agent to “build a feature” from a short prompt works for prototypes, not for business-critical systems. Internal platforms depend on specific rules: who can approve what, which data is visible to whom, how changes are audited, and how edge cases should behave.</p><p>Without clear requirements, AI fills the gaps by guessing. The result may look functional, but it can easily miss the actual business logic.</p><p>That is why we started using <strong>Spec-Driven Development (SDD)</strong> with Get.Ship.Done (GSD). Instead of relying on vague prompts, we define the work through structured requirements, technical plans, and execution steps.</p><p>GSD gives AI the context it needs before implementation starts. Humans define the intent and constraints; GSD turns them into clear specs; AI agents build within those boundaries. This makes AI useful not just for faster coding, but for controlled, maintainable product development.</p><h3>Spec-Driven Development with GSD</h3><p>SDD treats the specification as the main source of truth for implementation. Business rules, technical constraints, expected behavior, acceptance criteria, security requirements, and architectural decisions are defined before code is generated. This gives developers and AI agents a clear contract to work against.</p><p>This is especially important with AI agents. When requirements are vague, the agent must fill the gaps with assumptions. The result may compile, pass basic tests, and still implement the wrong business behavior. SDD reduces this risk by making critical decisions explicit before execution begins.</p><p>GSD implements this approach by storing project knowledge as versioned artifacts in the .planning/ directory. Requirements, roadmap phases, implementation decisions, UI specifications, plans, project state, and verification results become shared project memory rather than temporary chat context. These artifacts can be reviewed, committed to Git, and reused by different agents across multiple sessions.</p><p>The workflow moves through a clear sequence. First, the project is divided into phases with defined goals and success criteria. The discussion step resolves unclear areas such as permissions, state transitions, validation, data ownership, audit rules, and integrations. GSD records these decisions in CONTEXT.md, then converts them into executable plans with requirement references, file-level tasks, dependencies, and acceptance criteria.</p><p>During execution, specialized agents can work on separate parts of the solution, such as the database, backend, frontend, and tests, while using the same approved specification. Verification then checks the implementation against the original requirements and records evidence in verification and UAT artifacts. A phase is complete not when the code simply exists, but when the expected behavior has been proven.</p><p>In practice, GSD acts as the execution framework for SDD. The specification defines what must be built and which boundaries must be respected. GSD turns that specification into phases, plans, implementation tasks, verification evidence, and finally a pull request. This creates a traceable path from the original requirement to the delivered code, making AI-assisted development a controlled engineering process rather than an uncontrolled code-generation loop.</p><h3>Preparing the Initial Context for GSD</h3><p>Before starting the GSD workflow, we usually collect high-level requirements in a free-form document. At this stage, the goal is not to produce a complete specification. We capture the business problem, target users, key workflows, expected functionality, known constraints, and preferred technology stack in a format that is quick to create and easy for stakeholders to review.</p><p>We use these requirements as the input prompt for generating two initial artifacts: an Architecture Vision document and a set of UI mockups. The Architecture Vision formalizes the system boundaries, main business flows, technology choices, initial data model, security assumptions, and key architectural decisions. The mockups provide a visual reference for navigation, screen structure, user actions, and the overall product experience.</p><p>Together, these artifacts become the starting context for GSD. The Architecture Vision defines what the system is and how it should be structured, while the mockups show how users are expected to interact with it. GSD then uses this shared reference to generate requirements, roadmap phases, phase context, UI specifications, and implementation plans without starting from an abstract or ambiguous prompt.</p><h3>The GSD Advantage</h3><p>GSD provides the operating model that AI development often lacks. It shifts project knowledge from fleeting chat sessions into versioned, durable artifacts in a .planning/ directory.</p><p>This prevents the “context rot” typical of standard AI chats. When a session is cleared, the decisions don’t vanish. Instead, the workflow follows a repeatable loop: <strong>Discuss → Plan → Execute → Verify → Ship</strong>. Every step consumes artifacts from the previous one, making the entire process traceable and resilient to context resets.</p><h3>GSD Installed in the Project</h3><pre>npx @opengsd/gsd-core@latest<br>/gsd-new-project - auto @internal-platform-brief.md</pre><p><strong>Caption:<br></strong><em>We initialize GSD within the repository and provide the initial product brief for the internal automation platform.</em></p><p>In practice, we start by initializing GSD and defining a structured brief — transforming a vague idea into a “versioned project memory.” This ensures the AI isn’t just improvising, but working from a consistent, documented source of truth.</p><h3>From Idea to Requirements</h3><p>For example, suppose we want to build an <strong>Employee Access Request</strong> module.</p><p>In a traditional AI workflow, we might prompt:</p><blockquote>Build a page where employees can request access to internal systems, and managers can approve it.</blockquote><p>That is too vague.</p><p>With GSD, we start from a brief:</p><pre># Employee Access Request Module<br>Employees should be able to request access to internal systems.<br>Managers should approve or reject requests.<br>IT admins should see approved requests and mark provisioning as completed.<br>All status changes must be auditable.<br>Authentication must use Google SSO.<br>The frontend is React SPA.<br>The backend is .NET.<br>The database is PostgreSQL.<br>The infrastructure runs in Azure.<br>GSD turns this into structured planning artifacts.</pre><p>The generated .planning/ directory becomes the project memory. According to the docs, GSD Core stores project state in .planning/ as human-readable Markdown and JSON files that can be committed to git.</p><h3>The .planning/ Directory</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KtRRthlxjGneNlc_mjh2dg.png" /></figure><p><strong>Caption:<br></strong><em>Instead of keeping decisions inside chat history, GSD stores them as project artifacts that developers and agents can both read.</em></p><p>The root artifacts include PROJECT.md, REQUIREMENTS.md, ROADMAP.md, STATE.md, and config.json; the quickstart documentation describes these as project vision, scoped requirements, roadmap, living memory, and workflow configuration.</p><p>This is one of the biggest differences from vibe coding. The AI is no longer working from a temporary prompt. It is working from a versioned project memory.</p><h3>Discuss Before Planning</h3><p>The discussion phase clarifies the “gray areas” — delegation rules, audit depth, or identity mapping — before a single line of code is written. These answers are recorded in a CONTEXT.md file so subsequent agents don’t need to ask the same questions again.</p><pre>/gsd-discuss-phase 1 - power</pre><p>For our Employee Access Request module, this discussion may clarify questions like:</p><ul><li>Should requests support only one system at a time or multiple systems?</li><li>Can managers delegate approvals?</li><li>Should IT admins be able to reject already approved requests?</li><li>Do we need a full audit log or only the latest status?</li><li>Should Google groups map to application roles?</li><li>Should backend authorization be role-based, policy-based, or both?</li></ul><p>The important part is that these answers are written down before the agent plans the implementation.</p><p>GSD’s <em>/gsd-discuss-phase</em> command loads the existing context, scouts the codebase for reusable patterns, identifies unresolved gray areas, and writes a phase CONTEXT.md file so downstream agents do not need to ask the same questions again.</p><h3>Phase Context</h3><pre># Phase 1: Platform, Access, Setup, and Audit Foundations - Context<br>**Gathered:** 2026–06–24<br>**Status:** Ready for planning<br>&lt;domain&gt;<br>## Phase Boundary<br>Phase 1 establishes the React/TypeScript frontend, .NET Web API backend, EF Core/PostgreSQL persistence, Firebase-backed authentication, project-scoped authorization, setup/admin data management, and foundational audit records. It must enable users to access the app, operate within scoped permissions, maintain clients/projects/users/reporting periods/lookups, and trust that financial mutations are traceable.<br>&lt;/domain&gt;<br>&lt;decisions&gt;<br>## Implementation Decisions<br>### Identity Provider<br>- **D-01:** Use Firebase Authentication by Google for user sign-in.<br>- **D-02:** React frontend obtains Firebase ID tokens and sends them to the .NET Web API on authenticated requests.<br>- **D-03:** The .NET Web API verifies Firebase ID tokens server-side using Firebase Admin SDK token verification before creating the request user context.<br>- **D-04:** Firebase UID is the external identity key; application user profiles, organization membership, scoped roles, and project assignments live in PostgreSQL.<br>- **D-05:** Do not use Firebase custom claims as the source of truth for project-scoped RBAC in Phase 1.<br>### Project Roles and Scopes<br>- **D-06:** Implement the full MVP role set in Phase 1: Project Manager, Delivery Manager, and Admin.<br>- **D-07:** Authorization must support scoped assignments, not just global roles.<br>- **D-08:** Assignments can apply at project, portfolio, or organization scope, so the Delivery Manager and Finance workflows are not forced into one-project assignments.<br>- **D-09:** Backend authorization policies must enforce organization/project visibility and mutation permissions for every command/query/export path introduced in Phase 1.<br>### Setup Workflow<br>- **D-10:** Build user-facing admin screens in Phase 1 for setup data rather than seed/API-only setup.<br>- **D-11:** Admin screens should cover users, roles/assignments, clients, projects, reporting periods, and lookup values needed by MVP workflows.<br>- **D-12:** Setup screens should support create, edit, and archive/soft-delete behavior; avoid hard deletes for business records.<br>- **D-13:** Seed data is still useful for local development and tests, but it must not be the only way to configure MVP setup data.<br>### the agent&#39;s Discretion<br>- The planner may choose exact Firebase UI patterns, token attachment mechanics, and backend middleware structure as long as the decisions above are preserved.<br>- The planner may choose an exact admin screen layout and routing details consistent with the React application structure.<br>- After research, the planner may choose whether validation uses built-in ASP.NET Core validation, FluentValidation, or another standard .NET approach.<br>&lt;/decisions&gt;<br>&lt;canonical_refs&gt;<br>## Canonical References<br>**Downstream agents MUST read these before planning or implementing.**<br>### Project Scope<br>- `.planning/PROJECT.md` - Project vision, constraints, preferred stack, and locked architecture decisions.<br>- `.planning/REQUIREMENTS.md` - Phase 1 requirement IDs and traceability.<br>- `.planning/ROADMAP.md` - Phase 1 goal, requirement mapping, and success criteria.<br>- `.planning/research/STACK.md` - Corrected stack decision: React frontend, .NET Web API backend, EF Core DAL, PostgreSQL database.<br>### Firebase Authentication<br>- Firebase docs via Context7 `/websites/firebase_google` - Firebase client sends ID tokens from `getIdToken()`; trusted .NET backend verifies using Firebase Admin SDK `VerifyIdTokenAsync`.<br>&lt;/canonical_refs&gt;<br>&lt;code_context&gt;<br>## Existing Code Insights<br>### Reusable Assets<br>- None. The repository is greenfield and currently has no `src/` or `apps/` source files.<br>### Established Patterns<br>- No implementation patterns exist yet. Planning should establish the initial solution structure, frontend app structure, backend layering, EF Core migration pattern, authorization policy conventions, and audit persistence pattern.<br>### Integration Points<br>- New code will introduce an initial React app, a .NET solution/project, PostgreSQL/EF Core infrastructure, Firebase authentication integration, admin routes/screens, and foundational API endpoints.<br>&lt;/code_context&gt;<br>&lt;specifics&gt;<br>## Specific Ideas<br>- Identity provider preference is Firebase Authentication by Google.<br>- PostgreSQL remains the authority for app-specific user profiles and scoped RBAC; Firebase only proves identity.<br>- Admin setup screens are part of Phase 1 because setup data should be manageable in the product, not only through seeds.<br>&lt;/specifics&gt;<br>&lt;deferred&gt;<br>## Deferred Ideas<br>None - discussion stayed within the scope of the phase.<br>&lt;/deferred&gt;</pre><p><strong>Caption:<br></strong><em>The discussion phase converts business assumptions into explicit implementation decisions.</em></p><p>This is where GSD becomes practical. It forces us to answer the questions that AI would otherwise guess.</p><h3>UI Contracts for Frontend Work</h3><p>For frontend-heavy phases, we can add a UI specification before planning:</p><p><em>/gsd-ui-phase 2</em></p><p>This is useful for a React SPA because UI work is often where vague prompts create inconsistent output. A UI contract can define the page structure, components, states, validation behavior, empty states, loading states, and error messages.</p><p>The GSD workflow command reference describes /gsd-ui-phase as a command that produces a UI-SPEC.md design contract for frontend phases, covering component structure, interaction patterns, and brand alignment.</p><h3>UI Specification for the React SPA</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Zw-DvuodqM1e7Zt90FqNg.png" /></figure><p><strong>Caption:<br></strong><em>For React work, GSD can create a UI contract before implementation so the agent does not invent screens ad hoc.</em></p><h3>Planning: From Requirements to Atomic Tasks</h3><p>GSD then breaks the requirements into vertical slices — database, API, UI, and tests — generating executable PLAN.md files. These plans are explicit enough to prevent the agent from guessing business rules, including acceptance criteria and mandatory file references.</p><pre>/gsd-plan-phase 1 - mvp - tdd</pre><p>This is where GSD breaks the work into executable plans.</p><p>For our module, a plan may be split into vertical slices:</p><p>Plan 1: database schema and domain model.</p><p>Plan 2: backend API and authorization policies.</p><p>Plan 3: React UI for submitting and reviewing requests.</p><p>Plan 4: tests and verification.</p><p>The public docs say /gsd-plan-phase runs the planning pipeline, including optional research, the planner agent, and the plan-checker verification loop; it produces executable PLAN.md files and research artifacts when research runs.</p><p>A good plan is not just a task list. GSD plans include acceptance criteria, initial reading files, requirement links, and dependency ordering. The plan-checker verifies plans before they are approved.</p><h3>Generated Plan with Acceptance Criteria</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fQjbaWtssbEPJ9P98ZVQ7w.png" /></figure><p><strong>Caption:<br></strong><em>GSD plans are explicit enough for an executor agent to implement without having to guess the business rules.</em></p><p>This is the key improvement over vibe coding. The agent is no longer improvising from a loose request. It is executing a reviewed plan.</p><h3>Cross-AI Review Before Implementation</h3><p>For important phases, we can ask other AI tools to review the plan before execution:</p><pre>/gsd-plan-review-convergence 1 - codex - gemini</pre><p>This gives us another safety layer. Different models may find different problems: missing edge cases, weak acceptance criteria, security concerns, or architectural inconsistencies.</p><p>GSD supports a cross-AI convergence loop where external AI reviewers review a phase plan, high-severity findings are fed back into the planner, and the cycle repeats until no high-severity concerns remain or a cycle limit is reached.</p><h3>Cross-AI Review Findings</h3><pre># REVIEWS.md<br>Reviewer: Codex<br>Severity: HIGH<br>Finding: The plan does not specify whether rejected requests can be resubmitted.<br>Reviewer: Gemini<br>Severity: MEDIUM<br>Finding: Audit log retention requirements are not defined.<br>Resolution:<br>- Added status transition rule for resubmission.<br>- Added audit retention assumption to CONTEXT.md.</pre><p><strong>Caption:<br></strong><em>Before coding starts, another model can challenge the plan and force missing decisions to become explicit.</em></p><h3>Execution with Fresh-Context Agents</h3><p>Once the plan is approved, we execute:</p><pre>/gsd-execute-phase 1</pre><p>This is where GSD’s agent model matters.</p><p>Instead of one long-running chat session trying to remember everything, GSD uses specialist subagents with fresh context windows. The docs describe GSD Core as using thin orchestrators that spawn specialized agents for research, planning, execution, verification, and audits. Each agent receives only the artifacts it needs and writes its output back to disk.</p><p>For execution, GSD groups plans into dependency-aware waves and spawns executor agents. The workflow reference says <em>/gsd-execute-phase</em> discovers plan files, analyzes dependencies, groups work into parallel waves, spawns subagents, and collects results.</p><p>This is a practical productivity boost. One executor can work on the PostgreSQL migration. Another can work on the .NET API. Another can work on the React UI. But they all work from the same requirements, context, and plan.</p><h3>Parallel Execution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2QdFQuslRWhjhfcnpgX68Q.png" /></figure><p><strong>Caption:<br></strong><em>GSD can execute phase plans in parallel waves while keeping each agent focused on a small, explicit scope.</em></p><p>After execution, each executor writes a summary of what was built, including file paths, commits, deviations from the plan, and issues encountered. The verifier later reads these summaries and cross-checks them against the actual codebase.</p><h3>Verification Is Part of the Workflow</h3><p>In vibe coding, verification is often informal: run the app, click around, fix visible bugs.</p><p>With GSD, verification is a required phase.</p><pre>/gsd-verify-work 1</pre><p>The verifier checks what was built against the requirements and writes evidence. Then the human can run structured UAT.</p><p>GSD’s quickstart describes verification as a step that reads VERIFICATION.md, walks the user through acceptance testing, records results in UAT.md, and generates targeted fix plans if problems are found.</p><h3>Verification and UAT</h3><pre># 01-VERIFICATION.md<br>Verdict: PASS<br>Evidence:<br>- AccessRequestsController created.<br>- Authorization policies enforced.<br>- PostgreSQL migration applied.<br>- AuditLog table receives status transition entries.<br>- Unit tests pass.<br>- API integration tests pass.<br># 01-UAT.md<br>Scenario: Employee submits an access request.<br>Result: PASS<br>Scenario: Manager approves the request.<br>Result: PASS<br>Scenario: A non-manager attempts to approve the request.<br>Result: PASS - received 403.</pre><p><strong>Caption:<br></strong><em>The phase is not considered done because code exists. It is done when requirements are verified, and UAT is recorded.</em></p><p>Verification is a mandatory phase that moves away from the informal “click-and-fix” approach. GSD walks us through structured UAT, recording evidence in VERIFICATION.md. A feature isn’t finished because the code exists; it’s finished when the requirements are proven to be met.</p><h3>Shipping the Phase</h3><p>After verification, we ship:</p><pre>/gsd-ship 1 - draft</pre><p>The shipping step creates a structured pull request body based on the artifacts produced during the phase: plans, summaries, verification results, and UAT notes.</p><p>The workflow reference describes /gsd-ship as the step that pushes the branch, creates a PR with an auto-generated body, optionally triggers review, and closes the plan → execute → verify → ship loop.</p><h3>What GSD Gives Us in Practice</h3><p>Execution happens through specialized subagents with fresh context windows. This parallel approach keeps agents focused on small, manageable scopes while maintaining alignment with the broader requirements.</p><p>First, it gives us <strong>persistent project memory</strong>. The important decisions live in .planning/, not in a temporary AI chat. STATE.md tracks current position, blockers, decisions, and recovery state, while continue-here.md supports session handoff and resumption.</p><p>Second, it gives us <strong>requirement traceability</strong>. Requirements are assigned stable IDs, mapped to roadmap phases, referenced by plans, and checked during verification. The docs describe this as a chain from requirement to phase to plan to implementation to verification.</p><p>Third, it gives us <strong>fresh context</strong> for execution. Heavy research, planning, execution, and verification work are handled by specialized subagents rather than in a single overloaded conversation. This directly addresses context rot, where long-running AI sessions degrade as the context window fills up.</p><p>Fourth, it gives us <strong>quality gates</strong>. Plans are checked before implementation. Execution writes summaries. Verification produces evidence. UAT records human acceptance testing. Code review, security review, UI review, test generation, debugging, and cross-AI review can also be run as separate quality commands.</p><p>Fifth, it gives us <strong>a real shipping loop</strong>. We do not stop at “the agent-generated code”. We stop at a pull request with a clear body, evidence, verification notes, and a path to review.</p><h3>The Trade-Off: More Structure Means More Tokens</h3><p>Spec-driven AI engineering gives us much more control than casual coding, but it also comes with a cost.</p><p>The more structured the workflow becomes, the more AI context we need to process. Architecture Vision, requirements, roadmap, phase discussions, UI specifications, implementation plans, agent execution, code review, verification, UAT notes, and PR summaries all consume tokens. Every additional agent, every review cycle, every cross-check, and every regenerated plan increases the total token usage.</p><p>This is the other side of the medal. With GSD, we are not sending a single short prompt and getting a single answer. We are running a multi-step engineering workflow. The model reads existing artifacts, reasons about requirements, generates plans, checks implementation, writes summaries, and verifies results against the original intent. This produces better alignment, but it also means higher token consumption.</p><p>In practice, it is very easy to hit limits. Long context windows, multiple agents, repeated planning loops, cross-AI reviews, and large codebases can quickly exhaust subscription quotas or tool-specific limits. At some point, the bottleneck is not only engineering complexity but also the amount of useful reasoning we can afford to run through AI tools.</p><p>That forced us to be pragmatic. We learned that SDD does not mean describing everything in infinite detail. The goal is not to create a perfect specification for every possible future scenario. The goal is to provide enough clarity for agents to work safely and productively.</p><p>So we tried to keep the process balanced. The Architecture Vision should define the important boundaries, but it should not become a hundred-page document before the first phase starts. GSD phases should be small enough to execute and verify, but not so small that the process becomes pure overhead. Planning should clarify real uncertainty, but it should not turn into an endless loop where models keep improving the same document without shipping anything.</p><p>We also tried to avoid unnecessary model recursion. For example, we do not need a cross-AI review for every tiny UI adjustment. We do not need deep research for well-understood CRUD functionality. We do not need to regenerate the entire roadmap when only one implementation detail changes. The process should be strong where the risk is high and lightweight where the task is simple.</p><p>A practical rule emerged: use AI reasoning when it reduces ambiguity, risk, or manual effort — not just because it is available. We also paid attention to the economic model of AI tooling. For heavy workflows, fixed subscriptions can become limiting because they often come with usage caps, rate limits, or hidden constraints. Whenever possible, we preferred API-based usage with pay-per-token pricing. This gave us more predictable scaling for agentic workflows: instead of being blocked by subscription limits, we could treat AI usage more like infrastructure cost.</p><p>That does not mean API usage is always cheaper. It means it is often easier to control, measure, and optimize. Over time, token usage became something we had to manage deliberately, just as we do cloud resources. We started thinking about which artifacts should be included in context, which files agents really need to read, which review loops are worth running, and where human judgment is cheaper and faster than another AI pass.</p><h3>The Bigger Lesson</h3><p>Our transition from AppSheet to custom development was not a rejection of no-code. AppSheet helped us validate ideas, understand real business processes, and discover requirements that would have been difficult to define upfront. It was the right tool for exploration, but not for the level of control, performance, and flexibility the system eventually required.</p><p>Custom development removed those platform limitations, while AI helped us increase delivery speed. However, faster code generation did not solve the main engineering problem: turning incomplete business ideas into reliable software. Without enough context, AI agents still make assumptions, overlook edge cases, and produce implementations that may be technically correct but functionally wrong.</p><p>SDD with GSD gave us a practical way to control the process. We start with high-level requirements, turn them into an Architecture Vision and UI references, and then use GSD to refine the work into requirements, phases, plans, implementation tasks, verification, and UAT. The result is a traceable workflow from the original business need to a reviewed and verified implementation.</p><p>This approach is not free. It requires more preparation, more context, more tokens, and more discipline than casual prompting. It also needs to be applied selectively. Not every small change requires deep planning, multiple review agents, or a full specification cycle. The process should be detailed where the business risk is high and lightweight where the task is straightforward.</p><p>The main outcome for us was a more mature way of building internal products. No-code supported discovery, custom development provided control, AI improved speed, and GSD added the structure needed to use that speed responsibly. The value is not in replacing engineers with agents, but in giving engineers a repeatable way to direct, constrain, and verify AI-assisted development.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab0d159b2196" width="1" height="1" alt=""><hr><p><a href="https://medium.com/emerline-tech-talk/from-no-code-to-ai-engineering-with-sdd-ab0d159b2196">From No-Code to AI Engineering with SDD</a> was originally published in <a href="https://medium.com/emerline-tech-talk">Emerline Tech Talk</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Extended Our Automation Stack with Google Cloud]]></title>
            <link>https://medium.com/emerline-tech-talk/how-we-extended-our-automation-stack-with-google-cloud-26ad1f945bc9?source=rss-be0afd09aaa1------2</link>
            <guid isPermaLink="false">https://medium.com/p/26ad1f945bc9</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[chatbots]]></category>
            <dc:creator><![CDATA[Viachaslau Kitsun]]></dc:creator>
            <pubDate>Thu, 02 Apr 2026 07:18:27 GMT</pubDate>
            <atom:updated>2026-04-03T11:26:55.311Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*unE-Y3dWosvGBe0M1Hs2kg.png" /></figure><h3>When AppSheet Was No Longer Enough: How We Extended Our Automation Stack with Google Cloud</h3><p>There is a moment in almost every internal automation project when the first success creates a new problem.</p><p>At first, the goal is simple: remove manual work, make a process less fragile, and give people a tool they will actually use. That’s why we started with AppSheet. It gave us a fast way to turn routines into internal apps. We did not need a full custom development effort. For the first phase, it was the right decision. We moved quickly, automated real processes, and proved that even lightweight tools can create immediate value.</p><p>But as a few workflows began to function reliably, the project entered a new phase.</p><p>When people stop seeing automation as a one-off improvement, they start treating it as infrastructure. They want richer integrations, more control, better messaging, cleaner backend logic, and fewer things tied to individual user accounts. At that point, our AppSheet-based setup began to expand beyond AppSheet alone.</p><p>We did not want to replace it. We wanted to extend it.</p><p>This transition required:</p><ul><li>Broader use of Google Cloud</li><li>Tighter integration with Apps Script</li><li>Service-account-based Google API access</li><li>A Google Chat bot user interface</li><li>Eventually, a lightweight AI-assisted layer over practical workflows.</li></ul><p>In summary, this article covers the key lessons from our transition: how we evolved our automation from AppSheet to Google Cloud, why the shift was necessary, and the architectural improvements achieved.</p><h3><strong>AppSheet Was Still the Foundation</strong></h3><p>One important detail is worth stating upfront: AppSheet did not become the problem.</p><p>It remained the solution’s business layer and did its job well. It housed the app, surfaced process logic to users, and kept business data structured and accessible. In other words, we were not “moving away” from AppSheet. We were building around it.</p><p>That distinction mattered because it influenced the architecture. Instead of rebuilding everything as a custom application, we focused on strengthening the technical layer around it. We needed better API access, more robust authentication, more reusable backend logic, and a way to deliver actions to users in tools they already use every day.</p><p>Given these needs, the logical next step was to move toward Google Cloud.</p><h3><strong>The First Real Step Was Creating a Dedicated Google Cloud Project</strong></h3><p>Once the requirements started expanding, the first concrete move was to create a dedicated Google Cloud project for this automation stream.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SvcZf5pPZFgEf6wXw3npog.png" /><figcaption>Google Cloud project creation</figcaption></figure><p>This gave us a proper boundary for APIs, IAM configuration, service accounts, and future deployment workflows. It also structured our Apps Script projects. They no longer felt like isolated scripts solving local tasks, but rather components of a broader application stack.</p><p>The next step was to bind the Google Cloud project to our Apps Script projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BNMjuEUHsolmF3P7ChQW7w.png" /><figcaption>Connect GAS to the created GCP project</figcaption></figure><p>On the surface, this appears to be a minor technical action. In reality, it changes a lot.</p><p>It offers centralized API management, cleaner Google Cloud integration, and a robust script management framework. For us, this marked the shift from scattered automations to a reusable platform pattern.</p><h3><strong>One Unexpected Bonus: Cloud Logging Became Part of the Toolkit</strong></h3><p>There was also a very practical bonus we got from attaching a Google Cloud project to Apps Script: script execution logs became much more useful.</p><p>Once the project was connected, we gained access to logging through Google Cloud’s infrastructure. This improved our ability to observe script behavior by enabling centralized storage, searching, and filtering of execution logs in Cloud Logging, making debugging and monitoring more efficient than relying solely on local script-level logs.</p><p>That gave us a better view of what was happening in production: which functions were triggered, where errors occurred, what the execution flow looked like, and how often certain issues repeated.</p><p>More importantly, these logs were not just readable — they were analyzable.</p><p>We could filter them, inspect patterns, parse structured data, isolate problems, and use logs for deeper troubleshooting. For business-critical automation, this is a big upgrade. Logging stops being a developer convenience and becomes an operational asset.</p><p>That was not the original reason we connected Apps Script to Google Cloud, but it quickly became one of the most useful side effects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5lyPTbHg6TIfpgFeQwviqQ.png" /><figcaption>Google Cloud Logs Explorer</figcaption></figure><h3>Identity Was the Next Thing That Needed to Grow Up</h3><p>As soon as more integrations entered the picture, identity management became impossible to ignore.</p><p>Many internal automation projects start with human-bound access–a script runs under the creator’s account or someone stores a token somewhere. Everything seems fine until the workflow becomes important. At that point, the cracks show quickly. Ownership is blurry, permissions are hard to audit, and operations become fragile.</p><p>We wanted a cleaner model.</p><p>We integrated a service account via Google IAM and used it as a non-human identity for backend operations. This provided two immediate technical benefits: enhanced access control and easier automation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oav6ppmT6FEAvDCYCYHWiA.png" /><figcaption>Service account creation for the project</figcaption></figure><p>First, it made API interactions more stable and predictable. Instead of relying on a personal account, we could explicitly authorize requests. This made information more manageable, accessible and easier to maintain.</p><p>Second, it provided us with a stronger foundation for CI/CD workflows and future reuse. Once automation is part of a broader system, you want the backend identity to belong to the system, not a person.</p><p>To support those interactions, we also used OAuth 2.0 when needed. That allowed Apps Script to authenticate and call Google APIs in a more structured way. This was not always visible to end users, but it made a huge difference in maintainability and operational confidence.</p><pre>function getServiceAccountToken() {<br>  var header = JSON.stringify({ &quot;alg&quot;: &quot;RS256&quot;, &quot;typ&quot;: &quot;JWT&quot; });<br>  var now = Math.floor(Date.now() / 1000);<br>  var payload = JSON.stringify({<br>  &quot;iss&quot;: AUTH_CLIENT_EMAIL,<br>  &quot;sub&quot;: AUTH_CLIENT_EMAIL,<br>  &quot;scope&quot;: &quot;https://www.googleapis.com/auth/chat.memberships.readonly https://www.googleapis.com/auth/chat.messages.create https://www.googleapis.com/auth/userinfo.email&quot;,<br>  &quot;aud&quot;: &quot;https://oauth2.googleapis.com/token&quot;,<br>  &quot;exp&quot;: now + 3600,<br>  &quot;iat&quot;: now<br>  });<br><br>  var signature = Utilities.computeRsaSha256Signature(<br>  Utilities.base64EncodeWebSafe(header) + &quot;.&quot; + Utilities.base64EncodeWebSafe(payload),<br>  AUTH_PRIVATE_KEY<br>  );<br><br>  var jwt = Utilities.base64EncodeWebSafe(header) + &quot;.&quot; +<br>  Utilities.base64EncodeWebSafe(payload) + &quot;.&quot; +<br>  Utilities.base64EncodeWebSafe(signature);<br><br>  var response = UrlFetchApp.fetch(&quot;https://oauth2.googleapis.com/token&quot;, {<br>  method: &quot;post&quot;,<br>  payload: {<br>  &quot;grant_type&quot;: &quot;urn:ietf:params:oauth:grant-type:jwt-bearer&quot;,<br>  &quot;assertion&quot;: jwt<br>  }<br>  });<br><br>  console.log(&quot;Getting JWT token..&quot;);<br>  <br>return JSON.parse(response.getContentText()).access_token;<br>}</pre><h3>Google Chat Became the Natural Interface for Action</h3><p>Once we had the Cloud project, the IAM setup, and API access in place, the next logical step was to think about where the workflow should meet the user.</p><p>The answer was obvious.</p><p>It should NOT be in yet another tab people forget to open.</p><p>If internal processes need timely action, the best place to surface them is where teams already communicate. That is why we enabled the Google Chat API and built a Google Chat bot with Apps Script as the backend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NUM16kkHJaCpTKjjixkPoQ.png" /><figcaption>Google Chat Bot creation</figcaption></figure><p>From an architectural standpoint, this was an efficient and productive combination. Apps Script provided an event-driven backend that could respond when the bot was added to or removed from a space, mentioned in a message, or invoked via a command. Google Chat gave us the interaction surface.</p><p>Together, they created an integration pattern: AppSheet for data and workflows, Apps Script for orchestration, Google Cloud for identity and API management, and Google Chat for user-facing actions. This approach enabled seamless, lightweight yet powerful collaboration across platforms.</p><pre>function onAddedToSpace(e) {<br>  console.log(JSON.stringify(e));<br>  console.log(&quot;Bot added&quot;);<br>}<br><br>function onMessage(e) {<br>  //some logic for handling message event<br>  return;<br>}<br><br>function onRemovedFromSpace(e) {<br>  console.log(JSON.stringify(e));<br>  console.log(&quot;Bot deleted&quot;);<br>}</pre><h3>The First Use Case Was Simple, but Very Real</h3><p>The first scenario we implemented was not flashy. It was meant to solve a real operational problem.</p><p>We needed to identify employees who had not completed a required form in AppSheet and send them a reminder that would actually prompt them to take action.</p><p>This task sounds simple–until you see how it usually works. Someone checks the data manually, compiles a list, sends a reminder, follows up later, and repeats the same effort because there is no good action loop.</p><p>We wanted to automate that loop end-to-end.</p><p>Inside the bot, we made an API call to AppSheet, got a list of employees with pending submissions, and generated a Google Chat card for each. The card wasn’t just a notification; it was a prompt for action. It explained what was missing and included a button that took users directly to AppSheet to quickly complete the required form.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wfWt5dwot8MtrvosidQrKQ.jpeg" /><figcaption>Google Chat bot card</figcaption></figure><p>That detail mattered more than it may seem.</p><p>The value of automation is often not in detecting the issue, but in reducing the number of steps between the issue and the fix. If a reminder arrives in the place where the user already works, and the next action is one click away, completion rates tend to improve for very practical reasons: less context switching, less delay, fewer excuses to return to it later.</p><h3>We Added Gemini Thoughtfully, but not Everywhere</h3><p>Once the reminder flow was working, we added one more layer: Gemini.</p><p>But we were careful about where we used it.</p><p>We did not want a language model to decide who should receive a reminder, whether someone was compliant with the process, or what the business rule meant. Those parts remained deterministic and fully under the system’s control. The list of employees still came from AppSheet data. The trigger conditions were still explicit. The logic stayed predictable.</p><p>What Gemini added was a small human layer on top of that logic.</p><p>We used it to generate a short motivational suggestion to embed in the reminder. The goal was simple: make the message feel less robotic and slightly more engaging, without turning the workflow into something unpredictable.</p><p>This turned out to be a good fit for AI in internal tooling.</p><p>When AI is used for wording, tone, and presentation — and not for the underlying operational truth — it becomes easier to add value without weakening the reliability of the process. That was exactly the balance we wanted. The workflow remained stable. The message became more flexible.</p><p>And just as importantly, the feature was treated as an enhancement, not a dependency. If AI generation failed or returned something unhelpful, the system still sent a normal reminder. That fallback is essential in production automation. Internal tools have to keep working even when the “smart” part is unavailable.</p><pre>function getGeminiAdvice(managerName, projects) {<br>  const model = &quot;gemini-2.5-flash&quot;;<br>  const url = &quot;https://generativelanguage.googleapis.com/v1beta/models/&quot; + model + &quot;:generateContent?key=&quot; + GEMINI_API_KEY;<br>  const prompt = //your prompt here<br>  const payload = {<br>  &quot;contents&quot;: [{<br>  &quot;parts&quot;: [{<br>  &quot;text&quot;: prompt}]<br>   }]<br>  };<br>  const options = {<br>  &quot;method&quot;: &quot;post&quot;,<br>  &quot;contentType&quot;: &quot;application/json&quot;,<br>  &quot;headers&quot;: {<br>  &quot;X-goog-api-key&quot;: GEMINI_API_KEY<br>  },<br>  &quot;payload&quot;: JSON.stringify(payload),<br>  &quot;muteHttpExceptions&quot;: true<br>  };<br><br>  try {<br>    const response = UrlFetchApp.fetch(url, options);<br>    const responseText = response.getContentText();<br>    const json = JSON.parse(responseText);<br>    <br>    if (json.candidates &amp;&amp; json.candidates.length &gt; 0 <br>        &amp;&amp; json.candidates[0].content) {<br>        return json.candidates[0].content.parts[0].text.trim();<br>    } else {<br>      console.error(&quot;Gemini Error: &quot; + responseText);<br>      return &quot;Error!&quot;;<br>    }<br>  } catch (e) {<br>    console.error(&quot;Fetch Error: &quot; + e);<br>    return &quot;Error!&quot;;<br>  }<br>}</pre><h3>What This Changed for Us</h3><p>In the end, the most valuable outcome was not just the bot itself.</p><p>What we really gained was a reusable pattern for building the next generation of internal automations. We now had a way to keep AppSheet as the business-facing application layer, while extending it with:</p><ul><li>Google Cloud-managed APIs</li><li>IAM-backed service identities</li><li>Google Chat interactions</li><li>Improved observability through Cloud Logging</li><li>Selective AI support where it actually made sense.</li></ul><p>That combination gave us more than extra features. It gave us better control.</p><p>We had cleaner authentication and a better place to manage APIs. We had stronger logging and a more realistic way to analyze production behavior. We had a user interface for action that lived inside the company’s daily communication flow. And we had a low-risk way to add AI without compromising business logic.</p><p>Most importantly, we did not have to throw away the work that had already been done in AppSheet to get there.</p><p>That is probably the central lesson from this project.</p><p>You do not always need to replace a no-code or low-code foundation when your requirements grow. Sometimes the smarter move is to keep the layer that already works for business users and strengthen everything around it: identity, integrations, observability, messaging, and automation orchestration.</p><p>That was exactly our path.</p><p>AppSheet helped us get automation off the ground. Google Cloud helped us turn it into something more robust, more extensible, and much closer to production-grade engineering.</p><p>Are you thinking about transforming your business? Get your free consultation at <a href="https://emerline.com/services"><strong>Emerline</strong></a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26ad1f945bc9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/emerline-tech-talk/how-we-extended-our-automation-stack-with-google-cloud-26ad1f945bc9">How We Extended Our Automation Stack with Google Cloud</a> was originally published in <a href="https://medium.com/emerline-tech-talk">Emerline Tech Talk</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Internal Process Automation with Google AppSheet: What We Learned in 3 Months]]></title>
            <link>https://medium.com/emerline-tech-talk/building-internal-process-automation-with-google-appsheet-what-we-learned-in-3-months-80f0a04f2243?source=rss-be0afd09aaa1------2</link>
            <guid isPermaLink="false">https://medium.com/p/80f0a04f2243</guid>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[small-business]]></category>
            <category><![CDATA[hiring]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[Viachaslau Kitsun]]></dc:creator>
            <pubDate>Thu, 05 Feb 2026 09:06:39 GMT</pubDate>
            <atom:updated>2026-02-05T09:06:39.421Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QEriW1qVPb9IGdXBxMif3Q.png" /></figure><p>Over the last 3 months, we tackled a deceptively simple but costly challenge: automating routine operations. In many companies, management overhead (e.g. reporting, updating metrics, approvals, and access requests) adds up to an ongoing drag on productivity.</p><p>In our case, the pain point was clear: every month, reporting and metrics were assembled manually. Data lived in different places, owners changed, definitions drifted, and the process relied on people remembering to do the right thing at the right time. Even when everything went “well,” it consumed hours of valuable time across multiple roles.</p><p>At the same time, we already had a strong foundation we weren’t fully using: our Google Workspace subscription. That meant we had access to AppSheet as a low-code platform, with built-in data storage and integrations that work well with the Google ecosystem. This combination became the catalyst for designing and implementing an internal automation system on top of Google AppSheet.</p><p>This article provides concise key takeaways from our AppSheet automation journey: identifying practical benefits, outlining the main challenges, and describing the scenarios you are likely to encounter when pursuing similar automation efforts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RmgnNtKccow2y27Cqg8xyg.png" /></figure><h3>The Core Problem: Manual Reporting Is a System Smell</h3><p>Manual reporting typically indicates deeper issues than “we need a nicer spreadsheet.” It’s often a sign that the organization lacks a stable data backbone and that coordination work is being done informally.</p><p>Before we touched any tooling, we wrote down the most visible symptoms we kept running into month after month:</p><ul><li><strong>Data fragmentation: </strong>Metrics stored across spreadsheets, chats, docs, task trackers and personal notes.</li><li><strong>No single source of truth: </strong>Even when a “main spreadsheet” exists, it’s not authoritative as people copy and paste, then version it.</li><li><strong>Inconsistent definitions: </strong>What counts as “done,” “blocked,” “active” or even “headcount” changes between teams.</li><li><strong>High coordination cost: </strong>Reminders, approvals and folow-ups happen ad hoc in chat.</li><li><strong>Low auditability:</strong> You can’t reliably answer “Who changed what and when?” without detective work.</li><li><strong>Dependency on individuals: </strong>The process works because a few people carry it in their heads.</li></ul><p>Most importantly, manual reporting doesn’t scale. As teams multiply, the reporting burden grows disproportionately, highlighting the need for effective automation and leading us to consider the right tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8tUQMsbZBfjGUmdGSkhuVQ.png" /></figure><h3>Why AppSheet (Especially If You Already Have It)?</h3><p>AppSheet sits in a useful middle ground: it’s regulated enough to enforce structure, but flexible enough to iterate without a full engineering cycle.</p><p>Here’s the practical “why” for choosing AppSheet:</p><ul><li>Faster than building custom internal tools from scratch</li><li>More structured than “just spreadsheets”</li><li>Easier to govern than a collection of scripts maintained by a few people</li><li>Strong enough for many operational workflows (forms, approvals, state machines, notifications, dashboards, and role-based access)</li></ul><p>If your organization is already using Google Workspace, AppSheet can become a “force multiplier”, because it fits naturally into how people already work:</p><ul><li>Many users already authenticate with Google.</li><li>Collaboration and access control patterns are familiar.</li><li>Data often already lives in Sheets/Drive.</li><li>Integration paths (Gmail, Calendar, Drive, Sheets, and beyond) are straightforward.</li></ul><p>While AppSheet may not solve every challenge, it’s often a high-ROI starting point for internal operations. This conclusion led to the design of our own system, which we describe next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2xc2kunoCRmqoJVtgxt6ig.png" /><figcaption>Internal ops app: one entry point for recurring routines.</figcaption></figure><p><strong>What We Designed: An Internal Automation System</strong></p><p>When you say “system,” it’s easy to imagine a single monolith. In practice, we approached it as a <strong>“platform + apps”</strong> model, so we could grow in modules, without creating one inflexible mega-app.</p><p><strong>1) A data layer (the backbone)</strong></p><p>We started by translating “reporting” into a small set of durable entities (tables) that remain stable even when teams and owners change:</p><ul><li>Teams / projects</li><li>Reporting periods (weeks/months)</li><li>KPIs / metric definitions</li><li>Metric submissions</li><li>Status updates</li><li>Approvals / exceptions</li><li>Ownership and responsibility matrices</li></ul><p>The key decision: <strong>treat metrics and reporting as data, not documents</strong>.</p><p><strong>2) A workflow layer (how work moves)</strong></p><p>With the data model in place, we encoded the correct process directly into the workflow by specifying inputs, validations, approvals, locks, and reminders:</p><ul><li>Inputs (who provides what, by when)</li><li>Validation (required fields, allowed ranges, references)</li><li>Approvals (manager review, escalation)</li><li>Locking periods (freeze after cutoff)</li><li>Reminders and nudges (before and after deadlines)</li></ul><p><strong>3) An app layer (user-facing experiences)</strong></p><p>Different roles need different screens, and adoption depends heavily on making each role’s path short and obvious:</p><ul><li><strong>Contributors:</strong> quick submission screens, minimal fields, mobile-friendly</li><li><strong>Managers:</strong> approvals, completeness overview, exceptions</li><li><strong>Operations:</strong> admin screens, definitions, access control</li><li><strong>Leadership:</strong> dashboards and trend views</li></ul><p><strong>4) Reporting outputs (where the story is told)</strong></p><p>AppSheet offers built-in views and charts, but in many orgs, the reporting layer eventually benefits from a BI tool. A common pattern is:</p><ul><li>AppSheet handles data capture + workflow</li><li>A reporting tool handles visualization and deeper analytics</li></ul><h3>AppSheet Features That Make This Practical</h3><p>AppSheet is more than a UI builder; its workflow engine and integrations enable your app to actively participate in your process.</p><p><strong>Event-driven automations</strong> are particularly useful for operational flows. You can trigger actions when records are created or updated, when a field changes state, or when an input reaches a specific condition (e.g., a metric switches from <em>Draft</em> to <em>Submitted</em>). In practice, that means you can automatically notify a manager for review, create follow-up tasks, write a structured log entry, or branch the workflow based on validation results — without relying on humans to remember the next step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DSiPwBX1izK6Eb2yoah5VA.png" /></figure><p>In addition to event triggers, AppSheet supports <strong>time-based automations</strong>, which are essential for recurring management routines. Typical examples include:</p><ul><li>Sending reminders two days before a reporting deadline escalating overdue inputs after a cutoff</li><li>Generating a “monthly snapshot” at a fixed time</li><li>Periodically checking for anomalies (e.g., missing owners, gaps in inputs)</li></ul><p>Your process keeps moving, even when your attention is focused elsewhere. This makes your process even more resilient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K-NkMzjNyP3iSG_8YbsDVQ.png" /></figure><p>Another powerful capability is <strong>embedding your workflow into Google Chat</strong>. AppSheet apps can be integrated as a chatbot experience, which inspires positive changes in user behavior. Instead of “go to the app,” users interact within the tool where coordination already happens. You can implement flows like “submit metric,” “approve report,” “show missing items,” or “give me this month’s dashboard link” directly in Chat. For internal adoption, this is a big deal because it reduces context switching, which often matters more than adding features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2vx_V5FAHEG68D8px64a2Q.png" /></figure><p>Finally, AppSheet is not limited to the Google ecosystem. For integration with external systems, you can use <strong>webhooks</strong> to push events outward (e.g., “metric approved” → notify another system) or pull data in with custom endpoints depending on your architecture. And if you standardize on AppSheet’s database layer, you can also integrate other services by connecting directly to the <strong>AppSheet database</strong> as a data source, turning it into a lightweight internal backend for operational data. This enables patterns, for example, AppSheet manages data capture and workflow, while other services (BI, scripts, internal portals) consume the same tables for analysis, exports, or cross-system synchronization.</p><h3>Benefits We Saw (Beyond “Saving Time”)</h3><p>Time savings matter, but the biggest gains were structural, including clearer processes, better auditability, and more resilience.</p><p><strong>1) Single source of truth</strong></p><p>Once definitions and inputs move into structured tables, you reduce the three most common failure modes of spreadsheet-driven processes:</p><ul><li>Copy/paste errors</li><li>Version conflicts</li><li>“Which sheet is the latest?” debates</li></ul><p><strong>2) Standardization with flexibility</strong></p><p>With a central model, you can enforce rules and a consistent cadence, while still allowing teams to define metrics that match their reality.</p><p><strong>3) Auditability and accountability</strong></p><p>Tracking inputs as records means you can answer common “management questions” without investigative work:</p><ul><li>Who submitted late and why?</li><li>What changed between versions?</li><li>Where are the bottlenecks in approvals?</li></ul><p><strong>4) Faster iteration on processes</strong></p><p>A manual process is challenging to improve because every change requires retraining people and updating templates. With an app, you can gradually evolve the workflow.</p><p><strong>5) Reduced coordination overhead</strong></p><p>Automated reminders and status views reduce a lot of “pinging” in chat. These are not just convenient; they change the emotional texture of the process.</p><h3>Concrete Scenarios Where AppSheet Shines</h3><p>AppSheet shines most when the workflow is repeatable, has clear ownership, and benefits from structure — but does not yet require a full custom-build.</p><p>The following workflow archetypes fit AppSheet particularly well:</p><p><strong>Scenario A: Monthly metric collection (the obvious one)</strong></p><ul><li>Owners get an assigned list of metrics for the period</li><li>Inputs are validated at entry (format, thresholds, dependencies)</li><li>Missing items trigger automated reminders</li><li>Managers approve or request corrections</li><li>Period closes, data is locked</li></ul><p><strong>Scenario B: Team health checks / pulse surveys</strong></p><ul><li>Lightweight recurring forms</li><li>Automated aggregation</li><li>Anonymous or semi-anonymous modes (depending on policy)</li><li>Trend views by team or department</li></ul><p><strong>Scenario C: Intake and triage workflows</strong></p><p>For internal requests such as:</p><ul><li>Access requests</li><li>Procurement requests</li><li>Staffing needs</li><li>Hiring pipeline signals</li><li>Incident/risk reporting</li></ul><p>AppSheet is excellent at:</p><ul><li>Structured intake</li><li>Routing to owners</li><li>Tracking state transitions</li><li>Building transparency around queue status</li></ul><p><strong>Scenario D: Operational checklists (onboarding/offboarding)</strong></p><ul><li>Task templates by role</li><li>Assigned owners and deadlines</li><li>Verification steps</li><li>Completion evidence (links/files)</li></ul><p><strong>Scenario E: Lightweight asset/config tracking</strong></p><ul><li>Inventory-like tables</li><li>Ownership</li><li>Lifecycle states</li><li>Renewal reminders</li></ul><h3>The Hard Parts (And How to Prepare for Them)</h3><p>Low-code reduces engineering effort, but it doesn’t remove complexity — it shifts it. The key is to expect the complexity early and decide where you’ll be strict and where you’ll be flexible.</p><p><strong>1) Data modeling matters more than you think</strong></p><p>Spreadsheets encourage informal structure. Apps punish it.</p><p>To avoid scaling into chaos, you’ll need to think in terms of:</p><ul><li>Keys, references, and relationships</li><li>Normalization vs convenience</li><li>What changes over time (history tables) vs. what is the current state</li></ul><p>A good data model is the difference between an app that scales and one that collapses into workarounds.</p><p><strong>2) Governance: who owns definitions?</strong></p><p>Metrics without ownership are noise. Automating the collection process forces an important question set that many orgs postpone:</p><ul><li>Who defines the metric?</li><li>Who owns it?</li><li>Who approves changes?</li><li>What is the canonical formula?</li></ul><p>You’ll want a “metric registry” concept: definitions, owners, cadence, and allowed values.</p><p><strong>3) Access control and privacy</strong></p><p>Internal automation quickly touches sensitive areas: performance signals, staffing, risks. If you design privacy late, you’ll either over-restrict (hurting adoption) or leak context (hurting trust).</p><p>Plan early for:</p><ul><li>Role-based access</li><li>Per-team visibility rules</li><li>Separation between “admin” and “viewer”</li><li>Audit logs and change tracking</li></ul><p><strong>4) Change management is half the project</strong></p><p>Even if the app is great, adoption fails if:</p><ul><li>Users don’t understand why it exists</li><li>It adds friction compared to old habits</li><li>It feels like “extra reporting” rather than removing the burden</li></ul><p>A practical approach:</p><ul><li>Start with one team and one recurring pain point</li><li>Prove time saved</li><li>Expand gradually with templates</li></ul><p><strong>5) Platform boundaries</strong></p><p>At some point, you’ll hit limits. The important part is to treat those limits as architectural signals, not as AppSheet failures.</p><p>Typical boundaries include:</p><ul><li>Complex analytics and modeling</li><li>High-scale transactional workloads</li><li>Advanced custom UI needs</li></ul><p>That’s not failure, it’s a boundary. A healthy strategy is to treat AppSheet as:</p><ul><li>An operational workflow layer</li><li>Feeding into more specialized systems when needed</li></ul><h3>Practical Design Principles We Followed</h3><p>The difference between a “nice app” and a “reliable internal system” usually comes down to discipline in a few boring design choices.</p><p>If you’re starting a similar initiative, the following principles pay off quickly:</p><ol><li><strong>Start with one workflow and make it boringly reliable: </strong>Don’t try to automate “all operations”; pick the process that recurs with the highest frequency.</li><li><strong>Define “done” as a state machine: “</strong>Draft” → “Submitted” → “Reviewed” → “Approved” → “Locked”.</li><li><strong>Make the default path fast: </strong>Most users should complete their task in under a minute.</li><li><strong>Bake in validation: </strong>Prevent bad data at entry rather than correcting it later.</li><li><strong>Separate definitions from inputs:</strong> Keep a stable registry of metrics and a periodic table of submissions.</li><li><strong>Design for exceptions: </strong>Late inputs, missing owners, changed teams — build escape hatches with traceability.</li></ol><h3>What Success Looks Like</h3><p>To keep this grounded, it helps to define success in terms of outcomes (not features). For us, “success” became a short checklist we could validate after each cycle.</p><p>A good internal automation system should deliver:</p><ul><li>Predictable reporting cycles with less manual chasing</li><li>Stable, explainable metrics</li><li>Clear ownership and accountability</li><li>Consistent workflows across teams</li><li>Enough flexibility to evolve without chaos</li></ul><p>And perhaps most importantly, it should give people back the time they currently spend assembling status updates instead of doing meaningful work.</p><h3>Closing Thoughts</h3><p>Building internal automation on AppSheet is not just a tooling choice — it’s an organizational design choice. It forces clarity about ownership, definitions, access, and process boundaries. That can feel uncomfortable at first, but it’s exactly why AppSheet is valuable to organizations.</p><p>If your company already pays for Google Workspace, there’s a good chance you’re sitting on an underused platform capable of solving many of your internal operational challenges. The trick is to treat it like an engineering effort: model the data, design the workflow, govern the system, and iterate with real users.</p><p>If you do that, “monthly reporting” stops being a recurring fire drill and becomes a predictable, mostly automated routine.</p><p>Are you thinking about transforming your processes? Get your free consultation at <a href="https://emerline.com/?utm_source=Building+Internal+Process+Automation+&amp;utm_medium=Medium&amp;utm_content=techblog">Emerline</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80f0a04f2243" width="1" height="1" alt=""><hr><p><a href="https://medium.com/emerline-tech-talk/building-internal-process-automation-with-google-appsheet-what-we-learned-in-3-months-80f0a04f2243">Building Internal Process Automation with Google AppSheet: What We Learned in 3 Months</a> was originally published in <a href="https://medium.com/emerline-tech-talk">Emerline Tech Talk</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Solving Complex Data Challenges with GraphQL and Hot Chocolate]]></title>
            <link>https://medium.com/emerline-tech-talk/solving-complex-data-challenges-with-graphql-and-hot-chocolate-69eeccd2270f?source=rss-be0afd09aaa1------2</link>
            <guid isPermaLink="false">https://medium.com/p/69eeccd2270f</guid>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[dotnet]]></category>
            <category><![CDATA[hot-chocolate]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[c-sharp-programming]]></category>
            <dc:creator><![CDATA[Viachaslau Kitsun]]></dc:creator>
            <pubDate>Thu, 09 Oct 2025 14:02:32 GMT</pubDate>
            <atom:updated>2025-10-09T14:02:32.131Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VdEEJd5Fl1_OaPA6GnPjMg.png" /></figure><h3>Introduction</h3><p>Imagine being tasked with evolving a functional, in-house system into a powerful, enterprise-grade platform. The goal is ambitious: to enable massive scalability and deep customization for each client, but without the luxury of a ground-up rewrite. The existing system, a solid foundation for its initial purpose, now faces a new reality of expansive client expectations and the need for a highly flexible, multi-tenant architecture.</p><p>This is the story of how we transformed our supply chain management system to meet exactly those challenges. The core demand was apparent: the product catalog had to become infinitely adaptable. Every client needed the ability to define a unique, dynamic set of custom attributes for their products, which would then be used for critical operations such as searching, filtering, and sorting — all while delivering data efficiently with pagination.</p><p>The immediate question was, how do we build this on top of the already existing architecture? Their current REST API and relational database (Postgres) served well for fixed data models, but “dynamic attributes” threatened to become a scalability nightmare. The obvious, yet costly, path would be to alter the database schema constantly or create sprawling, sparse tables. While using a JSON column in Postgres offered a way to store dynamic data, it introduced a significant new hurdle: how would we efficiently query, filter, and sort against this JSON structure without writing immense amounts of complex, custom code for every potential field?</p><p>We needed a technology that could elegantly bridge the gap between a rigid backend and a fluid frontend requirement. Our solution emerged from a strategic blend of two powerful paradigms:</p><ol><li>A Modular Monolith with REST API: This forms our administrative backbone, allowing for the secure and structured management of custom product columns. It gives each tenant complete control to define their unique data schema without disrupting the data of others.</li><li>GraphQL with Hot Chocolate: For the product list and detailed views, where flexibility is paramount. We introduced a GraphQL layer. This was the key. It empowers the client to request exactly the data it needs, including any combination of those dynamic custom attributes, with built-in support for filtering, sorting, and pagination right out of the box.</li></ol><p>This article details our journey from a system facing growth constraints to one built for scale and customization, demonstrating that with the right architectural choices, you can dramatically evolve your platform without starting from scratch.</p><h3>GraphQL and Hot Chocolate</h3><p>Let’s pause for a moment and learn more about what GraphQL is and what opportunities it offers us. GraphQL is a powerful query language and runtime for APIs that allows clients to specify exactly what data they need, making data fetching more efficient and flexible than traditional REST APIs. Developed by Facebook and now widely adopted, GraphQL enables communication with your backend through a single endpoint, reducing over-fetching and under-fetching of data.</p><h4>Pros of GraphQL</h4><ul><li><strong>Efficient data fetching:</strong> Clients can request only the data they need, reducing network payload and improving performance, especially in mobile and complex web applications.</li><li><strong>Strongly-typed schema:</strong> GraphQL enforces a schema defined by the developer, providing clear contracts and type safety, which enhances error detection and development speed.</li><li><strong>Flexible evolution:</strong> APIs can evolve without breaking existing clients, since queries only return explicitly requested fields, simplifying versioning and maintenance.</li><li><strong>One endpoint for all data:</strong> Access to multiple data sources and complex relationships is unified under a single query endpoint, simplifying API integration and data management.</li><li><strong>Real-time updates:</strong> GraphQL supports subscriptions for real-time data, enabling features like live updates in applications.</li></ul><h4>Cons of GraphQL</h4><ul><li><strong>Increased complexity</strong>: Setting up and maintaining a GraphQL server and schema is more complex than a simple REST API, potentially increasing development time and costs.</li><li><strong>Steep learning curve</strong>: Developers familiar with REST may need additional training to master GraphQL’s concepts and query structure.</li><li><strong>Performance challenges:</strong> Complex queries may lead to inefficient data processing or excessive resource consumption if not carefully designed.</li><li><strong>Caching difficulties:</strong> Dynamic query responses complicate caching strategies compared to REST APIs with fixed endpoints.</li><li><strong>Security concerns:</strong> Flexible queries introduce risks such as denial-of-service (DoS) attacks if expensive queries are not properly managed.</li><li><strong>Possible over-fetching:</strong> Though designed to avoid this issue, clients may still inadvertently request more data than required, affecting application performance.</li></ul><p>In summary, GraphQL provides flexible and efficient data fetching for modern applications, but it requires careful management and expertise to address complexity and performance concerns.</p><p>Additionally, a few words about Hot Chocolate: Hot Chocolate is a highly efficient, open-source GraphQL server library for C# and .NET, offering various developer-friendly features and flexible schema design options for building modern APIs.</p><h4>Key features</h4><ul><li><strong>Multiple schema design approaches:</strong> Hot Chocolate supports annotation-based, code-first, and schema-first development, letting developers choose between C# attributes, explicit code, or SDL files to define their schema. Annotation-based and code-first options enable C# types to be mapped directly to GraphQL types, reducing boilerplate and eliminating the need to learn SDL syntax.</li><li><strong>Seamless ASP.NET Core integration:</strong> It works with ASP.NET Core’s dependency injection and middleware model, making setup and configuration intuitive for .NET developers.</li><li><strong>Advanced querying:</strong> The built-in middleware provides automatic filtering, sorting, and paging, streamlining complex data access without manual query logic.</li><li><strong>Real-time support and IDE tools:</strong> Native support for subscriptions enables real-time updates, and the integrated Nitro IDE provides visual schema exploration and testing tools for developers.</li></ul><h4>Advantages</h4><ul><li><strong>Developer productivity:</strong> Less boilerplate with annotation-based development; familiar C# types map seamlessly to GraphQL schema, speeding up initial setup and code maintenance.</li><li><strong>Self-documenting and strongly-typed:</strong> Auto-generated documentation and IDE autocomplete make APIs easier to understand and consume.</li><li><strong>High flexibility: </strong>Supports combining schema design approaches within the same project, allowing tailored solutions for different needs.</li><li><strong>Performance:</strong> Optimized for minimal overhead and efficient query handling, suitable for both small and enterprise-scale applications.</li><li><strong>Rich ecosystem:</strong> Integrates with Entity Framework and supports projections, making it powerful for data-heavy, relational apps.</li></ul><h4>Disadvantages</h4><ul><li><strong>Learning curve:</strong> Developers new to GraphQL or .NET dependency injection may need time to familiarize themselves with Hot Chocolate’s conventions and integration points.</li><li><strong>Abstraction over SDL:</strong> Relying on C# attributes may hide the details of the generated schema, making advanced fine-tuning less transparent.</li><li><strong>Parallel query issues:</strong> Some reports note challenges with Hot Chocolate’s handling of parallel queries, which could affect complex request scenarios.</li><li><strong>Complexity in large projects:</strong> Mixing schema approaches or heavy customization may complicate maintenance and onboarding for new team members.</li></ul><p>Hot Chocolate is an excellent choice for C# developers building modern, flexible GraphQL APIs, combining performance, ease of use, and advanced data access features.</p><h3>Implementation</h3><p>Based on these assumptions, we designed the following approach: each company in our system has a set of dynamic attributes for its products, which can be customized to support different data types, and these attributes are stored in a database with many-to-many relationships. To query that product list, we are implementing a Hot Chocolate query, which builds dynamic queries based on GraphQL requests and then uses Entity Framework Core, utilizing its PostgreSQL provider, to transform the query expression tree into SQL. As easy and straight forward as it looks, the devil is in the details.</p><p>Firstly, because we immediately inherited a relational database and did not have time to change it, the JSONB approach would also have required a significant amount of time. We opted for an approach of storing dynamic models in a relational database, so all the values of the dynamic attributes of the product are stored in a string field of the database at the intersection of many-to-many relationships.</p><p>This, in turn, led to the need to solve such several other problems, such as:</p><ul><li>We can’t perform all operations on the database side using IQueryable. For example, we store some product-related images in blob storage and need to enrich the response to include the link to the cover images for products, which will be rendered on the client side.</li><li>Sorting data according to its type.</li><li>Filters provided by Hot Chocolate are not as flexible as we need.</li><li>The inability to build SQL through EF Core out of the box.</li><li>The limitations of GraphQL in building reports and exporting data from the system.</li></ul><p>Let’s take it one step at a time.</p><h3>Resolving Performance Challenges</h3><p>The first challenge was to materialize the query in memory to enrich the response with additional data as it causes massive memory consumption and performance issues. To fix this problem, we implemented the following a workaround: filtering, sorting, and pagination from the GraphQL side applied to the query outside the execution pipeline, and after materializing, we are enriching only the already filtered, sorted, and paginated result collection:</p><pre>var filteredQuery = dbContext.Products<br>.AsNoTrackingWithIdentityResolution()<br>.Include(x =&gt; x.CustomColumnsWithValues)<br>.ThenInclude(x =&gt; x.CustomColumn)<br>.Where(x =&gt; x.TenantId == currentUserService.TenantId)<br>.Select(g =&gt; new ProductDto<br>{<br>Id = g.Id,<br>//other fields projection<br>}).Filter(resolverContext).Sort(resolverContext);<br><br>if(skipCustom.HasValue &amp;&amp; takeCustom.HasValue)<br>{<br>  filteredQuery = filteredQuery.Skip(skipCustom.Value).Take(takeCustom.Value);<br>}<br>var query = await filteredQuery.AsSplitQuery().ToListAsync();<br><br>foreach (var item in query)<br>{<br>  if (!item.CoverImage.IsNullOrEmpty() &amp;&amp; Guid.TryParse(item.CoverImage, out var parsedId))<br>  {<br>    var fileBlobName = allFilesMetadata.FirstOrDefault(x =&gt; x.Id == parsedId)?.FileBlobName;<br>    coverImagesTasks.Add(imagesFileService.GetLinkToFileWithSasAsync(fileBlobName, parsedId));<br>  }<br>  <br>  permissionsTasks.Add(permissionsService.AssignPermissionsForProductAsync(item, ResourceConstants.TenantProduct));<br><br>  item.CreatedBy = item.GetUserByAuditId(item.CreatedBy.Id, users);<br>  item.ModifiedBy = item.GetUserByAuditId(item.ModifiedBy?.Id, users);<br>}<br><br>if (coverImagesTasks.Count &gt; 0)<br>{<br>  var coverImages = await Task.WhenAll(coverImagesTasks);<br>  foreach (var item in query)<br>  {<br>    if (!item.CoverImage.IsNullOrEmpty() &amp;&amp; Guid.TryParse(item.CoverImage, out var parsedId))<br>    {<br>      var coverImage = coverImages.FirstOrDefault(x =&gt; x.Item1 == parsedId);<br>      if (coverImage != default)<br>      {<br>        item.CoverImage = coverImage.Item2.ToString();<br>      }<br>    }<br>  }<br>}</pre><p>Here, we are also utilizing a powerful EF Core feature: query splitting. It is a feature that breaks a single LINQ query with multiple includes into several more minor SQL queries, rather than one large query with multiple joins, thereby reducing redundant data and optimizing performance for complex object graphs.</p><h3>How Query Splitting works</h3><ul><li>When using .AsSplitQuery(), each included navigation property results in its own SQL query, allowing EF Core to stitch together the results in memory rather than retrieving everything with a single large SQL join.</li><li>This helps avoid issues like the Cartesian Explosion, where joins on large tables can duplicate data and dramatically increase result size and memory usage.</li></ul><h3>Sorting is not a problem</h3><p>The second challenge was:</p><p>We need to sort data by type. To dynamically display a set of attributes for each product from each company on the client side, we developed so-called “metadata” for each product column (attribute). This metadata includes the column name, its type, validation rules, length, format, and any other relevant details.</p><p>Based on this metadata, we added additional parameters to the product retrieval query, specifying the name of the column by which we sort and the sorting direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*XV1ODSlxjXI5-7NU_HhRcA.png" /><figcaption>GraphQL schema</figcaption></figure><p>Then, based on the type of this column, as determined by its metadata, we can build an extension for the IQueryable collection.</p><pre>if (!string.IsNullOrEmpty(sortByCustomColumnName))<br>{<br>  var metadata = await dbContext.CustomColumns.AsNoTrackingWithIdentityResolution()<br>                      .Where(x =&gt; x.TenantId == currentUserService.TenantId &amp;&amp; x.Name == sortByCustomColumnName)<br>                      .Select(x =&gt; new { x.DataType, x.Id })<br>                      .FirstOrDefaultAsync();<br>  if (metadata != null)<br>  {<br>    filteredQuery = filteredQuery<br>    .OrderByCustomColumn(sortByCustomColumnName, sortOrder, metadata.Id, metadata.DataType)<br>    .ThenBy(x =&gt; x.Id);<br>  }<br>}<br><br>private static IOrderedQueryable&lt;ProductDto&gt; OrderByCustomColumn(this IQueryable&lt;ProductDto&gt; query,<br>string columnName,<br>Guid columnId,<br>ColumnDataType dataType) =&gt;<br>dataType switch<br>{<br>  ColumnDataType.Numeric =&gt; query.OrderBy(p =&gt;<br>  decimal.Parse(p.CustomColumns<br>  .FirstOrDefault(cc =&gt; cc.Name == columnName &amp;&amp; cc.Id == columnId)!.Value)),<br>  ColumnDataType.DateTime =&gt; query.OrderBy(p =&gt;<br>  DateTimeOffset.Parse(p.CustomColumns<br>  .FirstOrDefault(cc =&gt; cc.Name == columnName &amp;&amp; cc.Id == columnId)!.Value)),<br>  ColumnDataType.Checkbox =&gt; query.OrderBy(p =&gt;<br>  bool.Parse(p.CustomColumns<br>  .FirstOrDefault(cc =&gt; cc.Name == columnName &amp;&amp; cc.Id == columnId)!.Value)),<br>  _ =&gt; query.OrderBy(p =&gt; p.CustomColumns<br>  .FirstOrDefault(cc =&gt; cc.Name == columnName &amp;&amp; cc.Id == columnId)!.Value)<br>};</pre><h3>Filtering</h3><p>Additionally, we experienced a problem with the limited filtering functionality provided by the default in Hot Chocolate. There are several variations of filtering that should be applied to our queries. For example, the ‘contains’ filter in the default implementation works as a case-sensitive search, but in our case, we need a case-insensitive one. To achieve this, we added our own override for the operator and built our own expression tree for transforming it to SQL and also for in-memory operations.</p><pre>public class QueryableStringInvariantContainsHandler(InputParser inputParser)<br>: QueryableStringOperationHandler(inputParser)<br>{<br>  protected override int Operation =&gt; DefaultFilterOperations.Contains;<br><br>  public override Expression HandleOperation(<br>  QueryableFilterContext context,<br>  IFilterOperationField field,<br>  IValueNode value,<br>  object parsedValue)<br>  {<br>  var property = context.GetInstance();<br>  if (parsedValue is string str)<br>  {<br>    var toLowerMethod = typeof(string).GetMethod(nameof(string.ToLower), Type.EmptyTypes);<br>    var containsMethod = typeof(string).GetMethod(nameof(string.Contains),<br>    new[] {typeof(string)});<br>    // Handle null property case<br><br>    var propertyNotNull = Expression.NotEqual(property, Expression.Constant(null));<br>    var propertyExpression = Expression.Condition(<br>    propertyNotNull,<br>    Expression.Call(property, toLowerMethod),<br>    Expression.Constant(string.Empty));<br>    <br>    var valueExpression = Expression.Constant(str.ToLower());<br>    if (context.InMemory)<br>    {<br>      return Expression.AndAlso(<br>        propertyNotNull,<br>        Expression.Call(propertyExpression, containsMethod, valueExpression));<br>  }<br><br>  var method = typeof(DbFunctionsExtensions).GetMethod(nameof(DbFunctionsExtensions.Like),<br>  new[] {typeof(DbFunctions), typeof(string), typeof(string)});<br>  <br>  var dbFunctions = Expression.Constant(EF.Functions);<br>  var likeExpression = Expression.Constant($&quot;%{str.ToLower()}%&quot;);<br>   return Expression.AndAlso(<br>    propertyNotNull,<br>    Expression.Call(method, dbFunctions, propertyExpression, likeExpression));<br>  }<br>  throw new InvalidOperationException();<br>  }<br>}</pre><p>In other cases, we need to have filters for a range of values, which should be within a specific range or not. For this, we implemented another override that includes type casting according to the type of the custom column based on its metadata.</p><pre>public class QueryableStringLessThanOrEqualsHandler(InputParser inputParser)<br>: QueryableStringOperationHandler(inputParser)<br>{<br>  protected override int Operation =&gt; DefaultFilterOperations.LowerThanOrEquals;<br>  <br>  public override Expression HandleOperation(<br>  QueryableFilterContext context,<br>  IFilterOperationField field,<br>  IValueNode value,<br>  object parsedValue)<br>  {<br>  var property = context.GetInstance();<br>  if (decimal.TryParse(parsedValue.ToString(), out var decimalValue))<br>  {<br>    var propertyAsDecimal = Expression.Condition(<br>    Expression.Equal(property, Expression.Constant(null, property.Type)),<br>    Expression.Constant(0m, typeof(decimal)),<br>    Expression.Call(typeof(decimal), nameof(decimal.Parse), null, Expression.Convert(property, typeof(string)))<br>  );<br><br>  return Expression.LessThanOrEqual(propertyAsDecimal, Expression.Constant(decimalValue));<br>  }<br><br>  if (DateTimeOffset.TryParse(parsedValue.ToString(), CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal, out var dateTimeValue))<br>  {<br>    var propertyAsDateTime = Expression.Condition(<br>    Expression.Equal(property, Expression.Constant(null, property.Type)),<br>    Expression.Constant(DateTimeOffset.MinValue, typeof(DateTimeOffset)), // Default value for null<br>    Expression.Call(typeof(DateTimeOffset), nameof(DateTimeOffset.Parse), null, Expression.Convert(property, typeof(string)))<br>    );<br><br>    return Expression.LessThanOrEqual(propertyAsDateTime, Expression.Constant(dateTimeValue));<br>  }<br><br>   throw new InvalidOperationException();<br> }<br>}</pre><h3>Translate Custom Statements Into SQL</h3><p>Here, we encountered another challenge: the EF Core provider for Postgres was unable to translate expressions such as Decimals.Parse into SQL. Here, we implemented the following solution: we wrote our own custom functions for PostgreSQL to handle type casting and the corresponding mapping between the C# expression and the database function call.</p><pre>CREATE OR REPLACE FUNCTION public.cast_numeric(input_value text)<br>RETURNS numeric<br><br>LANGUAGE &#39;plpgsql&#39;<br>COST 100<br>IMMUTABLE PARALLEL UNSAFE<br>AS $BODY$<br>BEGIN<br>  RETURN input_value::NUMERIC;<br>EXCEPTION<br>WHEN invalid_text_representation THEN<br>  RETURN NULL;<br>END;<br>$BODY$;</pre><pre>modelBuilder.HasDbFunction(<br>typeof(decimal).GetMethod(nameof(decimal.Parse), new[] {typeof(string)}<br>)).HasTranslation(args =&gt;<br>   new SqlFunctionExpression(<br>   functionName: CastToNumericFunctionName,<br>   arguments:<br>    new[]{<br>         args<br>         },<br>  nullable: true,<br>  argumentsPropagateNullability: new[] {true},<br>  typeof(decimal),<br>  typeMapping: null<br>  )<br>);</pre><p>After that, we got a correctly constructed query and decent performance speed.</p><h3>Exporting Data to the Excel Spreadsheet</h3><p>However, we still have one problem: exporting data by query results into another format, such as xlsx. As we gathered from GraphQL and Hot Chocolate documentation, it cannot return anything other than the usual JSON. However, in our case, we need to build the binary file and return it to the client side based on the query execution results and the already applied filters and sorting. To solve this problem, we found that Hot Chocolate can execute more than one query segment in a single run.</p><p>Another helpful thing we found is that if you mark your query segment method implementation with the serial attribute, it will be executed in sequential order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jUbjLFwmCiwPLu65tj-6cQ.png" /><figcaption>Query segment method signature</figcaption></figure><p>So, if we want to obtain a binary format answer from a GraphQL query, we are calling two queries in sequential mode. The first query retrieves the data response based on filtering and sorting, and the second query prepares the XLSX file and returns it to the client in binary format. However, the following technical challenge is to share data between these two queries, and for that, we are introducing an additional hack. Hot Chocolate works in the same manner as <a href="http://asp.net/">ASP.NET</a> Core Web API, with its own middleware pipeline, and we are going to inject our own logic for managing a shared state inside of that pipeline using an interceptor, which does the following:</p><p>Check if the GraphQL query contains a segment related to the export operation, and if yes, create in the HttpContext object of the current request the unique identifier of the operation:</p><pre>public class ExportOperationInterceptor : DefaultHttpRequestInterceptor<br>{<br>  <br>  public override async ValueTask OnCreateAsync(<br>  HttpContext context,<br>  IRequestExecutor requestExecutor,<br>  OperationRequestBuilder requestBuilder,<br>  CancellationToken cancellationToken)<br>  {<br><br>  await base.OnCreateAsync(context, requestExecutor, requestBuilder, cancellationToken);<br><br>  var clonedBuilder = CloneRequestBuilder(requestBuilder);<br>  var documentString = clonedBuilder.Build().Document?.ToString();<br>  if (!documentString.IsNullOrEmpty())<br>  {<br>    var parsedDocument = Utf8GraphQLParser.Parse(documentString);<br>    if (parsedDocument.Definitions.OfType&lt;OperationDefinitionNode&gt;()<br>        .Any(definition =&gt; definition.SelectionSet.Selections<br>        .OfType&lt;FieldNode&gt;()<br>        .Any(field =&gt; field.Name.Value.Contains(Constants.ExportOperationName))))<br>        {<br>          context.Items[Constants.OperationId] = Guid.NewGuid().ToString();<br>        }<br>   }<br> }<br>  <br>private static OperationRequestBuilder CloneRequestBuilder(OperationRequestBuilder original)<br>{<br>  var clonedBuilder = OperationRequestBuilder.New();<br>  var fields = typeof(OperationRequestBuilder).GetFields(BindingFlags.NonPublic | BindingFlags.Instance);<br>  foreach (var field in fields)<br>  {<br>    var value = field.GetValue(original);<br>    field.SetValue(clonedBuilder, value);<br>  }<br>  <br>  return clonedBuilder;<br>}<br>}</pre><p>After that, the two segments start sequential execution, and in the first query handler, we have a check to see if that operation identifier is present in the current HTTP context, which helps us understand whether we need to share the query result with the following query to perform a transformation to the binary format. For now, we are caching that result in the memory cache (with possible options to use Redis in the future) and attempting to parse and process it in the second query segment handler, which will be executed immediately afterward.</p><pre>var operationId = httpContextAccessor.GetOperationId();<br>if (!operationId.IsNullOrEmpty())<br>{<br>  var operationResult = query.ToList();<br>  await memoryCache.GetOrCreateAsync(operationId, entry =&gt;<br>        {<br>          entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(1);<br>          return Task.FromResult(operationResult);<br>        });<br>}</pre><h3>Conclusion</h3><p>As a result, we implemented a pragmatic approach to building a complex supply chain management system within demanding constraints. By strategically combining a REST API for core management with GraphQL and Hot Chocolate for flexible data querying, the team successfully navigated the challenges inherent in dynamic product attributes, complex filtering and sorting, and efficient data export.</p><p>The solutions presented — from optimizing query materialization with EF Core’s split queries to implementing custom filtering logic, handling EF Core translation limitations with database functions, and devising a unique GraphQL export mechanism — underscore the adaptability and extensibility of these technologies. This case study serves as a testament that even with pre-existing relational databases and tight budgets, sophisticated and performant systems can be delivered by leveraging the power of GraphQL and Hot Chocolate, coupled with inventive problem-solving. Ultimately, the success lay in understanding the tools deeply and extending their capabilities to precisely fit the project’s unique requirements, proving that complex problems can indeed be solved with thoughtful and innovative technical implementations.</p><p>Level up your tech project! Explore <a href="https://emerline.com/services/application-modernization"><strong>our expertise</strong></a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69eeccd2270f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/emerline-tech-talk/solving-complex-data-challenges-with-graphql-and-hot-chocolate-69eeccd2270f">Solving Complex Data Challenges with GraphQL and Hot Chocolate</a> was originally published in <a href="https://medium.com/emerline-tech-talk">Emerline Tech Talk</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Enhance Performance Using Redis and Diverse Data Structures]]></title>
            <link>https://medium.com/emerline-tech-talk/how-to-enhance-performance-using-redis-and-diverse-data-structures-0a21f7f7b33a?source=rss-be0afd09aaa1------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a21f7f7b33a</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[performance]]></category>
            <dc:creator><![CDATA[Viachaslau Kitsun]]></dc:creator>
            <pubDate>Mon, 09 Sep 2024 08:20:31 GMT</pubDate>
            <atom:updated>2024-09-25T18:51:06.033Z</atom:updated>
            <content:encoded><![CDATA[<h4>Real-world example of using Redis hashes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3-ZkNorvzviEwLDAcAZJoQ.png" /></figure><h3>Introduction</h3><p>Redis is a robust in-memory data structure store, often used as a database, cache, and message broker. While it’s primarily known for its speed and efficiency in handling in-memory data, Redis also offers persistence repository options, making it suitable for use as a persistent storage solution.</p><p>Let’s look at a real-world example of Redis being used as persistent storage. To start with some context, we have several microservices hosted as separate pods in Azure Kubernetes Service. These microservices exchange data and process user requests in asynchronous mode, as the query execution time can be long. The user sends a request that is run for execution and periodically pings the server to ask if the result of the request is ready.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqIPizx9h0nvN5h3c1tLDw.png" /><figcaption>Real world example of using Redis hashes</figcaption></figure><p>For the fast execution of user requests, we used Redis as persistent storage for each user request execution context, and we improved overall system performance and stability by using a special Redis data type.</p><h3>Problem and Context</h3><p>For fast data exchange between the microservices that process the user request, Redis was chosen as persistence storage to transfer the request context. This context is a JSON object size &gt;256 kilobytes; it was not possible to transfer it using a message broker due to the 256-kilobyte limit.</p><p>Initially, Redis was used as a distributed cache service that simply stored JSON objects as strings, and the object was updated during the flow execution. An update was implemented by replacing one JSON object with an updated one using a cache key. However, during performance testing of the implemented system, we faced the issue of vast amounts of data being transferred through Redis and reaching the limit of Redis’ overall throughput, which caused Redis’ response time to increase.</p><p>There are a few ways to solve such an issue:</p><ol><li>Upgrade the Azure Cache for Redis to the next tier, increasing its throughput, along with the price.</li><li>Use the Redis Modules, such as Redis.Json, to support extended API for working with JSON documents in Redis.</li><li>Use another Redis data structure and its APIs to decrease the amount of data transferred while updating Redis. This was also the least expensive option.</li></ol><h3>Redis Data Types</h3><p>Redis’s architecture is designed to handle various use cases, from simple caching mechanisms to complex real-time analytics and geospatial queries.</p><p>One of the key strengths of Redis is its ability to persist data to disk, ensuring durability while maintaining high-speed access. This makes it suitable for applications requiring both speed and reliability. Additionally, Redis supports replication, allowing data to be copied across multiple servers for high availability and fault tolerance. This feature is crucial for maintaining service continuity in distributed systems.</p><p>Redis also offers advanced features like Lua scripting, transactions, and pub/sub messaging, which enable developers to build sophisticated applications with minimal effort.</p><p>This variety of features is based on different data types:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*toVLKQHaQmGsRwWkncqE5Q.png" /><figcaption>Redis Data Types</figcaption></figure><h3>How to use Redis hashes to enhance performance</h3><h4>Prerequisites</h4><p>To start working with Redis hashes, you need a C# library that implements the Redis client and encapsulates work with the Redis API. In the current examples, we are using <a href="https://github.com/StackExchange/StackExchange.Redis">StackExchange.Redis</a> installed as a Nuget package.</p><h4>Code examples</h4><p>To solve the problem described in the example above, we are using more than just simple strings. Each entity we store and update in Redis will become an array of hashes. Refer to the following implementation examples:</p><p>First, we will define the interface that should be implemented. This interface must support storing in Redis as a collection of hashes and also reading a collection of hashes to the entity object:</p><pre>public interface IHashEntryConvertible&lt;out T&gt;<br>{<br>    T ConvertFromHashEntries(HashEntry[] entries);<br>    ICollection&lt;HashEntry&gt; ConvertToHashEntries();<br>}</pre><blockquote><em>NOTE: This is only one example of the options that can be implemented. You could use Reflection, or another system of your choice, but explicit interface implementation for each type is the best solution when you want to have fast write and read operations, and when the count of your entities is not more than 10.</em></blockquote><p>Next we create the an entity, which implements the interface declared above:</p><pre>public record OrderProcessingItem: IHashEntryConvertible&lt;OrderProcessingItem&gt;<br>{<br>    public Guid OrderId { get; init; }<br>    public string OrgKey { get; init; }<br>    public string UserKey { get; init; }<br>    public string PostBackUrl { get; init; }<br>   <br>    public OrderProcessingItem ConvertFromHashEntries(HashEntry[] entries)<br>    {<br>        var entriesList = entries.ToList();<br>        return new OrderProcessingItem<br>        {<br>            OrderId = Guid.Parse(entriesList<br>              .Find(x =&gt; x.Name == nameof(OrderId)).Value),<br>            OrgKey = entriesList.Find(x =&gt; x.Name == nameof(OrgKey)).Value,<br>            UserKey = entriesList.Find(x =&gt; x.Name == nameof(UserKey )).Value,<br>            PostBackUrl = entriesList<br>              .Find(x =&gt; x.Name == nameof(PostBackUrl)).Value,<br>        };<br>    }<br><br>    public ICollection&lt;HashEntry&gt; ConvertToHashEntries() =&gt;<br>        new List&lt;HashEntry&gt;()<br>        {<br>            new(nameof(OrderId), OrderId.ToString()),<br>            new(nameof(OrgKey), OrgKey),<br>            new(nameof(UserKey ), UserKey),<br>            new(nameof(PostBackUrl), PostBackUrl ?? string.Empty)<br>        };<br>}</pre><p>The final step is to implement a services that allows for operation creation, retrieval, and updating on the Redis hashes.</p><pre>public class StorageService&lt;TConfig&gt; : IStorageService <br>  where TConfig : ICacheConfig<br>{<br>    private readonly IDatabase _database;<br>    private readonly TConfig _cacheConfig;<br><br>    public StorageService(IOptions&lt;RedisConfiguration&gt; redisConfiguration, <br>      TConfig cacheConfig)<br>    {<br>        _cacheConfig = cacheConfig;<br>        if (redisConfiguration != null)<br>        {<br>            _database = ConnectionMultiplexer<br>              .Connect(redisConfiguration.Value.ConnectionString)<br>              .GetDatabase();<br>        }<br>    }<br><br>    public async Task&lt;T&gt; GetValueAsync&lt;T&gt;(string key, <br>      CancellationToken cancellationToken = default)<br>        where T : IHashEntryConvertible&lt;T&gt;, new()<br>    {<br>        var hashEntries = await _database.HashGetAllAsync(key)<br>          .ConfigureAwait(false);<br><br>        return hashEntries?.Length &gt; 0 ? new T()<br>          .ConvertFromHashEntries(hashEntries) : default;<br>    }<br><br>    public async Task&lt;T&gt; GetFieldValueAsync&lt;T&gt;(string key, <br>      string field, CancellationToken cancellationToken = default)<br>    {<br>        var fieldValue = await _database.HashGetAsync(key, field)<br>          .ConfigureAwait(false);<br><br>        return fieldValue.IsNullOrEmpty ? <br>          default : <br>          SerializationHelper.Deserialize&lt;T&gt;(fieldValue);<br>    }<br><br>    public async Task SetValueAsync&lt;T&gt;(string key, <br>      T value, CancellationToken cancellationToken = default)<br>        where T : IHashEntryConvertible&lt;T&gt;, new()<br>    {<br>        await _database.HashSetAsync(key, <br>          value.ConvertToHashEntries()<br>            .ToArray())<br>            .ConfigureAwait(false);<br><br>        await _database.KeyExpireAsync(key, <br>          TimeSpan.FromSeconds(_cacheConfig.DefaultExpirationTimeInSeconds))<br>            .ConfigureAwait(false);<br>    }<br><br>    public async Task SetFieldValueAsync&lt;T&gt;(string key, <br>      string field, T value,<br>        CancellationToken cancellationToken = default)<br>    {<br>        await _database.HashSetAsync(key, field,<br>            value.GetType().IsEnum ? <br>            value.ToString() : <br>            SerializationHelper.Serialize(value))<br>              .ConfigureAwait(false);<br><br>        await _database.KeyExpireAsync(key, <br>          TimeSpan.FromSeconds(_cacheConfig.DefaultExpirationTimeInSeconds))<br>            .ConfigureAwait(false);<br>    }</pre><p>As these examples show, we can set data to Redis as a JSON object, as well as get or set values for object fields. This is possible because we store objects as collections of hashes and can operate each field separately.</p><h3>Conclusion</h3><p>Redis, a distributed cache and persistent storage solution, offers significant advantages for backend developers and data engineers. Developers can optimize data transfer and storage efficiency by understanding and utilizing Redis’s diverse data structures, such as hashes. This approach addresses performance bottlenecks and enhances the scalability and responsiveness of microservices architectures.</p><p>This article has demonstrated, through practical examples and implementation strategies, how to effectively use Redis to manage large JSON objects and improve system performance. By adopting these techniques, developers can ensure their applications remain robust, efficient, and capable of handling high volumes of data with minimal latency.</p><p>Redis’s versatility and powerful features make it an invaluable tool for modern distributed systems. Mastering its advanced usage can significantly elevate a developer’s skill set and application performance.</p><p>Level up your tech project! Explore <a href="https://emerline.com/?utm_source=medium.com&amp;utm_medium=article&amp;utm_content=techical_blog">our expertise</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a21f7f7b33a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/emerline-tech-talk/how-to-enhance-performance-using-redis-and-diverse-data-structures-0a21f7f7b33a">How to Enhance Performance Using Redis and Diverse Data Structures</a> was originally published in <a href="https://medium.com/emerline-tech-talk">Emerline Tech Talk</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>