<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Enes Bulut on Medium]]></title>
        <description><![CDATA[Stories by Enes Bulut on Medium]]></description>
        <link>https://medium.com/@enes_bulut?source=rss-106e99c6238f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IluHgw5GaRlncp9OV2QYnA.jpeg</url>
            <title>Stories by Enes Bulut on Medium</title>
            <link>https://medium.com/@enes_bulut?source=rss-106e99c6238f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 20:29:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@enes_bulut/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Hybrid AI Automation Engine: Deterministic Rules + LLM Intelligence in Production]]></title>
            <link>https://medium.com/@enes_bulut/building-a-hybrid-ai-automation-engine-deterministic-rules-llm-intelligence-in-production-6755cf48710c?source=rss-106e99c6238f------2</link>
            <guid isPermaLink="false">https://medium.com/p/6755cf48710c</guid>
            <category><![CDATA[workflow-automation]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[agentic-workflow]]></category>
            <dc:creator><![CDATA[Enes Bulut]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 14:22:02 GMT</pubDate>
            <atom:updated>2026-07-07T14:26:19.245Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k1cjq3KuK1oUKfgsmuQuyg.png" /></figure><p><em>How we combined a visual rule engine, an AI workflow generator with self-correcting validation, and proactive data-driven recommendations to automate digital advertising at scale.</em></p><p>Recent research on production AI agents reveals a sobering reality: <strong>68% of errors come from incorrect parameter values</strong>, 21% from wrong tool selection, and 11% from hallucinated data. When we set out to build an automation engine for digital advertising — one that manages real budgets across Google Ads and Meta — we knew that “just use GPT” wasn’t an option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y1xO8nHudq_kHxnHJd8pMg.png" /></figure><p>This article is Part 2 of our engineering series. In <a href="https://medium.com/@enes_bulut/scalable-resilient-distributed-the-architectural-decisions-and-blueprint-for-a-modern-7e6363d05432">Part 1</a>, we covered our distributed data synchronization pipeline. Here, we dissect the automation layer that sits on top of it — a system that turns user intent into validated, executable advertising workflows.</p><p>💡 <em>Spoiler: The answer isn’t “more AI.” It’s knowing exactly where AI helps and where it hurts.</em></p><h3>Chapter 1: 🏗️ Architecture Overview — Why Hybrid Beats Pure AI</h3><p>Every ad-tech startup claims to be “AI-powered.” Most of them mean they wrapped a prompt around an LLM and called it a product. We took a different path.</p><p>To be fair, the fully-agentic end of the spectrum is no longer vaporware. Agencies like <a href="https://battlebridge.com/blog/the-architecture-of-an-agentic-marketing-system-how-we-built-10-autonomous-ai-agents/">BattleBridge</a> report running production marketing operations on autonomous agents — self-reporting 99.7% uptime over 8 months and content output 847% higher than traditional methods. Take vendor-published numbers with the usual grain of salt, but the direction is real. The question for us was never <em>whether</em> AI can act autonomously — it’s <em>where</em> autonomy is acceptable when the actions mutate live ad budgets.</p><p>Our system is built on three distinct layers, each chosen for a specific reason:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AX3g7rnL2ooLCwUoNXTz5Q.png" /></figure><h3>The Three Layers</h3><p><strong>1. Deterministic Rule Engine</strong> — The foundation. A visual flow builder where users construct automation workflows as directed graphs: Source → Condition → Action. Every condition evaluation, every metric comparison, every action dispatch follows deterministic, testable logic. No LLM involved.</p><p><strong>2. AI Augmentation Layer</strong> — The accelerator. Users can describe what they want in natural language (“pause campaigns when CPC exceeds the 7-day average”), and an LLM generates the corresponding workflow steps. But — and this is critical — the generated output must pass through the same deterministic validation pipeline as manually created workflows. The LLM doesn’t get special treatment.</p><p><strong>3. Proactive Intelligence Layer</strong> — The advisor. A cron-based recommendation engine that continuously evaluates user data against predefined template conditions. It’s entirely deterministic — no LLM in the evaluation loop — but the recommendations are surfaced through an AI-powered conversational interface.</p><h3>Why This Separation Matters</h3><p>The key insight is that each layer optimizes for a different quality:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8jiCiK52mBPw9qrDqYE9Qg.png" /></figure><p>This isn’t a compromise — it’s a deliberate architectural decision. We use AI where it adds value (translating intent, explaining complexity) and deterministic logic where correctness is non-negotiable (evaluating conditions, executing mutations).</p><p>Industry data supports this bet on AI-assisted creation — as long as quality is enforced. <a href="https://support.google.com/google-ads/answer/14143250">Google reports</a> that advertisers who improve their Performance Max Ad Strength to “Excellent” see 6% more conversions on average, and that advertisers using AI asset generation during campaign setup are 63% more likely to reach a “Good” or “Excellent” Ad Strength rating. The common thread: AI-generated output paired with a quality gate outperforms either alone.</p><h3>The Tech Stack</h3><ul><li><strong>Framework:</strong> NestJS 10 (modular DI, perfect for the registry patterns we’ll discuss)</li><li><strong>Queue:</strong> BullMQ + Redis (action execution, approval workflow)</li><li><strong>ORM:</strong> Prisma (multi-tenant, config versioning)</li><li><strong>AI:</strong> OpenAI GPT-4.1 (workflow generation &amp; editing) + GPT-4o / GPT-4o-mini (explanations, summaries)</li><li><strong>Monorepo:</strong> pnpm workspaces with @wask/ shared packages</li></ul><h3>Production Numbers</h3><p>Before diving into the architecture, here’s what this system handles in production:</p><ul><li><strong>Thousands of active configurations</strong> across our advertiser base</li><li>Execution runs processed at production volume</li><li><strong>Dozens of pre-built templates</strong> across 6 languages</li><li><strong>20+ distinct action types</strong> across Google Ads and Meta platforms</li></ul><h3>Chapter 2: ⚙️ The Rule Engine — Source, Condition, Action</h3><p>At its core, every automation workflow is a directed graph of three node types. This isn’t novel — what matters is how we made it production-grade.</p><h3>The Step Graph</h3><p>Every workflow follows a strict sequence: <strong>Source → Condition → Action</strong>, with conditions supporting recursive branching via then/else paths.</p><pre>const STEP_TYPE = {<br>  SOURCE: &#39;SOURCE&#39;,<br>  CONDITION: &#39;CONDITION&#39;,<br>  ACTION: &#39;ACTION&#39;,<br>} as const;<br><br>const ENTITY_LEVEL = {<br>  CAMPAIGN: &#39;CAMPAIGN&#39;,<br>  ADGROUP: &#39;ADGROUP&#39;,<br>  AD: &#39;AD&#39;,<br>} as const;<br>type EntityLevel = (typeof ENTITY_LEVEL)[keyof typeof ENTITY_LEVEL];<br>// Discriminated union - the runtime knows exactly which shape to expect<br>type Step = SourceStep | ConditionStep | ActionStep;<br><br>interface SourceStep {<br>  id: string;<br>  type: typeof STEP_TYPE.SOURCE;<br>  source: {<br>    level: EntityLevel;<br>    accounts: AccountSelection[];<br>  };<br>}<br>interface ConditionStep {<br>  id: string;<br>  type: typeof STEP_TYPE.CONDITION;<br>  conditions: ConditionGroup[]; // Recursive AND/OR groups<br>  then: Step[]; // Entities that pass<br>  else: Step[]; // Entities that fail<br>}<br>interface ActionStep {<br>  id: string;<br>  type: typeof STEP_TYPE.ACTION;<br>  action_type: ActionType;<br>  target: { id: string }; // References a previous step&#39;s output<br>  parameters: Record&lt;string, unknown&gt;;<br>}</pre><p>Here’s what a real production workflow looks like (structure taken from production, identifiers anonymized):</p><pre>[<br>  {<br>    &quot;id&quot;: &quot;1-source&quot;,<br>    &quot;type&quot;: &quot;SOURCE&quot;,<br>    &quot;source&quot;: {<br>      &quot;level&quot;: &quot;CAMPAIGN&quot;,<br>      &quot;accounts&quot;: [<br>        {<br>          &quot;platform&quot;: &quot;FACEBOOK&quot;,<br>          &quot;accountId&quot;: &quot;act_1234567890&quot;,<br>          &quot;selectedEntities&quot;: {<br>            &quot;CAMPAIGN&quot;: [{ &quot;id&quot;: &quot;1234567890123&quot;, &quot;name&quot;: &quot;Summer Sale Campaign&quot; }]<br>          }<br>        }<br>      ]<br>    }<br>  },<br>  {<br>    &quot;id&quot;: &quot;1-condition&quot;,<br>    &quot;type&quot;: &quot;CONDITION&quot;,<br>    &quot;conditions&quot;: [<br>      {<br>        &quot;logical_operator&quot;: &quot;AND&quot;,<br>        &quot;conditions&quot;: [<br>          {<br>            &quot;level&quot;: &quot;ADGROUP&quot;,<br>            &quot;metric&quot;: &quot;cpc&quot;,<br>            &quot;parameters&quot;: {<br>              &quot;operator&quot;: &quot;GREATER_THAN&quot;,<br>              &quot;date_preset&quot;: &quot;LAST_7_DAYS&quot;,<br>              &quot;comparison_target&quot;: &quot;CAMPAIGN_AVERAGE&quot;,<br>              &quot;offset_type&quot;: &quot;PERCENTAGE&quot;,<br>              &quot;offset_value&quot;: 20<br>            }<br>          }<br>        ]<br>      }<br>    ],<br>    &quot;then&quot;: [<br>      {<br>        &quot;id&quot;: &quot;1-action&quot;,<br>        &quot;type&quot;: &quot;ACTION&quot;,<br>        &quot;target&quot;: { &quot;id&quot;: &quot;1-condition&quot; },<br>        &quot;action_type&quot;: &quot;DECREASE_AD_SET_BID_AMOUNT&quot;,<br>        &quot;parameters&quot;: { &quot;value&quot;: 10, &quot;value_type&quot;: &quot;PERCENTAGE&quot; }<br>      }<br>    ],<br>    &quot;else&quot;: []<br>  }<br>]</pre><p>This workflow reads: <em>“For selected campaigns, find ad sets where CPC in the last 7 days exceeds the campaign average by 20% or more. For those that match, decrease the bid amount by 10%.”</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FkLkKhBa6y3w%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DkLkKhBa6y3w&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FkLkKhBa6y3w%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/23a8fe83214a52708dd0fa111a63274c/href">https://medium.com/media/23a8fe83214a52708dd0fa111a63274c/href</a></iframe><h3>The Condition Engine</h3><p>The condition engine is where the real engineering lives. It’s not just “is X greater than Y” — it’s a multi-level metric evaluation system with intelligent comparison targets.</p><p><strong>Eight comparison targets</strong>, each with different data requirements:</p><pre>const COMPARISON_TARGET = {<br>  CAMPAIGN_AVERAGE: &#39;CAMPAIGN_AVERAGE&#39;,<br>  ADGROUP_AVERAGE: &#39;ADGROUP_AVERAGE&#39;,<br>  PREVIOUS_PERIOD: &#39;PREVIOUS_PERIOD&#39;,<br>  CUSTOM: &#39;CUSTOM&#39;,<br>} as const;<br><br>switch (comparisonTarget) {<br>  case COMPARISON_TARGET.CAMPAIGN_AVERAGE:<br>    // Weighted average of all campaigns in the account<br>    // CPC = sum(cost) / sum(clicks), not avg(cpc) - prevents skew from low-traffic entities<br>    return this.calculateWeightedAverage(condition, entity, context);<br>  case COMPARISON_TARGET.ADGROUP_AVERAGE:<br>    // Weighted average across sibling ad groups in the same campaign<br>    return this.calculateAdGroupAverage(condition, entity, context);<br>  case COMPARISON_TARGET.PREVIOUS_PERIOD:<br>    // Same entity, previous time window (e.g., last 7 days vs prior 7 days)<br>    // Uses prefetched cache - zero API calls at evaluation time<br>    return this.calculatePreviousPeriod(condition, entity, context);<br>  case COMPARISON_TARGET.CUSTOM:<br>    // Direct numeric comparison (e.g., CPC &gt; 5.00)<br>    return this.getCustomValue(condition);<br>  // Also: AD_AVERAGE, CREATIVE_AVERAGE, KEYWORD_AVERAGE, SEARCH_TERM_AVERAGE<br>}</pre><p><strong>Why weighted averages matter:</strong> A naive arithmetic average of CPC across campaigns would give equal weight to a campaign with 2 clicks and one with 20,000. Our weighted average (sum(cost) / sum(clicks)) prevents low-traffic entities from distorting the baseline. This same logic is shared between the runtime condition engine and the template recommendation evaluator — one formula, two consumers.</p><p><strong>The prefetch strategy</strong> eliminates N+1 queries. Before evaluating any conditions, the orchestrator analyzes what data will be needed:</p><pre>// Before condition evaluation begins:<br>// 1. Analyze all conditions to determine required comparison targets<br>// 2. Bulk-fetch previous period data, sibling entities, parent metrics<br>// 3. Cache everything in ExecutionContext.prefetchedData<br>// 4. Evaluate conditions using only cached data - zero API calls<br>const context: ExecutionContext = {<br>  prefetchedData: {<br>    // Map&lt;Level, Map&lt;parentId, Entity[]&gt;&gt;<br>    CAMPAIGN: campaignCache,<br>    ADGROUP: adGroupCache,<br>  },<br>  prefetchedPreviousPeriodData: {<br>    CAMPAIGN: prevPeriodCache, // Map&lt;entityId, Entity&gt;<br>  },<br>};</pre><h3>The Action Registry Pattern</h3><p>With 20+ action types across 2 platforms and 6 entity levels, we needed a way to add new actions without modifying the executor. The solution is a Strategy Pattern backed by a runtime registry:</p><pre>@Injectable()<br>export class ActionRegistryService {<br>  private readonly registry = new Map&lt;string, BaseAction&gt;();<br>  register(<br>    actionType: ActionType,<br>    platform: Platform,<br>    level: Level,<br>    service: BaseAction,<br>  ): void {<br>    const key = `${actionType}_${platform}_${level}`;<br>    this.registry.set(key, service);<br>  }<br>  get(<br>    actionType: ActionType,<br>    platform: Platform,<br>    level: Level,<br>  ): BaseAction | undefined {<br>    return this.registry.get(`${actionType}_${platform}_${level}`);<br>  }<br>}</pre><p>Each action service self-registers during module initialization:</p><pre>const PLATFORM = { GOOGLE: &#39;GOOGLE&#39;, FACEBOOK: &#39;FACEBOOK&#39; } as const;<br>const ENTITY_LEVEL = { CAMPAIGN: &#39;CAMPAIGN&#39; } as const;<br><br>@Injectable()<br>export class BudgetActionService extends BaseAction implements OnModuleInit {<br>  onModuleInit() {<br>    // One service, multiple registrations<br>    this.registry.register(<br>      &#39;INCREASE_CAMPAIGN_BUDGET&#39;,<br>      PLATFORM.GOOGLE,<br>      ENTITY_LEVEL.CAMPAIGN,<br>      this,<br>    );<br>    this.registry.register(<br>      &#39;INCREASE_CAMPAIGN_BUDGET&#39;,<br>      PLATFORM.FACEBOOK,<br>      ENTITY_LEVEL.CAMPAIGN,<br>      this,<br>    );<br>    this.registry.register(<br>      &#39;DECREASE_CAMPAIGN_BUDGET&#39;,<br>      PLATFORM.GOOGLE,<br>      ENTITY_LEVEL.CAMPAIGN,<br>      this,<br>    );<br>    this.registry.register(<br>      &#39;DECREASE_CAMPAIGN_BUDGET&#39;,<br>      PLATFORM.FACEBOOK,<br>      ENTITY_LEVEL.CAMPAIGN,<br>      this,<br>    );<br>    // ... more registrations<br>  }<br>}</pre><p>Adding a new action type is now a single-file change: write the service, register it, done. The executor, the validator, and the API layer don’t need to change. This is the Open/Closed Principle in practice.</p><h3>Chapter 3: 🤖 AI Workflow Generation — The Self-Correcting Pipeline</h3><p>Here’s the moment most articles would say <em>“and then we called GPT-4 and it just worked.”</em> It didn’t. The interesting engineering starts after the LLM responds.</p><h3>The Problem</h3><p>Users want to type <em>“Check the CPC of my campaigns daily and reduce the budget for those that are increasing”</em> and get a working automation. The LLM is good at understanding this intent. It’s terrible at producing output that passes our 47 custom validators.</p><h3>The Self-Correcting Validation Loop</h3><p>Our solution treats the LLM as a junior developer who writes code that needs review. If the review fails, we send the errors back and ask it to fix them — up to 5 times.</p><pre>async generateFromPrompt(prompt: string, platform: Platform): Promise&lt;GenerationResult&gt; {<br>  const businessRules = getBusinessRules();<br>  const constants = getAllConstants(platform);<br>  let validationErrors: string[] = [];<br>  let attempts = 0;<br>  while (attempts &lt; MAX_VALIDATION_RETRIES) { // MAX_VALIDATION_RETRIES = 5<br>    attempts++;<br>    // Generate steps from the LLM provider - pass previous errors on retry<br>    const rawSteps = await this.llmProvider.generateConfigSteps(<br>      prompt,<br>      platform,<br>      businessRules,<br>      constants,<br>      attempts &gt; 1 ? validationErrors : undefined, // Feed errors back<br>    );<br>    // Transform raw JSON to typed DTOs<br>    const steps = this.transformToStepDtos(rawSteps);<br>    // Run through the SAME validation pipeline as manually created configs<br>    const errors = await this.validateSteps(steps, platform);<br>    if (errors.length === 0) {<br>      return { steps, validationAttempts: attempts };<br>    }<br>    // Validation failed - feed errors back to LLM on next iteration<br>    validationErrors = errors;<br>  }<br>  // All retries exhausted<br>  return { steps: [], finalValidationErrors: validationErrors };<br>}</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FFoRZNUOVsEM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DFoRZNUOVsEM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FFoRZNUOVsEM%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5398afd2326a9393a30fbc6844a99eea/href">https://medium.com/media/5398afd2326a9393a30fbc6844a99eea/href</a></iframe><p><strong>Why this works better than a bigger prompt:</strong> We tried expanding the system prompt with every validation rule. The prompt grew to 12,000 tokens and the LLM still made mistakes — just different ones. The retry loop with structured error feedback converges faster and handles edge cases the prompt can’t anticipate.</p><p><strong>What the LLM receives on retry:</strong></p><pre>Previous attempt failed validation with these errors:<br>- steps.1-condition.conditions[0]: metric &quot;click_through_rate&quot; is not valid. Use &quot;ctr&quot; instead.<br>- steps.2-action: action_type &quot;PAUSE&quot; is not valid for platform GOOGLE at level CAMPAIGN. Use &quot;PAUSE_CAMPAIGN&quot;.</pre><pre>Please fix these issues and regenerate the steps.</pre><p>The LLM sees its own mistakes in context and almost always corrects them. In production, <strong>~78% of generations succeed on the first attempt</strong>, and <strong>97% succeed within 3 attempts</strong>.</p><h3>Source Sanitization</h3><p>Even with validation, the LLM occasionally produces structurally valid but semantically wrong steps — like generating a SOURCE with level: &quot;SEARCH_TERM&quot; (which isn&#39;t a valid source level) or creating duplicate SOURCE steps:</p><pre>const STEP_TYPE = { SOURCE: &#39;SOURCE&#39; } as const;<br>const ENTITY_LEVEL = { CAMPAIGN: &#39;CAMPAIGN&#39; } as const;<br>const VALID_SOURCE_LEVELS = new Set([<br>  ENTITY_LEVEL.CAMPAIGN,<br>  ENTITY_LEVEL.ADGROUP,<br>  ENTITY_LEVEL.AD,<br>]);<br><br>function sanitizeSourceSteps(steps: unknown[]): unknown[] {<br>  const result: unknown[] = [];<br>  let validSourceCount = 0;<br>  for (const step of steps) {<br>    if (step.type !== STEP_TYPE.SOURCE) {<br>      result.push(step);<br>      continue;<br>    }<br>    // Drop sources with invalid levels (KEYWORD, SEARCH_TERM, etc.)<br>    if (!VALID_SOURCE_LEVELS.has(step.source?.level)) continue;<br>    // Keep only the first valid source<br>    if (validSourceCount &gt; 0) continue;<br>    result.push({ ...step, id: &#39;1-source&#39; });<br>    validSourceCount++;<br>  }<br>  return result;<br>}</pre><p>This is defensive coding born from production incidents. The LLM doesn’t know our domain constraints — but our sanitizer does.</p><h3>AI Explains What It Built</h3><p>Generation is only half the story. Users need to understand what a workflow does before activating it. We use separate, purpose-tuned AI calls for explanation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LNSVmkEA-SUyG5MSp4_2pg.png" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F0zx-LHVt4Cc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D0zx-LHVt4Cc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F0zx-LHVt4Cc%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3fa61aaa3ac1f0b045df9edbd0c3fc55/href">https://medium.com/media/3fa61aaa3ac1f0b045df9edbd0c3fc55/href</a></iframe><p>Notice the deliberate model and temperature choices: explanation tasks use higher temperature for natural language quality, while the review summary uses a smaller, cheaper model with lower temperature for factual accuracy. The review summary runs after every execution cycle, so cost optimization matters.</p><p>Putting the whole chapter together — here&#39;s the full journey from a user&#39;s sentence to an activated workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4GT1Sv6r85_PA2Mp7iaNig.png" /></figure><h3>Chapter 4: 🔍 Proactive Template Recommendations — Intelligence That Doesn’t Wait</h3><p>Most automation platforms wait for users to ask “what should I automate?” We flip this: we continuously evaluate their advertising data and proactively suggest what to optimize.</p><h3>The Design Philosophy</h3><p>The critical decision here was: <strong>no LLM in the evaluation loop.</strong> Template recommendations are entirely deterministic. This means:</p><ul><li><strong>Reproducible</strong>: Same data, same recommendations, every time</li><li><strong>Explainable</strong>: We can show exactly why a template was recommended</li><li><strong>Fast</strong>: No API calls to external services during evaluation</li><li><strong>Cheap</strong>: Runs on every account via cron without LLM costs</li></ul><h3>Template Definitions as Code</h3><p>Each recommendation rule is a TypeScript object with structured conditions:</p><pre>const SEARCH_TERM_TEMPLATES: TemplateDefinition[] = [<br>  {<br>    slug: &#39;rec-search-term-exclude-high-cpc&#39;,<br>    enabled: true,<br>    name: {<br>      en: &#39;Exclude High-CPC Zero-Converting Search Terms&#39;,<br>      tr: &#39;Dönüşümü sıfır ve CPC yüksek arama terimlerini hariç tut&#39;,<br>    },<br>    filters: {<br>      platform: &#39;GOOGLE&#39;,<br>      type: &#39;SEARCH&#39;,<br>      objective: &#39;CONVERSIONS&#39;,<br>    },<br>    metricLevel: &#39;SEARCH_TERM&#39;,<br>    dataSource: &#39;API&#39;, // Requires Google Ads API (not available in platform DB)<br>    datePeriod: &#39;LAST_7_DAYS&#39;,<br>    conditions: [<br>      { metric: &#39;conversions&#39;, operator: &#39;EQUALS&#39;, value: 0 },<br>      {<br>        metric: &#39;cpc&#39;,<br>        operator: &#39;GREATER_THAN&#39;,<br>        compareWith: &#39;SEARCH_TERM_AVERAGE&#39;,<br>      },<br>    ],<br>  },<br>];</pre><p>This reads: <em>“For Google Search campaigns targeting conversions, find search terms with zero conversions in the last 7 days where CPC exceeds the average across all search terms. Recommend excluding them.”</em></p><h3>The Evaluation Pipeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gTP4MUvXC91B3QiojYbfcg.png" /></figure><p>The evaluator uses the same weighted average logic as the runtime condition engine:</p><pre>private weightedAverageForMetric(metric: MetricKey, pool: MetricRow[]): number | undefined {<br>  if (pool.length === 0) return undefined;<br>  const totalCost = pool.reduce((s, p) =&gt; s + p.cost, 0);<br>  const totalClicks = pool.reduce((s, p) =&gt; s + p.clicks, 0);<br>  const totalImpressions = pool.reduce((s, p) =&gt; s + p.impressions, 0);<br>  if (metric === &#39;cpc&#39;) return totalClicks &gt; 0 ? totalCost / totalClicks : undefined;<br>  if (metric === &#39;cpm&#39;) return totalImpressions &gt; 0 ? (totalCost / totalImpressions) * 1000 : undefined;<br>  if (metric === &#39;ctr&#39;) return totalImpressions &gt; 0 ? (totalClicks / totalImpressions) * 100 : undefined;<br>  // ... more metrics<br>}</pre><h3>Template Enrichment — One-Click Activation</h3><p>When a recommendation matches, we don’t just tell the user “you should do X.” We pre-populate the workflow template with their specific campaigns:</p><pre>const STEP_TYPE = { SOURCE: &#39;SOURCE&#39; } as const;<br>const ENTITY_LEVEL = { CAMPAIGN: &#39;CAMPAIGN&#39; } as const;<br><br>private enrichMatchWithTemplate(match: TemplateMatch, accountId: string, platform: string) {<br>  const uniqueCampaigns = this.deduplicateCampaigns(match.matchedEntities);<br>  const clonedSteps: Step[] = JSON.parse(JSON.stringify(dbTemplate.steps));<br>  // Inject the matched campaigns into the SOURCE step<br>  const sourceStep = clonedSteps.find(s =&gt; s.type === STEP_TYPE.SOURCE);<br>  if (sourceStep) {<br>    sourceStep.source = {<br>      level: ENTITY_LEVEL.CAMPAIGN,<br>      accounts: [{<br>        platform,<br>        accountId,<br>        selectedEntities: { [ENTITY_LEVEL.CAMPAIGN]: uniqueCampaigns },<br>      }],<br>    };<br>  }<br>  return { ...templateMeta, steps: clonedSteps };<br>}</pre><p>The user sees a recommendation card with their actual campaign names. One click creates a fully configured automation — no manual setup required.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FPZeNqU-3pMA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPZeNqU-3pMA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FPZeNqU-3pMA%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ec6437b94a04180cff172a40d6ca7b94/href">https://medium.com/media/ec6437b94a04180cff172a40d6ca7b94/href</a></iframe><h3>The Hybrid Presentation</h3><p>The recommendation engine is deterministic, but its <strong>presentation</strong> is AI-powered. When the AI Marketing Copilot (covered in Part 3) responds to a user question about campaign performance, it checks pre-loaded cron recommendations and surfaces relevant templates:</p><pre>User: &quot;My CPC has been increasing lately, what should I do?&quot;</pre><pre>AI Copilot: [analyzes campaign data via tools]<br>&quot;Your CPC increased 23% week-over-week, primarily driven by the &#39;Brand Search&#39; campaign...&quot;<br>[Appends an inline template-recommendation token — format illustrative: [[template_rec:rec-search-term-exclude-high-cpc]]]</pre><p>The inline token is stripped from the displayed response and rendered as an interactive template card in the UI. The AI decides <em>when</em> to recommend; the cron engine decides <em>what</em> to recommend. Neither substitutes for the other.</p><h3>Chapter 5: 🔒 Human-in-the-Loop — The Approval Architecture</h3><p>Google’s own documentation acknowledges that 10–20% of advertising decisions require human oversight. We built approval as a first-class feature, not an afterthought — and the implementation uses a creative trick with our job queue.</p><p>There’s a regulatory dimension too. <a href="https://artificialintelligenceact.eu/article/14/">Article 14 of the EU AI Act</a> requires high-risk AI systems to be designed so that natural persons can effectively oversee them — including the ability to interpret outputs, override decisions, and stop the system. Those high-risk obligations were scheduled to apply from August 2, 2026 (a pending “Digital Omnibus” agreement may push that date out), and while classification will be debated per product and jurisdiction, the direction of travel is clear: systems that act autonomously on someone’s behalf need built-in human oversight, not bolted-on disclaimers. An approval-first architecture is that principle made concrete.</p><h3>The Delayed Job Pattern</h3><p>When a workflow has autoApply: false, users want to review proposed changes before they&#39;re executed. The challenge: how do you &quot;pause&quot; a distributed job system mid-execution?</p><p>Our solution: <strong>schedule jobs with a 365-day delay.</strong> They exist in the queue but won’t execute until explicitly promoted.</p><pre>// In ActionChunkPlanningService - when creating BullMQ jobs:<br>const job = await queue.add(<br>  jobName,<br>  { entities: chunk, actionType, accountId, configId, runHistoryId },<br>  {<br>    jobId: `action:chunk:${accountId}:${category}:${runHistoryId}:${chunkIndex}`,<br>    delay: isAutoApply<br>      ? 0 // Execute immediately<br>      : 365 * 24 * 60 * 60 * 1000, // 1 year - effectively &quot;paused&quot;<br>  },<br>);</pre><p><strong>Why 365 days and not infinity?</strong> BullMQ requires a finite delay value — a delayed job is a scheduled entry, not a separate “paused” state. One year is a sentinel: no approval flow lives anywhere near that long, dismissals cancel the jobs, and unresolved reviews expire well before the delay could ever elapse. It’s a pragmatic choice.</p><h3>The Approve → Promote Flow</h3><p>When the user approves changes on the review screen, we promote the delayed jobs to immediate execution:</p><pre>// In RunHistoryApprovedListener - triggered by approval event:<br>async onRunHistoryApproved(event: RunHistoryApprovedEvent) {<br>  // Check if this run has worker jobs (delayed in queue)<br>  const hasWorkerJobs = actionLogs.some(log =&gt; log.jobIds?.length &gt; 0);<br>  if (hasWorkerJobs) {<br>    // Promote all delayed jobs for this run - they execute immediately<br>    const result = await this.jobCancellationService.promoteRunHistoryJobs(<br>      event.runHistoryId,<br>    );<br>    // Result: { promoted: 42, failed: 0 }<br>  } else {<br>    // No worker jobs - execute actions synchronously from saved logs<br>    await this.executeSavedActions(event);<br>  }<br>}</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FFDuKbEyAXAk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DFDuKbEyAXAk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FFDuKbEyAXAk%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c83b683cb724f50e4b81b71daefac09/href">https://medium.com/media/5c83b683cb724f50e4b81b71daefac09/href</a></iframe><p>If the user <strong>dismisses</strong> the review, the jobs are simply cancelled — they never ran, no rollback needed.</p><h3>Event-Driven Module Decoupling</h3><p>The config module and automation module have a natural circular dependency: configs trigger automation, automation updates configs. We break this cycle with events:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cpTe4cIUpf22bD8RdqhRYg.png" /></figure><p>No forwardRef, no circular imports. The event bus (@nestjs/event-emitter) keeps modules cleanly separated.</p><h3>Execution Paths</h3><p>Not all actions need the worker queue. The system chooses the right execution path based on the action type:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CaG1EnuQuT9gEpCDjR4ETQ.png" /></figure><p>For the worker path, entities are chunked based on platform and action type — Google status changes use chunks of 50, creative operations use chunks of 10. Each chunk becomes a BullMQ job with its own retry policy. For details on our BullMQ infrastructure, see <a href="https://medium.com/@enes_bulut/scalable-resilient-distributed-the-architectural-decisions-and-blueprint-for-a-modern-7e6363d05432">Part 1</a> of this series.</p><h3>Chapter 6: 📝 Lessons Learned &amp; The Trade-Offs</h3><h3>1. Hybrid &gt; Pure AI</h3><p>The most important architectural decision we made was <strong>not</strong> going full AI. Deterministic rules for condition evaluation mean we can debug, test, and audit every decision. AI for workflow generation means users don’t need to learn our DSL. This separation gives us reliability where it matters and accessibility where it helps.</p><p>The counterargument is complexity — maintaining two systems is more work than one. We accept this trade-off because the alternative (trusting an LLM with budget mutations on live ad accounts) is not a production-grade strategy.</p><h3>2. Self-Correcting Beats Self-Generating</h3><p>We spent weeks trying to craft the perfect prompt that would generate valid workflows on the first attempt. Success rate plateaued at ~65%. The validation retry loop got us to 97% with a simpler prompt and less engineering effort. The lesson: <strong>invest in the feedback loop, not the prompt.</strong></p><h3>3. Proactive &gt; Reactive</h3><p>Template recommendations required significant upfront investment (data pipeline, evaluation engine, enrichment logic). But the payoff is outsized: users who receive proactive recommendations are significantly more likely to activate their first automation. Don’t wait for users to ask “what should I optimize?” — show them.</p><h3>4. Use Your Infrastructure Creatively</h3><p>The delayed job pattern for approvals wasn’t in any architecture textbook. It emerged from asking: “We already have BullMQ — can we use it as an approval mechanism?” The answer saved us from building a separate approval queue system. Look at your existing infrastructure before reaching for new tools.</p><h3>5. Explanation Is a Feature</h3><p>It’s tempting to treat AI explanation endpoints as nice-to-haves. They’re not. When a user sees <em>“decrease bid amount by 10% for ad sets where CPC in the last 7 days exceeds the campaign average by 20%”</em> as a natural language summary, they trust the system. When they see raw JSON, they don’t. We use three separate AI configurations (gpt-4o at temp 0.7, gpt-4o at temp 0.5, gpt-4o-mini at temp 0.4) each tuned for its specific explanation task — it’s worth the cost.</p><h3>Acknowledging the Trade-Offs</h3><ul><li><strong>Validation retry adds latency.</strong> Each retry is another LLM round-trip (~2–5 seconds). For the 22% of cases that need retry, generation takes 5–15 seconds instead of 3. We accept this because a slow correct result beats a fast wrong one.</li><li><strong>Deterministic recommendations have blind spots.</strong> Our template definitions cover common optimization patterns, but they can’t discover novel strategies. We mitigate this by continuously expanding the template library based on what users build manually.</li><li><strong>The approval workflow adds friction.</strong> Some users want fully autonomous optimization. We default to autoApply: false for new users and let them opt into full automation after they trust the system. Safety over speed.</li></ul><h3>What’s Next 🚀</h3><p>This engine processes workflows, but it doesn’t <em>converse</em>. In <strong>Part 3</strong>, we’ll tell the story of our AI Marketing Copilot — which started life as a single tool-calling orchestrator bolted onto our API monolith, and was rebuilt from the ground up as a dedicated multi-agent microservice: a supervisor orchestrating 5 specialized subagents, 39 tools, real-time streaming over Redis Streams, and human-in-the-loop approvals that can pause a run mid-flight and resume it later.</p><p>And it doesn’t just talk about your campaigns — it ships them. By the end of Part 3, you’ll see how a chat conversation turns into a real, published Google Ads campaign. Stay tuned. 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6755cf48710c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scalable, Resilient, Distributed: The Architectural Decisions and Blueprint for a Modern…]]></title>
            <link>https://medium.com/@enes_bulut/scalable-resilient-distributed-the-architectural-decisions-and-blueprint-for-a-modern-7e6363d05432?source=rss-106e99c6238f------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e6363d05432</guid>
            <dc:creator><![CDATA[Enes Bulut]]></dc:creator>
            <pubDate>Tue, 04 Nov 2025 08:42:48 GMT</pubDate>
            <atom:updated>2025-11-04T08:42:48.316Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scalable, Resilient, Distributed: The Architectural Decisions and Blueprint for a Modern High-Throughput Data Sync Pipeline 🚀</h3><h3>The Problem: When Scale Exposes Architectural Gaps</h3><p>Our initial data synchronization system worked well for the first few accounts — a straightforward monolithic service that fetched data from external APIs and wrote it to the database. But as we scaled to hundreds of accounts processing millions of data points daily, fundamental problems emerged.</p><p>The architecture had a single queue where lightweight planning operations and heavy API calls competed for the same worker pool. Monolithic workers handled orchestration, data fetching, and persistence in one codebase, with no clear separation of concerns. Each worker consumed gigabytes of RAM by loading unnecessary modules. Large accounts monopolized all workers, bulk operations hit PostgreSQL’s parameter limits, and database connection pools were exhausted.</p><p>We had no retry mechanisms — transient API errors became permanent failures. No circuit breakers meant we hammered failing endpoints indefinitely. No distributed locking led to data corruption from concurrent syncs. No watchdog recovered stuck jobs. Observability was basic logging without structure or real-time monitoring. Debugging meant SSH-ing into servers and grepping logs. We were scaling blind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ofx7tD1CddMPlskk0228Wg.png" /></figure><h3>The Solution: A Distributed Architecture Built on Proven Patterns</h3><p>Rather than patching individual problems, we rebuilt from first principles using battle-tested distributed systems patterns. The result is a platform with separated orchestration and execution layers, specialized workers for each job type, intelligent retry and circuit breaking, fair resource allocation through bulkheads and concurrency groups, and comprehensive observability. Every architectural decision in this document exists to solve one of the problems above. This is production-tested infrastructure processing billions of data points and counting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vGVPACV4LziStd95bwsyyQ.png" /></figure><p>In modern data engineering, building a pipeline that is merely functional is not enough. The true challenge lies in creating a system that is <strong>resilient, scalable, and fault-tolerant</strong>, especially when dealing with high-volume data streams from disparate and often unreliable external APIs.</p><p>This is a technical case study of how we engineered a distributed data synchronization platform, focusing on the architectural patterns and technology choices that enable it to process <strong>millions of data points with high reliability</strong>.</p><p>This article will bypass high-level business logic to focus on the core engineering principles that make our system tick. We will dissect our approach to concurrency, fault tolerance, dynamic scalability, and the rationale behind our tech stack. This is for the engineers in the trenches who appreciate the art of building robust, production-grade systems.</p><blockquote>💡 <strong>Spoiler alert</strong>: There’s no silver bullet, but there are some pretty good patterns.</blockquote><h3>Chapter 1: Architectural Blueprint: A System of Specialized Components</h3><p>Our architecture is founded on the principle of <strong>separation of concerns</strong>, taken to its logical conclusion. Think of it as the Marie Kondo method for distributed systems — every component has its place and a single, well-defined purpose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LZoNcv6N8iDMlXw-XeT7PQ.png" /></figure><h3>System Components</h3><ol><li><strong>🧠 The Orchestration Layer (</strong><strong>sync-manager):</strong> Acts as the central control plane. It receives sync requests, creates parent flow jobs, and dispatches them via RabbitMQ. It does <strong>no data processing itself</strong>, which is crucial for independent scaling.</li><li><strong>💪 The Processing Layer (</strong><strong>sync-worker):</strong></li></ol><ul><li><strong>[jobType]-batch-worker (Planning Layer):</strong> Receives sync jobs from RabbitMQ, plans chunk distribution, and creates thousands of chunk jobs in BullMQ. These workers are I/O-light but planning-intensive.</li><li><strong>[jobType]-chunk-queue(BullMQ/Redis):</strong> The staging ground where chunk jobs wait to be processed. Provides advanced features like priority queues, rate limiting, and concurrency groups.</li><li><strong>[jobType]-chunk-worker (Execution Layer):</strong> Processes individual chunks by fetching data from external APIs and performing bulk upserts to PostgreSQL. These workers are I/O-heavy and CPU-intensive.</li><li><strong>sync-events-listener (Event Driven Orchertrator &amp; Bridge):</strong> Listens to BullMQ events and triggers follow-up actions. Example: When Phase 1 completes, it signals the orchestrator to start Phase 2.</li></ul><h3>Chapter 2: Core Architectural Patterns in Action</h3><p>Beyond the components, several powerful patterns ensure our system is both scalable and resilient.</p><h4>🤝 Orchestrator/Worker &amp; Fan-Out: A Symphony of Parallelism</h4><p>Our system strictly adheres to the Orchestrator/Worker pattern. The clear separation between the sync-manager (planning) and the sync-worker fleet (execution) is the cornerstone of our scalability. It allows us to tune and scale each layer independently, preventing bottlenecks in one from affecting the other.</p><p>This separation directly enables the <strong>Fan-Out Pattern</strong>, where a single sync request is “fanned out” into thousands of individual jobs processed in parallel by the worker fleet. This is how we achieve high throughput.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qabyQZ65GOKy9vbV-_sLyA.png" /></figure><h4>🛡️ Bulkhead Pattern: Isolation &amp; Fair Scheduling</h4><p>In a multi-tenant system, one tenant’s problem must not sink the whole ship. We implement the Bulkhead pattern both reactively and proactively.</p><ul><li><strong>Reactive Isolation:</strong> When an external API rate-limits a specific ad account, the job for that account is gracefully moved to a delayed state, and a temporary lock is placed in Redis. Workers immediately pick up jobs for other, healthy accounts, preventing a single point of failure from halting the entire system.</li><li><strong>Proactive Scheduling with Concurrency Groups:</strong> To prevent a “<em>noisy neighbor</em>” problem, we leverage <strong>BullMQ’s Concurrency Groups</strong>. We treat each adAccountId as a unique group and configure workers to process a maximum number of jobs <em>per account</em> concurrently (e.g., 5), even if the worker&#39;s total capacity is 500. This ensures fair resource allocation, maximizes overall system throughput, and prevents any single large account from causing a rate-limit bottleneck.</li></ul><h3>📊 Staged Processing &amp; Resource-Aware Chunking</h3><p>Our pipeline separates tasks with different performance characteristics (CPU-bound planning vs. I/O-bound execution) into specialized stages. This allows each part of the system to scale independently and operate at peak efficiency. The handoff between these stages is managed by our queuing infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PdBojjHBe66c44CG_L7w9w.png" /></figure><p>A key intelligence in this process is <strong>Resource-Aware Chunking</strong>. The size of each data chunk isn’t arbitrary. It’s <strong>dynamically calculated</strong> based on PostgreSQL’s hard limit of <em>~32,000 parameters per query</em>. This resource-aware logic prevents runtime database errors, optimizes write throughput, and is a prime example of our <em>“Resource Awareness”</em> design philosophy.</p><h4>⚡ Dynamic Module Loading: Runtime Efficiency</h4><p>One of our proudest achievements is runtime module assembly based on environment variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uHUWFYDLgH_BI5sFabmYUA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Azm1c4HuzKyGUBAEDUJ10Q.png" /></figure><p><strong>The Payoff</strong>: <strong>68% memory savings per worker</strong> (from 2.5GB to 800MB). In a world where memory is money, this adds up fast.</p><h3>Chapter 3: The Two-Phase Initial Sync Pattern</h3><p>One of the most significant challenges in data synchronization is handling the “initial sync” for a new account. An account might have years of historical data, and attempting to sync it all at once can overwhelm both our system and the external API, leading to a poor user experience.</p><p>We solve this with a <strong>Two-Phase Initial Sync</strong> pattern, a workflow designed to deliver value to the user as quickly as possible while protecting system stability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uKYja7RjntjtL2ewzc4sSw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Hm7fbp_AFIOWc8e7Bhe4A.png" /></figure><h4>How It Works</h4><ol><li><strong>Phase 1 (The Sprint):</strong> When a new account is connected, the OrchestratorService immediately schedules high-priority jobs to sync the most recent and relevant data—typically the last 30 days. This includes both details (campaigns, ads) and performance metrics. The goal is to populate the user&#39;s dashboard with actionable data within seconds.</li><li><strong>Event-Driven Trigger:</strong> Once Phase 1 completes, BullMQ emits a parent flow completion event. A dedicated <strong>sync-events-listener</strong> service, which is constantly monitoring the queue, catches this specific event.</li><li><strong>Phase 2 (The Marathon):</strong> Upon receiving the completion event, the sync-events-listener sends a message back to the OrchestratorService (typically via RabbitMQ) to trigger the second phase. The orchestrator then schedules a new set of low-priority, long-running jobs to backfill the remaining historical data (e.g., the 11 months preceding Phase 1). Crucially, this phase often only syncs performance metrics, as the core entity details were already captured in Phase 1.</li></ol><h4>The Code in Action</h4><p>The logic is orchestrated via a parent INITIAL_SYNC_FLOW job that tracks the overall progress.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5qyT3SIvaQF_x1vDjFuvWw.png" /></figure><p>This pattern is a powerful example of how to architect for both <strong>perceived performance</strong> and <strong>system resilience</strong>. It ensures a responsive user experience without compromising the stability required to process massive datasets.</p><h3>Chapter 4: Battle-Tested Resilience Engineering</h3><p>Building distributed systems taught us that failure isn’t a bug — it’s a feature. The question isn’t <em>if</em> something will break, but <em>when</em> and <em>how gracefully</em> you handle it.</p><h4>📦 Idempotency by Design: The Foundation of Safety</h4><p>The most critical aspect of our resilience strategy is that all our jobs are <strong>idempotent</strong>. Running the same job multiple times produces the same result as running it once, thanks to INSERT ... ON CONFLICT (upsert) operations. This simplifies our entire error-handling model, as the default safe action is always to <strong>retry</strong>. This is why we don&#39;t need the complexity of a Saga pattern; a failure in one job doesn&#39;t require a complex rollback of others—it just needs to be retried.</p><p><strong>Aggressive Retry Strategy:</strong> Because our operations are idempotent, we can afford to be aggressive with retries:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cr-HAn6PSH-aBaJ1_y4RWQ.png" /></figure><p>Eight attempts might seem excessive, but in a world of flaky APIs and transient network issues, it’s a lifesaver. A job that fails due to a momentary network blip at 3 AM will automatically recover without waking anyone up. The retention policy keeps enough history for debugging without bloating Redis memory.</p><h4>💡 The Intelligent Circuit Breaker</h4><p>External APIs are unreliable. Our circuit breaker handles this gracefully by wrapping our outgoing API calls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FzaTQ226e-qJUP6LQth_5g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MbhEPPgt5J3JZxYhid7O8Q.png" /></figure><p>This prevents us from hammering an already-struggling external service, allowing it time to recover.</p><p><strong>Critical Timeout Configuration:</strong> Our system enforces strict timeouts at multiple levels to prevent jobs from hanging indefinitely:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eOrq5ZpiETmHSH1sczQoqw.png" /></figure><h4>🔒 Distributed Coordination with Redis</h4><p>In a distributed system, coordinating work between dozens of workers is essential to prevent chaos. We use Redis for two distinct but related patterns: <strong>semaphores</strong> for managing system-wide load and <strong>locks</strong> for ensuring resource exclusivity.</p><h4>🚦 System-Wide Concurrency with Semaphores</h4><p>To prevent the entire system from being overwhelmed (e.g., by starting too many sync planning operations at once), we use a semaphore pattern. This acts like a counter that limits the number of concurrent processes. We use Redis’s atomic INCR and DECR operations to implement this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7tZ2LFHRls_EombSN1zIjg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-setkn6iElD02DeMOFXrvA.png" /></figure><h4>🔐 Resource Locking with SETNX</h4><p>While a semaphore limits access to a pool of resources, a lock (or mutex) ensures exclusive access to a single resource. For this, we use Redis’s atomic SET command with the NX (Not eXists) option. This is the modern, industry-standard implementation of the <strong>SETNX pattern</strong>, which atomically performs two actions:</p><ol><li>Sets the key <strong>only if it does not already exist</strong>.</li><li>Sets a Time-To-Live (TTL) on the key in the same operation.</li></ol><p>This atomicity is critical. Using separate SETNX and EXPIRE commands would create a race condition where a system crash between the two could lead to a permanent lock (deadlock). Our approach guarantees that every lock will eventually expire.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i-xGFJ22-qr4AgMLM8Cbag.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OwwQgtdbMvI3gQDrLLp_tw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k0vWP5kMCtNfZL1WKepJ7Q.png" /></figure><p><strong>Production-Tested Lock TTL Values:</strong> Our lock durations are calibrated based on real-world sync patterns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Iy9HzaiRz0sEUx2A56VySA.png" /></figure><p>The ACCOUNT_SYNC_LOCK prevents the same account from running multiple syncs simultaneously, a scenario that caused data corruption in our early days. The HISTORICAL_SYNC_LOCK ensures only one system-wide historical backfill runs at a time—a critical safeguard when processing months of historical data. Note the 1-hour TTL paired with a heartbeat mechanism; the lock auto-renews while the job is active, preventing deadlocks if a worker crashes.</p><p>This distinction between limiting overall throughput (semaphore) and ensuring exclusive access (lock) is fundamental to our system’s stability.</p><h4>🛡️ Database-Level Resilience with Optimistic Locking</h4><p>To prevent race conditions during concurrent database updates (e.g., two processes trying to update the same AccountSync record), we use an optimistic locking strategy. Instead of locking the database row, we add a version column.</p><p>The update operation checks if the version has changed since the data was read. If it has, the transaction fails, and our utility function retries the entire read-update operation with exponential backoff.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hs8UgHsfzCE3afcqzzVYGA.png" /></figure><p>This approach is more performant than pessimistic locking as it avoids long-lived database locks, making it ideal for our high-throughput system.</p><h4>🔍 Proactive Reconciliation</h4><p>Beyond simple error handling, our system performs intelligent post-job data verification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oEZG22jAl5AzONG9pOfRMQ.png" /></figure><ul><li><strong>Data Completeness Checks:</strong> After a sync, we query the source and destination to validate expected vs. actual record counts.</li><li><strong>Automatic Backfill:</strong> If discrepancies are found, the system automatically schedules jobs to recover the missing data.</li><li><strong>Quality Assurance:</strong> This prevents silent data loss, one of the most insidious bugs in any data pipeline.</li></ul><h3>🔍 The Watchdog: Detecting and Recovering from Zombie Jobs</h3><p>Despite all our safeguards, jobs can still get “stuck” — a worker crashes mid-execution, a database connection hangs, or a rare bug causes an infinite loop. Our <strong>Watchdog service</strong> runs periodic sweeps to detect and recover these “stale” jobs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wPXQDH-X1fh4mvKn7v0m9g.png" /></figure><p>The 2-hour threshold was chosen empirically — our longest-running legitimate chunks complete in ~30 minutes, so 2 hours provides a generous buffer. The watchdog has saved us countless times, automatically recovering jobs that would have required manual intervention in our old system.</p><h3>Chapter 5: The Tech Stack That Just Works</h3><p>Every technology was chosen deliberately to solve a specific problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z_vM4hq2OLqOc4G770rMsw.png" /></figure><h4>🏗️ Framework Stack</h4><ul><li><strong>NestJS</strong>: Its modular architecture and powerful dependency injection are lifesavers in complex systems.</li><li><strong>pnpm + Monorepo</strong>: For efficient dependency management and sane code sharing across our services.</li></ul><h4>📬 Message Queue Strategy</h4><ul><li><strong>RabbitMQ</strong> 📮 (sync-manager ↔ BATCH workers): Reliable inter-service messaging with durable queues and service decoupling.</li><li><strong>BullMQ/Redis</strong> ⚡ (BATCH workers → CHUNKS workers): Intra-service job queue with priority scheduling, rate limiting, and retry logic.</li></ul><h4>🗄️ Our Database Philosophy: Performance by Design</h4><p>Our database isn’t just a data store; it’s a high-performance engine tuned for our specific workload. We combine the type-safety of an ORM with the raw power of SQL and advanced database features.</p><p>1.<strong> The Right Tool for the Job: Prisma + Raw SQL:</strong> We use <strong>Prisma</strong> for type safety in most operations, but we don’t hesitate to drop down to <strong>hand-tuned Raw SQL</strong> for complex, performance-critical queries where every millisecond counts.</p><p>2.<strong> Multi-Tenancy with HASH Partitioning:</strong> To ensure that one large client account doesn’t slow down queries for others, we partition our core tables (like campaigns, ads) by account_id. This physically separates data, meaning a query for one account only scans a small, dedicated partition instead of the entire multi-terabyte table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ca8tU-WoIlMin7mF-ySoLg.png" /></figure><p>3.<strong> Time-Series Data with RANGE Partitioning:</strong> For metrics tables (campaign_metrics), data is partitioned by date. This makes time-based queries incredibly fast and simplifies data lifecycle management—deleting old data is as simple as dropping a partition, an instantaneous operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oea29WidTqo6r4w-0CKeEw.png" /></figure><p>4. <strong>Surgical Indexing for Peak Performance:</strong> We design indexes to match our most common query patterns:</p><ul><li><strong>Query-Driven Composite Indexes →</strong> An index on [campaign_id, date] allows the database to instantly locate data for a specific campaign within a date range, avoiding slow, full-table scans.</li><li><strong>Indexes for Idempotency →</strong> A comprehensive unique index on sync_jobs allows the Orchestrator to instantly check if an identical job has already been requested, preventing duplicate work.</li><li><strong>Unique Indexes for Data Integrity →</strong> Multi-column @@unique constraints prevent duplicate data entry, acting as our safety net against race conditions and processing errors.</li><li><strong>GIN Indexes for Full-Text Search →</strong> For fields like campaign.name, we use PostgreSQL&#39;s GIN index with pg_trgm for lightning-fast partial text searches (LIKE &#39;%search_term%&#39;).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yjTxIl3zQfc82Sis4BFRUw.png" /></figure><p>5.<strong> OLTP vs. OLAP Separation:</strong></p><ul><li><strong>🐘 PostgreSQL (OLTP):</strong> Our primary transactional database. It’s the <strong>source of truth</strong>, optimized for high-throughput writes and reads using all the techniques mentioned above.</li><li><strong>📊 ClickHouse (OLAP):</strong> Fed by a real-time CDC pipeline from PostgreSQL, this columnar store is our analytics powerhouse, delivering sub-second queries on billions of rows without impacting the transactional workload.</li><li><strong>Connection Pool Sizing: The Goldilocks Zone:</strong> Connection pools that are too small cause bottlenecks; too large, and you exhaust database resources. We sized ours based on workload characteristics:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3-l6qw2do8v8Fw-LWZ0pGg.png" /></figure><p>Why only 6 connections per worker? Because we have 50+ workers. If each worker opened 50 connections, we’d have 2,500 connections competing for database resources. Instead, 6 connections × 50 workers = 300 total connections, well within PostgreSQL’s max_connections limit. The key insight: <strong>in a distributed system, connection pool size per instance × number of instances = total database load</strong>. We optimize globally, not locally.</p><h3>Chapter 6: Observability That Actually Helps</h3><p>You can’t fix what you can’t see. We built observability into our system from day one.</p><h4>The Monitoring Stack 📊</h4><ul><li><strong>Queue Health:<br>Tool:</strong> BullBoard<br><strong>Purpose:</strong> Real-time job monitoring, debugging, and manual retries.</li><li><strong>Performance Metrics:<br>Stack:</strong> Prometheus + Grafana<br><strong>Metrics:</strong> Job throughput, queue latency, API error rates, DB query times.</li><li><strong>Error Tracking:<br>Tool:</strong> Sentry<br><strong>Purpose:</strong> Real-time error alerts with stack traces and context.</li><li><strong>Structured Logging:<br>Tool:</strong> Pino (JSON format)<br><strong>Purpose:</strong> High-performance logging, ready to be ingested by ELK or similar.</li></ul><h3>🔔 Real-Time Operational Alerts with Slack</h3><p>Dashboards show trends, but for immediate operational awareness, we pipe critical events directly into Slack. This isn’t about spam; it’s about delivering actionable, real-time information to the team.</p><ul><li><strong>Sync Lifecycle Events:</strong> We get instant notifications for Initial Sync Started, Job Completed, and, most importantly, Initial Sync Flow Completed.</li><li><strong>Rich Context:</strong> Each alert is packed with context like AccountId, JobType, Platform, and performance metrics such as Duration, as seen in the image below.</li></ul><p>This proactive alerting transforms observability from a passive, “go look at the dashboard” model to an active, push-based system. It allows us to spot and diagnose issues in seconds, often before they impact the wider system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AT2g2iJM5br9BUHwCWnjVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*az2pSLnoJvNH-p-0Ln46xw.png" /></figure><p><em>Our custom Slack alerts provide instant visibility into the sync pipeline.</em></p><h4>Health Checks That Matter ❤️</h4><p>Our /health endpoint is more than just a 200 OK. It&#39;s a comprehensive dashboard of the service&#39;s current state</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ql4HVa6AQ85jrcq5GIvjHQ.png" /></figure><h4>Proactive Health Checks: Self-Regulating Queues</h4><p>Reactive monitoring is good, but proactive, self-regulating systems are better. A sudden flood of sync requests can cause queue backlogs, overwhelm workers, and risk database stability. We prevent this with a QueueMonitorService.</p><p>This service continuously analyzes BullMQ queue metrics like latency and waiting job counts to calculate an overall “system health” score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZQEWa3FO4Sfe3PbTMGLKaQ.png" /></figure><p>Based on this real-time health assessment, the OrchestratorService can make intelligent decisions <em>before</em> publishing new jobs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7BZ02h_OxHJ5p6ZawcHD4Q.png" /></figure><p>This acts as an automatic fuse, preventing the system from overwhelming itself. It’s a simple but incredibly powerful pattern for building resilient, self-healing systems.</p><h3>Chapter 7: Lessons Learned (The Hard Way)</h3><h4>Design Philosophy 🎯</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rJzQzM103ALN5MorCa30qQ.png" /></figure><ol><li><strong>Failure is the Default</strong>: Design every component assuming its dependencies will fail.</li><li><strong>Strict Separation of Concerns</strong>: Each service has one job and does it well.</li><li><strong>Resource Awareness</strong>: Build logic that adapts dynamically to system constraints (e.g., dynamic chunking).</li><li><strong>Observability First</strong>: If you can’t measure it, you can’t improve it.</li></ol><h4>Technical Principles 🔧</h4><ol><li><strong>Idempotency by Design</strong>: Makes retries safe and simple.</li><li><strong>Stateless Workers</strong>: Enables effortless horizontal scaling.</li><li><strong>Distributed Coordination</strong>: Use Redis for locking, semaphores, and rate limiting.</li><li><strong>Smart Defaults</strong>: Configuration that works well out of the box but is tunable.</li></ol><h3>Acknowledging the Trade-Offs: No Silver Bullets</h3><p>Every architectural decision brings with it a set of trade-offs. While this article focuses on the benefits of the chosen patterns, it’s important to acknowledge the challenges and alternative paths:</p><h4>🔀 <strong>Microservices vs. Monolith</strong></h4><p>This distributed architecture offers enormous advantages in terms of independent scaling and development of components. However, this comes at the cost of increased operational complexity, a more challenging development process, and higher initial costs. For a smaller-scale or early-stage project, a well-designed monolithic architecture can be both faster to develop and cheaper to operate.<strong> This architecture is a deliberate choice to solve problems that arise at a certain scale</strong>.</p><h4>📨<strong> The Queuing Strategy (RabbitMQ + BullMQ + Kafka)</strong></h4><p>Why use two different queue systems instead of a single technology? RabbitMQ is excellent for <strong>reliable and durable inter-service messaging</strong> (Orchestrator → Worker). BullMQ, with its Redis-based structure, offers much richer and more flexible control over <strong>job prioritization, delay, rate limiting, and retries</strong>. This was critical for the layer where we manage thousands of small “chunk” jobs. For a simpler system, a single service like RabbitMQ or AWS SQS might have been sufficient. Kafka, on the other hand, is more optimized for <strong>high-volume event streaming</strong> and would have been overly complex for our “job queue” needs.</p><h4>⚠️ <strong>Initial Complexity and “Over-Engineering” Risk</strong></h4><p>This blueprint is undoubtedly complex and costly for a new project. The patterns described here provide a roadmap for systems that are facing or aiming to face challenges like high data volume, concurrency, and fault tolerance.<strong> Not every project needs this level of architecture</strong>.</p><p><em>Our advice is always to start simple and evolve the architecture as the system’s needs grow.</em></p><h3>Chapter 8: Our Infrastructure Philosophy (The Great Migration)</h3><h4>Going Against the Current 🌊</h4><p>In 2025, suggesting you’ll host your own infrastructure is like showing up to a Tesla convention with a horse and buggy. But hear us out — sometimes the old ways work better.</p><p><strong>The Numbers Don’t Lie:</strong></p><ul><li><strong>AWS/Supabase managed services</strong>: ~$XX,XXX/month for our workload</li><li><strong>Self-hosted VPS infrastructure</strong>: ~$X,XX/month for the better? performance</li><li><strong>Savings</strong>: <strong>4x cost reduction</strong> with infinitely better control</li></ul><p>But it wasn’t just about money. Running our own PostgreSQL and Redis instances gave us the kind of fine-grained control that makes database administrators weep with joy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6kIYEPWUcR1MtBDlT2nJlQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_hcLU6T2Q2CbrXJGvurgVA.png" /></figure><p><strong>Application-Level Redis Configuration:</strong> Beyond server settings, our application-level Redis client is tuned for distributed system reliability:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1tIghyUtO2UtOzYsrEnB5A.png" /></figure><p>The ENABLE_OFFLINE_QUEUE: false is particularly important. When Redis goes down, we want to <strong>fail fast</strong> and alert the team, not silently queue commands in memory and give the illusion that everything is fine. The 5-second TCP keepalive prevents intermediate firewalls from killing &quot;idle&quot; connections—a lesson learned after mysterious connection drops in production.</p><h3>The Final Word</h3><p>Building this system has been a masterclass in distributed systems engineering. We’ve learned that there’s no perfect architecture — only a series of trade-offs that align with your specific needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/0*V6AQzVaB65QSA6bS.gif" /></figure><h4>The Numbers That Matter 📈</h4><p>6 months in production:</p><ul><li><strong>99.7% uptime</strong> (including planned maintenance)</li><li><strong>4.2 billion data points processed</strong> and counting</li><li><strong>Zero data loss incidents</strong> (knock on wood)</li><li><strong>~$XXK saved</strong> in infrastructure costs compared to managed service alternatives</li></ul><p>But the most important metric? <strong>Developer confidence</strong>. Our team deploys fearlessly because we trust the system to handle whatever production throws at it.</p><h3>What’s Next? 🚀</h3><p>The journey continues:</p><ul><li><strong>Full containerization</strong> with Docker and Kubernetes for even better scaling.</li><li><strong>ML-powered optimization</strong> for smarter job scheduling and resource allocation.</li><li><strong>Multi-region deployment</strong> for global scale and disaster recovery.</li></ul><p><em>This has been our journey from a simple script to a production-grade distributed system. Every line of code represents a lesson learned, often the hard way. We hope sharing our experience helps you build better systems and avoid some of the pitfalls we discovered along the way.</em></p><p><strong>Happy scaling! 🚀</strong></p><p>Github Repository For Code Examples: <a href="https://github.com/enescloud/data-ingestion-code-samples">https://github.com/enescloud/data-ingestion-code-samples</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e6363d05432" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>