A Past-Life Regression and the Emotions I Carried

I wanted to try past-life regression because I believed that many of our emotions and decisions might be shaped by experiences accumulated across past lifetimes. I wanted to explore whether any past experiences were affecting my current life.

I reached out to a psychotherapist who specializes in past-life regression. She explained that there were no guarantees my consciousness would reveal anything. Still, I was curious. I felt there was something there worth exploring.

I felt very lost. I still do. I often struggle with anxiety and emptiness, too afraid to join in life. An observer of life, watching from the sidelines.

Our first session lasted about two hours. I shared many of the things that were troubling me, especially the persistent nudges from my intuition that kept demanding my attention — and still do. But that isn’t the heart of this story.

During our second session, she guided me into a meditative, hypnotic state and asked me to recall an event from as far back as I could remember.

I saw myself running down a street in Montreal. I was about two or three years old. I’m 65 now. I was running to see my Aunt Joyce when I tripped and struck my forehead on a concrete step leading up to a second-floor apartment.

I remembered being in her arms while my mother called the police. They took my mother and me to the Children’s Hospital.

A nurse brought me into a room. My mother wasn’t allowed inside, and I was terrified. The nurse held my head down on the examination bed while the doctor injected a local anesthetic before stitching the wound. The injection hurt terribly.

I associated that experience with the fear of needles and injections that followed me through much of my life, even after the incident itself had faded into the fog of time.

Then my therapist, Tina, asked me to let another memory or image come to mind without forcing it.

I saw myself — or, at least, someone whose perspective I seemed to inhabit — standing on a dirt road, looking out over a beautiful meadow. The grass was a luminous lime green, a shade words cannot adequately describe. Flowers bloomed in every colour of the rainbow beneath a turquoise sky.

It was deeply peaceful.

Tina asked whether the place seemed familiar. I said no.

She asked what I was doing.

Nothing. I was simply standing there, admiring the beauty of the seemingly endless meadow.

She asked me to go further back.

Now I was standing on another dirt road, outside a tower built of rough-hewn stone. Again, I was simply looking.

Soon afterward, the scene changed. I seemed to be inside someone else’s body, experiencing what felt like a reflection of a past event.

I was kneeling in a dungeon. Both wrists were shackled above my head, held in iron cuffs attached to heavy chains secured to the stone wall.

I felt frightened and trapped. A tight knot of anxiety gripped my chest — a feeling painfully familiar from my current life.

I longed to escape. To be free.

I was terrified of what was coming. I didn’t want to be there.

Tina asked me to go further back.

I saw myself fighting with another man in a pub. We were pushing each other near the bar. I pulled out a knife and stabbed him.

As he lay dying on the floor, I stared at him, unable to believe what had happened in a single instant.

I was in shock.

My life was over.

Tina asked me to go back again.

This time, I was with a young woman who was crying. She had been hurt, but I couldn’t see what had happened or who had hurt her.

Then the scene shifted once more.

I was standing before a magistrate, being sentenced to death by hanging.

As the noose was placed around my neck, my entire body trembled with fear and adrenaline.

The trapdoor opened.

And then I was back on the dirt road beside that beautiful meadow.

I felt as though I had crossed a threshold between lives. In that moment, I understood the meadow as a place of transition, and the emotions from the life I had witnessed seemed connected to those I carried in this one.

When I came out of hypnosis, I was deeply troubled. The images and feelings stayed with me for several days.

Yet alongside that distress came a sense of recognition. I felt that I finally had a way to understand some of the fear, anxiety, and longing for freedom that had followed me throughout my life. I always looked to escape, to be free. I still do now, but I know why.

I came away believing that I had been carrying those traumatic memories in my consciousness, and that their emotional weight had shaped many of my choices.

Beyond Wishful Thinking: How to Use the “TEAM” Framework to Turn Thoughts Into Reality

We have all heard the age-old New Age maxim: “Thoughts are things.” It sounds beautiful on a vision board, but for many, it breeds frustration. You spend weeks visualizing a better career, a healthier relationship, or financial freedom, only to look around and find your reality completely unchanged.

The missing link isn’t the concept itself — it is the execution. Manifestation is not a magical event; it is a systematic process. True manifestation “requires a combination of positivity and action, which can only be facilitated by shifting your own perspective” The truth about manifestation | Practical Growth — Medium.

To bridge the gap between abstract imagination and tangible reality, you need an operational system. That is where the TEAM Framework of Manifestation comes in.

By treating your mind, body, and actions as a unified team, you can reliably translate mental blueprints into physical form. Here is how the four-part formula works.

🧠 T — Thoughts: The Initial Mindset and Vision

Every physical structure on Earth — from the smartphone in your hand to the skyscraper down the street — began as a single, invisible thought. Thoughts are the architectural blueprints of your reality.

If your thoughts are vague, your outcomes will be chaotic. Manifestation requires high-definition clarity. When you define exactly what you want and why, you program your “Reticular Activating System to highlight relevant cues and opportunities that align with your goal.” The Neuroscience of Manifestation: What the Brain Really Does

• The Trap: Most people spend their mental energy focusing on what they don’t want. They obsess over debts, past failures, or worst-case scenarios.
• The Framework Shift: You must intentionally direct your conscious mindset toward a specific vision. Crafting a hyper-clear mental image primes your brain to spot paths that a negative mindset would completely miss.

✨ E — Emotions & Energy: The Emotional Drive and Vibrational Alignment

A blueprint is useless without an engine to power it. In the TEAM framework, your Emotions generate the Energy that brings your thoughts to life.

If you think about a promotion but constantly feel insecure, anxious, or undeserving, your dominant energetic state is lack. Because “manifesting happens on the level of energy,” how to manifest: the do’s and don’ts of manifesting — Gabby Bernstein you cannot attract a high-level outcome from a low-level emotional state.

• The Trap: Thinking positively while feeling deeply stressed or cynical.
• The Framework Shift: You must emotionally align with your goal before it physically arrives. To manifest effectively, you have to “feel emotions and identify with our dreams.” MANIFESTATION AND VISUALIZATION. THE PRACTICAL GUIDE — Medium Cultivate the relief, pride, or gratitude of the completed goal right now. This emotional alignment transforms passive wishing into an unstoppable internal drive.

🎬 A — Actions: The Physical Execution and Steps Taken

This is where traditional law-of-attraction teachings often fall short, and where the TEAM framework demands accountability. Manifestation is not a substitute for work; it is the catalyst for inspired action.

You cannot sit on the couch, think happy thoughts, and expect life to transform. The universe rewards motion.

• The Trap: Relying strictly on “cosmic delivery” without getting your hands dirty.
• The Framework Shift: When your thoughts are clear and your energy is high, the actions you need to take will no longer feel like grinding chores. They become obvious, urgent next steps. You write the application, make the cold call, or step into the gym because your internal momentum demands it.

🎯 M — Manifestation: The Tangible Results Achieved

Manifestation is not the start of the journey; it is the inevitable end product. The outcomes you experience in your physical world are the direct, measurable output of your mental, emotional, and physical alignment.

Framework ComponentRole in the ProcessPhysical EquivalentThoughtsThe BlueprintArchitectural SketchEmotions / EnergyThe FuelThe Engine & ElectricityActionsThe ConstructionLaying the BricksManifestationThe ResultThe Finished Building

If you dislike the results you are currently getting, you do not try to fix the manifestation itself. Instead, you look backward up the chain. Did the breakdown happen because your thoughts lacked clarity? Did your energy drop into fear? Or did you skip the necessary action steps?

➡️ The Bottom Line

You are always manifesting your reality, whether you realize it or not. Every circumstance you experience is a reflection of how your mind, emotions, and habits work together.

Stop treating manifestation like a lottery ticket. Treat it like a TEAM sport. When you align your Thoughts, ignite your Emotions, and execute your Actions, the final Manifestation handles itself.

Most client portal problems do not start with software.

They start with scattered communication.

Project updates are buried in email. Files are spread across Google Drive, Dropbox, or SharePoint. Approvals happen in Slack, comments, or long email threads. Invoices live in accounting software. Access permissions are rarely reviewed until something goes wrong.

For agencies, consultants, freelancers, creative studios, and professional service teams, this creates a familiar problem:

Clients want visibility, but the team does not have a clear system for providing it.

That is why I created the Client Portal Readiness Scorecard.

It is a simple Excel and Google Sheets tool that helps service businesses audit whether they are ready to launch a client portal.

Why Client Portal Readiness Matters

A client portal sounds simple.

Give the client one place to find project status, files, approvals, invoices, contacts, and next actions.

But if the internal process is messy, a portal can make the problem more visible instead of solving it.

Before launching a client portal, teams should answer a few important questions:

• Can clients find the latest project status without asking?
• Are shared files organized and clearly labeled?
• Are approval requests tracked with owners and due dates?
• Can clients see invoice status and payment links?
• Do you know who has access to client information?
• Is there a repeatable process for creating and maintaining each portal?

If the answer is no, the business may not need more software yet.

It may need a better operating system.

What Is The Client Portal Readiness Scorecard?

The Client Portal Readiness Scorecard is a 10-minute audit for service teams that want better client visibility.

It helps you score your current client operations across six key areas:

1. Project status visibility
2. Shared files and resources
3. Client approvals
4. Invoice visibility
5. Portal access and security
6. Repeatable portal operations

Each section includes practical questions that reveal whether your client-facing process is clear, consistent, and safe to share.

At the end, the scorecard gives you:

• Total readiness score
• Readiness percentage
• Maturity stage
• Risk level
• Weakest category
• Recommended next step
• Action plan

It works in both Excel and Google Sheets.

Who This Scorecard Is For

This tool is built for:

• Agencies
• Consultants
• Freelancers
• Creative studios
• Implementation teams
• Professional service firms
• Client success teams
• Operations managers
• Fractional service providers

If your business manages client work, client files, client approvals, or client invoices, this scorecard can help you find the weak spots before they become client experience problems.

The Six Areas Every Client Portal Needs

A strong client portal is not just a page with links.

It needs structure.

1. Project Status Visibility

Clients should not have to ask, “Where are we at?”

A good client portal shows the current project status, next milestone, owner, due date, and whether the client needs to take action.

Without this, the portal becomes another place clients ignore.

2. Shared Files and Resources

Files are one of the biggest sources of confusion in client work.

The scorecard checks whether your team has a clear file index, client-facing file labels, version context, and permission checks.

This matters because clients often approve, review, or act on the wrong file when file systems are unclear.

3. Client Approvals

Approvals slow down projects when ownership is vague.

A reliable client portal should show:

• What needs approval
• Who needs to approve it
• When it is due
• What file or deliverable is attached
• Whether the decision has been recorded

The scorecard helps you see whether your approval process is ready for client visibility.

4. Invoice Visibility

Invoices should not feel disconnected from the rest of the client relationship.

The scorecard checks whether clients can see invoice numbers, due dates, statuses, payment links, and related projects.

This does not mean your portal replaces QuickBooks, Xero, Stripe, or FreshBooks.

It means the client has a clear view of what matters.

5. Portal Access and Security

Client portals create risk if access is not managed.

The scorecard asks whether you track:

• Who has access
• What level of access they have
• When they were invited
• When access was last reviewed
• Whether access was removed after completion

This is especially important for agencies and consultants managing multiple clients.

6. Repeatable Portal Operations

A client portal should not be rebuilt from scratch every time.

The best teams use templates, checklists, and maintenance routines.

The scorecard checks whether your business has a repeatable process for launching, updating, reviewing, and archiving client portals.

Why Use A Scorecard Before Building A Portal?

Because a client portal is only as good as the process behind it.

If status updates are stale, files are disorganized, approvals are unclear, and access is unmanaged, the portal will not solve the problem.

It will simply put the mess in one place.

A readiness scorecard helps you slow down and ask:

“What needs to be true before clients can rely on this?”

That question can save hours of cleanup later.

What Happens After You Take The Scorecard?

Your result will fall into one of four maturity stages:

Not Ready

Your client portal process needs foundational work before clients should rely on it.

Start with status updates, file organization, and approval ownership.

Foundational

You have some pieces in place, but they are not yet consistent.

This is the stage where templates, owners, and a weekly maintenance rhythm make the biggest difference.

Portal Ready

You are ready for a structured client portal system.

At this stage, the next step is to move from scattered trackers into a complete client portal operating system.

Scale Ready

Your process is strong enough to support automation, reminders, integrations, and possibly branded login.

This is where tools like Softr, Super, Memberstack, SharePoint, Notion, or a custom portal can make sense.

The Next Step: Full Client Portal Generator

The Scorecard tells you where the gaps are.

The Client Portal Generator helps you build the system.

The full Client Portal Generator includes structured areas for:

• Clients
• Client portals
• Projects
• Files
• Approvals
• Invoices
• Contacts
• Status updates
• Access logs
• Portal setup checklists
• Governance notes
• Client-facing portal preview

If the scorecard shows that your team is Foundational or Portal Ready, the full generator is the logical next step.

It turns the audit into an operating system.

Why This Matters For Client Experience

Clients do not need to see everything.

They need to see the right things clearly.

A strong client portal helps clients answer:

• What is happening?
• What do I need to do?
• Where are the files?
• What needs approval?
• What invoices are open?
• Who should I contact?
• What happens next?

When those answers are easy to find, clients feel more confident.

When clients feel more confident, your team spends less time chasing updates, resending links, and explaining the same things repeatedly.

Download The Client Portal Readiness Scorecard

If you manage client work and want a cleaner, more professional client experience, start with the scorecard.

It is a simple way to find out whether your business is ready for a client portal before investing in more tools, templates, or software.

Download the Client Portal Readiness Scorecard for Excel and Google Sheets here:
https://3531209778980.gumroad.com/l/pvmzyw

Use it to score your current process, identify your weakest area, and create a practical action plan.

Then, when you are ready to build the complete system, use the Client Portal Generator to turn your scorecard results into a working client portal operating system.

Start with the scorecard. Build the portal with confidence.

Build the data pipeline.
Define the KPIs.
Create the charts.
Publish the dashboard.

Once that work is done, the organization is supposed to become “data-driven.”

But in practice, that is often where the real questions begin.

A dashboard can tell a sales director that revenue is down 8%. It can show that customer churn increased, that conversion dropped, or that one region is underperforming.

What it usually cannot tell them is what to do next.

That is the gap modern analytics needs to close.

Dashboards answer an important question

At their best, dashboards answer:

What happened?

That matters. Reliable reporting is still essential.

A company needs to know whether sales are rising or falling. It needs to track margins, customer growth, operational performance, marketing results, inventory, service levels, and dozens of other measures.

Dashboards make those numbers visible.

They provide a shared view of the business. They help people monitor performance. They reduce the time spent gathering figures from spreadsheets, emails, and disconnected systems.

None of that is going away.

The problem is that visibility alone is not enough.

Once someone sees a problem on a dashboard, the next question is rarely, “Can you make the chart bigger?”

The next questions are usually much harder.

Why did this happen?

Is the change temporary, or is it part of a larger trend?

Which customers, products, channels, or regions contributed most to the result?

What is likely to happen next?

What action should we take?

And after we act, how will we know whether it worked?

A dashboard may reveal the symptom. It does not automatically provide the diagnosis.

More reporting does not always create more understanding

One of the most common reactions to a business problem is to create another report.

Revenue is down, so someone asks for a regional report.

Then they ask for a product report.

Then a customer-segment report.

Then a weekly comparison.

Soon there are five dashboards, twelve filters, and several different versions of the truth.

The organization has more information, but not necessarily more understanding.

This happens because reporting and analysis are not the same thing.

Reporting describes the current or historical state of the business.

Analysis tries to explain relationships, causes, patterns, and likely outcomes.

A dashboard can show that churn increased. A diagnostic analysis looks at whether churn increased among new customers, long-term customers, specific products, certain geographic regions, or customers who experienced service problems.

Those are different levels of work.

The dashboard tells us where to look.

The analysis helps us understand what we are seeing.

Modern analytics must answer a sequence of questions

A more useful way to think about analytics is as a progression.

What happened?

This is descriptive analytics.

It includes dashboards, reports, scorecards, trend lines, summaries, and standard KPIs.

Examples:

• Revenue declined by 8%.
• Conversion fell from 4.2% to 3.7%.
• Customer churn increased during the last quarter.
• Inventory levels dropped below the target range.

This is the foundation. Without trustworthy descriptive analytics, everything else becomes unreliable.

Why did it happen?

This is diagnostic analytics.

It looks for the factors behind a change.

Examples:

• Revenue fell because one high-value customer segment reduced its order frequency.
• Conversion dropped mainly on mobile devices after a website update.
• Churn increased among customers who contacted support more than twice.
• Inventory shortages were concentrated in a small number of fast-moving products.

This is where techniques such as variance analysis, contribution analysis, segmentation, cohort analysis, funnel analysis, correlation, and drill-down become useful.

What may happen next?

This is predictive analytics.

It uses historical patterns to estimate future outcomes.

Examples:

• Which customers are most likely to churn?
• What will demand look like next month?
• Which leads are most likely to convert?
• Which products are likely to run out of stock?
• How much revenue can we reasonably expect next quarter?

Prediction does not remove uncertainty. It helps us manage it.

Even a simple forecast can improve planning if it is more useful than relying on intuition alone.

What should we do?

This is prescriptive analytics.

It moves from prediction to recommendation.

Examples:

• Which customers should receive a retention offer?
• Which campaign should receive more budget?
• Which product should be recommended next?
• How should inventory be reallocated?
• Which operational issue should be addressed first?

This is the point where analytics starts to influence action directly.

Did the action work?

This is the question many analytics systems forget.

A recommendation is only useful if its outcome is measured.

Did the retention campaign reduce churn?

Did the pricing change increase profit or only increase volume?

Did the inventory decision reduce stockouts?

Did the recommended action improve the customer experience?

Without this feedback loop, organizations repeat the same decisions without learning whether those decisions were effective.

A mature analytics system should not stop at recommendation. It should capture the decision, track the outcome, compare the result to the objective, and feed that learning back into future decisions.

The real goal is not more dashboards

The goal is better decisions.

That sounds obvious, but it changes how analytics systems should be designed.

Instead of starting with, “What dashboard should we build?” we should often start with:

• What decision needs to be made?
• What information supports that decision?
• What patterns would explain the situation?
• What future outcomes matter?
• What actions are available?
• What constraints or risks must be considered?
• How will success be measured?

This produces a very different type of analytics environment.

The dashboard still has a role, but it becomes one component in a larger system.

That system may include:

• trusted KPI definitions
• diagnostic SQL patterns
• forecasting models
• recommendation logic
• business rules
• decision workflows
• alerts
• human review
• outcome tracking
• feedback loops

In other words, the dashboard becomes part of the decision process rather than the end of it.

Why reusable patterns matter

Many analytics teams repeatedly solve the same kinds of problems.

They calculate revenue growth.

They compare actuals to targets.

They analyze funnels.

They measure retention.

They segment customers.

They investigate unusual changes.

They build simple forecasts.

They rank opportunities.

They define recommendation rules.

They evaluate outcomes.

These problems often appear in different industries and different systems, but the underlying structures are similar.

That is why reusable analytics patterns are so valuable.

A pattern is not simply a saved SQL query. It is a repeatable approach to a recurring business problem.

A strong pattern may include:

• the business question
• the assumptions
• the required data
• the SQL logic
• the interpretation
• common variations
• limitations
• implementation guidance
• follow-up actions

Patterns help teams avoid reinventing the same work.

They also make analytics easier to teach, review, govern, and improve.

Instead of relying on isolated expertise, the organization starts building a shared analytical language.

From business intelligence to decision intelligence

Business intelligence helped organizations see what was happening.

The next step is decision intelligence: connecting data, analytics, predictions, recommendations, business rules, and human judgment to the decisions an organization actually makes.

This does not mean handing every decision to an algorithm.

In many cases, the best approach is a partnership between systems and people.

The system can identify patterns, surface risks, compare scenarios, and recommend actions.

The human can apply context, experience, ethics, and judgment.

That combination is more powerful than either side alone.

The future of analytics is not simply automated decision-making.

It is better-supported decision-making.

Dashboards are still useful

The argument is not that dashboards should disappear.

They remain one of the most effective ways to communicate performance.

The problem is expecting them to do a job they were never designed to do.

A dashboard is excellent at summarizing information.

It is less effective at explaining complex causes, forecasting uncertain outcomes, recommending actions, or learning from decisions.

Those capabilities require a broader analytics architecture.

The organizations that make the greatest progress will not necessarily be the ones with the most dashboards.

They will be the ones that connect reporting to diagnosis, diagnosis to prediction, prediction to action, and action to learning.

That is how analytics becomes part of the operating system of the business.

Not just a collection of charts.

Not just a monthly reporting process.

A practical system for making better decisions.

This article is part of the thinking behind my upcoming Leanpub book, Analytics Patterns for Intelligent Decision Systems, where I organize analytics, SQL, forecasting, recommendations, and decisioning into reusable patterns.

Read more about the book on Leanpub:

https://leanpub.com/analyticspatternsforintelligentdecisionsystems

How to design Node.js applications that reference optional modules, install or load them only when needed, and keep dependency governance under control.

Most Node.js applications treat dependencies as a fixed up-front cost.

If a feature might need a package, we add it to package.json, install it during setup, and load it when the application starts or when a file is imported. That works well for small projects, but it becomes painful as applications grow.

Over time, the dependency tree becomes heavier. Install times increase. Cold starts slow down. Optional features quietly become mandatory. Packages used by one rarely visited workflow are installed for every user, every environment, and every deployment.

There is a better pattern:

Application code should depend on capabilities, not packages.

Instead of importing a package directly, a feature should request a capability. The application can then decide whether that capability is already available, whether it needs to be installed, whether it is allowed, and how it should be injected.

This is the idea behind On-Demand Module Injection for Node.js.

The Problem With Standard Node.js Dependency Loading

In a standard Node.js application, we usually do something like this:

import sharp from "sharp";

That line looks innocent, but it creates a hard dependency.

The application now assumes:

• sharp is installed
• the correct native binaries are available
• the module is needed in this environment
• the feature using it is part of the current runtime path
• every deployment should carry that dependency

For core functionality, that is fine. For optional functionality, it is often wasteful.

Examples of optional dependencies include:

• image processing
• PDF generation
• Markdown rendering
• YAML parsing
• AI provider SDKs
• database adapters
• cloud storage adapters
• payment provider integrations
• analytics exporters

Many applications have features that only some users or environments need. Yet the default dependency model installs everything for everyone.

The Architecture: On-Demand Module Injection

The architecture is simple:

Application code
   ↓
Feature asks for a capability
   ↓
Module registry checks if a provider is available
   ↓
If installed: import it
If missing: install or download it
   ↓
Validate the module
   ↓
Inject it into the feature
   ↓
Cache it for later use

The feature does not say:

import marked from "marked";

It says:

const markdown = await modules.resolve("markdown");

That small change creates a powerful boundary.

The application is no longer tightly coupled to a package. It is coupled to a capability. The registry owns the package-level details.

Core Components

This architecture has five main parts.

1. A Module Manifest

The manifest declares which capabilities the application supports and which npm packages provide them.

{
  "modules": {
    "markdown": {
      "package": "marked",
      "version": "^12.0.0",
      "entry": "marked",
      "optional": true
    },
    "yaml": {
      "package": "yaml",
      "version": "^2.4.0",
      "entry": "yaml",
      "optional": true
    }
  }
}

This gives the application a controlled vocabulary.

Features ask for markdown or yaml. They do not decide which package to install. They do not run arbitrary imports. They do not pull package names from user input.

That matters for maintainability, and it matters even more for security.

2. A Module Registry

The registry is the central resolver.

It is responsible for:

• reading the manifest
• validating requested capabilities
• checking whether a package is allowed
• determining whether the package is installed
• installing it when needed
• dynamically importing it
• caching the result

Here is a minimal reference implementation:

import fs from "node:fs/promises";
import { createRequire } from "node:module";
import { execFile } from "node:child_process";
import { promisify } from "node:util";

const execFileAsync = promisify(execFile);
const require = createRequire(import.meta.url);

export class ModuleRegistry {
  constructor({ manifestPath, allowedModules }) {
    this.manifestPath = manifestPath;
    this.allowedModules = new Set(allowedModules);
    this.cache = new Map();
  }

  async loadManifest() {
    const raw = await fs.readFile(this.manifestPath, "utf8");
    return JSON.parse(raw).modules;
  }

  async resolve(name) {
    if (this.cache.has(name)) {
      return this.cache.get(name);
    }

    const modules = await this.loadManifest();
    const config = modules[name];

    if (!config) {
      throw new Error(`Unknown module capability: ${name}`);
    }

    if (!this.allowedModules.has(config.package)) {
      throw new Error(`Package is not allowed: ${config.package}`);
    }

    await this.ensureInstalled(config.package, config.version);

    const imported = await import(config.entry);
    const moduleValue = imported.default ?? imported;

    this.cache.set(name, moduleValue);
    return moduleValue;
  }

  async ensureInstalled(packageName, version) {
    try {
      require.resolve(packageName);
      return;
    } catch {
      const spec = `${packageName}@${version}`;
      await execFileAsync("npm", ["install", spec, "--save-optional"], {
        stdio: "pipe"
      });
    }
  }
}

This is intentionally small. A production implementation should add stronger controls around installation, integrity, logging, and environment behavior.

3. An Injection Container

The registry can be exposed through a small container:

import { ModuleRegistry } from "./moduleRegistry.js";

export const modules = new ModuleRegistry({
  manifestPath: new URL("./module-manifest.json", import.meta.url),
  allowedModules: ["marked", "yaml", "pdf-lib"]
});

Feature code can now depend on the container instead of directly depending on npm packages:

import { modules } from "./container.js";

export async function renderMarkdown(input) {
  const marked = await modules.resolve("markdown");
  return marked.parse(input);
}

The feature asks for Markdown rendering. The registry decides how to satisfy that request.

4. A Lazy Installer

The installer only runs when a requested capability is missing.

For example:

npm install marked@^12.0.0 --save-optional

This can be useful in development, plugin systems, internal tools, and modular platforms where not every capability is required at startup.

However, runtime installation should be treated carefully in production. For many systems, production should use a stricter mode:

Development:
  Missing module can be installed automatically

Production:
  Missing module fails clearly, or is pulled from a controlled internal package cache

In other words, lazy loading is almost always useful. Lazy installing is environment-dependent.

5. A Security Gate

The security model is not optional.

If the application can install packages dynamically, it must never install arbitrary package names from untrusted input.

At minimum, the system should enforce:

• declared capabilities only
• an explicit package allowlist
• pinned major versions
• trusted registry configuration
• lockfile integrity
• audit checks in CI
• installation logs
• production controls
• offline mode

The manifest is not just configuration. It is a policy boundary.

How This Fits Into package.json

For applications, optionalDependencies is a good fit:

{
  "dependencies": {
    "core-app-runtime": "^1.0.0"
  },
  "optionalDependencies": {
    "marked": "^12.0.0",
    "yaml": "^2.4.0"
  }
}

For libraries, optional peer dependencies are usually better:

{
  "peerDependencies": {
    "marked": "^12.0.0",
    "yaml": "^2.4.0"
  },
  "peerDependenciesMeta": {
    "marked": {
      "optional": true
    },
    "yaml": {
      "optional": true
    }
  }
}

The distinction matters.

Applications own their runtime. Libraries should avoid surprising the host application by installing or bundling more than necessary.

Production Structure

A production version should split responsibilities cleanly:

/core
  moduleRegistry
  manifestLoader
  installer
  integrityPolicy
  moduleCache

/features
  markdownFeature
  pdfFeature
  yamlFeature

/module-manifest.json

You can also add a small CLI:

node tools/modules.js list
node tools/modules.js install markdown
node tools/modules.js prefetch

The prefetch command is especially useful for production deployments. It allows the deployment process to install all required optional modules ahead of time while preserving the capability-based architecture at runtime.

Rollout Timeline

This can be implemented gradually.

Week 1: Prototype

Define the manifest format, build the first version of the registry, support dynamic imports, and test two optional capabilities such as Markdown and YAML.

Week 2: Feature Integration

Refactor selected features to request capabilities instead of importing packages directly. Add cache behavior and clear errors for missing, failed, or disallowed modules.

Week 3: Security and Deployment Controls

Add lockfile verification, production mode, offline mode, installation logs, and internal registry support if needed.

Week 4: Tooling and Documentation

Add CLI commands, document the manifest schema, explain how teams add optional modules, and build a small reference app.

Weeks 5–6: Hardening

Add retries, install timeouts, package integrity checks, CI policy checks, and usage telemetry.

When This Pattern Is Worth It

This architecture is useful when:

• your application has optional features
• install size matters
• cold start performance matters
• different deployments need different capabilities
• you support plugins or extensions
• you want stronger dependency governance
• you need to avoid forcing every package onto every user

It may be unnecessary for small applications with a stable, minimal dependency set.

Like most architecture patterns, the value appears when the boundary solves a real problem.

Final Thought

Node.js makes it easy to install and import packages. That is one of its great strengths.

But as applications grow, direct imports can quietly turn optional capabilities into permanent obligations.

On-demand module injection gives us a cleaner model:

Feature code asks for what it needs.
The platform decides how to provide it.

That separation makes Node.js applications more modular, more adaptable, and easier to govern at scale.

The future of large Node.js systems should not be every module installed everywhere, all the time.

It should be capability-first, policy-controlled, and loaded only when needed.

Stop Installing Every Node.js Module Up Front: A Better Architecture for On-Demand Dependencies was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

Most people think of Agile and Kanban as tools for software teams. But the truth is far more interesting: Agile is a philosophy for navigating complexity, and life is nothing if not complex. When you apply Agile principles to your personal and professional growth, you stop drifting and start iterating your way toward the person you want to become.

This isn’t about rigid systems or productivity hacks. It’s about building a life that adapts as fast as you do.

Let’s break it down.

🌱 Why Agile Works for Personal Growth

Agile is built on a few core ideas:

• Respond to change over following a fixed plan
• Deliver value in small increments
• Reflect and adjust regularly
• Limit work in progress to reduce overwhelm

These principles map beautifully onto personal development. Life changes fast. Goals evolve. Priorities shift. Agile gives you a structure that bends instead of breaks.

Here’s a polished, Medium‑ready article that shows how to apply Agile + Kanban to personal and professional growth — written in a clean, engaging, human tone that fits perfectly with your style. I’ll also weave in visuals using image tags to make the piece feel more like a modern Medium article.

Agile Personal Growth: How Kanban Can Transform Your Life and Career

Most people think of Agile and Kanban as tools for software teams. But the truth is far more interesting: Agile is a philosophy for navigating complexity, and life is nothing if not complex. When you apply Agile principles to your personal and professional growth, you stop drifting and start iterating your way toward the person you want to become.

This isn’t about rigid systems or productivity hacks. It’s about building a life that adapts as fast as you do.

Let’s break it down.

🌱 Why Agile Works for Personal Growth

Agile is built on a few core ideas:

• Respond to change over following a fixed plan
• Deliver value in small increments
• Reflect and adjust regularly
• Limit work in progress to reduce overwhelm

These principles map beautifully onto personal development. Life changes fast. Goals evolve. Priorities shift. Agile gives you a structure that bends instead of breaks.

🧭 Step 1: Create Your Personal Backlog

Your backlog is simply a living list of things you want to improve, learn, or accomplish. Think of it as your growth inventory.

Break it into categories:

• Health & Fitness
• Career & Skills
• Relationships
• Finances
• Creativity
• Inner Work / Mindset

Then add items like:

• “Improve my cycling endurance”
• “Publish two Medium articles per week”
• “Learn advanced data modeling”
• “Set healthier boundaries with family”
• “Meditate 10 minutes daily”

The key: don’t judge or prioritize yet. Just capture.

🧩 Step 2: Use Kanban to Visualize Your Growth

Kanban is the engine that makes Agile personal development work.

Create a simple board with three columns:

To Do → Doing → Done

That’s it.

Move items from your backlog into To Do, but only a few at a time. This is where most people sabotage themselves — they overload.

Kanban forces you to limit Work In Progress (WIP) so you can focus deeply and finish what matters.

⚙️ Step 3: Set WIP Limits (Your Secret Weapon)

WIP limits prevent overwhelm and context switching.

Examples:

• Personal WIP limit: 2 active habits
• Professional WIP limit: 1 major skill + 1 project
• Health WIP limit: 1 routine change at a time

This is where transformation happens. You stop trying to “fix everything at once” and start making meaningful, compounding progress.

🔄 Step 4: Run Weekly Sprints

A sprint is simply a one‑week focus cycle.

Each week:

1. Pull a few items from To Do into Doing
2. Commit to small, achievable outcomes
3. Track progress visually
4. Move completed items to Done

This creates momentum and a sense of accomplishment — something most personal growth systems lack.

🪞 Step 5: Hold a Weekly Retrospective

This is the heart of Agile personal growth.

Ask yourself:

• What worked well this week
• What didn’t
• What did I learn about myself
• What will I change next week

Retrospectives turn experience into wisdom. Without them, you’re just doing tasks. With them, you’re evolving.

🚀 Step 6: Apply Continuous Improvement

Agile isn’t about perfection. It’s about iteration.

Every week, you refine:

• Your backlog
• Your priorities
• Your habits
• Your systems
• Your mindset

This is how you grow without burning out.

🧠 Real-Life Example: Professional Growth with Agile

Let’s say you want to level up as a cloud data engineer.

Your backlog might include:

• Learn dbt
• Master Monte Carlo simulations
• Build a modular reporting tool
• Publish a technical course
• Improve architecture documentation skills

Your weekly sprint might include:

• Watch 2 dbt tutorials
• Build a small prototype in Streamlit
• Draft one section of your course

Small, consistent, visible progress beats big, inconsistent bursts every time.

💪 Real-Life Example: Personal Growth with Agile

Maybe you’re working on health, boundaries, and emotional autonomy.

Your backlog might include:

• Daily cycling
• High-protein Mediterranean meals
• Meditation practice
• Setting boundaries with family
• Reducing guilt-driven habits

Your sprint might include:

• 3 cycling sessions
• Prep 3 Mediterranean meals
• One boundary-setting conversation
• 5 minutes of meditation daily

Again: small, focused, achievable.

✨ Why This Works

Because Agile mirrors how humans actually grow:

• We learn in cycles
• We improve through reflection
• We thrive with visual clarity
• We succeed when we limit overwhelm
• We evolve when we adapt

Agile isn’t a productivity system. It’s a self-evolution framework.

📝 Conclusion: Build a Life That Iterates

When you combine Agile principles with a Kanban board, you create a personal operating system that is:

• Flexible
• Sustainable
• Motivating
• Data-driven
• Emotionally intelligent

You stop trying to “become your best self” in one giant leap and instead grow through continuous, intentional iteration.

That’s the real power of Agile.

It was sometime in my twenties, in the quiet rolling hills of Quebec’s Eastern Townships, that I stepped into an experience I still struggle to fully explain. The place itself felt removed from ordinary life — a secluded, almost monastic compound tucked between forest and sky. There were no signs, no distractions, no connection to the outside world. Just a small group of seekers, each carrying something unspoken.

From the moment we arrived, it was clear this wasn’t a typical retreat. We were asked to surrender not only our routines, but our identities. Everyone wore long, simple robes — identical, flowing, unadorned. No other clothing. It created a strange equality. No status, no fashion, no external markers. Just bodies and breath and whatever we had come there to face.

We slept in communal rooms. Rows of beds, quiet breathing in the dark. No one spoke much outside of the guided sessions. There was a shared understanding that something deeper was unfolding — something that didn’t need words.

The core of the weekend was rebirthing breathwork. Long, continuous breathing cycles designed to unlock what was buried beneath the surface — emotion, memory, tension, something older than thought. At first, it felt physical. Tingling hands. Tightness in the chest. Then waves — emotional surges rising without warning.

And then, at some point, everything shifted.

I left.

Not physically — my body was still there — but my awareness detached and accelerated into something vast and incomprehensible. I found myself moving rapidly through what I can only describe as a dark void, yet it wasn’t empty. It was filled with points of light — countless, like stars stretching infinitely in all directions.

There was no fear. Just movement. Speed. Expansion.

Then I wasn’t alone.

Two figures appeared — not suddenly, but as if they had always been there, waiting just beyond perception. They were older men, calm, grounded, completely at ease in that space. One of them stood out. Long white — almost silver — hair, and a robe that seemed to carry its own soft luminosity.

He didn’t speak in the usual sense, yet I understood him perfectly.

Everything is ok.

All is forgiven.

There was no judgment in it. No conditions. Just a deep, undeniable sense of acceptance that seemed to dissolve something I didn’t even realize I was carrying.

For a moment — or maybe much longer — I stayed there. Suspended between motion and stillness, between self and something beyond self.

Then I heard voices.

Faint at first. Then urgent.

The guides from the retreat. Calling me back. Repeatedly. Insistently.

It felt like being pulled across dimensions. The vastness collapsed. The movement stopped. The stars disappeared. And suddenly, I was back — breathing heavily, body tense, lying on the floor among others going through their own journeys.

The room felt smaller than before. Denser. Real, but less real at the same time.

No one fully explained what had happened. There were no grand interpretations offered. Just gentle grounding, water, quiet integration.

But something had changed.

Not in a dramatic, outward way — but internally. A shift in perspective. A sense that whatever we think reality is, it’s only a fragment. That there are layers — accessible, but not easily controlled. And that sometimes, when you push far enough inward, you don’t just find yourself…

You find something waiting.

Even now, years later, I can still recall that moment — the stillness of those figures, the certainty in that message, and the feeling of being seen in a way that required nothing in return.

Whether it was psychological, spiritual, or something else entirely, I don’t claim to know.

But it was real.

And it never quite leaves you.

Build Yourself While You Build Software: How Web Development Became My Most Powerful Tool for Personal Growth

I realized that every time I sat down to code, I wasn’t just building software — I was building myself. The habits that make you a great developer are the same habits that make you a stronger, more resilient human being.

Once I saw that connection, everything changed.
Web development stopped being a technical skill and became a personal transformation engine.

Here’s how you can fuse the two and turn your coding journey into a path of real growth.

1. Treat Every Project as a Personal Growth Experiment

Most tutorials teach you what to build.
Few teach you who you become while building it.

I started designing projects around the inner qualities I wanted to strengthen:

• Discipline → ship one tiny feature a day
• Focus → build single‑purpose micro‑apps
• Creativity → design UI components from scratch
• Emotional resilience → debug without spiraling
• Leadership → document code so others can follow

Suddenly, every project became a training ground for my character and my goals.

2. Bring Mindfulness Into Your Workflow

Mindfulness isn’t just meditation.
It’s awareness during action.

Before coding, I take 30 seconds to set an intention.
During coding, I notice frustration or boredom instead of reacting to it.
After coding, I reflect on what emotion drove my decisions.

This simple practice made me a calmer developer — and a calmer person.

3. Build Tools That Support Your Growth

One of the most powerful ways to grow is to build the tools you need.

Some ideas:

• A habit tracker
• A journaling app
• A goal visualization dashboard
• A meditation timer
• A micro‑commit tracker

When you use tools you built yourself, your growth becomes self‑reinforcing. You’re literally coding your future self into existence.

4. Turn Your Learning Journey Into Content

Teaching forces mastery.
Sharing forces clarity.

Write about:

• What you learned
• What frustrated you
• What surprised you
• How debugging mirrors life
• How building systems changed your mindset

Your growth becomes someone else’s roadmap — and your writing becomes a mirror for your own evolution.

5. Apply Developer Principles to Your Life

The parallels are uncanny:

• Version control → reflect without shame
• Refactoring → improve habits without starting over
• Modular design → break big goals into small tasks
• Testing → experiment before committing
• Documentation → journal your patterns

When you live like a developer, growth becomes systematic.

6. Build a Personal Growth Roadmap Like a Dev Roadmap

You already know how to plan a technical roadmap.
Use the same structure for your inner development.

• Quarterly themes
• Weekly sprints
• Daily micro‑commits

You’re not just managing tasks — you’re managing your evolution.

Final Thought

Web development isn’t just a career.
It’s a mirror.
A discipline.
A philosophy.

When you combine personal growth with web development, you don’t just build better software — you build a better version of yourself.

I hope helps you in some way.

© 2026 — Kevin Languedoc

Should we use Snowflake or Databricks?
SQL or Python?
dbt or Airflow?

These are implementation questions.
But successful data systems begin somewhere else entirely:

Design.

The biggest problem in modern data engineering is not tooling, infrastructure, or compute.

It’s lack of design.

The Silent Problem in Data Engineering

Walk into most data teams and you’ll see something familiar.

Pipelines everywhere.

• ETL scripts
• Airflow DAGs
• dbt models
• notebooks
• random SQL queries

Individually they work. But collectively they become a fragile ecosystem.

You start noticing symptoms:

• duplicated business logic
• inconsistent metrics
• unclear table ownership
• pipelines nobody understands
• schemas that evolve randomly

Over time, complexity explodes.

This isn’t a tooling issue.

It’s design debt.

Just like technical debt slows software teams, design debt slows data teams.

Why Data Systems Are Built Backwards

In traditional software engineering, development usually follows a structure:

Design → Model → Implementation

But data engineering often works like this:

Query → Pipeline → Table → Repeat

In other words:

We build first and design later.

Tables appear because someone needed a query.

Pipelines grow because someone needed a dashboard.

Schemas evolve because someone added a column.

Eventually, the architecture becomes an accident.

The Idea of Design-Driven Data Engineering

Design-Driven Data Engineering flips this process.

Instead of starting with code or pipelines, we start with design artifacts.

A design-driven workflow looks like this:

Business Domain
      ↓
Conceptual Model
      ↓
Logical Data Model
      ↓
Schemas and Pipelines
      ↓
Analytics and Applications

Everything in the system traces back to the business domain.

The goal is simple:

Every table, transformation, and metric should exist because it represents a real business concept.

Step 1: Model the Business Domain

Before writing SQL, you need to understand what the system represents. What business or functional problem are to trying to solve?

For example, imagine an e-commerce platform.

The domain might include entities like:

• Customer
• Order
• Product
• Payment

And relationships like:

• Customers place Orders
• Orders contain Products
• Payments belong to Orders

This conceptual model becomes the foundation of the data system.

Instead of starting with tables, we start with meaning.

Step 2: Choose the Right Data Model Pattern

Once the domain is understood, the next step is choosing the right data modeling pattern.

Different systems require different models.

Normalized Models (OLTP)

Best for transactional systems.

Customers
Orders
OrderItems
Products

Pros:

• strong data integrity
• minimal redundancy

Cons:

• complex analytical queries

Dimensional Models (Analytics)

Popular in data warehouses.

FactSales
DimCustomer
DimProduct
DimDate

Pros:

• simple BI queries
• fast analytics

Cons:

• less flexible for evolving schemas

Data Vault (Enterprise Warehouses)

Designed for auditability and historical tracking.

Components:

• Hubs — business entities
• Links — relationships
• Satellites — historical attributes

Pros:

• full lineage
• scalable ingestion

Cons:

• more complexity

Medallion Architecture (Lakehouses)

Modern data platforms often organize pipelines into layers:

Each layer represents progressive refinement of the design.

Step 3: Design the Pipeline (Before Coding It)

A pipeline isn’t just code.

It’s a data flow system.

Instead of writing scripts first, define the stages:

When you visualize the pipeline as a graph, its logic becomes obvious.

Raw Events
    ↓
Clean Orders
    ↓
Enriched Orders
    ↓
Sales Analytics Model

The code then becomes an implementation of the design, not the design itself.

Step 4: Treat Schema as Code

Traditional data teams treat schemas like static artifacts.

Design-driven teams treat them like versioned code.

For example, instead of manually writing SQL DDL:

CREATE TABLE customers (
  id INT PRIMARY KEY,
  email TEXT,
  join_date DATE
);

You might define a model:

class Customer(Table):
    id: int = Column(primary_key=True)
    email: str
    join_date: date

Which generates the SQL automatically.

This approach enables:

• schema versioning
• automated migrations
• CI/CD validation
• design consistency

Step 5: Build Testing Into the Design

Data testing should not be an afterthought.

Every dataset should have design-level expectations.

Examples:

• primary keys must not be null
• order totals must be positive
• timestamps must be valid

Instead of writing tests later, they are defined alongside the model:

expectations:
  - no_nulls(order_id)
  - total_amount >= 0

This makes data quality part of the architecture.

Design Is the Missing Discipline in Data Engineering

The most successful software systems are designed carefully before implementation.

Data systems deserve the same discipline.

When design becomes the center of development:

• schemas become consistent
• pipelines become modular
• governance becomes easier
• teams move faster

Most importantly:

The data system begins to reflect the business itself.

The Future: Design as Code

Modern data engineering is moving toward design-as-code.

Instead of diagrams sitting in documents, designs become:

• YAML domain models
• schema generators
• declarative pipeline definitions
• automated tests
• CI/CD workflows

The design itself becomes executable.

In that world:

Design is no longer documentation.
Design becomes the control plane of the data platform.

Final Thought

Many teams think they have a data engineering problem.

Often, they actually have a design problem.

If we start designing data systems the way software engineers design software, we can build systems that are:

• easier to evolve
• easier to govern
• easier to understand

And far less painful to maintain.

If you’re interested in learning more about design-driven data engineering, I’m wrote a book called Design Driven Data Engineering discussing to principles of design first approach and building a toolkit showing how to turn domain models into schemas, pipelines, and automated tests.

Code examples:
https://github.com/Logikev/DesignDrivenDataEngineering

© 2026 kevin Languedoc

Artificial Intelligence systems rely heavily on data retrieval, feature engineering, and scalable storage. Because of this, SQL databases often become the backbone of many AI pipelines.

However, when teams start building AI systems on top of traditional relational databases, they quickly discover that SQL was never originally designed for AI workloads.

Problems appear when dealing with:

• vector embeddings
• semantic search
• large document retrieval
• real-time feature engineering
• AI hallucinations caused by poor data retrieval

The good news is that modern PostgreSQL capabilities make it possible to solve many of these problems without adopting an entirely new database stack.

In this article, we will explore 10 common pain points when using SQL in AI systems — and how PostgreSQL solves them. All the code for the following PostgreSQL examples are located in the following GitHub repository:

https://github.com/Logikev/sql_ai_pain_points.git

Before developing and testing the following code examples, you will need to install PostgreSQL and create a database if you don’t already have PostgreSQL installed on your computer. Also, I am using Visual Studio Code and the following extensions for development and testing.

ms-ossdata.vscode-pgsql (client from Microsoft)

cweijan.vscode-postgresql-client2 (alternative database client)

From the commandline

If needed, create a database at the postgre=# prompt or use the default. I will be using the default for the examples in this article.

You may want to run a test query to make sure everything is running correctly.

1. Storing and Querying Embeddings

Modern AI systems rely on embeddings, which convert text, images, or documents into high-dimensional vectors.

Traditional SQL databases were not built for vector storage or similarity search.

Without specialized support, teams end up storing vectors as arrays and performing extremely slow comparisons.

Solution: Use the pgvector extension

Download and install the extension from the GitHub repo and follow the installation instructions. (https://github.com/pgvector/pgvector)

You will need to build from source and install it before you can use. The readme provides easy instructions to follow. If you are on Windows, make sure the c++ build tools are installed and you run the command from the native 64 command line shell with admin permissions.

PostgreSQL now supports vector operations through the pgvector extension. You will need to install it first using the following command:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE documents (
id SERIAL PRIMARY KEY,
content TEXT,
embedding VECTOR(3) 
);

To test you can use the following sample data or your own based on the following example. Insert some data:

INSERT INTO documents (content, embedding)
VALUES ('PostgreSQL supports vector search', '[0.0123, 0.9876, 0.4567]');

Perform similarity search:

SELECT id, content
FROM documents
ORDER BY embedding <=> '[0.01, 0.98, 0.45]'
LIMIT 5;

This allows PostgreSQL to function as a vector database for AI retrieval systems. If you want to test on a large dataset, you can generate larger embeddings using a query like this:

INSERT INTO documents (content, embedding)
SELECT
'test document',
array_to_string(
ARRAY(
SELECT random()
FROM generate_series(1,1536)
),
','
)::vector;

2. Semantic Search Becomes Slow at Scale

A naive similarity search scans every row in the table.

When your system contains millions of embeddings, query performance quickly degrades.

Solution: Use Vector Indexes

PostgreSQL supports specialized indexes for vector search.

CREATE INDEX ON documents
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);

For even better performance with large datasets, newer versions support HNSW indexing:

CREATE INDEX ON documents
USING hnsw (embedding vector_cosine_ops);

These indexes dramatically reduce search time and allow PostgreSQL to scale to millions of embeddings.

3. Traditional Text Search Is Not Enough

Modern AI applications — especially retrieval systems used in chatbots and question-answering systems — rely heavily on retrieving the right information from a database before sending it to a language model.

Two main retrieval approaches exist:

• Keyword search
• Semantic search

Both solve different problems, and relying on only one often leads to poor results.

Solution: PostgreSQL Full-Text Search

ALTER TABLE documents
ADD COLUMN tsv tsvector
GENERATED ALWAYS AS (to_tsvector('english', content)) STORED;

CREATE INDEX idx_tsv ON documents USING GIN(tsv);
SELECT id, content
FROM documents
WHERE tsv @@ plainto_tsquery('english', 'vector database');

Full-text search enables powerful keyword-based filtering before semantic ranking.

4. Poor Retrieval Leads to AI Hallucinations

Large Language Models (LLMs) do not inherently know the contents of your databases, internal documents, or proprietary knowledge bases. Instead, they rely on external context retrieved at runtime. This process is commonly implemented through a technique known as Retrieval-Augmented Generation (RAG).

In a RAG pipeline, the system first searches a database or document store for information related to the user’s query. The retrieved text is then inserted into the prompt that is sent to the language model. The model uses this additional context to generate a response that is grounded in real data.

However, the quality of the final answer is only as good as the retrieval step.

If the retrieval system returns irrelevant, incomplete, or poorly ranked documents, the model has no reliable information to work with. When this happens, the model may attempt to fill in the gaps by generating plausible-sounding text based on patterns it learned during training. This behavior is often referred to as hallucination, where the model produces confident answers that are incorrect or unsupported by the underlying data.

Poor retrieval can occur for several reasons, such as:

• irrelevant documents being ranked higher than relevant ones
• missing context due to improper document chunking
• keyword search failing to match semantically related text
• similarity search returning results that are only loosely related

For example, imagine a user asking:

“How does PostgreSQL perform vector similarity search?”

If the retrieval system returns documents about general indexing strategies rather than vector search with pgvector, the model may still attempt to answer the question. Since it lacks the correct context, the response may contain fabricated explanations or outdated information.

Improving retrieval quality is therefore one of the most important steps when building reliable AI systems. Techniques such as semantic search, hybrid search (keyword + vector search), relevance thresholds, and better document chunking can dramatically reduce hallucinations by ensuring that the model receives highly relevant context.

In practice, the most reliable AI systems focus less on making the model “smarter” and more on making the retrieval pipeline more accurate. When the right information is provided, even relatively small language models can generate precise and trustworthy answers.

Solution: Apply a Similarity Threshold

SELECT id,
content,
1 - (embedding <=> '[0.01, 0.98, 0.45]') AS similarity
FROM documents
WHERE 1 - (embedding <=> '[0.01, 0.98, 0.45]') > 0.80
ORDER BY embedding <=> '[0.01, 0.98, 0.45]'
LIMIT 5;

Filtering low-similarity results ensures that the model receives only relevant context.

5. Large Documents Break Retrieval Pipelines

Since Section 4 already contains that explanation and example, Section 5 should focus specifically on the design problem of large documents and chunking, rather than repeating the causes of poor retrieval.

Below is a rewritten Section 5 that avoids repeating Section 4 while keeping the concept clear.

5. Large Documents Can Break Retrieval Pipelines

Another common issue in AI retrieval systems is the way large documents are stored and processed. Many knowledge bases contain long technical documents, reports, manuals, or entire database schemas. When these documents are passed directly into a retrieval pipeline without careful design, they can significantly reduce the accuracy of the system.

Most Retrieval-Augmented Generation (RAG) systems do not search entire documents at once. Instead, documents are split into smaller segments, often called chunks, which are then indexed in a vector database or search system. Each chunk is converted into an embedding so that the system can find passages that are semantically related to a user’s question.

If documents are too large or are divided poorly, important information may be scattered across multiple chunks. When the system retrieves only one of those pieces, the language model receives incomplete context. Even though the correct information exists somewhere in the knowledge base, the model cannot access enough of it to generate a reliable answer.

For example, a technical guide might explain a concept across several paragraphs. If those paragraphs are split into separate chunks, the retrieval system may return only part of the explanation. The model then attempts to answer the question using a fragment of the original document, which can lead to misleading or incomplete responses.

Designing the document structure carefully is therefore essential. Effective retrieval pipelines typically apply several strategies:

• splitting documents into logically coherent chunks
• preserving surrounding context where necessary
• overlapping chunks to avoid losing important information
• storing metadata that helps the system retrieve related sections

These design choices may seem like implementation details, but they often determine whether a RAG system works reliably or produces inconsistent answers.

Well-designed retrieval pipelines treat documents not simply as files, but as structured sources of knowledge that must be prepared specifically for machine retrieval. When documents are organized and chunked thoughtfully, the language model receives richer context and can produce more accurate responses.

Example schema for document chunking

CREATE TABLE document_chunks (
id SERIAL PRIMARY KEY,
document_id INT,
chunk_index INT,
content TEXT,
embedding VECTOR(1536)
);

To test the addition, you can perform this retrieval query after inserting some sample data using this example INSERT:

INSERT INTO document_chunks (document_id, chunk_index, content, embedding)
SELECT d, c, 'Sample text for document ' || d || ' chunk ' || c,
(
SELECT ('[' || string_agg(random()::text, ',') || ']')::vector
FROM generate_series(1,1536)
)
FROM generate_series(1,3) d
CROSS JOIN generate_series(1,1536) c;

Here is the sample SELECT query and output:

SELECT document_id,
array_agg(content ORDER BY chunk_index)
FROM document_chunks
WHERE document_id IN (
SELECT document_id
FROM document_chunks
ORDER BY embedding <=> (
SELECT ('[' || string_agg(random()::text, ',') || ']')::vector
FROM generate_series(1,1536)
)
LIMIT 3
)
GROUP BY document_id;

Chunking dramatically improves retrieval quality in RAG systems.

6. Feature Engineering Queries Become Complex

Machine learning models rarely operate directly on raw transactional data. Instead, they rely on engineered features that summarize user behavior over time. These behavioral features help models detect patterns, trends, and anomalies that would otherwise be hidden in large datasets.

Common examples include:

• Rolling averages, which capture how metrics change over a recent time window
• Purchase frequency, which measures how often a customer makes purchases
• Time-based patterns, such as seasonal behavior or recent activity trends

These features transform raw event data into signals that a model can use for prediction.

For example, an e-commerce recommendation model might use features such as:

• average purchase value over the last 30 days
• number of purchases in the last week
• time since the last purchase
• total spending over the past year

Similarly, fraud detection models often rely on features like:

• number of transactions in the last hour
• sudden changes in spending patterns
• geographic anomalies in recent transactions

SQL is a powerful tool for generating these kinds of features because relational databases are optimized for aggregations and time-based analysis. Using techniques such as window functions, partitions, and time-based filters, SQL can efficiently compute complex behavioral metrics across millions of records.

However, without careful design, feature engineering queries can quickly become difficult to maintain. As more features are added, SQL scripts often grow into long chains of nested queries, joins, and aggregations. This complexity can introduce several problems:

• queries become harder to read and debug
• performance can degrade as datasets grow
• small changes may require modifying multiple parts of the query
• inconsistencies may appear between training data and real-time inference pipelines

For example, computing multiple rolling metrics for different time windows may lead to large queries with repeated logic and multiple joins. Over time, these queries become difficult for teams to manage, especially when new features must be added frequently.

To address this challenge, many teams adopt structured approaches to feature engineering. These include organizing queries into modular components, using reusable views or feature tables, and relying on SQL features such as window functions to simplify calculations.

When designed carefully, SQL can serve as a powerful foundation for building scalable feature pipelines that feed machine learning models with reliable behavioral signals.

Solution: Window Functions

To demonstrate, create this table and insert some sample data as in the following example:

CREATE TABLE events (
id SERIAL PRIMARY KEY,
user_id INT,
event_type TEXT,
amount NUMERIC(10,2),
event_time TIMESTAMP
);

Next, insert some sample data:

INSERT INTO events (user_id, event_type, amount, event_time) VALUES
(1,'view',0,'2026–01–01 10:00'),
(1,'purchase',120,'2026–01–01 11:00'),
(1,'view',0,'2026–01–02 09:00'),
(1,'purchase',80,'2026–01–03 14:00'),
(1,'view',0,'2026–01–04 15:00'),
(2,'view',0,'2026–01–01 12:00'),
(2,'purchase',200,'2026–01–02 13:00'),
(2,'purchase',150,'2026–01–03 16:00'),
(2,'view',0,'2026–01–04 10:00'),
(3,'view',0,'2026–01–01 09:00'),
(3,'view',0,'2026–01–02 10:00'),
(3,'purchase',50,'2026–01–03 11:00');

This SELECT query calculates:

• total events per user
• purchase count
• rolling spend

SELECT
user_id,
event_time,
event_type,
amount,
COUNT(*) OVER (PARTITION BY user_id) AS total_events,
COUNT(*) FILTER (WHERE event_type='purchase')
OVER (PARTITION BY user_id) AS purchases,
SUM(amount) OVER (
PARTITION BY user_id
ORDER BY event_time
ROWS BETWEEN 30 PRECEDING AND CURRENT ROW
) AS rolling_spend
FROM events
ORDER BY user_id, event_time;

Window functions enable powerful feature engineering directly in SQL.

7. Training Data Leakage

One of the most common ML mistakes is data leakage, where training data accidentally includes future information.

This leads to unrealistic model accuracy and hallucinatons;

Solution: Time-based filtering

SELECT *
FROM events
WHERE event_time < '2025–01–01';

Maintaining strict time boundaries ensures that models train on historically valid data.

8. Overusing JSON for AI Metadata

When AI systems log model activity, they often store additional information such as prompts, model parameters, responses, or configuration settings. Because this information can vary from one request to another, many teams store it in a JSONB column rather than creating many optional table columns.

For example, a logging table might contain structured fields such as the model name or response time, along with a flexible JSONB column for metadata:

Hybrid schema example

CREATE TABLE ai_logs (
id SERIAL PRIMARY KEY,
model_name TEXT,
latency_ms INT,
created_at TIMESTAMP DEFAULT NOW(),
metadata JSONB
);

The metadata column can store different attributes depending on the request, such as:

{
"temperature": 0.7,
"prompt_tokens": 250,
"response_tokens": 120,
"user_id": 42
}

Using an insert like this:

INSERT INTO ai_logs (model_name, latency_ms, metadata)
VALUES (
'gpt-4',
120,
'{
"temperature": 0.7,
"prompt_tokens": 250,
"response_tokens": 120,
"user_id": 42
}'::jsonb
);

While JSONB provides flexibility, querying it efficiently requires proper indexing. Without an index, PostgreSQL would need to scan every row in the table to find matching JSON values, which becomes slow as the table grows.

To solve this, PostgreSQL provides GIN (Generalized Inverted Index) indexes, which are optimized for searching inside composite structures such as arrays and JSON documents.

The following index enables efficient queries on the metadata column:

CREATE INDEX idx_metadata
ON ai_logs
USING GIN (metadata);

A GIN index works by indexing the individual keys and values inside the JSONB document. Instead of storing a single index entry per row, it creates index entries for each key-value pair. This allows PostgreSQL to quickly locate rows that contain specific attributes inside the JSON structure.

For example, once the index is created, queries like the following can execute much faster:

SELECT *
FROM ai_logs
WHERE metadata ->> 'temperature' = '0.7';
Or
SELECT *
FROM ai_logs
WHERE metadata ? 'prompt_tokens';

9. AI Workloads Are Read-Heavy

Retrieval pipelines and embedding searches can generate thousands of read queries.

Single-node databases quickly become a bottleneck.

Solution: Read Replicas

A common architecture:

• Writes → primary database
• Vector retrieval → read replicas

This allows AI systems to scale while keeping operational complexity manageable.

10. SQL Alone Cannot Train Models

SQL is fantastic for pulling and shaping data. It makes feature engineering — like calculating rolling averages, counts, and other behavioral metrics — fast and scalable. But SQL wasn’t built to train machine learning models: it lacks support for iterative optimization, GPU acceleration, and advanced algorithms. That’s why most AI systems use SQL to prepare the data and Python to train, evaluate, and deploy models, combining the strengths of both worlds for reliable, production-ready AI.

Example using Python:

import pandas as pd
import psycopg2
conn = psycopg2.connect("dbname=mydb user=postgres")
df = pd.read_sql("""
SELECT user_id,
COUNT(*) AS total_events,
SUM(amount) AS total_spend
FROM events
GROUP BY user_id
""", conn)

This workflow allows PostgreSQL to handle data preparation while Python handles model training.

Final Thoughts

SQL may not have been designed with AI workloads in mind, but modern PostgreSQL capabilities have transformed it into a powerful foundation for AI systems.

With the right patterns, PostgreSQL can support:

• vector similarity search
• hybrid semantic retrieval
• RAG pipelines
• feature engineering
• scalable AI data pipelines

Instead of introducing an entirely new database layer, many teams can leverage PostgreSQL as both a relational database and a vector store.

For data engineers and AI practitioners alike, learning these patterns can significantly simplify the architecture of modern AI systems.

SQL Meets AI: The Problems Nobody Warns You About was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.