Federated Learning on Snowflake: Training ML Models Across Accounts Without Sharing Data

When a single hospital develops a diabetes prediction model, the amount of available training data is inherently limited. In general, pooling patient data across multiple hospitals increases the dataset size and is expected to improve model accuracy. However, data sharing between healthcare institutions faces significant hurdles — regulatory requirements, patient privacy concerns, and institutional boundaries all stand in the way.

Federated learning offers a solution: hospitals can collaboratively train a model without ever sharing their underlying data. In this article, I’ll show you how to build this entirely with Snowflake native features — no external servers, no S3 buckets, no VPNs required.

We ran a full experiment across 3 Snowflake accounts spanning AWS and Azure with two model types: Logistic Regression (FedAvg) and XGBoost (Federated Forest). Both federated models outperformed every local baseline — and every single client saw improved accuracy on its own data, even with Differential Privacy noise applied.

Full source code: The complete implementation (DDL, Stored Procedures, orchestration scripts, and data pipeline) is available on GitHub: https://github.com/Snowflake-Labs/horizontal-federated-learning

TL;DR

• Federated Learning lets multiple organizations collaboratively train ML models by exchanging only model parameters — never raw data.
• We built an end-to-end HFL system across 3 Snowflake accounts (2 cloud providers, 3 regions), 10 clients, using only Private Listings, Stored Procedures, and Model Registry.
• We tested two aggregation strategies: FedAvg (Logistic Regression) and Federated Forest (XGBoost tree concatenation + pruning). Both achieved the ideal ordering: Federated > DP-Federated > all Local.

The Problem: Data Silos Meet Privacy Regulations

Healthcare data is invaluable for disease prediction. Combining patient records from multiple hospitals would dramatically improve model accuracy — but privacy regulations like HIPAA, GDPR, and institutional data governance policies make direct data sharing between organizations nearly impossible. Even within a single country, the legal and ethical barriers to centralizing patient data from independent institutions are often insurmountable.

This challenge is not unique to healthcare. The same pattern appears across industries — finance (fraud detection), telecommunications (network anomaly detection), and manufacturing (quality prediction) all share a common problem: pooling data would create significant value, but sharing it is not an option.

The traditional approach — “pool all data into one centralized location” — simply isn’t an option when regulatory, competitive, or ethical constraints exist. Federated learning addresses this directly. Each organization trains on its own local data, and only model parameters are exchanged. A central orchestrator aggregates these into a global model that benefits from everyone’s data — without anyone seeing anyone else’s raw records.

What is Horizontal Federated Learning?

In Horizontal FL (HFL), every participant has the same features but different samples. Think of 10 hospitals running the same standardized lab tests on different patient populations across different geographic regions. Each hospital has the same columns in their dataset (blood pressure, BMI, age, smoking history, etc.) but different rows (different patients). This is in contrast to Vertical FL, where participants share the same samples but hold different features — for example, a bank and an insurance company holding different data about the same customers.

The aggregation strategy depends on the model type:

• FedAvg (for linear models like Logistic Regression): Each client’s model weights are averaged, weighted by sample count: w_global = Σ(n_i × w_i) / Σ(n_i). After multiple rounds of local training followed by global aggregation, the model converges to something better than any single client could achieve alone.
• Federated Forest (for tree-based models like XGBoost): Weight averaging doesn’t apply to decision trees. Instead, all clients’ trees are concatenated into a single ensemble and pruned to retain only the most impactful trees. Each client trains 50 trees locally; the orchestrator combines all 500 trees (10 clients × 50) and prunes to the top 100 by prediction impact.

Architecture: 3 Accounts, Zero External Infrastructure

A typical FL deployment requires gRPC servers, cloud storage with cross-account IAM policies, and VPN tunnels. We replaced all of it with Snowflake native features:

Three Snowflake accounts across AWS and Azure exchange model parameters via Private Listings. Raw data never leaves the owner’s account.

Our setup uses 3 Snowflake accounts across 2 cloud providers and 3 regions:

• Account 1 (AWS AP-Northeast-1): Orchestrator — aggregation, evaluation, Model Registry. Holds no training data.
• Account 2 (AWS US-West-2): 5 clients (C1–C5), each with 1,000 training samples from assigned U.S. states.
• Account 3 (Azure Japan East): 5 clients (C6–C10), same structure on a different cloud provider.

The data comes from CDC BRFSS 2024 (Behavioral Risk Factor Surveillance System) — a public health survey with 457,670 records covering 53 U.S. states and territories. We use 15 health and lifestyle features (BMI, general health status, exercise habits, smoking history, prior conditions, etc.) to predict diabetes as a binary classification task.

All 53 states are distributed across the 10 clients via greedy bin packing, ensuring balanced pools of approximately 28,000 rows each, from which each client subsamples just 1,000 rows for training. This geographic Non-IID split is critical — it simulates real-world scenarios where each organization’s patient demographics and health patterns differ by region, creating the kind of data heterogeneity that makes federated learning both challenging and valuable.

Note: In production, each client would ideally have its own Snowflake account. We co-locate 5 clients per account for cost efficiency, but no data is shared between clients within the same account — each client trains on its own isolated table.

The Data Exchange: Private Listing Tables

The key to making HFL work on Snowflake is designing the tables that are shared via Private Listings. Two tables carry the model parameters between accounts:

HFL_CLIENT_UPDATES — shared from each client account to the orchestrator. Contains both noise-free and DP-noised model parameters per client per round:

CREATE TABLE HFL_APP_DB.HFL_LOCAL.HFL_CLIENT_UPDATES (
    ROUND_ID     NUMBER,
    CLIENT_ID    STRING,
    WEIGHTS_RAW  VARIANT,   -- Raw model params (no noise)
    WEIGHTS_DP   VARIANT,   -- DP model params (Gaussian noise applied)
    NUM_SAMPLES  NUMBER,
    IS_BASELINE  BOOLEAN,
    MODEL_TYPE   STRING,    -- 'LOGREG' or 'XGBOOST'
    CREATED_AT   TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);

HFL_GLOBAL_WEIGHTS — shared from the orchestrator to all clients. Contains the aggregated global model per round:

CREATE TABLE HFL_APP_DB.HFL_META.HFL_GLOBAL_WEIGHTS (
    MODEL_ID      STRING,
    ROUND_ID      NUMBER,
    MODEL_VERSION STRING,
    WEIGHTS       VARIANT,   -- Aggregated global model params
    AGG_METHOD    STRING,    -- 'WEIGHTED_AVG' or 'TREE_CONCAT'
    CLIENT_COUNT  NUMBER,
    NOISE_MODE    STRING,    -- 'NONE' or 'DP'
    MODEL_TYPE    STRING,    -- 'LOGREG' or 'XGBOOST'
    CREATED_AT    TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);

The VARIANT column is the key design choice — it stores arbitrarily structured JSON, enabling us to use the same table schema for both Logistic Regression coefficients and base64-encoded XGBoost tree ensembles. Private Listing auto-syncs these tables across accounts with approximately 1-minute latency.

Important: Use TRUNCATE + INSERT for data updates — CREATE OR REPLACE TABLE breaks Private Listing share references.

The HFL Workflow

Each federated round follows six steps, repeating for R=3 rounds.

Round 0 — Initialization: The orchestrator inserts an initial global model into HFL_GLOBAL_WEIGHTS. For LogReg, this is a zero-weight vector; for XGBoost, an empty model with no trees.

Each Round (1, 2, 3):

1. Clients fetch the previous round’s global model from the shared GLOBAL_HFL_DB (mounted via Private Listing)
2. 10 clients train locally — LogReg uses warm_start from global weights; XGBoost trains 50 fresh trees (max_depth=6, eta=0.1)
3. Clients upload both noise-free and DP-noised parameters to HFL_CLIENT_UPDATES as JSON in VARIANT columns
4. Orchestrator polls Private Listing shared databases until all 10 clients’ updates arrive (with timeout/retry logic)
5. Orchestrator aggregates via FedAvg (LogReg) or Federated Forest (XGBoost)
6. Orchestrator registers the global model in Snowflake Model Registry

The shell script that orchestrates this across all 3 accounts is straightforward — each round calls client SPs via snow sql -c <connection>, then triggers aggregation on the orchestrator:

# Federated round (simplified — full script in GitHub repo)
for ROUND in 1 2 3; do
  # 10 clients train in parallel across 2 accounts
  for i in 1 2 3 4 5; do
    snow sql -c AWS_ACCOUNT_2 -q \
      "CALL SP_TRAIN_LOCAL_FEDERATED($ROUND, 'C${i}', 0.05, 'XGBOOST')"
  done
  for i in 6 7 8 9 10; do
    snow sql -c AZURE_ACCOUNT_3 -q \
      "CALL SP_TRAIN_LOCAL_FEDERATED($ROUND, 'C${i}', 0.05, 'XGBOOST')"  
  done

  # Orchestrator aggregates (polls until all 10 clients' updates arrive)
  snow sql -c AWS_ACCOUNT_1 -q \
    "CALL SP_AGGREGATE_GLOBAL_MODEL($ROUND, 'NONE', 600, 15, 'XGBOOST', 100)"
  snow sql -c AWS_ACCOUNT_1 -q \
    "CALL SP_AGGREGATE_GLOBAL_MODEL($ROUND, 'DP', 600, 15, 'XGBOOST', 100)"
done

Aggregation: The Core of Federated Learning

The aggregation Stored Procedure is the heart of the system — it receives model parameters from all 10 clients via Private Listing shared databases, combines them into a single global model, and stores the result back in HFL_GLOBAL_WEIGHTS for the next round.

Get Yoshitaka Inoue’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Before aggregation can begin, the orchestrator must wait for all 10 clients’ updates to replicate via Private Listing. The SP implements a polling loop that checks the two shared databases (AWS2_HFL_CLIENT_DB and AZ1_HFL_CLIENT_DB) every 15 seconds, with a 600-second timeout. Once all 10 client updates are visible, aggregation proceeds.

We implemented two aggregation strategies side by side, selectable via the MODEL_TYPE parameter.

FedAvg (Logistic Regression)

FedAvg computes a sample-weighted average of all clients’ coefficients, intercepts, and scaler parameters:

# Inside SP_AGGREGATE_GLOBAL_MODEL (LogReg path)
def aggregate_logreg(updates_df):
    total_samples = 0
    weighted_coef = None
    weighted_intercept = 0.0
    weighted_scaler_mean = None
    weighted_scaler_scale = None

    for _, row in updates_df.iterrows():
        w = json.loads(row["WEIGHTS"]) if isinstance(row["WEIGHTS"], str) \
            else row["WEIGHTS"]
        n = int(row["NUM_SAMPLES"])
        coef = np.array(w["coef"])
        s_mean = np.array(w["scaler_mean"])
        s_scale = np.array(w["scaler_scale"])
        if weighted_coef is None:
            weighted_coef = coef * n
            weighted_scaler_mean = s_mean * n
            weighted_scaler_scale = s_scale * n
        else:
            weighted_coef += coef * n
            weighted_scaler_mean += s_mean * n
            weighted_scaler_scale += s_scale * n
        weighted_intercept += float(w["intercept"]) * n
        total_samples += n

    global_weights = {
        "model_type": "LOGREG",
        "coef": (weighted_coef / total_samples).tolist(),
        "intercept": weighted_intercept / total_samples,
        "scaler_mean": (weighted_scaler_mean / total_samples).tolist(),
        "scaler_scale": (weighted_scaler_scale / total_samples).tolist(),
    }
    return global_weights

Each client shares not just model coefficients but also its local StandardScaler parameters (mean_, scale_), which are averaged the same way. This ensures the global model is evaluated under a consistent scaling.

Federated Forest (XGBoost)

For tree-based models, weight averaging doesn’t apply. Instead, we concatenate all clients’ decision trees into one ensemble and prune to keep only the most impactful trees:

# Inside SP_AGGREGATE_GLOBAL_MODEL (XGBoost path)
def aggregate_xgboost(session, updates_df, max_trees):
    # 1. Parse all client models from base64-encoded JSON
    model_dicts = []
    for _, row in updates_df.iterrows():
        w = json.loads(row["WEIGHTS"]) if isinstance(row["WEIGHTS"], str) \
            else row["WEIGHTS"]
        raw_bytes = base64.b64decode(w["model_b64"])
        model_dicts.append(json.loads(raw_bytes))

    # 2. Concatenate all trees (10 clients × 50 trees = 500)
    base = model_dicts[0]
    all_trees = list(base["learner"]["gradient_booster"]["model"]["trees"])
    all_tree_info = list(
        base["learner"]["gradient_booster"]["model"]["tree_info"])
    for md in model_dicts[1:]:
        all_trees.extend(
            md["learner"]["gradient_booster"]["model"]["trees"])
        all_tree_info.extend(
            md["learner"]["gradient_booster"]["model"]["tree_info"])

    # 3. Prune: keep top MAX_TREES by prediction impact
    if len(all_trees) > max_trees:
        eval_df = session.table(
            "HFL_APP_DB.HFL_META.EVAL_DATA_SAMPLE"
        ).select(RAW_COLS + ["DIABETES_LABEL"]).to_pandas().dropna()
        X_eval = engineer_features(eval_df)

        tree_scores = []
        for i, tree in enumerate(all_trees):
            # Build single-tree model, predict, score by abs(pred - 0.5)
            single_model = build_single_tree_model(base, tree, all_tree_info[i])
            tmp = xgb.Booster()
            tmp.load_model(bytearray(json.dumps(single_model), "utf-8"))
            preds = tmp.predict(xgb.DMatrix(X_eval, feature_names=FEATURE_NAMES))
            tree_scores.append((i, float(np.mean(np.abs(preds - 0.5)))))

        tree_scores.sort(key=lambda x: x[1], reverse=True)
        keep = sorted([ts[0] for ts in tree_scores[:max_trees]])
        all_trees = [all_trees[i] for i in keep]

    # 4. Assemble pruned global model and serialize
    base["learner"]["gradient_booster"]["model"]["trees"] = all_trees
    merged_b64 = base64.b64encode(json.dumps(base).encode()).decode()
    return {"model_type": "XGBOOST", "model_b64": merged_b64,
            "num_trees": len(all_trees)}

The pruning criterion — mean(abs(pred - 0.5)) per tree — measures how much each individual tree shifts predictions away from the 0.5 decision boundary. Trees that produce predictions closer to 0 or 1 (i.e., more confident and decisive) score higher and are retained. Trees that produce predictions near 0.5 (essentially noise) are discarded. This is evaluated on the orchestrator's held-out 10K evaluation dataset, which covers all 53 states.

A key implementation detail for XGBoost JSON manipulation: after pruning, tree IDs must be re-indexed sequentially, and the model's num_trees, tree_info, and iteration_indptr metadata fields must be updated to match. Without this, the XGBoost Booster will fail to load the assembled model.

Differential Privacy: Protecting Model Parameters

Even though raw data never leaves each client’s account, model parameters can theoretically leak information about the training data. Techniques like model inversion and membership inference attacks can extract characteristics of individual training records from shared model weights. Differential Privacy (DP) addresses this by adding calibrated Gaussian noise to the parameters before they leave the client’s account.

The noise injection strategy differs by model type:

• LogReg: Gaussian noise (Normal(0, noise_scale)) is added to each element of the coefficient vector and the intercept
• XGBoost: Noise is added specifically to leaf values (base_weights in the JSON tree structure). Leaf nodes are identified by checking left_children[i] == -1 in the tree's array representation. Internal split conditions are left unchanged to preserve the tree's routing logic.

Both use noise_scale=0.05. The system produces two global models per round: FEDERATED_NONE (aggregated from clean parameters) and FEDERATED_DP (aggregated from DP-noised parameters). This paired design enables direct measurement of the privacy-accuracy tradeoff without needing to re-run the experiment.

Results

Global Evaluation

FEDERATED_NONE (0.8053) > FEDERATED_DP (0.8042) > all 10 LOCAL_ONLY models. The best local model reaches only 0.7787.

LogReg achieves higher absolute ROC-AUC because simpler linear models generalize better with only 1,000 samples per client. But XGBoost shows a 4.5× larger federation benefit (+0.027 vs +0.006) — XGBoost’s expressiveness is a double-edged sword. With 50 trees of depth 6, it has enough capacity to memorize patterns in 1,000 samples that don’t generalize. Federation provides the diversity of data that XGBoost needs to learn robust, population-wide patterns instead of local noise.

Both models achieve the ideal ordering: FEDERATED_NONE > FEDERATED_DP > all LOCAL_ONLY. The DP-noised federated model loses less than 0.002 ROC-AUC compared to the clean version — a negligible cost for meaningful privacy protection.

Client-Local Evaluation

Global metrics tell one story, but each client cares most about: “Does the federated model perform better on my specific patients?”

All 10 clients see improved accuracy on their own geographic data, with a mean improvement of +0.039.

All 10 clients improved — without exception. The improvement ranges from +0.023 (C3, which already had the strongest local model) to +0.049 (C8, whose local model struggled most with just 1,000 samples). The mean improvement is +0.039 — roughly 3.2× larger than what we observed with Logistic Regression (+0.012), confirming that tree-based models benefit disproportionately from federation because they are more prone to overfitting on small local datasets.

Fed column shows the higher of FEDERATED_NONE / FEDERATED_DP. Evaluated on each client’s own geographic distribution (states assigned to that client, held-out months).

This is perhaps the most compelling result for real-world adoption: federation doesn’t just help “on average” — it helps every single participant, even those whose local model was already performing reasonably well. No client is worse off for participating.

Why Build This on Snowflake?

Before settling on this architecture, I considered using an S3 bucket for parameter exchange — it would have worked, but it meant stepping outside Snowflake’s governance boundary and managing IAM roles, bucket policies, and cross-account permissions separately. Everything needed turned out to already exist natively:

• Zero additional infrastructure: Private Listing + Stored Procedures + Model Registry handle the entire workflow. No gRPC servers, no S3 buckets, no Kubernetes clusters.
• Cross-region, cross-cloud by default: Private Listing handles AWS-to-Azure synchronization transparently. Our experiment spans 3 regions across 2 cloud providers, yet the code contains zero network configuration.
• Built-in data governance: RBAC, audit logs, and sharing policies enforced at the platform level. “Data never leaves the owner’s account” is architecturally guaranteed.
• Leverage existing infrastructure: Most organizations already have data in Snowflake. No ETL to external ML systems required.
• Operationally simple: The entire workflow runs from a single shell script using the Snowflake CLI. Model versions are automatically tracked in the Model Registry.

Key Takeaways

Running this experiment across three accounts sharpened a few things I hadn’t fully appreciated before:

• Federated Learning is production-viable on Snowflake today using only native features: Private Listings, Python Stored Procedures, VARIANT columns, and Model Registry.
• Tree-based models benefit enormously from federation — XGBoost showed a 4.5× larger improvement than LogReg, because complex models suffer more from limited local data.
• Differential Privacy is practically free — the DP-noised federated model still outperformed every local baseline, with less than 0.002 ROC-AUC cost.
• Every client benefits — not just on average, but individually. All 10 clients saw meaningful improvement on their own geographic data.
• The pattern is universal — while we demonstrated with healthcare data, the same architecture applies wherever data sensitivity is high but collective intelligence creates value.

Full source code: https://github.com/Snowflake-Labs/horizontal-federated-learning

Data source: CDC BRFSS 2024 — Behavioral Risk Factor Surveillance System (457K records, 53 U.S. states, 15 health/lifestyle features for diabetes prediction).

Snowflake features used: Private Listing, Python Stored Procedures, VARIANT columns, Model Registry, Snowflake CLI.

The views expressed in this article are my own and do not represent the official position of my employer.