DLT

DLT Source vs Resource: When Do You Actually Need Both?

Understanding the building blocks of DLT pipelines

You started with @dlt.resource. It worked. You passed it to pipeline.run(), data landed in your destination, and you moved on.

Then you looked at every DLT example out there. They all have @dlt.source wrapping the resources. You wondered — is this just boilerplate, or am I missing something?

Let me break it down.

This is the third article in a series on dlt (data load tool). The first article covered incremental loading and deduplication. The second explored custom per-feed cursors when a global cursor isn’t enough. This one zooms out to the structural question: when do you need a source, and when is a resource enough?

Start with Verified Sources

Before writing any code, check the verified sources. dltHub maintains 50+ pre-built sources — SQL databases, REST APIs, cloud storage, SaaS platforms — all packaged as sources you can use directly:

from dlt.sources.sql_database import sql_database

# That's it. No @dlt.resource, no @dlt.source, no extraction code.
pipeline.run(sql_database(credentials="postgresql://...").with_resources("users", "orders"))

Verified sources handle pagination, authentication, schema inference, and incremental loading. You configure them with credentials and options; they return a source with resources ready to load.

When verified sources are enough:

• Standard APIs (Salesforce, HubSpot, Stripe, GitHub, etc.)
• SQL databases
• File systems (S3, GCS, Azure)
• You want zero extraction code

When you write custom resources:

• Internal APIs with no verified source
• Complex transformation logic during extraction
• Data sources requiring custom authentication
• You need fine-grained control over pagination or state

The distinction in this article matters most when you’re building your own sources. If a verified source covers your use case, use it — you skip both the resource and source implementation.

So… you decided to build it yourself. Keep reading.

Photo by Panos Sakalakis on Unsplash

A Resource Is All You Need (Sometimes)

Here’s the simplest possible dlt pipeline:

import dlt

@dlt.resource(name="events", write_disposition="replace")
def events():
    yield {"id": 1, "name": "Event 1"}
    yield {"id": 2, "name": "Event 2"}
p = dlt.pipeline("simple_pipeline", destination="duckdb")
p.run(events())

That’s it. No @dlt.source. Just a decorated generator function passed directly to pipeline.run().

This works perfectly for:

• Single-table pipelines
• Quick scripts and prototypes
• When you don’t need shared state or configuration

The data loads. The schema infers. Life is simple.

So why do sources exist at all?

What a Source Actually Does

A source is a container — a logical grouping of related resources. An abstraction around data providers.

The @dlt.source decorator gives you four things:

1. Logical Grouping
2. Shared Configuration
3. State Namespace
4. Schema Control

1. Logical Grouping

When you’re pulling from an API that has multiple endpoints — users, orders, products — you group them under one source:

@dlt.source(name="crm")
def crm_source(api_key):
    @dlt.resource(name="users")
    def users():
        yield from fetch_users(api_key)
    
    @dlt.resource(name="orders")  
    def orders():
        yield from fetch_orders(api_key)
    
    return [users, orders]

All tables land under the crm schema namespace. Related data stays together.

2. Shared Configuration

Notice api_key in the example above. The source function accepts configuration once and passes it to all resources. You don't have to repeat credentials or settings for each resource.

This also integrates with dlt’s config system — you can use dlt.secrets.value or dlt.config.value to pull from environment variables or config files:

@dlt.source(name="crm")
def crm_source(api_key=dlt.secrets.value):
    # api_key is automatically loaded from secrets
    ...

3. State Namespace

When you use dlt.current.source_state(), the state is scoped to the source name. This matters for incremental loading and custom cursor patterns (covered in the previous article).

Each source gets its own state dictionary. If you have two sources — crm and analytics — their states don't collide.

4. Schema Control

The source decorator accepts max_table_nesting, which controls how dlt handles nested data:

@dlt.source(name="events_api", max_table_nesting=0)
def events_source():
    @dlt.resource(name="events")
    def events():
        yield {
            "id": 1,
            "metadata": {"key": "value", "nested": {"deep": "data"}},
            "tags": ["tag1", "tag2"]
        }
    return events

With max_table_nesting=0, nested structures like metadata and tags are stored as JSON columns — no separate tables created. Without it, dlt would create events__tags and events__metadata tables to normalize the data.

Typical settings:

• 0 — No nested tables at all. Everything becomes JSON.
• 1 — One level of nested tables. Nested lists in those tables become JSON.
• Default (1000) — dlt normalizes everything recursively.

Photo by Susan Holt Simpson on Unsplash

The Key Difference: Eager vs Lazy Execution

Here’s one subtle but important distinction:

Get Yerachmiel Feltzman’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The source function runs immediately when you call it. The resource generator runs later, when pipeline.run() extracts the data.

@dlt.source(name="my_source")
def my_source():
    print("[SOURCE] This runs NOW")
    
    @dlt.resource(name="events")
    def events():
        print("[RESOURCE] This runs during pipeline.run()")
        yield {"id": 1}
    
    return events

print("About to call my_source()...")
source = my_source()  # [SOURCE] prints here
print("Returned from my_source()")
p = dlt.pipeline("demo_pipeline", destination="duckdb")
print("About to call pipeline.run()...")
p.run(source)  # [RESOURCE] prints here
print("Done")

Output:

About to call my_source()...
[SOURCE] This runs NOW
Returned from my_source()
About to call pipeline.run()...
[RESOURCE] This runs during pipeline.run()
Done

Why does this matter?

1. Do setup work in the source body. Create database connections, API clients, reflect table schemas, validate credentials. This runs eagerly, so you get fail-fast behavior — errors surface immediately when the source is instantiated, not deep into pipeline.run().
2. Don’t extract data in the source body. API calls that fetch rows, database queries that stream data, any work that scales with data volume — these belong in the resource generator. If you put them in the source, they run whether or not pipeline.run() is ever called, and you lose dlt's extraction metrics, retries, and parallelization.
3. Resources are lazy by design. This enables dlt to parallelize extraction, manage memory, and apply optimizations. Your resource generator yields data incrementally; dlt processes it in batches.

Dynamic Resource Creation

Sometimes you don’t know your resources ahead of time. Maybe you’re reflecting a database and want one resource per table. Maybe an API returns a list of available endpoints.

Sources support dynamic resource creation:

@dlt.source(name="database")
def database_source(connection_string):
    table_names = reflect_tables(connection_string)  # Get list of tables
    
    resources = []
    for table_name in table_names:
        @dlt.resource(name=table_name, write_disposition="replace")
        def table_resource(tn=table_name):  # Capture table_name in closure
            yield from fetch_table_data(connection_string, tn)
        resources.append(table_resource)
    
    return resources

The source function creates resources programmatically. Each table becomes its own resource, and you load them all with one pipeline.run(database_source(...)).

Selecting Resources

When a source returns multiple resources, you can choose which ones to load:

from my_sources import crm_source

source = crm_source()
# Load only users and orders, skip products
pipeline.run(source.with_resources("users", "orders"))

You can also access resources individually:

# Load just the users resource
pipeline.run(source.users)

# Or iterate over resource data without loading
for user in source.users:
    print(user)

This is useful for:

• Partial loads during development
• Separating heavy tables into different pipeline runs
• Debugging specific resources

Decision Framework: Do You Need a Source?

Photo by Jon Tyson on Unsplash

Here’s the quick test:

Use just @dlt.resource when:

• You have a single table/endpoint
• No shared configuration needed
• Simple, standalone extraction

Use @dlt.source when:

• Multiple resources from the same API/database
• Shared credentials or configuration
• You want schema-level control (max_table_nesting)
• You need state scoping (source_state())
• You want to select/filter resources at runtime

In practice, I start with a standalone resource for quick scripts. When the pipeline grows — more tables, shared config, incremental state — I wrap it in a source.

Common Patterns

Pattern 1: Single Resource Source

The simplest source wraps one resource. You get the namespace and configuration benefits without much ceremony:

@dlt.source(name="weather_alerts", max_table_nesting=0)
def weather_source(api_key=dlt.secrets.value):
    @dlt.resource(name="alerts", write_disposition="append", primary_key="alert_id")
    def alerts():
        yield from fetch_weather_alerts(api_key)
    return alerts

Pattern 2: Mode-Based Resources

Some pipelines have different modes — historical (full backfill) vs incremental. You can return different resource implementations:

@dlt.source(name="crm")
def crm_source(mode="incremental", api_key=dlt.secrets.value):
    if mode == "historical":
        @dlt.resource(name="users", write_disposition="replace")
        def users():
            yield from fetch_all_users(api_key)
    else:
        @dlt.resource(name="users", write_disposition="merge", primary_key="user_id")
        def users(
            updated_at=dlt.sources.incremental("updated_at", initial_value="1970-01-01")
        ):
            yield from fetch_users_since(api_key, updated_at.last_value)
    
    return users

Historical mode does a full replace. Incremental mode uses merge with a cursor.

Pattern 3: Configuration-Driven Multi-Table

When you have many tables with similar extraction logic, generate resources from configuration:

@dlt.source(name="api_mirror", max_table_nesting=0)
def api_source(endpoints: list[str], api_key=dlt.secrets.value):
    def make_resource(endpoint):
        @dlt.resource(name=endpoint, write_disposition="append")
        def resource():
            yield from fetch_endpoint(api_key, endpoint)
        return resource
    
    return [make_resource(ep) for ep in endpoints]

# Usage
pipeline.run(api_source(endpoints=["users", "orders", "products"]))

One source, many resources, minimal code duplication.

State Scoping in Detail

I covered source_state() in the previous article for per-feed cursors. Here's the scoping model:

State scoping model

If you’re using a standalone resource (no source), resource_state() still works — it's scoped to the resource name within the default source namespace.

Important timing note: state modifications in the source body (which runs eagerly) and state access in resource generators (which run lazily during pipeline.run()) happen at different times. Keep this in mind when coordinating state across the source/resource boundary.

Takeaways

1. Check verified sources first. 50+ pre-built sources mean you often don’t need to write extraction code at all. Use them when they fit your use case.
2. A standalone @dlt.resource is enough for simple custom pipelines. You don't need a source for single-table extractions.
3. @dlt.source is a container, not a requirement. Use it when you need grouping, shared config, state namespacing, or schema control.
4. Sources execute eagerly; resources execute lazily. Put setup (connections, clients, reflection) in the source body for fail-fast behavior. Put data extraction in the resource generator for retries, metrics, and parallelization.
5. max_table_nesting controls JSON vs nested tables. Set it at the source level to apply across all resources.
6. Resources can be created dynamically. Generate them from configuration, database reflection, or API metadata.
7. Select resources at runtime with with_resources(). Useful for partial loads and debugging.
8. State scoping follows the source/resource hierarchy. source_state() is shared; resource_state() is isolated.

Start simple. Add structure when you need it. The decorator pattern makes it easy to wrap a standalone resource in a source later — you’re not locked in.

Happy pipelines. 🚀

I’m a Senior Software Engineer at Tikal — a company of tech experts providing hands-on consultancy and scaling R&D teams with cutting-edge technologies. If you enjoyed this article, you should check Tikal’s publication for articles where breath and depth come together.