Deduplicating Logs at the Agent: OpenTelemetry vs. Vector, Fluent Bit, vlagent, Grafana Alloy, and Filebeat

What these tools actually mean by “deduplication,” which ones reduce traffic before storage, and what happened when I sent the same 100,000 events through each pipeline.

The same stack trace printed 50,000 times does not make an incident investigation 50,000 times more useful. It does, however, increase network traffic, storage consumption, and query time by a very real amount.

The obvious solution is to enable deduplication in the log agent. The problem is that different log shippers mean very different things by “deduplication.” One aggregates repeated events and preserves their count. Another silently keeps only the first event. A third can only enforce a rate limit. A fourth merely hides similar lines in the UI — after every duplicate has already been transmitted and stored.

My name is Anton Kasimov. I work at Gals Software and run Monitorim IT, a Russian-language Telegram channel about observability. I compared six familiar log collection tools:

- OpenTelemetry Collector

- Vector

- vlagent

- Fluent Bit

- Grafana Alloy

- Filebeat

I then sent the same stream of data through the implementations that support deduplication. Only two of the six provide genuine, built-in content deduplication: OpenTelemetry Collector and Vector. Fluent Bit can do it with a custom script. Filebeat, vlagent, and Grafana Alloy have no general-purpose built-in mechanism for it.

First, what exactly counts as a duplicate?

Consider these two records:

{"service":"billing","level":"error","message":"database timeout","request_id":"a-17","timestamp":"2026–09–18T09:00:00Z"}
{"service":"billing","level":"error","message":"database timeout","request_id":"b-42","timestamp":"2026–09–18T09:00:01Z"}

They are different byte for byte. Semantically, however, they may represent the same recurring failure. To group them, an agent needs a deduplication key — for example:

(service, level, message)

If we include `timestamp` and `request_id` in that key, every log becomes unique and deduplication achieves nothing. If we keep only `message`, we may accidentally merge events from different services or tenants.

It helps to separate the available mechanisms into four categories:

1. Fingerprint-based deduplication. The agent remembers selected fields and removes or aggregates repeats within a window.

2. Rate limiting. The agent allows no more than *N* records per second. It does not care whether those records are identical.

3. Protection against rereading. A registry or checkpoint stores the current read position in a file.

4. Query- or display-time deduplication. The backend stores everything, while the UI collapses similar lines for the user.

Only the first category is true pre-backend content deduplication. The others are useful, but they solve different problems.

The short answer

Log deduplication capabilities

LRU stands for least recently used. When the cache is full, the entry that has not been used for the longest time is evicted.

1. OpenTelemetry Collector: aggregate repeats without losing the count

The Contrib distribution includes the log_dedup processor. At the time of writing, its logs stability level is alpha.

The mechanism is straightforward:

1. The processor groups identical records during a configured interval — 10 seconds by default.

2. At the end of the interval, it emits one record.

3. It adds `log_count`, `first_observed_timestamp`, and `last_observed_timestamp`.

This is the most honest form of deduplication in the comparison. Instead of 100 identical errors, I get one error carrying the information that it occurred 100 times.

How the key is built

Without additional configuration, uniqueness is determined from the body, resource attributes, severity, and log attributes. That is usually too strict for real structured logs: one changing `request_id` is enough to make every record unique.

I prefer an explicit allowlist using `include_fields`:

processors:
 log_dedup:
 interval: 30s
 include_fields:
 - attributes.service
 - attributes.level
 - attributes.message
 log_count_attribute: repeat_count

The alternative is `exclude_fields`, but the two options are mutually exclusive. There is another unpleasant detail: excluded fields are also removed from the resulting aggregated record. An allowlist is usually easier to reason about and safer.

With `conditions`, you can also limit deduplication to selected records — for example, errors only.

A complete minimal example

After `json_parser` runs in this example, the parsed fields are stored in attributes, which is why the paths begin with `attributes.*`:

receivers:
 tcp_log:
 listen_address: 0.0.0.0:9000
 operators:
 - type: json_parser
 parse_from: body
processors:
 log_dedup:
 interval: 10s
 include_fields:
 - attributes.service
 - attributes.level
 - attributes.message
 log_count_attribute: duplicate_count
exporters:
 otlphttp/logs:
 endpoint: https://logs.example.com
service:
 pipelines:
 logs:
 receivers: [tcp_log]
 processors: [log_dedup]
 exporters: [otlphttp/logs]

What to keep in mind

- A record is delayed until the `interval` closes. That is the price of aggregation.

- The timestamp of the emitted log is the time at which the deduplicator released it, not the timestamp of the first original record. The source range is available in `first_observed_timestamp` and `last_observed_timestamp`.

- High cardinality can consume memory before deduplication saves anything.

- After a Collector restart, the active window starts from scratch. An exporter queue does not make the `log_dedup` state persistent.

- The old processor name, `logdedup`, is still accepted but deprecated. The current name is `log_dedup`.

Where it wins

OpenTelemetry is the best fit when I need to reduce transmitted volume without losing the number of occurrences. I can build alerts and statistics from `repeat_count`. The trade-offs are the processor’s alpha status, the aggregation delay, and the need to monitor cardinality carefully so memory usage does not grow dramatically.

— -

2. Vector: the first event gets through

Vector provides a stable, stateful dedupe transform. It stores recent fingerprints in an LRU cache and drops repeated events.

A minimal configuration for structured logs looks like this:

transforms:
 dedupe_errors:
 type: dedupe
 inputs: [input]
 fields:
 match:
 - service
 - level
 - message
 cache:
 num_events: 2000
 time_settings:
 max_age_ms: 60000
 refresh_on_drop: false

Why I would change the defaults

If `fields` is not specified, Vector compares the `timestamp`, `host`, and `message` fields from its global schema. Two otherwise identical messages with different timestamps are therefore different events. Deduplication may be enabled and still deduplicate nothing.

There are two ways to build the key:

- `fields.match` includes only the listed fields.

- `fields.ignore` includes everything except the listed fields.

I recommend `match`. It documents the meaning of an event more clearly and uses memory more efficiently: Vector stores only the values involved in the comparison. With `ignore`, it may need to retain almost the entire record.

Vector’s window is not the same as OpenTelemetry’s interval

`cache.num_events` defaults to 5,000. That is the number of events retained for comparison. With high cardinality, useful fingerprints are evicted quickly, allowing older repeats through again. `time_settings.max_age_ms` adds an age boundary.

These controls work together. A duplicate is dropped only while its fingerprint is still in the cache and has not exceeded `max_age_ms`.

`refresh_on_drop` changes the meaning of the window:

- With `false`, age is measured from the first accepted event. After one minute, the next copy passes through again.

- With `true`, every dropped repeat refreshes the key. A continuous log storm can be suppressed indefinitely, until the gap between repeats exceeds `max_age_ms`.

That is a small option with a large operational effect. In the first mode, I periodically see evidence that the problem still exists. In the second, every event after the first disappears while the storm continues.

What gets lost

Vector keeps the first event and drops the rest. The transform has no built-in equivalent of `repeat_count`. The technical metric `component_discarded_events_total` can show how many events were discarded by the component, but it is not broken down by individual log signature.

The LRU cache lives in memory. After a restart, the same log is new again. The same is true when Vector is scaled horizontally: two instances do not share a cache, so there is no global deduplication across replicas.

Where it wins

Vector is a strong choice when I need mature, compact, and predictable duplicate suppression — and nobody needs the exact repeat count.

— -

3. Fluent Bit: build it yourself

The current Fluent Bit filter catalog has no general-purpose stateful deduplication filter. The `throttle` filter is not a substitute: it limits the number of records without comparing their contents.

One practical option is the Lua filter. Its callback returns `-1` when a record should be dropped.

Example configuration:

service:
 flush: 1
 log_level: info
pipeline:
 inputs:
 - name: tail
 path: /var/log/containers/*.log
 tag: kubernetes.*
filters:
 - name: lua
 match: kubernetes.*
 script: /fluent-bit/scripts/dedupe.lua
 call: dedupe
 protected_mode: true
outputs:
 - name: forward
 match: '*'
 host: logs.example.com
 port: 24224

Here is a simplified Lua cache with a TTL and a size limit:

local seen = {}
local fifo = {}
local head = 1
local tail = 0
local max_entries = 2000
local ttl_seconds = 60
local function evict(now)
 while head <= tail do
 local item = fifo[head]
 local over_capacity = (tail - head + 1) > max_entries
 local expired = (now - item.seen_at) >= ttl_seconds
if not over_capacity and not expired then
 break
 end
if seen[item.key] == item.seen_at then
 seen[item.key] = nil
 end
fifo[head] = nil
 head = head + 1
 end
end
function dedupe(tag, timestamp, record)
 local now = os.time()
 local key = table.concat({
 tostring(record["service"]),
 tostring(record["level"]),
 tostring(record["message"]),
 }, "\31")
evict(now)
if seen[key] ~= nil and (now - seen[key]) < ttl_seconds then
 return -1, 0, 0
 end
seen[key] = now
 tail = tail + 1
 fifo[tail] = { key = key, seen_at = now }
 return 0, timestamp, record
end

What you have to solve yourself

- Key composition and type canonicalization

- TTL and eviction policy

- Memory limits

- Hash collisions, if you use a hash instead of the complete key

- Counters for dropped records

- Restart behavior

- Consistent behavior across replicas

- Tests for log schema changes

This example keeps state only in memory and loses it after a restart. Large Lua tables also consume real RAM. An external store is theoretically possible, but it introduces another dependency and makes the pipeline much less simple.

Where it wins

Fluent Bit makes sense when it is already deployed everywhere and the rule is narrow and easy to test — for example, suppressing a specific known log storm on a particular set of nodes. For a general deduplication strategy, I would look at Vector or OpenTelemetry first: less custom code and a more reusable implementation.

— -

4. vlagent: reliable delivery, not content deduplication

vlagent can read files and Kubernetes logs, add and remove fields, replicate a stream, and buffer data to disk while VictoriaLogs is unavailable. It does not have a general-purpose cache of log fingerprints.

Two mechanisms are easy to confuse with deduplication:

- A checkpoint remembers the read position in a file and helps prevent already processed lines from being read again after a restart.

- `-remoteWrite.tmpDataPath` stores undelivered data and sends it after the backend recovers.

Neither mechanism decides that two different records with the same `service`, `level`, and `_msg` are duplicates. If an application writes a message twice, vlagent sends both copies. If the same record arrives through two independent sources, a checkpoint cannot help either.

Here is an example of a reliable-delivery configuration:

./vlagent-prod \
 -fileCollector.glob='/var/log/apps/*.log' \
 -fileCollector.checkpointsPath=/var/lib/vlagent/file-checkpoints.json \
 -tmpDataPath=/var/lib/vlagent \
 -remoteWrite.tmpDataPath=/var/lib/vlagent/remote-write \
 -remoteWrite.maxDiskUsagePerURL=20GiB \
 -remoteWrite.url=http://victoria-logs:9428/insert/native

This is useful, but it is not content deduplication.

On the VictoriaLogs side, LogsQL can collapse query results:

_time:15m
| uniq by (service, level, _msg) with hits

uniq … with hits displays unique combinations and their counts. By that point, however, every record has already been accepted, transmitted, and stored. It saves neither network traffic nor storage.

Get Anton I. Kasimov’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

VictoriaMetrics — the metrics backend — has a `-dedup.minScrapeInterval` option that deduplicates samples from the same time series after ingestion. VictoriaLogs does not currently offer an equivalent feature for logs.

Where it wins

The advantage of vlagent is not deduplication. It is an efficient delivery path into VictoriaLogs.

— -

5. Grafana Alloy: filtering and rate limits, but no deduplication

The current Alloy component catalog has no equivalent of `log_dedup`, and loki.process has no mechanism for comparing log fingerprints over time.

The closest components solve different problems:

- `stage.drop` removes a record based on a condition or regular expression, but knows nothing about the previous event.

- `stage.limit` rate-limits logs within `loki.process`.

- otelcol.processor.filter evaluates each log independently and keeps no history.

For example, this configuration protects a backend from a stream exceeding 100 lines per second, but identical and unrelated logs are treated exactly the same:

loki.process "rate_limit" {
 stage.limit {
 rate = 100
 burst = 200
 drop = true
 }
forward_to = [loki.write.backend.receiver]
}

Another source of confusion is the **Deduplication** control in Grafana Explore. The `Exact`, `Numbers`, and `Signature` modes do hide similar lines, but this is a display setting. It does not reduce the volume Alloy sends or Loki stores.

What Alloy users can do

If I need real deduplication before the backend, the practical options are:

1. Put Vector with its `dedupe` transform in the pipeline before Loki.

2. Use a separate OpenTelemetry Collector Contrib instance with `log_dedup`, while keeping Alloy for the rest of the pipeline.

3. Suppress noise with `stage.drop` or `stage.limit`, while describing it accurately as filtering or rate limiting.

4. Deduplicate inside the application, where the event semantics are known best.

An Alloy module can neatly reuse a combination of built-in components, but it cannot create a deduplication mechanism that those components do not have.

Where it wins

Alloy is convenient as a unified agent for the Grafana stack and OpenTelemetry components. I would not choose it specifically for log deduplication. Its strengths are routing, parsing, labels, filtering, and delivery — not caching repeated events.

— -

6. Filebeat: idempotent indexing rather than a duplicate cache

Filebeat has no built-in stateful processor that keeps a fingerprint cache and drops matching messages within a time window. It does have another useful mechanism: Filebeat can pass a stable document identifier to Elasticsearch through `metadata._id`.

This addresses a property of at-least-once delivery. Imagine that Elasticsearch accepts a bulk request, but its acknowledgement is lost because of a network failure. Filebeat cannot know whether the document was stored, so it sends the event again. If Elasticsearch generates the ID, the index receives two documents. If Filebeat sends the same stable ID again, the new operation overwrites the document with that ID instead of creating another one.

The flow looks like this:

same fingerprint → same @metadata._id → same Elasticsearch _id

Configuration with a fingerprint

The fingerprint processor calculates an ID from selected fields:

filebeat.inputs:
 - type: filestream
 id: app-json
 paths:
 - /var/log/app/*.json
 parsers:
 - ndjson:
 target: ""
 add_error_key: true
processors:
 - fingerprint:
 fields:
 - service.name
 - log.level
 - message
 target_field: "@metadata._id"
 method: sha256
 encoding: hex
output.elasticsearch:
 hosts:
 - https://elasticsearch.example:9200

Filebeat supports `md5`, `sha1`, the `sha2` family, and `xxhash`; the default is `sha256`. For deduplication, choosing the correct fields matters more than choosing the cryptographic algorithm.

If the application already emits a unique `event_id`, I would use it instead of building a key from message text. For JSON stored in the `message` field, the ID can be extracted while decoding:

processors:
 - decode_json_fields:
 fields: [message]
 target: ""
 document_id: event_id

`document_id` takes the value of the selected field and places it in `metadata._id`. If Logstash sits between Filebeat and Elasticsearch, the metadata must be passed explicitly in the output configuration:

output {
 elasticsearch {
 hosts => ["https://elasticsearch.example:9200"]
 document_id => "%{[@metadata][_id]}"
 }
}

Otherwise, the stable ID is lost at the final pipeline stage and Elasticsearch goes back to creating documents with automatically generated IDs.

What counts as a duplicate here?

Filebeat has no time window. If two events receive the same `_id` and enter the same concrete index, the second replaces the first — whether they arrived five seconds or five months apart.

That creates two very different scenarios:

- A key based on a real `event_id` protects against redelivery of the same event.

- A key based on `(service.name, log.level, message)` suppresses repeated messages even if they represent two genuine, separate failures.

The second option is more dangerous than it looks. It does not preserve `repeat_count`, offers no time window, and can overwrite a useful later event. An `_id` is also unique only within one concrete index; when a new daily index is created, the same fingerprint creates a separate document again.

Filebeat also has an `add_id` processor that creates a unique ID for an event. This helps when Filebeat resends the same queued event because the ID remains attached to it. But if the line is read again after the registry file is lost, or the application writes it again, `add_id` generates a different value. Persistent idempotency requires either a deterministic fingerprint or a natural ID from the event itself.

Do not confuse this with file identity fingerprints

The filestream input also supports `file_identity.fingerprint`. It identifies a file from a fingerprint of part of its contents and protects against problems involving rotation, inode reuse, and renaming. That is the identity of a file, not a log record. It reduces the risk of rereading but does not compare messages with one another.

Resource behavior

This approach requires almost no state in the agent: the fingerprint is calculated on the fly, no LRU cache grows, and the same input produces the same ID after a restart. But this is not free pre-backend deduplication. The repeat still passes through Filebeat, the network, and the Elasticsearch ingest pipeline. The saving is in the number of final documents, while Elasticsearch still pays for the replacement indexing operation.

Where it wins

Filebeat is especially useful when avoiding a second document caused by redelivery is critical and the destination is Elasticsearch.

— -

Testing every implementation with the same stream

To verify the behavior in practice, I built a small Docker-based demo. The generator sent 100,000 NDJSON records containing 1,000 logical signatures, each repeated 100 times. The `timestamp`, `request_id`, and `seq` fields changed in every record, while the key remained identical across implementations:

(service, level, message)

The expected result for correct deduplication was 1,000 output records — a 99% reduction in event count.

Results

100k event deduplication test

*These figures describe this small test environment, not a general benchmark. Agent versions, allocators, input and output components, traffic shape, and measurement timing can all affect memory use.

The most useful mistake in the test

In my first OpenTelemetry run, I configured:

include_fields:
 - body.service
 - body.level
 - body.message

But `json_parser` had placed the parsed values in `attributes`, while the original JSON remained in the body. As a result, changing `request_id`, `timestamp`, and `seq` values made every record unique. All 100,000 records remained in the output, and the container’s memory usage grew to roughly 600 MB. Not exactly the result I was hoping for.

After changing the paths to `attributes.service`, `attributes.level`, and `attributes.message`, the result matched expectations: 1,000 records, a total occurrence count of 100,000, and roughly 84 MB of container memory after the run.

A bad fingerprint can eliminate every saving and turn a deduplicator into a high-cardinality record generator at the same time.

— -

How to test this in your own environment

A stream made entirely of identical messages is not enough. I recommend testing at least the following scenarios.

1. Repeated events and discarded-event accounting

Check how many records remain and whether multiplicity is preserved. With OpenTelemetry, the sum of `log_count` should match the input count. With Vector and Fluent Bit, measure discarded events.

2. A nearly unique stream

Send logs with a high-cardinality key. Deduplication will save almost nothing, but the test will reveal the upper bound of RAM and CPU consumption. In practice, this case can be more dangerous than an idealized storm of identical messages.

3. The window boundary

Send an identical record immediately before and after `interval` or `max_age_ms`. Confirm that the resulting semantics match your expectations. Test Vector’s `refresh_on_drop` especially carefully.

4. An agent restart

Send an event, restart the agent, and send it again. The cache will forget the previous event. That may be acceptable when deduplication is only a defense against event storms, but it is not enough to guarantee exactly-once delivery.

5. Multiple replicas

Split identical messages between two agents. Their local caches cannot see each other, so each agent passes its own first copy. If you need global deduplication, it belongs in a shared stateful layer or in a backend that supports an idempotency key.

And always test false positives. Losing a useful event because the key is too broad is worse than storing an extra 100 MB of logs.

What I would choose

I would choose OpenTelemetry Collector when preserving the repeat count matters. It is the only implementation in this comparison that performs true aggregation: one representative record plus the number of occurrences. Its alpha status still makes me cautious.

Vector is the most convenient ready-made option when the goal is simply to drop repeats. Its LRU behavior is transparent and the transform is stable, but per-signature occurrence counts are lost.

Fluent Bit with Lua is reasonable for a narrow, known pattern when a large Fluent Bit fleet is already in place. I would not treat custom scripting as a universal deduplication layer, although it can work well after careful testing.

I would choose vlagent when the destination is VictoriaLogs and efficient delivery is the priority. Pre-backend content deduplication requires an additional processing layer. LogsQL can use `uniq … with hits` at query time, as long as everyone understands that the duplicates remain stored in the backend.

Grafana Alloy does not currently provide content-based log deduplication. `drop`, `limit`, and Explore Dedup are useful, but they are filtering, rate limiting, and display-time collapsing respectively.

Filebeat offers a different and valuable guarantee: deterministic document IDs can prevent duplicate Elasticsearch documents caused by redelivery. That is idempotent indexing, not an in-agent time-window deduplication cache.

The larger lesson is that “deduplication” is not one feature. Before enabling anything, decide which guarantee you actually need:

- Fewer bytes sent over the network

- Fewer documents stored

- Protection against at-least-once redelivery

- A visible count of repeated failures

- Or simply a less noisy UI

Once that requirement is explicit, the tool choice becomes much easier.

Further reading

- OpenTelemetry Collector Contrib: Log DeDuplication Processor

- Vector: Dedupe transform

- Fluent Bit: filters

- Fluent Bit: Lua filter

- VictoriaLogs: vlagent

- VictoriaLogs LogsQL: uniq pipe

- Grafana Alloy: component reference

- Grafana Alloy: loki.process

- Grafana: logs in Explore

- Filebeat: deduplicate data

- Filebeat: fingerprint processor

- Filebeat: filestream input and file identity

Thanks for reading. If you need help designing an observability platform, feel free to reach out to me on Medium.