Redis Data Integration configuration file

Redis Data Integration configuration file reference

Configuration file for Redis Data Integration (RDI) source collectors and target connections

Properties

Name Type Description Required
sources
(Source collectors)		 object Defines source collectors and their configurations. Each key represents a unique source identifier, and its value contains specific configuration for that collector
processors
(Data processing configuration)		 object, null Configuration settings that control how data is processed, including batch sizes, error handling, and performance tuning
targets
(Target connections)		 object Configuration for target Redis databases where processed data will be written
secret-providers
(Secret providers)		 object

sources: Source collectors

Defines source collectors and their configurations. Each key represents a unique source identifier, and its value contains specific configuration for that collector

Properties (Pattern)

Name Type Description Required
.*

processors: Data processing configuration

Configuration settings that control how data is processed, including batch sizes, error handling, and performance tuning

Properties

Name Type Description Required
on_failed_retry_interval
(Retry interval on failure)		 integer, string Number of seconds to wait before retrying a failed operation
Default: 5
Pattern: ^\${.*}$
Minimum: 1
read_batch_size integer, string Maximum number of records to process in a single batch
Default: 2000
Pattern: ^\${.*}$
Minimum: 1
debezium_lob_encoded_placeholder
(Debezium LOB placeholder)		 string Placeholder value for LOB fields in Debezium
Default: "__debezium_unavailable_value"
dedup
(Enable deduplication)		 boolean Enable the deduplication mechanism to handle duplicate records
Default: false
dedup_max_size
(Deduplication set size)		 integer Maximum number of entries to store in the deduplication set
Default: 1024
Minimum: 1
dedup_strategy
(Deduplication strategy)		 string (DEPRECATED)
Property 'dedup_strategy' is now deprecated. The only supported strategy is 'ignore'. Please remove from the configuration.
Default: "ignore"
Enum: "reject", "ignore"
duration
(Batch duration limit)		 integer, string Maximum time in milliseconds to wait for a batch to fill before processing
Default: 100
Pattern: ^\${.*}$
Minimum: 1
write_batch_size integer, string Maximum number of records to write to target Redis database in a single batch
Default: 200
Pattern: ^\${.*}$
Minimum: 1
error_handling
(Error handling strategy)		 string Strategy for handling errors: ignore to skip errors, dlq to store rejected messages in dead letter queue
Default: "dlq"
Pattern: ^\${.*}$|ignore|dlq
dlq_max_messages
(DLQ message limit)		 integer, string Maximum number of messages to store in dead letter queue per stream
Default: 1000
Pattern: ^\${.*}$
Minimum: 1
target_data_type
(Target Redis data type)		 string Data type to use in Redis: hash for Redis Hash, json for RedisJSON (requires RedisJSON module)
Default: "hash"
Pattern: ^\${.*}$|hash|json
json_update_strategy string (DEPRECATED)
Property 'json_update_strategy' will be deprecated in future releases. Use 'on_update' job-level property to define the json update strategy.
Default: "replace"
Pattern: ^\${.*}$|replace|merge
use_native_json_merge
(Use native JSON merge)		 boolean Controls whether to use the native JSON.MERGE command (when true) or Lua scripts (when false) for JSON merge operations. Introduced in RDI 1.15.0. The native command provides 2x performance improvement but handles null values differently:

Previous behavior (Lua merge): When merging {"field1": "value1", "field2": "value2"} with {"field2": null, "field3": "value3"}, the result was {"field1": "value1", "field2": null, "field3": "value3"} (null value is preserved)

New behavior (JSON.MERGE): The same merge produces {"field1": "value1", "field3": "value3"} (null value removes the field, following RFC 7396)

Note: The native JSON.MERGE command requires RedisJSON 2.6.0 or higher. If the target database has an older version of RedisJSON, RDI will automatically fall back to using Lua-based merge operations regardless of this setting.

Impact: If your application logic distinguishes between a field with a null value and a missing field, you may need to adjust your data handling. This follows the JSON Merge Patch RFC standard but differs from the previous Lua implementation. Set to false to revert to the previous Lua-based merge behavior if needed.
Default: true
initial_sync_processes integer, string Number of parallel processes for performing initial data synchronization
Default: 4
Pattern: ^\${.*}$
Minimum: 1
Maximum: 32
idle_sleep_time_ms
(Idle sleep interval)		 integer, string Time in milliseconds to sleep between processing batches when idle
Default: 200
Pattern: ^\${.*}$
Minimum: 1
Maximum: 999999
idle_streams_check_interval_ms
(Idle streams check interval)		 integer, string Time in milliseconds between checking for new streams when processor is idle
Default: 1000
Pattern: ^\${.*}$
Minimum: 1
Maximum: 999999
busy_streams_check_interval_ms
(Busy streams check interval)		 integer, string Time in milliseconds between checking for new streams when processor is busy
Default: 5000
Pattern: ^\${.*}$
Minimum: 1
Maximum: 999999
wait_enabled
(Enable replica wait)		 boolean Enable verification that data has been written to replica shards
Default: false
wait_timeout
(Replica wait timeout)		 integer, string Maximum time in milliseconds to wait for replica write verification
Default: 1000
Pattern: ^\${.*}$
Minimum: 1
retry_on_replica_failure boolean Continue retrying writes until successful replication to replica shards is confirmed
Default: true

Additional Properties: not allowed

targets: Target connections

Configuration for target Redis databases where processed data will be written

Properties (Pattern)

Name Type Description Required
.*

secret-providers: Secret providers

Properties (Pattern)

Name Type Description Required
.*
(Secret provider entry)		 object yes

secret-providers..*: Secret provider entry

Properties

Name Type Description Required
type
(Provider type)		 string Enum: "aws", "vault"
 yes
parameters
(Provider parameters)		 object yes

Additional Properties: not allowed
Example

parameters:
  objects:
    - {}

secret-providers..*.parameters: Provider parameters

Properties

Name Type Description Required
objects
(Secrets objects array)		 object[] yes

Example

objects:
  - {}

secret-providers..*.parameters.objects[]: Secrets objects array

Items: Secret object

No properties.

Example

- {}
RATE THIS PAGE
Back to top ↑