Remove fields from a key

By default, RDI adds fields to hash or JSON objects in the target database for each of the columns of the source table. The examples below show how to omit some of those fields from the target data with the remove_field transformation.

Remove a single field

The first example removes a single field from the data. The source section selects the employee table of the chinook database (the optional db field here corresponds to the sources.<source-name>.connection.database field defined in config.yaml).

In the transform section, the remove_field transformation removes the hiredate field.

The output section specifies hash as the data_type to write to the target, which overrides the default setting of target_data_type defined in config.yaml. Also, the output.with.key section specifies a custom key format of the form emp:<employeeid>. Note that any fields you remove in the transform section are not available for the key calculation in the output section.

The full example is shown below:

source:
  db: chinook
  table: employee
transform:
  - uses: remove_field
    with:
      field: hiredate
output:
  - uses: redis.write
    with:
      connection: target
      data_type: hash
      key:
        expression: concat(['emp:', employeeid])
        language: jmespath

If you queried the generated target data from the default transformation using redis-cli, you would see something like the following:

> hgetall emp:8
 1) "employeeid"
 2) "8"
 3) "lastname"
 4) "Callahan"
 5) "firstname"
 6) "Laura"
 7) "title"
 8) "IT Staff"
 9) "reportsto"
10) "6"
11) "birthdate"
12) "-62467200000000"
13) "hiredate"
14) "1078358400000000"
15) "address"
16) "923 7 ST NW"
.
.

Using the job file above, the data omits the hiredate field:

 > hgetall emp:8
 1) "employeeid"
 2) "8"
 3) "lastname"
 4) "Callahan"
 5) "firstname"
 6) "Laura"
 7) "title"
 8) "IT Staff"
 9) "reportsto"
10) "6"
11) "birthdate"
12) "-62467200000000"
13) "address"
14) "923 7 ST NW"
.
.

Remove multiple fields

The remove_field transformation can also remove multiple fields at the same time if you specify them under a fields subsection. The example below is similar to the previous one but also removes the birthdate field:

source:
  db: chinook
  table: employee
transform:
  - uses: remove_field
    with:
      fields:
        - field: hiredate
        - field: birthdate
output:
  - uses: redis.write
    with:
      connection: target
      data_type: hash
      key:
        expression: concat(['emp:', employeeid])
        language: jmespath

If you query the data, you can see that it also omits the birthdate field:

> hgetall emp:8
 1) "employeeid"
 2) "8"
 3) "lastname"
 4) "Callahan"
 5) "firstname"
 6) "Laura"
 7) "title"
 8) "IT Staff"
 9) "reportsto"
10) "6"
11) "address"
12) "923 7 ST NW"
.
.

Using remove_field with add_field

The remove_field transformation is very useful in combination with add_field. For example, if you use add_field to concatenate a person's first and last names, you may not need separate firstname and lastname fields, so you can use remove_field to omit them. See Using add_field with remove_field for an example of how to do this.