It’s odd when you put your thoughts out into the world, how people see things differently. I posted two articles last month, one of which I thought had some good ideas, and one of which was a dry and painful walk though my own code. Based on statistics, people prefer dry and painful.

Fortunately for me, also based on my statistics, I don’t do this for a living and can write whatever makes me happy to write. I don’t know if I could handle writing mirror to my code stuff all the time.

Today I want to finish the refactoring of the code that I property tested in the last article. But before I do, I want better unit tests.

The code in question is the partiphify!/2 code from the morphix library. Morphix is a small Elixir library of convenience functions for Enums. partiphify!/2 takes a list l and an integer k, and divides the list as evenly as possible into k sublists. In the language of property testing,Given an input list l and an integer k,the output o of partiphify!/2 has 4 invariants:

1. o is a list of length k.
2. Every item in l is in some list in o.
3. The sum of the lengths of the sublists of o is equal to the length of l.
4. No sublist of o has a length that differs by more than 1 from any other sublist.

There are some other invariants, actually. o must be a list of lists, for example. I’m fairly certain that’s covered by the other 4, but strictly speaking it might be better to actually state it. I’m not a terribly strict person, so I’m going to stick with the invariants I’ve already got.

The code I’ve currently got in the library looks like this:

def partiphify!(list, k) when is_list(list) and is_integer(k) do
    ceil_div = fn a, b -> Float.ceil(a / b) end

    with chunk_size when chunk_size > 0 <-
           list
           |> Enum.count()
           |> Integer.floor_div(k),
         true <-
           list
           |> Enum.count()
           |> Integer.mod(k)
           |> ceil_div.(chunk_size) > 0 do
      list
      |> into_buckets(k, chunk_size)
      |> distribute_extra()
    else
      0 ->
        list = Enum.chunk_every(list, 1, 1, [])
        empty_buckets = k - Enum.count(list)
        Enum.reduce(1..empty_buckets, list, fn _, acc -> acc ++ [[]] end)

      false ->
        chunk_size =
          list
          |> Enum.count()
          |> Integer.floor_div(k)

        Enum.chunk_every(list, chunk_size, chunk_size, [])
    end
end

defp into_buckets(list, k, chunk_size) do
    chunks = Enum.chunk_every(list, chunk_size, chunk_size, [])
    extra_buckets = Enum.take(chunks, -(Enum.count(chunks) - k))
    k_buckets = chunks -- extra_buckets
    {extra_buckets, k_buckets}
end

defp distribute(list, buckets) do
    Enum.reduce(list, buckets, fn item, buckets ->
      [current_bucket | rest_of_buckets] = buckets
      new_bucket = [item | current_bucket]
      rest_of_buckets ++ [new_bucket]
    end)
end

defp distribute_extra({lists, buckets}) do
    with false <- Enum.empty?(lists) do
      [current_list | rest] = lists
      new_buckets = distribute(current_list, buckets)
      distribute_extra({rest, new_buckets})
    else
      _ -> buckets
    end
end

If that code doesn’t immediately make sense to you, that’s okay. I dissect it in a previous post. It made perfect sense when it was being written, and it’s written this way in order to avoid as many calls to ++ as possible. We’re going to refactor it to avoid any calls to ++.

But our unit tests are still in our way. Here’s an example unit test for this method:

iex> Morphix.partiphify!([:a, :b, :c, :d, :e, :f], 4)
[[:c], [:d], [:e, :a], [:f, :b]]

All the unit tests for this method are actual DocTests. Not good practice, but what’s even worse is that any change to the method that alters the order in which the output occurs will cause the test to fail, even though the order in which the output occurs is really not important at all.

DocTests are documentation. It’s in the name. They aren’t tests at all, really. So before we refactor, can we write some unit tests that actually test the method?

Let’s write some unit tests that test our invariants:

Invariant 1: given a list (l) and an integer (k), the output should be a list of length k.

We can probably write a test for that. Here is a test that is that test, but it isn’t a very good test:

test "the output list should be the same lenght as the input integer" do
    k = (1..100) |> Enum.random()
    l = (0..1000)
    |> Enum.to_list()
    actual = Morphix.partiphify!(l, k)
    assert Enum.count(actual) == k
end

We could improve the test by covering some more cases, for example, we probably want to handle the empty list and a few numbers that are larger or smaller than our value for k.

test "the output list should be the same length as the input integer" do
    k = (1..100) |> Enum.random()
    [0, 10, 13, 111, 683, 1013]
    |> Enum.each(fn length ->
      l = random_list(length)
      actual = Morphix.partiphify!(l, k)
      assert Enum.count(actual) == k
    end)
end

defp random_list(length) do
    (0..length)
    |> Enum.map(fn _ ->
      random_item()
    end)
end

defp random_item() do
    [random_string(), random_atom(), random_int()]
    |> Enum.random()
end

defp random_string() do
    [Faker.Lorem.sentence(), Faker.Lorem.word()]
end

defp random_atom() do
    Faker.Lorem.word()
    |> String.to_atom()
end

defp random_int() do
    :random.uniform(10000)
end

So we have a little randomization, and we’re working through a few different possible examples. This test passes with our current implementation, and we can make it break by changing the implementation to modify the value of k.

Invariant 2: given a list (l) and an integer (k), every item in l should be in the output.

We can leverage the work that we’ve already done to write this test.

test "all the items in the input list are in the output list" do
    k = (1..100) |> Enum.random()
    [0, 10, 13, 111, 1013]
    |> Enum.each(fn length ->
      l = random_list(length)
      actual = Morphix.partiphify!(l, k)
      assert Enum.all?(l, fn item ->
        Enum.any?(actual, fn sublist ->
          Enum.member?(sublist, item)
        end)
      end)
    end)
end

Breaking this test requires modifying our return value. I split the return list and returned the tail, which gave me a failing value. I like to see things fail.

Invariant 3: given a list (l) and an integer (k), the sum of the lengths of the output is equal to the length of l.

Again, our prior work helps us write a test here:

test "the sum of the lengths of the outputs is equal to the length of the input" do
    k = (1..100) |> Enum.random()
    [0, 10, 13, 111, 1013]
    |> Enum.each(fn length ->
      l = random_list(length)
      actual = Morphix.partiphify!(l, k)
      assert actual
            |> Enum.map(fn l -> Enum.count(l) end)
            |> Enum.sum() == Enum.count(l)
    end)
end

Invariant 4: No sublist of the output has a length that differs by more than 1 from any other sublist.

test "the lengths of the partitions are within 1 of each other" do
    k = (1..100) |> Enum.random()
    [0, 10, 13, 111, 1013]
    |> Enum.each(fn length ->
      l = random_list(length)
      actual = Morphix.partiphify!(l, k)

      {min, max} = actual
                   |> Enum.map(fn l -> Enum.count(l) end)
                   |> Enum.min_max()

      assert abs(max - min) <= 1
    end)
end

Are we ready to refactor yet? Finally, yes. A key point here is that, if, like me, you’re a proponent of test driven development, you might be thinking that we should have been ready to refactor as soon as we wanted to, because our tests should have already been there to allow us to refactor with wild abandon.

Exactly.

Preparing to refactor with better Unit Tests was originally published in Perplexinomicon on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you can think in properties, with or without a property testing library, your tests will support refactoring rather than being a barrier to it.

I didn’t figure out what this post is about until I finished it, but here it is: This post is about why and how Unit Testing can be broken in your application. It’s about the difference between good DocTests and good Unit Tests. And ultimately, it’s about thinking in properties. We take a while to get there, and there are some detours, but ultimately, that’s where we’ll land.

So a bit ago I wrote about the partiphify method, a part of the morphix library for Elixir. My conclusion was that I wasn’t particularly happy about the code I originally wrote for that method. It seemed clumsy. So I wanted to refactor it.

But that led to another problem, which is example code, and unit testing. What the partiphify! method does is to take a list and divide it up into a fixed number of sublists of equal length, or, as equal length as possible. Given a list like [1, 2,3,4,5,6,7] and the integer 3, partiphify! should return three lists, which might be [[1,2,3], [4,5], [6,7]], but not [[1,2,3],[4,5,6],[7]]. The second partitioning doesn’t keep the lists as close in size as possible, but the first partitioning does.

The thing about writing example code and unit tests for this kind of a method is that implementation matters. Here’s an example from the DocTests:

iex> Morphix.partiphify!([:a, :b, :c, :d, :e], 4)
[[:b], [:c], [:d], [:e, :a]]

Any implementation that ordered the list differently (for example [[:a,:b], [:c],[:d],[e]], would fail that unit test, even though the implementation might be correct in all the ways that matter. Which is okay in a doctest, which is more of an example than a test, but not great in a unit test.

Which made this a great time to add property tests to the library. A property test is a test where the developer determines things that are true about the code (these are called invariants) and example inputs are generated by the testing code and tested against these invariants.

So the first task involved in writing a property test is to figure out what your invariants are. In the case of partiphify!, we know that the function should take two arguments, a list and an integer, and it should divide the list into k sublists that are as close in size to each other possible.

Thinking about it a bit, we (hat tip to Jessica Kerr) came up with a list of 4 properties that make up the partiphify! function, given that there is an input list l, an input integer k, and an output value out.

1. The length of out should be equal to k.
2. Every item in l should be in out.
3. The sum of the lengths of the items in out should equal the length of l.
4. The length of each list in out should be at most +-1 of any other list in out.

And that’s it. If those properties match, then we know that we have divided our input list into partitions correctly. All the items are in the output, no item is in more than one sublist, we have the right number of sublists, and the sublists are balanced.

I wrote property tests, initially, using PropCheck, and then later rewrote those same tests using StreamData. PropCheck is a property testing library for Elixir that provides a wrapper around PropER, a property based testing library for Erlang.

To set up either library, you first want to add the library to your deps(), then add the application to your extra_applications: [] or applications: [] Keyword lists. I wanted to make sure these only ran in test, so for my application the changes to my mix.exs looked like this:

def application do
  [applications: applications(Mix.env())]
end

defp applications(:test) do
  applications(:default) ++ [:stream_data, :propcheck]
end

defp applications(_) do
  [:logger]
end

defp deps do
  [
    {:excoveralls, "~> 0.8", only: [:dev, :test]},
    {:ex_doc, "~> 0.18", only: :dev},
    {:credo, "~> 0.9.1", only: [:dev, :test]},
    {:stream_data, "~> 0.5.0", only: [:dev, :test]},
    {:propcheck, "~> 1.1", only: [:dev, :test]}
  ]
end

I have the changes for adding both StreamData and PropCheck here, but really you only need one of them.

In your test itself, you’ll want to add the dependencies:

defmodule PartiphifyPropTest do
  use ExUnit.Case
  use PropCheck

Or:

defmodule PartiphificationTest do
  use ExUnit.Case
  use ExUnitProperties

So for our first property, this is the test I came up in PropCheck:

# number of partitions should be correct
property "number of partitions is correct" do
  forall {list, p} <- {list(), integer(1, 10)} do
    partitioned = Morphix.partiphify!(list, p)
    Enum.count(partitioned) == p
  end
end

The property block acts like a test statement. The first line is the forall statement, which names two variables, list and p, and assigns generators to those variables. The first generator is list(), which will produce lists of any valid term, and integer(1,10), which will produce integers in the range of 1 to 10. The general form of this statement is

forall {var_1, var_2, …, var_n} <- {generator_1, generator_2, ..., generator_n}
....
end

Inside the forall statement, we want to have code that results in either a true or a false value. In this case, once we’ve partitioned the input list, we want to see that the number of partitions is equal to the input integer.

In StreamData, the test looks like this:

property "number of partitions is correct" do
  check all list <- list_of(term(), min_length: 0, max_length: 100),
            part <- integer(1..27) do
    assert list
           |> Morphix.partiphify!(part)
           |> Enum.count()
           |> Kernel.==(part)
  end
end

So, again, we have a property statement, and inside that statement we have a check all statement instead of a forall statement. The check all statement adds variables one at a time rather than in tuples. So in this case, the first variable will be list, which is generated by list_of(term(), [options]), so will be lists of between 0 and 100 of type term() , and part, which is generated by integer(1..27). The general form of this statement is:

 check all var_1 <- generator_1,
           var_2 <- generator_2,
           ...,
           var_n <- generator_n do
    ...
end

Inside the check all statement, we use assert to test the property. The code here looks a little different because I’m using a pipechain, but all the same code is being called in the same order as in the PropCheck example.

Our second invariant is that every item that is in the original list is in the partitions, somewhere. For this property, the generators are the same, but the code that checks the invariant is fairly complex and might be worth looking at:

assert Enum.reduce(list, true, fn list_item, acc_a ->
  in_a_list = 
    Enum.reduce(partitioned, false, fn partition, acc_b ->
      Enum.member?(partition, list_item) || acc_b
  end)

  in_a_list && acc_a
end)

Warning: In this next section I go pretty deep in the reduction logic for this invariant. I’m doing this mostly because using Enum.reduce/3 is not always intuitive for me, and I found it initially hard to wrap my head around it for uses other than adding up numbers. If you find the invariant code blindingly obvious, feel free to skip the next few paragraphs.

Let’s look at this code in detail.

Starting with the outside reduction, we’re reducing on true, and we can simplify a bit… let’s say I wanted to establish that every integer in a list of integers was greater than 10. I could write something like this:

small_integers = Enum.filter(list, fn i -> i <= 10 end)
less_than_tens = Enum.empty?(small_integers)

and less_than_tens would be either true or false. Or I could write this:

less_than_tens = 
  Enum.reduce(list, true, fn i, acc -> i <= 10 && acc end)

Where once some integer has flipped the accumulator from true to false, it will remain false for the rest of the list.

The outside reduction does just that, we look at each item in the input list, and check to see whether it is in a partition. If it is not, the value in_a_list && acc_a will become false, and it stays false for the remainder of the list.

What about finding the value of in_a_list? In this case, I have a list of lists, and I want to see if some value is in one of those lists. There are a lot of ways to do this as well. For example, I could write:

(i is some integer we're checking)
lists_of_bools =
  Enum.map(list_of_lists, fn list -> Enum.member?(list, i))
in_a_list = Enum.member?(list_of_bools, true)

If true is a member of our list of bools, than we found i in some list.

Alternately, I can use a reduction:

in_a_list = 
  Enum.reduce(list_of_lists, false, fn list, acc -> 
    Enum.member?(list, i) || acc
  end)

In this case, because of the || statement, once the item is found in a list, the accumulator becomes true, and it will remain true even if the item is not found in any subsequent list.

End of Warning Section.

So our outer reduction looks at each item in the input list, and if every item is found in any partition (by the inner reduction) it will return true as it’s accumulated value. If any item is not found, we return false and the invariant will fail. Notice, this invariant would pass under a number of bad implementations. For example, if partiphify had the following implementation:

def partiphify!(list, k), do: [[list]]

Then this invariant would pass. Similarly, if the implementation were:

def partiphify!(list, k) do: (1..k) |> Enum.map(fn _i -> [] end)

Then the first invariant would pass.

def partiphify!(list, k) do
  emptys = (2..k) |> Enum.map(fn _i -> [] end)
  [list] ++ emptys
end

Would pass both of these property tests, and the third invariant as well. The only invariant it would fail would be the fourth one, because the differences in sizes between partitions would exceed 1. I could pass that invariant and the 1st and 2nd invariants with this code:

def partiphify!(list, k), do: (1..k) |> Enum.map(fn _i -> [list] end)

Now all the partitions are the same size, and I have the right number of partitions, and every item in the input list is in some partition. But I’ll fail the third invariant, because I’ll have k * Enum.count(list) items if I add up the count of the partitions, and I only want Enum.count(list) items.

Each invariant limits the implementation in different ways, and each of them limits the implementation in ways that are relevant to the function. Contrast this to our DocTest:

iex> Morphix.partiphify!([:a, :b, :c, :d, :e], 4)
[[:b], [:c], [:d], [:e, :a]]

The DocTest gives an output example that meets all four invariants, but there are a lot of irrelevant factors in it as well. It tests ordering, and grouping, and the ordering inside groupings. We don’t care about any of those things, and there are many correct implementations of the method that would fail this test. Which means that rather than being a help to refactoring, this test stands in the way of refactoring!

An interesting question is whether we could write better unit tests around this code that would help us refactor. And of course, we could test these properties without using a property test library. We could even add some randomness to the test.

test "the number of partitions should be correct" do
  list = [1,2,3,4,5,6,7,8,9,10]
  k = (1..27) |> Enum.random()
  parts = Morphix.partiphify!(list, k)
  assert Enum.count(parts) == k
end

With a little work, I could (and might) generate random lists as well. I could even change the implementation to go through every number from 1 to 27 against every one of a number of randomly generated lists, or hit some set of numbers I though was import (1, the list count / k, list count-1, stuff like that). Once I built those cases for number of partitions, it would be a simple matter to test the other invariants.

These would be better Unit Tests. They might not make great DocTests, but they would be much better unit tests than testing that the input exactly matches the output. A good DocTest should be a simple example that tells people what will happen if they use the function being documented. In Elixir, it’s there to clarify and support your documentation. It’s not there to test your code.

A good Unit Test is there to test your code. So the goals of the two tests are not the same. Something I’ve often heard and have sometimes said is that “If your tests stand in the way of refactoring, they probably aren’t very good.” You’ve probably also heard “Don’t test the implementation” when writing unit tests. But what does the second statement mean, and how do we avoid having tests stand in the way of implementation?

I think the answer is this: “Try to think in properties, not examples”. That doesn’t mean adding a property testing library to your code. It means understanding what your invariants are for a piece of code, and attempting to test those invariants. It means not testing things that are not invariants as well!

.If you can think in properties, with or without a property testing library, your tests will support refactoring rather than being a barrier to it.

Preparing to Refactor with Property Tests was originally published in Perplexinomicon on Medium, where people are continuing the conversation by highlighting and responding to this story.

Happily, for the most part, I do.

Except for one function, which for reasons that are actually reasonable I named partifify. partifify/2 is part of my Morphix library, which is a tiny library of functions to help you deal with Enumerables in Elixir. I started writing it as a port of another library that I wrote several years ago in Ruby, but it’s grown since then into a fairly nice library that a surprising (read, greater than zero) number of people download and use.

Morphix has functions for getting rid of nil values, converting string keys to atoms (with safe options to keep from creating new atoms) converting atom keys to strings, flattening, checking if two lists have all the same items even if they are in different orders, and of course partifify. Most of these functions have a recursive form which is nice if you need to convert a bunch of atom keys to strings in a nested data structure, or you want to know if a nested map has all the same elements at every level and one of those levels has a list which isn’t strictly ordered.

partifify/2 takes a list and divides it into k sublists. Here’s an example:

iex> Morphix.partiphify!([1,2,3,4,5,6], 4)  
[[3], [4], [5, 1], [6, 2]]

The key thing about partifify is that the lists should be as balanced as possible. If I divide a list of 14 items into 4 sublists, I don’t want three lists of size 4 and one list of size 2. I want two lists of size 3 and two lists of size 4. For example:

iex(70)> Morphix.partiphify!([:a, :b, :c, :d, :e, :f, :g, :h, :i, :j, :k, :l, :m, :n], 4)

[[:g, :h, :i], [:j, :k, :l], [:m, :a, :b, :c], [:n, :d, :e, :f]]

Why do I want balanced lists? That goes back to what I was doing when I wrote this function. The system I was building was drawing data from a number of sources, and comparing that data to a sink. This was an audit system, essentially, checking the correctness of one system against another. Because there were a number of sources, we wanted to distribute the work across several servers, but because the number of sources wasn’t fixed, we wanted to distribute this work flexibly and equally across all servers. Oh, and the number of servers was also not fixed.

So, given a list of sources, divide them up evenly to the number of servers in operation and let each server begin its auditing work. In a word, partifify/2.

That’s what we wanted to do. So how were we going to do it? I remember having the problem. I remember solving the problem. I don’t remember HOW I solved the problem.

Let’s start with the admission that it is entirely possible that there exists a function in Elixir that can take a list of 6 items and output four balanced sublists. I may just be very bad at reading documentation and missed something super obvious. I will say that I looked, and I looked hard, and I couldn’t find a simple way to do this with Enum.chunk*.

With that admission out of the way, let’s look at the basic algorithm.

The algorithm runs in a few steps.

1. Find the size of the smallest sublist.
2. Divide the list into a number of sublists based on the smallest sublist size.
3. If there are extra sublists, distribute them evenly across the other sublists in a round robin fashion.

Is there an easier algorithm? How about:

1. Make n sublists that are empty.
2. Distribute the original list across the n sublists in a round robin fashion.

We’ll take a look at the original implementation, and then at an alternate implementation.

Find the smallest sublist

Finding the smallest sublist is done with some Integer math:

list |> Enum.count() |> Integer.floor_div(k)

Where k is the desired number of partitions. If the result of that is greater than zero, we move on:

with chunk_size when chunk_size > 0 <- 
    list 
    |> Enum.count() 
    |> Integer.floor_div(k), 
  true <- list 
          |> Enum.count() 
          |> Integer.mod(k) 
          |> ceil_div.(chunk_size) > 0

ceil_div is a helper function that just takes the ceiling of dividing two numbers. If we look at an input of ([1,2,3,4,5,6], 4), at this point our value for chunk size is 1 and the output of the second with condition is true, because the 6 modulo 4 is 2, and the ceiling of 2/1 is 2, which is greater than zero. What we’re establishing with this code is that we’re going to have uneven sublists, and we’re going to have more than one sublist. I think the ceil_div function is unnecessary here, because the only time the value is going to be less than or equal to zero is if Integer.mod(k) is zero. chunk_size can’t be negative unless k is negative, and we shouldn’t allow that as an input. We don’t need to be able to divide lists into negative numbers of sublists.

With our smallest sublist in hand, we move to the next line:

...do      
list      
|> into_buckets(k, chunk_size)
|> distribute_extra()

These are steps two and three of the algorithm, dividing the list and distributing the extra.

Divide the list into sublists of the smallest sublist size

This is done using the following function:

defp into_buckets(list, k, chunk_size) do
chunks = Enum.chunk_every(list, chunk_size, chunk_size, [])
extra_buckets = Enum.take(chunks, -(Enum.count(chunks) - k))
k_buckets = chunks -- extra_buckets
{extra_buckets, k_buckets}
end

We use the Enum.chunk_every/4 method to split the original list into sublists based on the smallest chunk size, then we split the result into two lists… one, the extra_buckets are additional lists that we need to distribute, the other, the k_buckets, will be the sublists that we are distributing to. At this point, for the inputs ([1,2,3,4,5,6], 4), we have the values:

extra_buckets = [[5],[6]]]
k_buckets = [[1],[2],[3],[4]]

It took me a minute looking at this code to figure out what:

extra_buckets = Enum.take(chunks, -(Enum.count(chunks) - k))

Meant, but it just means: “take the additional buckets off the end of the bucket list and assign them to a variable”.

Distribute the extra sublists

So we’ve returned a tuple to our initial function that contains extra lists and sublists. We want to distribute the extra lists efficiently across the sublists.

defp distribute_extra({lists, buckets}) do
  with false <- Enum.empty?(lists) do
    [current_list | rest] = lists
    new_buckets = distribute(current_list, buckets)
    distribute_extra({rest, new_buckets})
  else
    _ -> buckets    
  end  
end

lists is an unfortunate name for the first variable in this function, because those are the extra lists. If there aren’t any, we return the buckets and we’re done. If there are extra lists, we take the first extra list and distribute it across all the buckets, then call this function recursively with any remaining extra lists. In our case of making 4 sublists from the list [1,2,3,4,5.6], we call this function with {[[5],[6]], [[1],[2],[3],[4]]}. So the first call to distribute will be with [5], [[1],[2],[3],[4]].

defp distribute(list, buckets) do
  Enum.reduce(list, buckets, fn item, buckets ->
    [current_bucket | rest_of_buckets] = buckets
    new_bucket = [item | current_bucket]
    rest_of_buckets ++ [new_bucket]
  end)
end

Step one of the reduction is to grab the first bucket off of our list of buckets. In the example we’re using, the first bucket is [1]. Then we take the current item (in our example, 5) and add it to the front of the bucket. That bucket is moved to the back of the list, so on the next round of the reduction, the accumulator will be [[2], [3], [4], [5, 1]]. The function returns the accumulator when it has processed all the items in the list, and the recursive distribute_extra function calls itself back with any remaining extra lists and the new buckets. The first recursion will be called with:

{[[6]], [[2], [3], [4], [5, 1]]}

and the last call will be made with:

{[], [[3], [4], [5, 1], [6, 2]]}

That last all will terminate the recursion and return our completed buckets.

Why do things this way?

Looking back, we had a couple of motives for writing the code in this way. One was just that this is how it evolved and we never got around to refactoring it. We started trying to use Enum.chunk_by and a lot of decisions resulted from that initial idea. Another is that ++ is an expensive operation, and we wanted to minimize its use. In the simple example we just went through, ++ is called twice. If we naively used ++ to implement a fully round robin solution based around distribute and distribute_extra, there are always n calls to ++ . That probably is a premature optimization, though, because with practical numbers of lists that’s just not enough calls to matter.

It’s probably worth re-examining this code and trying to refactor it. If I were to encounter this library code as a third party, I don’t know that I’d be impressed with this implementation at all. I wrote it, and looking back, I have questions.

A Function of Parts was originally published in Perplexinomicon on Medium, where people are continuing the conversation by highlighting and responding to this story.

1. Review of the previous two posts.
2. Small world networks have short distances between similarly connected nodes.
3. You can extract a small world of data from a small world network with a breadth-first search.
4. Indexes are your friend.
5. The Postgres COPY command exports data as well as it imports data.

Example Code

Introduction

This is part 2 of 2 in a 2 part sub-series and the third in an N part series about microservices.

In the first post of the series, we downloaded three tab separated files from the Internet Movie DataBase and loaded them into a single Postgres Database, in three tables, title_basics, name_basics, and title_principals. We showed a little bit of how these tables related to each other, for example, we can use these three tables to find information about movies (titles) and actors (names) by way of the title_principals_table. So if we want to learn the names of the principal actors in the movie "Thor: The Dark World":

data_imdb=# select primary_name from name_basics
data_imdb-# where nconst in(
data_imdb(# select p.nconst from title_principals p,
data_imdb(# title_basics t
data_imdb(# where t.primary_title ilike('thor%dark%')
data_imdb(# and t.title_type='movie'
data_imdb(# and p.tconst=t.tconst
data_imdb(# and p.category ilike('act%'));
  primary_name    
-------------------
 Natalie Portman
 Stellan Skarsgård
 Tom Hiddleston
 Chris Hemsworth
(4 rows)

We also created a Phoenix application called DataMunger with three Schemas, TitlePrincipal, TitleBasic, and NameBasic. Using these Schemas and Ecto, we can find all the movies in which Natalie Portman played a principal role:

iex(2)> q = Ecto.Query.from(
...(2)> n in DataMunger.NameBasic,
...(2)> where: ilike(n.primary_name, "Natalie Portman"),
...(2)> select: n.nconst)

iex(4)> titles_from_nconst = fn(n) ->  
...(4)> Ecto.Query.from(               
...(4)> p in DataMunger.TitlePrincipal,
...(4)> where: p.nconst == ^n and      
...(4)> ilike(p.category, "act%"),     
...(4)> select: p.tconst) |>           
...(4)> DataMunger.ImdbRepo.all() end

iex(8)> q |> 
...(8)> DataMunger.ImdbRepo.one() |>
...(8)> titles_from_nconst.() |>         
...(8)> Enum.map(fn(p) ->                
...(8)>  Ecto.Query.from(                 
...(8)>  t in DataMunger.TitleBasic,      
...(8)>  where: t.tconst == ^p,           
...(8)>  select: t.primary_title) |>      
...(8)>  DataMunger.ImdbRepo.all() end)   
[["Leon: The Professional"], [...], ...]

In the second post, we created three new Schemas, Actors, Movies, and ActorMovies with matching database tables, and we created changesets to convert from one repo to the other.

In a future post, we’ll be looking into the problem of finding the same relationships when the three tables are split across three microservices. To do this, we’re going to split our database into three databases, and launch three services on Heroku to act as API backends. We can’t do that as things stand, because I don’t want to pay to host several million rows in a Postgres database on Heroku. In fact, I’d like to stay on the free tier, so we’re going to need to lose almost all of the data.

We can’t just truncate the tables, because we want to maintain relationships.

title     title        name     title      title
          name                  name         
-----     -----        ----     ----       -----
t1 -----> t1:n1 -----> n1 ----> n1:t1 ---> t1 
    \ \-> t1:n4 --     n2   \-> n1:t2 ---> t2 
     \--> t1:nb - \    n3   |-> n1:t3 ---> t3   
                 \ \-> n4-  |-> n1:t4 ---> t4
                  |    n5 \ \-> n1:t5 ---> t5
                  |    n6  \--> n4:t1      t6
                  |    n7   |-> n4:t2      t7
                  |    n8   |-> n4:t4      t8
                  |    n9   \-> n4:tf --   t9
                  |    na   /-> nb:t1  |   ta
                   \-> nb ----> nb:t4  |   tb
                            \-> nb:t5  |   ta
                                       |   tb
                                       |   tc
                                       |   td
                                       |   te
                                       \-> tf

In the figure, t1 has 3 principal actors, n1, n4, and nb. Between the three actors, we have 6 movies, which would lead us to select even more actors, and so on. But if we only selected, say, 9 rows from each database, we would end up with titles in the title selection that didn't show up in the title_name selection, or title_name selections that didn't show up in names. So it's important that we select data algorithmically when trying to extract a useful subset of the data.

In the rest of this post, we’ll be using Ecto to get a subset of the data that meets these constraints.

Getting the Data Subset

There are a couple of approaches we could use, and probably there’s an optimal algorithm for getting just exactly the data we want, but this is a case where the intuitive approach will probably do. Looking at the figure, it seems that by just following the arrows from name -> title_principal -> title -> title_principal -> name and so on, we should be able to rapidly expand our selection of names and titles until we have a nicely connected selection across all three databases. In this post, we'll write queries to extract the records we want from the initial IMDb database, transform them, load them into our new databases, and then export those databases to CSV files.

Extracting the Data

Extracting the data means solving the problem of getting a sparse, connected data set of fewer than 10000 rows for the largest database. We’ll start in Postgres and explore the data. The queries we use to do this will be refined and converted into Ecto when we have a better idea what we want to do.

Just as a reminder, we’re dealing with six tables here:

The IMDb data is loaded into an ImdbRepo, into three tables, name_basics, title_basics, and title_principals. title_basics contains information about video productions: movies, documentaries, whatever. Each title_basics record has a tconst identifier and other information, such as title, release date, and title_type. name_basics has the same sort of information about people, name, nconst, birth_year. title_principals is a join table, it keeps the tconst field and the nconst field from title_basics and name_basics, as well as information about what job the person had on the film. The tables we want to load into are the actor, movie, and actor_movie tables in the Repo.

Q for Query

Let’s start by asking a simple question: How many movies has Natalie Portman starred in? We’ll start with getting a universal key:

data_imdb=# select nconst from name_basics
data_imdb-# where primary_name 
data_imdb-# ilike('natalie portman');
  nconst   
-----------
 nm0000204

Then use that universal key to answer the question:

data_imdb=# select count(*) from title_principals p
data_imdb-# join title_basics t
data_imdb-# on (p.tconst=t.tconst)
data_imdb-# where nconst='nm0000204' and
data_imdb-# category ilike('act%') and
data_imdb-# title_type='movie';
 count 
-------
    36
(1 row)

That’s a fair number of movies. How many costars does that get us?

data_imdb=# select count(distinct p.nconst) 
data_imdb=# from title_principals p where 
data_imdb=# p.tconst in
data_imdb=# (select p.tconst from title_principals p
data_imdb=# join title_basics t
data_imdb=# on (p.tconst=t.tconst)
data_imdb=# where nconst='nm0000204' and
data_imdb=# category ilike('act%') and
data_imdb=# title_type='movie');
 count 
-------
   276
(1 row)

So, if we want to limit our extraction to 10,000 rows, we won’t be able to go 2 degrees from the source before passing our limit, because 200 squared is 40,000. If we include all the movies for all the actors, we’re going to end up with a network that has a small number of extremely tightly coupled nodes, which isn’t really the same shape as the IMDb database.

In the IMDb data, yes, there are 276 actors who are costars of Natalie Portman, but there are 1.6 million actors who are not. But for most of these 1.6 million actors, there is a fairly short path to Natalie Portman, or to any other actor. If an actor has 100 costars in a career, and each of these has a 100 costars, that’s 10,000 actors who have co-starred with a co-star of the first actor, and 1,000,000 who have co-starred with a co-star of a co-star. A network like this, where most nodes are not neighbors, but the path length between any two nodes is short, is called a “small world” network. This is the property that we want to maintain in our own data. One thing we could try is limiting the number of costars by limiting the number of movies:

data_imdb=# select count(distinct p.nconst) 
data_imdb=# from title_principals p where 
data_imdb=# p.tconst in
data_imdb=# (select p.tconst from title_principals p
data_imdb=# join title_basics t
data_imdb=# on (p.tconst=t.tconst)
data_imdb=# where nconst='nm0000204' and
data_imdb=# category ilike('act%') and
data_imdb=# title_type='movie' limit 3);
 count 
-------
   22
(1 row)

22 cubed is 10,648, so theoretically we won’t even be able to get 3 degrees from the source before passing our limit. However, there’s probably a good deal of overlap (that’s also a characteristic of small-world networks), so 3 seems like a reasonable starting place for limiting movies per actor.

At this point we’re passing my ability to write SQL, so lets see if we can do some exploring in Ecto. A query for finding three movies for an actor:

iex(15)> tfa = fn(a) -> Ecto.Query.from(                 
...(15)> p in DataMunger.TitlePrincipal,                 
...(15)> join: t in DataMunger.TitleBasic,               
...(15)> on: p.tconst == t.tconst,                       
...(15)> where: p.nconst == ^a and                       
...(15)> t.title_type=="movie" and                       
...(15)> ilike(p.category, "act%"),                      
...(15)> select: p.tconst,                             
...(15)> limit: 3) |> 
...(15)> DataMunger.ImdbRepo.all() end 
#Function<6.52032458/1 in :erl_eval.expr/5>
iex(16)> tfa.("nm0000204")
["tt2798920", "tt2180351", "tt0947798"]

A query for actors in a movie:

iex(20)> aft = fn(t) -> Ecto.Query.from(
...(20)> p in DataMunger.TitlePrincipal,
...(20)> where: p.tconst == ^t and      
...(20)> ilike(p.category, "act%"),     
...(20)> select: p.nconst) 
...(20)> |> DataMunger.ImdbRepo.all() end          
#Function<6.52032458/1 in :erl_eval.expr/5>
iex(21)> aft.("tt2798920")
["nm0000204", "nm0000492", "nm1935086", "nm0938950"]

The movies for actor tfa/1 query takes about 5 seconds to return, and the aft/1 query takes half a second. We'll be happier if we add a couple of indexes.

imdb# create index on title_principals 
imdb# (tconst, nconst, category);
imdb# create index on title_principals 
imdb# (tconst, category);
imdb# create index on title_basics
imdb# (tconst, title_type);
imdb# create index on name_basics
imdb# (nconst);

On my machine, that drops both queries into the 1–2 millisecond range. The last index on name_basics we won't need until the end, but we might as well add it now.

Then we can write a function to chain our two queries together.

iex(81)> xs_from_ys = fn(vals, f) ->
...(81)> Enum.flat_map(vals, fn(v) ->
...(81)> f.(v) end) |> Enum.uniq end
#Function<12.52032458/2 in :erl_eval.expr/5>
iex(82)> aft.("tt0110413") |> xs_from_ys.(mfa)
["tt0085426", "tt0095250", "tt0099789", "tt0091954",
"tt0093776", "tt0096294", "tt0080605", "tt0081145", 
"tt0082177", "tt0110413", "tt0120915", "tt0121765"]

So now we can find how many movies and titles we have at various distances from our seed actor.

iex(103)> tfa.("nm0000204") |> xs_from_ys.(aft) |>
...(103)> xs_from_ys.(tfa) |> xs_from_ys.(aft) |> 
...(103)> xs_from_ys.(tfa) |> xs_from_ys.(aft) |>
...(103)> xs_from_ys.(tfa) |> xs_from_ys.(aft) |>
...(103)> Enum.count()
2927
iex(103)> tfa.("nm0000204") |> xs_from_ys.(aft) |>
...(103)> xs_from_ys.(tfa) |> xs_from_ys.(aft) |> 
...(103)> xs_from_ys.(tfa) |> xs_from_ys.(aft) |>
...(103)> xs_from_ys.(tfa) |> xs_from_ys.(aft) |>
...(103)> xs_from_ys.(tfa) |> xs_from_ys.(aft) |>
...(103)> Enum.count()
12325

This pattern of moving from movie to actor to movie to actor and so on should look familiar, we’re basically recreating the logic from our ascii figure above.

After player around with this a bit, it seems like we can safely go about 4 degrees away from the seed if we limit the number of movies to three. This is imprecise, but it is a strategy that should build a sparse, connected, small worldy network.

The Professional (Query)

We’ll clean up our queries and add them to a GraphTraversemodule.

defmodule DataMunger.GraphTraverseTest do
  use ExUnit.Case, async: true
  alias DataMunger.GraphTraverse
  alias DataMunger.ImdbRepo
  alias DataMunger.TitleBasic
  alias DataMunger.TitlePrincipal
              
  describe "movies for actor" do
    test "given a name, finds 3 movie tconsts" do
      ["nm0000203", "nm0000702", "nm0000204", "nm0000102"]
      |> Enum.each(fn(n) ->
        GraphTraverse.movies_for_actor(n)
        |> Enum.map(fn(m) ->
          ImdbRepo.get(TitleBasic, m)
        end)
        |> Enum.map(fn(mov) ->
          assert mov.title_type == "movie"
        end)
        |> (&(assert Enum.count(&1) == 3)).()
      end)
    end
  end
end

Our testing strategy is basically to call back to TitleBasic to make sure we got the right number of movies. We should also check that the movies we got back directly related to each name as an Actor in the TitlePrincipal table. We won't show that test here, but it is in the example code.

We can pass these tests with the following code:

defmodule DataMunger.GraphTraverse do
  alias DataMunger.TitlePrincipal
  alias DataMunger.TitleBasic
  alias DataMunger.ImdbRepo
  import Ecto.Query, only: [from: 2]
              
  def movies_for_actor(actor, count \\ 3) do
    from(p in TitlePrincipal,
    join: t in TitleBasic,
    on: p.tconst == t.tconst,
    where: p.nconst == ^actor and
    t.title_type=="movie" and
    ilike(p.category, "act%"),
    select: p.tconst,
    limit: ^count)
    |> ImdbRepo.all()
  end
end

The code and tests for the complementary actors_for_moviefunction and the principals_from_movies function are very similar so we won't show them here. We are interested in the code that puts these together. We'll test the functionality of finding many movies and actors from a single actor. In test/data_munger/graph_traverse_test:

describe "gets actors and movies from one actor" do
  test "returns uniq collections" do
    {actors, movies} = ["nm0000204"]
                       |> GraphTraverse.movies_and_actors([], 2)
    actors
    |> Enum.each(fn(a) ->
         assert String.match?(a, ~r/nm[0-9]{7}/)
    end)
    assert actors 
           |> Enum.uniq()
           |> Enum.count() == actors 
                              |> Enum.count()
    assert Enum.count(actors) > 18
    movies
    |> Enum.each(fn(m) ->
        assert String.match?(m, ~r/tt[0-9]{7}/)
    end)
    assert movies 
           |> Enum.uniq()
           |> Enum.count() == movies 
                              |> Enum.count()
    assert Enum.count(movies) > 6
  end
end

The API we’re assuming here is that the movies_and_actorsfunction is going to take three arguments, actors, movies, and a depth. The idea is to find all the costars of the first set of actors, then the costars of all the costars (for three movies) out to a certain search depth into the graph. In /lib/data_munger/graph_traverse.ex:

def movies_and_actors(actors, movies, 0) do
  {actors, movies}
end
              
def movies_and_actors(actors, _movies, depth) do
  new_movies = actors
  |> Enum.flat_map(fn(a) -> 
       movies_for_actor(a) end)
  |> Enum.uniq()
  new_actors = new_movies
               |> Enum.flat_map(fn(m) -> 
                    actors_for_movie(m) end)
               |> Enum.uniq()
  movies_and_actors(new_actors, new_movies, depth-1)
end

A bit of explanation: we’re defining this as a recursive function, using the first pattern (actors, movies, 0) as the terminating condition, and the second pattern (actors, _movies, depth) is the recurrent procedure. In the recurring procedure, we use the list of actors to find 3 movies for each actor, and then we get all the actors for each of those movies. We go to the next recurrence with this new group of actors and movies. The depth variable keeps track of how deep we've gone into the recursion, ensuring that we'll exit when that value hits zero.

This solution could be improved in a couple of ways. The most obvious is that we don’t use the results of previous searches to limit the current search, so we make a lot of repetitive calls to the database. In fact, we only carry the movies from one recursion to the next so that we can return them when we hit 0. If we were concerned with performance we’d have to use a technique such as memoization to avoid making repetitive calls, but since we aren’t the naive version of the algorithm is fine. It’s worth noting in any case, because recursion is used a lot in Elixir code and it’s good to be aware of when you are falling into a common recursion pitfall.

We can improve this slightly by adding at least one other seed actor and extract our sparse subgraph:

iex(326)> DataMunger.GraphTraverse.movies_and_actors(
...(326)> ["nm0000204", "nm0000702"], [], 4) 
{["nm0427136", "nm0301556", "nm0473218", ...],
["tt0084627", "tt0085426", "tt0275909",  ...]}
iex(327)> {actors, movies} = v()
{["nm0427136", "nm0301556", "nm0473218", ...],
["tt0084627", "tt0085426", "tt0275909",  ...]}
iex(329)> princs = movies |>                               
...(329)> DataMunger.GraphTraverse.principals_from_movies()
[714013, 714015, 714014, ...]
iex(330)> Enum.count(actors)
4787
iex(331)> Enum.count(movies)
1966
iex(332)> Enum.count(princs)
8137

It’s not perfect, but it’s fine.

Extract, Transform, Load

We’re now just about done. All that’s left is using the results from the functions in GraphTraverse to load data into the schemas we created in the last post. We'll look in depth at just title_principal -> movie_actor. Again, we start with a test (test/data_munger/etl_test.exs):

defmodule DataMunger.EtlTest do
  use ExUnit.Case, async: true
  require Ecto.Query
  alias Ecto.Query
  alias DataMunger.Etl
  alias DataMunger.Repo
  alias DataMunger.ImdbRepo
  alias DataMunger.Movie
  alias DataMunger.TitlePrincipal
  alias DataMunger.Actor
  alias DataMunger.MovieActor
              
  setup do
    :ok = Ecto.Adapters.SQL.Sandbox.checkout(DataMunger.Repo)
  end
              
  test "adds the actor-movies" do
    DataMunger.Etl.load_from_seed(["nm0000204"], 1)
              
    ids = [937006, 937007, 937009, 937008, 1024551, 
           1024552, 1024554, 1024553, 1029911, 
           1029913, 1029914, 1029912]
    title_principals = Query.from(
                       p in TitlePrincipal,
                       where: p.id in ^ids,
                       select: {p.nconst, p.tconst})
                       |> ImdbRepo.all()
    movie_actors = Query.from(
                   ma in MovieActor, 
                   select: {ma.nconst, ma.tconst})
                   |> Repo.all()
                  
    assert title_principals -- movie_actors == []
    assert movie_actors -- title_principals == []
    assert Enum.count(movie_actors) == 12
  end
end

So obviously there’s some cheating going on here, I’m counting on the movies being returned from the database to always be returned in order, and I checked before hand to see which ids I would get back from a call to GraphTraverse.movies_and_actors/3. But essentially, I want to know that I'm creating records that match on the relevant transformed fields (nconst and tconst). We put all of this together in data_munger/lib/data_munger/etl.ex:

defmodule DataMunger.Etl do
  alias DataMunger.GraphTraverse
  alias DataMunger.TitlePrincipal
  alias DataMunger.MovieActor
  alias DataMunger.Repo
  alias DataMunger.ImdbRepo
  import Ecto.Query, only: [from: 2]
              
  def load_from_seed(seeds, depth) do
    {names, movies} = GraphTraverse.movies_and_actors(seeds, [], depth)
    principals = GraphTraverse.principals_from_movies(movies)
    from(p in TitlePrincipal,
    where: p.id in ^principals,
    select: p)
    |> ImdbRepo.all()
    |> Enum.each(fn(p) ->
         movie_actor_from_title_principal(p)
    end)
  end
              
  defp movie_actor_from_title_principal(principal) do
    %MovieActor{}
    |> MovieActor.changeset(principal)
    |> Repo.insert()
  end
end

Similar strategies will load the names and movies tables. With this code written, we're ready to actually execute the ETL in development.

data_imdb=# \c data_munger_dev
You are now connected to database "data_munger_dev" as
data_munger_dev=# select count(*) from actors;
 count 
-------
     0
(1 row)
              
data_munger_dev=# select count(*) from movies;
 count 
-------
     0
(1 row)
              
data_munger_dev=# select count(*) from movie_actors;
 count 
-------
     0
(1 row)

So at this point, if you haven’t created the indexes we created earlier and you want to run this step, definitely create the indexes! Otherwise this will absolutely not run in a reasonable period of time:

iex(1)> DataMunger.Etl.load_from_seed(["nm0000204", "nm0000702"], 4)

If we check our work in psql:

data_munger_dev=# select count(*) from actors;
 count 
-------
  4787
(1 row)
              
data_munger_dev=# select count(*) from movies;
 count 
-------
  1966
(1 row)
              
data_munger_dev=# select count(*) from movie_actors;
 count 
-------
  8137
(1 row)

Now that we have the data loaded into separate databases and the data set shrunk down to the size we wanted, we can export each database. The <filepath> here has to be an absolute filepath.

data_munger_dev=# copy actors to <filepath>;
COPY 4787
data_munger_dev=# copy movies to <filepath>;                    
COPY 1966
data_munger_dev=# copy movie_actors to <filepath>;        
COPY 8137

And finally, we’re done! Next time, we’ll look at creating our three phoenix services and launching them to heroku. Code from this post here.

A Tale of Graphs and Darkness was originally published in Perplexinomicon on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is part 1 of a 2 part series in an N part series about about microservices. Our overall goal is to create three microservices with related data, host them on Heroku, and then look at strategies for coordinating those services.

In the last post, we downloaded three tab separated files from the Internet Movie DataBase and loaded them into a single Postgres Database, in three tables, title_basics, name_basics, and title_principals. We created a Phoenix application called DataMunger with three Schemas, TitlePrincipal, TitleBasic, and NameBasic. We can use these three tables to find information about movies (titles) and actors (names) by way of the title_principals_table. For example, in SQL, we can learn the names of the principal actors in the movie "Thor: The Dark World"

data_imdb=# select primary_name from name_basics
data_imdb-# where nconst in(
data_imdb(# select p.nconst from title_principals p,
data_imdb(# title_basics t
data_imdb(# where t.primary_title ilike('thor%dark%')
data_imdb(# and t.title_type='movie'
data_imdb(# and p.tconst=t.tconst
data_imdb(# and p.category ilike('act%'));
  primary_name    
-------------------
 Natalie Portman
 Stellan Skarsgård
 Tom Hiddleston
 Chris Hemsworth
(4 rows)

Or, we can find the titles of movies where Kirsten Dunst plays a principal role with Ecto:

iex(2)> q = Ecto.Query.from(
...(2)> n in DataMunger.NameBasic,
...(2)> where: ilike(n.primary_name, "Kirsten Dunst"),
...(2)> select: n.nconst)

iex(4)> titles_from_nconst = fn(n) ->  
...(4)> Ecto.Query.from(               
...(4)> p in DataMunger.TitlePrincipal,
...(4)> where: p.nconst == ^n and      
...(4)> ilike(p.category, "act%"),     
...(4)> select: p.tconst) |>           
...(4)> DataMunger.ImdbRepo.all() end 

iex(8)> q |> 
...(8)> DataMunger.ImdbRepo.one() |>
...(8)> titles_from_nconst.() |>         
...(8)> Enum.map(fn(p) ->                
...(8)>  Ecto.Query.from(                 
...(8)>  t in DataMunger.TitleBasic,      
...(8)>  where: t.tconst == ^p,           
...(8)>  select: t.primary_title) |>      
...(8)>  DataMunger.ImdbRepo.all() end)   
[["The Virgin Suicides"], ["Get Over It"],
["Luckytown"], ["The Devil's Arithmetic"],
["The Animated Adventures of Tom Sawyer"],
["Woodshock"], ["The Beguiled"], [...], ...]
iex(9)> v() |> Enum.count()
71

What we’d like to do in this post is to take the data that we have in the three tables of the imdb_data database and prepare to move a subselection of that data into a new database. In a future post we’ll extract the data we want, load it into the new tables, and export it from the new tables.

In the rest of this post, we’ll be primarily using Ecto to transform data from one table into another table.

Create new tables

So in our final service, we’re going to use a severely truncated subset of the data. Therefore, the database we have has some fields and some data that we know we aren’t interested in.

For example, there are a number of title_types in the title_basics table:

data_imdb=# select distinct(title_type) from title_basics limit 5;
 title_type  
--------------
 short
 tvMiniSeries
 movie
 tvEpisode
 video
(5 rows)

But we’re ultimately only interested in one title_type, movie. Similarly, in title_principals, there are a number of categories:

data_imdb=# select distinct(category) from title_principals limit 5;
    category     
-----------------
 writer
 archive_footage
 composer
 archive_sound
 cinematographer
(5 rows)

But we’re only interested in actor or actress. So we'll create three new schemas that reflect our use case:

Create the migrations:

$ mix ecto.gen.migration add_movies_table
$ mix ecto.gen.migration add_actors_table
$ mix ecto.gen.migration add_movies_actors_table

Define the migrations for movies:

defmodule DataMunger.Repo.Migrations.AddMoviesTable do
  use Ecto.Migration
              
  def change do
    create table("movies") do
      add :title, :string, size: 480
      add :tconst, :string, size: 9
      add :year, :integer
    end
  end
end

Actors:

defmodule DataMunger.Repo.Migrations.AddActorsTable do
  use Ecto.Migration
              
  def change do
    create table("actors") do
      add :name, :string, size: 480
      add :nconst, :string, size: 9
      add :birth_year, :integer
      add :death_year, :integer
    end
  end
end

MovieActors:

defmodule DataMunger.Repo.Migrations.AddMoviesActorsTable do
  use Ecto.Migration
              
    def change do
      create table("movie_actors") do
      add :nconst, :string, size: 9
      add :tconst, :string, size: 9
      add :characters, {:array, :string}
    end
  end
end

Run the migrations:

$ mix ecto.migrate

Transform a single record

We’re going to want to transform a lot of different records from one type to another, TitleBasic to Movie, NameBasic to Actor, and TitlePrincipal to MovieActor, which means we're going to write some custom changesets. We can use tests to help us here, by defining what we want to have happen before we try to make it happen. The first two transformations are fairly straightforward:

defmodule DataMunger.ActorTest do
  use ExUnit.Case, async: true
  alias DataMunger.Actor
  import Ecto.Query
              
  setup do
    :ok = Ecto.Adapters.SQL.Sandbox.checkout(DataMunger.Repo)
  end
              
  test "can convert a name_basic record to an actor record" do
    ["Humphrey Bogart", "Natalie Portman"]
    |> Enum.each(fn(who) -> 
      [name_basic] = from(n in DataMunger.NameBasic,
                     where: ilike(n.primary_name, ^who),
                     select: n) 
                     |> DataMunger.ImdbRepo.all() 
      change = Actor.changeset(%Actor{}, name_basic)
              
      {:ok, actor} = DataMunger.Repo.insert(change)
      assert actor.name == name_basic.primary_name
      assert actor.nconst == name_basic.nconst
      assert actor.birth_year == name_basic.birth_year
      assert actor.death_year == name_basic.death_year
    end)
  end
end

Here, we are looking up 2 actors by name to get their NameBasic structs, creating an Actor changeset for each one, inserting that changeset into the database, and comparing the relevant keys between NameBasic and Actor.

In data_munger/lib/data_munger/actor.ex:

def changeset(%Actor{} = actor, 
  attrs = %DataMunger.NameBasic{}) do
  %{}
  |> Map.put(:nconst, attrs.nconst)
  |> Map.put(:name, attrs.primary_name)
  |> Map.put(:birth_year, attrs.birth_year)
  |> Map.put(:death_year, attrs.death_year)
  |> (&(changeset(actor, &1))).()
end
              
def changeset(%Actor{} = actor, attrs) do
  actor
  |> cast(attrs, [:nconst, :name, :birth_year, :death_year])
  |> validate_required([:nconst, :name])
end

We can take a similar approach to movies, but the MovieActortransformation is a little different. When we defined this schema, we made the characters field an array type. Most of our records, however, will probably involve actors who only played one character in a particular movie. To check that we’re handling the array properly, we’ll add a constraint on char_length to attempt to get responses in our test for actors who played multiple characters in a movie.

test "can convert a name_basic record to an actor record" do
  from(p in DataMunger.TitlePrincipal, 
  where: ilike(p.category, "act%") and 
  fragment("char_length(?)", p.characters) > 100, 
  select: p, limit: 3)
  |> DataMunger.ImdbRepo.all()
  |> Enum.each(fn(principal) -> 
    change = MovieActor.changeset(%MovieActor{}, principal)
              
    {:ok, movie_actor} = DataMunger.Repo.insert(change)
    assert movie_actor.nconst == principal.nconst
    assert movie_actor.tconst == principal.tconst
    assert Enum.count(movie_actor.characters) > 1
  end)
end

But when we run this query, we’re going to see a problem in our results:

iex(309)> Ecto.Query.from(p in DataMunger.TitlePrincipal, 
...(309)> select: p.characters,                          
...(309)> where: ilike(p.category, "act%") and           
...(309)> fragment("char_length(?)", p.characters) > 100, 
...(309)> limit: 3) |>                                   
...(309)> DataMunger.ImdbRepo.all()                       
["[\"Housekeeper (segment \"The Ghost of Murder Hill\")\",
\"Susan Castaneda (segment \"The Sailor's Secret\")\"]",
.....

Specifically, the first result contains nested quotes. Now, we could probably parse the results in such a way as to keep that data, but in the interests of time, we’ll just drop any sort of segment or episode information, by removing anything that is between ( and ).

In lib/data_munger/movie_actor.ex:

def changeset(%MovieActor{} = movie_actor, 
              %DataMunger.TitlePrincipal{} = attrs) do
  attrs = Map.from_struct(attrs)
  changeset(movie_actor, attrs)
end
              
def changeset(%MovieActor{} = movie_actor, attrs) do
  movie_actor
  |> cast(attrs, [:tconst, :nconst])
  |> add_category(attrs)
  |> validate_required([:nconst, :tconst])
end
              
@doc false 
def add_category(changeset, attrs) do
  case attrs
       |> Map.get(:characters) do              
    nil -> changeset
    character_string -> 
      characters = character_string
                   |> String.replace(~r/\ \(.*\)/, "")
                   |> Poison.decode!()
      changeset 
      |> Ecto.Changeset.put_change(:characters, characters)
  end
rescue
  _ -> changeset
       |> Ecto.Changeset.put_change(:characters, [])
end

Now we can convert one single record of any kind to its corresponding record in our new tables. See the github repository for details on how the Movie schema is handled.

In the next post we’ll look at how to extract, load, and export the data.

Just the ‘T’ from ‘ETL’ was originally published in Perplexinomicon on Medium, where people are continuing the conversation by highlighting and responding to this story.

1. Download compressed .tsv files
2. Import tsv files into postgres database
3. Create phoenix.ecto project
4. Connect project to tables from import step
5. Code Here

Tech Stack

Postgres 10.x, Elixir 1.4+, Erlang 19+

Motivation

This is a post about microservices. Actually, it’s a series of posts about microservices, because if I could put everything into one POST, it wouldn’t be a microservice.

Where we’re going in the future is to put three microservices up on Heroku, with Postgres databases, each one serving a small subset of the Internet Movie DataBase. Then we’re going to look at the problem of querying data across them, possibly creating a very small oracle of bacon along the way.

The three datasets are title.basics, which contains the names of movies and a unique identifier for each; name.basics, which contains the names of actors and crew and a unique identifier for each, and title.principals, which contains all the unique name/title combinations as well as some other data. Side note: On the imdb.com/datasets website, title.principals is not described this way, and I've downloaded that data three times and gotten two different schemas, so YMMV.

These datasets are big enough that, if we want to do this project on Heroku’s free tier (which I do), we’re going to need to filter out a lot of data. But we want to filter out that data such that we have connections (via the title.principals dataset) between all the actors and movies that we select. Given that we’re taking a dataset with twenty-three million or so rows and breaking it down to a dataset of ten thousand, we could easily end up with disjoint actors and movies.

We’ll get into that requirement in more detail in a future post, in this post, we’ll set up to filter that data with Elixir. We’ll download the datasets, import them into a postgres database, create a phoenix ecto project, and write the schemas and configuration we need to connect to the existing tables.

Let us cut Footloose

Get The Data into Postgres:

The first step is to get the data and prepare it for import. This involves using head and tail to split the .tsv file in two:

$ wget https://datasets.imdbws.com/title.basics.tsv.gz
$ gunzip title.basics.tsv.gz 
$ head -n 1 title.basics.tsv > title.basics.headers.tsv
$ tail -n +2 title.basics.tsv > trunc.title.basics.tsv
$ rm title.basics.tsv

Repeat with title.principals.tsv.gz and name.basics.tsv.gz. We should end up with six files, three containing the column headers, and three containing the column data. This will make importing into Postgres easier.

Now we want to create a postgres database. I created one owned by the postgres user, although I’m not sure that it matters. You have to execute the copy command as a root user, so we end up altering the grants on the tables anyway.

$ createdb -O postgres imdb_data
$ psql imdb_data

Create a table for the title.basics data. We can refer to the title.basics.headers.tsv file we created earlier for the column names.

imdb=# create table title_basics(
imdb(# tconst varchar(9),
imdb(# title_type varchar(80),
imdb(# primary_title varchar(512),
imdb(# original_title varchar(512),
imdb(# is_adult boolean,
imdb(# start_year smallint,
imdb(# end_year smallint,
imdb(# runtime_minutes int,
imdb(# genres varchar(80)
imdb(# )
CREATE TAB,

Now that we’ve created a table for the title.basics data, we can import the trunc.title.basics.csv file that we created. Remember, this is the data without the header columns.

imdb=# COPY title_basics FROM '/path/to/trunc.title.basics.tsv';
COPY 4817652

Now update the privileges on the title_basics table so that the postgres user can access it.

imdb=# grant all on title_basics to postgres;
GRANT

Now we repeat this process for trunc.title.principals.tsv and trunc.name.basics.tsv. When I was doing this, I ran into some errors, either because the ordering of my columns didn’t match the tsv column ordering, or because I was using smallint where I should have used int, or I wasn’t allowing the varchar size to be large enough. Eventually, my final schemas looked like this:

data_imdb=# \d title_principals
                Table "public.title_principals"
  Column    |          Type          | Collation | Nullable | Defau 
------------+------------------------+-----------+----------+----
 tconst     | character varying(9)   |           |          | 
 ordering   | smallint               |           |          | 
 nconst     | character varying(9)   |           |          | 
 category   | character varying(80)  |           |          | 
 job        | character varying(480) |           |          | 
 characters | character varying(480) |           |          | 
              
data_imdb=# \d name_basics 
                 Table "public.name_basics"
       Column       |          Type          | Collati
--------------------+------------------------+--------
 nconst             | character varying(9)   |       
 primary_name       | character varying(240) |       
 birth_year         | smallint               |       
 death_year         | smallint               |       
 primary_profession | character varying(180) |       
 known_for_titles   | character varying(180) |       
              
data_imdb=# \d title_basics
                   Table "public.title_basics"
     Column      |          Type          | Collation |
-----------------+------------------------+-----------+
 tconst          | character varying(9)   |           |
 title_type      | character varying(80)  |           |
 primary_title   | character varying(512) |           |
 original_title  | character varying(512) |           |
 is_adult        | boolean                |           |
 start_year      | smallint               |           |
 end_year        | smallint               |           |

Make sure you have all your data loaded and your privileges set:

data_imdb=# select count(*) from title_basics;
  count  
---------
 4817652
(1 row)
              
data_imdb=# select count(*) from name_basics;
  count  
---------
 8435455
(1 row)
              
data_imdb=# select count(*) from title_principals;
  count   
----------
 27109331
(1 row)
              
data_imdb=# \dp
                 Access privile
 Schema |       Name       |   Type   |   Access privil
--------+-------------------------+----------+---------
 public | name_basics      | table    | paul=arwdDxt/pa
        |                  |          | postgres=arwdDx
 public | title_basics     | table    | paul=arwdDxt/pa
        |                  |          | postgres=arwdDx
 public | title_principals | table    | paul=arwdDxt/pa
        |                  |          | postgres=arwdDx
              (3 rows)

If that looks more or less like what you’re looking at, you’re ready to move on.

Connect the database to a Phoenix project

We’re going to use Ecto for the next part, and for convenience, we’re going to create an Phoenix ecto project. We’ll probably put the microservices we’ll be building in a later post into this same umbrella project.

$ mix new --umbrella dataday
$ cd dataday/apps/
$ mix phx.new.ecto data_munger

This project assumes that there is a data_munger database, which we will create next time. The config for that database is already set up. To connect to our imdb_data database, we need to add configuration to /data_munger/config/dev.exs.

config :name_basics, DataMunger.ImdbRepo,
  adapter: Ecto.Adapters.Postgres,
  username: "postgres",
  password: "postgres",
  database: "imdb_data",
  hostname: "localhost",
  pool_size: 10

We also need to add the new DataMunger.ImdbRepo to our supervision tree. In data_munger/lib/data_munger/application.ex

def start(_type, _args) do
  import Supervisor.Spec, warn: false
              
  Supervisor.start_link([
    supervisor(DataMunger.Repo, []),
    supervisor(DataMunger.ImdbRepo, []),
  ], strategy: :one_for_one, name: DataMunger.Supervisor)
end

Now if we write a schema for one of the tables, we should be able to access it. Create a file data_munger/lib/data_munger/name_basic.ex:

defmodule DataMunger.NameBasic do
  use Ecto.Schema
  import Ecto.Changeset
  alias DataMunger.NameBasic           
              
  @primary_key{:nconst, :string, []}
  schema "name_basics" do
    field :birth_year, :integer
    field :death_year, :integer
    field :primary_name, :string 
      
  end
              
  @doc false
  def changeset(%NameBasic{} = name, attrs) do
    name
    |> cast(attrs, [:nconst, 
                    :primary_name,
                    :birth_year,
                    :death_year])
    |> validate_required([:nconst, :primary_name])
  end
end

You might notice the line @primary_key{:nconst, :string, []} above the schema definition. When we created this table, we didn't give it an autoincrement id, but there is a good natural key we can use in the dataset, the nconst value, so we can instruct Ecto that we'll use that as the primary key.

Let’s test our new schema in iex:

iex(1)> require Ecto.Query
Ecto.Query
iex(2)> Ecto.Query.from(                                 
...(2)> n in DataMunger.NameBasic,                      
...(2)> where: ilike(n.primary_name, "Natalie Portman"),
...(2)> select: n) |>
...(2)> DataMunger.ImdbRepo.all()
              
00:37:47.847 [debug] QUERY OK source="name_basics" db=1
722.7ms
SELECT n0."nconst", n0."birth_year", n0."death_year", n
0."primary_name" FROM "name_basics" AS n0 WHERE (n0."pr
imary_name" ILIKE 'Natalie Portman') []
%DataMunger.NameBasic{__meta__: #Ecto.Schema.Metadata 
<:loaded, "name_basics">,
birth_year: 1981, death_year: nil, nconst: "nm0000204"
, primary_name: "Natalie Portman"}

Fun fact, Natalie Portman’s Bacon/Erdős number is lower than Kevin Bacon’s, but not lower than Paul Erdős’s.

We can repeat this with the other two tables. One thing is, we don’t have a natural key for the title_principals table, so we should go ahead and add an automatic autoincrement column to that table, if we don't, Postgrex will not be happy and we won't be able to search the table.

data_imdb=# alter table title_principals
data_imdb-# add column id
data_imdb-# serial primary key;
ALTER TABLE

Once we’ve added an id column to the table, we can add the schema file /data_munger/lib/data_munger/title_principal.ex with the contents:

defmodule DataMunger.TitlePrincipal do
  use Ecto.Schema
  import Ecto.Changeset
  alias DataMunger.TitlePrincipal
              
  schema "title_principals" do
    field :nconst, :string
    field :tconst, :string
    field :category, :string
    field :characters, :string
    field :ordering, :integer

  end
              
  @doc false
  def changeset(%TitlePrincipal{} = title, attrs) do
    title
    |> cast(attrs, [:nconst,
                    :category,
                    :tconst,
                    :characters,
                    :ordering])
    |> validate_required([:nconst, :tconst])
  end
end

And test our new schema in IEx:

iex(6)> Ecto.Query.from(                
...(6)> n in DataMunger.TitlePrincipal, 
...(6)> where: n.nconst == "nm0000204",
...(6)> select: n,
...(6)> limit: 1) |>
...(6)> DataMunger.ImdbRepo.all()
              
01:22:47.033 [debug] QUERY OK source="title_princip....
SELECT t0."id", t0."nconst", t0."tconst", t0."categ....
[%DataMunger.TitlePrincipal{__meta__: #Ecto.Schema.....
category: "actress", characters: "[\"Nina\"]",
id: 1437846, nconst: "nm0000204", ordering: 9,
tconst: "tt0185275"}]

I’ve truncated some of the output, but it should look something like that.

Finally, we add the title_basics schema (/data_munger/lib/data_munger/title_basic.ex), this time using the tconst field as a natural key:

defmodule DataMunger.TitleBasic do
  use Ecto.Schema
  import Ecto.Changeset
  alias DataMunger.TitleBasic
              
              
  @primary_key{:tconst, :string, []}
  schema "title_basics" do
    field :primary_title, :string
    field :original_title, :string
    field :start_year, :integer
    field :end_year, :integer           

  end
              
  @doc false
  def changeset(%TitleBasic{} = name, attrs) do
    name
    |> cast(attrs, [:nconst,
                    :primary_name,
                    :birth_year,
                    :death_year])
    |> validate_required([:nconst, :primary_name])
  end
end

And verify that we have access:

iex(9)> Ecto.Query.from(               
...(9)> t in DataMunger.TitleBasic,    
...(9)> where: t.tconst == "tt0185275",
...(9)> select: t) |>                  
...(9)> DataMunger.ImdbRepo.all()      
              
03:28:42.722 [debug] QUERY OK source="title_basic.....
SELECT t0."tconst", t0."primary_title", t0."origi.....
[%DataMunger.TitleBasic{__meta__: #Ecto.Schema.Me.....
end_year: nil, original_title: "Developing",
primary_title: "Developing", start_year: 1994,
tconst: "tt0185275"}]

Next, we’re going to work on how to start filtering through this data. The code up to this point is available here, and my first pass at solving all these problems is available here if you want to read ahead.

The Kevin Bacon Problem was originally published in Perplexinomicon on Medium, where people are continuing the conversation by highlighting and responding to this story.

1. An interface module is created to wrap around the dependency.
2. A test version of the interface module is created.
3. Test responses are written in the test interface

Our implementation of this in the near earth wraps the HTTPoison library. There are some issues with this, but before we get to those I want to look at how we’re building and retrieving mock responses. Right now those responses are built into the test module.

This post is concerned with moving our test responses out of the test interface module and into the tests themselves. This is useful because we don’t have to move from file to file while we’re working on the tests. It should simplify our test code somewhat.

To do this, we’re going to use a library called Stash, which is a wrapper around ETS, Elixir’s build in key-value store. Adding Stash is simple, of course we add stash to our mix.exs file:

defp deps do
  [
    {:env_helper, "~> 0.0.4"},
    {:httpoison, "~> 0.10.0"},
    {:poison, "~> 3.0"},
    {:stash, "~> 1.0"}
  ]
end

And then change the implementation of test/support/http.ex. Previously, we only had a mock response for the 404 case:

defmodule NearEarth.TestHttp do
  use HTTPoison.Base
              
  def get("https://api.nasa....?api_key=DEMO_KEY") do
    {:ok, %HTTPoison.Response{headers: [], status_code: 404}}
  end
              
 def get(args) do
   HTTPoison.get(args)
 end
end

But now we’re going to redefine the get method to use stash:

def get(route) do
  Stash.get(:near_earth, route)
end

So whatever value is stored in the ETS table :near_earth for the key route, that's what the get method will return.

Obviously, to get our current tests working we’re going to need to do some work. Right now, we have two tests. The first tests our response to a 404, the second actually makes a real call out to the near earth api to get an asteroid and then tests the parsing of that JSON file. To get the 404 test working, we add a response to the setup:

setup do
  not_found = %HTTPoison.Response{status_code: 404}
  {:ok, %{not_found: not_found}}
end

And then add that response to the :near_earth stash table in the test:

test "will return not found if 404", context do
  Stash.set(:near_earth, "#{Sets.neo_url}fakevalue?api_key=#{Sets.api_key}",
           {:ok, context.not_found})
  assert {:error, :not_found} == NearEarth.get_asteroid("fakevalue")
end

So we’ve set the value of "#{Sets.neo_url}fakevalue?api_key=#{Sets.api_key}" to be an HTTPoison response with a status of 404. The Sets module is where I'm defining helpers to application and environment variables using the env_helper library.

The get_asteroid call uses the HTTP module:

def get_asteroid asteroid do
  url = "#{Sets.neo_url}#{asteroid}?api_key=#{Sets.api_key}"
  case Sets.http.get(url) do
    {:ok, response = %{status_code: 200}} -> 
      Poison.decode(response.body)
    {:ok, %{status_code: 404}} ->
      {:error, :not_found}
  end
end

And because we’re using the test http module in our test environment, we get back the fake 404 response and the test passes.

A similar strategy will work for getting the asteroid, although in that case we will need a fake asteroid JSON response to work with. My solution to this, for the moment, was to add that as a json file in the test/support folder and then add a module stubs:

defmodule NearEarthTest.Stubs do
  def asteroid do
    {:ok, asteroid} = File.read("test/support/asteroid.json")
    asteroid
  end
end

Add the stub to the test setup:

asteroid_response = %HTTPoison.Response{status_code: 200, 
                                        body: Stubs.asteroid}
              
{:ok, %{asteroid: asteroid_response,
        not_found: not_found}}

And then put that in the test:

test "retrieves asteroid by designation", context do
  id = "3542519"
  Stash.set(:near_earth, "#{Sets.neo_url}#{id}?api_key=#{Sets.api_key}",
           {:ok, context.asteroid})
  {:ok, asteroid_response} = NearEarth.get_asteroid(id)
  assert asteroid_response["name"] == "465633 (2009 JR5)"
end

And again, we should have a passing test.

One obvious problem with this implementation is that I’m making everything very tightly coupled to the HTTPoison library. In a future post, we’ll change the http module so that this coupling disappears, but for now we'll live with it.

We’ve refactored, and now we want to add an endpoint. In particular, I’d like to get all the asteroids that will make their closest approach today. Fortunately, NASA has a useful /feedendpoint that takes a start and end date as arguments. To write the failing test, we'll add another library to our codebase. Faker is a good library for generating fake data of various kinds, such as dates, addresses, and of course styles of beer. We use Faker at work mostly for dates, addresses and to generate fake car make and model names. Here, we're interested in the date functionality.

We’ll also want to add another stub, this one with a real JSON response from the feed api. We’ll again use the Stubsmodule to import the file, and add the stub to the test setup:

asteroids_response = %HTTPoison.Response{status_code: 200, 
                                         body: Stubs.asteroids}
{:ok, %{asteroids: asteroids_response,
        asteroid: asteroid_response,
        not_found: not_found}}

And then test against the stub by adding the fake response to stash:

test "retrieves today's asteroids", context do
  today = Date.utc_today
          |> Date.to_string
  tomorrow = Faker.Date.forward(1)
             |> Date.to_string
  Stash.set(:near_earth, 
            "#{Sets.neo_url}feed?start_date=#{today}&end_date=#{tomorrow}&api_key=#{Sets.api_key}", 
            {:ok, context.asteroids})
  {:ok, asteroids} = NearEarth.get_today()
  assert asteroids == context.asteroids.body 
                      |> Poison.decode!
end

Since we haven’t implemented get_today at all, it's pretty clear we're going to get a failing test here. So, we'll get it passing:

def get_today do
  today = Date.utc_today()
          |> Date.to_string()
  tomorrow = :calendar.universal_time
             |> :calendar.datetime_to_gregorian_seconds
             |> (&(&1 + 86400)).()
             |> :calendar.gregorian_seconds_to_datetime
             |> elem(0)
             |> Date.from_erl
             |> elem(1)
             |> Date.to_string 
 case Sets.http.get("#{Sets.neo_url}feed?start_date=#{today}&end_date=#{tomorrow}&api_key=#{Sets.api_key}") do
   {:ok, response = %{status_code: 200}} ->  
      Poison.decode(response.body)
 end

Granted, that could be the most convoluted method ever of getting a date string for tomorrow, but I think that this one will at least work. As always, a working version of this code is available on github.

Mock Responses with Stash was originally published in Perplexinomicon on Medium, where people are continuing the conversation by highlighting and responding to this story.

But we don’t exclusively write big things, sometimes we write smaller things. Sometimes, you’re at work, and you’re faced with a problem like “convert these 20 basically identical yaml files into 20 basically identical xml files”. The question is, what tool do you reach for when that happens?

Because even if you’re working in a PhP shop out in Bakersfield, you can use Elixir for scripting stuff. And it can make your life pretty good if you do. The Sendalot tool I’ve talked about before is an example of this, a little script that I can open up, mess around with, and start sending hundreds of thousands of simultaneous requests from my laptop to a server. Is it pretty? No. But it gets that specific job done and we break it out from time to time.

A week or so ago, I was faced with a little scripting job where we had to take about 20 basically identical YAML files, and convert them to 20 basically identical XML files. My pair that day and I are both big fans of Elixir, so naturally we solved this with a little Elixir script.

So the first thing we needed was a library that could turn Elixir data structures into XML, or YAML into Elixir data structures, or possibly both. A quick check on Hex led to the yaml_elixir library, which is pretty effective at reading YAML files and turning them into Elixir maps.

def file_to_map(path) do
  File.cwd!
  |> Path.join(path)
  |> YamlElixir.read_from_file
end

This takes a relative path (like “assets/test.yml”) to a .yml file and returns a map, when run from the top level of the project. So that’s the first half of the problem solved.

Unfortunately, writing the XML was not going to be quite so simple, because we needed to join multiple YAML files into a single XML file, and because we only need a part of each, and the nesting of the eventual XML file is going to be slightly different than the nesting of the YAML file.

All of which leads to this code:

def get_env_entries(map) do
  get_env_entries({}, 
  Map.get(map, "env_entries"))
end
              
defp get_env_entries(acc, env_entries) do
  with false <- Enum.empty?(Map.keys(env_entries)),
       entry_name <- env_entries
                     |> Map.keys()
                     |> List.first(),
       entry_map <- create_entry(env_entries, entry_name) do
    acc
    |> Tuple.append(entry_map)
    |> get_env_entries(Map.delete(env_entries, entry_name))
  else
    _ -> acc
         |> Tuple.to_list
  end
end

Where we are using the with statement in a recursive method, with the exit condition being the terminal condition of the recursion. To be honest, I didn't actually see this as a possibility until my pair pointed it out, but it makes perfect sense, the with/do/else control structure is just that, a control structure, so clearly we can use it as one.

In this case, get_env_entries/1 acts as the entry point to the recursion, taking as its argument the map produced from the original yaml file. It passes on an empty tuple (the initial accumulator), and the sub map under the "env_entries" key.

This sub map is actually composed of several sub maps, and the goal of the code is to take each key and generate a flattened and transformed version of that nested map. That is what the create_entry/2 method will do, but that code isn't terribly important. It takes a map and a key and returns a map. So, we get the first key from the "env_entries" sub map, and send that key and the “env_entries” map to the create_entry/2transformer, which returns entry_map. entry_map gets appended onto the accumulator, and then we make the recursive call back to get_env_entries/2, calling Map.delete/2 to remove the key we just transformed. When we run out of keys, the Enum.empty?/1 call that we make at the start of the with statement returns true, and we convert the accumulator from a tuple of maps into a list of maps and return it.

When I write it up like that, it feels sort of complicated. But the funny thing is this was developed mostly using iex, so once we had the recursive call with the delete, the empty check, and the return taken care of we were able to sort of REPL-drive the rest very quickly and easily, just filling in transforms in the middle until the correct accumulator was returned. It was fast, and it was extremely fun.

In any case, there was obviously more to be done once we had these property lists (remember, we had two files, so there was another get_entries type of method) we had to turn them into XML. The tool we used for this was xml_builder.

defp generate_env_elements(env) do
  env
  |> Enum.reduce({}, 
                 fn(lst, acc) -> XmlBuilder.element(:Env,lst) 
                                 |> (&(Tuple.append(acc, &1))).()
                 end)
  |> Tuple.to_list
end

This code is more my style, for better or worse. I’m a fan of the reduce/3 function and of anonymous functions. I don't know if this is more readable than a recursive approach to this same problem, but in any case, we take the list we built in the last stage, and for each entry in the list we create an XML element tuple. The reduce returns a tuple of tuples which we convert back into a list, because for whatever reason, it seems like the XmlBuilder API takes in lists and returns tuples, except when it takes tuples and returns an XML string.

So, I think that in the first method we could have used Enum.reduce/3, but I'm not 100% positive that the internal function would have been readable, and we might have ended up with a nested with statement anyway. The second method was much more straightforward and the with statement would have been overkill. But in any case, a few hours of hacking along, 100 lines of code or so later, and we had that yak shaved and the YAML converted to XML. It definitely wasn't harder than it would have been in any other language, and I think that the control structures and functional nature of Elixir made it an excellent choice for this problem… after all, we were only transforming some data from one form into another, and that's an area where Elixir really shines.

Elixir for the Little Things was originally published in Perplexinomicon on Medium, where people are continuing the conversation by highlighting and responding to this story.

But in the process, I was forced to ask myself what good all my side work, the blogging, the meetups, the presentations, etc, were doing. Was this time well spent? I could be spending more time with my family. I could be writing a novel. I could be exercising more, or learning to cook French cuisine, or getting more sleep.

So I stopped. Not on purpose, but out of a just general sense of ennui. I lost my motivation to side job. And for two months, I haven’t written a blog post or worked on my coding chops or generally done anything like that.

And you know what? I didn’t spend more time with my family. I didn’t work on my novel. I didn’t exercise more, I didn’t learn French cooking, I didn’t get more sleep. I did watch more TV. I did read a lot of fiction. I did gain ten pounds. I’m not happier, and I’m probably actually less useful at work.

I don’t think that this is a general lesson for everyone. It just is for me, which is this: I like puzzles. I like typing. I like writing code, and I like sharing it with other people. There’s a certain kind of side project that appeals to someone like me, and if I’m not working on that I stop working on other stuff too. For me, personally, not working on coding projects means that parts of my creative brain aren’t running, and I’m not working on anything else, either.

Of course, I’m speaking of myself here, and not in the general sense. In the general sense, the question is “Should coders, in general, have some sort of side projects going on?” And you know what? I’ll say no. It’s fine if you aren’t involved in the community, or don’t have anything to show on GitHub, or aren’t going to meetups or presenting at conferences or whatever. Especially if you have a lot of experience, and there are plenty of people to recommend you for the job I’m interviewing you for.

But if you don’t have that experience, ask yourself what seperates you from the person I’m interviewing next. If it is that they met me at a meetup and impressed me into going to my boss and saying “HIRE THIS PERSON”, than that might not be great for you. If it’s that the person I’m interviewing next ran a startup in college and can show me how they learned from that experience, and you didn’t, that might not be great for you. If it’s that the person I’m interviewing next got some documentation pull requests into a project I’m familiar with and you didn’t, that might not be great for you.

After you’ve been working for a couple of years, this stuff is less important. Because honestly, the industry is savagely hungry for talent and once you’ve shown you can do the job, you’ll probably get another job. But hiring is hard, and expensive, and people are pretty reticent about hiring unknown quantities.

I think that this is the core of this whole debate. If you have been working as a coder for a decade, you don’t need my advice. You’ve got this. But if you’re still trying to move into or from your first junior job, it can’t hurt to put in some extra effort.

As for me, I’m just happier when I’m working on my own stuff in my spare time. It’s become a habit, and it’s not a bad habit to have.

In Defense (sort of) of Side Projects was originally published in Perplexinomicon on Medium, where people are continuing the conversation by highlighting and responding to this story.