In this post I’ll share some strategies for improving the speed of recompilation. It’s important you understand the fundamentals first, so please check out part 1 and ideally part 2 first!

Reducing compile time dependencies

Avoiding module attributes calling other modules

The big thing here is to avoid using module attributes that call other modules. It’s a great way to reduce compile time dependencies. Instead just call those functions where you need them.

The only downside here is that you can’t call functions in your module headers (for example if you want to pattern match on them). Instead consider using a cond or read on to see some other ways to handle this.

Use alias instead of import

When you use import you create an export dependency between 2 files. By simply using alias instead no export dependencies are created.

Sometimes you have to have compile-time dependencies though right?

It’s absolutely true that sometimes you have to have compile time dependencies. Perhaps you have want to use module attributes as you want to pattern match against them in your function headers like this:

@active Status.active()
@disabled Status.disabled()
@old Status.old()

def handle_status(@active), do: ...
def handle_status(@disabled), do: ...
def handle_status(@old), do: ...

Or maybe you are implementing a behaviour, using the use keyword, or otherwise calling a macro.

In these cases the key things to make sure of are

• The file you have a compile-time dependency on has as few runtime dependencies as possible (ideally none)
• And any runtime dependencies it does have aren’t part of a dependency cycle

Let’s take an example. Imagine you have these files:

# a.ex
defmodule A do
 @active Status.active()
  ...
end

# status.ex
defmodule Status do
  def active, do: "active"
  def do_other_stuff, do: B.other_stuff()
end

# b.ex
defmodule B do
  # Let's imagine B calls other modules that form a circular dependency
  def other_stuff, do: ...
end

# ...etc. Assume modules C and D here that form a dependency cycle

This results in the following dependencies:

This is now truly awful! B, C, and D are now transitive compile time dependencies of A. That means anytime B, C, or D change A will also need to be recompiled.

How could we fix this? In this case simply splitting the Status module in 2 — one containing our constants, and one containing functions that call out to other modules, like this:

# a.ex
defmodule A do
 @active Status.Consts.active()
  ...
end

# status/consts.ex
defmodule Status.Consts do
  def active, do: "active"
end

# status.ex
defmodule Status do
  def do_other_stuff, do: B.other_stuff()
end

# b.ex
...etc

Now the only changes that will cause a.ex to be recompiled are changes to a.ex itself or changes to status/const.ex. Woohoo!

A similar principle applies for defining behaviours or macros — try to remove as many runtime dependencies from those modules as possible!

The mix xref graph — label compile-connected command can help you find these troublesome cases where you have transitive compile-time dependencies.

Reducing dependency cycles

As mentioned in part 1, dependency cycles are bad, especially big ones. They can lead to many more transitive compile-time dependencies, meaning more files need to be recompiled when you modify files. Here’s how to stop this happening:

Step 1 — identify your longest dependency cycles

mix xref will come in handy for this:

mix xref graph --format cycles

Step 2 — identify where dependencies don’t make sense

Looking at a long cycle of dependencies is a bit intimidating. Start by trying to identify a single call-chain that doesn’t feel like it would be necessary. For example some common smells are things like:

• A sub-module depending on it’s parent
• A module reaching into a context that doesn’t seem strictly related
• Generally any dependency where it doesn’t make intuitive sense from the file names, when you just think “but why would this need to depend on that??”

Let’s take a real example from a cycle in our codebase:

lib/platform/companies/events/company_created/publisher.ex
lib/platform/companies/events/json_serialiser.ex
lib/platform/companies/company.ex
lib/platform/roles/role.ex
lib/platform/roles/slug.ex
lib/platform/companies/companies.ex
lib/platform/companies/events/company_created/publisher.ex

One thing jumps out to me here — should slug (which generates a string to use in the URL) really need to know about companies? Surely it would get passed everything it needs to generate a slug?

Step 3 — Remove those dependencies where you can

Generally this means refactoring the code, which could mean:

• Changing where you call functions from to remove a troublesome dependency
• Changing where the function causing the dependency cycle lives — perhaps moving it into a new module for example

In the example above, it turned out that a function in slug.ex was being passed a company_id, and then using the companies context to look up the details for the company. By instead passing the required company details to the function we could remove the function call in slug, and hence remove this dependency cycle.

Step 4— check your progress

In smaller projects running mix xref graphs --format cycles is easy enough to see progress.

However if you have a lot of cycles it can be hard to feel like you’re making progress — often fixes don’t even reduce the number of loops, just reduce the number of steps per loop. You can use this nifty one liner to count the total steps across all your loops. You’ll be amazed how fast you can drive this number down:

mix xref graph --format cycles | grep "Cycle of length" | sed 's/[^0-9]//g' | paste -sd+ - | bc

Step 5— stop dependency cycles re-occuring in your CI pipeline

mix xref has this handy command which you can use in your CI pipeline to make sure you’re not introducing new dependency cycles.

mix xref graph --format cycles --fail-above X

Identifying slow-compiling files

This command will help you identify slow-compiling files:

mix compile --force --profile time

Use this to help focus your efforts on refactoring, or at least reducing compile-time dependencies for the slowest files. Some other tools you might find helpful for debugging your dependencies are:

• mix compile --verbose will tell you which files are being recompiled, so you can change a single file and run a compilation in verbose mode to see what files that causes to be recompiled
• mix xref trace FILE for identifying what dependencies a given file has

Tips for Phoenix compile times

When you call a macro (such as the get macro used in Phoenix routers) if you pass a full module name it creates a compile-time dependency on that module. For example this code has a compile-time dependency on CandidatesWeb.LoginController.

scope "/candidates" do    
  get "/", CandidatesWeb.LoginController, :delete
end

With Phoenix we can pass the outer module to scope. So with this code there is no compile-time dependency!

scope "/candidates", CandidatesWeb do    
  get "/", LoginController, :delete
end

NOTE: I believe this will be fixed in a future Phoenix release so the first example above doesn’t create a compile-time dependency — watch this space!

Inevitable dependency cycles with Phoenix

Phoenix has a design that naturally creates dependency cycles, because:

• A router contains the routes for your application, and depends on controllers to handle them
• Controllers have the logic for the route, and depend on contexts and views to display things to the user
• Views depend on templates to render things to the user
• Templates often contain links to other places in the site. These links tend to use Phoenix path helpers to ensure they are up to date. And of course, these path helpers depend on the router, and thus the cycle is formed
• Just to note, this dependency cycle still exists even if you use the new Verified Routes functionality instead of the usual path helpers

So far we’ve not found a way to avoid these. Instead we follow the advice earlier in this guide and ensure that there are never compile-time dependencies on a file that is part of a runtime dependency cycle.

Summary

I hope you’ve found this series helpful, and would love to hear any suggestions for more strategies to improve compilation times!

Also Multiverse is hiring so if you love working on interesting technical challenges and would like to help us build an outstanding alternative to university we’d love to hear from you!

How to speed up your Elixir compile times (part 3) — strategies to improve your compile times was originally published in multiverse-tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

This post gives a few examples to help you get your head around how the Elixir compiler deals with dependencies. Try your best to work out the answers without reading ahead — it’ll do wonders for your understanding! You can find the examples on Github. I also recommend reading part 1 of the series which explains some key details of how Elixir compilation works. Part 3 is now available which covers strategies for improving your compile times.

Challenges (solutions below)

1. Starting simple

If we edit a.ex below which files will be recompiled?

# a.ex
defmodule A do
  def a, do: "a"
end

# b.ex
defmodule B do
  def b, do: A.a()
end

2. Little harder…

Part a) If we edit a.ex below which files will be recompiled?
Part b) If we edit b.ex below which files will be recompiled?

# a.ex
defmodule A do
  def a, do: "a"
end

# b.ex
defmodule B do
  @a A.a()
  def b, do: @a
end

3. Keep that brain working…
If file a.ex is changed which files get recompiled?

# a.ex
defmodule A do
  def a, do: "a"
end

# b.ex
defmodule B do
  def b, do: A.a()
end

# c.ex
defmodule C do
  @b B.b()
  def c, do: @b
end

4. Still with me?
If file b.ex is changed which files will be recompiled?

# a.ex
defmodule A do
  def a, do: "a"
end

# b.ex
defmodule B do
  def b, do: A.a()
end

# c.ex
defmodule C do
  def c, do: B.b()
end

# d.ex
defmodule D do
  @a A.a()
  def d, do: @a
end

5. Let’s go round again…
Part a)
If file b.ex is changed which files will be recompiled?

# a.ex
defmodule A do
  def a, do: C.c()
end

# b.ex
defmodule B do
  def b, do: A.a()
end

# c.ex
defmodule C do
  def c, do: B.b()
end

# d.ex
defmodule D do
  @a A.a()
  def d, do: @a
end

Part b)
Same question but with the following tweak to the files:

# a.ex
defmodule A do
  def a, do: C5.c()
  def a2, do: "a"
end

# b.ex
defmodule B do
  def b, do: A.a()
end

# c.ex
defmodule C do
  def c, do: B.b()
end

# d.ex
defmodule D do
  @a A.a2()
  def d, do: @a
end

6. Let’s think about structs

# a.ex
defmodule A do
  defstruct [:name, :age]

  def a, do: "a"
end

#b.ex
defmodule B6 do
  def b(%A{name: name}), do: name
end

Part a) If the field height were added to the struct defined in a.ex which file(s) would be recompiled?

Part b) If A.a() were updated to return "b" then which file(s) would be recompiled?

7. More structs…

If the field height were added to the struct defined in a.ex which file(s) would be recompiled?

# a.ex
defmodule A do
  defstruct [:name, :age]

  def a, do: "a"
end

#b.ex
defmodule B6 do
  def b(%{name: name}), do: name
end

8. What about using import?

# a.ex
defmodule A do
  def a, do: "a"
end

#b.ex
defmodule B6 do
  import A
  def b, do: a()
end

Part a) Which files would be recompiled if change A.a() so it takes 1 argument?

Part b) Which files would be recompiled if we add a new function foo to the A module?

Part c) Which files would be recompiled if we make a return "a2" instead of just returning "a"?

9. And Macros?

# a.ex
defmodule A9 do
  defmacro a(_clause), do: "X"

  def woo, do: "woo"
end

# b.ex
defmodule B9 do
  alias A9
  require A9
  def b, do: A9.a("some clause")
end

Part a) If we change the implementation of the a macro which files will be recompiled?

Part b) If we change the implementation of the woo function which files will be recompiled?

Solutions

Have you answered all of the above challenges? Check below to see how you got on!

1. Only a.ex will be recompiled. b.ex only has a runtime dependency on a.ex, not a compile-time dependency
2. a)a.ex and b.ex will both be recompiled because b.ex has a compile-time dependency on a.ex. Remember that any code outside of a method will be called at compile-time.
   b) Only b.ex will be recompiled — the dependency is only from b to a, not the other way around
3. a.ex will be recompiled because it was modified. c.ex will be recompiled because it has a transitive compile-time dependency on C. Remember anything with a compile-time dependency on another file, will also have a transitive compile-time dependency on all the other files runtime dependencies
4. Only b.ex will be recompiled, no other files have a compile-time or transitive compile-time dependency on it!
5. Part a) This is a bit of a trick question. What will happen is the compiler will hang indefinitely in an infinite loop. At compile-time module D needs to call A.a() , which calls C.c(), which calls B.b() which calls A.a(), and so on in a never-ending loop.
   Part b) Fixes this issue, so in that case b.ex and d.ex will both be recompiled. d.ex will be recompiled because it has a compile-time dependency on a.ex. Because a.ex has a runtime dependency on c.ex, and c.ex depends on b.ex, d.ex therefore has transitive compile-time dependencies on a.ex, b.ex, and c.ex. You may have noticed a, b, and c form a dependency cycle, which is something to avoid if you can!
6. Part a) Both a.ex and b.ex would be recompiled — a.ex because it has been modified. b.ex because it depends on the struct from a.ex and that struct has been modified.
   Part b) Only a.ex would be recompiled. Because b.ex has an export dependency (because it depends on the struct from A), it will need to be recompiled only if the struct itself changes. Other details of A can change and not impact B at all
7. Only a.ex would be recompiled — as B no longer references the struct from A it doesn’t have any dependencies on A.
8. Part a) a.ex and b.ex would both be recompiled. When you use import then if any of the function signatures (i.e. names of functions or the number of arguments) change then
   Part b) a.ex and b.ex would both be recompiled. Again because we used import any change in the function signatures (including adding a new function)
   Part c) Only a.ex would be recompiled because only the implementation changed — the function signatures in A didn’t change (all function names and numbers of arguments remained the same)
9. Part a) b.ex has a compile time dependency on a.ex because it calls the macro from a.ex, which is executed at compile-time
   Part b) As with part a, both files will be recompiled. Remember compile-time dependencies are just known about at the file level, so even though we haven’t changed the macro itself, b.ex still needs to be recompiled. Of course the compiled version of b.ex will be identical, but the compiler doesn’t know that

What next?

In the next final article in this series I’ll share strategies to help improve compile times. Also Multiverse is hiring so if you love working on interesting technical challenges and would like to help us build an outstanding alternative to university we’d love to hear from you!

How to speed up your Elixir compile times (part 2) — test your understanding! was originally published in multiverse-tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Struggling with slow Elixir compile times? If you’re working in a large Elixir codebase you may find that the compile times start to suffer. You change just a single file and re-run your test, but it compiles tens of files and takes ages. Part 1 of these posts will explain why this happens. Part 2 will help you check your understanding, and part 3 will provide a practical guide for reducing your incremental compile times.

Why does Elixir need to compile so many files when I change just one?

Run-time vs compile time

When you write Elixir code some of it will be run at compile-time, and some will be run at runtime. For example if you use module attributes they are evaluated at compile time, and the output is written into the .beam files that the compiler produces. For example this:

defmodule Dinner do
 @dinner_steps ["make food", "make drinks"]
 def dinner_steps, do: @dinner_steps
end

Will be evaluated, so the code that would be at runtime would be just:

defmodule Dinner do
  def dinner_steps, do: ["make food", "make drinks"]
end

Of course this would be a .beam file rather than Elixir code, but you get the point. In this example Dinner doesn’t depend on any other module, so the only time it ever needs to be recompiled is if this file itself is modified.

Compile-time dependencies

But what if our module attribute depended on another module?

defmodule Dinner do
  @dinner_steps Food.make() ++ Drinks.make()
  def dinner_steps, do: @dinner_steps
end

Now if the Food or Drinks modules change it could change the .beam file produced for Dinner! Also, this applies even if you change something other than the make function in Food or Drink — the Elixir compiler only looks at file level dependencies and file level changes. So now if we modify Food it will cause both Food and Dinner to be recompiled. We say that Dinner has a compile-time dependency on both Food and Drinks.

Transitive compile-time dependencies

Now imagine the Food module calls on another module MainCourse . We say that Food has a runtime dependency on MainCourse.

defmodule Food do
  def make, do: MainCourse.make()
end

Now if MainCourse changes that may impact what the Dinner module gets compiled to, so Dinner will be recompiled every time MainCourse changes too. We can say Dinner has a transitive compile-time dependency on MainCourse. To summarise the dependency tree now looks like:

As you can imagine, the more compile-time or transitive compile-time dependencies a file has, the more likely it will need to be recompiled when other files change. If you have too many then even changing a single file can cause 10s or hundreds of other files to get recompiled, slowing you down.

The plot thickens — circular dependencies

Now imagine we wrote some code like this in our MainCourse module:

defmodule MainCourse do
  def make, do: Dinner.microwave("pot noodle")
end

Now MainCourse has a runtime dependency on Dinner and we’ve created a circular dependency. This is bad!

What it means is that every file in the loop has a runtime dependency on every other file in the loop, massively increasing the chances of transitive compile-time dependencies.

In our example it actually doesn’t have an impact because Dinner already had a transitive compile-time dependency on the other files. But imagine if the loop were 100 files long. Now any file that has a compile-time dependency on any of those files will also have a transitive compile-time dependency on the other 99 files. Overall it increases the number of files needing to be recompiled with each change, which is bad news for your incremental compile times.

In the example diagram below you can see there is a runtime dependency cycle between A -> B -> C -> D -> E -> A -> ...etc

F has a compile time dependency on D, but because of the dependency cycle it means F also has a transitive compile-time dependency on A, B, C, and E! Now whenever any of those files change, F would also need to be recompiled.

How do I know what type my dependencies are?

There are actually 3 types of dependencies:

• Compile-time, which happen in 2 main cases:
  1) When writing any code in the module body — i.e. outside of a function, like our example with module attributes
  2) When using Macros
• Run-time, which happen for any usual function calls
• Export dependencies — these are compile-time dependencies, but re-compilation is only needed for them when the module API changes (i.e. function args/arities change or a struct definition changes). They occur when you reference a struct from another module, or import the module

What next?

In the next 2 articles we’ll go through examples to help test your understanding, before finally sharing strategies to help improve compile times. Also Multiverse is hiring so if you love working on interesting technical challenges and would like to help us build an outstanding alternative to university we’d love to hear from you!

How to speed up your Elixir compile times (part 1) — understanding Elixir compilation was originally published in multiverse-tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

I’ve developed an open-source data testing and a quality tool called data-flare. It aims to help data engineers and data scientists assure the data quality of large datasets using Spark. In this post I’ll share why I wrote this tool, why the existing tools weren’t enough, and how this tool may be helpful to you.

Who spends their evenings writing a data quality tool?

In every data-driven organisation, we must always recognise that without confidence in the quality of our data, that data is useless. Despite that there are relatively few tools available to help us ensure our data quality stays high.

What I was looking for was a tool that:

• Helped me write high performance checks on the key properties of my data, like the size of my datasets, the percentage of rows that comply with a condition, or the distinct values in my columns
• Helped me track those key properties over time, so that I can see how my datasets are evolving, and spot problem areas easily
• Enabled me to write more complex checks to check other facets of my data that weren’t simple to incorporate in a property, and enabled me to compare between different datasets
• Would scale to huge volumes of data

The tools that I found were more limited, constraining me to simpler checks defined in yaml or json, or only letting me check simpler properties on a single dataset. I wrote data-flare to fill in these gaps, and provide a one-stop-shop for our data quality needs.

Show me the code

data-flare is a Scala library built on top of Spark. It means you will need to write some Scala, but I’ve tried to keep the interface simple so that even a non-Scala developer could quickly pick it up.

Let’s look at a simple example. Imagine we have a dataset containing orders, with the following attributes:

• CustomerId
• OrderId
• ItemId
• OrderType
• OrderValue

We can represent this in a Dataset[Order] in Spark, with our order being:

case class Order(customerId: String, orderId: String, itemId: String, orderType: String, orderValue: Int)

Checks on a single dataset

We want to check that our orders are all in order, including checking:

• orderType is “Sale” at least 90% of the time
• orderTypes of “Refund” have order values of less than 0
• There are 20 different items that we sell, and we expect orders for each of those
• We have at least 100 orders

We can do this as follows (here orders represents our Dataset[Order]):

As you can see from this code, everything starts with a ChecksSuite. You can then pass in all of your checks that operate on single datasets using the singleDsChecks. We’ve been able to do all of these checks using SingleMetricChecks — these are efficient and perform all checks in a single pass over the dataset.

What if we wanted to do something that we couldn’t easily express with a metric check? Let’s say we wanted to check that no customer had more than 5 orders with an orderType of “Flash Sale”. We could express that with an Arbitrary Check like so:

The ability to define arbitrary checks in this way gives you the power to define any check you want. They won’t be as efficient as the metric based checks, but the flexibility you get can make it a worthwhile trade-off.

Checks on a pair of datasets

Let’s imagine we have a machine learning algorithm that predicts which item each customer will order next. We are returned another Dataset[Order] with predicted orders in it.

We may want to compare metrics on our predicted orders with metrics on our original orders. Let’s say that we expect to have an entry in our predicted orders for every customer that has had a previous order. We could check this using Flare as follows:

We can pass in dualDsChecks to a ChecksSuite. Here we describe the datasets we want to compare, the metrics we want to calculate for each of those datasets, and a MetricComparator which describes how those metrics should be compared. In this case we want the number of distinct customerIds in each dataset to be equal.

What happens when you run your checks?

When you run your checks all metrics are calculated in a single pass over each dataset, and check results are calculated and returned. You can then decide yourself how to handle those results. For example if one of your checks gives an error you could fail the spark job, or send a failure notification.

What else can you do?

• Store your metrics and check results by passing in a metricsPersister and qcResultsRepository to your ChecksSuite (ElasticSearch supported out the box, and it’s extendable to support any data store)
• Graph metrics over time in Kibana so you can spot trends
• Write arbitrary checks for pairs of datasets

For more information check out the documentation and the code!

Why I Built an Opensource Tool for Big Data Testing and Quality Control was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

There is a lot to love about the X2, and for some that might be enough. For those who want to be able to put in a decent amount of their own power the gearing will likely be a turn off. For those after a convenient, low effort and speedy commute option it could be just the thing.

For me — I’m still undecided. I love that I can commute in my work clothes sweat free, saving about 25% of time compared to my touring bike, but the gearing is a constant annoyance. We’ll see if I return it within the 30 day return period..!

Pros:

• Fantastic ride quality
• Well designed app and bike technology (locking, power control, battery indicator, speedo)
• Use of the boost button makes up for some of the weaker points
• Excellent customer service (London store anyway)
• Built in lights

Cons:

• 2 speed automatic gear box shifts too early, making your cadence very low at times and putting undue pressure on your knees
• Some aspects of design not thought through — no water bottle holder, rear rack not ideal

Ride Quality

The ride quality on the X2 is superb. Power is smoothly applied, the seating position is comfortable. It’s probably one of the most comfortable bikes I’ve ridden! The boost button is also handy for getting quickly up to speed off the lights, whilst still feeling controlled and natural. I’ve not had anyone overtake me since I got the bike! Also generally it just feels like a well-designed, premium product.

The anti-theft technology

The X2 comes with Vanmoof’s famous anti-theft technology — a clever integrated lock which will alert you and the Vanmoof team if your bike is stolen. Thankfully I’ve not had to put this into action but certainly from a day to day use perspective it seems well thought through.

The gearing

The X2 has 2 speed automatic (yep you read that right) hub gearing. While this makes your cadence quite low when starting off, it’s not a big issue as you very quickly can up to a higher speed.

The HUGE issue with the gearing is that it shifts up to the higher gear at far too low a speed. Just as your cadence starts being reasonable the gear shift hits you. It’s slightly irritating when accelerating on the flat, pretty annoying on a small hill, but is an absolute killer on a big hill. You simply can’t put any power in with your legs without killing your knees, leading to gradual deceleration unless you just hammer the boost button on big hills.

Once I’m up to 15mph the ride cadence in the higher gear is ok, but from 8/9mph upwards it’s terrible. You can’t configure when it shifts, there is no option to adjust it, and for me it’s a real pity.

Other minor points

The X2 has built-in lights — great for a commuter who doesn’t want to worry about constantly removing and replacing lights, running out of batteries, etc.

The chain guard — it’s a minor point but genuinely really helpful, especially when commuting in your work clothes. One could be fitted to most other bikes though.

There are no bosses for attaching a water bottle. You could probably rig something up but I still feel I shouldn’t have to — this has never been an issue with previous bikes!

The rear rack is generally ok, but with some niggles. First off it is quite slim, though for most people that would be ok providing you don’t want to strap large boxes down on it. The bigger issue is my Ortlieb panniers are in danger of hitting the locking button on the wheel (in the shop they advised putting my pannier on the other side!), which would be rather terrifying. However their engineers did offer to fit a bracket to stop that happening, an offer which I’ve not yet taken them up on.

Following on from the introductory post on serialization with spark, this post gets right into the thick of it with a tricky example of serialization with Spark. I highly recommend attempting to get this working yourself first, you’ll learn a lot!

Each example steps through some ways you may try to debug the problems, eventually resulting in a working solution. Once again the code samples can be found on ONZO’s Github, and the numbering on this article should match up with the code there :)

9 — base example

**FAILS**

Now for some practice! This example is relatively complex and needs a few changes to work successfully. Can you figure out what they are? Kudos if so! The next few examples walk through a solution step by step, and some things you may try.

10 — make classes serializable

**FAILS**

One approach to serialization issues can be to make everything Serializable. However in this case you will find it doesn’t solve the issue. You’ll find it easier (but not that easy..!) to spot why if you look at the complete examples. It’s because when trying to serialize the classes it will find references to testRddand also the shouldBe method. This will trigger serialization of the test class (you can see the full code in Github) that contains these, and the test class is not serializable.

11a — use anon function

**FAILS**

In order to debug this you might try simplifying things by replacing the WithFunction class with a simple anonymous function. However in this case we still have an a failure, can you spot the issue now?

11b — use anon function, with enclosing

**PASSES**

Did you spot it? By enclosing the reduceInts method the map function can now access everything it needs in that one closure, no need to serialize the other classes!

12a — use function with def

**FAILS**
Taking small steps, we now replace the anonymous function with a function declared with a def. Again you will find this fails, but seeing why isn’t easy. It is because of the intricacies of how def works. Essentially a method defined with def contains an implicit reference to this, which in this case is an object
which can’t be serialized. You can find out more about the differences between def and val here.

12b — use function with val

**PASSES**
Declaring the method with val works. A val method equates to a Function1 object, which is serializable, and doesn’t contain an implicit reference to this, stopping the attempted serialization of the Example object.

12c — use function with val explained part 1

**FAILS**
This example serves to illustrate the point more clearly. Here the addOnefunction references the one value. As we saw earlier this will cause the whole Exampleobject to be serialized, which will fail.

**BONUS POINTS**
One helpful experiment to try here is to resolve this by making the Exampleobject serializable.
You will note that you still get a serialization error. Can you see why? There are actually 2 reasons:
1) testRddis referenced inside the WithSparkMap class, leading to the whole Spec trying to be serialized (please see Github link for full code which will explain this more!)
2) The shouldBemethod is also referenced, again leading to the whole Spec trying to be serialized

12d —use function with val explained part 2

**PASSES**
As above, the best way to fix the issue is to reference values only in the more immediate scope. Here we have added oneEnc, which prevents the serialization of the whole Exampleobject.

13 — back to the problem, no class params

**PASSES**
Coming back from the issue we originally had, now we understand a little more let's introduce our WithFunction class back in. To simplify things we've taken out the constructor parameter here. We're also using a val for the method rather than a def. No serialization issues now!

14 — back to the problem, with class params

**FAILS**
We’ve now added back in the class params. Can you spot why this fails? The plusOnefunction references num, outside of the immediate scope,
again causing more objects to be serialized which is failing.

15a — back to the problem, with class params, and enclosing

**PASSES**
This is now a simple fix, and we can enclose the num value with encNum which resolves the last of our serialization issues. Finally, this is a complete working example that is equivalent to our first implementation that failed!

15b — adding some complexity e.g.7b — testing understanding

**FAILS**
One more failing example! Can you see why the above fails?

The issue is that encNumwon’t be evaluated until plusOne is actually called, effectively within the map function. At this point then the numvalue will need to be accessed, causing additional serialization of the containing object and the failure here.

Conclusion

Hopefully these examples have made a little clearer how serialization of functions works with Spark and Scala, good luck with your Spark serialization challenges!

Serialization challenges with Spark and Scala — Part 2— Now for something really challenging… was originally published in ONZO Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Apache Spark is a great tool for high performance, high volume data analytics. When working with Spark and Scala you will often find that your objects will need to be serialized so they can be sent to the Spark worker nodes. Whilst the rules for serialization seem fairly simple, interpreting them in a complex code base can be less than straightforward! If you get things wrong then far more than you intended can end up being Serialized, and this can easily lead to run time exceptions where the objects aren’t serializable.

This post will talk through a number of motivating examples to help explain what will be serialized and why. There will shortly be a follow up post to work through a much more complex example too if you would like a challenge!

Serialization Rules

Before we get into examples let’s explore the basic rules around serialization with respect to Spark code.

When will objects need to be Serialized?

When you perform a function on an RDD (Spark’s Resilient Distributed Dataset), or on anything that is an abstraction on top of this (e.g. Dataframes, Datasets), it is common that this function will need to be serialized so it can be sent to each worker node to execute on its segment of the data.

What gets Serialized?

The rules for what is Serialized are the same as in Java more generally — only objects can be serialized.

The function being passed to map (or similar Spark RDD function) itself will need to be Serialized (note this function is itself an object). If references to other objects are made within this function then those objects will also need to be serialized. The whole of these objects will be serialized, even when accessing just one of their fields.

Examples

Examples including code and explanations follow, though I strongly encourage you to try running the examples yourself and trying to figure out why each one works or doesn’t work — you’ll learn much more this way! All the examples along with explanations can be found on ONZO’s Github here.

For each of these examples assume we have a testRdd containing Integers.

val testRdd: RDD[Int]

Basic(ish) Examples

We’ll start with some basic examples that draw out the key principles of Serialization in Spark.

1 — basic spark map

object Example {
  def myFunc = testRdd.map(_ + 1)
}

**PASSES**

A very simple example — in this case the only thing that will be serialized is a Function1 object which has an apply method that adds 1 to it’s input. The Example object won’t be serialized.

2 — spark map with external variable

object Example {
  val num = 1
  def myFunc = testRdd.map(_ + num)
}

**FAILS**

Very similar to the above, but this time within our anonymous function we’re accessing the num value. Therefore the whole of the containing Example object will need to be serialized, which will actually fail because it isn’t serializable.

3 — spark map with external variable — the first way to fix it

object Example extends Serializable {
  val num = 1 
  def myFunc = testRdd.map(_ + num)
}

**PASSES**

One solution people often jump to is to make the object in question Serializable. It works, but may not be desirable as ideally we want to be serializing as little as possible.

4 — spark map with external variable — a flawed way to fix it

object Example {
  val num = 1
  def myFunc = {
    lazy val enclosedNum = num
    testRdd.map(_ + enclosedNum)
  }
}

**FAILS**

In this case we create an enclosedNum value inside the scope of myFunc — when this is referenced it should stop trying to serialize the whole object because it can access everything required the scope of myFunc. However because enclosedNum is a lazy val this still won’t work, as it still requires knowledge of num and hence will still try to serialize the whole of the Example object.

5 — spark map with external variable — properly fixed!

object Example {
  val num = 1
  def myFunc = {
    val enclosedNum = num
    testRdd.map(_ + enclosedNum)
  }
}

**PASSES**

Similar to the previous example, but this time with enclosedNum being a val, which fixes the previous issue.

Upping the difficulty — examples with nested objects

The same principles apply in the following examples, just with the added complexity of a nested object.

6 — nested objects, a simple example

object Example {
  val outerNum = 1
  object NestedExample extends Serializable {
    val innerNum = 10
    def myFunc = testRdd.map(_ + innerNum)
  }
}

**PASSES**

A slightly more complex example but with the same principles. Here innerNum is being referenced by the map function. This triggers serialization of the whole of the NestedExample object. However this is fine because it extends Serializable. You could use the same enclosing trick as before to stop the serialization of the NestedExample object too.

7 — nested objects gone wrong

object Example {
  val outerNum = 1
  object NestedExample extends Serializable {
    val innerNum = 10
    def myFunc = testRdd.map(_ + outerNum)
  }
}

**FAILS**
In this case outerNum is being referenced inside the map function. This means the whole Example object would have to be serialized, which will fail as it isn't Serializable.

8 — nested objects, using enclosing in the inner object

object Example {
  val outerNum = 1
  object NestedExample extends Serializable {
    val innerNum = 10
    val encOuterNum = outerNum
    def myFunc = testRdd.map(_ + encOuterNum)
  }
}

**PASSES**

In this example we have fixed the previous issue by providing encOuterNum. Now the map references only values in the NestedExample object, which can be serialized.

What’s next?

Stay tuned for the next post which will walk through a much more complex example, truly testing your understanding of serialization in Spark.

New post now available here!
https://medium.com/onzo-tech/serialization-challenges-with-spark-and-scala-part-2-now-for-something-really-challenging-bd0f391bd142

Serialization challenges with Spark and Scala was originally published in ONZO Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.