This year, I proposed to change the MJIT’s design to simplify the Ruby VM, which will ultimately make the YJIT development easier. Since the new version is no longer a “M”ethod “JIT”, which MJIT was named after, I changed the project name to RJIT. RJIT is a pure-Ruby JIT compiler, so it’s called “R”uby “JIT”.

RJIT by k0kubun · Pull Request #7448 · ruby/ruby

Why does Ruby have two JITs?

It’s not rare that a language supports multiple JITs. For example, the Java HotSpot VM has the JVM Compiler Interface (JVMCI) that allows you to integrate a JIT like the Graal Compiler.

In the case of Ruby, the reference implementation CRuby doesn’t have public interface to integrate a third-party JIT. However, while CRuby already has YJIT that optimizes production pretty well, people have still developed other CRuby JITs, such as vnmakarov/ruby (sir-mirjit), tenderlove/tenderjit, and jhawthorn/hawthjit.

Since Ruby 3.2 MJIT was partly rewritten in Ruby, it became possible to integrate a third-party JIT like those by monkey-patching it. It’s always fun to write your own compiler. If you just remove MJIT, you won’t be able to enjoy that anymore. It’s also a great place to experiment with ideas for JIT optimization that can later be applied to YJIT. So I decided to keep this project while my primary interest remains to be making YJIT the best JIT for Ruby.

Which JIT should I use in production?

YJIT. Period.

YJIT is written in Rust while RJIT is in Ruby. While its peak performance could be as fast as YJIT if RJIT generates the same code as YJIT, there will always be a drawback in the warmup speed and memory footprint. I don’t think RJIT will ever beat this trade-off. Fast warmup and low memory footprint is crucial for companies like Shopify where developers deploy Ruby applications quite often to massive-scale clusters.

Furthermore, Shopify use YJIT in their tier-1 applications. They are business-critical, so if a problem is found in YJIT, the YJIT team fixes it very quickly. You can’t expect the same thing from a hobby project maintained in a single person’s spare time, RJIT. The YJIT team is a team full of kind and talented engineers. I’d like to once again thank the team and Shopify for having me as a teammate.

Conclusion

RJIT is there for your fun. Enjoy your JIT development in Ruby. If you’re interested, my RubyKaigi 2023 talk will cover how to develop and/or contribute to a Ruby JIT. See you in Matsumoto!

"MJIT Does Not Improve Rails Performance"

As I wrote in Ruby 3.0.0 Release Note and my previous post, we thought:

it is still not ready for optimizing workloads like Rails, which often spend time on so many methods and therefore suffer from i-cache misses exacerbated by JIT.

We thought Ruby 3’s JIT a.k.a. MJIT Does Not Improve Rails Performance and Rails Ruby Bench gets slower, not faster. That’s what I observed last year too.

Because of that and to support environments without compilers, Matz is thinking about introducing JIT that does not invoke compilers, MIR or YJIT.

Why does Rails become slower on Ruby 3’s JIT?

People have thought:

• They were very focused on this one synthetic benchmark called Optcarrot.
• GCC is not really equipped to optimize dynamically typed code.
• There are a lot of useful optimizations that got lost in the shuffle, and a lot of code that hasn’t yet been rewritten to use it.

I mean, I've literally thought about optimizing Rails by MJIT for over 3 years, and I've introduced several optimizations designed for such workloads. We do rely on GCC to ease our maintenance, but the reason why it optimizes some benchmarks is that it implements various Ruby VM-level optimizations that don’t really depend on GCC, not just that it uses GCC. Most of RTL-MJIT’s optimizations were already ported to or experimented with YARV-MJIT after we shuffled the code to make it work with YARV, and it’s now faster than RTL-MJIT because of that and many other improvements after it.

Then… why?

Is it because we use C compilers?

MJIT writes C code and invokes a C compiler to generate native code. Slow compilation time aside, which doesn't impact the peak performance, I see two performance problems in using C compilers as a JIT backend.

First, using a C compiler for code generation makes it hard to implement some low-level techniques, such as calculating a VM program counter from a native instruction pointer, retrieving VM stack values from a native stack, patching machine code, etc. One such optimization that YJIT implements and MJIT doesn’t is dispatching JIT-ed code from direct threading just like normal VM instructions. I ported the technique to MJIT and even tested generating a whole method with an inline assembler, but it didn’t seem to help the situation and MJIT’s dispatch wasn’t really slower than that.

Second, there’s an overhead caused by using dlopen as a loader. Whenever you call dlopen, it creates multiple memory pages for different ELF sections. More importantly, methods generated by different dlopen calls are on different pages. However, we removed the latter problem by periodically recompiling all methods to a single binary, which I call "JIT compaction". Also, function calls from a shared object are a bit slow. In fact, Ruby becomes a little slower if you compile Ruby with --enable-shared. The same overhead applies to MJIT. Though it’s usually just a couple of cycles after the first call is made and dynamic resolution is finished. I also tested loading .o files using shinh/objfcn, but compiling all methods in a single file and using dlopen weren’t slower than loading .o files.

i-cache misses by compiling many methods?

I’ve measured JIT-ed code with Linux perf so many times. I even wrote a perf plugin for it. Through the investigation, I observed cycles stalled for filling i-cache. Because the number of i-cache misses generally increases when you compile more methods, we changed the default of --jit-max-cache from 1000 to 100 in Ruby 2.7 to address the problem. And it indeed helped Rails performance. So I’ve believed JIT-ing a lot of methods is a bad thing.

However, Shopify’s YJIT, which compiles every method called more than once, achieved the same performance as the interpreter after compiling 4000 methods. Which, doesn’t really make sense if you assume the more methods you JIT, the slower the code runs.

What if we just compile all methods like YJIT? I tested it in older versions and it didn’t go well, which is why I haven’t tried it again until very recently. But in Ruby 3.0, I implemented an optimization that eliminates most of the code duplication across different methods and significantly reduces code size. With that, could MJIT-generated code perform well just like YJIT?

The “compile all” magic

Yes, that’s what we needed. If you compile all benchmarked methods instead of just the top 100 methods, the JIT makes Rails faster.

See this gist for benchmark details. Note that some benchmarks are forked by me for reasons explained later. Following results are measured under a single-threaded condition and run at a Rack level.

Sinatra Benchmark

This benchmark was from an article saying "enabling the JIT makes Sinatra 11% slower!", which just returns a static string from Sinatra.

Guess what? JIT makes it 11% faster instead of slower now.

Rails Simpler Bench

Rails Simpler Bench was authored by Noah Gibbs. The default benchmark endpoint returns a static text. So it's a Rails version of the above benchmark.

JIT makes it 1.04x faster. The difference is not as significant as Sinatra, but this is the first time we see the JIT makes Rails faster, not slower.

Railsbench

Railsbench measures the performance of HTML #show action of a controller generated by "rails g scaffold". It was originally headius/pgrailsbench used by the JRuby team, I forked it as k0kubun/railsbench to upgrade Rails, and Shopify/yjit-bench uses that too.

Despite its simplicity and benchmark convenience, it captures many real-world Rails characteristics. It uses ActiveRecord, for instance.

It's 1.03x faster. Not significant, but it's way better than a slowdown, right?

Discourse

Discourse is one of the most popular open-source Rails applications in the world. You can see it at Twitter Developers, GitHub Support, CircleCI Discuss, and more.

So it's pretty clear that Discourse is a real-world Ruby workload. The important thing in this context is that it has its own benchmark harness. Noah Gibbs's Rails Ruby Bench is based on Discourse too.

JIT can make Discourse 1.03x faster! One complication here, however, is that Discourse enables TracePoint and all JIT-ed code is canceled when TracePoint is enabled. Because Zeitwerk uses TracePoint, I needed to switch its autoloader back to the classic autoloader. So it's not ready to use the JIT for Discourse yet. But this proves the JIT is useful for real-world application logic.

Why does compiling everything make it faster?

To spot locations impacting metrics like i-cache misses, I've tried tools like perf and cachegrind before, but because the slowness appears everywhere and no single place shows a significant difference, it's hard to tell the exact reason.

Let's take a look at how the number of compiled methods impacts the performance with Railsbench.

The number of benchmarked methods was roughly 1200. When you have only VM interpretation, it's faster than most of the VM/JIT-mixed execution. But once you compile everything and use JIT-ed code most of the time, it actually becomes faster than the baseline.

What Maxime, the author of YJIT, thought was that thrashing between VM and JIT is the biggest issue because they could evict each other from i-cache and less-predictable VM dispatch makes it harder to prefetch code. She also proved in her 30k_methods benchmark that large code can run faster as long as you stay in linear predictable JIT-ed code. MJIT would share many of these characteristics too even while its code may not be as linear as YJIT's.

I think our significant code size reduction in Ruby 3.0 contributed to making this possible as well.

So, should I use JIT on Rails?

We're almost there, but we have some stuff to be fixed before bringing this to production.

Ruby 3.0.1 bug that's not in 3.0.0

One of the backports to Ruby 3.0.1 causes JIT compilation failures. 3.0.1 is significantly slower than 3.0.0 if you use JIT. Let's wait for the next release.

Ruby 3 bug that stops compilation in the middle

There's a bug that the JIT worker stops compiling the remaining methods in edge cases. You cannot reproduce the above speed if it happens. Please wait until this fix is backported and released.

Incompatibility with Zeitwerk / TracePoint

In ruby 2.5, Koichi, the author of TracePoint, implemented an optimization that makes overall Ruby execution faster as long as you never use TracePoint. I've thought you should not enable TracePoint if you care about performance. Thus MJIT has not supported TracePoint.

However, Byroot has found that it doesn't have a significant impact on VM performance nowadays. So the problem is only associated with JIT. Given that Zeitwerk disables TracePoint after eager loading finishes, now I'm thinking about re-enabling all JIT-ed code when all TracePoints get disabled.

Incompatibility with GC.compact

We either need to compromise performance by making every memory access indirect or recompile many methods whenever GC.compact is called. It has not been implemented, so all JIT-ed code is canceled for now.

The default value of --jit-max-cache

To compile everything, the default 100 of --jit-max-cache was too small. I'll change it to maybe 10000 in Ruby 3.1, but you'd need to override the default for Ruby 3.0.

Scalability of "JIT compaction"

If you want to reproduce the above benchmark results, you have to see the "JIT compaction" log at the end of --jit-verbose=1 logs. However, because the compacted binary has all methods and we currently have no way to distinguish which version of JIT-ed code is in use, the compacted code is almost always not GC-ed properly. Because of that, we restricted the max number of "JIT compaction" to 10% of --jit-max-cache (10 times by default). If JIT continues to compile methods after the last JIT compaction, it'll be slow.

Also, it takes a lot of time before "JIT compaction" is triggered. See this benchmark result for how long MJIT takes to reach the peak performance. If you measure MJIT's performance while GCC is running or before "JIT compaction", it would be often slower than the baseline.

I hope using MIR instead of GCC will shorten the warmup time in the future.

Next steps

I don't care which JIT backend we use to generate machine code out of MJIT, MIR, or YJIT. What I do care about the most is the performance in real-world workloads and optimization ideas implemented in the JIT compiler on top of the backend, which will continue to be useful even if you replace MJIT with MIR and/or YJIT.

In addition to approaching the problems discussed above, I'm thinking about working on the following things, which were slightly changed from the previous article.

Ruby-based JIT compiler

I considered "Ractor-based JIT worker" in the previous article because process invocation is something Ruby is good at and I thought doing that at a Ruby level might make it easier to maintain locks, in addition to using Ruby to easily maintain complicated optimization logic.

However, I found that running multiple Ractors has its own overhead unlike running only the default Ractor. So now I'm thinking to continue to maintain the JIT worker in C, but starting and stopping a Ractor whenever the JIT compilation happens so that we won't see the multi-Ractor overhead once it reaches the peak performance. We'd need to check if the overhead is insignificant compared to GCC's compilation time first though.

Faster deoptimization

One theme in the previous article was "On-Stack Replacement", but I split it to "Faster deoptimization" and the next "Lazy stack frame push" this time. Ultimately, On-Stack Replacement is just one way to implement those optimizations, and whether we use OSR or not doesn't really matter.

One of the reasons why we don't support TracePoint and GC.compact was to avoid adding too many branches in the code for deoptimization. If there's no extra overhead for doing it, it'll be easier to introduce the support.

Lazy stack frame push

From Ruby 3.0, we started to annotate methods that don't need any stack frame to be pushed. The number of methods we can annotate as such is limited for now because object allocation might raise NoMemoryError and a backtrace needs the frame. If we can lazily push the frame when it's being raised, we can skip pushing the frame. It's tricky to implement that with MJIT, but it'd be useful for reducing the method call overheads.

Sponsors

My contribution to OSS like the JIT, ERB, Haml, and IRB and my own projects like Hamlit, sqldef, pp, mitamae, and rspec-openapi has been supported by the following GitHub Sponsors. Thank you!

Ruby 3.0 JIT is the fastest JIT ever for MRI. However, despite Ruby 3.0's big improvement in reducing i-cache misses, it's still not ready for optimizing Rails applications. Stay tuned for Ruby 3.1.

"Is Ruby 3 Actually Three Times Faster?"

Ruby 3 is indeed 3x faster than Ruby 2.0 in a Ruby 3x3 benchmark, Optcarrot:

As proposed by Eregon, we measured performance after 3000 frames to measure peak time performance with Optcarrot.

Note that Ruby's JIT compiler methodology using a C compiler called "MJIT" was originally implemented by Vladimir Makarov. The JIT of Ruby 2.6 / 2.7 was slower than his implementation because we only partially merged his work and replaced the compiler to mitigate risks.

I've implemented lots of optimizations in Ruby 3, and finally it performs better than his original version. Ruby 3 JIT is the fastest JIT which MRI has ever had, which of course couldn't be achieved without his great work.

However, Optcarrot is just one of the two official Ruby 3 benchmarks. Please see Is Ruby 3 Actually Three Times Faster? for details, which was written by Noah Gibbs, the author of the other benchmark rails_ruby_bench.

Should I enable JIT on my Rails applications?

Not yet.

Let me quote what I wrote in the Ruby 3.0.0 release note:

As of Ruby 3.0, JIT is supposed to give performance improvements in limited workloads, such as games (Optcarrot), AI (Rubykon), or whatever application that spends majority of time in calling a few methods many times.

Although Ruby 3.0 significantly decreased a size of JIT-ed code, it is still not ready for optimizing workloads like Rails, which often spend time on so many methods and therefore suffer from i-cache misses exacerbated by JIT. Stay tuned for Ruby 3.1 for further improvements on this issue.

The JIT compiler generates native code and Ruby needs to run the JIT-ed code in addition to existing VM instructions. Apparently Rails uses more features than Optcarrot, e.g. Optcarrot doesn't trigger GC, and therefore a CPU's i-cache has little space for accommodating extra code.

As explained above, Ruby 3 decreased a generated code size for example from 1.3MB to 260KB in 100 methods of a Sinatra benchmark, which improved performance of the Sinatra benchmark, but still it was not enough.

Ruby 3.1 JIT

I still haven't given up to optimize web applications made by Rails or whatever frameworks by JIT, because that's where many people and I use Ruby for production.

I know some people are working on a lightweight JIT compiler project, such as MIR. Even if we introduce such a lightweight JIT as another tier of MRI JIT, for the best performance of long running applications like Rails, a heavyweight JIT should generate ideal native code for any method.

Given Ruby 3.0, I have some battle plans for the Ruby 3.1 JIT development.

Ractor-based JIT worker

The MJIT worker thread has been implemented using C, because Ruby has GVL for threads and we didn't want to block Ruby threads by what MJIT does.

Now we have Ractor. We can rewrite the MJIT worker implementation with Ractor to develop more complicated optimizations with Ruby. For example, if we annotate a method as side-effect free, we could call the method on compilation time to calculate the result beforehand, which we didn't want to do before because calling a Ruby method in an MJIT worker meant to acquire GVL.

On-Stack Replacement

On-Stack Replacement is a technique to replace code on stack. This is useful for eliminating various checks and branches in JIT-ed code because we can abandon too-optimized code outside the optimized code, for example when we redefine a method.

I know this is very hard to implement. It may be impossible as long as we use a C compiler for MJIT. Though I'm very convinced we need it for optimizing Ruby 3 further. For instance, because MRI has debug_inspector API which allows users to fetch any local variable in any frame from any method, MJIT currently doesn't optimize Ruby's local variables at all. If we can lazily move values from a native stack to Ruby's local variables when debug_inspector API is called, we can optimize them.

Reduce a code size further

Once we start using Ruby and have On-Stack Replacement, we should be able to reduce a JIT-ed code size further. I believe this effort will allow us to optimize a wider range of applications like Rails.

I'll also look for VM implementations that touch many cache lines and try to reduce their code size.

Sponsors

I can't thank my wife enough for supporting my development of Ruby. Ruby 3x3 couldn't be achieved without her help.

Also GitHub Sponsors, thank you for sponsoring my OSS development! Your support keeps me motivated even more.

We focused to improve Rails application performance in Ruby 2.7 JIT, but the last year's assumption was wrong and Ruby 2.7 JIT didn't meet the goal. We'll change the approach for it in Ruby 3.0.

Ruby 2.7 is released!

Merry Christmas! The new Ruby is out. As I've sometimes promised to make performance improvements in Ruby 2.7 JIT, let me explain how it went. See also The method JIT compiler for Ruby 2.6 and Ruby 2.6 JIT - Progress and Future in case you're not familiar with the context.

Since my original motivation to work on JIT is to improve Rails application's performance, I focused to improve JIT on such a workload. I prepared some more benchmarks using Rack, Sinatra, Rails and experimented some patches I believed were effective for Rails with them.

However, none of them didn't have a major impact on Rails benchmarks. I particularly spent a lot of time for "Optimized JIT-ed code dispatch", which will be explained later, as I thought it's very important last year. Therefore I changed my plan to stop thinking about it and optimize code more, but it was too late and the time is over to improve the Rails benchmark in Ruby 2.7.

Now I'm hoping to improve things in Ruby 3.0. To share the current progress towards that, let me summarize what I was doing in Ruby 2.7 development.

What's implemented in Ruby 2.7?

Default option changes

The easiest change, but maybe the largest impact on some benchmarks. Here's what's changed:

• --jit-max-cache: 1,000 → 100
• --jit-min-calls: 5 → 10,000

--jit-max-cache limits the number of methods to be JIT-ed. This is related to later "Optimized JIT-ed code dispatch" topic, because having a lot of JIT-ed code tends to increase the JIT-ed code dispatch overhead and therefore it impacts long-running application's performance a lot.

--jit-min-calls requires methods to be called such times before letting JIT worker consider compiling it. Because running a C compiler has a very big performance impact, Ruby's JIT is performing badly in most of existing Ruby benchmarks just for the C compiler's resource consumption. While I'm focusing on performance after all compilations, I thought changing this value has some easy benefits on such situations, at least skipping methods called only on startup.

These values are kind of influenced by OpenJDK's C2 compiler options.

Deoptimized recompilation

https://speakerdeck.com/k0kubun/rubyrussia-2019?slide=44

JIT has its own deoptimization mechanism. When some assumption like "this JIT-ed method is used only for this class" is not met, JIT-ed code fallbacks to VM interpretation because it's optimized using such an assumption.

While Ruby 2.6 continues to make the fallback forever, Ruby 2.7's JIT marks such methods to be recompiled and it generates less-optimized code which doesn't fallback to VM for the same assumption.

This was built in the original MJIT which was not merged in Ruby 2.6. In Ruby 2.7 development, we thought it's valuable for reducing the number of VM fallbacks, especially because Rails has many polymorphic class usages.

Frame-omitted method inlining

https://speakerdeck.com/k0kubun/rubyrussia-2019?slide=56

When we call a method, we push a frame to a call stack, evaluate the method, and pop the frame. The push and the pop of the frame has some overhead, especially for very fast operations like Integer#+. Therefore eliminating such frame manipulation has been implemented in VM level for some popular methods.

This "frame-omitted" method inlining introduces the same optimization for various more methods by JIT. For doing this, we need to know an inlined method has no chance to rely on the call stack. The CRuby's VM developers and I have developed VM instruction's attributes to examine such kind of things on methods and JIT is using the mechanism.

However, not so many methods are considered as frame-omittable for now. While we indeed spent time for making some instructions "pure" (no side effect) in Ruby 2.7 which is used for the optimization, we didn't make the optimization available for methods with a normal level of complexity. There are some rooms to be improved here for now.

What's not committed in Ruby 2.7?

Optimized JIT-ed code dispatch

https://speakerdeck.com/k0kubun/rubyrussia-2019?slide=64

This was the most promising thing to improve Rails performance on JIT. As mentioned in the --jit-max-cache explanation, it is the fact that a JIT-ed method dispatch becomes slow when there are many JIT-ed methods. Because I've benchmarked Rails with all methods compiled (up to --jit-max-cache), I thought JIT-ed code dispatch can be a bottleneck in the benchmark.

I managed to make a patch which offsets JIT-ed code dispatch overhead up to 100 methods, but in reality it didn't improve JIT's performance on Rails benchmark. So the bottleneck seems to live in another place.

Given that, I stopped working on eliminating low-level overhead. I changed my plan in Ruby 3.0 to just focus on optimizing compiled code instead.

Method inlining for more methods

The most important thing missing in the above "frame-omitted method inlining" now is that a method is not inlined if the method has local variables.

This is not super hard to solve, but I prioritized "Optimized JIT-ed code dispatch" after implementing the method inlining capability and making some important instructions pure.

Another important topic is writing core-class methods in Ruby. For example, if we define Numeric#zero? as `self == 0`, the JIT can consider the method as pure and it can perform frame-omitted method inlining on it, which can be a resolution for [Bug #15589].

Because there was "Write a Ruby interpreter in Ruby for Ruby 3" project by ko1, I was just waiting for his work. But it turned out that the current form of his work is basically inlining C code in Ruby code and therefore the JIT is blocked to inline such methods because it cannot examine whether C code is pure or not.

We're planning to mark C code with attributes like "pure" to make that happen. I thought it was for Ruby 2.7, but it just wasn't.

Stack-based object allocation

https://speakerdeck.com/k0kubun/rubyrussia-2019?slide=74

The optimization, allocating objects on stack instead of heap, is known to require a hard implementation, escape analysis. Though I thought it's possible to implement it in some very basic cases and I made an experimental implementation.

But even in such a very easy case, we need to know whether a method's receiver or arguments escapes or not, and this was blocked by the C code attributes work by ko1. I could have implemented a tentative version of it dedicated for JIT, but my time was spent for the "Optimized JIT-ed code dispatch" project and ko1 was working on many other things.

What will k0kubun do in Ruby 3.0?

I still don't think that the JIT can't optimize Rails. While it'd be important to achieve Ruby 3x3 in Optcarrot benchmark for some workloads, I'll continue to work on JIT in Ruby 3.0 to avoid slowing down Rails and optimize such a workload further.

Instead of focusing on the "Optimized JIT-ed code dispatch" project, I'll try to introduce optimizations which work well on Rails benchmarks, with the following strategies:

More per-instruction optimizations

You know more sophisticated JITs like HotSpot's C2 compiler or Graal are doing a very complicated task on optimizations. They'd be doing optimization over multiple instructions or methods. At some point Ruby's JIT should somehow have the thing to have the same level of optimization.

However, even in a single instruction level, the current implementation of Ruby JIT has a special optimization only in some very limited instructions. It optimizes a normal method call and instruction variables, but most of others are untouched and not doing per-instruction optimization, while C compilers can perform multi-instruction optimizations for some limited extent.

Once per-instruction optimizations are done well, we'll have more chance that C compilers can perform multi-instruction optimizations. Even in C code manipulated by JIT, it'd be harder to generate a simple code if per-instruction code were not already simplified.

The things I'm currently planning to try are:

• Revisit type-profiling instruction operands and generate type-specific code; The last attempt did not went well maybe because it increases the number of VM instructions. Maybe we'd be able to profile the same thing without having many instructions.
• Instance variable optimization for core-classes and their subclasses (VM-level preparation is done in Ruby 2.7, but JIT-level support is not done yet)
• Optimize getconstant; We haven’t touched this in JIT yet and maybe we can do something. To make constant folding happen in the future, we should be able to optimize constant references in per-instruction level first.

More inlining

Some more places are theoretically inlinable but not implemented to be inlined yet. Example things are:

• `super`; We attempted to implement VM-level method cache (prerequisite for JIT-level inlining) in Ruby 2.7, but we got a bug and abandoned it. We should debug the bug and introduce it again.
• C methods; Because it's hard to just rewrite C methods to Ruby methods without degrading the method's performance on VM, we'd need to continue to maintain things in C. Then we should consider supporting method inlining for C methods too, at least for methods used very often.
• Inlining Ruby methods with local variables; Explained above. It's just waiting to be implemented.

Not only inlining unblocks C compiler's optimizations, but also specializing code for inlining itself sometimes gives faster code because some values will be inlined in generated code instead of read from memory.

More C method's metadata

Things like allocating objects on stack and omitting frames for inlined C method would be blocked by the inability to know the behavior of C code. To unblock these optimizations, Ruby 3.0 should have a way to identify C method's behavior automatically.

As the first step, I'd like to have following attributes in C methods:

• pure (a.k.a. "leaf" in YARV instructions); It doesn't call another arbitrary method.
• noescape; We'd like to know whether a specific receiver or argument is not escaped by calling a C method.

Just adding the attributes is easy, but to avoid providing false information, we need to somehow automatically verify the information. @shyouhei did a good job for verifying "leaf" attribute in VM instructions on CI, and I hope we can do a similar thing in these new attributes too.

The method JIT compiler for Ruby 2.6

Fortunately my effort has not been reverted, so it will be officially released next week!

Over 10 months, Ruby’s JIT has evolved for the following focuses:

• Performance
• Portability
• Stability
• Security

Let me show you what we’ve achieved for each of them in Ruby 2.6.

Performance

At least for Ruby, JIT is introduced to improve performance, specifically for speed in my understanding. So, if it doesn’t make your application fast, it would be useless for you, right? Given the whole year, I believed I would be able to achieve small but meaningful performance improvement on real-world applications in 2.6. However, I failed to make it.

In an early stage of 2.6, some micro benchmarks have had the good progress.

Comparison:
                    while
         trunk+JIT: 493.8 i/s
2.6.0-preview1+JIT: 246.4 i/s - 2.00x  slower
    2.6.0-preview1:  86.8 i/s - 5.69x  slower
             trunk:  86.2 i/s - 5.73x  slower
             2.5.0:  80.9 i/s - 6.11x  slower
             2.0.0:  78.4 i/s - 6.30x  slower

And until 2.6.0-preview3, we've continued to improve the Optcarrot benchmark to measure Ruby 3x3 project, which hopes to let Ruby 3 be 3 times faster than Ruby 2.0.

Comparison:
             Optcarrot Lan_Master.nes
      2.6.0-preview3+JIT:        86.6 fps
      2.6.0-preview2+JIT:        73.9 fps - 1.17x  slower
      2.6.0-preview1+JIT:        59.2 fps - 1.46x  slower
          2.6.0-preview3:        54.6 fps - 1.59x  slower
          2.6.0-preview2:        53.3 fps - 1.62x  slower
          2.6.0-preview1:        53.0 fps - 1.63x  slower
                   2.5.3:        48.5 fps - 1.78x  slower
                   2.0.0:        34.6 fps - 2.50x  slower

This looks very great, and this is gonna be released as is in Ruby 2.6. It doesn't look so bad.

Guess what? It turned out that the NES emulator Optcarrot does not reflect a typical real-world workload in Ruby like Rails application so much. In Optcarrot, 2~3 methods which can be JIT-ed in a very short time are called very frequently, and the most busy method is large enough to be optimized well with MJIT's current optimization strategy.

In a typical Rails application, there's no single busy CPU-intensive method that can be optimized well by current MJIT. Noah Gibbs has written articles about MJIT's performance on Rails, and the latest article showed the current JIT does not make Rails fast.

After preview3, I called for Ruby benchmarks that are made slow by JIT:

k0kubun on Twitter

People, please give me Ruby benchmark which is made slower by passing --jit option on Ruby 2.6.0-preview3. That's super helpful at this moment because we still have 1~2 months to fix it before the 2.6.0 release. Examples (thanks!): https://t.co/HZD3JMLpdF https://t.co/EmFMIyS7RY

I collected them to experiment an easy fix that can be done within 1 month prior to the final 2.6 release, but I seem to have underestimated the difficulty of it, time flies, and I suddenly realized that this can't be fixed by a small change that can be rushed after 2.6.0-rcX releases.

Therefore, none of them hasn't been improved yet and apparently JIT on Ruby 2.6 failed to be ready for production. The only improvements we made in rc1 and rc2 were MJIT support in forked child processes and a bug fix for JIT on bootsnap.

So, it succeeded to be an interesting toy to experiment possible great future, but I failed to provide meaningful performance improvements as of Ruby 2.6. (Skip to the last "Future" section if you're interested in only performance.)

Portability

As written in the release note of Ruby 2.6-rc2, it aims to support JIT when it's built with following C compilers on non-EOL platforms.

• GCC
• Clang
• Microsoft Visual C++

When MJIT infrastructure is published by Vladimir Makarov, it had support for only GCC or Clang running on Unix environments, and it actually had some issues on non-Linux platforms like macOS or Windows MinGW.

As of my previous article, JIT on macOS was running properly, but we just passed builds on some other RubyCI environments or skipped test failures. MinGW had a fatal SEGV which had been hard to fix.

With a lot of investment on debugging Windows environments, I'm pleased to say that MJIT infrastructure is working fine on MinGW and Microsoft Visual C++. Both of them had its own difficult challenges. For example, we're relying on pre-processor-only mode of GCC and Clang but VC++ doesn't have that, then I ended up creating completely different build system. Who cares?

Helped by Nobuyoshi Nakada and Usa Nakamura, now MJIT is running for the above compilers at least on platforms listed in RubyCI.

Another good news is that now Heroku supports MJIT. Thank you so much for the effort on it!

Richard Schneeman 🤠 Beto for Senate 2020 on Twitter

Ruby 2.6.0 JIT now works on @heroku thanks to GCC being added to the stack image https://t.co/oAgO50lOGL ~ $ ruby --jit -e "RubyVM::MJIT.pause(wait: false)" ; echo $? 0

Stability

We've invested testing environment of MJIT as well. To prevent bugs, we have:

• http://ci.rvm.jp/results/trunk-mjit-wait@silicon-docker
• http://ci.rvm.jp/results/trunk-mjit@silicon-docker
• http://ci.rvm.jp/results/trunk-no-mjit@silicon-docker
• https://app.wercker.com/ruby/ruby/runs

ci.rvm.jp is a continuous testing environment maintained by Sasada Koichi. It's running for 24 hours even while any commit is not made. It has been revealed bugs which rarely occur.

On that, we have 3 types of MJIT-related builds. trunk-mjit-wait is for running all tests with "--jit-wait", which blocks when a method is queued to be JIT-ed and synchronously waits for JIT compilation, so that it exposes VM or JIT compiler-related bugs. trunk-mjit is running with "--jit" because synchronous compilation can't test SEGV or deadlocks on race condition. trunk-no-mjit is for testing configure flag --disable-mjit-support.

We also configured Wercker CI, which is dedicated for MJIT testing at this repository. It's running both --jit and --jit-wait testing. It's integrated with GitHub, and I think it's convenient for those who mainly watch GitHub for Ruby core development, including me.

These CI environments have reported many bugs which were not easy to fix. Have you ever considered what if Ruby script waits all child processes including MJIT’s C compiler process? Can you imagine what could happen on multi-threaded environment without GVL (for MJIT worker thread) that forks and execs on each thread and needs to deal with asynchronous signal handler? I can't count how many deadlocks, SEGVs, and esoteric test failures we've fixed. But now it's stable enough at least on the above CI environments, with a large help from Eric Wong.

We created the following continuous benchmark environments as well so that we can prevent unexpected performance regression:

• https://rubybench.org/
• https://benchmark-driver.github.io/

RubyBench has existed for a long time. This year original RubyBench developers added a support for measuring --jit, and I joined the team and maintained the --jit testing and benchmark driver part.

benchmark-driver.github.io is originally created for testing my benchmark tool benchmark_driver.gem at RubyGrant 2017, but it now exists for covering more test cases along with RubyBench. Benchmarking is hard because we should use the same machine without any parallelism forever and increasing the number of test cases in one benchmark system could require a lot of time for measuring one commit.

I succeeded to catch some unexpected performance regressions of MJIT at https://benchmark-driver.github.io/benchmarks/optcarrot/commits.html, and sometimes it contributed to expose random failure on a specific benchmark program.

Security

Ruby’s JIT compiler translates Ruby method to C code on /tmp, invokes C compiler with it, and dynamically loads the generated object file. Doesn’t it sound like dangerous?

We can’t make everything safe. For example, if attackers already have access to root user, they could do almost anything. So we want the JIT to be somewhat reasonably secure.

As a user, I would like to be able to prevent at least the following situations:

1. C compiler is replaced to an arbitrary binary
2. C header files or libraries are replaced
3. Another user’s program on the same server overwrites C code to be JIT-ed

For 1, Ruby binary remembers the full path of C compiler which is used to build Ruby interpreter, and Ruby’s JIT uses it on runtime. If C compiler is securely installed, unless an attacker has access to replace the original C compiler or somehow can introduce unexpected chroot on start, it would be hard to spoil it.

For 2, Ruby’s JIT uses only one header file in Ruby’s install directory and just links to usual libraries like ones used for normal C extensions. An attacker may need to access to Ruby’s install directory or C libraries to break it.

For 3, we create C source file with O_EXCL|O_CREAT to prevent any other program from changing the permission by creating a file with the same name beforehand. This part has been fine since I wrote my previous article.

Actually these parts were mostly improved by Nobuyoshi Nakada after the initial merge. Kudos to @nobu for a lot of improvements on build system!

Future

While I failed to make Ruby's JIT production-ready in 2.6, I don't regret what we've done this year. I needed to develop or lead all of the above things for Ruby 2.6, but I believe I can invest my time mainly on performance improvement in Ruby 2.7.

I assume the main reason of slow down on real-world applications is that the overhead of loading JIT-ed code and increasing the times to invoke new Virtual Machine over JIT-ed frame becomes larger than what we can optimize now.

I'm thinking about taking following 2 strategies to approach this:

• Inline Ruby methods as further as possible to avoid loading code or creating a VM frame many times
• Optimize object allocations which we've not invested so much in JIT so far

The former is relatively easy for me because I experimented method inlining and have a patch for 1-level inlining. I just need to change it to support multi-level inlining. I'm also planning to take time to learn other JIT implementation's strategy to decide what kind of unit to compile.

The latter might require to implement escape analysis for allocating objects on stack, which is known to be difficult to implement. But I'm sure that there's a room to improve in it and real-world applications allocate more objects than Optcarrot.

Also there may be an ongoing experiment for MRI JIT by another person and it might be published for Ruby 2.7 or 3.0.

I hope that we'll be able to say "MJIT makes Rails X times faster" in near future, until we hit version 3.0.

benchmark_driver.gem's example usage that outputs a graph:

Project Summary

Here is a copy of the project summary which I sent to Ruby Association:

This project aims to improve benchmark_driver.gem, which is built to compare performance of different Ruby binaries easily and precisely. It also aims to increase the number of benchmark test cases to cover more Ruby core features, and make it easier to optimize Ruby 3x faster by preparing an environment to continuously benchmark the tests with the tool.

I also created some detailed milestones for it and achieved all of them during the project period.

What’s benchmark_driver.gem?

benchmark_driver.gem is originally designed as a successor of benchmark/driver.rb in Ruby language repository, which can measure a Ruby script performance accurately by subtracting time taken for while loop from a looped script to measure, and which can benchmark multiple Ruby binaries at the same time and compare them.

While the original benchmark/driver.rb hard-codes the loop count of while loop to be subtracted for some specific benchmark name prefixes, benchmark_driver.gem has the count as loop_count by design and thus you can adjust the loop count as you like. It can also automatically calculate the loop count which is expected to run for 3s (changeable by --run-duration option).

One great part of benchmark_driver.gem is a variety of built-in output formats and measurable metrics, and extensibility of them.
benchmark_driver.gem not only can measure execution time, i/s and memory, but also can integrate custom metrics of existing Ruby benchmark
like Optcarrot's fps. And you can output the results in a sexy comparison output similar to benchmark-ips, markdown or a graph image. Both metrics and outputs are fully pluggable.

Basic usage: Ruby interface

benchmark_driver.gem has 2 benchmark definition formats: Ruby and YAML.
Ruby interface is designed for a usual Ruby benchmark or a use case to generate a benchmark target script dynamically, while YAML format is good for minimizing effort to write benchmark definitions and for flexibly changing parameters on the fly without modifying code.

Let’s see the difference between an ordinary benchmark tool and benchmark_driver.gem.

This is a benchmark example with benchmark-ips.gem:

require 'benchmark/ips'

class Array
  alias_method :blank?, :empty?
end

Benchmark.ips do |x|
  array = []
  x.report('Array#empty?') { array.empty? }
  x.report('Array#blank?') { array.blank? }
  x.compare!
end

With Ruby 2.5, It outputs a result like:

Warming up --------------------------------------
        Array#empty?   524.659k i/100ms
        Array#blank?   495.794k i/100ms
Calculating -------------------------------------
        Array#empty?     15.497M (± 2.5%) i/s -     77.650M in   5.013969s
        Array#blank?     14.171M (± 2.1%) i/s -     70.899M in   5.005282s

Comparison:
        Array#empty?: 15497274.4 i/s
        Array#blank?: 14171303.0 i/s - 1.09x  slower

From the result, you may understand ActiveSupport’s `Array#blank?` is not so slow compared to Ruby’s built-in `Array#empty?`. Let’s try measuring the same thing in benchmark_driver.gem.

Here is the example of benchmark_driver.gem usage with Ruby interface:

require 'benchmark_driver'

Benchmark.driver do |x|
  x.prelude %{
    class Array
      alias_method :blank?, :empty?
    end
    array = []
  }
  x.report 'Array#empty?', %{ array.empty? }
  x.report 'Array#blank?', %{ array.blank? }
end

When you run the above script with Ruby 2.5.0 (on Linux to see clocks/i), you will get an output like:

Warming up --------------------------------------
        Array#empty?    56.340M i/s
        Array#blank?    42.795M i/s
Calculating -------------------------------------
        Array#empty?   181.135M i/s -    169.019M times in 0.933111s (5.52ns/i, 23clocks/i)
        Array#blank?    99.275M i/s -    128.386M times in 1.293235s (10.07ns/i, 44clocks/i)

Comparison:
        Array#empty?: 181134991.8 i/s
        Array#blank?:  99275008.9 i/s - 1.82x  slower

So this result shows `Array#empty?` is actually about 1.8x faster than `Array#blank?`. The result is reasonable because Ruby’s optimized instruction for `Array#empty?` (opt_empty_p) is applied only if method name is "empty?".

Why does this result difference happen? There are 2 reasons:

• Overhead of calling a block has large overhead, compared to just running a part of while loop. benchmark_driver.gem takes a benchmark definition as string to dynamically generate such a loop script, instead of taking a script as a block.
• As I said before, it subtracts while loop overhead.

You may claim that benchmark-ips.gem also takes a string instead of a block for a measured script, which actually is exactly the same interface as benchmark_driver.gem’s `x.report`, but it doesn’t have `x.prelude` and so you can’t have a predefined local variable. I assume that it’s not designed for such usage.

Like `Array#empty?`, there are some methods which can run even faster than the overhead of calling a block. As such methods are optimized because they are frequently used, it’s important to measure performance of such methods accurately.

Comparing multiple Ruby binaries

Another reason why benchmark_driver.gem is good for measuring Ruby’s performance is that it can run any Ruby binaries at the same time and compare the benchmark results.

Here is the example to compare the performance between multiple Ruby implementations. In `x.rbenv`, you can specify Ruby binaries managed by rbenv as "[shown name]::[rbenv name],[arg1],[arg2]…".

require 'benchmark_driver'

Benchmark.driver do |x|
  x.prelude %{
    def script
      i = 0
      while i < 1000_000
        i += 1
      end
      i
    end
  }
  x.report 'while', %{ script }
  x.loop_count 2000

  x.rbenv(
    '2.0.0::2.0.0-p0',
    '2.5.0',
    '2.6.0-dev',
    '2.6.0-dev+JIT::2.6.0-dev,--jit',
  )
  x.verbose
end

And here is the output:

2.0.0: ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-linux]
2.5.0: ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux]
2.6.0-dev: ruby 2.6.0dev (2018-03-21 trunk 62870) [x86_64-linux]
2.6.0-dev+JIT: ruby 2.6.0dev (2018-03-21 trunk 62870) +JIT [x86_64-linux]
Calculating -------------------------------------
                          2.0.0       2.5.0   2.6.0-dev  2.6.0-dev+JIT
               while     77.952      80.325      87.239        491.907 i/s -      2.000k times in 25.656691s 24.898879s 22.925498s 4.065807s

Comparison:
                            while
       2.6.0-dev+JIT:       491.9 i/s
           2.6.0-dev:        87.2 i/s - 5.64x  slower
               2.5.0:        80.3 i/s - 6.12x  slower
               2.0.0:        78.0 i/s - 6.31x  slower

Obviously this tool is convenient to know: what times is the current Ruby faster than Ruby 2.0? Please try comparing performance with JRuby, Rubinius and truffleruby.

Not only plain text output like this, but also you can visualize this as a graph using a plugin (explained later).

Advanced usage: YAML and CLI

benchmark_driver.gem has CLI like this (descriptions are omitted because the width of medium is too small...):

$ benchmark-driver -h
Usage: benchmark-driver [options] [YAML]
    -r, --runner [TYPE]              
    -o, --output [TYPE]              
    -e, --executables [EXECS]        
        --rbenv [VERSIONS]           
        --repeat-count [NUM]         
        --bundler                    
        --filter [REGEXP]            
        --verbose [LEVEL]            
        --run-duration [SECONDS]     

It takes YAMLs as its arguments.

Here is an example of a benchmark definition in YAML.

prelude: |
  large_a = "Hellooooooooooooooooooooooooooooooooooooooooooooooooooo"
  large_b = "Wooooooooooooooooooooooooooooooooooooooooooooooooooorld"
  small_a = "Hello"
  small_b = "World"

benchmark:
  large: '"#{large_a}, #{large_b}!"'
  small: '"#{small_a}, #{small_b}!"'

If you save it as “benchmark.yml”, you can run it like:

$ benchmark-driver benchmark.yml --rbenv '2.4.3;2.5.0'
Warming up --------------------------------------
               large     3.693M i/s
               small     9.913M i/s
Calculating -------------------------------------
                          2.4.3       2.5.0
               large     3.895M      5.485M i/s -     11.079M times in 2.844249s 2.019943s
               small    11.755M     11.103M i/s -     29.740M times in 2.529966s 2.678612s

Comparison:
                            large
               2.5.0:   5484705.8 i/s
               2.4.3:   3895155.8 i/s - 1.41x  slower

small
               2.4.3:  11755264.9 i/s
               2.5.0:  11102923.0 i/s - 1.06x  slower

This shows characteristics of Ruby 2.5’s string interpolation performance improvement.

In YAML output, you can simplify the definition by its format, and you can specify Ruby executable without modifying code. Please use either of them depending on your use case.

Plugin System

You can customize metrics to be measured and a format for output as you like,
and build your own versions if you want.

The plugin interface has evolved dramatically and it’s not so stable yet,
which is the only reason I don’t release it as 1.x, but you'll be able to continue using the same interface unless you're a plugin developer.

To know what plugin is available, refer to benchmark_driver's README: https://github.com/k0kubun/benchmark_driver#output-options

You may be interested in a graph output, markdown output, memory runner or the "command_stdout" runner to integrate any existing Ruby benchmarks without building a runner plugin.

benchmark-driver.github.io

benchmark-driver

The goals of this project included that “preparing an environment to continuously benchmark the tests with the tool”. I wish I could use https://rubybench.org or its temporary fork to use benchmark_driver.gem, but a server which I could borrow for benchmark can’t be reached from the Internet.

To make sure my goal is achieved in a short period using the server, instead of contributing to RubyBench, I decided to create a system that can generate a static page to be published on GitHub pages. As we can easily create a plugin
to integrate benchmark_driver.gem with any services, we’ll be able to merge my works to RubyBench project later. After that, benchmark-driver.github.io might be obsoleted. The major difference between RubyBench and the site for now is that it includes JIT results.

If you are interested in how it works, you may want to see:
https://github.com/benchmark-driver/skybench
https://github.com/benchmark-driver/benchmark-driver.github.io

In this section, I’ll mainly describe about the benchmark sets which are included in the site.

Ruby Core

benchmark-driver/ruby-benchmarks

They are benchmarks in Ruby repository, but they are converted to YAML format to abstract away the while loop which I mentioned above.

While the benchmark results are also avalable on RubyBench, you can see JIT-ed results on benchmark-driver.github.io. But note that current Ruby’s JIT is method JIT and many of those benchmarks don’t create a method to be JIT-ed. In such a situation, JIT results in these benchmark might be useless and you may want to see “MJIT benchmarks” instead.

Ruby Method

benchmark-driver/ruby-method-benchmarks

This is originally created by @Watson1978 at https://github.com/Watson1978/ruby-method-benchmarks. He has actively improved Ruby’s performance by measuring many Ruby core features and find parts to work on, using the benchmark set.

Having a good coverage of benchmarked features is good to catch the performance regression. You may be able to find Ruby core features to be improved in the results.

MJIT

benchmark-driver/mjit-benchmarks

The benchmark set is created by Vladimir Makarov. As I said above, a benchmark hotspot needs to be a method to measure method JIT’s performance. At the same time, we want to omit the overhead of method call as much as possible. The benchmark set seems to be designed for measuing method JIT performance correctly.

After benchmark is finished for recent revisions, you’ll be able to see latest JIT performance improvements.

Optcarrot

benchmark-driver/optcarrot

It’s created by @mame and the repository adds only benchmark.yml to it.
You may want to see https://www.slideshare.net/mametter/optcarrot-a-pureruby-nes-emulator for the details of the amazing program for benchmark.

Future works

Outside the scope which was originally proposed for this project, we have several things to work on to make the best use of the outcome.

Sophisticating plugin interface

Right now the interface between Runner and Output is like this:

module BenchmarkDriver
  Metrics = ::BenchmarkDriver::Struct.new(
    :value,      # @param [Float]
    :executable, # @param [BenchmarkDriver::Config::Executable]
    :duration,   # @param [Float,nil]
  )

  Metrics::Type = ::BenchmarkDriver::Struct.new(
    :unit,          # @param [String]
    :larger_better, # @param [TrueClass,FalseClass]
    :worse_word,    # @param [String]
    defaults: { larger_better: true, worse_word: 'slower' },
  )
end

This was good enough to express all of execution time, memory consumption as max resident set and optcarrot fps, and to make a comparison or a graph from it.

But what if want to show real time and user/system CPU time at the same time? What about showing 50/75/90/99 percentile in one graph?

To achieve such use case, I’m planning to change only one Float value to a Hash as { Symbol => Float }. It should have a Symbol that should express the main metrics, and I’ll define some well-known Symbols to achieve special outputs available for limited plugins.

Add more benchmarks to benchmark-driver.github.io

Only measuring micro benchmarks and NES emulator performance sounds not enough to achieve Ruby 3x3. But adding benchmarks is a little challenging because it’s already taking a lot of time to finish running all benchmarks for one revision. Also, installing many gems for each revision will trigger a disk consumption problem if we keep many Ruby revisions for future benchmark addition.

Anyway, I’ll show some future canditates to be added to benchmark-driver.github.io.

Discourse

https://github.com/noahgibbs/rails_ruby_bench
https://github.com/discourse/discourse

Discourse is a popular Rails application and has script/bench.rb to measure its performance. And Noah Gibbs created its improved version as rails_ruby_bench.

Since one of major Ruby use cases is Rails, we definitely need to measure Ruby performance with Rails. This is blocked by “showing 50/75/90/99 percentile” issue and a disk consumption problem for now. Rails has too many dependencies compared to Optcarrot…

Fluentd

https://github.com/benchmark-driver/fluentd-benchmark

Fluentd is a log collector written in Ruby, which is used by so many large-scale services https://www.fluentd.org.So Fluentd might be a good real-world Ruby use case whose performance is very important.
Disclaimer: Note that I’m an employee of Treasure Data, a company building Fluentd.

I modified one_forward benchmark in https://github.com/fluent/fluentd-benchmark so that we can use it from benchmark_driver.gem. Currently it’s not on benchmark-driver.github.io because the benchmark result does not look affected by Ruby’s performance. I’m not sure why, and we need to investigate the cause to help Fluentd become faster.

Integrating derailed_benchmarks.gem

You may want to use your Rails application for a benchmark test case for Ruby 3x3. If I integrate derailed_benchmarks.gem with benchmark_driver.gem, we may be able to compare Ruby’s performance using the Rails application easily.

Recently @schneems kindly gave me a commit bit of the gem. So I’ll be able to improve the situation from both sides.

Conclusion

benchmark_driver.gem is good for many use cases of Ruby's benchmark.
Please try it and give me a feedback to achieve Ruby 3x3.

Acknowledgements

The success of this project would be impossible without help from Ruby Association. Especially Koichi Sasada, a mentor of this project, gave me many good ideas to improve it. Thank you so much.

k0kubun on Twitter

I've just committed the initial JIT compiler for Ruby. It's not still so fast yet (especially it's performing badly with Rails for now), but we have much time to improve it until Ruby 2.6 (or 3.0) release. https://t.co/7mO5FZM80C

Then I gave a talk about it yesterday at my company's conference.

As it isn't ultra-fast yet, I haven't shared it deeply until the conference. But some of you guys kindly wrote great articles about it.

• Playing with ruby's new JIT: MJIT
• Ruby's New JIT

Probably they are easier to understand than my article. But I found several things to point out or announce. So I decided to write this article.

What's "MJIT"?

Let me share the summary about recent JIT topics on Ruby. Those who read the above articles may want to skip this section.

The RTL instruction project

In July 2016, Vladimir Makarov, or Vlad, who optimized Hash at Ruby 2.4 filed a ticket "VM performance improvement proposal" (Feature #12589). It's a project to replace all Ruby VM's stack-based instruction set with register-based one, which is named RTL (Register Transfer Language) instructions.

He is also known as the author of GCC's register allocator and a maintainer of its instruction scheduler. I assumed that he wanted to apply some optimization knowledge in GCC to Ruby.

While it's such a big change to replace many parts of Ruby VM, he successfully passed `make test` with RTL instruction implementation. It succeeded to improve some micro benchmarks, but a larger benchmark named Optcarrot wasn't improved.

MJIT: JIT compiler that runs on RTL instructions

To approach the problem, in March 2017, he also published JIT compiler which compiles RTL instructions, which is named MJIT (MRI JIT).

He discussed some possible JIT methods in rtl_mjit_branch and chose the approach that generates C code to a disk, lets C compiler compile it to .so and loads machine code from the object file.

JIT succeeded to make the Optcarrot result 2 or 3 times faster.

On the other hand, as a side effect of largely rewriting the implementation of VM, although the `make test` is passed, a more severe test such as make test-all` is not passed regardless of whether JIT is enabled or disabled, and Rails fails to start on the RTL instructions.

YARV-MJIT: Yet another JIT compiler that runs on YARV instructions

In this way, RTL instruction conversion was a somewhat risky change, but my company implements cloud service as Rails application and provides it, so very high availability is required. If high risk changes were made to Ruby, upgrading Ruby would be very difficult.

After Vlad's session at RubyKaigi last year, I thought about how to reduce the risk to introduce JIT compiler to Ruby. Then I decided to implement another approach called YARV-MJIT, which compiles original stack-based YARV instructions instead of RTL instructions.

The JIT approach is very safe because it can be achieved without modifying VM implementation at all. Even if you find some bug after next Ruby release, you can ensure Ruby works as before by disabling optional JIT compiler.

As it doesn't change bytecode dynamically to avoid risks, it can't reach the performance of original MJIT. But with some improvements, I succeeded to make its performance comparable with MJIT.

By removing some premature aggressive optimizations, YARV-MJIT could pass all Ruby test cases with and without JIT enablement, keeping the performance improvement in some level. I proposed to merge it to Ruby in late 2017, and it's successfully merged this month after fixing some potential bugs and random crashes.

Ruby's JIT status

Ruby 2.6 merged following 2 components.

• JIT Compiler: YARV-MJIT
• JIT Infrastructure: MJIT

I call the infrastructure part as "MJIT infrastructure" in source code or ticket because "MJIT" was originally the combination of the part and the built-in JIT compiler. Sometimes people call both of them just as MJIT, but I think it's okay. Even Ruby committers don't say YARV-MJIT.

I'll share how Ruby's JIT implementation is going.

JIT compiler benchmarks

Even though the conservative JIT compiler has the aim to minimize risks of JIT introduction, of course it must improve the performance because the change is for optimization. I'll share some benchmark results on my machine: Intel 4.0GHz i7-4790K 8 Cores, 16GB memory, x86_64 Linux.

Optcarrot

This is the latest result of Optcarrot benchmark. While YARV-MJIT achieved about 68fps previously, it became a little slower because it dropped some premature optimizations.

In my Pull Request to merge JIT compiler, I attached the benchmark result with 63fps. Shannon's article seems to quote the result, but as I previously updated in the PR's description, it's temporarily changed to 58fps. I know the cause of the gap between 63fps and 58fps, and now I'm working on fixing it.

While we can't know which commit contributes to VM performance because recently RubyBench stops to be updated, recent shyouhei's VM improvements (like this) seem to make Ruby 2.6 much faster even without JIT compiler.

It's a shame for me that my JIT compiler doesn't contribute to Ruby 2.6's performance improvements so much, but the improvements of VM performance sounds a good news for sure.

Benchmark in “Playing with ruby’s new JIT”

Is the JIT compiler's optimization limited to only about 11%?

The article "Playing with ruby's new JIT: MJIT" has a benchmark script. And the author of truffleruby shared improved version of the benchmark.

def calculate(a, b, n = 40_000_000)
  i = 0
  c = 0
  while i < n
    a = a * 16807 % 2147483647
    b = b * 48271 % 2147483647
    c += 1 if (a & 0xffff) == (b & 0xffff)
    i += 1
  end
  c
end

Benchmark.ips do |x|
  x.iterations = 3
  x.report("calculate") do |times|
    calculate(65, 8921, 100_000)
  end
end

Here is its result:

$ ruby -v
ruby 2.6.0dev (2018-02-15 trunk 62410) [x86_64-linux]
$ ruby bench.rb
Warming up --------------------------------------
    calculate    13.000  i/100ms
    calculate    13.000  i/100ms
    calculate    13.000  i/100ms
Calculating -------------------------------------
    calculate    1.800k (± 2.7%) i/s - 8.996k in   5.002504s
    calculate    1.785k (± 7.4%) i/s - 8.853k in   5.003616s
    calculate    1.802k (± 4.0%) i/s - 8.996k in   5.006199s
$ ruby --jit bench.rb
Warming up --------------------------------------
    calculate    13.000  i/100ms
    calculate    18.000  i/100ms
    calculate    27.000  i/100ms
Calculating -------------------------------------
    calculate    7.182k (± 9.1%) i/s - 35.397k in   5.000332s
    calculate    7.296k (± 2.9%) i/s - 36.450k in   5.001392s
    calculate    7.295k (± 3.1%) i/s - 36.450k in   5.002572s

1.802k is improved to 7.296k. It's about 4x faster. It seems that we already achieved Ruby 3x3!

Obviously the benchmark looks designed for JIT propaganda, but it suggests the potential performance improvements in the real world if we tune the benchmark for it.

Even for such an arbitrarily created benchmark, we should aim to improve CRuby in the order of hundred because truffleruby shows 370x performance on my machine.

Other benchmarks

The commit merging JIT compiler shows other benchmark results in the commit message. Some of micro benchmarks are improved about 2x faster compared to Ruby 2.6's VM execution.

It also includes discourse's benchmark result. I wanted to use rails_ruby_bench by Noah Gibbs, but for now I'm failing to run it on my machine and so using discourse's script/bench.rb. For now, JIT compiler even makes Rails a little slower. Sam Saffron pointed out that Unicorn is forking and MJIT is disabled in the child process. There are many things to be improved. I continue to investigate Rails performance with JIT.

VM-Generated JIT Compiler

As JIT compiler needs to generate code that works in the same way as VM implementation, a very simple JIT compiler would have many copy-pastes of VM implementation. It would be bad for maintainability.

Ruby VM is generated by the special template format "insns.def". ko1 suggested to generate JIT compiler from insns.def too. Just before merging JIT compiler to Ruby trunk, I tried it and it worked fine.

It's so simple that you can understand immediately. Ruby has mjit_compile.inc.erb like following one. (%-starting line is the special mode of ERB. Did you know that?)

switch (insn) {
% RubyVM::BareInstructions.to_a.each do |insn|
  case BIN(<%= insn.name %>):
<%= render 'mjit_compile_insn', locals: { insn: insn } -%>
  break;
% end
}

Branching by the type of VM instruction, _mjit_compile_insn_body.erb prints the content of insns.def with the following code.

% expand_simple_macros.call(insn.expr.expr).each_line do |line|
%   if line =~ /\A\s+JUMP\((?<dest>[^)]+)\);\s+\z/
      /* Dynamic generation of JUMP code */
%   else
      fprintf(f, <%= to_cstr.call(line) %>);
%   end
% end

It converts macros for JIT beforehand, and then JUMP macro is dynamically generated because it requires special conversion. From this ERB file, following C code is generated, which runs as JIT compiler on runtime.

switch (insn) {
  case BIN(nop):
    fprintf(f, "{\n");
    {
        fprintf(f, "    reg_cfp->pc = original_body_iseq + %d;\n", pos);
        fprintf(f, "    reg_cfp->sp = (VALUE *)reg_cfp->bp + %d;\n", b->stack_size + 1);
        fprintf(f, "    {\n");
        fprintf(f, "        /* none */\n");
        fprintf(f, "    }\n");
        b->stack_size += attr_sp_inc_nop();
    }
    fprintf(f, "}\n");
    break;
  case BIN(getlocal):
    /* ... */
}

Just listing many `fprintf`s. It's so easy to understand, right? When Ruby's main method requests to JIT a method in hotspot, the code in MJIT thread generates the following code.

VALUE
_mjit0(rb_execution_context_t *ec, rb_control_frame_t *reg_cfp)
{
    VALUE *stack = reg_cfp->sp;
    static const VALUE *const original_body_iseq = (VALUE *)0x5643d9a852a0;
    if (reg_cfp->pc != original_body_iseq) {
        return Qundef;
    }

label_0: /* nop */
{
    reg_cfp->pc = original_body_iseq + 1;
    reg_cfp->sp = (VALUE *)reg_cfp->bp + 2;
    {
        /* none */
    }
}

/* snip... */

} /* end of _mjit0 */

`rb_execution_context_t` is a kind of Thread context and `rb_control_frame_t` is one frame of call stack. The interface allows to execute a frame that holds a method's bytecode on a specific Thread context.

By compiling this and calling `dlopen` and `dlsym`, you can get the function pointer of it and VM calls the function instead of executing bytecode on VM.

Why is it made fast? Please read the slides on the top of this article if you are interested.

How is JIT infrastructure?

Support status of platforms

I could easily port pthread to Windows native thread using existing thread abstraction layer for Ruby.

Instead, it was so hard to successfully build a C header file which is used on JIT compilation on some platforms. After merging JIT infrastructure, following platforms were reported to be failing or broken on RubyCI.

• NetBSD
• Solaris
• AIX
• Intel C/C++ Compiler
• Old Visual Studio

I've tested the JIT infrastructure with Linux, macOS, MinGW and newer Visual Studio. But it wasn't sufficient for Ruby's portability requirement. After installing some VMs or using CI environments, I succeeded to fix them. Now we can build the JIT header on many platforms.

Then I added `--jit-wait` option for JIT compiler testing and wrote unit tests for it. Linux+gcc usually works perfectly, but some tests fail on clang and MinGW environments. Not only the test failure, but also JIT performance with clang is a little worse than gcc. So it may be hard to see improvement on macOS. And MinGW is skipping to transform the JIT header due to bugs in transformation. It results in the slowness of JIT compilation on MinGW. I'm working to fix those problems.

Security

Other core committers (nobu, usa) has improved many parts of JIT infrastructure. One of them is security.

MJIT worker thread does `fopen` with a path like `/tmp/_ruby_mjit_p123u4` when writing a C file to a disk. First of all, initial implementation didn't control the permission of the file (it was a known issue). It also didn't restrict opening the existing file. It means that the opened file may have arbitrary permission. It's fixed to force creating a new file.

When you test the JIT on production before release, please own your risk and be careful.

Others

About "Startup Time"

The article Ruby's New JIT pointed out that:

Startup time is one thing to take into consideration when considering using the new JIT. Starting up Ruby with the JIT takes about six times (6x) longer.

You found a very important part. The cause of it is that `--jit` option always starts to precompile the header for JIT on initializer, and it waits for the finish on Ruby process finalizer. So unfortunately it takes at least the time to compile one C header.

We have following 2 approaches to solve it.

• Building precompiled header on Ruby's build, not on runtime
• Immediately cancel JIT compiler thread regardless of its state

Even though it's named "precompiled header", it's actually not precompiled. That's because I suspect that the format of precompiled header may change after upgrading a C compiler used for JIT. If it's turned out to be wrong, we can take the approach. For now, I took the conservative strategy.

We can immediately cancel JIT compiler thread ignoring the state of it, but it might be dangerous in some situations.

`--jit-cc` does no longer exist

Sometimes C compiler fails to compile a header generated by another C compiler. It's not Ruby's fault and we're not going to support it.

Now Ruby trunk automatically uses C compiler which was used to build Ruby itself. If you want to test performance with clang, you need to build Ruby with clang.

Acknowledgements

I'm still not sure if my JIT compiler will be in the release of Ruby 2.6, but I want to say "Thank you" for following people.

• The inventor of MJIT: Vladimir Makarov
• Ruby's father: Matz
• Reviewers of YARV-MJIT code: ko1, mame
• Many bug reports and fixes: wanabe
• The first MinGW support patch: Lars Kannis
• Maintainer of many build environments: hsbt
• Core committers fixing MJIT infrastructure: nobu, usa, znz, knu
• Recent VM improvements: shyouhei

What's next?

We're preparing to release very early preview of Ruby 2.6.0. The performance may not be improved so much at the release, but I hope you enjoy hacking MJIT there.