Performance Tips

Ignite .NET In-Memory Data Grid performance and throughput vastly depends on the features and the settings you use. In almost any use case the cache performance can be optimized by simply tweaking the cache configuration.

Disable Internal Events Notification

Ignite has rich event system to notify users about various events, including cache modification, eviction, compaction, topology changes, and a lot more. Since thousands of events per second are generated, it creates an additional load on the system. This can lead to significant performance degradation. Therefore, it is highly recommended to enable only those events that your application logic requires. By default, event notifications are disabled.

var cfg = new IgniteConfiguration
{
    IncludedEventTypes =
    {
        EventType.TaskStarted,
        EventType.TaskFinished,
        EventType.TaskFailed
    }
};

<igniteConfiguration>
    <includedEventTypes>
        <int>TaskStarted</int>
        <int>TaskFinished</int>
        <int>TaskFailed</int>
    </includedEventTypes>
</igniteConfiguration>

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    ... 
    <!-- Enable only some events and leave other ones disabled. -->
    <property name="includeEventTypes">
        <list>
            <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED"/>
            <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/>
            <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/>
        </list>
    </property>
    ...
</bean>

Tune Cache Start Size

In terms of size and capacity, Ignite's internal cache map acts exactly like a normal .NET Hashtable or Dictionary: it has some initial capacity (which is pretty small by default), which doubles as data arrives. The process of internal cache map resizing is CPU-intensive and time-consuming, and if you load a huge dataset into cache (which is a normal use case), the map will have to resize a lot of times. To avoid that, you can specify the initial cache map capacity, comparable to the expected size of your dataset. This will save a lot of CPU resources during the load time, because the map won't have to resize. For example, if you expect to load 100 million entries into cache, you can use the following configuration:

var cfg = new IgniteConfiguration
{
    CacheConfiguration = new[]
    {
        new CacheConfiguration
        {
            StartSize = 100 * 1024 * 1024
        }
    }
};

<igniteConfiguration>
    <cacheConfiguration>
        <cacheConfiguration startSize="104857600" />
    </cacheConfiguration>
</igniteConfiguration>

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    ...
    <property name="cacheConfiguration">
        <bean class="org.apache.ignite.configuration.CacheConfiguration">
            ...
            <!-- Set initial cache capacity to ~ 100M. -->
            <property name="startSize" value="#{100 * 1024 * 1024}"/> 
            ...
        </bean>
    </property>
</bean>

The above configuration will save you from log₂(10⁸) − log₂(1024) ≈ 16 cache map resizes (1024 is an initial map capacity by default). Remember, that each subsequent resize will be on average 2 times longer than the previous one.

Turn Off Backups

If you use PARTITIONED cache, and the data loss is not critical for you (for example, when you have a backing cache store), consider disabling backups for PARTITIONED cache. When backups are enabled, the cache engine has to maintain a remote copy of each entry, which requires network exchange and is time-consuming. To disable backups, use the following configuration:

var cfg = new IgniteConfiguration
{
    CacheConfiguration = new[]
    {
        new CacheConfiguration
        {
            CacheMode = CacheMode.Partitioned,
            Backups = 0
        }
    }
};

<igniteConfiguration>
    <cacheConfiguration>
        <cacheConfiguration cacheMode="Partitioned" backups="0" />
    </cacheConfiguration>
</igniteConfiguration>

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    ...
    <property name="cacheConfiguration">
        <bean class="org.apache.ignite.configuration.CacheConfiguration">
            ...
            <!-- Set cache mode. -->
            <property name="cacheMode" value="PARTITIONED"/>
            <!-- Set number of backups to 0-->
            <property name="backups" value="0"/>
            ...
        </bean>
    </property>
</bean>

🚧
Possible Data Loss
If you don't have backups enabled for PARTITIONED cache, you will loose all entries cached on a failed node. It may be acceptable for caching temporary data or data that can be otherwise recreated. Make sure that such data loss is not critical for application before disabling backups.

Tune Eviction Policy

Evictions are disabled by default. If you do need to use evictions to make sure that data in cache does not overgrow beyond allowed memory limits, consider choosing the proper eviction policy. An example of setting the LRU eviction policy with maximum size of 100000 entries is shown below:

var cfg = new IgniteConfiguration
{
    CacheConfiguration = new[]
    {
        new CacheConfiguration
        {
            EvictionPolicy = new LruEvictionPolicy { MaxSize = 1000000 }
        }
    }
};

<igniteConfiguration>
    <cacheConfiguration>
        <cacheConfiguration>
          	<evictionPolicy type='LruEvictionPolicy' maxSize='1000000' />
        </cacheConfiguration>
    </cacheConfiguration>
</igniteConfiguration>

<bean class="org.apache.ignite.cache.CacheConfiguration">
    ...
    <property name="evictionPolicy">
        <!-- LRU eviction policy. -->
        <bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
            <!-- Set the maximum cache size to 1 million (default is 100,000). -->
            <property name="maxSize" value="1000000"/>
        </bean>
    </property>
    ...
</bean>

Regardless of which eviction policy you use, cache performance will depend on the maximum amount of entries in cache allowed by eviction policy - if cache size overgrows this limit, the evictions start to occur.

Tune Cache Data Rebalancing

When a new node joins topology, existing nodes relinquish primary or back up ownership of some keys to the new node so that keys remain equally balanced across the grid at all times. This may require additional resources and hit cache performance. To tackle this possible problem, consider tweaking the following parameters:

Configure rebalance batch size, appropriate for your network. Default is 512KB which means that by default rebalance messages will be about 512KB. However, you may need to set this value to be higher or lower based on your network performance.
Configure rebalance throttling to unload the CPU. If your data sets are large and there are a lot of messages to send, the CPU or network can get over-consumed, which consecutively may slow down the application performance. In this case you should enable data rebalance throttling which helps tune the amount of time to wait between rebalance messages to make sure that rebalancing process does not have any negative performance impact. Note that application will continue to work properly while rebalancing is still in progress.
Configure rebalance thread pool size. As opposite to previous point, sometimes you may need to make rebalancing faster by engaging more CPU cores. This can be done by increasing the number of threads in rebalance thread pool (by default, there are only 2 threads in pool).

Below is an example of setting all of the above parameters in cache configuration:

var cfg = new IgniteConfiguration
{
    CacheConfiguration = new[]
    {
        new CacheConfiguration
        {
            RebalanceBatchSize = 1024 * 1024,
            RebalanceThrottle = TimeSpan.Zero  // disable throttling
        }
    }
};

<igniteConfiguration>
    <cacheConfiguration>
        <cacheConfiguration rebalanceBatchSize="1048576" rebalanceThrottle="0:0:0" />
    </cacheConfiguration>
</igniteConfiguration>

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    ...
    <property name="cacheConfiguration">
        <bean class="org.apache.ignite.configuration.CacheConfiguration">             
            <!-- Set rebalance batch size to 1 MB. -->
            <property name="rebalanceBatchSize" value="#{1024 * 1024}"/>
 
            <!-- Explicitly disable rebalance throttling. -->
            <property name="rebalanceThrottle" value="0"/>
            ... 
        </bean
    </property>
</bean>

Configure Thread Pools

By default, Ignite has it's main thread pool size set to the 2 times the available CPU count. In most cases keeping 2 threads per core will result in faster application performance, since there will be less context switching and CPU caches will work better. However, if you are expecting that your jobs will block for I/O or any other reason, it may make sense to increase the thread pool size. Below is an examples of how you configure thread pools:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    ... 
    <!-- Configure internal thread pool. -->
    <property name="publicThreadPoolSize" value="64"/>
    
    <!-- Configure system thread pool. -->
    <property name="systemThreadPoolSize" value="32"/>
    ...
</bean>

Use IBinarizable Whenever Possible

It is a very good practice to have every object that is transferred over network implement Apache.Ignite.Core.Binary.IBinarizable. These may be cache keys or values, jobs, job arguments, or anything else that will be sent across network to other grid nodes. Implementing IBinarizable may sometimes result in over 10x performance boost over standard serialization.

Force Timestamp format for DateTime values

Platform Interoperability explains how DateTime can be serialized in two ways, either as Timestamp (8 bytes) or as an object.

Timestamp format is preferred:

More efficient and compact
Interoperable with Ignite on other platforms (Java, C++, Python, etc)

We recommend enforcing Timestamp format globally:

new IgniteConfiguration
{
    ...
    BinaryConfiguration = new BinaryConfiguration
    {
        Serializer = new BinaryReflectiveSerializer
        {
            ForceTimestamp = true
        }
    }
};

new IgniteClientConfiguration
{
    ...
    BinaryConfiguration = new BinaryConfiguration
    {
        Serializer = new BinaryReflectiveSerializer
        {
            ForceTimestamp = true
        }
    }
};

Use Collocated Computations

Ignite enables you to execute MapReduce computations in memory. However, most computations usually work on some data which is cached on remote grid nodes. Loading that data from remote nodes is very expensive in most cases and it is a lot more cheaper to send the computation to the node where the data is. The easiest way to do it is to use ICompute.AffinityRun() method. There are other ways, including ICacheAffinity.MapKeysToNodes() methods. The topic of collocated computations is covered in much detail in Affinity Collocation, which contains proper code examples.

Use Data Streamer

If you need to upload lots of data into cache, use IDataStreamer to do it. Data streamer will properly batch the updates prior to sending them to remote nodes and will properly control number of parallel operations taking place on each node to avoid thrashing. Generally it provides performance of 10x than doing a bunch of single-threaded updates. See Data Loading section for more detailed description and examples.

Batch Up Your Messages

If you can send 10 bigger jobs instead of 100 smaller jobs, you should always choose to send bigger jobs. This will reduce the amount of jobs going across the network and may significantly improve performance. The same regards cache entries - always try to use API methods that take collections of keys or values, instead of passing them one-by-one.

Tune Garbage Collection

If you are seeing spikes in your throughput due to Garbage Collection (GC), then you should tune JVM settings. The following JVM settings have proven to provide fairly smooth throughput without large spikes:

-XX:+UseParNewGC 
-XX:+UseConcMarkSweepGC 
-XX:+UseTLAB 
-XX:NewSize=128m 
-XX:MaxNewSize=128m 
-XX:MaxTenuringThreshold=0 
-XX:SurvivorRatio=1024 
-XX:+UseCMSInitiatingOccupancyOnly 
-XX:CMSInitiatingOccupancyFraction=60

Disable Internal Events Notification

Tune Cache Start Size

Turn Off Backups

🚧Possible Data Loss

Tune Eviction Policy

Tune Cache Data Rebalancing

Configure Thread Pools

Use IBinarizable Whenever Possible

Force Timestamp format for DateTime values

Use Collocated Computations

Use Data Streamer

Batch Up Your Messages

Tune Garbage Collection

🚧
Possible Data Loss