Collections Evolution in .NET

In the evolving landscape of the .NET platform, efficient data structures are crucial in building high-performance applications. The framework has long provided various collection types, from simple arrays to sophisticated concurrent collections.

Traditional .NET collections fall into two main categories: mutable collections like List<T> and Dictionary<TKey, TValue>, which offer excellent performance but require careful handling in concurrent scenarios, and immutable collections like ImmutableList<T> and ImmutableDictionary<TKey, TValue>, which provide thread-safety at the cost of performance and memory overhead.

Get Vladyslav Bardin’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The journey of .NET collections reflects the evolving needs of modern software development, particularly in concurrent programming and functional paradigms. Let’s trace this evolution through its key stages.

Early Days: Mutable Collections

When .NET was first introduced, it provided a set of essential mutable collections in the System. Collections namespace. These collections were designed with simplicity and direct memory access in mind.

ArrayList list = new ArrayList();
list.Add("string");
list.Add(42); // Mixed types allowed, leading to potential runtime errors

With generics in .NET 2.0, type-safe collections are introduced in the System.Collections.Generic:

List<string> strings = new List<string>();
Dictionary<string, int> map = new Dictionary<string, int>();

The Concurrency Challenge

As multi-threaded programming became more prevalent, developers faced challenges with mutable collections:

// Potential race conditions
Dictionary<string, int> sharedData = new Dictionary<string, int>();


// Multiple threads accessing the dictionary
Task.WhenAll(
   Task.Run(() =>
   {
       if (!sharedData.ContainsKey("key")) sharedData.Add("key", 1);
   }),
   Task.Run(() =>
   {
       if (!sharedData.ContainsKey("key")) sharedData.Add("key", 2);
   })
);

This led to the introduction of thread-safe collections in the System.Collections.Concurrent:

ConcurrentDictionary<string, int> threadSafeData = new ConcurrentDictionary<string, int>();
// Thread-safe operations, but with performance overhead
threadSafeData.TryAdd("key", 1)

The Rise of Immutability

With the growing influence of functional programming and the need for thread-safe data structures, .NET Framework 4.5 introduced immutable collections:

var builder = ImmutableDictionary.CreateBuilder<string, int>();
builder.Add("one", 1);
builder.Add("two", 2);
ImmutableDictionary<string, int> immutable = builder.ToImmutable();

var immutable2 = ImmutableDictionary<string, int>.Empty
   .Add("one", 1)
   .Add("two", 2);

These collections offered several advantages:

• Thread safety by design.
• Predictable behavior in concurrent scenarios.
• Support for functional programming patterns.

However, they also came with notable drawbacks:

var immutable = ImmutableDictionary.CreateBuilder<string, int>()
   .ToImmutable();


// Each modification creates a new copy
var newDictionary = immutable.Add("three", 3); // Original remains unchanged


// Memory overhead for each modification
// Performance impact due to tree-based implementation

The Memory Efficiency Gap

While immutable collections solved many concurrency problems, they introduced memory and performance overhead:

// Memory profiling example
var mutable = new Dictionary<string, int>(10000);
var immutable = ImmutableDictionary.CreateRange(mutable);


/*
Typical memory usage comparison:
  – Mutable Dictionary: ~1x memory
  – Immutable Dictionary: ~2.5x memory due to tree structure
*/

This gap created a need for more efficient immutable collections, especially in scenarios where data is rarely modified but frequently read:

var initialSettings = new Dictionary<string, string>();
var configService = new ConfigService(initialSettings);

// Common pattern leading to inefficiency
public class ConfigService
{
   private readonly ImmutableDictionary<string, string> _settings;


   public ConfigService(IDictionary<string, string> initialSettings)
   {
       // Memory overhead for conversion
       // Performance overhead for tree-based lookups
       _settings = initialSettings.ToImmutableDictionary();
   }
}

Enter Frozen Collections

This evolution culminated in the introduction of Frozen Collections in .NET 8, which addressed the memory efficiency gap while maintaining immutability:

// Modern approach with Frozen Collections
var frozen = new Dictionary<string, int>
{
   ["one"] = 1,
   ["two"] = 2
}.ToFrozenDictionary();

Among the strengths of the frozen collections are:

• Zero-copy optimization(if applicable).
• Dense memory layout.
• Fast lookup performance.
• Thread-safe by design.

The evolution from mutable to immutable collections and finally to frozen collections represents a continuous improvement in balancing the competing needs of type and thread safety, memory consumptions & performance, as well developers convenience.

As .NET continues to evolve, these collections set a solid foundation for robust and scalable software, catering to ever-growing demands in software development.

For more insights on .NET and software engineering, connect with me on LinkedIn. You can also follow my YouTube channel, @bardin_speaks, and deep dive into advanced topics.