Description
Is there an existing issue for this?
- I have searched the existing issues
Describe the bug
Large File Upload Issue (still) with MultipartReader in ASP.NET Core -8+
- Problem Summary
- Why This Is an Issue
- Previous Reports of This Issue
- Violation of the Single Responsibility Principle
- Struggles and Final Thoughts
- Conclusion
- What Did I Do in the End?
- Issue Description
- References
Problem (one of many) Summary
The real problem even noting the below is that .NET / ASP.NET is simply incapable of uploading large files 4GB+ which frankly is rather a joke in 2025. With .NET 9 on the horizon and yet this fundamental web task still seems to be ignored.
Note: This is running in Visual Studio 2022 only using Kestrel. No other web server is involved. Before anybody starts commenting about the various Kestrel settings that can be changed I have pretty much tried them all in some form. (unless there is some hidden strange one not commonly known.)
- The
LengthLimit
is hard coded toHeadersLengthLimit
(16KB) during the constructor ofMultipartReader
. - Developers cannot modify
HeadersLengthLimit
before it's used because it's set after the constructor completes. - As a result, any multipart data (Header) exceeding 16KB causes an
InvalidDataException
, preventing large file uploads.
Why This Is an Issue
- The inability to adjust
HeadersLengthLimit
before it's enforced means developers cannot handle large file uploads. Given the violation of the Single Responsibility Principle discussed below, this limitation shouldn't even exist. - According to RFC 7578, which governs
multipart/form-data
, there is no specified maximum size for multipart uploads or headers. Therefore, this hard coded limit imposes an unnecessary restriction that is not compliant with the RFC. - Uploading large files is fundamental to many web applications. The lack of configuration in the
MultipartReader
class hinders developers from implementing essential functionality.
Previous Reports of This Issue (still after seven years, I mean really)
This issue has been reported to yourselves multiple times over the years:
- Issue #1948 - Mar 4, 2017
- Issue #10503 - May 23, 2019
- Issue #14518 - Sep 27, 2019
- Issue #15036 - Oct 15, 2019
- Issue #15098 - Oct 17, 2019
- Issue #17441 - Nov 27, 2019
- Issue #50252 - Aug 22, 2023
Violation of the Single Responsibility Principle
Now I always note "Rules are for the obedience of fools and the guidance of wise men." but in this case it seems a good call.
- The
MultipartReader
class is responsible for parsing multipart data but also imposes hard coded limits on headers and maybe body sizes as well without allowing developers to adjust them . - This mixing of responsibilities violates the Single Responsibility Principle.
- Reasoning: The class should focus solely on parsing multipart data. Enforcing limits should be a separate concern, configurable by the developer or handled by another component (e.g., the web server, which is the case generally).
- By hardcoding these limits and not providing a way to configure them, the class restricts legitimate use cases (e.g., large file uploads) and reduces flexibility.
Issue Description:
try
{
// Code removed for brevity
// Initialize the MultipartReader with the boundary, request body stream, and custom buffer size
MultipartReader multipartReader = new MultipartReader(
HeaderUtilities.RemoveQuotes(contentType.Boundary).Value!, // The multipart boundary
httpRequest.Body, // Stream containing multipart data
customBufferSize // The minimum buffer size to use
);
// When MultipartReader is instantiated, the following defaults are set:
// - HeadersCountLimit is set to DefaultHeadersCountLimit (16)
// - HeadersLengthLimit is set to DefaultHeadersLengthLimit (16,384 bytes or 16KB)
//
// These defaults are established as:
// public const int DefaultHeadersCountLimit = 16;
// public const int DefaultHeadersLengthLimit = 1024 * 16;
// public int HeadersLengthLimit { get; set; } = DefaultHeadersLengthLimit;
//
// Also, in the constructor, the private field _currentStream is initialized:
// private MultipartReaderStream _currentStream;
//
// The initialization is done with LengthLimit set to HeadersLengthLimit:
// _currentStream = new MultipartReaderStream(_stream, _boundary)
// {
// LengthLimit = HeadersLengthLimit
// };
//
// Note: There's a comment in the code indicating a known issue:
// // TODO: HeadersLengthLimit can't be modified until after the constructor.
//
// This means we cannot adjust HeadersLengthLimit before it's used in _currentStream.
// **** THIS IS WHERE IT FAILS IF THE FILE UPLOAD IS TOO LARGE ****
MultipartSection? multipartSection = await multipartReader.ReadNextSectionAsync(cancellationToken);
// Explanation of the failure:
//
// Inside ReadNextSectionAsync(), the first operation is to drain any preamble data:
// await _currentStream.DrainAsync(cancellationToken);
//
// The DrainAsync() method is defined in StreamHelperExtensions.cs:
//
// public static Task DrainAsync(this Stream stream, CancellationToken cancellationToken)
// {
// return stream.DrainAsync(ArrayPool<byte>.Shared, null, cancellationToken);
// }
//
// This calls the overloaded DrainAsync() method:
//
// public static async Task DrainAsync(this Stream stream, ArrayPool<byte> bytePool, long? limit, CancellationToken cancellationToken)
// {
// cancellationToken.ThrowIfCancellationRequested();
// var buffer = bytePool.Rent(_maxReadBufferSize);
// long total = 0;
// try
// {
// var read = await stream.ReadAsync(buffer.AsMemory(), cancellationToken); // ** NOTE THIS CALL **
//
// while (read > 0)
// {
// cancellationToken.ThrowIfCancellationRequested();
// if (limit.HasValue && limit.GetValueOrDefault() - total < read)
// {
// throw new InvalidDataException($"The stream exceeded the data limit {limit.GetValueOrDefault()}.");
// }
// total += read;
// read = await stream.ReadAsync(buffer.AsMemory(), cancellationToken);
// }
// }
// finally
// {
// bytePool.Return(buffer);
// }
// }
//
// During the first read operation, the stream's ReadAsync method is called, which eventually leads to MultipartReaderStream's ReadAsync.
// Inside MultipartReaderStream.cs:
// private int UpdatePosition(int read)
// {
// _position += read;
//
// if (_observedLength < _position)
// {
// _observedLength = _position;
//
// // The LengthLimit is set to HeadersLengthLimit (16KB) in the constructor and cannot be modified before this point.
// // Since we cannot adjust HeadersLengthLimit before it's enforced, any large file upload exceeding 16KB will cause an exception.
//
// if (LengthLimit.HasValue && _observedLength > LengthLimit.GetValueOrDefault())
// {
// throw new InvalidDataException($"Multipart body length limit {LengthLimit.GetValueOrDefault()} exceeded.");
// }
// }
//
// return read;
// }
//
// This is where the exception is thrown above when a large file is uploaded:
// if (LengthLimit.HasValue && _observedLength > LengthLimit.GetValueOrDefault())
//
// Code removed for brevity
}
catch (Exception ex)
{
// Handle the exception appropriately
// ...
}
Struggles and Final Thoughts
After struggling with this issue, I decided to write my own Multipart handler. But first, I wrote some code to directly take the uploaded stream and write it straight to a file without loading everything into memory. I tested it by attempting to upload a 5 GB file using Postman.
The following overly commented and overly verbose (not for production) code exists because if you cancel the upload in Postman or simply close Postman halfway through, Kestrel just keeps chugging along like nothing is wrong. It continues until it empties what I assume is its cache, and then just freezes. I’m sure there must be some timeout configuration to address this, but whatever it is, it’s incredibly long—almost like a DDoS/hacker's dream.
Once I got the code running somewhat correctly, it still would not allow the file to progress past 3,854,123 bytes. It just sits there until my own timer runs out.
[HttpPost]
[Route("api/testupload")]
[Every Possible Version of size limits and disable form stuff ect and in program.cs]
public async Task<IActionResult> TestLargeSingleFileUploadAsync()
{
// There is a lot of stuff for logging and just trying to test and bodge the thing to work.
// Don't use this in production without going throuugh it carefully first.
CancellationToken cancellationToken = HttpContext.RequestAborted;
bool uploadCompleted = false;
try
{
Log.Information("User is attempting to upload a file.");
// Define the path and filename where the incoming file will be saved.
string uploadingFilePath = Path.Combine(_appsettings.PublicUploadsDirectoryPath, "uploadedFile.txt");
// Use a reasonable buffer size for large file uploads.
const int bufferSize = 16 * 1024 * 1024; // 16 MB buffer size.
// Set a logging threshold of 200 MB.
const long loggingThreshold = 200 * 1024 * 1024; // 200 MB
// Keep track of the next logging threshold.
long nextLoggingThreshold = loggingThreshold;
// Use a FileStream to write the incoming data with a specified buffer size.
using (FileStream fileStream = new(
uploadingFilePath,
FileMode.Create,
FileAccess.Write,
FileShare.Read,
bufferSize: bufferSize, // Use the defined buffer size.
FileOptions.Asynchronous)) // Asynchronous I/O without WriteThrough for better performance
{
PipeReader bodyReader = HttpContext.Request.BodyReader;
long totalBytesRead = 0;
long totalBytesWritten = 0;
while (true)
{
// Read from the body reader using the cancellation token.
Task<ReadResult> readTask = bodyReader.ReadAsync(cancellationToken).AsTask();
// Check if the read task completes within the timeout (e.g., 5 seconds).
if (await Task.WhenAny(readTask, Task.Delay(10000, cancellationToken)) != readTask)
{
Log.Warning("Read operation timed out, possibly because the client has stopped sending data.");
throw new TimeoutException("Read operation timed out, possibly because the client has stopped sending data.");
}
ReadResult readResult = await readTask;
ReadOnlySequence<byte> buffer = readResult.Buffer;
long bytesRead = buffer.Length;
if (readResult.IsCompleted && buffer.IsEmpty)
{
Log.Information("Completed reading from the request body. Total bytes read: {TotalBytesRead} bytes.", totalBytesRead);
break;
}
totalBytesRead += bytesRead;
// Write the buffer segments to the file stream.
foreach (ReadOnlyMemory<byte> segment in buffer)
{
await fileStream.WriteAsync(segment, cancellationToken);
totalBytesWritten += segment.Length;
// Log only after exceeding the next 200 MB threshold.
if (totalBytesWritten >= nextLoggingThreshold)
{
Log.Information("Total bytes written to file so far: {TotalBytesWritten} bytes.", totalBytesWritten);
nextLoggingThreshold += loggingThreshold; // Update to the next 200 MB threshold.
}
}
// Mark the buffer as consumed.
bodyReader.AdvanceTo(buffer.End);
// Check for cancellation before continuing the next read.
cancellationToken.ThrowIfCancellationRequested();
}
// Perform a single flush at the end of the upload.
await fileStream.FlushAsync(cancellationToken);
Log.Information("Final flush completed. Total bytes written to disk: {TotalBytesWritten} bytes.", totalBytesWritten);
Log.Information("User has successfully uploaded the file. Total bytes written: {TotalBytesWritten} bytes.", totalBytesWritten);
uploadCompleted = true; // Mark the upload as successfully completed.
}
return NoContent();
}
catch (OperationCanceledException)
{
Log.Warning("File upload was cancelled by the client.");
return StatusCode(StatusCodes.Status499ClientClosedRequest, "File upload cancelled by client.");
}
catch (TimeoutException timeoutException)
{
Log.Warning(timeoutException, "File upload timed out.");
return StatusCode(StatusCodes.Status408RequestTimeout, "File upload timed out.");
}
catch (Exception exception)
{
Log.Error(exception, "An unexpected error occurred during file upload.");
return StatusCode(StatusCodes.Status500InternalServerError, exception.Message);
}
finally
{
// Delete the incomplete file only if the upload did not complete successfully.
// if (!uploadCompleted)
// {
// string incompleteFilePath = Path.Combine(_appsettings.PublicUploadsDirectoryPath, "uploadedFile.txt");
// if (System.IO.File.Exists(incompleteFilePath))
// {
// try
// {
// System.IO.File.Delete(incompleteFilePath);
// Log.Information("Incomplete file '{FilePath}' has been deleted.", incompleteFilePath);
// }
// catch (Exception fileException)
// {
// Log.Error(fileException, "Failed to delete incomplete file '{FilePath}'.", incompleteFilePath);
// }
// }
// }
}
}
Conclusion
After spending a week battling with this, I realised that ASP.NET and Kestrel fall short when it comes to large file uploads. The lack of clear documentation and flexibility makes it evident that they are not well-suited for handling this use case. This isn’t just a small oversight—it’s a fundamental flaw that I think requires a complete redesign, not just a minor fix.
What Did I Do in the End?
Although my Rust programming skills are a bit rough around the edges, I used Axum, and it just works. Hardly any code, really fast, and even though Rust is difficult to program, it didn’t take long to get it working.
I only add this for anyone in 2024 trying to do this with .NET: Don’t waste the week I’ve spent trying to get it to work. Consider alternatives like Axum if you need reliable large file upload support.
If somebody has managed to get this to work in .NET (without loading the whole file in memory for note) I would love to see a Full working example including all config around it, .NET and Kestrel.
References
- Uploading Files in ASP.NET Core
- ASP.NET Core Source Code - WebUtilities
- RFC 7578: Returning Values from Forms: multipart/form-data
- Kestrel Server Options in ASP.NET Core
- Axum GitHub Repository
Expected Behavior
To be able to Upload Large files...
Steps To Reproduce
No response
Exceptions (if any)
No response
.NET Version
.NET 8
Anything else?
No response