Describe the issue
On MacOS there is no way to allow for more than 10240 open files, no matter what the value of ulimit -Sn or ulimit -Hn or launchctl limit maxfiles. The number of open files is always limits to 10240.

This is a result of the standard practice across many (all?) jvms including GraalVM of setting the current max files limit to the hard max if possible. But on MacOS due to a compatibility note in the man page for setrlimit the max file limit is always set to OPEN_MAX from syslimits.h which has the value 10240.

In jvm's derived from OpenJDK, you can work around this by skipping the calls to setrlimit for max files with -XX:-MaxFDLimit. And in more recent version of OpenJDK they have started ignoring the COMPATIBILITY note from the man page because that note seems to not have been true since ~10.6.

So in GraalVM can something be done to allow for more than 10240 open files?

I'm not strongly opinionated as to how:

• an option to skip setting the limit
• some method to enable it being set to higher than OPEN_MAX

I've included lots of pointers to code/docs/tickets in the More Details section.

This is something I found as part of digging way too deep into why I couldn't use mvnd to build a very large project in this ticket apache/maven-mvnd#710

Steps to reproduce the issue
Please include both build steps as well as run steps

1. Set your local ulimit to higher than 10240
2. Create a java program that opens 10241 files.
3. Run the java program see that it fails with:

Too many open files

(Sorry for the minimal repro steps, hopefully the code links/discussion makeup for it.)

Describe GraalVM and your environment:

• GraalVM version (latest snapshot builds can be found here), or commit id if built from source: [e.g. EE 19.3, CE 20.0, CE 20.1.0-dev-20200304_0848]
• JDK major version: 17 & 21
• OS: macOS Sequoia 15.1.1 (24B91)**
• Architecture: Apple M1 aarch64

More details
setrlimit Man page link:
https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/setrlimit.2.html

A somewhat recent 3rd party hosted copy of syslimits.h:
https://github.com/phracker/MacOSX-SDKs/blob/11.3/MacOSX11.3.sdk/usr/include/sys/syslimits.h#L96

The code that does this in GraalVM is here (latest tag as of posting to give a static reference location):
https://github.com/oracle/graal/blob/jdk-25+21/substratevm/src/com.oracle.svm.core.posix/src/com/oracle/svm/core/posix/PosixNativeLibraryFeature.java#L86

The equivalent code from OpenJDK is here (latest jdk 21) which behaves exactly the same as GraalVM does today:
https://github.com/openjdk/jdk/blob/jdk-21%2B35/src/hotspot/os/bsd/os_bsd.cpp#L2005

The current equivalent code from OpenJDK is here (Latest tag as of posting - JDK 25) this no longer tries to enforce OPEN_MAX though it does include it as fallback logic:
https://github.com/openjdk/jdk/blob/jdk-25%2B21/src/hotspot/os/bsd/os_bsd.cpp#L2117

These changes were the result of this ticket (with many related tickets):
https://bugs.openjdk.org/browse/JDK-8324577

The summary is that through testing it has been confirmed that the compatibility note is no longer correct. However, they don't actually set it to RLIM_INFINITY because that ran into issues with certain other processes in the system that would break if RLIMIT_NOFILE was greater than int32 even though the underlying type is actually uint64. Then they limited it down further to match the typical limit in linux of 0x100000 because they discovered several cases where people were iterating over all possible fds trying to close them and with the limit set to int32 max some of these were causing timeouts.