Skip to content

Dependency "OpenBLAS" not found when mvn package #495

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ziyilin opened this issue Apr 10, 2025 · 23 comments
Closed

Dependency "OpenBLAS" not found when mvn package #495

ziyilin opened this issue Apr 10, 2025 · 23 comments

Comments

@ziyilin
Copy link

ziyilin commented Apr 10, 2025

I prepared a maven project with graalpy. But failed to mvn package the project.
This is my pom settings:

 <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <graalpy.version>24.1.2</graalpy.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.graalvm.python</groupId>
                <artifactId>graalpy-maven-plugin</artifactId>
                <version>${graalpy.version}</version>
                <configuration>
                    <packages>
                        <package>optuna==3.2.0</package>
                        <package>pandas==1.5.2</package>
                        <package>scikit-learn==1.2.2</package>
                        <package>numpy==1.24.3</package>
                    </packages>
                    <externalDirectory>
                        ${project.basedir}/python-resources
                    </externalDirectory>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>process-graalpy-resources</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

mvn package process failed with the following error message:

[INFO] --- graalpy-maven-plugin:24.1.2:process-graalpy-resources (default) @ tuning-driver ---
[INFO] Creating GraalPy 24.1.2 venv
[INFO] /home/cengfeng.lzy/codebase/mljava/tuning-driver/target/classes/org.graalvm.python.vfs/venv/bin/graalpy -m pip install scikit-learn==1.2.2 pandas==1.5.2 numpy==1.24.3 optuna==3.2.0
Looking in indexes: http://yum.tbsite.net/pypi/simple/, https://www.graalvm.org/python/wheels/
Collecting scikit-learn==1.2.2
  Using cached http://yum.tbsite.net/pypi/packages/c9/fa/8e158d81e3602da1e7bafbd4987938bc003fe4b0f44d65681e7f8face95a/scikit-learn-1.2.2.tar.gz (7.3 MB)
Looking for GraalPy patches for scikit-learn
Patching package scikit-learn using /home/cengfeng.lzy/.cache/org.graalvm.polyglot/python/python-home/7f99e649dc51b21db2b83ce56bc9742e402b87c1/lib/graalpy24.1/patches/scikit-learn/scikit-learn-1.2.2.patch
patching file pyproject.toml
patching file sklearn/_build_utils/__init__.py
patching file sklearn/datasets/_svmlight_format_fast.pyx
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [92 lines of output]
      Looking in indexes: http://yum.tbsite.net/pypi/simple/, https://www.graalvm.org/python/wheels/, https://www.graalvm.org/python/wheels/
      Ignoring numpy: markers 'python_version == "3.10" and platform_system == "Windows" and platform_python_implementation != "PyPy"' don't match your environment
      Collecting setuptools
        Using cached http://yum.tbsite.net/pypi/packages/54/21/f43f0a1fa8b06b32812e0975981f4677d28e0f3271601dc88ac5a5b83220/setuptools-78.1.0-py3-none-any.whl (1.3 MB)
      Collecting wheel
        Using cached http://yum.tbsite.net/pypi/packages/7d/cd/d7460c9a869b16c3dd4e1e403cce337df165368c71d6af229a74699622ce/wheel-0.43.0-py3-none-any.whl (65 kB)
      Collecting Cython<3,>=0.29.24
        Using cached http://yum.tbsite.net/pypi/packages/7e/26/9d8de10005fedb1eceabe713348d43bae1dbab1786042ca0751a2e2b0f8c/Cython-0.29.37-py2.py3-none-any.whl (989 kB)
      Collecting oldest-supported-numpy
        Using cached http://yum.tbsite.net/pypi/packages/dc/5c/e3c84cfdd488701aa074b22cf5bd227fb15d26e1d55a66d9088c39afa123/oldest_supported_numpy-2023.12.21-py3-none-any.whl (4.9 kB)
      Collecting scipy>=1.3.2
        Using cached http://yum.tbsite.net/pypi/packages/84/a9/2bf119f3f9cff1f376f924e39cfae18dec92a1514784046d185731301281/scipy-1.10.1.tar.gz (42.4 MB)
      auto-patching C API usages in /tmp/pip-install-lg_sqgji/scipy_61e5a5affd64483386ee2d3ee0995d24/scipy/_lib/boost/boost/python/suite/indexing/detail/indexing_suite_detail.hpp
      auto-patching C API usages in /tmp/pip-install-lg_sqgji/scipy_61e5a5affd64483386ee2d3ee0995d24/scipy/_lib/boost/boost/python/detail/wrap_python.hpp
      auto-patching C API usages in /tmp/pip-install-lg_sqgji/scipy_61e5a5affd64483386ee2d3ee0995d24/scipy/_lib/boost/boost/mpi/python/skeleton_and_content.hpp
      auto-patching C API usages in /tmp/pip-install-lg_sqgji/scipy_61e5a5affd64483386ee2d3ee0995d24/scipy/_lib/boost/boost/mpi/python/serialize.hpp
      Looking for GraalPy patches for scipy
      Patching package scipy using /home/cengfeng.lzy/.cache/org.graalvm.polyglot/python/python-home/7f99e649dc51b21db2b83ce56bc9742e402b87c1/lib/graalpy24.1/patches/scipy/scipy-1.10.1.patch
      patching file meson.build
      patching file mesonpy_wrapper.py
      patching file pyproject.toml
      patching file scipy/_lib/_ccallback_c.pyx
      patching file scipy/_lib/setup.py
      patching file scipy/_lib/src/messagestream_config.h.in
      patching file scipy/sparse/linalg/meson.build
      patching file scipy/special/meson.build
      patching file tools/cythonize.py
        Installing build dependencies: started
        Installing build dependencies: still running...
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): still running...
        Preparing metadata (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error

        × Preparing metadata (pyproject.toml) did not run successfully.
        │ exit code: 1
        ╰─> [42 lines of output]
            + meson setup --prefix=/home/cengfeng.lzy/.cache/org.graalvm.polyglot/python/python-home/7f99e649dc51b21db2b83ce56bc9742e402b87c1 /tmp/pip-install-lg_sqgji/scipy_61e5a5affd64483386ee2d3ee0995d24 /tmp/pip-install-lg_sqgji/scipy_61e5a5affd64483386ee2d3ee0995d24/mesonbuild-graalpy --native-file=/tmp/pip-install-lg_sqgji/scipy_61e5a5affd64483386ee2d3ee0995d24/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2
            The Meson build system
            Version: 1.2.2
            Source dir: /tmp/pip-install-lg_sqgji/scipy_61e5a5affd64483386ee2d3ee0995d24
            Build dir: /tmp/pip-install-lg_sqgji/scipy_61e5a5affd64483386ee2d3ee0995d24/mesonbuild-graalpy
            Build type: native build
            Project name: SciPy
            Project version: 1.10.1
            C compiler for the host machine: cc (gcc 10.2.1 "cc (GCC) 10.2.1 20200825 (Alibaba 10.2.1-3.8 2.32)")
            C linker for the host machine: cc ld.bfd 2.35-12
            C++ compiler for the host machine: c++ (gcc 10.2.1 "c++ (GCC) 10.2.1 20200825 (Alibaba 10.2.1-3.8 2.32)")
            C++ linker for the host machine: c++ ld.bfd 2.35-12
            Cython compiler for the host machine: cython (cython 0.29.37)
            Host machine cpu family: x86_64
            Host machine cpu: x86_64
            Compiler for C supports arguments -Wno-unused-but-set-variable: YES
            Compiler for C supports arguments -Wno-unused-function: YES
            Compiler for C supports arguments -Wno-conversion: YES
            Compiler for C supports arguments -Wno-misleading-indentation: YES
            Compiler for C supports arguments -Wno-incompatible-pointer-types: YES
            Library m found: YES
            Fortran compiler for the host machine: gfortran (gcc 10.2.1 "GNU Fortran (GCC) 10.2.1 20200825 (Alibaba 10.2.1-3.8 2.32)")
            Fortran linker for the host machine: gfortran ld.bfd 2.35-12
            Compiler for Fortran supports arguments -Wno-conversion: YES
            Compiler for Fortran supports arguments -fallow-argument-mismatch: YES
            Checking if "-Wl,--version-script" : links: YES
            Program cython found: YES (/tmp/pip-build-env-3v8a54ez/overlay/bin/cython)
            Program python found: YES (/home/cengfeng.lzy/codebase/mljava/tuning-driver/target/classes/org.graalvm.python.vfs/venv/bin/graalpy)
            Run-time dependency python found: YES 3.11
            Program pythran found: YES (/tmp/pip-build-env-3v8a54ez/overlay/bin/pythran)
            Run-time dependency threads found: YES
            Library npymath found: YES
            Library npyrandom found: YES
            Found pkg-config: /usr/bin/pkg-config (1.4.2)
            Found CMake: /usr/bin/cmake (3.26.5)
            WARNING: CMake Toolchain: Failed to determine CMake compilers state
            Run-time dependency openblas found: NO (tried pkgconfig and cmake)
            Run-time dependency openblas found: NO (tried pkgconfig and cmake)

            ../scipy/meson.build:134:7: ERROR: Dependency "OpenBLAS" not found, tried pkgconfig and cmake

            A full log can be found at /tmp/pip-install-lg_sqgji/scipy_61e5a5affd64483386ee2d3ee0995d24/mesonbuild-graalpy/meson-logs/meson-log.txt
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: metadata-generation-failed

      × Encountered error while generating package metadata.
      ╰─> See above for output.

      note: This is an issue with the package mentioned above, not pip.
      hint: See above for details.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

But I have installed openblas:

$sudo yum install openblas
Last metadata expiration check: 1:19:54 ago on Thu 10 Apr 2025 01:51:08 PM CST.
Package openblas-0.3.15-6.al8.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!

$ldconfig -p | grep openblas
        libopenblas.so.0 (libc6,x86-64) => /lib64/libopenblas.so.0

How can I get this fixed? Thanks.

@msimacek
Copy link
Contributor

Hi @ziyilin. It's a problem with the Fedora/RHEL/Centos openblas package where the maintainer refuses to ship the upstream pkg-config file, so scipy can't find it. You can find more on the scipy issue about this. I use the workaround described in this comment on the same issue. Note you also need to have openblas-devel installed.

@ziyilin
Copy link
Author

ziyilin commented Apr 11, 2025

@msimacek It works, thank you very much. But after 1 hour's building, another error is reported:

Compiling '/tmp/pip-install-vt2ycuy6/scipy_ec0df56da8e54379b4592e58f85d23f2/.mesonpy-69_gzq4e/install/home/cengfeng.lzy/.cache/org.graalvm.polyglot/python/python-home/7f99e649dc51b21db2b83ce56bc9742e402b87c1/lib/python3.11/site-packages/scipy/io/_harwell_boeing/tests/test_hb.py'...
            Traceback (most recent call last):
              File "/home/cengfeng.lzy/codebase/mljava/tuning-driver/target/classes/org.graalvm.python.vfs/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
                main()
              File "/home/cengfeng.lzy/codebase/mljava/tuning-driver/target/classes/org.graalvm.python.vfs/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
                json_out['return_val'] = hook(**hook_input['kwargs'])
              File "/home/cengfeng.lzy/codebase/mljava/tuning-driver/target/classes/org.graalvm.python.vfs/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
                whl_basename = backend.build_wheel(metadata_directory, config_settings)
              File "/tmp/pip-install-vt2ycuy6/scipy_ec0df56da8e54379b4592e58f85d23f2/mesonpy_wrapper.py", line 79, in build_wheel
                return mesonpy.build_wheel(wheel_directory, config_settings=config_settings, metadata_directory=metadata_directory)
              File "/tmp/pip-build-env-b1w8b9v3/overlay/lib/python3.11/site-packages/mesonpy/__init__.py", line 1054, in wrapper
                return func(*args, **kwargs)
              File "/tmp/pip-build-env-b1w8b9v3/overlay/lib/python3.11/site-packages/mesonpy/__init__.py", line 1116, in build_wheel
                return project.wheel(out).name
              File "/tmp/pip-build-env-b1w8b9v3/overlay/lib/python3.11/site-packages/mesonpy/__init__.py", line 955, in wheel
                wheel = self._wheel_builder.build(self._build_dir)
              File "/tmp/pip-build-env-b1w8b9v3/overlay/lib/python3.11/site-packages/mesonpy/__init__.py", line 513, in build
                whl.writestr(f'{self.distinfo_dir}/METADATA', self._project.metadata)
              File "/home/cengfeng.lzy/.cache/org.graalvm.polyglot/python/python-home/7f99e649dc51b21db2b83ce56bc9742e402b87c1/lib/python3.11/functools.py", line 1001, in __get__
                val = self.func(instance)
              File "/tmp/pip-build-env-b1w8b9v3/overlay/lib/python3.11/site-packages/mesonpy/__init__.py", line 868, in metadata
                core_metadata.headers['Version'] = [self.version]
            AttributeError: 'RFC822Message' object has no attribute 'headers'
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: metadata-generation-failed

      × Encountered error while generating package metadata.
      ╰─> See above for output.

      note: This is an issue with the package mentioned above, not pip.
      hint: See above for details.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

By the way, is there any option can keep the temporary files? They are deleted when the error is reported.

@msimacek
Copy link
Contributor

msimacek commented Apr 11, 2025

AttributeError: 'RFC822Message' object has no attribute 'headers'

That sounds like a version mismatch between meson and its dependencies (we used to patch meson and that forces an older version). I think it should be fixed on GraalPy 24.2.0, can you update?

By the way, is there any option can keep the temporary files? They are deleted when the error is reported.

For debugging build issues it's easier to download the native standalone version of graalpy, create a virtualenv and try to pip install the same packages in it. Then you can pass --no-clean to pip too keep the temporary files. When the builds succeed, the wheels get cached, so when you run the maven build afterwards, it shouldn't rebuild the packages again but use the cached ones (if the graalpy versions matched).

@ziyilin
Copy link
Author

ziyilin commented Apr 11, 2025

I have switched to Graalpy 24.2.0, but with no luck. I still got the exactly the same error.

@msimacek
Copy link
Contributor

That's strange, I tried building scipy with 24.2 just now and it worked fine. Can you try running pip install -v 'scipy>=1.3.2' on 24.2 native standalone and post the whole log?

@ziyilin
Copy link
Author

ziyilin commented Apr 14, 2025

24.2 native standalone is good to run pip install -v 'scipy>=1.3.2'.
This is the log: graalpy_log.txt

@msimacek
Copy link
Contributor

So, then if you run the maven build with 24.2 it should pick the wheels you just built from the cache. Does that work? If not, please post the build log.

@ziyilin
Copy link
Author

ziyilin commented Apr 14, 2025

Running with mvn pacakge, but get the same error again.
This is the log: graalpy_error.txt

@msimacek
Copy link
Contributor

Hmm, it's picking up scipy 1.10.1, but there should be no for graalpy reason to pick that old version. It seems like it's still somehow using 24.1, as if the virtualenv didn't get rebuilt when you changed the version (it could be a bug in the maven plugin). When you run /home/cengfeng.lzy/codebase/mljava/tuning-driver/python-resources/venv/bin/graalpy --version what does it say? If it still says graalpy241-311, can you please try mvn clean package?

@ziyilin
Copy link
Author

ziyilin commented Apr 14, 2025

It only provides python version.

$/home/cengfeng.lzy/codebase/mljava/tuning-driver/python-resources/venv/bin/graalpy --version
GraalPy 3.11.7 (Interpreted JVM Development Build)

@msimacek
Copy link
Contributor

Ah, sorry, I pasted the wrong thing, I meant /home/cengfeng.lzy/codebase/mljava/tuning-driver/python-resources/venv/bin/graalpy -c 'import sys;print(sys.implementation.cache_tag)'

@ziyilin
Copy link
Author

ziyilin commented Apr 15, 2025

Thanks for clarifying. It was graalpy241-311. I have mvn cleaan package and now is graalpy242-311.
But it still fails with the error: log.txt
Is it because my gcc is not compatible?
My cc info is:

$cc -v
Using built-in specs.
COLLECT_GCC=cc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/10/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --disable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.2.1 20200825 (Alibaba 10.2.1-3.8 2.32) (GCC)

@msimacek
Copy link
Contributor

It's not because of the compiler. It's trying to install a numpy version that doesn't have a graalpy patch (the latest patch is for numpy 2.0.2, but it tries to install 2.2.4). Pip should normally prefer a version that does have a patch (and it does when I try it locally). I see you're using a different repository than pypi.org, is it possible that it doesn't contain numpy 2.0.2?

@ziyilin
Copy link
Author

ziyilin commented Apr 16, 2025

I manually switched to numpy 2.0.2, but still got the same error.
I executed python-resources/venv/bin/pip3 install numpy==2.0.2. The error log is :

pip_install.log

@msimacek
Copy link
Contributor

That's really weird, after Looking for GraalPy patches for numpy, there should normally be Patching package numpy using numpy-2.0.0.patch. Do you change any environment variables or configuration for pip, besides the index URL? Do you have the patch command installed? Can you try it on the native standalone? You don't have to run the whole build, just the first minute or so, you can kill it after it prints Installing build dependencies and just tell me if there was anything different right after the Looking for GraalPy patches for numpy message.

@ziyilin
Copy link
Author

ziyilin commented Apr 17, 2025

Image
It seems the graalpy used by mvn package didn't patch numpy as standalone native graalpy.

Is it possible to use native standalone graalpy in mvn package ?

@msimacek
Copy link
Contributor

Yes, here's an example project that manages the virtualenv manually. Another option is to rely on the caching - build the packages with standalone and the maven build should pick up the cached wheels without rebuilding. Yet another option is to build the wheels with pip wheel and directly refer to the wheel files as dependencies (i.e. putting the wheel paths as <package>).
But before you do that, could you please post your graalpy-maven-plugin configuration? Because the patching should normally work under the plugin, so I'd like to reproduce the bug so that we can fix it.

@ziyilin
Copy link
Author

ziyilin commented Apr 17, 2025

Thanks, this is my pom configuration.

<properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <graalpy.version>24.2.0</graalpy.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.graalvm.python</groupId>
                <artifactId>graalpy-maven-plugin</artifactId>
                <version>${graalpy.version}</version>
                <configuration>
                    <packages>
                        <package>optuna==3.2.0</package>
                        <package>pandas==1.5.2</package>
                        <package>scikit-learn==1.2.2</package>
                        <package>numpy==1.24.3</package>
                    </packages>
                    <externalDirectory>
                        ${project.basedir}/python-resources
                    </externalDirectory>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>process-graalpy-resources</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.graalvm.polyglot</groupId>
            <artifactId>polyglot</artifactId>
            <version>${graalpy.version}</version>
        </dependency>
        <dependency>
            <groupId>org.graalvm.python</groupId>
            <artifactId>python-embedding</artifactId>
            <version>${graalpy.version}</version>
        </dependency>
        <dependency>
            <groupId>org.graalvm.polyglot</groupId>
            <artifactId>python</artifactId>
            <version>${graalpy.version}</version>
            <type>pom</type>
        </dependency>
    </dependencies>

@steve-s
Copy link
Member

steve-s commented Apr 30, 2025

@ziyilin I also cannot reproduce the issue locally.

Could you please run the script below with ./python-resources/venv/bin/graalpy and share the output?

from pip._internal.utils import graalpy
repo = graalpy.get_patch_repository()
print(f"{repo=}")
if hasattr(repo, 'patches_url'):
    print(f"{repo.patches_url=}")
rule = repo.get_matching_rule('numpy', '2.0.2', 'sdist')
print(f"{rule=}")
patch = rule.get('patch')
print(f"{patch=}")
with repo.resolve_patch(patch) as patch_path:
    print(f"{patch_path=}")
    with open(patch_path, 'r+') as f:
        print(f.read()[:100])

on my system, for example, it prints:

repo=<pip._internal.utils.graalpy.RemotePatchRepository object at 0x7b313fdf>
repo.patches_url='https://raw.githubusercontent.com/oracle/graalpython/refs/heads/github/patches/24.2.0/graalpython/lib-graalpython/patches/'
rule={'version': '>= 2.0.0rc1, < 2.1', 'patch': 'numpy-2.0.0.patch', 'license': 'BSD-3-Clause', 'dist-type': 'sdist'}
patch='numpy-2.0.0.patch'
patch_path=PosixPath('/tmp/tmp8szc7_bg/numpy-2.0.0.patch')
diff --git a/numpy/_core/include/numpy/ndarrayobject.h b/numpy/_core/include/numpy/ndarrayobject.h

Could you also try running the numpy 2.0.2 installation with env. variable PIP_GRAALPY_PATCHES_URL set to empty string, e.g.:

PIP_GRAALPY_PATCHES_URL= python-resources/venv/bin/pip3 install numpy==2.0.2

@ziyilin
Copy link
Author

ziyilin commented May 6, 2025

I have manually changed the graalpy at ./python-resources/venv/bin to native standalone version since my last reply and got everything good.
I have deleted the entire python-resources directory and then run mvn package to get the JVM version graalpy installed again to try @steve-s' script. But now everything is good.

$./python-resources/venv/bin/graalpy ./test.py
repo=<pip._internal.utils.graalpy.RemotePatchRepository object at 0x28bbac99>
repo.patches_url='https://raw.githubusercontent.com/oracle/graalpython/refs/heads/github/patches/24.2.0/graalpython/lib-graalpython/patches/'
rule={'version': '>= 2.0.0rc1, < 2.1', 'patch': 'numpy-2.0.0.patch', 'license': 'BSD-3-Clause', 'dist-type': 'sdist'}
patch='numpy-2.0.0.patch'
patch_path=PosixPath('/tmp/tmpm_5mlcf5/numpy-2.0.0.patch')
diff --git a/numpy/_core/include/numpy/ndarrayobject.h b/numpy/_core/include/numpy/ndarrayobject.h
i

$PIP_GRAALPY_PATCHES_URL= ./python-resources/venv/bin/pip3 install numpy==2.0.2
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple, https://www.graalvm.org/python/wheels/
Collecting numpy==2.0.2
  Using cached numpy-2.0.2-graalpy311-graalpy242_311_native-linux_x86_64.whl
Installing collected packages: numpy
Successfully installed numpy-2.0.2

Is it because it now uses the cached file? How can I remove the cached numpy-2.0.2-graalpy311-graalpy242_311_native-linux_x86_64.whl and try again?

@msimacek
Copy link
Contributor

msimacek commented May 6, 2025

Is it because it now uses the cached file? How can I remove the cached numpy-2.0.2-graalpy311-graalpy242_311_native-linux_x86_64.whl and try again?

It could be. You can remove the cached file with pip cache remove numpy (has to be graalpy's pip) or deleting it manually in ~/.cache/pip-graalpy/wheels/.

@ziyilin
Copy link
Author

ziyilin commented May 6, 2025

I have uninstalled the numpy and clear the cache. But it seems working fine now.

[[email protected] /home/cengfeng.lzy/codebase/mljava/tuning-driver]
$./python-resources/venv/bin/pip3 uninstall numpy==2.0.2
Found existing installation: numpy 2.0.2
Uninstalling numpy-2.0.2:
  Would remove:
    /home/cengfeng.lzy/codebase/mljava/tuning-driver/python-resources/venv/bin/f2py
    /home/cengfeng.lzy/codebase/mljava/tuning-driver/python-resources/venv/bin/numpy-config
    /home/cengfeng.lzy/codebase/mljava/tuning-driver/python-resources/venv/lib/python3.11/site-packages/numpy-2.0.2.dist-info/*
    /home/cengfeng.lzy/codebase/mljava/tuning-driver/python-resources/venv/lib/python3.11/site-packages/numpy/*
Proceed (Y/n)? y
  Successfully uninstalled numpy-2.0.2

[[email protected] /home/cengfeng.lzy/codebase/mljava/tuning-driver]
$ls ~/.cache/pip-graalpy/wheels/
1f  26  29  3c  54  5a  75  8d  a6  b7  c9  d5  f7

[[email protected] /home/cengfeng.lzy/codebase/mljava/tuning-driver]
$mv ~/.cache/pip-graalpy/wheels/ ~/.cache/pip-graalpy/wheels.bak

[[email protected] /home/cengfeng.lzy/codebase/mljava/tuning-driver]
$./python-resources/venv/bin/graalpy ./test.py
repo=<pip._internal.utils.graalpy.RemotePatchRepository object at 0x2af97fb9>
repo.patches_url='https://raw.githubusercontent.com/oracle/graalpython/refs/heads/github/patches/24.2.0/graalpython/lib-graalpython/patches/'
rule={'version': '>= 2.0.0rc1, < 2.1', 'patch': 'numpy-2.0.0.patch', 'license': 'BSD-3-Clause', 'dist-type': 'sdist'}
patch='numpy-2.0.0.patch'
patch_path=PosixPath('/tmp/tmpctrap626/numpy-2.0.0.patch')
diff --git a/numpy/_core/include/numpy/ndarrayobject.h b/numpy/_core/include/numpy/ndarrayobject.h
i

[[email protected] /home/cengfeng.lzy/codebase/mljava/tuning-driver]
$PIP_GRAALPY_PATCHES_URL= ./python-resources/venv/bin/pip3 install numpy==2.0.2
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple, https://www.graalvm.org/python/wheels/
Collecting numpy==2.0.2
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a9/75/10dd1f8116a8b796cb2c737b674e02d02e80454bda953fa7e65d8c12b016/numpy-2.0.2.tar.gz (18.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.9/18.9 MB 4.1 MB/s eta 0:00:00
auto-patching C API usages in /tmp/pip-install-nd9vl89q/numpy_ca0260732ae143749d7a176c47efaf91/numpy/_core/src/umath/override.c
auto-patching C API usages in /tmp/pip-install-nd9vl89q/numpy_ca0260732ae143749d7a176c47efaf91/numpy/_core/src/umath/_rational_tests.c
auto-patching C API usages in /tmp/pip-install-nd9vl89q/numpy_ca0260732ae143749d7a176c47efaf91/numpy/_core/src/multiarray/methods.c
auto-patching C API usages in /tmp/pip-install-nd9vl89q/numpy_ca0260732ae143749d7a176c47efaf91/numpy/_core/src/multiarray/dtype_transfer.c
auto-patching C API usages in /tmp/pip-install-nd9vl89q/numpy_ca0260732ae143749d7a176c47efaf91/numpy/_core/src/multiarray/dlpack.c
auto-patching C API usages in /tmp/pip-install-nd9vl89q/numpy_ca0260732ae143749d7a176c47efaf91/numpy/_core/src/multiarray/compiled_base.c
auto-patching C API usages in /tmp/pip-install-nd9vl89q/numpy_ca0260732ae143749d7a176c47efaf91/numpy/_core/src/multiarray/stringdtype/dtype.c
auto-patching C API usages in /tmp/pip-install-nd9vl89q/numpy_ca0260732ae143749d7a176c47efaf91/numpy/_core/include/numpy/npy_3kcompat.h
auto-patching C API usages in /tmp/pip-install-nd9vl89q/numpy_ca0260732ae143749d7a176c47efaf91/numpy/_core/include/numpy/ndarrayobject.h
  Looking for GraalPy patches for numpy
  Patching package numpy using numpy-2.0.0.patch
patching file numpy/_core/include/numpy/ndarrayobject.h
patching file numpy/_core/src/multiarray/compiled_base.c
patching file numpy/_core/src/multiarray/shape.c
patching file numpy/_core/src/multiarray/stringdtype/dtype.c
Hunk #1 succeeded at 832 (offset 15 lines).
patching file numpy/_core/src/multiarray/temp_elide.c
patching file numpy/_core/src/npymath/ieee754.c.src
patching file numpy/_core/src/npymath/ieee754.cpp
patching file vendored-meson/meson/mesonbuild/utils/universal.py
Hunk #1 succeeded at 728 (offset 1 line).
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... -

@msimacek
Copy link
Contributor

msimacek commented May 7, 2025

The upgrade path issue (changing graalpy version in the pom wouldn't rebuild the virtualenv) should be fixed by 69de2bf. I belive that resolves the issue.

@msimacek msimacek closed this as completed May 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants