[GR-62603] Remove Sulong support.

PullRequest: graalpython/3706

Author: timfel

.gitignore

@@ -96,3 +96,4 @@ pom-mx.xml
 .venv
 !graalpython/com.oracle.graal.python.test/src/tests/standalone/gradle/gradle-test-project/gradle/wrapper/gradle-wrapper.jar
 /*-venv/
+.aider*

CHANGELOG.md

@@ -8,6 +8,8 @@ language runtime. The main focus is on user-observable behavior of the engine.
 * `GRAALPY_VERSION_NUM` C macro now inlcudes the release level and serial number at the end to conform to the `hexversion` format. This shouldn't break any existing comparisons.
 * `dir(foreign_object)` now returns both foreign methods and Python methods (it used to return only foreign methods).
 * Support `__name__`, `__doc__`, `__text_signature__` fields on foreign executables to serve as their proper counterparts on the Python side. This is useful to, for example, use Java functional interfaces in lieu of Python functions for things like LangChain's `@tool` annotation that want to inspect the underlying function.
+* Remove support for running C extensions as LLVM bitcode. This also removes the related options `python.UseSystemToolchain` and `python.NativeModules`.
+* Remove built-in HPy module. HPy can now be installed and used from the upstream sources.
 
 ## Version 24.2.0
 * Updated developer metadata of Maven artifacts.

docs/contributor/DEV_TASKS.md

@@ -3,7 +3,7 @@
 ### Updating dependencies
 
 We can use the following command to update our CI jsonnet as well as all
-dependencies (truffle, sulong, ...) in one go:
+dependencies (truffle, ...) in one go:
 
     mx python-update-import
 
@@ -26,35 +26,6 @@ It prints a fairly long help. Note that you'll need to make sure you also pushed
 your `python-import` branch after doing the update, so that any conflicts you
 resolved don't have to be resolved again by the next person.
 
-### Updating hpy
-
-Follow these steps to update HPy.
-
-
-  1. Merge updated hpy sources. To do so, clone hpy somewhere next to
-     graalpython. Then run the following command on a new branch of graalpython:
-
-        mx python-update-hpy-import --pull /path/to/clone/of/hpy
-
-     Follow the instructions.
-  2. We need to fix compilation. We patch the hpy sources, and the merge may
-     have introduced new API or types, and for these we need to apply
-     patches. At the time of this writing, we redefine hpy types conditionally
-     on `#ifdef GRAALVM_PYTHON_LLVM` (grep for this to find some
-     examples). Also, we use macros to convert between the structured hpy types
-     and plain pointers for our native interface, see the uses of the `WRAP` and
-     `UNWRAP` macros.
-  3. Once compilation is working, we try to run the tests and go on fixing
-     them. If new API was added, `GraalHPyContext` needs to be adapted with the
-     new signatures and/or types. This may include:
-
-      - Updating the HPyContextMember enum
-      - Updating the HPyContextSignatureType enum
-      - Adding `GraalHPyContextFunction` implementations for the new APIs
-      - Updating the `createMembers` method to assign the appropriate
-        implementations to the context members
-      - Updating hpy.c to assign new context members to their native locations
-
 ### Updating patch branch
 GraalPy's `pip` has an ability to download newer versions of patches from our
 GitHub so that we can update patches outside of the release cycle. There should
@@ -64,3 +35,13 @@ release tag commit and then it should change the CI overlay version to point to
 the head of branch `graalpy-patch-branch` in order to disable unnecessary gates
 on it. The GitHub sync needs to be manually enabled in the mirroring service
 configuration.
+
+### Updating hpy
+
+1. Switch to the `hpy-import` branch
+2. Delete `graalpython/hpy`
+3. Copy the sources from the hpy repo into `graalpython/hpy` (e.g git clone
+   them there, then delete the `graalpython/hpy/.git` folder)
+4. Go back to your previous branch and merge hpy-import.
+5. Go to `graalpython/hpy/build.py` and update the `VERSION` constant to
+   whatever you updated to.

docs/user/Native-Images-with-Python.md

@@ -54,7 +54,6 @@ These are:
   This uses an `ExecutorService` with a thread pool.
   If you want to disallow such extra threads or avoid pulling in `ExecutorService` and related classes, then set this property to `false` and retrieve the `PollPythonAsyncActions` object from the context's polyglot bindings.
   This object is executable and can be used to trigger Python asynchronous actions at the locations you desire.
-* `python.WithoutJNI=true` - This option removes any code that uses JNI. As a consequence, you cannot use the HPy JNI backend and maybe other parts that rely on JNI.
 
 
 ### Removing Pre-initialized Python Heap

graalpython/com.oracle.graal.python.cext/CEXT-WINDOWS-README.md

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2023, 2024, Oracle and/or its affiliates.
+# Copyright (c) 2023, 2025, Oracle and/or its affiliates.
 #
 # All rights reserved.
 #
@@ -37,17 +37,9 @@ endfunction()
 
 set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
 
-if(MSVC)
-    message(FATAL_ERROR "C API cannot be built with MSVC")
-endif()
-
-if(WIN32)
-    require_var(GRAALVM_LLVM_LIB_DIR)
-endif()
 require_var(GRAALPY_PARENT_DIR)
 require_var(CAPI_INC_DIR)
 require_var(PYCONFIG_INCLUDE_DIR)
-require_var(TRUFFLE_H_INC)
 require_var(TRUFFLE_NFI_H_INC)
 require_var(GRAALPY_EXT)
 
@@ -61,31 +53,67 @@ set(TARGET_LIBPYTHON "python-native")
 # common variables and compile/link options (for all build targets)
 ######################################################################
 
-set(CFLAGS_WARNINGS -Wall -Werror,-Wunknown-warning-option -Wno-unused-function -Wno-unused-variable -Wno-unused-const-variable
+if (MSVC)
+  SET(CMAKE_BUILD_TYPE "RelWithDebInfo" CACHE STRING "" FORCE)
+  SET(CFLAGS_WARNINGS /Wall /WX
+    # Many warnings that are just MSVC being a bit pedantic
+    /wd4255 # no function prototype given: converting '()' to '(void)'
+    /wd4820 # padding added after data member
+    /wd4100 # unreferenced formal parameter
+    /wd4200 # nonstandard extension used: zero-sized array in struct/union
+    /wd4996 # This function or variable may be unsafe ... see _CRT_SECURE_NO_WARNINGS
+    /wd4668 # ... is not defined as a preprocessor macro, replacing with '0' for '#if/#elif'
+    /wd4115 # named type definition in parentheses
+    /wd4152 # nonstandard extension, function/data pointer conversion in expression
+    /wd5045 # Compiler will insert Spectre mitigation for memory load if /Qspectre switch specified
+    /wd4047 # 'int64_t' differs in levels of indirection from 'char []' / 'void *'
+    /wd4242 # conversion from 'int' to 'char', possible loss of data
+    /wd4244 # conversion from 'long' to 'char', possible loss of data
+    /wd4267 # conversion from 'size_t' to 'int', possible loss of data
+    /wd4127 # conditional expression is constant
+    /wd4702 # unreachable code
+    /wd4101 # unreferenced local variable
+    /wd4456 # declaration hides previous local declaration
+    /wd4459 # declaration hides global declaration
+    /wd4061 # enumerator X in switch of enum Y is not explicitly handled by a case label
+    /wd4464 # relative include path contains '..'
+    /wd4710 # 'fprintf': function not inlined
+    /wd4706 # assignment within conditional expression
+
+    # Some that I'm not so happy about
+    /wd4232 # sre: nonstandard extension used: 'ml_meth': address of dllimport 'Py_GenericAlias' is not static, identity not guaranteed
+    /wd4191 # sre and sqlite: 'type cast': like unsafe conversion from 'PyObject *(__cdecl *)(MatchObject *,PyObject *const *,Py_ssize_t)' to 'void (__cdecl *)(void)'
+    /wd4918 # sre: invalid character in pragma optimization list
+    /wd4701 # cpython_unicodedata: potentially uninitialized local variable 'rc' used
+    /wd4574 # sqlite: 'SQLITE_ATOMIC_INTRINSICS' is defined to be '0': did you mean to use '#if SQLITE_ATOMIC_INTRINSICS'?
+    /wd4310 # xmlparse: cast truncates constant value
+  )
+else()
+  set(CFLAGS_WARNINGS -Wall -Werror -Wno-unused-function -Wno-unused-variable -Wno-unused-const-variable
+                    -Wno-unknown-warning-option
+                    -Wno-discarded-qualifiers                         # _testbuffer.c
                     -Wno-tautological-constant-out-of-range-compare   # fileutils.c: wchar_t > MAX_UNICODE is always false on Windows
                     -Wno-unused-but-set-variable                      # sqlite.c: BOOL bRc
                     -Wno-ignored-pragmas                              # sre.c: #pragma optimize("agtw", on)
-                    -Wno-discarded-qualifiers                         # longobject.c: *pend = str; (gcc only hence: '-Wunknown-warning-option')
                     -Wno-int-to-pointer-cast -Wno-int-conversion -Wno-void-pointer-to-int-cast
-                    -Wno-incompatible-pointer-types-discards-qualifiers -Wno-pointer-type-mismatch
+                    -Wno-incompatible-pointer-types-discards-qualifiers
                     -Wno-braced-scalar-init -Wno-deprecated-declarations)
+  add_compile_options(-ffile-prefix-map=${GRAALPY_PARENT_DIR}=.)
+endif()
 
-add_compile_options(-ffile-prefix-map=${GRAALPY_PARENT_DIR}=.)
 
 # preprocessor defines for all platforms
 add_compile_definitions(
     NDEBUG
-    GRAALVM_PYTHON_LLVM
 )
 
-add_compile_definitions(GRAALVM_PYTHON_LLVM_NATIVE)
-
 
 if(WIN32)
     add_compile_definitions(
         MS_WINDOWS
         Py_ENABLE_SHARED
         HAVE_DECLSPEC_DLL
+        
     )
 endif()
 
@@ -155,7 +183,6 @@ include_directories(
     "${SRC_DIR}/include"
     "${CAPI_INC_DIR}"
     "${PYCONFIG_INCLUDE_DIR}"
-    "${TRUFFLE_H_INC}"
     "${TRUFFLE_NFI_H_INC}"
 )
 
@@ -261,9 +288,9 @@ target_compile_definitions(${TARGET_LIBPYTHON} PRIVATE Py_BUILD_CORE Py_BUILD_CO
 target_compile_options(${TARGET_LIBPYTHON} PRIVATE ${CFLAGS_WARNINGS})
 
 if(WIN32)
-    target_link_directories(${TARGET_LIBPYTHON} PRIVATE ${GRAALVM_LLVM_LIB_DIR})
-    target_compile_options(${TARGET_LIBPYTHON} PRIVATE "-fmsc-version=1920")
-    target_link_libraries(${TARGET_LIBPYTHON} sulong-native graalvm-llvm)
+    if (NOT MSVC)
+        target_compile_options(${TARGET_LIBPYTHON} PRIVATE "-fmsc-version=1920")
+    endif()
 else()
     # Link to math library; required for functions like 'hypot' or similar
     target_link_libraries(${TARGET_LIBPYTHON} m)

graalpython/com.oracle.graal.python.cext/CMakeLists.txt

@@ -42,4 +42,6 @@
 #include <memory.h>
 #include <string.h>
 
+#include "expat_config.h"
+
 #endif /* ndef WINCONFIG_H */

graalpython/com.oracle.graal.python.cext/expat/winconfig.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2022, 2024, Oracle and/or its affiliates.
+/* Copyright (c) 2022, 2025, Oracle and/or its affiliates.
  * Copyright (C) 1996-2022 Python Software Foundation
  *
  * Licensed under the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
@@ -174,7 +174,8 @@ struct _gc_runtime_state {
 
 extern void _PyGC_InitState(struct _gc_runtime_state *);
 
-extern Py_ssize_t _PyGC_CollectNoFail(PyThreadState *tstate);
+// GraalPy change: exporting _PyGC_CollectNoFail
+extern PyAPI_FUNC(Py_ssize_t) _PyGC_CollectNoFail(PyThreadState *tstate);
 
 
 // Functions to clear types free lists

graalpython/com.oracle.graal.python.cext/include/internal/pycore_gc.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2024, Oracle and/or its affiliates.
+/* Copyright (c) 2024, 2025, Oracle and/or its affiliates.
  * Copyright (C) 1996-2023 Python Software Foundation
  *
  * Licensed under the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
@@ -53,6 +53,7 @@ struct _Py_global_objects {
         _PyGC_Head_UNUSED _tuple_empty_gc_not_used;
         PyTupleObject tuple_empty;
 */
+        char GraalDummyField;
     } singletons;
 };
 

graalpython/com.oracle.graal.python.cext/include/internal/pycore_global_objects.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2022, 2024, Oracle and/or its affiliates.
+/* Copyright (c) 2022, 2025, Oracle and/or its affiliates.
  * Copyright (C) 1996-2022 Python Software Foundation
  *
  * Licensed under the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
@@ -251,7 +251,7 @@ _PyType_PreHeaderSize(PyTypeObject *tp)
 
 void _PyObject_GC_Link(PyObject *op);
 
-void _GraalPyObject_GC_NotifyOwnershipTransfer(PyObject *op);
+PyAPI_FUNC(void) _GraalPyObject_GC_NotifyOwnershipTransfer(PyObject *op);
 
 // Usage: assert(_Py_CheckSlotResult(obj, "__getitem__", result != NULL));
 extern int _Py_CheckSlotResult(

graalpython/com.oracle.graal.python.cext/include/internal/pycore_object.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2018, 2023, Oracle and/or its affiliates.
+/* Copyright (c) 2018, 2025, Oracle and/or its affiliates.
  * Copyright (C) 1996-2020 Python Software Foundation
  *
  * Licensed under the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
@@ -27,11 +27,13 @@ PyAPI_DATA(int) (*PyOS_InputHook)(void);
    to an 8k margin. */
 #define PYOS_STACK_MARGIN 2048
 
+#if 0 // GraalPy change
 #if defined(WIN32) && !defined(MS_WIN64) && !defined(_M_ARM) && defined(_MSC_VER) && _MSC_VER >= 1300
 /* Enable stack checking under Microsoft C */
 // When changing the platforms, ensure PyOS_CheckStack() docs are still correct
 #define USE_STACKCHECK
 #endif
+#endif
 
 #ifdef USE_STACKCHECK
 /* Check that we aren't overflowing our stack */

graalpython/com.oracle.graal.python.cext/include/pythonrun.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2018, 2024, Oracle and/or its affiliates.
+/* Copyright (c) 2018, 2025, Oracle and/or its affiliates.
  * Copyright (C) 1996-2020 Python Software Foundation
  *
  * Licensed under the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
@@ -1052,19 +1052,6 @@ _sre_SRE_Pattern_split_impl(PatternObject *self, PyObject *string,
 
 }
 
-__attribute__((always_inline))
-inline static void *_memchr(const void *s, int c, Py_ssize_t n) {
-    unsigned char *p = (unsigned char*)s;
-    while(n--) {
-        if(*p != (unsigned char)c) {
-            p++;
-        } else {
-            return p;
-        }
-    }
-    return 0;
-}
-
 static PyObject*
 pattern_subx(_sremodulestate* module_state,
              PatternObject* self,
@@ -1099,7 +1086,7 @@ pattern_subx(_sremodulestate* module_state,
         ptr = getstring(ptemplate, &n, &isbytes, &charsize, &view);
         if (ptr) {
             if (charsize == 1)
-                literal = _memchr(ptr, '\\', n) == NULL;
+                literal = memchr(ptr, '\\', n) == NULL;
             else
                 literal = PyUnicode_FindChar(ptemplate, '\\', 0, n, 1) == -1;
         } else {

graalpython/com.oracle.graal.python.cext/modules/_cpython_sre/sre.c

@@ -952,7 +952,7 @@ binary_op(PyObject *v, PyObject *w, const int op_slot, const char *op_name)
     if (result == Py_NotImplemented) {
         Py_DECREF(result);
 
-        if (op_slot == NB_SLOT(nb_rshift) &&
+        if (op_slot == (int)NB_SLOT(nb_rshift) &&
             PyCFunction_CheckExact(v) &&
             strcmp(((PyCFunctionObject *)v)->m_ml->ml_name, "print") == 0)
         {

graalpython/com.oracle.graal.python.cext/src/abstract.c

@@ -1,4 +1,4 @@
-/* Copyright (c) 2024, Oracle and/or its affiliates.
+/* Copyright (c) 2024, 2025, Oracle and/or its affiliates.
  * Copyright (C) 1996-2024 Python Software Foundation
  *
  * Licensed under the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
@@ -166,7 +166,7 @@ PyBytes_FromFormatV(const char *format, va_list vargs)
                                              (__buffer)[(__pos)++] = (__spec);\
                                          } while(0)
 
-    for(int i=0; i < sizeof(buffer); i++) {
+    for(int i=0; i < (int)sizeof(buffer); i++) {
     	buffer[i] = '\0';
     }