Py_Finalize() function, why it does not finalise Python??

[AlexSoft73]

The following code shows a bug in Py_Finalize(). It does not finalize python interpreter, the second round fires an error.

Thanks in advance to all of those of you that want's a better python
Alexei

Python code (copy it and save it as a .py file)
[Python code]
import numpy as np

The variable <char *pcharPathName> change it to the proper pathname of the python code. Test it
[c]

#ifdef _DEBUG
#define ALEX_PYTHON_TWEEK _DEBUG
#undef _DEBUG
#endif
//#define PY_SSIZE_T_CLEAN
//needed to define the following directive to avoid static linking
//#define Py_LIMITED_API 0x03110000
#include <Python.h>
//needed to avoid static linking
//#pragma comment (lib,"python311.lib")
//restore _DEBUB macro after python include
#ifdef ALEX_PYTHON_TWEEK
#define _DEBUG ALEX_PYTHON_TWEEK
#endif
#include <stdlib.h>


char* charFromWStr(wchar_t* pWChar,SIZE_T uLen)
{
	char* pChar;
	
	if (uLen)
	{
		//Alocare string len including End of Str char
		pChar = (char*)malloc(uLen  + 1);
		if (pChar)
		{
			memset(pChar, 0, uLen + 1);
			if (pChar)
			{
				SIZE_T I;
				for (I = 0; I < uLen; I++)
				{
					pChar[I] = (char)pWChar[I];
				}
				return pChar;
			}
		}
	}
	return NULL;
}

void PythonObject_DecodeString(char** ppcharValue, PyObject* pPyValue)
{
	SIZE_T sLen;
	wchar_t* pwcDymmy;
	if (ppcharValue)
	{
		if (pPyValue)//
		{
			sLen = (PyUnicode_GetLength(pPyValue) + 1);
			SIZE_T byteSize = sLen * sizeof(wchar_t);
			pwcDymmy = malloc(byteSize);
			SIZE_T Z = PyUnicode_AsWideChar(pPyValue, pwcDymmy, byteSize);

			if (Z) *ppcharValue = charFromWStr(pwcDymmy, sLen);
			free(pwcDymmy);
			return;
		}
		else *ppcharValue = NULL;
	}
}
void ReportPythonError()
{
	PyObject* pType, * pValue, * pTraceBack, * PyObjAsStr;
	PyErr_Fetch(&pType, &pValue, &pTraceBack);
	PyObjAsStr = PyObject_Str(pValue);
	if (PyObjAsStr)
	{
		char* pwc;
		PythonObject_DecodeString(&pwc, PyObjAsStr);
		if (pwc)
		{
			printf(pwc);
		}else printf("Could not retrieve error!");
	
	}
}
void RunCodeFromFile(PyObject* pModule,char* charPathName)
{
	int f = open(charPathName, 0,"r");
	if (f)
	{
		int l = 1024 * 1024;
		char* pBuff = malloc(l);
		if (pBuff)
		{
			memset(pBuff, 0, l);
			int ll = read(f, pBuff, l);
			if (ll)
			{
				PyErr_Clear();
				PyObject *pPyCode = Py_CompileString(pBuff,"", Py_file_input);
				if (pPyCode)
				{
					PyObject *pPyGlobals = PyModule_GetDict(pModule);//borrowwed Ref
					PyObject *pPyResult  = PyEval_EvalCode(pPyCode, pPyGlobals, pPyGlobals);
					if (pPyResult)
					{
						Py_DecRef(pPyResult);
					}
					Py_DecRef(pPyCode);
				}
				else ReportPythonError();

			}
			free(pBuff);
		}
		close(f);
	}
}
char* pcharPathName = "D:\\Programming\\VC\\2019\\_COMCLASSES\\Scripting\\Demo\\test.py";

int main(int argc, char* argv[])
{
	Py_InitializeEx(0);
	if (Py_IsInitialized())
	{
		PyObject *pPyMain = PyModule_New("__main__");
		if (pPyMain)
		{
			RunCodeFromFile(pPyMain, pcharPathName);
			Py_DecRef(pPyMain);
		}
		Py_Finalize();
	}
	//2nd round
	Py_InitializeEx(0);
	
	if (Py_IsInitialized())
	{
		PyObject* pPyMain = PyModule_New("__main__");
		if (pPyMain)
		{
			RunCodeFromFile(pPyMain, pcharPathName);
			Py_DecRef(pPyMain);
		}
		Py_Finalize();
	}
	return 0;
}

The 2nd round reports the following:
[ERROR Report]
Traceback (most recent call last):
File "", line 3, in
File "C:\Python\Python311\Lib\site-packages\numpy_init_.py", line 140, in
from . import core
File "C:\Python\Python311\Lib\site-packages\numpy\core_init_.py", line 23, in
from . import multiarray
File "C:\Python\Python311\Lib\site-packages\numpy\core\multiarray.py", line 10, in
from . import overrides
File "C:\Python\Python311\Lib\site-packages\numpy\core\overrides.py", line 6, in
from numpy.core._multiarray_umath import (
SystemError: D:_w\1\s\Objects\structseq.c:476: bad argument to internal function

[kumaraditya303]

numpy does not supports multiple interpreter initializations, it is not a bug in CPython. You may report this to numpy tracker.

[AlexSoft73]

Thanks kumaraditya303

CPython does have memory leaks, it does not need to be tested with numpy, that was just another sample of memoryt leaks.

By just executing:

Py_Initialize(...)
Py_Finalise()

you get memory leaks. But these days the CPyton "core developers"coalition is committee to close every thread that talks about CPython memory leaks taken any escuse as reason, this time numpy.

[arhadthedev]

CPython does have memory leaks, it does not need to be tested with numpy, that was just another sample of memoryt leaks.

[...] CPython "core developers"c oalition is committee to close every thread that talks about CPython memory leaks taken any escuse as reason, [...]

Because memory leaks in legacy code left for older plugins.

CPython now confidently transitions from global variables that actually cause memory leaks (because nobody cleaned objects that are supposed to die with the program anyway) to dynamically allocated structures destroyed with a single, simple and thus always presented function call. These structures are "heap types" and "support of multiple interpreters inside the same process".

Unfortunately for all of us such a transition is a mess that will be here for a long time until everything involved (both all CPython modules and every third party module) gets converted so nothing uses that legacy code anymore. In the worst case it can be akin to Python 2-to-3 transition lasting for a decade.

So this problem is well known (so reports repeating over and over again are closed) and is being steadely resolved, but not as fast as desired.

[kumaraditya303]

you get memory leaks. But these days the CPyton "core developers"coalition is committee to close every thread that talks about CPython memory leaks taken any escuse as reason, this time numpy.

It is a known problem and the work is ongoing, if you want to help read https://peps.python.org/pep-0687/ and work on it, creating issues for the same thing over and over is not helpful.