# C++ Plugin Documentation

This document is intended for people who want to understand, maintain or
extend the C++ part of this plugin.

**NOTE:** The plugins C++ sources are located at `plugins/pythonextensions`.

## Files
The following files exist as part of the C++ plugin:

 * `pyutil.[h|cpp]`
   - Contains utilities for working with the embedded Python interpreter
   - All the direct interaction with Shiboken / Python happens
     in the .cpp file to prevent a known error
   - These files were originally based of the ['scriptableapplication'](http://code.qt.io/cgit/pyside/pyside-setup.git/tree/examples/scriptableapplication)
     example from the PySide project
 * `pythonextensionsplugin.[h|cpp]`
   - Contains the IPlugin class and manages the life-cycle of Python
     extensions
   - This name is the standard for Qt Creator plugins
 * `pythonextensions_global.h`
   - standard plugin file
 * `typesystem_qtcreator.xml`
   - Typesystem for Shiboken
   - This file describes the classes and types that are bound to
     Python
 * `wrappedclasses.h`
   - This file contains the headers that Shiboken will parse when generating
     the bindings
 * `glue` (directory)
   - This directory contains any glue code needed for the bindings
   - Currently it looks like this will mainly be needed to overcome the missing
     support for C++ function pointers in Shiboken

## Buildsystem
The following is part of the buildsystem. The buildsystem has so far only been
tested on Linux.

 * `pythonextensions.pro`
   - QMake project file
   - The projects dependencies on the Qt Creator side are handled in
     the file `pythonextensions_dependencies.pri`
   - The current build system emerged in part from the standard plugin build
     setup, combined with some .pro code from the 'scriptableapplication' and
     a lot of bits and pices added during the initial development phase (mainly
     through trial and error)
 * `pyside2.pri`
   - This file is originally based of the 'scriptableapplication'
     example from the PySide project
   - It is responsible for detecting the PySide2 configuration
 * `tools/pyside2_config.py`
   - This script is a copy of a script found in the PySide2 examples
   - It detects the PySide2 configuration and is called by the .pri file
   - **Note:** This script will be officially included as a PySide util in the future.

### Goals of the buildsystem
The buildsystem aims to achieve two things

 1. Build the C++ plugin, linking against any relevant libraries
 2. Install the generated shared object, as well as copy the
    `/python` directory into the Qt Creator plugin directory. This
    directory contains the included extension manager, which is implemented
    as a Python extension

## General architecture of the plugin
During initialization, the plugin does several things:

 1. Initialize the Python bindings (this happens during the plugin's `initialize`
    call)
    1. initialize the embedded Python
    2. bind the module object generated by Shiboken
       - Note that this requires a patch for Shiboken, that
         exposes this generated object in a way that allows
         the `PyUtils` functions to access it
    3. bind the PythonExtensionsPlugin instance itself

 2. Run the `setup.py` script for every extension that provides it (this happens
    during the `delayedInitialize` call)
    - This is run here to not block the Qt Creator ui while loading the extensions
    - It is separated from the main extension runtime, to provide a facility to
      extensions that depend on Python modules that are not part of the standard
      Python library (for an example, see `examples/numpysetup` or the extension
      loader)
    - This script fails without an explicit warning in the plugin, however, all
      Python errors are printed to stdout, so they can be inspected there during
      extension development
    - Note that the extension manager attempts to run an extensions `main.py`
      script after installing it, but does not consider the `setup.py` script.
      If `main.py` than fails to run, the extension manager advises the user to
      restart Qt Creator
    - There is currently no facility for scripts to detect if they have loaded
      before their setup was called
    - Because the setup is run every time, the setup script needs to detect if
      e.g. attempting to install dependencies is necessary (e.g. trying to import
      them)

 3. Attempt to run the `main.py` script for every extension (this happens
    during the `delayedInitialize` call)
    - An extension that does not provide a `main.py` is considered to have failed
      loading
    - An extension for which the `main.py` fails to run is considered to have not
      loaded
    - Provided `main.py` succeeds, the plugin is
      considered to have loaded (even if the setup script failed)

The extension manager provides functions to find the extension directory, list
all installed extensions and list all loaded extensions. (The loaded extensions
are tracked by the PythonExtensionPlugin instance.) For details on the signatures
of these functions, see `pythonextensionsplugin.h`.

The plugin directories that are searched for the extension directory (currently
`/python`) are determined using `ExtensionSystem::PluginManager::pluginPaths()`.

## Important points / workarounds for problems regarding the various dependencies

When executing Python code, be sure that Python was initialized (this is typically
done by the provided helper functions). Usually, any extension code should be
executed using `runScriptWithPath()` (never use `runScript()`, as it does not
provide any facilities for separating extensions).

The util function `runScriptWithPath()` runs a supplied script in a way that
allows to import Python packages / modules from the provided path. It also
isolates the script by cleaning the module namespace (and removing all
modifications to sys.path) after the script finishes execution. Currently this is
achieved by wrapping the supplied script with some special Python code, that
implements these precautions. Because of this, no variables which have
`mchawrioklpilnjajqkfl` as part of their name should be used in extension
scripts. This function is used for both executing `setup.py` and `main.py` and
should be used whenever executing user supplied scripts.

The Python initialization needs to manually link the Python shared library due
to a bug in CPython. This is done in `init()` (`PyUtils.cpp`) and has some
facilities for detecting the required version and linking against the correct
shared object. However, this needs to be more extensively tested and will probably
have to be amended with several special cases for different platforms / versions.

In the `PyUtil.cpp` file, the macros `signals` and `slots` are undefined due to
compatibility reasons with the CPython headers. Due to this incompatibility, any
Qt Creator code and Shiboken code needs to be well separated. (These symbols need
to be undefined **BEFORE** including anything Python related.)

The glue code for the MacroExpander functions `registerVariable()` and `registerPrefix()`
stores a reference to a Python object that can be called from other functions of the
MacroExpander. For this to work, there may not be an unmatched call to `PyEval_SaveThread()`
in any of the bindings for these functions. To achieve this, all those functions are modified
using a code injection in the `typesystem_qtcreator.xml`.

### My build suddenly starts seg-faulting for no apparent reason
**DON'T PANIC**, did you change the bindings? Check if the value of
```
SBK_PYTHONEXTENSIONS_INTERNAL_PYTHONEXTENSIONSPLUGIN_IDX
```
changed. (See pyutil.h and the `..._python.h` in the generated bindings.)

## Thinks that could be broken on other platforms

This is _not_ an exhaustive list, _nor_ is it backed by any actual experience with
building this project on non-linux operating systems. However, the following things
are quite hacky and might not work on other systems:

 * Different Python versions might not work
 * Wired Python setups might trip it up (e.g. on macOS)
   - Especially watch out for the manually linked Python in pyutils.cpp
 * The pip install stuff might not select the correct version of Python on every machine