In this article we explore how to call Python scripts from within C++ code, and how to send arguments and receive arguments back to C++. I have used this technique for some algorithmic implementations when I coded up the active learning library for the Flexfringe-tool.
In that implementation (currently found through source/active_learning) I once coded up algorithms to infer surrogate automata from neural networks. The flexfringe tool was written in C++, however, getting a good interface for the neural networks including all the data management seemed intimidating, especially given how assumptions about how data is being dished out changes quickly from dataset to dataset. I decided to give some generic C++ interfaces, and leave the brunt of the neural network and data management in Python scripts. This way one only has to know which C++ to use, and then adapt the Python script in a way that keeps conventions intact.
In this article will learn how I did this, and therefore also a bit about CPython. You will also hopefully learn a technique to make your life easier or build better systems. So let's dive in.
I will give a small primer on CPython that will be enough to understand all the contents of this article. For a more thorough discussion please refer to the official documentation.
But what is CPython? We already know that Python is an interpreter language. That means that unlike e.g. a C++ program a Python-script does not need to get compiled first to be run on a machine. However, in order to run a set of other programs have to run in to get the Python script translated into machine executable code. These programs are written in other languages, and chances are on your machine that language is C. We call this standard implementation of Python CPython.
The basis for any object in CPython is the PyObject. We get access to this datatype by first
including Python.h. To do so we have to tell the preprocessor where to find the Python includes.
In CMake this is done via find_package(PythonLibs REQUIRED), and a subsequent
include_directories(${PYTHON_INCLUDE_DIRS}). I note that I had trouble with this
on a new Apple-native-CPU MacBook, I refer to the CMake of Flexfringe in case that this is
what you are using. The CPython documentation further recommends us to copy-paste the line
#define PY_SSIZE_T_CLEAN before including the Python header.
Once we have access to the Python headers we first have to run Py_Initialize();.
This line of code then simply starts the Python interpreter. We can then already issue
our first commands to the interpreter via PyRun_SimpleString();. I used this
command to first import the os- and sys-modules in Python, so that I could manipulate the
syspath of the interpreter to make sure it actually finds and can include my Python script
that I want to read from and write to. In my implementation this looked like the following:
// init
Py_Initialize();
PyRun_SimpleString("import sys");
PyRun_SimpleString("import os");
// manipulate syspath
stringstream cmd;
cmd << "sys.path.append( os.path.join(os.getcwd(), \"";
cmd << PYTHON_SCRIPT_PATH;
cmd << "\") )\n";
cmd << "sys.path.append( os.path.join(os.getcwd(), \"source/active_learning/system_under_learning/neural_network_suls/python/util\") )";
PyRun_SimpleString(cmd.str().c_str());
We can now import the Python module and gain access to it, but first we need to understand garbage collection and error handling in CPYthon.
To understand garbage collection we first have to have a look at the PyObject-type. In the
official source code of CPython we find the typedef
typedef struct _object PyObject;. Investigating the
_object struct we find the following two properties:
Py_ssize_t ob_refcnt; // could also have a different data type depending on flagsPyTypeObject *ob_type;
The rules for reference counts can also be neatly found in the
CPython documentation. Interesting for us is the rule to "never declare an
automatic or static variable of type PyObject", because all PyObjects are allocated on the heap.
For us this means that we should always call a native CPython function when creating a
CPython-object. Apart from that we want to point our attention towards the two macros
Py_INCREF(PyObject* p) and Py_DECREF(PyObject* p). Here, Py_INCREF()
simply increases the reference count for the given object by one. Py_DECREF() decreases
the reference count by one, checks if it reached zero, and calls the deallocator function
of the object if necessary, the CPython equivalent of a destructor in C++. For instance,
the deallocator of a Python list makes sure that objects that it holds get processed when the
list goes out of scope.
The garbage collector works just as you'd expect then: Objects with a reference count of zero
are to be removed. Therefore the reference count is a proxy for ownership to an object. A
caller receiving an object increases its count by one, and once it relinquishes ownership it
decreases the count by one again. Pitfalls arise when functions 'steal' references. A notable
example mentioned in the documentation are PyList_SetItem() and
PyTuple_SetItem(). These two populate a list or tuple in Python with objects,
assuming that ownership is handed from the caller to the list or tuple respectively. On the
contrary, PyList_GetItem() does not pass ownership to the caller, while
PySequence_GetItem() does. It is therefore important to read the documentation
of the function whenever in doubt, or risk segfaults or memory-leaks.
Error handling is much simpler to understand than reference counting: Exception-states are
stored on a per-thread basis, and the occurance of an exception can be checked via
PyErr_Occurred(). Additionally, typically functions of the CPython-API will
have a return value. Usually, the return value can be checked on NULL,
which typically indicates that an error occured.
Now that we understand memory-management and error handling, we can load our Python code
into our C++ program. To do so, first we need to load the Python module. We first translate
our std::string PYTHON_MODULE_NAME; // the full path into a PyObject via
PyObject* p_name = PyUnicode_FromString(PYTHON_MODULE_NAME.c_str());.
We want to perform error checking: if (p_name == NULL){//handle error}.
We then import the module via PyObject* p_module = PyImport_Import(p_name);,
and we do error checking again. Next we want to get a function pointer for our
Python function. We assume the following function (a function that actually exists
in our code, and that we use to query the output of neural networks):
def do_query(x: list):
# do something
return some_list_of_ints
We can load this function as a function pointer via PyObject* query_func =
PyObject_GetAttrString(p_module, "do_query");. This time we also want to
make sure that we actually do have a function, therefore we do an extra check. For brevity
purposes here is an example for the full error handling of this call (obviously there
is some boilerplate in CPython):
query_func = PyObject_GetAttrString(p_module, "do_query");
if (query_func == NULL || !PyCallable_Check(query_func)) {
Py_DECREF(p_name);
Py_DECREF(p_module);
cerr << "Problem in loading the query function. Terminating program." << endl;
exit(1);
}
Lastly we want to call the Python script and translate its return value back into a C++
native array in the form of an std::vector<int>. Assume we have a
C++ list vector<int> cpp_list;, and we want to send it. Firstly
we need to create a new Python list:
PyObject* p_list = PyList_New(cpp_list.size()). We then have to use
PyList_SET_ITEM() to populate the newly allocated Python-list, where we
first need to thoroughly translate each value of our C++ list into a PyObject via
PyLong_FromLong().
Once that it done we can call our function via
PyObject_CallOneArg() and get the argument back as a PyObject-pointer,
as per usual. The CPython-API provides multiple functions for calling Python methods,
out of which the one with the most fitting signature should be used. For us, since we had only one
argument, this one was the best. A list of all the possible function calls can be found on
the documentation.
Once we received our result we want to do some error checking, and subsequently translate it back to our C++ vector format. We also have to remember to free the space occupied by our code holding the newly created Python objects that we used as calling arguments, and whose ownership was transferred to us from the Python script. The code snippet below exemplary shows how all this is done. For brevity purposes we omit the error checking when translating our vector to a Python list and back.
// step 1: Prepare the input
PyObject* p_list = PyList_New(cpp_list.size());
for(int i=0; i<cpp_list.size(); i++){
PyObject p_int = PyLong_FromLong(cpp_list[i]);
// some error checking here
PyList_SET_ITEM(p_list, i, p_int); // this function does no error checking
}
// step 2: call function and check for errors
PyObject* p_result = PyObject_CallOneArg(query_func, p_list);
if (p_result == NULL || !PyList_Check(p_result)){ // PyList_Check() to see if we got expected return type
PyErr_Print();
// do error handling
}
// step 3: translate back to C++ data
// more on the size can be found here: https://docs.python.org/3/c-api/intro.html#c.Py_ssize_t
size_t p_result_size = static_cast<size_t>(PyList_GET_SIZE(p_result));
std::vector<int> res(p_result_size);
for(int i=0; i<query_traces.size(); i++){
PyObject* p_type = PyList_GET_ITEM(p_result, i);
// you can check the output p_type here for additional code safety
res[i] = PyLong_AsLong(p_type); // we expect int-values from Python
}
// step 4: free up space to avoid memory leaks
Py_DECREF(p_list);
Py_DECREF(p_result);
That concludes our brief tutorial. We touched upon various topics and aspects of safely calling Python code from within C++, such as compilation, memory management, error handling, initialization and the CPython calling API. This enables us to play out the strengths of both languages. For a thorough example of how I used this technique I refer to this project. For feedback or questions please do not hesitate to contact me, e.g. via LinkedIn.