faq.rst 13 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307
  1. Frequently asked questions
  2. ##########################
  3. "ImportError: dynamic module does not define init function"
  4. ===========================================================
  5. 1. Make sure that the name specified in PYBIND11_MODULE is identical to the
  6. filename of the extension library (without suffixes such as ``.so``).
  7. 2. If the above did not fix the issue, you are likely using an incompatible
  8. version of Python that does not match what you compiled with.
  9. "Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``"
  10. ========================================================================
  11. See the first answer.
  12. "SystemError: dynamic module not initialized properly"
  13. ======================================================
  14. See the first answer.
  15. The Python interpreter immediately crashes when importing my module
  16. ===================================================================
  17. See the first answer.
  18. .. _faq_reference_arguments:
  19. Limitations involving reference arguments
  20. =========================================
  21. In C++, it's fairly common to pass arguments using mutable references or
  22. mutable pointers, which allows both read and write access to the value
  23. supplied by the caller. This is sometimes done for efficiency reasons, or to
  24. realize functions that have multiple return values. Here are two very basic
  25. examples:
  26. .. code-block:: cpp
  27. void increment(int &i) { i++; }
  28. void increment_ptr(int *i) { (*i)++; }
  29. In Python, all arguments are passed by reference, so there is no general
  30. issue in binding such code from Python.
  31. However, certain basic Python types (like ``str``, ``int``, ``bool``,
  32. ``float``, etc.) are **immutable**. This means that the following attempt
  33. to port the function to Python doesn't have the same effect on the value
  34. provided by the caller -- in fact, it does nothing at all.
  35. .. code-block:: python
  36. def increment(i):
  37. i += 1 # nope..
  38. pybind11 is also affected by such language-level conventions, which means that
  39. binding ``increment`` or ``increment_ptr`` will also create Python functions
  40. that don't modify their arguments.
  41. Although inconvenient, one workaround is to encapsulate the immutable types in
  42. a custom type that does allow modifications.
  43. An other alternative involves binding a small wrapper lambda function that
  44. returns a tuple with all output arguments (see the remainder of the
  45. documentation for examples on binding lambda functions). An example:
  46. .. code-block:: cpp
  47. int foo(int &i) { i++; return 123; }
  48. and the binding code
  49. .. code-block:: cpp
  50. m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); });
  51. How can I reduce the build time?
  52. ================================
  53. It's good practice to split binding code over multiple files, as in the
  54. following example:
  55. :file:`example.cpp`:
  56. .. code-block:: cpp
  57. void init_ex1(py::module_ &);
  58. void init_ex2(py::module_ &);
  59. /* ... */
  60. PYBIND11_MODULE(example, m) {
  61. init_ex1(m);
  62. init_ex2(m);
  63. /* ... */
  64. }
  65. :file:`ex1.cpp`:
  66. .. code-block:: cpp
  67. void init_ex1(py::module_ &m) {
  68. m.def("add", [](int a, int b) { return a + b; });
  69. }
  70. :file:`ex2.cpp`:
  71. .. code-block:: cpp
  72. void init_ex2(py::module_ &m) {
  73. m.def("sub", [](int a, int b) { return a - b; });
  74. }
  75. :command:`python`:
  76. .. code-block:: pycon
  77. >>> import example
  78. >>> example.add(1, 2)
  79. 3
  80. >>> example.sub(1, 1)
  81. 0
  82. As shown above, the various ``init_ex`` functions should be contained in
  83. separate files that can be compiled independently from one another, and then
  84. linked together into the same final shared object. Following this approach
  85. will:
  86. 1. reduce memory requirements per compilation unit.
  87. 2. enable parallel builds (if desired).
  88. 3. allow for faster incremental builds. For instance, when a single class
  89. definition is changed, only a subset of the binding code will generally need
  90. to be recompiled.
  91. "recursive template instantiation exceeded maximum depth of 256"
  92. ================================================================
  93. If you receive an error about excessive recursive template evaluation, try
  94. specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The
  95. culprit is generally the generation of function signatures at compile time
  96. using C++14 template metaprogramming.
  97. .. _`faq:hidden_visibility`:
  98. "'SomeClass' declared with greater visibility than the type of its field 'SomeClass::member' [-Wattributes]"
  99. ============================================================================================================
  100. This error typically indicates that you are compiling without the required
  101. ``-fvisibility`` flag. pybind11 code internally forces hidden visibility on
  102. all internal code, but if non-hidden (and thus *exported*) code attempts to
  103. include a pybind type (for example, ``py::object`` or ``py::list``) you can run
  104. into this warning.
  105. To avoid it, make sure you are specifying ``-fvisibility=hidden`` when
  106. compiling pybind code.
  107. As to why ``-fvisibility=hidden`` is necessary, because pybind modules could
  108. have been compiled under different versions of pybind itself, it is also
  109. important that the symbols defined in one module do not clash with the
  110. potentially-incompatible symbols defined in another. While Python extension
  111. modules are usually loaded with localized symbols (under POSIX systems
  112. typically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default
  113. can be changed, but even if it isn't it is not always enough to guarantee
  114. complete independence of the symbols involved when not using
  115. ``-fvisibility=hidden``.
  116. Additionally, ``-fvisibility=hidden`` can deliver considerably binary size
  117. savings. (See the following section for more details.)
  118. .. _`faq:symhidden`:
  119. How can I create smaller binaries?
  120. ==================================
  121. To do its job, pybind11 extensively relies on a programming technique known as
  122. *template metaprogramming*, which is a way of performing computation at compile
  123. time using type information. Template metaprogramming usually instantiates code
  124. involving significant numbers of deeply nested types that are either completely
  125. removed or reduced to just a few instructions during the compiler's optimization
  126. phase. However, due to the nested nature of these types, the resulting symbol
  127. names in the compiled extension library can be extremely long. For instance,
  128. the included test suite contains the following symbol:
  129. .. only:: html
  130. .. code-block:: none
  131. _​_​Z​N​8​p​y​b​i​n​d​1​1​1​2​c​p​p​_​f​u​n​c​t​i​o​n​C​1​I​v​8​E​x​a​m​p​l​e​2​J​R​N​S​t​3​_​_​1​6​v​e​c​t​o​r​I​N​S​3​_​1​2​b​a​s​i​c​_​s​t​r​i​n​g​I​w​N​S​3​_​1​1​c​h​a​r​_​t​r​a​i​t​s​I​w​E​E​N​S​3​_​9​a​l​l​o​c​a​t​o​r​I​w​E​E​E​E​N​S​8​_​I​S​A​_​E​E​E​E​E​J​N​S​_​4​n​a​m​e​E​N​S​_​7​s​i​b​l​i​n​g​E​N​S​_​9​i​s​_​m​e​t​h​o​d​E​A​2​8​_​c​E​E​E​M​T​0​_​F​T​_​D​p​T​1​_​E​D​p​R​K​T​2​_
  132. .. only:: not html
  133. .. code-block:: cpp
  134. __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_
  135. which is the mangled form of the following function type:
  136. .. code-block:: cpp
  137. pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28])
  138. The memory needed to store just the mangled name of this function (196 bytes)
  139. is larger than the actual piece of code (111 bytes) it represents! On the other
  140. hand, it's silly to even give this function a name -- after all, it's just a
  141. tiny cog in a bigger piece of machinery that is not exposed to the outside
  142. world. So we'll generally only want to export symbols for those functions which
  143. are actually called from the outside.
  144. This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC
  145. and Clang, which sets the default symbol visibility to *hidden*, which has a
  146. tremendous impact on the final binary size of the resulting extension library.
  147. (On Visual Studio, symbols are already hidden by default, so nothing needs to
  148. be done there.)
  149. In addition to decreasing binary size, ``-fvisibility=hidden`` also avoids
  150. potential serious issues when loading multiple modules and is required for
  151. proper pybind operation. See the previous FAQ entry for more details.
  152. How can I properly handle Ctrl-C in long-running functions?
  153. ===========================================================
  154. Ctrl-C is received by the Python interpreter, and holds it until the GIL
  155. is released, so a long-running function won't be interrupted.
  156. To interrupt from inside your function, you can use the ``PyErr_CheckSignals()``
  157. function, that will tell if a signal has been raised on the Python side. This
  158. function merely checks a flag, so its impact is negligible. When a signal has
  159. been received, you must either explicitly interrupt execution by throwing
  160. ``py::error_already_set`` (which will propagate the existing
  161. ``KeyboardInterrupt``), or clear the error (which you usually will not want):
  162. .. code-block:: cpp
  163. PYBIND11_MODULE(example, m)
  164. {
  165. m.def("long running_func", []()
  166. {
  167. for (;;) {
  168. if (PyErr_CheckSignals() != 0)
  169. throw py::error_already_set();
  170. // Long running iteration
  171. }
  172. });
  173. }
  174. CMake doesn't detect the right Python version
  175. =============================================
  176. The CMake-based build system will try to automatically detect the installed
  177. version of Python and link against that. When this fails, or when there are
  178. multiple versions of Python and it finds the wrong one, delete
  179. ``CMakeCache.txt`` and then add ``-DPYTHON_EXECUTABLE=$(which python)`` to your
  180. CMake configure line. (Replace ``$(which python)`` with a path to python if
  181. your prefer.)
  182. You can alternatively try ``-DPYBIND11_FINDPYTHON=ON``, which will activate the
  183. new CMake FindPython support instead of pybind11's custom search. Requires
  184. CMake 3.12+, and 3.15+ or 3.18.2+ are even better. You can set this in your
  185. ``CMakeLists.txt`` before adding or finding pybind11, as well.
  186. Inconsistent detection of Python version in CMake and pybind11
  187. ==============================================================
  188. The functions ``find_package(PythonInterp)`` and ``find_package(PythonLibs)``
  189. provided by CMake for Python version detection are modified by pybind11 due to
  190. unreliability and limitations that make them unsuitable for pybind11's needs.
  191. Instead pybind11 provides its own, more reliable Python detection CMake code.
  192. Conflicts can arise, however, when using pybind11 in a project that *also* uses
  193. the CMake Python detection in a system with several Python versions installed.
  194. This difference may cause inconsistencies and errors if *both* mechanisms are
  195. used in the same project.
  196. There are three possible solutions:
  197. 1. Avoid using ``find_package(PythonInterp)`` and ``find_package(PythonLibs)``
  198. from CMake and rely on pybind11 in detecting Python version. If this is not
  199. possible, the CMake machinery should be called *before* including pybind11.
  200. 2. Set ``PYBIND11_FINDPYTHON`` to ``True`` or use ``find_package(Python
  201. COMPONENTS Interpreter Development)`` on modern CMake (3.12+, 3.15+ better,
  202. 3.18.2+ best). Pybind11 in these cases uses the new CMake FindPython instead
  203. of the old, deprecated search tools, and these modules are much better at
  204. finding the correct Python.
  205. 3. Set ``PYBIND11_NOPYTHON`` to ``TRUE``. Pybind11 will not search for Python.
  206. However, you will have to use the target-based system, and do more setup
  207. yourself, because it does not know about or include things that depend on
  208. Python, like ``pybind11_add_module``. This might be ideal for integrating
  209. into an existing system, like scikit-build's Python helpers.
  210. How to cite this project?
  211. =========================
  212. We suggest the following BibTeX template to cite pybind11 in scientific
  213. discourse:
  214. .. code-block:: bash
  215. @misc{pybind11,
  216. author = {Wenzel Jakob and Jason Rhinelander and Dean Moldovan},
  217. year = {2017},
  218. note = {https://github.com/pybind/pybind11},
  219. title = {pybind11 -- Seamless operability between C++11 and Python}
  220. }