exceptions.rst 17 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401
  1. Exceptions
  2. ##########
  3. Built-in C++ to Python exception translation
  4. ============================================
  5. When Python calls C++ code through pybind11, pybind11 provides a C++ exception handler
  6. that will trap C++ exceptions, translate them to the corresponding Python exception,
  7. and raise them so that Python code can handle them.
  8. pybind11 defines translations for ``std::exception`` and its standard
  9. subclasses, and several special exception classes that translate to specific
  10. Python exceptions. Note that these are not actually Python exceptions, so they
  11. cannot be examined using the Python C API. Instead, they are pure C++ objects
  12. that pybind11 will translate the corresponding Python exception when they arrive
  13. at its exception handler.
  14. .. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}|
  15. +--------------------------------------+--------------------------------------+
  16. | Exception thrown by C++ | Translated to Python exception type |
  17. +======================================+======================================+
  18. | :class:`std::exception` | ``RuntimeError`` |
  19. +--------------------------------------+--------------------------------------+
  20. | :class:`std::bad_alloc` | ``MemoryError`` |
  21. +--------------------------------------+--------------------------------------+
  22. | :class:`std::domain_error` | ``ValueError`` |
  23. +--------------------------------------+--------------------------------------+
  24. | :class:`std::invalid_argument` | ``ValueError`` |
  25. +--------------------------------------+--------------------------------------+
  26. | :class:`std::length_error` | ``ValueError`` |
  27. +--------------------------------------+--------------------------------------+
  28. | :class:`std::out_of_range` | ``IndexError`` |
  29. +--------------------------------------+--------------------------------------+
  30. | :class:`std::range_error` | ``ValueError`` |
  31. +--------------------------------------+--------------------------------------+
  32. | :class:`std::overflow_error` | ``OverflowError`` |
  33. +--------------------------------------+--------------------------------------+
  34. | :class:`pybind11::stop_iteration` | ``StopIteration`` (used to implement |
  35. | | custom iterators) |
  36. +--------------------------------------+--------------------------------------+
  37. | :class:`pybind11::index_error` | ``IndexError`` (used to indicate out |
  38. | | of bounds access in ``__getitem__``, |
  39. | | ``__setitem__``, etc.) |
  40. +--------------------------------------+--------------------------------------+
  41. | :class:`pybind11::key_error` | ``KeyError`` (used to indicate out |
  42. | | of bounds access in ``__getitem__``, |
  43. | | ``__setitem__`` in dict-like |
  44. | | objects, etc.) |
  45. +--------------------------------------+--------------------------------------+
  46. | :class:`pybind11::value_error` | ``ValueError`` (used to indicate |
  47. | | wrong value passed in |
  48. | | ``container.remove(...)``) |
  49. +--------------------------------------+--------------------------------------+
  50. | :class:`pybind11::type_error` | ``TypeError`` |
  51. +--------------------------------------+--------------------------------------+
  52. | :class:`pybind11::buffer_error` | ``BufferError`` |
  53. +--------------------------------------+--------------------------------------+
  54. | :class:`pybind11::import_error` | ``ImportError`` |
  55. +--------------------------------------+--------------------------------------+
  56. | :class:`pybind11::attribute_error` | ``AttributeError`` |
  57. +--------------------------------------+--------------------------------------+
  58. | Any other exception | ``RuntimeError`` |
  59. +--------------------------------------+--------------------------------------+
  60. Exception translation is not bidirectional. That is, *catching* the C++
  61. exceptions defined above will not trap exceptions that originate from
  62. Python. For that, catch :class:`pybind11::error_already_set`. See :ref:`below
  63. <handling_python_exceptions_cpp>` for further details.
  64. There is also a special exception :class:`cast_error` that is thrown by
  65. :func:`handle::call` when the input arguments cannot be converted to Python
  66. objects.
  67. Registering custom translators
  68. ==============================
  69. If the default exception conversion policy described above is insufficient,
  70. pybind11 also provides support for registering custom exception translators.
  71. Similar to pybind11 classes, exception translators can be local to the module
  72. they are defined in or global to the entire python session. To register a simple
  73. exception conversion that translates a C++ exception into a new Python exception
  74. using the C++ exception's ``what()`` method, a helper function is available:
  75. .. code-block:: cpp
  76. py::register_exception<CppExp>(module, "PyExp");
  77. This call creates a Python exception class with the name ``PyExp`` in the given
  78. module and automatically converts any encountered exceptions of type ``CppExp``
  79. into Python exceptions of type ``PyExp``.
  80. A matching function is available for registering a local exception translator:
  81. .. code-block:: cpp
  82. py::register_local_exception<CppExp>(module, "PyExp");
  83. It is possible to specify base class for the exception using the third
  84. parameter, a ``handle``:
  85. .. code-block:: cpp
  86. py::register_exception<CppExp>(module, "PyExp", PyExc_RuntimeError);
  87. py::register_local_exception<CppExp>(module, "PyExp", PyExc_RuntimeError);
  88. Then ``PyExp`` can be caught both as ``PyExp`` and ``RuntimeError``.
  89. The class objects of the built-in Python exceptions are listed in the Python
  90. documentation on `Standard Exceptions <https://docs.python.org/3/c-api/exceptions.html#standard-exceptions>`_.
  91. The default base class is ``PyExc_Exception``.
  92. When more advanced exception translation is needed, the functions
  93. ``py::register_exception_translator(translator)`` and
  94. ``py::register_local_exception_translator(translator)`` can be used to register
  95. functions that can translate arbitrary exception types (and which may include
  96. additional logic to do so). The functions takes a stateless callable (e.g. a
  97. function pointer or a lambda function without captured variables) with the call
  98. signature ``void(std::exception_ptr)``.
  99. When a C++ exception is thrown, the registered exception translators are tried
  100. in reverse order of registration (i.e. the last registered translator gets the
  101. first shot at handling the exception). All local translators will be tried
  102. before a global translator is tried.
  103. Inside the translator, ``std::rethrow_exception`` should be used within
  104. a try block to re-throw the exception. One or more catch clauses to catch
  105. the appropriate exceptions should then be used with each clause using
  106. ``PyErr_SetString`` to set a Python exception or ``ex(string)`` to set
  107. the python exception to a custom exception type (see below).
  108. To declare a custom Python exception type, declare a ``py::exception`` variable
  109. and use this in the associated exception translator (note: it is often useful
  110. to make this a static declaration when using it inside a lambda expression
  111. without requiring capturing).
  112. The following example demonstrates this for a hypothetical exception classes
  113. ``MyCustomException`` and ``OtherException``: the first is translated to a
  114. custom python exception ``MyCustomError``, while the second is translated to a
  115. standard python RuntimeError:
  116. .. code-block:: cpp
  117. static py::exception<MyCustomException> exc(m, "MyCustomError");
  118. py::register_exception_translator([](std::exception_ptr p) {
  119. try {
  120. if (p) std::rethrow_exception(p);
  121. } catch (const MyCustomException &e) {
  122. exc(e.what());
  123. } catch (const OtherException &e) {
  124. PyErr_SetString(PyExc_RuntimeError, e.what());
  125. }
  126. });
  127. Multiple exceptions can be handled by a single translator, as shown in the
  128. example above. If the exception is not caught by the current translator, the
  129. previously registered one gets a chance.
  130. If none of the registered exception translators is able to handle the
  131. exception, it is handled by the default converter as described in the previous
  132. section.
  133. .. seealso::
  134. The file :file:`tests/test_exceptions.cpp` contains examples
  135. of various custom exception translators and custom exception types.
  136. .. note::
  137. Call either ``PyErr_SetString`` or a custom exception's call
  138. operator (``exc(string)``) for every exception caught in a custom exception
  139. translator. Failure to do so will cause Python to crash with ``SystemError:
  140. error return without exception set``.
  141. Exceptions that you do not plan to handle should simply not be caught, or
  142. may be explicitly (re-)thrown to delegate it to the other,
  143. previously-declared existing exception translators.
  144. Note that ``libc++`` and ``libstdc++`` `behave differently under macOS
  145. <https://stackoverflow.com/questions/19496643/using-clang-fvisibility-hidden-and-typeinfo-and-type-erasure/28827430>`_
  146. with ``-fvisibility=hidden``. Therefore exceptions that are used across ABI
  147. boundaries need to be explicitly exported, as exercised in
  148. ``tests/test_exceptions.h``. See also:
  149. "Problems with C++ exceptions" under `GCC Wiki <https://gcc.gnu.org/wiki/Visibility>`_.
  150. Local vs Global Exception Translators
  151. =====================================
  152. When a global exception translator is registered, it will be applied across all
  153. modules in the reverse order of registration. This can create behavior where the
  154. order of module import influences how exceptions are translated.
  155. If module1 has the following translator:
  156. .. code-block:: cpp
  157. py::register_exception_translator([](std::exception_ptr p) {
  158. try {
  159. if (p) std::rethrow_exception(p);
  160. } catch (const std::invalid_argument &e) {
  161. PyErr_SetString("module1 handled this")
  162. }
  163. }
  164. and module2 has the following similar translator:
  165. .. code-block:: cpp
  166. py::register_exception_translator([](std::exception_ptr p) {
  167. try {
  168. if (p) std::rethrow_exception(p);
  169. } catch (const std::invalid_argument &e) {
  170. PyErr_SetString("module2 handled this")
  171. }
  172. }
  173. then which translator handles the invalid_argument will be determined by the
  174. order that module1 and module2 are imported. Since exception translators are
  175. applied in the reverse order of registration, which ever module was imported
  176. last will "win" and that translator will be applied.
  177. If there are multiple pybind11 modules that share exception types (either
  178. standard built-in or custom) loaded into a single python instance and
  179. consistent error handling behavior is needed, then local translators should be
  180. used.
  181. Changing the previous example to use ``register_local_exception_translator``
  182. would mean that when invalid_argument is thrown in the module2 code, the
  183. module2 translator will always handle it, while in module1, the module1
  184. translator will do the same.
  185. .. _handling_python_exceptions_cpp:
  186. Handling exceptions from Python in C++
  187. ======================================
  188. When C++ calls Python functions, such as in a callback function or when
  189. manipulating Python objects, and Python raises an ``Exception``, pybind11
  190. converts the Python exception into a C++ exception of type
  191. :class:`pybind11::error_already_set` whose payload contains a C++ string textual
  192. summary and the actual Python exception. ``error_already_set`` is used to
  193. propagate Python exception back to Python (or possibly, handle them in C++).
  194. .. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}|
  195. +--------------------------------------+--------------------------------------+
  196. | Exception raised in Python | Thrown as C++ exception type |
  197. +======================================+======================================+
  198. | Any Python ``Exception`` | :class:`pybind11::error_already_set` |
  199. +--------------------------------------+--------------------------------------+
  200. For example:
  201. .. code-block:: cpp
  202. try {
  203. // open("missing.txt", "r")
  204. auto file = py::module_::import("io").attr("open")("missing.txt", "r");
  205. auto text = file.attr("read")();
  206. file.attr("close")();
  207. } catch (py::error_already_set &e) {
  208. if (e.matches(PyExc_FileNotFoundError)) {
  209. py::print("missing.txt not found");
  210. } else if (e.matches(PyExc_PermissionError)) {
  211. py::print("missing.txt found but not accessible");
  212. } else {
  213. throw;
  214. }
  215. }
  216. Note that C++ to Python exception translation does not apply here, since that is
  217. a method for translating C++ exceptions to Python, not vice versa. The error raised
  218. from Python is always ``error_already_set``.
  219. This example illustrates this behavior:
  220. .. code-block:: cpp
  221. try {
  222. py::eval("raise ValueError('The Ring')");
  223. } catch (py::value_error &boromir) {
  224. // Boromir never gets the ring
  225. assert(false);
  226. } catch (py::error_already_set &frodo) {
  227. // Frodo gets the ring
  228. py::print("I will take the ring");
  229. }
  230. try {
  231. // py::value_error is a request for pybind11 to raise a Python exception
  232. throw py::value_error("The ball");
  233. } catch (py::error_already_set &cat) {
  234. // cat won't catch the ball since
  235. // py::value_error is not a Python exception
  236. assert(false);
  237. } catch (py::value_error &dog) {
  238. // dog will catch the ball
  239. py::print("Run Spot run");
  240. throw; // Throw it again (pybind11 will raise ValueError)
  241. }
  242. Handling errors from the Python C API
  243. =====================================
  244. Where possible, use :ref:`pybind11 wrappers <wrappers>` instead of calling
  245. the Python C API directly. When calling the Python C API directly, in
  246. addition to manually managing reference counts, one must follow the pybind11
  247. error protocol, which is outlined here.
  248. After calling the Python C API, if Python returns an error,
  249. ``throw py::error_already_set();``, which allows pybind11 to deal with the
  250. exception and pass it back to the Python interpreter. This includes calls to
  251. the error setting functions such as ``PyErr_SetString``.
  252. .. code-block:: cpp
  253. PyErr_SetString(PyExc_TypeError, "C API type error demo");
  254. throw py::error_already_set();
  255. // But it would be easier to simply...
  256. throw py::type_error("pybind11 wrapper type error");
  257. Alternately, to ignore the error, call `PyErr_Clear
  258. <https://docs.python.org/3/c-api/exceptions.html#c.PyErr_Clear>`_.
  259. Any Python error must be thrown or cleared, or Python/pybind11 will be left in
  260. an invalid state.
  261. Chaining exceptions ('raise from')
  262. ==================================
  263. Python has a mechanism for indicating that exceptions were caused by other
  264. exceptions:
  265. .. code-block:: py
  266. try:
  267. print(1 / 0)
  268. except Exception as exc:
  269. raise RuntimeError("could not divide by zero") from exc
  270. To do a similar thing in pybind11, you can use the ``py::raise_from`` function. It
  271. sets the current python error indicator, so to continue propagating the exception
  272. you should ``throw py::error_already_set()``.
  273. .. code-block:: cpp
  274. try {
  275. py::eval("print(1 / 0"));
  276. } catch (py::error_already_set &e) {
  277. py::raise_from(e, PyExc_RuntimeError, "could not divide by zero");
  278. throw py::error_already_set();
  279. }
  280. .. versionadded:: 2.8
  281. .. _unraisable_exceptions:
  282. Handling unraisable exceptions
  283. ==============================
  284. If a Python function invoked from a C++ destructor or any function marked
  285. ``noexcept(true)`` (collectively, "noexcept functions") throws an exception, there
  286. is no way to propagate the exception, as such functions may not throw.
  287. Should they throw or fail to catch any exceptions in their call graph,
  288. the C++ runtime calls ``std::terminate()`` to abort immediately.
  289. Similarly, Python exceptions raised in a class's ``__del__`` method do not
  290. propagate, but are logged by Python as an unraisable error. In Python 3.8+, a
  291. `system hook is triggered
  292. <https://docs.python.org/3/library/sys.html#sys.unraisablehook>`_
  293. and an auditing event is logged.
  294. Any noexcept function should have a try-catch block that traps
  295. class:`error_already_set` (or any other exception that can occur). Note that
  296. pybind11 wrappers around Python exceptions such as
  297. :class:`pybind11::value_error` are *not* Python exceptions; they are C++
  298. exceptions that pybind11 catches and converts to Python exceptions. Noexcept
  299. functions cannot propagate these exceptions either. A useful approach is to
  300. convert them to Python exceptions and then ``discard_as_unraisable`` as shown
  301. below.
  302. .. code-block:: cpp
  303. void nonthrowing_func() noexcept(true) {
  304. try {
  305. // ...
  306. } catch (py::error_already_set &eas) {
  307. // Discard the Python error using Python APIs, using the C++ magic
  308. // variable __func__. Python already knows the type and value and of the
  309. // exception object.
  310. eas.discard_as_unraisable(__func__);
  311. } catch (const std::exception &e) {
  312. // Log and discard C++ exceptions.
  313. third_party::log(e);
  314. }
  315. }
  316. .. versionadded:: 2.6