Re: PyGILState_Ensure() deadlocks, why?

On 2024-07-07 20:40, Tomas Ukkonen via Python-list wrote:

Hi
 There was a bug in the example code. I fixed it and it STILL deadlocks (my larger software project deadlocks when I call python from C++).
 Updated code:
 /* * This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
  *
  * g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
  * ./a.out
  *
  * uname:
  * Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
  */
 #include <Python.h>
#include <thread>
#include <vector>
#include <iostream>
  void thread_function() {
     // Ensure this thread has the GIL
     PyGILState_STATE gstate = PyGILState_Ensure();
      // Execute some Python code
     PyRun_SimpleString("print('Hello from std::thread!')");
      // Release the GIL
     PyGILState_Release(gstate);
}
 int main() {
     // Initialize the Python Interpreter
     Py_Initialize();
      // Create a vector of threads
     std::vector<std::thread> threads;
      // Launch threads
     for (int i = 0; i < 5; ++i) {
         threads.push_back(std::thread(thread_function));
     }
      // Join threads
     for (auto& t : threads) {
         t.join();
     }
      // Finalize the Python Interpreter
     Py_Finalize();
      return 0;
}
sunnuntaina 7. heinäkuuta 2024 klo 10:24 ip, Tomas Ukkonen <tomas.ukkonen@protonmail.ch> kirjoitti:
 

Hello
Is this python c api bug? The following C++ code (g++) deadlocks on Ubuntu Linux.
 

 

/* * This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
 *
 * g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
 * ./a.out
 *
 * uname:
 * Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
 */
 

 

#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>
 

 

void perform_long_operation() {
    // Simulate a long-running task
    std::this_thread::sleep_for(std::chrono::seconds(5));
}
 

 

void thread_function() {
    // Ensure this thread has the GIL
    PyGILState_STATE gstate = PyGILState_Ensure();
 

 

    // Execute some Python code
    PyRun_SimpleString("print('Hello from std::thread!')");
 

 

    // Release the GIL for long operation
    Py_BEGIN_ALLOW_THREADS
    perform_long_operation();
    Py_END_ALLOW_THREADS
 

 

    // Re-acquire the GIL and execute more Python code
    gstate = PyGILState_Ensure();
    PyRun_SimpleString("print('Thread operation completed!')");
 

 

    // Release the GIL
    PyGILState_Release(gstate);
}
 

 

int main() {
    // Initialize the Python Interpreter
    Py_Initialize();
 

At this point, there's only one thread (the main thread) and it owns the GIL.

 

    // Create a vector of threads
    std::vector<std::thread> threads;
 

 

    // Launch threads
    for (int i = 0; i < 5; ++i) {
        threads.push_back(std::thread(thread_function));
    }
 

The threads will each try to acquire and release the GIL, but it's still owned by the main thread.

 

    // Join threads
    for (auto& t : threads) {
        t.join();
    }

The main thread is waiting for the sub-threads to finish, and the threads waiting for the GIL, but the main thread still owns the GIL, so they'll be waiting forever. Deadlock.

 

 

    // Finalize the Python Interpreter
    Py_Finalize();
 

 

    return 0;
}