This is an OpenPGP/MIME signed message (RFC 4880 and 3156) -----------------------9e29034e00ea96b7d10a477458ea4ed4 Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;charset=utf-8

Hi

There was a bug in the example code. I fixed it and it STILL deadlocks (my larger software project deadlocks when I call python from C++).

Updated code:

/* * This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
 *
 * g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
 * ./a.out
 *
 * uname:
 * Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
 */

#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>


void thread_function() {
    // Ensure this thread has the GIL
    PyGILState_STATE gstate = PyGILState_Ensure();

    // Execute some Python code
    PyRun_SimpleString("print('Hello from std::thread!')");

    // Release the GIL
    PyGILState_Release(gstate);
}

int main() {
    // Initialize the Python Interpreter
    Py_Initialize();

    // Create a vector of threads
    std::vector<std::thread> threads;

    // Launch threads
    for (int i = 0; i < 5; ++i) {
        threads.push_back(std::thread(thread_function));
    }

    // Join threads
    for (auto& t : threads) {
        t.join();
    }

    // Finalize the Python Interpreter
    Py_Finalize();

    return 0;
}
sunnuntaina 7. heinäkuuta 2024 klo 10:24 ip, Tomas Ukkonen <[email protected]> kirjoitti:

Hello
Is this python c api bug? The following C++ code (g++) deadlocks on Ubuntu Linux.

/* * This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
 *
 * g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
 * ./a.out
 *
 * uname:
 * Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
 */

#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>

void perform_long_operation() {
    // Simulate a long-running task
    std::this_thread::sleep_for(std::chrono::seconds(5));
}

void thread_function() {
    // Ensure this thread has the GIL
    PyGILState_STATE gstate = PyGILState_Ensure();

    // Execute some Python code
    PyRun_SimpleString("print('Hello from std::thread!')");

    // Release the GIL for long operation
    Py_BEGIN_ALLOW_THREADS
    perform_long_operation();
    Py_END_ALLOW_THREADS

    // Re-acquire the GIL and execute more Python code
    gstate = PyGILState_Ensure();
    PyRun_SimpleString("print('Thread operation completed!')");

    // Release the GIL
    PyGILState_Release(gstate);
}

int main() {
    // Initialize the Python Interpreter
    Py_Initialize();

    // Create a vector of threads
    std::vector<std::thread> threads;

    // Launch threads
    for (int i = 0; i < 5; ++i) {
        threads.push_back(std::thread(thread_function));
    }

    // Join threads
    for (auto& t : threads) {
        t.join();
    }

    // Finalize the Python Interpreter
    Py_Finalize();

    return 0;
}

Tomas Ukkonen

-----------------------9e29034e00ea96b7d10a477458ea4ed4--

-----BEGIN PGP SIGNATURE-----
Version: ProtonMail

wsBzBAEBCAAnBYJmiu86CZCaEYwMt4b97BYhBJbmzyegv3Wo8O5gpZoRjAy3 hv3sAAAGZQgAry3/hOkYMNAsE4FgfSHzeGmxt9xdeejo6X8TYT+6SqxtPxZw cIdIBxG2HodcJ625hz0Ouj0Ft+TZZOhwOrk6uPZUsya0Q1CoMfALl4jpaigh k2jB5alPivARIrTbg0jreVhrXd+HjSmIie0Y3j1e5Vg2v6u+z+hOH2JXhlyx +CFwCvBdKTERIy5HaT2CQNBBBpt/AOUjT2L4v333qOBqcu7BHihN+v5cMQe0 RyzOeA/W7yVpB9d89CedS78PnrCu86Hgky1zisWJ2TBw4gomZcF1hdJnwNuQ LrK3piMdNfCo2Ib08Hc7cVDy58/L7PdUXuUf8QTigw7m7QLWOpJKIQ==
=7W+t
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) -----------------------c50b1a0d029d02ab0c0c864f7ca0375d Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;charset=utf-8

Hello
Is this python c api bug? The following C++ code (g++) deadlocks on Ubuntu Linux.

/* * This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
 *
 * g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
 * ./a.out
 *
 * uname:
 * Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
 */

#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>

void perform_long_operation() {
    // Simulate a long-running task
    std::this_thread::sleep_for(std::chrono::seconds(5));
}

void thread_function() {
    // Ensure this thread has the GIL
    PyGILState_STATE gstate = PyGILState_Ensure();

    // Execute some Python code
    PyRun_SimpleString("print('Hello from std::thread!')");

    // Release the GIL for long operation
    Py_BEGIN_ALLOW_THREADS
    perform_long_operation();
    Py_END_ALLOW_THREADS

    // Re-acquire the GIL and execute more Python code
    gstate = PyGILState_Ensure();
    PyRun_SimpleString("print('Thread operation completed!')");

    // Release the GIL
    PyGILState_Release(gstate);
}

int main() {
    // Initialize the Python Interpreter
    Py_Initialize();

    // Create a vector of threads
    std::vector<std::thread> threads;

    // Launch threads
    for (int i = 0; i < 5; ++i) {
        threads.push_back(std::thread(thread_function));
    }

    // Join threads
    for (auto& t : threads) {
        t.join();
    }

    // Finalize the Python Interpreter
    Py_Finalize();

    return 0;
}



Tomas Ukkonen
-----------------------c50b1a0d029d02ab0c0c864f7ca0375d--

-----BEGIN PGP SIGNATURE-----
Version: ProtonMail

wsBzBAEBCAAnBYJmiutFCZCaEYwMt4b97BYhBJbmzyegv3Wo8O5gpZoRjAy3 hv3sAACH0wgAk6xX7goQyDG6b2zgFvqJ0HoSqH5qjNyXyKb8JrdWMS08dw53 aOwYhRvRRvNK17zv1RPp8G8ROjg9S3zjuURb4ChofqUZk8iHKDFG782JTdPG EE1XP8e/j0VNHwRJZyif/cKcu6JUnVlISv6xMKCvBxCC3dDmVEi2S3YQEC3T VJCw2XKpP1TEVXoDpnXgBIbHaGlm+EGCADIuZqXRI3idHnsPM+6tQVRX8R8y L3/7jooYtXx7UIoiX6t3o5TWc42lD/eCmWyDNHAxZHz+8KiolxCXi+IDy9RD Kpwwvj3gPUaQlk7HjTb7EkuoU9xl4V63QwfruvCtDX3gJak4+ac4sg==
=0jND
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 7 Jul 2024, at 22:09, Tomas Ukkonen via Python-list <[email protected]> wrote:

Py_Initialize();

You also need to tell python to init threading.
I think you are missing more python setup code before you can use threads.
Also i think you need to tell python that your thread wants to call into python.
But I an not near my dev system to research this for you.

I have code to use python from C++ in my pysvn project.
See the code starting a line 354 in https://sourceforge.net/p/pysvn/code/HEAD/tree/trunk/pysvn/Extension/Source/pysvn.cpp
That saves the thread state and restores it.

But in my case python creates the threads and I release and acquire the GIL.

Barry

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2024-07-07 20:40, Tomas Ukkonen via Python-list wrote:

Hi

There was a bug in the example code. I fixed it and it STILL deadlocks (my larger software project deadlocks when I call python from C++).

Updated code:

/* * This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
 *
 * g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
 * ./a.out
 *
 * uname:
 * Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
 */

#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>

void thread_function() {
    // Ensure this thread has the GIL
    PyGILState_STATE gstate = PyGILState_Ensure();

    // Execute some Python code
    PyRun_SimpleString("print('Hello from std::thread!')");

    // Release the GIL
    PyGILState_Release(gstate);
}

int main() {
    // Initialize the Python Interpreter
    Py_Initialize();

    // Create a vector of threads
    std::vector<std::thread> threads;

    // Launch threads
    for (int i = 0; i < 5; ++i) {
        threads.push_back(std::thread(thread_function));
    }

    // Join threads
    for (auto& t : threads) {
        t.join();
    }

    // Finalize the Python Interpreter
    Py_Finalize();

    return 0;
}
sunnuntaina 7. heinäkuuta 2024 klo 10:24 ip, Tomas Ukkonen <[email protected]> kirjoitti:

Hello
Is this python c api bug? The following C++ code (g++) deadlocks on Ubuntu Linux.

/* * This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
 *
 * g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
 * ./a.out
 *
 * uname:
 * Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
 */

#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>

void perform_long_operation() {
    // Simulate a long-running task
    std::this_thread::sleep_for(std::chrono::seconds(5));
}

void thread_function() {
    // Ensure this thread has the GIL
    PyGILState_STATE gstate = PyGILState_Ensure();

    // Execute some Python code
    PyRun_SimpleString("print('Hello from std::thread!')");

    // Release the GIL for long operation
    Py_BEGIN_ALLOW_THREADS
    perform_long_operation();
    Py_END_ALLOW_THREADS

    // Re-acquire the GIL and execute more Python code
    gstate = PyGILState_Ensure();
    PyRun_SimpleString("print('Thread operation completed!')");

    // Release the GIL
    PyGILState_Release(gstate);
}

int main() {
    // Initialize the Python Interpreter
    Py_Initialize();

At this point, there's only one thread (the main thread) and it owns the
GIL.

    // Create a vector of threads
    std::vector<std::thread> threads;

    // Launch threads
    for (int i = 0; i < 5; ++i) {
        threads.push_back(std::thread(thread_function));
    }

The threads will each try to acquire and release the GIL, but it's still
owned by the main thread.

    // Join threads
    for (auto& t : threads) {
        t.join();
    }

The main thread is waiting for the sub-threads to finish, and the
threads waiting for the GIL, but the main thread still owns the GIL, so
they'll be waiting forever. Deadlock.

    // Finalize the Python Interpreter
    Py_Finalize();

    return 0;
}

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 7 Jul 2024, at 23:21, Barry via Python-list <[email protected]> wrote:

On 7 Jul 2024, at 22:09, Tomas Ukkonen via Python-list <[email protected]> wrote:

Py_Initialize();

You also need to tell python to init threading.

I'm in front of my dev machine now and checking up on threading.

There is no longer any extra init for threads required.

I think you are missing more python setup code before you can use threads. Also i think you need to tell python that your thread wants to call into python.
But I an not near my dev system to research this for you.

You are right to use PyGILState_Ensure()

But as MRAB says the main thread is holding the GIL.

I have code to use python from C++ in my pysvn project.
See the code starting a line 354 in https://sourceforge.net/p/pysvn/code/HEAD/tree/trunk/pysvn/Extension/Source/pysvn.cpp
That saves the thread state and restores it.

You still might find the classes I wrong to manage GIL acquire and release interesting.
I have the C++ type system enforcing the rules of acquire and release.
As well as RAII ensuring never to leave a block with the GIL in the wrong state.

Barry

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)