Hi all,

I've been working on embedding Python and have an interesting case around locking with PyEval_RestoreThread which wasn't quite doing what I expect, hoping someone can explain what I should expect here.

I have a little example (I'm running this in parallel from two different threads; I have some more C code for that but I don't think it's super interesting):

void run_python(PyThreadState* thread) {
LOG("Restoring thread %p...", thread);
PyEval_RestoreThread(thread);
LOG("Restored thread %p", thread);
PyRun_SimpleString("import time; print('sleeping'); time.sleep(3.0)");
LOG("Saving thread...");
PyThreadState* saved_thread = PyEval_SaveThread();
LOG("Saved thread %p", saved_thread);
}

This produces output like
11:46:48.110058893: Restoring thread 0xabc480...
11:46:48.110121656: Restored thread 0xabc480
11:46:48.110166060: Restoring thread 0xabc480...
sleeping
11:46:48.110464194: Restored thread 0xabc480
sleeping
11:46:51.111307541: Saving thread...
11:46:51.111361075: Saved thread 0xabc480
11:46:51.113116633: Saving thread...
11:46:51.113177605: Saved thread 0xabc480

The thing that surprises me is that both threads seem to be able to pass PyEval_RestoreThread before either reaches the corresponding
PyEval_SaveThread call, which I wasn't expecting to happen; I assumed that since RestoreThread acquires the GIL, that thread state would remain locked until it's released.

I understand that the system occasionally switches threads, which I guess
might well happen with that time.sleep() call, but I wasn't expecting the
same thread to become usable somewhere else. Maybe I am just confusing
things by approaching the same Python thread from multiple OS threads concurrently and should be managing my own locking around that?

Thanks in advance,

Peter

--
Thought Machine Group Limited, a company registered in England & Wales. Registered number: 11114277. 
Registered Office: 5 New Street Square,
London EC4A 3TW <https://maps.google.com/?q=5+New+Street+Square,+London+EC4A+3TW&entry=gmail&source=g>.


The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of
this message with any third party, without a written consent of the sender.
If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not
occur in the future.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2023-09-26 14:20, Peter Ebden via Python-list wrote:

Hi all,

I've been working on embedding Python and have an interesting case around locking with PyEval_RestoreThread which wasn't quite doing what I expect, hoping someone can explain what I should expect here.

I have a little example (I'm running this in parallel from two different threads; I have some more C code for that but I don't think it's super interesting):

void run_python(PyThreadState* thread) {
LOG("Restoring thread %p...", thread);
PyEval_RestoreThread(thread);
LOG("Restored thread %p", thread);
PyRun_SimpleString("import time; print('sleeping'); time.sleep(3.0)");
LOG("Saving thread...");
PyThreadState* saved_thread = PyEval_SaveThread();
LOG("Saved thread %p", saved_thread);
}

This produces output like
11:46:48.110058893: Restoring thread 0xabc480...
11:46:48.110121656: Restored thread 0xabc480
11:46:48.110166060: Restoring thread 0xabc480...
sleeping
11:46:48.110464194: Restored thread 0xabc480
sleeping
11:46:51.111307541: Saving thread...
11:46:51.111361075: Saved thread 0xabc480
11:46:51.113116633: Saving thread...
11:46:51.113177605: Saved thread 0xabc480

The thing that surprises me is that both threads seem to be able to pass PyEval_RestoreThread before either reaches the corresponding PyEval_SaveThread call, which I wasn't expecting to happen; I assumed that since RestoreThread acquires the GIL, that thread state would remain locked until it's released.

I understand that the system occasionally switches threads, which I guess might well happen with that time.sleep() call, but I wasn't expecting the same thread to become usable somewhere else. Maybe I am just confusing
things by approaching the same Python thread from multiple OS threads concurrently and should be managing my own locking around that?

Storing the result of PyEval_SaveThread in a local variable looks wrong
to me.

In the source for the regex module, I release the GIL with
PyEval_SaveThread and save its result. Then, when I want to claim the
GIL, I pass that saved value to PyEval_RestoreThread.

You seem to be releasing the GIL and discarding the result, so which
thread are you resuming when you call PyEval_RestoreThread?

It looks like you're resuming the same thread twice. As it's already
resumed the second time, no wonder it's not blocking!

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

+1 843 620 1451 Hash Online Shop Europe.

+1 843 620 1451
[email protected]
Los Angeles , California (USA)

Hashish is produced from the resin glands (trichomes) of the cannabis plant. These trichomes are collected, processed, and compacted into a solid form, creating hash. The result is a highly potent substance known for its unique aroma, flavor, and effects.

Why Buy Hash Online?
Purchasing hash online provides several advantages. It offers convenience, a wider selection, and access to reputable suppliers. Moreover, online dispensaries often provide detailed information about their products, allowing you to make informed decision

Legality of Hashish in Europe
Before you buy hash online, it's crucial to be aware of the legal status of hashish in your European country. Laws regarding cannabis products vary, and staying compliant with local regulations is essential.

Traditional Hash
This is the classic form of hash, typically found in brick or slab form. It has a distinctive aroma and a smooth, earthy flavor.

Bubble Hash
Bubble hash is created using ice water and agitation to separate the trichomes from the plant material. It's known for its purity and potency.

Afghan Hash
Afghan hash is renowned for its strong, sedative effects and earthy taste. It's often dark and sticky in texture.

Moroccan Hash
Moroccan hash is known for its sweet, spicy flavor and a mellow, relaxing high. It's usually light in color and has a crumbly texture.

How to Order Hash Online
Ordering hashish from HashandCannabis.com is a straightforward process. Simply browse their selection, add your desired products to the cart, and follow the checkout instructions. They accept various payment methods, making it convenient for customers
worldwide.

https://hashandcannabis.com/product-category/buy-cannabis-oil-online/ https://hashandcannabis.com/product-category/buy-vape-cartridges-online/ https://hashandcannabis.com/product-category/buy-concentrates-online/ https://hashandcannabis.com/product-category/buy-hashish-online/ https://hashandcannabis.com/product-category/buy-hybrid-strains-online/ https://hashandcannabis.com/product-category/buy-indica-strains-online/ https://hashandcannabis.com/product-category/buy-moon-rock-eu-online/ https://hashandcannabis.com/product-category/buy-sativa-strains-online/ https://hashandcannabis.com/product-category/buy-shatter-wax-online/

Hash Culture in Europe
Hash has a long-standing cultural significance in Europe, and it continues to be enjoyed by people from all walks of life. Learning about its cultural context can deepen your appreciation for this substance.

Buying hash online in Europe can be a rewarding experience when approached with knowledge and caution. Remember to research the legalities in your region, choose a reputable seller, and consume responsibly. With these guidelines, you can enjoy the world
of hash safely and responsibly.

buy cheap hash online uk +1 843 620 1451
deep purple cannabis +1 843 620 1451
hash online shop +1 843 620 1451
Hash UK Delivery +1 843 620 1451
how to buy hash +1 843 620 1451
buy real hash online +1 843 620 1451
hash dispensary +1 843 620 1451
hash deep purple +1 843 620 1451
deep purple hash +1 843 620 1451
Hash Online Shop Europe +1 843 620 1451

+1 843 620 1451
[email protected]
Los Angeles , California (USA)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

The thread variable I'm passing in is the one I originally got from calling Py_NewInterpreter. I'd assumed that I didn't need to particularly track the
one I get back from SaveThread since it should always be the one I restored previously (which does seem to be the case).

It looks like you're resuming the same thread twice. As it's already

resumed the second time, no wonder it's not blocking!

That isn't how I read the docs though? It says "If the lock has been
created, the current thread must not have acquired it, otherwise deadlock ensues." That suggests to me that it should try to acquire the GIL again
and wait until it can (although possibly also that it's not an expected use
and Python thread states are expected to be more 1:1 with C threads).

On Wed, Sep 27, 2023 at 3:53 AM MRAB via Python-list <[email protected]> wrote:

On 2023-09-26 14:20, Peter Ebden via Python-list wrote:

Hi all,

I've been working on embedding Python and have an interesting case around locking with PyEval_RestoreThread which wasn't quite doing what I expect, hoping someone can explain what I should expect here.

I have a little example (I'm running this in parallel from two different threads; I have some more C code for that but I don't think it's super interesting):

void run_python(PyThreadState* thread) {
LOG("Restoring thread %p...", thread);
PyEval_RestoreThread(thread);
LOG("Restored thread %p", thread);
PyRun_SimpleString("import time; print('sleeping'); time.sleep(3.0)");
LOG("Saving thread...");
PyThreadState* saved_thread = PyEval_SaveThread();
LOG("Saved thread %p", saved_thread);
}

This produces output like
11:46:48.110058893: Restoring thread 0xabc480...
11:46:48.110121656: Restored thread 0xabc480
11:46:48.110166060: Restoring thread 0xabc480...
sleeping
11:46:48.110464194: Restored thread 0xabc480
sleeping
11:46:51.111307541: Saving thread...
11:46:51.111361075: Saved thread 0xabc480
11:46:51.113116633: Saving thread...
11:46:51.113177605: Saved thread 0xabc480

The thing that surprises me is that both threads seem to be able to pass PyEval_RestoreThread before either reaches the corresponding PyEval_SaveThread call, which I wasn't expecting to happen; I assumed

that

since RestoreThread acquires the GIL, that thread state would remain

locked

until it's released.

I understand that the system occasionally switches threads, which I guess might well happen with that time.sleep() call, but I wasn't expecting the same thread to become usable somewhere else. Maybe I am just confusing things by approaching the same Python thread from multiple OS threads concurrently and should be managing my own locking around that?

Storing the result of PyEval_SaveThread in a local variable looks wrong
to me.

In the source for the regex module, I release the GIL with
PyEval_SaveThread and save its result. Then, when I want to claim the
GIL, I pass that saved value to PyEval_RestoreThread.

You seem to be releasing the GIL and discarding the result, so which
thread are you resuming when you call PyEval_RestoreThread?

It looks like you're resuming the same thread twice. As it's already
resumed the second time, no wonder it's not blocking!

--
https://mail.python.org/mailman/listinfo/python-list

--
Thought Machine Group Limited, a company registered in England & Wales. Registered number: 11114277. 
Registered Office: 5 New Street Square,
London EC4A 3TW <https://maps.google.com/?q=5+New+Street+Square,+London+EC4A+3TW&entry=gmail&source=g>.


The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of
this message with any third party, without a written consent of the sender.
If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not
occur in the future.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)