Re: Threads across programming languages

On Fri, 3 May 2024 10:34:11 +0200
David Brown <david.brown@hesbynett.no> wrote:

On 03/05/2024 01:58, Chris M. Thomasson wrote:

On 5/2/2024 4:15 PM, Lawrence D'Oliveiro wrote: 

On Thu, 2 May 2024 13:28:15 -0700, Chris M. Thomasson wrote:
 

On 5/1/2024 10:39 PM, Lawrence D'Oliveiro wrote: 

>
On Wed, 1 May 2024 22:20:47 -0700, Chris M. Thomasson wrote:
 

On 5/1/2024 1:34 PM, Lawrence D'Oliveiro wrote:
 

Remember, we’re talking about maximizing I/O throughput here,
so CPU is not the bottleneck. 

>
It can be if your thread synchronization scheme is sub par. 

>
Another reason to avoid threads. 

>
Why? Believe it or not, there are ways to create _highly_ scalable
thread synchronization schemes. 

>
I’m sure there are. But none of that is relevant when the CPU
isn’t the bottleneck anyway. 

 
The CPU can become a bottleneck. Depends on how the programmer
implements things.
 
 

So long as your async tasks have an await call somewhere in
their main loops, that should be sufficient to avoid most
bottlenecks. 

>
async tasks are using threads... No? 

>
No. They are built on coroutines. Specifically, the “stackless”
variety.
>
<https://gitlab.com/ldo/python_topics_notebooks/-/blob/master/Generators%20&%20Coroutines.ipynb?ref_type=heads>
 

 
So, there is no way to take advantage of multiple threads on
Python? Heck, even JavaScript has WebWorkers... ;^) 

 
Python supports multi-threading.  It uses a global lock (the "GIL")
in the Python interpreter - thus only one thread can be running
Python code at a time.  However, if you are doing anything serious
with Python, much of the time will be spend either blocked (waiting
for network, IO, etc.) or using compiled or external code (using your
favourite gui toolkit, doing maths with numpy, etc.).  The GIL is
released while executing such code.
 
Thus if you are using Python for cpu-intensive work (and doing so
sensibly), you have full multi-threading.  If you are using it for
IO-intensive work, you have full multi-threading.  It's not going to
be as efficient as well-written compiled code, even with JIT and
pypy, but in practice it gets pretty close while being very
convenient and developer friendly.
 
If you really need parallel running of Python code, or better
separation between tasks, Python has a multi-processing module that
makes it simple to control and pass data between separate Python
processes, each with their own GIL.
 
 
 

A typical scenario is that you started you python program while
thinking that it wouldn't e CPU-intensive. And then it grew and became
CPU-intensive.
That's actually a good case, because it means that your program is used
and is doing something worthwhile.