Re: Threads across programming languages

On 5/3/2024 8:47 PM, Chris M. Thomasson wrote:

On 5/3/2024 8:44 PM, Chris M. Thomasson wrote:

On 4/30/2024 2:04 AM, Stefan Ram wrote:

ram@zedat.fu-berlin.de (Stefan Ram) wrote or quoted:

The GIL only prevents multiple Python statements from being
interpreted simultaneously, but if you're waiting on inputs (like
sockets), it's not active, so that could be distributed across
multiple cores.

>
   Disclaimer: This is not on-topic here as it discusses Python,
   not C or C++.
>
   FWIW, here's some multithreaded Python code modeled after what
   I use in an application.
>
   I am using Python to prepare a press review for me, getting article
   headers from several newssites, removing all headers matching a list
   of regexps, and integrating everything into a single HTML resource.
   (I do not like to read about Lindsay Lohan, for example, so articles
   with the text "Lindsay Lohan" will not show up on my HTML review.)
>
   I'm usually downloading all pages at once using Python threads,
   which will make sure that a thread uses the CPU while another
   thread is waiting for TCP/IP data. This is the code, taken from
   my Python program and a bit simplified:
>
from multiprocessing.dummy import Pool
>
...
>
with Pool( 9 if fast_internet else 1 )as pool:
     for i in range( 9 ):
         content[ i ] = pool.apply_async( fetch,[ uris[ i ] ])
     pool.close()
     pool.join()
>
   . I'm using my "fetch" function to fetch a single URI, and the
   loop starts nine threads within a thread pool to fetch the
   content of those nine URIs "in parallel". This is observably
   faster than corresponding sequential code.
>
   (However, sometimes I have a slow connection and have to download
   sequentially in order not to overload the slow connection, which
   would result in stalled downloads. To accomplish this, I just
   change the "9" to "1" in the first line above.)
>
   In case you wonder about the "dummy":
>
|The multiprocessing.dummy module module provides a wrapper
|for the multiprocessing module, except implemented using
|thread-based concurrency.
|
|It provides a drop-in replacement for multiprocessing,
|allowing a program that uses the multiprocessing API to
|switch to threads with a single change to import statements.
>
   . So, this is an area where multithreading the Python way is easy
   to use and enhances performance even in the presence of the GIL!

>
Agreed. However, its a very small sample. Try to download 60,000 files concurrently from different sources all at once. This can be where the single lock messes with performance...

 Certain sources are faster than others. That's always fun... Think of timeout logic... ;^D

Create a death client, that creates a shit load of connections that do nothing. Some of them barely transmit data, some are very fast.... Some are both, then turn into others... Send very fast, then quite for a while, then send hyper fast again...