Re: Threads across programming languages

ram@zedat.fu-berlin.de (Stefan Ram) wrote or quoted:

The GIL only prevents multiple Python statements from being
interpreted simultaneously, but if you're waiting on inputs (like
sockets), it's not active, so that could be distributed across
multiple cores.

  Disclaimer: This is not on-topic here as it discusses Python,
  not C or C++.

  FWIW, here's some multithreaded Python code modeled after what
  I use in an application.

  I am using Python to prepare a press review for me, getting article
  headers from several newssites, removing all headers matching a list
  of regexps, and integrating everything into a single HTML resource.
  (I do not like to read about Lindsay Lohan, for example, so articles
  with the text "Lindsay Lohan" will not show up on my HTML review.)

  I'm usually downloading all pages at once using Python threads,
  which will make sure that a thread uses the CPU while another
  thread is waiting for TCP/IP data. This is the code, taken from
  my Python program and a bit simplified:

from multiprocessing.dummy import Pool

...

with Pool( 9 if fast_internet else 1 )as pool:
    for i in range( 9 ):
        content[ i ] = pool.apply_async( fetch,[ uris[ i ] ])
    pool.close()
    pool.join()

  . I'm using my "fetch" function to fetch a single URI, and the
  loop starts nine threads within a thread pool to fetch the
  content of those nine URIs "in parallel". This is observably
  faster than corresponding sequential code.

  (However, sometimes I have a slow connection and have to download
  sequentially in order not to overload the slow connection, which
  would result in stalled downloads. To accomplish this, I just
  change the "9" to "1" in the first line above.)

  In case you wonder about the "dummy":

|The multiprocessing.dummy module module provides a wrapper
|for the multiprocessing module, except implemented using
|thread-based concurrency.
|
|It provides a drop-in replacement for multiprocessing,
|allowing a program that uses the multiprocessing API to
|switch to threads with a single change to import statements.

  . So, this is an area where multithreading the Python way is easy
  to use and enhances performance even in the presence of the GIL!