Re: Is Parallel Programming Hard, And, If So, What Can You Do About It?

mitchalsup@aol.com (MitchAlsup1) writes:

On Tue, 20 May 2025 20:06:13 +0000, Stefan Monnier wrote:

FWIW, I think these kinds of things usually fall in the scope of
concurrency rather than parallelism.

>
When I run 20-copies of a FEM CFD application, each uni-process::
am I running concurrently ?? or in parallel ?? or both ??

That's an example of an embarrasingly parallel problem, and you can
run it on a machine with 20 processors or 20 machines with 1 processor
each without needing any communication between the processes.

Concurrency is concerned with the coordination of simultaneous
processes (or threads) that need to communicate, even if it is just to
coordinate the access to a shared resource.  The classical example is
how to coordinate the simulteous access to a bank account.  If the
account contains EUR 100 (and allows no overdraft), and two people
want to withdraw EUR 100 from it at the same time, what happens?

Concurrency is an issue already when only a single core is involved
(if the account is checked before the withdrawal happens, and there
can be a task switch to the other withdrawal, the account can be
overdrafted; this is a classic race condition), but becomes harder
when parallel hardware is involved.

I have not read the book, but it seems that a more appropriate title
would be "Is Concurrent Programming Hard ...".

Parallel computing, OTOH seems to be more concerned with building
computers with many processors that execute HPC applications quickly,
and programming the HPC applications such that they make good use of
these computers.

That leaves the question open what HPC applications are.  It's
somewhat cyclic: Those that can make good use of parallel
(super-)computers.  The non-cyclical part is that those are
applications that are important enough to someone for financing such
supercomputers.  The example of CFP (computational fluid dynamics) is
maybe _the_ original HPC application: the nuclear weapons laboratories
in the USA were the first customers of supercomputers and parallel
computers.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
  Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>