Re: Python recompile

bart <bc@freeuk.com> wrote:

On 06/03/2025 19:21, Waldek Hebisch wrote:

In comp.lang.c Muttley@dastardlyhq.org wrote:

On Tue, 4 Mar 2025 18:16:25 +0000
bart <bc@freeuk.com> wibbled:

The CPython source bundle doesn't come with any makefiles. The first
step appears to be to run a 35,000-line 'configure' script. Part of its

>
Frankly any build system that has a 35K configure file needs revisiting.
No package is so complex as to require that much setup. OS specific code
should be dealt with some appropriate ifdefs in the source and libraries
and linking should be a few 10s of lines.
>
Back in the day packages used to hve different preprepared Makefiles for
each system they could build on and IME that tended to work a lot better
than configure scripts that tried to figure things out on the fly.

 
I remember times when source package came with README file saying
edit this and that to configure for your system.  Typically one
had to edit Makefile and possibly something like config.h.  For
me it worked quite well.  But those times are gone.  Most people
now do not know essential information about system they use, so
are unable to provide sensible values.  And modern programs are
bigger and more complicated, so such hand editing is too much work
for rare people capable of doing this.
 
Per platform Makefile-s do not scale when one wants to support
multiple system and multiple configurations (there is exponential
growth of possible combinations).  And even of single configuration
for supposedly single system like Linux there are troubles.
In one project there was someting like 20 Makefile.linux_x files
where x represented one Linux flavour.  Yet regularly somebody
would come and say: "build fails on my Linux x.y.z".  If enough
information was provided new Makefile was added, or possibly
some similar Makefile was modified to cover more cases.
Those troubles mostly vanished when project switched to configure
script.  Now there is about 1500 lines of hand written code
in configure machinery (biggest is 933 lines long configure.ac)
and 8564 lines long generated configure script.  Most build
troubles is now gone or diagnosed early.
 
Writing "diagnosed early" I mean that most user errors are
detected quite early.  For example, a lot of people try to
compile program without having a C compiler.  And when they
have compiler they fail to install needed libraries (there
is list which says what is needed but people fail to follow
instructions).
 
It is certainly silly when generated configure file is much
bigger than actual program code.  This happens frequently
when lazy or brainwashed folks use automake (which apparently
puts all checks that it knows into generated configure.ac).
But Python have reasons to check for a lot of things, so
the size is somewhat reasonable.

 
So, why doesn't the configuration script itself have a configuration
script? Will the language used in the script run on EVERY Linux and Unix
system?

Depends on meaning of EVERY, Linux and Unix.  If you mean just
Linux kernel, then it was possible to set up Linux kernel +
system without normal shell.  Since there is no shell, shell
script will not run.  IIUC first version of Unix from Bell Labs
had significantly different shell, incompatible with modern ones.
But if we take normal Linux distributions or for example
Android or Unices that are not long obsolete then mostly yes.

I wrote mostly because shell code generated by Autoconf macros
is very portable.  There is good chance that what automake
produces is very portable too.  However, when using hand
written configure.ac there is hand written shell code inside.
In the case I mention above it is probably about 400-500 lines
of hand written shell code.  Such hand written code is as portable
as its author make it.  Which sometimes means quite unportable.
In this case all I can say that nobody reported any trouble
with shell code.

If so, why doesn't the same apply to the main language of the
application? (Why isn't the application written in that language!)

You like benchmarks, so try some benchmarks of speed of shell
programs.  Or believe me that for computational problems shell
programs are dog slow (of order of 100 times slower than normal
interpreted languages like Python).

Portable shell programs may be rather verbose and ugly (I hope
you agree that typical configure script are verbose and
and ugly).

Despite of the above, historically there were applications
written as shell programs.  In open source world they
were not frequent but existed.

I've also seen a project where the configuration program was a C file,
one which didn't need any special options, whose output was used to
build the main app.

Yes, that is a possiblity.  Some time ago I was thinking about
similar thing.  Basically, configure scripts are slow, ugly and
long due to limitations of shell and need to work with "any"
shell.  Could we write resonably small "better shell" to
execute simpler version of configure script?  I am not sure
if this would work well, basically pre-computations are easy
to do in C code, but typical configure must do several tests
and this means using various system interfaces.  System
interfaces vary from system to system and detecting
available system interfaces is part of configure job.
One could try to depend on Posix interfaces, they are
available on many systems.  But shell is part of Posix
and may be availble on systems which do not provide
Posix interface at C level.  So it is not clear if C based
configuration program would simplify things.

Again, why can't the main app be made to work like the small one?

Modern applications depend on libraries.  Let me mention a dll
for Windows that I created many years ago.  My own code was
about 2000 (wc) lines of code, part of it numeric code that
provided main functionality (image transformation), rest was
glue code to the libraries.  Input data came from .tiff files
and I used libtiff to read them.  I needed some geometry code
which I found on the net.  I needed linear equation solver,
that I got from Lapack.  So there were 4 libraries (3 that
I used directly and libjpeg used by libtiff), and most
code in "my" dll actually came from the other libraries.

In this dll was doing rather special thing used as a part
of bigger process.  In modern times application like gimp
connects several things of similar complexity.  There is
a lot of image formats, so many libraries just to read and
write graphic files.  I did not look at gimp build process,
but at configure stage is must verify if all needed libraries
(probably tens if not more) are present. Libraries need to be
in sufficiently new versions and export needed functionality
(libraries frequently may be compiled skipping part of
functionality).  I am pretty sure that parts of gimp are
optional, that is you may include or skip them.  And some
other projects may wish to use parts of gimp, so build
probaly created quite a few of libraries (that is separate
things that need to be linked).

In modern times applications are supposed to be internationalized.
That is various messages should appear in selected language.
Normal practise in modern times is to have separate build
step to expract english messages from source files and
create connetion to translations.  That alone means that
build is more complex than simply compiling C files.

Similarly, usefulness of applications depend on good
documentation.  When application evolves documentaion
should be updated to show new results from examples.
Good documentaion build will extract examples from
documentation, run them trough program and then merge
back results.  So this is dependent on C compilation,
but contains quite a different steps.

I've played with projects that had nearly 20 different makefiles, none
of which was right for my own compiler. It is a ridiculous situation.

Yes, there is trouble with multiple Makefiles, change something
and you need next one.  configure scripts tend to be better
here, as long as fundamental assumptions hold they can
adapt.  But if say you refuse to use shell, or compiler can
not produce object files (but only executables), then there
are troubles: simply such change is much bigger than expected.

--
                              Waldek Hebisch