Re: Python recompile

On 07/03/2025 01:07, Waldek Hebisch wrote:

bart <bc@freeuk.com> wrote:

On 06/03/2025 19:21, Waldek Hebisch wrote:

In comp.lang.c Muttley@dastardlyhq.org wrote:

On Tue, 4 Mar 2025 18:16:25 +0000
bart <bc@freeuk.com> wibbled:

The CPython source bundle doesn't come with any makefiles. The first
step appears to be to run a 35,000-line 'configure' script. Part of its

>
Frankly any build system that has a 35K configure file needs revisiting.
No package is so complex as to require that much setup. OS specific code
should be dealt with some appropriate ifdefs in the source and libraries
and linking should be a few 10s of lines.
>
Back in the day packages used to hve different preprepared Makefiles for
each system they could build on and IME that tended to work a lot better
than configure scripts that tried to figure things out on the fly.

>
I remember times when source package came with README file saying
edit this and that to configure for your system.  Typically one
had to edit Makefile and possibly something like config.h.  For
me it worked quite well.  But those times are gone.  Most people
now do not know essential information about system they use, so
are unable to provide sensible values.  And modern programs are
bigger and more complicated, so such hand editing is too much work
for rare people capable of doing this.
>
Per platform Makefile-s do not scale when one wants to support
multiple system and multiple configurations (there is exponential
growth of possible combinations).  And even of single configuration
for supposedly single system like Linux there are troubles.
In one project there was someting like 20 Makefile.linux_x files
where x represented one Linux flavour.  Yet regularly somebody
would come and say: "build fails on my Linux x.y.z".  If enough
information was provided new Makefile was added, or possibly
some similar Makefile was modified to cover more cases.
Those troubles mostly vanished when project switched to configure
script.  Now there is about 1500 lines of hand written code
in configure machinery (biggest is 933 lines long configure.ac)
and 8564 lines long generated configure script.  Most build
troubles is now gone or diagnosed early.
>
Writing "diagnosed early" I mean that most user errors are
detected quite early.  For example, a lot of people try to
compile program without having a C compiler.  And when they
have compiler they fail to install needed libraries (there
is list which says what is needed but people fail to follow
instructions).
>
It is certainly silly when generated configure file is much
bigger than actual program code.  This happens frequently
when lazy or brainwashed folks use automake (which apparently
puts all checks that it knows into generated configure.ac).
But Python have reasons to check for a lot of things, so
the size is somewhat reasonable.

>
So, why doesn't the configuration script itself have a configuration
script? Will the language used in the script run on EVERY Linux and Unix
system?

 Depends on meaning of EVERY, Linux and Unix.  If you mean just
Linux kernel, then it was possible to set up Linux kernel +
system without normal shell.  Since there is no shell, shell
script will not run.  IIUC first version of Unix from Bell Labs
had significantly different shell, incompatible with modern ones.
But if we take normal Linux distributions or for example
Android or Unices that are not long obsolete then mostly yes.
 I wrote mostly because shell code generated by Autoconf macros
is very portable.  There is good chance that what automake
produces is very portable too.  However, when using hand
written configure.ac there is hand written shell code inside.
In the case I mention above it is probably about 400-500 lines
of hand written shell code.  Such hand written code is as portable
as its author make it.  Which sometimes means quite unportable.
In this case all I can say that nobody reported any trouble
with shell code.
 

If so, why doesn't the same apply to the main language of the
application? (Why isn't the application written in that language!)

 You like benchmarks, so try some benchmarks of speed of shell
programs.  Or believe me that for computational problems shell
programs are dog slow (of order of 100 times slower than normal
interpreted languages like Python).

That wasn't a serious suggestion. But why not have a language that can run anywhere? A compiler is a program that reads in some files and writes one or more output files - you can't get anything more portable!

Portable shell programs may be rather verbose and ugly (I hope
you agree that typical configure script are verbose and
and ugly).
 Despite of the above, historically there were applications
written as shell programs.  In open source world they
were not frequent but existed.
 

I've also seen a project where the configuration program was a C file,
one which didn't need any special options, whose output was used to
build the main app.

 Yes, that is a possiblity.  Some time ago I was thinking about
similar thing.  Basically, configure scripts are slow, ugly and
long due to limitations of shell and need to work with "any"
shell.  Could we write resonably small "better shell" to
execute simpler version of configure script?

You'd still have to sell me on the need for a configuration script at all.
I accept that sometimes scripting is needed to orchestrate certain builds or to perform installation, but such programs would be 2-3 magnitudes smaller than that 35Kloc monstrosity.

I am not sure
if this would work well, basically pre-computations are easy
to do in C code, but typical configure must do several tests
and this means using various system interfaces.  System
interfaces vary from system to system and detecting
available system interfaces is part of configure job.
One could try to depend on Posix interfaces, they are
available on many systems.  But shell is part of Posix
and may be availble on systems which do not provide
Posix interface at C level.  So it is not clear if C based
configuration program would simplify things.

I mentioned my transpiler producing C code that is for either Windows or Linux. At one time I also supported a neutral OS, where exactly the same code compiled on either (or any) OS.
While there were limitations (for example my interpreter needs dlsym() from Linux or GetProcAddress() from Windows, here there was a workaround), enough would work to run a project like a compiler.

Again, why can't the main app be made to work like the small one?

 Modern applications depend on libraries.  Let me mention a dll
for Windows that I created many years ago.  My own code was
about 2000 (wc) lines of code, part of it numeric code that
provided main functionality (image transformation), rest was
glue code to the libraries.  Input data came from .tiff files
and I used libtiff to read them.  I needed some geometry code
which I found on the net.  I needed linear equation solver,
that I got from Lapack.  So there were 4 libraries (3 that
I used directly and libjpeg used by libtiff), and most
code in "my" dll actually came from the other libraries.
 In this dll was doing rather special thing used as a part
of bigger process.  In modern times application like gimp
connects several things of similar complexity.  There is
a lot of image formats, so many libraries just to read and
write graphic files.  I did not look at gimp build process,
but at configure stage is must verify if all needed libraries
(probably tens if not more) are present. Libraries need to be
in sufficiently new versions and export needed functionality
(libraries frequently may be compiled skipping part of
functionality).  I am pretty sure that parts of gimp are
optional, that is you may include or skip them.  And some
other projects may wish to use parts of gimp, so build
probaly created quite a few of libraries (that is separate
things that need to be linked).

Some DLLS will be part of the OS. Some are easy to obtain by the user. Some could be bundled.
However my apps (those mentioned below) would not fail if one was missing; only if specific functions from a DLL were needed. (I think I only needed one, to do with JPEG handling).
If I was still making apps, I would not have a dependency of an elusive library. One such is GMPxxx.DLL; that was so difficult to source, or even to build from source code (yes it has its own 30Kloc configure file!) that I found it easier to create my own library.
Another hard one I've bypassed is LIBFFI, with my own solution.

In modern times applications are supposed to be internationalized.
That is various messages should appear in selected language.
Normal practise in modern times is to have separate build
step to expract english messages from source files and
create connetion to translations.  That alone means that
build is more complex than simply compiling C files.

Actually my 1990s apps were internationalised (working in English, French, German, Dutch). A dictionary of translations was provided by distributors. There was some scripting to help with that, but the script language was built-in to the app.
It was all taken care of. No need for makefiles or any stuff like that. Supporting a different language didn't need a new build; just a data file.