Re: question about linker

On 06/12/2024 23:26, Scott Lurndal wrote:

David Brown <david.brown@hesbynett.no> writes:

On 04/12/2024 23:48, Scott Lurndal wrote:

David Brown <david.brown@hesbynett.no> writes:

On 04/12/2024 17:55, Scott Lurndal wrote:

>

13 minutes by the wall clock, a bit over an hour
of CPU time.
>

>
Is that a "typical" build - or an atypical full clean build?
>

>
That was a build that touched a key header file, so maybe
85% full.  A full build adds a minute or two wall time.
>
A single source file build (for most source files) takes
about 28 seconds real, a large portion of that in make as it
figures out what to rebuild in a project with thousands
of source files and associated dependencies.

>
That sounds like your dependency handling in make is not ideal.  Store
your dependencies in individual files per source file, where both the
object file and the dependency file itself are dependent on the headers
used by the C file.  (The makefile should also have both the object file
and the dependency file dependent on the C source file.)

 We use the standard makedepend support built into GCC.  It creates
a .d file for each source file that contains the dependencies
for that source file.

This is part of the makefile macro I use when generating dependency files:
        $(CCDEP) $(combined_cflags) $(2) $(3) -MM -MP \
                       -MT $(obj_dir)$(1)$$(@F:%.d=%.o) \
                       -MT $(dep_dir)$(1)$$(@F:%.o=%.o) \
                       -MF $$@ $$<
The point is that because the dependency files themselves are also targets in these dependency files, they don't need to be re-build unnecessarily.

 Scale matters, (with two thousand+ source and header files, that's a lot
of 'access', 'read' and 'stat' system calls just to build the dependency
graph.   On NFS filesystems, which we use in most environments, the make
takes even more time.
 

If the issue is stat'ing the times for those files in order to check the dependencies, then I understand the bottleneck.  But I would not expect people to be sharing a source tree on NFS like that - it is certainly not the usual way of collaborating on projects.
Certainly it all sounds like you are doing something seriously inefficient.  You've only got ten times the number of files that I have, but massively slower update builds.  On the other hand, it might simply be that the way your headers are used, and the type of C++ code you have (it is mainly C++, is it not?) mean that individual file compiles are much more demanding than on my system.

Thus your build only generates new dependency files when either the C
source file changes, or any of the headers included (recursively) by
that C file change.  Change a C file, and it's dependencies will be
re-calculated - but not the dependencies for anything else.  Change a
header, and you only need to re-calculate the dependencies for the
source files that actually use it.
>

 In the case above, over 1000 source files were dependent upon that
key project-wide header file which had been modified.

Yes, when that happens there will be a lot of re-compilation.  But usually that doesn't happen very often.