Re: "RESET"

On Wed, 4 Jun 2025 12:58:21 +0200, David Brown
<david.brown@hesbynett.no> wrote:

On 30/05/2025 19:39, Joe Gwinn wrote:

On Fri, 30 May 2025 17:53:59 +0200, David Brown
<david.brown@hesbynett.no> wrote:
 

On 28/05/2025 18:07, Joe Gwinn wrote:

>

I recall those days.  Some managers thought that if they decreed that
no module could have a complexity (computed in various ways) exceeding
some arbitrary limit.  The problem was that real-world problems are
vastly more complex, causing atomization of the inherent complexity
into a bazillion tiny modules, hiding the structure and imposing large
added processing overheads from traversing all those inter-module
interfaces.
>

>
The problem with any generalisation and rules is that they are sometimes
inappropriate.  /Most/ functions, modules, pages of schematic diagram,
or whatever, should have a low complexity however you compute it.  But
there are always some that are exceptions, where the code is clearer
despite being "complex" according to the metrics you use.

 
No, all of the complexity metrics were blown away by practical
software running on practical hardware.  Very few modules were that
simple, because too many too small modules carry large inter-module
interface overheads.
 

>
That changes nothing of the principles.
>
You aim for low and controlled complexity, at all levels, so that you
can realistically test, verify, and check the code and systems at the
different levels.  (Checking can be automatic, manual, human code
reviews, code coverage tools, etc., - usually in combination.)  Any part
with particularly high complexity is going to take more specialised
testing and checking - that costs more time and money, and is higher
risk.  Sometimes it is still the right choice, because alternatives are
worse (such as the "too many small modules" issues you mention) or
because there are clear and reliable ways to test dues to particular
patterns (as you might get in a very large "dispatch" function).

In theory, sure.  In practice, it didn't help enough to make it
worthwhile.

You don't just throw your hands in the air and say it's better with
spaghetti in a module than spaghetti between modules, and therefore you
can ignore complexity!  I don't believe that is what you are actually
doing, but it sounds a little like that.

Peer review of the code works better, because no pattern scanning tool
can tell spaghetti from inherent complexity.

And this goes double for operating system kernel code, which violate
essentially all of the coding standards developed for user-level
application code.

Other tools that can be useful in testing are code coverage tools - you
can check that your test setups check all paths through the code.

>
We still do this, but the limitation is that all such tools yield far
more false alarms then valid hits, so all hits must be manually
verified.
>

>
A false alarm for a code coverage report would mean code that is not
reported as hit, but actually /is/ hit when the code is run.  How does
that come about?

 
The code coverage vendors hold the details close, so we usually don't
know how hits are declared, and probably never will.
 

>
Do the gcc and gcov developers hold their details secret?  I'm sure
there are many good reasons for picking different code coverage tools,
and I'm not suggesting that gcov is in any way the "best" (for many
reasons, code coverage tools would be of very limited use for most of my
work).  And there are all sorts of different coverage metrics.  But it
would surprise me if major vendors keep information about the prime
purpose of the tool a secret.  Who would buy a coverage tool that
doesn't tell you what it measures?

I was dealing with a proprietary code coverage tool that management
was quite enamored with and so was pressuring us to use.  But we had
only a sales brochure to go from, and I point-blank refused to use it
without knowing what it did and how.  This caused a copy of the
requirements document of the scanner to appear. 

I don't think gcov existed then.  We used gcc, so the software folk
would have used it were it both available and mature enough.

The one that I did manage to obtain the details turned out to be
looking for certain combinations of certain words and arrangements. It
had zero understanding of what the code did, never mind why.
 

>
I am not sure what kind of tool you are referring to here. 

It's the proprietary coverage tool mentioned above.

   Code
coverage tools track metrics about the functions, blocks and code lines
that are run.  Different tools (or options) track different metrics -
counts, times, or just "at least once".  They might track things at
different levels.  Some are intrusive and accurate, others are
non-intrusive but statistical based.  If you are wanting to use code
coverage tools in combination with branch testing, you just want to know
that during your test suite runs, every branch is tested at least once
in each direction.

Yep.  The longer-term outlook is that this proprietary tool is just
one of a stable of coverage tools that are used to generate lists for
manual disposition by the authors of the code being scanned.

Maybe modern AI will do better, but may be too expensive to make
business sense.
 

>
We can pretty much guarantee that commercial vendors will add claims of
AI to their tools and charge more for them.  Whether or not they will be
better for it, is another matter.

Yes.  Don't forget Quantum.

I would expect AI to be more useful in the context of static error
checkers, simulators, and fuzz testers rather than code coverage at
run-time.

Why?  I would think that a LLM could follow the thread far better than
any static checker.

The US financial firm Morgan Stanley is using AI to analyze and
summarize nine million lines of code (in languages such as COBOL) for
re-implementation in modern languages.  This from The Wall Street
Journal, 3 June 2025 issue:

"Morgan Stanley is now aiming artificial intelligence at one of
enterprise software's biggest pain points, and one it said Big Tech
hasn't quite nailed yet: helping rewrite old, outdated code into
modern coding languages.

In January, the company rolled out a tool known as DevGen.AI, built
in-house on OpenAI's GPT models. It can translate legacy code from
languages like COBOL into plain English specs that developers can then
use to rewrite it. 

So far this year it's reviewed nine million lines of code, saving
developers 280,000 hours, said Mike Pizzi, Morgan Stanley's global
head of technology and operations."

But it is certainly true that any kind of automatic testing or
verification is only going to get you so far - false hits or missed
cases are inevitable.

 
Yes, in fact the false hits dominate by a large factor, and the main
expense in using such tools is the human effort needed to extract
those few true hits.
 

>
Just to be clear - are you using non-intrusive statistical code coverage
tools (i.e., a background thread, timer, etc., that samples the program
counter of running code?  Or are you using a tool that does
instrumentation when compiling?  I'm trying to get an understanding of
the kinds of "false hits" you are seeing.
>
>

 

But remember that testing cannot prove the absence of bugs - only their
presence.  And it only works on the assumption that the hardware is
correct - even when the software is perfect, you might still need that
reset button or a watchdog!

>
Absolutely.  This was true in the days of uniprocessors with one
megahertz clocks and kilobyte memories.  Now it's hundreds of
processors with multi-gigahertz clocks and terabyte physical memories.
>

>
And it's still true - modern systems give more scope for hardware issues
than simpler systems (as well as more scope for subtle software bugs).
A cosmic ray in the wrong place can render all your software
verification void.

 
I must say that there was much worry about cosmic ray hits back in the
day, but they never turned out to matter in practice, except in space
systems.

>
I guess there are many factors for that.  If something weird happens,
and you have no explanation and it never happens again, then you can
easily say it was probably a cosmic ray - without any direct evidence.
It is also the case that many systems or subsystems are tolerant of an
occasional single-event upset - be it from cosmic rays or anything else.
 If you have ECC memory, or other kinds of redundancy or error
checking, rare errors there are not an issue.  So many types of memory,
buses, and communication protocols are effectively immune to such
things.  However, critical parts of the system will still be vulnerable
to hardware glitches.  It is not without justification that
safety-critical electronics often has two cores running in lockstep or
other types of redundancy.

What happened is that semiconductor technology progressed to the point
that the amount of charge (or whatever) that distinguished symbols
became very small and thus vulnerable to random errors, for which an
error-correcting code had to be built in.  At this point, cosmic rays
were lost in the random noise, so to speak. So ECC is now inherent,
not an extra-cost bolt-on.

The dominant source of errors turned out to be electrical cross-talk
and interference in the backplanes, and meta-stability in interfaces
between logic clock domains in the larger hardware system.
 

>
Sure.  Cosmic rays were only an example (pulled out of thin air :.) ).
Glitches on power lines, unlucky coincidences on bit patterns,
production flaws or ESD damage eroding electrical tolerances - there are
lots of possibilities.  I'm not trying to suggest relative likelihoods
here, as that will be highly variable.

And this is still true.

I vaguely recall doing an analysis on this issue, some decades ago.
 

>
I recall something of the opposite - a long time ago, we had to add a
variety of "safety" features to a product to fulfil a customer's safety
/ reliability checklist, without regard to how realistic the failure
scenarios were and without spending time and money on analysis.  The
result was, IMHO, lower reliability because it was more likely for the
extra monitoring and checking hardware and software to fail than for the
original functional stuff to fail.  Many of these extra checks were in
themselves impossible to test.

Yes.  I recall directly testing the issue with ECC as implemented in
early DEC VAX computers, in the early 1980s.  We had a customer who
specified ECC, so we had ECC.  And soon discovered that the computer
was more reliable with ECC disabled than enabled.  That was the end of
ECC.

Joe