Re: "RESET"

On 5/27/2025 4:13 PM, Joe Gwinn wrote:

On Tue, 27 May 2025 14:13:02 -0700, Don Y
<blockedofcourse@foo.invalid> wrote:
 

On 5/25/2025 12:33 PM, Joe Gwinn wrote:

Exactly.  I recall a customer wanting us to verify all possible paths
through a bit of air traffic control radar software, about 100,000
lines of plain C.  Roughly one in five executable line was an IF
statement, which is 20,000 IF statements.  So there are 2^20000 =
10^6020 such paths.

>
And probably 99.9% of them are superfluous.

>
[snip]
 The problem is that you have no way to know which cases are
irrelevant.  And practical hardware will have many things able to
retain state.

The design of the specification becomes important.
The *implementor* can further refine the test cases based
on other potential vulnerabilities that his implementation
may create.
E.g., if you implement a multiply routine with repeated
additions of the multiplicand, the case where the multiplier
is zero can be buggy -- because you may have examined
the "remaining multiplier" in a place that assumes at
least one iteration of the multiplicand's processing.
Or, a negative multiplier (how do you run a loop some
negative number of times?)

Experience teaches you to construct your code so that testability is
enhanced.  Instead of waiting until it seems to be "done" and then
trying to reassure yourself that it works as intended -- usually by
throwing EXPECTED conditions at it and hoping for the expected
results (that's not testing).

 It is not usually the expected that causes trouble:  It ain't what you
don't know that matters, it what you know that ain't so that's the
problem.

Which is why it is so easy to prove code doesn't work!  You
identify what the likely assumptions were and then challenge
each of those.  People treat assumptions as so ingrained
that they never actually consider their validity:  Will
this value always be non-zero?  will the datalogger
always be powered on?  will "spare" memory always be
available?
Attack one and watch the code flail about -- because it's
expectations have been defied.

You need 2^20000 if there are 2^20000 distinct outcomes (leaf nodes) in
your code.  I strongly doubt that to be the case.

 So assume only 1000 IF statements, so it's 2^1000 or 10^600 or so.
You'll still run out of lifetime.

If you have 1000 if statements in a module, then your module is
at least 16 pages long -- containing no OTHER code.  This is why
you make small modules so their complete behavior can be "groked"
without having ludicrous levels of complexity.
if (x==0) {
   //
} else if (x==1) {
   //
} else if (x==2) {
   //
} else if (x==3) {
   //
} else {
   //
}
Four conditionals.  6 test cases for 100% coverage (<0, 0, 1, 2, 3, >=4)
Not 16.
I.e., assuming complexity correlates with a geometric weighing of
the number of conditionals is specious reasoning.

The testing campaign will have only scratched the surface when the Sun
runs out of hydrogen and goes supernova.  Tomorrow's problem.

>
How do you test an electronic circuit?  Let's impose an infinite number of
discrete voltages on each of the input signals and verify the correct
outputs for each?  (Do you deliberately verify all ranges of signal values
and frequencies?  Or, just say "operation outside of these conditions is
indeterminate"?)

 You do all the tests for required behavior - does it meet stated
requirements.

But, you only test the *boundary* conditions.  I.e., if something is
suposed to work over a range of 0 - 12 volts, you test 0, 12, something
in between and then < 0 and > 12.  You don't test 0, 0.1, 0.2, 0.3, 0.4,
0.5, 0.50000000000001, 0.5000000000002, etc.
Because you know the circuit is well-behaved BETWEEN these conditions.
A UNIX pathname can be thousands of characters long.  Do you test:
    a
    b
    c
    d
    ...
    z
    A
    ...
    Z
    0
    ...
    9
    aa
    ab
    ...
    az
    ...
Of course not!  Yet, a test case like:
    /0/1/2/3/4/../../../../../../../../../../../../../../../..
could tickle a bug in the implementation.
UNLESS, the implementor took a shortcut that exposed some buried
condition that needs to be exposed to testing.  E.g., if he
routed the signal through two different paths based on whether
or not it exceeded 5 volts.
The specification likely won't have anticipated such an "optimization"
so wouldn't have drawn attention to "5V" as a particularly important case.
The implementor/designer has to understand how his implementation could
potentially break in ways that the specification couldn't catch.

Then you random probe it for weeks and see what goes Bang!
 One form of this is Fuzzing.
 .<https://www.usenix.org/conference/usenixsecurity22/presentation/trippel>

You still rely on "luck" to stumble on bugs.  You need good specifications,
"bite size" modules (where complexity is limited) and implementors who
are aware of testing IN their design choices -- so they don't meet
the published specifications (and likely test cases) but still fail,
miserably when their internal quirks are tickled.