Re: Validation in non-regulated industries/markets

On 11/12/2024 4:27 PM, Niocláiſín Cóilín de Ġloſtéir wrote:

On 8th November 2024, David Brown wrote:
"Unfortunately, in some regulated markets, or for some types of "safety
certification", the rule-makers don't understand how this works.  The result is
that they insist on extra fault-checking hardware and/or software that actually
decreases the total reliability of the system, and introduces new parts that in
themselves cannot be checked (systematically, in production, and/or in the
field)."
  Professor William H. Sanders from the University of Illinois at Urbana-Champaign dishonestly boasted in a lecture to us on what he pretends to be "Validating computer system and network trustworthiness" that he has solved a problem to guarantee successes against faults and that NASA had complained that this problem is not solvable. He showed us this sham supposed solution. So I immediately accused him that this proposal does not succeed. So he admitted this accusation when he said "Who checks the checker?" Many papers exist with a similar title to this question. They are supposedly named after a supposedly common question about Ancient Roman soldiers.

VALIDATION is simply ensuring the product DELIVERED meets the needs of the
customer to which it is delivered.
TESTING simply ensures that a product meets its specification.
There can -- and often is -- a disconnect between the specification
and "what the customer wants/needs".  Because the spec author often
has incomplete domain knowledge OR makes assumptions that aren't
guaranteed by anything.
Because you are trying to prove the device meets the customer's needs,
you have to have THE product -- not a simulation of it or a bunch of
"test logs" where you threw test cases at the code and "proved" that
the system did what it should.
[I patched a message in a binary for a product many years ago.  The
customer -- an IBM division -- aborted the lengthy validation test
when my message appeared ("That is not supposed to be the message
so we KNOW that this isn't the actual system that we contracted to
purchase!")]
Because you have to validate the actual product, the types of things
you can do to the hardware to facilitate fault injection are sorely
limited.  "Are you SURE this doesn't have any secondary impact on
the product that changes WHAT we are testing?"
E.g., tablet presses produce up to 200 tablets per second.  One of
the control systems is responsible for "ensuring" that the *weights*
of the tablets are correct.  But, you can't weigh individual tablets
in 5ms.  And, you can't alter the weight of a tablet once it is produced!
So, you have to watch the (mechanical) process and convince yourself that
the product coming off the press is /highly likely/ to meet the weight
constraints.  You do this by watching the forces exerted on the "powder"
(granulation) in a fixed geometry cavity for each tablet.  Obviously,
if more material was present, the force would go up; down if less.
[Alternatively, you can allow the geometry to vary and watch how
LARGE the resulting tablets are -- again, at 5ms intervals]
How do you simulate a PARTICULAR tablet being filled with too much -- or
too little -- granulation?  You can't stop the press and add (or subtract)
a little material -- the dynamics of the process would be altered.
But, you can alter the geometry of a particular cavity so that it
takes on more (or less) material.  And, further alter it so that
it leaves a visible mark on the affected tablets (e.g., put a score
line on the top surface of THOSE tablets and no score line on the
others.
Now, run the press and make sure ONLY scored tablets are in the "reject"
pile and NO scored tablets are in the "accept" pile.  You've validated
the controller's ability to discriminate between good and bad tablet
weights -- without altering the software or hardware in the controller.
This lets issues that may have been omitted in the specification
percolate to a level of awareness that DOES affect the product's
applicability.
"What if the feeder runs out of material?  Or, the material has
the wrong moisture content and 'clumps'?"
"What if the mechanism that is responsible for physically sorting
the tablets (at 5ms intervals) fails?"
"What if the tablet press has been improperly configured and
the geometry isn't guaranteed to be fixed (e.g., the process's
compression force is creeping into the "overload" setting
which changes the physical dimensions of each compression
event, dynamically -- so tablet #1 and tablet #2 experience
different compression conditions)"
I was interviewed by a prospective client to work on an
"electronic door lock" system (think: hotels).  During the
demo of their prototype, I reached over and unplugged a cable
(that I strongly suspected would inject a fault... that they
would NOT detect!).  Sure as shit, their system didn't notice
what I had done and blindly allowed me to produce several
"master keys" while my host watched in horror.  All without
the system having a record of my actions!
"Oooops!  Wanna bet that wasn't part of the specification?!"
Validation exists for a reason -- separate from and subsequent to
product testing.  Because these sorts of SNAFUs happen all the
time!