Re: Validation in non-regulated industries/markets

On 13/11/2024 01:21, Don Y wrote:

On 11/12/2024 4:27 PM, Niocláiſín Cóilín de Ġloſtéir wrote:

On 8th November 2024, David Brown wrote:
"Unfortunately, in some regulated markets, or for some types of "safety
certification", the rule-makers don't understand how this works.  The result is
that they insist on extra fault-checking hardware and/or software that actually
decreases the total reliability of the system, and introduces new parts that in
themselves cannot be checked (systematically, in production, and/or in the
field)."
>
>

VALIDATION is simply ensuring the product DELIVERED meets the needs of the
customer to which it is delivered.
 

By "validation", I would normally think that the /customer/ is accepting the product as working well enough for their needs.  This is roughly what you wrote, but the emphasis is on who does the validation.

TESTING simply ensures that a product meets its specification.

Nope.
Testing only ever shows the presence of bugs - it can never show their absence.  Only in very limited circumstances can you use testing to show that something meets its specifications - basically, you need to be able to test the system for every possible valid input (i.e., every input that is within the usage specifications).  That is sometimes possible for small software functions, but very rarely feasible for larger functions or anything involving hardware.
And even then, you are not "ensuring" it meets the specifications - you are "demonstrating" it.  The way you "ensure" you meet the specifications is my having a rigorous enough development procedure (which will include testing of many types) that you can be highly confident of correct behaviour.  Testing is vital to this, but it is not remotely sufficient.

 There can -- and often is -- a disconnect between the specification
and "what the customer wants/needs".  Because the spec author often
has incomplete domain knowledge OR makes assumptions that aren't
guaranteed by anything.
 

Absolutely true.  That's part of what makes development fun!

Because you are trying to prove the device meets the customer's needs,
you have to have THE product -- not a simulation of it or a bunch of
"test logs" where you threw test cases at the code and "proved" that
the system did what it should.
 [I patched a message in a binary for a product many years ago.  The
customer -- an IBM division -- aborted the lengthy validation test
when my message appeared ("That is not supposed to be the message
so we KNOW that this isn't the actual system that we contracted to
purchase!")]
 Because you have to validate the actual product, the types of things
you can do to the hardware to facilitate fault injection are sorely
limited.  "Are you SURE this doesn't have any secondary impact on
the product that changes WHAT we are testing?"

Like I said - you do fault injection and similar testing as a systematic test of the design and fully replicable parts of the system (like software).  You don't do it on final products.
And customers may be involved in, or have insight into, such fault injection tests.
You talk about "proving" the device meets the customer's needs, and "ensuring" it works according to specification.  That's just wrong. This is not a pass/fail binary thing where you have proof of correctness - its a matter of risk reduction, failure mitigation, and confidence. You are never trying to make something that is "perfect" in the true sense - because you can never succeed.  What you are trying to do is get a high confidence that there is a low risk of the product not fulfilling its specifications, and that the ill-effects of any failings are minimal within the time and cost budgets.
You achieve this by combining many methods, each of which reduces the risks.  That starts with good specification methods, passes through design and development phases, test phases (including fault injection and other systematic tests), production tests, long-term tests, artificial ageing tests, follow-up testing of deployed devices, and whatever else suits for the type of project and budget.

I was interviewed by a prospective client to work on an
"electronic door lock" system (think: hotels).  During the
demo of their prototype, I reached over and unplugged a cable
(that I strongly suspected would inject a fault... that they
would NOT detect!).  Sure as shit, their system didn't notice
what I had done and blindly allowed me to produce several
"master keys" while my host watched in horror.  All without
the system having a record of my actions!
 "Oooops!  Wanna bet that wasn't part of the specification?!"
 Validation exists for a reason -- separate from and subsequent to
product testing.  Because these sorts of SNAFUs happen all the
time!
 

That's not validation.  Validation /does/ exist for a reason - it's what lets the developer get paid and not sued even if the customer later on thinks of an extra clause that they should have had in their specifications.  "Validation" is the point when the customer takes on the risks for the product not actually working according to their needs (which they might not be fully aware of).