Re: Validation in non-regulated industries/markets

On 08/11/2024 05:10, Don Y wrote:

In *regulated* industries (FDA, aviation, etc.), products are
validate (hardware and software) in their "as sold" configurations.
This adds constraints to what can be tested, and how.  E.g.,
invariants in code need to remain in the production configuration
if relied upon during validation.
 But, *testing* (as distinct from validation) is usually more
thorough and benefits from test-specific changes to the
hardware and software.  These to allow for fault injection
and observation.
 In *unregulated* industries (common in the US but not so abroad),
how much of a stickler is the validation process for this level
of "purity"?
 E.g., I have "test" hardware that I use to exercise the algorithms
in my code to verify they operate as intended and detect the
faults against which they are designed to protect.  So, I can inject
EDAC errors in my memory interface, SEUs, multiple row/column
faults, read/write disturb errors, pin/pad driver faults, etc.
 These are useful (essential?) to proving the software can
detect these faults -- without having to wait for a "naturally
occurrence".  But, because they are verified/validated on non
production hardware, they wouldn't "fly" in regulated
markets.
 Do you "assume" your production hardware/software mimics
the "test" configuration, just by a thought exercise
governing the differences between the two situations?
 Without specialty devices (e.g., bond-outs), how can you
address these issues, realistically?
 

I think perhaps this is confusing systems testing with product testing. You need to make a clear distinction between the two.
Systems testing is about checking that a /design/ is correct.  Much of that is usually about software testing, but it applies to hardware too. This will often be done using modified hardware so that you can, for example, inject /realistic/ faults and check that the hardware and software function as expected.  Depending on the application, you might also run test boards at high temperatures or otherwise abuse them to confirm the design.
Production testing is about ensuring that the products made are correct according to the design.  You don't check that the memory works, or the ECC handler works - you check that you have correctly mounted and soldered the memory chip and that the memory chip supplier has checked for production faults.
There are some products where the likelihood of developing partial faults in the field are high and the consequences of that are serious but it is useful to be able to keep a partially failed system in action.   There are also products with user-serviceable parts.  Then it is often helpful to have some kind of self-test to identify failing subsystems.
Unfortunately, in some regulated markets, or for some types of "safety certification", the rule-makers don't understand how this works.  The result is that they insist on extra fault-checking hardware and/or software that actually decreases the total reliability of the system, and introduces new parts that in themselves cannot be checked (systematically, in production, and/or in the field).
How do you deal with it?  You follow the rules, even though some of them were written by muppets.