Re: Unit Testing: from theory to real case

Il 30/08/2024 14:21, Don Y ha scritto:

On 8/30/2024 1:18 AM, pozz wrote:

When you write, test for this, test for that, what happens if the client uses the module in a wrong way, what happens when the system clock changes a little or a big, and when the task missed the exact timestamp of an event?
>
I was trying to write tests for *all* of those situations, but it seemed to me a very, VERY, *VERY* big job. The implementation of the calendar module took me a couple of days, tests seem an infinite job.

 Because there are lots of ways your code can fail.  You have to prove
that it doesn't fail in ANY of those ways.

So you're confirming it's a very tedious and long job.

uint multiply(multiplicand, multiplier) {
     return(6)
}
 works well for the test cases:
   2,3
   3,2
   6,1
   1,6
but not so well for:
   8,5
   17,902
   1,1
etc.

This is a simpler case. Just test for a couple of different result and you're almost sure it works for normal case (positive integers).

I have four types of events, for each test I should check the correct behaviour for each type.
>
What happen if the timestamp of an event was already expired when it is added to the system? I should write 4 tests, one for each type.
>
AddOneShotEventWithExpiredTimestamp_NoActions
AddWeeklyEventWithExpiredTimestamp_NoActions
AddMonthlyEventWithExpiredTimestamp_NoActions
AddYearlyEventWithExpiredTimestamp_NoActions

 Chances are, there is one place in your code that is aware of the fact that
the event is scheduled for a PAST time.  So, you only need to create one test.
(actually, two -- one that proves one behavior for time *almost* NOT past and
another for time JUST past)

I read that tests shouldn't be written for the specific implementation, but should be generic enough to work well even if the implementation changes.

Your goal (having already implemented the modules) is to exercise each
path through the code.
 whatever() {
    ...
    if (x > y) {
       // do something
    } else {
       // do something else
    }
    ...
}
 Here, there are only two different paths through the code:
- one for x > y
- one for !(x > y)
So, you need to create test cases that will exercise each path.

Now I really know there are only two paths in the current implementation, but I'm not sure this will stay the same in the future. What happens if a developer changes the code in:
   if (event_type == SINGLE) {
     if (event_timestamp < current_time) {
       // Event expired, ignore it
     } else {
       add_event_to_active_events();
     }
   } else if (event_type == MONTHLY) {
     if (event_timestamp <= current_time) {
       ...
     } else {
       ...
     }
   } else ...
The paths could multiply.

To verify your "x > y" test, you would want to pick an x that is
just detectably larger than y.  And, another case where x is as
large as possible WITHOUT exceeding y.  You can view this as defining
the "edge" between the two routes.

Yes, and the test cases proliferate like ants... sigh :-(

If, for example, you picked x = 5 and x = 3 as your test cases
(where y = 4), then you WOULD exercise both paths.  But, if you had
mistakenly coded this as
    if (x >= y) {
       // do something
    } else {
       // do something else
    }
you wouldn't be able to detect that fault, whereas using x = 5 and x = 4
would cause you to wonder why "do something else" never got executed!
 

What does it mean "expired timestamp"? Suppose the event timestamp is

 A time that is "in the past".  If it is time 't', now, what happens if the
client specifies an event to happen at time t-1?  Should you *immediately*
activate the event (because NOW it is t > t-1?)  Or, should you discard it
because it was SUPPOSED to happen 1 second ago?
 What if t-495678?  Is there a different type of action you expect if the
time is "a long time ago" vs. "just recently"?
 Do events happen at *instants* in time?  Or, in CONDITIONS of time?
 If they happen at instants, then you have to ensure you can discern one
instant from another.

Yes, yes, yes, please add tests, tests, tests.
My trouble isn't inventing new tests for normal and edge and corner cases, but limiting them to the real necessity, otherwise the software development stucks on tests.

"01/01/2024 10:00:00". This timestamp could be expired for a few seconds, a few minutes or one day or months or years. Maybe the module performs well when the system time has a different date, but bad if the timestamp expired in the same day, for example "01/01/2024 11:00:00" or "01/01/2024 10:00:01".
Should I add:
>
AddOneShotEventWithExpiredTimestamp1s_NoActions
AddOneShotEventWithExpiredTimestamp1m_NoActions
AddOneShotEventWithExpiredTimestamp1h_NoActions
AddOneShotEventWithExpiredTimestamp1d_NoActions
AddWeeklyEventWithExpiredTimestamp1s_NoActions
AddWeeklyEventWithExpiredTimestamp1m_NoActions
AddWeeklyEventWithExpiredTimestamp1h_NoActions
AddWeeklyEventWithExpiredTimestamp1d_NoActions
AddMonthlyEventWithExpiredTimestamp1s_NoActions
AddMonthlyEventWithExpiredTimestamp1m_NoActions
AddMonthlyEventWithExpiredTimestamp1h_NoActions
AddMonthlyEventWithExpiredTimestamp1d_NoActions
AddYearlyEventWithExpiredTimestamp1s_NoActions
AddYearlyEventWithExpiredTimestamp1m_NoActions
AddYearlyEventWithExpiredTimestamp1h_NoActions
AddYearlyEventWithExpiredTimestamp1d_NoActions
>
They are 16 tests for just a single stupid scenario. If I continue this way, I will thousands of tests. I don't think this is the way to make testing, do I?

 You declare what scenario you are testing for as a (commentary) preface
to the test stanza.
 If you are testing to ensure "NoActions" is handled correctly, then
you look to see how many ways the "NoActions" criteria can tickle
the code.
 If there is only ONE place where "NoActions" alters the flow through
the code, then you only need one test (actually, two as you need
to cover "SomeAction" to show that "NoAction" is different).
 In a different test scenaario, you would test that 1s, 1m, 1h, 1d,
etc. are all handled correctly IF EACH OF THOSE PASSED THROUGH YOUR
CODE OVER DIFFERENT PATHWAYS.
 And, elsewhere, you might test to see that "repeated" events
operate correctly.
 You "prove" that one scenario is handled correctly and then
don't need to reexamine those various tests again in any other
scenario UNLESS THEY ALTER THE PATH THROUGH THE CODE.
 Your understanding of how the code would LIKELY be crafted lets
you determine some of these tests before you've written ANY
code.  E.g., I suggested "expired events" because I am reasonably
sure that SOMEWHERE your code is looking at "event time" vs. "now"
so you would need to test that comparison.
 Your knowledge of how the code is *actually* crafted lets you
refine your test cases to cover specifics of YOUR implementation.
 Note that test cases that are applied to version 1 of the code should
yield the same results in version 305, even if the implementation
changes dramatically.  Because the FUNCTIONALITY shouldn't be
changing.

Ok, but if you create tests knowing how you will implement functionalities (execution paths), it's possible they will not be sufficient when the implementation change at version 305.

So, you can just keep adding test cases to your test suite;
you don't ever need to remove any.
 [If a test STOPS working, you have to ask yourself how you have
BROKEN/changed the function of the module]
 For example, I could implement a "long" integer multiplication routine
by adding the multiplicand to an accumulator a number of times
dictated by the multiplier.  I can create test cases for this
implementation.
 Later, I could revise the routine to use a shift-and-add approach.
 BUT, THE ORIGINAL TEST CASES SHOULD STILL PASS!  I might,
however, have to add some other tests to identify failures in
the shift logic in this new approach (e.g., if I only examined
the rightmost 26 of the bits in the multiplier, then a large
value multiplier would fail in this approach but could pass
in the repeated addition implementation).  This would be
evident to me in looking at the code because there would be
a different path through the code when the "last bit" had been
checked.

I agree with you. Indeed the beautiful theory that tests shouldn't take into account real implementation is imho false.
I know a test case should verify a functionality that is indipendent of implementation, but the number and type of test cases could change *depending* on the current implementation.
Just as a limit case, suppose you ahave a  function that calculate the square of a number in the range 0-15, so the result can be returned in a unsigned char. The return value is undefined if the parameter is greater than 15.
   unsigned char square(unsigned char num);
Before implementing the function I can imagine the following test cases:
   assert(square(0) == 0)
   assert(square(1) == 1)
   assert(square(2) == 4)
   assert(square(15) == 225)
Now the developer writes the function this way:
   unsigned char square(unsigned char num) {
     if (num == 0) return 0;
     if (num == 1) return 1;
     if (num == 2) return 4;
     if (num == 3) return 9;
     if (num == 4) return 16;
     if (num == 5) return 35;
     if (num == 6) return 36;
     if (num == 7) return 49;
     ...
     if (num == 15) return 225;
   }
My tests pass, but the implementation is wrong. To avoid this I, writing tests, should add so many test cases that I get a headache.