Standard testsuite (was: single-xt approach in the standard)

On 2024-09-19 00:59, Gerry Jackson wrote:

On 18/09/2024 09:44, Ruvim wrote:

On 2024-09-17 22:58, Anthony Howe wrote:

On 2024-09-17 07:59, Ruvim wrote:

On 2024-09-17 15:20, minforth wrote:

I would like to see an "officially recognized" standard reference
system, speed and number of xt's per word are of no importance.
>
Are you planning to make one?

>
I think, having the single standard reference implementation is a big step back in Forth standardization process. Because implementation details of the particular implementation will be used as requirements.

>
I think it should be recognised in some capacity, given Forth's origins, but whether there should be a reference implementation probably not.
>
>

I think, the standard conformance test suite is enough.

>
One would think so, but the test suite is:
>
* incomplete
* as published in draft 19-1 has typos and possible errors
* test cases often test multiple words at once that have not be tested separately
* assumes the entire draft with all the optional sets are present, rather than separate the word sets into separate unit tests

>
There is another testsuite:
https://github.com/gerryjackson/forth2012-test-suite

 Just to put the record straight, The Forth Test suite in the Forth 200X standard draft 19.1 incorporates most of my test suite with the addition of, for example, tests for new words such as FIND-NAME and FIND-NAME-IN.
 

>
>
I have an idea for a testsuite that:
   — is a ready-to-use program;
   — does not use words from optional word sets (for that, all the source files are transpiled into a single file that can be passed to stdin);
   — does not change the host Forth system (for example, does not add the missed standard words);
   — has an external program that parses output of the testsuite from the host's Forth system stdout and generates a report in text/xml/ xhtml form;
   — includes into the report general information such as implemented words and word sets, implementation options (that can be inferred by a standard program), the behavior of some words in edge cases, etc., along with the lists of passed and failed tests;
   — has a configuration for a number of known systems (that is extended by the community);
>

Some of these, particularly the reporting has been suggested before but, as you pointed out elsewhere everybody is an unpaid volunteer and nobody has found the time or motivation to undertake such extensions. Are you volunteering? I've just requested that the Forth 200X committee find someone else to take over my test suite - so feel free to volunteer.

I want to spend some time on such a testsuite. I even created a PoC to filter out the output of the testsuite (I can create a public Git repository for review now is anyone is interested). But at the moment I have some higher priority tasks regarding the standardization process than continue to develop the testsuite.

One thing not included in the Test Suite in the standard, as far as I can see,  and not mentioned in your wish list is that the Hayes tester assumes so much of the system is working before the tests actually start

I don't think that all the source codes of a testsuite should be included into the standard text. It is enough that the standard contains tests for all words and some of their combinations (as it is now). There is no need to even include a definition for `t{` — it is enough to describe the notation that is used for the test cases.

- how do you test the tester? I thought about this and developed a preliminary test based on the assumption that for automated testing at least the interpreter must work. So it starts by using SOURCE TYPE ( comments ) and CR to output messages and include a commentary of how the tests progress. If the displays are correct it tests >IN manipulation and uses it as an interpretive IF to actually detect errors. Then it goes on to do a basic test of the words used in the Hayes tester and so on. This is included in the link above.
 

It's good enough, I think.
We can try to use some *very* small amount of words to implement the testsuite engine itself (for example, we can even avoid using the control-flow words from the host system). But this will significantly impact performance. I do not sure that this approach is worth implementing.
It seems, there is a *very* small chance that the testsuite will produce false positive or false negative results due to bugs in some words of the host Forth system and then terminate normally. It is much more likely that the testsuite will be aborted during loading/compilation, or will halt/hang and be interrupted by an external watchdog in such a case.
--
Ruvim