Re: slow fileutil::foreachLine

Mark Summerfield <mark@qtrac.eu> wrote:

I have this function:
 
proc ws::get_words {wordfile} {
    set in [open $wordfile r]
    try {
        while {[gets $in line] >= 0} {
            if {[regexp {^[a-z]+$} $line matched]} {
                lappend ::ws::Words [string tolower $matched]
            }
        }
    } finally {
        close $in
    }
}
 
It reads about 100_000 lines and ends up keeping about 65_000 of them
(from /usr/share/dict/words)
 
I tried replacing it with:
 
proc ws::get_words {wordfile} {
    ::fileutil::foreachLine line $wordfile {
        if {[regexp {^[a-z]+$} $line matched]} {
            lappend ::ws::Words [string tolower $matched]
        }
    }
}
 
The first version loads "instantly"; but the second version (with
foreachLine) takes seconds.

If you check the implementation of fileutil::foreachLine, you find:

   set code [catch {uplevel 1 $cmd} result options]

Where "$cmd" is a variable holding a string of the "command" passed to
foreachLine.

Your original copy is all in a single procedure, so it will be bytecode
compiled, and for all but the first execution will run that compiled
bytecode.

The foreachLine version, since the "cmd" is a string, will receive
little to no byte code compiling, and the difference in time is the
overhead of not being able to bytecode compile the "command" string
passed to foreachLine.