Re: 2GB limitation

On Wed, 24 Jul 2024 16:22:53 -0000 (UTC)
Andreas Leitgeb <avl@logic.at> wrote:

Emiliano <emiliano@example.invalid> wrote:

In 9.0 the type of the 'length' member of the Tcl_Obj struct (the number of
bytes at '*bytes' member, not including the terminating null) has changed
from int to ptrdiff_t, so it will remain (1<<31)-1 => 2147483647 bytes on
32 bit platforms (unsurprisingly) and (1<<63)-1 => 9223372036854775807
(9,22 exabyte) on 64 bit platforms.

 
My hearsay was "generally 64 bit (minus the sign-bit)".
Are you sure that length-type is *always* ptrdiff_t, and
that this may be 32bit?

In 9.X, it is ptrdiff_t. In 8.Y is still int.

See https://core.tcl-lang.org/tcl/file?ci=trunk&name=generic/tcl.h&ln=325-333
and
https://core.tcl-lang.org/tcl/file?ci=trunk&name=generic/tcl.h&ln=740-752

ptrdiff_t can still be a 32 bits wide value. See below.
 

The "64bit'ness" of a platform is also a bit more complicated...
There are platforms, where pointers are 64bit, but ints are
still 32 (despite machine words being all 64bit) - in those
cases, I'd expect ptrdiff_t to be 64 bit, but on a real old
32bit machine, I don't really know for sure...

This is what I mean when say "on 32-bit platforms is still 2GB",
since i386-i686 platform has a 32 bit ptrdiff_t.

On my ancient i686 machine:

$ uname -m
i686
$ tclsh9.0
% expr {(1 << (8 * $tcl_platform(pointerSize))-1) - 1}
2147483647
% package provide Tcl
9.0b3
% set tcl_platform(pointerSize)
4

IIUC that's also the (new) number of elements for a Tcl list.
In practice the number will be less, since the length of the
string representation of such list will hit the '*bytes' max
length first.

 
Not all lists are ever turned to string-rep.  While they are
semantically "just strings", well written programs can avoid
the actual obtainment of the string rep, at least for those
really long lists that may be relevant here.

Yes, but that's an optimization. Tcl semantics are still defined
in terms of strings operations. I prefer not to depend on internals.

--
Emiliano