Re: [gawk] Handling variants of CSV input data formats

On 8/26/2024 8:39 PM, Janis Papanagnou wrote:

On 27.08.2024 02:49, Ed Morton wrote:

On 8/26/2024 7:54 AM, Janis Papanagnou wrote:

snip>
I'd have liked to provide more concrete information here, but I'm at
the moment even unable to reproduce Awk's behavior as documented in
its manual; I've tried the following command with various locales
>
$ echo 4,321 | LC_ALL=en_DK.utf-8 gawk '{ print $1 + 1 }'
-| 5,321
>
but always got just  5  as result.

>
You need to specifically TELL gawk to use your locale to read input
numbers:
>
$ echo 4,321 | LC_ALL=en_DK.utf-8 gawk '{ print $1 + 1 }'
5
>
$ echo 4,321 | POSIXLY_CORRECT=1 LC_ALL=en_DK.utf-8 gawk '{ print $1 + 1 }'
5,321
>
$ echo 4,321 | LC_ALL=en_DK.utf-8 gawk -N '{ print $1 + 1 }'
5,321
>
See
https://www.gnu.org/software/gawk/manual/gawk.html#Locale-influences-conversions
for more info on that.

 Thanks. That's actually where I got above example from.
 I've missed that there was an explicit
$ export POSIXLY_CORRECT=1
set on the very top of these examples. Gee!
 Feels anyway strange that an explicit LC_* setting is ineffective
without the additional POSIXLY_CORRECT variable. And the page also
says: "The POSIX standard says that awk always uses the period as
the decimal point when reading the awk program source code".
So despite POSIX saying that, you have to use a variable named
POSIXLY_CORRECT. - Do I need some more coffee to understand that?

POSIXLY_CORRECT=1 (or equivalently `--posix` aka `-P`) affects numbers in the input your script reads (as shown in the previous post) and strings being converted to numbers in your code, it doesn't affect literal numbers in the source code for your script that awk reads.
In the source code the decimal separator for a literal number (as opposed to a string being converted to a number) is always `.`.
You can't use, say, a comma as the decimal separator in a literal number because a comma already means something in the awk syntax, e.g. `print 4,321` means the same as "print 4 OFS 321`.
For example, this code compares a literal number 4.321 to another literal number 4.1 and prints "bigger" because 4.321 is a bigger number than 4.1:
$ awk 'BEGIN{ if (4.321 > 4.1) print "bigger"; else print "not" }'
bigger
while this attempt to use `,` as the decimal separator is a syntax error because `,` has a meaning in awk syntax:
$ awk 'BEGIN{ if (4,321 > 4.1) print "bigger"; else print "not" }'
awk: cmd. line:1: BEGIN{ if (4,321 > 4.1) print "bigger"; else print "not" }
awk: cmd. line:1:             ^ syntax error
You can't just write the number as a string because then you're doing a string comparison and comparing the collation order of `.` vs `,`, not the values of the numbers:
$ awk 'BEGIN{ if ("4,321" > 4.1) print "bigger"; else print "not" }'
not
You might think you can just add `0` to convert the string to a number:
$ awk 'BEGIN{ if (("4,321"+0) > 4.1) print "bigger"; else print "not" }'
not
but by default in gawk that conversion truncates everything from the `,` on and so you end up comparing the number 4 to the number 4.1.
To convert `"4,321"` to the number `4.321` in gawk you need to once again set your locale and tell gawk to use it:
$ LC_ALL=en_DK.utf-8 awk 'BEGIN{ if (("4,321"+0) > 4.1) print "bigger"; else print "not" }'
not
$ LC_ALL=en_DK.utf-8 awk -N 'BEGIN{ if (("4,321"+0) > 4.1) print "bigger"; else print "not" }'
bigger
$ LC_ALL=en_DK.utf-8 awk -P 'BEGIN{ if (("4,321"+0) > 4.1) print "bigger"; else print "not" }'
bigger
or in gawk instead of adding 0 you could use `strtonum()` on a string containing that value to convert it to a number. For example:
$ LC_ALL=en_DK.utf-8 awk 'BEGIN{ if (strtonum("4,321") > 4.1) print "bigger"; else print "not" }'
not
$ LC_ALL=en_DK.utf-8 awk -N 'BEGIN{ if (strtonum("4,321") > 4.1) print "bigger"; else print "not" }'
bigger
$ LC_ALL=en_DK.utf-8 awk -P 'BEGIN{ if (strtonum("4,321") > 4.1) print "bigger"; else print "not" }'
awk: cmd. line:1: fatal: function `strtonum' not defined
In that last example above we see why `-N` is better than `-P` since they both do what you want with the number but `-P` also disables gawk extensions while `-N` doesn't.

 And I see there's an additional GNU Awk option '--use-lc-numeric'.

That's just the long form of `-N`, identical in meaning.

What a mess!
 (I suppose current status can only be explained by the mentioned
forth-and-back during history of various GNU Awk versions.)

Right.

 What's worth the LC_* variables if they are ignored (or maybe not).

They have their uses but using them for everything by default apparently isn't the way people most frequently want to use awk so you need options to tell gawk when to use them in specific situations.
Ed.

 Janis
 

>
Regards,
>
     Ed