gcc arm inline asm: how to output value for .set directive?

How do I convince ARM GCC's extended asm() to emit a value that can be
used in a .set directive?  Here's a simplified example:

    #include <stdint.h>
   
    typedef struct
    {
      uint8_t  what[32];
      uint32_t ever[2];
      unsigned foo;
      uint8_t bar[20];
    } shm_t;
   
    void foo(void)
    {
      asm("\t.global foo_offset");
      asm("\t.set foo_offset, 40");
      asm("\t.set foo_offset, %[off]" : : [off] "i" ( __builtin_offsetof(shm_t, foo) ) : );
    }

The first of the two .set directive works.  The second one doesn't.
The offsetof() value is being used as desired, but gcc is prefixing
the value with a '#' (presumably it thinks it's an instruction operand
and not a directive operand). Here's an excerpt from the assembly
listing:

  28              @ 13 "foo.c" 1
  29              .global foo_offset
  30              @ 0 "" 2
  31              @ 14 "foo.c" 1
  32              .set foo_offset, 40
  33              @ 0 "" 2
  34              @ 15 "foo.c" 1
  35              .set foo_offset, #40
  36              @ 0 "" 2

And of course the assembler chucks a wobbly at line 35:

/tmp/cc1TaWVh.s: Assembler messages:
/tmp/cc1TaWVh.s:35: Error: bad expression
/tmp/cc1TaWVh.s:35: Error: junk at end of line, first unrecognized character is `4'

I've tried a bunch of gcc extended assembly input value constraints
and modifiers, but can't find the magic code that emits the value "40"
without the '#' on the front.

Any ideas?