flex — the lexical analyser generator, for WASIX

flex 2.6.4, built for WASIX. Turns a .l file into a C scanner that compiles anywhere — the output is portable C, it is only the generator that runs inside the sandbox.

wasmer run icedcoffee/flex -- -o scan.c scan.l
wasmer run icedcoffee/flex -- -t scan.l > scan.c
wasmer run icedcoffee/flex -- --header-file=scan.h -o scan.c scan.l

What is in the package

flex. The whole generator: %% rule sections, start conditions (%s, %x, BEGIN), REJECT, yymore()/yyless(), trailing context (r/s), <<EOF>> rules, %option handling (noyywrap, reentrant, bison-bridge, prefix=, outfile=, header-file=, 8bit/7bit, case-insensitive, stack, array, pointer, yylineno, nodefault, perf-report, …), %{ … %} user code, named definitions, -o, -t, -l, -i, -s, -v, -B, -I, -8, -C table-compression options, -+ (C++ scanners), --version, --help.

The generated scanner is ordinary C with no WASIX dependency; compile it with whatever toolchain you like.

Verified

$ cat scan.l
%option noyywrap
%%
[0-9]+      { printf("NUM(%s)\n", yytext); }
[a-zA-Z]+   { printf("WORD(%s)\n", yytext); }
[ \t\n]+    ;
.           { printf("CHAR(%s)\n", yytext); }
%%
int main(void) { yylex(); return 0; }

$ flex -o scan.c scan.l        # 44 KiB of C, 13 mentions of yylex
$ flex -t scan.l | wc -c       # same output on stdout

The generated scan.c was then compiled and run outside the sandbox as a correctness check:

$ printf 'abc 123 !\n' | ./scan_test
WORD(abc)
NUM(123)
CHAR(!)

What was needed

• A newer config.sub/config.guess. flex 2.6.4 ships 2015-vintage copies that reject wasm32-wasi outright (machine 'wasm32' not recognized). Replaced with current versions before configuring.
• ac_cv_func_malloc_0_nonnull=yes ac_cv_func_realloc_0_nonnull=yes. Without them configure decides libc's realloc is broken (it cannot run its realloc(p,0) test inside the sandbox) and points realloc at gnulib's rpl_realloc, which is not part of flex's build.
• M4=/bin/m4 at configure time. flex pipes its skeleton through m4 and bakes the m4 path in; the default /usr/bin/m4 does not exist in a WASIX image. It also falls back to $M4 from the environment at run time. This package declares the dependency on icedcoffee/m4, which installs /bin/m4.
• asyncify. Driving m4 is a fork/exec plus a pipe pair, and on WASIX that runs through the module's asyncify exports; without them every run exits 45. 435 KiB raw, 1.1 MB asyncified.

Building

M4=/bin/m4 ac_cv_func_malloc_0_nonnull=yes ac_cv_func_realloc_0_nonnull=yes \
./configure --host=wasm32-wasi --build=x86_64-linux-gnu --disable-nls
make
wasm-opt --asyncify --enable-threads --enable-bulk-memory src/flex -o flex.wasm

with -matomics -mbulk-memory -mmutable-globals -pthread -mthread-model posix -ftls-model=local-exec, -D_WASI_EMULATED_{MMAN,SIGNAL,PROCESS_CLOCKS,GETPID}, -lwasi-emulated-{mman,process-clocks,getpid}, -Wl,--shared-memory -Wl,--max-memory=4294967296 -Wl,--import-memory -Wl,-z,stack-size=2097152, and the __heap_base/__stack_pointer/__wasm_init_tls/__tls_* exports.

License: BSD-2-Clause, as upstream.