TurboVM Bytecode Engine
=======================

TurboVM is an optional high-performance normalization engine for liblognorm.
It compiles rulebases into bytecode at startup and executes them through a
linear virtual machine with SIMD-accelerated parsing primitives. When enabled,
it provides significant throughput improvements over the default recursive
parser, especially on high-volume log streams.

Overview
--------

The default liblognorm normalization engine (the "recursive walker") traverses
the parse DAG node-by-node for each log message, allocating json-c objects as
fields are extracted. TurboVM replaces this with:

- **Bytecode compilation**: rulebases are compiled into a compact instruction
  sequence at load time. Each rule becomes a linear program.
- **Arena allocation**: all per-message memory comes from a single pre-allocated
  arena (~16 KB), fitting in L1 cache. Zero malloc/free per message.
- **SIMD parsing**: character scanning, delimiter search, whitespace skipping,
  and IP address parsing use SSE4.2 or NEON intrinsics when available.
- **Output parity**: field *values* match the standard engine (numeric
  fields are strings by default, as in the standard parser). The JSON
  *document* nests dotted names; see "Known differences" below.
- **Nested JSON**: dotted field names (e.g. ``source.ip``) produce
  nested JSON objects (``{"source":{"ip":"..."}}``), which is the ECS
  (Elastic Common Schema) shape.

Building with TurboVM
---------------------

TurboVM is an optional build feature, disabled by default::

    ./configure --enable-turbo

The build system automatically detects the CPU architecture and enables the
appropriate SIMD instruction set:

- **x86-64**: SSE4.2 (Intel Nehalem+, AMD Bulldozer+)
- **ARM64**: NEON (all ARMv8-A processors, including Apple M1/M2)
- **Other**: scalar fallback (functional but without SIMD acceleration)

No additional dependencies are required.

Using with lognormalizer
------------------------

The ``lognormalizer`` command-line tool supports turbo mode via the
``-oturbo`` option::

    $ lognormalizer -r rules.rb -e json -oturbo < messages.log

In turbo mode:

- Normalization uses the TurboVM bytecode engine
- Output is compact JSON with nested objects for dotted field names
- Field values match the standard engine (numeric fields are strings by
  default, or native JSON numbers with ``format="number"``)
- Tags are emitted at the root as ``tags``, the ECS spelling, and only when
  the matched rule carries any. The standard engine uses a flat
  ``event.tags`` key instead; see "Known differences" below
- The ``getline()`` system call is used for input (more efficient than
  ``fgets()`` for large-scale processing)

If a rulebase cannot be compiled to bytecode (e.g. it uses unsupported
parser types), lognormalizer falls back to standard normalization
automatically. The same happens per message whenever the bytecode engine
declines one.

That fallback is what makes turbo safe to enable, and also what makes a
turbo defect hard to see: the standard engine quietly produces the right
answer, so comparing output tells you nothing about whether the bytecode
engine did any work. ``-oturbostrict`` turns the fallback off and reports
the failure instead::

    $ lognormalizer -r rules.rb -oturbostrict < messages.log

Use it when measuring turbo coverage or writing parity tests; a message
reported unparsed under ``-oturbostrict`` but parsed under ``-oturbo`` is
one the bytecode engine declined.

Library API
-----------

To enable TurboVM in your application, set the ``LN_CTXOPT_TURBO`` option
on the normalization context before loading rules::

    #include <liblognorm.h>

    ln_ctx ctx = ln_initCtx();
    ln_setCtxOpts(ctx, LN_CTXOPT_TURBO);
    ln_loadSamples(ctx, "/path/to/rules.rb");

After loading, verify that compilation succeeded::

    if (ln_turbo_is_available(ctx)) {
        /* bytecode is ready; ln_normalize_to_str() runs it */
    }

For direct string output (bypassing json-c entirely)::

    char *json_str = NULL;
    size_t json_len = 0;
    int r = ln_normalize_to_str(ctx, msg, msg_len, &json_str, &json_len);
    if (r == 0 && json_str) {
        /* json_str contains the normalized JSON string */
        free(json_str);
    }

Only ``ln_normalize_to_str()``, ``ln_turbo_normalize_to_str()`` and
``ln_turbo_normalize_raw()`` execute the bytecode and honour
``LN_CTXOPT_TURBO_STRICT``. ``ln_normalize()`` always uses the recursive
walker. Comparing those two APIs with turbo enabled compares the walker
with itself and hides a turbo decline.

High-performance API (lognorm-turbo.h)
--------------------------------------

Consumers that want to avoid json-c construction entirely (for example
rsyslog's ``mmnormalize`` worker hot path) can use the curated public
header ``lognorm-turbo.h``. It is installed alongside ``liblognorm.h``
and ``lognorm-features.h`` and gated on ``LOGNORM_TURBO_SUPPORTED``::

    #include <liblognorm/lognorm-features.h>
    #if defined(LOGNORM_TURBO_SUPPORTED)
    #include <liblognorm/lognorm-turbo.h>
    #endif

The header exposes only opaque types and a function-level contract; the
internal ``turbo*.h`` headers and the fast-result/snapshot struct layouts
are **not** installed and may change between releases without affecting the
ABI. The contract covers:

- ``ln_turbo_normalize_raw()``: normalize into an opaque, context-owned
  result (valid until the next normalize call on that context). A
  truncated result is still returned, with
  ``ln_fast_result_is_truncated()`` set; the string path refuses that
  result and falls back to the walker.
- ``ln_turbo_snapshot_result()`` / ``ln_fast_result_snapshot_get()`` /
  ``ln_fast_result_snapshot_free()``: retain a result beyond the next
  call. The snapshot copies used field slots plus the result tail
  (tags, match info). Unused slots past ``n_fields`` are not copied.
  The allocation still covers a full ``ln_fast_result_t`` because
  ``ln_fast_result_snapshot_get()`` returns a pointer to that object.
- typed accessors ``ln_fast_result_field_count()``,
  ``ln_fast_result_get_field()``, ``ln_fast_result_get_field_typed()``
  (preserves ``LN_FTYPE_*`` value type and the ``LN_FFIELD_NESTED`` flag),
  ``ln_fast_result_get_string()`` / ``_get_int()``, the tag accessors and
  ``ln_fast_result_get_rule_id()``.

Supported Parsers
-----------------

TurboVM compiles every v2 parser type:

- **Text**: ``word``, ``alpha``, ``string``, ``rest``, ``char-to``,
  ``char-separated``, ``string-to``, ``op-quoted-string``,
  ``quoted-string``, ``literal``
- **Numeric**: ``number``, ``float``, ``hexnumber``
- **Network**: ``ipv4``, ``ipv6``, ``mac48``
- **Date/Time**: ``date-rfc3164``, ``date-rfc5424``, ``date-iso``,
  ``time-24hr``, ``time-12hr``, ``duration``, ``kernel-timestamp``
- **Structured**: ``json``, ``cee-syslog``, ``cef``, ``v2-iptables``,
  ``checkpoint-lea``, ``name-value-list``, ``repeat``
- **Special**: ``whitespace``, ``cisco-interface-spec``

A construct the compiler cannot express still fails the whole rulebase
(compilation is all-or-nothing). Per-message declines fall back to the
recursive walker unless ``LN_CTXOPT_TURBO_STRICT`` / ``-oturbostrict``
is set.

Known differences from the standard engine
-------------------------------------------

The JSON *document* TurboVM writes is not the one the standard engine
writes, by design:

- Dotted field names nest. ``%source.ip:word%`` gives
  ``{"source":{"ip":"..."}}`` under TurboVM and a flat ``"source.ip"`` key
  under the standard engine. This is the ECS-output feature described above,
  not an accident, but it does mean the two engines produce different
  documents for any rulebase that uses dotted names.
- Tags follow from that. TurboVM puts them at the root as ``"tags"``, the
  ECS spelling; the standard engine adds a flat ``"event.tags"`` key. Writing
  ``"event.tags"`` in the nested serializer would place the same logical path
  both inside and outside the ``event`` object whenever a rulebase also
  carries an ``event.*`` field.

Consumers of ``ln_turbo_normalize_raw()`` and the typed field accessors see
none of this: it is the string serializer only.

A few parsers keep a deliberate match with the standard engine:

- ``string-to`` with a single-character delimiter never matches. That is
  the standard parser's behaviour (its inner comparison loop cannot run
  for a delimiter shorter than two bytes) and TurboVM replicates it so a
  rulebase behaves the same on both engines.
- ``json`` field names follow the same three contracts as the standard
  parser (see configuration.rst "Special field names"): ``-`` is matched
  and discarded; ``.`` inlines object keys at the current context; a
  real name stores the value as one nested JSON object. Named
  ``%field:json%`` keeps arrays, booleans and nulls as JSON types. The
  ``.`` inline path walks the object into the turbo arena (dotted leaves,
  capped at ``LN_FAST_MAX_FIELDS``). Nested arrays on that path are kept
  as JSON arrays (``LN_FFIELD_RAW_JSON``), not flattened to ``.0`` /
  ``.1`` keys.
- The JSON scanner matches libfastjson: trailing whitespace after a
  value is consumed, single-quoted keys and strings are accepted, and
  ``true`` / ``false`` / ``null`` are case-insensitive. A named JSON
  field whose span contains a single quote is re-emitted as RFC JSON so
  the document stays valid.

Extracted field names (for example a name-value-list key) are
JSON-escaped the same way as string values. An unescaped quote in a key
would terminate the object syntax.

A rule that binds the same name twice keeps every binding, as libfastjson
does. The JSON document lists them newest first (duplicate keys).
``ln_fast_result_get_string()`` returns the last, matching
``json_object_object_get_ex``. A RFC JSON parser that last-wins on
duplicate keys therefore sees the first-added value, which is what
``json.loads`` of the walker document also returns.

Performance Notes
-----------------

Throughput improvements depend on the rulebase complexity and message
format. Typical observations:

- Simple rulebases (5-10 rules): 2-3x throughput improvement
- Complex rulebases (50+ rules with alternatives): 5-10x improvement
- The ``ln_normalize_to_str()`` path avoids json-c entirely and provides
  the highest throughput for applications that consume JSON as strings

TurboVM adds no overhead when disabled (``--disable-turbo`` or default).
Compilation is all-or-nothing per rulebase: the compiler walks one shared
parse DAG, so a construct it cannot express means the whole rulebase runs
on the recursive walker. When that happens the partially emitted program is
discarded, ``ln_turbo_is_available()`` reports false, and no per-message
work is wasted.

Limits
------

Two fixed-size structures bound what a single message can carry:

- ``LN_FAST_MAX_FIELDS`` (128) fields per result. Rulebases over CSV-shaped
  sources reach this: a PAN-OS TRAFFIC record is 97 columns before
  annotations. Exceeding it is not silent: ``ln_normalize_to_str()``
  refuses the result and falls back to the recursive walker, which has no
  field limit. Callers of ``ln_turbo_normalize_raw()`` get the partial
  result plus ``ln_fast_result_is_truncated()`` and decide for themselves.
- ``LN_FAST_MAX_TAGS`` (16) tags per result, with the same contract.

A third limit is on the rulebase rather than the message. Linear
non-terminal chains (one parser, not a terminal) compile in a loop, so a
long sequential rule does not consume one C stack frame per field.
Recursion is reserved for forks, prefix-terminals, and custom-type /
repeat bodies, and is capped at ``LN_TURBO_MAX_COMPILE_DEPTH`` (1024).
Exceeding it discards the program. Compilation is all-or-nothing, so one
such rule puts the whole rulebase on the recursive walker.
``-oturbostrict`` and the ``turbo: compilation failed`` debug line are
the way to notice.

Field names, annotation values and literal texts are *not* limited by the
size of the opcode buffers they normally live in: anything that does not
fit in the 60-byte inline slot is interned in the program string pool
(``OP_LITERAL_EXT`` for long compacted literals).
