#Regular expressions without SQLite extensions

The default compiler implements a deliberately limited MongoDB-compatible regex grammar entirely in SQL. $regex queries and $regexMatch expressions use stock SQLite capabilities, including those available to Expo. They require no loadable extension, JavaScript SQLite function, Node API, or regex package. Native PCRE2 remains an explicit optional capability for the full grammar and captures. Runtime validation covers Node and actual Expo web SQLite; native Expo device execution remains unverified.

This is a supported subset, not a translation of arbitrary regex into LIKE or GLOB. Unsupported patterns fail during compilation with SILT_UNSUPPORTED. Invalid options, NUL patterns, invalid Unicode patterns, and reversed character ranges fail with SILT_VALIDATION.

#Supported grammar

Use a literal pipe for alternation: cat|dog. The alternatives in ^cat|dog$ have their own anchors: the pattern means “starts with cat or ends with dog.” Parentheses are not supported, so write ^cat$|^dog$ for two complete-string alternatives.

Inside a class, escape literal [, ], and -. A caret negates a class only in its initial position; escaping it is also supported. Ranges compare Unicode code point values. A dot consumes one code point, not one grapheme: ^.$ matches 😀 but not e followed by a combining accent or a family emoji joined with ZWJ.

$ follows MongoDB/PCRE2 behavior: without m, it matches at the end or immediately before one final LF. Consequently, ^cat$ matches both "cat" and "cat\n", but not "cat\n\n". CR, CRLF, NEL, and Unicode line/paragraph separators receive no additional newline treatment. With m, $ also matches before internal LF and ^ matches after LF when another code point follows. The initial position remains valid for ^, including an empty subject.

The portable matcher rejects i (case folding), x (extended mode), groups/captures, lookaround, backreferences, inline options, counted repetition, lazy/possessive quantifiers, shorthand classes such as \d, \w, and \s, Unicode properties, word boundaries, POSIX classes, and other escapes. Use [0-9] for ASCII digits and explicit classes for the intended alphabet. Enabling nativeRegex delegates all patterns to the optional PCRE2 extension; the default never silently falls back to another engine.

General patterns allow at most 64 alternatives, 256 consuming atoms across all branches, and 256 class members/ranges across all classes. Pure literal/anchor alternatives without m have the same 64-alternative limit but no 256-atom limit because they use the literal fast path. Exceeding these limits is an explicit unsupported-pattern error.

#Queries and expressions

import { compileFind, compileAggregate } from '@silt-db/compiler';

compileFind('users', {
  username: { $regex: '^[a-zA-Z][a-zA-Z0-9_]*$' },
});

compileFind('events', {
  message: { $regex: '^error:.*$', $options: 'm' },
});

compileAggregate('users', [
  {
    $project: {
      validUsername: {
        $regexMatch: {
          input: '$username',
          regex: '^[a-zA-Z][a-zA-Z0-9_]*$',
        },
      },
    },
  },
]);

Query matching retains MongoDB field traversal, array-element matching, $elemMatch, $not, and $expr behavior. Nonstring query values do not match. $regexMatch requires a string input and fails for missing/null/nonstring inputs.

Portable $regexMatch requires a compile-time string pattern and compile-time string options. Plain literal strings and { $literal: '...' } work. Patterns beginning with $ need $literal because ordinary aggregation strings beginning with $ are field references. Dynamic field/computed patterns, $regexFind, and $regexFindAll require the optional native extension. The portable API continues to accept string patterns in JSON, not JavaScript RegExp objects.

#Optional native PCRE2

Choose the native extension on Node when your queries need case-insensitive flags, captures, lookaround, dynamic expression patterns, or throughput beyond the portable SQL matcher. It executes PCRE2 inside SQLite and leaves document matching and traversal in the SQL compiler. It is not required by the default packages and is not loaded by Expo.

Build from the repository root with a C11 compiler and the PCRE2 8-bit development library installed:

npm run build:native

Linux output is packages/native/build/silt_regex.so; macOS output is packages/native/build/silt_regex.dylib. CC, PCRE2_INCLUDE_DIR, and PCRE2_LIB_DIR can select the toolchain. There is no install hook, automatic download, or second SQLite engine. Rebuild for each deployment platform; Windows has not been validated. The extension links the system PCRE2 library, so syntax support follows that linked version.

import { openDatabase } from '@silt-db/sqlite/better-sqlite3';

const db = await openDatabase('app.sqlite', {
  regexExtension: '/absolute/path/to/silt_regex.so',
});
const users = await db
  .collection('users')
  .find({
    name: { $regex: '^a\\w+$', $options: 'i' },
  })
  .toArray();

For a supplied node:sqlite connection, allow extension loading when creating the native handle; disable further loading after installing the extension. Standalone compilation with nativeRegex: true only declares that the execution connection already has the functions. It does not load a binary.

Native mode supports flags i, m, s, x, and u, with UTF-8 always enabled. Unicode character properties are not enabled by default for word classes; PCRE2 (*UCP) is a separate opt-in. $regexFind/$regexFindAll return { match, idx, captures } values, with code-point indices and null for unmatched optional captures.

Silt-specific limits include 1 MiB pattern text, 256 nested parentheses, 1,000,000 match steps, depth 1,000, a 16 MiB match heap, and 100,000 capture-all matches. Exceeding a limit aborts the operation. Invalid PCRE patterns fail when evaluated, but an empty input or a short-circuited predicate may not evaluate them. SELECT silt_regexp_validate(pattern, flags) provides explicit validation. Full MongoDB error timing and resource-limit parity are not promised.

Build requirements, SQL function signatures, licensing, and further native tests are in the native package README. JavaScript coverage does not measure C branch coverage.

#SQLite and text requirements

Use a UTF-8 SQLite database (the default for new SQLite databases). An inexpensive constant SQL check rejects UTF-16 database encoding instead of interpreting its bytes as UTF-8. A rejected encoding produces the compiler's usual SQLite expression failure (malformed JSON), with the UTF-8 requirement visible in the generated error branch.

Subjects must be well-formed Unicode strings. Embedded NUL in subjects is supported, including matching after NUL, negated classes containing NUL, and dots consuming NUL. Patterns containing literal NUL remain rejected, matching the existing query contract. The matcher uses BLOB slices and UTF-8 code point boundaries to avoid SQLite TEXT length/substr truncation at NUL. Pattern text, literals, and automaton transitions are SQL parameters.

The SQL uses ordinary JSON functions, recursive CTEs, and AS MATERIALIZED (SQLite 3.35+); the surrounding compiler requires the newer JSON SQL features documented in the project contract. It does not require SQLite's optional regexp function. The implementation was executed on stock SQLite 3.53.3; native device/browser availability still depends on the SQLite version supplied by the embedding application.

#Cost and verification

Literal substring, prefix, suffix, and literal alternatives use direct SQLite BLOB instr/substr/equality operations. General patterns compile to an automaton carried by a bound JSON parameter. A recursive CTE visits each (automaton state, byte offset) at most once using UNION deduplication, avoiding exponential regex backtracking. Fully anchored patterns omit the unanchored search loop.

General matching still incurs SQL recursion, transition materialization, deduplication, and class-range lookup overhead for each candidate string. State count and subject length determine the visited-state bound; large classes add range-check work. Regex is a residual predicate and does not itself create an index. Large collections benefit from indexed non-regex filters that reduce candidate documents. Use native PCRE2 when advanced syntax or high regex throughput justifies a custom build.

A local diagnostic on SQLite 3.53.3 ran 1,000 short strings of the form prefix-123-café-😀 or wrong-124-café-😀, using prepared statements and taking the median of five passes. These are runtime-specific observations, not Expo device performance guarantees:

test/portable-regex.test.js contains:

• Stock-SQLite execution and query traversal/expression checks, including NUL, astral code points, combining marks, escaping, empty matches, and LF rules.
• Explicit rejection and UTF-16 encoding checks, plus a repeated-state no-match regression.
• 13,005 PCRE2 comparisons: 51 patterns × 5 option combinations × 51 subjects (when the optional extension is built).
• The same 13,005 comparisons against real MongoDB query results and another 13,005 against MongoDB $regexMatch results (when MONGODB_URI is supplied).

node --test test/portable-regex.test.js
MONGODB_URI=mongodb://127.0.0.1:27017 node --test test/portable-regex.test.js
# Or use an already installed binary with an isolated temporary database:
MONGODB_BINARY=/absolute/path/to/mongod node scripts/with-mongo.mjs --test test/portable-regex.test.js

Reference semantics: MongoDB $regex, MongoDB $regexMatch, and SQLite recursive CTE deduplication.