Log parsers silently break the moment upstream microservices update their logging schemas. ParseForge is a backend system that automates log parser repair by pairing an LLM agent with a restricted execution sandbox and an objective evaluation pipeline operating in a closed feedback loop.
In modern distributed infrastructure, log ingestion pipelines depend heavily on structured log parsing. However, parsers are notoriously brittle. A subtle change in upstream log output—an added timestamp key, a renamed field, an extra JSON envelope, or a multi-line exception stack trace—causes existing parsing routines to fail.
Traditionally, resolving a broken log parser requires an engineer to inspect raw log streams, debug stack traces, and hand-craft revised regular expressions or string manipulation code. This manual process is time-consuming and introduces operational risk into production telemetry pipelines.
Asking a standard conversational AI to "fix this parser" relies on the model's self-reported confidence. Without executing the generated code against actual ground-truth data, an LLM cannot verify if its proposed fix parses edge cases accurately or introduces silent extraction bugs.
ParseForge shifts the paradigm from conversational prompting to an autonomous closed feedback loop. Instead of trusting LLM completions outright, ParseForge treats parser repair as an environment interaction:
Driven by an explicit state machine API, tracking attempt budgets, parser code state, runtime tracebacks, and stagnation conditions.
Candidate code proposed by the agent is compiled and executed inside a restricted Python sandbox with strict module allowlists and thread time-budgeting.
Parsed results are scored against ground-truth datasets using granular metrics. The LLM is never allowed to judge its own correctness.
The system is designed with a strict single-responsibility separation along trust boundaries. Code execution, scoring, agent decisions, and state orchestration operate independently.
Owns episode state, step counters, and state-consistency verification. Coordinates sandbox execution and evaluation without coupling to LLM logic.
Manages conversation history and model fallback chains. Implements resilient output extraction to recover structured actions from raw completions.
Safely executes untrusted candidate parsers. Restricts builtin functions, filters imported modules, and enforces time limits on execution.
Pure mathematical scoring module that computes granular accuracy, field coverage, and key-value pair precision and recall.
Every repair attempt follows a deterministic 5-stage lifecycle. If candidate parser code fails or underperforms, the exact execution error or missing field diagnostics are fed back into the next attempt.
Before calling the agent, the environment evaluates the initial or current parser against sample raw log lines to identify crashing lines, missing schema fields, or structural mismatches.
The LLM agent receives an observation containing the raw log sample, current parser code, target schema expectations, and previous runtime stack traces, proposing a revised replacement parser.
The proposed Python function is compiled and executed line-by-line across all sample logs inside the sandbox. Unhandled exceptions or syntax errors are caught gracefully.
Extracted records are compared against ground-truth definitions. Metric scores are computed to determine structural validity, field coverage, and extraction precision.
The step loop evaluates termination criteria: deployment threshold success, attempt budget exhaustion, or stagnation detection. If unfulfilled, feedback loops to step 1.
To ensure correctness without human intervention, ParseForge replaces subjective scoring with a weighted composite reward derived from three complementary metrics:
Measures structural stability—the ratio of log lines where the parser executed successfully and returned a valid, non-empty dictionary without throwing exceptions.
Measures schema recall—the average proportion of required ground-truth fields successfully extracted and normalized per log entry.
The strictest metric—micro-averaged precision and recall over all key-value pairs. Crucially penalizes extra, hallucinated, or incorrectly parsed fields.
ParseForge includes a diverse suite of benchmark tasks covering real-world log parsing failure modes:
Extending basic regex patterns to extract user identifiers, HTTP methods, URIs, and response sizes from web server access streams.
Handling complex quoted header fields, HTTP referers, user-agent strings, and dash placeholders in combined log lines.
Repairing broken JSON log streams containing trailing commas, unquoted enum keys, or single-quoted string values.
Parsing application logs with multi-line Java exception stack traces requiring stateful line aggregation.
Two-stage parsing: unwrap JSON container envelopes then parse embedded bracketed timestamp and log level strings.
Detecting and routing heterogeneous log streams (syslog, Apache, JSON) to line-specific parsing routines.
Key architectural principles that ensure system reliability, security, and maintainability:
Restricts untrusted LLM-generated code by stripping dangerous builtins and enforcing module allowlists during execution.
Employs extraction strategies to reliably recover code actions from LLM completions while preserving raw regex metacharacters.
Guarantees that syntax errors, execution exceptions, or model parsing failures never crash the orchestration loop.
Eliminates model self-evaluation. All deployment decisions rely strictly on code-verifiable metrics against ground truth.
Maintains strict acyclic separation between environment state, agent reasoning, sandbox execution, and metric evaluation.
Verified by an extensive offline unit and integration test suite covering environment state transitions and scoring math.
Explore the complete source code, test suite, and benchmark task definitions on GitHub: