CSV Is a State Machine

Parsing and repairing messy imports in the browser without hiding data loss.

A CSV file looks like lines split by commas until a customer pastes an address containing a comma, an export includes a quoted newline, or a spreadsheet changes the delimiter for a different locale. At that point, a convenient text.split('\n').map(row => row.split(',')) is not merely incomplete. It silently changes the table.

This article develops a small browser-only preflight pipeline. It detects common encoding failures, chooses a likely delimiter, parses quoted records with a state machine, reports structural problems, and produces normalized CSV. The important design choice is that diagnosis and repair are separate. A tool should say what it inferred, what it changed, and what remains ambiguous.

Start with a narrow contract

“Support CSV” is too vague to test. The implementation accepts UTF-8 text with an optional byte-order mark and recognizes comma, tab, semicolon, or pipe delimiters. It supports quoted fields, escaped quotes, and newlines inside quoted fields. It reports empty or duplicate headers, rows with unexpected column counts, and duplicate data rows.

It deliberately does not guess character encodings beyond surfacing UTF-16 markers, stream multi-gigabyte files, infer column types, or decide which duplicate row is authoritative. Those are separate product decisions with different failure modes. Keeping the boundary explicit prevents a normalizer from becoming a quiet data-loss machine.

Gate bytes before parsing text

A browser File can be read as an ArrayBuffer. That is the right moment to inspect byte signatures. Once malformed bytes have already been decoded with replacement characters, the parser cannot reconstruct what was lost.

export function detectEncoding(bytes) {
  if (bytes[0] === 0xef && bytes[1] === 0xbb && bytes[2] === 0xbf)
    return { encoding: "UTF-8 BOM", offset: 3, supported: true };
  if (bytes[0] === 0xff && bytes[1] === 0xfe)
    return { encoding: "UTF-16 LE", offset: 2, supported: false };
  if (bytes[0] === 0xfe && bytes[1] === 0xff)
    return { encoding: "UTF-16 BE", offset: 2, supported: false };
  try {
    new TextDecoder("utf-8", { fatal: true }).decode(bytes);
    return { encoding: "UTF-8 (no BOM)", offset: 0, supported: true };
  } catch {
    return { encoding: "Unknown / not valid UTF-8", offset: 0, supported: false };
  }
}

The fatal option matters. Without it, TextDecoder may substitute the Unicode replacement character and make a damaged identifier look valid. The browser API behavior is documented by MDN. Rejecting UTF-16 is not a claim that UTF-16 is invalid; it is an honest statement that this small tool has no conversion contract for it.

Delimiter detection is a ranking problem

A delimiter guess should be evidence that the parser exposes, not a fact it pretends to know. A useful lightweight heuristic samples up to twenty non-empty physical lines and scores each candidate by consistency and useful width. A candidate that produces the same number of fields repeatedly is stronger than one that appears often but produces erratic widths.

Physical-line sampling has a known weakness: a quoted field can span lines. A line inside that field may contain commas that are content, not separators. The production parser handles that case correctly, but this cheap detector can still rank the wrong delimiter. The UI therefore displays the inferred delimiter before the user downloads normalized output. The current prototype does not yet provide an override, so ambiguous files should not be normalized.

General rule: if an inference can change the table shape, surface it as a choice. Confidence hidden inside a helper function is not a user contract.

Parse characters with explicit state

The parser only needs a few states, but they must be explicit. Track the current field, current row, whether the cursor is inside a quoted field, and whether the last character was a carriage return. Each character then has a bounded meaning.

for (let i = 0; i < text.length; i += 1) {
  const char = text[i];
  if (insideQuotes) {
    if (char === '"' && text[i + 1] === '"') {
      field += '"';
      i += 1;
    } else if (char === '"') {
      insideQuotes = false;
    } else {
      field += char;
    }
    continue;
  }
  if (char === '"' && field.length === 0) insideQuotes = true;
  else if (char === delimiter) finishField();
  else if (char === '\n') finishRow();
  else if (char !== '\r') field += char;
}

Inside quotes, a delimiter is ordinary data and a newline belongs to the field. Two consecutive quote characters represent one literal quote. Outside quotes, the delimiter finishes a field and the record separator finishes a row. A quote that begins after unquoted content is not automatically treated as an opening quote; doing so would reinterpret malformed data.

The parser also has to define end-of-file behavior. It flushes the final field and row even when the file does not end with a newline. If the file ends while still inside quotes, it reports an unclosed-quote error. Returning partial rows while hiding that error would make downstream validation misleading.

RFC 4180 is a useful reference for the common quoted-field shape, but real exports vary. The implementation treats its supported dialect as a local contract rather than claiming universal CSV compliance.

Validate structure before values

Once rows exist, structural checks are cheap and explainable. The first row is the header. Trimmed empty names are reported by column position. Duplicate headers are compared after trimming because two visually identical names should not create competing keys. Every later row is compared with the header width.

Duplicate-row detection uses a collision-safe representation rather than joining values with a delimiter that may itself occur in a field. Serializing the row array to JSON is sufficient for this bounded client-side tool. A streaming production importer would likely use an incremental hash plus collision confirmation to avoid retaining every row.

These checks should produce issues, not automatic deletions. A duplicate may be a legitimate repeated transaction. A short row may encode omitted trailing values, or it may be corruption. Preflight can show the evidence and preserve the original row number; only an importer-specific policy can decide what the data means.

Normalization must be lossless within the contract

Serialization is simpler than parsing but still needs one invariant: parsing the normalized output must recover the same field strings. Quote any field that contains the selected delimiter, a quote, carriage return, or newline. Inside a quoted field, double every quote.

function escapeField(value, delimiter) {
  const text = String(value);
  if (![delimiter, '"', '\r', '\n'].some(token => text.includes(token))) return text;
  return `"${text.replaceAll('"', '""')}"`;
}

The normalizer standardizes record separators and quoting style, and replaces empty or repeated header names with unique placeholders. It does not trim data cells, coerce dates, remove duplicates, pad short rows, or truncate long rows. Those changes can alter meaning. The accompanying error report lists each detected problem so a human or domain-aware policy can make the next decision.

Test counterexamples, not just happy paths

A parser test suite earns its keep with named counterexamples. The current suite covers a multiline quoted field with an escaped quote, semicolon detection, empty and duplicate headers, uneven row widths, duplicate rows, round-trip quoting, UTF-8 BOM removal, UTF-16 markers, and invalid UTF-8.

const rows = parseCsv('id,note\n1,"line one\nline two: ""ok"""', ',');
assert.deepEqual(rows, [
  ['id', 'note'],
  ['1', 'line one\nline two: "ok"']
]);

This test fails immediately with line splitting, naive comma splitting, or incorrect escaped-quote handling. By contrast, a test containing only a,b\n1,2 can pass many broken parsers. Good fixtures distinguish implementations that preserve the contract from those that merely work on a demo.

Publish the uncomfortable benchmark

Correctness does not make this small parser automatically fast. A recorded local run parsed a deterministic 2,667,802-byte file containing 100,001 simple rows. On Node.js 22.23.2 and an Apple M3 Pro, eleven measured runs after three warmups produced a median of 24.49 ms for the state machine and 15.86 ms for naive line-and-comma splitting. On that deliberately easy input, the correct parser was 1.54 times slower.

The faster implementation is still not interchangeable. Both outputs match on the simple benchmark file, but the same naive splitter fails the checked fixture containing a quoted comma and quoted newline. The useful conclusion is bounded: on this machine and input, correctness costs measurable CPU time, while naive speed buys a parser that changes valid records. It is not a universal browser-performance claim.

The benchmark rig generates the input, asserts both implementations agree before timing, runs the named counterexample, and records the median rather than the fastest result. The checked-in result file includes the runtime, CPU, byte count, run count, and exact measurements. Reproduce it with npm run benchmark; use npm run benchmark:record only when intentionally replacing the recorded evidence.

Keep files on the device by construction

The browser application reads the selected file into memory, analyzes it locally, and creates downloads with Blob and object URLs. It has no upload endpoint, analytics script, external font, or runtime dependency. The data flow is inspectable: input bytes enter through the file picker or paste event and output leaves only through a user-triggered download.

This is “local-only,” not a broad security guarantee. The prototype has not completed a full content-security-policy audit, worker isolation, memory-pressure testing, or cross-browser accessibility review. Large files are processed in one allocation and can freeze a tab. A production version should move parsing to a Web Worker, add cancellation and size limits, and stream records where browser APIs permit.

A practical preflight sequence

  1. Read bytes and reject encodings outside the declared contract.
  2. Rank delimiter candidates and show the selected value.
  3. Parse with an explicit quoted-field state machine.
  4. Stop on syntax errors such as an unclosed quote.
  5. Report header, width, and duplicate-row issues with locations.
  6. Serialize fields without semantic coercion.
  7. Offer normalized CSV and a separate issue report.

The result is intentionally modest. It does not make bad data good. It makes the parser's assumptions visible, catches errors that would otherwise shift columns silently, and preserves enough evidence for a human to choose the right repair.

The complete dependency-free implementation and tests are available in the CSV Preflight repository, and the live tool runs entirely in the browser.