go-kata/02-performance-allocation/11-ndjson-stream-reader/README.md

Kata 11: The NDJSON Reader That Survives Long Lines

Target Idioms: Streaming I/O (io.Reader), bufio.Reader vs Scanner, Handling ErrBufferFull, Low Allocation
Difficulty: 🟡 Intermediate

🧠 The "Why"

Seasoned devs reach for bufio.Scanner and it “works”… until production sends a line > 64K and you get: bufio.Scanner: token too long.

This kata forces you to implement a streaming reader that can handle arbitrarily large lines without falling over.

🎯 The Scenario

You ingest NDJSON logs from stdin or a file. Lines can be huge (hundreds of KB). You must process line-by-line.

🛠 The Challenge

Implement:

• func ReadNDJSON(ctx context.Context, r io.Reader, handle func([]byte) error) error

1. Functional Requirements

• Call handle(line) for each line (without the trailing newline).
• Stop immediately on handle error.
• Stop immediately on ctx.Done().

2. The "Idiomatic" Constraints (Pass/Fail Criteria)

• Must NOT rely on default bufio.Scanner behavior.
• Must use bufio.Reader and correctly handle ReadSlice('\n') returning ErrBufferFull.
• Must avoid per-line allocations where possible (reuse buffers).
• Wrap errors with line number context using %w.

🧪 Self-Correction (Test Yourself)

• If a 200KB line crashes with “token too long”: you failed.
• If cancellation doesn’t stop promptly: you failed.
• If you allocate a new buffer each line: you failed the low-allocation goal.

📚 Resources

• https://pkg.go.dev/bufio
• https://pkg.go.dev/io