bufioScanner

bufio.Scanner is a convenience type in Go’s standard library (package bufio) that provides a simple way to tokenize input read from an io.Reader. It repeatedly returns tokens, as defined by a split function, making it useful for line-by-line or word-by-word processing of streams such as files, standard input, or network data.

The typical usage pattern is to create a scanner with NewScanner, loop with Scan to advance tokens, and retrieve the token with Text or Bytes. For example, scanning lines is common: for scanner.Scan() { line := scanner.Text() … }. After the loop, any error encountered during scanning should be checked with scanner.Err(). The default split function is ScanLines, which yields line-delimited tokens and handles end-of-line markers appropriately.

Split functions determine how data is tokenized. The package provides several built-in splitters, including ScanLines (default), ScanWords, and ScanRunes. You can also supply a custom Split function to customize token boundaries. The choice of splitter affects what constitutes a token and when a token is returned.

Buffer size and token length are important considerations. The scanner has a default token size limit of about 64K. To process longer tokens, increase the buffer size using scanner.Buffer(buf, maxCapacity), where maxCapacity specifies the maximum token length. The first argument is a reusable buffer that the scanner will use to store data between calls.

Notes: Text returns a string that references the scanner’s internal buffer, which may be overwritten by the next Scan. If you need to retain tokens beyond the next scan, copy the data. For large or binary data, or when tokenization requirements exceed what Scanner is comfortable with, other approaches such as bufio.Reader may be more appropriate.