Skip to content

The BOM is kept unless you ask

Level: 101 · anyone reading a CSV that came out of a spreadsheet

One line: Ruby reads a UTF-8 byte order mark as an ordinary first character, U+FEFF, so the text does not start_with?("id") and a CSV's first header is not "id"; encoding: "BOM|UTF-8" removes it, and set_encoding_by_bom detects it on a file opened in binary mode.

A UTF-8 byte order mark is the three bytes EF BB BF at the start of a file. UTF-8 has no byte order to mark, but spreadsheet exports write one anyway — the Encodings library's A BOM in a CSV ↗ has the details. The mark is invisible in an editor and in puts, which is what makes it expensive.

Verified output of the_bom_is_kept_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

on disk, first 8 bytes: efbbbf69642c6e61

File.read(path, encoding: "UTF-8")
  first code points      U+FEFF U+0069 U+0064
  start_with?("id")      false
File.read(path, encoding: "BOM|UTF-8")
  first code points      U+0069 U+0064 U+002C
  start_with?("id")      true

CSV.read(path, encoding: "UTF-8", headers: true).first
  first header           U+FEFF U+0069 U+0064
  row["id"]              nil
CSV.read(path, encoding: "BOM|UTF-8", headers: true).first
  row["id"]              "7"

File.open(path, "rb") then f.set_encoding_by_bom
  returns                #<Encoding:UTF-8>
  then f.gets            "id,name\n"
the same on a file with no mark
  returns                nil
  then f.gets            "id,name\n"

Reading the output

  • Plain encoding: "UTF-8" keeps the mark as U+FEFF, the first character of the string, so start_with?("id") is false.
  • CSV inherits it. The first header is U+FEFF followed by id. It prints exactly like id, and row["id"] is nil — not an exception, a missing value.
  • BOM|UTF-8 removes a mark if there is one and reads UTF-8 either way. It works in File.read, in File.open modes ("r:BOM|UTF-8") and in CSV.read.
  • set_encoding_by_bom on a handle opened with "rb" consumes a mark and returns the encoding it indicates, or returns nil and consumes nothing.

What to do

Read every CSV, and every text file that may have come from Windows, with BOM|UTF-8. On a file without a mark it costs nothing.

If you are coming from another language

Python's utf-8 codec keeps the mark too; utf-8-sig is the codec that strips it:

Verified output of the_bom_is_kept_py.py — regenerated by tools/run_examples.py, never hand-typed.

  encoding='utf-8'      first header starts U+FEFF, row.get('id') is None
  encoding='utf-8-sig'  first header starts U+0069, row.get('id') is '7'

Java keeps it as well — the Java text library's The BOM is not stripped ↗. The Encodings library's Byte order and the BOM ↗ explains what a byte order mark is for in the encodings that do have a byte order.