The NUL byte¶

Level: 201 · working knowledge

One line: U+0000 is an ordinary character that encodes as one ordinary byte and passes every UTF-8 validator — and almost nothing that carries text will carry it, which is why the same zero byte is a string terminator, a binary-file tripwire, the one separator you can trust, and a ValueError at four different doors.

Control characters introduces NUL as the first of the first 32 codes and shows what strlen does with it. This page is about that one byte on its own, because it turns up in more places than any other character in this library and it is doing something different in each of them.

One byte, five jobs¶

Where you meet it	What the zero byte means there	Written up in
C, and everything with a C interface underneath	the string ends here	Control characters
`grep`, `rg`, `git diff`, `file`	this is not text — stop showing it	`grep` on text that is not ASCII
`find -print0`, `xargs -0`, `sort -z`	the record ends here — the one separator a filename cannot contain	`find`, and filenames that are bytes
A certificate, a log line, a path	somebody else's punctuation, inside your value	The byte that means something to somebody else
Python, Rust, Java, any `str`	a character like any other	this page

Those five are not a miscellany, and the last row is the reason. NUL is the only byte that every text encoding says is fine and half the world's text APIs cannot represent. UTF-8 encodes it in one byte; a validator accepts it; a str holds it; JSON has a spelling for it. Then it reaches C — or anything whose interface is C, which is the operating system, the filesystem, most database drivers and every argv — and there it is not data at all, it is the mark that says stop reading.

So the same value is content on one side of a call and structure on the other, and every bug on this page lives in that crossing. That is also why the fix is never "escape it": there is nothing to escape it to. The layer below is not parsing your string, it is measuring it.

Nothing rejects it, which is the problem¶

It is tempting to file NUL with the malformed bytes — the overlongs, the lone surrogates, the F5 above U+10FFFF. It does not belong there. str::from_utf8(&[0]) is Ok. b'\x00'.decode('utf-8') returns a string. Validation is a real boundary and this byte walks straight through it, so the check that stops a NUL is never the encoding check — it is a separate rule, imposed by whoever owns the next layer, and each of them imposes a different one.

One system took the other way out. Java's Modified UTF-8 encodes U+0000 as the overlong C0 80 precisely so that a NUL never appears as a zero byte inside a string, which keeps the surrounding C happy — at the price of no longer being UTF-8. That trade, and what it costs, is on Overlong sequences.

Why it is the one safe separator¶

A separator has to be a byte the data cannot contain. Newline fails that test for filenames — a newline in a filename is perfectly legal, which is why find … | wc -l counts lines and answers a question about files that nobody asked.

On a Unix system a pathname component may hold any byte at all except two: /, which is busy separating directories, and NUL. That leaves exactly one candidate, and it is not a convention anybody chose. It falls out of execve(2) ↗: the kernel takes its arguments as NUL-terminated strings, so a filename with a NUL in it could not be passed to a program even if a filesystem agreed to store one. The byte that ruins a C string is the byte that makes a safe record separator, for the same reason, and -print0 / -0 / -z / --null-data are all one idea wearing four spellings.

In Python¶

Verified output of the_nul_byte_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. U+0000 IS AN ORDINARY CHARACTER, AND ORDINARY UTF-8
   ord(nul)                        = 0
   category is 'Cc' (a control)    = True
   nul.encode('utf-8')             = b'\x00'   one byte, no escape hatch needed
   b'\x00'.decode('utf-8')         = '\x00'   every validator accepts it
   nul.encode('utf-16-le')         = b'\x00\x00'
   unicodedata.name(nul)            -> ValueError: no such name
   unicodedata.lookup('NULL')      = '\x00'   it has no Name, but it has an alias

2. PYTHON HOLDS IT WITHOUT COMPLAINT
   s = 'ab\x00cd'
   len(s)                          = 5   five characters; NUL is the third
   s.encode().decode() == s        = True
   s.split(chr(0))                 = ['ab', 'cd']
   sorted(['b', nul, 'a'])         = ['\x00', 'a', 'b']   it sorts before everything
   Nothing above is special-cased. To Python it is a character like any other.

3. FOUR PLACES PYTHON STOPS YOU, AND ALL FOR THE SAME REASON
   open('a\x00b')                   -> ValueError: embedded null byte
   os.stat('a\x00b')                -> ValueError
   subprocess.run(['echo', s])      -> ValueError: embedded null byte
   os.environ['A\x00B'] = 'x'       -> ValueError: embedded null byte
   Each of those hands the string to the operating system, whose interface is
   NUL-terminated C strings. Python refuses rather than let the value be cut.
   os.stat is the one printed without its message, because CPython words
   it differently per platform: macOS says 'stat: embedded null character
   in path', Linux says 'embedded null byte'. Same refusal, same class,
   two sentences -- so the class is the part that may go in a key.

4. THE CONTAINERS: WHO WILL CARRY A NUL?
   json.dumps(s)                   = "ab\u0000cd"   escaped, and legal JSON
   json.loads(that) == s           = True
   json.loads('"a\x00b"')           -> JSONDecodeError: Invalid control character at: line 1 column 3 (char 2)
   (a RAW control character inside a JSON string is invalid; the escape is not)
   csv round-trip                  = True   csv carries it and says nothing
   ET.tostring(...)                = b'<a>ab\x00cd</a>'
   ET.fromstring(that)              -> ParseError: not well-formed (invalid token): line 1, column 5
   ET.fromstring('<a>&#0;</a>')     -> ParseError: reference to invalid character number: line 1, column 3
   XML 1.0 has no way to spell U+0000 at all -- not as a raw byte and not as a
   character reference -- so Python's writer produced a document that Python's
   own parser refuses. Written, and unreadable.
   sqlite: value comes back whole   = True
   sqlite: length(s) says           = 2   ...for a five-character string
   sqlite: hex(s) says              = 6162006364   all five bytes are stored
   The row is intact; SQL's own string function stopped at the NUL. That is
   sizeof against strlen again, one layer up, inside a database.

5. WHY IT IS THE ONE SAFE SEPARATOR
   two filenames, one with a newline in it: ['holiday\nphotos.txt', 'notes.txt']
   split on '\n'  -> ['holiday', 'photos.txt', 'notes.txt']   three names, and only two files
   split on '\0'  -> ['holiday\nphotos.txt', 'notes.txt']   right, because a name cannot hold a NUL
   A separator has to be a byte the data cannot contain. A Unix filename may
   hold any byte but two -- '/' and NUL -- and '/' is busy separating directories.

Section 4 is the one to keep. ET.tostring wrote a raw NUL into the XML, and ET.fromstring refuses to read the bytes it just produced — Python's own writer against Python's own parser, disagreeing about the same document. XML 1.0 has no spelling for U+0000 at all: it is outside the Char production ↗, so not even  is legal, and a serializer that emits one has produced a file rather than a document. This is the write/read asymmetry that Byte order and the BOM meets in a different costume, and the rule is the same: the format you can write is not always the format you can read back.

And SQLite is the C lesson one layer up. The row comes back whole — hex(s) shows all five bytes — while length(s) answers 2, because SQL's own string function is a C string function underneath. Nothing was lost, and nothing you compute in SQL about that column is right.

In the terminal¶

Verified output of the_nul_byte_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

1. A NUL TRAVELS THROUGH A FILE AND A PIPE JUST FINE

$ printf 'a\0b' | xxd
00000000: 6100 62                                  a.b

$ printf 'a\0b' | wc -c | tr -d ' '
3

2. BUT IT CANNOT SURVIVE A SHELL VARIABLE

$ v=$(printf 'a\0b'); printf %s "$v" | xxd
00000000: 6162                                     ab
   Two bytes, not three. The NUL did not survive the assignment, and neither
   would it survive being passed to a command: argv is NUL-terminated strings
   all the way down to execve(2), so no program can ever receive one.

3. WHICH IS EXACTLY WHY THE -0 AND -z FLAGS EXIST
   Two files. One of them has a newline in its name, which is legal.

$ printf 'holiday\nphotos.txt\nnotes.txt\n' | xargs -n1 echo ' item:'
 item: holiday
 item: photos.txt
 item: notes.txt
   Three items, two files. Now the same two names, NUL-separated:

$ printf 'holiday\nphotos.txt\0notes.txt\0' | xargs -0 -n1 echo ' item:'
 item: holiday
photos.txt
 item: notes.txt

4. THE FAMILY OF FLAGS THAT MEANS 'NUL-SEPARATED'

$ printf 'pear\0apple\0' | sort -z | xxd
00000000: 6170 706c 6500 7065 6172 00              apple.pear.
   find -print0    xargs -0        sort -z         grep -z
   tar --null -T -                 rg --null-data  read -d ''
   All one idea: use the byte the data cannot hold.

5. ON REAL FILES, WITH A REAL find

$ find . -type f | wc -l | tr -d ' '
3
   Two files, and 'how many lines' answered three.

$ find . -type f -print0 | tr -dc '\0' | wc -c | tr -d ' '
2
   Counting NULs instead of lines answers two, which is the number of files.

Section 2 is the shell's whole relationship with this byte: a NUL is fine in a file and fine on a pipe, and cannot exist in a variable, because bash strings are C strings. Two things follow that are easy to get wrong. Command substitution does not fail — it silently returns the bytes before the NUL, so v=$(…) is the quietest truncation in this library; bash 5 warns on stderr and bash 3.2, which is what macOS ships, says nothing at all, so the example closes stderr to keep the two machines' output identical. And there is no flag to fix it: the value cannot reach a program as an argument either, so a shell pipeline that must carry NULs has to keep them on the pipe from end to end. That is what the -0 family is for.

In Rust¶

Verified output of the_nul_byte_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

1. '\0' IS AN ORDINARY char, AND ORDINARY UTF-8
   char::from_u32(0)            = Some('\0')
   nul as u32                   = 0
   nul.len_utf8()               = 1   one byte, like any ASCII character
   str::from_utf8(&[0])         = Ok("\0")
   nul.is_control()             = true
   No validator on this page rejects it. Being valid is not the problem.

2. A str HOLDS IT WITHOUT COMPLAINT
   s = "ab\0cd"
   s.len()                      = 5   bytes
   s.chars().count()            = 5   characters
   s.split('\0')                = ["ab", "cd"]

3. THE BOUNDARIES WHERE RUST STOPS YOU
   CString::new(s)              -> Err, nul_position = 2
   File::open("a\0b")           -> Err, kind = InvalidInput
   Rust refuses at the door rather than hand C a string it will cut short.
   Python raises ValueError at the same two places, for the same reason.

4. READING A C STRING BACK OUT OF A BUFFER
   buffer -- 16 bytes of a fixed-width record field:
     6e 61 6d 65 2e 74 78 74 00 00 00 00 00 00 00 00
   CStr::from_bytes_until_nul   -> "name.txt", 8 bytes
   The padding is still in the buffer. The NUL is where the value ends.

5. NUL AS THE SEPARATOR
   split on b'\0'               -> ["holiday\nphotos.txt", "notes.txt"]
   Two names, and the first one contains a newline. That is what
   `find -print0` produces and what any -0 reader has to expect.

CString::new refusing with a nul_position is the same event as Python's ValueError, moved into the type system: the boundary to C is a conversion, the conversion is fallible, and you cannot forget to handle it. File::open gives an InvalidInput error at runtime for the same reason a level down (on the machine and toolchain this page was written with, its message reads "file name contained an unexpected NUL byte", which is worth recognising but is std's wording, not an API promise).

Section 4 is the job nobody names. CStr::from_bytes_until_nul is what you reach for when a fixed-width record hands you sixteen bytes and only the first eight are the value — a COBOL extract, a binary log, an xstring from an RFC. The padding is real, the NUL is the length, and reading it back out is the honest inverse of everything else on this page.

If you are coming from Python or ABAP¶

Python. A str holds a NUL, so Python is happy right up to the last possible moment and then raises at the door — open, os.stat, subprocess, os.environ, all ValueError. Take two habits from that. First, if a value is going to become a path, an argument or an environment variable, the place to reject a NUL is where you accept the value, not where you use it, or the traceback will point at code that did nothing wrong. Second, csv will carry one and say nothing, and sqlite3 will store one and then miscount it, so a NUL that arrives in your data can travel a long way before anything complains — Python text in practice is the wider version of that argument.

ABAP. An ABAP string is internally UCS-2 — two bytes a character, always — and can hold U+0000 without complaint, the same as Python's, so the interesting question is never the variable — it is the boundary. OPEN DATASET … IN BINARY MODE moves bytes and does not care; IN TEXT MODE and the conversion classes (cl_abap_conv_*, cl_abap_codepage) are where a NUL becomes somebody else's problem, and an RFC to an external, C-based system is where it becomes theirs silently. The nearest ABAP-native version of the surprise on this page is a different byte doing the same job: a c field pads and then drops trailing blanks, so what you stored and what you read back differ by a rule the type imposed rather than one you wrote. If you need a NUL constant, take it from cl_abap_char_utilities and verify the name against your system rather than trusting a page. (Not machine-checked — CI cannot run ABAP.)

Try it¶

cd 02_Characters/the_nul_byte/examples
python3 the_nul_byte_py.py
bash the_nul_byte_sh.sh
rustc --edition 2024 the_nul_byte_rs.rs -o /tmp/nulbyte && /tmp/nulbyte

Then on something real. grep -c a file you suspect and see whether it matched while refusing to show you the line. Take any script that pipes find into xargs and check whether both ends say -0. And the question worth sitting with: your application accepts a display name from a form and stores it. Which of the layers between the form and the disk would notice a %00 in it, and which would quietly hand on a shorter string than the one it was given?

Practice¶

Five doors. For "a\x00b", predict which of these accept it and which refuse: .encode('utf-8'), int(), open() as a filename, re.compile(), .split().

Then two questions the pattern answers. Is the string valid UTF-8 — and if it is, why does anything refuse it? And why is NUL the one separator find -print0 can rely on?

Answers

Verified output of the_nul_byte_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

the string     'a\x00b'   len 3
utf-8          610062   three bytes
round trip     True   -- valid UTF-8, in and out

FIVE DOORS
   str.encode('utf-8')        accepted
   int('1\x002')              ValueError -- refused
   open(name)                 ValueError -- refused
   re.compile                 accepted
   'a\x00b'.split()           accepted

The encoder does not care -- U+0000 is assigned, and its UTF-8 form is
the single byte 00, which every validator accepts because there is
nothing invalid about it. Python's own str carries it without comment.

The refusals are all at BOUNDARIES, and every one of them is refusing on
behalf of something else: a filename crosses into a C API where 00 ends
the string, so Python raises rather than silently truncating -- which is
the good behaviour and the reason you meet a ValueError instead of a
file called 'a'.

WHY IT IS ALSO THE SEPARATOR YOU CAN TRUST
   ['one two', 'three\nfour', "five'six"]
   joined with NUL, split back: ['one two', 'three\nfour', "five'six"]
   Every other separator can occur inside a filename -- space, newline,
   quote, tab. NUL cannot, because the kernel's own API cannot express
   it. That is exactly why find -print0 and xargs -0 exist: the one byte
   that is guaranteed absent from the data is the one safe delimiter.