The byte that means something to somebody else¶

Level: 301 · deep dive

One line: A character that means stop here or what follows is an instruction to one program is ordinary data to another, so putting it inside a field makes the two disagree about where the field ends — and one of them is usually the one that acts.

The shape¶

you store a VALUE
somebody downstream parses a SENTENCE
your value contained the sentence's punctuation

The list of punctuation is short and you already know it: NUL ends a C string, ' ends a SQL literal, / ends a path segment, CR LF ends an HTTP header and a log record, ${ opens an expression, ESC opens a terminal command, % opens a printf conversion. None of them is dangerous. Each is dangerous in exactly one place — inside a field that somebody parses instead of storing.

What makes this its own page rather than a footnote to injection is the encoding half: the two parties often disagree about where the field ends because they disagree about what a string is.

One NUL, one valid certificate¶

The best example ever built is Moxie Marlinspike's null-prefix attack, presented at Black Hat USA in 2009.

Request a certificate for the name www.paypal.com\0.thoughtcrime.org. The certificate authority parses ASN.1, where a string carries its own length, so it reads all 32 bytes, sees a subdomain of a domain you demonstrably control, and signs it. The browser pulls the Common Name out and hands it to a C string API, which stops at the NUL and compares www.paypal.com.

Nobody is wrong. Both parties read the field correctly under their own model of what a string is, and the models differ: one counts, the other searches. The attacker did not break either; they wrote down a name that is two names.

The same shape without certificates is the PHP null-byte bug: a check on /etc/passwd\0.jpg reads the whole PHP string, sees it ends in .jpg, and allows it; open() receives a C string and opens /etc/passwd. Note which way round it is — the strict reader is the one that gets fooled, because it reads past the end of the name the weak reader will actually use. PHP stopped accepting a NUL in a path argument in 5.3.4.

One newline, one log entry that never happened¶

A log file's entire structure is one delimiter. Anyone who can put a newline in a field can write a record the program never emitted, and the forged record is a real line by every measure any reader has — grep finds it, the shipper indexes it, the dashboard counts it, the alert rule matches it. It is CWE-117 ↗, and it is worth more to an attacker than it looks: the log is the artefact the incident responder trusts.

The version that is genuinely invisible uses ESC. ESC [ 2K erases the current line and CR returns to its start, so a log entry can unwrite itself on any terminal that renders it — the analyst runs tail -f, sees a clean line, and the damning bytes are in the file the whole time. Terminals have been talked into worse than that; some historically supported reporting their own window title back onto standard input, which turns reading a file into running a command.

One `${`, and the filter that cannot win¶

Log4Shell is this page's shape with an extra turn. The logger interpolated the message it was given, so a value that reached a log statement was executed rather than recorded — data in the sentence, exactly as above.

The turn is what happened next. Filters went in for the keyword, and the payloads assembled the keyword out of lookups: ${${lower:j}ndi:...}, ${${::-j}${::-n}${::-d}${::-i}:...}. The banned string is not in the input; instructions for building it are, and the builder is the interpolator itself. You cannot block a substring that your own code is going to assemble. (Log4j also matched lookup names case-insensitively, so ${jNdI:...} walked past a rule for jndi with no nesting at all — two independent ways past one filter, which is roughly what a keyword filter is worth.)

The C view¶

Verified output of in_band_signals_c.c — regenerated by tools/run_examples.py, never hand-typed.

1. ONE FIELD, TWO LENGTHS
   bytes stored   32   (the length the format carries)
   strlen(name)   14   (the length C computes by searching)
   first 20      77 77 77 2E 70 61 79 70 61 6C 2E 63 6F 6D 00 2E 74 68 6F 75 
   The 0x00 at offset 14 is a byte like any other to whoever counted
   to 32. To strlen it is the end of the world.

2. WHAT EACH PARTY CHECKS, AND BOTH ARE RIGHT
   the issuer walks all 32 bytes and asks: does this end in
   ".thoughtcrime.org", a domain the requester controls?
     answer: yes -- issue it
   the client reads the same field with str* and asks: is this
   "www.paypal.com"?
     answer: yes -- trust it
   Neither party has a bug. They are reading two different strings
   out of one buffer, because they disagree about where it ends.

3. THE SAME SHAPE, WITHOUT CERTIFICATES
   the field      "/etc/passwd\0.jpg"   (16 bytes)
   the CHECK is length-prefixed: does it end in ".jpg"?  yes -- allowed
   the OPEN is NUL-terminated:   what file is that?      "/etc/passwd"
   That is the PHP null-byte bug, and note which way round it is:
   the strict reader is the one that gets FOOLED, because it reads
   past the end of the name the weak reader will actually use.
   PHP stopped accepting a NUL in a path argument in 5.3.4.
   Anywhere a length-prefixed world meets a NUL-terminated one --
   a database column, a protocol field, a filename from a zip, an
   environment variable, a Java String handed to a C library --
   the same two readings are available and somebody picks each.

4. WHAT ACTUALLY FIXES IT
   not: search harder for the NUL
   but: REJECT the field. A NUL inside a domain name, a filename or
        a header is not data that needs handling -- it is a claim
        that two readings exist, and the only safe answer is no.
   memchr(name, 0, stored) != NULL  ->  contains an embedded NUL: refuse

C is where this lives, because C is where the length of a string is computed by searching rather than carried. Section 1 is the whole disagreement in two numbers: 32 bytes stored, 14 bytes long. Section 4 is the fix, and it is not "search harder" — an embedded NUL in a name, a path or a header is not data needing careful handling. It is a claim that two readings exist, and the only safe answer is refusal.

In Python¶

Verified output of in_band_signals_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. MARKERS: A CHARACTER THAT MEANS SOMETHING TO SOMEBODY ELSE
   bytes    char   read by whom, as what
   00       NUL    C: the string stops here
   27       '      SQL: the literal stops here
   2F       /      a path: the segment stops here
   0D 0A    CR LF  HTTP and log files: the line stops here
   24 7B    ${     a logger or template: what follows is an expression
   1B       ESC    a terminal: what follows is a command
   25       %      printf: what follows names an argument
   None of these is dangerous. Each is dangerous in exactly one place:
   inside a field that somebody downstream parses instead of storing.

2. A LOGGER THAT INTERPOLATES WHAT IT LOGS
   log('user=' + 'alice')
     -> user=alice
   log('user=' + '${env:AWS_SECRET}')
     -> user=sk-live-0000-1111
   The logger was asked to record a username. It read the username as a
   program, because the two live in the same string and nothing marks
   which part came from outside. That is Log4Shell's shape -- with a
   lookup that fetched a remote class instead of an environment variable.

3. AND A FILTER ON THE MARKER LOSES
   input          ${env:AWS_SECRET}
     filter sees 'env:'?  True
     result        sk-live-0000-1111
   input          ${${lower:E}nv:AWS_SECRET}
     filter sees 'env:'?  False
     result        sk-live-0000-1111
   input          ${${lower:E}${lower:N}${lower:V}:AWS_SECRET}
     filter sees 'env:'?  False
     result        sk-live-0000-1111
   The second and third never contain the blocked string. They contain
   instructions for BUILDING it, and the builder is the interpolator
   itself -- so the marker exists only after the filter has finished.
   You cannot block a substring that your own code is going to assemble.
   Log4j also matched lookup names case-insensitively, so ${jNdI:...} got
   through a rule for 'jndi' with no nesting at all. Two independent ways
   past the same filter, which is what a filter on a keyword is worth.

4. THE RULE
   The bug is not the payload and not the missing filter. It is that a
   value from outside was placed where a parser was going to look.
     wrong:  log('user=' + name)          format string built from data
     right:  log('user={}', name)         data passed beside the format
   Same for SQL (parameters, not concatenation), for shells (argv, not a
   command line), for HTML (a text node, not markup) and for printf
   (a literal format, never a variable). One rule, five syntaxes:
   KEEP THE DATA OUT OF THE SENTENCE.

The interpolator in section 2 is fifteen lines and has the one property the real ones have: its output is fed back through it. That single fact is what makes filtering the input insufficient, and section 3 shows two payloads that never contain the blocked string.

Section 4 is the rule in five syntaxes. log('user=' + name) builds a sentence out of data; log('user={}', name) passes the data beside the sentence. Parameters instead of concatenation in SQL, argv instead of a command line for a shell, a text node instead of markup in HTML, a literal format string for printf. One idea: keep the data out of the sentence.

In the terminal¶

Verified output of in_band_signals_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

1. ONE LOG CALL, TWO LOG LINES
   log calls made      2
   lines in the file   3
   grep -c 'login ok'  1   <- a line no log statement produced
   The second username contained a newline. The log format's only
   structure IS the newline, so the field became a record.

2. WHAT IS ACTUALLY IN THE FILE
     login failed user=alice$
     login failed user=bob$
     login ok user=root$
   cat -vet marks the ends of lines with $ and shows control bytes as ^X.
   The forged record is a real line by every measure a reader has.

3. THE PART YOU CANNOT SEE AT ALL
   the file, as bytes:
     00000000: 6c6f 6769 6e20 6661 696c 6564 2075 7365  login failed use
     00000010: 723d 6d61 6c6c 6f72 791b 5b32 4b0d 6c6f  r=mallory.[2K.lo
     00000020: 6769 6e20 6f6b 2075 7365 723d 726f 6f74  gin ok user=root
     00000030: 0a                                       .
   the file, with control bytes shown:
     login failed user=mallory^[[2K^Mlogin ok user=root$
   ^[ is ESC. On a terminal, ESC [ 2K erases the line and CR returns to
   its start, so anyone who runs 'cat' or 'tail -f' on this file is shown
   'login ok user=root' and never sees the word mallory at all.
   The bytes are in the file. The screen is a renderer, and it was
   following instructions the file gave it.

4. HOW TO READ SOMETHING YOU DID NOT WRITE
   cat -vet file      control bytes as ^X, line ends as $
   xxd file           no interpretation at all
   less -R OFF        (plain 'less' already escapes control bytes)
   and on the writing side: escape or reject CR, LF and ESC in any field
   that goes into a line-delimited format. A log line is a record, and a
   record separator inside a value is the same bug as a quote inside SQL.

Two log calls, three lines in the file. Section 3 is the one to run yourself on a real terminal, because the output above is cat -vet and xxd — deliberately, since a page that printed the raw escapes would erase its own line. Run the example, then cat the file the ordinary way, and watch the username disappear.

The reading habit that follows is small and worth keeping: anything you did not write gets cat -vet, xxd, or less with its escaping left on. Your terminal is a renderer, and a renderer follows instructions in the thing it renders.

If you are coming from Python or ABAP¶

Python. str has no terminator, so the certificate bug cannot happen inside Python — but Python is usually the length-prefixed half of a pair, which is the half that gets fooled. Every subprocess, every ctypes call, every filename handed to the OS crosses into a NUL-terminated world, and Python already knows: open("a\0b") raises ValueError: embedded null byte rather than truncating, and so does subprocess. Treat that error as a model rather than an annoyance — it is refusal, at the boundary, which is section 4 of the C example implemented for you. For the sentence half, the ones that matter are subprocess.run([...]) and never shell=True, parameterised SQL and never %-formatting, and logging.info("user=%s", name) and never logging.info("user=" + name) — the last one is not about performance, whatever the style guides say.

ABAP. Three concrete places. CALL TRANSFORMATION and any string-built dynamic WHERE clause are the sentence-building constructs; Open SQL host variables (@lv_name) are parameters and are safe, so the rule is simply never to assemble the clause yourself. CALL 'SYSTEM' and external OS commands cross into the NUL-terminated world exactly like Python's subprocess, so a value with a NUL or a newline in it needs rejecting before it goes near one. And for logging: an application log entry (BAL_LOG_MSG_ADD) that concatenates user data into the message text carries the same forged-record risk as any other log — put the value in a parameter of the message rather than in the text. Verify the specific call names against your own release rather than a document. (Not machine-checked — CI cannot run ABAP.)

Try it¶

cd 12_Adversarial/in_band_signals/examples
cc -std=c11 -Wall -Wextra in_band_signals_c.c -o /tmp/ibs && /tmp/ibs
python3 in_band_signals_py.py
bash in_band_signals_sh.sh

Watch a log line erase itself. This writes a file and then reads it two ways:

printf 'user=mallory\033[2K\rlogin ok user=root\n' > /tmp/demo.log
cat -vet /tmp/demo.log     # the truth: user=mallory^[[2K^Mlogin ok user=root$
cat /tmp/demo.log          # what your terminal decides to show you

Then look for the punctuation in your own fields. Pick one value that travels — a username, a filename, a header, a display name — and list every format it is embedded into on the way: SQL, a path, a log line, JSON, HTML, a shell command, a terminal. Each of those has its own punctuation, and the value needs to be safe in all of them or escaped separately for each. Choosing "escape once at input" is choosing to be wrong in every format but one.

And one to reason about. The C example refuses a field containing a NUL rather than truncating at it. Why is refusal the right answer here when the overlong page gives the same advice about repairing invalid UTF-8 — and what do the two situations have in common that makes "fix it up and carry on" the wrong instinct in both?

Practice¶

Five bytes that end a field for somebody. For 00, 0a, ,, " and a leading =, say what each means, and to whom — the reader that acts on it is never the same program as the one that stored it.

Then take the value O"Brien, Jr.\nadmin and say how it must be escaped for a CSV cell, a JSON string and a shell command. Three different answers for one unchanged value — say what that proves about where escaping belongs, and name the two defences, only one of which scales.

Answers

Verified output of in_band_signals_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

THE SHAPE
   A character that means 'stop here' or 'what follows is an
   instruction' to one program is ordinary data to another. Put it in a
   field and the two disagree about where the field ends -- and one of
   them is the one that acts.

   byte   char  means                      to
   00     NUL   ends a C string            the kernel and every C library
   0a     LF    ends a record              line-oriented tools, log parsers, HTTP headers
   2c     ,     ends a field               every CSV reader
   22     "     ends a quoted field        CSV, JSON, shell
   3d     =     ends a formula's name      a spreadsheet, if it is the FIRST byte

THE ONE THAT IS NOT A BYTE AT ALL
   A leading =, +, - or @ in a CSV cell is data to your program and a
   FORMULA to a spreadsheet. Nothing in the file changed; the meaning
   was assigned by the reader. Prefixing the cell with a quote or a
   space is the fix, and it is a fix in the WRITER, because the reader
   is not yours.

WHY ESCAPING IS NOT ONE PROBLEM
   the value  'O"Brien, Jr.\nadmin'
   To put that in a CSV cell you double the quote and wrap the field.
   To put it in a JSON string you backslash the quote and the newline.
   To put it in a shell command you do something different again.
   Three destinations, three escapings, and the value is unchanged in
   all three -- so escaping is a property of the CHANNEL, never of the
   data, and a single 'sanitise' function that runs early is the bug.

THE TWO DEFENCES, AND ONLY ONE OF THEM SCALES
   1. Escape at the boundary, per destination, with the destination's
      own library. csv.writer, json.dumps, shlex.quote, a parameterised
      query -- each knows its own in-band signals.
   2. Get the data out of the band entirely: a length-prefixed field, a
      separate argument, a bound parameter. Then no byte can mean
      'field ends here' because the length already said where.

   Everything else -- stripping, blocklists, replacing quotes on input
   -- is guessing which channel the value will eventually cross, and
   values usually cross more than one.