Two readers, one byte string¶

Level: 301 · deep dive

One line: A byte string does not record which alphabet it is written in, so two programs can read the same bytes as different characters — and an attacker's entire job is to find an input where the program that checks and the program that acts disagree.

The shape¶

Every one of these bugs is three lines long:

stage A reads the bytes with table X   and reports: nothing here
stage B reads the bytes with table Y   and acts on what it finds
the attacker chose bytes for which X and Y disagree

Neither stage is broken. Both are doing exactly what they were written to do, and each in isolation passes its own tests. The defect is the pair, which is why it survives review: a reviewer reads one function at a time, and there is nothing wrong with either function.

This library has already told one of these stories in full. Overlong sequences is the case where X and Y are two implementations of the same encoding, one strict and one lenient, and it ends with a 2001 worm. This page is the more common case, where X and Y are simply two different tables, and nothing is malformed at all.

The trail byte that ate the backslash¶

Escaping a quote is byte surgery: find 27, put 5C in front of it. That is what PHP's addslashes() does, and it is correct for every single-byte encoding and for UTF-8.

It is not correct for GBK, where BF 5C is one character. Send the two bytes BF 27; the escaper sees an ASCII quote at offset 1 and inserts a backslash; the database receives BF 5C 27 and — reading GBK — takes the first two bytes as one Chinese character and hands you the third as a live, unescaped quote. Big5, Shift-JIS and several others have the same property, because 5C is a legal trail byte in all of them. (In Shift-JIS 5C is also the lead byte drawn as ¥, which is the same collision seen from the other end.)

The version of this that actually shipped is worse, because it defeats the function everyone was told to use instead. mysql_real_escape_string() is charset-aware — but it asks the client library, and setting the connection charset with the SQL statement SET NAMES gbk changes the server's view without telling the client anything. The client keeps escaping by Latin-1 rules while the server parses GBK. The fix was mysql_set_charset(), which tells both; the write-up that made this widely known is Chris Shiflett's, in 2006.

Worth noticing what makes the whole family impossible: in UTF-8 no continuation byte is ever below 0x80, so 5C cannot be the tail of anything. That is self-synchronisation, and this bug is one of the things it buys you. "Use utf8mb4" is not fashion advice here — it removes the collision rather than escaping around it.

The page that was never labelled¶

The second case needs no exotic table at all, only a missing declaration.

UTF-7 (RFC 2152 ↗) encodes non-ASCII by escaping into a +…- run of base64 — and it is allowed to escape ASCII characters too. So < may be written +ADw- and > as +AD4-, and +ADw-script+AD4- is sixteen bytes of unremarkable ASCII that a UTF-7 decoder reads as <script>.

Internet Explorer used to work out a page's encoding from its bytes when no charset was declared, and UTF-7 was one of the candidates. A payload with no angle bracket in it passed every filter, and then the browser decided the page was UTF-7 and the tag existed. It was demonstrated against Google's own 404 page in 2006. IE8 dropped UTF-7 from its detection, and the HTML standard now forbids sniffing it at all — but the general lesson outlived the specific browser: an undeclared encoding is an encoding chosen by the attacker.

In Python¶

Verified output of parser_differentials_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. ONE BYTE STRING, THREE TABLES, THREE READINGS
   the bytes            BF 5C 27
   read as utf-8        invalid -- invalid start byte
   read as latin-1      3 characters:  ¿ \ '
   read as gbk          2 characters:  縗 '
   Latin-1 reads three characters and one of them is a backslash.
   GBK reads two, and the backslash is the SECOND HALF of the first one.
   Nothing is corrupt. The bytes simply do not say how they should be cut.

2. THE ESCAPER THAT HANDS OVER THE QUOTE
   attacker sends       BF 27        2 bytes
   the escaper sees     a quote (27) at offset 1, and puts 5C in front of it
   the database gets    BF 5C 27     3 bytes
   read as latin-1      3 characters:  ¿ \ '
   read as gbk          2 characters:  縗 '
   Under Latin-1 the quote is escaped, exactly as intended.
   Under GBK the backslash was eaten as half of a character and the
   quote came out live. The escaper never wrote a bug; it was reading
   a different alphabet from the one the database was reading.

   the statement, as GBK characters:
     SELECT * FROM users WHERE name = '縗'' AND admin = 0
   The closing quote is now early, and everything after it is code.

3. THE PAGE THAT WAS NEVER LABELLED
   payload bytes        2B 41 44 77 2D 73 63 72 69 70 74 2B 41 44 34 2D
   as ASCII             '+ADw-script+AD4-'
   contains b'<script>' ? False
   contains b'<' ?        False
   decoded as utf-7     '<script>'
   A filter looking for the eight bytes of '<script>' finds nothing --
   nor does one looking for a bare '<', which is what most filters check,
   because in UTF-7 those two characters are spelled +ADw- and +AD4-.
   The tag does not exist until somebody decides the page is UTF-7 --
   which a browser used to be willing to work out from the bytes alone.

   note: encoding is not forced to use the escape --
   '<script>'.encode('utf-7') = b'<script>'
   so the two spellings are not symmetric: an encoder picks one, and a
   decoder must accept both. Every 'many spellings' bug lives in that gap.

4. THE SHAPE, WITHOUT THE STORY
   Both cases are the same three lines:
     stage A reads the bytes with table X  and says: nothing here
     stage B reads the bytes with table Y  and acts on what it finds
     the attacker chose the bytes so that X and Y disagree
   So the question to ask of a pipeline is never 'is this input safe'.
   It is: WHO DECODES, WITH WHICH TABLE, AND IN WHAT ORDER --
   and the fix is to make the answer the same at every stage, once,
   at the top, instead of letting each stage work it out for itself.

Section 1 is the entire idea in three rows. Nothing about BF 5C 27 is corrupt; the bytes simply do not say where the characters begin, and Latin-1 and GBK make different, defensible choices. Section 4 is the question to ask a design, and it is not "is this input safe" — it is who decodes, with which table, and in what order.

In the terminal¶

Verified output of parser_differentials_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

1. THE PAYLOAD IS ORDINARY ASCII
   $ printf %s '+ADw-script+AD4-' | xxd -p
     2b4144772d7363726970742b4144342d
   16 bytes, every one of them below 0x80.

2. EVERY BYTE FILTER LETS IT THROUGH
   grep -c '<script>'     0
   grep -c '<'            0
   grep -c '>'            0
   grep -c 'javascript'   0
   grep -c 'script'       1   <- the only thing a filter can see
   No angle bracket exists in the file. There is nothing to escape,
   nothing to strip, and nothing for a rule to match on.

3. AND IT IS VALID UTF-8, SO VALIDATION SAYS YES TOO
   iconv -f UTF-8 -t UTF-8   exit 0   (0 = well-formed)
   iconv -f ASCII -t UTF-8   exit 0   (0 = well-formed)
   Both say yes, and both are right. Well-formedness is a question about
   ONE table. It cannot tell you that the same bytes spell something else
   in a second table -- here UTF-7, where +ADw- is '<' and +AD4- is '>'.

4. SO THE DEFENCE IS NOT A FILTER, IT IS A DECLARATION
   wrong:  serve the bytes and let the reader work out the encoding
   right:  Content-Type: text/html; charset=utf-8

   A filter guards a spelling. Naming the table removes the second
   spelling entirely, which is the only move that scales: there is
   always another table, and you cannot enumerate them.

This one is here to kill a comfortable belief. The repo's validation page shows that deciding "these bytes are UTF-8" at a boundary makes everything downstream simpler, and it does. Section 3 shows what it does not do: the UTF-7 payload is well-formed UTF-8 and well-formed ASCII, and both iconv runs exit 0, correctly. Well-formedness is a question about one table. It cannot tell you the same bytes spell something else in a second one.

So validation and declaration are two different jobs. Validation says these bytes are a legal spelling. Declaration says and this is the alphabet they are spelled in. You need both, and only the second one closes this page's bugs.

If you are coming from Python or ABAP¶

Python. You have the whole demonstration in the standard library, which is the good news and the trap: bytes.decode(table) makes the table an argument, so it is visible in the diff and somebody chose it. The place it goes wrong is where the argument is implicit — open() without encoding= takes the locale's answer, which differs between your laptop and the container, and str.encode() defaults to UTF-8 while a subprocess on the other side may not. When you are the boundary, pass encoding= explicitly every single time; when you are checking somebody else's boundary, grep for the calls that omit it. locale.getpreferredencoding(False) tells you what the omission currently resolves to, which is worth printing once in a start-up log.

ABAP. The nearest equivalent of this whole page is the gap between an xstring and a string. cl_abap_codepage=>convert_from( source = xs codepage = '...' ) names the table, and the fact that it has to name it is a feature — the moment you can see the code page in the source, two stages naming different ones is a diff you can read. The dangerous shape is the same as Python's: an OPEN DATASET ... IN LEGACY TEXT MODE with no CODE PAGE ↗, or IN TEXT MODE ENCODING NON-UNICODE, whose table is not in the statement but in TCP0C, looked up for the text environment your logon language sets ↗, so a colleague who logs on in another language gets a different table without a line of code changing. And for the SQL half specifically: Open SQL host variables are parameters, not concatenation, so the escaping problem above does not arise — until somebody builds a WHERE clause in a string and passes it to dynamic SQL, at which point every word of this page applies again. Verify code page numbers against your own system rather than a document. (Not machine-checked — CI cannot run ABAP.)

Try it¶

cd 12_Adversarial/parser_differentials/examples
python3 parser_differentials_py.py
bash parser_differentials_sh.sh

Watch the backslash vanish for yourself. Your iconv can do this without Python:

printf '\xbf\x5c' | iconv -f GBK -t UTF-8 | xxd

One three-byte character comes out, and the backslash is inside it. (Recorded on macOS; the shell example keeps GBK out of its answer key because iconv's table names are not guaranteed identical on both platforms.)

Then find the second reader in something you own. Pick any request that reaches a database, and write down every stage between the socket and the SQL parser — framework decode, validation, ORM, driver, connection charset, column collation. You are looking for two stages that name a table, and for the stages that name none. The ones that name none are where the next one of these lives.

And one to reason about. The Python example notes that '<script>'.encode('utf-7') returns plain ASCII: Python's encoder declines to use the +ADw- form even though it is legal. Why is an encoder allowed to choose, while a decoder is not — and what does that asymmetry imply about testing a filter by round-tripping payloads through your own encoder?

Practice¶

One byte string, two readers. Decode b"caf\xe9" as latin-1, cp1252 and utf-8 and say what distinguishes the three outcomes. Then the classic: a filter searches for the byte 2f in c0 af — predict whether it finds it, and what a lenient decoder later produces.

Then two you can run today in one library: what json.loads does with a duplicate key, and what it does with a lone surrogate. Finish with the single defence that works, which is not "validate more".

Answers

Verified output of parser_differentials_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. THE SAME BYTES, TWO ALPHABETS
   as latin-1   'café'
   as cp1252    'café'
   as utf-8     UnicodeDecodeError -- refuses
   Nothing in the bytes says which. A checker that decodes one way and
   an actor that decodes another are looking at different strings.

2. THE CLASSIC: A CHECK IN BYTES, AN ACTION IN CHARACTERS
   filter looks for the byte 2f in c0 af -> not found, passes
   a STRICT decoder refuses this outright
   a LENIENT decoder that assembles the payload bits yields '/'
   Two components, one input, opposite conclusions. The check passed
   because it was true about the bytes; the action happened because it
   was true about the characters.

3. A JSON DIFFERENTIAL YOU CAN RUN TODAY
   {"role": "user", "role": "admin"}
   Python json.loads -> {'role': 'admin'}
   Duplicate keys are not an error in the JSON spec; it says nothing
   about which wins. Python keeps the LAST. Other parsers keep the
   first. A gateway that validates with one library and a service that
   acts with another can disagree about a single request -- and neither
   is violating the specification.

4. AND ONE MORE, IN THE SAME LIBRARY
   json.loads(\'"\ud800"\') accepts a lone surrogate: '\ud800'
   ...and the resulting str cannot be encoded as UTF-8 at all
   So a document that parses is not a document that can be re-emitted
   in the encoding its own RFC requires.

THE DEFENCE
   One parser. If the checking stage and the acting stage cannot share
   an implementation, they must share a CANONICAL FORM: parse once,
   re-serialise, and pass the re-serialised value on -- so the second
   stage never sees the attacker's spelling, only yours.