The check that ran too early¶

Level: 301 · deep dive

One line: A check tells you something true about the string you checked, and every transformation that runs afterwards can hand the next stage a different string — so the order of canonicalise and check is the security property, and neither one is safe alone.

The shape¶

check(input)                 -> looks fine
transform(input)             -> now it is the thing the check forbids

The previous page's bugs came from two readers disagreeing about the same bytes. This page's come from one reader, running twice, at two different moments — and the second run is almost always somebody being helpful. Normalising Unicode, folding case, resolving .., decoding a percent-escape, converting to the local code page: every one is correct hygiene, every one is recommended somewhere, and every one manufactures characters that were not in the string when the gate looked at it.

The fix is one word long — first — and it is the same word the overlong sequences page arrives at from the other direction. Canonicalise, then check, then pass on the canonical form. Three verbs, one order.

One character, one slash¶

The sharpest example needs no encoding trick at all. ℀ is a single code point, U+2100 ACCOUNT OF, and its compatibility decomposition is three characters: a, /, c.

A filter scanning for / cannot see a slash that does not exist yet. Then a database column with a normalising collation, a filename lookup on a normalising filesystem, a JSON library, or a template engine runs NFKC — and the slash appears. Nobody attacked the filter. The filter answered correctly, about a string that no longer exists.

The same holds for the whole fullwidth block: ＜script＞ normalises to <script>, and ．．／ to ../, so a path check and an HTML filter can both be walked past by typing the payload in the wrong-width Latin letters.

Two that do not normalise, and the table that gets them anyway¶

∕ U+2215 DIVISION SLASH and ⁄ U+2044 FRACTION SLASH look like separators and NFKC leaves both alone — Unicode says they are not slashes, and normalisation agrees. So normalising is not a defence against them either; it is not the relevant transformation.

The relevant transformation is a best-fit mapping: when a string is converted to an encoding that has no exact match for a character, some converters substitute one that looks similar rather than failing. Windows' WideCharToMultiByte does this by default, and its table maps a long list of look-alikes onto ASCII — fullwidth ＆ to &, quotation variants to ', and slash-like characters to /. An application validates a clean Unicode string, passes it to an API, and the conversion below the application produces the metacharacters the validation just certified were absent.

Orange Tsai and splitline presented this as Worst Fit at Black Hat Europe 2024, with path traversal, argument injection and remote code execution across a range of widely deployed software. The defensive knob exists and is off by default: WC_NO_BEST_FIT_CHARS makes the conversion fail instead of approximating. Nothing on this page's programs can demonstrate it, because it is a Win32 behaviour and this library's examples run on macOS and Ubuntu — but it is the reason the rule is canonicalise then check rather than normalise then check: normalisation is one of the transformations, not the definition of them.

The oldest member of the same family is double decoding. A gate rejects ../, so the attacker sends %252e%252e%252f; the gate decodes once and sees %2e%2e%2f, which contains no slash; a later stage decodes again and gets ../. Same shape, no Unicode required.

In Python¶

Verified output of canonicalize_then_check_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. THE GATE SAYS YES
   a tag                        gate(...) = True
   a path                       gate(...) = True
   one character, three after   gate(...) = True
   blocked substrings: '<' '>' '/' '\\' '..'
   None of these strings contains any of them. The gate is not broken,
   not bypassed, and not misconfigured. It is answering correctly.

2. AND THEN SOMETHING NORMALISES
   a tag                        ＜script＞
                                  -> NFKC -> '<script>'
                                  gate would now say: False
   a path                       ．．／．．／etc／passwd
                                  -> NFKC -> '../../etc/passwd'
                                  gate would now say: False
   one character, three after   ℀
                                  -> NFKC -> 'a/c'
                                  gate would now say: False
   Nobody attacked the gate. A later stage -- a database column with a
   normalising collation, a JSON library, a filename lookup, a template
   engine -- did the one thing everybody agrees is correct hygiene, and
   manufactured the exact characters the gate exists to reject.

   The third one is worth staring at. It is a SINGLE code point:
     ℀  U+2100  ACCOUNT OF
   NFKC expands it to three characters, one of which is a path separator.
   No filter that scans for '/' can see a slash that does not exist yet.

3. THE SAME TWO STEPS, SWAPPED
   def safe(s): return gate(unicodedata.normalize('NFKC', s))
   a tag                        safe(...) = False
   a path                       safe(...) = False
   one character, three after   safe(...) = False
   Same gate, same strings, same normalisation. Only the order changed.
   Canonicalise, then check, then STORE THE CANONICAL FORM -- so nothing
   downstream is ever handed the spelling you did not check.

4. TWO THAT NFKC DOES NOT TOUCH
   U+2215  ∕  DIVISION SLASH
           NFKC -> '∕'   (unchanged)
   U+2044  ⁄  FRACTION SLASH
           NFKC -> '⁄'   (unchanged)
   They look like a slash and Unicode says they are not one, so
   normalising is not a defence against them either. What turns THESE
   into '/' is a different kind of table: a 'best fit' mapping, which
   substitutes a similar-looking character when a target encoding has
   no exact match. That conversion happens below the application, and
   the page has the story.

5. THE RULE, IN ONE LINE
   Any transformation that runs AFTER your check is part of your
   attack surface, and normalisation is a transformation.
   Decode once, canonicalise once, check the canonical form, and pass
   the canonical form on. Three verbs, one order, no second spelling.

Section 3 is the whole fix and it is three words rearranged. Section 5's last clause is the part people leave out: store the canonical form. Checking the canonical form and storing the original puts the unchecked spelling back into the system, and the next stage that reads it starts the page over.

Every mapping in that program is a compatibility decomposition of a character assigned in Unicode 1.1, which the normalization stability policy freezes forever — so those results are safe in an answer key. A count over the whole table is not, which is what the exercise below is about.

In Rust¶

Verified output of canonicalize_then_check_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

1. THE GATE IS CORRECT, AND ASCII-NARROW
   reserved name      "kelvin"
   candidate          "Kelvin"   (6 chars, 8 bytes)
   ...which is         U+212A U+0065 U+006C U+0076 U+0069 U+006E
   ...written as       "\u{212A}elvin"
   Printed, it is indistinguishable from the ASCII word. That is
   not a limitation of this program; it is the entire technique.
   to_ascii_lowercase "Kelvin"
   equal to reserved? false
   eq_ignore_ascii_case? false
   So the registration is allowed, and every line above is right:
   U+212A is not an ASCII letter and an ASCII fold must not touch it.

2. THE STAGE AFTER IT IS NOT NARROW
   to_lowercase       "kelvin"
   equal to reserved? true
   Full Unicode lowercasing maps U+212A KELVIN SIGN to ASCII 'k'.
   Two accounts now canonicalise to one name. Which of them the
   next lookup returns is a question about a hash table, not policy.

3. AND CASE MAPPING CHANGES LENGTH
   "ß"  1 chars  2 bytes  ->  upper "SS" 2 chars   lower "ß" 1 chars
   "ﬁ"  1 chars  3 bytes  ->  upper "FI" 2 chars   lower "ﬁ" 1 chars
   "İ"  1 chars  2 bytes  ->  upper "İ" 1 chars   lower "i\u{307}" 2 chars
   A length limit checked BEFORE a case fold does not hold after it.
   Both directions exist: 'fi' as one ligature grows, and a string
   trimmed to fit can be cut through the middle of a character.

4. WHAT std GIVES YOU, AND WHAT IT DOES NOT
   to_lowercase / to_uppercase   full Unicode, in std
   to_ascii_lowercase            ASCII only, and the name says so
   eq_ignore_ascii_case          ASCII only, and the name says so
   NFC / NFD / NFKC / NFKD       not in std at all -- a crate
   Rust names its narrow operations honestly, which is a real
   advantage: you cannot reach for an ASCII fold by accident.
   But the wide one is a dependency, so the compiler has no opinion
   about WHERE it runs -- and where it runs is the whole question.

5. THE ORDER THAT WORKS
   let canonical = input.to_lowercase();      // canonicalise
   if canonical == reserved { reject() }       // then check
   store(&canonical);                         // then store THAT
   canonical = "kelvin" (U+006B U+0065 U+006C U+0076 U+0069 U+006E)  -> rejected: true
   The third line is the one people leave out. Checking the
   canonical form and storing the original puts the two spellings
   back in the database, and the next lookup gets to choose.

Rust makes the first half of this harder and the second half stranger.

Harder, because std names its narrow operations honestly. to_ascii_lowercase and eq_ignore_ascii_case say ASCII in the name, so you cannot reach for an ASCII fold thinking you got a Unicode one — which is exactly the confusion section 1 is built on, and in Rust you have to write it deliberately.

Stranger, because std has no normalisation at all. NFC, NFKC and their siblings live in the unicode-normalization crate, so on a Rust codebase the canonicalisation step is a dependency — and a dependency runs wherever somebody happened to call it, which is precisely the thing this page says must be decided once. The type system has nothing to say about it: String guarantees well-formed UTF-8 and makes no claim at all about normal form.

Section 3 is the quieter trap. ß uppercases to two characters, ﬁ to two, İ lowercases to two — so a length limit checked before a fold does not hold after it, in both directions. A field that fits in 64 characters when you validated it can arrive at storage as 128, and a naive truncation back to 64 can cut through the middle of a character.

If you are coming from Python or ABAP¶

Python. unicodedata.normalize is an explicit call, which is the good part; the bad part is everything that normalises without one. pathlib and open() hand filenames to a filesystem that may normalise (macOS does), str.lower() is a fold you did not describe as one, and email.utils / idna / a JSON round-trip through a database can each return a differently-spelled string from the one you handed over. The habit that survives contact with all of it: build one function — def canonical(s): return unicodedata.normalize("NFKC", s).casefold() — call it at exactly one place, and let every check and every stored key be computed from its output. If two places in your code call normalize with different forms, that is the bug, and it is greppable.

ABAP. Two halves, and only one has a built-in. For case, TRANSLATE ... TO UPPER CASE and the to_upper/to_lower built-ins fold with the system's rules, so the ordering rule applies unchanged: fold, then check, then store what you folded. For normalisation there is no ABAP equivalent of unicodedata.normalize in the language, so a Unicode system's string holds whatever spelling arrived — which means the canonicalisation has to happen at the interface, before the data reaches ABAP, or be done by whatever the receiving system uses (a database collation, a middleware step). Say which, in writing, on the interface spec; "the data is normalised" with no named actor is how two systems each assume the other did it. And the practical version of section 3's length trap is the classic ABAP one: a CHAR(40) field is 40 characters in a Unicode system and the fold can change the count, so validate the length after the fold, never before. (Not machine-checked — CI cannot run ABAP.)

Try it¶

cd 12_Adversarial/canonicalize_then_check/examples
python3 canonicalize_then_check_py.py
rustc --edition 2024 canonicalize_then_check_rs.rs -o /tmp/cc && /tmp/cc

Search the table yourself. The program deliberately does not print a count, because "how many code points normalise into a string containing /" is a fact about your Python's Unicode version, not about Unicode. So go and get the number for yours:

import sys, unicodedata as ud
for target in ("/", "<", "'", ".."):
    hits = [chr(cp) for cp in range(sys.maxunicode + 1)
            if target in ud.normalize("NFKC", chr(cp)) and chr(cp) != target]
    print(f"{target!r:6} {len(hits):4}  {''.join(hits[:24])}")

Four lines, four short lists — which is the useful surprise: this is not a vast attack surface, it is a specific and enumerable one — and nobody enumerates it, because the filter is written before anyone asks the question.

Then check an order you own. Find any place in your code where a value is validated and later lowercased, normalised, url-decoded, or written to a filesystem. Write down which happens first. That single sentence is the finding; you do not need a payload to file it.

Practice¶

The same guard, two orders. A blocklist refuses /etc/passwd. Predict which of /etc/passwd, /etc/./passwd, /var/../etc/passwd, /etc/%70asswd and /tmp/ok get through when the check runs before canonicalisation, and when it runs after.

Then say why "check harder" is not the fix. Finish with the Unicode version of the same bug — four spellings of admin — and say why neither folding nor not-folding is safe on its own.

Answers

Verified output of canonicalize_then_check_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

input                    check first                canonicalise first
/etc/passwd              REFUSED                    REFUSED
/etc/./passwd            opened /etc/passwd         REFUSED
/var/../etc/passwd       opened /etc/passwd         REFUSED
/etc/%70asswd            opened /etc/passwd         REFUSED
/tmp/ok                  opened /tmp/ok             opened /tmp/ok

   Four of the five reach /etc/passwd, and the guard that runs FIRST
   stops exactly one of them -- the one spelled the obvious way. The
   check was not wrong: it told the truth about the string it was given.
   Everything after it handed the next stage a different string.

THE RULE, AND WHY IT IS NOT 'CHECK HARDER'
   You cannot enumerate the spellings. Percent-encoding, dot segments,
   symlinks, case folding, Unicode normalization and overlong UTF-8 all
   produce the same resource from different bytes, and a blocklist has
   to be right about all of them at once. Canonicalising first makes the
   check compare ONE value against ONE value.

THE UNICODE VERSION OF THE SAME BUG
   stored user   'ADMIN'
   admin    -> NFKC+casefold 'admin'    MATCHES the reserved name
   ADMIN    -> NFKC+casefold 'admin'    MATCHES the reserved name
   ａdmin    -> NFKC+casefold 'admin'    MATCHES the reserved name
   Ⓐdmin    -> NFKC+casefold 'admin'    MATCHES the reserved name
   If you compare before folding, all four are different users. If you
   fold before comparing, all four are one. Neither is 'safe' on its
   own -- what makes it safe is that ONE of them is the stored form and
   everything is converted to it before any comparison happens.

AND THE ORDER THAT IS STILL WRONG
   canonicalise -> check -> canonicalise again is not belt and braces;
   the second canonicalisation can move the value again. Do it once, as
   early as possible, and pass the CANONICAL value downstream -- never
   the original alongside it, or something will use the wrong one.