UTF-7, and the seven-bit transport¶
Level: 201 → 301 · for interface work, and for anyone who has met UTF-7 in a security report
One line: UTF-7 spells the whole of Unicode in printable ASCII because a 1990s mail relay would not carry a byte above 0x7F — a sound answer to a real constraint — and it is deprecated everywhere today for one clause in its own specification: escaping is optional, so one eight-character string has 2,961 legal spellings and a decoder must accept every one an encoder will never write.
The transport would not carry your bytes¶
This chapter has already met two channels that take only ASCII, and two different answers to them. Binary to text re-cuts the whole byte stream into six-bit pieces and spends legibility to guarantee that every byte survives. Escaping into ASCII leaves the text readable and rewrites only the characters the channel objects to. UTF-7 is the third answer, and it is worth understanding what it was trying to beat.
SMTP and NNTP in the 1990s were seven-bit transports. A relay was entitled to strip the high bit off every byte it handled, so a message body containing a C3 was not a message body that arrived. Base64 got the bytes through and made the English unreadable along with the Polish. A MIME encoded-word got a header through and named its charset while doing it, which is still the best design in the family — but it wraps a word, not a document, and it needs the sender and receiver to agree on the inner table anyway.
UTF-7 (RFC 2152 ↗, May 1997, Informational) is the answer that keeps ASCII looking like ASCII and carries all of Unicode with no second table to agree on. caf+AOk- is café: three letters that were already fine, and one shift sequence for the one that was not. That is a real advantage over both neighbours, and the RFC is candid about where it applies — it says UTF-7 should normally be used only for seven-bit transports such as mail, and that straight Unicode or UTF-8 is preferred elsewhere.
+ shifts in, - shifts out, and the middle is UTF-16¶
The mechanism is three sets and one state bit.
| set | what is in it | how it is written |
|---|---|---|
| Set D, direct | the letters, the digits, and nine punctuation marks | as themselves |
| Set O, optional direct | twenty more punctuation marks, listed below | as themselves, or escaped — the encoder chooses |
| Set B | the Base64 alphabet without = |
the payload of a shift sequence |
Set O is the whole story of this page. Here are its twenty characters, and the two to look for are the angle brackets:
Space, tab, carriage return and line feed may be written directly too. \ and ~ are in neither set, deliberately — the RFC leaves them out because variants of ASCII redefined them — so a conforming encoder has to escape those two, and it has to escape + because + is the shift character itself. Those three are the whole of what Python's encoder escapes in printable ASCII, which is what the program below counts. A + shifts into a modified Base64 run — modified because the = padding is dropped, the length being recoverable from where the run ends. The run ends at a -, which is absorbed and produces no character, or at the first byte that is not in the Base64 alphabet, which is kept. +- is how you write a literal +.
What the run carries is the part people guess wrong: it is Base64 of UTF-16BE code units, not of code points and not of UTF-8 bytes. So é is +AOk- and 😀 is +2D3eAA- — twice as long, because it goes through a surrogate pair on its way into a 1997 mail body. UTF-7 is two encodings this chapter has already taught, stacked, plus a shift; and it inherits everything that comes with the inner one.
In Python¶
Verified output of utf7_and_the_seven_bit_transport_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. THE CONSTRAINT, AND FOUR ANSWERS TO IT
------------------------------------------------------------------------
SMTP and NNTP in the 1990s were SEVEN-BIT transports: a relay was
entitled to strip the high bit off every byte it carried. So a
payload with any byte above 0x7F could not be sent as itself.
the string 'café'
as UTF-8 63 61 66 c3 a9 highest byte 0xC3
...which is exactly the byte the transport would not carry.
scheme output what it does
base64 Y2Fmw6k= whole stream, unreadable, +33% always
quoted-printable caf=C3=A9 per byte, ASCII stays legible
MIME encoded-word =?utf-8?b?Y2Fmw6k=?= names its own charset -- alone in this table
UTF-7 caf+AOk- per character, ASCII stays legible, no charset named
Highest byte across all four outputs: 0x77. Every one is ASCII,
which was the whole requirement.
UTF-7 is the entry that keeps ASCII readable AND carries the rest of
Unicode without a second header naming a table. That is a genuine
advantage over the other three, and it is why the format existed.
2. THE MECHANISM: + SHIFTS IN, - SHIFTS OUT, THE MIDDLE IS UTF-16
------------------------------------------------------------------------
code point UTF-16BE code units base64 UTF-7 character
U+00E9 00 e9 AOk +AOk- é
U+20AC 20 ac IKw +IKw- €
U+1F600 d8 3d de 00 2D3eAA +2D3eAA- 😀
The second column is the finding. A UTF-7 run is Base64 of UTF-16
CODE UNITS -- not of code points, and not of UTF-8 bytes -- so an
emoji travels through a surrogate pair on its way into a mail body,
and its run is twice as long as the two BMP characters above it.
The '=' padding Base64 would add is stripped. A run ends at '-', or
at the first byte that is not in the Base64 alphabet, or at the end
of the input; a '-' that ends a run is absorbed and is not output.
3. THE ASYMMETRY: THE DECODER ACCEPTS WHAT THE ENCODER NEVER WRITES
------------------------------------------------------------------------
decode b'+ADw-script+AD4-' -> '<script>'
encode '<script>' -> b'<script>' <- unchanged
Both are correct. RFC 2152 puts '<' and '>' in Set O, the OPTIONAL
direct characters: an encoder may pass them through or escape them,
and a decoder must accept either. Round-tripping a payload through
your own encoder therefore proves nothing about your decoder.
printable ASCII this encoder escapes: + \ ~ (3 of 95)
Everything else, angle brackets included, is passed through. That is
one encoder's policy, not the format's rule.
Three spellings of one character:
b'<' -> '<'
b'+ADw-' -> '<'
b'+ADw' -> '<'
Direct; shifted in and explicitly out; shifted in and ended by the
end of the input.
Now the whole string '<script>', enumerated under that rule:
byte strings constructed 2961
of those, decoded back to the string 2961
rejected, or decoded to something else 0
spellings this encoder will ever write 1 (b'<script>')
shortest b'<script>'
longest, at 40 bytes b'+ADw-+AHM-+AGM-+AHI-+AGk-+AHA-+AHQ-+AD4-'
That ratio IS the security problem. A filter that searches bytes is
guarding one spelling out of thousands, and the attacker chooses
which one to send. Compare overlong UTF-8, where the same argument
ends the other way: one character, one legal spelling, and every
other candidate ill-formed and rejected on sight.
4. WHAT THIS DECODER REFUSES, AND WHAT IT CARRIES
------------------------------------------------------------------------
refused:
b'+ADx-' -> UnicodeDecodeError 'ADx' carries '<' plus two padding bits that are not zero
b'+AGEA-' -> UnicodeDecodeError 22 payload bits: one code unit and six bits of nothing
carried, and each of these is five printable ASCII bytes:
b'+AAA-' -> '\x00' a NUL, through a channel that takes printable ASCII only
b'+AB8-' -> '\x1f' U+001F, a C0 control
b'+2D0-' -> '\ud83d' a LONE SURROGATE -- half of the emoji above
and that last one, re-encoded to UTF-8: UnicodeEncodeError -- a surrogate is not a scalar value
So a UTF-7 decode that SUCCEEDS can hand you a string your next
stage cannot write out at all -- the UTF-16 layer showing through.
The codec is not lax, though. RFC 2152 calls a run ill-formed when
the discarded bits are non-zero, and the two refusals above are
exactly that rule and the incomplete-run rule. The danger was never
a sloppy implementation; it is the format's own optional escape,
faithfully implemented. Which is why deprecating the format was the
fix, and hardening the decoder was not.
5. AND THE CODEC IS STILL HERE
------------------------------------------------------------------------
codecs.lookup('utf-7').name -> 'utf-7'
It takes no flag to reach and prints no warning when used. .NET
marked UTF7Encoding obsolete and browsers stopped sniffing UTF-7,
but a decoder kept alive for compatibility is still a decoder
somebody's input can reach. Hence the rule this page ends on: name
the encoding at the boundary, so nothing downstream is left to
choose a second reading of the same bytes.
Section 3 is the finding, and it is worth stating plainly because it is the shape of every parser differential: the decoder accepts a spelling the encoder never produces. b'+ADw-script+AD4-'.decode('utf-7') is '<script>', and '<script>'.encode('utf-7') is b'<script>', unchanged. Neither is a bug. < and > are Set O, so passing them through is a legal encoder policy and accepting the escape is a mandatory decoder duty, and the two do not meet.
The consequence is the count. Cut the eight characters into consecutive groups, write each group directly or as one Base64 run, and every one of the 2,961 byte strings that construction produces decodes back to <script> — while the encoder in front of you writes exactly one of them. That ratio is why a byte filter cannot win here: it is guarding one spelling and the attacker picks which of the other 2,960 to send.
This is the same argument overlong sequences makes about UTF-8, with the opposite ending. There, a code point has exactly one legal spelling and every padded alternative is ill-formed, so a decoder that follows the rules destroys the ambiguity for you. Here the alternatives are legal, and no conforming decoder may reject them. Non-canonical spelling is a security bug is the shared sentence; UTF-8 fixed it in the format and UTF-7 could not.
Section 4 adds the part that makes it worse than a spelling problem. A UTF-7 run can carry a NUL, a C0 control, and a lone surrogate — through a channel whose entire premise is that it carries printable ASCII. b'+2D0-' is five innocuous bytes and decodes to a str that cannot then be encoded to UTF-8 at all.
In Rust¶
Verified output of utf7_and_the_seven_bit_transport_rs.rs — regenerated by tools/run_examples.py, never hand-typed.
1. WHAT std HAS, AND WHAT IT DOES NOT
------------------------------------------------------------------------
Rust's standard library converts between UTF-8 and Unicode's
own other forms, and offers no other character table at all:
std::str::from_utf8(&[63, 61, 66, c3, a9])
-> "café"
String::from_utf16(&[0063, 0061, 0066, 00e9])
-> "café"
std::str::from_utf7 / Encoding::UTF7 / anything charset-shaped
-> does not exist. Not deprecated, not feature-gated: absent.
That absence is the finding, and it is deliberate. std carries
the encoding forms Unicode defines and stops there; every legacy
table -- UTF-7, Shift-JIS, cp1252 -- lives in a crate. Nothing
in a Rust program can decide a byte string is UTF-7 by accident,
because nothing in a Rust program knows what UTF-7 is.
2. SO HERE IS THE NARROW CASE, HAND-ROLLED IN std
------------------------------------------------------------------------
caf+AOk- -> "café"
+ADw-script+AD4- -> "<script>"
<script> -> "<script>"
+2D3eAA- -> "😀"
1 +- 1 -> "1 + 1"
Line two is the whole security story in one row: sixteen bytes
of unremarkable ASCII, and a tag on the other side. Line three
is the same tag spelled directly, which is what an encoder
writes. Both are legal UTF-7 and they are not the same bytes.
3. WHERE RUST AND PYTHON PART, AND WHY
------------------------------------------------------------------------
"+2D0-" -> refused: unpaired surrogate U+D83D -- not a character, so not a `char`
char::from_u32(0xD83D) -> None
The Python block on this page accepts those same five bytes
and hands back a str holding one lone surrogate -- a string
that cannot then be encoded to UTF-8. Rust cannot reach that
state: `char` is a Unicode SCALAR value, the surrogate range
is a hole in it, and so the refusal happens at the decode
instead of two stages later. Same input, same specification,
two different places to find out -- which is a differential
between languages, inside a format built to be carried
between machines.
There is no UTF-7 in Rust's standard library, and that absence is the finding rather than a gap this page works around. std ships the conversions Unicode itself defines — std::str::from_utf8, String::from_utf16, char::decode_utf16 — and no charset registry at all; every legacy table lives in a crate. So nothing in a Rust program can decide a byte string is UTF-7 by accident, because nothing in a Rust program knows what UTF-7 is. That is the same property, arrived at from the other direction, that Python still shipping the codec does not have.
Writing the decoder by hand rather than reaching for a crate is what exposes the last difference. Rust's char is a Unicode scalar value, so the surrogate range is a hole in it: b'+2D0-' has to be refused at the decode, where Python hands back a lone surrogate and lets the failure surface two stages later. Same input, same specification, two different places to find out — a differential between languages, inside a format designed to be carried between machines.
Why it is gone¶
Three removals, from three different kinds of authority, and none of them is a browser vendor's blog post.
.NET deprecated it outright. Encoding.UTF7 and the UTF7Encoding constructors are marked obsolete starting in .NET 5, and using them raises SYSLIB0001 ↗ at compile time. Microsoft's stated reasons are the three on this page: the encoding is no longer in wide use, specifications forbid it in interchange, and it has been used as an attack vector against applications that never expected it. The class reference ↗ adds a fourth in its own words — UTF7Encoding provides no error detection, so for security you should be using UTF8Encoding with detection switched on. The obsoletion is still in force through .NET 10 and 11.
The web forbids it, and the mechanism is worth reading closely. The WHATWG Encoding Standard ↗ does not mention UTF-7 at all; it publishes a table of every encoding and label a user agent must support and then closes the list — "User agents must not support any other encodings or labels." UTF-7 is prohibited by not being on a closed list, which is a stronger and more durable move than naming it. The HTML Standard ↗ then spells the consequence out in §13.2.3.3: "The above prohibits supporting, for example, CESU-8, UTF-7, BOCU-1, SCSU, EBCDIC, and UTF-32." Browsers stopped sniffing UTF-7 before that — Internet Explorer 8 dropped it from detection — but sniffing was only ever the delivery mechanism. The rule that closed it is the closed list.
Python still ships the codec. codecs.lookup('utf-7') succeeds, bytes.decode('utf-7') takes no flag and prints no warning, and the codec is a faithful implementation: it enforces the RFC's rule that non-zero padding bits make a run ill-formed. That is the uncomfortable half. A language keeping a dangerous decoder alive for compatibility is not doing anything wrong — mail from 1997 still exists, and IMAP4rev1 mailbox names are a UTF-7 variant (RFC 3501 §5.1.3 ↗, with & for the shift and , for /, which IMAP4rev1 servers still speak, even though IMAP4rev2 ↗ replaced it with Net-Unicode in 2021) — but the decoder is reachable from any string in any program, and the format's own design says a decoder must accept every spelling. Deprecating the format was the fix precisely because hardening the decoder is not available: there is nothing in a conforming decoder to harden.
The general rule this joins is the one Two readers, one byte string states from the attacker's side: an undeclared encoding is an encoding chosen by somebody else. Naming the charset at the boundary is the move that closes it, and item 32 in the backlog — what a charset label actually means, and what the Encoding Standard does with one — is the page that will take that apart properly.
The one tool not to check this with¶
iconv knows UTF-7 on both platforms this library tests, and the two decoders disagree. Measured 2026-09-08 — six inputs decoded, plus the encoder in both builds, which writes caf+AOk- for café either way:
iconv -f UTF-7 -t UTF-8, six inputs, both builds
input BSD, Darwin 25.6.0 GNU, glibc 2.39 agree?
(UTF-7 bytes) macOS 26.6.2 ubuntu:24.04
------------------ ---------------------------- ------------------------- ------
+AOk- c3 a9 2d c3 a9 NO
caf+AOk- 63 61 66 c3 a9 2d 63 61 66 c3 a9 NO
+ADw-script+AD4- 3c 73 63 72 69 70 74 3e 2d 3c 73 63 72 69 70 74 3e NO
a+AOk-b 61 c3 a9 62 61 c3 a9 62 yes
+AOk-+AOk- c3 a9 c3 a9 2d c3 a9 c3 a9 NO
a+AOk-b+AOk 61 c3 a9 62 c3 a9 61 c3 a9 62 c3 a9 yes
exit status 0 on every row, on both builds
RFC 2152 says a - that terminates a shift sequence is absorbed. BSD iconv absorbs it everywhere except at the end of the input, where it emits the byte 2d as a character — so iconv -f UTF-8 -t UTF-7 | iconv -f UTF-7 -t UTF-8 does not round-trip its own output on a Mac, and both halves exit 0. Mid-string terminators are handled correctly on both builds, which is why the bug survives casual testing: it only shows on the last character of the file. That is why this page has no ## In the terminal section and no recorded iconv key — there is no key that matches both runners — and it is the same shape as the rest of CONTRIBUTING's platform list: the wrong answer, silently, with a successful exit status, on the machine that is on your desk.
If you are coming from Python or ABAP¶
Python. The codec is right there and takes no ceremony, which is the risk and the teaching aid at once. Two habits follow. Never treat x.encode('utf-7').decode('utf-7') == x as evidence about a filter: the encoder chose one spelling out of thousands and your test never saw the others, which is what section 3 counts. And treat a successful decode('utf-7') as unfinished work rather than a result — the string it returns may hold a NUL, a C0 control or a lone surrogate, so the UnicodeEncodeError arrives at the next stage, in code that never mentioned UTF-7. If you are decoding IMAP mailbox names, note that the variant there is not this codec: it uses , in place of / and & in place of +, so 'utf-7' is the wrong argument even where UTF-7 is the right idea.
ABAP. Ask your own system what it knows before you assume anything here, because a catalogue is a decision somebody made and SAP's numbering is its own. What is worth expecting is the shape Rust's standard library has: the tables a platform carries are the tables its inputs can be read with, and that list is a security boundary rather than a convenience. Where this page reaches an ABAP system is inbound mail and inbound HTTP, not storage. A Content-Type header arriving with a charset your system does not know is the case to handle explicitly rather than let a default answer, because the default is a table somebody else chose; and an xstring that reached you over a seven-bit path may have been through a transformation nothing in the payload records. Verify any code page number against your own system rather than a document. (Not machine-checked — CI cannot run ABAP.)
Try it¶
cd 03_Encodings/utf7_and_the_seven_bit_transport/examples
python3 utf7_and_the_seven_bit_transport_py.py
rustc --edition 2024 utf7_and_the_seven_bit_transport_rs.rs -o /tmp/u7 && /tmp/u7
Reproduce the iconv split on your own machine, because the table above is a claim about two builds and yours is one of them:
Five bytes back means your iconv follows the RFC; six, with a 2d on the end, means it does not. Then check the exit status of both halves.
Grep your own code for the codec. grep -rn "utf-7\|utf7\|UTF7\|65000" . across a service you own, including its dependencies' vendored copies. Every hit is either mail handling — where it may be correct — or a place where an encoding nobody chose has become reachable. The Windows code page number 65000 is the one that will not show up in a search for the word.
Then find the stage that decides. Take one request path and write down where the charset is named and where it is guessed. UTF-7 is gone from the browsers, but the pattern that made it dangerous is not: any stage that infers an encoding rather than being told one is a stage an attacker gets a vote in.
Practice¶
Five byte strings, one tag, and the one an encoder will never write. Here are five candidates. Predict, before running anything, which of them a UTF-7 decoder turns into <script> — and for the one that fails, say what rule it breaks, given that it differs from a working candidate by a single letter.
Then: which of the five will '<script>'.encode('utf-7') ever produce, and why is that not a contradiction?
Last, a width question. 'é'.encode('utf-7') is five bytes and '€'.encode('utf-7') is five bytes. Write down how many bytes '😀'.encode('utf-7') is, and derive the number rather than guessing it.
Answers
Verified output of utf7_and_the_seven_bit_transport_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. WHICH OF THE FIVE DECODE TO THE TAG
------------------------------------------------------------------------
YES <script> -> '<script>'
YES +ADw-script+AD4- -> '<script>'
YES +ADw-script+AD4 -> '<script>'
no +ADx-script+AD4- -> UnicodeDecodeError -- non-zero padding bits
YES +ADwAcwBjAHIAaQBwAHQAPg- -> '<script>'
Four of the five, and the four are not near-misses of each other:
1 direct, no shift sequence at all
2 each bracket in its own run, explicitly shifted out
3 the same, with the last run ended by the end of the input
5 every character of the tag inside ONE run
The one that fails is 4, and it fails for a reason that has nothing
to do with angle brackets: 'ADx' carries the same 16 payload bits as
'ADw' plus two padding bits that are not zero, and RFC 2152 calls
that ill-formed. A rule about arithmetic, not about characters.
2. WHAT THE ENCODER PRODUCES
------------------------------------------------------------------------
'<script>'.encode('utf-7') -> b'<script>'
One of the five, and it is the one with no shift sequence in it.
'<' and '>' are Set O in RFC 2152 -- OPTIONAL direct characters --
so passing them through is a legal choice this encoder makes and a
decoder is not allowed to make. That is the asymmetry: feeding your
own encoder's output back through your own filter tests one spelling
out of the family, and it is the spelling nobody attacks with.
3. THE WIDTH QUESTION
------------------------------------------------------------------------
code point UTF-16BE code units UTF-7 width char
U+00E9 00 e9 1 +AOk- 5 bytes é
U+20AC 20 ac 1 +IKw- 5 bytes €
U+1F600 d8 3d de 00 2 +2D3eAA- 8 bytes 😀
Five bytes for the two BMP characters and eight for the emoji, and
the arithmetic says why. A run carries UTF-16 code units: 16 bits
each, re-cut into 6-bit base64 symbols, so one unit needs ceil(16/6)
= 3 symbols and two units need ceil(32/6) = 6. Add the '+' and the
'-' and you get 5 and 8.
The trap in the question is the word 'character'. The emoji is one
character and one code point, and it is TWO code units -- so its run
is twice as long, exactly as it is in UTF-16 and for the same reason.
A 1997 mail-safe format inherits the surrogate pair whole.
4. THE CHECK
------------------------------------------------------------------------
Base64 of the UTF-16BE bytes, done by hand and compared:
b64(UTF-16BE) = AOk codec said AOk same é
b64(UTF-16BE) = 2D3eAA codec said 2D3eAA same 😀
Nothing in a UTF-7 run is a UTF-7 invention. It is Base64, over
UTF-16, with the padding removed -- two encodings this library has
already taught, stacked. The only thing UTF-7 adds is the shift, and
the shift is the part that turned out to be dangerous.
See also¶
- Binary to text — the first answer to the same seven-bit constraint: re-cut every byte, and lose legibility to keep the guarantee
- Escaping into ASCII — the second answer, and the one scheme in the family that writes down which table it used
- The alphabet is not the encoding — what "modified Base64" means, and why a changed alphabet is not a changed encoding
- Overlong sequences — the same "one string, several spellings" argument about UTF-8, where the format ruled the alternatives ill-formed instead
- UTF-16 and surrogates — the layer inside every UTF-7 shift sequence, and where the lone surrogate comes from
- Two readers, one byte string — the attack this page explains the why of, alongside the GBK trail byte that eats a backslash
- Validation is a boundary — why "these bytes are well-formed UTF-8" is true of the payload and answers nothing
- RFC 2152: UTF-7 ↗ — the specification, its three character sets, and its own advice to use it only for mail
UTF7Encoding↗ and SYSLIB0001 ↗ — the .NET obsoletion, and Microsoft's three reasons for it- Encoding Standard ↗ and HTML §13.2.3.3 ↗ — the closed list, and the sentence that names UTF-7 as excluded by it