DEX¶
Level: 201 · for anyone with a hex editor open
One line: A DEX header names its own byte order with a constant, 0x12345678, checksums itself twice — Adler-32 then SHA-1, over two ranges that both stop short of the magic — and stores every string as a ULEB128 count of UTF-16 code units followed by modified UTF-8, the same MUTF-8 as a Java class file, where a NUL is two bytes and an emoji is six.
What the loader reads¶
Dalvik Executable is the bytecode container Android compiles Java and Kotlin into — classes.dex inside every APK — specified in the Android Open Source Project's Dalvik executable format ↗ page. The header is 112 bytes, 0x70, and every field after the signature is a 32-bit integer in the order the header itself declares:
| offset | field | width | what it says |
|---|---|---|---|
| 0 | magic |
8 | dex\n035\0 — the word, a newline, three version digits, a NUL |
| 8 | checksum |
4 | Adler-32 of everything from offset 12 to the end |
| 12 | signature |
20 | SHA-1 of everything from offset 32 to the end |
| 32 | file_size |
4 | |
| 36 | header_size |
4 | 0x70 |
| 40 | endian_tag |
4 | 0x12345678; a reader that sees 0x78563412 has the order wrong |
| 44 | link_size, link_off |
4 each | |
| 52 | map_off |
4 | |
| 56 | string_ids_size, string_ids_off |
4 each | the string table: one 32-bit offset per string |
| 64 | type_ids, proto_ids, field_ids, method_ids, class_defs |
8 each | five more size/offset pairs |
| 104 | data_size, data_off |
4 each | where the strings and the code are |
The versions Ghidra's DexConstants knows are 035, 036, 037, 038, 039 and 040, plus the pre-release 009; the loader compares only dex\n and lets the digits vary.
In Python¶
Verified output of dex_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. THE HEADER NAMES ITS OWN BYTE ORDER
------------------------------------------------------------------------
magic 64 65 78 0a 30 33 35 00 b'dex\n035\x00'
endian_tag 78 56 34 12 read LE 0x12345678 read BE 0x78563412
ENDIAN_CONSTANT is 0x12345678. A reader that gets 0x78563412 has
read a big-endian file with a little-endian rule, and the spec
names that value too, REVERSE_ENDIAN_CONSTANT -- the same trick as
Mach-O's CIGAM, as a 32-bit field instead of the magic itself.
2. TWO CHECKSUMS OVER TWO RANGES
------------------------------------------------------------------------
checksum at 8 d0 17 ef 91 adler32 of bytes 12..end recomputed 0x91ef17d0 ok
signature at 12 4809cbd3fc2e7974a5c4db8b... sha1 of bytes 32..end ok
flip one bit in the last string checksum FAILS signature FAILS
change the version digit at byte 4 checksum ok signature ok
Each check covers everything after itself and nothing before: the
magic and the version are inside neither, so 'dex\n935' verifies.
3. MODIFIED UTF-8 IS UTF-8 WITH TWO EXCEPTIONS
------------------------------------------------------------------------
string UTF-8 MUTF-8 note
'café' 63 61 66 c3 a9 63 61 66 c3 a9 identical below U+10000, except NUL
'😀' f0 9f 98 80 ed a0 bd ed b8 80 one code point, two surrogates, six bytes
'a\x00b' 61 00 62 61 c0 80 62 NUL is c0 80, so no byte is zero
c0 80 is an overlong sequence and a UTF-8 decoder must reject it;
ed a0 bd is a surrogate and so must that. Modified UTF-8 wants both,
for one reason: a string can then be NUL-terminated in C and hold
any character. The Java class file uses the same encoding.
4. A STRING IS A COUNT OF UTF-16 UNITS, THEN MUTF-8, THEN NUL
------------------------------------------------------------------------
string_id[0] -> offset 124 uleb128 04 = 4 UTF-16 units 5 bytes of MUTF-8 63 61 66 c3 a9
string_id[1] -> offset 131 uleb128 02 = 2 UTF-16 units 6 bytes of MUTF-8 ed a0 bd ed b8 80
string_id[2] -> offset 139 uleb128 03 = 3 UTF-16 units 4 bytes of MUTF-8 61 c0 80 62
utf16_size is not a byte count and not a code-point count: café is
4 and 5 bytes, the emoji is 2 and 6 bytes. And the terminator that
ends the search is a real 00, which is why NUL inside is c0 80.
5. ULEB128, THE LENGTH FIELD THAT GROWS
------------------------------------------------------------------------
4 -> 04
127 -> 7f
128 -> 80 01
300 -> ac 02
16384 -> 80 80 01
Seven bits per byte, low bits first, top bit set on every byte but
the last. 127 is one byte and 128 is two, which is the same shape
as UTF-8 itself: the length of the length depends on the value.
Modified UTF-8¶
Section 3 is the page's subject. Modified UTF-8 is UTF-8 with two deliberate violations: U+0000 is written as the two-byte sequence c0 80, and a code point above U+FFFF is written as its two UTF-16 surrogates, each encoded as three bytes — six in all where UTF-8 uses four. Both are things a conforming UTF-8 decoder must reject: c0 80 is the overlong form of NUL, and ed a0 bd is a surrogate, which UTF-16 and surrogates explains has no business in UTF-8 at all.
The reason is one property: a modified-UTF-8 string never contains a zero byte, so it can be handed to C code that stops at NUL and still hold every character. The Java class file invented the encoding for its constant pool, Dalvik inherited it, and Ghidra's own GZF container has it in its item names because Java's writeUTF writes it. Three formats on this list, one non-standard encoding, and CESU-8 — the same surrogate trick without the NUL rule — is what SAP HANA stores.
Three lengths for one string¶
A string_data_item is a ULEB128 number, then the MUTF-8 bytes, then a real NUL. The number is utf16_size: not the byte count, not the code-point count, but how many 16-bit units the string takes in UTF-16 — because that is the length a Java String reports, and the runtime wants it without decoding. Section 4 reads three: café is 4 units and 5 bytes, the emoji 2 units and 6 bytes, a\0b 3 units and 4 bytes. A code point is not a character counted five rulers over one string; a DEX file carries two of them for every string and derives the third from the terminator.
ULEB128 itself is section 5: seven bits per byte, low bits first, the high bit as a continuation flag — the length of the length depends on the value, which is UTF-8's own shape applied to integers.
Two checksums that do not cover the magic¶
The checksum covers bytes 12 to the end and the signature covers 32 to the end, so each protects everything after itself and nothing before. Section 2 flips one bit in the last string and both fail; it changes the version digit at byte 4 and both pass — the magic, the checksum field and the signature field are outside both ranges by construction, since a hash cannot cover the bytes it is stored in. dex\n935\0 verifies. What a format's integrity check does not cover is the second half of adding one, as Framing a format found for Intel HEX.
application/octet-stream
Dalvik dex file version 035
Identical on both builds, and the version is read out of the magic. Neither checks the checksum.
What Ghidra checks¶
DexLoader ↗, in the FileFormats module, parses a DexHeader and accepts the file when the first four bytes are dex\n; it then loads under a fixed language, Dalvik, since a DEX has no machine field to consult. Its name is Dalvik Executable (DEX), and a sibling CDexLoader handles the compact variant. The APK loader is this one with a ZIP in front.
If you are coming from Python or ABAP¶
Python. There is no MUTF-8 codec in the standard library, and the twelve-line encoder in dex_py.py is the whole of it; decoding is the same in reverse, with c0 80 mapped to \x00 and a surrogate pair reassembled. s.encode('utf-8', 'surrogatepass') is not it — it writes a lone surrogate as three bytes, which is half of the rule, and keeps NUL as one byte, which is the other half wrong. len(s.encode('utf-16-le')) // 2 is utf16_size. zlib.adler32 and hashlib.sha1 are both standard library, which is why the program can verify a real header.
ABAP. (Not machine-checked — CI cannot run ABAP.) The three lengths are the point that transfers. strlen on a string counts UTF-16 units — an emoji is 2 — and xstrlen on its UTF-8 xstring counts bytes, so a DEX file is carrying an ABAP strlen beside the bytes. Modified UTF-8 has no code page number in the system and cl_abap_codepage will not produce it; convert to UTF-8 and patch 00 and the surrogates by hand if a DEX ever has to be written from ABAP, and verify any code-page number against the system.
Try it¶
- Unzip any APK and
xxd -l 112 classes.dex. Read the version out of bytes 4–7 andendian_tagout of bytes 40–43. python3 -c "import zlib,sys; d=open('classes.dex','rb').read(); print(hex(zlib.adler32(d[12:])), d[8:12][::-1].hex())". The two should match.- Find
string_ids_offand follow the first offset. Decode the ULEB128 by hand and count the bytes to the NUL. grep -c $'\xc0\x80' classes.dexunderLC_ALL=C. Every hit is a NUL inside a string.- Compile a Java class with an emoji constant,
javap -vit, and compare theUtf8bytes with whatdex_py.pybuilds for the same string.
See also¶
- Java class file — the format that invented modified UTF-8, with a 16-bit byte count where DEX has a ULEB128 unit count
- Android APK — the ZIP this file travels in
- Overlong sequences — why
c0 80is forbidden in UTF-8 and wanted here - UTF-16 and surrogates — where the two three-byte halves of an emoji come from
- A code point is not a character — the five rulers, two of which a DEX string carries
- ELF — the other format here with a byte-order field, one byte wide instead of four
- Dalvik executable format ↗ — the header table and the
string_data_itemdefinition this page was checked against