file guesses¶
Level: 101 · for anyone with a terminal
One line: file reports what an encoding looks like in the first 64 KiB, not what it is — utf-8 is an inference, iso-8859-1 is a proof of a negative, us-ascii is a statement that no experiment could contradict (bar one byte file waves through), and a file that is UTF-8 from byte 65536 onward reports as ASCII.
The mechanism is next door. File type is four questions has the magic database, the three test classes and their fixed order, and the point where a signature file knows runs out and a heuristic it is running takes over. This page is the heuristic — the last stage, where there is nothing to look up because an encoding leaves no signature, and file has to reason from the bytes alone.
What there is to reason from¶
A file does not record its encoding. There is no header, no field, no convention that a general-purpose tool could read — outside the handful of formats that carry one, and outside the BOM, which is a mark somebody chose to write and most files do not have. So file has exactly one input, the bytes, and it asks three questions of them in order:
- Is there a mark? A BOM is a fact in the file. This is the only branch that produces evidence rather than inference — and section 4 below finds two asterisks on it.
- Are they all under 128? Then say
us-asciiand stop, because every 8-bit table agrees down there and no further question has an answer. One byte above 127 passes this test anyway, and section 6 finds it by trying all of them. - Do the high bytes form valid UTF-8? If yes,
utf-8. If no, the file has high bytes and is not UTF-8, which narrows it to some 8-bit table and no further —iso-8859-1if the bytes look like a plausible one,unknown-8bitif they do not.
That is the whole ladder, and none of it is statistics. file does not count letter frequencies or compare against a language model; it validates, and falls through when validation fails.
In the terminal¶
Verified output of file_guesses_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
1. FIVE ANSWERS, AND ONLY TWO OF THEM ARE EVIDENCE
ascii.txt us-ascii 48 65 6c 6c 6f 2c 20 70
utf8.txt utf-8 63 61 66 c3 a9 20 e2 82
utf8_bom.txt utf-8 ef bb bf 63 61 66 c3 a9
latin1.txt iso-8859-1 63 61 66 e9 20 a4 20 31
cp1252.txt unknown-8bit 61 80 62 0a
utf16le_bom.txt utf-16le ff fe 63 00 61 00 66 00
utf16le_nobom.txt binary 63 00 61 00 66 00 e9 00
blob.bin binary 00 01 02 ff fe 7f 00
empty.txt binary
us-ascii and utf-8 are inferences from the bytes. iso-8859-1 is a
proof of a NEGATIVE — not valid UTF-8, so some 8-bit table, and file
does not pretend to know which. unknown-8bit is a weaker negative
still: 8-bit, and not even a plausible ISO-8859 one. binary is a
surrender, and the two utf-16 rows are the only lines here that rest
on something written IN the file.
2. WHY A PURE-ASCII FILE IS EVERY ENCODING AT ONCE
read as US-ASCII -> 48 65 6c 6c 6f 2c 20 70 6c 61 69 6e 20 41 53 43 49 49 2e 0a
read as ISO-8859-1 -> 48 65 6c 6c 6f 2c 20 70 6c 61 69 6e 20 41 53 43 49 49 2e 0a
read as ISO-8859-2 -> 48 65 6c 6c 6f 2c 20 70 6c 61 69 6e 20 41 53 43 49 49 2e 0a
read as CP1252 -> 48 65 6c 6c 6f 2c 20 70 6c 61 69 6e 20 41 53 43 49 49 2e 0a
read as CP850 -> 48 65 6c 6c 6f 2c 20 70 6c 61 69 6e 20 41 53 43 49 49 2e 0a
Five tables, five identical answers. Below 0x80 they agree by
construction, so there is no experiment that could tell them apart
on this file. 'us-ascii' is not file being cautious — it is the
strongest true statement available: every byte is under 128.
3. THE WHOLE VERDICT TURNS ON ONE BYTE
one.txt 63 61 66 c3 a9 0a utf-8
two.txt 63 61 66 e9 0a iso-8859-1
three.txt 63 61 66 65 0a us-ascii
Same word, three spellings. c3 a9 is a legal UTF-8 pair, so file
validates it and says utf-8. e9 alone cannot start a UTF-8 sequence,
so file falls through to 'it has a high byte and is not UTF-8'. That
is the entire UTF-8-versus-Latin-1 decision: a validity check with a
fallback, not a table lookup and not statistics.
4. A BOM IS A FACT — WITH TWO ASTERISKS
UTF-8 BOM, then invalid bytes ef bb bf e9 e9 e9 0a iso-8859-1
ff fe, then A B in UTF-16LE ff fe 41 00 42 00 utf-16le
ff fe, then two zero bytes ff fe 00 00 41 00 00 00 utf-32le
First asterisk: a UTF-8 BOM does not settle it. file read the body,
found bytes that are not UTF-8, and reported iso-8859-1 anyway — the
mark is evidence about intent, not a licence to skip the check.
Second asterisk: ff fe IS the UTF-16LE mark and ALSO the first half
of the UTF-32LE one. Two more zero bytes and the same prefix means a
different encoding, so 'the BOM is a fact' is a fact about four
bytes, not two.
5. FILE DOES NOT READ YOUR FILE — IT READS THE FIRST 64 KIB OF IT
65534 bytes of ASCII, then c3 a9 -> utf-8
65535 bytes of ASCII, then c3 a9 -> iso-8859-1
65536 bytes of ASCII, then c3 a9 -> us-ascii
Three files that differ only in padding, and three different answers.
At 65534 both bytes of the e-acute fall inside the window and file
says utf-8. At 65535 the window ends BETWEEN them: file sees a c3
with nothing after it, which is not valid UTF-8, and reports
iso-8859-1 for a file that is perfectly good UTF-8. At 65536 the
character is outside the window entirely and the file reads as pure
ASCII. The verdict is about a prefix, and the file is not the prefix.
6. THE ONE BYTE ABOVE 127 THAT FILE CALLS us-ascii
nel.txt us-ascii 57 61 69 74 85 20 77 68 61 74 0a
read as CP1252 -> 57 61 69 74 e2 80 a6 20 77 68 61 74 0a
read as ISO-8859-1 -> 57 61 69 74 c2 85 20 77 68 61 74 0a
read as CP850 -> 57 61 69 74 c3 a0 20 77 68 61 74 0a
read as US-ASCII -> iconv refuses it
all 128 high bytes, each between an a and a b with a newline after:
96 iso-8859-1, 31 unknown-8bit, and us-ascii for: 85
Section 2 said us-ascii means every byte is under 128. This file has
an 85 in it and gets us-ascii anyway: file counts 85 — NEL, the C1
control that EBCDIC's newline turns into — as a plain text byte. The
file is not ASCII, iconv refuses it as ASCII, and three tables read
three different characters: an ellipsis, the control itself, and an
a with a grave accent. The sweep tried all 128 high bytes and 85 is
the only one. In Windows-1252 it is the ellipsis, so an otherwise
ASCII file with a single … in it gets exactly this answer.
7. TWENTY-FIVE BYTES MAKE A FILE binary, AND NUL IS ONLY ONE OF THEM
binary 00-06 0e-1a 1c-1f 7f 25 values
us-ascii 07-0d 1b 20-7e 103 values
The 128 low bytes, each between an a and a b with a newline after.
NUL is one byte in twenty-five: the C0 control characters other than
bell, backspace, tab, line feed, vertical tab, form feed, carriage
return and escape, plus DEL. So a single 01 with no NUL anywhere
makes a file binary to file, while utf16le_bom.txt in section 1 is
full of NULs and is utf-16le. 'Has a NUL' and 'is binary' are
separate questions, and file does not ask the first one at all.
with the newline (above): 25 binary
without it: 21 binary, ebcdic for 05 15 16 7f
Before file gives up on a byte string it tries it as EBCDIC text,
and in 'a?b' with no newline these four pass that test; add the
newline and they are binary again. So twenty-five is a count for
this shape of file, and one byte nobody was asking about moves it.
Section 1 has five answers where the stub for this page predicted four. unknown-8bit is the fifth, and it is a weaker negative than iso-8859-1: not valid UTF-8, and not a plausible ISO-8859 file either. The byte that triggered it is 0x80 — the euro sign in Windows-1252 and an unassigned C1 control in Latin-1 — which is exactly the byte on which cp1252 and Latin-1 differ. So unknown-8bit is often not an exotic file at all; it is a perfectly ordinary Windows CSV.
Section 2 is why us-ascii is the strongest true answer, not the cautious one. The same file decoded through five different tables produced five identical results, because all five agree below 0x80 by construction. There is no experiment that distinguishes them on that file. So us-ascii does not mean "probably ASCII"; it means "every byte here is under 128, and therefore this file is simultaneously valid ASCII, Latin-1, Latin-2, CP1252 and CP850, with the same meaning under all of them." The answer is complete. It is the question that has no more to give. Section 6 is the one byte for which none of that is true.
Section 3 is the UTF-8-versus-Latin-1 decision in three lines. c3 a9 is a legal two-byte UTF-8 sequence, so it validates and file says utf-8 — an inference, and a strong one, because random 8-bit text almost never accidentally forms valid UTF-8. e9 alone cannot begin a UTF-8 sequence, so validation fails and file falls back to high bytes, not UTF-8. Neither branch involved a table of languages.
Section 5 is the finding, and it is the reason to distrust the answer on anything large. file reads a bounded prefix — 64 KiB — and three files differing only in how much ASCII padding comes first got three different verdicts. The middle one is the sharp edge: at 65535 bytes of padding the window ends between the two bytes of an é, so file sees a c3 with nothing after it, calls that invalid UTF-8, and reports iso-8859-1 for a file that is perfectly good UTF-8. One byte of padding either way and the answer changes twice. Every verdict on this page is a verdict about a prefix, and on a log file or a database export the prefix is not the file.
Section 6 is where us-ascii stops being a proof, and it is file that breaks it. A file whose only high byte is 85 comes back us-ascii, because file counts 85 — NEL, the C1 control that EBCDIC's newline becomes in Unicode — among the bytes of plain text; the default wording even reports ASCII text, with LF, NEL line terminators. The file is not ASCII: iconv refuses it as ASCII, and the three tables that accept it read three different characters, which is exactly the situation section 2 says us-ascii rules out. The sweep beside it asks about all 128 high bytes one at a time, and 85 is the only one that comes back us-ascii. It is not an exotic case. 85 is the ellipsis in Windows-1252, and Word's AutoCorrect turns three typed dots into one, so an otherwise-ASCII file exported from a Windows program with a single … in it is precisely the file that gets the wrong answer — identical on file-5.41 (macOS), 5.44 (Debian 12) and 5.45 (Ubuntu 24.04). strings meets the same character from the other side: in UTF-8 it is three bytes, and one build of strings deals them across two lines.
Section 7 is the verdict this page otherwise leaves alone: binary. Twenty-five of the 128 low bytes make file give up — the C0 control characters other than bell, backspace, tab, line feed, vertical tab, form feed, carriage return and escape, plus DEL — and the NUL is only one of them. The count needs its newline: each byte goes between an a and a b with a newline after, and without that newline four of the twenty-five — 05, 15, 16 and 7f — come back ebcdic, because file tries a byte string as EBCDIC text before it gives up. The sweep prints both counts. A single 01 with no NUL anywhere makes a file binary, while section 1's UTF-16 file is full of NULs and comes back utf-16le, because it carries a byte-order mark. So has a NUL is not what file means by binary, and the tools that test for a NUL instead — Binary is a verdict, not a property measures git and grep — disagree with file in both directions. Identical on file-5.41, 5.44 and 5.45.
Ask for the MIME form, always¶
Every measurement above uses --mime-encoding. That is not a style preference.
file-5.41 macOS 26.6.2
file-5.44 debian:bookworm-slim
file-5.45 ubuntu:24.04
--mime-encoding identical on all three, on every file
--mime-type identical on all three, on every file
the default prose identical on these files — and NOT stable in general:
upstream moved the word 'executable' from one noun to
another between 5.41 and 5.44, so macOS says
'POSIX shell script text executable, ASCII text' where
Debian and Ubuntu say
'POSIX shell script, ASCII text executable'
The default output of file is English prose written for a human, and upstream edits it. A script that greps it is depending on a release note. This page's stub was written because of that, months before anyone measured it — its last bullet said "file's wording changes between versions, so the example asks --mime-encoding only" — and the measurement, when it was finally taken across three builds and nine files, agreed with the guess exactly.
There is one more reason to prefer the long spelling, and it is worse than a wording change:
macOS (file-5.41) ubuntu:24.04 (file-5.45)
file -i f f: regular file exit 0 f: text/plain; charset=utf-8
file -I f f: text/plain; … file: invalid option -- 'I' exit 1
file --mime f f: text/plain; charset=utf-8 f: text/plain; charset=utf-8
-i and -I are swapped, and the two failures are not symmetrical. Getting it wrong on Ubuntu is loud — an invalid-option error and exit 1. Getting it wrong on macOS is silent: file -i there means do not look inside, so it answers regular file, exits 0, and a script that greps the output for charset= simply finds nothing and concludes whatever its else branch says. Same genre as base64 -i, which means input file on macOS and ignore garbage on GNU. Write --mime-encoding, which is spelled the same everywhere.
The one thing that is evidence, and its two asterisks¶
Section 4 put two dents in "a BOM is a fact", and both are worth carrying:
- A UTF-8 BOM does not settle the question.
ef bb bffollowed by bytes that are not UTF-8 reportediso-8859-1—fileread the body and let it overrule the mark. That is the right call: a BOM records what somebody intended, and intent is not a guarantee about the rest of the file. ff feis two marks. It is the UTF-16LE BOM, and it is also the first half of the UTF-32LE one (ff fe 00 00). Two zero bytes after it and the verdict flips fromutf-16letoutf-32le— which means a genuine UTF-16LE file whose first character happens to beU+0000is indistinguishable from a UTF-32LE file by its mark alone. The fact is about four bytes, not two.
The validator to reach for next, and the one place it lies¶
file triages; iconv decides, because it converts every byte or stops. The idiom this library uses for UTF-8 is iconv -f UTF-8 -t UTF-8, judged on its exit status, and it behaves identically on both platforms. Generalising it to ASCII does not:
macOS 26.6.2 ubuntu:24.04
iconv -f US-ASCII -t US-ASCII exit 0 ✗ exit 1 ✓
iconv -f US-ASCII -t UTF-8 exit 1 ✓ exit 1 ✓
With ASCII named on both sides, Apple's iconv takes a substituting path — it writes 61 3f 62, a question mark where the byte was — and reports success. GNU refuses. Measured across six inputs, macOS returns 0 for every one where the invalid byte is not the entire file; the only case the two agree on is a file consisting of nothing but the bad byte. Naming a different target avoids the path entirely, and -f US-ASCII -t UTF-8 was identical on both platforms on every input tried. The general rule is narrower than it looked: -f X -t X is a validator for UTF-8, not for every X.
If you are coming from Python or ABAP¶
Python. There is no file in the standard library and that absence is honest — the ladder above is fifteen lines you can write yourself, and writing it is the fastest way to stop treating the answer as authoritative:
def sniff(data: bytes) -> str: # the same three questions
if data[:3] == b"\xef\xbb\xbf": return "utf-8-sig" # a mark: evidence
if data[:2] in (b"\xff\xfe", b"\xfe\xff"): return "utf-16"
if not any(b > 0x7F for b in data): return "us-ascii"
try: data.decode("utf-8"); return "utf-8" # validates: inference
except UnicodeDecodeError: return "some 8-bit table" # a negative
Note what it cannot return: which 8-bit table. That is the same wall file hits, and no amount of code gets past it. The third-party chardet and charset-normalizer go further by guessing with statistics — letter frequencies against language models — which is a genuinely different kind of answer and comes with a confidence score for the honest reason that it can be wrong. When you find yourself wanting one, the real fix is upstream: get the encoding written down.
ABAP. There is no equivalent and the gap is architectural rather than accidental. OPEN DATASET ... IN LEGACY TEXT MODE CODE PAGE '1100' takes the code page as an input, never as a question — the system does what you say and produces a string either way, so a wrong code page here is precisely the silent damage a wrong -f is to iconv. There is no CL_ABAP_* class that will sniff a file for you. What you can do is the second half of section 2 above: read the file as an xstring, look for any byte above 0x7F, and if there are none the code-page question does not arise. If there are, you need the sender to tell you — and on an interface that is a line in the contract, not a runtime decision. (Not machine-checked — CI cannot run ABAP.)
Try it¶
- Run
file --mime-encodingover a directory of real files — an export folder, a repo,/etc— and count how many come backus-ascii. That number is how much of your data the question does not apply to. - Take the largest text file you have and compare
file --mime-encoding big.txtagainsticonv -f US-ASCII -t UTF-8 big.txt >/dev/null; echo $?. If the first saysus-asciiand the second says1, you have found section 5 in the wild. - Run
file -iandfile -Ion the same file on a Mac and on Linux. One of the four answers is wrong and does not say so. - Find something in your own tooling that branches on
file's output. Check whether it greps the English prose, and whether it would survive the wordexecutablemoving.
Practice¶
Three files, one verdict. Each of these reports us-ascii from file --mime-encoding:
a.txt 12 bytes, all under 0x80
b.txt 65536 bytes of 'a', then 63 61 66 c3 a9 0a
c.txt 65536 bytes of 'a', then 63 61 66 e9 0a
Say why all three get the same answer, which of them are actually ASCII, and write down one command that tells them apart. Then the second half: of every answer file --mime-encoding can give, which single one rests on something written in the file rather than read off it?
Answers
Verified output of file_guesses_kata_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
1. THREE FILES, ONE VERDICT
a_really.txt 12 bytes file says us-ascii
b_utf8_late.txt 65542 bytes file says us-ascii
c_latin1_late.txt 65541 bytes file says us-ascii
Only the first one is ASCII. The other two hold a non-ASCII character
past byte 65536, which is outside the window file(1) reads, so all
three get the same answer and two of them are wrong.
2. THE TOOL THAT READS THE WHOLE FILE
a_really.txt all ASCII? exit=0 valid UTF-8? exit=0
b_utf8_late.txt all ASCII? exit=1 valid UTF-8? exit=0
c_latin1_late.txt all ASCII? exit=1 valid UTF-8? exit=1
iconv converts every byte or stops, so its answer is about the file
rather than about a prefix. Read the two columns together: 0/0 is
ASCII, 1/0 is UTF-8 with something above 127 in it, 1/1 is neither —
an 8-bit table, and no tool in the terminal can tell you which.
Note the FIRST command: it is -f US-ASCII -t UTF-8, not the -f X -t X
form this library uses to validate UTF-8. Apple's iconv does not
refuse a high byte when ASCII is named on BOTH sides — it substitutes
a question mark and exits 0. Naming a different target avoids that
path and the two platforms agree. The measurement is on the page.
3. WHICH ANSWER WAS EVIDENCE
ff fe 41 00 utf-16le
the three files above us-ascii, three times
The BOM row is the only claim on this page that rests on something
written IN the file. us-ascii, utf-8 and iso-8859-1 are all read OFF
the bytes — the first two by validating, the third by failing to.
A guess that reads 64 KiB is still a guess; a guess that reads the
whole file is a slower guess. Only a mark, a manifest, a Content-Type
header or a contract makes it a fact.
4. WHAT TO DO ABOUT IT
* Ask --mime-encoding, never the English wording: the prose moved
the word 'executable' between file 5.41 and 5.44 and the MIME
output did not change at all.
* Use file(1) to triage and iconv to decide.
* If you control the producer, write the encoding down somewhere a
program can read: that is the only way the answer stops being an
inference.
See also¶
- File type is four questions — the stages before this one, and the four mechanisms that answer "what kind of file is this?"
- Inspecting a file — the three shapes of
file's prose answer, beside the dumps they describe iconv— the tool that reads the whole file, and the family it still gets wrong- Byte order and the BOM — the mark, and what it does and does not promise
- Windows-1252 vs Latin-1 — the byte behind most
unknown-8bitanswers - The table has a version — the same failure one layer up: an out-of-date table answers
Cnwith no hedge either stringshas a printable set, not an encoding — the third tool for poking at an unknown file, and what it counts as readable