Locale and LC_CTYPE

Level: 201 · working knowledge

One line: The locale is not one setting but six independent variables, LC_CTYPE is the one that decides what "a character" means, and the two things worth knowing are that almost every command-line tool obeys it and that Python, since 3.7, does not.

Everywhere else in this chapter a tool has one behaviour and you learn it. Here the same tool, on the same file, in the same second, gives two different answers depending on a variable nobody set on purpose. That is not a bug in wc; it is wc doing what it was told by an environment that came from three layers away — your Region setting, your terminal profile, your Dockerfile, or nothing at all.

The locale is six variables, not one

LC_CTYPE is the one this library cares about: it decides which byte sequences are characters, which are printable, and what uppercase means. The others are independent of it and of each other — LC_COLLATE for sort order, LC_NUMERIC for the decimal separator, LC_TIME, LC_MONETARY, LC_MESSAGES. Setting one says nothing about the rest, which is section 3 of the run below and is the part that surprises people.

Three variables decide what any of them end up as, and they are consulted in a fixed order: LC_ALL overrides everything, a specific LC_* comes next, and LANG is the fallback for whatever was not named. LC_ALL is the sledgehammer — set it and nothing else can be consulted, which is exactly why scripts that want determinism set it and nothing else.

The terminal's own settings are not the locale

Apple's help page for this — Display high-bit characters in Terminal on Mac — walks you through three checkboxes, and it is worth reading once for what they are, as long as you notice that they belong to two different layers and only one of them is visible to a program.

• "Text encoding" (Profiles ▸ Advanced ▸ International) is Terminal decoding the bytes a program writes to it, and encoding what you type. That is a display decision, made by the emulator, about bytes it did not produce. wc cannot see it. Neither can python3, nor any example in this library.
• "Set locale environment variables on startup" sits in the same panel and is a completely different mechanism: it puts LANG / LC_* into the environment of the shell Terminal launches — derived from your Region in System Settings, not from the encoding menu above it. This is the one every program reads, and the only one this page is about.
• "Escape non-ASCII input with Control-V" (Advanced ▸ Input) touches the input path only — a literal-next prefix for line editors that needed one before a high byte.

Set the first without the second and you get a terminal decoding, correctly, bytes the program was never told to produce. That is the failure the page's title is really about, and it is why the fix for mojibake is almost never the encoding menu.

Measured 2026-09-06, before trusting any of the three: the Basic profile carries no encoding, locale, or escape key at all — not in defaults read com.apple.Terminal, and not in the shipped Basic.terminal inside the app bundle — so all three sit at built-in defaults and cannot be read back from a script. Terminal.app 2.15, macOS 26.6.2. An audit of these settings has to read the GUI, which is worth knowing before promising to check them across a fleet.

In the terminal

Verified output of locale_and_lc_ctype_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

1. THREE VARIABLES, ONE ANSWER — and the order they are consulted in
     LANG=<utf8>                        -> UTF-8
     LANG=<utf8>  LC_CTYPE=C            -> not UTF-8   <- LC_CTYPE beats LANG
     LANG=C  LC_CTYPE=C  LC_ALL=<utf8>  -> UTF-8       <- LC_ALL beats everything
     LANG is the fallback, LC_CTYPE is the specific setting, LC_ALL is the
     override. Set LC_ALL in a script and nothing else can be consulted.

2. IN THE C LOCALE THERE IS NO SUCH THING AS A CHARACTER

$ LC_ALL=C     wc -c < ./_demo.txt | tr -d ' '
6

$ LC_ALL=C     wc -m < ./_demo.txt | tr -d ' '
6
   ^ two different flags, the same answer. 'Character' means 'byte' here,
     so the question wc -m exists to ask cannot be asked.

$ LC_ALL=$U    wc -m < ./_demo.txt | tr -d ' '
5
   ^ same file, same tool, same second. One byte of it is now half a letter.

3. LC_CTYPE IS NOT LC_COLLATE — the variables really are independent

$ LC_ALL=C           sort ./_sort.txt | tr '\n' ' '; echo
A B aa az b 

$ LC_CTYPE=$U        sort ./_sort.txt | tr '\n' ' '; echo
A B aa az b 
   ^ identical. Switching the CHARACTER TYPE to UTF-8 says nothing about
     sort order, which is LC_COLLATE's job and was never asked to change.
     'The locale' is six variables; a tool reads the one it needs.

4. SO WHICH TOOLS CHANGE?
     wc -m      yes   it is a question about characters
     wc -c      no    a byte is a byte in every locale
     sort       yes, but to LC_COLLATE, not to LC_CTYPE
     od -a      yes   it asks isprint() -- see 'Inspecting a file'
     tr         it depends on WHOSE tr, which is the point of the table
                on the page: BSD tr obeys the locale and GNU tr cannot.

Section 2 is the whole lesson in three numbers. wc -c and wc -m are separate flags because they are separate questions — and in the C locale they collapse into one answer, because the C locale has no concept of a multi-byte character. It is not that wc -m gets the wrong number; it is that the question it exists to ask cannot be phrased. Switch LC_CTYPE and the same six bytes become five characters.

Section 3 is the one that catches people. Turning the character type to UTF-8 did nothing to the sort order, because sort order is LC_COLLATE and was never asked to change. "Set the locale to UTF-8" is not a single act; a program that needs both has to say both, or say LC_ALL.

Section 1 is why a script sets LC_ALL and not LANG. Setting LANG leaves any LC_* already in the environment in charge, and there usually is one — your terminal put it there. Setting LC_ALL cannot be second-guessed.

In Python

Verified output of locale_and_lc_ctype_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. A C LOCALE, AND NOTHING ELSE SET
     LC_ALL=C  LANG=C  PYTHONUTF8 unset  -- the default in a container,
     a cron job, a systemd unit and most CI runners.

2. WHAT THE LOCALE SAYS, AND WHAT PYTHON DOES ANYWAY
     locale.getencoding() is utf-8             False
     locale.getpreferredencoding() is utf-8    True
     sys.flags.utf8_mode                       1
     sys.stdout.encoding is utf-8              True
     sys.getfilesystemencoding() is utf-8      True
     UTF-8 Mode turned ITSELF on, because the locale is C (PEP 540).
     Nothing in the environment asked for it. Python read the same
     setting `wc` read and decided a machine claiming to be ASCII-only
     is describing its own configuration, not its data.

3. WHAT THAT IS PROTECTING YOU FROM
     PYTHONUTF8=1   utf8_mode 1   open() read it: OK
     PYTHONUTF8=0   utf8_mode 0   open() read it: UnicodeDecodeError
     Same file, same locale, one variable apart. With the mode off,
     open() takes its default encoding from the C locale -- ASCII --
     and the second byte of the e-acute ends the program. That is the
     crash 3.7 stopped shipping, and it is still one variable away.

4. THE ONLY SPELLING THAT IS NOT A GUESS
     open(path, encoding='utf-8')    says what it means, everywhere
     open(path)                      asks the locale, unless UTF-8 Mode
     open(path, encoding=None)       the same guess, spelled worse
     Pass encoding= and nothing on this page can reach your program.

Python read the same C locale and declined to believe it. UTF-8 Mode enables itself when the locale is C or POSIX — PEP 540 ↗, shipped in 3.7 — so sys.flags.utf8_mode is 1 with nothing set, and open(), sys.stdout and the filesystem encoding are all UTF-8 while locale.getencoding() still honestly reports ASCII. The reasoning is that a machine claiming to be ASCII-only in this decade is describing its own missing configuration, not its data.

That gap between the two locale.* calls is the thing to remember. locale.getencoding() answers what the locale says; locale.getpreferredencoding() answers what Python will actually do. They disagree here, and reaching for the first one to decide how to open a file is a bug that only appears on a machine whose locale is unset — which is to say, only in production.

Section 3 shows it is still one variable away. PYTHONUTF8=0 restores the pre-3.7 behaviour, open() goes back to asking the C locale, and the second byte of é raises UnicodeDecodeError. People do set that variable, usually to fix something else. The defence is not the mode; it is passing encoding= and never finding out.

The one tool that cannot be recorded here

The stub this page replaces promised tr alongside wc. It is not in the run above, because the two platforms disagree about whether tr obeys the locale at all — so no single answer key could match both. Measured 2026-09-06, same file, same command:

Measured 2026-09-06 — abridged; printf 'caf\303\251\n' | tr '[:lower:]' '[:upper:]' | od -An -tx1

                              macOS 26.6 (BSD tr)   ubuntu:24.04 (GNU tr, glibc 2.39)
  under LC_ALL=C              43 41 46 c3 a9 0a     43 41 46 c3 a9 0a
  under a UTF-8 locale        43 41 46 c3 89 0a     43 41 46 c3 a9 0a
                                       ^^ É                  ^^ unchanged

BSD tr uppercased é to É; GNU tr left it alone. GNU tr is byte-oriented and has never supported multi-byte characters, so its behaviour is locale-independent — which means the portable statement is not "tr follows LC_CTYPE" but "tr follows LC_CTYPE on one of the two machines you are likely to run it on." If you need case conversion on non-ASCII text in a shell pipeline, this is the moment to stop and use a language that ships a Unicode table.

Why every other example in this library pins LC_ALL=C

tools/run_examples.py sets LC_ALL=C, LANG=C and PYTHONUTF8=1 for every example it runs — and this page is the reason. An answer key records what a program printed; if the program consults the environment, the key records the environment too, and then it only matches the machine that wrote it. Pinning makes the key a property of the code.

The cost is that a lesson about the locale cannot inherit the pin, so both examples above set their own, in view, on the line above each measurement. The Python one goes further and takes every reading in a child interpreter with a constructed environment, so its output is identical whether you run it through the runner or type it at a prompt — which is the discipline any example that reads os.environ has to keep.

In Rust

There is nothing to run, and the absence is the point. String and &str are UTF-8 by definition, chars() counts scalar values, and to_uppercase() uses the Unicode tables compiled into the binary. No environment variable changes any of it, on any machine. LC_ALL is, to a Rust program, an ordinary string you can read with std::env::var and nothing more.

That is a genuine trade rather than a free win. The behaviour Rust gives up is the one a locale exists to provide: "i".to_uppercase() is "I" even in Turkish, where the correct answer is "İ". Rust's position is that a silent, environment-dependent answer is worse than a consistent one you can override deliberately — which is the same argument PEP 540 makes, reached from the other direction.

If you are coming from Python or ABAP

ABAP has a locale too, but it is not an environment variable and it does not belong to a process. It is the text environment ↗: every program in an internal session shares one, the logon language sets it when the session opens, and SET LOCALE LANGUAGE ↗ changes it. What it cannot change is the code page that text is held in, which on a Unicode system is UTF-16 in every text environment, so nothing per session can make two programs disagree about what a character in memory is. What corresponds to the bug on this page is the boundary. OPEN DATASET ... IN TEXT MODE will not compile without ENCODING ↗, and ENCODING DEFAULT has meant UTF-8 on every system since 7.50, so neither of those is where the bug lives. ENCODING NON-UNICODE is, and so is IN LEGACY TEXT MODE or IN LEGACY BINARY MODE with no CODE PAGE ↗: both convert through the non-Unicode code page that table TCP0C assigns to the current text environment, so two sessions whose users logged on in different languages can read one legacy file as two different strings. That code page is the LC_CTYPE of the ABAP world — invisible, inherited, and correct until the file comes from somewhere else. Name the encoding explicitly there for the same reason you pass encoding= in Python — ENCODING UTF-8, or a CODE PAGE on a legacy open — and verify any code-page number on the system rather than trusting one from documentation. (Not machine-checked — CI cannot run ABAP.)

Try it

1. Run locale in your own terminal. If it prints LC_CTYPE="C" you have just found out that your shell tools are counting bytes; if it prints a UTF-8 locale, find where that came from — Terminal's checkbox, your shell profile, or an ssh session that forwarded it.
2. ssh to any machine and run locale there. LC_* variables are forwarded by default on many configurations, so a locale that does not exist on the remote end arrives anyway and every tool falls back silently. This is the single most common way the setting arrives without anyone choosing it.
3. Take the wc -m line from section 2 and find the equivalent for awk, grep -o . and expr length. Which of them agree with wc -m, and which are counting something else entirely?
4. Set PYTHONUTF8=0 in a shell, run a script that opens a UTF-8 file without encoding=, and read the traceback. Then add encoding="utf-8" and watch the variable stop mattering.

Practice

Six variables, and the one that matters here. Name all six LC_* categories and say what each decides. Then say which one changes whether grep's . matches one byte or one character, and what LC_ALL=C does to that.

Then the part people get wrong: does setting LC_CTYPE change how many characters Python thinks a string has? Answer for Python 3.7 and later, say what the locale can still affect, and explain why a shell pipeline and the Python script inside it can legitimately disagree about a line's length.

Answers

Verified output of locale_and_lc_ctype_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

THE SIX CATEGORIES, AND WHAT EACH ONE DECIDES
   LC_CTYPE      what a CHARACTER is: case, class, the multibyte encoding
   LC_COLLATE    what ORDER strings sort in
   LC_NUMERIC    the decimal separator -- comma or point
   LC_TIME       date and time formatting, and the first day of the week
   LC_MONETARY   the currency symbol and where it goes
   LC_MESSAGES   the language of a program's own output

   They are INDEPENDENT. LANG sets a default for all six, LC_ALL
   overrides all six, and any single one can be set on its own -- so a
   session sorting in Polish while formatting numbers in English is a
   normal configuration, not a broken one.
   Python's list, straight from the module: ['LC_ALL', 'LC_COLLATE', 'LC_CTYPE', 'LC_MESSAGES', 'LC_MONETARY', 'LC_NUMERIC', 'LC_TIME']

WHY LC_CTYPE IS THE ONE THIS LIBRARY CARES ABOUT
   It decides what a tool means by 'a character', which decides:
     * whether grep's . matches one byte or one character
     * whether tr, cut -c and wc -m count bytes or characters
     * whether a tool refuses an invalid sequence or skips the line
   In the C locale a character IS a byte, which is why LC_ALL=C is the
   escape hatch for searching -- and why it is the wrong answer for
   anything that REARRANGES text, such as rev.

AND PYTHON, SINCE 3.7, DOES NOT ASK
   sys.getdefaultencoding()      'utf-8'   always, everywhere
   str is code points and never bytes, so LC_CTYPE cannot change what a
   character means inside the language. What the locale COULD still
   affect is the boundary -- the default encoding for open() and for
   stdio -- and PEP 538/540 made even that predictable: UTF-8 mode, on
   by default in a POSIX locale since 3.7, so a container with no
   locales behaves like a workstation with all of them.

   The practical consequence is worth stating plainly. A pipeline of
   shell tools changes behaviour when LC_ALL changes; the Python script
   in the middle of that pipeline does not. If the two disagree about
   how many characters a line has, they are not both misconfigured --
   one of them is asking the environment and the other never does.

   locale.getpreferredencoding(False) still reports the boundary
   default, and it is the one call whose answer legitimately differs
   per machine -- which is exactly why it is not printed here.

See also

• Inspecting a file — od -a and wc -m, two more tools whose output changes with LC_CTYPE
• printf writes bytes — putting known bytes on a pipe, so the locale is the only variable left
• Opening a file — the encoding= argument, in full