The shell has no string type¶
Level: 201 · for anyone writing a shell script over text
One line: A shell variable holds bytes and nothing else — no encoding, no validation, and no second type to convert to — so ${#var} is a question about LC_CTYPE rather than about the value, and the sh on a Mac and the sh on Ubuntu answer it differently for the same script.
One type, and what that costs¶
Every other language in this library's cast keeps two: Python has bytes and str, Rust has Vec<u8> and String, and between them sits a decode that can fail and tell you so. The shell has one type. A variable is a byte string, an argument is a byte string, a filename is a byte string, and a line off a pipe is a byte string — the same thing four times, with no conversion anywhere and therefore nothing that can refuse.
That sounds like freedom and is the source of every problem on this page. A decode is where the encoding question gets asked once. With no decode there is no such place, so the question does not go away — it moves into every operation individually, and each one answers it on its own by asking the locale. ${#var} asks. ${var:0:1} asks. case's ? asks. sed and grep and cut each ask separately, which is what the rest of this chapter is about. Nothing coordinates the answers, and nothing records which one was given.
Two bytes are then reserved out from under you, and not by the shell — by the machinery either side of it. NUL cannot be in an argument at all, because execve(2) ↗ takes NUL-terminated strings — and it does not survive a POSIX shell's variable either, which is the whole reason find -print0 | xargs -0 exists. (One shell does hold it in a variable and still cannot pass it on; that grid is below.) Newline is the default field separator of every text channel a shell has, and it is a perfectly legal byte in a filename. So the two bytes you cannot put in a shell string are the byte a filename may not contain and the byte it very much may.
In the terminal¶
The script sets LC_ALL itself, section by section, because the locale is the subject here — it cannot inherit the runner's LC_ALL=C like every other example in this library. It looks the UTF-8 locale up rather than naming one, because macOS and ubuntu:24.04 share no UTF-8 locale name that is guaranteed present on both, and it never prints the name it found. USER is pinned to ada so section 1 is the same on every machine; there is no clock, no $HOME and no hostname anywhere in the file.
Verified output of sh_holds_bytes_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
1. A VARIABLE HOLDS BYTES, AND NOTHING CHECKS THEM
good=636166c3a9 bad=61fffe62
$USER=ada (pinned) its bytes=616461
The second assignment did not fail, warn, or replace anything: no
UTF-8 sequence begins with ff or fe, and the shell stored them
anyway. There is no other value here -- no str, no String, no
bytes-versus-text pair. A shell variable is a byte string, and an
environment variable is the same byte string one execve further on.
2. SO WHAT DOES ${#var} COUNT?
LC_ALL=C ${#café}=5 ${#ż}=2
a UTF-8 locale ${#café}=4 ${#ż}=1
the bytes, either way: café=5 ż=2
${#var} is not a length. It is a question about the locale, and it
has two answers for one unchanged variable. The byte count is the
one number that does not move, and the portable way to ask for it is
the third line: printf '%s' "$v" | wc -c.
3. AND EVERY OTHER EXPANSION ASKS THE SAME QUESTION
LC_ALL=C ${café:3:1}=c3 ${ż#?}=bc case ż in ?) no match
a UTF-8 locale ${café:3:1}=c3a9 ${ż#?}= case ż in ?) matches
In the C locale the substring took HALF of the é and the ? stripped
half of the ż, leaving an orphan bc that is not text any more. This
is tr's failure mode with no tool involved: the cut happened inside
the shell, in an expansion that looks like string slicing.
4. THE FIRST BYTE THE SHELL CANNOT CARRY: NUL
on disk 6100620a
through $(...) 6162 len=2
through $'a\x00b' 61 len=1
out through an execve 6162
Three different amputations of one three-byte string, none of them an
error. $(...) drops the NUL and keeps going; $'...' stops dead at it;
and execve could not have carried it in the first place, which is the
kernel's rule rather than the shell's. That is why -print0 exists.
5. THE SECOND: NEWLINE, WHICH IS THE DEFAULT FIELD SEPARATOR
two files on disk: 2e2f706c61696e2e7478740a2e2f74776f0a6c696e65732e7478740a
for f in $(ls) 3 words
for f in * 2 items
find -print | xargs 3 runs
find -print0 | xargs -0 2 runs
The loop is not the bug. $(ls) and a bare xargs both went through a
channel whose record separator is a newline, and one of these files
contains a newline -- so two names arrived as three. The glob never
serialised at all, and -print0 serialised with the one separator a
filename cannot contain, so both of those counted two.
6. IFS IS A SET OF BYTES, NOT A DELIMITER
LC_ALL=C IFS=é on "aéb" -> 3 fields: [61] [] [62]
a UTF-8 locale IFS=é on "aéb" -> 2 fields: [61] [62]
Three fields in the C locale, and the middle one is empty: both bytes
of the é were separators, so one character became two delimiters with
nothing between them. IFS is a SET, exactly like tr's argument, and
the locale is the only thing deciding whether it is a set of bytes or
a set of characters. Use a single-byte separator and the question
never comes up.
7. WHICH IS WHY ONE UNQUOTED EXPANSION IS THE WHOLE CLASS
var='*.txt' set -- $var -> 2 | set -- "$var" -> 1
var='one two' set -- $var -> 2 | set -- "$var" -> 1
An unquoted expansion is not a value being read. It is a value being
re-parsed: split on IFS, then matched against the filesystem. Quoting
it turns both off, which is the whole of the advice -- and the reason
the advice is not 'quote when the value might have spaces'.
8. echo IS NOT PORTABLE. printf IS.
printf "caf\303\251" 636166c3a9
echo "caf\303\251" 6361665c3330335c3235310a
$'caf\xc3\xa9' 636166c3a9
Line one is six bytes on every shell this library has measured. Line
two is bash's echo leaving the backslashes alone -- and dash's echo,
which is /bin/sh on Ubuntu, expands them instead, so the same command
writes a different file. Line three is bash's own $'...' quoting,
which dash does not have at all: it prints a dollar sign and the
backslashes, and says nothing. printf with octal escapes is the one
spelling that means the same thing everywhere.
Which sh?¶
sh is not a program. It is a name that several unrelated programs answer to, and the two this library's CI runs on are two different ones:
${#v} in LC_ALL=C ${#v} in a UTF-8 locale
macOS /bin/sh = bash 3.2 2 1
Ubuntu /bin/sh = dash 2 2
either bash 2 1
either dash 2 2
macOS zsh 5.9 2 1
Read the first two rows together. The same #!/bin/sh script, over the same bytes, in the same UTF-8 locale, gets 1 on a Mac and 2 on Ubuntu — not because the platforms differ but because sh resolves to bash on one and dash on the other, and bash grew multi-byte support while dash never did. Nothing in the script, the locale or the file changed. A Mac is the only one of the two where sh counts characters, which is the opposite of the direction most people guess.
So ${#var} is decided by two things a script does not control — the locale it inherits, and which program sh turned out to be — and the two answers are not marked as different in any way a reader of the script could see. The one that does not move is the byte count, and there is exactly one portable way to ask for it:
If you want characters, the shell is the wrong tool and the honest move is to leave it — python3 -c or awk with a pinned locale, and awk is three programs is the page about how much that helps.
echo is not portable and printf is¶
printf writes bytes is that argument's own page and makes it in full — including the echo that cannot print the string -n. What belongs here is the same question asked of the shells rather than of the builtins, because sh being two programs turns one of its findings inside out:
echo "caf\303\251" echo -n hi echo '-n'
bash 3.2, bash 5.2, zsh 5.9 caf\303\251 (12 bytes) hi (nothing at all)
dash = Ubuntu's /bin/sh café ( 6 bytes) hi (nothing at all)
bash-as-sh = a Mac's /bin/sh caf\303\251 (12 bytes) -n hi -n
POSIX leaves the first column open on purpose: if any argument contains a backslash the behaviour is unspecified, and the XSI description expands the escapes. So row two is dash taking the XSI reading and row one is bash keeping its own — neither is a bug, and the same command writes a different file.
Row three is the one nobody expects, and it is the mirror of the other page's headline. Bash invoked as sh enters POSIX mode and stops recognising options, so echo -n hi writes -n hi and a newline — three characters added at the front, and the byte you were trying to remove still on the end. In the same breath that makes a Mac's /bin/sh the only shell measured here that can print the string -n at all. One shell, two opposite bugs, decided by which name you invoked it under.
printf 'caf\303\251' put the same six bytes on the pipe in all seven shell-and-platform combinations measured here. So does printf '\303\251' for the bare character. The bash-only spelling is worth knowing precisely because it looks portable:
Under dash, $'caf\xc3\xa9' is not a quoting form at all. The $ is an ordinary character, so you get a dollar sign followed by the literal backslashes — twelve bytes instead of six, no error, no status. It is the failure shape this chapter keeps meeting: the wrong answer that costs nothing to produce.
NUL, one layer at a time¶
The NUL byte is that byte's own page and has the full treatment; what belongs here is that the shell loses it in three different places for three different reasons, and one shell shows you which is which:
bash 3.2 bash 5.2 dash zsh 5.9
v=$(cat file-with-a-NUL) 6162 6162 (*) 6162 610062
IFS= read -r v < that file 61 6162 6162 610062
"$v" out to an external 6162 6162 6162 61
(*) bash 5 warns on stderr; bash 3.2 says nothing at all.
Three things fall out of that grid. Command substitution drops the NUL and keeps going in every POSIX shell, and whether you are told is a bash version question rather than a platform one — that table is on the printf page, and its consequence is that no script can portably detect the loss. read cannot agree with itself: bash 3.2 truncates at the NUL and bash 5.2 drops it, which are different wrong answers to the same call. And zsh is the counter-example that proves the rule is not about shells at all: zsh's variables are counted rather than NUL-terminated, so it holds all three bytes through both the assignment and the read — and still hands 61 to the next program, because that boundary is execve's and no shell can argue with it.
That last row is the reason -print0 and -0 are a pipe convention and not a variable one. A pipeline that must carry arbitrary filenames has to keep them on the pipe from end to end; the moment a name enters a variable or an argv, the two reserved bytes are gone.
The one unquoted expansion¶
Everything above is about what a value is. This is about what happens when you mention it, and it is the single largest class of shell bug there is:
cp $src $dst # two expansions, four re-parses, no strings involved
cp "$src" "$dst" # the fix, and it is the whole fix
An unquoted expansion is not a value being read. It is a value being re-parsed: split into fields on IFS, then each field matched against the filesystem as a glob. Both happen after the variable is expanded and neither is visible in the source. The generated block above measures both — a variable holding *.txt becomes two arguments in a directory with two .txt files, and a variable holding one two becomes two arguments with no filesystem involved at all.
And IFS is a set of bytes, not a delimiter — the same distinction that separates sed from tr, living inside the shell itself. Section 6 of the block sets IFS to the two bytes of é and splits aéb: in the C locale that is two separators, so one character produces an empty field between them, exactly as tr -d 'é' damages the word next door. A UTF-8 locale makes it one separator. Whitespace separators collapse runs and every other separator does not, which is why the empty field appears at all — the same rule that makes paste -d 'é' cut in two places on GNU. Use a single-byte separator and none of it can happen.
If you are coming from Python or ABAP¶
Python: len(s) is unambiguous because s is a str — a sequence of code points that was decoded, once, at a boundary you can point at, and that boundary is where the failure happened if there was one. ${#var} is the same expression with the boundary deleted: no decode ever ran, so the count is computed fresh on every call from bytes that were never checked, against whatever LC_CTYPE says at that moment. The nearest Python to a shell variable is bytes, and len(b'caf\xc3\xa9') is 5 — which is what ${#var} gives you in the C locale and what printf '%s' "$v" | wc -c gives you always. os.fsencode / os.fsdecode and surrogateescape exist precisely because Python has to represent the byte strings this page is about, including the ones that are not text; the shell needs no such machinery because it never left. What does not transfer is the quoting: subprocess.run([cmd, src, dst]) passes a list and cannot word-split, so the largest bug class here has no Python equivalent unless you reach for shell=True, which recreates it exactly.
ABAP (Not machine-checked — CI cannot run ABAP.) ABAP is the far end of the spectrum from the shell: string is decoded characters with a length in characters, xstring is bytes with a length in bytes, and the two are different types the compiler will not let you confuse — so strlen( ) and xstrlen( ) are two functions where the shell has one expansion with two meanings. Converting between them is cl_abap_codepage, an explicit call naming an encoding, which is the decode boundary the shell does not have. The place the shell's world does reach ABAP is the same place it reaches Python: shelling out. Anything that hands a value to an external command — a SUBMIT with a filename, a job step, a transport script — is handing bytes to a program that will word-split and glob them, and ABAP's type safety stops at that call.
Try it¶
v=$(printf 'caf\303\251'); echo ${#v}in your own terminal, then again withLC_ALL=Cin front of it. If the two agree, printlocale charmapand find out which one you are in.readlink -f /bin/sh— or/bin/sh -c 'echo ${BASH_VERSION:-not bash}'. Whichever it names, now grep your own#!/bin/shscripts for${#,${var:and$'.- Take the worst filename on your disk — a space, a quote, an accent — and run your most-used pipeline over it. Then
touch "$(printf 'two\nlines.txt')"in a scratch directory and run it again. grep -n '\$[A-Za-z_][A-Za-z0-9_]*' somescript.shand read every hit for whether it is quoted. The ones inside[ ]and after=are the ones people skip.- Run
echo -nandecho 'a\tb'underbash, undersh, and underdashif you have it. Count how many of your scripts assume the answer you just got.
Practice¶
How long is this variable, and which two bytes never arrive? Predict, in order, before running anything.
z=$(printf '\305\274')holdsż. What is${#z}underLC_ALL=C, and under a UTF-8 locale? What does${z#?}leave in each — and is what it leaves still a character? What is the one count that is the same in both, and how do you ask for it in a script that must run undersh?bad=$(printf 'a\377\376b').ffandfedecode to nothing in UTF-8. What is${#bad}in a UTF-8 locale — 2, 4, or an error?- A file holds the three bytes
61 00 62. Afterv=$(cat file), what is${#v}, and what would tell a script that something was lost? - A directory holds two files, one of whose names contains a newline. How many words does
for f in $(ls)see, how many items doesfor f in *see, and how many times doesfind . -type f -print | xargs -n1 …run the command? p='*.txt'in a directory holdinga.txtandb.txt. How many arguments doesset -- $pproduce, and how many doesset -- "$p"? Then: withIFS=:, how many fields doesa::bsplit into, and why is that number different from what the defaultIFSdoes toa b?
Answers
Verified output of sh_kata_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
1. ONE VARIABLE, TWO LENGTHS
LC_ALL=C ${#ż}=2 ${ż#?} leaves [bc]
a UTF-8 locale ${#ż}=1 ${ż#?} leaves []
the byte count, either way: 2
Two answers for one unchanged variable, because ${#} is a question
about LC_CTYPE and not about the value. The C-locale ? stripped one
BYTE and left bc, which is half a ż and not a character at all.
printf '%s' "$v" | wc -c is the count that does not move, and the
one every shell on both platforms agreed on.
2. AND THE BYTES THAT ARE NOT A CHARACTER IN ANY LOCALE
LC_ALL=C held=61fffe62 ${#bad}=4
a UTF-8 locale held=61fffe62 ${#bad}=4
Four both times, and no complaint either time. No UTF-8 sequence
begins with ff or fe, so there is nothing there for a UTF-8 locale to
count -- bash falls back to one per undecodable byte and moves on.
Python's str and Rust's String both refuse these bytes at the decode.
The shell has no decode: there is one type here, and it is bytes.
3. THE BYTE THAT NEVER ARRIVES
the file 6100620a the variable 6162 ${#v}=2
Two, not three. The NUL was not stored and no status said so -- the
warning is bash 5's, on stderr, and macOS's bash 3.2 prints nothing
at all, so a script cannot portably detect this. It is not a bash
limitation to route around either: execve takes NUL-terminated
arguments, so the value could not be passed on even if it were held.
4. WHICH IS WHY THE OTHER RESERVED BYTE IS A NEWLINE
for f in $(ls) 3 words
for f in * 2 items
find -print | xargs 3 runs
find -print0 | xargs -0 2 runs
Two files, three words. Newline is the default record separator of
every text channel a shell has, and it is a legal filename byte, so
any name-carrying pipeline needs a separator that is not legal in a
name. There is exactly one, and -print0 / -0 / -z / --null-data are
its four spellings. The glob got two because it never serialised.
5. AND THE SPLIT THAT HAPPENS EVEN WHEN NOTHING IS WRONG
var='*.txt' set -- $var -> 2 set -- "$var" -> 1
default IFS on "a b" -> 2 fields: [a] [b]
IFS=: on "a::b" -> 3 fields: [a] [] [b]
An unquoted expansion is re-parsed twice: split on IFS, then matched
against the filesystem. Quoting turns both off, which is why the rule
is quote everything rather than quote when it might have spaces --
the *.txt case has no space in it and still came apart.
The last two lines are the rule behind the empty field: a run of
WHITESPACE separators collapses into one, and a run of any other
separator does not. So one two-byte character used as IFS becomes two
separators with an empty field between them, which is the shell doing
to a delimiter exactly what tr does to a character.
See also¶
sedmatches patterns, not bytes — sequences against sets, which is the distinctionIFSgets wrongawkis three programs — where to go when you actually need to count characters, and what it costs- The NUL byte — the byte the shell cannot carry, in full
- Locale and
LC_CTYPE— the variable every expansion on this page consults printfwrites bytes — the portable way to type a byte, and theechothis page asks of four shells instead of onexargssplits on the wrong things — the same two reserved bytes, one process further ontrandsortwork a byte at a time — the byte-set failure this page reproduces insideIFS