Control characters¶

Level: 101 · for anyone starting from zero

One line: The first 32 codes are commands to a teletype, not letters, and three of them — TAB, LF and CR — decide how every text file is cut into lines and columns; a fourth, NUL, is where a C string ends, which is why some of your text disappears the moment it crosses into C.

Commands to a machine that no longer exists¶

ASCII was designed for a teletype: a typewriter driven by a wire. Most of its 128 codes print something. The first 32 do something instead — ring the bell, back up a column, feed a line, return the carriage to the left margin, eject the page. Code 127, DEL, punches every hole in a paper tape, which is how you deleted a character you had already punched.

 0  0x00  NUL  ^@   nothing — and the end of a C string
 7  0x07  BEL  ^G   ring the bell (a terminal still beeps)
 8  0x08  BS   ^H   move back one column
 9  0x09  TAB  ^I   jump to the next tab stop
10  0x0A  LF   ^J   line feed: down one line            — the Unix line end
13  0x0D  CR   ^M   carriage return: back to column 0    — half of the Windows line end
27  0x1B  ESC  ^[   escape: the next bytes are a command to the terminal
127 0x7F  DEL  ^?   delete

The ^M column is caret notation, and it is arithmetic, not a naming convention: Ctrl + a letter clears bit 6 of the letter, the same way Shift clears bit 5 (the case bit). M is 0x4D; 0x4D & 0x1F is 13; 13 is CR. That is why vim shows a Windows file with ^M at the end of every line, why od -c prints \r, and why pressing Ctrl-M in a terminal acts like Return.

The three that matter every week¶

TAB is a jump to the next tab stop, not "some spaces"; how far it jumps is the reader's decision, which is why a file indented with tabs looks different in every editor.

LF and CR were two separate motions on a teletype — down a line, then back to the left — so a new line was two characters, CR LF. Unix decided one was enough and kept LF. DOS kept both. Classic Mac OS kept CR alone. Every text file you meet uses one of those three, and the bytes tell you which:

Unix          6f 6e 65 0a                     one⏎
Windows       6f 6e 65 0d 0a                  one⏎ with the CR first
Classic Mac   6f 6e 65 0d                     rare now; still produced by some spreadsheets

A file with the wrong one is not corrupt. It is a file whose 0d bytes the reader did not expect, and they show up as a phantom column in a CSV, a ^M in vim, an \r on the end of every string you read, and git's LF will be replaced by CRLF warning. CRLF vs LF is the whole repair kit; this page is only what the bytes are.

In Python¶

Verified output of control_characters_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. THE ONES YOU WILL MEET: code, hex, Python escape, caret, meaning
     0  0x00  NUL  '\x00'  ^@  nothing — and the end of a C string
     7  0x07  BEL  '\x07'  ^G  ring the bell (a terminal still beeps)
     8  0x08  BS   '\x08'  ^H  move back one column
     9  0x09  TAB  '\t'    ^I  jump to the next tab stop
    10  0x0A  LF   '\n'    ^J  line feed: move down one line  — the Unix line end
    11  0x0B  VT   '\x0b'  ^K  vertical tab (nobody uses it)
    12  0x0C  FF   '\x0c'  ^L  form feed: eject the page (printers still do)
    13  0x0D  CR   '\r'    ^M  carriage return: back to column 0  — half of the Windows line end
    27  0x1B  ESC  '\x1b'  ^[  escape: the next bytes are a command to the terminal
   127  0x7F  DEL  '\x7f'  ^?  delete (punch every hole in the paper tape)

2. THE CARET IS ARITHMETIC: Ctrl+letter clears bit 6, the way Shift clears bit 5
   Ctrl-I  =  0x49 & 0x1F  =   9  =  TAB
   Ctrl-J  =  0x4a & 0x1F  =  10  =  LF
   Ctrl-M  =  0x4d & 0x1F  =  13  =  CR
   Ctrl-[  =  0x5b & 0x1F  =  27  =  ESC
   So ^M is CR, ^I is TAB, ^J is LF, ^[ is ESC — the caret names vim and od use.

3. THREE LINE ENDINGS, ONE FILE EACH
   Unix  LF         6f 6e 65 0a 74 77 6f 0a           8 bytes  split('\n') -> ['one', 'two', '']
   Windows  CR LF   6f 6e 65 0d 0a 74 77 6f 0d 0a    10 bytes  split('\n') -> ['one\r', 'two\r', '']
   Classic Mac  CR  6f 6e 65 0d 74 77 6f 0d           8 bytes  split('\n') -> ['one\rtwo\r']
   Only LF counts as a line to split('\n'); the CR stays glued to the word before it.

4. splitlines() KNOWS ALL OF THEM, AND MORE THAN YOU WANT
   'a\nb\r\nc\rd\x0be\x0cf\x1cg\x85h\u2028i'
   split('\n')  -> 3 pieces
   splitlines() -> 9 pieces: ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i']
   VT, FF, FS, NEL and LINE SEPARATOR all count. A CSV with a stray \x85 in a cell splits there.

5. NUL: PYTHON KEEPS IT, C STOPS AT IT
   len('ab\x00cd') = 5   (Python: five characters, NUL is just a character)
   libc strlen(b'ab\x00cd') = 2   (C: the string ended at the NUL)
   Every C API you hand a Python string to sees the first half only. See the C example.

6. ESC IS STILL A LIVE PROTOCOL
   '\x1b[31mred\x1b[0m'
   bytes: 1b 5b 33 31 6d 72 65 64 1b 5b 30 6d
   ESC [ 31 m  =  'switch to red';  ESC [ 0 m  =  'reset'.  A terminal draws 3 letters; a file holds 12 bytes.

Section 4 is the trap worth remembering. str.split('\n') splits on one character. str.splitlines() splits on every line boundary Unicode defines — LF, CR, CR LF, VT, FF, the C1 control NEL (\x85), the three ASCII separators \x1c–\x1e, and the Unicode line and paragraph separators. A CSV that arrives with a stray \x85 inside a cell, which Windows-1252 text produces easily, splits into an extra row under splitlines() and not under split('\n'). Neither is wrong; they answer different questions, and the csv module asks a third one.

Section 5 uses ctypes to call the real strlen from Python: the same five characters, and C reports two. That is the entire content of the C example below.

In the terminal¶

Verified output of control_characters_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

1. od -c NAMES THE COMMON ONES; xxd JUST SHOWS A DOT

$ printf 'a\tb\nc\r\n' | od -An -tx1 -c | tidy
  61  09  62  0a  63  0d  0a
   a  \t   b  \n   c  \r  \n

$ printf 'a\tb\nc\r\n' | xxd
00000000: 6109 620a 630d 0a                        a.b.c..

2. THE SAME TWO LINES, THREE WAYS TO END THEM

$ printf 'one\ntwo\n'     | xxd
00000000: 6f6e 650a 7477 6f0a                      one.two.

$ printf 'one\r\ntwo\r\n' | xxd
00000000: 6f6e 650d 0a74 776f 0d0a                 one..two..

$ printf 'one\rtwo\r'     | xxd
00000000: 6f6e 650d 7477 6f0d                      one.two.

3. wc -l COUNTS LF BYTES, NOTHING ELSE

$ printf 'one\ntwo\n'     | wc -l | tr -d ' '
2

$ printf 'one\r\ntwo\r\n' | wc -l | tr -d ' '
2

$ printf 'one\rtwo\r'     | wc -l | tr -d ' '
0

4. cat -v SHOWS THE CR AS ^M — THE THING YOU SEE IN vim

$ printf 'one\r\ntwo\r\n' | cat -v
one^M
two^M

5. STRIP THEM: tr is always installed

$ printf 'one\r\ntwo\r\n' | tr -d '\r' | xxd
00000000: 6f6e 650a 7477 6f0a                      one.two.

6. THE CARET ARITHMETIC, IN BASH

$ echo $(( 0x49 & 0x1F ))   # Ctrl-I
9

$ echo $(( 0x4D & 0x1F ))   # Ctrl-M
13

7. NUL CANNOT LIVE IN A SHELL VARIABLE, BUT IT CAN LIVE IN A PIPE

$ printf 'ab\0cd' | xxd
00000000: 6162 0063 64                             ab.cd

$ printf 'ab\0cd' | wc -c | tr -d ' '
5

od -c is the tool for this page, because it names the common control characters (\t, \n, \r, \0) where xxd prints a dot. wc -l counts LF bytes and nothing else — a Classic-Mac file has zero lines to it. cat -v shows CR as ^M on both macOS and Linux (the tab-showing flag differs between them, -t and -T, so it is not used here). And a NUL cannot be stored in a shell variable — bash strings are C strings — but it passes through a pipe untouched, which section 7 proves with wc -c.

In Rust¶

Verified output of control_characters_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

1. THE ESCAPES A LITERAL ACCEPTS
   "tab\there\nnew line\r\ncrlf\0nul\u{1b}ESC"
   31 bytes, 31 chars — the escapes are single bytes once compiled
   (Rust has no \a \b \f \v: write \x07 \x08 \x0C \x0B)

2. TWO QUESTIONS, TWO METHODS
   '\t'       U+0009  is_ascii_control=true  is_control=true
   '\n'       U+000A  is_ascii_control=true  is_control=true
   '\r'       U+000D  is_ascii_control=true  is_control=true
   '\0'       U+0000  is_ascii_control=true  is_control=true
   '\u{1b}'   U+001B  is_ascii_control=true  is_control=true
   '\u{7f}'   U+007F  is_ascii_control=true  is_control=true
   'A'        U+0041  is_ascii_control=false is_control=false
   '\u{85}'   U+0085  is_ascii_control=false is_control=true
   '\u{2028}' U+2028  is_ascii_control=false is_control=false
   is_ascii_control is the 0..=31 + 127 table; is_control is Unicode's Cc category, which adds 0x80..=0x9F.

3. lines() SPLITS ON \n AND STRIPS A TRAILING \r — AND ON NOTHING ELSE
   "one\r\ntwo\nthree\rfour"
   lines() -> ["one", "two", "three\rfour"]
   (the lone \r in 'three\rfour' is not a line end to Rust)

4. THE CARET ARITHMETIC
   Ctrl-I  = 0x49 & 0x1F =  9 = '\t'
   Ctrl-J  = 0x4a & 0x1F = 10 = '\n'
   Ctrl-M  = 0x4d & 0x1F = 13 = '\r'
   Ctrl-[  = 0x5b & 0x1F = 27 = '\u{1b}'

5. NUL IS A CHARACTER TO RUST, TOO — UNTIL IT MEETS C
   "ab\0cd".len() = 5
   std::ffi::CString::new("ab\0cd") -> Err(NulError(2, [97, 98, 0, 99, 100]))
   Rust refuses to make a C string with an interior NUL, because C would silently truncate it.

Two things Rust decides for you. str::lines() splits on \n and strips one trailing \r, so a Windows file reads cleanly and a Classic-Mac file reads as one line. And CString::new refuses a string with an interior NUL rather than truncating it, because the C side would: the same fact as Python's section 5, turned into an error at the boundary instead of a surprise past it.

The C view¶

C is where a string is the bytes, so it is the place to see what NUL actually does.

control_characters_c.c in full — pasted here by tools/run_examples.py from the file CI runs.

/* The C view: a string is bytes up to the first NUL, and nothing else.
 *
 * Build & run:  cc -std=c11 -Wall -Wextra control_characters_c.c -o nul && ./nul
 */
#include <stdio.h>
#include <string.h>

int main(void) {
    char s[] = "ab\0cd";           /* six bytes: a b NUL c d NUL */

    printf("1. THE ARRAY AND THE STRING ARE DIFFERENT LENGTHS\n");
    printf("   sizeof(s) = %zu   (the compiler stored six bytes, terminator included)\n", sizeof s);
    printf("   strlen(s) = %zu   (strlen counted until the first NUL)\n", strlen(s));
    printf("   printf(\"%%s\") -> \"%s\"   (so does printf)\n", s);
    printf("\n");

    printf("2. THE BYTES ARE ALL STILL THERE\n   ");
    for (size_t i = 0; i < sizeof s; i++) {
        printf("%s%02x", i ? " " : "", (unsigned char)s[i]);
    }
    printf("\n   Only the C string functions stop early. The memory does not.\n\n");

    printf("3. A 'char' IS A BYTE, NOT A CHARACTER\n");
    const char *cafe = "caf\xc3\xa9";
    printf("   \"caf\\xc3\\xa9\" has strlen %zu: C counts bytes and has never heard of \xc3\xa9\n", strlen(cafe));
    printf("\n");

    printf("4. THE CONTROL CHARACTERS C CAN SPELL\n");
    const char *names[] = {"\\a", "\\b", "\\t", "\\n", "\\v", "\\f", "\\r", "\\e (not standard)", "\\0"};
    const char values[] = {'\a', '\b', '\t', '\n', '\v', '\f', '\r', 27, '\0'};
    for (size_t i = 0; i < sizeof values; i++) {
        printf("   %-18s = %2d = 0x%02X\n", names[i], values[i], values[i]);
    }
    return 0;
}

Verified output of control_characters_c.c — regenerated by tools/run_examples.py, never hand-typed.

1. THE ARRAY AND THE STRING ARE DIFFERENT LENGTHS
   sizeof(s) = 6   (the compiler stored six bytes, terminator included)
   strlen(s) = 2   (strlen counted until the first NUL)
   printf("%s") -> "ab"   (so does printf)

2. THE BYTES ARE ALL STILL THERE
   61 62 00 63 64 00
   Only the C string functions stop early. The memory does not.

3. A 'char' IS A BYTE, NOT A CHARACTER
   "caf\xc3\xa9" has strlen 5: C counts bytes and has never heard of é

4. THE CONTROL CHARACTERS C CAN SPELL
   \a                 =  7 = 0x07
   \b                 =  8 = 0x08
   \t                 =  9 = 0x09
   \n                 = 10 = 0x0A
   \v                 = 11 = 0x0B
   \f                 = 12 = 0x0C
   \r                 = 13 = 0x0D
   \e (not standard)  = 27 = 0x1B
   \0                 =  0 = 0x00

sizeof counts the bytes the compiler stored; strlen counts until the first zero byte; every string function in the C library does what strlen does. The memory after the NUL is still there — section 2 prints it — but nothing that takes a char * will ever look. That is what "C API truncates" means: not that data was destroyed, but that the receiver stopped reading. Section 3 is the other half of the C view, and the reason this library exists: char is one byte, and strlen("café") is 5.

If you are coming from Python or ABAP¶

Python. The escapes you know — \t \n \r \0 \a \b \f \v \x1b — are all here, and repr() is how you see them. Two things to take away. splitlines() is not split('\n') (section 4). And a str can hold anything, including NUL, which means Python is happy right up until it hands the string to something that is not Python: ctypes, an os. call taking a path, a database driver. ValueError: embedded null byte from open() is Python doing what Rust's CString::new does, at the last possible moment.

ABAP. Control characters cannot be typed into a literal, so they are constants: cl_abap_char_utilities=>newline (LF), =>cr_lf, =>horizontal_tab, =>form_feed, =>vertical_tab, =>backspace. The line-ending question is decided at the file, not in the string: OPEN DATASET … IN TEXT MODE takes WITH UNIX LINEFEED, WITH WINDOWS LINEFEED, WITH NATIVE LINEFEED or WITH SMART LINEFEED, and READ DATASET strips whichever it was told to expect — so a file written with the wrong addition is the ABAP form of the phantom \r column. A string holds NUL without complaint, like Python's; whether the other end of an RFC or a download does is the other end's business. (Not machine-checked — CI cannot run ABAP.)

Try it¶

cd 02_Characters/control_characters/examples
python3 control_characters_py.py
bash control_characters_sh.sh
rustc --edition 2024 control_characters_rs.rs -o /tmp/ctrl && /tmp/ctrl
cc -std=c11 -Wall -Wextra control_characters_c.c -o /tmp/nul && /tmp/nul

Then on a real file: xxd some.csv | grep -m3 '0d0a' tells you whether it came from Windows. printf 'x\r\n' | cat -v shows you the ^M. And without the machine: what is Ctrl-[? What is Ctrl-@? Check both with echo $(( 0x5B & 0x1F )) and section 2.

Practice¶

How many lines is this string?

S = "a\tb\r\nc\rd\ne\x0bf"

Predict len(S), len(S.split('\n')) and len(S.splitlines()) before running any of them. Two of those three disagree; say how many things splitlines() cuts on and why that is a parser differential inside one language.

Then "a\x00b": give its Python len, its UTF-8 bytes, and what C's strlen would say.

Answers

Verified output of control_characters_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

the string   'a\tb\r\nc\rd\ne\x0bf'
len          12   characters, every control included

S.split('\n')       -> ['a\tb\r', 'c\rd', 'e\x0bf']
S.splitlines()      -> ['a\tb', 'c', 'd', 'e', 'f']
len(splitlines())   -> 5

split('\n') cuts on ONE byte and gives 3 pieces. splitlines() cuts on
36 + 1 different things -- CR, LF, CRLF as one, plus VT, FF, the
three file separators, NEL, and Unicode's own LINE and PARAGRAPH
SEPARATOR -- and gives more. Same string, two correct answers, and the
difference is a parser differential inside one language.

THE FOUR CONTROLS THE PAGE IS ABOUT
   \t     U+0009  moves to the next tab stop -- a layout instruction, not spaces
   \n     U+000A  ends a line on Unix; one byte, 0x0a
   \r     U+000D  returns the carriage; on a terminal the NEXT text overwrites this line
   \x00   U+0000  ends a C string -- see below

AND THE NUL
   'a\x00b'   Python len = 3
   utf-8 bytes = 610062   -- three bytes, perfectly valid
   C strlen would say 1. The byte is not removed and not rejected; the
   next layer simply stops reading there, which is why text that crosses
   into C comes back shorter with nothing raised anywhere.