Skip to content

Three history file formats

Level: 201 · you want to read, grep or clean up a history file outside the shell that wrote it

One line: bash's history file is one command per line, in the bytes you typed, plus a #<seconds> line before each command once HISTTIMEFORMAT is set. zsh's is one command per line too, or : <start>:<elapsed>;command with EXTENDED_HISTORY, and it escapes some of the bytes in UTF-8, so grep jaźń ~/.zsh_history finds nothing. fish's is a list of - cmd: records with a when: line each, in UTF-8 as typed, with newlines and backslashes escaped.

Measured

bash first. The sessions are interactive bashes as in the other lessons, and --noediting turns the line editor off, so bash takes the typed bytes exactly as they arrive. perl prints the file's bytes in hex, one line of hex per line of file. TZ is UTC in every example, and a timestamp bash writes during the run is replaced with <epoch>.

Verified output of histfile_sh.sh, identical on Linux and macOS — regenerated by tools/run_examples.py, never hand-typed.

$ HISTFILE=$PWD/hist bash --norc --noediting -i >/dev/null 2>/dev/null <<'EOF'
echo zażółć gęślą jaźń
EOF

$ cat hist
echo zażółć gęślą jaźń

$ perl -ne 'printf "%*v02x\n", " ", $_' hist
65 63 68 6f 20 7a 61 c5 bc c3 b3 c5 82 c4 87 20 67 c4 99 c5 9b 6c c4 85 20 6a 61 c5 ba c5 84 0a

$ printf '#1700000000\necho one\n#1700000060\necho two\n' > stamped

$ HISTFILE=$PWD/stamped bash --norc -i 2>/dev/null <<'EOF'
history | head -n 2
HISTTIMEFORMAT='%F %T ' history | head -n 2
EOF
    1  echo one
    2  echo two
    1  2023-11-14 22:13:20 echo one
    2  2023-11-14 22:14:20 echo two

$ HISTFILE=$PWD/stamped bash --norc -i >/dev/null 2>/dev/null <<'EOF'
HISTTIMEFORMAT='%F %T '
echo three
EOF

$ sed 's/^#[0-9][0-9]*$/#<epoch>/' stamped
#<epoch>
echo one
#<epoch>
echo two
history | head -n 2
HISTTIMEFORMAT='%F %T ' history | head -n 2
#<epoch>
HISTTIMEFORMAT='%F %T '
#<epoch>
echo three
  • One command, one line, the bytes as typed. The hex is plain UTF-8: ż is c5 bc, ć is c4 87, ń is c5 84. Nothing is escaped, so grep on ~/.bash_history finds what you typed.
  • # followed by digits is a timestamp, not a command. The hand-written file has four lines, and history lists two entries. bash read the #1700000000 lines as times even with HISTTIMEFORMAT unset. With it set, history prints them: 2023-11-14 22:13:20, in UTC here and in your own time zone in your terminal.
  • Timestamps are written only for commands run while HISTTIMEFORMAT is set. The reading session never set it, so its two commands were appended bare. The next session set it, so both of its commands, the assignment included, got a #<epoch> line. One file can hold both kinds of line.

On a Mac

Nothing changes. All three keys on this page are shared: /bin/bash 3.2, zsh 5.9 and fish 4.3.2 on macOS wrote the same bytes as on Ubuntu. That includes zsh's escaping below, and the hex itself: the perl line prints bytes identically on both machines.

In zsh and fish

zsh writes one command per line, but not always the bytes you typed. The sessions are zsh -d -i with a scratch ZDOTDIR, so the history file is read and written:

Verified output of histfile_zsh.zsh, identical on Linux and macOS — regenerated by tools/run_examples.py, never hand-typed.

$ zsh -d -i >/dev/null 2>/dev/null <<'EOF'
echo zażółć gęślą jaźń
EOF

$ printf '%s\n' 'echo zażółć gęślą jaźń' | perl -ne 'printf "%*v02x\n", " ", $_'
65 63 68 6f 20 7a 61 c5 bc c3 b3 c5 82 c4 87 20 67 c4 99 c5 9b 6c c4 85 20 6a 61 c5 ba c5 84 0a

$ perl -ne 'printf "%*v02x\n", " ", $_' hist
65 63 68 6f 20 7a 61 c5 bc c3 b3 c5 82 c4 83 a7 20 67 c4 83 b9 c5 83 bb 6c c4 83 a5 20 6a 61 c5 ba c5 83 a4 0a

$ grep -c 'zażó' hist; grep -c 'jaźń' hist
1
0

$ zsh -d -i 2>/dev/null <<'EOF'
LC_ALL=C.UTF-8 fc -ln 1 1
EOF
echo zażółć gęślą jaźń

$ zsh -d -i -o extendedhistory >/dev/null 2>/dev/null <<'EOF'
echo one
EOF

$ sed 's/^: [0-9]*:/: <epoch>:/' hist
: <epoch>:0;echo one

$ printf ': 1700000000:5;sleep 5\n: 1700000060:0;echo two\necho plain\n' > hist

$ zsh -d -i 2>/dev/null <<'EOF'
fc -l -t '%F %T' -D 1 2
fc -ln 1 3
EOF
    1  2023-11-14 22:13:20  0:05  sleep 5
    2  2023-11-14 22:14:20  0:00  echo two
sleep 5
echo two
echo plain
  • The file is not the text's UTF-8. Compare the two hex lines. Where the text has c4 87 (ć), the file has c4 83 a7. zsh wrote the byte 0x87 as 0x83 followed by 0x87 XOR 0x20, which is 0xa7. The same happened to 0x99 in ę, 0x9b in ś, 0x85 in ą and 0x84 in ń. The second bytes of ż (0xbc), ó (0xb3), ł (0x82) and ź (0xba) went through unchanged. zsh's source calls this metafying, and 0x83 is the escape byte.
  • So a byte search of the file misses some words. grep -c 'zażó' found the line, because none of those bytes were escaped. grep -c 'jaźń' found nothing, because ń is stored as c5 83 a4.
  • zsh reads its own escaping back. fc -ln printed the command as typed. LC_ALL=C.UTF-8 is there so zsh prints the text as UTF-8 rather than spelling out the non-ASCII bytes.
  • EXTENDED_HISTORY adds : <start>:<elapsed>;: the command's start time in seconds, and how many seconds it ran.
  • zsh tells the two line shapes apart itself. The hand-written file mixes them. fc -l -t '%F %T' -D shows the start time and the elapsed 0:05 of sleep 5, and the plain line reads back as a command like the others.

fish writes records:

Verified output of histfile_fish.fish, identical on Linux and macOS — regenerated by tools/run_examples.py, never hand-typed.

$ cat add.fish
set -e fish_private_mode; set -e fish_history
history append "echo zażółć gęślą jaźń"
history append "echo two
echo lines"
history append "echo back\\slash"

$ fish --no-config add.fish

$ string replace -r "when: \d+" "when: <epoch>" < ~/.local/share/fish/fish_history
- cmd: echo zażółć gęślą jaźń
  when: <epoch>
- cmd: echo two\necho lines
  when: <epoch>
- cmd: echo back\\slash
  when: <epoch>

$ head -n 1 ~/.local/share/fish/fish_history | perl -ne 'printf "%*v02x\n", " ", $_'
2d 20 63 6d 64 3a 20 65 63 68 6f 20 7a 61 c5 bc c3 b3 c5 82 c4 87 20 67 c4 99 c5 9b 6c c4 85 20 6a 61 c5 ba c5 84 0a

$ cat data/fish/fish_history
- cmd: echo one
  when: 1700000000
- cmd: echo two
  when: 1700000060

$ XDG_DATA_HOME=$PWD/data fish --no-config -c 'set -e fish_private_mode; set -e fish_history; history --show-time="%F %T  "'
2023-11-14 22:14:20  echo two
2023-11-14 22:13:20  echo one
  • Each command is a - cmd: line with a when: line under it, in seconds since 1970. A command typed at a terminal can also get a paths: list of the path arguments that existed, like the cp in The history is a list in memory.
  • A newline in a command is written \n, and a backslash \\, so every record stays on its own lines.
  • Other bytes are written as they are. After the seven bytes of - cmd:, the hex is the same UTF-8 that bash wrote, with nothing escaped.
  • The file lives in $XDG_DATA_HOME/fish, or ~/.local/share/fish when that is unset. history --show-time prints the when: times in the TZ time zone (UTC here), newest first.
bash zsh fish
where $HISTFILE, ~/.bash_history $HISTFILE; ~/.zsh_history from macOS's /etc/zshrc $XDG_DATA_HOME/fish/fish_history, else ~/.local/share/fish/fish_history
one command one line one line a - cmd: record
its time a #<seconds> line, if HISTTIMEFORMAT is set : <start>:<elapsed>; with EXTENDED_HISTORY when:, always
non-ASCII the bytes typed metafied the bytes typed
newline, backslash not measured here not measured here \n, \\

If you are coming from another library

See also