uni — the character's name¶
Level: 101 → 201 · for anyone who has stared at c5bc and still not known what it is
One line: Every dump tool on your machine answers "how is this stored?"; uni is the one that answers "what is this?" — the code point, the decimal, the UTF-8 bytes, the HTML entity and the character's name, in one row, from one command.
The column no dumper has¶
printf '%s = ' "$(cat f)"; xxd -p f prints ż = c5bc, which is the shortest useful thing a terminal can tell you about text. It is also only half an answer. c5bc is the character's spelling in UTF-8. It is not the character's number — that is U+017C — and neither of them is its name.
For A all three coincide closely enough to hide the distinction for an entire career: 41, U+0041, LATIN CAPITAL LETTER A. Above U+007F they diverge completely, and no amount of staring at c5bc produces 017C, let alone the name.
That is the gap uni fills, and it is the one tool in the install list that earns its place the day you install it.
$ uni identify 'żé€'
Dec UTF8 HTML Name
'ż' U+017C 380 c5 bc ż LATIN SMALL LETTER Z WITH DOT ABOVE
'é' U+00E9 233 c3 a9 é LATIN SMALL LETTER E WITH ACUTE
'€' U+20AC 8364 e2 82 ac € EURO SIGN
The UTF8 column is the one xxd -p gives you. The other four — the code point, Dec, HTML and Name — are the ones it cannot, and Name is the answer to the question you actually had. Note that the code point is the column with no heading, printed against the character itself.
It goes both ways¶
identify takes characters and gives you facts. The other two directions are what make it a tool rather than a lookup:
$ uni print U+017C
Dec UTF8 HTML Name
'ż' U+017C 380 c5 bc ż LATIN SMALL LETTER Z WITH DOT ABOVE
$ uni print 0x41-0x45 -c
'A' U+0041 65 41 A LATIN CAPITAL LETTER A
'B' U+0042 66 42 B LATIN CAPITAL LETTER B
'C' U+0043 67 43 C LATIN CAPITAL LETTER C
'D' U+0044 68 44 D LATIN CAPITAL LETTER D
'E' U+0045 69 45 E LATIN CAPITAL LETTER E
$ uni search polish -c
'ƍ' U+018D 397 c6 8d ƍ LATIN SMALL LETTER TURNED DELTA [reversed Polish-hook o]
'Ꟁ' U+A7C0 42944 ea 9f 80 Ꟁ LATIN CAPITAL LETTER OLD POLISH O
'ꟁ' U+A7C1 42945 ea 9f 81 ꟁ LATIN SMALL LETTER OLD POLISH O
'💅' U+1F485 128133 f0 9f 92 85 💅 NAIL POLISH [manicure, nail care]
print takes a code point or a range — the thing you want when a hex dump has handed you U+017C and nothing else. search takes words and looks through every name in the database, which is the direction no xxd, od or hexdump can go at all. It also searches honestly rather than cleverly, as NAIL POLISH demonstrates.
And it reads stdin, so it drops into a pipeline where the rest of this chapter lives:
The use that saves an afternoon¶
This is the one to remember, because it ends an argument that otherwise runs for a day:
$ uni identify 'é' # typed on a Mac with option-e, e
'e' U+0065 101 65 e LATIN SMALL LETTER E
'◌́' U+0301 769 cc 81 ́ COMBINING ACUTE ACCENT [stress mark, Greek oxia, tonos]
$ uni identify 'é' # pasted from a web page
'é' U+00E9 233 c3 a9 é LATIN SMALL LETTER E WITH ACUTE
Two rows against one. That is the whole of why find -name cannot see a file you can cat, why uniq counts one word twice, and why a search that should match does not. uni is how you find out in five seconds instead of by reading hex.
Note the ◌ that uni draws under the combining acute: a mark with nothing to sit on would otherwise land on whatever character preceded it in the output, mangling the table. uni -r turns that off when you want the raw character.
In Python¶
Everything above except the substring search is in the standard library, which matters for two reasons: it is the answer on a machine where you cannot install anything, and it is what lets this page be checked at all.
Verified output of naming_a_character_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. EVERY COLUMN uni PRINTS, FROM unicodedata
char code point dec utf-8 html name
A U+0041 65 41 A LATIN CAPITAL LETTER A
é U+00E9 233 c3 a9 é LATIN SMALL LETTER E WITH ACUTE
ż U+017C 380 c5 bc ż LATIN SMALL LETTER Z WITH DOT ABOVE
€ U+20AC 8364 e2 82 ac € EURO SIGN
ß U+00DF 223 c3 9f ß LATIN SMALL LETTER SHARP S
The utf-8 column is what `xxd -p` gives you. The other four are the
ones no dump tool has, and the last one — the NAME — is the answer to
'what is this character?', which is a different question from 'how is
it stored?' and has a different answer for every character above A.
One column is not identical to uni's: ż comes out as the numeric
ż here and as ż there, because html.entities carries the
HTML 4 set and uni carries HTML 5's. Both render the same character.
2. THE COLUMN THAT SETTLES AN NFC/NFD ARGUMENT
NFC 'é' 1 code point(s), 2 bytes
U+00E9 LATIN SMALL LETTER E WITH ACUTE
NFD 'é' 2 code point(s), 3 bytes
U+0065 LATIN SMALL LETTER E
U+0301 COMBINING ACUTE ACCENT
One é and one é. Reading the names is how you find out which one is
in front of you, and it is the fastest end to the argument about why
a filename or a search 'does not match' when it plainly should.
3. GOING THE OTHER WAY: FROM A NAME TO A CHARACTER
EURO SIGN -> '€' U+20AC e2 82 ac
LATIN SMALL LETTER Z WITH DOT ABOVE -> 'ż' U+017C c5 bc
NO-BREAK SPACE -> '\xa0' U+00A0 c2 a0
unicodedata.lookup() needs the name EXACTLY, which is the difference
from `uni search`: uni does a substring search over the whole database
and Python has no index to do that with. The standard library answers
'what is this?'; the tool also answers 'what was it called again?'.
4. THE PROPERTIES BEHIND THE OTHER TOOLS' BEHAVIOUR
char category combining bidi numeric
A Lu 0 L -
é Ll 0 L -
◌́ Mn 230 NSM -
€ Sc 0 ET -
5 Nd 0 EN 5.0
'Mn' is a non-spacing mark — the combining acute — and its combining
class of 230 is why it draws on top of the previous letter instead of
beside it. That single row is why 'e' + U+0301 takes one column on
screen and two positions in every string type in this library.
Installing it, and one caveat¶
brew install uni # macOS
go install zgo.at/uni/v2@latest # anywhere with Go — it is not in Ubuntu's archive
uni ships its own copy of the Unicode database rather than asking the system for one:
That is a feature — the answers do not change when someone updates a library underneath you — and a thing to check before quoting a result. Your Python's unicodedata is pinned to a different revision, chosen by whoever built your Python, so for a character added in the last few years the two can disagree about whether it exists at all. For anything in this library's cast the question never arises; every one of them has had the same name since the 1990s.
Homepage: arp242/uni ↗. MIT licensed. Every flag, command and column on its help screen is explained, keyword by keyword, in uni -h, line by line.
If you are coming from Python or ABAP¶
Python: unicodedata.name(ch) is the Name column, ord(ch) the Dec, f'U+{ord(ch):04X}' the code point, ch.encode('utf-8').hex(' ') the UTF8. The one thing with no standard-library answer is uni search — unicodedata.lookup() needs the name exactly, because there is no name index to search, only a table to consult. That asymmetry is worth noticing: going from a character to its name is a lookup, and going from a word to a set of characters is a search, and only one of the two is cheap.
ABAP (Not machine-checked — CI cannot run ABAP.) There is no name database in the ABAP runtime at all. cl_abap_conv_* will give you the bytes and cl_abap_codepage=>convert_to the xstring, which is the UTF8 column; the code point you compute yourself from the UCS-2 representation. For the name you leave the system — which is precisely the argument for having uni on the machine you actually debug from, since the character that arrived in a field is usually the thing you need to name before anyone can decide what to do about it.
Try it¶
uni identify "$(pbpaste)"on whatever is in your clipboard right now. On Linux,xclip -o.- Take a filename that "does not match" and run
ls | uni identify -c. Look forCOMBINING. uni search space -cand count how many kinds of space there are. Then look up which one is in the CSV that will not parse.uni print U+FEFFanduni print U+FFFD— the two invisible characters this library talks about most.
Practice¶
The column no dump has. For é, write down the row uni would print: code point, decimal, UTF-8 bytes, HTML entity, category, and name. Then say which of those six columns xxd, od and hexdump can all give you, and which one none of them can.
Then the reverse direction: given only the words "latin small letter z with dot above", produce the character — and say why the name, and not the compose sequence you use to type it, is the identifier to put in a bug report.
Answers
Verified output of uni_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.
ch code pt decimal utf-8 html cat name
A U+0041 65 41 A Lu LATIN CAPITAL LETTER A
é U+00E9 233 c3 a9 é Ll LATIN SMALL LETTER E WITH ACUTE
ż U+017C 380 c5 bc ż Ll LATIN SMALL LETTER Z WITH DOT ABOVE
Ж U+0416 1046 d0 96 Ж Lu CYRILLIC CAPITAL LETTER ZHE
😀 U+1F600 128512 f0 9f 98 80 😀 So GRINNING FACE
THE COLUMN THAT MATTERS IS THE LAST ONE
Every dump on your machine answers 'how is this stored?'. xxd, od and
hexdump all print the same bytes in different arrangements. None of
them will tell you WHAT the character is, because none of them has
the table -- and the name is the only identifier that is a standard
rather than one project's shorthand.
SEARCH BY NAME, WHICH IS THE OTHER HALF
LATIN SMALL LETTER Z WITH DOT ABOVE -> ż U+017C
GRINNING FACE -> 😀 U+1F600
ud.lookup is the reverse of ud.name, and it is exact: the name is a
normative property, stable for the life of the code point, which is
why it is safe to write in source and in a bug report.
THE FOUR NAMES A CHARACTER HAS, AND WHICH ONE TO USE
the Unicode Name LATIN SMALL LETTER E WITH ACUTE -- normative
the code point U+00E9 -- normative
a compose sequence <Compose> e ' -- X11's table
a Vim digraph e' -- Vim's table
The first two identify the character to any program. The last two
are input methods, they disagree with each other, and neither is a
name the character actually has.
WHEN uni IS NOT INSTALLED
python3 -c "import unicodedata as u,sys;c=sys.argv[1];print(f'U+{ord(c):04X}',u.name(c))" é
That is the row this key printed, and it needs nothing but Python.
See also¶
uni -h, line by line — the help screen verbatim, every keyword linked to the page that explains it, and five places the screen and the program disagree- A character and its bytes on one line — the two columns you already have, without installing anything
- Unicode code points — reading
U+XXXXas an address rather than a number find, and filenames that are bytes — the problemuni identifydiagnoses fastest- The five worth installing — the rest of the optional toolbox