Interfaces and storage¶

Level: 201 · for anyone starting from zero

One line: Every protocol has exactly one place where the encoding is declared, and the whole job is knowing where that place is — because three of the eight channels below have nowhere to put it at all, and when nobody declares it, somebody guesses.

The closing rule, in four clauses¶

Everything in this library reduces to one sentence, and this page is that sentence with the failures shown:

Decode at the boundary. Hold text in the middle. Encode at the boundary. Write the encoding down in the contract.

The first three clauses are the sandwich, and they are the part everybody has heard. The fourth is the one that gets skipped, and it is the only one that survives you leaving the team — because the first three describe your program and the fourth describes the agreement, which is the thing the other system is also reading.

The reason it gets skipped is that it sounds like paperwork. It is not: a boundary you did not declare is a boundary where two programs each picked a default, and defaults are per-machine, per-language and per-decade. The open() with no encoding= on a colleague's Windows box, the database column whose utf8 is three bytes wide, the CSV nobody put a BOM on — each of those is one undeclared boundary, and each produces a bug that reproduces on exactly one machine.

Where each channel keeps the answer¶

Verified output of interfaces_and_storage_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. ONE NAME, EIGHT ROWS, AND HOW MANY DIFFERENT SIZES
------------------------------------------------------------------------
   the text itself            4 characters   Łódź
   UTF-8 bytes                     7 bytes   c5 81 c3 b3 64 c5 ba
   JSON, default                  31 bytes   {"city": "\u0141\u00f3d\u017a"}
   JSON, ensure_ascii=False       19 bytes   {"city": "Łódź"}
   CSV, no BOM                    13 bytes   'city,Łódź\n'
   CSV, utf-8-sig                 16 bytes   'city,Łódź\n'
   URL query string               24 bytes   city=%C5%81%C3%B3d%C5%BA
   CHAR(10), byte-counted         10 bytes   c5 81 c3 b3 64 c5 ba 20 20 20

   8 rows and 7 different byte sizes for one name. Every
   pipeline above is correct; each is measuring a different layer.
   A field width, a length limit or a quota applies to exactly ONE
   of these rows, and it is rarely the first -- which is the
   question to ask before sizing a column, not after the insert
   fails in production.

2. WHERE EACH CHANNEL DECLARES THE ENCODING
------------------------------------------------------------------------
   channel       the declaration                     who decides
   HTTP          Content-Type: ...; charset=utf-8    in the header, and it wins
   HTML          <meta charset> -- only if no header the header outranks it
   JSON          nowhere: RFC 8259 fixes UTF-8       no charset parameter exists
   XML           <?xml version encoding=...?>        in the document, self-describing
   CSV           nowhere at all                      a BOM, or a document, or a guess
   fixed-width   nowhere at all                      the interface specification
   URL query     nowhere: %XX escapes bytes          RFC 3986 can only recommend
   email header  =?utf-8?B?...?= names it inline     the only one that always does

   Three groups, not two. 4 of the 8 carry the answer INSIDE the
   data, where a reader can find it without being told. JSON has
   nowhere to put it and does not need one, because the format FIXES
   it -- one encoding, no parameter, no negotiation, which is the
   design worth copying. And 3 -- CSV, fixed-width, URL query --
   have nowhere to put it AND nothing fixing it, so the answer lives
   in a document or in somebody's memory. Those three are the ones
   that produce a ticket saying 'the file is corrupt'.

3. JSON IS UTF-8 BY THE RFC, SO 'JSON IN ANOTHER ENCODING' IS NOT A THING
------------------------------------------------------------------------
   encoding this record as cp1252     UnicodeEncodeError
     the character it stopped on      'Ł'  U+0141
     of Polish's nine letters cp1252 holds  ó   (1 of 9)
   So the question 'what if this JSON were cp1252' cannot even be
   asked of this record -- the encode fails before any JSON exists.
   ('ó' survives because it is in the Latin-1 half that cp1252 kept;
   the other eight are the ones Latin-1 never had.) Take a value it
   CAN hold, and the answer arrives one layer later instead:

     as UTF-8    7b 22 63 69 74 79 22 3a 20 22 63 61 66 c3 a9 22 7d
     as cp1252   7b 22 63 69 74 79 22 3a 20 22 63 61 66 e9 22 7d
     json.loads(bytes encoded utf-8  ) -> {'city': 'café'}
     json.loads(bytes encoded cp1252 ) -> UnicodeDecodeError

   RFC 8259 section 8.1 fixes the encoding of an exchanged JSON text
   to UTF-8, and the media type registration defines no charset
   parameter -- so `Content-Type: application/json; charset=cp1252`
   is not a JSON document in another encoding, it is a broken one
   carrying a parameter nothing is required to read. json.loads
   takes bytes and decodes them itself, which is why the refusal
   above is a DECODE error rather than a syntax error: the parser
   never got far enough to have an opinion about the JSON.

   The escape form is the other half of that. json.dumps defaults to
   ensure_ascii=True, which is not an encoding choice -- it is a
   channel choice, for a pipe that will only carry ASCII:
     default             {"city": "\u0141\u00f3d\u017a"}
     ensure_ascii=False  {"city": "Łódź"}
     same parsed value?  True
   Same document, same parsed value, different size on the wire, and
   a \uXXXX escape that is a UTF-16 code unit inside a UTF-8 format.

4. CSV IS THE ONE WITH NOWHERE TO PUT IT
------------------------------------------------------------------------
   Read the BOM'd file with the wrong codec and the damage lands in
   the one place a program will not look -- the first column NAME:
     decode as utf-8      header[0] = '\ufeffcity'   len 5
     decode as utf-8-sig  header[0] = 'city'   len 4

   Both decodes SUCCEEDED. Nothing raised, nothing warned, and one of
   them produced a column called something no lookup will match.
   'utf-8-sig' strips a BOM if there is one and is harmless if there
   is not, so it is the right codec for reading somebody else's CSV;
   plain 'utf-8' is the right one for writing yours.

   The other two arguments that belong on every csv open():
     newline=''       the module handles line endings itself, and a
                      quoted field may CONTAIN one
     encoding='utf-8' because the default is the machine's, and the
                      machine writing the file is not the one reading

5. THE SANDWICH, AND THE ONE HOP THAT USUALLY MISSES IT
------------------------------------------------------------------------
   Decode at the boundary, hold text in the middle, encode at the
   boundary. The failure is never the whole program -- it is one hop
   where a value stayed bytes and got treated as text anyway:

   bytes arriving from a socket    c5 81 c3 b3 64 c5 ba
     .upper() on the BYTES         b'\xc5\x81\xc3\xb3D\xc5\xba'
     .upper() on the TEXT          'ŁÓDŹ'

   bytes.upper() did not raise and did not refuse. It is documented
   as ASCII-only, so it changed exactly 1 byte of 7 -- the 'd' --
   and left every byte of 'Ł', 'ó' and 'ź' as it found them. A
   half-uppercased value, no error, no log line, and it will compare
   unequal to the properly uppercased one forever.

   Note what did NOT differ: both answers are the same length here.
     'Łódź'.upper()   4 -> 4 characters, 7 -> 7 bytes
   Case mapping is a table lookup and the table is free to change
   the length -- 'ß'.upper() is 'SS', 1 character becoming 2 --
   so 'uppercasing does not resize a field' is a fact about THIS
   string, not a rule you can size a column with.

6. AND WHAT 'WRITE IT DOWN' ACTUALLY MEANS
------------------------------------------------------------------------
   Seven lines, and every one of them is a question somebody will
   otherwise answer for you, differently, at three in the morning:

     encoding                                    UTF-8
     normalization form                          NFC (this record already is: True)
     line ending                                 LF
     BOM                                         absent
     field width unit                            bytes
     what happens to a value that does not fit   reject the record
     what happens to an undecodable byte         reject the record

   Note the last two. 'Which encoding' is the famous question and it
   is the easy one; the ones that decide whether an interface is
   debuggable are what it does when the rule is broken. An
   errors='ignore' nobody wrote down is a policy too -- it is just a
   policy chosen by whoever typed fastest.

Section 2 is the map, and the useful thing about it is that it has three groups rather than two.

Four channels carry the declaration inside the data, where a reader finds it without being told: HTTP's Content-Type: …; charset=utf-8, XML's declaration, HTML's <meta charset> — which the header outranks, so a page that renders wrongly for you and correctly for the developer is usually one where only the meta tag was edited — and the MIME encoded-word in an email header, which is the only one of the eight that always names its charset. (Escaping into ASCII has all four escape schemes and which layer each one wraps.)

One channel has no place for a declaration and does not need one, and it is the design worth copying. RFC 8259 §8.1 ↗ requires JSON exchanged between systems to be UTF-8, and the media type registration in §11 ↗ notes that no charset parameter is defined for it — adding one has no effect on a compliant recipient. So application/json; charset=cp1252 is not JSON in another encoding; it is a broken document carrying a parameter nothing is required to read. Section 3 of the run shows what that means in practice: json.loads takes bytes and decodes them itself, so a cp1252 payload fails as a decode error, before the parser is far enough in to have an opinion about the JSON.

And three — CSV, fixed-width, and a URL query string — have nowhere to put it and nothing fixing it. For those the encoding lives in a document, a convention, or somebody's memory, and they are the three that generate the ticket that says the file is corrupt.

The three with nowhere to put it¶

CSV is the format that carries a table of text and cannot say what text is. In practice the declaration is a BOM, which is Excel's de-facto marker and a parser's silent bug — the same three bytes doing opposite jobs, which A BOM in a CSV is entirely about. Section 4 of the run shows the specific damage: read a BOM'd file as plain utf-8 and both decodes succeed, but the first column is now named 'city' and every lookup against it misses. Read with utf-8-sig, which strips a BOM if there is one and is harmless if there is not; write with plain utf-8; and pass newline='' to the open() because the csv module handles line endings itself and a quoted field may contain one.

A fixed-width field declares a number and not a unit, and the number means bytes on one side of the interface and characters on the other. That is its own page, and the short version is that the width belongs in the contract next to the encoding, because neither is any use without the other.

A URL query string percent-escapes bytes and has no syntax for saying which encoding produced them. RFC 3986 §2.5 ↗ can only recommend UTF-8 for new schemes; an old form's %F3 from Latin-2 is exactly as well-formed as a modern %C3%B3 and means a different letter. The server has to know.

Databases: the storage half¶

The same rule, with the boundary in a place people forget is a boundary. Four facts worth carrying, none of which this library's examples can measure because no database is installed on either CI runner:

MySQL's utf8 is not UTF-8. It is a three-byte subset that cannot store a character above U+FFFF, so an emoji or a rare CJK ideograph fails to insert or is truncated depending on the mode. The real one is utf8mb4, which has been the server default since MySQL 8.0. A schema that says utf8 is a schema written before that and never revisited.
PostgreSQL's encoding is per-database and fixed at CREATE DATABASE, and client_encoding is a separate, per-session setting that converts on the way in and out — so a client with the wrong client_encoding produces mojibake in a database that is internally correct, which is a different bug from a database that is internally wrong, and the fix is not the same.
VARCHAR(50) counts different things in different engines — characters in PostgreSQL and in MySQL, bytes in some Oracle configurations (VARCHAR2(50 BYTE) against VARCHAR2(50 CHAR), and which one you get depends on a session parameter). This is the fixed-width unit problem wearing a schema.
A collation is a versioned data file, and anything stored in sorted order depends on that version. Sorting and collation has the measurement, and the two-year window in which PostgreSQL could not warn about it.

Filesystems, which are a boundary you did not choose¶

On Unix a filename is bytes — any bytes but NUL and / — so a name need not be valid UTF-8 at all, and Python hands you one through surrogateescape rather than refusing to list the directory. On Windows it is UTF-16, and may contain unpaired surrogates. macOS is the interesting one: HFS+ stored filenames decomposed (NFD), so a Polish or French filename written on a Mac and copied to Linux compares unequal to the same name typed on Linux — the bytes genuinely differ, and only normalization makes them equal. APFS no longer normalizes, but it is normalization-insensitive and refuses a name that is not valid UTF-8 outright, which is a third behaviour again. find, and filenames that are bytes measures the ones that can be measured here.

SAP, where the contract is the deliverable¶

An interface specification in this world already has a field for all of this, which is the good news; the bad news is that leaving it blank still produces a file. The system code page of a Unicode SAP system is UTF-16, while the ABAP language is UCS-2 and a character is always two bytes — so a c LENGTH 10 is ten characters and twenty bytes, and neither of those numbers is the width of the file you are writing. OPEN DATASET … IN TEXT MODE cannot leave the encoding blank — ENCODING is mandatory there, and DEFAULT has meant UTF-8 since 7.50 ↗ — but a LEGACY open with no CODE PAGE ↗ can, and then it converts through whatever code page table TCP0C assigns to the logon language of whoever runs the job, which is a choice disguised as an absence. ENCODING NON-UNICODE makes the same choice out loud. The code-page number — 1100, 4110, 4103 — belongs to the interface specification rather than to the file, because nothing in the file records it; SAP code pages has the table, and every number on it should be verified on the system rather than quoted from a document.

If you are coming from Python or ABAP¶

Python. The whole rule is three habits. encoding='utf-8' on every open(), even where it is already the default, because the default is the machine's and the machine writing the file is not the one reading it. errors= chosen on purpose — strict for data you own, surrogateescape to carry somebody else's bytes through unharmed, replace only for a log line, never ignore. And newline='' for CSV. Python text in practice has the rest, including the interpreter switch that finds every open() you missed.

Section 5 of the run shows the hop where the sandwich usually leaks, and it is worth seeing because it does not raise: bytes.upper() on a UTF-8 payload changed exactly one byte of seven — the ASCII d — and left Ł, ó and ź untouched. The method is documented as ASCII-only, so this is correct behaviour producing a half-uppercased value with no error and no log line, which will then compare unequal to the properly uppercased one forever. Bytes that look like text are the whole hazard, and the defence is decoding at the door rather than checking every call.

ABAP. The pairing is string / c for text and xstring / x for bytes, exactly as str / bytes pair in Python, and the conversion between them is the boundary: cl_abap_conv_in_ce / cl_abap_conv_out_ce on older systems, cl_abap_conv_codepage on newer ones, and both take a code page as an argument rather than assuming one. That argument is the declaration. Do the conversion once, at the edge, and keep string in the middle — the ABAP shape of the sandwich, and the reason a program that passes xstring around internally is the one that later has a field it cannot explain. cl_abap_char_utilities is where the newline and tab constants live, which matters because the line ending is part of the contract too. Verify any code-page number against the system. (Not machine-checked — CI cannot run ABAP.)

What to actually write down¶

Section 6 of the run is a seven-line specification, and the last two lines are the ones that get left out:

line	why it is not obvious
encoding	the famous one, and the easy one
normalization form	`café` and `café` are different bytes and the same word
line ending	LF or CRLF, and git may be rewriting it under you
BOM	present or absent, decided rather than inherited
field width unit	bytes or characters — the number alone says neither
what happens to a value that does not fit	truncate, reject, or corrupt
what happens to an undecodable byte	reject, replace, or silently drop

"Which encoding" is the question everybody asks. The two that decide whether an interface is debuggable are the last pair, because an errors='ignore' that nobody wrote down is still a policy — it is just a policy chosen by whoever typed fastest, applied to data nobody will ever see again.

Try it¶

Open the last interface specification you were sent and look for the encoding. If it is not there, you now know the most useful thing about that interface; if it says "ASCII", find out what happens to the first customer named Müller.
curl -sI <your own API> | grep -i content-type. Does it name a charset? For application/json it should not, and if it does, find out whether anything reads it.
Take a CSV somebody sent you and run head -c 3 file | xxd. Then open it with encoding='utf-8' and with encoding='utf-8-sig' and print reader.fieldnames[0] both ways.
SHOW VARIABLES LIKE 'character_set%' on any MySQL you own, or SHOW server_encoding; SHOW client_encoding; on any PostgreSQL. If you find a utf8 that is not utf8mb4, you have found a column that cannot hold an emoji.
Write the seven lines above for one interface you maintain, and send them to the team on the other end. The disagreement that comes back is the bug you were going to have next quarter.