Fixed-width byte fields¶
Level: 201 → 301 · for interface work
One line: A fixed-width interface allocates bytes, not characters, so a 10-byte field holds ten ASCII letters or five Polish ones — and cutting a value at the tenth byte can land inside a character, which does not shorten the field, it stops the file being text.
Two units, one number, and only one of them is written down¶
CHAR(10). PIC X(10). LENGTH 10. Every fixed-width format in use says the width the same way — one number — and the number means bytes on one side of the interface and characters on the other, with nothing in the syntax to say which. That is the whole problem, and it is not solved by care: a field holding Nowak is ten bytes and five characters and works under either reading, so the mismatch is invisible until a value arrives with a ż in it.
The two readings diverge in both directions. Read a byte width as characters and you allocate a column that half your data does not fit in. Read a character width as bytes and you write past the field. And the moment somebody fixes the overflow with a truncation, the second bug arrives underneath the first.
Truncation is where it stops being a length problem¶
Cutting an ASCII value short loses information and leaves text. Cutting a UTF-8 value short at an arbitrary byte can leave half a character — a leading byte whose high bits promise a two-byte sequence, followed by the end of the field. What is on disk is no longer text in any encoding, and every reader downstream has to decide what to do about it, separately, with a different default.
The reason this reaches production rather than being caught in review is in section 2 of the Python run below. For the seventeen-character Polish pangram this page uses, 9 of the 26 possible field widths cut inside a character and 17 do not — and the 17 safe ones are not a coincidence, they are exactly the character count, because a byte prefix decodes precisely when it ends on a character boundary. So a naive value.encode()[:width] is correct for every ASCII value it was tested on, correct for most accented ones too, and wrong for the rest. Whether your code is correct is a property of the data and the width you were given, which means testing it against your own sample proves nothing at all.
What the split looks like¶
UTF-8's leading byte states its own length in its high bits — 110xxxxx promises two bytes, 1110xxxx three, 11110xxx four (UTF-8 by hand has the table). A truncation keeps the promise and removes the rest of it, so the resulting bytes are not wrong about their length; they are incomplete, which is a different failure and the one a decoder can name to the exact byte.
That precision is a feature and it is worth noticing what it costs elsewhere. Cut a code-page file at the same offset and nothing is detectable — every byte is a character, so a truncated Latin-2 record is a shorter, perfectly valid, silently wrong record. UTF-8's structure is what turns a length bug into an error message.
In Python¶
Verified output of fixed_width_byte_fields_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. THE FIELD IS BYTES; THE VALUE IS CHARACTERS
----------------------------------------------------------------------
the value 'zażółć gęślą jaźń'
len(value) 17 characters
len(value.encode()) 26 bytes
bytes per character 11222211221211122
So one CHAR(10) field, counted in bytes, holds either:
10 ASCII letters 'aaaaaaaaaa' 10 bytes
5 Polish letters 'żółćę' 10 bytes
Same field, half the letters, and the type name says neither.
2. THE NAIVE CUT IS RIGHT MOST OF THE TIME, WHICH IS WHY IT SHIPS
----------------------------------------------------------------------
value.encode()[:n], for every n from 1 to 26:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
. . X . X . X . X . . . X . X . . X . . . . X . X .
decodes: 17 splits a character: 9
the widths that split it: [3, 5, 7, 9, 13, 15, 18, 23, 25]
The 17 safe widths are not a coincidence: a prefix decodes exactly
when it ends on a character boundary, and this value has 17
characters. Safe widths = characters; the other 9 are the gaps
inside multi-byte sequences, one per byte a character has past
its first.
Width 10 is a clean cut for THIS value. The width decides
whether the code is correct, and the width is in somebody else's
specification. That is the whole shape of the bug: it is a
property of the data, so testing with your own data proves nothing.
3. WHAT A SPLIT SEQUENCE LOOKS LIKE AS BYTES
----------------------------------------------------------------------
cut at 3 bytes 7a 61 c5
one byte later 7a 61 c5 bc
^^ the byte the cut left behind
The last byte kept is 0xc5, which is 11000101 in binary.
A UTF-8 leading byte states its own length in its high bits:
110xxxxx promises two bytes, 1110xxxx three, 11110xxx four.
This one promises 2 and the field ends after 1.
So the sequence is not WRONG about its length; it is INCOMPLETE,
which is a different failure and the one a decoder can name to the
exact byte. That precision is the feature -- a cut in a code page
with no structure produces a different letter and says nothing.
4. WHAT EACH READER DOES WITH THE HALF CHARACTER
----------------------------------------------------------------------
.decode() UnicodeDecodeError
.start .end 2 3 -- byte offsets, not characters
the bytes it names c5
errors='replace' 'za�'
3 characters; re-encoding gives 5 bytes, round trip is LOST
errors='ignore' 'za'
2 characters; re-encoding gives 2 bytes, round trip is LOST
errors='backslashreplace' 'za\\xc5'
6 characters; re-encoding gives 6 bytes, round trip is LOST
errors='surrogateescape' 'za\udcc5'
3 characters; re-encoding gives 3 bytes, round trip holds
Only surrogateescape survives the round trip. It parks each stray
byte in an unpaired surrogate and hands the same byte back on the
way out, so a value you cannot read is still a value you can
forward. 'replace' and 'ignore' both destroy the byte, and they
destroy a different NUMBER of them, so neither length is the
original -- which is why neither belongs in a pipeline that
re-emits the record.
5. CUTTING IN THE RIGHT PLACE
----------------------------------------------------------------------
Five spellings of 'fit this value into the field', at width
13, where the naive one is wrong:
value[:13].encode() 20 bytes OVERFLOWS the field
value.encode()[:13] 13 bytes does NOT decode
...[:13].decode(ignore).encode() 12 bytes decodes, and hides a real error
loop over characters 12 bytes decodes 'zażółć g'
back up off continuation bytes 12 bytes decodes 'zażółć g'
The first counts the wrong unit and can overflow the field by up
to three bytes per character. The second is the bug. The third is
the tempting fix and the dangerous one: errors='ignore' cannot
tell a character YOU cut in half from bytes that arrived broken,
so it silences the truncation and the corruption with one word.
The last two are the same decision made in the two units, and
they agree byte for byte -- the loop is clearer, the back-up is
what you write when all you have is the buffer.
6. PADDING IS PART OF THE WIDTH, AND THE PAD IS AN ENCODED CHARACTER
----------------------------------------------------------------------
value 'żółw' 4 characters, 7 bytes
ljust(10) then encode 13 bytes padded in CHARACTERS, overflows
encode then pad to 10 10 bytes padded in BYTES, correct
c5 bc c3 b3 c5 82 77 20 20 20
The pad is a character too, so it is encoded like one -- and in a
two-byte encoding a field cannot be padded with single 0x20 bytes:
'żółw' as UTF-16BE 01 7c 00 f3 01 42 00 77 (8 bytes)
+ 2 ASCII spaces 20 20 -> 'żółw†'
+ 3 ASCII spaces 20 20 20 -> UnicodeDecodeError
+ 4 ASCII spaces 20 20 20 20 -> 'żółw††'
Two spaces are not two spaces: 0x20 0x20 is one UTF-16 code unit,
U+2020, which is a DAGGER. An odd number of them leaves the field
an odd number of bytes long and no decoder will read it at all.
The correct pad is the space ENCODED -- 00 20 per slot:
+ 2 encoded spaces 00 20 00 20 -> 'żółw '
In UTF-8 a space is one byte, so this particular mistake cannot
happen -- which is exactly why nobody thinks about it until the
field on the other side of the interface is UTF-16.
Section 4 is the part to take to work. Four errors= handlers, four different answers, and only surrogateescape returns the same bytes it was given. replace and ignore both destroy the stray byte and they destroy a different number of them, so neither result has the original length — which is why neither belongs in a pipeline that re-emits the record. Encode, decode and errors works through all eight handlers.
And section 5 has the fix people actually reach for, which is the dangerous one. raw[:width].decode('utf-8', errors='ignore').encode('utf-8') produces the right answer here, and it is still wrong, because ignore cannot tell a character you cut in half from bytes that arrived broken. One word silences the truncation you caused and the corruption you were sent, and the log line looks the same either way. Cut on a boundary instead — either by walking characters until the byte budget is spent, or by backing up off continuation bytes in the buffer. The run shows both, and they agree byte for byte.
In Rust¶
Verified output of fixed_width_byte_fields_rs.rs — regenerated by tools/run_examples.py, never hand-typed.
1. THE SAME ARITHMETIC, IN THE TYPE THAT ENFORCES IT
----------------------------------------------------------------------
VALUE.len() 26 bytes <- len() is BYTES on a &str
VALUE.chars().count() 17 characters <- and this one is O(n)
'ż'.len_utf8() 2
A &str is UTF-8 by definition, so the byte length is the
field width and the character count is what the user typed.
Rust makes you pick which one you meant at every call site.
2. WHICH WIDTHS ARE BOUNDARIES
----------------------------------------------------------------------
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
. . X . X . X . X . . . X . X . . X . . . . X . X .
is_char_boundary says 9 of 26 widths cut inside a character:
[3, 5, 7, 9, 13, 15, 18, 23, 25]
The same map the Python run prints, from a different library.
3. THE THREE WAYS TO ASK FOR A PREFIX
----------------------------------------------------------------------
at width 13, which is not a boundary:
VALUE.get(..13) None
&VALUE[..13] panicked: true
VALUE.floor_char_boundary(13) 12
floor_by_hand(raw, 13) 12
`get` returns None -- a value you can handle. The index
operator panics, because a &str that is not UTF-8 is a
contradiction rather than an error case, and there is no
value it could return. The two boundary functions agree,
which is the point of showing the hand-rolled one.
4. WHAT `from_utf8` SAYS ABOUT A CUT SEQUENCE
----------------------------------------------------------------------
the bytes 7a 61 c5 bc c3 b3 c5 82 c4 87 20 67 c4
Err(Utf8Error)
valid_up_to() 12
error_len() None
the valid prefix "zażółć g"
and for bytes that are wrong rather than cut short:
7a 61 c5 41 7a valid_up_to() 2 error_len() Some(1)
error_len() is None for exactly one reason: the input ended
in the middle of a sequence that was otherwise fine. That is
the truncation signature, and it is a VALUE -- so a reader
can say 'this record was cut, ask for more bytes' and mean
something different by 'this record is corrupt'. Python
distinguishes the same two cases only in the text of
UnicodeDecodeError.reason; .start and .end are identical for
both, and the text is a diagnostic that may be reworded.
5. THE LOSSY READER, AND WHAT IT COSTS
----------------------------------------------------------------------
from_utf8_lossy "zażółć g�"
characters 9
bytes if re-encoded 15
U+FFFD in it 1
One replacement character stands in for the one byte that
was left behind, and it costs three bytes to write. So the
repaired value is LONGER than the field it came out of --
which is how a lossy read of a fixed-width file overflows
the very column it is being loaded into.
6. CUTTING BY CHARACTERS INSTEAD
----------------------------------------------------------------------
char_indices() walk "zażółć g" (12 bytes)
floor_char_boundary "zażółć g" (12 bytes)
Same answer. Reach for floor_char_boundary when you have a
byte budget and the walk when you also need the count, the
offsets, or a rule about which characters may be dropped.
Section 4 is the sharpest thing either language says about this. Utf8Error::error_len() returns None for exactly one condition: the input ended in the middle of a sequence that was otherwise fine. That is the truncation signature as a value, so a reader can act on "this record was cut short, ask for more bytes" and mean something genuinely different by "this record is corrupt". Python distinguishes the same two cases only in the wording of UnicodeDecodeError.reason — measured here, .start and .end are 2 and 3 for a truncated sequence, an invalid continuation byte and a lone continuation byte alike — and a diagnostic string is not a property of your data.
Section 5 is the one that surprises people at load time. from_utf8_lossy repairs the value by substituting U+FFFD, which costs three bytes to write where the byte it replaced cost one. A lossy read of a 13-byte field produces a 15-byte value. The repair overflows the column it is being loaded into, and the error surfaces two systems away from the truncation that caused it.
And &s[..13] panics rather than returning anything. A &str that is not UTF-8 is a contradiction in Rust rather than an error case, so there is no value the index operator could hand back. The message is worth reading once, because it names the character and its byte range rather than just the offset:
thread 'main' panicked at fixed_width_byte_fields_rs.rs:67:54:
end byte index 13 is not a char boundary; it is inside 'ę' (bytes 12..14 of string)
The non-panicking spellings are get(..n), which returns Option<&str>, and floor_char_boundary(n), which is std as of Rust 1.91 — the example carries the hand-rolled walk beside it, because that is what the same decision looks like on an older compiler and in C. Slicing by byte is the same panic met from the other direction.
Padding is part of the width, and the pad is an encoded character¶
Section 6 of the Python run. Padding in characters and then encoding overflows the field by exactly the bytes the accents cost; encoding and then padding to the byte width is the correct order, and the difference is silent.
The sharper case is a field in a two-byte encoding. Padding a UTF-16 field with single 0x20 bytes does not add spaces: 20 20 is one UTF-16 code unit, U+2020, which is a dagger. An odd number of them leaves the field an odd number of bytes long and no decoder will read it at all. The correct pad is the space encoded — 00 20 per slot. In UTF-8 a space is one byte, so this mistake cannot happen, which is precisely why nobody thinks about it until the field on the other side of the interface is UTF-16.
If you are coming from Python or ABAP¶
The Python half is above; the words that transfer are str against bytes, and the rule that a field width belongs to the bytes side.
ABAP has the same two units and names them in the type, which is the good news. A c LENGTH 10 is ten characters, and on a Unicode system it occupies exactly twenty bytes, because the ABAP language is UCS-2 and a character is always two — so the width you declare and the width the file needs are related by a constant you can compute, right up until a character outside the BMP arrives and the constant is wrong. string is the variable-length type and has no field width at all. xstring (and x LENGTH n) is the byte side, and the pairing to hold on to is c/string for text and x/xstring for bytes, exactly as str/bytes pair in Python.
The conversion between them is where the width changes, and it is explicit: cl_abap_conv_out_ce / cl_abap_conv_in_ce (newer systems: cl_abap_conv_codepage) take a code page and hand back an xstring, and that length is the one a fixed-width file is measured in. So the discipline is the same as the Python one — do the truncation on the xstring after the conversion, not on the c field before it, and back up off a continuation byte rather than cutting on the byte the width names. OPEN DATASET … IN BINARY MODE gives you the byte view; IN TEXT MODE ENCODING does the conversion for you and applies the width in characters, which is the reading you have to be sure the other system agrees with. Verify any code-page number against the system rather than a document. (Not machine-checked — CI cannot run ABAP.)
And the mainframe case is worth one line, because it changes which byte is which rather than how many there are: an EBCDIC fixed-width record is single-byte throughout, so no character can be split — and the pad is 0x40, not 0x20, and even A is a different byte. A truncation there is a length bug with no error message attached, which is the older and quieter version of everything on this page. SAP code pages has the tables.
Try it¶
- Take the widest
CHARcolumn in an interface you actually maintain and find out, from the specification rather than the code, whether the number is bytes or characters. If the document does not say, you have found the bug before it fired. - Run
awk '{ print length($0) }' fileandwc -L fileover a fixed-width export with accented data, then compare both againstLC_ALL=C awk '{ print length($0) }'. The three answers tell you which of your tools is counting bytes. - Cut one of your own records with
head -c N, then runiconv -f UTF-8 -t UTF-8 < cut > /dev/null; echo $?. That is the validator, and its exit status is the same on both platforms. - Search your codebase for
[:applied to something you.encode()d, and forsubstring/LEFTapplied to a value that later gets encoded. Each hit is a width in the wrong unit until you prove otherwise.
See also¶
- Packing a record — the sibling: the same byte budget in a binary record, where the layout is not in the file either, and where the answer is to refuse rather than to truncate
- Slicing by byte — the same boundary, met as a panic
- SAP code pages — the code page a fixed-width interface is measured in
- Encode, decode and errors — all eight
errors=handlers, and what each costs - Validation is a boundary — checking at the door rather than downstream
- Escaping into ASCII — which row your length limit is measured on
- Interfaces and storage — where the width and the encoding are declared