Where a line may break¶

Level: 301 · for anyone who has wrapped text they could not read

One line: Wrapping is not splitting on spaces — Japanese decides by a rule about the two adjacent characters, Thai decides by a dictionary, and textwrap implements neither, so it is wrong for a large fraction of the world's text.

import textwrap
textwrap.wrap('日本語は「スペース」を使いません。', 8)
# ['日本語は「スペー', 'ス」を使いません', '。']   <- a full stop, alone, starting a line

Every whitespace-based wrapper carries the same buried assumption: that a space is where a line may break, and that text therefore arrives pre-marked with its own break opportunities. For English that is close enough to true that nobody notices it is an assumption. For Japanese and Thai — written with no spaces between words at all — there is nothing to find, and the tool does not say so. It returns a plausible list of lines.

The right question is not where are the spaces but where may this line break, and UAX #14 ↗ is the document that answers it. Its answer has two halves, and the two are not the same kind of thing.

	how a break opportunity is found	can a small table do it?
Japanese	a rule over the classes of two adjacent characters	yes — and this page does
Thai	a dictionary: the boundary is between two words	no, and UAX #14 says so

The Japanese half is a rule, and rules can be written down¶

Japanese typesetting has a name for this — kinsoku shori, "forbidden-character handling" — and its most basic rule is the one textwrap breaks above: a full stop may not begin a line. Nor may a closing bracket, an exclamation mark, or a small kana. Nor may a line end with an opening bracket.

UAX #14 encodes exactly that, by giving every character a line-break class and stating rules over pairs of them. Four classes and three rules are enough for one ordinary sentence:

never break BEFORE   CL  closing bracket, 。 、        (LB13)
                     EX  ！ ？                          (LB13)
                     NS  ー and the small kana          (LB21)
never break AFTER    OP  opening bracket                (LB14)
otherwise                between two ideographs, break  (LB31)

Nothing there consults a space, a font, or a dictionary. It asks which class each of two neighbouring characters is in, and that is the whole mechanism.

The Thai half is a dictionary, and this page cannot supply one¶

ภาษาไทย is two words — ภาษา (language) and ไทย (Thai) — set solid, with no space and no punctuation between them. The only correct break is in the middle, and no rule about adjacent characters can find it, because the boundary is not a property of the two characters either side of it. It is a fact about Thai vocabulary.

UAX #14 does not pretend otherwise. It assigns Thai the class SA, Complex Context Dependent, and hands the problem to a lexical analyser — that is, to a dictionary, or to a model trained on one.

The instructive part is what textwrap does here, because it is not simply wrong:

width 3  ->  ['ภาษ', 'าไท', 'ย']
width 4  ->  ['ภาษา', 'ไทย']      <- correct
width 5  ->  ['ภาษาไ', 'ทย']
width 6  ->  ['ภาษาไท', 'ย']

Width 4 is right. It is also luck — it is the one width whose greedy cut happens to land on the boundary, and every width either side of it splits a word down the middle. A wrapper that is correct at one width and wrong at the next knows nothing; and a test written at one width would have passed.

In Python¶

Verified output of where_a_line_may_break_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. THE STRING, AND WHY `SPLIT ON WHITESPACE` HAS NOTHING TO WORK WITH
------------------------------------------------------------------------
   日本語は「スペース」を使いません。
   17 characters, 51 bytes, and:
     JA.split()      -> ['日本語は「スペース」を使いません。']
     ' ' in JA       -> False

   One 'word' as far as every whitespace-based tool is concerned.
   It is an ordinary sentence: `Japanese does not use spaces.`

2. WHAT textwrap DOES WITH IT
------------------------------------------------------------------------
   textwrap.wrap(text, 8):
     1  |日本語は「スペー|
     2  |ス」を使いません|
     3  |。|

   Look at the last line. It is a full stop, alone, at the start of
   a line. In Japanese typesetting that is the first rule anyone
   learns -- kinsoku shori, `forbidden-character handling` -- and it
   is forbidden in every style guide, every word processor and every
   browser. `textwrap` is not broken; it was asked a question about
   spaces and answered it.

3. THE SAME STRING, BROKEN BY RULE
------------------------------------------------------------------------
   Three rules, and the class of the two adjacent characters decides:

     never break BEFORE  CL (closing bracket, 。 、)  EX (！？)  NS (ー, small kana)
     never break AFTER   OP (opening bracket)
     otherwise, between two ideographs, break freely

   Every position in the sentence, and the verdict:

   pos  left   right  classes    break?
     1  日      本      ID-ID      yes
     2  本      語      ID-ID      yes
     3  語      は      ID-ID      yes
     4  は      「      ID-OP      yes
     5  「      ス      OP-ID      NO
     6  ス      ペ      ID-ID      yes
     7  ペ      ー      ID-NS      NO
     8  ー      ス      NS-ID      yes
     9  ス      」      ID-CL      NO
    10  」      を      CL-ID      yes
    11  を      使      ID-ID      yes
    12  使      い      ID-ID      yes
    13  い      ま      ID-ID      yes
    14  ま      せ      ID-ID      yes
    15  せ      ん      ID-ID      yes
    16  ん      。      ID-CL      NO

   wrap_by_rule(text, 8):
     1  |日本語は「スペー|
     2  |ス」を使いませ|
     3  |ん。|

   The full stop stayed with its sentence, and the closing bracket
   stayed with the word it closes. Nothing here consulted a space,
   a dictionary or a font -- only which of five classes each of two
   neighbouring characters is in.

4. AND NOW THAI, WHERE THE RULE DOES NOT EXIST
------------------------------------------------------------------------
   ภาษาไทย   -- 7 characters, 21 bytes, no space
   It is two words: ภาษา (language) + ไทย (Thai). The only correct
   break is between them, at position 4.

   Every position, through the same machinery as section 3:

   pos  left   right  classes    break?
     1  ภ      า      ID-ID      yes
     2  า      ษ      ID-ID      yes
     3  ษ      า      ID-ID      yes
     4  า      ไ      ID-ID      yes
     5  ไ      ท      ID-ID      yes
     6  ท      ย      ID-ID      yes

   Six positions, six yeses. The rules cannot see the one answer that
   matters, because in Thai the boundary is not a fact about two
   adjacent characters -- it is a fact about the vocabulary. UAX #14
   says so itself: it puts Thai in class SA, `Complex Context
   Dependent`, and hands the problem to a lexical analyser.

   textwrap on the same string, at four widths:
     width 3  ->  ['ภาษ', 'าไท', 'ย']
     width 4  ->  ['ภาษา', 'ไทย']
     width 5  ->  ['ภาษาไ', 'ทย']
     width 6  ->  ['ภาษาไท', 'ย']

   Width 4 is CORRECT. It is also luck: it is the only width whose
   greedy cut happens to land on the boundary, and the three either
   side of it split a word down the middle. A wrapper that is right
   at one width and wrong at the next is not a wrapper that knows
   anything -- which is the most useful thing on this page, because
   testing at one width would have shown a pass.

5. WHAT THE STANDARD LIBRARY HAS, AND WHAT IT DOES NOT
------------------------------------------------------------------------
   properties `unicodedata` exposes that concern line breaking: 0

   It has category, combining class, decomposition, bidirectional,
   east_asian_width and normalize. It does not have Line_Break, so
   there is nothing in the standard library to look a class up in --
   which is why the table at the top of this file is a table at the
   top of this file.

   Two more things that were NOT done here, deliberately:

     * Nothing measured a COLUMN. This program's `width` counts
       characters, and a real wrapper counts columns or pixels --
       every one of the CJK characters above is two columns wide.
       Measuring that means reading east_asian_width, which is a
       fact about the machine, and this page keeps such facts out
       of its answer key.
     * Nothing here is the whole of UAX #14. It defines dozens of
       classes and thirty-odd numbered rules; this is four classes
       and three rules, chosen to be exactly enough for one sentence.
       It is a demonstration of the mechanism, not an implementation
       of the annex.

   And one place where even this small table took a side. The
   prolonged sound mark ー and the small kana are class CJ,
   `Conditional Japanese Starter`, which UAX #14 resolves to NS in
   STRICT mode and to ID in NORMAL and LOOSE mode. The table above
   picked strict. Under the loose reading, position 7 becomes a legal
   break and this sentence wraps differently -- so `where may this
   line break` does not have one answer even after you have the
   annex, and CSS exposes the choice as `line-break: strict | normal
   | loose` precisely because publishers disagree about it.

What the crate — and ICU — adds¶

The hand-rolled breaker above is four classes out of dozens and three rules out of thirty-odd. Naming what it leaves out is the honest half of the page:

The rest of UAX #14. Hyphens, non-breaking glue, numeric sequences, Korean syllable blocks, regional indicators, emoji ZWJ sequences, the rules for what happens around spaces. The annex is thirty-odd numbered rules applied in order, and their order matters the way stringprep's four steps do.
The table. Every character's class, for every assigned code point. That is data, not cleverness — the same conclusion the Rust half of Preparing a string reaches, and the reason these things ship as libraries.
The dictionary. For Thai, Lao, Khmer, Burmese and Japanese-quality segmentation, a word list plus a model over it. ICU ships a Burmese/Khmer/Lao/Thai dictionary and a CJK one; that is a data file, and it is why the library is measured in megabytes.
The tailoring. Even with the annex in hand, the answer is not unique. The prolonged sound mark ー and the small kana are class CJ, which resolves to NS in strict mode and to ID in normal and loose mode — so the sentence on this page wraps two different ways depending on which published mode you pick. CSS exposes the choice as line-break: strict | normal | loose precisely because publishers disagree.

That last one is worth sitting with. It is not a gap in the implementation; it is the standard declining to have one answer, and shipping three.

This is the same shape as the ICU claim at the foot of "Handles Unicode" is four questions: most languages' text handling is ICU wearing a hat, and where a language's standard library is not ICU, the missing piece is almost always a table or a dictionary rather than an algorithm.

If you are coming from Python or ABAP¶

Python. textwrap is an ASCII-era tool and its docstring does not claim otherwise: it splits on whitespace and hyphens, and break_long_words breaks between arbitrary code points. There is no UAX #14 anywhere in the standard library — unicodedata exposes category, combining class, decomposition, bidirectional and east_asian_width, and not Line_Break, so there is nothing to look a class up in. If you are wrapping text you did not write in a language you do not know, use uniseg ↗ or PyICU ↗, and if you are rendering to a terminal remember that textwrap's width counts characters while your terminal counts columns — 日 is one of the first and two of the second.

ABAP. (Not machine-checked — CI cannot run ABAP.) There is no UAX #14 in ABAP either. Wrapping in classic reports is column arithmetic over a c field, which is a character count and not a column count, so CJK text in a fixed-width report list is already misaligned before any break rule is considered — the same one-character-one-column assumption, one layer lower down. SPLIT … AT SPACE has the identical blind spot to textwrap. Where output goes to Smart Forms or Adobe Forms the renderer does the breaking and generally does it properly, which is the practical advice: for CJK and Thai output, let the form renderer wrap and do not pre-wrap in ABAP, because a pre-wrapped string arrives with its bad breaks already baked in and the renderer cannot undo them.

Try it¶

Take the longest non-English string in your own data — a Japanese product name, a Thai address — and run your language's wrapper over it at three widths in a row. If the results are not all defensible, you have found the bug this page is about.
Find where your codebase wraps text for a fixed-width surface (an email, a terminal table, a PDF) and ask what it measures: characters, columns, or pixels. Those are three different numbers for the same string.
In a browser, put a Japanese sentence in a narrow element and toggle line-break: strict against line-break: loose in dev tools. Watch a legal break appear and disappear on the same text.
Run python3 -c "import unicodedata; print([n for n in dir(unicodedata) if 'break' in n])" — the empty list is the reason this page hand-rolls a table.
If any system you own stores a "line 1 / line 2" pre-wrapped address, work out which locales it was wrapped for, and what happens when a Thai address arrives.