Splitting text¶

One line: Splitting on a pattern looks like the simplest regex operation and disagrees across engines on three points — whether a captured delimiter comes back as a piece (Python, Perl, JavaScript, Ruby: yes; Java, Go, Rust: no), whether an empty pattern splits between every character or refuses, and how many empty pieces an edge produces — so a split that has to be portable uses a non-capturing pattern and strips its own empties.

Level: 201 · you have delimited text and want a list

What the driver asks¶

Every engine here was asked to split with "keep everything" semantics — Perl's -1 limit, Java's -1, Go's -1, Ruby's -1 — so the differences below are about the pattern, not about a limit argument. In your own code the defaults differ again: Perl and Ruby drop trailing empty pieces unless told not to; Java's String.split drops them too; Python, JavaScript, Go and Rust keep them.

Measured¶

Verified output of topic_splitting_text_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

construct                 python       perl         node         ruby         java      go       rust       fancy
plain delimiter           [a,b,c]      [a,b,c]      [a,b,c]      [a,b,c]      [a,b,c]   [a,b,c]  [a,b,c]    [a,b,c]
captured delimiter        [a,;,b,;,c]  [a,;,b,;,c]  [a,;,b,;,c]  [a,;,b,;,c]  [a,b,c]   [a,b,c]  [a,b,c]    [a,b,c]
trailing empty piece      [a,b,]       [a,b,]       [a,b,]       [a,b,]       [a,b,]    [a,b,]   [a,b,]     [a,b,]
leading empty piece       [,a]         [,a]         [,a]         [,a]         [,a]      [,a]     [,a]       [,a]
run of delimiters         [a,b]        [a,b]        [a,b]        [a,b]        [a,b]     [a,b]    [a,b]      [a,b]
on whitespace             [a,b,c]      [a,b,c]      [a,b,c]      [a,b,c]      [a,b,c]   [a,b,c]  [a,b,c]    [a,b,c]
zero-width split          [a,ba,b]     [a,ba,b]     [a,ba,b]     [a,ba,b]     [a,ba,b]  -        -          [a,ba,b]
split on every character  [,a,b,c,]    [a,b,c,]     [a,b,c]      [a,b,c,]     [a,b,c,]  [a,b,c]  [,a,b,c,]  [,a,b,c,]

Reading the table¶

Rows 1, 3–6 are unanimous once the limit is set to keep everything: three pieces for a;b;c, an empty trailing piece for a;b;, an empty leading piece for ;a, one split for a run of delimiters when the pattern is ;+, and whitespace runs as one delimiter.
Row 2, a captured delimiter. (;) on a;b;c: Python, Perl, JavaScript and Ruby return five pieces — the delimiters are included, [a,;,b,;,c]. Java, Go and Rust return three — the capture is ignored. This is the most-ported split bug there is: code that relies on getting the delimiters back moves from Python to Go and silently loses them; code that does not expect them moves the other way and gets a list twice as long.
Row 7, a zero-width pattern. (?=b) splits before each b in the six engines that have lookahead — [a,ba,b]. Go and Rust refuse the lookahead, not the split.
Row 8, the empty pattern. (?:) matches the empty string everywhere, and splitting on it produces four different answers: [,a,b,c,] with empties at both ends (Python, Rust), [a,b,c,] with one at the end (Perl, Ruby, Java), and [a,b,c] with none (JavaScript, Go). No engine refuses. The portable way to split into characters is not a regex.

The portable shape¶

split on   (?:DELIM)       — non-capturing, so no engine returns the delimiter
then       drop empties    — in the language, not by relying on the split's limit

If you want the delimiters back, and the engine is Java, Go or Rust, match instead: find every [^;]* or every [^;]+|; and sort the pieces in the loop. If you want them in Python or JavaScript, the capture does it — and now the code is not portable, which is worth a comment.

Keywords on this page¶

Capturing group · Non-capturing group · Lookahead · *, +, ? — the empty match again.