Skip to content

An Enumerator has a cursor: next, peek, rewind

Level: 201 · a Python programmer who knows iter() and next() and wants the Ruby spelling, plus the two things Ruby adds

One line: Any each called without a block returns an Enumerator; next pulls one value from it, peek looks at the next one without taking it, StopIteration marks the end and carries the iteration's return value in result, rewind starts over and loop rescues StopIteration for you — where Python's iterators have next() and StopIteration only, no peek, no rewind, and one cursor that a for loop shares.

Internal iteration hands each element to a block and keeps control; external iteration gives control to the caller, who asks for elements one at a time — which is what you need to walk two collections in step (the kata merges two sorted arrays that way), to read a header before the body, or to hand a half-consumed sequence to someone else. Ruby keeps the two kinds apart: the cursor that next moves belongs to the Enumerator object, and the same object's to_a, map or each start from the beginning without touching it, because next runs the underlying each on a fiber that is suspended between calls — the enumerators run on fibers page measures that; here it is only the reason (Not machine-checked here.).

Verified output of external_enumerators_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

next, peek, and the end
 1. e = [1, 2].each; e.class                                       Enumerator
 2. e.next                                                         1
 3. e.peek  (looks, does not advance)                              2
 4. e.next                                                         2
 5. e.next at the end raises: class, message, ex.result            [StopIteration, "iteration reached an end", [1, 2]]
 6. e.next again: still exhausted                                  StopIteration
 7. e.rewind; e.next                                               1
 8. [1, 2].each.size, [1, 2].cycle.size, Enumerator.new {}.size    [2, Infinity, nil]

loop rescues StopIteration silently, and its value is the result
 9. seen; loop { e2.next }.equal?(arr)  (each returned arr)        [[10, 20, 30], true]

The external cursor is separate from internal iteration
10. e3.next; e3.to_a; e3.next  (to_a starts over, next does not)   [10, [10, 20, 30], 20]
11. [1, 2].each.with_index(1).to_a                                 [[1, 1], [2, 2]]

Enumerators built by hand, and what they leave in result
12. Enumerator.new { y << 1; y.yield 2; :done }: next x2, result   [[1, 2], :done]
13. numbers (to_enum): loop { em.next } is the method's value      :the_return
14. a raise inside the block comes out of next                     ["a", "RuntimeError: boom"]
15. Enumerator.produce(1) { |x| x * 2 }.take(5)                    [1, 2, 4, 8, 16]
16. 3.times.next, "abc".each_char.next, {a: 1}.each.next           [0, "a", [:a, 1]]
17. StopIteration.ancestors.take(3)  (a bare rescue catches it)    [StopIteration, IndexError, StandardError]
18. e2.peek on the e2 that loop drained                            StopIteration

Reading the output

Rows 1–8, the cursor: each with no block returns an Enumerator; next gives 1; peek shows 2 and leaves it there, so the following next gives 2 too. At the end next raises StopIteration with the message iteration reached an end, and the exception carries result — the value the underlying each returned, which for an Array is the Array itself (row 5). An exhausted enumerator stays exhausted (row 6) until rewind (row 7). Row 8 is size, answered without iterating: 2 for an array's each, Infinity for cycle, and nil for a hand-built Enumerator.new that was given no size.

Row 9 is the idiom that makes external iteration pleasant: loop rescues StopIteration silently, and its own value is the exception's result, so loop { e.next } walks an enumerator to the end and returns what each returned (equal?(arr) is true). Rows 10–11 are the separation of cursors: after one next, to_a returns all three elements from the start, and the next next still returns the second element — internal iteration never moved the cursor. with_index(1) chains onto any enumerator.

Rows 12–18, enumerators built by hand: Enumerator.new gets a yielder, y << v and y.yield v both hand a value out, and the block's own value becomes the result (row 12: :done). A method turned into an enumerator with to_enum (row 13) behaves the same — loop { em.next } returns the method's return value. An exception raised inside the block comes out of the next that ran it (row 14). Enumerator.produce makes an endless enumerator from a seed and a step (row 15), and every blockless iterator method is an enumerator source — 3.times, each_char, Hash#each (row 16). Row 17 places StopIteration in the hierarchy: under IndexError, so a bare rescue catches it, which is what loop does. Row 18: peek on an exhausted enumerator raises the same way next does.

Compared with Python

iter(x) and next(it) are the same protocol, and the twin's rows line up until Ruby adds something. There is no peek (row 3): the standard-library answers are itertools.tee or a one-item buffer, and the third-party more-itertools package's peekable (Not machine-checked here.). There is no rewind (row 7): iter(it) is it, an iterator's __iter__ returns itself, so the only way back to the start is a fresh iter(x). StopIteration.value is None for a list iterator (row 5) but holds a generator's return value (row 12) — and yield from captures it inside another generator (row 13), where a for loop or list() throws it away. next(it, default) is the Python spelling for "or stop", and operator.length_hint is the polite question about size: 2 for a list iterator, 0 for count(), a hint rather than a size, while len refuses (row 8).

The one real difference is the cursor. A for loop catches StopIteration silently like Ruby's loop (row 9, though as a statement it has no value to return), but it drives the same cursor: in row 10, list(it3) after one next returns only the two remaining elements and leaves the iterator exhausted, where Ruby's to_a started over and left the cursor alone. In Python an iterator is a position; in Ruby an Enumerator is a recipe with a position attached. produce is a three-line generator (row 15); every container gives an iterator through iter() (row 16); StopIteration derives from Exception (row 17), so except Exception catches it — and, by PEP 479, a StopIteration raised inside a generator body is turned into RuntimeError rather than ending the generator (Not machine-checked here.).

Verified output of external_enumerators_py.py — regenerated by tools/run_examples.py, never hand-typed.

next, and the end
 1. it = iter([1, 2]); type(it).__name__                           'list_iterator'
 2. next(it)                                                       1
 3. hasattr(it, "peek")  (no peek: see the prose)                  False
 4. next(it)                                                       2
 5. next(it) at the end raises: class, ex.value                    ['StopIteration', None]
 6. next(it) again: still exhausted; next(it, "default")           ['StopIteration', 'default']
 7. hasattr(it, "rewind"), iter(it) is it, next(iter([1, 2]))      [False, True, 1]
 8. length_hint(iter([1, 2])), length_hint(count()), len raises    [2, 0, 'TypeError']

for catches StopIteration silently, and a statement has no value
 9. seen; (a for loop has no value to compare)                     [[10, 20, 30], None]

There is one cursor: list() and for continue from where next left it
10. next(it3); list(it3); next(it3, "exhausted")                   [10, [20, 30], 'exhausted']
11. list(enumerate(iter([1, 2]), 1))                               [(1, 1), (2, 2)]

Generators built by hand, and what they leave in StopIteration.value
12. yield 1; yield 2; return 'done': next x2, ex.value             [[1, 2], 'done']
13. for/list drop the return value; yield from captures it         [[1, 2], [1, 2, 'the_return']]
14. a raise inside the generator comes out of next                 ['a', 'RuntimeError: boom']
15. list(islice(produce(1, lambda x: x * 2), 5))                   [1, 2, 4, 8, 16]
16. next(iter(x)) for range(3), "abc", {"a": 1}.items()            [0, 'a', ('a', 1)]
17. StopIteration.__mro__ names  (except Exception catches it)     ['StopIteration', 'Exception', 'BaseException']
18. next(it2, "exhausted") on the it2 that for drained             'exhausted'

The Python docs describe the protocol under Iterator Types ↗; the Ruby side is the Enumerator ↗ class page.

Try it

  1. Write merge(a, b), which merges two sorted arrays into one sorted array using only each enumerators, peek, next and loop — no indexes — and interleave(a, b), which alternates elements from two arrays and, when the shorter runs out, appends the rest of the longer. Then show the trap: call next once on an enumerator, then to_a, then next again.
  2. File.foreach(path) without a block returns an Enumerator over a file's lines. Use next to read a two-line header, then hand the enumerator to a method that eaches the body. Does the body start at line one or line three? Try rewind in between and explain what it did to the file.
Solution to 1

external_enumerators_kata_rb.rb in full — pasted here by tools/run_examples.py from the file CI runs.

# Kata: merge two sorted arrays, and interleave two arrays of different lengths,
# with nothing but external enumerators: next, peek and StopIteration.
def row(n, label, value) = puts(format("%2d. %-44s %s", n, label, value.inspect))

def merge(a, b)
  ea, eb = a.each, b.each
  out = []
  loop { out << (ea.peek <= eb.peek ? ea.next : eb.next) }   # ends when either peek raises StopIteration
  loop { out << ea.next }                                    # drain whichever side is left...
  loop { out << eb.next }                                    # ...the other loop ends at once
  out
end

def interleave(a, b)
  ea, eb = a.each, b.each
  out = []
  loop do
    out << ea.next
    out << eb.next
  end
  loop { out << ea.next }
  loop { out << eb.next }
  out
end

row 1, "merge([1, 4, 9], [2, 3, 10, 11])", merge([1, 4, 9], [2, 3, 10, 11])
row 2, "merge([], [5, 6])", merge([], [5, 6])
row 3, "interleave(%w[a b c], [1, 2])", interleave(%w[a b c], [1, 2])
row 4, "interleave([1], %w[x y z])", interleave([1], %w[x y z])
e = [1, 2, 3].each
e.next
row 5, "the trap: to_a after next starts over", [e.to_a, e.next]

Verified output of external_enumerators_kata_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

 1. merge([1, 4, 9], [2, 3, 10, 11])             [1, 2, 3, 4, 9, 10, 11]
 2. merge([], [5, 6])                            [5, 6]
 3. interleave(%w[a b c], [1, 2])                ["a", 1, "b", 2, "c"]
 4. interleave([1], %w[x y z])                   [1, "x", "y", "z"]
 5. the trap: to_a after next starts over        [[1, 2, 3], 2]

See also