Skip to content

Enumerator#next runs the block on its own Fiber

Level: 301 · anyone who has called next on an enumerator and wondered how Ruby paused the block in the middle

One line: each, to_a, map and first run an Enumerator's block where they are called, on the calling fiber; next and peek run the same block on a private fiber that is resumed one value at a time — which is why next costs a fiber switch, why rewind starts the block over, and why a Thread.current[] value set outside is invisible inside it.

Enumerator.new { |y| … } takes a block and hands it a Yielder; every y << v or y.yield v is one value produced. How that block runs depends on who asks. Internal iteration — each, to_a, map, first, take, lazy — calls the block right there, so Fiber.current inside it is the root fiber (rows 1 and 11). External iteration — next, peek — has to stop the block after each value and come back later, and Ruby does that with a Fiber: the first next creates one and resumes it, y << v becomes a Fiber.yield, and the fiber sits suspended between calls (rows 2 and 3). When the block returns, next raises StopIteration whose result is the block's return value (rows 4 and 5), and keeps raising it (row 6); rewind throws the private fiber away so the next next starts the block from the top (row 7). peek resumes the fiber once and keeps the value for the following next (row 8), and loop rescues StopIteration for you, returning its result (row 9) — which for [10, 20].each is the array each returns.

Row 10 is a detail worth one look: inside the block, y.yield(v) evaluates to whatever the block given to each returned; under map, which keeps that value for its result array, and under next, which has no block at all, it is nil. The other rows fix the rest of the picture: size is nil unless the enumerator was told its size (row 12), the private fiber runs on the calling thread (row 13), and the Yielder has three methods of its own (row 14). The cost is the switch: each next is a fiber resume and a fiber yield, so external iteration is slower than each by a constant factor, and Enumerator::Lazy — which chains blocks without fibers — is the tool when you want laziness rather than a hand-driven cursor (Not machine-checked here.). The previous page showed the Fiber itself; the next shows the trap this design sets for Thread.current[].

Verified output of enumerators_run_on_fibers_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

 1. to_a (internal): the block runs on the root fiber?   [true, true]
 2. next (external): on the root fiber?                  false
 3. next again: the same private fiber, resumed          false
 4. next past the end                                    StopIteration: iteration reached an end
 5. StopIteration#result is the block's return value     :finished
 6. next once more                                       StopIteration
 7. rewind, then next starts the block over              false
 8. peek / next / next on [10, 20].each                  10 / 10 / 20
 9. loop rescues StopIteration and returns its result    [10, 20]
10. Yielder#yield returns: under each / map / next       [1, "yield returned 10"] / [10, "yield returned nil"] / [1, "yield returned nil"]
11. first / take(1) / lazy.first(1): on the root fiber?  true / [true] / [true]
12. size: Enumerator.new { } / Enumerator.new(3) { }     nil / 3
13. under next, the block is on the calling thread       true
14. Enumerator::Yielder's own methods                    [:<<, :to_proc, :yield]

Reading the output

Row 1 prints [true, true]: driven by to_a, both values were produced on the root fiber. Rows 2 and 3 print false twice: driven by next, the block ran on another fiber, and the second next resumed that same fiber rather than starting a new one — the block did not run from the top again, it continued from its first y <<. Rows 4 to 6 are the end: StopIteration: iteration reached an end, with result equal to the block's last expression :finished, and every later next raises again. Row 7 prints false after rewind: a fresh private fiber, the block started over.

Rows 8 and 9 use the enumerator every array already has. peek shows 10 without consuming it; the two next calls give 10 and 20; loop { ext.next } ends quietly when next raises and returns the result, [10, 20]. Row 10 shows y.yield(1) returning 10 under each — the block's v * 10 — while map collected the 10 into its result and handed nil back to the yielder, and next handed back nil too. Row 11 confirms that first, take and lazy.first are internal: all on the root fiber. Row 12 is nil / 3, row 13 true, and row 14 lists [:<<, :to_proc, :yield].

Compared with Python

Python has no internal/external split because it has no each: a generator is always driven from outside, one next() at a time, and list() is just a loop of next() calls. There is also no fiber. The generator's block is a frame that the interpreter suspends and resumes on the calling thread — rows 1, 2, 3, 11 and 13 all print True for "on the calling thread", and row 3 adds gi_suspended True between calls. So the closest Python has to Ruby's row 2 is the observation that nothing changes: the same thread, the same frame, no switch to pay for.

The end of the frame matches Ruby's protocol exactly: next() raises StopIteration, the generator's return value rides on StopIteration.value (rows 4 and 5), and later next() calls raise again with None (row 6). What Python lacks is rewind — a finished generator cannot be restarted; call the generator function again (row 7) — and peek, which needs a wrapper or itertools.tee (row 8). for swallows StopIteration the way loop does, but has no value; next(it, default) is the idiom for a quiet end (row 9). Row 10 is Ruby's row 10 from the other side: the yield expression evaluates to what send() passed, None under plain next() — the consumer decides, as each's block does in Ruby. There is no size at all — len() is a TypeError and operator.length_hint says 0 (row 12) — and the generator object exposes its frame directly as gi_frame, along with gi_code, gi_running, gi_suspended and gi_yieldfrom (row 14).

Verified output of enumerators_run_on_fibers_py.py — regenerated by tools/run_examples.py, never hand-typed.

 1. list() (all at once): on the calling thread?         [True, True]
 2. next() (one at a time): on the calling thread?       True
 3. next() again: the same frame, resumed                True (gi_suspended True)
 4. next() past the end                                  StopIteration (state GEN_CLOSED)
 5. StopIteration.value is the return value              'finished'
 6. next() once more                                     StopIteration (value None)
 7. no rewind: gen() again, then next()                  True
 8. (no peek) / next / next on iter([10, 20])            (no peek) / 10 / 20
 9. for swallows StopIteration; next(it, default)        'the default'
10. the yield expression: under list() / after send(10)  [1, 'yield returned None'] / [1, 'yield returned 10']
11. next(gen()) / islice / list: on the calling thread?  True / [True] / [True]
12. size: len(gen()) / operator.length_hint(gen())       TypeError / 0
13. under next(), the frame is on the calling thread     True
14. a generator object's gi_ attributes                  ['gi_code', 'gi_frame', 'gi_running', 'gi_suspended', 'gi_yieldfrom']

Try it

  1. Build an Enumerator over the words of a sentence and read it from the outside with next and peek, printing each word with the one that follows it. Let loop end the iteration, but rescue the StopIteration that peek raises on the last word yourself, and print what next raises after the loop.
  2. Write an Enumerator whose block records Fiber.current at each y <<, then compare what to_a and three next calls record: how many distinct fibers appear in each case, and what does rewind add? Print counts, not fibers.
Solution to 1

enumerators_run_on_fibers_kata_rb.rb in full — pasted here by tools/run_examples.py from the file CI runs.

# Kata: an Enumerator over the words of a sentence, read from the outside with
# next and peek so that each word is printed with the one that follows it.
# The StopIteration that ends the loop is the one `loop` rescues; the one
# from peek is rescued by hand.
words = Enumerator.new do |y|
  "the quick brown fox".split.each { |w| y << w }
end

loop do
  word = words.next
  following = begin
    words.peek
  rescue StopIteration
    "(none)"
  end
  puts "#{word.ljust(6)} -> next: #{following}"
end
puts "after the loop: next raises #{(words.next rescue $!.class)}"

Verified output of enumerators_run_on_fibers_kata_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

the    -> next: quick
quick  -> next: brown
brown  -> next: fox
fox    -> next: (none)
after the loop: next raises StopIteration

See also