Skip to content

An iterator is used up

Level: 201 · for Python programmers, especially ones coming from ABAP or C

One line: An iterator is a position in a walk, not the collection being walked, and once it reaches the end it stays there — so sum(it) twice gives 6 and then 0, zip, map and a generator expression each work exactly once, and 4 in evens quietly moves the position past 4.

nums = [1, 2, 3]
it = iter(nums)
sum(it)         # 6
sum(it)         # 0
sum(nums)       # 6

Python has two protocols where most languages have one. An iterable is anything iter() accepts, a list, a string, a dict, a file, and it can hand out as many iterators as it is asked for, each starting at the beginning. An iterator is the object next() advances, and asked for an iterator it hands out itself. A for loop calls iter() once and then next() until StopIteration. So a loop over a list starts at the front every time, and a loop over an iterator starts wherever the iterator is, which after one full loop is the end.

What makes this a page rather than a footnote is that the end is silent. An exhausted iterator is not an error; it is an empty sequence, so a second for runs zero times, a second sum is 0 and a second list() is [], and nothing says why. The bug shows up as missing output, somewhere downstream of the loop that consumed the data.

The run

Verified output of an_iterator_is_used_up_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. A LIST CAN BE WALKED TWICE; ITS ITERATOR CANNOT
     nums = [1, 2, 3]
     sum(nums), sum(nums)                   (6, 6)                     a list starts every walk at the beginning
     it = iter(nums)
     sum(it), sum(it)                       (6, 0)                     the second sum found the iterator at its end
     it = iter(nums)
     sum(it)                                6
     max(it)                                ValueError                 not a complaint about iterators: max of nothing
     len(iter(nums))                        TypeError                  no length: the only way to count is to consume

2. WHAT for DOES: iter() ONCE, THEN next() UNTIL StopIteration
     it = iter([10, 20])
     next(it)                               10
     next(it)                               20
     next(it)                               StopIteration              the signal that the walk is over
     next(it, 'done')                       'done'                     a default instead of the exception
     iter(it) is it                         True                       an iterator's iter() is itself, so for cannot rewind it
     iter(nums) is iter(nums)               False                      a list hands out a fresh iterator every time

3. WHICH BUILT-INS HAND YOU AN ITERATOR
     pairs = zip('ab', [1, 2])
     list(pairs)                            [('a', 1), ('b', 2)]
     list(pairs)                            []                         zip is an iterator: used up
     squares = map(lambda n: n * n, [1, 2, 3])
     list(squares), list(squares)           ([1, 4, 9], [])            so is map, and filter, reversed, enumerate
     evens = (n for n in range(10) if n % 2 == 0)
     4 in evens                             True                       in walked the generator as far as 4
     list(evens)                            [6, 8]                     and left it there
     r = range(3)
     list(r), list(r)                       ([0, 1, 2], [0, 1, 2])     range is a sequence, not an iterator
     d = {'a': 1}
     items = d.items()
     list(items), list(items)               ([('a', 1)], [('a', 1)])   a dict view is a view, not an iterator

4. HALF-CONSUMED BY A SHORT-CIRCUIT
     it = iter([1, 2, 3, 4])
     any(n > 1 for n in it)                 True                       any() stopped at 2
     list(it)                               [3, 4]                     the rest is still there, for whoever asks next

5. A GENERATOR RUNS ONLY WHEN ASKED, AND ONLY ONCE
     def gen():
         print('     (gen: started)')
         yield 1
         print('     (gen: resumed)')
         yield 2
         print('     (gen: finished)')
     g = gen()
     (gen: started)
     next(g)                                1                          nothing ran until this call
     (gen: resumed)
     next(g)                                2
     (gen: finished)
     next(g)                                StopIteration              the body ran to its end, then signalled
     list(g)                                []                         spent; gen() again is the only way back

6. A FILE IS AN ITERATOR TOO
     f = io.StringIO('a\nb\n')
     len(f.readlines())                     2
     len(f.readlines())                     0                          at the end, like any iterator
     f.seek(0)
     len(f.readlines())                     2                          a file has seek(); an iterator in general does not

7. WALKING TWICE ON PURPOSE
     it = iter([1, 2, 3])
     a, b = itertools.tee(it)
     list(a), list(b)                       ([1, 2, 3], [1, 2, 3])     tee buffers what one side has seen and the other has not
     list(it)                               []                         and the original is spent either way
     saved = list(iter([1, 2, 3]))
     sum(saved), sum(saved)                 (6, 6)                     or keep a list, which is what tee is doing quietly

What the run shows

Section 1: a list can be walked twice; its iterator cannot. sum(nums) is 6 both times; sum(it) is 6 and then 0. The max(it) row is the shape the bug takes in real code: the exception is ValueError, and its message complains about an empty sequence, which is true and unhelpful, because the sequence was not empty until sum walked it. len(iter(nums)) raises TypeError: an iterator has no length, so counting it and using it are the same act.

Section 2: what for does. next(it) gives 10, then 20, then raises StopIteration, which is the protocol's only way of saying the walk is over. A second argument to next substitutes a default for the exception. iter(it) is it is True, and that identity is the whole reason a for loop cannot rewind an iterator: when the loop calls iter() on it, it gets the same object back, at the same position. iter(nums) is iter(nums) is False, because a list builds a fresh iterator on every request.

Section 3: which built-ins hand you an iterator. zip, map, filter, reversed and enumerate return iterators, so each lists once and then lists []. A generator expression is one too, and in walks it: 4 in evens is True, and the walk stopped at 4, so list(evens) is [6, 8]. The two rows at the end are the false friends. range is a sequence, and list(r) twice gives the same three numbers; a dict's items() is a view, which is re-iterable and follows the dict's later changes. Neither is an iterator, and both are safe to loop over twice.

Section 4: half-consumed by a short-circuit. any(n > 1 for n in it) stopped as soon as it saw 2, and list(it) shows what was left: [3, 4]. The same is true of all, of in, and of a for loop with a break: the iterator is left exactly where the consumer stopped, and the next consumer starts there.

Section 5: a generator runs only when asked, and only once. g = gen() printed nothing: calling a generator function runs none of its body. The first next(g) ran it as far as the first yield and printed (gen: started); the second resumed it; the third ran it to the end, printed (gen: finished), and raised StopIteration. list(g) is [], and the only way to run the body again is to call gen() again, which makes a new generator. A generator is an iterator that writes its next() as a function body, and it is used up like any other.

Section 6: a file is an iterator too. readlines() reads two lines the first time and none the second, because the position is at the end. A file is the one common iterator with a position you can move, seek(0), and that is a property of files, not of iterators: zip and map have nothing like it.

Section 7: walking twice on purpose. itertools.tee splits one iterator into two that can each be walked in full, by buffering whatever one side has seen and the other has not, and the original is spent either way. Keeping a list does the same job in plain sight, and for anything that fits in memory it is the clearer choice.

If you are coming from Rust

Rust has one Iterator trait, with a next() that returns Option<Item>, and it makes the whole of this page a compile-time fact. Consuming an iterator moves it, so a second it.sum() after the first is E0382, use of a moved value, rather than a zero. for x in v moves the vector too, and v is gone after the loop unless you wrote &v, which is the iter(nums) against nums distinction with the compiler enforcing it. The Rust library's Iterators are lazy ↗ is section 5 from that side, and iter, iter_mut and into_iter ↗ is the choice Python makes for you at iter().

If you are coming from C

C has no iterators. A walk is an index or a pointer, "used up" is an index equal to the length, and going back is assigning zero to it. Python's iterator hides the index and offers no assignment to it; the file's seek() is the one exception, and it exists because a file has an offset the operating system keeps. The instinct to reset and go again has nothing to reset.

If you are coming from ABAP

LOOP AT itab starts at the first row every time, and READ TABLE … INDEX goes to any row by number, so an internal table is a list in this page's sense: it can be walked as often as you like. The ABAP object that behaves like a Python iterator is a database cursor. After OPEN CURSOR the rows come out through FETCH NEXT CURSOR once each, a second pass needs the cursor opened again, and a SELECT … ENDSELECT loop is the same thing with the cursor hidden. Read zip, map and a generator as cursors over their inputs rather than as tables, and section 1 stops being a surprise. (Not machine-checked — CI cannot run ABAP.)

Try it

  1. Search a codebase you maintain for a zip(, map( or filter( result assigned to a name, then look for a second use of that name. Section 3 says the second use saw nothing.
  2. Pass one generator to two functions in a row. Add a print(len(list(arg))) to each and see which one printed 0.
  3. Open a CSV of yours and run csv.reader over the same file object twice. Count the rows each time, then add f.seek(0) between the two and count again.
  4. Call sorted() on an iterator you built, then list() on the same iterator. sorted had to walk it all, so section 1 applies.
  5. Put an any() or an in in front of a loop over the same iterator, as section 4 does, and count how many items the loop still sees.

See also