Skip to content

A closure captures the variable, not the value

Level: 201 · for Python programmers

One line: A function defined inside another holds a reference to the enclosing variable and reads it when it runs, not when it was made — so three lambdas built in one loop all return the loop's last value, and every fix works by giving each of them a variable of its own.

fs = [lambda: i for i in range(3)]
[f() for f in fs]           # [2, 2, 2]

fs = [lambda i=i: i for i in range(3)]
[f() for f in fs]           # [0, 1, 2]

When Python compiles a function it decides, for every name in the body, where that name lives: in the function's own locals, in an enclosing function, or in the module. It does not decide the value. A name that lives in an enclosing function is looked up through a cell, a small box the two functions share, and the lookup happens each time the inner function runs. So the inner function does not hold 2 or 0; it holds the box, and it reads whatever the box holds at the moment of the call.

A for loop does not make a new box per iteration. It rebinds one variable, so every lambda made inside it shares that one box, and after the loop the box holds the last value. That is the whole of the trap, and it is also why every fix on this page is a way of making a new box.

The run

Verified output of a_closure_captures_the_variable_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. THREE LAMBDAS, ONE VARIABLE
     fs = [lambda: i for i in range(3)]
     [f() for f in fs]                            [2, 2, 2]          all three read i when called, and i ended at 2

2. THE LOOKUP HAPPENS WHEN THE FUNCTION RUNS
     x = 1
     def f(): return x
     x = 2
     f()                                          2                  f reads x now, not when def ran
     del x
     f()                                          NameError          and if x is gone by then, it fails now

3. WHAT A CLOSURE HOLDS: A CELL, WHICH CAN BE SHARED
     def counter():
         n = 0
         def inc():
             nonlocal n
             n += 1
             return n
         def get():
             return n
         return inc, get
     inc, get = counter()
     inc(), inc(), get()                          (1, 2, 2)          get sees what inc did: one cell
     inc.__closure__[0] is get.__closure__[0]     True
     get.__closure__[0].cell_contents             2                  the cell holds the current value

4. THREE WAYS TO CAPTURE THE VALUE INSTEAD
     fs = [lambda i=i: i for i in range(3)]
     [f() for f in fs]                            [0, 1, 2]          a default is computed once, when the lambda is made
     fs = [functools.partial(lambda i: i, i) for i in range(3)]
     [f() for f in fs]                            [0, 1, 2]          partial stores the argument now
     def keep(i): return lambda: i
     fs = [keep(i) for i in range(3)]
     [f() for f in fs]                            [0, 1, 2]          each call to keep is a new scope, so a new cell

5. THE LOOP VARIABLE OUTLIVES THE LOOP
     for k in range(3): pass
     k                                            2                  a for loop makes no scope of its own
     squares = [k * k for k in range(10)]
     k                                            2                  a comprehension does: its k never leaked

6. AND ASSIGNING TO THE NAME INSIDE MAKES IT LOCAL
     total = 0
     def bump(): total += 1
     bump()                                       UnboundLocalError  the += made total local to bump, so the read finds nothing
     def bump():
         global total
         total += 1
     bump()
     total                                        1

What the run shows

Section 1: three lambdas, one variable. The comprehension binds i three times and makes three lambdas, each holding the same cell. By the time any of them is called, the cell holds 2, so all three return 2.

Section 2: the lookup happens when the function runs. f was defined while x was 1, and f() returns 2, because f reads the module's x at the call. After del x, the same call raises NameError: there was never a copy inside f to fall back on. A global name and an enclosing name follow the same late-binding rule; only the mechanism differs.

Section 3: what a closure holds. inc and get were made in one call to counter, so they share one cell for n: inc.__closure__[0] is get.__closure__[0] is True, get() reports what inc() did, and cell_contents shows the current value. nonlocal is what lets inc assign to the shared variable instead of making a local one, which is section 6.

Section 4: three ways to capture the value instead. lambda i=i: i uses the fact that a default is computed once, when the lambda is made, so each lambda gets its own i. functools.partial stores the argument at the moment partial is called. A factory function, keep, gives each lambda a fresh scope, and so a fresh cell, because every call to a function is a new scope. All three return [0, 1, 2], and the third is the one that reads best when the closure is more than a line.

Section 5: the loop variable outlives the loop. After for k in range(3), k is 2: a for statement makes no scope. A comprehension does, in Python 3, so the k inside [k * k for k in range(10)] never leaks, and the outer k is still 2. That scope is also why section 1's lambdas share a cell: the comprehension is one scope with one i, not three.

Section 6: assigning to the name inside makes it local. def bump(): total += 1 fails with UnboundLocalError. The compiler saw an assignment to total in the body, so total is local throughout the body, and the read that += starts with finds nothing. global total tells the compiler otherwise, and the second bump() changes the module's total. Assignment makes a name local is that rule on its own.

If you are coming from Rust

A Rust closure also captures variables, and the compiler chooses how: by shared reference, by mutable reference, or by move, whichever is the least it can get away with, and move forces the last. What it will not do is let the shape in section 1 compile: a closure that borrows the loop variable and outlives the iteration is a lifetime error, and one that moves it takes the value, which is section 4's behaviour by default. The Rust library's What a closure is ↗ and The move keyword ↗ are the two halves.

If you are coming from C

C has no closures. The pattern is a function pointer plus a void * context, and "the captured variable" is whatever you copied into the context struct at the time you filled it in. That is capture by value, done by hand, so a C programmer's instinct is section 4's answer, and section 1 is the surprise. The C library's Strings chapter ↗ is where its callbacks live.

If you are coming from ABAP

ABAP has no closures and no lambdas. The stand-in is an object: a class with the "captured" values as attributes set in the constructor, and a method for the body. Those attributes are copies, made once, so ABAP's version of this page is section 4 with nothing to fix. What transfers is section 6's lesson in reverse: a method that reads a class attribute and a method that assigns to it both name the same thing in ABAP, whereas in Python the assignment quietly changes which variable the name means. (Not machine-checked — CI cannot run ABAP.)

Try it

  1. Search a codebase you maintain for a lambda inside a for loop or a comprehension. For each one, list the names it reads that the loop rebinds. Section 1 says every one of them is read late.
  2. Find a callback registered in a loop, a GUI button, a signal handler, a functools.partial that someone already added, and check whether the value it needs was captured or will be looked up.
  3. Take a closure of yours and print f.__code__.co_freevars and [c.cell_contents for c in f.__closure__]. That is the list of boxes it holds and what they hold now.
  4. Turn one for loop in your code into a comprehension, then search below it for the loop variable. Section 5 says the code below just lost it.
  5. Make an UnboundLocalError on purpose, with a function that reads a module-level counter and then increments it, and decide which of global, nonlocal and returning the new value the real code should use.

See also