Closures and Decorators in Python

def keyword

⋆ The def keyword binds the body of a function to a name in such a way that functions are simply objects like everything else in Python.

⋆ def is executed at run time ⇒ ⚑ functions are created at run time

Methods

⋆ Functions defined within module scopes or inside classes are referred to as methods.

Local functions

⋆ Functions defined inside other functions are referred to as local functions since they are defined ⚑ local to a specific function’s scope.

⚑ Local functions are not members of enclosing functions, they are simply defined within the scope of the enclosing function.

def func(a, b):
  def local_func(x, y):
    return x * y
  return local_func(a, b)

In the above example, local_func is local to scope of func. So we can say local_func is a local function inside func.

☛ Each call to func results in a new definition of local_func. So, local_func is bound to a separate function body each time it’s called.☚

Let’s check it out:

store = []

def func(a, b):
  def local_func(x, y):# store the function instances in the store list
    store.append(local_func)
    print(local_func)
    print(x * y)
  return local_func(a, b)

func(2, 3)
func(2, 6)

This is the output:

<function func.<locals>.local_func at 0x0596DBB8>
6
<function func.<locals>.local_func at 0x0596DC00>
12

As we can infer from the output, the two function addresses are indeed different. Thus, a new function body is created for local_func on every call to func.

Scoping rules for local functions

☛ Local functions are subject to the same scoping rules as other functions:

★ L — Local

★ E — Enclosing

★ G — Global

★ B — Built-in

✔ What are local functions used for?

♥ Useful for specialized, one-off functions

♥ Help in code organization and readability

♥ Similar to lambdas, but more general as they may contain multiple expressions and may contain statements unlike lambda expressions

Returning functions from functions

In the following example, we are returning a function from an enclosing function which we then bind to a named variable and finally call like any other function.

def outer():
  def inner():
    print('inner')
  return inner

x = outer()
x()

Output is:

inner

Now, we can take down two important definitions:

★ First-class functions: Functions that can be treated like any other entity such as being passed as an argument, returned from another function or assigned to a variable.

★ Higher-order functions: Functions that take other functions as arguments or return other functions as their results.

Closures

There may be a situation where a local function uses variables from the enclosing scope. When the enclosing scope returns the local function, local function can still access those variables from the enclosing scope. This is where closures come into play.

⚑ A closure is a record storing a function and its environment (where it is created, i.e., the enclosing function’s scope).

Before we move further, let’s try to understand what exactly the term environment imply.

⚑ An environment is a mapping associating each free variable within the function to a value or a storage location to which the variable was bound to when the closure was created.

A closure, unlike a plain function, has access to those free variables even when called from outside their scope.

⚑ The local function closes over the variables it needs preventing them from being garbage-collected.

Let us look at an example:

def outer():
  x = 'closure1'
  y = 'closure2'
  def inner():
    print(x, y)
  return inner

local_func = outer()
local_func()

print(local_func.__closure__)

Here is the output:

closure1 closure2

(<cell at 0x02A5B3F0: str object at 0x02B2E2A0>, <cell at 0x02A5B470: str object at 0x02B3A520>)

The line print(local_func.__closure__) results in a set containing two objects. The first is x ⇒ ‘closure1’ and second is y ⇒ ‘closure2’.

Now that we know what closures are, let’s explore what a function factory is.

Function factory

☛ A function factory is a function that returns a new, specialized function which performs an operation with its own arguments as well as arguments passed to the function factory.

def outer(x):
  def inner(y):
    return(y + " " + x)
  return inner

# This line assigns local_func to inner
# but at the same time assigns argument x of outer to 'Manoj'
# Once outer returns inner, all the variables that inner refers
# to are stored in an environment
local_func = outer('Manoj')

print(local_func('Hello'))

print(local_func.__closure__)

Output is :

Hello Manoj
(<cell at 0x03DD2710: str object at 0x03D8B3E0>,)

The second line of output is a set with one element which is a string object with value ‘Manoj’ referenced by argument x of the outer function.

How to access global and enclosing scope variables inside the local function?

⚑ LEGB rules don't apply when making new name bindings. ☹

However, we can access and modify global and enclosing scope variables inside the local function with global and nonlocal keywords ☺and thus, avoid creating new variables with the same name as that of global and enclosing scope variables.

Function Decorators

Decorators are a way of modifying or enhancing existing functions in a non-intrusive and maintainable way.

On a simpler note, ⚑ decorators are implemented as callables that take a function (or a callable object) as argument and returns a function (or a callable object).

Following is a diagram showing how a decorator works:

Points to remember about a decorator:

★ Decorator replaces, enhances or modifies an existing function

★ The original function definition remains unchanged

★ Decorator uses the modified function’s original name

def escape_unicode(f):
  def wrap(*args, **kwargs):
    x = f(*args, **kwargs)
    return ascii(x)
  return wrap

@escape_unicode
def unic():
  return 'TromŒ'

print(unic())

The output is:

'Trom\u0152'

Thus, the escape_unicode function modifies the function and returns a new callable function wrap which is the new function body for unic.

Before we move on some of you might be wondering what are *args and **kwargs. Let me explain.

def args_func(*args):
   print(args)

print(args_func(['a', 'b']))         # (['a', 'b'],)

# Iterates through the elements 
# of the list
print(args_func(*['a', 'b']))          # ('a', 'b')

def kwargs_func(**kwargs):
   print(kwargs)

print(kwargs_func(a=1, b=2))          # {'a': 1, 'b': 2}

print(kwargs_func(**{'a': 1, 'b': 2}))   # {'a': 1, 'b': 2}

args stores all the positional arguments passed to a function and stores dictionaries and key-value pairs.

Class Decorators

⋆ Classes can also be used as decorators.

⋆ Classes are callable and a new instance will be created when a class is called.

class CallCount:
  def __init__(self, f):
    self.f = f
    self.count = 0

  def __call__(self, *args, **kwargs):
    self.count += 1
    return self.f(*args, **kwargs)

@CallCount
def hello(name):
  print('Hello {}'.format(name))

In the above example, when hello is called say as hello(‘Michael’), then a new instance of CallCount will be created. The function object created by calling hello will be assigned to attribute f of CallCount. The instance will then be immediately returned which means hello will now refer to the returned CallCount instance.

hello('Manoj')    
Hello Manoj

hello('William')   
Hello William

hello.count
=> 2 

hello.f('Manoj')
Hello Manoj # This won't increment hello.count

Instance Decorators

☛ Instance decorators are instances called immediately with function passed as an argument to the __call__ function of the instance’s class and return a callable.

class Trace:

  def __init__(self):
    self.enabled = True

  def __call__(self, f):
    def wrap(*args, **kwargs):
      """
      A wrapper function
      """
      if self.enabled:
        print('Calling {}'.format(f))
      return f(*args, **kwargs)
    return wrap

tracer = Trace()

@tracer
def rotate_list(l):
  """
  Rotates a list
  """
  return l[1:] + [l[0]]

l = [1,2,3]

print(rotate_list(l))   

# Calling <function rotate_list at 0x7f86fe38a488>
# => [2, 3, 1]

tracer.enabled = False

print(rotate_list(l))

# => [2, 3, 1]

That is all about decorators. We can also use

☛ multiple decorators where the decorators are executed in the reverse order of application to the function.

So, we are done. Well, not yet. An important concept is still to discuss.

⚠ Decorators help us write modular code but at the same time, they change the metadata of the original function when a callable is returned from the original.

Don’t believe me? Check for yourself.

From the above example, when we print rotate_list.__name__, instead of getting rotate_list, we get wrap. Similarly, when we print rotate_list.__doc__, instead of “Rotates a list” we see “A wrapper function”. This is not good.

⚑ One way to solve this issue is to update __doc__ and __name__ of the wrap function. But there is a better way.

.wrap()

☛ .wraps() properly updates metadata on wrapped functions.

import functools

class Trace:
  def __init__(self):
    self.enabled = True

  def __call__(self, f):
    @functools.wraps(f)
    def wrap(*args, **kwargs):
      """
      Wrap the function
      """
      if self.enabled:
        print('Calling {}'.format(f))
      return f(*args, **kwargs)
    
    return wrap

tracer = Trace()

@tracer
def rotate_list(l):
  """
  Rotates a list
  """
  l = l[1:] + [l[0]]
  return l

Now rotate_list.__doc__ and rotate_list.__name__ would give the correct results.

Decorators for validating arguments

Decorators can be used for validating function arguments. Note here check_hero is a function which returns a decorator named validator.

def check_hero(name):
  def validator(f):
    def wrap(*args):
      if args[0] == name:
        raise ValueError(
          '{} is a villain'.format(name)
        )
      return f(*args)
    return wrap
  return validator  

@check_hero('Thanos')
def check(value):
  return '{} is a hero'.format(value)

check('Tony')  # Tony is a hero
check('Thanos') # Throws ValueError: Thanos is a villain

Thanks for reading.