16.advanced-functions.md

Advanced Function Usage

Let us continue exploring the related knowledge of defining and using functions. Through the previous lesson, we already know that functions have parameters and return values, and parameters can be any data type, and return values can also be any data type. Then there is a small question here: can we use a function as the parameter of another function, and can we use a function as the return value of another function? Let us first give the conclusion: functions in Python are first-class functions. So-called first-class functions mean that functions can be assigned to variables, functions can be used as the parameters of other functions, and functions can also be used as the return values of other functions. The usage of taking one function as the parameter or return value of another function is usually called a higher-order function.

Higher-Order Functions

Let us go back to an example we talked about before: design a function that receives any number of arguments and adds up the elements whose type is int or float. We make a small change to the previous code to make it more compact, as shown below.

def calc(*args, **kwargs):
    items = list(args) + list(kwargs.values())
    result = 0
    for item in items:
        if type(item) in (int, float):
            result += item
    return result

If we want the calc function above not only to do addition for many arguments, but also to do more binary operations or even custom binary operations, what should we do? The code above can only do addition because the function uses the += operator, and this makes the function coupled with addition. If we can remove this coupling, the universality and flexibility of the function will become better. The way to remove the coupling is to turn the + operator into a function call, and then design it as a parameter of the function.

def calc(init_value, op_func, *args, **kwargs):
    items = list(args) + list(kwargs.values())
    result = init_value
    for item in items:
        if type(item) in (int, float):
            result = op_func(result, item)
    return result

Please note that the function above adds two parameters. Here, init_value means the initial value of the operation, and op_func means a binary operation function. To call the changed function, let us first define functions that do addition and multiplication, as shown below.

def add(x, y):
    return x + y


def mul(x, y):
    return x * y

If we want to do addition, we can call the calc function in the following way.

print(calc(0, add, 1, 2, 3, 4, 5))  # 15

If we want to do multiplication, we can call the calc function in the following way.

print(calc(1, mul, 1, 2, 3, 4, 5))  # 120

The calc function above turns the operator into a function parameter, and in this way it removes the coupling with addition. This is a very smart and practical programming skill, but beginners may feel it is hard to understand, so it is worth reading carefully. One thing to note is that there is a clear difference between passing a function as a parameter to another function and directly calling a function. To call a function, we need parentheses after the function name. But when passing a function as a parameter, we only need the function name itself.

If we do not define add and mul in advance, we can also use the add and mul functions provided by the operator module in Python's standard library. They represent binary addition and multiplication operations, so we can use them directly, as shown below.

import operator

print(calc(0, operator.add, 1, 2, 3, 4, 5))  # 15
print(calc(1, operator.mul, 1, 2, 3, 4, 5))  # 120

Python also includes a number of built-in higher-order functions. The filter and map functions we mentioned earlier are higher-order functions. The former can filter elements in a sequence, and the latter can map elements in a sequence. For example, if we want to remove the odd numbers from a list of integers and square all the even numbers to obtain a new list, we can directly use these two functions to do so.

def is_even(num):
    """Check whether num is even"""
    return num % 2 == 0


def square(num):
    """Calculate the square"""
    return num ** 2


old_nums = [35, 12, 8, 99, 60, 52]
new_nums = list(map(square, filter(is_even, old_nums)))
print(new_nums)  # [144, 64, 3600, 2704]

Of course, we can also do the same thing with a list comprehension, and the list-comprehension way is simpler and more elegant.

old_nums = [35, 12, 8, 99, 60, 52]
new_nums = [num ** 2 for num in old_nums if num % 2 == 0]
print(new_nums)  # [144, 64, 3600, 2704]

Let us also discuss a built-in function, sorted. It can sort the elements of container data types such as lists and dictionaries. We talked about the sort method of the list type before. From the function side, sorted is not different from the sort method of a list, but it returns a sorted list object instead of directly modifying the original list. We call this function design without side effects, which means calling the function will not affect the state of the program or the external environment except for producing a return value. When using the sorted function to sort, we can customize the sorting rule in the form of a higher-order function.

old_strings = ['in', 'apple', 'zoo', 'waxberry', 'pear']
new_strings = sorted(old_strings)
print(new_strings)

But if we want to sort the list elements by the length of the strings instead of dictionary order, we can pass a parameter named key to the sorted function, and set the key parameter to the function len, which gets the length of a string. We talked about this function before. The code is shown below.

old_strings = ['in', 'apple', 'zoo', 'waxberry', 'pear']
new_strings = sorted(old_strings, key=len)
print(new_strings)  # ['in', 'zoo', 'pear', 'apple', 'waxberry']

Note: The sort method of the list type has the same key parameter. Interested readers can try it by themselves.

Lambda Functions

When using higher-order functions, if the function used as a parameter or return value is itself very simple, one line of code is enough to finish it, and there is no need to think about reusing the function, then we can use a lambda function. Lambda functions in Python are functions without names, so many people also call them anonymous functions. A lambda function can have only one line of code, and the result of the expression in that line is the return value of the anonymous function.

In the earlier code, the is_even and square functions we wrote both have only one line of code, so we can consider replacing them with lambda functions, as shown below.

old_nums = [35, 12, 8, 99, 60, 52]
new_nums = list(map(lambda x: x ** 2, filter(lambda x: x % 2 == 0, old_nums)))
print(new_nums)  # [144, 64, 3600, 2704]

From the code above, we can see that the keyword for defining a lambda function is lambda. After it come the function parameters. If there are many parameters, they are separated by commas. The part after the colon is the function body, and it is usually an expression. The result of the expression is the return value of the lambda function, so there is no need to write the return keyword.

As we said before, functions in Python are first-class functions, so a function can be directly assigned to a variable. After learning lambda functions, some functions we wrote before can now be written in one line. Look at the one-line factorial function and prime-checking function below and see whether you can understand them.

import functools
import operator

fac = lambda n: functools.reduce(operator.mul, range(2, n + 1), 1)
is_prime = lambda x: all(map(lambda f: x % f, range(2, int(x ** 0.5) + 1)))

print(fac(6))        # 720
print(is_prime(37))  # True

Tip 1: functools.reduce performs a reduction over a sequence. Together with filter and map, it forms a powerful trio for data processing: filtering, mapping, and reducing.

Tip 2: The lambda function above that checks whether a number is prime uses the range function to build the range from 2 to $\small{\sqrt{x}}$, and checks whether this range has a factor of x. The all function is also a built-in Python function. If all Boolean values in the sequence passed to it are True, all returns True; otherwise, it returns False.

Partial Functions

A partial function means fixing some parameters of a function and generating a new function, so that we do not need to pass the same parameters every time we call the function. In Python, we can use the partial function in the functools module to create partial functions.

For example, by default the int function can convert a string as a decimal integer. If we change its base parameter, we can define three new functions to convert binary, octal, and hexadecimal strings into integers, as shown below.

import functools

int2 = functools.partial(int, base=2)
int8 = functools.partial(int, base=8)
int16 = functools.partial(int, base=16)

print(int('1001'))    # 1001

print(int2('1001'))   # 9
print(int8('1001'))   # 513
print(int16('1001'))  # 4097

Did you notice that both the first parameter and the return value of the partial function are functions? It processes the function passed in and returns a new function. By building partial functions, we can combine real usage scenes and turn the original function into a new function that is easier to use. This is quite interesting.

Summary

Functions in Python are first-class functions. They can be assigned to variables, and they can also be used as the parameters and return values of other functions. This means we can use higher-order functions in Python. The concept of higher-order functions is not very friendly to beginners, but it brings flexibility to function design. If the function we want to define is very simple, has only one line of code, and does not need a function name for reuse, we can use a lambda function.