Python Good Practices Part 3 – Writing faster code

There are many ways to write code in order to achieve same results. This article will show you how different approaches affect the code execution in the scope of time duration. This should give you a hint on what choices should be made but mostly to encourage you to always consider time execution as a substantial matter.

For comparison purposes, timeit module is used. Usually, to achieve best results the input will be from 0 to 1,000,000 with the step  20000. Every step is executed 10 times. To get the real average time of code execution we should get the time from y axis for given number of elements from x axis and divide it by 10.  If those input data differs, it will be noted.

If applicable, every comparison will be tested for worst scenario.

1. List initialization

There are two ways that allows you to initialize a collection like list or dictionary in python.

empty_list = []
empty_list = list()

The first way uses literal whilst the second one uses function invocation and symbol lookup. It may seems that there are no difference between them, however if you take a closer look you will notice a difference in the scope of time execution.

To measure performance of those two lists initialization methods, the below code is going to be considered:

# []
for _ in range(1000000):
    list_ = []

# list()
for _ in range(1000000):
    list_ = list()

Performance plot has been presented below:

Above graph shows that using literal is around 3 times faster. This is because Python is able to build a list using literal with a single bytecode instruction while using list() requires more operations such us symbol lookup and function call. The module ‘dis’ allows as to take a peek at generated bytecode.

dis.dis('[]')

  1           0 BUILD_LIST               0
              2 RETURN_VALUE

dis.dis('list()')

  1           0 LOAD_NAME                0 (list)
              2 CALL_FUNCTION            0
              4 RETURN_VALUE

2. List comprehension

If you would like to create a collection from another one, you could loop over every element from the base collection. However, Python provides a construct named comprehension that helps us to do it in a more concise way.

These two approaches will be used to create a list with elements multiplied by two from another list.

# Setup
numbers_list = [number for number in range(0, 1000000)]

# For loop
multiplied = []
for number in numbers_list:
    multiplied.append(number*2)
    
# List comprehension
multiplied = [number*2 for number in numbers_list]

As we can see, the list comprehension is faster. The reason is that it does not need to invoke append() method on every iteration.

3. Reversing list

In Python there are multiple ways to reverse the list. You can use a reverse() method of a list object or a built-in function – reversed(). The first option reverses the list in place mutating it whilst the second option returns iterator for the reversed sequence (if we want to have a list as a final result we need to cast the reversed() function to list).

Another option to reverse a list that the comparison will include is by list slicing i.e. [::-1].

The below code snippets are going to be benchmarked:

# Setup
from random import random
random_list = [random() for _ in range(1000000)]

# list.reverse() method
random_list.reverse()

# [::-1] 
reversed_list = random_list[::-1]

# list(reversed())
reversed_list = list(reversed(random_numbers))

As we can see in the graph above, the fastest is the usage of reverse() method of list object and the list(reversed()) options is the slowest one. But the choose should depend on what we want to achieve.

If we need to mutate a list or we do not care about it, we should choose reverse() method. If we do not want to mutate a list, we should choose [::-1] slicing operator. But if we do not want to mutate a list and we just want to iterate over reversed sequence, we should choose reversed() function and use it directly in a loop as it returns an iterator.

4. Sorting lists

Similarly to lists reversal there are two options to sort them out:

– sort() method of a list object that mutates it,

– built-in sorted() function that returns new, sorted list.

Let’s compare these two options on a collection with random numbers:

# Setup
from random import random
random_list = [random() for _ in range(1000000)]

# list.sort() method
random_list.sort()

# sorted() function
sorted_list = sorted(random_list)

You can see benchmark results on the below graph:

Here we can see that sorted() function is slower than sort() method. The difference is significant for vast number of elements. To sum up we should choose list.sort() over sorted() solution, if we do not need to have the original list preserved.

5. String concatenation

String concatenation is very common procedure in programming. Having collection of strings, we can concatenate them in different ways in Python.  Two of them will be compared in the scope of time execution:

• using += operator
• string’s join() method

# += operator
strings = ''
for i in range(1000000):
    strings += 'a'  

# join()
strings = ''.join('a' for i in range(1000000))

List method join() is much faster than += operator. The reason is that strings are immutable and using += requires to load bigger and bigger strings on every iteration into a memory buffer. And as we can see in the graph, the function for += rises exponentially.

6. Checking value in if statement

Very often we want to check whether some value equals to True, is not empty, is not null or is a non-zero value. In Python we can check this in three different ways i.e.:

if element:
    pass
if element == True:
    pass
if element is True:
    pass

At first let’s create a list of elements to check with random values (True or False):

# Setup
import random
random_list = [bool(random.getrandbits(1)) for _ in range(1000000)]

Then we are going to check those values in three different ways that were mentioned before.  It applies only in a situation when we want to check whether the value is different than the default value of a given type (not None for any object, non-zero value for int, True for Boolean, non-empty collection etc.).

Comparing these three if statements, we notice that the implicit check is the fastest. The ‘is’ keyword is the second in our podium, but the ‘==’ operator is the slowest one. If we do not want to make a check for a specific value but the default one, we should choose the implicit check.

7. Looping over generator and list comprehension

If we are working with the vast amount of elements in collections, we need to consider the memory usage. Python provides generators and generator expressions, which help us to save memory space. But what about time execution? To answer this question, let’s compare iterating over a list and generator.

Let’s implement a function to get a list and generator to yield values:

def list_comprehension():
    return [i for i in range(1000000)]

def generator():
    return (i for i in range(1000000))

Now let’s iterate over them and compare the time execution:

# list comprehension
for element in list_comprehension():
    pass

# generator
for element in generator():
    pass

Generators help us to save memory usage, but as we can see iterating over them is slower than  lists. This happens because of the number of generator invocations. In this case, memory usage and time execution should be considered to choose the better option.

8. Checking membership of a list

Very often we want to check whether the collection contains specific value. To achieve this in Python, you can compare every element with the desired value or use the ‘in’ keyword.

Let’s use this methods to check whether a list contains its last value (the worst scenario):

numbers = [number for number in range(0, 1000000)]
last_value = len(numbers) – 1

# for loop
for number in numbers:
    if number == last_value:
        break

# 'in' keyword
if last_value in numbers:
    pass

Using ‘in’ keyword is faster and it should always be a preferred way of checking the membership of a collection. Additionally, using ‘in’ keyword is assumed to be more Pythonic.

9. Checking all elements in a list

In order to check elements in a list there are two useful built-in functions:

• all() – checks whether all elements in an iterable results to True
• any() – checks whether any of the elements in an iterable results to True

In this example we are going to compare time execution of all() function with a conventional way of checking every element of a collection in a loop. The base list consists of only True values (worst scenario).

true_list = [True] * 1000000

# for loop
for element in true_list:
    if not element:
        break

# all()
all(true_list)

As we can see,  the built-in function is faster than using for loop and it should be always preferred way to check whether all values in the collection results to True.

10. Checking any element in a list

In this example we are going to make an analogical comparison for any() built-in function. The list we are going to check contains only False elements.

false_list = [False] * 1000000

# for loop
for element in false_list:
    if element:
        break

# any()
any(false_list)

As we could predict, any() function is as better as all() function while comparing it with ordinary ‘for loop’ solution.

11. Updating dictionary values with key existence check

Sometimes we need to create dictionary without the knowledge whether the existence of the key we want to update the value for. We can do it in multiple ways.

Let’s create a list of random numbers:

from random import randint
numbers = [randint(1, 500000) for _ in range(1000000)]

The dictionaries mapping elements from a list to the number of their occurrence will be created in 3 different ways:

• By checking the occurrence of element on every iteration
• By using dict.get() method
• By using defaultdict from collections module

# Checking keys on every iteration
number_ocurrences = {}
for number in numbers:
    if number not in number_ocurrences:
        number_ocurrences[number] = 0
    number_ocurrences[number] += 1

# Using dict.get() method
number_ocurrences = {}
for number in numbers:
    number_ocurrences[number] = number_ocurrences.get(number, 0) + 1

# Using defaultdict 
from collections import defaultdict
number_ocurrences = defaultdict(int)
for number in numbers:
    number_ ocurrences[number] += 1
number_ocurrences = dict(number_ocurrences)

In the plot we see that there is no best solution in the scope of time execution with the given data. Every solution has similar time execution for dictionary that we know that would have at most 500,000 elements. But what in case if we knew that there would be at most only 1,000 keys? Let’s compare it:

 from random import randint
numbers = [randint(1, 1000) for _ in range(1000000)]

In case of smaller collection we see that defaultdict is the fastest in execution, but using dict.get() method is the slowest. In conclusion, one solution can be better in one case while another is better in a different situation. The choice of algorithm should be driven by the experience and the knowledge of the input data.

Summary

In this article the comparison of the code snippets that are doing exactly the same was compared in the scope of time execution. The differences for small number of executions or number of elements might be considered irrelevant but as the input increases the time execution difference grows as well. It is always a good practice to write code which can be executed as fast as possible. Especially when it comes to big commercial applications, where the time execution is substantial.