ZetCode

Pandas Vectorization and Performance

last modified November 17, 2026 Python 3.15

Data work often means performing the same arithmetic on thousands or millions of values. A tempting approach is to walk over the rows with a Python for loop, but that is surprisingly slow. This article explains why and shows the faster tools that Pandas provides.

Vectorization means operating on whole arrays at once instead of one element at a time: an expression such as df['a'] * df['b'] is handed to compiled code that runs a tight loop over every value. We also cover eval, query and itertuples, and finish with a real timing comparison.

Why row-wise loops are slow

On each pass through a Python loop the interpreter performs bookkeeping: it looks up names, checks types and dispatches operations again for every single value. A vectorised expression does that work once and then lets compiled code process the whole column. The result is a new Series whose values line up with the input rows by position, which we can store straight back into the DataFrame as a new column.

vectorized_arithmetic.py
import pandas as pd

df = pd.DataFrame({'price': [10, 20, 30, 40], 'qty': [1, 2, 3, 4]})
df['total'] = df['price'] * df['qty']
print(df)

The multiplication is applied to both entire columns, producing 10, 40, 90 and 160 in one step. No explicit loop appears in our code, yet all four rows are processed. This is the basic pattern to reach for whenever values from columns need to be combined.

$ python vectorized_arithmetic.py
   price  qty  total
0     10    1     10
1     20    2     40
2     30    3     90
3     40    4    160

Conditional logic with np.where

Often a value depends on a condition. NumPy's np.where is the vectorised equivalent of an if-else expression: it checks a condition for every element and picks the first value where the condition is true and the second where it is false. The condition here is a whole Boolean Series, not a single Boolean, so the choice is made for all rows at once.

np_where.py
import numpy as np
import pandas as pd

df = pd.DataFrame({'price': [10, 25, 40, 15]})
df['label'] = np.where(df['price'] >= 25, 'high', 'low')
print(df)

Rows with a price of 25 or more are labelled high and the rest low. The Boolean Series df['price'] >= 25 is evaluated for the whole column, and np.where assigns a label to each row based on it.

$ python np_where.py
   price label
0     10   low
1     25  high
2     40  high
3     15   low

Computing with eval

The eval method evaluates an arithmetic expression written as a string. This is useful for long formulas because it keeps the expression readable and can avoid creating temporary Series for intermediate results. The column names in the string refer to columns of the DataFrame.

df_eval.py
import pandas as pd

df = pd.DataFrame({'price': [10, 20, 30, 40], 'qty': [1, 2, 3, 4]})
result = df.eval('total = price * qty')
print(result)

The assignment inside the string creates a total column, so the returned DataFrame has three columns with the same values we computed by hand earlier. Without the assignment, eval would return just the resulting Series instead.

$ python df_eval.py
   price  qty  total
0     10    1     10
1     20    2     40
2     30    3     90
3     40    4    160

Filtering with query

The query method filters rows using a string expression instead of bracket indexing. Column names can be written directly in the expression, which often reads more like a sentence. It is the string-based cousin of the Boolean indexing covered in the Pandas query tutorial.

df_query.py
import pandas as pd

df = pd.DataFrame({
    'product': ['apple', 'banana', 'cherry', 'apple'],
    'price': [10, 20, 30, 15]
})
selected = df.query('price >= 15 and product != "banana"')
print(selected)

Two conditions are combined with and, so only rows that satisfy both survive: the cherry and the second apple. The original row indices, 2 and 3, are kept in the output to show where the rows came from.

$ python df_query.py
  product  price
2  cherry     30
3   apple     15

When a loop is unavoidable: itertuples

Some work cannot be expressed as array arithmetic, for example calling a library that accepts one record at a time. In that case itertuples is the best row iterator: it yields a named tuple, a lightweight object whose fields are read by name such as row.price. This is much faster than iterrows, which builds a Series for every row.

itertuples_loop.py
import pandas as pd

df = pd.DataFrame({'price': [10, 20, 30], 'qty': [1, 2, 3]})
totals = [row.price * row.qty for row in df.itertuples()]
print(totals)

Each row is a named tuple, and multiplying its price and qty fields gives the same totals as the vectorised version. With index=False the index would not appear as an extra field; here we simply ignore it. Use this approach only when a genuine loop is needed.

$ python itertuples_loop.py
[10, 40, 90]

A timing comparison

The timeit module runs a piece of code many times and reports how long it took, which removes much of the noise from a single measurement. Here we double 100,000 values twice: once with a Python loop that appends to a list, and once with the vectorised expression s * 2. The script prints only which version was faster; raw timings would vary far too much between runs to be worth publishing.

timing.py
import timeit

import numpy as np
import pandas as pd

s = pd.Series(np.arange(100_000))

def loop():
    out = []
    for value in s:
        out.append(value * 2)
    return out

def vectorized():
    return s * 2

t_loop = timeit.timeit(loop, number=20)
t_vec = timeit.timeit(vectorized, number=20)
print('vectorized is faster than the Python loop:', t_vec < t_loop)

On the machine used to verify this example the vectorised version took roughly two orders of magnitude less time. Absolute timings depend heavily on the machine, the Python build and the current load, and the exact ratio changes from run to run, so the script deliberately prints only the comparison, whose result is stable. The takeaway is the size of the gap, not any particular number: prefer whole-column operations and reach for a loop only when the calculation genuinely requires one.

$ python timing.py
vectorized is faster than the Python loop: True

Source

Pandas Basics Documentation, Pandas DataFrame.eval Documentation, Pandas DataFrame.query Documentation, and Pandas DataFrame.itertuples Documentation

In this article we have seen why row-wise Python loops are slow, how to calculate with whole columns, how np.where, eval and query express common choices and filters, and how itertuples helps when a loop is necessary. For string-based row filtering see the Pandas query tutorial, and for other ways to transform data consult Pandas apply, map and transform.

Author

I am Jan Bodnar, a programmer and technical writer. I have been writing programming tutorials at ZetCode since 2007 and have published eight e-books. See about the author.

List all Pandas tutorials.