Pandas Vectorization and Performance
last modified November 17, 2026 Python 3.15
Data work often means performing the same arithmetic on thousands or millions
of values. A tempting approach is to walk over the rows with a Python
for loop, but that is surprisingly slow. This article explains
why and shows the faster tools that Pandas provides.
Vectorization means operating on whole arrays at once instead of one
element at a time: an expression such as df['a'] * df['b'] is handed
to compiled code that runs a tight loop over every value. We also cover
eval, query and itertuples, and finish
with a real timing comparison.
Why row-wise loops are slow
On each pass through a Python loop the interpreter performs bookkeeping: it looks up names, checks types and dispatches operations again for every single value. A vectorised expression does that work once and then lets compiled code process the whole column. The result is a new Series whose values line up with the input rows by position, which we can store straight back into the DataFrame as a new column.
import pandas as pd
df = pd.DataFrame({'price': [10, 20, 30, 40], 'qty': [1, 2, 3, 4]})
df['total'] = df['price'] * df['qty']
print(df)
The multiplication is applied to both entire columns, producing 10, 40, 90 and 160 in one step. No explicit loop appears in our code, yet all four rows are processed. This is the basic pattern to reach for whenever values from columns need to be combined.
$ python vectorized_arithmetic.py price qty total 0 10 1 10 1 20 2 40 2 30 3 90 3 40 4 160
Conditional logic with np.where
Often a value depends on a condition. NumPy's np.where is the
vectorised equivalent of an if-else expression: it checks a condition for every
element and picks the first value where the condition is true and the second
where it is false. The condition here is a whole Boolean Series, not a single
Boolean, so the choice is made for all rows at once.
import numpy as np
import pandas as pd
df = pd.DataFrame({'price': [10, 25, 40, 15]})
df['label'] = np.where(df['price'] >= 25, 'high', 'low')
print(df)
Rows with a price of 25 or more are labelled high and the rest
low. The Boolean Series df['price'] >= 25 is
evaluated for the whole column, and np.where assigns a label to
each row based on it.
$ python np_where.py price label 0 10 low 1 25 high 2 40 high 3 15 low
Computing with eval
The eval method evaluates an arithmetic expression written as a
string. This is useful for long formulas because it keeps the expression
readable and can avoid creating temporary Series for intermediate results. The
column names in the string refer to columns of the DataFrame.
import pandas as pd
df = pd.DataFrame({'price': [10, 20, 30, 40], 'qty': [1, 2, 3, 4]})
result = df.eval('total = price * qty')
print(result)
The assignment inside the string creates a total column, so the
returned DataFrame has three columns with the same values we computed by hand
earlier. Without the assignment, eval would return just the
resulting Series instead.
$ python df_eval.py price qty total 0 10 1 10 1 20 2 40 2 30 3 90 3 40 4 160
Filtering with query
The query method filters rows using a string expression instead of
bracket indexing. Column names can be written directly in the expression,
which often reads more like a sentence. It is the string-based cousin of the
Boolean indexing covered in the Pandas query
tutorial.
import pandas as pd
df = pd.DataFrame({
'product': ['apple', 'banana', 'cherry', 'apple'],
'price': [10, 20, 30, 15]
})
selected = df.query('price >= 15 and product != "banana"')
print(selected)
Two conditions are combined with and, so only rows that satisfy
both survive: the cherry and the second apple. The original row indices, 2 and
3, are kept in the output to show where the rows came from.
$ python df_query.py product price 2 cherry 30 3 apple 15
When a loop is unavoidable: itertuples
Some work cannot be expressed as array arithmetic, for example calling a
library that accepts one record at a time. In that case
itertuples is the best row iterator: it yields a
named tuple, a lightweight object whose fields are read by name such
as row.price. This is much faster than iterrows,
which builds a Series for every row.
import pandas as pd
df = pd.DataFrame({'price': [10, 20, 30], 'qty': [1, 2, 3]})
totals = [row.price * row.qty for row in df.itertuples()]
print(totals)
Each row is a named tuple, and multiplying its price and
qty fields gives the same totals as the vectorised version. With
index=False the index would not appear as an extra field; here we
simply ignore it. Use this approach only when a genuine loop is needed.
$ python itertuples_loop.py [10, 40, 90]
A timing comparison
The timeit module runs a piece of code many times and reports how
long it took, which removes much of the noise from a single measurement. Here
we double 100,000 values twice: once with a Python loop that appends to a list,
and once with the vectorised expression s * 2. The script prints
only which version was faster; raw timings would vary far too much between
runs to be worth publishing.
import timeit
import numpy as np
import pandas as pd
s = pd.Series(np.arange(100_000))
def loop():
out = []
for value in s:
out.append(value * 2)
return out
def vectorized():
return s * 2
t_loop = timeit.timeit(loop, number=20)
t_vec = timeit.timeit(vectorized, number=20)
print('vectorized is faster than the Python loop:', t_vec < t_loop)
On the machine used to verify this example the vectorised version took roughly two orders of magnitude less time. Absolute timings depend heavily on the machine, the Python build and the current load, and the exact ratio changes from run to run, so the script deliberately prints only the comparison, whose result is stable. The takeaway is the size of the gap, not any particular number: prefer whole-column operations and reach for a loop only when the calculation genuinely requires one.
$ python timing.py vectorized is faster than the Python loop: True
Source
Pandas Basics Documentation, Pandas DataFrame.eval Documentation, Pandas DataFrame.query Documentation, and Pandas DataFrame.itertuples Documentation
In this article we have seen why row-wise Python loops are slow, how to
calculate with whole columns, how np.where, eval and
query express common choices and filters, and how
itertuples helps when a loop is necessary. For string-based row
filtering see the Pandas query tutorial, and for
other ways to transform data consult
Pandas apply, map and transform.
Author
List all Pandas tutorials.