ZetCode

Pandas Memory Usage

last modified November 17, 2026 Python 3.15

Memory usage is the number of bytes a DataFrame occupies in RAM. Knowing it matters when a dataset approaches the available memory: a column stored as 64-bit integers can take eight times more space than the same values stored as 8-bit integers, and repeated text can take far more space than a categorical code. Pandas provides several ways to measure and reduce it.

The byte counts below come from real runs of pandas 3.1.1 with pyarrow 26.0.1 installed. Exact numbers depend on the platform, the Pandas version and the actual values, so treat them as illustrations rather than constants.

Shallow and deep memory_usage

The memory_usage method returns bytes used per column, including the index. By default it is shallow: for text columns it counts the references to strings, not the strings themselves. Passing deep=True counts the real string data as well, which is the number to trust for object-like and string columns.

mem_usage.py
import pandas as pd

df = pd.DataFrame({'name': ['apple', 'banana', 'cherry'] * 100})
print('shallow (per column):')
print(df.memory_usage())
print()
print('deep total:', df.memory_usage(deep=True).sum(), 'bytes')

The shallow view reports 4100 bytes for the 300 names plus 132 bytes for the index. The deep total, 4232 bytes, adds the actual characters. Calling sum() on the result turns the per-column Series into one total.

$ python mem_usage.py
shallow (per column):
Index     132
name     4100
dtype: int64

deep total: 4232 bytes

A quick overview with info

The info method prints a compact summary of a DataFrame: its class, the number of rows and columns, each column's non-null count and dtype, and a memory total. With memory_usage='deep' that total includes the real string bytes, equivalent to memory_usage(deep=True).sum().

mem_info.py
import pandas as pd

df = pd.DataFrame({'name': ['apple', 'banana', 'cherry'] * 100})
df.info(memory_usage='deep')

The output lists one column, name, with 300 non-null entries and dtype str, and ends with a memory usage of 4.1 KB. This is a fast way to inspect both dtypes and memory before deciding what to change.

$ python mem_info.py
<class 'pandas.DataFrame'>
RangeIndex: 300 entries, 0 to 299
Data columns (total 1 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   name    300 non-null    str  
dtypes: str(1)
memory usage: 4.1 KB

Downcasting numeric dtypes

A dtype describes how values are stored. Pandas often chooses a wide default, such as 64-bit integer (int64), even when the values would fit in something smaller. astype converts a column to a narrower dtype; here int8 stores each value in a single byte.

mem_downcast.py
import pandas as pd

df = pd.DataFrame({'x': [1, 2, 3] * 100})
print('int64:', df.memory_usage(deep=True)['x'], 'bytes')
small = df.astype({'x': 'int8'})
print('int8: ', small.memory_usage(deep=True)['x'], 'bytes')

The same 300 values need 2400 bytes as int64 and only 300 bytes as int8. Only downcast when every value fits: int8 holds numbers from -128 to 127. For floating-point columns, float32 is the corresponding smaller choice.

$ python mem_downcast.py
int64: 2400 bytes
int8:  300 bytes

Strings and StringDtype

In pandas 3 the default text dtype is str, shown as str in info output. An alternative is pd.StringDtype with storage='pyarrow', which keeps the strings in Arrow-backed arrays; pyarrow is installed in the environment used here. The dtype attribute tells you which representation a column uses.

mem_string.py
import pandas as pd

s = pd.Series(['apple', 'banana', 'cherry'] * 100)
print('default dtype:', s.dtype)
print('default deep: ', s.memory_usage(deep=True), 'bytes')
a = s.astype(pd.StringDtype(storage='pyarrow'))
print('arrow dtype:  ', a.dtype)
print('arrow deep:   ', a.memory_usage(deep=True), 'bytes')

The default dtype is str and the Arrow-backed one is string. On these short, repeated strings both report 4232 bytes, so there is no saving here: the optional dtype is chosen for its nullable semantics and Arrow interoperability, not as a guaranteed memory win.

$ python mem_string.py
default dtype: str
default deep:  4232 bytes
arrow dtype:   string
arrow deep:    4232 bytes

Categoricals for repeated values

When a column contains only a handful of distinct strings, converting it to a category dtype stores each distinct value once and uses small integer codes for the rows. This is often the biggest single saving available for text data, and it is described in more detail in the categorical data tutorial.

mem_cat.py
import pandas as pd

s = pd.Series(['apple', 'banana', 'cherry'] * 100)
print('object deep:  ', s.memory_usage(deep=True), 'bytes')
c = s.astype('category')
print('category deep:', c.memory_usage(deep=True), 'bytes')

The deep figure drops from 4232 bytes to 474 bytes because the three words are stored once instead of three hundred times. The saving grows with the number of rows and shrinks as the number of distinct values grows.

$ python mem_cat.py
object deep:   4232 bytes
category deep: 474 bytes

Reading large files in chunks

A chunk is one piece of a file read into memory at a time. Passing chunksize to read_csv makes it return an iterator of DataFrames instead of one huge frame. Each chunk is processed and then released, so peak memory depends on the chunk size rather than the file size.

mem_chunks.py
import pandas as pd

path = '/tmp/opencode/agentG_big.csv'
pd.DataFrame({'a': range(10000), 'b': ['x'] * 10000}).to_csv(path, index=False)

total_rows = 0
chunks = 0
for chunk in pd.read_csv(path, chunksize=2000):
    total_rows += len(chunk)
    chunks += 1
print('chunks:', chunks)
print('rows read:', total_rows)

The example writes a temporary ten-thousand-row CSV, then reads it five chunks of 2000 rows at a time and adds up their lengths. In a real program you would aggregate or filter each chunk before discarding it, keeping only the small result in memory.

$ python mem_chunks.py
chunks: 5
rows read: 10000

Source

Pandas DataFrame.memory_usage Documentation, Pandas DataFrame.info Documentation, Pandas Categorical Data Documentation, and Pandas read_csv Documentation

In this article we have measured shallow and deep memory, used info for an overview, downcast numeric columns, compared the default str dtype with a pyarrow-backed StringDtype, saved space with categoricals, and read a large file in chunks. For the categorical representation see Pandas categorical data, and for how sharing affects memory see Pandas copy-on-write.

Author

I am Jan Bodnar, a programmer and technical writer. I have been writing programming tutorials at ZetCode since 2007 and have published eight e-books. See about the author.

List all Pandas tutorials.