Pandas Duplicate Rows
last modified November 17, 2026 Python 3.15
A duplicate row is a row whose values match those of another row in every column. Real data sets often contain repeated records, and leaving them in can skew a calculation. Pandas gives us two closely related tools: duplicated, which reports where the repeats are, and drop_duplicates, which removes them.
Detecting and removing are separate jobs. Detection leaves the data untouched and answers the question “which rows repeat?”. Removal builds a new table with the extra copies stripped out. The examples below were verified with pandas 3.1.1.
Detecting duplicates with duplicated
A boolean mask is a Series of True and False values that lines up row by row with the data. Calling duplicated on a DataFrame returns exactly such a mask: True marks a row that has appeared earlier in the table, so only later copies are flagged.
import pandas as pd
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Alice', 'Carol', 'Bob'],
'score': [85, 90, 85, 78, 90]
})
print(df.duplicated())
The mask has one entry per row, indexed just like the DataFrame. Rows 2 and 4 are True because the pairs (Alice, 85) and (Bob, 90) were already seen at rows 0 and 1. Notice that the first occurrence of each pair is False.
$ python duplicated.py 0 False 1 False 2 True 3 False 4 True dtype: bool
Removing duplicates with drop_duplicates
drop_duplicates returns a new DataFrame with the flagged rows removed. By default it keeps the first occurrence of each repeated row — the same rows that duplicated marked False — and leaves the original index values in place.
import pandas as pd
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Alice', 'Carol', 'Bob'],
'score': [85, 90, 85, 78, 90]
})
print(df.drop_duplicates())
Only the three unique rows survive. The original labels 0, 1 and 3 are kept rather than renumbered, which tells you where each surviving row came from. We change that with ignore_index later.
$ python drop_duplicates.py
name score
0 Alice 85
1 Bob 90
3 Carol 78
Choosing which copy to keep
The keep parameter decides which member of a group of equal rows survives. It accepts 'first' (the default), 'last', or False. The last choice discards the entire group, keeping only rows that have no duplicate at all.
import pandas as pd
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Alice', 'Bob'],
'score': [85, 90, 85, 90]
})
print("keep='first':")
print(df.drop_duplicates(keep='first'))
print()
print("keep='last':")
print(df.drop_duplicates(keep='last'))
print()
print("keep=False:")
print(df.drop_duplicates(keep=False))
With 'first' rows 0 and 1 remain; with 'last' rows 2 and 3 remain, carrying the later index labels. With keep=False every row belongs to a duplicated pair, so nothing is left and an empty DataFrame is printed.
$ python keep.py
keep='first':
name score
0 Alice 85
1 Bob 90
keep='last':
name score
2 Alice 85
3 Bob 90
keep=False:
Empty DataFrame
Columns: [name, score]
Index: []
Limiting the check to selected columns
A subset is the list of columns that Pandas compares when deciding whether two rows match. Passing subset= to either method ignores every other column. This is useful when a table has an identifier column that should be unique, plus other columns that legitimately differ.
import pandas as pd
df = pd.DataFrame({
'name': ['Alice', 'Alice', 'Bob', 'Bob'],
'score': [85, 90, 78, 78]
})
print("duplicated(subset=['name']):")
print(df.duplicated(subset=['name']))
print()
print("drop_duplicates(subset=['name'], keep='first'):")
print(df.drop_duplicates(subset=['name'], keep='first'))
Here the scores differ, so on the full table no row is a duplicate. Comparing only the name column, however, flags the second Alice and the second Bob. Dropping with that subset leaves one row per name: Alice with her first score, 85, and Bob with his first score, 78.
$ python subset.py
duplicated(subset=['name']):
0 False
1 True
2 False
3 True
dtype: bool
drop_duplicates(subset=['name'], keep='first'):
name score
0 Alice 85
2 Bob 78
Resetting the index
The index is the row-label Series that identifies each row. Because drop_duplicates preserves the original labels, the result can have gaps such as 0, 1, 3. Setting ignore_index=True replaces them with a fresh continuous range starting at zero.
import pandas as pd
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Alice', 'Carol'],
'score': [85, 90, 85, 78]
})
print(df.drop_duplicates())
print()
print(df.drop_duplicates(ignore_index=True))
The first output keeps the gap at label 3. The second output renumbers the surviving rows 0, 1, 2. The data is identical; only the labels changed.
$ python ignore_index.py
name score
0 Alice 85
1 Bob 90
3 Carol 78
name score
0 Alice 85
1 Bob 90
2 Carol 78
Counting duplicates
Because duplicated returns a boolean mask, counting is simply a matter of adding up its True values — Python treats True as 1 and False as 0, and a Series provides a sum method. The number of unique rows is the length of the table minus that count, or the length of the result of drop_duplicates.
import pandas as pd
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Alice'],
'score': [85, 90, 85, 78, 90, 85]
})
mask = df.duplicated()
print("duplicate mask:", mask.to_list())
print("duplicate count:", int(mask.sum()))
print("unique count:", int(df.drop_duplicates().shape[0]))
The mask shows three True entries, so there are three extra copies: rows 2, 4 and 5. Subtracting them from the six original rows leaves three unique rows.
$ python counting.py duplicate mask: [False, False, True, False, True, True] duplicate count: 3 unique count: 3
Detecting versus removing on a Series
Both methods work on a Series as well as a DataFrame. On a one-dimensional Series there are no columns to choose from, so each value is compared directly with the values before it. This makes the difference between the two methods especially clear: one returns labels and booleans, the other returns the values that survive.
import pandas as pd
s = pd.Series(['x', 'y', 'x', 'z', 'y', 'x'])
print("detect (boolean mask):")
print(s.duplicated())
print()
print("remove (values kept):")
print(s.drop_duplicates())
The first call is a diagnostic: a boolean Series telling us that positions 2, 4 and 5 repeat. The second call is the cleanup: a shorter Series holding x, y and z, with their original labels 0, 1 and 3. Use duplicated to investigate, then drop_duplicates when you want the cleaned data.
$ python series_duplicates.py detect (boolean mask): 0 False 1 False 2 True 3 False 4 True 5 True dtype: bool remove (values kept): 0 x 1 y 3 z dtype: str
Source
Pandas DataFrame.duplicated Documentation, Pandas DataFrame.drop_duplicates Documentation, and Pandas Series.duplicated Documentation
In this article we have detected duplicate rows with duplicated, removed them with drop_duplicates, chosen which copy to keep, restricted the comparison to a subset of columns, reset the index, counted the repeats, and seen the difference between detecting and removing. For related clean-up steps see dropping missing values and filling missing values; for other ways to narrow a table down see boolean indexing.
Author
List all Pandas tutorials.