Python encode/decode
last modified October 30, 2026 Python 3.15
In this article we show how to encode and decode data in Python.
Strings and bytes
Python has two types for sequences of characters and bytes. A
str is a sequence of Unicode code points; a bytes
object is an immutable sequence of integers in the range 0 to 255. Files,
network sockets, and serial ports work with bytes. An encoding
defines how code points are turned into bytes and back.
str.encode(encoding='utf-8', errors='strict')
The str.encode method encodes the string into the
bytes type. The encoding defaults to UTF-8.
bytes.decode(encoding='utf-8', errors='strict')
The bytes.decode method decodes bytes into a string.
Python supports many standard encodings, including utf_8,
utf_16, ascii, latin_1,
iso8859_2, and cp1250. An encoding may have
several aliases; for instance, utf_8 can also be written as
utf8 or utf-8, and the names are case-insensitive.
Encoding a string
In the first example, we encode a message containing emoji characters.
text = 'one 🐘 and three 🐋'
print(text)
print(len(text))
for encoding in ['utf-8', 'utf-16', 'utf-16-le', 'utf-32']:
data = text.encode(encoding)
print(f'{encoding:9} {len(data):3} {data[:12]}')
We print the text and its number of characters, then encode it with four encodings and print the number of bytes and the first twelve bytes.
data = text.encode(encoding)
UTF-8 uses one byte for ASCII characters and four bytes for each emoji.
UTF-16 uses two bytes for most characters and four bytes (a surrogate pair)
for the emoji. The utf-16 and utf-32 codecs start
the output with a byte order mark (BOM); utf-16-le has a fixed
byte order and no BOM, so it is two bytes shorter.
$ python encode_text.py one 🐘 and three 🐋 17 utf-8 23 b'one \xf0\x9f\x90\x98 and' utf-16 40 b'\xff\xfeo\x00n\x00e\x00 \x00=\xd8' utf-16-le 38 b'o\x00n\x00e\x00 \x00=\xd8\x18\xdc' utf-32 72 b'\xff\xfe\x00\x00o\x00\x00\x00n\x00\x00\x00'
Code points and bytes
Each character is a code point, an integer written as U+XXXX.
UTF-8 encodes a code point in one to four bytes.
for char in 'aé€🐘':
utf8 = char.encode('utf-8')
print(f'{char} U+{ord(char):04X} {len(utf8)} {utf8.hex(" ")}')
print(chr(0x1F40B))
print(bytes.fromhex('e2 82 ac').decode('utf-8'))
The ord function returns the code point of a character and
chr does the opposite. The hex method shows the
bytes as hexadecimal numbers with an optional separator, and
bytes.fromhex parses them.
$ python code_points.py a U+0061 1 61 é U+00E9 2 c3 a9 € U+20AC 3 e2 82 ac 🐘 U+1F418 4 f0 9f 90 98 🐋 €
Decoding a file
In the following example, we read a file in binary mode and decode the data into a string.
one 🐘 and three 🐋
We have this data.txt file.
fname = 'data.txt'
with open(fname, mode='rb') as f:
contents = f.read()
print(type(contents))
print(contents)
print(contents.decode('utf-8'), end='')
with open(fname, encoding='utf-8') as f:
print(repr(f.read()))
In the rb mode, read returns bytes. We print their
type and representation, then the decoded text.
with open(fname, encoding='utf-8') as f:
print(repr(f.read()))
In text mode, the file object decodes the data for us. It is good practice to always pass the encoding explicitly.
$ python decode_file.py <class 'bytes'> b'one \xf0\x9f\x90\x98 and three \xf0\x9f\x90\x8b\n' one 🐘 and three 🐋 'one 🐘 and three 🐋\n'
Legacy encodings
Older systems use single-byte encodings, which can represent only 256
characters. Central European languages use cp1250 on Windows and
iso8859_2 elsewhere.
text = 'Žluťoučký kůň úpěl ďábelské ódy'
for encoding in ['utf-8', 'cp1250', 'iso8859_2']:
data = text.encode(encoding)
print(f'{encoding:10} {len(data):2} {data[:12]}')
try:
text.encode('latin-1')
except UnicodeEncodeError as e:
print(e)
We encode a Czech sentence. In UTF-8, each accented letter takes two bytes;
in the single-byte encodings, every character takes one byte, but the two
encodings assign different bytes to some letters, such as Ž. The
latin-1 encoding has no Ž, so encoding fails with a
UnicodeEncodeError.
$ python slovak.py utf-8 43 b'\xc5\xbdlu\xc5\xa5ou\xc4\x8dk\xc3' cp1250 31 b'\x8elu\x9dou\xe8k\xfd k\xf9' iso8859_2 31 b'\xaelu\xbbou\xe8k\xfd k\xf9' 'latin-1' codec can't encode character '\u017d' in position 0: ordinal not in range(256)
Mojibake
Bytes do not carry information about their encoding. Decoding with the wrong encoding produces garbled text, called mojibake, or an error.
text = 'Žluťoučký kůň'
data = text.encode('utf-8')
print(data.decode('cp1252', errors='replace'))
legacy = text.encode('cp1250')
try:
legacy.decode('utf-8')
except UnicodeDecodeError as e:
print(e)
print(legacy.decode('cp1250'))
Decoding UTF-8 bytes as cp1252 turns each two-byte sequence
into two Western European characters; the replace handler
substitutes bytes that are undefined in cp1252. Decoding
cp1250 bytes as UTF-8 fails, because they are not valid UTF-8.
Only the correct encoding gives the original text.
$ python mojibake.py ŽluÅ¥ouÄ�ký kůň 'utf-8' codec can't decode byte 0x8e in position 0: invalid start byte Žluťoučký kůň
Encoding error handlers
The errors parameter decides what happens with characters that
the encoding cannot represent. The default strict handler raises
an exception.
text = 'café ½ €'
for errors in ['ignore', 'replace', 'backslashreplace', 'xmlcharrefreplace',
'namereplace']:
print(f'{errors:18} {text.encode("ascii", errors=errors)}')
try:
text.encode('ascii')
except UnicodeEncodeError as e:
print(e)
We encode a text into ASCII. The ignore handler drops the
characters, replace substitutes question marks, and the
remaining handlers write escape sequences: Python escapes, XML character
references, and Unicode names.
$ python encode_errors.py
ignore b'caf '
replace b'caf? ? ?'
backslashreplace b'caf\\xe9 \\xbd \\u20ac'
xmlcharrefreplace b'café ½ €'
namereplace b'caf\\N{LATIN SMALL LETTER E WITH ACUTE} \\N{VULGAR FRACTION ONE HALF} \\N{EURO SIGN}'
'ascii' codec can't encode character '\xe9' in position 3: ordinal not in range(128)
Decoding error handlers
Error handlers also apply when invalid bytes are decoded.
data = b'caf\xc3\xa9 \xff\xfe ok'
for errors in ['ignore', 'replace', 'backslashreplace', 'surrogateescape']:
print(f'{errors:16} {data.decode("utf-8", errors=errors)!r}')
text = data.decode('utf-8', errors='surrogateescape')
print(text.encode('utf-8', errors='surrogateescape') == data)
The replace handler inserts the U+FFFD replacement character.
The surrogateescape handler maps each invalid byte to a lone
surrogate code point; encoding with the same handler restores the original
bytes. Python uses it for file names that are not valid in the file system
encoding.
$ python decode_errors.py ignore 'café ok' replace 'café �� ok' backslashreplace 'café \\xff\\xfe ok' surrogateescape 'café \udcff\udcfe ok' True
Byte order mark
A byte order mark is the U+FEFF character at the start of a text. Some Windows programs, such as Excel, write it at the start of UTF-8 files.
from pathlib import Path
Path('excel.csv').write_text('name,city\nJán,Košice\n', encoding='utf-8-sig')
data = Path('excel.csv').read_bytes()
print(data[:8])
print(repr(data.decode('utf-8')[:6]))
print(repr(data.decode('utf-8-sig')[:6]))
print('hi'.encode('utf-16'))
print(b'\xff\xfeh\x00i\x00'.decode('utf-16'))
print(b'\x00h\x00i'.decode('utf-16-be'))
The utf-8-sig codec writes the three BOM bytes when encoding
and removes them when decoding. With the plain utf-8 codec, the
BOM remains in the text as \ufeff, which breaks, for instance,
the first column name of a CSV file. The utf-16 codec uses the
BOM to detect the byte order; without one, we choose the
-be or -le variant.
$ python bom.py b'\xef\xbb\xbfname,' '\ufeffname,' 'name,c' b'\xff\xfeh\x00i\x00' hi hi
Unicode normalization
The same text can be written with different code points. The letter é is either a single code point or the letter e followed by a combining accent.
import unicodedata
composed = 'café'
decomposed = 'café'
print(composed, decomposed, composed == decomposed)
print(len(composed), len(decomposed))
print(composed.encode('utf-8'), decomposed.encode('utf-8'))
nfc = unicodedata.normalize('NFC', decomposed)
print(nfc == composed, len(nfc))
for char in decomposed[-2:]:
print(f'U+{ord(char):04X} {unicodedata.name(char)}')
The two strings look the same but they are not equal, they have different
lengths, and they encode to different bytes. The
unicodedata.normalize function converts the text to a
canonical form; NFC composes the characters. Text from different sources
should be normalized before it is compared.
$ python normalization.py café café False 4 5 b'caf\xc3\xa9' b'cafe\xcc\x81' True 4 U+0065 LATIN SMALL LETTER E U+0301 COMBINING ACUTE ACCENT
Encoding names
The codecs.lookup function finds a codec by any of its names.
import codecs
for name in ['utf8', 'UTF-8', 'u8', 'latin1', 'iso-8859-1', 'cp1250',
'windows-1250', 'ascii', 'us-ascii']:
print(f'{name:13} {codecs.lookup(name).name}')
We print the canonical name for several aliases.
$ python aliases.py utf8 utf-8 UTF-8 utf-8 u8 utf-8 latin1 iso8859-1 iso-8859-1 iso8859-1 cp1250 cp1250 windows-1250 cp1250 ascii ascii us-ascii ascii
Default encoding of open
When open is called in text mode without the
encoding argument, a default encoding is used. Before Python
3.15, it was the locale encoding, so the same program could write different
bytes on different systems. Since Python 3.15, the UTF-8 mode is enabled by
default (PEP 686), and open uses UTF-8 regardless of the
locale.
import locale
import sys
from pathlib import Path
print(f'utf8_mode: {sys.flags.utf8_mode}')
print(f'locale encoding: {locale.getencoding()}')
with open('note.txt', 'w') as f:
print(f'open() encoding: {f.encoding}')
f.write('žltý kôň')
print(Path('note.txt').read_bytes())
The program prints the UTF-8 mode flag, the locale encoding, and the
encoding chosen by open, and writes a Slovak text into a file.
We run the program under a legacy ISO-8859-2 locale. Python 3.15 writes
UTF-8, while Python 3.14 uses the locale encoding. The UTF-8 mode can be
disabled with -X utf8=0 or PYTHONUTF8=0, which
restores the old behaviour.
$ LC_ALL=sk_SK.ISO-8859-2 python default_encoding.py
utf8_mode: 1
locale encoding: ANSI_X3.4-1968
open() encoding: utf-8
b'\xc5\xbelt\xc3\xbd k\xc3\xb4\xc5\x88'
$ LC_ALL=sk_SK.ISO-8859-2 python3.14 default_encoding.py
utf8_mode: 1
locale encoding: ANSI_X3.4-1968
open() encoding: utf-8
b'\xc5\xbelt\xc3\xbd k\xc3\xb4\xc5\x88'
$ LC_ALL=sk_SK.ISO-8859-2 python -X utf8=0 default_encoding.py
utf8_mode: 0
locale encoding: ANSI_X3.4-1968
open() encoding: ANSI_X3.4-1968
Traceback (most recent call last):
File "/home/user/python/default_encoding.py", line 10, in <module>
f.write('\u017elt\xfd k\xf4\u0148')
~~~~~~~^^^^^^^^^^^^
UnicodeEncodeError: 'ascii' codec can't encode character '\u017e' in position 0: ordinal not in range(128)
Bytes from HTTP
The data on the network is transmitted as bytes. The requests
library gives us both the raw bytes and the decoded text.
import requests
resp = requests.get('https://example.com', timeout=10)
print(type(resp.content), len(resp.content))
print(resp.content[:46])
print(resp.headers['content-type'])
print(resp.encoding)
print(resp.text[:46])
print(resp.content.decode('utf-8') == resp.text)
The content attribute holds the body as bytes. The
text attribute decodes it with the encoding from the
Content-Type header, available in the encoding
attribute.
$ python http_bytes.py <class 'bytes'> 577 b'<!doctype html><html lang=en><head><meta chars' text/html; charset=utf-8 utf-8 <!doctype html><html lang=en><head><meta chars True
Bytes on a socket
At the socket level, we encode the request ourselves and decode the response.
import socket
request = 'HEAD / HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n'
with socket.create_connection(('example.com', 80), timeout=10) as sock:
sock.sendall(request.encode('ascii'))
chunks = []
while chunk := sock.recv(1024):
chunks.append(chunk)
response = b''.join(chunks)
status_line = response.split(b'\r\n')[0]
print(status_line)
print(status_line.decode('ascii'))
print(type(response).__name__, len(response) > 0)
We send an HTTP HEAD request to example.com. The
sendall method accepts only bytes, so we encode the request;
HTTP headers are ASCII. The recv method returns bytes; we join
the chunks, take the status line, and decode it.
$ python socket_bytes.py b'HTTP/1.1 200 OK' HTTP/1.1 200 OK bytes True
Source
Unicode HOWTO - Python documentation
In this article we have worked with the encode and decode methods in Python.
Author
List all Python tutorials.