ZetCode

Python encode/decode

last modified October 30, 2026 Python 3.15

In this article we show how to encode and decode data in Python.

Strings and bytes

Python has two types for sequences of characters and bytes. A str is a sequence of Unicode code points; a bytes object is an immutable sequence of integers in the range 0 to 255. Files, network sockets, and serial ports work with bytes. An encoding defines how code points are turned into bytes and back.

str.encode(encoding='utf-8', errors='strict')

The str.encode method encodes the string into the bytes type. The encoding defaults to UTF-8.

bytes.decode(encoding='utf-8', errors='strict')

The bytes.decode method decodes bytes into a string.

Python supports many standard encodings, including utf_8, utf_16, ascii, latin_1, iso8859_2, and cp1250. An encoding may have several aliases; for instance, utf_8 can also be written as utf8 or utf-8, and the names are case-insensitive.

Encoding a string

In the first example, we encode a message containing emoji characters.

encode_text.py
text = 'one 🐘 and three 🐋'
print(text)
print(len(text))

for encoding in ['utf-8', 'utf-16', 'utf-16-le', 'utf-32']:
    data = text.encode(encoding)
    print(f'{encoding:9} {len(data):3} {data[:12]}')

We print the text and its number of characters, then encode it with four encodings and print the number of bytes and the first twelve bytes.

data = text.encode(encoding)

UTF-8 uses one byte for ASCII characters and four bytes for each emoji. UTF-16 uses two bytes for most characters and four bytes (a surrogate pair) for the emoji. The utf-16 and utf-32 codecs start the output with a byte order mark (BOM); utf-16-le has a fixed byte order and no BOM, so it is two bytes shorter.

$ python encode_text.py
one 🐘 and three 🐋
17
utf-8      23 b'one \xf0\x9f\x90\x98 and'
utf-16     40 b'\xff\xfeo\x00n\x00e\x00 \x00=\xd8'
utf-16-le  38 b'o\x00n\x00e\x00 \x00=\xd8\x18\xdc'
utf-32     72 b'\xff\xfe\x00\x00o\x00\x00\x00n\x00\x00\x00'

Code points and bytes

Each character is a code point, an integer written as U+XXXX. UTF-8 encodes a code point in one to four bytes.

code_points.py
for char in 'aé€🐘':
    utf8 = char.encode('utf-8')
    print(f'{char} U+{ord(char):04X} {len(utf8)} {utf8.hex(" ")}')

print(chr(0x1F40B))
print(bytes.fromhex('e2 82 ac').decode('utf-8'))

The ord function returns the code point of a character and chr does the opposite. The hex method shows the bytes as hexadecimal numbers with an optional separator, and bytes.fromhex parses them.

$ python code_points.py
a U+0061 1 61
é U+00E9 2 c3 a9
€ U+20AC 3 e2 82 ac
🐘 U+1F418 4 f0 9f 90 98
🐋
€

Decoding a file

In the following example, we read a file in binary mode and decode the data into a string.

data.txt
one 🐘 and three 🐋

We have this data.txt file.

decode_file.py
fname = 'data.txt'

with open(fname, mode='rb') as f:
    contents = f.read()

print(type(contents))
print(contents)
print(contents.decode('utf-8'), end='')

with open(fname, encoding='utf-8') as f:
    print(repr(f.read()))

In the rb mode, read returns bytes. We print their type and representation, then the decoded text.

with open(fname, encoding='utf-8') as f:
    print(repr(f.read()))

In text mode, the file object decodes the data for us. It is good practice to always pass the encoding explicitly.

$ python decode_file.py
<class 'bytes'>
b'one \xf0\x9f\x90\x98 and three \xf0\x9f\x90\x8b\n'
one 🐘 and three 🐋
'one 🐘 and three 🐋\n'

Legacy encodings

Older systems use single-byte encodings, which can represent only 256 characters. Central European languages use cp1250 on Windows and iso8859_2 elsewhere.

slovak.py
text = 'Žluťoučký kůň úpěl ďábelské ódy'

for encoding in ['utf-8', 'cp1250', 'iso8859_2']:
    data = text.encode(encoding)
    print(f'{encoding:10} {len(data):2} {data[:12]}')

try:
    text.encode('latin-1')
except UnicodeEncodeError as e:
    print(e)

We encode a Czech sentence. In UTF-8, each accented letter takes two bytes; in the single-byte encodings, every character takes one byte, but the two encodings assign different bytes to some letters, such as Ž. The latin-1 encoding has no Ž, so encoding fails with a UnicodeEncodeError.

$ python slovak.py
utf-8      43 b'\xc5\xbdlu\xc5\xa5ou\xc4\x8dk\xc3'
cp1250     31 b'\x8elu\x9dou\xe8k\xfd k\xf9'
iso8859_2  31 b'\xaelu\xbbou\xe8k\xfd k\xf9'
'latin-1' codec can't encode character '\u017d' in position 0: ordinal not in range(256)

Mojibake

Bytes do not carry information about their encoding. Decoding with the wrong encoding produces garbled text, called mojibake, or an error.

mojibake.py
text = 'Žluťoučký kůň'
data = text.encode('utf-8')

print(data.decode('cp1252', errors='replace'))

legacy = text.encode('cp1250')

try:
    legacy.decode('utf-8')
except UnicodeDecodeError as e:
    print(e)

print(legacy.decode('cp1250'))

Decoding UTF-8 bytes as cp1252 turns each two-byte sequence into two Western European characters; the replace handler substitutes bytes that are undefined in cp1252. Decoding cp1250 bytes as UTF-8 fails, because they are not valid UTF-8. Only the correct encoding gives the original text.

$ python mojibake.py
Žluťou�ký kůň
'utf-8' codec can't decode byte 0x8e in position 0: invalid start byte
Žluťoučký kůň

Encoding error handlers

The errors parameter decides what happens with characters that the encoding cannot represent. The default strict handler raises an exception.

encode_errors.py
text = 'café ½ €'

for errors in ['ignore', 'replace', 'backslashreplace', 'xmlcharrefreplace',
               'namereplace']:
    print(f'{errors:18} {text.encode("ascii", errors=errors)}')

try:
    text.encode('ascii')
except UnicodeEncodeError as e:
    print(e)

We encode a text into ASCII. The ignore handler drops the characters, replace substitutes question marks, and the remaining handlers write escape sequences: Python escapes, XML character references, and Unicode names.

$ python encode_errors.py
ignore             b'caf  '
replace            b'caf? ? ?'
backslashreplace   b'caf\\xe9 \\xbd \\u20ac'
xmlcharrefreplace  b'caf&#233; &#189; &#8364;'
namereplace        b'caf\\N{LATIN SMALL LETTER E WITH ACUTE} \\N{VULGAR FRACTION ONE HALF} \\N{EURO SIGN}'
'ascii' codec can't encode character '\xe9' in position 3: ordinal not in range(128)

Decoding error handlers

Error handlers also apply when invalid bytes are decoded.

decode_errors.py
data = b'caf\xc3\xa9 \xff\xfe ok'

for errors in ['ignore', 'replace', 'backslashreplace', 'surrogateescape']:
    print(f'{errors:16} {data.decode("utf-8", errors=errors)!r}')

text = data.decode('utf-8', errors='surrogateescape')
print(text.encode('utf-8', errors='surrogateescape') == data)

The replace handler inserts the U+FFFD replacement character. The surrogateescape handler maps each invalid byte to a lone surrogate code point; encoding with the same handler restores the original bytes. Python uses it for file names that are not valid in the file system encoding.

$ python decode_errors.py
ignore           'café  ok'
replace          'café �� ok'
backslashreplace 'café \\xff\\xfe ok'
surrogateescape  'café \udcff\udcfe ok'
True

Byte order mark

A byte order mark is the U+FEFF character at the start of a text. Some Windows programs, such as Excel, write it at the start of UTF-8 files.

bom.py
from pathlib import Path

Path('excel.csv').write_text('name,city\nJán,Košice\n', encoding='utf-8-sig')

data = Path('excel.csv').read_bytes()
print(data[:8])

print(repr(data.decode('utf-8')[:6]))
print(repr(data.decode('utf-8-sig')[:6]))

print('hi'.encode('utf-16'))
print(b'\xff\xfeh\x00i\x00'.decode('utf-16'))
print(b'\x00h\x00i'.decode('utf-16-be'))

The utf-8-sig codec writes the three BOM bytes when encoding and removes them when decoding. With the plain utf-8 codec, the BOM remains in the text as \ufeff, which breaks, for instance, the first column name of a CSV file. The utf-16 codec uses the BOM to detect the byte order; without one, we choose the -be or -le variant.

$ python bom.py
b'\xef\xbb\xbfname,'
'\ufeffname,'
'name,c'
b'\xff\xfeh\x00i\x00'
hi
hi

Unicode normalization

The same text can be written with different code points. The letter é is either a single code point or the letter e followed by a combining accent.

normalization.py
import unicodedata

composed = 'café'
decomposed = 'café'

print(composed, decomposed, composed == decomposed)
print(len(composed), len(decomposed))
print(composed.encode('utf-8'), decomposed.encode('utf-8'))

nfc = unicodedata.normalize('NFC', decomposed)
print(nfc == composed, len(nfc))

for char in decomposed[-2:]:
    print(f'U+{ord(char):04X} {unicodedata.name(char)}')

The two strings look the same but they are not equal, they have different lengths, and they encode to different bytes. The unicodedata.normalize function converts the text to a canonical form; NFC composes the characters. Text from different sources should be normalized before it is compared.

$ python normalization.py
café café False
4 5
b'caf\xc3\xa9' b'cafe\xcc\x81'
True 4
U+0065 LATIN SMALL LETTER E
U+0301 COMBINING ACUTE ACCENT

Encoding names

The codecs.lookup function finds a codec by any of its names.

aliases.py
import codecs

for name in ['utf8', 'UTF-8', 'u8', 'latin1', 'iso-8859-1', 'cp1250',
             'windows-1250', 'ascii', 'us-ascii']:
    print(f'{name:13} {codecs.lookup(name).name}')

We print the canonical name for several aliases.

$ python aliases.py
utf8          utf-8
UTF-8         utf-8
u8            utf-8
latin1        iso8859-1
iso-8859-1    iso8859-1
cp1250        cp1250
windows-1250  cp1250
ascii         ascii
us-ascii      ascii

Default encoding of open

When open is called in text mode without the encoding argument, a default encoding is used. Before Python 3.15, it was the locale encoding, so the same program could write different bytes on different systems. Since Python 3.15, the UTF-8 mode is enabled by default (PEP 686), and open uses UTF-8 regardless of the locale.

default_encoding.py
import locale
import sys
from pathlib import Path

print(f'utf8_mode: {sys.flags.utf8_mode}')
print(f'locale encoding: {locale.getencoding()}')

with open('note.txt', 'w') as f:
    print(f'open() encoding: {f.encoding}')
    f.write('žltý kôň')

print(Path('note.txt').read_bytes())

The program prints the UTF-8 mode flag, the locale encoding, and the encoding chosen by open, and writes a Slovak text into a file.

We run the program under a legacy ISO-8859-2 locale. Python 3.15 writes UTF-8, while Python 3.14 uses the locale encoding. The UTF-8 mode can be disabled with -X utf8=0 or PYTHONUTF8=0, which restores the old behaviour.

$ LC_ALL=sk_SK.ISO-8859-2 python default_encoding.py
utf8_mode: 1
locale encoding: ANSI_X3.4-1968
open() encoding: utf-8
b'\xc5\xbelt\xc3\xbd k\xc3\xb4\xc5\x88'
$ LC_ALL=sk_SK.ISO-8859-2 python3.14 default_encoding.py
utf8_mode: 1
locale encoding: ANSI_X3.4-1968
open() encoding: utf-8
b'\xc5\xbelt\xc3\xbd k\xc3\xb4\xc5\x88'
$ LC_ALL=sk_SK.ISO-8859-2 python -X utf8=0 default_encoding.py
utf8_mode: 0
locale encoding: ANSI_X3.4-1968
open() encoding: ANSI_X3.4-1968
Traceback (most recent call last):
  File "/home/user/python/default_encoding.py", line 10, in <module>
    f.write('\u017elt\xfd k\xf4\u0148')
    ~~~~~~~^^^^^^^^^^^^
UnicodeEncodeError: 'ascii' codec can't encode character '\u017e' in position 0: ordinal not in range(128)

Bytes from HTTP

The data on the network is transmitted as bytes. The requests library gives us both the raw bytes and the decoded text.

http_bytes.py
import requests

resp = requests.get('https://example.com', timeout=10)

print(type(resp.content), len(resp.content))
print(resp.content[:46])
print(resp.headers['content-type'])
print(resp.encoding)
print(resp.text[:46])
print(resp.content.decode('utf-8') == resp.text)

The content attribute holds the body as bytes. The text attribute decodes it with the encoding from the Content-Type header, available in the encoding attribute.

$ python http_bytes.py
<class 'bytes'> 577
b'<!doctype html><html lang=en><head><meta chars'
text/html; charset=utf-8
utf-8
<!doctype html><html lang=en><head><meta chars
True

Bytes on a socket

At the socket level, we encode the request ourselves and decode the response.

socket_bytes.py
import socket

request = 'HEAD / HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n'

with socket.create_connection(('example.com', 80), timeout=10) as sock:
    sock.sendall(request.encode('ascii'))

    chunks = []
    while chunk := sock.recv(1024):
        chunks.append(chunk)

response = b''.join(chunks)
status_line = response.split(b'\r\n')[0]

print(status_line)
print(status_line.decode('ascii'))
print(type(response).__name__, len(response) > 0)

We send an HTTP HEAD request to example.com. The sendall method accepts only bytes, so we encode the request; HTTP headers are ASCII. The recv method returns bytes; we join the chunks, take the status line, and decode it.

$ python socket_bytes.py
b'HTTP/1.1 200 OK'
HTTP/1.1 200 OK
bytes True

Source

Unicode HOWTO - Python documentation

In this article we have worked with the encode and decode methods in Python.

Author

I am Jan Bodnar, a programmer and technical writer. I have been writing programming tutorials at ZetCode since 2007 and have published eight e-books. See about the author.

List all Python tutorials.