Python lxml
last modified November 16, 2026 Python 3.15
In this article we show how to parse, query, generate, and transform XML and HTML documents in Python with the lxml library.
lxml provides Python bindings for the C libraries libxml2 and
libxslt. It combines their speed and completeness (full XPath 1.0, XSLT, XML
Schema validation, a forgiving HTML parser) with the familiar ElementTree API
of the standard library's xml.etree.ElementTree module. The
lxml.etree module works with XML; the lxml.html
module adds HTML-specific parsing and helpers.
The examples were tested with lxml 6.1 built on libxml2 2.14. CSS selector
support requires the separate cssselect package.
$ pip install lxml cssselect
Sample HTML document
Most of the HTML examples work with the following document.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Words</title>
</head>
<body>
<h1 class="title">Words</h1>
<ul id="words">
<li>sky</li>
<li class="animal">bear</li>
<li>cup</li>
<li class="animal">wolf</li>
<li>water</li>
<li class="animal">falcon</li>
</ul>
<p>
See the <a href="/dict/">dictionary</a> or
<a href="https://example.com/about">about page</a>.
</p>
<div id="output">
No output yet
</div>
</body>
</html>
The document contains a heading, a list of words, some of them with a class,
a paragraph with a relative and an absolute link, and a div with
an id.
Iterating over elements
In the first example, we parse the document and go over all its elements.
from lxml import html
tree = html.parse('words.html')
for e in tree.iter():
print(e.tag, end=' ')
print()
The html.parse function parses a file and returns an
ElementTree. Its iter method walks all elements in
document order, and the tag attribute holds the element name. The
document type declaration is not an element and is not listed.
$ python tags.py html head meta title body h1 ul li li li li li li p a a div
The root element
The root element of a tree is returned by getroot. HTML
elements from lxml.html have extra properties such as
head and body and the text_content
method.
import re
from lxml import html
tree = html.parse('words.html')
root = tree.getroot()
print(root.tag, root.get('lang'))
print(root.head.tag, root.head.findtext('title'))
text = root.body.text_content()
print(re.sub(r'\s+', ' ', text).strip())
We print the root tag with its lang attribute, read with
get. The findtext method returns the text of the
first matching subelement.
text = root.body.text_content() print(re.sub(r'\s+', ' ', text).strip())
The text_content method returns the text of an element and all
its descendants, including the whitespace between the tags. We collapse the
whitespace with a regular expression. Note the raw string: in a normal string,
\s is an invalid escape sequence and produces a
SyntaxWarning.
$ python root.py html en head Words Words sky bear cup wolf water falcon See the dictionary or about page. No output yet
Finding elements with findall
The find, findall, findtext, and
iterfind methods take an ElementPath expression, a limited subset
of XPath supported by the standard ElementTree API.
from lxml import html
root = html.parse('words.html').getroot()
for e in root.findall('body/ul/li'):
print(e.text, end=' ')
print()
print(root.find('.//h1').text)
print(len(root.findall('.//li')))
print([e.text for e in root.iterfind('.//li[@class="animal"]')])
The body/ul/li path is relative to the root element and selects
the list items by their exact location. The .// prefix searches
at any depth below the current element. Predicates in square brackets can test
attributes; iterfind returns the matches lazily.
$ python find_all.py sky bear cup wolf water falcon Words 6 ['bear', 'wolf', 'falcon']
XPath queries
The xpath method evaluates a full XPath 1.0 expression. Depending
on the expression, it returns a list of elements, strings, or attribute
values, or a single number, string, or boolean.
from lxml import html
root = html.parse('words.html').getroot()
print(root.xpath('//li/text()'))
print(root.xpath('//li[@class="animal"]/text()'))
print(root.xpath('//a/@href'))
print(root.xpath('count(//li)'))
print(root.xpath('//li[position() <= 2]/text()'))
print(root.xpath('//li[starts-with(text(), "w")]/text()'))
print(root.xpath('string(//div[@id="output"])').strip())
The text() step selects text nodes and @href
selects attribute values, so the first queries return lists of strings. The
count function returns a float. XPath also supports positional
predicates, string functions such as starts-with, and the
string function that converts the first matching node to its
text.
$ python xpath_query.py ['sky', 'bear', 'cup', 'wolf', 'water', 'falcon'] ['bear', 'wolf', 'falcon'] ['/dict/', 'https://example.com/about'] 6.0 ['sky', 'bear'] ['wolf', 'water'] No output yet
CSS selectors
The cssselect method selects elements with CSS selectors, which
are often shorter than XPath for HTML. lxml translates the selector to XPath
with the cssselect package.
from lxml import html
root = html.parse('words.html').getroot()
print([e.text for e in root.cssselect('li.animal')])
print([e.text for e in root.cssselect('ul#words > li:nth-child(odd)')])
print([a.get('href') for a in root.cssselect('p a[href^="https"]')])
print(root.cssselect('h1.title')[0].text)
We select the list items with the animal class, the odd items of
the list with the words id, the links in a paragraph whose
href starts with https, and the heading with the
title class. The method always returns a list.
$ python css_select.py ['bear', 'wolf', 'falcon'] ['sky', 'cup', 'water'] ['https://example.com/about'] Words
Finding an element by id
HTML elements have the get_element_by_id method, which finds an
element by its id attribute.
from lxml import html
root = html.parse('words.html').getroot()
e = root.get_element_by_id('output')
print(e.tag, e.text.strip())
print(root.get_element_by_id('missing', None))
The program prints the tag and the text of the div element. If
no element has the id, the method raises a KeyError, unless a
default value is given as the second argument.
$ python find_by_id.py div No output yet None
Attributes and links
The get and set methods read and write attributes;
the attrib property is a dictionary-like view of all attributes
of an element. For HTML, lxml has helpers for working with links.
from lxml import html
root = html.parse('words.html').getroot()
for a in root.iter('a'):
print(a.get('href'), '|', a.text_content())
root.make_links_absolute('https://zetcode.com/')
for el, attr, link, pos in root.iterlinks():
print(el.tag, attr, link)
li = root.find('.//li')
li.set('class', 'sky')
print(li.attrib, dict(li.attrib), li.get('title', 'no title'))
We print the target and the text of each link. The
make_links_absolute method resolves relative links against a
base URL; afterwards, iterlinks yields tuples of the element, the
attribute, the link, and its position in the attribute value.
Finally, we set the class attribute of the first list item. The
get method accepts a default value for missing attributes.
$ python attributes.py
/dict/ | dictionary
https://example.com/about | about page
a href https://zetcode.com/dict/
a href https://example.com/about
{'class': 'sky'} {'class': 'sky'} no title
Parsing XML
XML documents are parsed with the lxml.etree module. The
following file contains a list of products.
<?xml version="1.0" encoding="UTF-8"?>
<products>
<product id="1" category="tools">
<name>Hammer</name>
<price currency="EUR">12.50</price>
<stock>34</stock>
</product>
<product id="2" category="garden">
<name>Rake</name>
<price currency="EUR">18.90</price>
<stock>0</stock>
</product>
<product id="3" category="tools">
<name>Screwdriver set</name>
<price currency="EUR">24.00</price>
<stock>12</stock>
</product>
<product id="4" category="garden">
<name>Watering can</name>
<price currency="EUR">9.99</price>
<stock>7</stock>
</product>
</products>
Each product has an id and a category attribute and three child elements.
from lxml import etree
tree = etree.parse('products.xml')
root = tree.getroot()
print(root.tag, len(root))
for product in root:
name = product.findtext('name')
price = float(product.findtext('price'))
print(f'{product.get("id")} {name:<16} {price:>6.2f} {product.get("category")}')
in_stock = root.xpath('//product[stock > 0]/name/text()')
print(in_stock)
total = sum(float(p) for p in root.xpath('//product[@category="tools"]/price/text()'))
print(f'{total:.2f}')
The etree.parse function returns a tree. An element behaves like
a list of its children, so we can call len on the root and
iterate over it. The findtext method reads the text of a child
element; we convert the price to a float.
in_stock = root.xpath('//product[stock > 0]/name/text()')
XPath predicates can compare element values. Here we select the names of the products that are in stock, and then sum the prices of all tools.
$ python parse_xml.py products 4 1 Hammer 12.50 tools 2 Rake 18.90 garden 3 Screwdriver set 24.00 tools 4 Watering can 9.99 garden ['Hammer', 'Screwdriver set', 'Watering can'] 36.50
Parsing from a string
The fromstring function parses XML from a string or bytes and
returns the root element. Malformed input raises an
XMLSyntaxError.
from lxml import etree
data = b'<user id="7"><name>Lucy</name><roles><role>admin</role><role>dev</role></roles></user>'
root = etree.fromstring(data)
print(root.get('id'), root.findtext('name'), [r.text for r in root.iter('role')])
try:
etree.fromstring('<user><name>Lucy</user>')
except etree.XMLSyntaxError as e:
print(f'error: {e}')
We read the attribute, the name, and the roles of a user. The
iter method with a tag name yields only the matching
descendants. The second document has a missing end tag; the exception message
says where the parser found the problem.
A string with an XML declaration that specifies an encoding must be passed as
bytes; fromstring refuses such a str with a
ValueError.
$ python from_string.py 7 Lucy ['admin', 'dev'] error: Opening and ending tag mismatch: name line 1 and user, line 1, column 24 (<string>, line 1)
Namespaces
Elements in XML namespaces have qualified names. lxml writes them in the
Clark notation, {namespace-uri}localname. In XPath
queries, we map prefixes of our choice to the namespace URIs.
from lxml import etree
data = b'''<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:media="http://search.yahoo.com/mrss/">
<title>News</title>
<entry><title>First</title><media:thumbnail url="a.png"/></entry>
<entry><title>Second</title><media:thumbnail url="b.png"/></entry>
</feed>'''
root = etree.fromstring(data)
print(root.tag)
ns = {'a': 'http://www.w3.org/2005/Atom', 'media': 'http://search.yahoo.com/mrss/'}
print(root.xpath('//a:entry/a:title/text()', namespaces=ns))
print(root.xpath('//media:thumbnail/@url', namespaces=ns))
print([e.text for e in root.findall('{http://www.w3.org/2005/Atom}title')])
print(root.findtext('a:entry/a:title', namespaces=ns))
The Atom feed uses a default namespace, so even elements without a prefix belong to it. The root tag is printed in the Clark notation.
ns = {'a': 'http://www.w3.org/2005/Atom', 'media': 'http://search.yahoo.com/mrss/'}
XPath 1.0 has no concept of a default namespace, so we must bind a prefix to
the Atom namespace, here a, and use it in every step. The
findall family accepts either the Clark notation or a
namespaces mapping.
$ python namespaces.py
{http://www.w3.org/2005/Atom}feed
['First', 'Second']
['a.png', 'b.png']
['News']
First
Creating XML documents
The Element and SubElement factories build a
document element by element.
from lxml import etree
users = [('Paul', 34, 'paul@example.com'), ('Lucy', 28, 'lucy@example.com')]
root = etree.Element('users')
for i, (name, age, email) in enumerate(users, start=1):
user = etree.SubElement(root, 'user', id=str(i))
etree.SubElement(user, 'name').text = name
etree.SubElement(user, 'age').text = str(age)
etree.SubElement(user, 'email').text = email
root.append(etree.Comment(' generated by lxml '))
data = etree.tostring(root, pretty_print=True, xml_declaration=True,
encoding='UTF-8')
print(data.decode(), end='')
etree.ElementTree(root).write('users.xml', pretty_print=True,
xml_declaration=True, encoding='UTF-8')
We create a users root element and add a user
element with three children for each record. Keyword arguments of
SubElement become attributes; their values must be strings. We
also append a comment.
data = etree.tostring(root, pretty_print=True, xml_declaration=True,
encoding='UTF-8')
The tostring function serializes an element. With an encoding,
it returns bytes, so we decode them for printing. The write
method of an ElementTree saves the document to a file.
$ python create_xml.py
<?xml version='1.0' encoding='UTF-8'?>
<users>
<user id="1">
<name>Paul</name>
<age>34</age>
<email>paul@example.com</email>
</user>
<user id="2">
<name>Lucy</name>
<age>28</age>
<email>lucy@example.com</email>
</user>
<!-- generated by lxml -->
</users>
Building HTML with the E-factory
The E factory from lxml.html.builder creates
elements with a function call per tag. Nested calls mirror the structure of
the document, which makes generating HTML concise.
from lxml import html
from lxml.html.builder import E
words = ['sky', 'bear', 'cup']
doc = E.html(
E.head(E.title('Generated page')),
E.body(
E.h1('Words', {'class': 'title'}),
E.ul(*[E.li(w) for w in words], id='words'),
E.p('Created with ', E.a('lxml', href='https://lxml.de/'), '.'),
),
lang='en',
)
print(html.tostring(doc, pretty_print=True, doctype='<!DOCTYPE html>').decode(), end='')
Positional arguments of a factory call are children: strings become text and
elements become subelements. A dictionary or keyword arguments become
attributes; a dictionary is needed for class, which is a Python
keyword. The html.tostring function serializes the tree with an
HTML document type.
$ python builder.py <!DOCTYPE html> <html lang="en"> <head><title>Generated page</title></head> <body> <h1 class="title">Words</h1> <ul id="words"> <li>sky</li> <li>bear</li> <li>cup</li> </ul> <p>Created with <a href="https://lxml.de/">lxml</a>.</p> </body> </html>
Modifying a document
Element trees are mutable. We can remove elements, change their text, add new ones, and serialize the result.
from lxml import etree
tree = etree.parse('products.xml')
root = tree.getroot()
for product in root.xpath('//product[stock = 0]'):
root.remove(product)
for price in root.iter('price'):
price.text = f'{float(price.text) * 1.1:.2f}'
new = etree.SubElement(root, 'product', id='5', category='tools')
etree.SubElement(new, 'name').text = 'Saw'
etree.SubElement(new, 'price', currency='EUR').text = '15.40'
etree.SubElement(new, 'stock').text = '3'
etree.indent(root, space=' ')
print(etree.tostring(root, encoding='unicode'))
We remove the products that are out of stock, raise all prices by ten
percent, and append a new product. Elements removed during iteration over an
XPath result are safe to delete, because xpath returns a
list.
etree.indent(root, space=' ')
The indent function re-indents the tree, so the new element is
formatted like the existing ones. With encoding='unicode',
tostring returns a str.
$ python modify.py
<products>
<product id="1" category="tools">
<name>Hammer</name>
<price currency="EUR">13.75</price>
<stock>34</stock>
</product>
<product id="3" category="tools">
<name>Screwdriver set</name>
<price currency="EUR">26.40</price>
<stock>12</stock>
</product>
<product id="4" category="garden">
<name>Watering can</name>
<price currency="EUR">10.99</price>
<stock>7</stock>
</product>
<product id="5" category="tools">
<name>Saw</name>
<price currency="EUR">15.40</price>
<stock>3</stock>
</product>
</products>
Parsing large files with iterparse
Parsing a whole document builds the entire tree in memory. For large files,
iterparse parses the document incrementally and reports elements
as they are completed, so we can process and discard them one by one.
from lxml import etree
count = 0
value = 0.0
for event, elem in etree.iterparse('products.xml', events=('end',), tag='product'):
count += 1
value += float(elem.findtext('price')) * int(elem.findtext('stock'))
elem.clear()
while elem.getprevious() is not None:
del elem.getparent()[0]
print(f'{count} products, stock value {value:.2f} EUR')
With events=('end',) and tag='product', we receive
each product once it has been fully parsed, including its children. We add
its stock value to the total.
elem.clear()
while elem.getprevious() is not None:
del elem.getparent()[0]
The tree is still being built in the background. Clearing the processed element and deleting its preceding siblings from the parent keeps the memory use constant regardless of the file size.
$ python iterparse_big.py 4 products, stock value 782.93 EUR
XSLT transformations
XSLT is a language for transforming XML documents into other documents. lxml runs XSLT 1.0 stylesheets with libxslt. The following stylesheet turns the products into an HTML table.
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="html" indent="yes"/>
<xsl:param name="category" select="'tools'"/>
<xsl:template match="/products">
<table>
<tr><th>Name</th><th>Price</th></tr>
<xsl:for-each select="product[@category = $category]">
<xsl:sort select="price" data-type="number"/>
<tr>
<td><xsl:value-of select="name"/></td>
<td><xsl:value-of select="price"/></td>
</tr>
</xsl:for-each>
</table>
</xsl:template>
</xsl:stylesheet>
The stylesheet selects the products of one category, sorts them by price, and
outputs a table row for each. The category is a parameter with the default
value tools.
from lxml import etree
doc = etree.parse('products.xml')
transform = etree.XSLT(etree.parse('report.xsl'))
result = transform(doc)
print(str(result), end='')
result = transform(doc, category=etree.XSLT.strparam('garden'))
print(str(result), end='')
The etree.XSLT class compiles the stylesheet into a callable
transformation. We apply it to the document twice: once with the default
parameter and once for the garden products. String parameters must be quoted
for XSLT, which XSLT.strparam does safely.
$ python xslt.py <table> <tr> <th>Name</th> <th>Price</th> </tr> <tr> <td>Hammer</td> <td>12.50</td> </tr> <tr> <td>Screwdriver set</td> <td>24.00</td> </tr> </table> <table> <tr> <th>Name</th> <th>Price</th> </tr> <tr> <td>Watering can</td> <td>9.99</td> </tr> <tr> <td>Rake</td> <td>18.90</td> </tr> </table>
Validating with XML Schema
An XML Schema (XSD) describes the allowed structure and data types of a
document. lxml validates documents against a schema with the
XMLSchema class.
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="products">
<xs:complexType>
<xs:sequence>
<xs:element name="product" maxOccurs="unbounded">
<xs:complexType>
<xs:sequence>
<xs:element name="name" type="xs:string"/>
<xs:element name="price">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:decimal">
<xs:attribute name="currency" type="xs:string" use="required"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
<xs:element name="stock" type="xs:nonNegativeInteger"/>
</xs:sequence>
<xs:attribute name="id" type="xs:positiveInteger" use="required"/>
<xs:attribute name="category" type="xs:string"/>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
The schema requires a name, a decimal price with a currency, and a non-negative stock for each product.
from lxml import etree
schema = etree.XMLSchema(etree.parse('products.xsd'))
doc = etree.parse('products.xml')
print(schema.validate(doc))
bad = etree.fromstring(b'''<products>
<product id="1"><name>Hammer</name><price>12.50</price><stock>-3</stock></product>
</products>''')
print(schema.validate(bad))
for error in schema.error_log:
print(f'line {error.line}: {error.message}')
The validate method returns a boolean. The products file is
valid; the second document has a price without a currency and a negative
stock. The error_log attribute lists the problems with their line
numbers. The assertValid method raises an exception instead.
$ python validate.py True False line 2: Element 'price': The attribute 'currency' is required but missing. line 2: Element 'stock': '-3' is not a valid value of the atomic type 'xs:nonNegativeInteger'.
Web scraping
lxml's HTML parser copes with real-world markup, which makes it a good tool for web scraping. To keep the example reproducible, we serve a small page with Python's built-in HTTP server and extract a table from it.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Most populated countries</title>
</head>
<body>
<table id="countries">
<thead>
<tr><th>Rank</th><th>Country</th><th>Population (millions)</th></tr>
</thead>
<tbody>
<tr><td>1</td><td>India</td><td>1 464</td></tr>
<tr><td>2</td><td>China</td><td>1 416</td></tr>
<tr><td>3</td><td>United States</td><td>347</td></tr>
<tr><td>4</td><td>Indonesia</td><td>286</td></tr>
<tr><td>5</td><td>Pakistan</td><td>255</td></tr>
<tr><td>6</td><td>Nigeria</td><td>238</td></tr>
<tr><td>7</td><td>Brazil</td><td>213</td></tr>
<tr><td>8</td><td>Bangladesh</td><td>176</td></tr>
</tbody>
</table>
</body>
</html>
The page contains a table of the most populated countries.
import sys
from urllib.request import urlopen
from lxml import html
url = sys.argv[1]
with urlopen(url, timeout=10) as resp:
doc = html.fromstring(resp.read(), base_url=url)
print(doc.findtext('.//title'))
for row in doc.xpath('//table[@id="countries"]/tbody/tr'):
rank, country, population = (td.text_content().strip() for td in row)
print(f'{rank:>2}. {country:<14} {population:>6}')
The program downloads the page with urllib.request.urlopen and
parses the bytes with html.fromstring; lxml detects the encoding
from the meta tag. The base_url argument would allow
resolving relative links later.
The XPath query selects the rows of the table body. Each row is an element whose children are the cells, so we can unpack them directly into three variables.
$ python scrape.py http://localhost:8000/countries.html Most populated countries 1. India 1 464 2. China 1 416 3. United States 347 4. Indonesia 286 5. Pakistan 255 6. Nigeria 238 7. Brazil 213 8. Bangladesh 176
Source
lxml – XML and HTML with Python
In this article we have processed XML and HTML documents in Python with lxml. See also Python ElementTree, Python BeautifulSoup, and Python pyquery.
Author
List all Python tutorials.