ZetCode

Python lxml

last modified November 16, 2026 Python 3.15

In this article we show how to parse, query, generate, and transform XML and HTML documents in Python with the lxml library.

lxml provides Python bindings for the C libraries libxml2 and libxslt. It combines their speed and completeness (full XPath 1.0, XSLT, XML Schema validation, a forgiving HTML parser) with the familiar ElementTree API of the standard library's xml.etree.ElementTree module. The lxml.etree module works with XML; the lxml.html module adds HTML-specific parsing and helpers.

The examples were tested with lxml 6.1 built on libxml2 2.14. CSS selector support requires the separate cssselect package.

$ pip install lxml cssselect

Sample HTML document

Most of the HTML examples work with the following document.

words.html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Words</title>
</head>
<body>

<h1 class="title">Words</h1>

<ul id="words">
    <li>sky</li>
    <li class="animal">bear</li>
    <li>cup</li>
    <li class="animal">wolf</li>
    <li>water</li>
    <li class="animal">falcon</li>
</ul>

<p>
See the <a href="/dict/">dictionary</a> or
<a href="https://example.com/about">about page</a>.
</p>

<div id="output">
    No output yet
</div>

</body>
</html>

The document contains a heading, a list of words, some of them with a class, a paragraph with a relative and an absolute link, and a div with an id.

Iterating over elements

In the first example, we parse the document and go over all its elements.

tags.py
from lxml import html

tree = html.parse('words.html')

for e in tree.iter():
    print(e.tag, end=' ')
print()

The html.parse function parses a file and returns an ElementTree. Its iter method walks all elements in document order, and the tag attribute holds the element name. The document type declaration is not an element and is not listed.

$ python tags.py
html head meta title body h1 ul li li li li li li p a a div 

The root element

The root element of a tree is returned by getroot. HTML elements from lxml.html have extra properties such as head and body and the text_content method.

root.py
import re

from lxml import html

tree = html.parse('words.html')
root = tree.getroot()

print(root.tag, root.get('lang'))
print(root.head.tag, root.head.findtext('title'))

text = root.body.text_content()
print(re.sub(r'\s+', ' ', text).strip())

We print the root tag with its lang attribute, read with get. The findtext method returns the text of the first matching subelement.

text = root.body.text_content()
print(re.sub(r'\s+', ' ', text).strip())

The text_content method returns the text of an element and all its descendants, including the whitespace between the tags. We collapse the whitespace with a regular expression. Note the raw string: in a normal string, \s is an invalid escape sequence and produces a SyntaxWarning.

$ python root.py
html en
head Words
Words sky bear cup wolf water falcon See the dictionary or about page. No output yet

Finding elements with findall

The find, findall, findtext, and iterfind methods take an ElementPath expression, a limited subset of XPath supported by the standard ElementTree API.

find_all.py
from lxml import html

root = html.parse('words.html').getroot()

for e in root.findall('body/ul/li'):
    print(e.text, end=' ')
print()

print(root.find('.//h1').text)
print(len(root.findall('.//li')))
print([e.text for e in root.iterfind('.//li[@class="animal"]')])

The body/ul/li path is relative to the root element and selects the list items by their exact location. The .// prefix searches at any depth below the current element. Predicates in square brackets can test attributes; iterfind returns the matches lazily.

$ python find_all.py
sky bear cup wolf water falcon 
Words
6
['bear', 'wolf', 'falcon']

XPath queries

The xpath method evaluates a full XPath 1.0 expression. Depending on the expression, it returns a list of elements, strings, or attribute values, or a single number, string, or boolean.

xpath_query.py
from lxml import html

root = html.parse('words.html').getroot()

print(root.xpath('//li/text()'))
print(root.xpath('//li[@class="animal"]/text()'))
print(root.xpath('//a/@href'))
print(root.xpath('count(//li)'))
print(root.xpath('//li[position() <= 2]/text()'))
print(root.xpath('//li[starts-with(text(), "w")]/text()'))
print(root.xpath('string(//div[@id="output"])').strip())

The text() step selects text nodes and @href selects attribute values, so the first queries return lists of strings. The count function returns a float. XPath also supports positional predicates, string functions such as starts-with, and the string function that converts the first matching node to its text.

$ python xpath_query.py
['sky', 'bear', 'cup', 'wolf', 'water', 'falcon']
['bear', 'wolf', 'falcon']
['/dict/', 'https://example.com/about']
6.0
['sky', 'bear']
['wolf', 'water']
No output yet

CSS selectors

The cssselect method selects elements with CSS selectors, which are often shorter than XPath for HTML. lxml translates the selector to XPath with the cssselect package.

css_select.py
from lxml import html

root = html.parse('words.html').getroot()

print([e.text for e in root.cssselect('li.animal')])
print([e.text for e in root.cssselect('ul#words > li:nth-child(odd)')])
print([a.get('href') for a in root.cssselect('p a[href^="https"]')])
print(root.cssselect('h1.title')[0].text)

We select the list items with the animal class, the odd items of the list with the words id, the links in a paragraph whose href starts with https, and the heading with the title class. The method always returns a list.

$ python css_select.py
['bear', 'wolf', 'falcon']
['sky', 'cup', 'water']
['https://example.com/about']
Words

Finding an element by id

HTML elements have the get_element_by_id method, which finds an element by its id attribute.

find_by_id.py
from lxml import html

root = html.parse('words.html').getroot()

e = root.get_element_by_id('output')
print(e.tag, e.text.strip())

print(root.get_element_by_id('missing', None))

The program prints the tag and the text of the div element. If no element has the id, the method raises a KeyError, unless a default value is given as the second argument.

$ python find_by_id.py
div No output yet
None

Attributes and links

The get and set methods read and write attributes; the attrib property is a dictionary-like view of all attributes of an element. For HTML, lxml has helpers for working with links.

attributes.py
from lxml import html

root = html.parse('words.html').getroot()

for a in root.iter('a'):
    print(a.get('href'), '|', a.text_content())

root.make_links_absolute('https://zetcode.com/')

for el, attr, link, pos in root.iterlinks():
    print(el.tag, attr, link)

li = root.find('.//li')
li.set('class', 'sky')
print(li.attrib, dict(li.attrib), li.get('title', 'no title'))

We print the target and the text of each link. The make_links_absolute method resolves relative links against a base URL; afterwards, iterlinks yields tuples of the element, the attribute, the link, and its position in the attribute value.

Finally, we set the class attribute of the first list item. The get method accepts a default value for missing attributes.

$ python attributes.py
/dict/ | dictionary
https://example.com/about | about page
a href https://zetcode.com/dict/
a href https://example.com/about
{'class': 'sky'} {'class': 'sky'} no title

Parsing XML

XML documents are parsed with the lxml.etree module. The following file contains a list of products.

products.xml
<?xml version="1.0" encoding="UTF-8"?>
<products>
    <product id="1" category="tools">
        <name>Hammer</name>
        <price currency="EUR">12.50</price>
        <stock>34</stock>
    </product>
    <product id="2" category="garden">
        <name>Rake</name>
        <price currency="EUR">18.90</price>
        <stock>0</stock>
    </product>
    <product id="3" category="tools">
        <name>Screwdriver set</name>
        <price currency="EUR">24.00</price>
        <stock>12</stock>
    </product>
    <product id="4" category="garden">
        <name>Watering can</name>
        <price currency="EUR">9.99</price>
        <stock>7</stock>
    </product>
</products>

Each product has an id and a category attribute and three child elements.

parse_xml.py
from lxml import etree

tree = etree.parse('products.xml')
root = tree.getroot()

print(root.tag, len(root))

for product in root:
    name = product.findtext('name')
    price = float(product.findtext('price'))
    print(f'{product.get("id")} {name:<16} {price:>6.2f} {product.get("category")}')

in_stock = root.xpath('//product[stock > 0]/name/text()')
print(in_stock)

total = sum(float(p) for p in root.xpath('//product[@category="tools"]/price/text()'))
print(f'{total:.2f}')

The etree.parse function returns a tree. An element behaves like a list of its children, so we can call len on the root and iterate over it. The findtext method reads the text of a child element; we convert the price to a float.

in_stock = root.xpath('//product[stock > 0]/name/text()')

XPath predicates can compare element values. Here we select the names of the products that are in stock, and then sum the prices of all tools.

$ python parse_xml.py
products 4
1 Hammer            12.50 tools
2 Rake              18.90 garden
3 Screwdriver set   24.00 tools
4 Watering can       9.99 garden
['Hammer', 'Screwdriver set', 'Watering can']
36.50

Parsing from a string

The fromstring function parses XML from a string or bytes and returns the root element. Malformed input raises an XMLSyntaxError.

from_string.py
from lxml import etree

data = b'<user id="7"><name>Lucy</name><roles><role>admin</role><role>dev</role></roles></user>'

root = etree.fromstring(data)
print(root.get('id'), root.findtext('name'), [r.text for r in root.iter('role')])

try:
    etree.fromstring('<user><name>Lucy</user>')
except etree.XMLSyntaxError as e:
    print(f'error: {e}')

We read the attribute, the name, and the roles of a user. The iter method with a tag name yields only the matching descendants. The second document has a missing end tag; the exception message says where the parser found the problem.

A string with an XML declaration that specifies an encoding must be passed as bytes; fromstring refuses such a str with a ValueError.

$ python from_string.py
7 Lucy ['admin', 'dev']
error: Opening and ending tag mismatch: name line 1 and user, line 1, column 24 (<string>, line 1)

Namespaces

Elements in XML namespaces have qualified names. lxml writes them in the Clark notation, {namespace-uri}localname. In XPath queries, we map prefixes of our choice to the namespace URIs.

namespaces.py
from lxml import etree

data = b'''<feed xmlns="http://www.w3.org/2005/Atom"
      xmlns:media="http://search.yahoo.com/mrss/">
  <title>News</title>
  <entry><title>First</title><media:thumbnail url="a.png"/></entry>
  <entry><title>Second</title><media:thumbnail url="b.png"/></entry>
</feed>'''

root = etree.fromstring(data)
print(root.tag)

ns = {'a': 'http://www.w3.org/2005/Atom', 'media': 'http://search.yahoo.com/mrss/'}

print(root.xpath('//a:entry/a:title/text()', namespaces=ns))
print(root.xpath('//media:thumbnail/@url', namespaces=ns))
print([e.text for e in root.findall('{http://www.w3.org/2005/Atom}title')])
print(root.findtext('a:entry/a:title', namespaces=ns))

The Atom feed uses a default namespace, so even elements without a prefix belong to it. The root tag is printed in the Clark notation.

ns = {'a': 'http://www.w3.org/2005/Atom', 'media': 'http://search.yahoo.com/mrss/'}

XPath 1.0 has no concept of a default namespace, so we must bind a prefix to the Atom namespace, here a, and use it in every step. The findall family accepts either the Clark notation or a namespaces mapping.

$ python namespaces.py
{http://www.w3.org/2005/Atom}feed
['First', 'Second']
['a.png', 'b.png']
['News']
First

Creating XML documents

The Element and SubElement factories build a document element by element.

create_xml.py
from lxml import etree

users = [('Paul', 34, 'paul@example.com'), ('Lucy', 28, 'lucy@example.com')]

root = etree.Element('users')

for i, (name, age, email) in enumerate(users, start=1):
    user = etree.SubElement(root, 'user', id=str(i))
    etree.SubElement(user, 'name').text = name
    etree.SubElement(user, 'age').text = str(age)
    etree.SubElement(user, 'email').text = email

root.append(etree.Comment(' generated by lxml '))

data = etree.tostring(root, pretty_print=True, xml_declaration=True,
                      encoding='UTF-8')
print(data.decode(), end='')

etree.ElementTree(root).write('users.xml', pretty_print=True,
                              xml_declaration=True, encoding='UTF-8')

We create a users root element and add a user element with three children for each record. Keyword arguments of SubElement become attributes; their values must be strings. We also append a comment.

data = etree.tostring(root, pretty_print=True, xml_declaration=True,
                      encoding='UTF-8')

The tostring function serializes an element. With an encoding, it returns bytes, so we decode them for printing. The write method of an ElementTree saves the document to a file.

$ python create_xml.py
<?xml version='1.0' encoding='UTF-8'?>
<users>
  <user id="1">
    <name>Paul</name>
    <age>34</age>
    <email>paul@example.com</email>
  </user>
  <user id="2">
    <name>Lucy</name>
    <age>28</age>
    <email>lucy@example.com</email>
  </user>
  <!-- generated by lxml -->
</users>

Building HTML with the E-factory

The E factory from lxml.html.builder creates elements with a function call per tag. Nested calls mirror the structure of the document, which makes generating HTML concise.

builder.py
from lxml import html
from lxml.html.builder import E

words = ['sky', 'bear', 'cup']

doc = E.html(
    E.head(E.title('Generated page')),
    E.body(
        E.h1('Words', {'class': 'title'}),
        E.ul(*[E.li(w) for w in words], id='words'),
        E.p('Created with ', E.a('lxml', href='https://lxml.de/'), '.'),
    ),
    lang='en',
)

print(html.tostring(doc, pretty_print=True, doctype='<!DOCTYPE html>').decode(), end='')

Positional arguments of a factory call are children: strings become text and elements become subelements. A dictionary or keyword arguments become attributes; a dictionary is needed for class, which is a Python keyword. The html.tostring function serializes the tree with an HTML document type.

$ python builder.py
<!DOCTYPE html>
<html lang="en">
<head><title>Generated page</title></head>
<body>
<h1 class="title">Words</h1>
<ul id="words">
<li>sky</li>
<li>bear</li>
<li>cup</li>
</ul>
<p>Created with <a href="https://lxml.de/">lxml</a>.</p>
</body>
</html>

Modifying a document

Element trees are mutable. We can remove elements, change their text, add new ones, and serialize the result.

modify.py
from lxml import etree

tree = etree.parse('products.xml')
root = tree.getroot()

for product in root.xpath('//product[stock = 0]'):
    root.remove(product)

for price in root.iter('price'):
    price.text = f'{float(price.text) * 1.1:.2f}'

new = etree.SubElement(root, 'product', id='5', category='tools')
etree.SubElement(new, 'name').text = 'Saw'
etree.SubElement(new, 'price', currency='EUR').text = '15.40'
etree.SubElement(new, 'stock').text = '3'

etree.indent(root, space='    ')
print(etree.tostring(root, encoding='unicode'))

We remove the products that are out of stock, raise all prices by ten percent, and append a new product. Elements removed during iteration over an XPath result are safe to delete, because xpath returns a list.

etree.indent(root, space='    ')

The indent function re-indents the tree, so the new element is formatted like the existing ones. With encoding='unicode', tostring returns a str.

$ python modify.py
<products>
    <product id="1" category="tools">
        <name>Hammer</name>
        <price currency="EUR">13.75</price>
        <stock>34</stock>
    </product>
    <product id="3" category="tools">
        <name>Screwdriver set</name>
        <price currency="EUR">26.40</price>
        <stock>12</stock>
    </product>
    <product id="4" category="garden">
        <name>Watering can</name>
        <price currency="EUR">10.99</price>
        <stock>7</stock>
    </product>
    <product id="5" category="tools">
        <name>Saw</name>
        <price currency="EUR">15.40</price>
        <stock>3</stock>
    </product>
</products>

Parsing large files with iterparse

Parsing a whole document builds the entire tree in memory. For large files, iterparse parses the document incrementally and reports elements as they are completed, so we can process and discard them one by one.

iterparse_big.py
from lxml import etree

count = 0
value = 0.0

for event, elem in etree.iterparse('products.xml', events=('end',), tag='product'):
    count += 1
    value += float(elem.findtext('price')) * int(elem.findtext('stock'))

    elem.clear()
    while elem.getprevious() is not None:
        del elem.getparent()[0]

print(f'{count} products, stock value {value:.2f} EUR')

With events=('end',) and tag='product', we receive each product once it has been fully parsed, including its children. We add its stock value to the total.

elem.clear()
while elem.getprevious() is not None:
    del elem.getparent()[0]

The tree is still being built in the background. Clearing the processed element and deleting its preceding siblings from the parent keeps the memory use constant regardless of the file size.

$ python iterparse_big.py
4 products, stock value 782.93 EUR

XSLT transformations

XSLT is a language for transforming XML documents into other documents. lxml runs XSLT 1.0 stylesheets with libxslt. The following stylesheet turns the products into an HTML table.

report.xsl
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="html" indent="yes"/>
  <xsl:param name="category" select="'tools'"/>

  <xsl:template match="/products">
    <table>
      <tr><th>Name</th><th>Price</th></tr>
      <xsl:for-each select="product[@category = $category]">
        <xsl:sort select="price" data-type="number"/>
        <tr>
          <td><xsl:value-of select="name"/></td>
          <td><xsl:value-of select="price"/></td>
        </tr>
      </xsl:for-each>
    </table>
  </xsl:template>
</xsl:stylesheet>

The stylesheet selects the products of one category, sorts them by price, and outputs a table row for each. The category is a parameter with the default value tools.

xslt.py
from lxml import etree

doc = etree.parse('products.xml')
transform = etree.XSLT(etree.parse('report.xsl'))

result = transform(doc)
print(str(result), end='')

result = transform(doc, category=etree.XSLT.strparam('garden'))
print(str(result), end='')

The etree.XSLT class compiles the stylesheet into a callable transformation. We apply it to the document twice: once with the default parameter and once for the garden products. String parameters must be quoted for XSLT, which XSLT.strparam does safely.

$ python xslt.py
<table>
<tr>
<th>Name</th>
<th>Price</th>
</tr>
<tr>
<td>Hammer</td>
<td>12.50</td>
</tr>
<tr>
<td>Screwdriver set</td>
<td>24.00</td>
</tr>
</table>
<table>
<tr>
<th>Name</th>
<th>Price</th>
</tr>
<tr>
<td>Watering can</td>
<td>9.99</td>
</tr>
<tr>
<td>Rake</td>
<td>18.90</td>
</tr>
</table>

Validating with XML Schema

An XML Schema (XSD) describes the allowed structure and data types of a document. lxml validates documents against a schema with the XMLSchema class.

products.xsd
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="products">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="product" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="name" type="xs:string"/>
              <xs:element name="price">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:decimal">
                      <xs:attribute name="currency" type="xs:string" use="required"/>
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
              </xs:element>
              <xs:element name="stock" type="xs:nonNegativeInteger"/>
            </xs:sequence>
            <xs:attribute name="id" type="xs:positiveInteger" use="required"/>
            <xs:attribute name="category" type="xs:string"/>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

The schema requires a name, a decimal price with a currency, and a non-negative stock for each product.

validate.py
from lxml import etree

schema = etree.XMLSchema(etree.parse('products.xsd'))

doc = etree.parse('products.xml')
print(schema.validate(doc))

bad = etree.fromstring(b'''<products>
  <product id="1"><name>Hammer</name><price>12.50</price><stock>-3</stock></product>
</products>''')

print(schema.validate(bad))

for error in schema.error_log:
    print(f'line {error.line}: {error.message}')

The validate method returns a boolean. The products file is valid; the second document has a price without a currency and a negative stock. The error_log attribute lists the problems with their line numbers. The assertValid method raises an exception instead.

$ python validate.py
True
False
line 2: Element 'price': The attribute 'currency' is required but missing.
line 2: Element 'stock': '-3' is not a valid value of the atomic type 'xs:nonNegativeInteger'.

Web scraping

lxml's HTML parser copes with real-world markup, which makes it a good tool for web scraping. To keep the example reproducible, we serve a small page with Python's built-in HTTP server and extract a table from it.

site/countries.html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Most populated countries</title>
</head>
<body>

<table id="countries">
<thead>
<tr><th>Rank</th><th>Country</th><th>Population (millions)</th></tr>
</thead>
<tbody>
<tr><td>1</td><td>India</td><td>1 464</td></tr>
<tr><td>2</td><td>China</td><td>1 416</td></tr>
<tr><td>3</td><td>United States</td><td>347</td></tr>
<tr><td>4</td><td>Indonesia</td><td>286</td></tr>
<tr><td>5</td><td>Pakistan</td><td>255</td></tr>
<tr><td>6</td><td>Nigeria</td><td>238</td></tr>
<tr><td>7</td><td>Brazil</td><td>213</td></tr>
<tr><td>8</td><td>Bangladesh</td><td>176</td></tr>
</tbody>
</table>

</body>
</html>

The page contains a table of the most populated countries.

scrape.py
import sys
from urllib.request import urlopen

from lxml import html

url = sys.argv[1]

with urlopen(url, timeout=10) as resp:
    doc = html.fromstring(resp.read(), base_url=url)

print(doc.findtext('.//title'))

for row in doc.xpath('//table[@id="countries"]/tbody/tr'):
    rank, country, population = (td.text_content().strip() for td in row)
    print(f'{rank:>2}. {country:<14} {population:>6}')

The program downloads the page with urllib.request.urlopen and parses the bytes with html.fromstring; lxml detects the encoding from the meta tag. The base_url argument would allow resolving relative links later.

The XPath query selects the rows of the table body. Each row is an element whose children are the cells, so we can unpack them directly into three variables.

$ python scrape.py http://localhost:8000/countries.html
Most populated countries
 1. India           1 464
 2. China           1 416
 3. United States     347
 4. Indonesia         286
 5. Pakistan          255
 6. Nigeria           238
 7. Brazil            213
 8. Bangladesh        176

Source

lxml – XML and HTML with Python

The lxml.etree tutorial

In this article we have processed XML and HTML documents in Python with lxml. See also Python ElementTree, Python BeautifulSoup, and Python pyquery.

Author

I am Jan Bodnar, a programmer and technical writer. I have been writing programming tutorials at ZetCode since 2007 and have published eight e-books. See about the author.

List all Python tutorials.