Perl Mojo::DOM

last modified November 7, 2026

Web scraping means downloading pages and pulling structured data out of their HTML. Regular expressions are a poor tool for this, because HTML is nested and written in many equivalent ways. Mojo::DOM, the HTML and XML parser of the Mojolicious framework, turns a document into a tree of elements and lets us query it with CSS selectors, the same language that style sheets and the browser's querySelectorAll use. It has no dependencies outside Mojolicious and handles the broken markup found on real web pages.

The module is part of Mojolicious, installed with cpanm Mojolicious. The examples were run with Perl 5.44 and Mojolicious 9.53 on Linux. Most of them parse a local file, so their output is stable; the last two download live pages, combining the parser with the non-blocking requests from the Mojo::UserAgent async article.

The sample document

The local examples use a small shop page with a navigation menu, a product list, a stock table, and an order form.

shop.html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Green Leaf Shop</title>
</head>
<body>
  <nav id="menu">
    <a href="/">Home</a>
    <a href="/products">Products</a>
    <a href="https://blog.example.com">Blog</a>
    <a href="mailto:info@example.com">Contact</a>
  </nav>

  <h1>Green Leaf Shop</h1>
  <p class="intro">Fresh <b>organic</b> herbs, delivered daily.</p>

  <ul class="products">
    <li class="product" data-id="101"><span class="name">Basil</span>
      <span class="price">2.50</span></li>
    <li class="product sale" data-id="102"><span class="name">Mint</span>
      <span class="price">1.80</span></li>
    <li class="product" data-id="103"><span class="name">Rosemary</span>
      <span class="price">3.20</span></li>
    <li class="product sold-out" data-id="104"><span class="name">Thyme</span>
      <span class="price">2.90</span></li>
  </ul>

  <table id="stock">
    <thead>
      <tr><th>Plant</th><th>Pots</th><th>Price</th></tr>
    </thead>
    <tbody>
      <tr><td>Basil</td><td>40</td><td>2.50</td></tr>
      <tr><td>Mint</td><td>25</td><td>1.80</td></tr>
      <tr><td>Rosemary</td><td>12</td><td>3.20</td></tr>
      <tr><td>Thyme</td><td>0</td><td>2.90</td></tr>
    </tbody>
  </table>

  <form id="order" action="/order" method="post">
    <input type="hidden" name="token" value="a1b2c3">
    <input type="text" name="customer" value="Jana Novak">
    <input type="email" name="email" value="jana@example.com">
    <select name="plant">
      <option value="basil">Basil</option>
      <option value="mint" selected>Mint</option>
      <option value="rosemary">Rosemary</option>
    </select>
    <input type="number" name="pots" value="3">
    <input type="checkbox" name="gift" value="yes" checked>
    <input type="radio" name="delivery" value="post">
    <input type="radio" name="delivery" value="courier" checked>
    <textarea name="note">Leave at the door</textarea>
    <button type="submit">Order</button>
  </form>
</body>
</html>

The products carry classes such as sale and sold-out and a data-id attribute, which gives the selectors something to work with. The form contains the common field types: hidden and text inputs, a select box, a checkbox, radio buttons, and a text area.

Parsing and the at method

A document is parsed by passing the HTML to the constructor. The at method returns the first element that matches a CSS selector, or undef when nothing matches.

parse.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::DOM;
use Mojo::File qw(path);

my $dom = Mojo::DOM->new(path('shop.html')->slurp);

say $dom->at('title')->text;

my $intro = $dom->at('p.intro');

say $intro->text;
say $intro->all_text;
say $intro->at('b')->to_string;
say $intro->tag;

my $first = $dom->at('li.product');

say $first->attr('data-id');
say $first->attr('class');
say $first->at('.name')->text;
say $first->all_text =~ s/\s+/ /gr;

say defined $dom->at('table.missing') ? 'found' : 'no match';

The slurp method of Mojo::File reads the file. The text method returns only the element's own text, so the word inside the b element is missing, while all_text includes the text of all descendants. The to_string method returns the element's markup and tag its name. The attr method reads an attribute; the class attribute is returned as written. Because all_text keeps the whitespace of the source, we collapse it with a substitution. A selector that matches nothing makes at return undef, so the result should be checked before calling a method on it.

$ perl parse.pl
Green Leaf Shop
Fresh  herbs, delivered daily.
Fresh organic herbs, delivered daily.
<b>organic</b>
p
101
product
Basil
Basil 2.50
no match

CSS selectors

The find method returns all matching elements. Mojo::DOM supports nearly all of CSS Selectors Level 3 and parts of Level 4, plus a few extensions.

selectors.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::DOM;
use Mojo::File qw(path);

my $dom = Mojo::DOM->new(path('shop.html')->slurp);

my @selectors = (
    'li.sale .name',
    '#menu a[href^="http"]',
    '#menu a[href^="/"]',
    'li[data-id="103"] .name',
    'li.product:not(.sold-out) .name',
    'ul.products > li:first-child .name',
    'ul.products > li:nth-child(even) .name',
    '#stock td:last-child',
    'li:has(.price) .name',
    'a:text(Blog)',
);

for my $sel (@selectors) {
    my @texts = $dom->find($sel)->map('text')->each;
    printf "%-40s %s\n", $sel, join ', ', @texts;
}

The program runs a list of selectors and prints the text of what they match. A space means a descendant at any depth, > a direct child. Attribute selectors test for a prefix with ^=, which separates absolute links from site-relative ones, or for an exact value with =. Pseudo-classes such as :not, :first-child, :nth-child(even), and :last-child filter by structure; :has selects elements that contain a match. The :text pseudo-class is a Mojo extension that matches elements by the text they contain.

$ perl selectors.pl
li.sale .name                            Mint
#menu a[href^="http"]                    Blog
#menu a[href^="/"]                       Home, Products
li[data-id="103"] .name                  Rosemary
li.product:not(.sold-out) .name          Basil, Mint, Rosemary
ul.products > li:first-child .name       Basil
ul.products > li:nth-child(even) .name   Mint, Thyme
#stock td:last-child                     2.50, 1.80, 3.20, 2.90
li:has(.price) .name                     Basil, Mint, Rosemary, Thyme
a:text(Blog)                             Blog

Collections: each, map, grep

The result of find is a Mojo::Collection, an array object with chainable methods for processing lists.

collection.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::DOM;
use Mojo::File qw(path);

my $dom = Mojo::DOM->new(path('shop.html')->slurp);

my $products = $dom->find('li.product');

say 'count: ', $products->size;

$products->each(sub {
    my ($e, $num) = @_;
    say "$num. ", $e->at('.name')->text, ' ', $e->at('.price')->text;
});

my $names = $products->map(sub { $_->at('.name')->text });
say $names->join(', ');

my $cheap = $products->grep(sub { $_->at('.price')->text < 3 });
say 'cheap: ', $cheap->map(sub { $_->at('.name')->text })->join(', ');

say 'ids: ', $products->map(attr => 'data-id')->join(' ');

my $total = 0;
$total += $_ for $dom->find('.price')->map('text')->each;
say "total: $total";

my $rose = $products->first(sub { $_->at('.name')->text eq 'Rosemary' });
say 'first match: ', $rose->attr('data-id');

say 'sorted: ', $names->sort->reverse->join(' > ');

The each method with a callback passes every element and its position, starting at 1; without a callback it returns the elements as a plain list, which is what the for loop uses. The map method accepts either a code reference, where the element is in $_, or a method name with arguments, as in map(attr => 'data-id'), which calls attr('data-id') on each element. Both map and grep return new collections, so calls can be chained and ended with join. The first method returns the first element for which the callback is true, and sort, reverse, and size work as their names suggest.

$ perl collection.pl
count: 4
1. Basil 2.50
2. Mint 1.80
3. Rosemary 3.20
4. Thyme 2.90
Basil, Mint, Rosemary, Thyme
cheap: Basil, Mint, Thyme
ids: 101 102 103 104
total: 10.4
first match: 103
sorted: Thyme > Rosemary > Mint > Basil

Extracting a table

Tables are a common source of scraped data. The usual approach is to read the header cells once and then turn every body row into a hash keyed by the headers.

table.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::DOM;
use Mojo::File qw(path);

my $dom = Mojo::DOM->new(path('shop.html')->slurp);
my $table = $dom->at('table#stock');

my @headers = $table->find('thead th')->map('text')->each;

my @rows;

for my $tr ($table->find('tbody tr')->each) {
    my @cells = $tr->find('td')->map('all_text')->each;
    my %row;
    @row{@headers} = @cells;
    push @rows, \%row;
}

say join ' | ', @headers;

for my $r (@rows) {
    say join ' | ', @$r{@headers};
}

my @out = grep { $_->{Pots} == 0 } @rows;
say 'out of stock: ', join ', ', map { $_->{Plant} } @out;

my $value = 0;
$value += $_->{Pots} * $_->{Price} for @rows;
printf "stock value: %.2f\n", $value;

The headers come from the th cells in thead. For each row, a hash slice, @row{@headers} = @cells, pairs the header names with the cell texts. We use all_text for cells, since cells often contain links or other markup. Once the data is an array of hashes, it is plain Perl: we filter the plants that are out of stock and compute the value of the stock.

$ perl table.pl
Plant | Pots | Price
Basil | 40 | 2.50
Mint | 25 | 1.80
Rosemary | 12 | 3.20
Thyme | 0 | 2.90
out of stock: Thyme
stock value: 183.40

Extracting form fields

Scrapers often need to read a form, for example to collect the hidden token that must be sent back with a login or search request. The val method returns the value of a form element: the value attribute of an input, the selected option of a select box, or the content of a text area.

form.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::DOM;
use Mojo::File qw(path);

my $dom = Mojo::DOM->new(path('shop.html')->slurp);
my $form = $dom->at('form#order');

say 'action: ', $form->attr('action');
say 'method: ', $form->attr('method');

my %fields;

for my $e ($form->find('input[name], select[name], textarea[name]')->each) {
    my $type = $e->attr('type') // $e->tag;

    next if ($type eq 'checkbox' || $type eq 'radio')
        && !exists $e->attr->{checked};

    $fields{ $e->attr('name') } = $e->val;
}

printf "%-9s %s\n", $_, $fields{$_} for sort keys %fields;

say 'options: ', $form->find('select[name=plant] option')->map(attr => 'value')->join(', ');

The selector picks every named input, select, and textarea element of the form. A checkbox or radio button is sent only when it is checked, so the loop skips the others; calling attr without arguments returns a hash of all attributes, in which the checked attribute exists with an undef value. The result is the same set of name and value pairs a browser would submit; it can be posted with $ua->post($url => form => \%fields).

$ perl form.pl
action: /order
method: post
customer  Jana Novak
delivery  courier
email     jana@example.com
gift      yes
note      Leave at the door
plant     mint
pots      3
token     a1b2c3
options: basil, mint, rosemary

Navigating the tree

Besides selectors, every element offers methods that move through the tree relative to it: to the parent, the siblings, the children, and the ancestors.

navigate.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::DOM;
use Mojo::File qw(path);

my $dom = Mojo::DOM->new(path('shop.html')->slurp);

my $mint = $dom->find('.name')->first(sub { $_->text eq 'Mint' });

say 'parent: ', $mint->parent->tag, ' ', $mint->parent->attr('class');
say 'next sibling: ', $mint->next->text;
say 'next product: ', $mint->parent->next->at('.name')->text;
say 'previous product: ', $mint->parent->previous->at('.name')->text;
say 'ancestors: ', $mint->ancestors->map('tag')->join(' < ');

my $cell = $dom->find('#stock td')->first(sub { $_->text eq 'Rosemary' });
say 'row: ', $cell->parent->children('td')->map('text')->join(', ');
say 'following cells: ', $cell->following('td')->map('text')->join(', ');

say $dom->at('ul.products')->children->size, ' list items';

Starting from the span with the text Mint, parent gives its list item and next the next sibling element, the price. The previous and next methods of the list item reach the neighbouring products. The ancestors method returns a collection running up to the root. The children and following methods accept an optional selector to filter the elements. Navigation is handy when the element we need has no class or id of its own but sits next to one that can be found, for example a value cell next to a label cell.

$ perl navigate.pl
parent: li product sale
next sibling: 1.80
next product: Rosemary
previous product: Basil
ancestors: li < ul < body < html
row: Rosemary, 12, 3.20
following cells: 12, 3.20
4 list items

Modifying a document

The tree can also be changed: elements removed, added, and rewritten. This is useful to clean a page before extracting text, or to rewrite HTML.

modify.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::DOM;

my $html = '<ul id="list"><li>Basil</li><li class="old">Thyme</li>'
    . '<script>track()</script></ul>';

my $dom = Mojo::DOM->new($html);

$dom->find('script')->map('remove');
$dom->at('li.old')->remove;
$dom->at('#list')->append_content('<li>Mint</li>');
$dom->find('li')->each(sub { $_->attr(class => 'herb') });
$dom->at('li')->content('Sweet basil');

say $dom->to_string;
say $dom->find('li')->map('text')->join(', ');

The remove method deletes an element; mapping it over a collection deletes all script elements. The append_content method adds HTML at the end of an element, attr with two arguments sets an attribute, and content replaces an element's content. The to_string method serializes the modified tree.

$ perl modify.pl
<ul id="list"><li class="herb">Sweet basil</li><li class="herb">Mint</li></ul>
Sweet basil, Mint

Scraping a live page

With Mojo::UserAgent, the response of a request can be parsed directly; its dom method returns a Mojo::DOM object for the body. This example reads the list of recent uploads from the CPAN home page.

recent.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::UserAgent;

my $ua = Mojo::UserAgent->new(max_redirects => 3);
my $dom = $ua->get('https://www.cpan.org/')->result->dom;

my $heading = $dom->find('h2')->first(sub { $_->text eq 'Recent Uploads' })
    or die "section not found\n";

for my $link ($heading->next->find('a[href*="/release/"]')->each) {
    my ($author) = $link->attr('href') =~ m{/release/([^/]+)/};
    printf "%-45s %s\n", $link->text, $author;
}

The list has no id or class of its own, so the program finds the h2 heading with the text Recent Uploads and moves to the next sibling element, the ul list, with next. Anchoring a scraper to visible text in this way is often more stable than relying on the layout. The author's PAUSE id is taken from the link with a regular expression, which is fine for a short, well-defined string like a URL path. The list changes every few minutes, so the output will differ.

$ perl recent.pl
CPANPLUS-Dist-Debora-0.019                    VOEGELAS
Async-Microservice-0.09                       JKUTEJ
Data-JSEmail-0.06                             BRONG
Text-JSContact-0.06                           BRONG
Text-JSCalendar-0.10                          BRONG
VPNDetection-3.11.0                           MSLM
Mojolicious-9.54                              SRI
DBD-Mock-Session-GenerateFixtures-1.16        UXYZAB
InternetData-1.12.0                           MSLM
Devel-Cover-Report-Coveralls-1.0.0            MIKIHOSHI

A concurrent scraper

A scraper that visits many pages spends most of its time waiting for the network. The final example takes the links from the CPAN home page, downloads every release page from MetaCPAN concurrently, and extracts the release name and abstract from each.

scraper.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Time::HiRes qw(time);
use Mojo::UserAgent;
use Mojo::Promise;
use Mojo::URL;

my $ua = Mojo::UserAgent->new(max_redirects => 3, request_timeout => 15);
$ua->transactor->name('ZetCodeScraper/1.0');

my $start = Mojo::URL->new('https://www.cpan.org/');
my $dom = $ua->get($start)->result->dom;

my $heading = $dom->find('h2')->first(sub { $_->text eq 'Recent Uploads' })
    or die "section not found\n";

my @urls = $heading->next->find('a[href*="/release/"]')
    ->map(sub { Mojo::URL->new($_->attr('href'))->to_abs($start)->to_string })
    ->each;

say 'scraping ', scalar @urls, ' release pages';

my $begin = time;
my %info;

Mojo::Promise->map({ concurrency => 4 }, sub {
    my $url = $_;

    return $ua->get_p($url)->then(sub {
        my $res = shift->result;
        die $res->code . "\n" unless $res->is_success;

        my $page = $res->dom;
        my $meta = sub {
            my $tag = $page->at(qq{meta[property="og:$_[0]"]});
            return $tag ? $tag->attr('content') : '';
        };

        $info{$url} = sprintf '%-42s %s', $meta->('title'), $meta->('description');
    })->catch(sub {
        $info{$url} = "$url failed: $_[0]";
    });
}, @urls)->wait;

say $info{$_} for @urls;
printf "done in %.1f s\n", time - $begin;

The first page is fetched with a blocking request; the links are made absolute with to_abs. Then Mojo::Promise->map runs a non-blocking get_p for each URL, at most four at a time, which keeps the load on the server reasonable. The then callback parses the response and reads the Open Graph meta tags, which many sites fill with clean titles and descriptions; they are often easier to scrape than the visible page. Any failure, a network error or a non-2xx status, is caught per URL, so one bad page does not stop the rest. The results are stored in a hash and printed in the original order, since the responses arrive in random order. The user agent has a name, set with transactor->name, and a request timeout.

$ perl scraper.pl
scraping 10 release pages
CPANPLUS-Dist-Debora-0.019                 Create Debian or RPM packages from Perl modules
Async-Microservice-0.09                    Async HTTP Microservice Moose Role
Data-JSEmail-0.06                          Convert between Email::MIME and the JSON format from RFC 8621 (JMAP Mail)
Text-JSContact-0.06                        Convert between vCard and JSContact (RFC 9553/9555)
Text-JSCalendar-0.10                       autogenerated timezone data for ME::CalDAVTalk
VPNDetection-3.11.0                        Official Perl client library for the VPNDetection API
Mojolicious-9.54                           Real-time web framework
DBD-Mock-Session-GenerateFixtures-1.16     This distribution generates fixtures for DBD-Mock-Session module
InternetData-1.12.0                        Official Perl client library for the InternetData API
Devel-Cover-Report-Coveralls-1.0.0         coveralls backend for Devel::Cover
done in 0.3 s

Best practices

Check at results: at returns undef when nothing matches; a page redesign must not crash the scraper with a method call on undef.
Prefer stable hooks: ids, data- attributes, meta tags, and visible headings change less often than layout classes and positions.
Use all_text for content: cells and links often contain nested markup that text skips; collapse the whitespace afterwards.
Turn tables into hashes: pair header names with cells so the code refers to columns by name.
Limit concurrency: use Mojo::Promise->map with a small concurrency value and set a timeout.
Be polite: identify the scraper with a User-Agent name, respect robots.txt and the site's terms, and prefer an official API when one exists.

In this article we have parsed HTML with Mojo::DOM, selected elements with CSS selectors, processed them with each, map, grep, and first, extracted tables and form fields, navigated and modified the tree, and built a concurrent scraper with Mojo::UserAgent.

Source

Mojo::DOM on MetaCPAN · Mojo::DOM::CSS · Mojo::Collection

Author

My name is Jan Bodnar, and I am a passionate programmer with extensive programming experience. I have been writing programming articles since 2007. To date, I have authored over 1,400 articles and 8 e-books. I possess more than ten years of experience in teaching programming.

List all Perl tutorials.

Ebooks