Perl Mojo::UserAgent async

last modified November 7, 2026

Most of the time an HTTP client spends is time waiting: for DNS, for the TLS handshake, and for the server to answer. A blocking client waits for each request in turn. Mojo::UserAgent, the HTTP client of the Mojolicious framework, can instead start many requests at once and handle each response when it arrives, all in a single process. It does so with an event loop, Mojo::IOLoop, and offers three styles on top of it: callbacks, promises, and async/await. In this article we use all three, combine requests with all, race, and any, limit the number of requests in flight, and handle timeouts and errors.

The modules are installed from CPAN with cpanm Mojolicious IO::Socket::SSL Future::AsyncAwait; IO::Socket::SSL adds HTTPS support and Future::AsyncAwait enables the async and await keywords. The examples were run with Perl 5.44, Mojolicious 9.53, and Future::AsyncAwait 0.71 on Linux. They talk to real servers, including httpbin.org, whose /delay/n endpoint answers after n seconds, so timings vary a little between runs.

A blocking request

Without a callback, get is blocking: it returns a transaction object after the response has arrived. The result method returns the response, or dies if the connection itself failed. The response gives access to the status line, the headers, and, for HTML documents, a parsed Mojo::DOM tree.

blocking.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::UserAgent;

my $ua = Mojo::UserAgent->new;

my $res = $ua->get('https://example.com')->result;

say $res->code, ' ', $res->message;
say $res->headers->content_type;
say $res->dom->at('title')->text;

The program fetches one page and prints its status code and message, the Content-Type header, and the text of the title element, which at finds with a CSS selector. This style is simple and fine for a single request; the rest of the article shows what to do when there are many.

$ perl blocking.pl
200 OK
text/html; charset=utf-8
Example Domain

The event loop

Asynchronous code in Mojolicious runs inside Mojo::IOLoop. The program registers interest in events, such as a timer expiring or a socket becoming readable, together with a callback, and then starts the loop. The loop waits for events and runs the matching callbacks; it stops by itself when there is nothing left to wait for.

timers.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::IOLoop;

for my $delay (3, 1, 2) {
    Mojo::IOLoop->timer($delay => sub {
        say "timer of $delay s fired";
    });
}

say 'timers registered';

Mojo::IOLoop->start;

say 'event loop stopped';

Three timers are registered in the order 3, 1, and 2 seconds. Registering them returns at once, so timers registered is printed first. Then start runs the loop: the callbacks fire in the order their timers expire, and when the last timer is gone the loop has no more work and start returns. The whole program takes three seconds, not six, because the timers count down at the same time.

$ perl timers.pl
timers registered
timer of 1 s fired
timer of 2 s fired
timer of 3 s fired
event loop stopped

A non-blocking request with a callback

Passing a callback as the last argument makes get non-blocking. It returns immediately, and the callback receives the user agent and the finished transaction once the response has arrived. Because the request is processed by the event loop, the loop has to be running.

callback.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::UserAgent;
use Mojo::IOLoop;

my $ua = Mojo::UserAgent->new;

$ua->get('https://www.perl.org' => sub {
    my ($ua, $tx) = @_;

    my $res = $tx->result;
    say $res->code, ' ', $res->dom->at('title')->text;
});

say 'request sent';

Mojo::IOLoop->start unless Mojo::IOLoop->is_running;

The message request sent appears before the response, which shows that get did not wait. The guard unless Mojo::IOLoop->is_running is a common idiom: it starts the loop in a stand-alone script, but leaves it alone when the same code runs inside a Mojolicious web application, where the loop is already running.

$ perl callback.pl
request sent
200 The Perl Programming Language - www.perl.org

Running requests concurrently

The real benefit appears with several requests. The next program asks httpbin.org for three delayed responses, first one after another with blocking calls, then all at once with callbacks.

concurrent.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Time::HiRes qw(time);
use Mojo::UserAgent;
use Mojo::IOLoop;

my $ua = Mojo::UserAgent->new;
my @urls = map { "https://httpbin.org/delay/$_" } 2, 1, 3;

my $begin = time;

for my $url (@urls) {
    $ua->get($url)->result;
}

printf "blocking: %.1f s\n", time - $begin;

$begin = time;
my $pending = @urls;

for my $url (@urls) {
    $ua->get($url => sub {
        my ($ua, $tx) = @_;

        printf "%s done after %.1f s\n", $tx->req->url->path, time - $begin;
        Mojo::IOLoop->stop unless --$pending;
    });
}

Mojo::IOLoop->start;

printf "non-blocking: %.1f s\n", time - $begin;

The blocking loop takes the sum of the delays plus the overhead of each request. In the non-blocking part all three requests are in flight together, and the responses come back in the order the server finishes them, not in the order they were sent. The counter $pending tracks unfinished requests; the last callback calls Mojo::IOLoop->stop. The total time is about the time of the slowest request.

$ perl concurrent.pl
blocking: 6.9 s
/delay/1 done after 1.6 s
/delay/2 done after 3.0 s
/delay/3 done after 3.6 s
non-blocking: 3.6 s

Promises

Callbacks nest quickly when one step depends on another. Every non-blocking method of Mojo::UserAgent also has a _p variant, such as get_p, that returns a Mojo::Promise. A promise stands for a value that is not available yet. The then method registers a handler for success, catch one for failure, and both return a new promise, so the steps form a flat chain.

promise.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::UserAgent;

my $ua = Mojo::UserAgent->new;

$ua->get_p('https://metacpan.org')->then(sub {
    my $tx = shift;

    my $res = $tx->result;
    return $res->dom->at('title')->text;
})->then(sub {
    my $title = shift;

    say "title: $title";
})->catch(sub {
    my $err = shift;

    warn "request failed: $err\n";
})->wait;

The first handler receives the transaction and returns the page title; the value it returns becomes the input of the next then. If the connection fails, the promise is rejected and the chain skips to catch. At the end, wait starts the event loop and returns once the promise has settled, so no explicit Mojo::IOLoop->start is needed.

$ perl promise.pl
title: Search the CPAN - metacpan.org

Waiting for all requests

Mojo::Promise->all takes several promises and returns a new one that is fulfilled when all of them are. Its handler receives one array reference per promise, in the order the promises were given, holding the values that promise was fulfilled with. If any promise is rejected, the combined promise is rejected at once with the same reason.

all.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::UserAgent;
use Mojo::Promise;

my $ua = Mojo::UserAgent->new;

my @urls = qw(
    https://www.perl.org
    https://metacpan.org
    https://example.com
);

my @promises = map { $ua->get_p($_) } @urls;

Mojo::Promise->all(@promises)->then(sub {
    my @results = @_;

    for my $result (@results) {
        my $tx = $result->[0];
        printf "%-24s %s\n", $tx->req->url->host, $tx->result->dom->at('title')->text;
    }
})->catch(sub {
    my $err = shift;
    warn "one of the requests failed: $err\n";
})->wait;

The three requests run concurrently, yet the results come back in the order of @urls, which makes them easy to match to their inputs. The transaction is the first element of each array reference, and its request object still knows which URL it was sent to.

$ perl all.pl
www.perl.org             The Perl Programming Language - www.perl.org
metacpan.org             Search the CPAN - metacpan.org
example.com              Example Domain

Handling errors

There are two kinds of failure. A connection error, such as an unknown host or a refused connection, rejects the promise. An HTTP error, such as 404 or 500, is still a valid response: the promise is fulfilled, and the status code has to be checked with is_success or is_error. Because all gives up at the first rejection, Mojo::Promise->all_settled is better for a batch where every outcome matters. It waits for all promises and describes each one with a hash holding status and either value or reason.

settled.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::UserAgent;
use Mojo::Promise;

my $ua = Mojo::UserAgent->new;

my @urls = qw(
    https://example.com
    https://httpbin.org/status/404
    https://no-such-host.invalid
);

my @promises = map { $ua->get_p($_) } @urls;

Mojo::Promise->all_settled(@promises)->then(sub {
    my @outcomes = @_;

    for my $i (0 .. $#urls) {
        my $outcome = $outcomes[$i];

        if ($outcome->{status} eq 'rejected') {
            say "$urls[$i]: connection error: $outcome->{reason}[0]";
            next;
        }

        my $res = $outcome->{value}[0]->result;

        if ($res->is_success) {
            say "$urls[$i]: ok, ", length $res->body, " bytes";
        }
        else {
            say "$urls[$i]: HTTP error ", $res->code;
        }
    }
})->wait;

The list contains a working page, a URL that answers 404, and a host in the reserved .invalid domain, which never resolves. Like value, the reason field is an array reference, and its first element is the error message. The program reports one line per URL, in the original order, whatever happened to each request.

$ perl settled.pl
https://example.com: ok, 577 bytes
https://httpbin.org/status/404: HTTP error 404
https://no-such-host.invalid: connection error: Can't connect: Name or service not known

Timeouts

A slow server should not hold a program forever. The request_timeout attribute limits the total time of a request, including connecting and reading the whole response. When it runs out, the promise is rejected with the message Request timeout. The user agent also has connect_timeout and inactivity_timeout for the individual phases.

timeout.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::UserAgent;
use Mojo::Promise;

my $ua = Mojo::UserAgent->new(request_timeout => 3);

my @promises = map {
    my $url = "https://httpbin.org/delay/$_";

    $ua->get_p($url)->then(
        sub { "$url: " . $_[0]->result->code },
        sub { "$url: $_[0]" },
    );
} 1, 5;

Mojo::Promise->all(@promises)->then(sub {
    say $_->[0] for @_;
})->wait;

Passing two handlers to then handles both outcomes at once: the first runs on success, the second on failure, and either one turns the outcome into a line of text. Since every promise is now fulfilled, all cannot fail. The one-second request finishes within the three-second limit; the five-second one is cancelled.

$ perl timeout.pl
https://httpbin.org/delay/1: 200
https://httpbin.org/delay/5: Request timeout

race, any, and timeout

Sometimes only the first answer matters, for instance when the same file is available from several mirrors. Mojo::Promise->race settles as soon as the first promise settles, with its value or its reason. Mojo::Promise->any waits for the first promise to be fulfilled and is rejected only when all of them fail. The class method timeout creates a promise that is rejected after a given time, which can be raced against other promises.

race.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::UserAgent;
use Mojo::Promise;

my $ua = Mojo::UserAgent->new;

my %mirrors = (
    slow    => 'https://httpbin.org/delay/3',
    medium  => 'https://httpbin.org/delay/1',
    broken  => 'https://no-such-host.invalid',
);

my @promises = map {
    my $name = $_;
    $ua->get_p($mirrors{$name})->then(sub { $name });
} sort keys %mirrors;

Mojo::Promise->race(@promises)->then(
    sub { say "race: $_[0] settled first" },
    sub { say "race: failed with $_[0]" },
)->wait;

Mojo::Promise->any(@promises)->then(
    sub { say "any: $_[0] answered first" },
    sub { say "any: all requests failed" },
)->wait;

Mojo::Promise->timeout(0.5 => 'gave up after 0.5 s')->then(
    sub { say 'timeout: done' },
    sub { say "timeout: $_[0]" },
)->wait;

The three requests stand for mirrors: one is slow, one answers after a second, and one points to a host that does not exist. race settles with the broken mirror, because its DNS error arrives before any response. any ignores the failure and returns the medium mirror, the first that answered. The last call shows the timeout promise on its own; it is rejected after half a second with the given message.

$ perl race.pl
race: failed with Can't connect: Name or service not known
any: medium answered first
timeout: gave up after 0.5 s

Limiting concurrency

Starting a thousand requests at once overloads both the client and the server, and many sites block clients that open too many connections. Mojo::Promise->map calls a function for each item of a list, like map, but the function returns a promise, and the concurrency option limits how many of them may be pending at the same time. The current item is in $_.

map.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Time::HiRes qw(time);
use Mojo::UserAgent;
use Mojo::Promise;

my $ua = Mojo::UserAgent->new;
my $begin = time;

Mojo::Promise->map({ concurrency => 2 }, sub {
    my $job = $_;

    printf "%.1f s start job %d\n", time - $begin, $job;

    return $ua->get_p('https://httpbin.org/delay/1')->then(sub {
        printf "%.1f s end   job %d\n", time - $begin, $job;
        return $_[0]->result->code;
    });
}, 1 .. 5)->then(sub {
    my @codes = map { $_->[0] } @_;
    say "status codes: @codes";
})->wait;

Five one-second jobs run with at most two in flight. The timestamps show that a new job starts only when one of the running jobs ends, so the jobs run in three rounds. The promise returned by map behaves like the one from all: its values are array references in the order of the input list.

$ perl map.pl
0.0 s start job 1
0.0 s start job 2
1.5 s end   job 1
1.5 s start job 3
1.6 s end   job 2
1.6 s start job 4
2.7 s end   job 3
2.7 s start job 5
2.7 s end   job 4
3.9 s end   job 5
status codes: 200 200 200 200 200

async and await

With Future::AsyncAwait installed, the -async_await flag of Mojo::Base adds the async and await keywords. An async sub always returns a promise, and inside it await suspends the sub until a promise settles, without blocking the event loop. Asynchronous code then reads like ordinary sequential code, with return for results and die for errors.

await.pl

#!/usr/bin/perl

use Mojo::Base -strict, -async_await;
use v5.34.0;
use Mojo::UserAgent;
use Mojo::Promise;

my $ua = Mojo::UserAgent->new;

async sub title_of {
    my $url = shift;

    my $tx = await $ua->get_p($url);
    my $res = $tx->result;

    die "$url: HTTP ", $res->code, "\n" unless $res->is_success;
    return $res->dom->at('title')->text;
}

async sub main {
    my @urls = qw(https://www.perl.org https://example.com);

    my @titles = await Mojo::Promise->all(map { title_of($_) } @urls);
    say $_->[0] for @titles;

    eval {
        await title_of('https://httpbin.org/status/500');
        1;
    } or print "error: $@";
}

main()->wait;

The -strict flag turns on strict, warnings, and utf8. The title_of sub awaits the response and dies on an HTTP error, which rejects the promise it returns. In main, await is applied to the combined promise from all, so both titles are fetched concurrently. A rejected await throws, so ordinary eval catches the error from the 500 response. await cannot be used at the top level of a script, which is why the code lives in main and the script ends with main()->wait.

$ perl await.pl
The Perl Programming Language - www.perl.org
Example Domain
error: https://httpbin.org/status/500: HTTP 500

Practical example: downloading web pages

The Parallel::ForkManager article downloads a list of pages with a pool of forked workers. The same task fits in one process with Mojo::UserAgent: Mojo::Promise->map keeps four downloads in flight, and Mojo::File reads the list and writes the pages. The URLs come from a text file.

urls.txt

https://example.com
https://httpbin.org/get
https://www.perl.org
https://metacpan.org
https://www.python.org
https://www.php.net
https://github.com
https://httpbin.org/status/404

The list mixes pages that return content with one URL that answers 404, so the failure path is exercised as well.

download.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::File qw(path);
use Mojo::URL;
use Mojo::UserAgent;
use Mojo::Promise;

my @urls = grep { /\S/ } split /\n/, path('urls.txt')->slurp;

my $ua = Mojo::UserAgent->new(max_redirects => 3, request_timeout => 10);
$ua->transactor->name('Perl script');

my $dir = path('files')->make_path;

say 'downloading ', scalar @urls, ' files';

Mojo::Promise->map({ concurrency => 4 }, sub {
    my $url = Mojo::URL->new($_);

    my $name = $url->host =~ s/^www\.//r;
    $name .= $url->path =~ tr{/}{_}r if $url->path ne '' && $url->path ne '/';

    return $ua->get_p($url)->then(sub {
        my $res = shift->result;

        if ($res->is_success) {
            $dir->child("$name.txt")->spew($res->body);
            return 1;
        }

        warn "$url: ", $res->code, ' ', $res->message, "\n";
        return 0;
    })->catch(sub {
        warn "$url: $_[0]\n";
        return 0;
    });
}, @urls)->then(sub {
    my $saved = grep { $_->[0] } @_;
    say "saved $saved of ", scalar @urls, ' files';
})->wait;

The user agent follows up to three redirects and gives each request ten seconds; transactor->name sets the User-Agent header. The file name comes from the parts of the Mojo::URL object: the host loses its www. prefix, and a path such as /get becomes _get. Each job returns 1 for a saved page and 0 for a failure, with catch turning connection errors into 0 as well, so one bad URL does not reject the whole batch. The final handler counts the successes. Unlike the forking version, the work happens in one process, and the results come back to the main program without any serialization.

$ perl download.pl
downloading 8 files
https://httpbin.org/status/404: 404 Not Found
saved 7 of 8 files
$ ls -1 files/
example.com.txt
github.com.txt
httpbin.org_get.txt
metacpan.org.txt
perl.org.txt
php.net.txt
python.org.txt

Practical example: checking links

A link checker shows blocking and non-blocking code side by side. The page itself is fetched with a blocking call, since nothing can happen before it arrives. Its links are then checked concurrently with HEAD requests, which ask only for the headers and so transfer far less data than GET.

links.pl

#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Mojo::URL;
use Mojo::UserAgent;
use Mojo::Promise;

my $page = Mojo::URL->new(shift // 'https://www.perl.org');

my $ua = Mojo::UserAgent->new(max_redirects => 5, request_timeout => 10);

my $res = $ua->get($page)->result;
die "$page: ", $res->code, "\n" unless $res->is_success;

my %seen;
my @links = grep { /^https?:/ && !$seen{$_}++ }
            map  { Mojo::URL->new($_)->to_abs($page)->fragment(undef)->to_string }
            $res->dom->find('a[href]')->map(attr => 'href')->each;

say 'checking ', scalar @links, " links on $page";
exit unless @links;

my %status;

Mojo::Promise->map({ concurrency => 6 }, sub {
    my $link = $_;

    return $ua->head_p($link)->then(sub {
        my $tx = shift;
        return $tx->result->code == 405 ? $ua->get_p($link) : $tx;
    })->then(sub {
        $status{$link} = $_[0]->result->code;
    })->catch(sub {
        $status{$link} = "error: $_[0]";
    });
}, @links)->then(sub {
    my @broken = grep { $status{$_} !~ /^2/ } sort keys %status;

    say @links - @broken, ' links ok';
    say "$status{$_}  $_" for @broken;
})->wait;

The find method of Mojo::DOM collects all a elements with an href, and to_abs resolves relative links against the page URL. The fragment is dropped, and a hash removes duplicates and keeps only HTTP links. Some servers reject HEAD with 405 Method Not Allowed, so the first handler retries those links with GET: returning a promise from then makes the chain wait for it. Every link ends up in %status with a code or an error, and the summary lists the links that did not return a 2xx status. The results depend on the current state of the checked sites.

$ perl links.pl
checking 35 links on https://www.perl.org
35 links ok
$ perl links.pl https://www.php.net
checking 156 links on https://www.php.net
153 links ok
error: Request timeout  https://wiki.php.net/todo/php86
403  https://www.linkedin.com/company/phpnet
403  https://www.linkedin.com/company/phpnet/

Best Practices

Prefer promises: use the _p methods and then chains, or async/await, rather than nested callbacks.
Check both kinds of failure: a rejected promise means the connection failed; a fulfilled one may still hold a 4xx or 5xx response, so test is_success.
Bound concurrency: use Mojo::Promise->map with concurrency instead of starting every request at once.
Set timeouts: give the user agent a request_timeout, so that one slow server cannot stall the whole batch.
Use all_settled for batches: all rejects at the first failure; all_settled, or a catch on each promise, keeps every result.
Never block inside the loop: sleep, blocking requests, or heavy computation in a callback stall every other request; use Mojo::IOLoop->timer or the non-blocking methods instead.
Keep the user agent alive: a Mojo::UserAgent object that goes out of scope cancels its pending requests, so store it in a variable that outlives them.
Start the loop once: call wait on the final promise, or Mojo::IOLoop->start unless Mojo::IOLoop->is_running, so the code also works inside a running application.
Be polite: set a User-Agent name and keep the number of connections to one site small.

In this article we have sent blocking and non-blocking requests with Mojo::UserAgent, explored the Mojo::IOLoop event loop, chained promises with then and catch, combined them with all, all_settled, race, and any, set timeouts, limited concurrency with Mojo::Promise->map, written asynchronous code with async and await, downloaded pages concurrently, and built a link checker.

Source

Mojo::UserAgent on MetaCPAN · Mojo::Promise · Mojolicious Cookbook: User agent

Author

My name is Jan Bodnar, and I am a passionate programmer with extensive programming experience. I have been writing programming articles since 2007. To date, I have authored over 1,400 articles and 8 e-books. I possess more than ten years of experience in teaching programming.

List all Perl tutorials.

Ebooks