Perl LWP programming
last modified October 8, 2026
In this article we show how to do WWW programming in Perl with LWP module.
LWP is a set of Perl modules which provides a simple and consistent
application programming interface (API) to the World-Wide Web. The main focus of
the library is to provide classes and functions to write WWW clients. LWP is
short for Library for WWW in Perl. HTTPS support requires the optional
LWP::Protocol::https module (install it with
cpanm LWP::Protocol::https).
LWP::Simple
LWP::Simple is a simple procedural interface to LWP. It contains a
few functions for easy working with web pages. The LWP::Simple
module is handy for simple cases. For more control over requests and responses,
use LWP::UserAgent, which supports features such as cookies,
authorization, and custom headers.
The get function
The get function fetches the document identified by the given URL
and returns it. It returns undef if it fails. The $url
argument can be either a string or a reference to a URI object.
#!/usr/bin/perl
use 5.30.0;
use warnings;
use LWP::Simple;
my $cont = get('https://example.com');
die 'Unable to get page' unless defined $cont;
say $cont;
The script fetches the content of https://example.com. The response
body depends on what the server returns.
Response content is supplied by the server and may change.
The following program demonstrates a simple tag-shaped text substitution on a fetched page.
#!/usr/bin/perl
use 5.30.0;
use warnings;
use LWP::Simple;
my $cont = get('https://example.com');
die 'Unable to get page' unless defined $cont;
foreach ($cont) {
s/<[^>]*>//g;
print;
}
This substitution removes tag-shaped text from the response, but it is not an HTML parser and should not be used to process arbitrary HTML.
The head function
The head function retrieves document headers. On success, it returns
the following five values: the content type, document length, modification time,
expiration time, and server. It returns an empty list if it fails.
#!/usr/bin/perl
use 5.30.0;
use warnings;
use LWP::Simple;
my ($content_type, $doc_length,
$mod_time, $expires, $server) = head("https://example.com");
say "Content type: $content_type";
say "Document length: $doc_length";
say "Modification time: $mod_time";
say "Expires: $expires";
say "Server: $server";
The example prints selected response headers from
https://example.com. Header values can change over time.
The getstore function
The getstore function retrieves a document identified by a URL and
stores it in the file. The return value is the HTTP response code.
#!/usr/bin/perl
use 5.30.0;
use warnings;
use LWP::Simple;
my $r = getstore('https://example.com', 'webcode.html');
die "Request failed with status $r" unless is_success($r);
say "Response code: $r";
The script fetches the contents of https://example.com and stores
them in webcode.html.
It is possible to check the return code with the is_success
function.
#!/usr/bin/perl use 5.30.0; use warnings; use LWP::Simple; my $url = 'https://httpbin.org/status/404'; my $r = getstore($url, 'webcode.html'); die "Request to $url returned status $r" unless is_success($r);
A missing resource normally returns a 404 status. The test uses
is_success, which is true for successful 2xx responses.
The LWP Class Model
The LWP Class Model contains classes for more complex work with the World-Wide Web.
The LWP::UserAgent class represents a web client. Create and
configure a user-agent object, then send an HTTP::Request (or use
a convenience method such as get, head,
post, put, or delete). The request
returns an HTTP::Response object. Use
decoded_content when you need text decoded according to the response
headers.
User agent
The LWP::UserAgent is a web user agent class.
$ cpanm Mojolicious
We install Mojolicious framework.
#!/usr/bin/perl
use Mojolicious::Lite -signatures;
get '/' => sub ($c) {
my $ua = $c->req->headers->user_agent;
$c->render(text => $ua);
};
app->start;
The server processes the client request, determines the user agent, and returns the user agent back to the client.
$ perl server.pl daemon Web application available at http://127.0.0.1:3000
We start our server; it listens on port 3000.
#!/usr/bin/perl
use 5.30.0;
use warnings;
use LWP::UserAgent;
use HTTP::Request;
my $ua = LWP::UserAgent->new(timeout => 10);
$ua->agent('Perl script');
my $req = HTTP::Request->new('GET' => 'http://localhost:3000');
my $res = $ua->request($req);
if ($res->is_success) {
say $res->decoded_content;
} else {
say $res->status_line;
}
This script creates a simple GET request to the localhost.
my $ua = LWP::UserAgent->new(timeout => 10);
An instance of the LWP::UserAgent is created.
$ua->agent("Perl script");
With the agent method, we set the name of the
agent.
my $req = HTTP::Request->new('GET' => 'http://localhost:3000');
A GET request to the localhost is created.
my $res = $ua->request($req);
The request method dispatches the request object. The return value
is a response object.
if ($res->is_success) {
say $res->decoded_content;
} else {
say $res->status_line;
}
The is_success method checks if the response has a success return
code. The decoded_content method returns text decoded using response
headers. status_line returns the status code and message.
$ ./agent.pl Perl script
The server responded with the name of the agent that we have sent with the request.
The get method
The user agent's get method is a convenience method to execute an
HTTP request. It saves some typing.
#!/usr/bin/perl
use 5.30.0;
use warnings;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new(timeout => 10);
$ua->agent("Perl script");
my $res = $ua->get('https://example.com');
if ($res->is_success) {
say $res->decoded_content;
} else {
say $res->status_line;
}
The script gets the contents of the https://example.com page with
the convenience get method.
The following example requests definitions of a term from
urbandictionary.com. It requires the
non-core HTML::TreeBuilder module (install it with
cpanm HTML::TreeBuilder).
#!/usr/bin/perl
use 5.30.0;
use warnings;
use LWP::UserAgent;
use HTML::TreeBuilder;
use URI;
my $word = shift || 'dog';
my %parameters = (term => $word);
my $url = URI->new('https://www.urbandictionary.com/define.php');
$url->query_form(%parameters);
my $ua = LWP::UserAgent->new(timeout => 10);
my $res = $ua->get($url);
die $res->status_line unless $res->is_success;
my $tree = HTML::TreeBuilder->new_from_content($res->decoded_content);
my @meanings = $tree->look_down(_tag => q{div}, 'class' => 'meaning');
foreach my $el (@meanings) {
say $el->as_text;
}
$tree->delete;
In this script, we find the definitions of the term dog on
urbandictionary.com. We display the definitions from the first
page. The HTML::TreeBuilder is used to parse the HTML code.
Posting a value
The post method dispatches a POST request on the given
URL, providing the key/value pairs for the fill-in form content.
#!/usr/bin/perl
use Mojolicious::Lite -signatures;
post '/' => sub ($c) {
my $name = $c->param('name');
$c->render(text => "Hello $name!");
};
app->start;
In the handler, we get the name parameter. From the parameter, we build a message and send it to the client.
#!/usr/bin/perl
use 5.30.0;
use warnings;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new(timeout => 10);
my $res = $ua->post('http://localhost:3000/',
['name' => 'Jan']);
if ($res->is_success) {
say $res->decoded_content;
} else {
say $res->status_line;
}
The script sends a request with a name key having Jan
value.
$ ./post_value.pl Hello Jan!
Credentials
The user agent's credentials method supplies a user name and
password for an authentication realm. This example uses localhost over HTTP;
Basic authentication does not encrypt credentials, so use HTTPS for remote
connections.
$ sudo apt-get install apache2-utils $ sudo htpasswd -c /etc/nginx/.htpasswd user7 New password: Re-type new password: Adding password for user user7
We use the htpasswd tool to create a user name and a password
for basic HTTP authentication.
location /secure {
auth_basic "Restricted Area";
auth_basic_user_file /etc/nginx/.htpasswd;
}
Inside the nginx /etc/nginx/sites-available/default configuration
file, we create a secured page. The name of the realm is "Restricted Area".
<!DOCTYPE html> <html lang="en"> <head> <title>Secure page</title> </head> <body> <p> This is a secure page. </p> </body> </html>
Inside the /usr/share/nginx/html/secure directory, we have
this HTML file.
#!/usr/bin/perl
use 5.30.0;
use warnings;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new(timeout => 10);
$ua->agent("Perl script");
$ua->credentials('localhost:80', 'Restricted Area', 'user7' => 's$cret');
my $res = $ua->get('http://localhost/secure/');
if ($res->is_success) {
say $res->decoded_content;
} else {
say $res->status_line;
}
The script accesses the protected route on the local test server and supplies the required credentials. This local HTTP example is not suitable for remote use; Basic authentication does not encrypt the credentials.
$ ./credentials.pl <!DOCTYPE html> <html lang="en"> <head> <title>Secure page</title> </head> <body> <p> This is a secure page. </p> </body> </html>
With the right credentials, the credentials.pl script returns
the secured page.
In this article we have worked with the Perl LWP module.
Author
List all Perl tutorials.