ZetCode

Perl Parallel::ForkManager

last modified November 7, 2026

When a program has hundreds of similar jobs to run, forking one child per job at once would exhaust the machine. The CPAN module Parallel::ForkManager manages a pool of worker processes: it forks children up to a fixed limit, reuses free slots as workers exit, and lets the parent collect their results through callbacks. In this article we build several complete examples with it, from a minimal pool to timeouts, retries, and a parallel word count.

The module is installed from CPAN with cpanm Parallel::ForkManager. The examples were run with Perl 5.44 and Parallel::ForkManager 2.04 on Linux.

Running jobs in parallel

A new fork manager is created with Parallel::ForkManager->new($n), where $n is the largest number of children allowed to run at the same time. Each job calls $pm->start($ident), which forks a child and returns the child's process ID in the parent and 0 in the child. The idiom $pm->start($ident) and next; makes the parent skip the body of the loop, so only the child runs the work. The child ends with $pm->finish, and the parent waits for all remaining children with $pm->wait_all_children.

basic.pl 
#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Time::HiRes qw(time);
use Parallel::ForkManager;

my $pm = Parallel::ForkManager->new(2);

$pm->run_on_start(sub {
    my (undef, $ident) = @_;
    say "job $ident started";
});

my $begin = time;

for my $id (1 .. 5) {
    $pm->start($id) and next;

    sleep 1;
    $pm->finish(0);
}

$pm->wait_all_children;
printf "all jobs finished in %.0f s\n", time - $begin;

Only two children run at a time; when one of the first two exits, the next job is dispatched into the free slot. The run_on_start callback runs in the parent, so its messages appear in the order the jobs are dispatched, and the last line is printed only after every child has been reaped. Five one-second jobs in two slots need three rounds, so we would expect about three seconds. The run takes twice as long, which the next section explains. 

$ perl basic.pl
job 1 started
job 2 started
job 3 started
job 4 started
job 5 started
all jobs finished in 6 s

Waiting without polling

By default, the parent does not block in waitpid. When the pool is full, it checks for finished children, sleeps for one second, and checks again. This pseudo-blocking wait keeps the module from swallowing exit notifications of processes it did not start, but it adds up to a second of latency every time the parent waits for a slot. If the program does not fork any other children, set_waitpid_blocking_sleep(0) switches to a true blocking wait.

blocking.pl 
#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Time::HiRes qw(time);
use Parallel::ForkManager;

my $pm = Parallel::ForkManager->new(2);
$pm->set_waitpid_blocking_sleep(0);

$pm->run_on_start(sub {
    my (undef, $ident) = @_;
    say "job $ident started";
});

my $begin = time;

for my $id (1 .. 5) {
    $pm->start($id) and next;

    sleep 1;
    $pm->finish(0);
}

$pm->wait_all_children;
printf "all jobs finished in %.0f s\n", time - $begin;

The only change is the call to set_waitpid_blocking_sleep right after the manager is created. The parent now wakes up the moment a child exits, so the next job starts immediately and the five jobs finish in the expected three seconds. The sleep period can also be a fraction of a second, such as 0.1, which keeps the polling behaviour but reduces the delay. 

$ perl blocking.pl
job 1 started
job 2 started
job 3 started
job 4 started
job 5 started
all jobs finished in 3 s

Callbacks and returning data

A child process gets its own copy of memory, so variables changed there are not visible in the parent. The supported way to send a result back is the second argument of finish, a reference to a data structure. The parent receives it in a run_on_finish callback, whose parameters are the child PID, the exit code, the identifier given to start, the exit signal, the core-dump flag, and the returned data reference. Declare both callbacks before the first call to start.

callbacks.pl 
#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Parallel::ForkManager;

my $pm = Parallel::ForkManager->new(2);

my %squares;

$pm->run_on_start(sub {
    my ($pid, $ident) = @_;
    say "started job $ident";
});

$pm->run_on_finish(sub {
    my ($pid, $exit_code, $ident, $signal, $core, $data) = @_;
    $squares{$ident} = $data->{value};
});

for my $n (1 .. 6) {
    $pm->start($n) and next;

    my $square = $n * $n;
    $pm->finish(0, { value => $square });
}

$pm->wait_all_children;

say "square of $_ is $squares{$_}" for sort { $a <=> $b } keys %squares;

Each child computes one square and sends it back as a hash reference. The parent stores the results under the identifier and prints them in numeric order once all workers are done. The order in which children finish is not guaranteed, which is why the results are sorted before they are shown. 

$ perl callbacks.pl
started job 1
started job 2
started job 3
started job 4
started job 5
started job 6
square of 1 is 1
square of 2 is 4
square of 3 is 9
square of 4 is 16
square of 5 is 25
square of 6 is 36

Running a subroutine as a child

When the work is a short block of code, start_child avoids the manual and next and the explicit finish. It takes an optional identifier and a code reference; the reference runs in the child, and whatever it returns is serialized and delivered to run_on_finish as the data argument. The return value has to be a reference, such as a hash or an array.

child.pl 
#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Parallel::ForkManager;

my $pm = Parallel::ForkManager->new(3);

my %sorted;

$pm->run_on_finish(sub {
    my (undef, undef, $ident, undef, undef, $data) = @_;
    $sorted{$ident} = $data->{value};
});

for my $word (qw(banana apple cherry)) {
    $pm->start_child($word => sub {
        return { value => join '', sort split //, $word };
    }) and next;
}

$pm->wait_all_children;

say "$_ -> $sorted{$_}" for sort keys %sorted;

The callback sorts the letters of each word and returns them in a hash reference. The parent receives one result per word and prints the keys in alphabetical order. Because the callback is the whole body of the child, this form reads like a plain parallel map. 

$ perl child.pl
apple -> aelpp
banana -> aaabnn
cherry -> cehrry

Debugging without forking

Bugs in forked code are hard to trace, because a debugger follows only one process and output from several children interleaves. Passing 0 to new turns the fork manager off: start does not fork, the job body runs in the parent, and finish returns instead of exiting. The callbacks still fire, so the same program runs sequentially without any code changes.

debug.pl 
#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Parallel::ForkManager;

my $procs  = shift // 2;
my $parent = $$;

my $pm = Parallel::ForkManager->new($procs);

$pm->run_on_finish(sub {
    my (undef, $exit_code, $ident, undef, undef, $data) = @_;
    say "$ident ran in the $data->{where}, exit code $exit_code";
});

for my $task (qw(alpha beta gamma)) {
    $pm->start($task) and next;

    my $where = $$ == $parent ? 'parent' : 'child';
    $pm->finish(0, { where => $where });
}

$pm->wait_all_children;

The number of workers comes from the command line and defaults to 2. Each job compares the current process ID with the one saved before the loop and reports where it ran. With the default pool every job runs in a child; with 0 every job runs in the parent, one after another, while run_on_finish still receives the exit code and the data. 

$ perl debug.pl
alpha ran in the child, exit code 0
beta ran in the child, exit code 0
gamma ran in the child, exit code 0
$ perl debug.pl 0
alpha ran in the parent, exit code 0
beta ran in the parent, exit code 0
gamma ran in the parent, exit code 0

Detecting failures

finish($exit_code) sets the status the parent later sees. In run_on_finish a non-zero exit code means the child reported a failure, while a non-zero exit signal means the child was killed rather than exiting normally. Checking both fields lets the parent react to each case separately instead of losing the distinction.

errors.pl 
#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Parallel::ForkManager;

my $pm = Parallel::ForkManager->new(3);

my @log;

$pm->run_on_finish(sub {
    my ($pid, $exit_code, $ident, $exit_signal) = @_;

    if ($exit_signal) {
        push @log, "$ident killed by signal $exit_signal";
    }
    else {
        push @log, "$ident exited with code $exit_code";
    }
});

for my $job (qw(ok fail crash)) {
    $pm->start($job) and next;

    $pm->finish(2)              if $job eq 'fail';
    kill 'KILL', $$             if $job eq 'crash';

    $pm->finish(0);
}

$pm->wait_all_children;
say for sort @log;

The three jobs take different paths: one exits cleanly, one exits with code 2, and one sends itself the KILL signal, so it never reaches finish. The callback records each outcome, and signal 9 marks the job that was killed. The messages are sorted for a stable listing, since the children finish in an arbitrary order. 

$ perl errors.pl
crash killed by signal 9
fail exited with code 2
ok exited with code 0

Limiting the run time of a job

A worker that hangs on a slow network or a stuck lock occupies a slot forever. Because every job is a separate process, the child can simply set an alarm with alarm. Without a $SIG{ALRM} handler, the ALRM signal terminates the child, and the parent sees signal 14 in run_on_finish.

timeout.pl 
#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Parallel::ForkManager;

my %jobs = (quick => 1, medium => 2, slow => 5);

my $pm = Parallel::ForkManager->new(3);
$pm->set_waitpid_blocking_sleep(0);

$pm->run_on_finish(sub {
    my (undef, $exit_code, $ident, $signal) = @_;

    if ($signal == 14) {
        say "$ident timed out";
    }
    else {
        say "$ident done, exit code $exit_code";
    }
});

for my $name (sort { $jobs{$a} <=> $jobs{$b} } keys %jobs) {
    $pm->start($name) and next;

    alarm 3;
    sleep $jobs{$name};

    $pm->finish(0);
}

$pm->wait_all_children;

The jobs simulate work with sleep for one, two, and five seconds, and each child gives itself three seconds. The first two finish in time; the slow one is killed by the alarm and reported as timed out. Blocking waits are enabled, so the messages appear as soon as each child ends, and the whole program takes three seconds instead of five. 

$ perl timeout.pl
quick done, exit code 0
medium done, exit code 0
slow timed out

Retrying failed jobs

Callbacks run in the parent, so run_on_finish can push a failed job back to the work queue. The inner loop consumes the queue while callbacks may refill it, and the outer loop repeats until no retry is left after wait_all_children, since the last failures are reported only while waiting for the final children.

retry.pl 
#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Parallel::ForkManager;

my $max_attempts = 3;

my %attempts;
my @queue = qw(a b c d);

my $pm = Parallel::ForkManager->new(2);
$pm->set_waitpid_blocking_sleep(0);

$pm->run_on_finish(sub {
    my (undef, $exit_code, $job) = @_;

    if ($exit_code == 0) {
        say "$job succeeded on attempt $attempts{$job}";
    }
    elsif ($attempts{$job} < $max_attempts) {
        say "$job failed on attempt $attempts{$job}, retrying";
        push @queue, $job;
    }
    else {
        say "$job failed $max_attempts times, giving up";
    }
});

while (@queue) {

    while (my $job = shift @queue) {
        my $attempt = ++$attempts{$job};

        $pm->start($job) and next;

        my $ok = $job eq 'b' ? $attempt >= 2
               : $job eq 'd' ? 0
               :               1;

        $pm->finish($ok ? 0 : 1);
    }

    $pm->wait_all_children;
}

The parent counts the attempts per job before forking, so the child knows which attempt it runs. Job b fails once and succeeds on the second try, job d always fails and is given up after three attempts, and the other two succeed at once. The exact order of the lines can vary between runs, because two workers finish independently. 

$ perl retry.pl
a succeeded on attempt 1
b failed on attempt 1, retrying
c succeeded on attempt 1
d failed on attempt 1, retrying
d failed on attempt 2, retrying
b succeeded on attempt 2
d failed 3 times, giving up

Practical example: parallel file hashing

Computing message digests is a good fit for the pool: the files are independent and the work is CPU-bound. The example creates three temporary files, then reads and hashes each one in a separate child with Digest::MD5. The parent collects the digests by file name through run_on_finish.

hashing.pl 
#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use File::Temp qw(tempdir);
use Digest::MD5 qw(md5_hex);
use Parallel::ForkManager;

my $dir = tempdir(CLEANUP => 1);

my @files;
for my $name (qw(one.txt two.txt three.txt)) {
    my $path = "$dir/$name";
    open my $out, '>', $path or die "open $path: $!";
    print {$out} "$name content\n";
    close $out;
    push @files, [$name, $path];
}

my $pm = Parallel::ForkManager->new(3);
my %digests;

$pm->run_on_finish(sub {
    my (undef, undef, $ident, undef, undef, $data) = @_;
    $digests{$ident} = $data->{digest};
});

for my $file (@files) {
    my ($name, $path) = @$file;
    $pm->start($name) and next;

    open my $in, '<', $path or $pm->finish(1, { digest => 'read-error' });
    binmode $in;
    my $content = do { local $/; <$in> };
    close $in;

    $pm->finish(0, { digest => md5_hex($content) });
}

$pm->wait_all_children;

say "$_ $digests{$_}" for sort keys %digests;

Each child opens one file, slurps it, and sends the digest back inside a hash reference; a file that cannot be opened ends the child with exit code 1. The parent stores the result under the file name and prints the lines in sorted order. The digest values depend only on the file contents, which is why the output is reproducible. 

$ perl hashing.pl
one.txt b41407d01b84546f2d60205c111c7bd1
three.txt c1ca8f487aa872f14af826a602aac000
two.txt 5d5a63ded39594313ba7b1e481d2e1c6

Practical example: counting words

Forking one process per item is wasteful when each item is tiny. A better pattern splits the input into a few chunks, lets each worker process a whole chunk, and merges the partial results in the parent: a small map-reduce. The example counts words in a block of text with three workers.

wordcount.pl 
#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Parallel::ForkManager;

my @lines = (<<'END') =~ /^(.+)$/mg;
the sky is blue and the sea is blue
a cat sat on the mat
the cat and the dog are friends
blue is the colour of the sky
a dog chased the cat up the tree
the tree is old and the sky is clear
END

my $workers = 3;
my $size    = int((@lines + $workers - 1) / $workers);

my $pm = Parallel::ForkManager->new($workers);
my %total;

$pm->run_on_finish(sub {
    my (undef, undef, $ident, undef, undef, $counts) = @_;
    say "chunk $ident counted ", scalar keys %$counts, " distinct words";
    $total{$_} += $counts->{$_} for keys %$counts;
});

my $chunk = 0;

while (my @part = splice @lines, 0, $size) {
    $chunk++;

    $pm->start($chunk) and next;

    my %counts;
    $counts{lc $_}++ for map { split ' ' } @part;

    $pm->finish(0, \%counts);
}

$pm->wait_all_children;

my @top = sort { $total{$b} <=> $total{$a} || $a cmp $b } keys %total;
printf "%-5s %d\n", $_, $total{$_} for @top[0 .. 4];

The text is split into non-empty lines with a multiline match on the here document, and splice removes one chunk of lines at a time, so each of the three workers gets two lines. A child counts its words in a private hash and returns a reference to it; the parent adds every partial count into %total. The chunks finish in an arbitrary order, but the merged totals are the same on every run, and ties in the top five are broken alphabetically. 

$ perl wordcount.pl
chunk 3 counted 12 distinct words
chunk 1 counted 11 distinct words
chunk 2 counted 11 distinct words
the   11
is    5
and   3
blue  3
cat   3

Practical example: downloading web pages in parallel

A downloader is the classic use for a bounded worker pool: each URL is independent, and the number of simultaneous connections has to stay under control. The example reads the URLs from a text file, derives a file name from each URL's host and path, and has each child fetch one page with LWP::UserAgent and write it into files/. The modules come from CPAN; install them with cpanm Parallel::ForkManager Path::Tiny LWP LWP::Protocol::https.

urls.txt 
https://example.com
https://httpbin.org/get
https://www.perl.org
https://metacpan.org
https://www.python.org
https://www.php.net
https://github.com
https://httpbin.org/status/404

The list mixes pages that return content with one URL that answers 404, so the failure path is exercised as well.

async_req.pl 
#!/usr/bin/perl

use strict;
use warnings;
use v5.34.0;
use Path::Tiny;
use URI;
use LWP::UserAgent;
use Parallel::ForkManager;

my @urls = grep { /\S/ } path('urls.txt')->lines_utf8({ chomp => 1 });

my $pm = Parallel::ForkManager->new(4);
my $ua = LWP::UserAgent->new(agent => 'Perl script', timeout => 10);

say "downloading ", scalar @urls, " files";

my $dir = path('files');
$dir->mkpath;

for my $link (@urls) {

    my $uri = URI->new($link);
    my $name = $uri->host =~ s/^www\.//r;
    $name .= $uri->path =~ tr{/}{_}r if $uri->path ne '' && $uri->path ne '/';

    $pm->start and next;

    my $resp = $ua->get($link);

    if ($resp->is_success) {
        $dir->child("$name.txt")->spew_utf8($resp->decoded_content);
    } else {
        warn "$link: ", $resp->status_line, "\n";
    }

    $pm->finish;
}

$pm->wait_all_children;

The parent computes the file name before forking, so the child only needs the link. The parts of the URI object give a safe name: the host loses its www. prefix, and a path such as /get becomes _get, so https://httpbin.org/get is stored as httpbin.org_get.txt. A failed transfer is reported with its URL and status line, and four workers keep the pool bounded while the pages download. Note that this is parallelism through processes, not asynchronous I/O; for true non-blocking requests on a single process, Mojo::UserAgent promises are the natural choice. 

$ perl async_req.pl
downloading 8 files
https://httpbin.org/status/404: 404 Not Found
$ ls -1 files/
example.com.txt
github.com.txt
httpbin.org_get.txt
metacpan.org.txt
perl.org.txt
php.net.txt
python.org.txt

Best Practices

In this article we have used Parallel::ForkManager to run jobs in parallel, limited the pool with new, removed the polling delay with blocking waits, collected results through run_on_finish, run subroutines with start_child, debugged without forking, handled exit codes and signals, limited job time with alarm, retried failed jobs, merged partial results of a chunked word count, and downloaded web pages in a bounded pool of workers.

Source

Parallel::ForkManager on MetaCPAN ยท perlipc

Author

My name is Jan Bodnar, and I am a passionate programmer with extensive programming experience. I have been writing programming articles since 2007. To date, I have authored over 1,400 articles and 8 e-books. I possess more than ten years of experience in teaching programming.

List all Perl tutorials.