Asynchronous HTTP requests

last modified November 4, 2026

The asynchronous HTTP requests tutorial shows how to send async HTTP requests in Go, C#, F#, Groovy, Python, Perl, JavaScript, Java, Kotlin, PHP, Rust, and Raku.

Most of the time spent on an HTTP request is waiting for the network. A synchronous client sends a request, waits for the response, and only then sends the next one, so the total time is the sum of all requests. An asynchronous client starts all requests at once and handles each response when it arrives, so the total time is close to the time of the slowest request.

Languages take different routes to this. C#, F#, Python, Perl, JavaScript, PHP, and Rust use non-blocking I/O driven by an event loop or the operating system. Java and Groovy use the asynchronous API of the JDK HttpClient; Kotlin wraps the same API in coroutines. Go writes ordinary blocking code in goroutines; the Go runtime parks a goroutine waiting on a socket and runs others in the meantime. Raku is shown both ways: the first example runs blocking requests on a thread pool, the second uses the non-blocking Cro client.

Each example also sets a timeout and reports errors per request, so a single slow or failing site does not hide the results of the others. The lists contain https://httpbin.org/status/404 and, in some examples, the unresolvable https://no-such-host.invalid to show both kinds of failure. The output depends on the network and the remote sites, so titles, server names, and the order of lines can differ between runs.

Go async requests

Go has goroutines for making asynchronous requests. A goroutine is a lightweight thread managed by the Go runtime.

package main

import (
    "fmt"
    "io"
    "net/http"
    "regexp"
    "sync"
    "time"
)

var titleRx = regexp.MustCompile(`(?is)<title[^>]*>\s*(.*?)\s*</title>`)

var client = &http.Client{Timeout: 10 * time.Second}

func main() {

    urls := []string{
        "https://example.com",
        "https://www.perl.org",
        "https://www.php.net",
        "https://www.python.org",
        "https://go.dev",
        "https://clojure.org",
        "https://github.com",
    }

    var wg sync.WaitGroup

    for _, url := range urls {

        wg.Go(func() {

            content, err := doReq(url)

            if err != nil {
                fmt.Printf("%s: %v\n", url, err)
                return
            }

            fmt.Printf("%s: %s\n", url, getTitle(content))
        })
    }

    wg.Wait()
}

func doReq(url string) (string, error) {

    resp, err := client.Get(url)

    if err != nil {
        return "", err
    }

    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)

    if err != nil {
        return "", err
    }

    return string(body), nil
}

func getTitle(content string) string {

    parts := titleRx.FindStringSubmatch(content)

    if len(parts) > 1 {
        return parts[1]
    }

    return "no title"
}

We make multiple asynchronous HTTP requests and print the contents of the title tag of each web page together with its URL.

var titleRx = regexp.MustCompile(`(?is)<title[^>]*>\s*(.*?)\s*</title>`)

var client = &http.Client{Timeout: 10 * time.Second}

The regular expression is compiled once. The i flag makes it case insensitive and the s flag lets . match newlines, so titles that span lines are found too. Unlike http.Get, which uses a client without a timeout, our client gives up after ten seconds.

var wg sync.WaitGroup

for _, url := range urls {

    wg.Go(func() {
        ...
    })
}

wg.Wait()

A WaitGroup waits for a collection of goroutines to finish. The Go method, available since Go 1.25, starts the function in a new goroutine and tracks it in the group. Since Go 1.22, each loop iteration has its own url variable, so the closure can use it directly.

content, err := doReq(url)

if err != nil {
    fmt.Printf("%s: %v\n", url, err)
    return
}

The doReq function returns the error to the caller instead of logging it, so we can report which URL failed.

$ go run .
https://www.perl.org: The Perl Programming Language - www.perl.org
https://www.php.net: PHP
https://example.com: Example Domain
https://clojure.org: Clojure
https://www.python.org: Welcome to Python.org
https://github.com: GitHub · Change is constant. GitHub keeps you ahead. · GitHub · GitHub
https://go.dev: The Go Programming Language

C# async requests

In C#, we use the HttpClient to generate asynchronous requests.

using System.Text.RegularExpressions;

string[] urls = [
    "https://example.com", "https://httpbin.org/html", "https://www.python.org",
    "https://www.php.net", "https://github.com", "https://httpbin.org/status/404"
];

var rx = new Regex(@"<title[^>]*>\s*(.+?)\s*</title>",
    RegexOptions.Compiled | RegexOptions.Singleline | RegexOptions.IgnoreCase);

using var client = new HttpClient { Timeout = TimeSpan.FromSeconds(10) };

async Task<string> FetchTitleAsync(string url)
{
    try
    {
        var content = await client.GetStringAsync(url);
        var match = rx.Match(content);

        return match.Success ? $"{url}: {match.Groups[1].Value}" : $"{url}: no title";
    }
    catch (Exception e) when (e is HttpRequestException or TaskCanceledException)
    {
        return $"{url}: {e.Message}";
    }
}

var titles = await Task.WhenAll(urls.Select(FetchTitleAsync));

foreach (var title in titles)
{
    Console.WriteLine(title);
}

We download the given web pages asynchronously and print their titles.

using var client = new HttpClient { Timeout = TimeSpan.FromSeconds(10) };

A single HttpClient is shared by all requests. The default timeout is 100 seconds; we lower it to ten.

var content = await client.GetStringAsync(url);
var match = rx.Match(content);

The GetStringAsync sends a GET request to the specified URL and returns the response body as a string in an asynchronous operation. We take the first match only; a page may contain other title elements, for instance inside embedded SVG images. The Groups[1] holds the text between the tags.

catch (Exception e) when (e is HttpRequestException or TaskCanceledException)
{
    return $"{url}: {e.Message}";
}

A failed request or an error status code throws an HttpRequestException; a timeout throws a TaskCanceledException. We turn both into a message, so one bad URL does not fail the whole batch.

var titles = await Task.WhenAll(urls.Select(FetchTitleAsync));

The Task.WhenAll creates a task that completes when all of the given tasks have completed. Awaiting it does not block a thread, and it returns the results in the order of the input URLs.

$ dotnet run
https://example.com: Example Domain
https://httpbin.org/html: no title
https://www.python.org: Welcome to Python.org
https://www.php.net: PHP
https://github.com: GitHub · Change is constant. GitHub keeps you ahead. · GitHub · GitHub
https://httpbin.org/status/404: Response status code does not indicate success: 404 (Not Found).

F# async requests with task expressions

The following example uses HttpClient and task expressions to fetch website titles asynchronously.

open System
open System.Net
open System.Net.Http
open System.Text.RegularExpressions
open System.Threading.Tasks

let client = new HttpClient(Timeout = TimeSpan.FromSeconds 10.0)

let rx = Regex(@"<title[^>]*>\s*(.+?)\s*</title>",
               RegexOptions.Singleline ||| RegexOptions.IgnoreCase)

let fetchTitleAsync (url: string) =
    task {
        try
            let! html = client.GetStringAsync(url)
            let m = rx.Match(html)

            return
                if m.Success then $"{url}: {WebUtility.HtmlDecode m.Groups[1].Value}"
                else $"{url}: no title"
        with e ->
            return $"{url}: {e.Message}"
    }

let sites =
    [| "https://example.com"
       "https://clojure.org"
       "https://www.python.org"
       "https://fsharp.org"
       "https://github.com"
       "https://httpbin.org/status/404" |]

let titles =
    (sites |> Array.map fetchTitleAsync |> Task.WhenAll).GetAwaiter().GetResult()

titles |> Array.iter (printfn "%s")

The example asynchronously retrieves the titles of the given URLs.

let client = new HttpClient(Timeout = TimeSpan.FromSeconds 10.0)

The client is created once and shared by all requests. Creating a new HttpClient for every request wastes connections.

let! html = client.GetStringAsync(url)
let m = rx.Match(html)

Inside the task expression, let! awaits the response body without blocking. The WebUtility.HtmlDecode then turns HTML entities in the title into plain characters.

let titles =
    (sites |> Array.map fetchTitleAsync |> Task.WhenAll).GetAwaiter().GetResult()

The Task.WhenAll combines the tasks into one. At the top level of a script we wait for it with GetAwaiter().GetResult().

$ dotnet fsi async_req.fsx
https://example.com: Example Domain
https://clojure.org: Clojure
https://www.python.org: Welcome to Python.org
https://fsharp.org: fsharp.org
https://github.com: GitHub · Change is constant. GitHub keeps you ahead. · GitHub · GitHub
https://httpbin.org/status/404: Response status code does not indicate success: 404 (Not Found).

F# async requests with async workflows

The second solution uses F# async workflows. The Async.Parallel runs the computations concurrently and lets us limit how many of them run at the same time.

open System
open System.Net
open System.Net.Http
open System.Text.RegularExpressions

let client = new HttpClient(Timeout = TimeSpan.FromSeconds 10.0)

let rx = Regex(@"<title[^>]*>\s*(.+?)\s*</title>",
               RegexOptions.Singleline ||| RegexOptions.IgnoreCase)

let fetchTitleAsync (url: string) =
    async {
        let! ct = Async.CancellationToken

        try
            let! html = client.GetStringAsync(url, ct) |> Async.AwaitTask
            let m = rx.Match(html)

            return
                if m.Success then $"{url}: {WebUtility.HtmlDecode m.Groups[1].Value}"
                else $"{url}: no title"
        with
        | :? AggregateException as e -> return $"{url}: {e.InnerException.Message}"
        | e -> return $"{url}: {e.Message}"
    }

let sites =
    [ "https://example.com"
      "https://clojure.org"
      "https://www.python.org"
      "https://fsharp.org"
      "https://github.com"
      "https://httpbin.org/status/404" ]

let titles =
    sites
    |> List.map fetchTitleAsync
    |> fun jobs -> Async.Parallel(jobs, maxDegreeOfParallelism = 3)
    |> Async.RunSynchronously

titles |> Array.iter (printfn "%s")

The example retrieves the same titles, at most three at a time.

let! ct = Async.CancellationToken

try
    let! html = client.GetStringAsync(url, ct) |> Async.AwaitTask

The Async.AwaitTask turns the task returned by HttpClient into an async computation. We pass the cancellation token of the workflow to the request, so cancelling the workflow cancels the request as well.

| :? AggregateException as e -> return $"{url}: {e.InnerException.Message}"

The Async.AwaitTask may wrap the error in an AggregateException; we print the message of the inner exception.

|> fun jobs -> Async.Parallel(jobs, maxDegreeOfParallelism = 3)
|> Async.RunSynchronously

The maxDegreeOfParallelism parameter limits the number of concurrent requests. The Async.RunSynchronously runs the combined computation and waits for its result.

$ dotnet fsi async_req2.fsx
https://example.com: Example Domain
https://clojure.org: Clojure
https://www.python.org: Welcome to Python.org
https://fsharp.org: fsharp.org
https://github.com: GitHub · Change is constant. GitHub keeps you ahead. · GitHub · GitHub
https://httpbin.org/status/404: Response status code does not indicate success: 404 (Not Found).

Groovy async requests

In Groovy, we use the asynchronous API of the JDK HttpClient.

import java.net.http.HttpClient
import java.net.http.HttpRequest
import java.net.http.HttpResponse
import java.time.Duration
import java.util.concurrent.CompletableFuture

def client = HttpClient.newBuilder()
    .connectTimeout(Duration.ofSeconds(10))
    .build()

def urls = [
    "https://example.com",
    "https://www.wikipedia.org",
    "https://groovy-lang.org",
    "https://www.python.org",
    "https://github.com",
    "https://httpbin.org/status/404",
    "https://no-such-host.invalid"
]

def futures = urls.collect { url ->

    def request = HttpRequest.newBuilder(URI.create(url))
        .timeout(Duration.ofSeconds(10))
        .header("User-Agent", "Groovy async example")
        .build()

    client.sendAsync(request, HttpResponse.BodyHandlers.discarding())
        .thenApply { res -> "${url}: ${res.statusCode()}" }
        .exceptionally { ex -> "${url}: ${ex.cause?.class?.simpleName ?: ex.class.simpleName}" }
}

CompletableFuture.allOf(futures as CompletableFuture[]).join()

futures.each { println it.join() }
println "finished"

The example makes asynchronous requests to the URLs and prints their status codes.

def futures = urls.collect { url ->

The collect method passes each URL to the closure as a parameter, so every request gets its own copy of the URL. The closure returns a future, and collect gathers the futures into a list.

client.sendAsync(request, HttpResponse.BodyHandlers.discarding())
    .thenApply { res -> "${url}: ${res.statusCode()}" }
    .exceptionally { ex -> "${url}: ${ex.cause?.class?.simpleName ?: ex.class.simpleName}" }

The sendAsync sends the request without blocking and returns a CompletableFuture. We discard the body, since we only need the status code. The exceptionally turns a failed request into a message instead of an exception.

CompletableFuture.allOf(futures as CompletableFuture[]).join()

futures.each { println it.join() }

The allOf waits for all futures to complete. We then print the results in the order of the input URLs.

$ groovy mul_async_req.gvy
https://example.com: 200
https://www.wikipedia.org: 200
https://groovy-lang.org: 200
https://www.python.org: 200
https://github.com: 200
https://httpbin.org/status/404: 404
https://no-such-host.invalid: ConnectException
finished

Python async requests

In Python, we use the httpx and asyncio modules.

$ pip install httpx

#!/usr/bin/python

import asyncio
import time

import httpx

urls = ['https://example.com', 'https://httpbin.org/get',
        'https://www.python.org', 'https://github.com',
        'https://httpbin.org/status/404', 'https://no-such-host.invalid']


async def get_status(client, url):

    try:
        resp = await client.get(url)
        return f'{url}: {resp.status_code}'
    except httpx.HTTPError as e:
        return f'{url}: {type(e).__name__}'


async def main():

    start = time.perf_counter()

    async with httpx.AsyncClient(timeout=10, follow_redirects=True) as client:
        async with asyncio.TaskGroup() as tg:
            tasks = [tg.create_task(get_status(client, url)) for url in urls]

    for task in tasks:
        print(task.result())

    print(f'finished in {time.perf_counter() - start:.2f} s')

asyncio.run(main())

The example makes asynchronous requests and prints the status code of each of the URLs, followed by the total time.

async with httpx.AsyncClient(timeout=10, follow_redirects=True) as client:

One AsyncClient is shared by all requests, so connections are reused. The httpx library does not follow redirects by default; we turn it on.

async with asyncio.TaskGroup() as tg:
    tasks = [tg.create_task(get_status(client, url)) for url in urls]

The TaskGroup runs the tasks concurrently and waits for all of them at the end of the async with block. If a task raises an unexpected exception, the group cancels the remaining tasks.

except httpx.HTTPError as e:
    return f'{url}: {type(e).__name__}'

Network errors and timeouts are subclasses of httpx.HTTPError. We handle them per request, so the other results are kept.

$ ./async_req.py
https://example.com: 200
https://httpbin.org/get: 200
https://www.python.org: 200
https://github.com: 200
https://httpbin.org/status/404: 404
https://no-such-host.invalid: ConnectError
finished in 0.72 s

Perl async requests

In Perl, we use the Mojo::UserAgent module from the Mojolicious framework. Its non-blocking requests run in an event loop and return promises.

$ cpanm Mojolicious

#!/usr/bin/perl

use v5.36;
use open qw(:std :encoding(UTF-8));
use Mojo::UserAgent;
use Mojo::Promise;

my @urls = qw(
    https://example.com
    https://httpbin.org/get
    https://www.perl.org
    https://metacpan.org
    https://www.python.org
    https://www.php.net
    https://github.com
    https://httpbin.org/status/404
);

my $ua = Mojo::UserAgent->new(max_redirects => 5, request_timeout => 10);

my @promises = map {
    my $url = $_;

    $ua->get_p($url)->then(sub ($tx) {
        my $res = $tx->result;
        return say "$url: ", $res->code, ' ', $res->message unless $res->is_success;

        my $title = $res->dom->at('title');
        say "$url: ", $title ? $title->text : 'no title';
    })->catch(sub ($err) {
        say "$url: $err";
    });
} @urls;

Mojo::Promise->all(@promises)->wait;

The example sends non-blocking requests to the URLs and prints the title of each page.

my $ua = Mojo::UserAgent->new(max_redirects => 5, request_timeout => 10);

The user agent follows up to five redirects and gives up on a request after ten seconds.

$ua->get_p($url)->then(sub ($tx) {

The get_p method starts a non-blocking GET request and returns a Mojo::Promise. The callback passed to then runs when the response arrives.

my $title = $res->dom->at('title');

Instead of a regular expression, we parse the page with the built-in HTML parser and select the title element with a CSS selector.

Mojo::Promise->all(@promises)->wait;

The all method creates a promise that is resolved when all the given promises are resolved. The wait method runs the event loop until then.

$ ./async_req.pl
https://www.perl.org: The Perl Programming Language - www.perl.org
https://example.com: Example Domain
https://www.php.net: PHP
https://metacpan.org: Search the CPAN - metacpan.org
https://www.python.org: Welcome to Python.org
https://github.com: GitHub · Change is constant. GitHub keeps you ahead. · GitHub · GitHub
https://httpbin.org/get: no title
https://httpbin.org/status/404: 404 Not Found

JavaScript async requests

Node.js 18 and newer has a built-in fetch function, so no external module is needed. The file has the .mjs extension, which makes it an ES module where we can use top-level await.

const urls = [
    'https://example.com',
    'https://httpbin.org/get',
    'https://clojure.org',
    'https://www.perl.org',
    'https://www.python.org',
    'https://github.com',
    'https://httpbin.org/status/404',
    'https://no-such-host.invalid'
];

async function fetchUrl(url) {

    const resp = await fetch(url, { signal: AbortSignal.timeout(10_000) });

    if (!resp.ok) {
        throw new Error(`HTTP ${resp.status}`);
    }

    return `${url} -> ${resp.headers.get('server') ?? 'unknown'}: ${resp.status}`;
}

const results = await Promise.allSettled(urls.map(fetchUrl));

results.forEach((res, i) => {

    if (res.status === 'fulfilled') {
        console.log(res.value);
    } else {
        console.log(`${urls[i]} failed: ${res.reason.cause?.code ?? res.reason.message}`);
    }
});

The example generates async requests to the given list of URLs. It prints the URL, server name, and status code of each site.

const resp = await fetch(url, { signal: AbortSignal.timeout(10_000) });

if (!resp.ok) {
    throw new Error(`HTTP ${resp.status}`);
}

The fetch function returns a promise. The AbortSignal.timeout aborts the request after ten seconds. Since fetch does not reject on HTTP error codes, we check the ok property ourselves.

const results = await Promise.allSettled(urls.map(fetchUrl));

The Promise.allSettled resolves after all of the given promises have either fulfilled or rejected. Each result has a status property, and a value or a reason. In contrast, Promise.all rejects as soon as one of the promises rejects.

$ node async_req.mjs
https://example.com -> cloudflare: 200
https://httpbin.org/get -> unknown: 200
https://clojure.org -> AmazonS3: 200
https://www.perl.org -> Combust/Plack (Perl): 200
https://www.python.org -> nginx: 200
https://github.com -> github.com: 200
https://httpbin.org/status/404 failed: HTTP 404
https://no-such-host.invalid failed: ENOTFOUND

Java async requests

The CompletableFuture is a high-level API for asynchronous programming in Java. The sendAsync method of the HttpClient returns one.

package com.zetcode;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.CompletableFuture;

public class AsyncReqEx {

    public static void main(String[] args) {

        List<URI> uris = List.of(
                "https://www.google.com/",
                "https://clojure.org",
                "https://www.rust-lang.org",
                "https://go.dev",
                "https://www.python.org",
                "https://github.com",
                "https://httpbin.org/status/404",
                "https://no-such-host.invalid"
        ).stream().map(URI::create).toList();

        try (HttpClient httpClient = HttpClient.newBuilder()
                .connectTimeout(Duration.ofSeconds(10))
                .followRedirects(HttpClient.Redirect.NORMAL)
                .build()) {

            var futures = uris.stream()
                    .map(uri -> verifyUri(httpClient, uri))
                    .toArray(CompletableFuture[]::new);

            CompletableFuture.allOf(futures).join();
        }
    }

    private static CompletableFuture<Void> verifyUri(HttpClient httpClient,
                                                     URI uri) {
        HttpRequest request = HttpRequest.newBuilder(uri)
                .timeout(Duration.ofSeconds(5))
                .build();

        return httpClient.sendAsync(request, HttpResponse.BodyHandlers.discarding())
                .thenApply(HttpResponse::statusCode)
                .thenAccept(status -> {
                    String result = status == 200 ? "SUCCESS" : "FAILURE";
                    System.out.printf("[%s] %s: %d%n", result, uri, status);
                })
                .exceptionally(ex -> {
                    System.out.printf("[FAILURE] %s: %s%n", uri,
                            ex.getCause().getClass().getSimpleName());
                    return null;
                });
    }
}

In the example, we check the status of the given web pages.

try (HttpClient httpClient = HttpClient.newBuilder()
        .connectTimeout(Duration.ofSeconds(10))
        .followRedirects(HttpClient.Redirect.NORMAL)
        .build()) {

Since Java 21, HttpClient is AutoCloseable. The NORMAL policy follows redirects except from HTTPS to HTTP.

return httpClient.sendAsync(request, HttpResponse.BodyHandlers.discarding())
        .thenApply(HttpResponse::statusCode)
        .thenAccept(status -> { ... })
        .exceptionally(ex -> { ... });

Each stage runs when the previous one completes. If the request fails, the exceptionally stage receives the exception; its cause tells us what went wrong.

CompletableFuture.allOf(futures).join();

The allOf returns a future that completes when all the given futures complete; join waits for it.

$ java com/zetcode/AsyncReqEx.java
[FAILURE] https://no-such-host.invalid: ConnectException
[SUCCESS] https://clojure.org: 200
[SUCCESS] https://www.python.org: 200
[SUCCESS] https://go.dev: 200
[FAILURE] https://httpbin.org/status/404: 404
[SUCCESS] https://www.google.com/: 200
[SUCCESS] https://github.com: 200
[SUCCESS] https://www.rust-lang.org: 200

Kotlin async requests

In Kotlin, we combine coroutines with the asynchronous API of the JDK HttpClient. The coroutines come from the kotlinx-coroutines-core library.

plugins {
    kotlin("jvm") version "2.3.0"
    application
}

repositories {
    mavenCentral()
}

dependencies {
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.10.2")
}

kotlin {
    jvmToolchain(25)
}

application {
    mainClass = "MainKt"
}

The Gradle build adds the coroutines library. Kotlin 2.3 is the first release that can target JDK 25.

import java.net.URI
import java.net.http.HttpClient
import java.net.http.HttpRequest
import java.net.http.HttpResponse
import java.time.Duration
import kotlin.time.measureTime
import kotlinx.coroutines.async
import kotlinx.coroutines.awaitAll
import kotlinx.coroutines.future.await
import kotlinx.coroutines.runBlocking

val urls = listOf(
    "https://example.com",
    "https://httpbin.org/get",
    "https://kotlinlang.org",
    "https://www.python.org",
    "https://github.com",
    "https://httpbin.org/delay/8",
    "https://httpbin.org/status/404",
    "https://no-such-host.invalid",
)

suspend fun getStatus(client: HttpClient, url: String): String {
    val request = HttpRequest.newBuilder(URI.create(url))
        .timeout(Duration.ofSeconds(5))
        .build()

    return try {
        val resp = client.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray()).await()
        "$url: ${resp.statusCode()} (${resp.body().size} bytes)"
    } catch (e: Exception) {
        "$url: ${e::class.simpleName}"
    }
}

fun main() = runBlocking {
    HttpClient.newBuilder()
        .connectTimeout(Duration.ofSeconds(5))
        .followRedirects(HttpClient.Redirect.NORMAL)
        .build().use { client ->

            val elapsed = measureTime {
                val results = urls.map { url -> async { getStatus(client, url) } }.awaitAll()
                results.forEach(::println)
            }

            println("finished in %.2f s".format(elapsed.inWholeMilliseconds / 1000.0))
        }
}

The example prints the status code and the body size of each URL, followed by the total time.

val resp = client.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray()).await()

The sendAsync method returns a CompletableFuture. The await extension function from kotlinx.coroutines.future suspends the coroutine until the future completes, without blocking a thread. A failed future is rethrown as an ordinary exception, which we catch in the same function.

val results = urls.map { url -> async { getStatus(client, url) } }.awaitAll()

The async builder starts a coroutine for each URL and returns a Deferred. The awaitAll function waits for all of them and returns the results in the order of the list.

fun main() = runBlocking {

The runBlocking builder bridges the blocking main function and the coroutine world; it returns when all coroutines started inside it have finished.

The delay/8 endpoint waits eight seconds before it answers; the five-second request timeout ends it with HttpTimeoutException.

$ gradle -q run
https://example.com: 200 (577 bytes)
https://httpbin.org/get: 200 (193 bytes)
https://kotlinlang.org: 200 (73629 bytes)
https://www.python.org: 200 (53029 bytes)
https://github.com: 200 (530402 bytes)
https://httpbin.org/delay/8: HttpTimeoutException
https://httpbin.org/status/404: 404 (0 bytes)
https://no-such-host.invalid: ConnectException
finished in 5.02 s

PHP async requests

In PHP, we use the multi interface of the cURL library.

<?php

$urls = [
    "https://example.com",
    "https://httpbin.org/get",
    "https://www.perl.org",
    "https://www.php.net",
    "https://www.python.org",
    "https://ifconfig.me",
    "https://httpbin.org/status/404",
    "https://no-such-host.invalid",
];

$options = [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_FOLLOWLOCATION => true,
    CURLOPT_CONNECTTIMEOUT => 5,
    CURLOPT_TIMEOUT => 10,
    CURLOPT_USERAGENT => 'PHP curl_multi example',
];

$mh = curl_multi_init();
$chs = [];

foreach ($urls as $url) {

    $ch = curl_init($url);
    curl_setopt_array($ch, $options);
    curl_setopt($ch, CURLOPT_PRIVATE, $url);
    curl_multi_add_handle($mh, $ch);
    $chs[$url] = $ch;
}

$results = [];

do {
    $status = curl_multi_exec($mh, $running);

    if ($running) {
        curl_multi_select($mh);
    }

    while ($info = curl_multi_info_read($mh)) {
        $results[curl_getinfo($info['handle'], CURLINFO_PRIVATE)] = $info['result'];
    }
} while ($running && $status === CURLM_OK);

foreach ($chs as $url => $ch) {

    if ($results[$url] !== CURLE_OK) {
        printf("%s: error: %s\n", $url, curl_strerror($results[$url]));
    } else {
        $code = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
        $size = strlen(curl_multi_getcontent($ch));
        printf("%s: %d (%d bytes)\n", $url, $code, $size);
    }

    curl_multi_remove_handle($mh, $ch);
}

curl_multi_close($mh);

We print the status code and the size of the body of each requested page.

$mh = curl_multi_init();

The curl_multi_init function creates a new multi handle, which allows the processing of multiple cURL handles asynchronously. Each handle is added to it with curl_multi_add_handle. The CURLOPT_PRIVATE option stores the URL on the handle.

do {
    $status = curl_multi_exec($mh, $running);

    if ($running) {
        curl_multi_select($mh);
    }
    ...
} while ($running && $status === CURLM_OK);

The curl_multi_exec performs the work that is possible without waiting. The curl_multi_select then sleeps until one of the connections has activity. Without it, the loop would keep the CPU busy.

while ($info = curl_multi_info_read($mh)) {
    $results[curl_getinfo($info['handle'], CURLINFO_PRIVATE)] = $info['result'];
}

The curl_multi_info_read returns the result of every finished transfer. For handles in a multi handle, curl_errno is not set, so this is how we learn about errors such as an unresolvable host.

$ php async_req.php
https://example.com: 200 (577 bytes)
https://httpbin.org/get: 200 (213 bytes)
https://www.perl.org: 200 (12401 bytes)
https://www.php.net: 200 (68912 bytes)
https://www.python.org: 200 (53029 bytes)
https://ifconfig.me: 200 (14 bytes)
https://httpbin.org/status/404: 404 (0 bytes)
https://no-such-host.invalid: error: Could not resolve hostname

Rust async requests

In Rust, we use the reqwest HTTP client on top of the tokio runtime.

$ cargo add tokio --features full
$ cargo add reqwest

use std::time::{Duration, Instant};

use tokio::task::JoinSet;

async fn get_status(client: reqwest::Client, url: &str) -> String {
    match client.get(url).send().await {
        Ok(resp) => {
            let status = resp.status();
            match resp.bytes().await {
                Ok(body) => format!("{url}: {} ({} bytes)", status.as_u16(), body.len()),
                Err(e) => format!("{url}: error: {e}"),
            }
        }
        Err(e) if e.is_timeout() => format!("{url}: timeout"),
        Err(e) if e.is_connect() => format!("{url}: connection error"),
        Err(e) => format!("{url}: error: {e}"),
    }
}

#[tokio::main]
async fn main() -> Result<(), reqwest::Error> {
    let urls = [
        "https://example.com",
        "https://httpbin.org/get",
        "https://www.rust-lang.org",
        "https://go.dev",
        "https://github.com",
        "https://httpbin.org/delay/8",
        "https://httpbin.org/status/404",
        "https://no-such-host.invalid",
    ];

    let client = reqwest::Client::builder()
        .timeout(Duration::from_secs(5))
        .user_agent("Rust reqwest example")
        .build()?;

    let start = Instant::now();
    let mut set = JoinSet::new();

    for url in urls {
        set.spawn(get_status(client.clone(), url));
    }

    while let Some(res) = set.join_next().await {
        match res {
            Ok(line) => println!("{line}"),
            Err(e) => eprintln!("task failed: {e}"),
        }
    }

    println!("finished in {:.2} s", start.elapsed().as_secs_f64());

    Ok(())
}

The example prints the status code and the body size of each URL as soon as its request finishes, followed by the total time.

let client = reqwest::Client::builder()
    .timeout(Duration::from_secs(5))
    .user_agent("Rust reqwest example")
    .build()?;

The Client keeps a connection pool. It is cheap to clone, because the clones share the same pool; each task gets its own clone. The timeout covers the whole request, including reading the body. Redirects are followed by default.

let mut set = JoinSet::new();

for url in urls {
    set.spawn(get_status(client.clone(), url));
}

Calling an async fn only creates a future; nothing runs until it is polled. The spawn method hands each future to the tokio runtime, which runs them concurrently. The JoinSet owns the spawned tasks.

while let Some(res) = set.join_next().await {

The join_next method returns the next task to finish, so the lines come in the order of completion. It returns None when the set is empty. The outer Result is an error only if the task panicked or was cancelled.

Err(e) if e.is_timeout() => format!("{url}: timeout"),
Err(e) if e.is_connect() => format!("{url}: connection error"),

A reqwest::Error tells us what kind of failure occurred. An HTTP 404 is not an error; send returns it as a normal response. To turn such statuses into errors, call error_for_status on the response.

$ cargo run -q
https://no-such-host.invalid: connection error
https://example.com: 200 (577 bytes)
https://httpbin.org/status/404: 404 (0 bytes)
https://httpbin.org/get: 200 (211 bytes)
https://go.dev: 200 (63613 bytes)
https://github.com: 200 (530371 bytes)
https://www.rust-lang.org: 200 (18594 bytes)
https://httpbin.org/delay/8: timeout
finished in 5.00 s

Raku async requests

Raku has promises built into the language. We run the blocking HTTP::UserAgent client in start blocks, which execute on the thread pool of the Raku runtime.

$ zef install HTTP::UserAgent IO::Socket::SSL

use HTTP::UserAgent;
use IO::Socket::SSL;

my @urls = <
    https://example.com
    https://httpbin.org/get
    https://raku.org
    https://www.python.org
    https://github.com
    https://httpbin.org/delay/8
    https://httpbin.org/status/404
    https://no-such-host.invalid
>;

my $start = now;

my @promises = @urls.map: -> $url {
    start {
        my $ua = HTTP::UserAgent.new(:useragent('Raku example'));
        $ua.get($url).status-line;
    }
}

await Promise.anyof(Promise.allof(@promises), Promise.in(5));

for @urls Z @promises -> ($url, $p) {
    given $p.status {
        when Kept    { say "$url: {$p.result}" }
        when Broken  { say "$url: error: {$p.cause.message.lines.head}" }
        when Planned { say "$url: timeout" }
    }
}

say "finished in {(now - $start).fmt('%.2f')} s";

The example prints the status line of each URL, followed by the total time.

use IO::Socket::SSL;

The HTTP::UserAgent loads the SSL module on its first HTTPS request. When many threads make their first request at the same time, this lazy loading can fail with Cannot find method 'protect' on object of type NQPMu. Loading the module at compile time avoids the race.

my @promises = @urls.map: -> $url {
    start {
        my $ua = HTTP::UserAgent.new(:useragent('Raku example'));
        $ua.get($url).status-line;
    }
}

The start block returns a Promise right away and runs its code on another thread. The value of the last statement becomes the result of the promise; an exception breaks it. Each block creates its own user agent, so the threads share no state.

await Promise.anyof(Promise.allof(@promises), Promise.in(5));

The timeout option of HTTP::UserAgent applies to connecting, not to waiting for the response, so we set a deadline for the whole batch. Promise.allof is kept when all promises have finished; Promise.in(5) is kept after five seconds. The anyof combinator waits for whichever comes first.

given $p.status {
    when Kept    { say "$url: {$p.result}" }
    when Broken  { say "$url: error: {$p.cause.message.lines.head}" }
    when Planned { say "$url: timeout" }
}

A finished promise is either Kept or Broken; the cause method returns the exception of a broken one. A promise that is still Planned after the deadline is reported as a timeout. Its thread keeps running, but the program ends after the last statement.

$ raku async_req.raku
https://example.com: 200 OK
https://httpbin.org/get: 200 OK
https://raku.org: 200 OK
https://www.python.org: 200 OK
https://github.com: 200 OK
https://httpbin.org/delay/8: timeout
https://httpbin.org/status/404: 404 Not Found
https://no-such-host.invalid: error: Failed to resolve host name 'no-such-host.invalid' with family 0.
finished in 5.02 s

Raku async requests with Cro

Cro is a set of Raku libraries for building HTTP services and clients. Unlike HTTP::UserAgent, its Cro::HTTP::Client uses non-blocking I/O and returns promises directly, so we do not need start blocks.

$ zef install Cro::HTTP

use Cro::HTTP::Client;

my @urls = <
    https://example.com
    https://httpbin.org/get
    https://raku.org
    https://www.python.org
    https://github.com
    https://httpbin.org/delay/8
    https://httpbin.org/status/404
    https://no-such-host.invalid
>;

my $client = Cro::HTTP::Client.new(
    timeout => %( connection => 5, headers => 5 ),
    headers => [ User-Agent => 'Raku Cro example' ],
);

my $start = now;

react {
    for @urls -> $url {
        whenever $client.get($url) -> $resp {
            whenever $resp.body-blob -> $body {
                say "$url: {$resp.status} ({$body.bytes} bytes)";
            }
            QUIT {
                when X::Cro::HTTP::Error { say "$url: {.response.status}" }
                when X::Cro::HTTP::Client::Timeout { say "$url: timeout" }
                default { say "$url: error: {.message.lines.head}" }
            }
        }
    }
}

say "finished in {(now - $start).fmt('%.2f')} s";

The example prints the status code and the body size of each URL as its response arrives, followed by the total time.

my $client = Cro::HTTP::Client.new(
    timeout => %( connection => 5, headers => 5 ),
    headers => [ User-Agent => 'Raku Cro example' ],
);

One client is shared by all requests. The timeout is set per phase: five seconds to connect and five seconds to receive the response headers. Cro follows redirects by default.

react {
    for @urls -> $url {
        whenever $client.get($url) -> $resp {
            ...
        }
    }
}

The get method returns a Promise that is kept when the response headers arrive. Inside a react block, each whenever subscribes to a promise and runs its block when the promise is kept. The react block ends when all subscriptions are done, which is when all requests have finished.

whenever $resp.body-blob -> $body {
    say "$url: {$resp.status} ({$body.bytes} bytes)";
}

The body is read separately. The body-blob method returns another promise, so we nest a second whenever.

QUIT {
    when X::Cro::HTTP::Error { say "$url: {.response.status}" }
    when X::Cro::HTTP::Client::Timeout { say "$url: timeout" }
    default { say "$url: error: {.message.lines.head}" }
}

A broken promise ends the whole react block, unless the whenever has a QUIT phaser, which handles the exception. Cro treats 4xx and 5xx responses as errors; the exception carries the response. A timeout and a DNS failure have their own exceptions.

$ raku cro_req.raku
https://no-such-host.invalid: error: Failed to resolve host name 'no-such-host.invalid' with family 0.
https://example.com: 200 (577 bytes)
https://www.python.org: 200 (53029 bytes)
https://httpbin.org/get: 200 (186 bytes)
https://httpbin.org/status/404: 404
https://raku.org: 200 (31429 bytes)
https://github.com: 200 (530207 bytes)
https://httpbin.org/delay/8: timeout
finished in 5.86 s

Comparison

The table summarizes the approaches used in this tutorial.

Language	Concurrency model	HTTP client	Waiting for all
Go	goroutines	net/http standard library	sync.WaitGroup
C#	async/await, tasks	HttpClient .NET	Task.WhenAll
F#	task expressions, async workflows	HttpClient .NET	Task.WhenAll, Async.Parallel
Groovy	CompletableFuture	HttpClient JDK	CompletableFuture.allOf
Python	asyncio coroutines	httpx.AsyncClient httpx	asyncio.TaskGroup
Perl	event loop, promises	Mojo::UserAgent Mojolicious	Mojo::Promise->all
JavaScript	event loop, promises	fetch built into Node.js 18+	Promise.allSettled
Java	CompletableFuture	HttpClient JDK	CompletableFuture.allOf
Kotlin	coroutines	HttpClient JDK, kotlinx.coroutines	awaitAll
PHP	cURL multi interface	curl_multi_* curl extension	curl_multi_exec and curl_multi_select loop
Rust	async/await, futures	reqwest::Client reqwest, tokio	JoinSet
Raku	promises on a thread pool	HTTP::UserAgent zef module	Promise.allof
Raku	promises, react/whenever	Cro::HTTP::Client Cro	react block

In this tutorial we have generated asynchronous web requests in Go, C#, F#, Groovy, Python, Perl, JavaScript, Java, Kotlin, PHP, Rust, and Raku.