ZetCode

Asynchronous HTTP requests

last modified January 10, 2023

The asynchronous HTTP requests tutorial shows how to create async HTTP requests in Go, C#, F#, Groovy, Python, Perl, Java, JavaScript, and PHP.

Asynchronous requests do not block the client and allow us to generate HTTP requests more efficiently.

Rather than generating requests one by one, waiting for the current request to finish before executing next one, we execute all requests quickly and then wait for all of them to finish.

Go async requests

Go has goroutines for making asynchronous requests. A goroutine is a lightweight thread managed by the Go runtime.

main.go
package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "regexp"
    "sync"
)

func main() {

    urls := []string{
        "http://webcode.mse",
        "https://example.com",
        "http://httpbin.org",
        "https://www.perl.org",
        "https://www.php.net",
        "https://www.python.org",
        "https://code.visualstudio.com",
        "https://clojure.org",
    }

    var wg sync.WaitGroup

    for _, u := range urls {

        wg.Add(1)
        go func(url string) {

            defer wg.Done()

            content := doReq(url)
            title := getTitle(content)
            fmt.Println(title)
        }(u)
    }

    wg.Wait()
}

func doReq(url string) (content string) {

    resp, err := http.Get(url)

    if err != nil {

        log.Println(err)
        return
    }

    defer resp.Body.Close()

    body, err := ioutil.ReadAll(resp.Body)

    if err != nil {

        log.Println(err)
        return
    }

    return string(body)
}

func getTitle(content string) (title string) {

    re := regexp.MustCompile("<title>(.*)</title>")

    parts := re.FindStringSubmatch(content)

    if len(parts) > 0 {
        return parts[1]
    } else {
        return "no title"
    }
}

We make multiple asynchronous HTTP requests. We get the contents of the title tag of each of the web pages. 

var wg sync.WaitGroup

WaitGroups are used to manage goroutines. It waits for a collection of goroutines to finish. 

go func(url string) {

  defer wg.Done()

  content := doReq(url)
  title := getTitle(content)
  fmt.Println(title)
}(u)

A goroutine is created with the go keyword. 

$ go run async_req.go 
The Perl Programming Language - www.perl.org
Welcome to Python.org
Visual Studio Code - Code Editing. Redefined
PHP: Hypertext Preprocessor
Example Domain
httpbin.org
Clojure
My html page

C# async requests

In C#, we use the HttpClient to generate asynchronous requests.

Program.cs
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using System.Text.RegularExpressions;

var urls = new string[] { "http://webcode.me", "http://example.com",
    "http://httpbin.org", "https://ifconfig.me", "http://termbin.com",
    "https://github.com"
};

var rx = new Regex(@"<title>\s*(.+?)\s*</title>",
  RegexOptions.Compiled);

using var client = new HttpClient();

var tasks = new List<Task<string>>();

foreach (var url in urls)
{
    tasks.Add(client.GetStringAsync(url));
}

Task.WaitAll(tasks.ToArray());

var data = new List<string>();

foreach (var task in tasks)
{
    data.Add(await task);
}

foreach (var content in data)
{
    var matches = rx.Matches(content);

    foreach (var match in matches)
    {
        Console.WriteLine(match);
    }
}

We download the given web pages asynchronously and print their HTML title tags. 

tasks.Add(client.GetStringAsync(url));

The GetStringAsync sends a GET request to the specified url and returns the response body as a string in an asynchronous operation. It returns a new task; in C# a task represents an asynchronous operation. 

Task.WaitAll(tasks.ToArray());

The Task.WaitAll waits for all of the provided tasks to complete execution. 

data.Add(await task);

The await keywords unwraps the result value. 

$ dotnet run
<title>My html page</title>
<title>Example Domain</title>
<title>httpbin.org</title>
<title>termbin.com - terminal pastebin</title>
<title>GitHub: Where the world builds software · GitHub</title>

F# async requests

The following example uses HttpClient and task expressions to fetch website titles asynchronously.

async_req.fsx
open System.Net.Http
open System.Text.RegularExpressions
open System.Threading.Tasks

let fetchTitleAsync (url: string) =

    task {

        use client = new HttpClient()
        let! html = client.GetStringAsync(url)
        let pattern = "<title>\s*(.+?)\s*</title>"

        let m = Regex.Match(html, pattern)
        return m.Value
    }

let sites =
    [| "http://webcode.me"
       "http://example.com"
       "https://bing.com"
       "http://httpbin.org"
       "https://ifconfig.me"
       "http://termbin.com"
       "https://github.com" |]

let titles =
    sites
    |> Array.map fetchTitleAsync
    |> Task.WhenAll
    |> Async.AwaitTask
    |> Async.RunSynchronously

titles
|> Array.iter (fun title -> printfn $"%s{title}")

The example asynchronously retrieves the titles of the given urls.

Another solution uses the WebRequest to generate a request. Its GetResponseStream returns a response to a request as an asynchronous operation.

async_req2.fsx
open System.Net
open System
open System.Text.RegularExpressions

let fetchTitleAsync url =

    async {
        let req = WebRequest.Create(Uri(url))
        use! resp = req.AsyncGetResponse()
        use stream = resp.GetResponseStream()

        use reader = new IO.StreamReader(stream)
        let html = reader.ReadToEnd()

        let pattern = "<title>\s*(.+?)\s*</title>"

        let m = Regex.Match(html, pattern)
        return m.Value
    }

let sites =
    [ "http://webcode.me"
      "http://example.com"
      "https://bing.com"
      "http://httpbin.org"
      "https://ifconfig.me"
      "http://termbin.com"
      "https://github.com" ]

let titles = sites
            |> List.map fetchTitleAsync
            |> Async.Parallel
            |> Async.RunSynchronously

titles |> Array.iter (fun title -> printfn $"%s{title}")

The example asynchronously retrieves the titles of the given urls.

Groovy async requests

In Groovy, we use ExecutorService and HttpClient.

mul_async_req.gvy
import java.util.concurrent.Executors
import java.util.concurrent.TimeUnit

import java.net.http.HttpClient
import java.net.http.HttpRequest
import java.net.http.HttpResponse

int nThreads = 30

def executor = Executors.newFixedThreadPool(nThreads)

def urls = [
    "https://crunchify.com",
    "https://yahoo.com",
    "https://www.ebay.com",
    "https://google.com",
    "https://www.example.co",
    "https://paypal.com",
    "http://bing.com/",
    "https://techcrunch.com/",
    "http://mashable.com/",
    "https://pro.crunchify.com/",
    "https://wordpress.com/",
    "https://wordpress.org/",
    "https://example.com/",
    "https://sjsu.edu/",
    "https://ask.crunchify.com/",
    "https://test.com.au/",
    "https://www.wikipedia.org/",
    "https://en.wikipedia.org"
]

for (String url in urls ) {

    executor.execute(() -> {

        worker(url)

        // try {
        //     worker(url)
        // } catch (Exception e) {
        //     e.printStackTrace()
        // }
    })
}

executor.shutdown()

executor.awaitTermination(30, TimeUnit.SECONDS)
println("finished")

def worker(url) {

    def client = HttpClient.newHttpClient()
    def request = HttpRequest.newBuilder()
        .uri(URI.create(url))
        .build()

    HttpResponse<Void> res = client.send(request,
            HttpResponse.BodyHandlers.discarding())

    println "${url}: ${res.statusCode()}"
}

The example makes multiple asynchronous requests to URLs and prints their status codes. 

$ groovy mul_async_req.gvy
http://mashable.com/: 301
http://bing.com/: 301
https://paypal.com: 302
https://en.wikipedia.org: 301
https://paypal.com: 302
https://en.wikipedia.org: 301
https://en.wikipedia.org: 301
https://google.com: 301
https://example.com/: 200
https://example.com/: 200
https://yahoo.com: 301
https://test.com.au/: 301
https://wordpress.com/: 200
https://techcrunch.com/: 200
https://www.ebay.com: 200
https://ask.crunchify.com/: 200
https://pro.crunchify.com/: 200
https://sjsu.edu/: 200
finished

Python async requests

In Python, we use the httpx and asyncio modules.

async_req.py
#!/usr/bin/python

import httpx
import asyncio

async def get_async(url):
    async with httpx.AsyncClient() as client:
        return await client.get(url)

urls = ['http://webcode.me', 'https://httpbin.org/get',
    'https://google.com', 'https://stackoverflow.com',
    'https://github.com']

async def launch():
    resps = await asyncio.gather(*map(get_async, urls))
    data = [resp.status_code for resp in resps]

    for status_code in data:
        print(status_code)

asyncio.run(launch())

The example makes asynchronous requests in Python. It prints the status code of all the provided urls. 

./async_req.py
200
200
200
200
200

Perl async requests

In Perl, we use the LWP module to generate requests and the Parallel::ForkManager module to make them asynchronous. 

$ cpanm Parallel::ForkManager LWP

We install the modules with cpanm.

urls.txt
http://webcode.me
https://example.com
http://httpbin.org
https://google.com
https://www.perl.org
https://fsharp.org
https://clojure.org
https://www.rust-lang.org
https://golang.org
https://www.python.org
https://code.visualstudio.com
https://ifconfig.me
http://termbin.com
https://github.com
https://stackoverflow.com
https://www.php.net/

The urls.txt contains a list of websites.

async_req.pl
#!/usr/bin/perl

use warnings;
use 5.30.0;
use Path::Tiny;
use LWP::UserAgent;
use Parallel::ForkManager;

my @urls = split "\n", path('urls.txt')->slurp_utf8;

my $pm = Parallel::ForkManager->new(4);
my $ua = LWP::UserAgent->new;
$ua->agent('Perl script');

say "downloading ", scalar @urls, " files";

my $dir = 'files/';
mkdir $dir if not -d $dir;

foreach my $link (@urls) {

    my $name = $1 if $link =~ m%https?://(.+)\.\w+%;
    my $file_name = "$dir/$name" . '.txt';

    $pm->start and next;

    my $resp = $ua->get($link);

    if ($resp->is_success) {

        path($file_name)->spew_utf8($resp->decoded_content);

    } else { warn $resp->status_line }

    $pm->finish;
}

$pm->wait_all_children;

The example reads the urls.txt file and gets the links. It generates async requests to the given urls. The contents of the web pages are written to files. 

$ ./async_req.pl
downloading 15 files
$ ls -1 files/
clojure.txt
code.visualstudio.txt
example.txt
fsharp.txt
github.txt
golang.txt
google.txt
httpbin.txt
ifconfig.txt
stackoverflow.txt
termbin.txt
webcode.txt
www.perl.txt
www.python.txt
www.rust-lang.txt

JS async requests

For JavaScript, we have chosen the axios module. 

$ npm i axios

We install the axios module.

async_req.js
const axios = require('axios');

async function makeRequests(urls) {

    const fetchUrl = (url) => axios.get(url);
    const promises = urls.map(fetchUrl);

    let responses = await Promise.all(promises);

    responses.forEach(resp => {
        let msg = `${resp.config.url} -> ${resp.headers.server}: ${resp.status}`;
        console.log(msg);
    });
}

let urls = [
    'http://webcode.me',
    'https://example.com',
    'http://httpbin.org',
    'https://clojure.org',
    'https://fsharp.org',
    'https://symfony.com',
    'https://www.perl.org',
    'https://www.php.net',
    'https://www.python.org',
    'https://code.visualstudio.com',
    'https://github.com'
];

makeRequests(urls);

The example generates async requests to the given list of urls. It prints the web site's url, server name, and status code. 

const fetchUrl = (url) => axios.get(url);

The axios.get makes an async request and returns a promise. 

let responses = await Promise.all(promises);

We collect all promises with Promise.All. The method resolves after all of the given promises have either fulfilled or rejected. 

$ node async_req.js
http://webcode.me -> nginx/1.6.2: 200
https://example.com -> ECS (dcb/7F83): 200
http://httpbin.org -> gunicorn/19.9.0: 200
https://clojure.org -> AmazonS3: 200
https://fsharp.org -> GitHub.com: 200
https://symfony.com -> cloudflare: 200
https://www.perl.org -> Combust/Plack (Perl): 200
https://www.php.net -> myracloud: 200
https://www.python.org -> nginx: 200
https://code.visualstudio.com -> Microsoft-IIS/10.0: 200
https://github.com -> GitHub.com: 200

Java async requests

The CompletableFuture a high-level API for asynchronous programming in Java.

com/zetcode/AsyncReqEx.java
package com.zetcode;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.stream.Stream;

import static java.util.stream.Collectors.toList;

public class AsyncReqEx {

    public static void main(String[] args) {

        List<URI> uris = Stream.of(
                "https://www.google.com/",
                "https://clojure.org",
                "https://www.rust-lang.org",
                "https://golang.org",
                "https://www.python.org",
                "https://code.visualstudio.com",
                "https://ifconfig.me",
                "http://termbin.com",
                "https://www.github.com/"
        ).map(URI::create).collect(toList());

        HttpClient httpClient = HttpClient.newBuilder()
                .connectTimeout(Duration.ofSeconds(10))
                .followRedirects(HttpClient.Redirect.ALWAYS)
                .build();

        var futures = uris.stream()
                .map(uri -> verifyUri(httpClient, uri))
                .toArray(CompletableFuture[]::new);

        CompletableFuture.allOf(futures).join();
    }

    private static CompletableFuture<Void> verifyUri(HttpClient httpClient,
                                                     URI uri) {
        HttpRequest request = HttpRequest.newBuilder()
                .timeout(Duration.ofSeconds(5))
                .uri(uri)
                .build();

        return httpClient.sendAsync(request, HttpResponse.BodyHandlers.discarding())
                .thenApply(HttpResponse::statusCode)
                .thenApply(statusCode -> statusCode == 200)
                .exceptionally(ex -> false)
                .thenAccept(valid -> {
                    if (valid) {
                        System.out.printf("[SUCCESS] Verified %s%n", uri);
                    } else {
                        System.out.printf("[FAILURE] Failed to verify%s%n", uri);
                    }
                });
    }
}

In the example, we have a list of urls. We check the status of the given web pages. The example uses HttpClient for making a web request and CompletableFuture for asynchronous execution. 

[SUCCESS] Verified http://termbin.com
[SUCCESS] Verified https://clojure.org
[SUCCESS] Verified https://www.google.com/
[SUCCESS] Verified https://ifconfig.me
[SUCCESS] Verified https://www.python.org
[SUCCESS] Verified https://code.visualstudio.com
[SUCCESS] Verified https://golang.org
[SUCCESS] Verified https://www.rust-lang.org
[SUCCESS] Verified https://www.github.com/

PHP async requests

In PHP, we use the cURL library.

async_req.php
<?php

$urls = [
    "http://webcode.me",
    "https://example.com",
    "http://httpbin.org",
    "https://www.perl.org",
    "https://www.php.net",
    "https://www.python.org",
    "https://code.visualstudio.com",
    "https://ifconfig.me"
];

$options = [CURLOPT_HEADER => true, CURLOPT_NOBODY => true,
    CURLOPT_RETURNTRANSFER => true];

$mh = curl_multi_init();
$chs = [];


foreach ($urls as $url) {

    $ch = curl_init($url);
    curl_setopt_array($ch, $options);
    curl_multi_add_handle($mh, $ch);
    $chs[] = $ch;
}

$running = false;

do {
    curl_multi_exec($mh, $running);
} while ($running);

foreach ($chs as $h) {

    curl_multi_remove_handle($mh, $h);
}

curl_multi_close($mh);

foreach ($chs as $h) {

    $status = curl_getinfo($h, CURLINFO_RESPONSE_CODE);
    echo $status . "\n";
}

foreach ($chs as $h) {

    echo "----------------------\n";
    echo curl_multi_getcontent($h);
}

We print the status codes and headers of the requested web pages. 

$ch = curl_init($url);

The curl_multi_init function creates a new multi handle, which allows the processing of multiple cURL handles asynchronously. 

$ php async_req.php
200
200
200
200
200
200
200
200
----------------------
HTTP/1.1 200 OK
Server: nginx/1.6.2
Date: Thu, 22 Jul 2021 13:14:22 GMT
Content-Type: text/html
Content-Length: 348
Last-Modified: Sat, 20 Jul 2019 11:49:25 GMT
Connection: keep-alive
ETag: "5d32ffc5-15c"
Accept-Ranges: bytes

----------------------
HTTP/2 200
content-encoding: gzip
accept-ranges: bytes
...

In this tutorial we have generated asynchronous web requests in Go, C#, F#, Python, Perl, Java, JavaScript, and PHP.