ZetCode

Go Pipe

last modified November 15, 2020

Go pipe tutorial shows how to work with pipes in Golang.

Pipe

A pipe is a form of redirection from one process to another process. It is a unidirectional data channel that can be used for interprocess communication.

The io.Pipe function creates a synchronous in-memory pipe. It can be used to connect code expecting an io.Reader with code expecting an io.Writer.

Go pipe simple example

The following example demonstrates the usage of the io.Pipe function.

simple.go
package main

import (
    "fmt"
    "io"
    "log"
    "os"
)

func main() {
    r, w := io.Pipe()

    go func() {
        fmt.Fprint(w, "Hello there\n")
        w.Close()
    }()

    _, err := io.Copy(os.Stdout, r)

    if err != nil {
        log.Fatal(err)
    }
}

In the example, we create a pipe with io.Pipe. We write to the pipe's writer in a goroutine and then copy the data from the pipe's reader to the standard output using io.Copy.

go func() {
    fmt.Fprint(w, "Hello there\n")
    w.Close()
}()

We write some data to the pipe's writer in a goroutine. Each write to the PipeWriter blocks until it has satisfied one or more reads from the PipeReader that fully consume the written data.

$ go run simple.go 
Hello there

This is the output.

Go cmd StdoutPipe

The StdoutPipe of a command returns a pipe that will be connected to the command's standard output when the command starts.

pingcmd.go
package main

import (
    "bufio"
    "fmt"
    "log"
    "os"
    "os/exec"
)

func main() {

    cmd := exec.Command("ping", "webcode.me")
    stdout, err := cmd.StdoutPipe()

    if err != nil {
        log.Fatal(err)
    }

    cmd.Start()

    buf := bufio.NewReader(stdout) 
    num := 0

    for {
        line, _, _ := buf.ReadLine()
        if num > 3 {
            os.Exit(0)
        }
        num += 1
        fmt.Println(string(line))
    }
}

In the example, we launch a ping command and read four lines of its output.

cmd := exec.Command("ping", "webcode.me")

We create a command which lauches ping to test the availability of the webcode.me website.

stdout, err := cmd.StdoutPipe()

We get the standard output of the command.

buf := bufio.NewReader(stdout) 

A reader from the standard output is created.

for {
    line, _, _ := buf.ReadLine()
    if num > 3 {
        os.Exit(0)
    }
    num += 1
    fmt.Println(string(line))
}

In a for loop, we read four lines and print them to the console.

$ go run pingcmd.go 
PING webcode.me (46.101.248.126) 56(84) bytes of data.
64 bytes from 46.101.248.126 (46.101.248.126): icmp_seq=1 ttl=54 time=29.7 ms
64 bytes from 46.101.248.126 (46.101.248.126): icmp_seq=2 ttl=54 time=35.9 ms
64 bytes from 46.101.248.126 (46.101.248.126): icmp_seq=3 ttl=54 time=37.4 ms

This is a sample output.

Go pipe POST JSON data

In the following example, we post JSON data to the https://httpbin.org/post.

post_json.go
package main

import (
    "encoding/json"
    "fmt"
    "io"
    "io/ioutil"
    "log"
    "net/http"
)

type PayLoad struct {
    Content string
}

func main() {

    r, w := io.Pipe()

    go func() {
        defer w.Close()

        err := json.NewEncoder(w).Encode(&PayLoad{Content: "Hello there!"})

        if err != nil {
            log.Fatal(err)
        }
    }()

    resp, err := http.Post("https://httpbin.org/post", "application/json", r)

    if err != nil {
        log.Fatal(err)
    }

    body, err := ioutil.ReadAll(resp.Body)

    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(string(body))
}

The example posts a JSON payload to the website and reads its body response.

go func() {
    defer w.Close()

    err := json.NewEncoder(w).Encode(&PayLoad{Content: "Hello there!"})

    if err != nil {
        log.Fatal(err)
    }
}()

We write a JSON payload to the PipeWriter in a goroutine.

resp, err := http.Post("https://httpbin.org/post", "application/json", r)

The http.Post method expects a reader as its third parameter; we pass a PipeReader there.

body, err := ioutil.ReadAll(resp.Body)

if err != nil {
    log.Fatal(err)
}

fmt.Println(string(body))

Finally, we read the response body and print it.

$ go run post_json.go 
{
  "args": {}, 
  "data": "{\"Content\":\"Hello there!\"}\n", 
  "files": {}, 
  "form": {}, 
  "headers": {
...

This is the output.

Go read standard input through pipe

The following example creates a Go program, which reads data passed from standard intput via the pipe.

read_stdin.go
package main

import (
    "bufio"
    "fmt"
    "io"
    "log"
    "os"
)

func main() {

    nBytes, nChunks := int64(0), int64(0)
    r := bufio.NewReader(os.Stdin)
    buf := make([]byte, 0, 4*1024)

    for {

        n, err := r.Read(buf[:cap(buf)])
        buf = buf[:n]

        if n == 0 {

            if err == nil {
                continue
            }

            if err == io.EOF {
                break
            }

            log.Fatal(err)
        }

        nChunks++
        nBytes += int64(len(buf))

        fmt.Println(string(buf))

        if err != nil && err != io.EOF {
            log.Fatal(err)
        }
    }

    fmt.Println("Bytes:", nBytes, "Chunks:", nChunks)
}

The example reads the data from the standard input and prints the data and the number of bytes and chunks read.

r := bufio.NewReader(os.Stdin)

We create a reader from the standard input.

buf := make([]byte, 0, 4*1024)

Here we create a buffer of 4KB.

n, err := r.Read(buf[:cap(buf)])
buf = buf[:n]

We read the data from the standard input into the buffer.

nChunks++
nBytes += int64(len(buf))

We calculate the number of chunks and bytes read.

fmt.Println(string(buf))

We print the contents of the buffer to the terminal.

$ date | go run read_stdin.go 
Sun 15 Nov 2020 01:08:13 PM CET

Bytes: 32 Chunks: 1

We pass the output of the | operator to our program. The Go program reads the data, counts the number of bytes and prints the data to the console.

Go Stat

The Stat function returns the FileInfo structure describing file. We can use it to check if there is some data from the pipe on the terminal.

hello.go
package main

import (
    "bufio"
    "fmt"
    "log"
    "os"
)

func main() {
    stat, _ := os.Stdin.Stat()

    if (stat.Mode() & os.ModeCharDevice) == 0 {

        var buf []byte
        scanner := bufio.NewScanner(os.Stdin)

        for scanner.Scan() {
            buf = append(buf, scanner.Bytes()...)
        }

        if err := scanner.Err(); err != nil {
            log.Fatal(err)
        }

        fmt.Printf("Hello %s!\n", buf)

    } else {
        fmt.Print("Enter your name: ")

        var name string
        fmt.Scanf("%s", &name)
        fmt.Printf("Hello %s!\n", name)
    }
}

The example takes the input from the user either via the pipe or from the prompt.

stat, _ := os.Stdin.Stat()

We get the FileInfo structure of the standard input.

if (stat.Mode() & os.ModeCharDevice) == 0 {

We check if the input data comes from a pipe.

var buf []byte
scanner := bufio.NewScanner(os.Stdin)

for scanner.Scan() {
    buf = append(buf, scanner.Bytes()...)
}

We read the data passed via the | pipe operator on the terminal.

} else {
    fmt.Print("Enter your name: ")

    var name string
    fmt.Scanf("%s", &name)
    fmt.Printf("Hello %s!\n", name)
}

If there is no data from the pipe, we read the data from the prompt.

$ echo "Peter" | go run hello.go 
Hello Peter!
$ go run hello.go 
Enter your name: Peter
Hello Peter!

First, we pass data to the program via the pipe, then via prompt.

Go pipe in HTTP handler

In the following example, we use a pipe in an HTTP handler.

hadler.go
package main

import (
    "fmt"
    "io"
    "net/http"
    "os/exec"
)

func handler(w http.ResponseWriter, r *http.Request) {

    cmd := exec.Command("date")

    pr, pw := io.Pipe()
    defer pw.Close()

    cmd.Stdout = pw
    cmd.Stderr = pw
    go io.Copy(w, pr)

    cmd.Run()
}

func main() {

    http.HandleFunc("/", handler)
    fmt.Println("server started on port 8080")
    http.ListenAndServe(":8080", nil)
}

The example starts a simple server, which returns today's date.

cmd := exec.Command("date")

We will execute the date command.

pr, pw := io.Pipe()
defer pw.Close()

We create a Go pipe.

cmd.Stdout = pw
cmd.Stderr = pw

We pass the PipeWriter to the command's standard output and standard error output.

go io.Copy(w, pr)

In a goroutine, we copy the contents of the PipeReader to the http.ResponseWriter.

cmd.Run()

The command is executed with Run.

$ go run handler.go 
server started on port 8080

We run the server.

$ curl localhost:8080
Sun 15 Nov 2020 02:18:07 PM CET

In a different terminal window, we create a GET request with curl.

In this tutorial, we have worked with a pipe in Go.

List all Go tutorials.