Python os.writev Function

Last modified April 11, 2025

This comprehensive guide explores Python's os.writev function, which performs scatter-gather I/O operations. We'll cover buffer management, performance benefits, and practical file writing examples.

Basic Definitions

The os.writev function writes data from multiple buffers to a file descriptor in a single system call. This is known as vectorized I/O or scatter-gather I/O.

Key parameters: fd (file descriptor), buffers (sequence of bytes-like objects). Returns total bytes written. Available on Unix-like systems (not Windows).

Basic File Writing with writev

This example demonstrates the simplest use of os.writev to write multiple buffers to a file. We'll create a file and write three separate chunks.

import os

# Create a test file
fd = os.open("output.txt", os.O_WRONLY | os.O_CREAT, 0o644)

# Prepare multiple buffers
buffers = [
    b"First line\n",
    b"Second line\n",
    b"Third line\n"
]

# Write all buffers at once
bytes_written = os.writev(fd, buffers)
print(f"Total bytes written: {bytes_written}")

os.close(fd)

This code opens a file, creates three byte buffers, and writes them in one operation. The function returns the total bytes written across all buffers.

The advantage over multiple write calls is reduced system call overhead, especially beneficial for high-performance applications.

Writing Different Buffer Types

os.writev accepts various bytes-like objects. This example shows writing different types: bytes, bytearray, and memoryview objects.

import os

fd = os.open("mixed_buffers.bin", os.O_WRONLY | os.O_CREAT, 0o644)

buffers = [
    b"Static bytes\n",
    bytearray(b"Bytearray content\n"),
    memoryview(b"Memoryview content\n")
]

bytes_written = os.writev(fd, buffers)
print(f"Total bytes written: {bytes_written}")

os.close(fd)

This demonstrates os.writev's flexibility with different bytes-like objects. All are written efficiently in a single system call.

Memoryview is particularly useful as it avoids copying large buffers while still working with the vectorized write operation.

Performance Comparison

This example compares os.writev with multiple os.write calls to demonstrate the performance benefit of vectorized I/O.

import os
import time

def test_writev(fd, buffers):
    start = time.perf_counter()
    os.writev(fd, buffers)
    return time.perf_counter() - start

def test_multiple_writes(fd, buffers):
    start = time.perf_counter()
    for buf in buffers:
        os.write(fd, buf)
    return time.perf_counter() - start

# Prepare test data
buffers = [b"x" * 1024 for _ in range(1000)]
fd = os.open("perf_test.bin", os.O_WRONLY | os.O_CREAT)

# Test writev
writev_time = test_writev(fd, buffers)
os.ftruncate(fd, 0)  # Reset file

# Test multiple writes
write_time = test_multiple_writes(fd, buffers)

os.close(fd)

print(f"writev time: {writev_time:.6f}s")
print(f"Multiple writes time: {write_time:.6f}s")
print(f"Speedup: {write_time/writev_time:.1f}x")

This benchmark creates 1000 buffers and writes them using both methods. On most systems, os.writev will be significantly faster.

The performance difference comes from reduced system call overhead - one call instead of many. The advantage grows with more buffers.

Partial Writes with writev

Like regular writes, os.writev may perform partial writes. This example shows how to handle such cases by retrying with remaining data.

import os

def robust_writev(fd, buffers):
    total = 0
    while buffers:
        try:
            written = os.writev(fd, buffers)
            total += written
            # Adjust buffers based on what was written
            while written > 0:
                if written >= len(buffers[0]):
                    written -= len(buffers[0])
                    buffers.pop(0)
                else:
                    buffers[0] = buffers[0][written:]
                    written = 0
        except BlockingIOError:
            # Would block - wait and retry
            time.sleep(0.1)
        except InterruptedError:
            # System call interrupted - retry
            continue
    return total

fd = os.open("partial_test.txt", os.O_WRONLY | os.O_CREAT | os.O_NONBLOCK)
buffers = [b"a" * 10000, b"b" * 20000, b"c" * 30000]

bytes_written = robust_writev(fd, buffers)
print(f"Successfully wrote {bytes_written} bytes")

os.close(fd)

This implementation handles partial writes by tracking how much was written and adjusting the buffers accordingly. It also handles non-blocking scenarios.

The function continues writing until all buffers are exhausted, making it robust against partial writes and interruptions.

Network Socket Writing

os.writev is particularly useful for network programming. This example shows writing HTTP response headers and body to a socket efficiently.

import os
import socket

def send_response(client_sock, headers, body):
    # Convert headers to bytes
    header_bytes = "\r\n".join(headers).encode("utf-8")
    
    # Prepare buffers - headers + blank line + body
    buffers = [
        header_bytes,
        b"\r\n\r\n",
        body if isinstance(body, (bytes, bytearray, memoryview)) else body.encode("utf-8")
    ]
    
    # Get raw file descriptor from socket
    fd = client_sock.fileno()
    
    # Write all at once
    try:
        total = os.writev(fd, buffers)
        print(f"Sent {total} bytes")
    except (BrokenPipeError, ConnectionResetError) as e:
        print(f"Connection error: {e}")

# Example usage
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(("localhost", 8080))
server_socket.listen(1)

client, addr = server_socket.accept()
print(f"Connection from {addr}")

headers = [
    "HTTP/1.1 200 OK",
    "Content-Type: text/plain",
    "Connection: close"
]
body = "Hello from writev example!"

send_response(client, headers, body)
client.close()
server_socket.close()

This demonstrates how os.writev can efficiently send HTTP responses by combining headers and body in one system call.

The technique reduces latency and CPU overhead compared to multiple send calls, especially important for high-performance servers.

Memory Efficiency with Memoryview

This example shows how to use memoryview with os.writev to write large files without copying data.

import os

def write_large_file(filename, chunks):
    fd = os.open(filename, os.O_WRONLY | os.O_CREAT, 0o644)
    
    # Convert chunks to memoryviews to avoid copying
    buffers = [memoryview(chunk) for chunk in chunks]
    
    total = 0
    while buffers:
        written = os.writev(fd, buffers)
        total += written
        
        # Adjust buffers based on what was written
        while written > 0 and buffers:
            chunk_len = len(buffers[0])
            if written >= chunk_len:
                written -= chunk_len
                buffers.pop(0)
            else:
                buffers[0] = buffers[0][written:]
                written = 0
    
    os.close(fd)
    return total

# Generate 1MB of data in 10 chunks
data_chunks = [os.urandom(100000) for _ in range(10)]
bytes_written = write_large_file("large_file.bin", data_chunks)
print(f"Wrote {bytes_written} bytes")

This approach is memory efficient because memoryview allows slicing without copying the underlying data. It's ideal for large files.

The function handles partial writes and tracks progress through the buffer list, ensuring all data is eventually written.

Error Handling

This example demonstrates comprehensive error handling for os.writev, covering common failure scenarios and edge cases.

import os
import errno

def safe_writev(fd, buffers):
    if not buffers:
        raise ValueError("Empty buffer list")
    
    for buf in buffers:
        if not isinstance(buf, (bytes, bytearray, memoryview)):
            raise TypeError(f"Buffer must be bytes-like, got {type(buf)}")
    
    try:
        return os.writev(fd, buffers)
    except OSError as e:
        if e.errno == errno.EBADF:
            raise ValueError("Invalid file descriptor") from e
        elif e.errno == errno.EINVAL:
            raise ValueError("Invalid argument") from e
        elif e.errno == errno.EFAULT:
            raise ValueError("Bad buffer address") from e
        elif e.errno in (errno.EAGAIN, errno.EWOULDBLOCK):
            raise BlockingIOError("Write would block") from e
        elif e.errno == errno.EINTR:
            raise InterruptedError("System call interrupted") from e
        raise  # Re-raise other errors

# Example usage
try:
    fd = os.open("error_test.txt", os.O_WRONLY | os.O_CREAT)
    buffers = [b"test", b"data"]
    
    try:
        written = safe_writev(fd, buffers)
        print(f"Wrote {written} bytes")
    finally:
        os.close(fd)
except Exception as e:
    print(f"Error: {e}")

This implementation provides detailed error checking and meaningful error messages for various failure scenarios that os.writev might encounter.

The function validates input buffers and translates system errors into appropriate Python exceptions with clear explanations.

Security Considerations

• File descriptors: Ensure descriptors are valid and properly opened
• Buffer validation: Verify all buffers are bytes-like objects
• Partial writes: Always handle cases where not all data is written
• Resource limits: Large writes may hit system limits
• Platform support: Not available on Windows systems

Best Practices

• Use memoryview: For large buffers to avoid copying
• Handle partial writes: Implement retry logic for robustness
• Benchmark: Compare with alternatives for your use case
• Error handling: Cover all possible failure scenarios
• Document assumptions: Note platform requirements clearly

Source References

• Python os.writev Documentation
• Linux writev(2) man page

Author

My name is Jan Bodnar, and I am a passionate programmer with extensive programming experience. I have been writing programming articles since 2007. To date, I have authored over 1,400 articles and 8 e-books. I possess more than ten years of experience in teaching programming.

List all Python tutorials.