Perl UTF-8 and encoding
last modified October 9, 2026
Perl has two kinds of strings. A byte string is a sequence of octets, while a
character string is a sequence of Unicode codepoints. Which one you hold depends
on how the data entered the program, and the difference becomes visible in
length, substr, and regular expressions.
Perl keeps a UTF8 flag on every string. When the flag is on, string operations
work on characters; when it is off, they work on bytes. The behaviour is
controlled by the use utf8 pragma for source code, by
binmode and the :encoding layer for I/O, by the core
Encode module for explicit conversion, and it can be inspected with
utf8::is_utf8.
In this tutorial, we move text between bytes and characters, read and write UTF-8 files, match and fold Unicode text, and report characters as codepoints so that the output stays portable.
Byte strings versus character strings
The same character can be stored as one codepoint or as several octets. Perl
tracks both cases with a flag, and the flag decides what
length counts.
#!/usr/bin/perl
use strict;
use warnings;
use v5.34.0;
use Encode;
# A byte string holds octets; a character string holds codepoints.
my $byte_string = "\xE2\x98\xBA"; # three octets, no UTF8 flag
my $char_string = "\x{263A}"; # one character, UTF8 flag set
say "byte string length: " . length($byte_string);
say "byte string flagged: " . (utf8::is_utf8($byte_string) ? "yes" : "no");
say "char string length: " . length($char_string);
say "char string flagged: " . (utf8::is_utf8($char_string) ? "yes" : "no");
my $octets = Encode::encode('UTF-8', $char_string);
say "same octets: " . (($octets eq $byte_string) ? "yes" : "no");
say "octets: " . join(' ', map { sprintf '%02X', ord } split //, $octets);
Both variables describe the same character, the smiley U+263A, and
once the character string is encoded they hold the same three octets. The flag
still makes them behave differently: length returns 3 for the byte
string and 1 for the character string. The utf8::is_utf8 function
reports the flag and is a debugging aid, not a validation test.
$ perl strings.pl byte string length: 3 byte string flagged: no char string length: 1 char string flagged: yes same octets: yes octets: E2 98 BA
The use utf8 pragma
When the source file itself contains non-ASCII characters, the parser needs to
know how they are encoded on disk. The use utf8 pragma declares the
source text to be UTF-8.
#!/usr/bin/perl
use strict;
use warnings;
use v5.34.0;
use utf8;
use Encode qw(encode);
my $greeting = "Grüße";
say "characters: " . length($greeting);
say "octets: " . length(encode("UTF-8", $greeting));
say "codepoints: " . join(" ", map { sprintf "%04X", ord } split //, $greeting);
Without the pragma the raw bytes of the file would be read as Latin-1 characters, so each accented letter would be counted twice and the codepoints would be wrong. The pragma affects literals in the file only. It does not touch any filehandle, so input and output still need their own encoding layers.
$ perl pragma.pl characters: 5 octets: 7 codepoints: 0047 0072 00FC 00DF 0065
Printing Unicode with binmode
By default Perl writes octets to a handle without encoding, and printing a
character above U+00FF produces a Wide character in
print warning. The binmode builtin installs an encoding
layer on the handle and encodes every character on the way out.
#!/usr/bin/perl
use strict;
use warnings;
use v5.34.0;
use utf8;
use Encode qw(encode);
my $text = "café";
binmode STDOUT, ":encoding(UTF-8)";
my @layers = PerlIO::get_layers(*STDOUT);
say "encoding layer on STDOUT: " . ((grep { /^encoding/ } @layers) ? "yes" : "no");
say "characters: " . length($text);
# An in-memory handle with the same layer shows the octets produced.
open my $mem, ">:encoding(UTF-8)", \my $bytes or die "open failed: $!";
print {$mem} $text;
close $mem;
say "encoded bytes: " . join(" ", map { sprintf "%02X", ord } split //, $bytes);
say "matches Encode: " . (($bytes eq encode("UTF-8", $text)) ? "yes" : "no");
The layer is now listed on STDOUT and the four characters are
written as five octets. The in-memory handle shows that the layer produces
exactly what Encode::encode produces. STDERR and any
other stream are separate and need their own binmode call.
$ perl binmode.pl encoding layer on STDOUT: yes characters: 4 encoded bytes: 63 61 66 C3 A9 matches Encode: yes
Reading a UTF-8 file
The three-argument open accepts a mode that includes an encoding
layer. With <:encoding(UTF-8) Perl reads octets from disk and
decodes them into characters.
#!/usr/bin/perl
use strict;
use warnings;
use v5.34.0;
use File::Temp qw(tempdir);
use Encode qw(encode);
my $dir = tempdir(CLEANUP => 1);
my $path = "$dir/greek.txt";
# Write raw UTF-8 octets so the file on disk is a plain byte stream.
open my $out, ">:raw", $path or die "cannot write: $!";
print {$out} encode("UTF-8", "\x{3B1}\x{3B2}\x{3B3}\n");
close $out;
# Read it back with an encoding layer decoding from UTF-8.
open my $in, "<:encoding(UTF-8)", $path or die "cannot read: $!";
my $line = <$in>;
close $in;
chomp $line;
say "length: " . length($line);
say "codepoints: " . join(" ", map { sprintf "U+%04X", ord } split //, $line);
say "is Greek: " . (($line =~ /\p{Greek}+/) ? "yes" : "no");
The file holds nine octets but the decoded line has three characters, the Greek
letters alpha, beta, and gamma. Without the layer length would
return the octet count and the pattern would not match. Prefer
:encoding(UTF-8) over the shorter :utf8, because the
former validates the input and the latter does not.
$ perl read_utf8.pl length: 3 codepoints: U+03B1 U+03B2 U+03B3 is Greek: yes
Writing a UTF-8 file
Writing is the mirror image of reading. The mode
>:encoding(UTF-8) encodes every character before it reaches the
disk.
#!/usr/bin/perl
use strict;
use warnings;
use v5.34.0;
use File::Temp qw(tempdir);
my $dir = tempdir(CLEANUP => 1);
my $path = "$dir/price.txt";
open my $out, ">:encoding(UTF-8)", $path or die "cannot write: $!";
print {$out} "\x{20AC} 100 and \x{4E2D}\x{6587}\n";
close $out;
# Read the same file as raw octets to see what was written.
open my $raw, "<:raw", $path or die "cannot read: $!";
my $bytes = do { local $/; <$raw> };
close $raw;
say "bytes on disk: " . length($bytes);
say "bytes: " . join(" ", map { sprintf "%02X", ord } split //, $bytes);
The euro sign occupies three octets and each CJK character takes three more, so nineteen octets land on disk. Reopening the file as a raw byte stream is the right way to inspect an encoding problem: dumping binary data straight to the terminal tells you much less than a hex listing.
$ perl write_utf8.pl bytes on disk: 19 bytes: E2 82 AC 20 31 30 30 20 61 6E 64 20 E4 B8 AD E6 96 87 0A
Encoding and decoding with Encode
The core Encode module performs the conversions explicitly. It is
the tool to reach for when the layer approach is not possible, for example when
data arrives from a socket or a database driver.
#!/usr/bin/perl
use strict;
use warnings;
use v5.34.0;
use Encode qw(encode decode is_utf8);
my $text = "\x{263A} smile";
my $octets = encode("UTF-8", $text);
say "octets: " . join(" ", map { sprintf "%02X", ord } split //, $octets);
say "octets flagged: " . (is_utf8($octets) ? "yes" : "no");
say "octet length: " . length($octets);
my $back = decode("UTF-8", $octets);
say "round trip: " . (($back eq $text) ? "yes" : "no");
say "decoded length: " . length($back);
# The same byte E9 means different characters in different encodings.
my $latin1_char = decode("ISO-8859-1", "\xE9");
say "latin1 codepoint: " . sprintf("U+%04X", ord $latin1_char);
encode turns characters into octets and clears the flag, while
decode turns octets into characters and sets it. The round trip is
lossless, but only because the same encoding is used twice. The last line shows
that the octet E9 means a different character in Latin-1 than it
does in UTF-8, which is why the encoding must always be stated. Invalid input
makes decode substitute U+FFFD and warn.
$ perl encode_decode.pl octets: E2 98 BA 20 73 6D 69 6C 65 octets flagged: no octet length: 9 round trip: yes decoded length: 7 latin1 codepoint: U+00E9
length, substr and use bytes
Once a string carries the UTF8 flag, the string functions count characters
rather than bytes. The use bytes pragma switches those operations
back to the internal octets for the enclosing block.
#!/usr/bin/perl
use strict;
use warnings;
use v5.34.0;
my $text = "\x{1F600}\x{4E2D}\x{6587}";
say "characters: " . length($text);
say "second char: " . sprintf("U+%04X", ord substr($text, 1, 1));
say "last char: " . sprintf("U+%04X", ord substr($text, -1));
{
use bytes;
say "octets (use bytes): " . length($text);
say "first octet: " . sprintf("%02X", ord substr($text, 0, 1));
}
say "characters again: " . length($text);
The string holds an emoji and two CJK characters, three characters in total.
substr slices by character, so the negative index reaches the last
character without arithmetic. Inside the block, use bytes reports
ten octets because the emoji needs four and each CJK character needs three. The
pragma is lexically scoped: outside the block the character semantics return.
$ perl lengths.pl characters: 3 second char: U+4E2D last char: U+6587 octets (use bytes): 10 first octet: F0 characters again: 3
Unicode-aware regular expressions
Perl regular expressions understand Unicode properties. The \p{...}
escape matches a script, a block, a general category, or a single named
character, and it works on characters as long as the string carries the flag.
#!/usr/bin/perl
use strict;
use warnings;
use v5.34.0;
use utf8;
my $greek = "αβγ";
my $mixed = "AΓα中";
my $combo = "e\x{0301}";
say "greek matches: " . (($greek =~ /\p{Greek}+/) ? "yes" : "no");
say "greek letters: " . (() = $greek =~ /\p{Greek}/g);
say "letters in mixed: " . (() = $mixed =~ /\p{L}/g);
say "uppercase: " . join(" ", map { sprintf "U+%04X", ord } ($mixed =~ /\p{Lu}/g));
say "greek block: " . (($mixed =~ /\p{InGreek}/) ? "yes" : "no");
say "combining marks: " . (() = $combo =~ /\p{Mn}/g);
say "characters: " . length($combo);
\p{Greek} matches by script and \p{InGreek} matches by
Unicode block, while \p{L} and \p{Lu} match general
categories: any letter and any uppercase letter. The pattern text itself
contains non-ASCII characters, so the program needs use utf8. The
combining acute accent is one code point of category Mn, so
e followed by the accent is a two character string that renders as
a single grapheme.
$ perl unicode_regex.pl greek matches: yes greek letters: 3 letters in mixed: 4 uppercase: U+0041 U+0393 greek block: yes combining marks: 1 characters: 2
Case conversion with fc, uc and lc
uc and lc change the case of a string, and
fc performs case folding, the operation intended for
case-insensitive comparison. The three differ on real Unicode text.
#!/usr/bin/perl
use strict;
use warnings;
use v5.34.0;
use utf8;
my $final_sigma = "ς";
my $normal_sigma = "σ";
my $sharp_s = "ß";
say "uc(final sigma) = " . sprintf("U+%04X", ord uc $final_sigma);
say "lc(SIGMA) = " . sprintf("U+%04X", ord lc "Σ");
say "lc compare: " . ((lc($final_sigma) eq lc($normal_sigma)) ? "yes" : "no");
say "fc compare: " . ((fc($final_sigma) eq fc($normal_sigma)) ? "yes" : "no");
say "fc(sharp s): " . join(" ", map { sprintf "U+%04X", ord } split //, fc $sharp_s);
say "fc equal: " . ((fc("STRASSE") eq fc("straße")) ? "yes" : "no");
The final sigma and the normal sigma are the two lowercase forms of the same
letter. Lowercasing keeps them apart, but case folding maps both to the normal
sigma, so only fc reports a match. Folding and uppercasing may also
expand one character into several, as the sharp s shows when it becomes two
s characters.
$ perl case.pl uc(final sigma) = U+03A3 lc(SIGMA) = U+03C3 lc compare: no fc compare: yes fc(sharp s): U+0073 U+0073 fc equal: yes
Reporting characters as codepoints
Diagnostics that print raw non-ASCII text depend on the terminal, the locale, and the pipeline. Reporting codepoints instead keeps the output ASCII-safe and identical everywhere.
#!/usr/bin/perl
use strict;
use warnings;
use v5.34.0;
use Encode qw(encode);
my $text = "\x{1F600}A\x{00E9}";
say "codepoints: " . join(" ", map { sprintf "U+%04X", ord } split //, $text);
say "decimal: " . join(" ", map { ord } split //, $text);
say "characters: " . length($text);
say "utf8 octets: " . uc unpack("H*", encode("UTF-8", $text));
say "octet list: " . join(" ", map { sprintf "%02X", ord } split //, encode("UTF-8", $text));
say "escaped: " . join("", map { sprintf "\\u{%04X}", ord } split //, $text);
Splitting the string into single characters and applying ord gives
one number per character, which sprintf "U+%04X" formats in the
conventional notation. unpack "H*" dumps the UTF-8 octets of the
whole string in one call, and the \u{...} form is an escape that
can be pasted back into Perl source. Note that the emoji is a single character
even though it occupies four octets.
$ perl codepoints.pl
codepoints: U+1F600 U+0041 U+00E9
decimal: 128512 65 233
characters: 3
utf8 octets: F09F988041C3A9
octet list: F0 9F 98 80 41 C3 A9
escaped: \u{1F600}\u{0041}\u{00E9}
Fixing mojibake
Mojibake appears when octets are decoded with the wrong encoding, usually because the text was already decoded once before. The repair reverses the wrong step and applies the correct one.
#!/usr/bin/perl
use strict;
use warnings;
use v5.34.0;
use Encode qw(encode decode);
# The letter e with acute accent (U+00E9) is C3 A9 in UTF-8.
# A broken pipeline decoded each octet as Latin-1 instead.
my $broken = "\x{C3}\x{A9}";
say "broken length: " . length($broken);
say "broken codepoints: " . join(" ", map { sprintf "U+%04X", ord } split //, $broken);
# Undo it: turn the characters back into octets, then decode as UTF-8.
my $octets = encode("ISO-8859-1", $broken);
my $fixed = decode("UTF-8", $octets);
say "fixed length: " . length($fixed);
say "fixed codepoint: " . sprintf("U+%04X", ord $fixed);
say "matches e acute: " . (($fixed eq "\x{E9}") ? "yes" : "no");
The two characters U+00C3 and U+00A9 are what the
octets C3 A9 look like when they are read as Latin-1. Encoding
those characters back to Latin-1 recovers the original octets, and decoding
them as UTF-8 yields the intended U+00E9. If a decode step fails,
the text was not simply double encoded, and the failure is more useful than the
substitution character.
$ perl mojibake.pl broken length: 2 broken codepoints: U+00C3 U+00A9 fixed length: 1 fixed codepoint: U+00E9 matches e acute: yes
Best Practices
- Declare the source encoding: put
use utf8;in every file that contains non-ASCII literals. - Set the layers at the boundary: use
binmodeor a three-argumentopenwith:encoding(UTF-8)so text is decoded on the way in and encoded on the way out. - Keep one representation: decode once at the edge, work with character strings, and encode only when writing.
- Validate untrusted input: prefer
:encoding(UTF-8)to:utf8, because only the former checks that the octets are valid. - Compare with fc: case-insensitive comparison should use
case folding rather than
lcoruc. - Print diagnostics as codepoints:
sprintf "U+%04X"andunpack "H*"keep test output ASCII-safe and reproducible.
In this article we have covered the byte and character string model of Perl,
the use utf8 pragma, encoding layers for input and output, the
Encode module, character based length and
substr, Unicode regular expressions, case folding, codepoint
diagnostics, and the repair of double encoded text.
Source
Unicode support in Perl · Perl Unicode introduction · Perl Unicode FAQ · The utf8 pragma · The Encode module · Perl character classes · The Perl open function
Author
List all Perl tutorials.