This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
perlunitut: clarification
[perl5.git] / pod / perlunitut.pod
CommitLineData
aadaa455
RGS
1=head1 NAME
2
3perlunitut - Perl Unicode Tutorial
4
5=head1 DESCRIPTION
6
7The days of just flinging strings around are over. It's well established that
8modern programs need to be capable of communicating funny accented letters, and
9things like euro symbols. This means that programmers need new habits. It's
10easy to program Unicode capable software, but it does require discipline to do
11it right.
12
13There's a lot to know about character sets, and text encodings. It's probably
14best to spend a full day learning all this, but the basics can be learned in
15minutes.
16
17These are not the very basics, though. It is assumed that you already
18know the difference between bytes and characters, and realise (and accept!)
19that there are many different character sets and encodings, and that your
20program has to be explicit about them. Recommended reading is "The Absolute
21Minimum Every Software Developer Absolutely, Positively Must Know About Unicode
22and Character Sets (No Excuses!)" by Joel Spolsky, at
23L<http://joelonsoftware.com/articles/Unicode.html>.
24
25This tutorial speaks in rather absolute terms, and provides only a limited view
26of the wealth of character string related features that Perl has to offer. For
27most projects, this information will probably suffice.
28
29=head2 Definitions
30
31It's important to set a few things straight first. This is the most important
32part of this tutorial. This view may conflict with other information that you
33may have found on the web, but that's mostly because many sources are wrong.
34
35You may have to re-read this entire section a few times...
36
37=head3 Unicode
38
39B<Unicode> is a character set with room for lots of characters. The ordinal
e1b711da
KW
40value of a character is called a B<code point>. (But in practice, the
41distinction between code point and character is blurred, so the terms often
42are used interchangeably.)
aadaa455 43
e1b711da 44There are many, many code points, but computers work with bytes, and a byte has
5f37c4c6
KW
45room for only 256 values. Unicode has many more characters than that,
46so you need a method to make these accessible.
aadaa455
RGS
47
48Unicode is encoded using several competing encodings, of which UTF-8 is the
49most used. In a Unicode encoding, multiple subsequent bytes can be used to
50store a single code point, or simply: character.
51
52=head3 UTF-8
53
54B<UTF-8> is a Unicode encoding. Many people think that Unicode and UTF-8 are
55the same thing, but they're not. There are more Unicode encodings, but much of
56the world has standardized on UTF-8.
57
58UTF-8 treats the first 128 codepoints, 0..127, the same as ASCII. They take
59only one byte per character. All other characters are encoded as two or more
60(up to six) bytes using a complex scheme. Fortunately, Perl handles this for
61us, so we don't have to worry about this.
62
63=head3 Text strings (character strings)
64
65B<Text strings>, or B<character strings> are made of characters. Bytes are
66irrelevant here, and so are encodings. Each character is just that: the
67character.
68
69On a text string, you would do things like:
70
71 $text =~ s/foo/bar/;
72 if ($string =~ /^\d+$/) { ... }
73 $text = ucfirst $text;
74 my $character_count = length $text;
75
76The value of a character (C<ord>, C<chr>) is the corresponding Unicode code
77point.
78
79=head3 Binary strings (byte strings)
80
81B<Binary strings>, or B<byte strings> are made of bytes. Here, you don't have
82characters, just bytes. All communication with the outside world (anything
83outside of your current Perl process) is done in binary.
84
85On a binary string, you would do things like:
86
87 my (@length_content) = unpack "(V/a)*", $binary;
88 $binary =~ s/\x00\x0F/\xFF\xF0/; # for the brave :)
89 print {$fh} $binary;
90 my $byte_count = length $binary;
91
92=head3 Encoding
93
94B<Encoding> (as a verb) is the conversion from I<text> to I<binary>. To encode,
95you have to supply the target encoding, for example C<iso-8859-1> or C<UTF-8>.
96Some encodings, like the C<iso-8859> ("latin") range, do not support the full
97Unicode standard; characters that can't be represented are lost in the
98conversion.
99
100=head3 Decoding
101
102B<Decoding> is the conversion from I<binary> to I<text>. To decode, you have to
103know what encoding was used during the encoding phase. And most of all, it must
104be something decodable. It doesn't make much sense to decode a PNG image into a
105text string.
106
107=head3 Internal format
108
109Perl has an B<internal format>, an encoding that it uses to encode text strings
110so it can store them in memory. All text strings are in this internal format.
111In fact, text strings are never in any other format!
112
113You shouldn't worry about what this format is, because conversion is
114automatically done when you decode or encode.
115
116=head2 Your new toolkit
117
118Add to your standard heading the following line:
119
120 use Encode qw(encode decode);
121
122Or, if you're lazy, just:
123
124 use Encode;
125
126=head2 I/O flow (the actual 5 minute tutorial)
127
128The typical input/output flow of a program is:
129
130 1. Receive and decode
131 2. Process
132 3. Encode and output
133
134If your input is binary, and is supposed to remain binary, you shouldn't decode
135it to a text string, of course. But in all other cases, you should decode it.
136
137Decoding can't happen reliably if you don't know how the data was encoded. If
138you get to choose, it's a good idea to standardize on UTF-8.
139
140 my $foo = decode('UTF-8', get 'http://example.com/');
141 my $bar = decode('ISO-8859-1', readline STDIN);
142 my $xyzzy = decode('Windows-1251', $cgi->param('foo'));
143
144Processing happens as you knew before. The only difference is that you're now
145using characters instead of bytes. That's very useful if you use things like
146C<substr>, or C<length>.
147
148It's important to realize that there are no bytes in a text string. Of course,
149Perl has its internal encoding to store the string in memory, but ignore that.
150If you have to do anything with the number of bytes, it's probably best to move
151that part to step 3, just after you've encoded the string. Then you know
152exactly how many bytes it will be in the destination string.
153
154The syntax for encoding text strings to binary strings is as simple as decoding:
155
156 $body = encode('UTF-8', $body);
157
158If you needed to know the length of the string in bytes, now's the perfect time
159for that. Because C<$body> is now a byte string, C<length> will report the
160number of bytes, instead of the number of characters. The number of
161characters is no longer known, because characters only exist in text strings.
162
163 my $byte_count = length $body;
164
165And if the protocol you're using supports a way of letting the recipient know
166which character encoding you used, please help the receiving end by using that
167feature! For example, E-mail and HTTP support MIME headers, so you can use the
168C<Content-Type> header. They can also have C<Content-Length> to indicate the
169number of I<bytes>, which is always a good idea to supply if the number is
170known.
171
172 "Content-Type: text/plain; charset=UTF-8",
173 "Content-Length: $byte_count"
174
aadaa455
RGS
175=head1 SUMMARY
176
177Decode everything you receive, encode everything you send out. (If it's text
178data.)
179
2575c402
JW
180=head1 Q and A (or FAQ)
181
182After reading this document, you ought to read L<perlunifaq> too.
183
aadaa455
RGS
184=head1 ACKNOWLEDGEMENTS
185
186Thanks to Johan Vromans from Squirrel Consultancy. His UTF-8 rants during the
187Amsterdam Perl Mongers meetings got me interested and determined to find out
188how to use character encodings in Perl in ways that don't break easily.
189
190Thanks to Gerard Goossen from TTY. His presentation "UTF-8 in the wild" (Dutch
191Perl Workshop 2006) inspired me to publish my thoughts and write this tutorial.
192
193Thanks to the people who asked about this kind of stuff in several Perl IRC
194channels, and have constantly reminded me that a simpler explanation was
195needed.
196
197Thanks to the people who reviewed this document for me, before it went public.
198They are: Benjamin Smith, Jan-Pieter Cornet, Johan Vromans, Lukas Mai, Nathan
199Gray.
200
201=head1 AUTHOR
202
740d4bb2 203Juerd Waalboer <#####@juerd.nl>
aadaa455
RGS
204
205=head1 SEE ALSO
206
2575c402 207L<perlunifaq>, L<perlunicode>, L<perluniintro>, L<Encode>
aadaa455 208