This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Fix comment typos principal → principle
[perl5.git] / pod / perluniintro.pod
CommitLineData
ba62762e
JH
1=head1 NAME
2
07fcf8ff 3perluniintro - Perl Unicode introduction
ba62762e
JH
4
5=head1 DESCRIPTION
6
7This document gives a general idea of Unicode and how to use Unicode
c8695642
KW
8in Perl. See L</Further Resources> for references to more in-depth
9treatments of Unicode.
ba62762e
JH
10
11=head2 Unicode
12
376d9008 13Unicode is a character set standard which plans to codify all of the
ba62762e
JH
14writing systems of the world, plus many other symbols.
15
c8695642
KW
16Unicode and ISO/IEC 10646 are coordinated standards that unify
17almost all other modern character set standards,
18covering more than 80 writing systems and hundreds of languages,
376d9008 19including all commercially-important modern languages. All characters
ba62762e
JH
20in the largest Chinese, Japanese, and Korean dictionaries are also
21encoded. The standards will eventually cover almost all characters in
22more than 250 writing systems and thousands of languages.
c8695642 23Unicode 1.0 was released in October 1991, and 6.0 in October 2010.
ba62762e
JH
24
25A Unicode I<character> is an abstract entity. It is not bound to any
376d9008
JB
26particular integer width, especially not to the C language C<char>.
27Unicode is language-neutral and display-neutral: it does not encode the
e1b711da 28language of the text, and it does not generally define fonts or other graphical
ba62762e
JH
29layout details. Unicode operates on characters and on text built from
30those characters.
31
32Unicode defines characters like C<LATIN CAPITAL LETTER A> or C<GREEK
376d9008
JB
33SMALL LETTER ALPHA> and unique numbers for the characters, in this
34case 0x0041 and 0x03B1, respectively. These unique numbers are called
c8695642
KW
35I<code points>. A code point is essentially the position of the
36character within the set of all possible Unicode characters, and thus in
37Perl, the term I<ordinal> is often used interchangeably with it.
ba62762e
JH
38
39The Unicode standard prefers using hexadecimal notation for the code
4c496f0c
JH
40points. If numbers like C<0x0041> are unfamiliar to you, take a peek
41at a later section, L</"Hexadecimal Notation">. The Unicode standard
42uses the notation C<U+0041 LATIN CAPITAL LETTER A>, to give the
43hexadecimal code point and the normative name of the character.
ba62762e
JH
44
45Unicode also defines various I<properties> for the characters, like
376d9008 46"uppercase" or "lowercase", "decimal digit", or "punctuation";
ba62762e
JH
47these properties are independent of the names of the characters.
48Furthermore, various operations on the characters like uppercasing,
376d9008 49lowercasing, and collating (sorting) are defined.
ba62762e 50
0111a78f
KW
51A Unicode I<logical> "character" can actually consist of more than one internal
52I<actual> "character" or code point. For Western languages, this is adequately
c670e63a 53modelled by a I<base character> (like C<LATIN CAPITAL LETTER A>) followed
0111a78f 54by one or more I<modifiers> (like C<COMBINING ACUTE ACCENT>). This sequence of
376d9008 55base character and modifiers is called a I<combining character
0111a78f 56sequence>. Some non-western languages require more complicated
c8695642
KW
57models, so Unicode created the I<grapheme cluster> concept, which was
58later further refined into the I<extended grapheme cluster>. For
59example, a Korean Hangul syllable is considered a single logical
60character, but most often consists of three actual
c670e63a
KW
61Unicode characters: a leading consonant followed by an interior vowel followed
62by a trailing consonant.
0111a78f
KW
63
64Whether to call these extended grapheme clusters "characters" depends on your
65point of view. If you are a programmer, you probably would tend towards seeing
c8695642
KW
66each element in the sequences as one unit, or "character". However from
67the user's point of view, the whole sequence could be seen as one
68"character" since that's probably what it looks like in the context of the
69user's language. In this document, we take the programmer's point of
70view: one "character" is one Unicode code point.
71
72For some combinations of base character and modifiers, there are
73I<precomposed> characters. There is a single character equivalent, for
4d2d7a4c 74example, for the sequence C<LATIN CAPITAL LETTER A> followed by
c8695642
KW
75C<COMBINING ACUTE ACCENT>. It is called C<LATIN CAPITAL LETTER A WITH
76ACUTE>. These precomposed characters are, however, only available for
77some combinations, and are mainly meant to support round-trip
78conversions between Unicode and legacy standards (like ISO 8859). Using
79sequences, as Unicode does, allows for needing fewer basic building blocks
80(code points) to express many more potential grapheme clusters. To
81support conversion between equivalent forms, various I<normalization
82forms> are also defined. Thus, C<LATIN CAPITAL LETTER A WITH ACUTE> is
83in I<Normalization Form Composed>, (abbreviated NFC), and the sequence
84C<LATIN CAPITAL LETTER A> followed by C<COMBINING ACUTE ACCENT>
85represents the same character in I<Normalization Form Decomposed> (NFD).
ba62762e
JH
86
87Because of backward compatibility with legacy encodings, the "a unique
376d9008
JB
88number for every character" idea breaks down a bit: instead, there is
89"at least one number for every character". The same character could
90be represented differently in several legacy encodings. The
91a927c1 91converse is not true: some code points do not have an assigned
376d9008
JB
92character. Firstly, there are unallocated code points within
93otherwise used blocks. Secondly, there are special Unicode control
94characters that do not represent true characters.
ba62762e 95
e9078126
KW
96When Unicode was first conceived, it was thought that all the world's
97characters could be represented using a 16-bit word; that is a maximum of
91a927c1
KW
98C<0x10000> (or 65,536) characters would be needed, from C<0x0000> to
99C<0xFFFF>. This soon proved to be wrong, and since Unicode 2.0 (July
4c496f0c 1001996), Unicode has been defined all the way up to 21 bits (C<0x10FFFF>),
e9078126
KW
101and Unicode 3.1 (March 2001) defined the first characters above C<0xFFFF>.
102The first C<0x10000> characters are called the I<Plane 0>, or the
103I<Basic Multilingual Plane> (BMP). With Unicode 3.1, 17 (yes,
104seventeen) planes in all were defined--but they are nowhere near full of
105defined characters, yet.
106
107When a new language is being encoded, Unicode generally will choose a
108C<block> of consecutive unallocated code points for its characters. So
109far, the number of code points in these blocks has always been evenly
110divisible by 16. Extras in a block, not currently needed, are left
111unallocated, for future growth. But there have been occasions when
30189633 112a later release needed more code points than the available extras, and a
82aed44a
KW
113new block had to allocated somewhere else, not contiguous to the initial
114one, to handle the overflow. Thus, it became apparent early on that
fc273927 115"block" wasn't an adequate organizing principle, and so the C<Script>
82aed44a
KW
116property was created. (Later an improved script property was added as
117well, the C<Script_Extensions> property.) Those code points that are in
118overflow blocks can still
e9078126
KW
119have the same script as the original ones. The script concept fits more
120closely with natural language: there is C<Latin> script, C<Greek>
121script, and so on; and there are several artificial scripts, like
122C<Common> for characters that are used in multiple scripts, such as
123mathematical symbols. Scripts usually span varied parts of several
124blocks. For more information about scripts, see L<perlunicode/Scripts>.
1bfb14c4 125The division into blocks exists, but it is almost completely
e9078126
KW
126accidental--an artifact of how the characters have been and still are
127allocated. (Note that this paragraph has oversimplified things for the
128sake of this being an introduction. Unicode doesn't really encode
129languages, but the writing systems for them--their scripts; and one
130script can be used by many languages. Unicode also encodes things that
131aren't really about languages, such as symbols like C<BAGGAGE CLAIM>.)
ba62762e
JH
132
133The Unicode code points are just abstract numbers. To input and
4c496f0c
JH
134output these abstract numbers, the numbers must be I<encoded> or
135I<serialised> somehow. Unicode defines several I<character encoding
91a927c1
KW
136forms>, of which I<UTF-8> is the most popular. UTF-8 is a
137variable length encoding that encodes Unicode characters as 1 to 4
e1b711da 138bytes. Other encodings
4c496f0c 139include UTF-16 and UTF-32 and their big- and little-endian variants
f321be7e 140(UTF-8 is byte-order independent). The ISO/IEC 10646 defines the UCS-2
4c496f0c 141and UCS-4 encoding forms.
ba62762e 142
376d9008
JB
143For more information about encodings--for instance, to learn what
144I<surrogates> and I<byte order marks> (BOMs) are--see L<perlunicode>.
ba62762e
JH
145
146=head2 Perl's Unicode Support
147
c2fb32ed
BF
148Starting from Perl v5.6.0, Perl has had the capacity to handle Unicode
149natively. Perl v5.8.0, however, is the first recommended release for
376d9008
JB
150serious Unicode work. The maintenance release 5.6.1 fixed many of the
151problems of the initial Unicode implementation, but for example
1bfb14c4 152regular expressions still do not work with Unicode in 5.6.1.
c2fb32ed 153Perl v5.14.0 is the first release where Unicode support is
c8695642 154(almost) seamlessly integrable without some gotchas (the exception being
91a927c1 155some differences in L<quotemeta|perlfunc/quotemeta>, and that is fixed
2e2b2571 156starting in Perl 5.16.0). To enable this
c8695642
KW
157seamless support, you should C<use feature 'unicode_strings'> (which is
158automatically selected if you C<use 5.012> or higher). See L<feature>.
159(5.14 also fixes a number of bugs and departures from the Unicode
160standard.)
161
c2fb32ed 162Before Perl v5.8.0, the use of C<use utf8> was used to declare
ba62762e 163that operations in the current block or file would be Unicode-aware.
376d9008 164This model was found to be wrong, or at least clumsy: the "Unicodeness"
1bfb14c4 165is now carried with the data, instead of being attached to the
c8695642 166operations.
c2fb32ed 167Starting with Perl v5.8.0, only one case remains where an explicit C<use
c8695642
KW
168utf8> is needed: if your Perl script itself is encoded in UTF-8, you can
169use UTF-8 in your identifier names, and in string and regular expression
376d9008 170literals, by saying C<use utf8>. This is not the default because
8f8cf39c 171scripts with legacy 8-bit data in them would break. See L<utf8>.
ba62762e
JH
172
173=head2 Perl's Unicode Model
174
376d9008 175Perl supports both pre-5.6 strings of eight-bit native bytes, and
c8695642
KW
176strings of Unicode characters. The general principle is that Perl tries
177to keep its data as eight-bit bytes for as long as possible, but as soon
178as Unicodeness cannot be avoided, the data is transparently upgraded
c2fb32ed 179to Unicode. Prior to Perl v5.14.0, the upgrade was not completely
c8695642
KW
180transparent (see L<perlunicode/The "Unicode Bug">), and for backwards
181compatibility, full transparency is not gained unless C<use feature
182'unicode_strings'> (see L<feature>) or C<use 5.012> (or higher) is
183selected.
ba62762e 184
4192de81 185Internally, Perl currently uses either whatever the native eight-bit
376d9008
JB
186character set of the platform (for example Latin-1) is, defaulting to
187UTF-8, to encode Unicode strings. Specifically, if all code points in
188the string are C<0xFF> or less, Perl uses the native eight-bit
189character set. Otherwise, it uses UTF-8.
4192de81 190
7ca610e8 191A user of Perl does not normally need to know nor care how Perl
20ba30f4 192happens to encode its internal strings, but it becomes relevant when
ac036724 193outputting Unicode strings to a stream without a PerlIO layer (one with
194the "default" encoding). In such a case, the raw bytes used internally
7ca610e8
JH
195(the native character set or UTF-8, as appropriate for each string)
196will be used, and a "Wide character" warning will be issued if those
197strings contain a character beyond 0x00FF.
4192de81
JH
198
199For example,
200
ae5648b3 201 perl -e 'print "\x{DF}\n", "\x{0100}\x{DF}\n"'
4192de81
JH
202
203produces a fairly useless mixture of native bytes and UTF-8, as well
1bfb14c4 204as a warning:
4192de81 205
1bfb14c4
JH
206 Wide character in print at ...
207
740d4bb2 208To output UTF-8, use the C<:encoding> or C<:utf8> output layer. Prepending
4192de81
JH
209
210 binmode(STDOUT, ":utf8");
211
376d9008 212to this sample program ensures that the output is completely UTF-8,
1bfb14c4 213and removes the program's warning.
ba62762e 214
8aa8f774
JH
215You can enable automatic UTF-8-ification of your standard file
216handles, default C<open()> layer, and C<@ARGV> by using either
217the C<-C> command line switch or the C<PERL_UNICODE> environment
218variable, see L<perlrun> for the documentation of the C<-C> switch.
219
c8695642
KW
220Note that this means that Perl expects other software to work the same
221way:
8aa8f774 222if Perl has been led to believe that STDIN should be UTF-8, but then
c8695642
KW
223STDIN coming in from another command is not UTF-8, Perl will likely
224complain about the malformed UTF-8.
b310b053 225
ec71e770
JH
226All features that combine Unicode and I/O also require using the new
227PerlIO feature. Almost all Perl 5.8 platforms do use PerlIO, though:
228you can see whether yours is by running "perl -V" and looking for
229C<useperlio=define>.
230
64c66fb6
JH
231=head2 Unicode and EBCDIC
232
0145df7d
KW
233Perl 5.8.0 added support for Unicode on EBCDIC platforms. This support
234was allowed to lapse in later releases, but was revived in 5.22.
235Unicode support is somewhat more complex to implement since additional
236conversions are needed. See L<perlebcdic> for more information.
64c66fb6 237
64c66fb6 238On EBCDIC platforms, the internal Unicode encoding form is UTF-EBCDIC
376d9008
JB
239instead of UTF-8. The difference is that as UTF-8 is "ASCII-safe" in
240that ASCII characters encode to UTF-8 as-is, while UTF-EBCDIC is
0145df7d
KW
241"EBCDIC-safe", in that all the basic characters (which includes all
242those that have ASCII equivalents (like C<"A">, C<"0">, C<"%">, I<etc.>)
243are the same in both EBCDIC and UTF-EBCDIC. Often, documentation
244will use the term "UTF-8" to mean UTF-EBCDIC as well. This is the case
245in this document.
ba62762e
JH
246
247=head2 Creating Unicode
248
0145df7d
KW
249This section applies fully to Perls starting with v5.22. Various
250caveats for earlier releases are in the L</Earlier releases caveats>
251subsection below.
ba62762e 252
0145df7d
KW
253To create Unicode characters in literals,
254use the C<\N{...}> notation in double-quoted strings:
ba62762e 255
0145df7d
KW
256 my $smiley_from_name = "\N{WHITE SMILING FACE}";
257 my $smiley_from_code_point = "\N{U+263a}";
ba62762e 258
0145df7d 259Similarly, they can be used in regular expression literals
ba62762e 260
0145df7d
KW
261 $smiley =~ /\N{WHITE SMILING FACE}/;
262 $smiley =~ /\N{U+263a}/;
ba62762e 263
0145df7d 264At run-time you can use:
ba62762e 265
0145df7d
KW
266 use charnames ();
267 my $hebrew_alef_from_name
268 = charnames::string_vianame("HEBREW LETTER ALEF");
269 my $hebrew_alef_from_code_point = charnames::string_vianame("U+05D0");
ba62762e 270
376d9008
JB
271Naturally, C<ord()> will do the reverse: it turns a character into
272a code point.
ba62762e 273
0145df7d
KW
274There are other runtime options as well. You can use C<pack()>:
275
276 my $hebrew_alef_from_code_point = pack("U", 0x05d0);
277
278Or you can use C<chr()>, though it is less convenient in the general
279case:
280
281 $hebrew_alef_from_code_point = chr(utf8::unicode_to_native(0x05d0));
282 utf8::upgrade($hebrew_alef_from_code_point);
283
284The C<utf8::unicode_to_native()> and C<utf8::upgrade()> aren't needed if
285the argument is above 0xFF, so the above could have been written as
286
287 $hebrew_alef_from_code_point = chr(0x05d0);
ba62762e 288
0145df7d 289since 0x5d0 is above 255.
ba62762e 290
0145df7d
KW
291C<\x{}> and C<\o{}> can also be used to specify code points at compile
292time in double-quotish strings, but, for backward compatibility with
293older Perls, the same rules apply as with C<chr()> for code points less
294than 256.
ba62762e 295
0145df7d
KW
296C<utf8::unicode_to_native()> is used so that the Perl code is portable
297to EBCDIC platforms. You can omit it if you're I<really> sure no one
298will ever want to use your code on a non-ASCII platform. Starting in
299Perl v5.22, calls to it on ASCII platforms are optimized out, so there's
300no performance penalty at all in adding it. Or you can simply use the
301other constructs that don't require it.
ba62762e 302
0145df7d
KW
303See L</"Further Resources"> for how to find all these names and numeric
304codes.
ba62762e 305
0145df7d 306=head3 Earlier releases caveats
8a5e5dd5 307
0145df7d
KW
308On EBCDIC platforms, prior to v5.22, using C<\N{U+...}> doesn't work
309properly.
1eda90df 310
0145df7d
KW
311Prior to v5.16, using C<\N{...}> with a character name (as opposed to a
312C<U+...> code point) required a S<C<use charnames :full>>.
771cd3b2 313
0145df7d
KW
314Prior to v5.14, there were some bugs in C<\N{...}> with a character name
315(as opposed to a C<U+...> code point).
316
317C<charnames::string_vianame()> was introduced in v5.14. Prior to that,
318C<charnames::vianame()> should work, but only if the argument is of the
319form C<"U+...">. Your best bet there for runtime Unicode by character
320name is probably:
321
322 use charnames ();
323 my $hebrew_alef_from_name
324 = pack("U", charnames::vianame("HEBREW LETTER ALEF"));
1eda90df 325
ba62762e
JH
326=head2 Handling Unicode
327
328Handling Unicode is for the most part transparent: just use the
329strings as usual. Functions like C<index()>, C<length()>, and
330C<substr()> will work on the Unicode characters; regular expressions
331will work on the Unicode characters (see L<perlunicode> and L<perlretut>).
332
0111a78f
KW
333Note that Perl considers grapheme clusters to be separate characters, so for
334example
ba62762e 335
9ba22424
KW
336 print length("\N{LATIN CAPITAL LETTER A}\N{COMBINING ACUTE ACCENT}"),
337 "\n";
ba62762e
JH
338
339will print 2, not 1. The only exception is that regular expressions
c8695642
KW
340have C<\X> for matching an extended grapheme cluster. (Thus C<\X> in a
341regular expression would match the entire sequence of both the example
342characters.)
ba62762e 343
376d9008
JB
344Life is not quite so transparent, however, when working with legacy
345encodings, I/O, and certain special cases:
ba62762e
JH
346
347=head2 Legacy Encodings
348
c8695642
KW
349When you combine legacy data and Unicode, the legacy data needs
350to be upgraded to Unicode. Normally the legacy data is assumed to be
351ISO 8859-1 (or EBCDIC, if applicable).
ba62762e 352
376d9008 353The C<Encode> module knows about many encodings and has interfaces
ba62762e
JH
354for doing conversions between those encodings:
355
2c9359a2
RGS
356 use Encode 'decode';
357 $data = decode("iso-8859-3", $data); # convert from legacy to utf-8
ba62762e
JH
358
359=head2 Unicode I/O
360
8baee566 361Normally, writing out Unicode data
ba62762e 362
8baee566 363 print FH $some_string_with_unicode, "\n";
ba62762e 364
8baee566 365produces raw bytes that Perl happens to use to internally encode the
376d9008
JB
366Unicode string. Perl's internal encoding depends on the system as
367well as what characters happen to be in the string at the time. If
368any of the characters are at code points C<0x100> or above, you will get
369a warning. To ensure that the output is explicitly rendered in the
370encoding you desire--and to avoid the warning--open the stream with
371the desired encoding. Some examples:
ba62762e 372
ec90690f
ST
373 open FH, ">:utf8", "file";
374
375 open FH, ">:encoding(ucs2)", "file";
376 open FH, ">:encoding(UTF-8)", "file";
377 open FH, ">:encoding(shift_jis)", "file";
1d7919c5 378
376d9008 379and on already open streams, use C<binmode()>:
1d7919c5
JH
380
381 binmode(STDOUT, ":utf8");
382
ec90690f
ST
383 binmode(STDOUT, ":encoding(ucs2)");
384 binmode(STDOUT, ":encoding(UTF-8)");
385 binmode(STDOUT, ":encoding(shift_jis)");
386
b5d8778e 387The matching of encoding names is loose: case does not matter, and
fae2c0fb 388many encodings have several aliases. Note that the C<:utf8> layer
1bfb14c4 389must always be specified exactly like that; it is I<not> subject to
c8695642 390the loose matching of encoding names. Also note that currently C<:utf8> is unsafe for
740d4bb2 391input, because it accepts the data without validating that it is indeed valid
367b3305
KW
392UTF-8; you should instead use C<:encoding(utf-8)> (with or without a
393hyphen).
b5d8778e 394
376d9008
JB
395See L<PerlIO> for the C<:utf8> layer, L<PerlIO::encoding> and
396L<Encode::PerlIO> for the C<:encoding()> layer, and
397L<Encode::Supported> for many encodings supported by the C<Encode>
398module.
ba62762e 399
a5f0baef 400Reading in a file that you know happens to be encoded in one of the
1bfb14c4
JH
401Unicode or legacy encodings does not magically turn the data into
402Unicode in Perl's eyes. To do that, specify the appropriate
fae2c0fb 403layer when opening files
ba62762e 404
740d4bb2 405 open(my $fh,'<:encoding(utf8)', 'anything');
8baee566
JF
406 my $line_of_unicode = <$fh>;
407
ec90690f 408 open(my $fh,'<:encoding(Big5)', 'anything');
8baee566 409 my $line_of_unicode = <$fh>;
ba62762e 410
fae2c0fb 411The I/O layers can also be specified more flexibly with
376d9008 412the C<open> pragma. See L<open>, or look at the following example.
ba62762e 413
9e5bbba0
KW
414 use open ':encoding(utf8)'; # input/output default encoding will be
415 # UTF-8
1d7919c5
JH
416 open X, ">file";
417 print X chr(0x100), "\n";
ba62762e 418 close X;
1d7919c5 419 open Y, "<file";
ba62762e
JH
420 printf "%#x\n", ord(<Y>); # this should print 0x100
421 close Y;
422
fae2c0fb 423With the C<open> pragma you can use the C<:locale> layer
ba62762e 424
12f98225 425 BEGIN { $ENV{LC_ALL} = $ENV{LANG} = 'ru_RU.KOI8-R' }
9e5bbba0
KW
426 # the :locale will probe the locale environment variables like
427 # LC_ALL
ba62762e
JH
428 use open OUT => ':locale'; # russki parusski
429 open(O, ">koi8");
430 print O chr(0x430); # Unicode CYRILLIC SMALL LETTER A = KOI8-R 0xc1
431 close O;
432 open(I, "<koi8");
433 printf "%#x\n", ord(<I>), "\n"; # this should print 0xc1
434 close I;
435
8baee566
JF
436These methods install a transparent filter on the I/O stream that
437converts data from the specified encoding when it is read in from the
a5f0baef 438stream. The result is always Unicode.
ba62762e
JH
439
440The L<open> pragma affects all the C<open()> calls after the pragma by
fae2c0fb
RGS
441setting default layers. If you want to affect only certain
442streams, use explicit layers directly in the C<open()> call.
ba62762e
JH
443
444You can switch encodings on an already opened stream by using
8baee566 445C<binmode()>; see L<perlfunc/binmode>.
ba62762e 446
c2fb32ed 447The C<:locale> does not currently work with
1ecefa54 448C<open()> and C<binmode()>, only with the C<open> pragma. The
8baee566 449C<:utf8> and C<:encoding(...)> methods do work with all of C<open()>,
1ecefa54 450C<binmode()>, and the C<open> pragma.
ba62762e 451
fae2c0fb 452Similarly, you may use these I/O layers on output streams to
a5f0baef
JH
453automatically convert Unicode to the specified encoding when it is
454written to the stream. For example, the following snippet copies the
455contents of the file "text.jis" (encoded as ISO-2022-JP, aka JIS) to
456the file "text.utf8", encoded as UTF-8:
ba62762e 457
c722ef0a
JH
458 open(my $nihongo, '<:encoding(iso-2022-jp)', 'text.jis');
459 open(my $unicode, '>:utf8', 'text.utf8');
0cf8a8d9 460 while (<$nihongo>) { print $unicode $_ }
ba62762e
JH
461
462The naming of encodings, both by the C<open()> and by the C<open>
2575c402
JW
463pragma allows for flexible names: C<koi8-r> and C<KOI8R> will both be
464understood.
ba62762e
JH
465
466Common encodings recognized by ISO, MIME, IANA, and various other
8baee566 467standardisation organisations are recognised; for a more detailed
1bfb14c4 468list see L<Encode::Supported>.
ba62762e
JH
469
470C<read()> reads characters and returns the number of characters.
471C<seek()> and C<tell()> operate on byte counts, as do C<sysread()>
472and C<sysseek()>.
473
8baee566 474Notice that because of the default behaviour of not doing any
fae2c0fb 475conversion upon input if there is no default layer,
ba62762e 476it is easy to mistakenly write code that keeps on expanding a file
1bfb14c4 477by repeatedly encoding the data:
ba62762e
JH
478
479 # BAD CODE WARNING
480 open F, "file";
8baee566 481 local $/; ## read in the whole file of 8-bit characters
ba62762e
JH
482 $t = <F>;
483 close F;
740d4bb2 484 open F, ">:encoding(utf8)", "file";
8baee566 485 print F $t; ## convert to UTF-8 on output
ba62762e
JH
486 close F;
487
488If you run this code twice, the contents of the F<file> will be twice
740d4bb2
JW
489UTF-8 encoded. A C<use open ':encoding(utf8)'> would have avoided the
490bug, or explicitly opening also the F<file> for input as UTF-8.
ba62762e 491
0c901d84 492B<NOTE>: the C<:utf8> and C<:encoding> features work only if your
cdf6c183
TH
493Perl has been built with L<PerlIO>, which is the default
494on most systems.
0c901d84 495
1ecefa54
JH
496=head2 Displaying Unicode As Text
497
498Sometimes you might want to display Perl scalars containing Unicode as
8baee566 499simple ASCII (or EBCDIC) text. The following subroutine converts
1ecefa54 500its argument so that Unicode characters with code points greater than
1bfb14c4
JH
501255 are displayed as C<\x{...}>, control characters (like C<\n>) are
502displayed as C<\x..>, and the rest of the characters as themselves:
1ecefa54 503
9e5bbba0 504 sub nice_string {
96cfa1df
KW
505 join("",
506 map { $_ > 255 # if wide character...
507 ? sprintf("\\x{%04X}", $_) # \x{...}
508 : chr($_) =~ /[[:cntrl:]]/ # else if control character...
509 ? sprintf("\\x%02X", $_) # \x..
510 : quotemeta(chr($_)) # else quoted or as themselves
511 } unpack("W*", $_[0])); # unpack Unicode characters
58c274a1
JF
512 }
513
514For example,
515
516 nice_string("foo\x{100}bar\n")
517
d0551e73 518returns the string
58c274a1 519
d0551e73
JH
520 'foo\x{0100}bar\x0A'
521
522which is ready to be printed.
1ecefa54 523
0145df7d
KW
524(C<\\x{}> is used here instead of C<\\N{}>, since it's most likely that
525you want to see what the native values are.)
526
ba62762e
JH
527=head2 Special Cases
528
529=over 4
530
531=item *
532
533Bit Complement Operator ~ And vec()
534
1bfb14c4
JH
535The bit complement operator C<~> may produce surprising results if
536used on strings containing characters with ordinal values above
537255. In such a case, the results are consistent with the internal
538encoding of the characters, but not with much else. So don't do
539that. Similarly for C<vec()>: you will be operating on the
540internally-encoded bit patterns of the Unicode characters, not on
541the code point values, which is very probably not what you want.
ba62762e
JH
542
543=item *
544
8baee566
JF
545Peeking At Perl's Internal Encoding
546
547Normal users of Perl should never care how Perl encodes any particular
a5f0baef 548Unicode string (because the normal ways to get at the contents of a
376d9008 549string with Unicode--via input and output--should always be via
fae2c0fb 550explicitly-defined I/O layers). But if you must, there are two
a5f0baef 551ways of looking behind the scenes.
ba62762e
JH
552
553One way of peeking inside the internal encoding of Unicode characters
f337b084
TH
554is to use C<unpack("C*", ...> to get the bytes of whatever the string
555encoding happens to be, or C<unpack("U0..", ...)> to get the bytes of the
556UTF-8 encoding:
ba62762e 557
8baee566 558 # this prints c4 80 for the UTF-8 bytes 0xc4 0x80
f337b084 559 print join(" ", unpack("U0(H2)*", pack("U", 0x100))), "\n";
ba62762e
JH
560
561Yet another way would be to use the Devel::Peek module:
562
563 perl -MDevel::Peek -e 'Dump(chr(0x100))'
564
1e54db1a 565That shows the C<UTF8> flag in FLAGS and both the UTF-8 bytes
376d9008 566and Unicode characters in C<PV>. See also later in this document
8800c35a 567the discussion about the C<utf8::is_utf8()> function.
ba62762e
JH
568
569=back
570
571=head2 Advanced Topics
572
573=over 4
574
575=item *
576
577String Equivalence
578
579The question of string equivalence turns somewhat complicated
376d9008 580in Unicode: what do you mean by "equal"?
ba62762e 581
07698885
RGS
582(Is C<LATIN CAPITAL LETTER A WITH ACUTE> equal to
583C<LATIN CAPITAL LETTER A>?)
ba62762e 584
a5f0baef
JH
585The short answer is that by default Perl compares equivalence (C<eq>,
586C<ne>) based only on code points of the characters. In the above
376d9008 587case, the answer is no (because 0x00C1 != 0x0041). But sometimes, any
c8695642 588CAPITAL LETTER A's should be considered equal, or even A's of any case.
ba62762e
JH
589
590The long answer is that you need to consider character normalization
e1b711da
KW
591and casing issues: see L<Unicode::Normalize>, Unicode Technical Report #15,
592L<Unicode Normalization Forms|http://www.unicode.org/unicode/reports/tr15> and
593sections on case mapping in the L<Unicode Standard|http://www.unicode.org>.
ba62762e 594
1bfb14c4 595As of Perl 5.8.0, the "Full" case-folding of I<Case
afba1538 596Mappings/SpecialCasing> is implemented, but bugs remain in C<qr//i> with them,
4d2d7a4c 597mostly fixed by 5.14, and essentially entirely by 5.18.
ba62762e
JH
598
599=item *
600
601String Collation
602
376d9008 603People like to see their strings nicely sorted--or as Unicode
ba62762e
JH
604parlance goes, collated. But again, what do you mean by collate?
605
07698885
RGS
606(Does C<LATIN CAPITAL LETTER A WITH ACUTE> come before or after
607C<LATIN CAPITAL LETTER A WITH GRAVE>?)
ba62762e 608
58c274a1 609The short answer is that by default, Perl compares strings (C<lt>,
ba62762e 610C<le>, C<cmp>, C<ge>, C<gt>) based only on the code points of the
1bfb14c4 611characters. In the above case, the answer is "after", since
da76a1f4 612C<0x00C1> > C<0x00C0>.
ba62762e
JH
613
614The long answer is that "it depends", and a good answer cannot be
615given without knowing (at the very least) the language context.
616See L<Unicode::Collate>, and I<Unicode Collation Algorithm>
2bbc8d55 617L<http://www.unicode.org/unicode/reports/tr10/>
ba62762e
JH
618
619=back
620
621=head2 Miscellaneous
622
623=over 4
624
625=item *
626
3ff56b75 627Character Ranges and Classes
ba62762e 628
0eb9ada1
KW
629Character ranges in regular expression bracketed character classes ( e.g.,
630C</[a-z]/>) and in the C<tr///> (also known as C<y///>) operator are not
631magically Unicode-aware. What this means is that C<[A-Za-z]> will not
632magically start to mean "all alphabetic letters" (not that it does mean that
633even for 8-bit characters; for those, if you are using locales (L<perllocale>),
634use C</[[:alpha:]]/>; and if not, use the 8-bit-aware property C<\p{alpha}>).
635
636All the properties that begin with C<\p> (and its inverse C<\P>) are actually
637character classes that are Unicode-aware. There are dozens of them, see
638L<perluniprops>.
639
0145df7d 640Starting in v5.22, you can use Unicode code points as the end points of
74fe8880
KW
641regular expression pattern character ranges, and the range will include
642all Unicode code points that lie between those end points, inclusive.
0145df7d
KW
643
644 qr/ [\N{U+03]-\N{U+20}] /x
645
646includes the code points
647C<\N{U+03}>, C<\N{U+04}>, ..., C<\N{U+20}>.
ba62762e 648
74fe8880
KW
649(It is planned to extend this behavior to ranges in C<tr///> in Perl
650v5.24.)
651
ba62762e
JH
652=item *
653
654String-To-Number Conversions
655
376d9008
JB
656Unicode does define several other decimal--and numeric--characters
657besides the familiar 0 to 9, such as the Arabic and Indic digits.
ba62762e 658Perl does not support string-to-number conversion for digits other
4d2d7a4c 659than ASCII C<0> to C<9> (and ASCII C<a> to C<f> for hexadecimal).
c8695642 660To get safe conversions from any Unicode string, use
67592e11 661L<Unicode::UCD/num()>.
ba62762e
JH
662
663=back
664
665=head2 Questions With Answers
666
667=over 4
668
818c4caa 669=item *
5cb3728c
RB
670
671Will My Old Scripts Break?
ba62762e
JH
672
673Very probably not. Unless you are generating Unicode characters
1bfb14c4
JH
674somehow, old behaviour should be preserved. About the only behaviour
675that has changed and which could start generating Unicode is the old
676behaviour of C<chr()> where supplying an argument more than 255
677produced a character modulo 255. C<chr(300)>, for example, was equal
678to C<chr(45)> or "-" (in ASCII), now it is LATIN CAPITAL LETTER I WITH
679BREVE.
ba62762e 680
818c4caa 681=item *
5cb3728c
RB
682
683How Do I Make My Scripts Work With Unicode?
ba62762e
JH
684
685Very little work should be needed since nothing changes until you
1bfb14c4
JH
686generate Unicode data. The most important thing is getting input as
687Unicode; for that, see the earlier I/O discussion.
c8695642
KW
688To get full seamless Unicode support, add
689C<use feature 'unicode_strings'> (or C<use 5.012> or higher) to your
690script.
ba62762e 691
818c4caa 692=item *
5cb3728c
RB
693
694How Do I Know Whether My String Is In Unicode?
ba62762e 695
c8695642
KW
696You shouldn't have to care. But you may if your Perl is before 5.14.0
697or you haven't specified C<use feature 'unicode_strings'> or C<use
52d1f2c9 6985.012> (or higher) because otherwise the rules for the code points
c8695642 699in the range 128 to 255 are different depending on
2bbc8d55 700whether the string they are contained within is in Unicode or not.
e1b711da 701(See L<perlunicode/When Unicode Does Not Happen>.)
ba62762e 702
2bbc8d55 703To determine if a string is in Unicode, use:
ba62762e 704
8800c35a 705 print utf8::is_utf8($string) ? 1 : 0, "\n";
ba62762e
JH
706
707But note that this doesn't mean that any of the characters in the
708string are necessary UTF-8 encoded, or that any of the characters have
709code points greater than 0xFF (255) or even 0x80 (128), or that the
710string has any characters at all. All the C<is_utf8()> does is to
711return the value of the internal "utf8ness" flag attached to the
376d9008 712C<$string>. If the flag is off, the bytes in the scalar are interpreted
3c1c8017 713as a single byte encoding. If the flag is on, the bytes in the scalar
0eb9ada1
KW
714are interpreted as the (variable-length, potentially multi-byte) UTF-8 encoded
715code points of the characters. Bytes added to a UTF-8 encoded string are
1e54db1a 716automatically upgraded to UTF-8. If mixed non-UTF-8 and UTF-8 scalars
c8695642 717are merged (double-quoted interpolation, explicit concatenation, or
3c1c8017
AT
718printf/sprintf parameter substitution), the result will be UTF-8 encoded
719as if copies of the byte strings were upgraded to UTF-8: for example,
720
721 $a = "ab\x80c";
722 $b = "\x{100}";
723 print "$a = $b\n";
724
a02b5feb
JH
725the output string will be UTF-8-encoded C<ab\x80c = \x{100}\n>, but
726C<$a> will stay byte-encoded.
ba62762e
JH
727
728Sometimes you might really need to know the byte length of a string
ce7675db 729instead of the character length. For that use either the
00d823b9 730C<Encode::encode_utf8()> function or the C<bytes> pragma
c8695642 731and the C<length()> function:
ba62762e
JH
732
733 my $unicode = chr(0x100);
734 print length($unicode), "\n"; # will print 1
ce7675db 735 require Encode;
9ba22424 736 print length(Encode::encode_utf8($unicode)),"\n"; # will print 2
00d823b9 737 use bytes;
1bfb14c4
JH
738 print length($unicode), "\n"; # will also print 2
739 # (the 0xC4 0x80 of the UTF-8)
0eb9ada1 740 no bytes;
ba62762e 741
818c4caa 742=item *
5cb3728c 743
c8695642
KW
744How Do I Find Out What Encoding a File Has?
745
968ee499 746You might try L<Encode::Guess>, but it has a number of limitations.
c8695642
KW
747
748=item *
749
5cb3728c 750How Do I Detect Data That's Not Valid In a Particular Encoding?
ba62762e 751
8baee566
JF
752Use the C<Encode> package to try converting it.
753For example,
ba62762e 754
bb2f379c 755 use Encode 'decode_utf8';
2bbc8d55 756
228ee848 757 if (eval { decode_utf8($string, Encode::FB_CROAK); 1 }) {
a365f2ce 758 # $string is valid utf8
ba62762e 759 } else {
a365f2ce 760 # $string is not valid utf8
ba62762e
JH
761 }
762
f337b084 763Or use C<unpack> to try decoding it:
ba62762e
JH
764
765 use warnings;
f337b084 766 @chars = unpack("C0U*", $string_of_bytes_that_I_think_is_utf8);
ba62762e 767
ae5648b3
RGS
768If invalid, a C<Malformed UTF-8 character> warning is produced. The "C0" means
769"process the string character per character". Without that, the
770C<unpack("U*", ...)> would work in C<U0> mode (the default if the format
771string starts with C<U>) and it would return the bytes making up the UTF-8
f337b084 772encoding of the target string, something that will always work.
ba62762e 773
818c4caa 774=item *
5cb3728c
RB
775
776How Do I Convert Binary Data Into a Particular Encoding, Or Vice Versa?
ba62762e 777
8baee566
JF
778This probably isn't as useful as you might think.
779Normally, you shouldn't need to.
ba62762e 780
1bfb14c4 781In one sense, what you are asking doesn't make much sense: encodings
376d9008 782are for characters, and binary data are not "characters", so converting
a5f0baef
JH
783"data" into some encoding isn't meaningful unless you know in what
784character set and encoding the binary data is in, in which case it's
376d9008 785not just binary data, now is it?
8baee566 786
1bfb14c4
JH
787If you have a raw sequence of bytes that you know should be
788interpreted via a particular encoding, you can use C<Encode>:
ba62762e
JH
789
790 use Encode 'from_to';
791 from_to($data, "iso-8859-1", "utf-8"); # from latin-1 to utf-8
792
1bfb14c4
JH
793The call to C<from_to()> changes the bytes in C<$data>, but nothing
794material about the nature of the string has changed as far as Perl is
795concerned. Both before and after the call, the string C<$data>
796contains just a bunch of 8-bit bytes. As far as Perl is concerned,
797the encoding of the string remains as "system-native 8-bit bytes".
8baee566
JF
798
799You might relate this to a fictional 'Translate' module:
800
801 use Translate;
802 my $phrase = "Yes";
803 Translate::from_to($phrase, 'english', 'deutsch');
804 ## phrase now contains "Ja"
ba62762e 805
8baee566 806The contents of the string changes, but not the nature of the string.
1bfb14c4
JH
807Perl doesn't know any more after the call than before that the
808contents of the string indicates the affirmative.
ba62762e 809
376d9008 810Back to converting data. If you have (or want) data in your system's
a5f0baef
JH
811native 8-bit encoding (e.g. Latin-1, EBCDIC, etc.), you can use
812pack/unpack to convert to/from Unicode.
ba62762e 813
f337b084
TH
814 $native_string = pack("W*", unpack("U*", $Unicode_string));
815 $Unicode_string = pack("U*", unpack("W*", $native_string));
ba62762e
JH
816
817If you have a sequence of bytes you B<know> is valid UTF-8,
818but Perl doesn't know it yet, you can make Perl a believer, too:
819
820 use Encode 'decode_utf8';
8baee566 821 $Unicode = decode_utf8($bytes);
ba62762e 822
f337b084
TH
823or:
824
825 $Unicode = pack("U0a*", $bytes);
ae5648b3 826
2bbc8d55
SP
827You can find the bytes that make up a UTF-8 sequence with
828
9e5bbba0 829 @bytes = unpack("C*", $Unicode_string)
2bbc8d55
SP
830
831and you can create well-formed Unicode with
832
9e5bbba0 833 $Unicode_string = pack("U*", 0xff, ...)
ba62762e 834
818c4caa 835=item *
5cb3728c
RB
836
837How Do I Display Unicode? How Do I Input Unicode?
ba62762e 838
2bbc8d55
SP
839See L<http://www.alanwood.net/unicode/> and
840L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e 841
818c4caa 842=item *
5cb3728c
RB
843
844How Does Unicode Work With Traditional Locales?
ba62762e 845
31f05a37
KW
846If your locale is a UTF-8 locale, starting in Perl v5.20, Perl works
847well for all categories except C<LC_COLLATE> dealing with sorting and
848the C<cmp> operator.
849
850For other locales, starting in Perl 5.16, you can specify
66cbab2c
KW
851
852 use locale ':not_characters';
853
31f05a37 854to get Perl to work well with them. The catch is that you
66cbab2c
KW
855have to translate from the locale character set to/from Unicode
856yourself. See L</Unicode IE<sol>O> above for how to
857
858 use open ':locale';
859
860to accomplish this, but full details are in L<perllocale/Unicode and
30189633 861UTF-8>, including gotchas that happen if you don't specify
66cbab2c 862C<:not_characters>.
ba62762e
JH
863
864=back
865
866=head2 Hexadecimal Notation
867
376d9008
JB
868The Unicode standard prefers using hexadecimal notation because
869that more clearly shows the division of Unicode into blocks of 256 characters.
ba62762e
JH
870Hexadecimal is also simply shorter than decimal. You can use decimal
871notation, too, but learning to use hexadecimal just makes life easier
1bfb14c4 872with the Unicode standard. The C<U+HHHH> notation uses hexadecimal,
076d825e 873for example.
ba62762e
JH
874
875The C<0x> prefix means a hexadecimal number, the digits are 0-9 I<and>
876a-f (or A-F, case doesn't matter). Each hexadecimal digit represents
877four bits, or half a byte. C<print 0x..., "\n"> will show a
878hexadecimal number in decimal, and C<printf "%x\n", $decimal> will
879show a decimal number in hexadecimal. If you have just the
376d9008 880"hex digits" of a hexadecimal number, you can use the C<hex()> function.
ba62762e
JH
881
882 print 0x0009, "\n"; # 9
883 print 0x000a, "\n"; # 10
884 print 0x000f, "\n"; # 15
885 print 0x0010, "\n"; # 16
886 print 0x0011, "\n"; # 17
887 print 0x0100, "\n"; # 256
888
889 print 0x0041, "\n"; # 65
890
891 printf "%x\n", 65; # 41
892 printf "%#x\n", 65; # 0x41
893
894 print hex("41"), "\n"; # 65
895
896=head2 Further Resources
897
898=over 4
899
900=item *
901
902Unicode Consortium
903
2bbc8d55 904L<http://www.unicode.org/>
ba62762e
JH
905
906=item *
907
908Unicode FAQ
909
2bbc8d55 910L<http://www.unicode.org/unicode/faq/>
ba62762e
JH
911
912=item *
913
914Unicode Glossary
915
2bbc8d55 916L<http://www.unicode.org/glossary/>
ba62762e
JH
917
918=item *
919
c8695642
KW
920Unicode Recommended Reading List
921
922The Unicode Consortium has a list of articles and books, some of which
923give a much more in depth treatment of Unicode:
924L<http://unicode.org/resources/readinglist.html>
925
926=item *
927
ba62762e
JH
928Unicode Useful Resources
929
2bbc8d55 930L<http://www.unicode.org/unicode/onlinedat/resources.html>
ba62762e
JH
931
932=item *
933
934Unicode and Multilingual Support in HTML, Fonts, Web Browsers and Other Applications
935
2bbc8d55 936L<http://www.alanwood.net/unicode/>
ba62762e
JH
937
938=item *
939
940UTF-8 and Unicode FAQ for Unix/Linux
941
2bbc8d55 942L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e
JH
943
944=item *
945
946Legacy Character Sets
947
2bbc8d55
SP
948L<http://www.czyborra.com/>
949L<http://www.eki.ee/letter/>
ba62762e
JH
950
951=item *
952
ba62762e
JH
953You can explore various information from the Unicode data files using
954the C<Unicode::UCD> module.
955
956=back
957
f6edf83b
JH
958=head1 UNICODE IN OLDER PERLS
959
960If you cannot upgrade your Perl to 5.8.0 or later, you can still
961do some Unicode processing by using the modules C<Unicode::String>,
962C<Unicode::Map8>, and C<Unicode::Map>, available from CPAN.
963If you have the GNU recode installed, you can also use the
376d9008 964Perl front-end C<Convert::Recode> for character conversions.
f6edf83b 965
aaef10c5 966The following are fast conversions from ISO 8859-1 (Latin-1) bytes
63de3cb2 967to UTF-8 bytes and back, the code works even with older Perl 5 versions.
aaef10c5
JH
968
969 # ISO 8859-1 to UTF-8
970 s/([\x80-\xFF])/chr(0xC0|ord($1)>>6).chr(0x80|ord($1)&0x3F)/eg;
971
972 # UTF-8 to ISO 8859-1
973 s/([\xC2\xC3])([\x80-\xBF])/chr(ord($1)<<6&0xC0|ord($2)&0x3F)/eg;
974
ba62762e
JH
975=head1 SEE ALSO
976
2575c402 977L<perlunitut>, L<perlunicode>, L<Encode>, L<open>, L<utf8>, L<bytes>,
4c496f0c
JH
978L<perlretut>, L<perlrun>, L<Unicode::Collate>, L<Unicode::Normalize>,
979L<Unicode::UCD>
ba62762e 980
376d9008 981=head1 ACKNOWLEDGMENTS
ba62762e
JH
982
983Thanks to the kind readers of the perl5-porters@perl.org,
984perl-unicode@perl.org, linux-utf8@nl.linux.org, and unicore@unicode.org
985mailing lists for their valuable feedback.
986
987=head1 AUTHOR, COPYRIGHT, AND LICENSE
988
91a927c1
KW
989Copyright 2001-2011 Jarkko Hietaniemi E<lt>jhi@iki.fiE<gt>.
990Now maintained by Perl 5 Porters.
ba62762e
JH
991
992This document may be distributed under the same terms as Perl itself.