This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
perluniintro: Fix comment, Encode::decode does not have to return string with UTF8...
[perl5.git] / pod / perluniintro.pod
CommitLineData
ba62762e
JH
1=head1 NAME
2
07fcf8ff 3perluniintro - Perl Unicode introduction
ba62762e
JH
4
5=head1 DESCRIPTION
6
7This document gives a general idea of Unicode and how to use Unicode
c8695642
KW
8in Perl. See L</Further Resources> for references to more in-depth
9treatments of Unicode.
ba62762e
JH
10
11=head2 Unicode
12
376d9008 13Unicode is a character set standard which plans to codify all of the
ba62762e
JH
14writing systems of the world, plus many other symbols.
15
c8695642
KW
16Unicode and ISO/IEC 10646 are coordinated standards that unify
17almost all other modern character set standards,
18covering more than 80 writing systems and hundreds of languages,
376d9008 19including all commercially-important modern languages. All characters
ba62762e
JH
20in the largest Chinese, Japanese, and Korean dictionaries are also
21encoded. The standards will eventually cover almost all characters in
22more than 250 writing systems and thousands of languages.
c8695642 23Unicode 1.0 was released in October 1991, and 6.0 in October 2010.
ba62762e
JH
24
25A Unicode I<character> is an abstract entity. It is not bound to any
376d9008
JB
26particular integer width, especially not to the C language C<char>.
27Unicode is language-neutral and display-neutral: it does not encode the
e1b711da 28language of the text, and it does not generally define fonts or other graphical
ba62762e
JH
29layout details. Unicode operates on characters and on text built from
30those characters.
31
32Unicode defines characters like C<LATIN CAPITAL LETTER A> or C<GREEK
376d9008
JB
33SMALL LETTER ALPHA> and unique numbers for the characters, in this
34case 0x0041 and 0x03B1, respectively. These unique numbers are called
c8695642
KW
35I<code points>. A code point is essentially the position of the
36character within the set of all possible Unicode characters, and thus in
37Perl, the term I<ordinal> is often used interchangeably with it.
ba62762e
JH
38
39The Unicode standard prefers using hexadecimal notation for the code
4c496f0c
JH
40points. If numbers like C<0x0041> are unfamiliar to you, take a peek
41at a later section, L</"Hexadecimal Notation">. The Unicode standard
42uses the notation C<U+0041 LATIN CAPITAL LETTER A>, to give the
43hexadecimal code point and the normative name of the character.
ba62762e
JH
44
45Unicode also defines various I<properties> for the characters, like
376d9008 46"uppercase" or "lowercase", "decimal digit", or "punctuation";
ba62762e
JH
47these properties are independent of the names of the characters.
48Furthermore, various operations on the characters like uppercasing,
376d9008 49lowercasing, and collating (sorting) are defined.
ba62762e 50
0111a78f
KW
51A Unicode I<logical> "character" can actually consist of more than one internal
52I<actual> "character" or code point. For Western languages, this is adequately
c670e63a 53modelled by a I<base character> (like C<LATIN CAPITAL LETTER A>) followed
0111a78f 54by one or more I<modifiers> (like C<COMBINING ACUTE ACCENT>). This sequence of
376d9008 55base character and modifiers is called a I<combining character
0111a78f 56sequence>. Some non-western languages require more complicated
c8695642
KW
57models, so Unicode created the I<grapheme cluster> concept, which was
58later further refined into the I<extended grapheme cluster>. For
59example, a Korean Hangul syllable is considered a single logical
60character, but most often consists of three actual
c670e63a
KW
61Unicode characters: a leading consonant followed by an interior vowel followed
62by a trailing consonant.
0111a78f
KW
63
64Whether to call these extended grapheme clusters "characters" depends on your
65point of view. If you are a programmer, you probably would tend towards seeing
c8695642
KW
66each element in the sequences as one unit, or "character". However from
67the user's point of view, the whole sequence could be seen as one
68"character" since that's probably what it looks like in the context of the
69user's language. In this document, we take the programmer's point of
70view: one "character" is one Unicode code point.
71
72For some combinations of base character and modifiers, there are
73I<precomposed> characters. There is a single character equivalent, for
4d2d7a4c 74example, for the sequence C<LATIN CAPITAL LETTER A> followed by
c8695642
KW
75C<COMBINING ACUTE ACCENT>. It is called C<LATIN CAPITAL LETTER A WITH
76ACUTE>. These precomposed characters are, however, only available for
77some combinations, and are mainly meant to support round-trip
78conversions between Unicode and legacy standards (like ISO 8859). Using
79sequences, as Unicode does, allows for needing fewer basic building blocks
80(code points) to express many more potential grapheme clusters. To
81support conversion between equivalent forms, various I<normalization
82forms> are also defined. Thus, C<LATIN CAPITAL LETTER A WITH ACUTE> is
83in I<Normalization Form Composed>, (abbreviated NFC), and the sequence
84C<LATIN CAPITAL LETTER A> followed by C<COMBINING ACUTE ACCENT>
85represents the same character in I<Normalization Form Decomposed> (NFD).
ba62762e
JH
86
87Because of backward compatibility with legacy encodings, the "a unique
376d9008
JB
88number for every character" idea breaks down a bit: instead, there is
89"at least one number for every character". The same character could
90be represented differently in several legacy encodings. The
91a927c1 91converse is not true: some code points do not have an assigned
376d9008
JB
92character. Firstly, there are unallocated code points within
93otherwise used blocks. Secondly, there are special Unicode control
94characters that do not represent true characters.
ba62762e 95
e9078126
KW
96When Unicode was first conceived, it was thought that all the world's
97characters could be represented using a 16-bit word; that is a maximum of
91a927c1
KW
98C<0x10000> (or 65,536) characters would be needed, from C<0x0000> to
99C<0xFFFF>. This soon proved to be wrong, and since Unicode 2.0 (July
4c496f0c 1001996), Unicode has been defined all the way up to 21 bits (C<0x10FFFF>),
e9078126
KW
101and Unicode 3.1 (March 2001) defined the first characters above C<0xFFFF>.
102The first C<0x10000> characters are called the I<Plane 0>, or the
103I<Basic Multilingual Plane> (BMP). With Unicode 3.1, 17 (yes,
104seventeen) planes in all were defined--but they are nowhere near full of
105defined characters, yet.
106
107When a new language is being encoded, Unicode generally will choose a
108C<block> of consecutive unallocated code points for its characters. So
109far, the number of code points in these blocks has always been evenly
110divisible by 16. Extras in a block, not currently needed, are left
111unallocated, for future growth. But there have been occasions when
30189633 112a later release needed more code points than the available extras, and a
82aed44a
KW
113new block had to allocated somewhere else, not contiguous to the initial
114one, to handle the overflow. Thus, it became apparent early on that
fc273927 115"block" wasn't an adequate organizing principle, and so the C<Script>
82aed44a
KW
116property was created. (Later an improved script property was added as
117well, the C<Script_Extensions> property.) Those code points that are in
118overflow blocks can still
e9078126
KW
119have the same script as the original ones. The script concept fits more
120closely with natural language: there is C<Latin> script, C<Greek>
121script, and so on; and there are several artificial scripts, like
122C<Common> for characters that are used in multiple scripts, such as
123mathematical symbols. Scripts usually span varied parts of several
124blocks. For more information about scripts, see L<perlunicode/Scripts>.
1bfb14c4 125The division into blocks exists, but it is almost completely
e9078126
KW
126accidental--an artifact of how the characters have been and still are
127allocated. (Note that this paragraph has oversimplified things for the
128sake of this being an introduction. Unicode doesn't really encode
129languages, but the writing systems for them--their scripts; and one
130script can be used by many languages. Unicode also encodes things that
131aren't really about languages, such as symbols like C<BAGGAGE CLAIM>.)
ba62762e
JH
132
133The Unicode code points are just abstract numbers. To input and
4c496f0c
JH
134output these abstract numbers, the numbers must be I<encoded> or
135I<serialised> somehow. Unicode defines several I<character encoding
91a927c1
KW
136forms>, of which I<UTF-8> is the most popular. UTF-8 is a
137variable length encoding that encodes Unicode characters as 1 to 4
e1b711da 138bytes. Other encodings
4c496f0c 139include UTF-16 and UTF-32 and their big- and little-endian variants
f321be7e 140(UTF-8 is byte-order independent). The ISO/IEC 10646 defines the UCS-2
4c496f0c 141and UCS-4 encoding forms.
ba62762e 142
376d9008
JB
143For more information about encodings--for instance, to learn what
144I<surrogates> and I<byte order marks> (BOMs) are--see L<perlunicode>.
ba62762e
JH
145
146=head2 Perl's Unicode Support
147
c2fb32ed
BF
148Starting from Perl v5.6.0, Perl has had the capacity to handle Unicode
149natively. Perl v5.8.0, however, is the first recommended release for
376d9008
JB
150serious Unicode work. The maintenance release 5.6.1 fixed many of the
151problems of the initial Unicode implementation, but for example
1bfb14c4 152regular expressions still do not work with Unicode in 5.6.1.
c2fb32ed 153Perl v5.14.0 is the first release where Unicode support is
d6c970c7
AC
154(almost) seamlessly integrable without some gotchas. (There are two
155exceptions. Firstly, some differences in L<quotemeta|perlfunc/quotemeta>
156were fixed starting in Perl 5.16.0. Secondly, some differences in
157L<the range operator|perlop/Range Operators> were fixed starting in
158Perl 5.26.0.)
159
160To enable this
c8695642
KW
161seamless support, you should C<use feature 'unicode_strings'> (which is
162automatically selected if you C<use 5.012> or higher). See L<feature>.
163(5.14 also fixes a number of bugs and departures from the Unicode
164standard.)
165
c2fb32ed 166Before Perl v5.8.0, the use of C<use utf8> was used to declare
ba62762e 167that operations in the current block or file would be Unicode-aware.
376d9008 168This model was found to be wrong, or at least clumsy: the "Unicodeness"
1bfb14c4 169is now carried with the data, instead of being attached to the
c8695642 170operations.
c2fb32ed 171Starting with Perl v5.8.0, only one case remains where an explicit C<use
c8695642
KW
172utf8> is needed: if your Perl script itself is encoded in UTF-8, you can
173use UTF-8 in your identifier names, and in string and regular expression
376d9008 174literals, by saying C<use utf8>. This is not the default because
8f8cf39c 175scripts with legacy 8-bit data in them would break. See L<utf8>.
ba62762e
JH
176
177=head2 Perl's Unicode Model
178
376d9008 179Perl supports both pre-5.6 strings of eight-bit native bytes, and
c8695642
KW
180strings of Unicode characters. The general principle is that Perl tries
181to keep its data as eight-bit bytes for as long as possible, but as soon
182as Unicodeness cannot be avoided, the data is transparently upgraded
c2fb32ed 183to Unicode. Prior to Perl v5.14.0, the upgrade was not completely
c8695642
KW
184transparent (see L<perlunicode/The "Unicode Bug">), and for backwards
185compatibility, full transparency is not gained unless C<use feature
186'unicode_strings'> (see L<feature>) or C<use 5.012> (or higher) is
187selected.
ba62762e 188
4192de81 189Internally, Perl currently uses either whatever the native eight-bit
376d9008
JB
190character set of the platform (for example Latin-1) is, defaulting to
191UTF-8, to encode Unicode strings. Specifically, if all code points in
192the string are C<0xFF> or less, Perl uses the native eight-bit
193character set. Otherwise, it uses UTF-8.
4192de81 194
7ca610e8 195A user of Perl does not normally need to know nor care how Perl
20ba30f4 196happens to encode its internal strings, but it becomes relevant when
ac036724 197outputting Unicode strings to a stream without a PerlIO layer (one with
198the "default" encoding). In such a case, the raw bytes used internally
7ca610e8
JH
199(the native character set or UTF-8, as appropriate for each string)
200will be used, and a "Wide character" warning will be issued if those
201strings contain a character beyond 0x00FF.
4192de81
JH
202
203For example,
204
ae5648b3 205 perl -e 'print "\x{DF}\n", "\x{0100}\x{DF}\n"'
4192de81
JH
206
207produces a fairly useless mixture of native bytes and UTF-8, as well
1bfb14c4 208as a warning:
4192de81 209
1bfb14c4
JH
210 Wide character in print at ...
211
740d4bb2 212To output UTF-8, use the C<:encoding> or C<:utf8> output layer. Prepending
4192de81
JH
213
214 binmode(STDOUT, ":utf8");
215
376d9008 216to this sample program ensures that the output is completely UTF-8,
1bfb14c4 217and removes the program's warning.
ba62762e 218
8aa8f774
JH
219You can enable automatic UTF-8-ification of your standard file
220handles, default C<open()> layer, and C<@ARGV> by using either
221the C<-C> command line switch or the C<PERL_UNICODE> environment
222variable, see L<perlrun> for the documentation of the C<-C> switch.
223
c8695642
KW
224Note that this means that Perl expects other software to work the same
225way:
8aa8f774 226if Perl has been led to believe that STDIN should be UTF-8, but then
c8695642
KW
227STDIN coming in from another command is not UTF-8, Perl will likely
228complain about the malformed UTF-8.
b310b053 229
ec71e770
JH
230All features that combine Unicode and I/O also require using the new
231PerlIO feature. Almost all Perl 5.8 platforms do use PerlIO, though:
232you can see whether yours is by running "perl -V" and looking for
233C<useperlio=define>.
234
64c66fb6
JH
235=head2 Unicode and EBCDIC
236
0145df7d
KW
237Perl 5.8.0 added support for Unicode on EBCDIC platforms. This support
238was allowed to lapse in later releases, but was revived in 5.22.
239Unicode support is somewhat more complex to implement since additional
240conversions are needed. See L<perlebcdic> for more information.
64c66fb6 241
64c66fb6 242On EBCDIC platforms, the internal Unicode encoding form is UTF-EBCDIC
376d9008
JB
243instead of UTF-8. The difference is that as UTF-8 is "ASCII-safe" in
244that ASCII characters encode to UTF-8 as-is, while UTF-EBCDIC is
0145df7d
KW
245"EBCDIC-safe", in that all the basic characters (which includes all
246those that have ASCII equivalents (like C<"A">, C<"0">, C<"%">, I<etc.>)
247are the same in both EBCDIC and UTF-EBCDIC. Often, documentation
248will use the term "UTF-8" to mean UTF-EBCDIC as well. This is the case
249in this document.
ba62762e
JH
250
251=head2 Creating Unicode
252
0145df7d
KW
253This section applies fully to Perls starting with v5.22. Various
254caveats for earlier releases are in the L</Earlier releases caveats>
255subsection below.
ba62762e 256
0145df7d
KW
257To create Unicode characters in literals,
258use the C<\N{...}> notation in double-quoted strings:
ba62762e 259
0145df7d
KW
260 my $smiley_from_name = "\N{WHITE SMILING FACE}";
261 my $smiley_from_code_point = "\N{U+263a}";
ba62762e 262
0145df7d 263Similarly, they can be used in regular expression literals
ba62762e 264
0145df7d
KW
265 $smiley =~ /\N{WHITE SMILING FACE}/;
266 $smiley =~ /\N{U+263a}/;
ba62762e 267
0145df7d 268At run-time you can use:
ba62762e 269
0145df7d
KW
270 use charnames ();
271 my $hebrew_alef_from_name
272 = charnames::string_vianame("HEBREW LETTER ALEF");
273 my $hebrew_alef_from_code_point = charnames::string_vianame("U+05D0");
ba62762e 274
376d9008
JB
275Naturally, C<ord()> will do the reverse: it turns a character into
276a code point.
ba62762e 277
0145df7d
KW
278There are other runtime options as well. You can use C<pack()>:
279
280 my $hebrew_alef_from_code_point = pack("U", 0x05d0);
281
282Or you can use C<chr()>, though it is less convenient in the general
283case:
284
285 $hebrew_alef_from_code_point = chr(utf8::unicode_to_native(0x05d0));
286 utf8::upgrade($hebrew_alef_from_code_point);
287
288The C<utf8::unicode_to_native()> and C<utf8::upgrade()> aren't needed if
289the argument is above 0xFF, so the above could have been written as
290
291 $hebrew_alef_from_code_point = chr(0x05d0);
ba62762e 292
0145df7d 293since 0x5d0 is above 255.
ba62762e 294
0145df7d
KW
295C<\x{}> and C<\o{}> can also be used to specify code points at compile
296time in double-quotish strings, but, for backward compatibility with
297older Perls, the same rules apply as with C<chr()> for code points less
298than 256.
ba62762e 299
0145df7d
KW
300C<utf8::unicode_to_native()> is used so that the Perl code is portable
301to EBCDIC platforms. You can omit it if you're I<really> sure no one
302will ever want to use your code on a non-ASCII platform. Starting in
303Perl v5.22, calls to it on ASCII platforms are optimized out, so there's
304no performance penalty at all in adding it. Or you can simply use the
305other constructs that don't require it.
ba62762e 306
0145df7d
KW
307See L</"Further Resources"> for how to find all these names and numeric
308codes.
ba62762e 309
0145df7d 310=head3 Earlier releases caveats
8a5e5dd5 311
0145df7d
KW
312On EBCDIC platforms, prior to v5.22, using C<\N{U+...}> doesn't work
313properly.
1eda90df 314
0145df7d
KW
315Prior to v5.16, using C<\N{...}> with a character name (as opposed to a
316C<U+...> code point) required a S<C<use charnames :full>>.
771cd3b2 317
0145df7d
KW
318Prior to v5.14, there were some bugs in C<\N{...}> with a character name
319(as opposed to a C<U+...> code point).
320
321C<charnames::string_vianame()> was introduced in v5.14. Prior to that,
322C<charnames::vianame()> should work, but only if the argument is of the
323form C<"U+...">. Your best bet there for runtime Unicode by character
324name is probably:
325
326 use charnames ();
327 my $hebrew_alef_from_name
328 = pack("U", charnames::vianame("HEBREW LETTER ALEF"));
1eda90df 329
ba62762e
JH
330=head2 Handling Unicode
331
332Handling Unicode is for the most part transparent: just use the
333strings as usual. Functions like C<index()>, C<length()>, and
334C<substr()> will work on the Unicode characters; regular expressions
335will work on the Unicode characters (see L<perlunicode> and L<perlretut>).
336
0111a78f
KW
337Note that Perl considers grapheme clusters to be separate characters, so for
338example
ba62762e 339
9ba22424
KW
340 print length("\N{LATIN CAPITAL LETTER A}\N{COMBINING ACUTE ACCENT}"),
341 "\n";
ba62762e
JH
342
343will print 2, not 1. The only exception is that regular expressions
c8695642
KW
344have C<\X> for matching an extended grapheme cluster. (Thus C<\X> in a
345regular expression would match the entire sequence of both the example
346characters.)
ba62762e 347
376d9008
JB
348Life is not quite so transparent, however, when working with legacy
349encodings, I/O, and certain special cases:
ba62762e
JH
350
351=head2 Legacy Encodings
352
c8695642
KW
353When you combine legacy data and Unicode, the legacy data needs
354to be upgraded to Unicode. Normally the legacy data is assumed to be
355ISO 8859-1 (or EBCDIC, if applicable).
ba62762e 356
376d9008 357The C<Encode> module knows about many encodings and has interfaces
ba62762e
JH
358for doing conversions between those encodings:
359
2c9359a2 360 use Encode 'decode';
2426b0d3 361 $data = decode("iso-8859-3", $data); # convert from legacy
ba62762e
JH
362
363=head2 Unicode I/O
364
8baee566 365Normally, writing out Unicode data
ba62762e 366
8baee566 367 print FH $some_string_with_unicode, "\n";
ba62762e 368
8baee566 369produces raw bytes that Perl happens to use to internally encode the
376d9008
JB
370Unicode string. Perl's internal encoding depends on the system as
371well as what characters happen to be in the string at the time. If
372any of the characters are at code points C<0x100> or above, you will get
373a warning. To ensure that the output is explicitly rendered in the
374encoding you desire--and to avoid the warning--open the stream with
375the desired encoding. Some examples:
ba62762e 376
ec90690f
TS
377 open FH, ">:utf8", "file";
378
379 open FH, ">:encoding(ucs2)", "file";
380 open FH, ">:encoding(UTF-8)", "file";
381 open FH, ">:encoding(shift_jis)", "file";
1d7919c5 382
376d9008 383and on already open streams, use C<binmode()>:
1d7919c5
JH
384
385 binmode(STDOUT, ":utf8");
386
ec90690f
TS
387 binmode(STDOUT, ":encoding(ucs2)");
388 binmode(STDOUT, ":encoding(UTF-8)");
389 binmode(STDOUT, ":encoding(shift_jis)");
390
b5d8778e 391The matching of encoding names is loose: case does not matter, and
fae2c0fb 392many encodings have several aliases. Note that the C<:utf8> layer
1bfb14c4 393must always be specified exactly like that; it is I<not> subject to
c8695642 394the loose matching of encoding names. Also note that currently C<:utf8> is unsafe for
740d4bb2 395input, because it accepts the data without validating that it is indeed valid
367b3305
KW
396UTF-8; you should instead use C<:encoding(utf-8)> (with or without a
397hyphen).
b5d8778e 398
376d9008
JB
399See L<PerlIO> for the C<:utf8> layer, L<PerlIO::encoding> and
400L<Encode::PerlIO> for the C<:encoding()> layer, and
401L<Encode::Supported> for many encodings supported by the C<Encode>
402module.
ba62762e 403
a5f0baef 404Reading in a file that you know happens to be encoded in one of the
1bfb14c4
JH
405Unicode or legacy encodings does not magically turn the data into
406Unicode in Perl's eyes. To do that, specify the appropriate
fae2c0fb 407layer when opening files
ba62762e 408
6d8e7450 409 open(my $fh,'<:encoding(UTF-8)', 'anything');
8baee566
JF
410 my $line_of_unicode = <$fh>;
411
ec90690f 412 open(my $fh,'<:encoding(Big5)', 'anything');
8baee566 413 my $line_of_unicode = <$fh>;
ba62762e 414
fae2c0fb 415The I/O layers can also be specified more flexibly with
376d9008 416the C<open> pragma. See L<open>, or look at the following example.
ba62762e 417
6d8e7450
P
418 use open ':encoding(UTF-8)'; # input/output default encoding will be
419 # UTF-8
1d7919c5
JH
420 open X, ">file";
421 print X chr(0x100), "\n";
ba62762e 422 close X;
1d7919c5 423 open Y, "<file";
ba62762e
JH
424 printf "%#x\n", ord(<Y>); # this should print 0x100
425 close Y;
426
fae2c0fb 427With the C<open> pragma you can use the C<:locale> layer
ba62762e 428
12f98225 429 BEGIN { $ENV{LC_ALL} = $ENV{LANG} = 'ru_RU.KOI8-R' }
9e5bbba0
KW
430 # the :locale will probe the locale environment variables like
431 # LC_ALL
ba62762e
JH
432 use open OUT => ':locale'; # russki parusski
433 open(O, ">koi8");
434 print O chr(0x430); # Unicode CYRILLIC SMALL LETTER A = KOI8-R 0xc1
435 close O;
436 open(I, "<koi8");
437 printf "%#x\n", ord(<I>), "\n"; # this should print 0xc1
438 close I;
439
8baee566
JF
440These methods install a transparent filter on the I/O stream that
441converts data from the specified encoding when it is read in from the
a5f0baef 442stream. The result is always Unicode.
ba62762e
JH
443
444The L<open> pragma affects all the C<open()> calls after the pragma by
fae2c0fb
RGS
445setting default layers. If you want to affect only certain
446streams, use explicit layers directly in the C<open()> call.
ba62762e
JH
447
448You can switch encodings on an already opened stream by using
8baee566 449C<binmode()>; see L<perlfunc/binmode>.
ba62762e 450
c2fb32ed 451The C<:locale> does not currently work with
1ecefa54 452C<open()> and C<binmode()>, only with the C<open> pragma. The
8baee566 453C<:utf8> and C<:encoding(...)> methods do work with all of C<open()>,
1ecefa54 454C<binmode()>, and the C<open> pragma.
ba62762e 455
fae2c0fb 456Similarly, you may use these I/O layers on output streams to
a5f0baef
JH
457automatically convert Unicode to the specified encoding when it is
458written to the stream. For example, the following snippet copies the
459contents of the file "text.jis" (encoded as ISO-2022-JP, aka JIS) to
460the file "text.utf8", encoded as UTF-8:
ba62762e 461
c722ef0a
JH
462 open(my $nihongo, '<:encoding(iso-2022-jp)', 'text.jis');
463 open(my $unicode, '>:utf8', 'text.utf8');
0cf8a8d9 464 while (<$nihongo>) { print $unicode $_ }
ba62762e
JH
465
466The naming of encodings, both by the C<open()> and by the C<open>
2575c402
JW
467pragma allows for flexible names: C<koi8-r> and C<KOI8R> will both be
468understood.
ba62762e
JH
469
470Common encodings recognized by ISO, MIME, IANA, and various other
8baee566 471standardisation organisations are recognised; for a more detailed
1bfb14c4 472list see L<Encode::Supported>.
ba62762e
JH
473
474C<read()> reads characters and returns the number of characters.
475C<seek()> and C<tell()> operate on byte counts, as do C<sysread()>
476and C<sysseek()>.
477
8baee566 478Notice that because of the default behaviour of not doing any
fae2c0fb 479conversion upon input if there is no default layer,
ba62762e 480it is easy to mistakenly write code that keeps on expanding a file
1bfb14c4 481by repeatedly encoding the data:
ba62762e
JH
482
483 # BAD CODE WARNING
484 open F, "file";
8baee566 485 local $/; ## read in the whole file of 8-bit characters
ba62762e
JH
486 $t = <F>;
487 close F;
6d8e7450 488 open F, ">:encoding(UTF-8)", "file";
8baee566 489 print F $t; ## convert to UTF-8 on output
ba62762e
JH
490 close F;
491
492If you run this code twice, the contents of the F<file> will be twice
6d8e7450 493UTF-8 encoded. A C<use open ':encoding(UTF-8)'> would have avoided the
740d4bb2 494bug, or explicitly opening also the F<file> for input as UTF-8.
ba62762e 495
0c901d84 496B<NOTE>: the C<:utf8> and C<:encoding> features work only if your
cdf6c183
TH
497Perl has been built with L<PerlIO>, which is the default
498on most systems.
0c901d84 499
1ecefa54
JH
500=head2 Displaying Unicode As Text
501
502Sometimes you might want to display Perl scalars containing Unicode as
8baee566 503simple ASCII (or EBCDIC) text. The following subroutine converts
1ecefa54 504its argument so that Unicode characters with code points greater than
1bfb14c4
JH
505255 are displayed as C<\x{...}>, control characters (like C<\n>) are
506displayed as C<\x..>, and the rest of the characters as themselves:
1ecefa54 507
9e5bbba0 508 sub nice_string {
96cfa1df
KW
509 join("",
510 map { $_ > 255 # if wide character...
511 ? sprintf("\\x{%04X}", $_) # \x{...}
512 : chr($_) =~ /[[:cntrl:]]/ # else if control character...
513 ? sprintf("\\x%02X", $_) # \x..
514 : quotemeta(chr($_)) # else quoted or as themselves
515 } unpack("W*", $_[0])); # unpack Unicode characters
58c274a1
JF
516 }
517
518For example,
519
520 nice_string("foo\x{100}bar\n")
521
d0551e73 522returns the string
58c274a1 523
d0551e73
JH
524 'foo\x{0100}bar\x0A'
525
526which is ready to be printed.
1ecefa54 527
0145df7d
KW
528(C<\\x{}> is used here instead of C<\\N{}>, since it's most likely that
529you want to see what the native values are.)
530
ba62762e
JH
531=head2 Special Cases
532
533=over 4
534
535=item *
536
537Bit Complement Operator ~ And vec()
538
1bfb14c4
JH
539The bit complement operator C<~> may produce surprising results if
540used on strings containing characters with ordinal values above
541255. In such a case, the results are consistent with the internal
542encoding of the characters, but not with much else. So don't do
543that. Similarly for C<vec()>: you will be operating on the
544internally-encoded bit patterns of the Unicode characters, not on
545the code point values, which is very probably not what you want.
ba62762e
JH
546
547=item *
548
8baee566
JF
549Peeking At Perl's Internal Encoding
550
551Normal users of Perl should never care how Perl encodes any particular
a5f0baef 552Unicode string (because the normal ways to get at the contents of a
376d9008 553string with Unicode--via input and output--should always be via
fae2c0fb 554explicitly-defined I/O layers). But if you must, there are two
a5f0baef 555ways of looking behind the scenes.
ba62762e
JH
556
557One way of peeking inside the internal encoding of Unicode characters
f337b084
TH
558is to use C<unpack("C*", ...> to get the bytes of whatever the string
559encoding happens to be, or C<unpack("U0..", ...)> to get the bytes of the
560UTF-8 encoding:
ba62762e 561
8baee566 562 # this prints c4 80 for the UTF-8 bytes 0xc4 0x80
f337b084 563 print join(" ", unpack("U0(H2)*", pack("U", 0x100))), "\n";
ba62762e
JH
564
565Yet another way would be to use the Devel::Peek module:
566
567 perl -MDevel::Peek -e 'Dump(chr(0x100))'
568
1e54db1a 569That shows the C<UTF8> flag in FLAGS and both the UTF-8 bytes
376d9008 570and Unicode characters in C<PV>. See also later in this document
8800c35a 571the discussion about the C<utf8::is_utf8()> function.
ba62762e
JH
572
573=back
574
575=head2 Advanced Topics
576
577=over 4
578
579=item *
580
581String Equivalence
582
583The question of string equivalence turns somewhat complicated
376d9008 584in Unicode: what do you mean by "equal"?
ba62762e 585
07698885
RGS
586(Is C<LATIN CAPITAL LETTER A WITH ACUTE> equal to
587C<LATIN CAPITAL LETTER A>?)
ba62762e 588
a5f0baef
JH
589The short answer is that by default Perl compares equivalence (C<eq>,
590C<ne>) based only on code points of the characters. In the above
376d9008 591case, the answer is no (because 0x00C1 != 0x0041). But sometimes, any
c8695642 592CAPITAL LETTER A's should be considered equal, or even A's of any case.
ba62762e
JH
593
594The long answer is that you need to consider character normalization
e1b711da
KW
595and casing issues: see L<Unicode::Normalize>, Unicode Technical Report #15,
596L<Unicode Normalization Forms|http://www.unicode.org/unicode/reports/tr15> and
597sections on case mapping in the L<Unicode Standard|http://www.unicode.org>.
ba62762e 598
1bfb14c4 599As of Perl 5.8.0, the "Full" case-folding of I<Case
afba1538 600Mappings/SpecialCasing> is implemented, but bugs remain in C<qr//i> with them,
4d2d7a4c 601mostly fixed by 5.14, and essentially entirely by 5.18.
ba62762e
JH
602
603=item *
604
605String Collation
606
376d9008 607People like to see their strings nicely sorted--or as Unicode
ba62762e
JH
608parlance goes, collated. But again, what do you mean by collate?
609
07698885
RGS
610(Does C<LATIN CAPITAL LETTER A WITH ACUTE> come before or after
611C<LATIN CAPITAL LETTER A WITH GRAVE>?)
ba62762e 612
58c274a1 613The short answer is that by default, Perl compares strings (C<lt>,
ba62762e 614C<le>, C<cmp>, C<ge>, C<gt>) based only on the code points of the
1bfb14c4 615characters. In the above case, the answer is "after", since
da76a1f4 616C<0x00C1> > C<0x00C0>.
ba62762e
JH
617
618The long answer is that "it depends", and a good answer cannot be
619given without knowing (at the very least) the language context.
620See L<Unicode::Collate>, and I<Unicode Collation Algorithm>
2bbc8d55 621L<http://www.unicode.org/unicode/reports/tr10/>
ba62762e
JH
622
623=back
624
625=head2 Miscellaneous
626
627=over 4
628
629=item *
630
3ff56b75 631Character Ranges and Classes
ba62762e 632
0eb9ada1
KW
633Character ranges in regular expression bracketed character classes ( e.g.,
634C</[a-z]/>) and in the C<tr///> (also known as C<y///>) operator are not
635magically Unicode-aware. What this means is that C<[A-Za-z]> will not
636magically start to mean "all alphabetic letters" (not that it does mean that
637even for 8-bit characters; for those, if you are using locales (L<perllocale>),
638use C</[[:alpha:]]/>; and if not, use the 8-bit-aware property C<\p{alpha}>).
639
640All the properties that begin with C<\p> (and its inverse C<\P>) are actually
641character classes that are Unicode-aware. There are dozens of them, see
642L<perluniprops>.
643
0145df7d 644Starting in v5.22, you can use Unicode code points as the end points of
74fe8880
KW
645regular expression pattern character ranges, and the range will include
646all Unicode code points that lie between those end points, inclusive.
0145df7d 647
77c8f263 648 qr/ [ \N{U+03} - \N{U+20} ] /xx
0145df7d
KW
649
650includes the code points
651C<\N{U+03}>, C<\N{U+04}>, ..., C<\N{U+20}>.
ba62762e 652
f3f4813b 653This also works for ranges in C<tr///> starting in Perl v5.24.
74fe8880 654
ba62762e
JH
655=item *
656
657String-To-Number Conversions
658
376d9008
JB
659Unicode does define several other decimal--and numeric--characters
660besides the familiar 0 to 9, such as the Arabic and Indic digits.
ba62762e 661Perl does not support string-to-number conversion for digits other
4d2d7a4c 662than ASCII C<0> to C<9> (and ASCII C<a> to C<f> for hexadecimal).
c8695642 663To get safe conversions from any Unicode string, use
67592e11 664L<Unicode::UCD/num()>.
ba62762e
JH
665
666=back
667
668=head2 Questions With Answers
669
670=over 4
671
818c4caa 672=item *
5cb3728c
RB
673
674Will My Old Scripts Break?
ba62762e
JH
675
676Very probably not. Unless you are generating Unicode characters
1bfb14c4
JH
677somehow, old behaviour should be preserved. About the only behaviour
678that has changed and which could start generating Unicode is the old
679behaviour of C<chr()> where supplying an argument more than 255
680produced a character modulo 255. C<chr(300)>, for example, was equal
681to C<chr(45)> or "-" (in ASCII), now it is LATIN CAPITAL LETTER I WITH
682BREVE.
ba62762e 683
818c4caa 684=item *
5cb3728c
RB
685
686How Do I Make My Scripts Work With Unicode?
ba62762e
JH
687
688Very little work should be needed since nothing changes until you
1bfb14c4
JH
689generate Unicode data. The most important thing is getting input as
690Unicode; for that, see the earlier I/O discussion.
c8695642
KW
691To get full seamless Unicode support, add
692C<use feature 'unicode_strings'> (or C<use 5.012> or higher) to your
693script.
ba62762e 694
818c4caa 695=item *
5cb3728c
RB
696
697How Do I Know Whether My String Is In Unicode?
ba62762e 698
c8695642
KW
699You shouldn't have to care. But you may if your Perl is before 5.14.0
700or you haven't specified C<use feature 'unicode_strings'> or C<use
52d1f2c9 7015.012> (or higher) because otherwise the rules for the code points
c8695642 702in the range 128 to 255 are different depending on
2bbc8d55 703whether the string they are contained within is in Unicode or not.
e1b711da 704(See L<perlunicode/When Unicode Does Not Happen>.)
ba62762e 705
2bbc8d55 706To determine if a string is in Unicode, use:
ba62762e 707
8800c35a 708 print utf8::is_utf8($string) ? 1 : 0, "\n";
ba62762e
JH
709
710But note that this doesn't mean that any of the characters in the
711string are necessary UTF-8 encoded, or that any of the characters have
712code points greater than 0xFF (255) or even 0x80 (128), or that the
713string has any characters at all. All the C<is_utf8()> does is to
714return the value of the internal "utf8ness" flag attached to the
376d9008 715C<$string>. If the flag is off, the bytes in the scalar are interpreted
3c1c8017 716as a single byte encoding. If the flag is on, the bytes in the scalar
0eb9ada1
KW
717are interpreted as the (variable-length, potentially multi-byte) UTF-8 encoded
718code points of the characters. Bytes added to a UTF-8 encoded string are
1e54db1a 719automatically upgraded to UTF-8. If mixed non-UTF-8 and UTF-8 scalars
c8695642 720are merged (double-quoted interpolation, explicit concatenation, or
3c1c8017
AT
721printf/sprintf parameter substitution), the result will be UTF-8 encoded
722as if copies of the byte strings were upgraded to UTF-8: for example,
723
724 $a = "ab\x80c";
725 $b = "\x{100}";
726 print "$a = $b\n";
727
a02b5feb
JH
728the output string will be UTF-8-encoded C<ab\x80c = \x{100}\n>, but
729C<$a> will stay byte-encoded.
ba62762e
JH
730
731Sometimes you might really need to know the byte length of a string
96b10823 732instead of the character length. For that use the C<bytes> pragma
c8695642 733and the C<length()> function:
ba62762e
JH
734
735 my $unicode = chr(0x100);
736 print length($unicode), "\n"; # will print 1
00d823b9 737 use bytes;
96b10823 738 print length($unicode), "\n"; # will print 2
1bfb14c4 739 # (the 0xC4 0x80 of the UTF-8)
0eb9ada1 740 no bytes;
ba62762e 741
818c4caa 742=item *
5cb3728c 743
c8695642
KW
744How Do I Find Out What Encoding a File Has?
745
968ee499 746You might try L<Encode::Guess>, but it has a number of limitations.
c8695642
KW
747
748=item *
749
5cb3728c 750How Do I Detect Data That's Not Valid In a Particular Encoding?
ba62762e 751
8baee566
JF
752Use the C<Encode> package to try converting it.
753For example,
ba62762e 754
8e179dd8 755 use Encode 'decode';
2bbc8d55 756
8e179dd8
P
757 if (eval { decode('UTF-8', $string, Encode::FB_CROAK); 1 }) {
758 # $string is valid UTF-8
ba62762e 759 } else {
8e179dd8 760 # $string is not valid UTF-8
ba62762e
JH
761 }
762
f337b084 763Or use C<unpack> to try decoding it:
ba62762e
JH
764
765 use warnings;
f337b084 766 @chars = unpack("C0U*", $string_of_bytes_that_I_think_is_utf8);
ba62762e 767
ae5648b3
RGS
768If invalid, a C<Malformed UTF-8 character> warning is produced. The "C0" means
769"process the string character per character". Without that, the
770C<unpack("U*", ...)> would work in C<U0> mode (the default if the format
771string starts with C<U>) and it would return the bytes making up the UTF-8
f337b084 772encoding of the target string, something that will always work.
ba62762e 773
818c4caa 774=item *
5cb3728c
RB
775
776How Do I Convert Binary Data Into a Particular Encoding, Or Vice Versa?
ba62762e 777
8baee566
JF
778This probably isn't as useful as you might think.
779Normally, you shouldn't need to.
ba62762e 780
1bfb14c4 781In one sense, what you are asking doesn't make much sense: encodings
376d9008 782are for characters, and binary data are not "characters", so converting
a5f0baef
JH
783"data" into some encoding isn't meaningful unless you know in what
784character set and encoding the binary data is in, in which case it's
376d9008 785not just binary data, now is it?
8baee566 786
1bfb14c4
JH
787If you have a raw sequence of bytes that you know should be
788interpreted via a particular encoding, you can use C<Encode>:
ba62762e
JH
789
790 use Encode 'from_to';
791 from_to($data, "iso-8859-1", "utf-8"); # from latin-1 to utf-8
792
1bfb14c4
JH
793The call to C<from_to()> changes the bytes in C<$data>, but nothing
794material about the nature of the string has changed as far as Perl is
795concerned. Both before and after the call, the string C<$data>
796contains just a bunch of 8-bit bytes. As far as Perl is concerned,
797the encoding of the string remains as "system-native 8-bit bytes".
8baee566
JF
798
799You might relate this to a fictional 'Translate' module:
800
801 use Translate;
802 my $phrase = "Yes";
803 Translate::from_to($phrase, 'english', 'deutsch');
804 ## phrase now contains "Ja"
ba62762e 805
8baee566 806The contents of the string changes, but not the nature of the string.
1bfb14c4
JH
807Perl doesn't know any more after the call than before that the
808contents of the string indicates the affirmative.
ba62762e 809
376d9008 810Back to converting data. If you have (or want) data in your system's
a5f0baef
JH
811native 8-bit encoding (e.g. Latin-1, EBCDIC, etc.), you can use
812pack/unpack to convert to/from Unicode.
ba62762e 813
f337b084
TH
814 $native_string = pack("W*", unpack("U*", $Unicode_string));
815 $Unicode_string = pack("U*", unpack("W*", $native_string));
ba62762e
JH
816
817If you have a sequence of bytes you B<know> is valid UTF-8,
818but Perl doesn't know it yet, you can make Perl a believer, too:
819
016b3422
P
820 $Unicode = $bytes;
821 utf8::decode($Unicode);
ba62762e 822
f337b084
TH
823or:
824
825 $Unicode = pack("U0a*", $bytes);
ae5648b3 826
2bbc8d55
SP
827You can find the bytes that make up a UTF-8 sequence with
828
9e5bbba0 829 @bytes = unpack("C*", $Unicode_string)
2bbc8d55
SP
830
831and you can create well-formed Unicode with
832
9e5bbba0 833 $Unicode_string = pack("U*", 0xff, ...)
ba62762e 834
818c4caa 835=item *
5cb3728c
RB
836
837How Do I Display Unicode? How Do I Input Unicode?
ba62762e 838
2bbc8d55
SP
839See L<http://www.alanwood.net/unicode/> and
840L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e 841
818c4caa 842=item *
5cb3728c
RB
843
844How Does Unicode Work With Traditional Locales?
ba62762e 845
31f05a37
KW
846If your locale is a UTF-8 locale, starting in Perl v5.20, Perl works
847well for all categories except C<LC_COLLATE> dealing with sorting and
848the C<cmp> operator.
849
850For other locales, starting in Perl 5.16, you can specify
66cbab2c
KW
851
852 use locale ':not_characters';
853
31f05a37 854to get Perl to work well with them. The catch is that you
66cbab2c
KW
855have to translate from the locale character set to/from Unicode
856yourself. See L</Unicode IE<sol>O> above for how to
857
858 use open ':locale';
859
860to accomplish this, but full details are in L<perllocale/Unicode and
30189633 861UTF-8>, including gotchas that happen if you don't specify
66cbab2c 862C<:not_characters>.
ba62762e
JH
863
864=back
865
866=head2 Hexadecimal Notation
867
376d9008
JB
868The Unicode standard prefers using hexadecimal notation because
869that more clearly shows the division of Unicode into blocks of 256 characters.
ba62762e
JH
870Hexadecimal is also simply shorter than decimal. You can use decimal
871notation, too, but learning to use hexadecimal just makes life easier
1bfb14c4 872with the Unicode standard. The C<U+HHHH> notation uses hexadecimal,
076d825e 873for example.
ba62762e
JH
874
875The C<0x> prefix means a hexadecimal number, the digits are 0-9 I<and>
876a-f (or A-F, case doesn't matter). Each hexadecimal digit represents
877four bits, or half a byte. C<print 0x..., "\n"> will show a
878hexadecimal number in decimal, and C<printf "%x\n", $decimal> will
879show a decimal number in hexadecimal. If you have just the
376d9008 880"hex digits" of a hexadecimal number, you can use the C<hex()> function.
ba62762e
JH
881
882 print 0x0009, "\n"; # 9
883 print 0x000a, "\n"; # 10
884 print 0x000f, "\n"; # 15
885 print 0x0010, "\n"; # 16
886 print 0x0011, "\n"; # 17
887 print 0x0100, "\n"; # 256
888
889 print 0x0041, "\n"; # 65
890
891 printf "%x\n", 65; # 41
892 printf "%#x\n", 65; # 0x41
893
894 print hex("41"), "\n"; # 65
895
896=head2 Further Resources
897
898=over 4
899
900=item *
901
902Unicode Consortium
903
2bbc8d55 904L<http://www.unicode.org/>
ba62762e
JH
905
906=item *
907
908Unicode FAQ
909
2bbc8d55 910L<http://www.unicode.org/unicode/faq/>
ba62762e
JH
911
912=item *
913
914Unicode Glossary
915
2bbc8d55 916L<http://www.unicode.org/glossary/>
ba62762e
JH
917
918=item *
919
c8695642
KW
920Unicode Recommended Reading List
921
922The Unicode Consortium has a list of articles and books, some of which
923give a much more in depth treatment of Unicode:
924L<http://unicode.org/resources/readinglist.html>
925
926=item *
927
ba62762e
JH
928Unicode Useful Resources
929
2bbc8d55 930L<http://www.unicode.org/unicode/onlinedat/resources.html>
ba62762e
JH
931
932=item *
933
934Unicode and Multilingual Support in HTML, Fonts, Web Browsers and Other Applications
935
2bbc8d55 936L<http://www.alanwood.net/unicode/>
ba62762e
JH
937
938=item *
939
940UTF-8 and Unicode FAQ for Unix/Linux
941
2bbc8d55 942L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e
JH
943
944=item *
945
946Legacy Character Sets
947
2bbc8d55
SP
948L<http://www.czyborra.com/>
949L<http://www.eki.ee/letter/>
ba62762e
JH
950
951=item *
952
ba62762e
JH
953You can explore various information from the Unicode data files using
954the C<Unicode::UCD> module.
955
956=back
957
f6edf83b
JH
958=head1 UNICODE IN OLDER PERLS
959
960If you cannot upgrade your Perl to 5.8.0 or later, you can still
961do some Unicode processing by using the modules C<Unicode::String>,
962C<Unicode::Map8>, and C<Unicode::Map>, available from CPAN.
963If you have the GNU recode installed, you can also use the
376d9008 964Perl front-end C<Convert::Recode> for character conversions.
f6edf83b 965
aaef10c5 966The following are fast conversions from ISO 8859-1 (Latin-1) bytes
63de3cb2 967to UTF-8 bytes and back, the code works even with older Perl 5 versions.
aaef10c5
JH
968
969 # ISO 8859-1 to UTF-8
970 s/([\x80-\xFF])/chr(0xC0|ord($1)>>6).chr(0x80|ord($1)&0x3F)/eg;
971
972 # UTF-8 to ISO 8859-1
973 s/([\xC2\xC3])([\x80-\xBF])/chr(ord($1)<<6&0xC0|ord($2)&0x3F)/eg;
974
ba62762e
JH
975=head1 SEE ALSO
976
2575c402 977L<perlunitut>, L<perlunicode>, L<Encode>, L<open>, L<utf8>, L<bytes>,
4c496f0c
JH
978L<perlretut>, L<perlrun>, L<Unicode::Collate>, L<Unicode::Normalize>,
979L<Unicode::UCD>
ba62762e 980
376d9008 981=head1 ACKNOWLEDGMENTS
ba62762e
JH
982
983Thanks to the kind readers of the perl5-porters@perl.org,
984perl-unicode@perl.org, linux-utf8@nl.linux.org, and unicore@unicode.org
985mailing lists for their valuable feedback.
986
987=head1 AUTHOR, COPYRIGHT, AND LICENSE
988
91a927c1
KW
989Copyright 2001-2011 Jarkko Hietaniemi E<lt>jhi@iki.fiE<gt>.
990Now maintained by Perl 5 Porters.
ba62762e
JH
991
992This document may be distributed under the same terms as Perl itself.