This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
perluniintro: Remove obsolete text
[perl5.git] / pod / perluniintro.pod
CommitLineData
ba62762e
JH
1=head1 NAME
2
07fcf8ff 3perluniintro - Perl Unicode introduction
ba62762e
JH
4
5=head1 DESCRIPTION
6
7This document gives a general idea of Unicode and how to use Unicode
c8695642
KW
8in Perl. See L</Further Resources> for references to more in-depth
9treatments of Unicode.
ba62762e
JH
10
11=head2 Unicode
12
376d9008 13Unicode is a character set standard which plans to codify all of the
ba62762e
JH
14writing systems of the world, plus many other symbols.
15
c8695642
KW
16Unicode and ISO/IEC 10646 are coordinated standards that unify
17almost all other modern character set standards,
18covering more than 80 writing systems and hundreds of languages,
376d9008 19including all commercially-important modern languages. All characters
ba62762e
JH
20in the largest Chinese, Japanese, and Korean dictionaries are also
21encoded. The standards will eventually cover almost all characters in
22more than 250 writing systems and thousands of languages.
c8695642 23Unicode 1.0 was released in October 1991, and 6.0 in October 2010.
ba62762e
JH
24
25A Unicode I<character> is an abstract entity. It is not bound to any
376d9008
JB
26particular integer width, especially not to the C language C<char>.
27Unicode is language-neutral and display-neutral: it does not encode the
e1b711da 28language of the text, and it does not generally define fonts or other graphical
ba62762e
JH
29layout details. Unicode operates on characters and on text built from
30those characters.
31
32Unicode defines characters like C<LATIN CAPITAL LETTER A> or C<GREEK
376d9008
JB
33SMALL LETTER ALPHA> and unique numbers for the characters, in this
34case 0x0041 and 0x03B1, respectively. These unique numbers are called
c8695642
KW
35I<code points>. A code point is essentially the position of the
36character within the set of all possible Unicode characters, and thus in
37Perl, the term I<ordinal> is often used interchangeably with it.
ba62762e
JH
38
39The Unicode standard prefers using hexadecimal notation for the code
4c496f0c
JH
40points. If numbers like C<0x0041> are unfamiliar to you, take a peek
41at a later section, L</"Hexadecimal Notation">. The Unicode standard
42uses the notation C<U+0041 LATIN CAPITAL LETTER A>, to give the
43hexadecimal code point and the normative name of the character.
ba62762e
JH
44
45Unicode also defines various I<properties> for the characters, like
376d9008 46"uppercase" or "lowercase", "decimal digit", or "punctuation";
ba62762e
JH
47these properties are independent of the names of the characters.
48Furthermore, various operations on the characters like uppercasing,
376d9008 49lowercasing, and collating (sorting) are defined.
ba62762e 50
0111a78f
KW
51A Unicode I<logical> "character" can actually consist of more than one internal
52I<actual> "character" or code point. For Western languages, this is adequately
c670e63a 53modelled by a I<base character> (like C<LATIN CAPITAL LETTER A>) followed
0111a78f 54by one or more I<modifiers> (like C<COMBINING ACUTE ACCENT>). This sequence of
376d9008 55base character and modifiers is called a I<combining character
0111a78f 56sequence>. Some non-western languages require more complicated
c8695642
KW
57models, so Unicode created the I<grapheme cluster> concept, which was
58later further refined into the I<extended grapheme cluster>. For
59example, a Korean Hangul syllable is considered a single logical
60character, but most often consists of three actual
c670e63a
KW
61Unicode characters: a leading consonant followed by an interior vowel followed
62by a trailing consonant.
0111a78f
KW
63
64Whether to call these extended grapheme clusters "characters" depends on your
65point of view. If you are a programmer, you probably would tend towards seeing
c8695642
KW
66each element in the sequences as one unit, or "character". However from
67the user's point of view, the whole sequence could be seen as one
68"character" since that's probably what it looks like in the context of the
69user's language. In this document, we take the programmer's point of
70view: one "character" is one Unicode code point.
71
72For some combinations of base character and modifiers, there are
73I<precomposed> characters. There is a single character equivalent, for
4d2d7a4c 74example, for the sequence C<LATIN CAPITAL LETTER A> followed by
c8695642
KW
75C<COMBINING ACUTE ACCENT>. It is called C<LATIN CAPITAL LETTER A WITH
76ACUTE>. These precomposed characters are, however, only available for
77some combinations, and are mainly meant to support round-trip
78conversions between Unicode and legacy standards (like ISO 8859). Using
79sequences, as Unicode does, allows for needing fewer basic building blocks
80(code points) to express many more potential grapheme clusters. To
81support conversion between equivalent forms, various I<normalization
82forms> are also defined. Thus, C<LATIN CAPITAL LETTER A WITH ACUTE> is
83in I<Normalization Form Composed>, (abbreviated NFC), and the sequence
84C<LATIN CAPITAL LETTER A> followed by C<COMBINING ACUTE ACCENT>
85represents the same character in I<Normalization Form Decomposed> (NFD).
ba62762e
JH
86
87Because of backward compatibility with legacy encodings, the "a unique
376d9008
JB
88number for every character" idea breaks down a bit: instead, there is
89"at least one number for every character". The same character could
90be represented differently in several legacy encodings. The
91a927c1 91converse is not true: some code points do not have an assigned
376d9008
JB
92character. Firstly, there are unallocated code points within
93otherwise used blocks. Secondly, there are special Unicode control
94characters that do not represent true characters.
ba62762e 95
e9078126
KW
96When Unicode was first conceived, it was thought that all the world's
97characters could be represented using a 16-bit word; that is a maximum of
91a927c1
KW
98C<0x10000> (or 65,536) characters would be needed, from C<0x0000> to
99C<0xFFFF>. This soon proved to be wrong, and since Unicode 2.0 (July
4c496f0c 1001996), Unicode has been defined all the way up to 21 bits (C<0x10FFFF>),
e9078126
KW
101and Unicode 3.1 (March 2001) defined the first characters above C<0xFFFF>.
102The first C<0x10000> characters are called the I<Plane 0>, or the
103I<Basic Multilingual Plane> (BMP). With Unicode 3.1, 17 (yes,
104seventeen) planes in all were defined--but they are nowhere near full of
105defined characters, yet.
106
107When a new language is being encoded, Unicode generally will choose a
108C<block> of consecutive unallocated code points for its characters. So
109far, the number of code points in these blocks has always been evenly
110divisible by 16. Extras in a block, not currently needed, are left
111unallocated, for future growth. But there have been occasions when
30189633 112a later release needed more code points than the available extras, and a
82aed44a
KW
113new block had to allocated somewhere else, not contiguous to the initial
114one, to handle the overflow. Thus, it became apparent early on that
fc273927 115"block" wasn't an adequate organizing principle, and so the C<Script>
82aed44a
KW
116property was created. (Later an improved script property was added as
117well, the C<Script_Extensions> property.) Those code points that are in
118overflow blocks can still
e9078126
KW
119have the same script as the original ones. The script concept fits more
120closely with natural language: there is C<Latin> script, C<Greek>
121script, and so on; and there are several artificial scripts, like
122C<Common> for characters that are used in multiple scripts, such as
123mathematical symbols. Scripts usually span varied parts of several
124blocks. For more information about scripts, see L<perlunicode/Scripts>.
1bfb14c4 125The division into blocks exists, but it is almost completely
e9078126
KW
126accidental--an artifact of how the characters have been and still are
127allocated. (Note that this paragraph has oversimplified things for the
128sake of this being an introduction. Unicode doesn't really encode
129languages, but the writing systems for them--their scripts; and one
130script can be used by many languages. Unicode also encodes things that
131aren't really about languages, such as symbols like C<BAGGAGE CLAIM>.)
ba62762e
JH
132
133The Unicode code points are just abstract numbers. To input and
4c496f0c
JH
134output these abstract numbers, the numbers must be I<encoded> or
135I<serialised> somehow. Unicode defines several I<character encoding
91a927c1
KW
136forms>, of which I<UTF-8> is the most popular. UTF-8 is a
137variable length encoding that encodes Unicode characters as 1 to 4
e1b711da 138bytes. Other encodings
4c496f0c 139include UTF-16 and UTF-32 and their big- and little-endian variants
f321be7e 140(UTF-8 is byte-order independent). The ISO/IEC 10646 defines the UCS-2
4c496f0c 141and UCS-4 encoding forms.
ba62762e 142
376d9008
JB
143For more information about encodings--for instance, to learn what
144I<surrogates> and I<byte order marks> (BOMs) are--see L<perlunicode>.
ba62762e
JH
145
146=head2 Perl's Unicode Support
147
c2fb32ed
BF
148Starting from Perl v5.6.0, Perl has had the capacity to handle Unicode
149natively. Perl v5.8.0, however, is the first recommended release for
376d9008
JB
150serious Unicode work. The maintenance release 5.6.1 fixed many of the
151problems of the initial Unicode implementation, but for example
1bfb14c4 152regular expressions still do not work with Unicode in 5.6.1.
c2fb32ed 153Perl v5.14.0 is the first release where Unicode support is
d6c970c7
AC
154(almost) seamlessly integrable without some gotchas. (There are two
155exceptions. Firstly, some differences in L<quotemeta|perlfunc/quotemeta>
156were fixed starting in Perl 5.16.0. Secondly, some differences in
157L<the range operator|perlop/Range Operators> were fixed starting in
158Perl 5.26.0.)
159
160To enable this
c8695642
KW
161seamless support, you should C<use feature 'unicode_strings'> (which is
162automatically selected if you C<use 5.012> or higher). See L<feature>.
163(5.14 also fixes a number of bugs and departures from the Unicode
164standard.)
165
c2fb32ed 166Before Perl v5.8.0, the use of C<use utf8> was used to declare
ba62762e 167that operations in the current block or file would be Unicode-aware.
376d9008 168This model was found to be wrong, or at least clumsy: the "Unicodeness"
1bfb14c4 169is now carried with the data, instead of being attached to the
c8695642 170operations.
c2fb32ed 171Starting with Perl v5.8.0, only one case remains where an explicit C<use
c8695642
KW
172utf8> is needed: if your Perl script itself is encoded in UTF-8, you can
173use UTF-8 in your identifier names, and in string and regular expression
376d9008 174literals, by saying C<use utf8>. This is not the default because
8f8cf39c 175scripts with legacy 8-bit data in them would break. See L<utf8>.
ba62762e
JH
176
177=head2 Perl's Unicode Model
178
376d9008 179Perl supports both pre-5.6 strings of eight-bit native bytes, and
c8695642
KW
180strings of Unicode characters. The general principle is that Perl tries
181to keep its data as eight-bit bytes for as long as possible, but as soon
182as Unicodeness cannot be avoided, the data is transparently upgraded
c2fb32ed 183to Unicode. Prior to Perl v5.14.0, the upgrade was not completely
c8695642
KW
184transparent (see L<perlunicode/The "Unicode Bug">), and for backwards
185compatibility, full transparency is not gained unless C<use feature
186'unicode_strings'> (see L<feature>) or C<use 5.012> (or higher) is
187selected.
ba62762e 188
4192de81 189Internally, Perl currently uses either whatever the native eight-bit
376d9008
JB
190character set of the platform (for example Latin-1) is, defaulting to
191UTF-8, to encode Unicode strings. Specifically, if all code points in
192the string are C<0xFF> or less, Perl uses the native eight-bit
193character set. Otherwise, it uses UTF-8.
4192de81 194
7ca610e8 195A user of Perl does not normally need to know nor care how Perl
20ba30f4 196happens to encode its internal strings, but it becomes relevant when
ac036724 197outputting Unicode strings to a stream without a PerlIO layer (one with
198the "default" encoding). In such a case, the raw bytes used internally
7ca610e8
JH
199(the native character set or UTF-8, as appropriate for each string)
200will be used, and a "Wide character" warning will be issued if those
201strings contain a character beyond 0x00FF.
4192de81
JH
202
203For example,
204
ae5648b3 205 perl -e 'print "\x{DF}\n", "\x{0100}\x{DF}\n"'
4192de81
JH
206
207produces a fairly useless mixture of native bytes and UTF-8, as well
1bfb14c4 208as a warning:
4192de81 209
1bfb14c4
JH
210 Wide character in print at ...
211
740d4bb2 212To output UTF-8, use the C<:encoding> or C<:utf8> output layer. Prepending
4192de81
JH
213
214 binmode(STDOUT, ":utf8");
215
376d9008 216to this sample program ensures that the output is completely UTF-8,
1bfb14c4 217and removes the program's warning.
ba62762e 218
8aa8f774
JH
219You can enable automatic UTF-8-ification of your standard file
220handles, default C<open()> layer, and C<@ARGV> by using either
221the C<-C> command line switch or the C<PERL_UNICODE> environment
222variable, see L<perlrun> for the documentation of the C<-C> switch.
223
c8695642
KW
224Note that this means that Perl expects other software to work the same
225way:
8aa8f774 226if Perl has been led to believe that STDIN should be UTF-8, but then
c8695642
KW
227STDIN coming in from another command is not UTF-8, Perl will likely
228complain about the malformed UTF-8.
b310b053 229
ec71e770
JH
230All features that combine Unicode and I/O also require using the new
231PerlIO feature. Almost all Perl 5.8 platforms do use PerlIO, though:
232you can see whether yours is by running "perl -V" and looking for
233C<useperlio=define>.
234
64c66fb6
JH
235=head2 Unicode and EBCDIC
236
0145df7d
KW
237Perl 5.8.0 added support for Unicode on EBCDIC platforms. This support
238was allowed to lapse in later releases, but was revived in 5.22.
239Unicode support is somewhat more complex to implement since additional
240conversions are needed. See L<perlebcdic> for more information.
64c66fb6 241
64c66fb6 242On EBCDIC platforms, the internal Unicode encoding form is UTF-EBCDIC
376d9008
JB
243instead of UTF-8. The difference is that as UTF-8 is "ASCII-safe" in
244that ASCII characters encode to UTF-8 as-is, while UTF-EBCDIC is
0145df7d
KW
245"EBCDIC-safe", in that all the basic characters (which includes all
246those that have ASCII equivalents (like C<"A">, C<"0">, C<"%">, I<etc.>)
247are the same in both EBCDIC and UTF-EBCDIC. Often, documentation
248will use the term "UTF-8" to mean UTF-EBCDIC as well. This is the case
249in this document.
ba62762e
JH
250
251=head2 Creating Unicode
252
0145df7d
KW
253This section applies fully to Perls starting with v5.22. Various
254caveats for earlier releases are in the L</Earlier releases caveats>
255subsection below.
ba62762e 256
0145df7d
KW
257To create Unicode characters in literals,
258use the C<\N{...}> notation in double-quoted strings:
ba62762e 259
0145df7d
KW
260 my $smiley_from_name = "\N{WHITE SMILING FACE}";
261 my $smiley_from_code_point = "\N{U+263a}";
ba62762e 262
0145df7d 263Similarly, they can be used in regular expression literals
ba62762e 264
0145df7d
KW
265 $smiley =~ /\N{WHITE SMILING FACE}/;
266 $smiley =~ /\N{U+263a}/;
ba62762e 267
0145df7d 268At run-time you can use:
ba62762e 269
0145df7d
KW
270 use charnames ();
271 my $hebrew_alef_from_name
272 = charnames::string_vianame("HEBREW LETTER ALEF");
273 my $hebrew_alef_from_code_point = charnames::string_vianame("U+05D0");
ba62762e 274
376d9008
JB
275Naturally, C<ord()> will do the reverse: it turns a character into
276a code point.
ba62762e 277
0145df7d
KW
278There are other runtime options as well. You can use C<pack()>:
279
280 my $hebrew_alef_from_code_point = pack("U", 0x05d0);
281
282Or you can use C<chr()>, though it is less convenient in the general
283case:
284
285 $hebrew_alef_from_code_point = chr(utf8::unicode_to_native(0x05d0));
286 utf8::upgrade($hebrew_alef_from_code_point);
287
288The C<utf8::unicode_to_native()> and C<utf8::upgrade()> aren't needed if
289the argument is above 0xFF, so the above could have been written as
290
291 $hebrew_alef_from_code_point = chr(0x05d0);
ba62762e 292
0145df7d 293since 0x5d0 is above 255.
ba62762e 294
0145df7d
KW
295C<\x{}> and C<\o{}> can also be used to specify code points at compile
296time in double-quotish strings, but, for backward compatibility with
297older Perls, the same rules apply as with C<chr()> for code points less
298than 256.
ba62762e 299
0145df7d
KW
300C<utf8::unicode_to_native()> is used so that the Perl code is portable
301to EBCDIC platforms. You can omit it if you're I<really> sure no one
302will ever want to use your code on a non-ASCII platform. Starting in
303Perl v5.22, calls to it on ASCII platforms are optimized out, so there's
304no performance penalty at all in adding it. Or you can simply use the
305other constructs that don't require it.
ba62762e 306
0145df7d
KW
307See L</"Further Resources"> for how to find all these names and numeric
308codes.
ba62762e 309
0145df7d 310=head3 Earlier releases caveats
8a5e5dd5 311
0145df7d
KW
312On EBCDIC platforms, prior to v5.22, using C<\N{U+...}> doesn't work
313properly.
1eda90df 314
0145df7d
KW
315Prior to v5.16, using C<\N{...}> with a character name (as opposed to a
316C<U+...> code point) required a S<C<use charnames :full>>.
771cd3b2 317
0145df7d
KW
318Prior to v5.14, there were some bugs in C<\N{...}> with a character name
319(as opposed to a C<U+...> code point).
320
321C<charnames::string_vianame()> was introduced in v5.14. Prior to that,
322C<charnames::vianame()> should work, but only if the argument is of the
323form C<"U+...">. Your best bet there for runtime Unicode by character
324name is probably:
325
326 use charnames ();
327 my $hebrew_alef_from_name
328 = pack("U", charnames::vianame("HEBREW LETTER ALEF"));
1eda90df 329
ba62762e
JH
330=head2 Handling Unicode
331
332Handling Unicode is for the most part transparent: just use the
333strings as usual. Functions like C<index()>, C<length()>, and
334C<substr()> will work on the Unicode characters; regular expressions
335will work on the Unicode characters (see L<perlunicode> and L<perlretut>).
336
0111a78f
KW
337Note that Perl considers grapheme clusters to be separate characters, so for
338example
ba62762e 339
9ba22424
KW
340 print length("\N{LATIN CAPITAL LETTER A}\N{COMBINING ACUTE ACCENT}"),
341 "\n";
ba62762e
JH
342
343will print 2, not 1. The only exception is that regular expressions
c8695642
KW
344have C<\X> for matching an extended grapheme cluster. (Thus C<\X> in a
345regular expression would match the entire sequence of both the example
346characters.)
ba62762e 347
376d9008
JB
348Life is not quite so transparent, however, when working with legacy
349encodings, I/O, and certain special cases:
ba62762e
JH
350
351=head2 Legacy Encodings
352
c8695642
KW
353When you combine legacy data and Unicode, the legacy data needs
354to be upgraded to Unicode. Normally the legacy data is assumed to be
355ISO 8859-1 (or EBCDIC, if applicable).
ba62762e 356
376d9008 357The C<Encode> module knows about many encodings and has interfaces
ba62762e
JH
358for doing conversions between those encodings:
359
2c9359a2
RGS
360 use Encode 'decode';
361 $data = decode("iso-8859-3", $data); # convert from legacy to utf-8
ba62762e
JH
362
363=head2 Unicode I/O
364
8baee566 365Normally, writing out Unicode data
ba62762e 366
8baee566 367 print FH $some_string_with_unicode, "\n";
ba62762e 368
8baee566 369produces raw bytes that Perl happens to use to internally encode the
376d9008
JB
370Unicode string. Perl's internal encoding depends on the system as
371well as what characters happen to be in the string at the time. If
372any of the characters are at code points C<0x100> or above, you will get
373a warning. To ensure that the output is explicitly rendered in the
374encoding you desire--and to avoid the warning--open the stream with
375the desired encoding. Some examples:
ba62762e 376
ec90690f
ST
377 open FH, ">:utf8", "file";
378
379 open FH, ">:encoding(ucs2)", "file";
380 open FH, ">:encoding(UTF-8)", "file";
381 open FH, ">:encoding(shift_jis)", "file";
1d7919c5 382
376d9008 383and on already open streams, use C<binmode()>:
1d7919c5
JH
384
385 binmode(STDOUT, ":utf8");
386
ec90690f
ST
387 binmode(STDOUT, ":encoding(ucs2)");
388 binmode(STDOUT, ":encoding(UTF-8)");
389 binmode(STDOUT, ":encoding(shift_jis)");
390
b5d8778e 391The matching of encoding names is loose: case does not matter, and
fae2c0fb 392many encodings have several aliases. Note that the C<:utf8> layer
1bfb14c4 393must always be specified exactly like that; it is I<not> subject to
c8695642 394the loose matching of encoding names. Also note that currently C<:utf8> is unsafe for
740d4bb2 395input, because it accepts the data without validating that it is indeed valid
367b3305
KW
396UTF-8; you should instead use C<:encoding(utf-8)> (with or without a
397hyphen).
b5d8778e 398
376d9008
JB
399See L<PerlIO> for the C<:utf8> layer, L<PerlIO::encoding> and
400L<Encode::PerlIO> for the C<:encoding()> layer, and
401L<Encode::Supported> for many encodings supported by the C<Encode>
402module.
ba62762e 403
a5f0baef 404Reading in a file that you know happens to be encoded in one of the
1bfb14c4
JH
405Unicode or legacy encodings does not magically turn the data into
406Unicode in Perl's eyes. To do that, specify the appropriate
fae2c0fb 407layer when opening files
ba62762e 408
740d4bb2 409 open(my $fh,'<:encoding(utf8)', 'anything');
8baee566
JF
410 my $line_of_unicode = <$fh>;
411
ec90690f 412 open(my $fh,'<:encoding(Big5)', 'anything');
8baee566 413 my $line_of_unicode = <$fh>;
ba62762e 414
fae2c0fb 415The I/O layers can also be specified more flexibly with
376d9008 416the C<open> pragma. See L<open>, or look at the following example.
ba62762e 417
9e5bbba0
KW
418 use open ':encoding(utf8)'; # input/output default encoding will be
419 # UTF-8
1d7919c5
JH
420 open X, ">file";
421 print X chr(0x100), "\n";
ba62762e 422 close X;
1d7919c5 423 open Y, "<file";
ba62762e
JH
424 printf "%#x\n", ord(<Y>); # this should print 0x100
425 close Y;
426
fae2c0fb 427With the C<open> pragma you can use the C<:locale> layer
ba62762e 428
12f98225 429 BEGIN { $ENV{LC_ALL} = $ENV{LANG} = 'ru_RU.KOI8-R' }
9e5bbba0
KW
430 # the :locale will probe the locale environment variables like
431 # LC_ALL
ba62762e
JH
432 use open OUT => ':locale'; # russki parusski
433 open(O, ">koi8");
434 print O chr(0x430); # Unicode CYRILLIC SMALL LETTER A = KOI8-R 0xc1
435 close O;
436 open(I, "<koi8");
437 printf "%#x\n", ord(<I>), "\n"; # this should print 0xc1
438 close I;
439
8baee566
JF
440These methods install a transparent filter on the I/O stream that
441converts data from the specified encoding when it is read in from the
a5f0baef 442stream. The result is always Unicode.
ba62762e
JH
443
444The L<open> pragma affects all the C<open()> calls after the pragma by
fae2c0fb
RGS
445setting default layers. If you want to affect only certain
446streams, use explicit layers directly in the C<open()> call.
ba62762e
JH
447
448You can switch encodings on an already opened stream by using
8baee566 449C<binmode()>; see L<perlfunc/binmode>.
ba62762e 450
c2fb32ed 451The C<:locale> does not currently work with
1ecefa54 452C<open()> and C<binmode()>, only with the C<open> pragma. The
8baee566 453C<:utf8> and C<:encoding(...)> methods do work with all of C<open()>,
1ecefa54 454C<binmode()>, and the C<open> pragma.
ba62762e 455
fae2c0fb 456Similarly, you may use these I/O layers on output streams to
a5f0baef
JH
457automatically convert Unicode to the specified encoding when it is
458written to the stream. For example, the following snippet copies the
459contents of the file "text.jis" (encoded as ISO-2022-JP, aka JIS) to
460the file "text.utf8", encoded as UTF-8:
ba62762e 461
c722ef0a
JH
462 open(my $nihongo, '<:encoding(iso-2022-jp)', 'text.jis');
463 open(my $unicode, '>:utf8', 'text.utf8');
0cf8a8d9 464 while (<$nihongo>) { print $unicode $_ }
ba62762e
JH
465
466The naming of encodings, both by the C<open()> and by the C<open>
2575c402
JW
467pragma allows for flexible names: C<koi8-r> and C<KOI8R> will both be
468understood.
ba62762e
JH
469
470Common encodings recognized by ISO, MIME, IANA, and various other
8baee566 471standardisation organisations are recognised; for a more detailed
1bfb14c4 472list see L<Encode::Supported>.
ba62762e
JH
473
474C<read()> reads characters and returns the number of characters.
475C<seek()> and C<tell()> operate on byte counts, as do C<sysread()>
476and C<sysseek()>.
477
8baee566 478Notice that because of the default behaviour of not doing any
fae2c0fb 479conversion upon input if there is no default layer,
ba62762e 480it is easy to mistakenly write code that keeps on expanding a file
1bfb14c4 481by repeatedly encoding the data:
ba62762e
JH
482
483 # BAD CODE WARNING
484 open F, "file";
8baee566 485 local $/; ## read in the whole file of 8-bit characters
ba62762e
JH
486 $t = <F>;
487 close F;
740d4bb2 488 open F, ">:encoding(utf8)", "file";
8baee566 489 print F $t; ## convert to UTF-8 on output
ba62762e
JH
490 close F;
491
492If you run this code twice, the contents of the F<file> will be twice
740d4bb2
JW
493UTF-8 encoded. A C<use open ':encoding(utf8)'> would have avoided the
494bug, or explicitly opening also the F<file> for input as UTF-8.
ba62762e 495
0c901d84 496B<NOTE>: the C<:utf8> and C<:encoding> features work only if your
cdf6c183
TH
497Perl has been built with L<PerlIO>, which is the default
498on most systems.
0c901d84 499
1ecefa54
JH
500=head2 Displaying Unicode As Text
501
502Sometimes you might want to display Perl scalars containing Unicode as
8baee566 503simple ASCII (or EBCDIC) text. The following subroutine converts
1ecefa54 504its argument so that Unicode characters with code points greater than
1bfb14c4
JH
505255 are displayed as C<\x{...}>, control characters (like C<\n>) are
506displayed as C<\x..>, and the rest of the characters as themselves:
1ecefa54 507
9e5bbba0 508 sub nice_string {
96cfa1df
KW
509 join("",
510 map { $_ > 255 # if wide character...
511 ? sprintf("\\x{%04X}", $_) # \x{...}
512 : chr($_) =~ /[[:cntrl:]]/ # else if control character...
513 ? sprintf("\\x%02X", $_) # \x..
514 : quotemeta(chr($_)) # else quoted or as themselves
515 } unpack("W*", $_[0])); # unpack Unicode characters
58c274a1
JF
516 }
517
518For example,
519
520 nice_string("foo\x{100}bar\n")
521
d0551e73 522returns the string
58c274a1 523
d0551e73
JH
524 'foo\x{0100}bar\x0A'
525
526which is ready to be printed.
1ecefa54 527
0145df7d
KW
528(C<\\x{}> is used here instead of C<\\N{}>, since it's most likely that
529you want to see what the native values are.)
530
ba62762e
JH
531=head2 Special Cases
532
533=over 4
534
535=item *
536
537Bit Complement Operator ~ And vec()
538
1bfb14c4
JH
539The bit complement operator C<~> may produce surprising results if
540used on strings containing characters with ordinal values above
541255. In such a case, the results are consistent with the internal
542encoding of the characters, but not with much else. So don't do
543that. Similarly for C<vec()>: you will be operating on the
544internally-encoded bit patterns of the Unicode characters, not on
545the code point values, which is very probably not what you want.
ba62762e
JH
546
547=item *
548
8baee566
JF
549Peeking At Perl's Internal Encoding
550
551Normal users of Perl should never care how Perl encodes any particular
a5f0baef 552Unicode string (because the normal ways to get at the contents of a
376d9008 553string with Unicode--via input and output--should always be via
fae2c0fb 554explicitly-defined I/O layers). But if you must, there are two
a5f0baef 555ways of looking behind the scenes.
ba62762e
JH
556
557One way of peeking inside the internal encoding of Unicode characters
f337b084
TH
558is to use C<unpack("C*", ...> to get the bytes of whatever the string
559encoding happens to be, or C<unpack("U0..", ...)> to get the bytes of the
560UTF-8 encoding:
ba62762e 561
8baee566 562 # this prints c4 80 for the UTF-8 bytes 0xc4 0x80
f337b084 563 print join(" ", unpack("U0(H2)*", pack("U", 0x100))), "\n";
ba62762e
JH
564
565Yet another way would be to use the Devel::Peek module:
566
567 perl -MDevel::Peek -e 'Dump(chr(0x100))'
568
1e54db1a 569That shows the C<UTF8> flag in FLAGS and both the UTF-8 bytes
376d9008 570and Unicode characters in C<PV>. See also later in this document
8800c35a 571the discussion about the C<utf8::is_utf8()> function.
ba62762e
JH
572
573=back
574
575=head2 Advanced Topics
576
577=over 4
578
579=item *
580
581String Equivalence
582
583The question of string equivalence turns somewhat complicated
376d9008 584in Unicode: what do you mean by "equal"?
ba62762e 585
07698885
RGS
586(Is C<LATIN CAPITAL LETTER A WITH ACUTE> equal to
587C<LATIN CAPITAL LETTER A>?)
ba62762e 588
a5f0baef
JH
589The short answer is that by default Perl compares equivalence (C<eq>,
590C<ne>) based only on code points of the characters. In the above
376d9008 591case, the answer is no (because 0x00C1 != 0x0041). But sometimes, any
c8695642 592CAPITAL LETTER A's should be considered equal, or even A's of any case.
ba62762e
JH
593
594The long answer is that you need to consider character normalization
e1b711da
KW
595and casing issues: see L<Unicode::Normalize>, Unicode Technical Report #15,
596L<Unicode Normalization Forms|http://www.unicode.org/unicode/reports/tr15> and
597sections on case mapping in the L<Unicode Standard|http://www.unicode.org>.
ba62762e 598
1bfb14c4 599As of Perl 5.8.0, the "Full" case-folding of I<Case
afba1538 600Mappings/SpecialCasing> is implemented, but bugs remain in C<qr//i> with them,
4d2d7a4c 601mostly fixed by 5.14, and essentially entirely by 5.18.
ba62762e
JH
602
603=item *
604
605String Collation
606
376d9008 607People like to see their strings nicely sorted--or as Unicode
ba62762e
JH
608parlance goes, collated. But again, what do you mean by collate?
609
07698885
RGS
610(Does C<LATIN CAPITAL LETTER A WITH ACUTE> come before or after
611C<LATIN CAPITAL LETTER A WITH GRAVE>?)
ba62762e 612
58c274a1 613The short answer is that by default, Perl compares strings (C<lt>,
ba62762e 614C<le>, C<cmp>, C<ge>, C<gt>) based only on the code points of the
1bfb14c4 615characters. In the above case, the answer is "after", since
da76a1f4 616C<0x00C1> > C<0x00C0>.
ba62762e
JH
617
618The long answer is that "it depends", and a good answer cannot be
619given without knowing (at the very least) the language context.
620See L<Unicode::Collate>, and I<Unicode Collation Algorithm>
2bbc8d55 621L<http://www.unicode.org/unicode/reports/tr10/>
ba62762e
JH
622
623=back
624
625=head2 Miscellaneous
626
627=over 4
628
629=item *
630
3ff56b75 631Character Ranges and Classes
ba62762e 632
0eb9ada1
KW
633Character ranges in regular expression bracketed character classes ( e.g.,
634C</[a-z]/>) and in the C<tr///> (also known as C<y///>) operator are not
635magically Unicode-aware. What this means is that C<[A-Za-z]> will not
636magically start to mean "all alphabetic letters" (not that it does mean that
637even for 8-bit characters; for those, if you are using locales (L<perllocale>),
638use C</[[:alpha:]]/>; and if not, use the 8-bit-aware property C<\p{alpha}>).
639
640All the properties that begin with C<\p> (and its inverse C<\P>) are actually
641character classes that are Unicode-aware. There are dozens of them, see
642L<perluniprops>.
643
0145df7d 644Starting in v5.22, you can use Unicode code points as the end points of
74fe8880
KW
645regular expression pattern character ranges, and the range will include
646all Unicode code points that lie between those end points, inclusive.
0145df7d 647
77c8f263 648 qr/ [ \N{U+03} - \N{U+20} ] /xx
0145df7d
KW
649
650includes the code points
651C<\N{U+03}>, C<\N{U+04}>, ..., C<\N{U+20}>.
ba62762e 652
f3f4813b 653This also works for ranges in C<tr///> starting in Perl v5.24.
74fe8880 654
ba62762e
JH
655=item *
656
657String-To-Number Conversions
658
376d9008
JB
659Unicode does define several other decimal--and numeric--characters
660besides the familiar 0 to 9, such as the Arabic and Indic digits.
ba62762e 661Perl does not support string-to-number conversion for digits other
4d2d7a4c 662than ASCII C<0> to C<9> (and ASCII C<a> to C<f> for hexadecimal).
c8695642 663To get safe conversions from any Unicode string, use
67592e11 664L<Unicode::UCD/num()>.
ba62762e
JH
665
666=back
667
668=head2 Questions With Answers
669
670=over 4
671
818c4caa 672=item *
5cb3728c
RB
673
674Will My Old Scripts Break?
ba62762e
JH
675
676Very probably not. Unless you are generating Unicode characters
1bfb14c4
JH
677somehow, old behaviour should be preserved. About the only behaviour
678that has changed and which could start generating Unicode is the old
679behaviour of C<chr()> where supplying an argument more than 255
680produced a character modulo 255. C<chr(300)>, for example, was equal
681to C<chr(45)> or "-" (in ASCII), now it is LATIN CAPITAL LETTER I WITH
682BREVE.
ba62762e 683
818c4caa 684=item *
5cb3728c
RB
685
686How Do I Make My Scripts Work With Unicode?
ba62762e
JH
687
688Very little work should be needed since nothing changes until you
1bfb14c4
JH
689generate Unicode data. The most important thing is getting input as
690Unicode; for that, see the earlier I/O discussion.
c8695642
KW
691To get full seamless Unicode support, add
692C<use feature 'unicode_strings'> (or C<use 5.012> or higher) to your
693script.
ba62762e 694
818c4caa 695=item *
5cb3728c
RB
696
697How Do I Know Whether My String Is In Unicode?
ba62762e 698
c8695642
KW
699You shouldn't have to care. But you may if your Perl is before 5.14.0
700or you haven't specified C<use feature 'unicode_strings'> or C<use
52d1f2c9 7015.012> (or higher) because otherwise the rules for the code points
c8695642 702in the range 128 to 255 are different depending on
2bbc8d55 703whether the string they are contained within is in Unicode or not.
e1b711da 704(See L<perlunicode/When Unicode Does Not Happen>.)
ba62762e 705
2bbc8d55 706To determine if a string is in Unicode, use:
ba62762e 707
8800c35a 708 print utf8::is_utf8($string) ? 1 : 0, "\n";
ba62762e
JH
709
710But note that this doesn't mean that any of the characters in the
711string are necessary UTF-8 encoded, or that any of the characters have
712code points greater than 0xFF (255) or even 0x80 (128), or that the
713string has any characters at all. All the C<is_utf8()> does is to
714return the value of the internal "utf8ness" flag attached to the
376d9008 715C<$string>. If the flag is off, the bytes in the scalar are interpreted
3c1c8017 716as a single byte encoding. If the flag is on, the bytes in the scalar
0eb9ada1
KW
717are interpreted as the (variable-length, potentially multi-byte) UTF-8 encoded
718code points of the characters. Bytes added to a UTF-8 encoded string are
1e54db1a 719automatically upgraded to UTF-8. If mixed non-UTF-8 and UTF-8 scalars
c8695642 720are merged (double-quoted interpolation, explicit concatenation, or
3c1c8017
AT
721printf/sprintf parameter substitution), the result will be UTF-8 encoded
722as if copies of the byte strings were upgraded to UTF-8: for example,
723
724 $a = "ab\x80c";
725 $b = "\x{100}";
726 print "$a = $b\n";
727
a02b5feb
JH
728the output string will be UTF-8-encoded C<ab\x80c = \x{100}\n>, but
729C<$a> will stay byte-encoded.
ba62762e
JH
730
731Sometimes you might really need to know the byte length of a string
ce7675db 732instead of the character length. For that use either the
00d823b9 733C<Encode::encode_utf8()> function or the C<bytes> pragma
c8695642 734and the C<length()> function:
ba62762e
JH
735
736 my $unicode = chr(0x100);
737 print length($unicode), "\n"; # will print 1
ce7675db 738 require Encode;
9ba22424 739 print length(Encode::encode_utf8($unicode)),"\n"; # will print 2
00d823b9 740 use bytes;
1bfb14c4
JH
741 print length($unicode), "\n"; # will also print 2
742 # (the 0xC4 0x80 of the UTF-8)
0eb9ada1 743 no bytes;
ba62762e 744
818c4caa 745=item *
5cb3728c 746
c8695642
KW
747How Do I Find Out What Encoding a File Has?
748
968ee499 749You might try L<Encode::Guess>, but it has a number of limitations.
c8695642
KW
750
751=item *
752
5cb3728c 753How Do I Detect Data That's Not Valid In a Particular Encoding?
ba62762e 754
8baee566
JF
755Use the C<Encode> package to try converting it.
756For example,
ba62762e 757
bb2f379c 758 use Encode 'decode_utf8';
2bbc8d55 759
228ee848 760 if (eval { decode_utf8($string, Encode::FB_CROAK); 1 }) {
a365f2ce 761 # $string is valid utf8
ba62762e 762 } else {
a365f2ce 763 # $string is not valid utf8
ba62762e
JH
764 }
765
f337b084 766Or use C<unpack> to try decoding it:
ba62762e
JH
767
768 use warnings;
f337b084 769 @chars = unpack("C0U*", $string_of_bytes_that_I_think_is_utf8);
ba62762e 770
ae5648b3
RGS
771If invalid, a C<Malformed UTF-8 character> warning is produced. The "C0" means
772"process the string character per character". Without that, the
773C<unpack("U*", ...)> would work in C<U0> mode (the default if the format
774string starts with C<U>) and it would return the bytes making up the UTF-8
f337b084 775encoding of the target string, something that will always work.
ba62762e 776
818c4caa 777=item *
5cb3728c
RB
778
779How Do I Convert Binary Data Into a Particular Encoding, Or Vice Versa?
ba62762e 780
8baee566
JF
781This probably isn't as useful as you might think.
782Normally, you shouldn't need to.
ba62762e 783
1bfb14c4 784In one sense, what you are asking doesn't make much sense: encodings
376d9008 785are for characters, and binary data are not "characters", so converting
a5f0baef
JH
786"data" into some encoding isn't meaningful unless you know in what
787character set and encoding the binary data is in, in which case it's
376d9008 788not just binary data, now is it?
8baee566 789
1bfb14c4
JH
790If you have a raw sequence of bytes that you know should be
791interpreted via a particular encoding, you can use C<Encode>:
ba62762e
JH
792
793 use Encode 'from_to';
794 from_to($data, "iso-8859-1", "utf-8"); # from latin-1 to utf-8
795
1bfb14c4
JH
796The call to C<from_to()> changes the bytes in C<$data>, but nothing
797material about the nature of the string has changed as far as Perl is
798concerned. Both before and after the call, the string C<$data>
799contains just a bunch of 8-bit bytes. As far as Perl is concerned,
800the encoding of the string remains as "system-native 8-bit bytes".
8baee566
JF
801
802You might relate this to a fictional 'Translate' module:
803
804 use Translate;
805 my $phrase = "Yes";
806 Translate::from_to($phrase, 'english', 'deutsch');
807 ## phrase now contains "Ja"
ba62762e 808
8baee566 809The contents of the string changes, but not the nature of the string.
1bfb14c4
JH
810Perl doesn't know any more after the call than before that the
811contents of the string indicates the affirmative.
ba62762e 812
376d9008 813Back to converting data. If you have (or want) data in your system's
a5f0baef
JH
814native 8-bit encoding (e.g. Latin-1, EBCDIC, etc.), you can use
815pack/unpack to convert to/from Unicode.
ba62762e 816
f337b084
TH
817 $native_string = pack("W*", unpack("U*", $Unicode_string));
818 $Unicode_string = pack("U*", unpack("W*", $native_string));
ba62762e
JH
819
820If you have a sequence of bytes you B<know> is valid UTF-8,
821but Perl doesn't know it yet, you can make Perl a believer, too:
822
823 use Encode 'decode_utf8';
8baee566 824 $Unicode = decode_utf8($bytes);
ba62762e 825
f337b084
TH
826or:
827
828 $Unicode = pack("U0a*", $bytes);
ae5648b3 829
2bbc8d55
SP
830You can find the bytes that make up a UTF-8 sequence with
831
9e5bbba0 832 @bytes = unpack("C*", $Unicode_string)
2bbc8d55
SP
833
834and you can create well-formed Unicode with
835
9e5bbba0 836 $Unicode_string = pack("U*", 0xff, ...)
ba62762e 837
818c4caa 838=item *
5cb3728c
RB
839
840How Do I Display Unicode? How Do I Input Unicode?
ba62762e 841
2bbc8d55
SP
842See L<http://www.alanwood.net/unicode/> and
843L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e 844
818c4caa 845=item *
5cb3728c
RB
846
847How Does Unicode Work With Traditional Locales?
ba62762e 848
31f05a37
KW
849If your locale is a UTF-8 locale, starting in Perl v5.20, Perl works
850well for all categories except C<LC_COLLATE> dealing with sorting and
851the C<cmp> operator.
852
853For other locales, starting in Perl 5.16, you can specify
66cbab2c
KW
854
855 use locale ':not_characters';
856
31f05a37 857to get Perl to work well with them. The catch is that you
66cbab2c
KW
858have to translate from the locale character set to/from Unicode
859yourself. See L</Unicode IE<sol>O> above for how to
860
861 use open ':locale';
862
863to accomplish this, but full details are in L<perllocale/Unicode and
30189633 864UTF-8>, including gotchas that happen if you don't specify
66cbab2c 865C<:not_characters>.
ba62762e
JH
866
867=back
868
869=head2 Hexadecimal Notation
870
376d9008
JB
871The Unicode standard prefers using hexadecimal notation because
872that more clearly shows the division of Unicode into blocks of 256 characters.
ba62762e
JH
873Hexadecimal is also simply shorter than decimal. You can use decimal
874notation, too, but learning to use hexadecimal just makes life easier
1bfb14c4 875with the Unicode standard. The C<U+HHHH> notation uses hexadecimal,
076d825e 876for example.
ba62762e
JH
877
878The C<0x> prefix means a hexadecimal number, the digits are 0-9 I<and>
879a-f (or A-F, case doesn't matter). Each hexadecimal digit represents
880four bits, or half a byte. C<print 0x..., "\n"> will show a
881hexadecimal number in decimal, and C<printf "%x\n", $decimal> will
882show a decimal number in hexadecimal. If you have just the
376d9008 883"hex digits" of a hexadecimal number, you can use the C<hex()> function.
ba62762e
JH
884
885 print 0x0009, "\n"; # 9
886 print 0x000a, "\n"; # 10
887 print 0x000f, "\n"; # 15
888 print 0x0010, "\n"; # 16
889 print 0x0011, "\n"; # 17
890 print 0x0100, "\n"; # 256
891
892 print 0x0041, "\n"; # 65
893
894 printf "%x\n", 65; # 41
895 printf "%#x\n", 65; # 0x41
896
897 print hex("41"), "\n"; # 65
898
899=head2 Further Resources
900
901=over 4
902
903=item *
904
905Unicode Consortium
906
2bbc8d55 907L<http://www.unicode.org/>
ba62762e
JH
908
909=item *
910
911Unicode FAQ
912
2bbc8d55 913L<http://www.unicode.org/unicode/faq/>
ba62762e
JH
914
915=item *
916
917Unicode Glossary
918
2bbc8d55 919L<http://www.unicode.org/glossary/>
ba62762e
JH
920
921=item *
922
c8695642
KW
923Unicode Recommended Reading List
924
925The Unicode Consortium has a list of articles and books, some of which
926give a much more in depth treatment of Unicode:
927L<http://unicode.org/resources/readinglist.html>
928
929=item *
930
ba62762e
JH
931Unicode Useful Resources
932
2bbc8d55 933L<http://www.unicode.org/unicode/onlinedat/resources.html>
ba62762e
JH
934
935=item *
936
937Unicode and Multilingual Support in HTML, Fonts, Web Browsers and Other Applications
938
2bbc8d55 939L<http://www.alanwood.net/unicode/>
ba62762e
JH
940
941=item *
942
943UTF-8 and Unicode FAQ for Unix/Linux
944
2bbc8d55 945L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e
JH
946
947=item *
948
949Legacy Character Sets
950
2bbc8d55
SP
951L<http://www.czyborra.com/>
952L<http://www.eki.ee/letter/>
ba62762e
JH
953
954=item *
955
ba62762e
JH
956You can explore various information from the Unicode data files using
957the C<Unicode::UCD> module.
958
959=back
960
f6edf83b
JH
961=head1 UNICODE IN OLDER PERLS
962
963If you cannot upgrade your Perl to 5.8.0 or later, you can still
964do some Unicode processing by using the modules C<Unicode::String>,
965C<Unicode::Map8>, and C<Unicode::Map>, available from CPAN.
966If you have the GNU recode installed, you can also use the
376d9008 967Perl front-end C<Convert::Recode> for character conversions.
f6edf83b 968
aaef10c5 969The following are fast conversions from ISO 8859-1 (Latin-1) bytes
63de3cb2 970to UTF-8 bytes and back, the code works even with older Perl 5 versions.
aaef10c5
JH
971
972 # ISO 8859-1 to UTF-8
973 s/([\x80-\xFF])/chr(0xC0|ord($1)>>6).chr(0x80|ord($1)&0x3F)/eg;
974
975 # UTF-8 to ISO 8859-1
976 s/([\xC2\xC3])([\x80-\xBF])/chr(ord($1)<<6&0xC0|ord($2)&0x3F)/eg;
977
ba62762e
JH
978=head1 SEE ALSO
979
2575c402 980L<perlunitut>, L<perlunicode>, L<Encode>, L<open>, L<utf8>, L<bytes>,
4c496f0c
JH
981L<perlretut>, L<perlrun>, L<Unicode::Collate>, L<Unicode::Normalize>,
982L<Unicode::UCD>
ba62762e 983
376d9008 984=head1 ACKNOWLEDGMENTS
ba62762e
JH
985
986Thanks to the kind readers of the perl5-porters@perl.org,
987perl-unicode@perl.org, linux-utf8@nl.linux.org, and unicore@unicode.org
988mailing lists for their valuable feedback.
989
990=head1 AUTHOR, COPYRIGHT, AND LICENSE
991
91a927c1
KW
992Copyright 2001-2011 Jarkko Hietaniemi E<lt>jhi@iki.fiE<gt>.
993Now maintained by Perl 5 Porters.
ba62762e
JH
994
995This document may be distributed under the same terms as Perl itself.