This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Encode::Guess is iffy
[perl5.git] / pod / perluniintro.pod
CommitLineData
ba62762e
JH
1=head1 NAME
2
07fcf8ff 3perluniintro - Perl Unicode introduction
ba62762e
JH
4
5=head1 DESCRIPTION
6
7This document gives a general idea of Unicode and how to use Unicode
c8695642
KW
8in Perl. See L</Further Resources> for references to more in-depth
9treatments of Unicode.
ba62762e
JH
10
11=head2 Unicode
12
376d9008 13Unicode is a character set standard which plans to codify all of the
ba62762e
JH
14writing systems of the world, plus many other symbols.
15
c8695642
KW
16Unicode and ISO/IEC 10646 are coordinated standards that unify
17almost all other modern character set standards,
18covering more than 80 writing systems and hundreds of languages,
376d9008 19including all commercially-important modern languages. All characters
ba62762e
JH
20in the largest Chinese, Japanese, and Korean dictionaries are also
21encoded. The standards will eventually cover almost all characters in
22more than 250 writing systems and thousands of languages.
c8695642 23Unicode 1.0 was released in October 1991, and 6.0 in October 2010.
ba62762e
JH
24
25A Unicode I<character> is an abstract entity. It is not bound to any
376d9008
JB
26particular integer width, especially not to the C language C<char>.
27Unicode is language-neutral and display-neutral: it does not encode the
e1b711da 28language of the text, and it does not generally define fonts or other graphical
ba62762e
JH
29layout details. Unicode operates on characters and on text built from
30those characters.
31
32Unicode defines characters like C<LATIN CAPITAL LETTER A> or C<GREEK
376d9008
JB
33SMALL LETTER ALPHA> and unique numbers for the characters, in this
34case 0x0041 and 0x03B1, respectively. These unique numbers are called
c8695642
KW
35I<code points>. A code point is essentially the position of the
36character within the set of all possible Unicode characters, and thus in
37Perl, the term I<ordinal> is often used interchangeably with it.
ba62762e
JH
38
39The Unicode standard prefers using hexadecimal notation for the code
4c496f0c
JH
40points. If numbers like C<0x0041> are unfamiliar to you, take a peek
41at a later section, L</"Hexadecimal Notation">. The Unicode standard
42uses the notation C<U+0041 LATIN CAPITAL LETTER A>, to give the
43hexadecimal code point and the normative name of the character.
ba62762e
JH
44
45Unicode also defines various I<properties> for the characters, like
376d9008 46"uppercase" or "lowercase", "decimal digit", or "punctuation";
ba62762e
JH
47these properties are independent of the names of the characters.
48Furthermore, various operations on the characters like uppercasing,
376d9008 49lowercasing, and collating (sorting) are defined.
ba62762e 50
0111a78f
KW
51A Unicode I<logical> "character" can actually consist of more than one internal
52I<actual> "character" or code point. For Western languages, this is adequately
c670e63a 53modelled by a I<base character> (like C<LATIN CAPITAL LETTER A>) followed
0111a78f 54by one or more I<modifiers> (like C<COMBINING ACUTE ACCENT>). This sequence of
376d9008 55base character and modifiers is called a I<combining character
0111a78f 56sequence>. Some non-western languages require more complicated
c8695642
KW
57models, so Unicode created the I<grapheme cluster> concept, which was
58later further refined into the I<extended grapheme cluster>. For
59example, a Korean Hangul syllable is considered a single logical
60character, but most often consists of three actual
c670e63a
KW
61Unicode characters: a leading consonant followed by an interior vowel followed
62by a trailing consonant.
0111a78f
KW
63
64Whether to call these extended grapheme clusters "characters" depends on your
65point of view. If you are a programmer, you probably would tend towards seeing
c8695642
KW
66each element in the sequences as one unit, or "character". However from
67the user's point of view, the whole sequence could be seen as one
68"character" since that's probably what it looks like in the context of the
69user's language. In this document, we take the programmer's point of
70view: one "character" is one Unicode code point.
71
72For some combinations of base character and modifiers, there are
73I<precomposed> characters. There is a single character equivalent, for
74example, to the sequence C<LATIN CAPITAL LETTER A> followed by
75C<COMBINING ACUTE ACCENT>. It is called C<LATIN CAPITAL LETTER A WITH
76ACUTE>. These precomposed characters are, however, only available for
77some combinations, and are mainly meant to support round-trip
78conversions between Unicode and legacy standards (like ISO 8859). Using
79sequences, as Unicode does, allows for needing fewer basic building blocks
80(code points) to express many more potential grapheme clusters. To
81support conversion between equivalent forms, various I<normalization
82forms> are also defined. Thus, C<LATIN CAPITAL LETTER A WITH ACUTE> is
83in I<Normalization Form Composed>, (abbreviated NFC), and the sequence
84C<LATIN CAPITAL LETTER A> followed by C<COMBINING ACUTE ACCENT>
85represents the same character in I<Normalization Form Decomposed> (NFD).
ba62762e
JH
86
87Because of backward compatibility with legacy encodings, the "a unique
376d9008
JB
88number for every character" idea breaks down a bit: instead, there is
89"at least one number for every character". The same character could
90be represented differently in several legacy encodings. The
35146e33 91converse is not also true: some code points do not have an assigned
376d9008
JB
92character. Firstly, there are unallocated code points within
93otherwise used blocks. Secondly, there are special Unicode control
94characters that do not represent true characters.
ba62762e 95
e1b711da 96A common myth about Unicode is that it is "16-bit", that is,
376d9008 97Unicode is only represented as C<0x10000> (or 65536) characters from
4c496f0c
JH
98C<0x0000> to C<0xFFFF>. B<This is untrue.> Since Unicode 2.0 (July
991996), Unicode has been defined all the way up to 21 bits (C<0x10FFFF>),
100and since Unicode 3.1 (March 2001), characters have been defined
101beyond C<0xFFFF>. The first C<0x10000> characters are called the
102I<Plane 0>, or the I<Basic Multilingual Plane> (BMP). With Unicode
1033.1, 17 (yes, seventeen) planes in all were defined--but they are
104nowhere near full of defined characters, yet.
ba62762e 105
e1b711da 106Another myth is about Unicode blocks--that they have something to
1bfb14c4
JH
107do with languages--that each block would define the characters used
108by a language or a set of languages. B<This is also untrue.>
109The division into blocks exists, but it is almost completely
110accidental--an artifact of how the characters have been and
e1b711da
KW
111still are allocated. Instead, there is a concept called I<scripts>, which is
112more useful: there is C<Latin> script, C<Greek> script, and so on. Scripts
113usually span varied parts of several blocks. For more information about
114scripts, see L<perlunicode/Scripts>.
ba62762e
JH
115
116The Unicode code points are just abstract numbers. To input and
4c496f0c
JH
117output these abstract numbers, the numbers must be I<encoded> or
118I<serialised> somehow. Unicode defines several I<character encoding
119forms>, of which I<UTF-8> is perhaps the most popular. UTF-8 is a
120variable length encoding that encodes Unicode characters as 1 to 6
e1b711da 121bytes. Other encodings
4c496f0c
JH
122include UTF-16 and UTF-32 and their big- and little-endian variants
123(UTF-8 is byte-order independent) The ISO/IEC 10646 defines the UCS-2
124and UCS-4 encoding forms.
ba62762e 125
376d9008
JB
126For more information about encodings--for instance, to learn what
127I<surrogates> and I<byte order marks> (BOMs) are--see L<perlunicode>.
ba62762e
JH
128
129=head2 Perl's Unicode Support
130
376d9008
JB
131Starting from Perl 5.6.0, Perl has had the capacity to handle Unicode
132natively. Perl 5.8.0, however, is the first recommended release for
133serious Unicode work. The maintenance release 5.6.1 fixed many of the
134problems of the initial Unicode implementation, but for example
1bfb14c4 135regular expressions still do not work with Unicode in 5.6.1.
c8695642
KW
136Perl 5.14.0 is the first release where Unicode support is
137(almost) seamlessly integrable without some gotchas (the exception being
138some differences in L<quotemeta|perlfunc/quotemeta>). To enable this
139seamless support, you should C<use feature 'unicode_strings'> (which is
140automatically selected if you C<use 5.012> or higher). See L<feature>.
141(5.14 also fixes a number of bugs and departures from the Unicode
142standard.)
143
144Before Perl 5.8.0, the use of C<use utf8> was used to declare
ba62762e 145that operations in the current block or file would be Unicode-aware.
376d9008 146This model was found to be wrong, or at least clumsy: the "Unicodeness"
1bfb14c4 147is now carried with the data, instead of being attached to the
c8695642
KW
148operations.
149Starting with Perl 5.8.0, only one case remains where an explicit C<use
150utf8> is needed: if your Perl script itself is encoded in UTF-8, you can
151use UTF-8 in your identifier names, and in string and regular expression
376d9008 152literals, by saying C<use utf8>. This is not the default because
8f8cf39c 153scripts with legacy 8-bit data in them would break. See L<utf8>.
ba62762e
JH
154
155=head2 Perl's Unicode Model
156
376d9008 157Perl supports both pre-5.6 strings of eight-bit native bytes, and
c8695642
KW
158strings of Unicode characters. The general principle is that Perl tries
159to keep its data as eight-bit bytes for as long as possible, but as soon
160as Unicodeness cannot be avoided, the data is transparently upgraded
161to Unicode. Prior to Perl 5.14, the upgrade was not completely
162transparent (see L<perlunicode/The "Unicode Bug">), and for backwards
163compatibility, full transparency is not gained unless C<use feature
164'unicode_strings'> (see L<feature>) or C<use 5.012> (or higher) is
165selected.
ba62762e 166
4192de81 167Internally, Perl currently uses either whatever the native eight-bit
376d9008
JB
168character set of the platform (for example Latin-1) is, defaulting to
169UTF-8, to encode Unicode strings. Specifically, if all code points in
170the string are C<0xFF> or less, Perl uses the native eight-bit
171character set. Otherwise, it uses UTF-8.
4192de81 172
7ca610e8 173A user of Perl does not normally need to know nor care how Perl
20ba30f4 174happens to encode its internal strings, but it becomes relevant when
ac036724 175outputting Unicode strings to a stream without a PerlIO layer (one with
176the "default" encoding). In such a case, the raw bytes used internally
7ca610e8
JH
177(the native character set or UTF-8, as appropriate for each string)
178will be used, and a "Wide character" warning will be issued if those
179strings contain a character beyond 0x00FF.
4192de81
JH
180
181For example,
182
ae5648b3 183 perl -e 'print "\x{DF}\n", "\x{0100}\x{DF}\n"'
4192de81
JH
184
185produces a fairly useless mixture of native bytes and UTF-8, as well
1bfb14c4 186as a warning:
4192de81 187
1bfb14c4
JH
188 Wide character in print at ...
189
740d4bb2 190To output UTF-8, use the C<:encoding> or C<:utf8> output layer. Prepending
4192de81
JH
191
192 binmode(STDOUT, ":utf8");
193
376d9008 194to this sample program ensures that the output is completely UTF-8,
1bfb14c4 195and removes the program's warning.
ba62762e 196
8aa8f774
JH
197You can enable automatic UTF-8-ification of your standard file
198handles, default C<open()> layer, and C<@ARGV> by using either
199the C<-C> command line switch or the C<PERL_UNICODE> environment
200variable, see L<perlrun> for the documentation of the C<-C> switch.
201
c8695642
KW
202Note that this means that Perl expects other software to work the same
203way:
8aa8f774 204if Perl has been led to believe that STDIN should be UTF-8, but then
c8695642
KW
205STDIN coming in from another command is not UTF-8, Perl will likely
206complain about the malformed UTF-8.
b310b053 207
ec71e770
JH
208All features that combine Unicode and I/O also require using the new
209PerlIO feature. Almost all Perl 5.8 platforms do use PerlIO, though:
210you can see whether yours is by running "perl -V" and looking for
211C<useperlio=define>.
212
64c66fb6
JH
213=head2 Unicode and EBCDIC
214
215Perl 5.8.0 also supports Unicode on EBCDIC platforms. There,
376d9008 216Unicode support is somewhat more complex to implement since
e1b711da
KW
217additional conversions are needed at every step.
218
219Later Perl releases have added code that will not work on EBCDIC platforms, and
220no one has complained, so the divergence has continued. If you want to run
221Perl on an EBCDIC platform, send email to perlbug@perl.org
64c66fb6 222
64c66fb6 223On EBCDIC platforms, the internal Unicode encoding form is UTF-EBCDIC
376d9008
JB
224instead of UTF-8. The difference is that as UTF-8 is "ASCII-safe" in
225that ASCII characters encode to UTF-8 as-is, while UTF-EBCDIC is
226"EBCDIC-safe".
ba62762e
JH
227
228=head2 Creating Unicode
229
376d9008
JB
230To create Unicode characters in literals for code points above C<0xFF>,
231use the C<\x{...}> notation in double-quoted strings:
ba62762e
JH
232
233 my $smiley = "\x{263a}";
234
376d9008 235Similarly, it can be used in regular expression literals
ba62762e
JH
236
237 $smiley =~ /\x{263a}/;
238
239At run-time you can use C<chr()>:
240
241 my $hebrew_alef = chr(0x05d0);
242
376d9008 243See L</"Further Resources"> for how to find all these numeric codes.
ba62762e 244
376d9008
JB
245Naturally, C<ord()> will do the reverse: it turns a character into
246a code point.
ba62762e 247
1bfb14c4
JH
248Note that C<\x..> (no C<{}> and only two hexadecimal digits), C<\x{...}>,
249and C<chr(...)> for arguments less than C<0x100> (decimal 256)
250generate an eight-bit character for backward compatibility with older
251Perls. For arguments of C<0x100> or more, Unicode characters are
a5f0baef
JH
252always produced. If you want to force the production of Unicode
253characters regardless of the numeric value, use C<pack("U", ...)>
254instead of C<\x..>, C<\x{...}>, or C<chr()>.
ba62762e
JH
255
256You can also use the C<charnames> pragma to invoke characters
376d9008 257by name in double-quoted strings:
ba62762e
JH
258
259 use charnames ':full';
260 my $arabic_alef = "\N{ARABIC LETTER ALEF}";
261
262And, as mentioned above, you can also C<pack()> numbers into Unicode
263characters:
264
265 my $georgian_an = pack("U", 0x10a0);
266
8a5e5dd5
JH
267Note that both C<\x{...}> and C<\N{...}> are compile-time string
268constants: you cannot use variables in them. if you want similar
fb121860 269run-time functionality, use C<chr()> and C<charnames::string_vianame()>.
8a5e5dd5 270
1eda90df 271If you want to force the result to Unicode characters, use the special
f337b084
TH
272C<"U0"> prefix. It consumes no arguments but causes the following bytes
273to be interpreted as the UTF-8 encoding of Unicode characters:
1eda90df 274
f337b084 275 my $chars = pack("U0W*", 0x80, 0x42);
771cd3b2 276
ae5648b3 277Likewise, you can stop such UTF-8 interpretation by using the special
771cd3b2 278C<"C0"> prefix.
1eda90df 279
ba62762e
JH
280=head2 Handling Unicode
281
282Handling Unicode is for the most part transparent: just use the
283strings as usual. Functions like C<index()>, C<length()>, and
284C<substr()> will work on the Unicode characters; regular expressions
285will work on the Unicode characters (see L<perlunicode> and L<perlretut>).
286
0111a78f
KW
287Note that Perl considers grapheme clusters to be separate characters, so for
288example
ba62762e
JH
289
290 use charnames ':full';
291 print length("\N{LATIN CAPITAL LETTER A}\N{COMBINING ACUTE ACCENT}"), "\n";
292
293will print 2, not 1. The only exception is that regular expressions
c8695642
KW
294have C<\X> for matching an extended grapheme cluster. (Thus C<\X> in a
295regular expression would match the entire sequence of both the example
296characters.)
ba62762e 297
376d9008
JB
298Life is not quite so transparent, however, when working with legacy
299encodings, I/O, and certain special cases:
ba62762e
JH
300
301=head2 Legacy Encodings
302
c8695642
KW
303When you combine legacy data and Unicode, the legacy data needs
304to be upgraded to Unicode. Normally the legacy data is assumed to be
305ISO 8859-1 (or EBCDIC, if applicable).
ba62762e 306
376d9008 307The C<Encode> module knows about many encodings and has interfaces
ba62762e
JH
308for doing conversions between those encodings:
309
2c9359a2
RGS
310 use Encode 'decode';
311 $data = decode("iso-8859-3", $data); # convert from legacy to utf-8
ba62762e
JH
312
313=head2 Unicode I/O
314
8baee566 315Normally, writing out Unicode data
ba62762e 316
8baee566 317 print FH $some_string_with_unicode, "\n";
ba62762e 318
8baee566 319produces raw bytes that Perl happens to use to internally encode the
376d9008
JB
320Unicode string. Perl's internal encoding depends on the system as
321well as what characters happen to be in the string at the time. If
322any of the characters are at code points C<0x100> or above, you will get
323a warning. To ensure that the output is explicitly rendered in the
324encoding you desire--and to avoid the warning--open the stream with
325the desired encoding. Some examples:
ba62762e 326
ec90690f
ST
327 open FH, ">:utf8", "file";
328
329 open FH, ">:encoding(ucs2)", "file";
330 open FH, ">:encoding(UTF-8)", "file";
331 open FH, ">:encoding(shift_jis)", "file";
1d7919c5 332
376d9008 333and on already open streams, use C<binmode()>:
1d7919c5
JH
334
335 binmode(STDOUT, ":utf8");
336
ec90690f
ST
337 binmode(STDOUT, ":encoding(ucs2)");
338 binmode(STDOUT, ":encoding(UTF-8)");
339 binmode(STDOUT, ":encoding(shift_jis)");
340
b5d8778e 341The matching of encoding names is loose: case does not matter, and
fae2c0fb 342many encodings have several aliases. Note that the C<:utf8> layer
1bfb14c4 343must always be specified exactly like that; it is I<not> subject to
c8695642 344the loose matching of encoding names. Also note that currently C<:utf8> is unsafe for
740d4bb2 345input, because it accepts the data without validating that it is indeed valid
c8695642
KW
346UTF8; you should instead use C<:encoding(UTF-8)> (unfortunately this
347specification needs to be in all upper-case with the dash to get the
348safety checking; C<:encoding(utf-8)>, for example, doesn't do the
349checking).
b5d8778e 350
376d9008
JB
351See L<PerlIO> for the C<:utf8> layer, L<PerlIO::encoding> and
352L<Encode::PerlIO> for the C<:encoding()> layer, and
353L<Encode::Supported> for many encodings supported by the C<Encode>
354module.
ba62762e 355
a5f0baef 356Reading in a file that you know happens to be encoded in one of the
1bfb14c4
JH
357Unicode or legacy encodings does not magically turn the data into
358Unicode in Perl's eyes. To do that, specify the appropriate
fae2c0fb 359layer when opening files
ba62762e 360
740d4bb2 361 open(my $fh,'<:encoding(utf8)', 'anything');
8baee566
JF
362 my $line_of_unicode = <$fh>;
363
ec90690f 364 open(my $fh,'<:encoding(Big5)', 'anything');
8baee566 365 my $line_of_unicode = <$fh>;
ba62762e 366
fae2c0fb 367The I/O layers can also be specified more flexibly with
376d9008 368the C<open> pragma. See L<open>, or look at the following example.
ba62762e 369
9e5bbba0
KW
370 use open ':encoding(utf8)'; # input/output default encoding will be
371 # UTF-8
1d7919c5
JH
372 open X, ">file";
373 print X chr(0x100), "\n";
ba62762e 374 close X;
1d7919c5 375 open Y, "<file";
ba62762e
JH
376 printf "%#x\n", ord(<Y>); # this should print 0x100
377 close Y;
378
fae2c0fb 379With the C<open> pragma you can use the C<:locale> layer
ba62762e 380
12f98225 381 BEGIN { $ENV{LC_ALL} = $ENV{LANG} = 'ru_RU.KOI8-R' }
9e5bbba0
KW
382 # the :locale will probe the locale environment variables like
383 # LC_ALL
ba62762e
JH
384 use open OUT => ':locale'; # russki parusski
385 open(O, ">koi8");
386 print O chr(0x430); # Unicode CYRILLIC SMALL LETTER A = KOI8-R 0xc1
387 close O;
388 open(I, "<koi8");
389 printf "%#x\n", ord(<I>), "\n"; # this should print 0xc1
390 close I;
391
8baee566
JF
392These methods install a transparent filter on the I/O stream that
393converts data from the specified encoding when it is read in from the
a5f0baef 394stream. The result is always Unicode.
ba62762e
JH
395
396The L<open> pragma affects all the C<open()> calls after the pragma by
fae2c0fb
RGS
397setting default layers. If you want to affect only certain
398streams, use explicit layers directly in the C<open()> call.
ba62762e
JH
399
400You can switch encodings on an already opened stream by using
8baee566 401C<binmode()>; see L<perlfunc/binmode>.
ba62762e 402
1ecefa54
JH
403The C<:locale> does not currently (as of Perl 5.8.0) work with
404C<open()> and C<binmode()>, only with the C<open> pragma. The
8baee566 405C<:utf8> and C<:encoding(...)> methods do work with all of C<open()>,
1ecefa54 406C<binmode()>, and the C<open> pragma.
ba62762e 407
fae2c0fb 408Similarly, you may use these I/O layers on output streams to
a5f0baef
JH
409automatically convert Unicode to the specified encoding when it is
410written to the stream. For example, the following snippet copies the
411contents of the file "text.jis" (encoded as ISO-2022-JP, aka JIS) to
412the file "text.utf8", encoded as UTF-8:
ba62762e 413
c722ef0a
JH
414 open(my $nihongo, '<:encoding(iso-2022-jp)', 'text.jis');
415 open(my $unicode, '>:utf8', 'text.utf8');
0cf8a8d9 416 while (<$nihongo>) { print $unicode $_ }
ba62762e
JH
417
418The naming of encodings, both by the C<open()> and by the C<open>
2575c402
JW
419pragma allows for flexible names: C<koi8-r> and C<KOI8R> will both be
420understood.
ba62762e
JH
421
422Common encodings recognized by ISO, MIME, IANA, and various other
8baee566 423standardisation organisations are recognised; for a more detailed
1bfb14c4 424list see L<Encode::Supported>.
ba62762e
JH
425
426C<read()> reads characters and returns the number of characters.
427C<seek()> and C<tell()> operate on byte counts, as do C<sysread()>
428and C<sysseek()>.
429
8baee566 430Notice that because of the default behaviour of not doing any
fae2c0fb 431conversion upon input if there is no default layer,
ba62762e 432it is easy to mistakenly write code that keeps on expanding a file
1bfb14c4 433by repeatedly encoding the data:
ba62762e
JH
434
435 # BAD CODE WARNING
436 open F, "file";
8baee566 437 local $/; ## read in the whole file of 8-bit characters
ba62762e
JH
438 $t = <F>;
439 close F;
740d4bb2 440 open F, ">:encoding(utf8)", "file";
8baee566 441 print F $t; ## convert to UTF-8 on output
ba62762e
JH
442 close F;
443
444If you run this code twice, the contents of the F<file> will be twice
740d4bb2
JW
445UTF-8 encoded. A C<use open ':encoding(utf8)'> would have avoided the
446bug, or explicitly opening also the F<file> for input as UTF-8.
ba62762e 447
0c901d84 448B<NOTE>: the C<:utf8> and C<:encoding> features work only if your
d0fadae5
JH
449Perl has been built with the new PerlIO feature (which is the default
450on most systems).
0c901d84 451
1ecefa54
JH
452=head2 Displaying Unicode As Text
453
454Sometimes you might want to display Perl scalars containing Unicode as
8baee566 455simple ASCII (or EBCDIC) text. The following subroutine converts
1ecefa54 456its argument so that Unicode characters with code points greater than
1bfb14c4
JH
457255 are displayed as C<\x{...}>, control characters (like C<\n>) are
458displayed as C<\x..>, and the rest of the characters as themselves:
1ecefa54 459
9e5bbba0
KW
460 sub nice_string {
461 join("",
462 map { $_ > 255 ? # if wide character...
463 sprintf("\\x{%04X}", $_) : # \x{...}
464 chr($_) =~ /[[:cntrl:]]/ ? # else if control character ...
465 sprintf("\\x%02X", $_) : # \x..
466 quotemeta(chr($_)) # else quoted or as themselves
f337b084 467 } unpack("W*", $_[0])); # unpack Unicode characters
58c274a1
JF
468 }
469
470For example,
471
472 nice_string("foo\x{100}bar\n")
473
d0551e73 474returns the string
58c274a1 475
d0551e73
JH
476 'foo\x{0100}bar\x0A'
477
478which is ready to be printed.
1ecefa54 479
ba62762e
JH
480=head2 Special Cases
481
482=over 4
483
484=item *
485
486Bit Complement Operator ~ And vec()
487
1bfb14c4
JH
488The bit complement operator C<~> may produce surprising results if
489used on strings containing characters with ordinal values above
490255. In such a case, the results are consistent with the internal
491encoding of the characters, but not with much else. So don't do
492that. Similarly for C<vec()>: you will be operating on the
493internally-encoded bit patterns of the Unicode characters, not on
494the code point values, which is very probably not what you want.
ba62762e
JH
495
496=item *
497
8baee566
JF
498Peeking At Perl's Internal Encoding
499
500Normal users of Perl should never care how Perl encodes any particular
a5f0baef 501Unicode string (because the normal ways to get at the contents of a
376d9008 502string with Unicode--via input and output--should always be via
fae2c0fb 503explicitly-defined I/O layers). But if you must, there are two
a5f0baef 504ways of looking behind the scenes.
ba62762e
JH
505
506One way of peeking inside the internal encoding of Unicode characters
f337b084
TH
507is to use C<unpack("C*", ...> to get the bytes of whatever the string
508encoding happens to be, or C<unpack("U0..", ...)> to get the bytes of the
509UTF-8 encoding:
ba62762e 510
8baee566 511 # this prints c4 80 for the UTF-8 bytes 0xc4 0x80
f337b084 512 print join(" ", unpack("U0(H2)*", pack("U", 0x100))), "\n";
ba62762e
JH
513
514Yet another way would be to use the Devel::Peek module:
515
516 perl -MDevel::Peek -e 'Dump(chr(0x100))'
517
1e54db1a 518That shows the C<UTF8> flag in FLAGS and both the UTF-8 bytes
376d9008 519and Unicode characters in C<PV>. See also later in this document
8800c35a 520the discussion about the C<utf8::is_utf8()> function.
ba62762e
JH
521
522=back
523
524=head2 Advanced Topics
525
526=over 4
527
528=item *
529
530String Equivalence
531
532The question of string equivalence turns somewhat complicated
376d9008 533in Unicode: what do you mean by "equal"?
ba62762e 534
07698885
RGS
535(Is C<LATIN CAPITAL LETTER A WITH ACUTE> equal to
536C<LATIN CAPITAL LETTER A>?)
ba62762e 537
a5f0baef
JH
538The short answer is that by default Perl compares equivalence (C<eq>,
539C<ne>) based only on code points of the characters. In the above
376d9008 540case, the answer is no (because 0x00C1 != 0x0041). But sometimes, any
c8695642 541CAPITAL LETTER A's should be considered equal, or even A's of any case.
ba62762e
JH
542
543The long answer is that you need to consider character normalization
e1b711da
KW
544and casing issues: see L<Unicode::Normalize>, Unicode Technical Report #15,
545L<Unicode Normalization Forms|http://www.unicode.org/unicode/reports/tr15> and
546sections on case mapping in the L<Unicode Standard|http://www.unicode.org>.
ba62762e 547
1bfb14c4 548As of Perl 5.8.0, the "Full" case-folding of I<Case
e1b711da 549Mappings/SpecialCasing> is implemented, but bugs remain in C<qr//i> with them.
ba62762e
JH
550
551=item *
552
553String Collation
554
376d9008 555People like to see their strings nicely sorted--or as Unicode
ba62762e
JH
556parlance goes, collated. But again, what do you mean by collate?
557
07698885
RGS
558(Does C<LATIN CAPITAL LETTER A WITH ACUTE> come before or after
559C<LATIN CAPITAL LETTER A WITH GRAVE>?)
ba62762e 560
58c274a1 561The short answer is that by default, Perl compares strings (C<lt>,
ba62762e 562C<le>, C<cmp>, C<ge>, C<gt>) based only on the code points of the
1bfb14c4 563characters. In the above case, the answer is "after", since
da76a1f4 564C<0x00C1> > C<0x00C0>.
ba62762e
JH
565
566The long answer is that "it depends", and a good answer cannot be
567given without knowing (at the very least) the language context.
568See L<Unicode::Collate>, and I<Unicode Collation Algorithm>
2bbc8d55 569L<http://www.unicode.org/unicode/reports/tr10/>
ba62762e
JH
570
571=back
572
573=head2 Miscellaneous
574
575=over 4
576
577=item *
578
3ff56b75 579Character Ranges and Classes
ba62762e 580
0eb9ada1
KW
581Character ranges in regular expression bracketed character classes ( e.g.,
582C</[a-z]/>) and in the C<tr///> (also known as C<y///>) operator are not
583magically Unicode-aware. What this means is that C<[A-Za-z]> will not
584magically start to mean "all alphabetic letters" (not that it does mean that
585even for 8-bit characters; for those, if you are using locales (L<perllocale>),
586use C</[[:alpha:]]/>; and if not, use the 8-bit-aware property C<\p{alpha}>).
587
588All the properties that begin with C<\p> (and its inverse C<\P>) are actually
589character classes that are Unicode-aware. There are dozens of them, see
590L<perluniprops>.
591
592You can use Unicode code points as the end points of character ranges, and the
593range will include all Unicode code points that lie between those end points.
ba62762e
JH
594
595=item *
596
597String-To-Number Conversions
598
376d9008
JB
599Unicode does define several other decimal--and numeric--characters
600besides the familiar 0 to 9, such as the Arabic and Indic digits.
ba62762e 601Perl does not support string-to-number conversion for digits other
58c274a1 602than ASCII 0 to 9 (and ASCII a to f for hexadecimal).
c8695642
KW
603To get safe conversions from any Unicode string, use
604L<Unicode::UCD/num()>.
ba62762e
JH
605
606=back
607
608=head2 Questions With Answers
609
610=over 4
611
818c4caa 612=item *
5cb3728c
RB
613
614Will My Old Scripts Break?
ba62762e
JH
615
616Very probably not. Unless you are generating Unicode characters
1bfb14c4
JH
617somehow, old behaviour should be preserved. About the only behaviour
618that has changed and which could start generating Unicode is the old
619behaviour of C<chr()> where supplying an argument more than 255
620produced a character modulo 255. C<chr(300)>, for example, was equal
621to C<chr(45)> or "-" (in ASCII), now it is LATIN CAPITAL LETTER I WITH
622BREVE.
ba62762e 623
818c4caa 624=item *
5cb3728c
RB
625
626How Do I Make My Scripts Work With Unicode?
ba62762e
JH
627
628Very little work should be needed since nothing changes until you
1bfb14c4
JH
629generate Unicode data. The most important thing is getting input as
630Unicode; for that, see the earlier I/O discussion.
c8695642
KW
631To get full seamless Unicode support, add
632C<use feature 'unicode_strings'> (or C<use 5.012> or higher) to your
633script.
ba62762e 634
818c4caa 635=item *
5cb3728c
RB
636
637How Do I Know Whether My String Is In Unicode?
ba62762e 638
c8695642
KW
639You shouldn't have to care. But you may if your Perl is before 5.14.0
640or you haven't specified C<use feature 'unicode_strings'> or C<use
6415.012> (or higher) because otherwise the semantics of the code points
642in the range 128 to 255 are different depending on
2bbc8d55 643whether the string they are contained within is in Unicode or not.
e1b711da 644(See L<perlunicode/When Unicode Does Not Happen>.)
ba62762e 645
2bbc8d55 646To determine if a string is in Unicode, use:
ba62762e 647
8800c35a 648 print utf8::is_utf8($string) ? 1 : 0, "\n";
ba62762e
JH
649
650But note that this doesn't mean that any of the characters in the
651string are necessary UTF-8 encoded, or that any of the characters have
652code points greater than 0xFF (255) or even 0x80 (128), or that the
653string has any characters at all. All the C<is_utf8()> does is to
654return the value of the internal "utf8ness" flag attached to the
376d9008 655C<$string>. If the flag is off, the bytes in the scalar are interpreted
3c1c8017 656as a single byte encoding. If the flag is on, the bytes in the scalar
0eb9ada1
KW
657are interpreted as the (variable-length, potentially multi-byte) UTF-8 encoded
658code points of the characters. Bytes added to a UTF-8 encoded string are
1e54db1a 659automatically upgraded to UTF-8. If mixed non-UTF-8 and UTF-8 scalars
c8695642 660are merged (double-quoted interpolation, explicit concatenation, or
3c1c8017
AT
661printf/sprintf parameter substitution), the result will be UTF-8 encoded
662as if copies of the byte strings were upgraded to UTF-8: for example,
663
664 $a = "ab\x80c";
665 $b = "\x{100}";
666 print "$a = $b\n";
667
a02b5feb
JH
668the output string will be UTF-8-encoded C<ab\x80c = \x{100}\n>, but
669C<$a> will stay byte-encoded.
ba62762e
JH
670
671Sometimes you might really need to know the byte length of a string
ce7675db 672instead of the character length. For that use either the
c8695642
KW
673C<Encode::encode_utf8()> function or the (deprecated) C<bytes> pragma
674and the C<length()> function:
ba62762e
JH
675
676 my $unicode = chr(0x100);
677 print length($unicode), "\n"; # will print 1
ce7675db
A
678 require Encode;
679 print length(Encode::encode_utf8($unicode)), "\n"; # will print 2
c8695642 680 use bytes; # DEPRECATED!
1bfb14c4
JH
681 print length($unicode), "\n"; # will also print 2
682 # (the 0xC4 0x80 of the UTF-8)
0eb9ada1 683 no bytes;
ba62762e 684
818c4caa 685=item *
5cb3728c 686
c8695642
KW
687How Do I Find Out What Encoding a File Has?
688
968ee499 689You might try L<Encode::Guess>, but it has a number of limitations.
c8695642
KW
690
691=item *
692
5cb3728c 693How Do I Detect Data That's Not Valid In a Particular Encoding?
ba62762e 694
8baee566
JF
695Use the C<Encode> package to try converting it.
696For example,
ba62762e 697
bb2f379c 698 use Encode 'decode_utf8';
2bbc8d55 699
228ee848 700 if (eval { decode_utf8($string, Encode::FB_CROAK); 1 }) {
a365f2ce 701 # $string is valid utf8
ba62762e 702 } else {
a365f2ce 703 # $string is not valid utf8
ba62762e
JH
704 }
705
f337b084 706Or use C<unpack> to try decoding it:
ba62762e
JH
707
708 use warnings;
f337b084 709 @chars = unpack("C0U*", $string_of_bytes_that_I_think_is_utf8);
ba62762e 710
ae5648b3
RGS
711If invalid, a C<Malformed UTF-8 character> warning is produced. The "C0" means
712"process the string character per character". Without that, the
713C<unpack("U*", ...)> would work in C<U0> mode (the default if the format
714string starts with C<U>) and it would return the bytes making up the UTF-8
f337b084 715encoding of the target string, something that will always work.
ba62762e 716
818c4caa 717=item *
5cb3728c
RB
718
719How Do I Convert Binary Data Into a Particular Encoding, Or Vice Versa?
ba62762e 720
8baee566
JF
721This probably isn't as useful as you might think.
722Normally, you shouldn't need to.
ba62762e 723
1bfb14c4 724In one sense, what you are asking doesn't make much sense: encodings
376d9008 725are for characters, and binary data are not "characters", so converting
a5f0baef
JH
726"data" into some encoding isn't meaningful unless you know in what
727character set and encoding the binary data is in, in which case it's
376d9008 728not just binary data, now is it?
8baee566 729
1bfb14c4
JH
730If you have a raw sequence of bytes that you know should be
731interpreted via a particular encoding, you can use C<Encode>:
ba62762e
JH
732
733 use Encode 'from_to';
734 from_to($data, "iso-8859-1", "utf-8"); # from latin-1 to utf-8
735
1bfb14c4
JH
736The call to C<from_to()> changes the bytes in C<$data>, but nothing
737material about the nature of the string has changed as far as Perl is
738concerned. Both before and after the call, the string C<$data>
739contains just a bunch of 8-bit bytes. As far as Perl is concerned,
740the encoding of the string remains as "system-native 8-bit bytes".
8baee566
JF
741
742You might relate this to a fictional 'Translate' module:
743
744 use Translate;
745 my $phrase = "Yes";
746 Translate::from_to($phrase, 'english', 'deutsch');
747 ## phrase now contains "Ja"
ba62762e 748
8baee566 749The contents of the string changes, but not the nature of the string.
1bfb14c4
JH
750Perl doesn't know any more after the call than before that the
751contents of the string indicates the affirmative.
ba62762e 752
376d9008 753Back to converting data. If you have (or want) data in your system's
a5f0baef
JH
754native 8-bit encoding (e.g. Latin-1, EBCDIC, etc.), you can use
755pack/unpack to convert to/from Unicode.
ba62762e 756
f337b084
TH
757 $native_string = pack("W*", unpack("U*", $Unicode_string));
758 $Unicode_string = pack("U*", unpack("W*", $native_string));
ba62762e
JH
759
760If you have a sequence of bytes you B<know> is valid UTF-8,
761but Perl doesn't know it yet, you can make Perl a believer, too:
762
763 use Encode 'decode_utf8';
8baee566 764 $Unicode = decode_utf8($bytes);
ba62762e 765
f337b084
TH
766or:
767
768 $Unicode = pack("U0a*", $bytes);
ae5648b3 769
2bbc8d55
SP
770You can find the bytes that make up a UTF-8 sequence with
771
9e5bbba0 772 @bytes = unpack("C*", $Unicode_string)
2bbc8d55
SP
773
774and you can create well-formed Unicode with
775
9e5bbba0 776 $Unicode_string = pack("U*", 0xff, ...)
ba62762e 777
818c4caa 778=item *
5cb3728c
RB
779
780How Do I Display Unicode? How Do I Input Unicode?
ba62762e 781
2bbc8d55
SP
782See L<http://www.alanwood.net/unicode/> and
783L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e 784
818c4caa 785=item *
5cb3728c
RB
786
787How Does Unicode Work With Traditional Locales?
ba62762e
JH
788
789In Perl, not very well. Avoid using locales through the C<locale>
4c496f0c
JH
790pragma. Use only one or the other. But see L<perlrun> for the
791description of the C<-C> switch and its environment counterpart,
792C<$ENV{PERL_UNICODE}> to see how to enable various Unicode features,
793for example by using locale settings.
ba62762e
JH
794
795=back
796
797=head2 Hexadecimal Notation
798
376d9008
JB
799The Unicode standard prefers using hexadecimal notation because
800that more clearly shows the division of Unicode into blocks of 256 characters.
ba62762e
JH
801Hexadecimal is also simply shorter than decimal. You can use decimal
802notation, too, but learning to use hexadecimal just makes life easier
1bfb14c4 803with the Unicode standard. The C<U+HHHH> notation uses hexadecimal,
076d825e 804for example.
ba62762e
JH
805
806The C<0x> prefix means a hexadecimal number, the digits are 0-9 I<and>
807a-f (or A-F, case doesn't matter). Each hexadecimal digit represents
808four bits, or half a byte. C<print 0x..., "\n"> will show a
809hexadecimal number in decimal, and C<printf "%x\n", $decimal> will
810show a decimal number in hexadecimal. If you have just the
376d9008 811"hex digits" of a hexadecimal number, you can use the C<hex()> function.
ba62762e
JH
812
813 print 0x0009, "\n"; # 9
814 print 0x000a, "\n"; # 10
815 print 0x000f, "\n"; # 15
816 print 0x0010, "\n"; # 16
817 print 0x0011, "\n"; # 17
818 print 0x0100, "\n"; # 256
819
820 print 0x0041, "\n"; # 65
821
822 printf "%x\n", 65; # 41
823 printf "%#x\n", 65; # 0x41
824
825 print hex("41"), "\n"; # 65
826
827=head2 Further Resources
828
829=over 4
830
831=item *
832
833Unicode Consortium
834
2bbc8d55 835L<http://www.unicode.org/>
ba62762e
JH
836
837=item *
838
839Unicode FAQ
840
2bbc8d55 841L<http://www.unicode.org/unicode/faq/>
ba62762e
JH
842
843=item *
844
845Unicode Glossary
846
2bbc8d55 847L<http://www.unicode.org/glossary/>
ba62762e
JH
848
849=item *
850
c8695642
KW
851Unicode Recommended Reading List
852
853The Unicode Consortium has a list of articles and books, some of which
854give a much more in depth treatment of Unicode:
855L<http://unicode.org/resources/readinglist.html>
856
857=item *
858
ba62762e
JH
859Unicode Useful Resources
860
2bbc8d55 861L<http://www.unicode.org/unicode/onlinedat/resources.html>
ba62762e
JH
862
863=item *
864
865Unicode and Multilingual Support in HTML, Fonts, Web Browsers and Other Applications
866
2bbc8d55 867L<http://www.alanwood.net/unicode/>
ba62762e
JH
868
869=item *
870
871UTF-8 and Unicode FAQ for Unix/Linux
872
2bbc8d55 873L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e
JH
874
875=item *
876
877Legacy Character Sets
878
2bbc8d55
SP
879L<http://www.czyborra.com/>
880L<http://www.eki.ee/letter/>
ba62762e
JH
881
882=item *
883
ba62762e
JH
884You can explore various information from the Unicode data files using
885the C<Unicode::UCD> module.
886
887=back
888
f6edf83b
JH
889=head1 UNICODE IN OLDER PERLS
890
891If you cannot upgrade your Perl to 5.8.0 or later, you can still
892do some Unicode processing by using the modules C<Unicode::String>,
893C<Unicode::Map8>, and C<Unicode::Map>, available from CPAN.
894If you have the GNU recode installed, you can also use the
376d9008 895Perl front-end C<Convert::Recode> for character conversions.
f6edf83b 896
aaef10c5 897The following are fast conversions from ISO 8859-1 (Latin-1) bytes
63de3cb2 898to UTF-8 bytes and back, the code works even with older Perl 5 versions.
aaef10c5
JH
899
900 # ISO 8859-1 to UTF-8
901 s/([\x80-\xFF])/chr(0xC0|ord($1)>>6).chr(0x80|ord($1)&0x3F)/eg;
902
903 # UTF-8 to ISO 8859-1
904 s/([\xC2\xC3])([\x80-\xBF])/chr(ord($1)<<6&0xC0|ord($2)&0x3F)/eg;
905
ba62762e
JH
906=head1 SEE ALSO
907
2575c402 908L<perlunitut>, L<perlunicode>, L<Encode>, L<open>, L<utf8>, L<bytes>,
4c496f0c
JH
909L<perlretut>, L<perlrun>, L<Unicode::Collate>, L<Unicode::Normalize>,
910L<Unicode::UCD>
ba62762e 911
376d9008 912=head1 ACKNOWLEDGMENTS
ba62762e
JH
913
914Thanks to the kind readers of the perl5-porters@perl.org,
915perl-unicode@perl.org, linux-utf8@nl.linux.org, and unicore@unicode.org
916mailing lists for their valuable feedback.
917
918=head1 AUTHOR, COPYRIGHT, AND LICENSE
919
c8695642 920Copyright 2001-2011 Jarkko Hietaniemi E<lt>jhi@iki.fiE<gt>
ba62762e
JH
921
922This document may be distributed under the same terms as Perl itself.