This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Nit in perluniintro.pod
[perl5.git] / pod / perluniintro.pod
CommitLineData
ba62762e
JH
1=head1 NAME
2
07fcf8ff 3perluniintro - Perl Unicode introduction
ba62762e
JH
4
5=head1 DESCRIPTION
6
7This document gives a general idea of Unicode and how to use Unicode
8in Perl.
9
10=head2 Unicode
11
376d9008 12Unicode is a character set standard which plans to codify all of the
ba62762e
JH
13writing systems of the world, plus many other symbols.
14
15Unicode and ISO/IEC 10646 are coordinated standards that provide code
376d9008 16points for characters in almost all modern character set standards,
ba62762e 17covering more than 30 writing systems and hundreds of languages,
376d9008 18including all commercially-important modern languages. All characters
ba62762e
JH
19in the largest Chinese, Japanese, and Korean dictionaries are also
20encoded. The standards will eventually cover almost all characters in
21more than 250 writing systems and thousands of languages.
4c496f0c 22Unicode 1.0 was released in October 1991, and 4.0 in April 2003.
ba62762e
JH
23
24A Unicode I<character> is an abstract entity. It is not bound to any
376d9008
JB
25particular integer width, especially not to the C language C<char>.
26Unicode is language-neutral and display-neutral: it does not encode the
e1b711da 27language of the text, and it does not generally define fonts or other graphical
ba62762e
JH
28layout details. Unicode operates on characters and on text built from
29those characters.
30
31Unicode defines characters like C<LATIN CAPITAL LETTER A> or C<GREEK
376d9008
JB
32SMALL LETTER ALPHA> and unique numbers for the characters, in this
33case 0x0041 and 0x03B1, respectively. These unique numbers are called
34I<code points>.
ba62762e
JH
35
36The Unicode standard prefers using hexadecimal notation for the code
4c496f0c
JH
37points. If numbers like C<0x0041> are unfamiliar to you, take a peek
38at a later section, L</"Hexadecimal Notation">. The Unicode standard
39uses the notation C<U+0041 LATIN CAPITAL LETTER A>, to give the
40hexadecimal code point and the normative name of the character.
ba62762e
JH
41
42Unicode also defines various I<properties> for the characters, like
376d9008 43"uppercase" or "lowercase", "decimal digit", or "punctuation";
ba62762e
JH
44these properties are independent of the names of the characters.
45Furthermore, various operations on the characters like uppercasing,
376d9008 46lowercasing, and collating (sorting) are defined.
ba62762e 47
0111a78f
KW
48A Unicode I<logical> "character" can actually consist of more than one internal
49I<actual> "character" or code point. For Western languages, this is adequately
c670e63a 50modelled by a I<base character> (like C<LATIN CAPITAL LETTER A>) followed
0111a78f 51by one or more I<modifiers> (like C<COMBINING ACUTE ACCENT>). This sequence of
376d9008 52base character and modifiers is called a I<combining character
0111a78f 53sequence>. Some non-western languages require more complicated
c670e63a
KW
54models, so Unicode created the I<grapheme cluster> concept, and then the
55I<extended grapheme cluster>. For example, a Korean Hangul syllable is
0111a78f 56considered a single logical character, but most often consists of three actual
c670e63a
KW
57Unicode characters: a leading consonant followed by an interior vowel followed
58by a trailing consonant.
0111a78f
KW
59
60Whether to call these extended grapheme clusters "characters" depends on your
61point of view. If you are a programmer, you probably would tend towards seeing
62each element in the sequences as one unit, or "character". The whole sequence
63could be seen as one "character", however, from the user's point of view, since
64that's probably what it looks like in the context of the user's language.
376d9008
JB
65
66With this "whole sequence" view of characters, the total number of
67characters is open-ended. But in the programmer's "one unit is one
68character" point of view, the concept of "characters" is more
f7d1603d 69deterministic. In this document, we take that second point of view:
0111a78f 70one "character" is one Unicode code point.
376d9008
JB
71
72For some combinations, there are I<precomposed> characters.
73C<LATIN CAPITAL LETTER A WITH ACUTE>, for example, is defined as
ba62762e 74a single code point. These precomposed characters are, however,
376d9008 75only available for some combinations, and are mainly
ba62762e 76meant to support round-trip conversions between Unicode and legacy
376d9008
JB
77standards (like the ISO 8859). In the general case, the composing
78method is more extensible. To support conversion between
ba62762e 79different compositions of the characters, various I<normalization
376d9008 80forms> to standardize representations are also defined.
ba62762e
JH
81
82Because of backward compatibility with legacy encodings, the "a unique
376d9008
JB
83number for every character" idea breaks down a bit: instead, there is
84"at least one number for every character". The same character could
85be represented differently in several legacy encodings. The
35146e33 86converse is not also true: some code points do not have an assigned
376d9008
JB
87character. Firstly, there are unallocated code points within
88otherwise used blocks. Secondly, there are special Unicode control
89characters that do not represent true characters.
ba62762e 90
e1b711da 91A common myth about Unicode is that it is "16-bit", that is,
376d9008 92Unicode is only represented as C<0x10000> (or 65536) characters from
4c496f0c
JH
93C<0x0000> to C<0xFFFF>. B<This is untrue.> Since Unicode 2.0 (July
941996), Unicode has been defined all the way up to 21 bits (C<0x10FFFF>),
95and since Unicode 3.1 (March 2001), characters have been defined
96beyond C<0xFFFF>. The first C<0x10000> characters are called the
97I<Plane 0>, or the I<Basic Multilingual Plane> (BMP). With Unicode
983.1, 17 (yes, seventeen) planes in all were defined--but they are
99nowhere near full of defined characters, yet.
ba62762e 100
e1b711da 101Another myth is about Unicode blocks--that they have something to
1bfb14c4
JH
102do with languages--that each block would define the characters used
103by a language or a set of languages. B<This is also untrue.>
104The division into blocks exists, but it is almost completely
105accidental--an artifact of how the characters have been and
e1b711da
KW
106still are allocated. Instead, there is a concept called I<scripts>, which is
107more useful: there is C<Latin> script, C<Greek> script, and so on. Scripts
108usually span varied parts of several blocks. For more information about
109scripts, see L<perlunicode/Scripts>.
ba62762e
JH
110
111The Unicode code points are just abstract numbers. To input and
4c496f0c
JH
112output these abstract numbers, the numbers must be I<encoded> or
113I<serialised> somehow. Unicode defines several I<character encoding
114forms>, of which I<UTF-8> is perhaps the most popular. UTF-8 is a
115variable length encoding that encodes Unicode characters as 1 to 6
e1b711da 116bytes. Other encodings
4c496f0c
JH
117include UTF-16 and UTF-32 and their big- and little-endian variants
118(UTF-8 is byte-order independent) The ISO/IEC 10646 defines the UCS-2
119and UCS-4 encoding forms.
ba62762e 120
376d9008
JB
121For more information about encodings--for instance, to learn what
122I<surrogates> and I<byte order marks> (BOMs) are--see L<perlunicode>.
ba62762e
JH
123
124=head2 Perl's Unicode Support
125
376d9008
JB
126Starting from Perl 5.6.0, Perl has had the capacity to handle Unicode
127natively. Perl 5.8.0, however, is the first recommended release for
128serious Unicode work. The maintenance release 5.6.1 fixed many of the
129problems of the initial Unicode implementation, but for example
1bfb14c4 130regular expressions still do not work with Unicode in 5.6.1.
ba62762e 131
2bbc8d55 132B<Starting from Perl 5.8.0, the use of C<use utf8> is needed only in much more restricted circumstances.> In earlier releases the C<utf8> pragma was used to declare
ba62762e 133that operations in the current block or file would be Unicode-aware.
376d9008 134This model was found to be wrong, or at least clumsy: the "Unicodeness"
1bfb14c4 135is now carried with the data, instead of being attached to the
376d9008
JB
136operations. Only one case remains where an explicit C<use utf8> is
137needed: if your Perl script itself is encoded in UTF-8, you can use
138UTF-8 in your identifier names, and in string and regular expression
139literals, by saying C<use utf8>. This is not the default because
8f8cf39c 140scripts with legacy 8-bit data in them would break. See L<utf8>.
ba62762e
JH
141
142=head2 Perl's Unicode Model
143
376d9008
JB
144Perl supports both pre-5.6 strings of eight-bit native bytes, and
145strings of Unicode characters. The principle is that Perl tries to
146keep its data as eight-bit bytes for as long as possible, but as soon
e1b711da
KW
147as Unicodeness cannot be avoided, the data is (mostly) transparently upgraded
148to Unicode. There are some problems--see L<perlunicode/The "Unicode Bug">.
ba62762e 149
4192de81 150Internally, Perl currently uses either whatever the native eight-bit
376d9008
JB
151character set of the platform (for example Latin-1) is, defaulting to
152UTF-8, to encode Unicode strings. Specifically, if all code points in
153the string are C<0xFF> or less, Perl uses the native eight-bit
154character set. Otherwise, it uses UTF-8.
4192de81 155
7ca610e8 156A user of Perl does not normally need to know nor care how Perl
20ba30f4 157happens to encode its internal strings, but it becomes relevant when
ac036724 158outputting Unicode strings to a stream without a PerlIO layer (one with
159the "default" encoding). In such a case, the raw bytes used internally
7ca610e8
JH
160(the native character set or UTF-8, as appropriate for each string)
161will be used, and a "Wide character" warning will be issued if those
162strings contain a character beyond 0x00FF.
4192de81
JH
163
164For example,
165
ae5648b3 166 perl -e 'print "\x{DF}\n", "\x{0100}\x{DF}\n"'
4192de81
JH
167
168produces a fairly useless mixture of native bytes and UTF-8, as well
1bfb14c4 169as a warning:
4192de81 170
1bfb14c4
JH
171 Wide character in print at ...
172
740d4bb2 173To output UTF-8, use the C<:encoding> or C<:utf8> output layer. Prepending
4192de81
JH
174
175 binmode(STDOUT, ":utf8");
176
376d9008 177to this sample program ensures that the output is completely UTF-8,
1bfb14c4 178and removes the program's warning.
ba62762e 179
8aa8f774
JH
180You can enable automatic UTF-8-ification of your standard file
181handles, default C<open()> layer, and C<@ARGV> by using either
182the C<-C> command line switch or the C<PERL_UNICODE> environment
183variable, see L<perlrun> for the documentation of the C<-C> switch.
184
185Note that this means that Perl expects other software to work, too:
186if Perl has been led to believe that STDIN should be UTF-8, but then
187STDIN coming in from another command is not UTF-8, Perl will complain
188about the malformed UTF-8.
b310b053 189
ec71e770
JH
190All features that combine Unicode and I/O also require using the new
191PerlIO feature. Almost all Perl 5.8 platforms do use PerlIO, though:
192you can see whether yours is by running "perl -V" and looking for
193C<useperlio=define>.
194
64c66fb6
JH
195=head2 Unicode and EBCDIC
196
197Perl 5.8.0 also supports Unicode on EBCDIC platforms. There,
376d9008 198Unicode support is somewhat more complex to implement since
e1b711da
KW
199additional conversions are needed at every step.
200
201Later Perl releases have added code that will not work on EBCDIC platforms, and
202no one has complained, so the divergence has continued. If you want to run
203Perl on an EBCDIC platform, send email to perlbug@perl.org
64c66fb6 204
64c66fb6 205On EBCDIC platforms, the internal Unicode encoding form is UTF-EBCDIC
376d9008
JB
206instead of UTF-8. The difference is that as UTF-8 is "ASCII-safe" in
207that ASCII characters encode to UTF-8 as-is, while UTF-EBCDIC is
208"EBCDIC-safe".
ba62762e
JH
209
210=head2 Creating Unicode
211
376d9008
JB
212To create Unicode characters in literals for code points above C<0xFF>,
213use the C<\x{...}> notation in double-quoted strings:
ba62762e
JH
214
215 my $smiley = "\x{263a}";
216
376d9008 217Similarly, it can be used in regular expression literals
ba62762e
JH
218
219 $smiley =~ /\x{263a}/;
220
221At run-time you can use C<chr()>:
222
223 my $hebrew_alef = chr(0x05d0);
224
376d9008 225See L</"Further Resources"> for how to find all these numeric codes.
ba62762e 226
376d9008
JB
227Naturally, C<ord()> will do the reverse: it turns a character into
228a code point.
ba62762e 229
1bfb14c4
JH
230Note that C<\x..> (no C<{}> and only two hexadecimal digits), C<\x{...}>,
231and C<chr(...)> for arguments less than C<0x100> (decimal 256)
232generate an eight-bit character for backward compatibility with older
233Perls. For arguments of C<0x100> or more, Unicode characters are
a5f0baef
JH
234always produced. If you want to force the production of Unicode
235characters regardless of the numeric value, use C<pack("U", ...)>
236instead of C<\x..>, C<\x{...}>, or C<chr()>.
ba62762e
JH
237
238You can also use the C<charnames> pragma to invoke characters
376d9008 239by name in double-quoted strings:
ba62762e
JH
240
241 use charnames ':full';
242 my $arabic_alef = "\N{ARABIC LETTER ALEF}";
243
244And, as mentioned above, you can also C<pack()> numbers into Unicode
245characters:
246
247 my $georgian_an = pack("U", 0x10a0);
248
8a5e5dd5
JH
249Note that both C<\x{...}> and C<\N{...}> are compile-time string
250constants: you cannot use variables in them. if you want similar
fb121860 251run-time functionality, use C<chr()> and C<charnames::string_vianame()>.
8a5e5dd5 252
1eda90df 253If you want to force the result to Unicode characters, use the special
f337b084
TH
254C<"U0"> prefix. It consumes no arguments but causes the following bytes
255to be interpreted as the UTF-8 encoding of Unicode characters:
1eda90df 256
f337b084 257 my $chars = pack("U0W*", 0x80, 0x42);
771cd3b2 258
ae5648b3 259Likewise, you can stop such UTF-8 interpretation by using the special
771cd3b2 260C<"C0"> prefix.
1eda90df 261
ba62762e
JH
262=head2 Handling Unicode
263
264Handling Unicode is for the most part transparent: just use the
265strings as usual. Functions like C<index()>, C<length()>, and
266C<substr()> will work on the Unicode characters; regular expressions
267will work on the Unicode characters (see L<perlunicode> and L<perlretut>).
268
0111a78f
KW
269Note that Perl considers grapheme clusters to be separate characters, so for
270example
ba62762e
JH
271
272 use charnames ':full';
273 print length("\N{LATIN CAPITAL LETTER A}\N{COMBINING ACUTE ACCENT}"), "\n";
274
275will print 2, not 1. The only exception is that regular expressions
0111a78f 276have C<\X> for matching an extended grapheme cluster.
ba62762e 277
376d9008
JB
278Life is not quite so transparent, however, when working with legacy
279encodings, I/O, and certain special cases:
ba62762e
JH
280
281=head2 Legacy Encodings
282
283When you combine legacy data and Unicode the legacy data needs
284to be upgraded to Unicode. Normally ISO 8859-1 (or EBCDIC, if
ae5648b3 285applicable) is assumed.
ba62762e 286
376d9008 287The C<Encode> module knows about many encodings and has interfaces
ba62762e
JH
288for doing conversions between those encodings:
289
2c9359a2
RGS
290 use Encode 'decode';
291 $data = decode("iso-8859-3", $data); # convert from legacy to utf-8
ba62762e
JH
292
293=head2 Unicode I/O
294
8baee566 295Normally, writing out Unicode data
ba62762e 296
8baee566 297 print FH $some_string_with_unicode, "\n";
ba62762e 298
8baee566 299produces raw bytes that Perl happens to use to internally encode the
376d9008
JB
300Unicode string. Perl's internal encoding depends on the system as
301well as what characters happen to be in the string at the time. If
302any of the characters are at code points C<0x100> or above, you will get
303a warning. To ensure that the output is explicitly rendered in the
304encoding you desire--and to avoid the warning--open the stream with
305the desired encoding. Some examples:
ba62762e 306
ec90690f
ST
307 open FH, ">:utf8", "file";
308
309 open FH, ">:encoding(ucs2)", "file";
310 open FH, ">:encoding(UTF-8)", "file";
311 open FH, ">:encoding(shift_jis)", "file";
1d7919c5 312
376d9008 313and on already open streams, use C<binmode()>:
1d7919c5
JH
314
315 binmode(STDOUT, ":utf8");
316
ec90690f
ST
317 binmode(STDOUT, ":encoding(ucs2)");
318 binmode(STDOUT, ":encoding(UTF-8)");
319 binmode(STDOUT, ":encoding(shift_jis)");
320
b5d8778e 321The matching of encoding names is loose: case does not matter, and
fae2c0fb 322many encodings have several aliases. Note that the C<:utf8> layer
1bfb14c4 323must always be specified exactly like that; it is I<not> subject to
740d4bb2
JW
324the loose matching of encoding names. Also note that C<:utf8> is unsafe for
325input, because it accepts the data without validating that it is indeed valid
326UTF8.
b5d8778e 327
376d9008
JB
328See L<PerlIO> for the C<:utf8> layer, L<PerlIO::encoding> and
329L<Encode::PerlIO> for the C<:encoding()> layer, and
330L<Encode::Supported> for many encodings supported by the C<Encode>
331module.
ba62762e 332
a5f0baef 333Reading in a file that you know happens to be encoded in one of the
1bfb14c4
JH
334Unicode or legacy encodings does not magically turn the data into
335Unicode in Perl's eyes. To do that, specify the appropriate
fae2c0fb 336layer when opening files
ba62762e 337
740d4bb2 338 open(my $fh,'<:encoding(utf8)', 'anything');
8baee566
JF
339 my $line_of_unicode = <$fh>;
340
ec90690f 341 open(my $fh,'<:encoding(Big5)', 'anything');
8baee566 342 my $line_of_unicode = <$fh>;
ba62762e 343
fae2c0fb 344The I/O layers can also be specified more flexibly with
376d9008 345the C<open> pragma. See L<open>, or look at the following example.
ba62762e 346
9e5bbba0
KW
347 use open ':encoding(utf8)'; # input/output default encoding will be
348 # UTF-8
1d7919c5
JH
349 open X, ">file";
350 print X chr(0x100), "\n";
ba62762e 351 close X;
1d7919c5 352 open Y, "<file";
ba62762e
JH
353 printf "%#x\n", ord(<Y>); # this should print 0x100
354 close Y;
355
fae2c0fb 356With the C<open> pragma you can use the C<:locale> layer
ba62762e 357
12f98225 358 BEGIN { $ENV{LC_ALL} = $ENV{LANG} = 'ru_RU.KOI8-R' }
9e5bbba0
KW
359 # the :locale will probe the locale environment variables like
360 # LC_ALL
ba62762e
JH
361 use open OUT => ':locale'; # russki parusski
362 open(O, ">koi8");
363 print O chr(0x430); # Unicode CYRILLIC SMALL LETTER A = KOI8-R 0xc1
364 close O;
365 open(I, "<koi8");
366 printf "%#x\n", ord(<I>), "\n"; # this should print 0xc1
367 close I;
368
8baee566
JF
369These methods install a transparent filter on the I/O stream that
370converts data from the specified encoding when it is read in from the
a5f0baef 371stream. The result is always Unicode.
ba62762e
JH
372
373The L<open> pragma affects all the C<open()> calls after the pragma by
fae2c0fb
RGS
374setting default layers. If you want to affect only certain
375streams, use explicit layers directly in the C<open()> call.
ba62762e
JH
376
377You can switch encodings on an already opened stream by using
8baee566 378C<binmode()>; see L<perlfunc/binmode>.
ba62762e 379
1ecefa54
JH
380The C<:locale> does not currently (as of Perl 5.8.0) work with
381C<open()> and C<binmode()>, only with the C<open> pragma. The
8baee566 382C<:utf8> and C<:encoding(...)> methods do work with all of C<open()>,
1ecefa54 383C<binmode()>, and the C<open> pragma.
ba62762e 384
fae2c0fb 385Similarly, you may use these I/O layers on output streams to
a5f0baef
JH
386automatically convert Unicode to the specified encoding when it is
387written to the stream. For example, the following snippet copies the
388contents of the file "text.jis" (encoded as ISO-2022-JP, aka JIS) to
389the file "text.utf8", encoded as UTF-8:
ba62762e 390
c722ef0a
JH
391 open(my $nihongo, '<:encoding(iso-2022-jp)', 'text.jis');
392 open(my $unicode, '>:utf8', 'text.utf8');
0cf8a8d9 393 while (<$nihongo>) { print $unicode $_ }
ba62762e
JH
394
395The naming of encodings, both by the C<open()> and by the C<open>
2575c402
JW
396pragma allows for flexible names: C<koi8-r> and C<KOI8R> will both be
397understood.
ba62762e
JH
398
399Common encodings recognized by ISO, MIME, IANA, and various other
8baee566 400standardisation organisations are recognised; for a more detailed
1bfb14c4 401list see L<Encode::Supported>.
ba62762e
JH
402
403C<read()> reads characters and returns the number of characters.
404C<seek()> and C<tell()> operate on byte counts, as do C<sysread()>
405and C<sysseek()>.
406
8baee566 407Notice that because of the default behaviour of not doing any
fae2c0fb 408conversion upon input if there is no default layer,
ba62762e 409it is easy to mistakenly write code that keeps on expanding a file
1bfb14c4 410by repeatedly encoding the data:
ba62762e
JH
411
412 # BAD CODE WARNING
413 open F, "file";
8baee566 414 local $/; ## read in the whole file of 8-bit characters
ba62762e
JH
415 $t = <F>;
416 close F;
740d4bb2 417 open F, ">:encoding(utf8)", "file";
8baee566 418 print F $t; ## convert to UTF-8 on output
ba62762e
JH
419 close F;
420
421If you run this code twice, the contents of the F<file> will be twice
740d4bb2
JW
422UTF-8 encoded. A C<use open ':encoding(utf8)'> would have avoided the
423bug, or explicitly opening also the F<file> for input as UTF-8.
ba62762e 424
0c901d84 425B<NOTE>: the C<:utf8> and C<:encoding> features work only if your
d0fadae5
JH
426Perl has been built with the new PerlIO feature (which is the default
427on most systems).
0c901d84 428
1ecefa54
JH
429=head2 Displaying Unicode As Text
430
431Sometimes you might want to display Perl scalars containing Unicode as
8baee566 432simple ASCII (or EBCDIC) text. The following subroutine converts
1ecefa54 433its argument so that Unicode characters with code points greater than
1bfb14c4
JH
434255 are displayed as C<\x{...}>, control characters (like C<\n>) are
435displayed as C<\x..>, and the rest of the characters as themselves:
1ecefa54 436
9e5bbba0
KW
437 sub nice_string {
438 join("",
439 map { $_ > 255 ? # if wide character...
440 sprintf("\\x{%04X}", $_) : # \x{...}
441 chr($_) =~ /[[:cntrl:]]/ ? # else if control character ...
442 sprintf("\\x%02X", $_) : # \x..
443 quotemeta(chr($_)) # else quoted or as themselves
f337b084 444 } unpack("W*", $_[0])); # unpack Unicode characters
58c274a1
JF
445 }
446
447For example,
448
449 nice_string("foo\x{100}bar\n")
450
d0551e73 451returns the string
58c274a1 452
d0551e73
JH
453 'foo\x{0100}bar\x0A'
454
455which is ready to be printed.
1ecefa54 456
ba62762e
JH
457=head2 Special Cases
458
459=over 4
460
461=item *
462
463Bit Complement Operator ~ And vec()
464
1bfb14c4
JH
465The bit complement operator C<~> may produce surprising results if
466used on strings containing characters with ordinal values above
467255. In such a case, the results are consistent with the internal
468encoding of the characters, but not with much else. So don't do
469that. Similarly for C<vec()>: you will be operating on the
470internally-encoded bit patterns of the Unicode characters, not on
471the code point values, which is very probably not what you want.
ba62762e
JH
472
473=item *
474
8baee566
JF
475Peeking At Perl's Internal Encoding
476
477Normal users of Perl should never care how Perl encodes any particular
a5f0baef 478Unicode string (because the normal ways to get at the contents of a
376d9008 479string with Unicode--via input and output--should always be via
fae2c0fb 480explicitly-defined I/O layers). But if you must, there are two
a5f0baef 481ways of looking behind the scenes.
ba62762e
JH
482
483One way of peeking inside the internal encoding of Unicode characters
f337b084
TH
484is to use C<unpack("C*", ...> to get the bytes of whatever the string
485encoding happens to be, or C<unpack("U0..", ...)> to get the bytes of the
486UTF-8 encoding:
ba62762e 487
8baee566 488 # this prints c4 80 for the UTF-8 bytes 0xc4 0x80
f337b084 489 print join(" ", unpack("U0(H2)*", pack("U", 0x100))), "\n";
ba62762e
JH
490
491Yet another way would be to use the Devel::Peek module:
492
493 perl -MDevel::Peek -e 'Dump(chr(0x100))'
494
1e54db1a 495That shows the C<UTF8> flag in FLAGS and both the UTF-8 bytes
376d9008 496and Unicode characters in C<PV>. See also later in this document
8800c35a 497the discussion about the C<utf8::is_utf8()> function.
ba62762e
JH
498
499=back
500
501=head2 Advanced Topics
502
503=over 4
504
505=item *
506
507String Equivalence
508
509The question of string equivalence turns somewhat complicated
376d9008 510in Unicode: what do you mean by "equal"?
ba62762e 511
07698885
RGS
512(Is C<LATIN CAPITAL LETTER A WITH ACUTE> equal to
513C<LATIN CAPITAL LETTER A>?)
ba62762e 514
a5f0baef
JH
515The short answer is that by default Perl compares equivalence (C<eq>,
516C<ne>) based only on code points of the characters. In the above
376d9008
JB
517case, the answer is no (because 0x00C1 != 0x0041). But sometimes, any
518CAPITAL LETTER As should be considered equal, or even As of any case.
ba62762e
JH
519
520The long answer is that you need to consider character normalization
e1b711da
KW
521and casing issues: see L<Unicode::Normalize>, Unicode Technical Report #15,
522L<Unicode Normalization Forms|http://www.unicode.org/unicode/reports/tr15> and
523sections on case mapping in the L<Unicode Standard|http://www.unicode.org>.
ba62762e 524
1bfb14c4 525As of Perl 5.8.0, the "Full" case-folding of I<Case
e1b711da 526Mappings/SpecialCasing> is implemented, but bugs remain in C<qr//i> with them.
ba62762e
JH
527
528=item *
529
530String Collation
531
376d9008 532People like to see their strings nicely sorted--or as Unicode
ba62762e
JH
533parlance goes, collated. But again, what do you mean by collate?
534
07698885
RGS
535(Does C<LATIN CAPITAL LETTER A WITH ACUTE> come before or after
536C<LATIN CAPITAL LETTER A WITH GRAVE>?)
ba62762e 537
58c274a1 538The short answer is that by default, Perl compares strings (C<lt>,
ba62762e 539C<le>, C<cmp>, C<ge>, C<gt>) based only on the code points of the
1bfb14c4 540characters. In the above case, the answer is "after", since
da76a1f4 541C<0x00C1> > C<0x00C0>.
ba62762e
JH
542
543The long answer is that "it depends", and a good answer cannot be
544given without knowing (at the very least) the language context.
545See L<Unicode::Collate>, and I<Unicode Collation Algorithm>
2bbc8d55 546L<http://www.unicode.org/unicode/reports/tr10/>
ba62762e
JH
547
548=back
549
550=head2 Miscellaneous
551
552=over 4
553
554=item *
555
3ff56b75 556Character Ranges and Classes
ba62762e 557
0eb9ada1
KW
558Character ranges in regular expression bracketed character classes ( e.g.,
559C</[a-z]/>) and in the C<tr///> (also known as C<y///>) operator are not
560magically Unicode-aware. What this means is that C<[A-Za-z]> will not
561magically start to mean "all alphabetic letters" (not that it does mean that
562even for 8-bit characters; for those, if you are using locales (L<perllocale>),
563use C</[[:alpha:]]/>; and if not, use the 8-bit-aware property C<\p{alpha}>).
564
565All the properties that begin with C<\p> (and its inverse C<\P>) are actually
566character classes that are Unicode-aware. There are dozens of them, see
567L<perluniprops>.
568
569You can use Unicode code points as the end points of character ranges, and the
570range will include all Unicode code points that lie between those end points.
ba62762e
JH
571
572=item *
573
574String-To-Number Conversions
575
376d9008
JB
576Unicode does define several other decimal--and numeric--characters
577besides the familiar 0 to 9, such as the Arabic and Indic digits.
ba62762e 578Perl does not support string-to-number conversion for digits other
58c274a1 579than ASCII 0 to 9 (and ASCII a to f for hexadecimal).
ba62762e
JH
580
581=back
582
583=head2 Questions With Answers
584
585=over 4
586
818c4caa 587=item *
5cb3728c
RB
588
589Will My Old Scripts Break?
ba62762e
JH
590
591Very probably not. Unless you are generating Unicode characters
1bfb14c4
JH
592somehow, old behaviour should be preserved. About the only behaviour
593that has changed and which could start generating Unicode is the old
594behaviour of C<chr()> where supplying an argument more than 255
595produced a character modulo 255. C<chr(300)>, for example, was equal
596to C<chr(45)> or "-" (in ASCII), now it is LATIN CAPITAL LETTER I WITH
597BREVE.
ba62762e 598
818c4caa 599=item *
5cb3728c
RB
600
601How Do I Make My Scripts Work With Unicode?
ba62762e
JH
602
603Very little work should be needed since nothing changes until you
1bfb14c4
JH
604generate Unicode data. The most important thing is getting input as
605Unicode; for that, see the earlier I/O discussion.
ba62762e 606
818c4caa 607=item *
5cb3728c
RB
608
609How Do I Know Whether My String Is In Unicode?
ba62762e 610
2bbc8d55 611You shouldn't have to care. But you may, because currently the semantics of the
0eb9ada1 612characters whose ordinals are in the range 128 to 255 are different depending on
2bbc8d55 613whether the string they are contained within is in Unicode or not.
e1b711da 614(See L<perlunicode/When Unicode Does Not Happen>.)
ba62762e 615
2bbc8d55 616To determine if a string is in Unicode, use:
ba62762e 617
8800c35a 618 print utf8::is_utf8($string) ? 1 : 0, "\n";
ba62762e
JH
619
620But note that this doesn't mean that any of the characters in the
621string are necessary UTF-8 encoded, or that any of the characters have
622code points greater than 0xFF (255) or even 0x80 (128), or that the
623string has any characters at all. All the C<is_utf8()> does is to
624return the value of the internal "utf8ness" flag attached to the
376d9008 625C<$string>. If the flag is off, the bytes in the scalar are interpreted
3c1c8017 626as a single byte encoding. If the flag is on, the bytes in the scalar
0eb9ada1
KW
627are interpreted as the (variable-length, potentially multi-byte) UTF-8 encoded
628code points of the characters. Bytes added to a UTF-8 encoded string are
1e54db1a 629automatically upgraded to UTF-8. If mixed non-UTF-8 and UTF-8 scalars
376d9008 630are merged (double-quoted interpolation, explicit concatenation, and
3c1c8017
AT
631printf/sprintf parameter substitution), the result will be UTF-8 encoded
632as if copies of the byte strings were upgraded to UTF-8: for example,
633
634 $a = "ab\x80c";
635 $b = "\x{100}";
636 print "$a = $b\n";
637
a02b5feb
JH
638the output string will be UTF-8-encoded C<ab\x80c = \x{100}\n>, but
639C<$a> will stay byte-encoded.
ba62762e
JH
640
641Sometimes you might really need to know the byte length of a string
ce7675db 642instead of the character length. For that use either the
2bbc8d55
SP
643C<Encode::encode_utf8()> function or the C<bytes> pragma and
644the C<length()> function:
ba62762e
JH
645
646 my $unicode = chr(0x100);
647 print length($unicode), "\n"; # will print 1
ce7675db
A
648 require Encode;
649 print length(Encode::encode_utf8($unicode)), "\n"; # will print 2
ba62762e 650 use bytes;
1bfb14c4
JH
651 print length($unicode), "\n"; # will also print 2
652 # (the 0xC4 0x80 of the UTF-8)
0eb9ada1 653 no bytes;
ba62762e 654
818c4caa 655=item *
5cb3728c
RB
656
657How Do I Detect Data That's Not Valid In a Particular Encoding?
ba62762e 658
8baee566
JF
659Use the C<Encode> package to try converting it.
660For example,
ba62762e 661
bb2f379c 662 use Encode 'decode_utf8';
2bbc8d55 663
228ee848 664 if (eval { decode_utf8($string, Encode::FB_CROAK); 1 }) {
a365f2ce 665 # $string is valid utf8
ba62762e 666 } else {
a365f2ce 667 # $string is not valid utf8
ba62762e
JH
668 }
669
f337b084 670Or use C<unpack> to try decoding it:
ba62762e
JH
671
672 use warnings;
f337b084 673 @chars = unpack("C0U*", $string_of_bytes_that_I_think_is_utf8);
ba62762e 674
ae5648b3
RGS
675If invalid, a C<Malformed UTF-8 character> warning is produced. The "C0" means
676"process the string character per character". Without that, the
677C<unpack("U*", ...)> would work in C<U0> mode (the default if the format
678string starts with C<U>) and it would return the bytes making up the UTF-8
f337b084 679encoding of the target string, something that will always work.
ba62762e 680
818c4caa 681=item *
5cb3728c
RB
682
683How Do I Convert Binary Data Into a Particular Encoding, Or Vice Versa?
ba62762e 684
8baee566
JF
685This probably isn't as useful as you might think.
686Normally, you shouldn't need to.
ba62762e 687
1bfb14c4 688In one sense, what you are asking doesn't make much sense: encodings
376d9008 689are for characters, and binary data are not "characters", so converting
a5f0baef
JH
690"data" into some encoding isn't meaningful unless you know in what
691character set and encoding the binary data is in, in which case it's
376d9008 692not just binary data, now is it?
8baee566 693
1bfb14c4
JH
694If you have a raw sequence of bytes that you know should be
695interpreted via a particular encoding, you can use C<Encode>:
ba62762e
JH
696
697 use Encode 'from_to';
698 from_to($data, "iso-8859-1", "utf-8"); # from latin-1 to utf-8
699
1bfb14c4
JH
700The call to C<from_to()> changes the bytes in C<$data>, but nothing
701material about the nature of the string has changed as far as Perl is
702concerned. Both before and after the call, the string C<$data>
703contains just a bunch of 8-bit bytes. As far as Perl is concerned,
704the encoding of the string remains as "system-native 8-bit bytes".
8baee566
JF
705
706You might relate this to a fictional 'Translate' module:
707
708 use Translate;
709 my $phrase = "Yes";
710 Translate::from_to($phrase, 'english', 'deutsch');
711 ## phrase now contains "Ja"
ba62762e 712
8baee566 713The contents of the string changes, but not the nature of the string.
1bfb14c4
JH
714Perl doesn't know any more after the call than before that the
715contents of the string indicates the affirmative.
ba62762e 716
376d9008 717Back to converting data. If you have (or want) data in your system's
a5f0baef
JH
718native 8-bit encoding (e.g. Latin-1, EBCDIC, etc.), you can use
719pack/unpack to convert to/from Unicode.
ba62762e 720
f337b084
TH
721 $native_string = pack("W*", unpack("U*", $Unicode_string));
722 $Unicode_string = pack("U*", unpack("W*", $native_string));
ba62762e
JH
723
724If you have a sequence of bytes you B<know> is valid UTF-8,
725but Perl doesn't know it yet, you can make Perl a believer, too:
726
727 use Encode 'decode_utf8';
8baee566 728 $Unicode = decode_utf8($bytes);
ba62762e 729
f337b084
TH
730or:
731
732 $Unicode = pack("U0a*", $bytes);
ae5648b3 733
2bbc8d55
SP
734You can find the bytes that make up a UTF-8 sequence with
735
9e5bbba0 736 @bytes = unpack("C*", $Unicode_string)
2bbc8d55
SP
737
738and you can create well-formed Unicode with
739
9e5bbba0 740 $Unicode_string = pack("U*", 0xff, ...)
ba62762e 741
818c4caa 742=item *
5cb3728c
RB
743
744How Do I Display Unicode? How Do I Input Unicode?
ba62762e 745
2bbc8d55
SP
746See L<http://www.alanwood.net/unicode/> and
747L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e 748
818c4caa 749=item *
5cb3728c
RB
750
751How Does Unicode Work With Traditional Locales?
ba62762e
JH
752
753In Perl, not very well. Avoid using locales through the C<locale>
4c496f0c
JH
754pragma. Use only one or the other. But see L<perlrun> for the
755description of the C<-C> switch and its environment counterpart,
756C<$ENV{PERL_UNICODE}> to see how to enable various Unicode features,
757for example by using locale settings.
ba62762e
JH
758
759=back
760
761=head2 Hexadecimal Notation
762
376d9008
JB
763The Unicode standard prefers using hexadecimal notation because
764that more clearly shows the division of Unicode into blocks of 256 characters.
ba62762e
JH
765Hexadecimal is also simply shorter than decimal. You can use decimal
766notation, too, but learning to use hexadecimal just makes life easier
1bfb14c4 767with the Unicode standard. The C<U+HHHH> notation uses hexadecimal,
076d825e 768for example.
ba62762e
JH
769
770The C<0x> prefix means a hexadecimal number, the digits are 0-9 I<and>
771a-f (or A-F, case doesn't matter). Each hexadecimal digit represents
772four bits, or half a byte. C<print 0x..., "\n"> will show a
773hexadecimal number in decimal, and C<printf "%x\n", $decimal> will
774show a decimal number in hexadecimal. If you have just the
376d9008 775"hex digits" of a hexadecimal number, you can use the C<hex()> function.
ba62762e
JH
776
777 print 0x0009, "\n"; # 9
778 print 0x000a, "\n"; # 10
779 print 0x000f, "\n"; # 15
780 print 0x0010, "\n"; # 16
781 print 0x0011, "\n"; # 17
782 print 0x0100, "\n"; # 256
783
784 print 0x0041, "\n"; # 65
785
786 printf "%x\n", 65; # 41
787 printf "%#x\n", 65; # 0x41
788
789 print hex("41"), "\n"; # 65
790
791=head2 Further Resources
792
793=over 4
794
795=item *
796
797Unicode Consortium
798
2bbc8d55 799L<http://www.unicode.org/>
ba62762e
JH
800
801=item *
802
803Unicode FAQ
804
2bbc8d55 805L<http://www.unicode.org/unicode/faq/>
ba62762e
JH
806
807=item *
808
809Unicode Glossary
810
2bbc8d55 811L<http://www.unicode.org/glossary/>
ba62762e
JH
812
813=item *
814
815Unicode Useful Resources
816
2bbc8d55 817L<http://www.unicode.org/unicode/onlinedat/resources.html>
ba62762e
JH
818
819=item *
820
821Unicode and Multilingual Support in HTML, Fonts, Web Browsers and Other Applications
822
2bbc8d55 823L<http://www.alanwood.net/unicode/>
ba62762e
JH
824
825=item *
826
827UTF-8 and Unicode FAQ for Unix/Linux
828
2bbc8d55 829L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e
JH
830
831=item *
832
833Legacy Character Sets
834
2bbc8d55
SP
835L<http://www.czyborra.com/>
836L<http://www.eki.ee/letter/>
ba62762e
JH
837
838=item *
839
840The Unicode support files live within the Perl installation in the
841directory
842
843 $Config{installprivlib}/unicore
844
ae5648b3 845in Perl 5.8.0 or newer, and
ba62762e
JH
846
847 $Config{installprivlib}/unicode
848
849in the Perl 5.6 series. (The renaming to F<lib/unicore> was done to
850avoid naming conflicts with lib/Unicode in case-insensitive filesystems.)
551b6b6f 851The main Unicode data file is F<UnicodeData.txt> (or F<Unicode.301> in
ba62762e
JH
852Perl 5.6.1.) You can find the C<$Config{installprivlib}> by
853
854 perl "-V:installprivlib"
855
ba62762e
JH
856You can explore various information from the Unicode data files using
857the C<Unicode::UCD> module.
858
859=back
860
f6edf83b
JH
861=head1 UNICODE IN OLDER PERLS
862
863If you cannot upgrade your Perl to 5.8.0 or later, you can still
864do some Unicode processing by using the modules C<Unicode::String>,
865C<Unicode::Map8>, and C<Unicode::Map>, available from CPAN.
866If you have the GNU recode installed, you can also use the
376d9008 867Perl front-end C<Convert::Recode> for character conversions.
f6edf83b 868
aaef10c5 869The following are fast conversions from ISO 8859-1 (Latin-1) bytes
63de3cb2 870to UTF-8 bytes and back, the code works even with older Perl 5 versions.
aaef10c5
JH
871
872 # ISO 8859-1 to UTF-8
873 s/([\x80-\xFF])/chr(0xC0|ord($1)>>6).chr(0x80|ord($1)&0x3F)/eg;
874
875 # UTF-8 to ISO 8859-1
876 s/([\xC2\xC3])([\x80-\xBF])/chr(ord($1)<<6&0xC0|ord($2)&0x3F)/eg;
877
ba62762e
JH
878=head1 SEE ALSO
879
2575c402 880L<perlunitut>, L<perlunicode>, L<Encode>, L<open>, L<utf8>, L<bytes>,
4c496f0c
JH
881L<perlretut>, L<perlrun>, L<Unicode::Collate>, L<Unicode::Normalize>,
882L<Unicode::UCD>
ba62762e 883
376d9008 884=head1 ACKNOWLEDGMENTS
ba62762e
JH
885
886Thanks to the kind readers of the perl5-porters@perl.org,
887perl-unicode@perl.org, linux-utf8@nl.linux.org, and unicore@unicode.org
888mailing lists for their valuable feedback.
889
890=head1 AUTHOR, COPYRIGHT, AND LICENSE
891
0f2f9b7d 892Copyright 2001-2002 Jarkko Hietaniemi E<lt>jhi@iki.fiE<gt>
ba62762e
JH
893
894This document may be distributed under the same terms as Perl itself.