This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Nits in perluniintro.pod
[perl5.git] / pod / perluniintro.pod
CommitLineData
ba62762e
JH
1=head1 NAME
2
07fcf8ff 3perluniintro - Perl Unicode introduction
ba62762e
JH
4
5=head1 DESCRIPTION
6
7This document gives a general idea of Unicode and how to use Unicode
8in Perl.
9
10=head2 Unicode
11
376d9008 12Unicode is a character set standard which plans to codify all of the
ba62762e
JH
13writing systems of the world, plus many other symbols.
14
15Unicode and ISO/IEC 10646 are coordinated standards that provide code
376d9008 16points for characters in almost all modern character set standards,
ba62762e 17covering more than 30 writing systems and hundreds of languages,
376d9008 18including all commercially-important modern languages. All characters
ba62762e
JH
19in the largest Chinese, Japanese, and Korean dictionaries are also
20encoded. The standards will eventually cover almost all characters in
21more than 250 writing systems and thousands of languages.
4c496f0c 22Unicode 1.0 was released in October 1991, and 4.0 in April 2003.
ba62762e
JH
23
24A Unicode I<character> is an abstract entity. It is not bound to any
376d9008
JB
25particular integer width, especially not to the C language C<char>.
26Unicode is language-neutral and display-neutral: it does not encode the
e1b711da 27language of the text, and it does not generally define fonts or other graphical
ba62762e
JH
28layout details. Unicode operates on characters and on text built from
29those characters.
30
31Unicode defines characters like C<LATIN CAPITAL LETTER A> or C<GREEK
376d9008
JB
32SMALL LETTER ALPHA> and unique numbers for the characters, in this
33case 0x0041 and 0x03B1, respectively. These unique numbers are called
34I<code points>.
ba62762e
JH
35
36The Unicode standard prefers using hexadecimal notation for the code
4c496f0c
JH
37points. If numbers like C<0x0041> are unfamiliar to you, take a peek
38at a later section, L</"Hexadecimal Notation">. The Unicode standard
39uses the notation C<U+0041 LATIN CAPITAL LETTER A>, to give the
40hexadecimal code point and the normative name of the character.
ba62762e
JH
41
42Unicode also defines various I<properties> for the characters, like
376d9008 43"uppercase" or "lowercase", "decimal digit", or "punctuation";
ba62762e
JH
44these properties are independent of the names of the characters.
45Furthermore, various operations on the characters like uppercasing,
376d9008 46lowercasing, and collating (sorting) are defined.
ba62762e 47
0111a78f
KW
48A Unicode I<logical> "character" can actually consist of more than one internal
49I<actual> "character" or code point. For Western languages, this is adequately
c670e63a 50modelled by a I<base character> (like C<LATIN CAPITAL LETTER A>) followed
0111a78f 51by one or more I<modifiers> (like C<COMBINING ACUTE ACCENT>). This sequence of
376d9008 52base character and modifiers is called a I<combining character
0111a78f 53sequence>. Some non-western languages require more complicated
c670e63a
KW
54models, so Unicode created the I<grapheme cluster> concept, and then the
55I<extended grapheme cluster>. For example, a Korean Hangul syllable is
0111a78f 56considered a single logical character, but most often consists of three actual
c670e63a
KW
57Unicode characters: a leading consonant followed by an interior vowel followed
58by a trailing consonant.
0111a78f
KW
59
60Whether to call these extended grapheme clusters "characters" depends on your
61point of view. If you are a programmer, you probably would tend towards seeing
62each element in the sequences as one unit, or "character". The whole sequence
63could be seen as one "character", however, from the user's point of view, since
64that's probably what it looks like in the context of the user's language.
376d9008
JB
65
66With this "whole sequence" view of characters, the total number of
67characters is open-ended. But in the programmer's "one unit is one
68character" point of view, the concept of "characters" is more
f7d1603d 69deterministic. In this document, we take that second point of view:
0111a78f 70one "character" is one Unicode code point.
376d9008
JB
71
72For some combinations, there are I<precomposed> characters.
73C<LATIN CAPITAL LETTER A WITH ACUTE>, for example, is defined as
ba62762e 74a single code point. These precomposed characters are, however,
376d9008 75only available for some combinations, and are mainly
ba62762e 76meant to support round-trip conversions between Unicode and legacy
376d9008
JB
77standards (like the ISO 8859). In the general case, the composing
78method is more extensible. To support conversion between
ba62762e 79different compositions of the characters, various I<normalization
376d9008 80forms> to standardize representations are also defined.
ba62762e
JH
81
82Because of backward compatibility with legacy encodings, the "a unique
376d9008
JB
83number for every character" idea breaks down a bit: instead, there is
84"at least one number for every character". The same character could
85be represented differently in several legacy encodings. The
86converse is also not true: some code points do not have an assigned
87character. Firstly, there are unallocated code points within
88otherwise used blocks. Secondly, there are special Unicode control
89characters that do not represent true characters.
ba62762e 90
e1b711da 91A common myth about Unicode is that it is "16-bit", that is,
376d9008 92Unicode is only represented as C<0x10000> (or 65536) characters from
4c496f0c
JH
93C<0x0000> to C<0xFFFF>. B<This is untrue.> Since Unicode 2.0 (July
941996), Unicode has been defined all the way up to 21 bits (C<0x10FFFF>),
95and since Unicode 3.1 (March 2001), characters have been defined
96beyond C<0xFFFF>. The first C<0x10000> characters are called the
97I<Plane 0>, or the I<Basic Multilingual Plane> (BMP). With Unicode
983.1, 17 (yes, seventeen) planes in all were defined--but they are
99nowhere near full of defined characters, yet.
ba62762e 100
e1b711da 101Another myth is about Unicode blocks--that they have something to
1bfb14c4
JH
102do with languages--that each block would define the characters used
103by a language or a set of languages. B<This is also untrue.>
104The division into blocks exists, but it is almost completely
105accidental--an artifact of how the characters have been and
e1b711da
KW
106still are allocated. Instead, there is a concept called I<scripts>, which is
107more useful: there is C<Latin> script, C<Greek> script, and so on. Scripts
108usually span varied parts of several blocks. For more information about
109scripts, see L<perlunicode/Scripts>.
ba62762e
JH
110
111The Unicode code points are just abstract numbers. To input and
4c496f0c
JH
112output these abstract numbers, the numbers must be I<encoded> or
113I<serialised> somehow. Unicode defines several I<character encoding
114forms>, of which I<UTF-8> is perhaps the most popular. UTF-8 is a
115variable length encoding that encodes Unicode characters as 1 to 6
e1b711da 116bytes. Other encodings
4c496f0c
JH
117include UTF-16 and UTF-32 and their big- and little-endian variants
118(UTF-8 is byte-order independent) The ISO/IEC 10646 defines the UCS-2
119and UCS-4 encoding forms.
ba62762e 120
376d9008
JB
121For more information about encodings--for instance, to learn what
122I<surrogates> and I<byte order marks> (BOMs) are--see L<perlunicode>.
ba62762e
JH
123
124=head2 Perl's Unicode Support
125
376d9008
JB
126Starting from Perl 5.6.0, Perl has had the capacity to handle Unicode
127natively. Perl 5.8.0, however, is the first recommended release for
128serious Unicode work. The maintenance release 5.6.1 fixed many of the
129problems of the initial Unicode implementation, but for example
1bfb14c4 130regular expressions still do not work with Unicode in 5.6.1.
ba62762e 131
2bbc8d55 132B<Starting from Perl 5.8.0, the use of C<use utf8> is needed only in much more restricted circumstances.> In earlier releases the C<utf8> pragma was used to declare
ba62762e 133that operations in the current block or file would be Unicode-aware.
376d9008 134This model was found to be wrong, or at least clumsy: the "Unicodeness"
1bfb14c4 135is now carried with the data, instead of being attached to the
376d9008
JB
136operations. Only one case remains where an explicit C<use utf8> is
137needed: if your Perl script itself is encoded in UTF-8, you can use
138UTF-8 in your identifier names, and in string and regular expression
139literals, by saying C<use utf8>. This is not the default because
8f8cf39c 140scripts with legacy 8-bit data in them would break. See L<utf8>.
ba62762e
JH
141
142=head2 Perl's Unicode Model
143
376d9008
JB
144Perl supports both pre-5.6 strings of eight-bit native bytes, and
145strings of Unicode characters. The principle is that Perl tries to
146keep its data as eight-bit bytes for as long as possible, but as soon
e1b711da
KW
147as Unicodeness cannot be avoided, the data is (mostly) transparently upgraded
148to Unicode. There are some problems--see L<perlunicode/The "Unicode Bug">.
ba62762e 149
4192de81 150Internally, Perl currently uses either whatever the native eight-bit
376d9008
JB
151character set of the platform (for example Latin-1) is, defaulting to
152UTF-8, to encode Unicode strings. Specifically, if all code points in
153the string are C<0xFF> or less, Perl uses the native eight-bit
154character set. Otherwise, it uses UTF-8.
4192de81 155
7ca610e8 156A user of Perl does not normally need to know nor care how Perl
20ba30f4 157happens to encode its internal strings, but it becomes relevant when
ac036724 158outputting Unicode strings to a stream without a PerlIO layer (one with
159the "default" encoding). In such a case, the raw bytes used internally
7ca610e8
JH
160(the native character set or UTF-8, as appropriate for each string)
161will be used, and a "Wide character" warning will be issued if those
162strings contain a character beyond 0x00FF.
4192de81
JH
163
164For example,
165
ae5648b3 166 perl -e 'print "\x{DF}\n", "\x{0100}\x{DF}\n"'
4192de81
JH
167
168produces a fairly useless mixture of native bytes and UTF-8, as well
1bfb14c4 169as a warning:
4192de81 170
1bfb14c4
JH
171 Wide character in print at ...
172
740d4bb2 173To output UTF-8, use the C<:encoding> or C<:utf8> output layer. Prepending
4192de81
JH
174
175 binmode(STDOUT, ":utf8");
176
376d9008 177to this sample program ensures that the output is completely UTF-8,
1bfb14c4 178and removes the program's warning.
ba62762e 179
8aa8f774
JH
180You can enable automatic UTF-8-ification of your standard file
181handles, default C<open()> layer, and C<@ARGV> by using either
182the C<-C> command line switch or the C<PERL_UNICODE> environment
183variable, see L<perlrun> for the documentation of the C<-C> switch.
184
185Note that this means that Perl expects other software to work, too:
186if Perl has been led to believe that STDIN should be UTF-8, but then
187STDIN coming in from another command is not UTF-8, Perl will complain
188about the malformed UTF-8.
b310b053 189
ec71e770
JH
190All features that combine Unicode and I/O also require using the new
191PerlIO feature. Almost all Perl 5.8 platforms do use PerlIO, though:
192you can see whether yours is by running "perl -V" and looking for
193C<useperlio=define>.
194
64c66fb6
JH
195=head2 Unicode and EBCDIC
196
197Perl 5.8.0 also supports Unicode on EBCDIC platforms. There,
376d9008 198Unicode support is somewhat more complex to implement since
e1b711da
KW
199additional conversions are needed at every step.
200
201Later Perl releases have added code that will not work on EBCDIC platforms, and
202no one has complained, so the divergence has continued. If you want to run
203Perl on an EBCDIC platform, send email to perlbug@perl.org
64c66fb6 204
64c66fb6 205On EBCDIC platforms, the internal Unicode encoding form is UTF-EBCDIC
376d9008
JB
206instead of UTF-8. The difference is that as UTF-8 is "ASCII-safe" in
207that ASCII characters encode to UTF-8 as-is, while UTF-EBCDIC is
208"EBCDIC-safe".
ba62762e
JH
209
210=head2 Creating Unicode
211
376d9008
JB
212To create Unicode characters in literals for code points above C<0xFF>,
213use the C<\x{...}> notation in double-quoted strings:
ba62762e
JH
214
215 my $smiley = "\x{263a}";
216
376d9008 217Similarly, it can be used in regular expression literals
ba62762e
JH
218
219 $smiley =~ /\x{263a}/;
220
221At run-time you can use C<chr()>:
222
223 my $hebrew_alef = chr(0x05d0);
224
376d9008 225See L</"Further Resources"> for how to find all these numeric codes.
ba62762e 226
376d9008
JB
227Naturally, C<ord()> will do the reverse: it turns a character into
228a code point.
ba62762e 229
1bfb14c4
JH
230Note that C<\x..> (no C<{}> and only two hexadecimal digits), C<\x{...}>,
231and C<chr(...)> for arguments less than C<0x100> (decimal 256)
232generate an eight-bit character for backward compatibility with older
233Perls. For arguments of C<0x100> or more, Unicode characters are
a5f0baef
JH
234always produced. If you want to force the production of Unicode
235characters regardless of the numeric value, use C<pack("U", ...)>
236instead of C<\x..>, C<\x{...}>, or C<chr()>.
ba62762e
JH
237
238You can also use the C<charnames> pragma to invoke characters
376d9008 239by name in double-quoted strings:
ba62762e
JH
240
241 use charnames ':full';
242 my $arabic_alef = "\N{ARABIC LETTER ALEF}";
243
244And, as mentioned above, you can also C<pack()> numbers into Unicode
245characters:
246
247 my $georgian_an = pack("U", 0x10a0);
248
8a5e5dd5
JH
249Note that both C<\x{...}> and C<\N{...}> are compile-time string
250constants: you cannot use variables in them. if you want similar
251run-time functionality, use C<chr()> and C<charnames::vianame()>.
252
1eda90df 253If you want to force the result to Unicode characters, use the special
f337b084
TH
254C<"U0"> prefix. It consumes no arguments but causes the following bytes
255to be interpreted as the UTF-8 encoding of Unicode characters:
1eda90df 256
f337b084 257 my $chars = pack("U0W*", 0x80, 0x42);
771cd3b2 258
ae5648b3 259Likewise, you can stop such UTF-8 interpretation by using the special
771cd3b2 260C<"C0"> prefix.
1eda90df 261
ba62762e
JH
262=head2 Handling Unicode
263
264Handling Unicode is for the most part transparent: just use the
265strings as usual. Functions like C<index()>, C<length()>, and
266C<substr()> will work on the Unicode characters; regular expressions
267will work on the Unicode characters (see L<perlunicode> and L<perlretut>).
268
0111a78f
KW
269Note that Perl considers grapheme clusters to be separate characters, so for
270example
ba62762e
JH
271
272 use charnames ':full';
273 print length("\N{LATIN CAPITAL LETTER A}\N{COMBINING ACUTE ACCENT}"), "\n";
274
275will print 2, not 1. The only exception is that regular expressions
0111a78f 276have C<\X> for matching an extended grapheme cluster.
ba62762e 277
376d9008
JB
278Life is not quite so transparent, however, when working with legacy
279encodings, I/O, and certain special cases:
ba62762e
JH
280
281=head2 Legacy Encodings
282
283When you combine legacy data and Unicode the legacy data needs
284to be upgraded to Unicode. Normally ISO 8859-1 (or EBCDIC, if
ae5648b3 285applicable) is assumed.
ba62762e 286
376d9008 287The C<Encode> module knows about many encodings and has interfaces
ba62762e
JH
288for doing conversions between those encodings:
289
2c9359a2
RGS
290 use Encode 'decode';
291 $data = decode("iso-8859-3", $data); # convert from legacy to utf-8
ba62762e
JH
292
293=head2 Unicode I/O
294
8baee566 295Normally, writing out Unicode data
ba62762e 296
8baee566 297 print FH $some_string_with_unicode, "\n";
ba62762e 298
8baee566 299produces raw bytes that Perl happens to use to internally encode the
376d9008
JB
300Unicode string. Perl's internal encoding depends on the system as
301well as what characters happen to be in the string at the time. If
302any of the characters are at code points C<0x100> or above, you will get
303a warning. To ensure that the output is explicitly rendered in the
304encoding you desire--and to avoid the warning--open the stream with
305the desired encoding. Some examples:
ba62762e 306
ec90690f
TS
307 open FH, ">:utf8", "file";
308
309 open FH, ">:encoding(ucs2)", "file";
310 open FH, ">:encoding(UTF-8)", "file";
311 open FH, ">:encoding(shift_jis)", "file";
1d7919c5 312
376d9008 313and on already open streams, use C<binmode()>:
1d7919c5
JH
314
315 binmode(STDOUT, ":utf8");
316
ec90690f
TS
317 binmode(STDOUT, ":encoding(ucs2)");
318 binmode(STDOUT, ":encoding(UTF-8)");
319 binmode(STDOUT, ":encoding(shift_jis)");
320
b5d8778e 321The matching of encoding names is loose: case does not matter, and
fae2c0fb 322many encodings have several aliases. Note that the C<:utf8> layer
1bfb14c4 323must always be specified exactly like that; it is I<not> subject to
740d4bb2
JW
324the loose matching of encoding names. Also note that C<:utf8> is unsafe for
325input, because it accepts the data without validating that it is indeed valid
326UTF8.
b5d8778e 327
376d9008
JB
328See L<PerlIO> for the C<:utf8> layer, L<PerlIO::encoding> and
329L<Encode::PerlIO> for the C<:encoding()> layer, and
330L<Encode::Supported> for many encodings supported by the C<Encode>
331module.
ba62762e 332
a5f0baef 333Reading in a file that you know happens to be encoded in one of the
1bfb14c4
JH
334Unicode or legacy encodings does not magically turn the data into
335Unicode in Perl's eyes. To do that, specify the appropriate
fae2c0fb 336layer when opening files
ba62762e 337
740d4bb2 338 open(my $fh,'<:encoding(utf8)', 'anything');
8baee566
JF
339 my $line_of_unicode = <$fh>;
340
ec90690f 341 open(my $fh,'<:encoding(Big5)', 'anything');
8baee566 342 my $line_of_unicode = <$fh>;
ba62762e 343
fae2c0fb 344The I/O layers can also be specified more flexibly with
376d9008 345the C<open> pragma. See L<open>, or look at the following example.
ba62762e 346
740d4bb2 347 use open ':encoding(utf8)'; # input/output default encoding will be UTF-8
1d7919c5
JH
348 open X, ">file";
349 print X chr(0x100), "\n";
ba62762e 350 close X;
1d7919c5 351 open Y, "<file";
ba62762e
JH
352 printf "%#x\n", ord(<Y>); # this should print 0x100
353 close Y;
354
fae2c0fb 355With the C<open> pragma you can use the C<:locale> layer
ba62762e 356
12f98225 357 BEGIN { $ENV{LC_ALL} = $ENV{LANG} = 'ru_RU.KOI8-R' }
1ecefa54 358 # the :locale will probe the locale environment variables like LC_ALL
ba62762e
JH
359 use open OUT => ':locale'; # russki parusski
360 open(O, ">koi8");
361 print O chr(0x430); # Unicode CYRILLIC SMALL LETTER A = KOI8-R 0xc1
362 close O;
363 open(I, "<koi8");
364 printf "%#x\n", ord(<I>), "\n"; # this should print 0xc1
365 close I;
366
8baee566
JF
367These methods install a transparent filter on the I/O stream that
368converts data from the specified encoding when it is read in from the
a5f0baef 369stream. The result is always Unicode.
ba62762e
JH
370
371The L<open> pragma affects all the C<open()> calls after the pragma by
fae2c0fb
RGS
372setting default layers. If you want to affect only certain
373streams, use explicit layers directly in the C<open()> call.
ba62762e
JH
374
375You can switch encodings on an already opened stream by using
8baee566 376C<binmode()>; see L<perlfunc/binmode>.
ba62762e 377
1ecefa54
JH
378The C<:locale> does not currently (as of Perl 5.8.0) work with
379C<open()> and C<binmode()>, only with the C<open> pragma. The
8baee566 380C<:utf8> and C<:encoding(...)> methods do work with all of C<open()>,
1ecefa54 381C<binmode()>, and the C<open> pragma.
ba62762e 382
fae2c0fb 383Similarly, you may use these I/O layers on output streams to
a5f0baef
JH
384automatically convert Unicode to the specified encoding when it is
385written to the stream. For example, the following snippet copies the
386contents of the file "text.jis" (encoded as ISO-2022-JP, aka JIS) to
387the file "text.utf8", encoded as UTF-8:
ba62762e 388
c722ef0a
JH
389 open(my $nihongo, '<:encoding(iso-2022-jp)', 'text.jis');
390 open(my $unicode, '>:utf8', 'text.utf8');
0cf8a8d9 391 while (<$nihongo>) { print $unicode $_ }
ba62762e
JH
392
393The naming of encodings, both by the C<open()> and by the C<open>
2575c402
JW
394pragma allows for flexible names: C<koi8-r> and C<KOI8R> will both be
395understood.
ba62762e
JH
396
397Common encodings recognized by ISO, MIME, IANA, and various other
8baee566 398standardisation organisations are recognised; for a more detailed
1bfb14c4 399list see L<Encode::Supported>.
ba62762e
JH
400
401C<read()> reads characters and returns the number of characters.
402C<seek()> and C<tell()> operate on byte counts, as do C<sysread()>
403and C<sysseek()>.
404
8baee566 405Notice that because of the default behaviour of not doing any
fae2c0fb 406conversion upon input if there is no default layer,
ba62762e 407it is easy to mistakenly write code that keeps on expanding a file
1bfb14c4 408by repeatedly encoding the data:
ba62762e
JH
409
410 # BAD CODE WARNING
411 open F, "file";
8baee566 412 local $/; ## read in the whole file of 8-bit characters
ba62762e
JH
413 $t = <F>;
414 close F;
740d4bb2 415 open F, ">:encoding(utf8)", "file";
8baee566 416 print F $t; ## convert to UTF-8 on output
ba62762e
JH
417 close F;
418
419If you run this code twice, the contents of the F<file> will be twice
740d4bb2
JW
420UTF-8 encoded. A C<use open ':encoding(utf8)'> would have avoided the
421bug, or explicitly opening also the F<file> for input as UTF-8.
ba62762e 422
0c901d84 423B<NOTE>: the C<:utf8> and C<:encoding> features work only if your
d0fadae5
JH
424Perl has been built with the new PerlIO feature (which is the default
425on most systems).
0c901d84 426
1ecefa54
JH
427=head2 Displaying Unicode As Text
428
429Sometimes you might want to display Perl scalars containing Unicode as
8baee566 430simple ASCII (or EBCDIC) text. The following subroutine converts
1ecefa54 431its argument so that Unicode characters with code points greater than
1bfb14c4
JH
432255 are displayed as C<\x{...}>, control characters (like C<\n>) are
433displayed as C<\x..>, and the rest of the characters as themselves:
1ecefa54 434
58c274a1
JF
435 sub nice_string {
436 join("",
437 map { $_ > 255 ? # if wide character...
8baee566 438 sprintf("\\x{%04X}", $_) : # \x{...}
58c274a1 439 chr($_) =~ /[[:cntrl:]]/ ? # else if control character ...
8baee566 440 sprintf("\\x%02X", $_) : # \x..
d0551e73 441 quotemeta(chr($_)) # else quoted or as themselves
f337b084 442 } unpack("W*", $_[0])); # unpack Unicode characters
58c274a1
JF
443 }
444
445For example,
446
447 nice_string("foo\x{100}bar\n")
448
d0551e73 449returns the string
58c274a1 450
d0551e73
JH
451 'foo\x{0100}bar\x0A'
452
453which is ready to be printed.
1ecefa54 454
ba62762e
JH
455=head2 Special Cases
456
457=over 4
458
459=item *
460
461Bit Complement Operator ~ And vec()
462
1bfb14c4
JH
463The bit complement operator C<~> may produce surprising results if
464used on strings containing characters with ordinal values above
465255. In such a case, the results are consistent with the internal
466encoding of the characters, but not with much else. So don't do
467that. Similarly for C<vec()>: you will be operating on the
468internally-encoded bit patterns of the Unicode characters, not on
469the code point values, which is very probably not what you want.
ba62762e
JH
470
471=item *
472
8baee566
JF
473Peeking At Perl's Internal Encoding
474
475Normal users of Perl should never care how Perl encodes any particular
a5f0baef 476Unicode string (because the normal ways to get at the contents of a
376d9008 477string with Unicode--via input and output--should always be via
fae2c0fb 478explicitly-defined I/O layers). But if you must, there are two
a5f0baef 479ways of looking behind the scenes.
ba62762e
JH
480
481One way of peeking inside the internal encoding of Unicode characters
f337b084
TH
482is to use C<unpack("C*", ...> to get the bytes of whatever the string
483encoding happens to be, or C<unpack("U0..", ...)> to get the bytes of the
484UTF-8 encoding:
ba62762e 485
8baee566 486 # this prints c4 80 for the UTF-8 bytes 0xc4 0x80
f337b084 487 print join(" ", unpack("U0(H2)*", pack("U", 0x100))), "\n";
ba62762e
JH
488
489Yet another way would be to use the Devel::Peek module:
490
491 perl -MDevel::Peek -e 'Dump(chr(0x100))'
492
1e54db1a 493That shows the C<UTF8> flag in FLAGS and both the UTF-8 bytes
376d9008 494and Unicode characters in C<PV>. See also later in this document
8800c35a 495the discussion about the C<utf8::is_utf8()> function.
ba62762e
JH
496
497=back
498
499=head2 Advanced Topics
500
501=over 4
502
503=item *
504
505String Equivalence
506
507The question of string equivalence turns somewhat complicated
376d9008 508in Unicode: what do you mean by "equal"?
ba62762e 509
07698885
RGS
510(Is C<LATIN CAPITAL LETTER A WITH ACUTE> equal to
511C<LATIN CAPITAL LETTER A>?)
ba62762e 512
a5f0baef
JH
513The short answer is that by default Perl compares equivalence (C<eq>,
514C<ne>) based only on code points of the characters. In the above
376d9008
JB
515case, the answer is no (because 0x00C1 != 0x0041). But sometimes, any
516CAPITAL LETTER As should be considered equal, or even As of any case.
ba62762e
JH
517
518The long answer is that you need to consider character normalization
e1b711da
KW
519and casing issues: see L<Unicode::Normalize>, Unicode Technical Report #15,
520L<Unicode Normalization Forms|http://www.unicode.org/unicode/reports/tr15> and
521sections on case mapping in the L<Unicode Standard|http://www.unicode.org>.
ba62762e 522
1bfb14c4 523As of Perl 5.8.0, the "Full" case-folding of I<Case
e1b711da 524Mappings/SpecialCasing> is implemented, but bugs remain in C<qr//i> with them.
ba62762e
JH
525
526=item *
527
528String Collation
529
376d9008 530People like to see their strings nicely sorted--or as Unicode
ba62762e
JH
531parlance goes, collated. But again, what do you mean by collate?
532
07698885
RGS
533(Does C<LATIN CAPITAL LETTER A WITH ACUTE> come before or after
534C<LATIN CAPITAL LETTER A WITH GRAVE>?)
ba62762e 535
58c274a1 536The short answer is that by default, Perl compares strings (C<lt>,
ba62762e 537C<le>, C<cmp>, C<ge>, C<gt>) based only on the code points of the
1bfb14c4 538characters. In the above case, the answer is "after", since
da76a1f4 539C<0x00C1> > C<0x00C0>.
ba62762e
JH
540
541The long answer is that "it depends", and a good answer cannot be
542given without knowing (at the very least) the language context.
543See L<Unicode::Collate>, and I<Unicode Collation Algorithm>
2bbc8d55 544L<http://www.unicode.org/unicode/reports/tr10/>
ba62762e
JH
545
546=back
547
548=head2 Miscellaneous
549
550=over 4
551
552=item *
553
3ff56b75 554Character Ranges and Classes
ba62762e 555
0eb9ada1
KW
556Character ranges in regular expression bracketed character classes ( e.g.,
557C</[a-z]/>) and in the C<tr///> (also known as C<y///>) operator are not
558magically Unicode-aware. What this means is that C<[A-Za-z]> will not
559magically start to mean "all alphabetic letters" (not that it does mean that
560even for 8-bit characters; for those, if you are using locales (L<perllocale>),
561use C</[[:alpha:]]/>; and if not, use the 8-bit-aware property C<\p{alpha}>).
562
563All the properties that begin with C<\p> (and its inverse C<\P>) are actually
564character classes that are Unicode-aware. There are dozens of them, see
565L<perluniprops>.
566
567You can use Unicode code points as the end points of character ranges, and the
568range will include all Unicode code points that lie between those end points.
ba62762e
JH
569
570=item *
571
572String-To-Number Conversions
573
376d9008
JB
574Unicode does define several other decimal--and numeric--characters
575besides the familiar 0 to 9, such as the Arabic and Indic digits.
ba62762e 576Perl does not support string-to-number conversion for digits other
58c274a1 577than ASCII 0 to 9 (and ASCII a to f for hexadecimal).
ba62762e
JH
578
579=back
580
581=head2 Questions With Answers
582
583=over 4
584
818c4caa 585=item *
5cb3728c
RB
586
587Will My Old Scripts Break?
ba62762e
JH
588
589Very probably not. Unless you are generating Unicode characters
1bfb14c4
JH
590somehow, old behaviour should be preserved. About the only behaviour
591that has changed and which could start generating Unicode is the old
592behaviour of C<chr()> where supplying an argument more than 255
593produced a character modulo 255. C<chr(300)>, for example, was equal
594to C<chr(45)> or "-" (in ASCII), now it is LATIN CAPITAL LETTER I WITH
595BREVE.
ba62762e 596
818c4caa 597=item *
5cb3728c
RB
598
599How Do I Make My Scripts Work With Unicode?
ba62762e
JH
600
601Very little work should be needed since nothing changes until you
1bfb14c4
JH
602generate Unicode data. The most important thing is getting input as
603Unicode; for that, see the earlier I/O discussion.
ba62762e 604
818c4caa 605=item *
5cb3728c
RB
606
607How Do I Know Whether My String Is In Unicode?
ba62762e 608
2bbc8d55 609You shouldn't have to care. But you may, because currently the semantics of the
0eb9ada1 610characters whose ordinals are in the range 128 to 255 are different depending on
2bbc8d55 611whether the string they are contained within is in Unicode or not.
e1b711da 612(See L<perlunicode/When Unicode Does Not Happen>.)
ba62762e 613
2bbc8d55 614To determine if a string is in Unicode, use:
ba62762e 615
8800c35a 616 print utf8::is_utf8($string) ? 1 : 0, "\n";
ba62762e
JH
617
618But note that this doesn't mean that any of the characters in the
619string are necessary UTF-8 encoded, or that any of the characters have
620code points greater than 0xFF (255) or even 0x80 (128), or that the
621string has any characters at all. All the C<is_utf8()> does is to
622return the value of the internal "utf8ness" flag attached to the
376d9008 623C<$string>. If the flag is off, the bytes in the scalar are interpreted
3c1c8017 624as a single byte encoding. If the flag is on, the bytes in the scalar
0eb9ada1
KW
625are interpreted as the (variable-length, potentially multi-byte) UTF-8 encoded
626code points of the characters. Bytes added to a UTF-8 encoded string are
1e54db1a 627automatically upgraded to UTF-8. If mixed non-UTF-8 and UTF-8 scalars
376d9008 628are merged (double-quoted interpolation, explicit concatenation, and
3c1c8017
AT
629printf/sprintf parameter substitution), the result will be UTF-8 encoded
630as if copies of the byte strings were upgraded to UTF-8: for example,
631
632 $a = "ab\x80c";
633 $b = "\x{100}";
634 print "$a = $b\n";
635
a02b5feb
JH
636the output string will be UTF-8-encoded C<ab\x80c = \x{100}\n>, but
637C<$a> will stay byte-encoded.
ba62762e
JH
638
639Sometimes you might really need to know the byte length of a string
ce7675db 640instead of the character length. For that use either the
2bbc8d55
SP
641C<Encode::encode_utf8()> function or the C<bytes> pragma and
642the C<length()> function:
ba62762e
JH
643
644 my $unicode = chr(0x100);
645 print length($unicode), "\n"; # will print 1
ce7675db
A
646 require Encode;
647 print length(Encode::encode_utf8($unicode)), "\n"; # will print 2
ba62762e 648 use bytes;
1bfb14c4
JH
649 print length($unicode), "\n"; # will also print 2
650 # (the 0xC4 0x80 of the UTF-8)
0eb9ada1 651 no bytes;
ba62762e 652
818c4caa 653=item *
5cb3728c
RB
654
655How Do I Detect Data That's Not Valid In a Particular Encoding?
ba62762e 656
8baee566
JF
657Use the C<Encode> package to try converting it.
658For example,
ba62762e 659
bb2f379c 660 use Encode 'decode_utf8';
2bbc8d55 661
228ee848 662 if (eval { decode_utf8($string, Encode::FB_CROAK); 1 }) {
a365f2ce 663 # $string is valid utf8
ba62762e 664 } else {
a365f2ce 665 # $string is not valid utf8
ba62762e
JH
666 }
667
f337b084 668Or use C<unpack> to try decoding it:
ba62762e
JH
669
670 use warnings;
f337b084 671 @chars = unpack("C0U*", $string_of_bytes_that_I_think_is_utf8);
ba62762e 672
ae5648b3
RGS
673If invalid, a C<Malformed UTF-8 character> warning is produced. The "C0" means
674"process the string character per character". Without that, the
675C<unpack("U*", ...)> would work in C<U0> mode (the default if the format
676string starts with C<U>) and it would return the bytes making up the UTF-8
f337b084 677encoding of the target string, something that will always work.
ba62762e 678
818c4caa 679=item *
5cb3728c
RB
680
681How Do I Convert Binary Data Into a Particular Encoding, Or Vice Versa?
ba62762e 682
8baee566
JF
683This probably isn't as useful as you might think.
684Normally, you shouldn't need to.
ba62762e 685
1bfb14c4 686In one sense, what you are asking doesn't make much sense: encodings
376d9008 687are for characters, and binary data are not "characters", so converting
a5f0baef
JH
688"data" into some encoding isn't meaningful unless you know in what
689character set and encoding the binary data is in, in which case it's
376d9008 690not just binary data, now is it?
8baee566 691
1bfb14c4
JH
692If you have a raw sequence of bytes that you know should be
693interpreted via a particular encoding, you can use C<Encode>:
ba62762e
JH
694
695 use Encode 'from_to';
696 from_to($data, "iso-8859-1", "utf-8"); # from latin-1 to utf-8
697
1bfb14c4
JH
698The call to C<from_to()> changes the bytes in C<$data>, but nothing
699material about the nature of the string has changed as far as Perl is
700concerned. Both before and after the call, the string C<$data>
701contains just a bunch of 8-bit bytes. As far as Perl is concerned,
702the encoding of the string remains as "system-native 8-bit bytes".
8baee566
JF
703
704You might relate this to a fictional 'Translate' module:
705
706 use Translate;
707 my $phrase = "Yes";
708 Translate::from_to($phrase, 'english', 'deutsch');
709 ## phrase now contains "Ja"
ba62762e 710
8baee566 711The contents of the string changes, but not the nature of the string.
1bfb14c4
JH
712Perl doesn't know any more after the call than before that the
713contents of the string indicates the affirmative.
ba62762e 714
376d9008 715Back to converting data. If you have (or want) data in your system's
a5f0baef
JH
716native 8-bit encoding (e.g. Latin-1, EBCDIC, etc.), you can use
717pack/unpack to convert to/from Unicode.
ba62762e 718
f337b084
TH
719 $native_string = pack("W*", unpack("U*", $Unicode_string));
720 $Unicode_string = pack("U*", unpack("W*", $native_string));
ba62762e
JH
721
722If you have a sequence of bytes you B<know> is valid UTF-8,
723but Perl doesn't know it yet, you can make Perl a believer, too:
724
725 use Encode 'decode_utf8';
8baee566 726 $Unicode = decode_utf8($bytes);
ba62762e 727
f337b084
TH
728or:
729
730 $Unicode = pack("U0a*", $bytes);
ae5648b3 731
2bbc8d55
SP
732You can find the bytes that make up a UTF-8 sequence with
733
734 @bytes = unpack("C*", $Unicode_string)
735
736and you can create well-formed Unicode with
737
738 $Unicode_string = pack("U*", 0xff, ...)
ba62762e 739
818c4caa 740=item *
5cb3728c
RB
741
742How Do I Display Unicode? How Do I Input Unicode?
ba62762e 743
2bbc8d55
SP
744See L<http://www.alanwood.net/unicode/> and
745L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e 746
818c4caa 747=item *
5cb3728c
RB
748
749How Does Unicode Work With Traditional Locales?
ba62762e
JH
750
751In Perl, not very well. Avoid using locales through the C<locale>
4c496f0c
JH
752pragma. Use only one or the other. But see L<perlrun> for the
753description of the C<-C> switch and its environment counterpart,
754C<$ENV{PERL_UNICODE}> to see how to enable various Unicode features,
755for example by using locale settings.
ba62762e
JH
756
757=back
758
759=head2 Hexadecimal Notation
760
376d9008
JB
761The Unicode standard prefers using hexadecimal notation because
762that more clearly shows the division of Unicode into blocks of 256 characters.
ba62762e
JH
763Hexadecimal is also simply shorter than decimal. You can use decimal
764notation, too, but learning to use hexadecimal just makes life easier
1bfb14c4 765with the Unicode standard. The C<U+HHHH> notation uses hexadecimal,
076d825e 766for example.
ba62762e
JH
767
768The C<0x> prefix means a hexadecimal number, the digits are 0-9 I<and>
769a-f (or A-F, case doesn't matter). Each hexadecimal digit represents
770four bits, or half a byte. C<print 0x..., "\n"> will show a
771hexadecimal number in decimal, and C<printf "%x\n", $decimal> will
772show a decimal number in hexadecimal. If you have just the
376d9008 773"hex digits" of a hexadecimal number, you can use the C<hex()> function.
ba62762e
JH
774
775 print 0x0009, "\n"; # 9
776 print 0x000a, "\n"; # 10
777 print 0x000f, "\n"; # 15
778 print 0x0010, "\n"; # 16
779 print 0x0011, "\n"; # 17
780 print 0x0100, "\n"; # 256
781
782 print 0x0041, "\n"; # 65
783
784 printf "%x\n", 65; # 41
785 printf "%#x\n", 65; # 0x41
786
787 print hex("41"), "\n"; # 65
788
789=head2 Further Resources
790
791=over 4
792
793=item *
794
795Unicode Consortium
796
2bbc8d55 797L<http://www.unicode.org/>
ba62762e
JH
798
799=item *
800
801Unicode FAQ
802
2bbc8d55 803L<http://www.unicode.org/unicode/faq/>
ba62762e
JH
804
805=item *
806
807Unicode Glossary
808
2bbc8d55 809L<http://www.unicode.org/glossary/>
ba62762e
JH
810
811=item *
812
813Unicode Useful Resources
814
2bbc8d55 815L<http://www.unicode.org/unicode/onlinedat/resources.html>
ba62762e
JH
816
817=item *
818
819Unicode and Multilingual Support in HTML, Fonts, Web Browsers and Other Applications
820
2bbc8d55 821L<http://www.alanwood.net/unicode/>
ba62762e
JH
822
823=item *
824
825UTF-8 and Unicode FAQ for Unix/Linux
826
2bbc8d55 827L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e
JH
828
829=item *
830
831Legacy Character Sets
832
2bbc8d55
SP
833L<http://www.czyborra.com/>
834L<http://www.eki.ee/letter/>
ba62762e
JH
835
836=item *
837
838The Unicode support files live within the Perl installation in the
839directory
840
841 $Config{installprivlib}/unicore
842
ae5648b3 843in Perl 5.8.0 or newer, and
ba62762e
JH
844
845 $Config{installprivlib}/unicode
846
847in the Perl 5.6 series. (The renaming to F<lib/unicore> was done to
848avoid naming conflicts with lib/Unicode in case-insensitive filesystems.)
551b6b6f 849The main Unicode data file is F<UnicodeData.txt> (or F<Unicode.301> in
ba62762e
JH
850Perl 5.6.1.) You can find the C<$Config{installprivlib}> by
851
852 perl "-V:installprivlib"
853
ba62762e
JH
854You can explore various information from the Unicode data files using
855the C<Unicode::UCD> module.
856
857=back
858
f6edf83b
JH
859=head1 UNICODE IN OLDER PERLS
860
861If you cannot upgrade your Perl to 5.8.0 or later, you can still
862do some Unicode processing by using the modules C<Unicode::String>,
863C<Unicode::Map8>, and C<Unicode::Map>, available from CPAN.
864If you have the GNU recode installed, you can also use the
376d9008 865Perl front-end C<Convert::Recode> for character conversions.
f6edf83b 866
aaef10c5 867The following are fast conversions from ISO 8859-1 (Latin-1) bytes
63de3cb2 868to UTF-8 bytes and back, the code works even with older Perl 5 versions.
aaef10c5
JH
869
870 # ISO 8859-1 to UTF-8
871 s/([\x80-\xFF])/chr(0xC0|ord($1)>>6).chr(0x80|ord($1)&0x3F)/eg;
872
873 # UTF-8 to ISO 8859-1
874 s/([\xC2\xC3])([\x80-\xBF])/chr(ord($1)<<6&0xC0|ord($2)&0x3F)/eg;
875
ba62762e
JH
876=head1 SEE ALSO
877
2575c402 878L<perlunitut>, L<perlunicode>, L<Encode>, L<open>, L<utf8>, L<bytes>,
4c496f0c
JH
879L<perlretut>, L<perlrun>, L<Unicode::Collate>, L<Unicode::Normalize>,
880L<Unicode::UCD>
ba62762e 881
376d9008 882=head1 ACKNOWLEDGMENTS
ba62762e
JH
883
884Thanks to the kind readers of the perl5-porters@perl.org,
885perl-unicode@perl.org, linux-utf8@nl.linux.org, and unicore@unicode.org
886mailing lists for their valuable feedback.
887
888=head1 AUTHOR, COPYRIGHT, AND LICENSE
889
0f2f9b7d 890Copyright 2001-2002 Jarkko Hietaniemi E<lt>jhi@iki.fiE<gt>
ba62762e
JH
891
892This document may be distributed under the same terms as Perl itself.