This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
perluniintro: revise text on blocks vs scripts
[perl5.git] / pod / perluniintro.pod
CommitLineData
ba62762e
JH
1=head1 NAME
2
07fcf8ff 3perluniintro - Perl Unicode introduction
ba62762e
JH
4
5=head1 DESCRIPTION
6
7This document gives a general idea of Unicode and how to use Unicode
c8695642
KW
8in Perl. See L</Further Resources> for references to more in-depth
9treatments of Unicode.
ba62762e
JH
10
11=head2 Unicode
12
376d9008 13Unicode is a character set standard which plans to codify all of the
ba62762e
JH
14writing systems of the world, plus many other symbols.
15
c8695642
KW
16Unicode and ISO/IEC 10646 are coordinated standards that unify
17almost all other modern character set standards,
18covering more than 80 writing systems and hundreds of languages,
376d9008 19including all commercially-important modern languages. All characters
ba62762e
JH
20in the largest Chinese, Japanese, and Korean dictionaries are also
21encoded. The standards will eventually cover almost all characters in
22more than 250 writing systems and thousands of languages.
c8695642 23Unicode 1.0 was released in October 1991, and 6.0 in October 2010.
ba62762e
JH
24
25A Unicode I<character> is an abstract entity. It is not bound to any
376d9008
JB
26particular integer width, especially not to the C language C<char>.
27Unicode is language-neutral and display-neutral: it does not encode the
e1b711da 28language of the text, and it does not generally define fonts or other graphical
ba62762e
JH
29layout details. Unicode operates on characters and on text built from
30those characters.
31
32Unicode defines characters like C<LATIN CAPITAL LETTER A> or C<GREEK
376d9008
JB
33SMALL LETTER ALPHA> and unique numbers for the characters, in this
34case 0x0041 and 0x03B1, respectively. These unique numbers are called
c8695642
KW
35I<code points>. A code point is essentially the position of the
36character within the set of all possible Unicode characters, and thus in
37Perl, the term I<ordinal> is often used interchangeably with it.
ba62762e
JH
38
39The Unicode standard prefers using hexadecimal notation for the code
4c496f0c
JH
40points. If numbers like C<0x0041> are unfamiliar to you, take a peek
41at a later section, L</"Hexadecimal Notation">. The Unicode standard
42uses the notation C<U+0041 LATIN CAPITAL LETTER A>, to give the
43hexadecimal code point and the normative name of the character.
ba62762e
JH
44
45Unicode also defines various I<properties> for the characters, like
376d9008 46"uppercase" or "lowercase", "decimal digit", or "punctuation";
ba62762e
JH
47these properties are independent of the names of the characters.
48Furthermore, various operations on the characters like uppercasing,
376d9008 49lowercasing, and collating (sorting) are defined.
ba62762e 50
0111a78f
KW
51A Unicode I<logical> "character" can actually consist of more than one internal
52I<actual> "character" or code point. For Western languages, this is adequately
c670e63a 53modelled by a I<base character> (like C<LATIN CAPITAL LETTER A>) followed
0111a78f 54by one or more I<modifiers> (like C<COMBINING ACUTE ACCENT>). This sequence of
376d9008 55base character and modifiers is called a I<combining character
0111a78f 56sequence>. Some non-western languages require more complicated
c8695642
KW
57models, so Unicode created the I<grapheme cluster> concept, which was
58later further refined into the I<extended grapheme cluster>. For
59example, a Korean Hangul syllable is considered a single logical
60character, but most often consists of three actual
c670e63a
KW
61Unicode characters: a leading consonant followed by an interior vowel followed
62by a trailing consonant.
0111a78f
KW
63
64Whether to call these extended grapheme clusters "characters" depends on your
65point of view. If you are a programmer, you probably would tend towards seeing
c8695642
KW
66each element in the sequences as one unit, or "character". However from
67the user's point of view, the whole sequence could be seen as one
68"character" since that's probably what it looks like in the context of the
69user's language. In this document, we take the programmer's point of
70view: one "character" is one Unicode code point.
71
72For some combinations of base character and modifiers, there are
73I<precomposed> characters. There is a single character equivalent, for
74example, to the sequence C<LATIN CAPITAL LETTER A> followed by
75C<COMBINING ACUTE ACCENT>. It is called C<LATIN CAPITAL LETTER A WITH
76ACUTE>. These precomposed characters are, however, only available for
77some combinations, and are mainly meant to support round-trip
78conversions between Unicode and legacy standards (like ISO 8859). Using
79sequences, as Unicode does, allows for needing fewer basic building blocks
80(code points) to express many more potential grapheme clusters. To
81support conversion between equivalent forms, various I<normalization
82forms> are also defined. Thus, C<LATIN CAPITAL LETTER A WITH ACUTE> is
83in I<Normalization Form Composed>, (abbreviated NFC), and the sequence
84C<LATIN CAPITAL LETTER A> followed by C<COMBINING ACUTE ACCENT>
85represents the same character in I<Normalization Form Decomposed> (NFD).
ba62762e
JH
86
87Because of backward compatibility with legacy encodings, the "a unique
376d9008
JB
88number for every character" idea breaks down a bit: instead, there is
89"at least one number for every character". The same character could
90be represented differently in several legacy encodings. The
35146e33 91converse is not also true: some code points do not have an assigned
376d9008
JB
92character. Firstly, there are unallocated code points within
93otherwise used blocks. Secondly, there are special Unicode control
94characters that do not represent true characters.
ba62762e 95
e9078126
KW
96When Unicode was first conceived, it was thought that all the world's
97characters could be represented using a 16-bit word; that is a maximum of
98C<0x10000> (or 65536) characters from C<0x0000> to C<0xFFFF> would be
99needed. This soon proved to be false, and since Unicode 2.0 (July
4c496f0c 1001996), Unicode has been defined all the way up to 21 bits (C<0x10FFFF>),
e9078126
KW
101and Unicode 3.1 (March 2001) defined the first characters above C<0xFFFF>.
102The first C<0x10000> characters are called the I<Plane 0>, or the
103I<Basic Multilingual Plane> (BMP). With Unicode 3.1, 17 (yes,
104seventeen) planes in all were defined--but they are nowhere near full of
105defined characters, yet.
106
107When a new language is being encoded, Unicode generally will choose a
108C<block> of consecutive unallocated code points for its characters. So
109far, the number of code points in these blocks has always been evenly
110divisible by 16. Extras in a block, not currently needed, are left
111unallocated, for future growth. But there have been occasions when
112a later relase needed more code points than available extras, and a new
113block had to allocated somewhere else, not contiguous to the initial one
114to handle the overflow. Thus, it became apparent early on that "block"
115wasn't an adequate organizing principal, and so the C<script> property
116was created. Those code points that are in overflow blocks can still
117have the same script as the original ones. The script concept fits more
118closely with natural language: there is C<Latin> script, C<Greek>
119script, and so on; and there are several artificial scripts, like
120C<Common> for characters that are used in multiple scripts, such as
121mathematical symbols. Scripts usually span varied parts of several
122blocks. For more information about scripts, see L<perlunicode/Scripts>.
1bfb14c4 123The division into blocks exists, but it is almost completely
e9078126
KW
124accidental--an artifact of how the characters have been and still are
125allocated. (Note that this paragraph has oversimplified things for the
126sake of this being an introduction. Unicode doesn't really encode
127languages, but the writing systems for them--their scripts; and one
128script can be used by many languages. Unicode also encodes things that
129aren't really about languages, such as symbols like C<BAGGAGE CLAIM>.)
ba62762e
JH
130
131The Unicode code points are just abstract numbers. To input and
4c496f0c
JH
132output these abstract numbers, the numbers must be I<encoded> or
133I<serialised> somehow. Unicode defines several I<character encoding
134forms>, of which I<UTF-8> is perhaps the most popular. UTF-8 is a
135variable length encoding that encodes Unicode characters as 1 to 6
e1b711da 136bytes. Other encodings
4c496f0c
JH
137include UTF-16 and UTF-32 and their big- and little-endian variants
138(UTF-8 is byte-order independent) The ISO/IEC 10646 defines the UCS-2
139and UCS-4 encoding forms.
ba62762e 140
376d9008
JB
141For more information about encodings--for instance, to learn what
142I<surrogates> and I<byte order marks> (BOMs) are--see L<perlunicode>.
ba62762e
JH
143
144=head2 Perl's Unicode Support
145
376d9008
JB
146Starting from Perl 5.6.0, Perl has had the capacity to handle Unicode
147natively. Perl 5.8.0, however, is the first recommended release for
148serious Unicode work. The maintenance release 5.6.1 fixed many of the
149problems of the initial Unicode implementation, but for example
1bfb14c4 150regular expressions still do not work with Unicode in 5.6.1.
c8695642
KW
151Perl 5.14.0 is the first release where Unicode support is
152(almost) seamlessly integrable without some gotchas (the exception being
153some differences in L<quotemeta|perlfunc/quotemeta>). To enable this
154seamless support, you should C<use feature 'unicode_strings'> (which is
155automatically selected if you C<use 5.012> or higher). See L<feature>.
156(5.14 also fixes a number of bugs and departures from the Unicode
157standard.)
158
159Before Perl 5.8.0, the use of C<use utf8> was used to declare
ba62762e 160that operations in the current block or file would be Unicode-aware.
376d9008 161This model was found to be wrong, or at least clumsy: the "Unicodeness"
1bfb14c4 162is now carried with the data, instead of being attached to the
c8695642
KW
163operations.
164Starting with Perl 5.8.0, only one case remains where an explicit C<use
165utf8> is needed: if your Perl script itself is encoded in UTF-8, you can
166use UTF-8 in your identifier names, and in string and regular expression
376d9008 167literals, by saying C<use utf8>. This is not the default because
8f8cf39c 168scripts with legacy 8-bit data in them would break. See L<utf8>.
ba62762e
JH
169
170=head2 Perl's Unicode Model
171
376d9008 172Perl supports both pre-5.6 strings of eight-bit native bytes, and
c8695642
KW
173strings of Unicode characters. The general principle is that Perl tries
174to keep its data as eight-bit bytes for as long as possible, but as soon
175as Unicodeness cannot be avoided, the data is transparently upgraded
176to Unicode. Prior to Perl 5.14, the upgrade was not completely
177transparent (see L<perlunicode/The "Unicode Bug">), and for backwards
178compatibility, full transparency is not gained unless C<use feature
179'unicode_strings'> (see L<feature>) or C<use 5.012> (or higher) is
180selected.
ba62762e 181
4192de81 182Internally, Perl currently uses either whatever the native eight-bit
376d9008
JB
183character set of the platform (for example Latin-1) is, defaulting to
184UTF-8, to encode Unicode strings. Specifically, if all code points in
185the string are C<0xFF> or less, Perl uses the native eight-bit
186character set. Otherwise, it uses UTF-8.
4192de81 187
7ca610e8 188A user of Perl does not normally need to know nor care how Perl
20ba30f4 189happens to encode its internal strings, but it becomes relevant when
ac036724 190outputting Unicode strings to a stream without a PerlIO layer (one with
191the "default" encoding). In such a case, the raw bytes used internally
7ca610e8
JH
192(the native character set or UTF-8, as appropriate for each string)
193will be used, and a "Wide character" warning will be issued if those
194strings contain a character beyond 0x00FF.
4192de81
JH
195
196For example,
197
ae5648b3 198 perl -e 'print "\x{DF}\n", "\x{0100}\x{DF}\n"'
4192de81
JH
199
200produces a fairly useless mixture of native bytes and UTF-8, as well
1bfb14c4 201as a warning:
4192de81 202
1bfb14c4
JH
203 Wide character in print at ...
204
740d4bb2 205To output UTF-8, use the C<:encoding> or C<:utf8> output layer. Prepending
4192de81
JH
206
207 binmode(STDOUT, ":utf8");
208
376d9008 209to this sample program ensures that the output is completely UTF-8,
1bfb14c4 210and removes the program's warning.
ba62762e 211
8aa8f774
JH
212You can enable automatic UTF-8-ification of your standard file
213handles, default C<open()> layer, and C<@ARGV> by using either
214the C<-C> command line switch or the C<PERL_UNICODE> environment
215variable, see L<perlrun> for the documentation of the C<-C> switch.
216
c8695642
KW
217Note that this means that Perl expects other software to work the same
218way:
8aa8f774 219if Perl has been led to believe that STDIN should be UTF-8, but then
c8695642
KW
220STDIN coming in from another command is not UTF-8, Perl will likely
221complain about the malformed UTF-8.
b310b053 222
ec71e770
JH
223All features that combine Unicode and I/O also require using the new
224PerlIO feature. Almost all Perl 5.8 platforms do use PerlIO, though:
225you can see whether yours is by running "perl -V" and looking for
226C<useperlio=define>.
227
64c66fb6
JH
228=head2 Unicode and EBCDIC
229
230Perl 5.8.0 also supports Unicode on EBCDIC platforms. There,
376d9008 231Unicode support is somewhat more complex to implement since
e1b711da
KW
232additional conversions are needed at every step.
233
234Later Perl releases have added code that will not work on EBCDIC platforms, and
235no one has complained, so the divergence has continued. If you want to run
236Perl on an EBCDIC platform, send email to perlbug@perl.org
64c66fb6 237
64c66fb6 238On EBCDIC platforms, the internal Unicode encoding form is UTF-EBCDIC
376d9008
JB
239instead of UTF-8. The difference is that as UTF-8 is "ASCII-safe" in
240that ASCII characters encode to UTF-8 as-is, while UTF-EBCDIC is
241"EBCDIC-safe".
ba62762e
JH
242
243=head2 Creating Unicode
244
376d9008
JB
245To create Unicode characters in literals for code points above C<0xFF>,
246use the C<\x{...}> notation in double-quoted strings:
ba62762e
JH
247
248 my $smiley = "\x{263a}";
249
376d9008 250Similarly, it can be used in regular expression literals
ba62762e
JH
251
252 $smiley =~ /\x{263a}/;
253
254At run-time you can use C<chr()>:
255
256 my $hebrew_alef = chr(0x05d0);
257
376d9008 258See L</"Further Resources"> for how to find all these numeric codes.
ba62762e 259
376d9008
JB
260Naturally, C<ord()> will do the reverse: it turns a character into
261a code point.
ba62762e 262
1bfb14c4
JH
263Note that C<\x..> (no C<{}> and only two hexadecimal digits), C<\x{...}>,
264and C<chr(...)> for arguments less than C<0x100> (decimal 256)
265generate an eight-bit character for backward compatibility with older
266Perls. For arguments of C<0x100> or more, Unicode characters are
a5f0baef
JH
267always produced. If you want to force the production of Unicode
268characters regardless of the numeric value, use C<pack("U", ...)>
269instead of C<\x..>, C<\x{...}>, or C<chr()>.
ba62762e
JH
270
271You can also use the C<charnames> pragma to invoke characters
376d9008 272by name in double-quoted strings:
ba62762e
JH
273
274 use charnames ':full';
275 my $arabic_alef = "\N{ARABIC LETTER ALEF}";
276
277And, as mentioned above, you can also C<pack()> numbers into Unicode
278characters:
279
280 my $georgian_an = pack("U", 0x10a0);
281
8a5e5dd5
JH
282Note that both C<\x{...}> and C<\N{...}> are compile-time string
283constants: you cannot use variables in them. if you want similar
fb121860 284run-time functionality, use C<chr()> and C<charnames::string_vianame()>.
8a5e5dd5 285
1eda90df 286If you want to force the result to Unicode characters, use the special
f337b084
TH
287C<"U0"> prefix. It consumes no arguments but causes the following bytes
288to be interpreted as the UTF-8 encoding of Unicode characters:
1eda90df 289
f337b084 290 my $chars = pack("U0W*", 0x80, 0x42);
771cd3b2 291
ae5648b3 292Likewise, you can stop such UTF-8 interpretation by using the special
771cd3b2 293C<"C0"> prefix.
1eda90df 294
ba62762e
JH
295=head2 Handling Unicode
296
297Handling Unicode is for the most part transparent: just use the
298strings as usual. Functions like C<index()>, C<length()>, and
299C<substr()> will work on the Unicode characters; regular expressions
300will work on the Unicode characters (see L<perlunicode> and L<perlretut>).
301
0111a78f
KW
302Note that Perl considers grapheme clusters to be separate characters, so for
303example
ba62762e
JH
304
305 use charnames ':full';
306 print length("\N{LATIN CAPITAL LETTER A}\N{COMBINING ACUTE ACCENT}"), "\n";
307
308will print 2, not 1. The only exception is that regular expressions
c8695642
KW
309have C<\X> for matching an extended grapheme cluster. (Thus C<\X> in a
310regular expression would match the entire sequence of both the example
311characters.)
ba62762e 312
376d9008
JB
313Life is not quite so transparent, however, when working with legacy
314encodings, I/O, and certain special cases:
ba62762e
JH
315
316=head2 Legacy Encodings
317
c8695642
KW
318When you combine legacy data and Unicode, the legacy data needs
319to be upgraded to Unicode. Normally the legacy data is assumed to be
320ISO 8859-1 (or EBCDIC, if applicable).
ba62762e 321
376d9008 322The C<Encode> module knows about many encodings and has interfaces
ba62762e
JH
323for doing conversions between those encodings:
324
2c9359a2
RGS
325 use Encode 'decode';
326 $data = decode("iso-8859-3", $data); # convert from legacy to utf-8
ba62762e
JH
327
328=head2 Unicode I/O
329
8baee566 330Normally, writing out Unicode data
ba62762e 331
8baee566 332 print FH $some_string_with_unicode, "\n";
ba62762e 333
8baee566 334produces raw bytes that Perl happens to use to internally encode the
376d9008
JB
335Unicode string. Perl's internal encoding depends on the system as
336well as what characters happen to be in the string at the time. If
337any of the characters are at code points C<0x100> or above, you will get
338a warning. To ensure that the output is explicitly rendered in the
339encoding you desire--and to avoid the warning--open the stream with
340the desired encoding. Some examples:
ba62762e 341
ec90690f
TS
342 open FH, ">:utf8", "file";
343
344 open FH, ">:encoding(ucs2)", "file";
345 open FH, ">:encoding(UTF-8)", "file";
346 open FH, ">:encoding(shift_jis)", "file";
1d7919c5 347
376d9008 348and on already open streams, use C<binmode()>:
1d7919c5
JH
349
350 binmode(STDOUT, ":utf8");
351
ec90690f
TS
352 binmode(STDOUT, ":encoding(ucs2)");
353 binmode(STDOUT, ":encoding(UTF-8)");
354 binmode(STDOUT, ":encoding(shift_jis)");
355
b5d8778e 356The matching of encoding names is loose: case does not matter, and
fae2c0fb 357many encodings have several aliases. Note that the C<:utf8> layer
1bfb14c4 358must always be specified exactly like that; it is I<not> subject to
c8695642 359the loose matching of encoding names. Also note that currently C<:utf8> is unsafe for
740d4bb2 360input, because it accepts the data without validating that it is indeed valid
367b3305
KW
361UTF-8; you should instead use C<:encoding(utf-8)> (with or without a
362hyphen).
b5d8778e 363
376d9008
JB
364See L<PerlIO> for the C<:utf8> layer, L<PerlIO::encoding> and
365L<Encode::PerlIO> for the C<:encoding()> layer, and
366L<Encode::Supported> for many encodings supported by the C<Encode>
367module.
ba62762e 368
a5f0baef 369Reading in a file that you know happens to be encoded in one of the
1bfb14c4
JH
370Unicode or legacy encodings does not magically turn the data into
371Unicode in Perl's eyes. To do that, specify the appropriate
fae2c0fb 372layer when opening files
ba62762e 373
740d4bb2 374 open(my $fh,'<:encoding(utf8)', 'anything');
8baee566
JF
375 my $line_of_unicode = <$fh>;
376
ec90690f 377 open(my $fh,'<:encoding(Big5)', 'anything');
8baee566 378 my $line_of_unicode = <$fh>;
ba62762e 379
fae2c0fb 380The I/O layers can also be specified more flexibly with
376d9008 381the C<open> pragma. See L<open>, or look at the following example.
ba62762e 382
9e5bbba0
KW
383 use open ':encoding(utf8)'; # input/output default encoding will be
384 # UTF-8
1d7919c5
JH
385 open X, ">file";
386 print X chr(0x100), "\n";
ba62762e 387 close X;
1d7919c5 388 open Y, "<file";
ba62762e
JH
389 printf "%#x\n", ord(<Y>); # this should print 0x100
390 close Y;
391
fae2c0fb 392With the C<open> pragma you can use the C<:locale> layer
ba62762e 393
12f98225 394 BEGIN { $ENV{LC_ALL} = $ENV{LANG} = 'ru_RU.KOI8-R' }
9e5bbba0
KW
395 # the :locale will probe the locale environment variables like
396 # LC_ALL
ba62762e
JH
397 use open OUT => ':locale'; # russki parusski
398 open(O, ">koi8");
399 print O chr(0x430); # Unicode CYRILLIC SMALL LETTER A = KOI8-R 0xc1
400 close O;
401 open(I, "<koi8");
402 printf "%#x\n", ord(<I>), "\n"; # this should print 0xc1
403 close I;
404
8baee566
JF
405These methods install a transparent filter on the I/O stream that
406converts data from the specified encoding when it is read in from the
a5f0baef 407stream. The result is always Unicode.
ba62762e
JH
408
409The L<open> pragma affects all the C<open()> calls after the pragma by
fae2c0fb
RGS
410setting default layers. If you want to affect only certain
411streams, use explicit layers directly in the C<open()> call.
ba62762e
JH
412
413You can switch encodings on an already opened stream by using
8baee566 414C<binmode()>; see L<perlfunc/binmode>.
ba62762e 415
1ecefa54
JH
416The C<:locale> does not currently (as of Perl 5.8.0) work with
417C<open()> and C<binmode()>, only with the C<open> pragma. The
8baee566 418C<:utf8> and C<:encoding(...)> methods do work with all of C<open()>,
1ecefa54 419C<binmode()>, and the C<open> pragma.
ba62762e 420
fae2c0fb 421Similarly, you may use these I/O layers on output streams to
a5f0baef
JH
422automatically convert Unicode to the specified encoding when it is
423written to the stream. For example, the following snippet copies the
424contents of the file "text.jis" (encoded as ISO-2022-JP, aka JIS) to
425the file "text.utf8", encoded as UTF-8:
ba62762e 426
c722ef0a
JH
427 open(my $nihongo, '<:encoding(iso-2022-jp)', 'text.jis');
428 open(my $unicode, '>:utf8', 'text.utf8');
0cf8a8d9 429 while (<$nihongo>) { print $unicode $_ }
ba62762e
JH
430
431The naming of encodings, both by the C<open()> and by the C<open>
2575c402
JW
432pragma allows for flexible names: C<koi8-r> and C<KOI8R> will both be
433understood.
ba62762e
JH
434
435Common encodings recognized by ISO, MIME, IANA, and various other
8baee566 436standardisation organisations are recognised; for a more detailed
1bfb14c4 437list see L<Encode::Supported>.
ba62762e
JH
438
439C<read()> reads characters and returns the number of characters.
440C<seek()> and C<tell()> operate on byte counts, as do C<sysread()>
441and C<sysseek()>.
442
8baee566 443Notice that because of the default behaviour of not doing any
fae2c0fb 444conversion upon input if there is no default layer,
ba62762e 445it is easy to mistakenly write code that keeps on expanding a file
1bfb14c4 446by repeatedly encoding the data:
ba62762e
JH
447
448 # BAD CODE WARNING
449 open F, "file";
8baee566 450 local $/; ## read in the whole file of 8-bit characters
ba62762e
JH
451 $t = <F>;
452 close F;
740d4bb2 453 open F, ">:encoding(utf8)", "file";
8baee566 454 print F $t; ## convert to UTF-8 on output
ba62762e
JH
455 close F;
456
457If you run this code twice, the contents of the F<file> will be twice
740d4bb2
JW
458UTF-8 encoded. A C<use open ':encoding(utf8)'> would have avoided the
459bug, or explicitly opening also the F<file> for input as UTF-8.
ba62762e 460
0c901d84 461B<NOTE>: the C<:utf8> and C<:encoding> features work only if your
d0fadae5
JH
462Perl has been built with the new PerlIO feature (which is the default
463on most systems).
0c901d84 464
1ecefa54
JH
465=head2 Displaying Unicode As Text
466
467Sometimes you might want to display Perl scalars containing Unicode as
8baee566 468simple ASCII (or EBCDIC) text. The following subroutine converts
1ecefa54 469its argument so that Unicode characters with code points greater than
1bfb14c4
JH
470255 are displayed as C<\x{...}>, control characters (like C<\n>) are
471displayed as C<\x..>, and the rest of the characters as themselves:
1ecefa54 472
9e5bbba0
KW
473 sub nice_string {
474 join("",
475 map { $_ > 255 ? # if wide character...
476 sprintf("\\x{%04X}", $_) : # \x{...}
477 chr($_) =~ /[[:cntrl:]]/ ? # else if control character ...
478 sprintf("\\x%02X", $_) : # \x..
479 quotemeta(chr($_)) # else quoted or as themselves
f337b084 480 } unpack("W*", $_[0])); # unpack Unicode characters
58c274a1
JF
481 }
482
483For example,
484
485 nice_string("foo\x{100}bar\n")
486
d0551e73 487returns the string
58c274a1 488
d0551e73
JH
489 'foo\x{0100}bar\x0A'
490
491which is ready to be printed.
1ecefa54 492
ba62762e
JH
493=head2 Special Cases
494
495=over 4
496
497=item *
498
499Bit Complement Operator ~ And vec()
500
1bfb14c4
JH
501The bit complement operator C<~> may produce surprising results if
502used on strings containing characters with ordinal values above
503255. In such a case, the results are consistent with the internal
504encoding of the characters, but not with much else. So don't do
505that. Similarly for C<vec()>: you will be operating on the
506internally-encoded bit patterns of the Unicode characters, not on
507the code point values, which is very probably not what you want.
ba62762e
JH
508
509=item *
510
8baee566
JF
511Peeking At Perl's Internal Encoding
512
513Normal users of Perl should never care how Perl encodes any particular
a5f0baef 514Unicode string (because the normal ways to get at the contents of a
376d9008 515string with Unicode--via input and output--should always be via
fae2c0fb 516explicitly-defined I/O layers). But if you must, there are two
a5f0baef 517ways of looking behind the scenes.
ba62762e
JH
518
519One way of peeking inside the internal encoding of Unicode characters
f337b084
TH
520is to use C<unpack("C*", ...> to get the bytes of whatever the string
521encoding happens to be, or C<unpack("U0..", ...)> to get the bytes of the
522UTF-8 encoding:
ba62762e 523
8baee566 524 # this prints c4 80 for the UTF-8 bytes 0xc4 0x80
f337b084 525 print join(" ", unpack("U0(H2)*", pack("U", 0x100))), "\n";
ba62762e
JH
526
527Yet another way would be to use the Devel::Peek module:
528
529 perl -MDevel::Peek -e 'Dump(chr(0x100))'
530
1e54db1a 531That shows the C<UTF8> flag in FLAGS and both the UTF-8 bytes
376d9008 532and Unicode characters in C<PV>. See also later in this document
8800c35a 533the discussion about the C<utf8::is_utf8()> function.
ba62762e
JH
534
535=back
536
537=head2 Advanced Topics
538
539=over 4
540
541=item *
542
543String Equivalence
544
545The question of string equivalence turns somewhat complicated
376d9008 546in Unicode: what do you mean by "equal"?
ba62762e 547
07698885
RGS
548(Is C<LATIN CAPITAL LETTER A WITH ACUTE> equal to
549C<LATIN CAPITAL LETTER A>?)
ba62762e 550
a5f0baef
JH
551The short answer is that by default Perl compares equivalence (C<eq>,
552C<ne>) based only on code points of the characters. In the above
376d9008 553case, the answer is no (because 0x00C1 != 0x0041). But sometimes, any
c8695642 554CAPITAL LETTER A's should be considered equal, or even A's of any case.
ba62762e
JH
555
556The long answer is that you need to consider character normalization
e1b711da
KW
557and casing issues: see L<Unicode::Normalize>, Unicode Technical Report #15,
558L<Unicode Normalization Forms|http://www.unicode.org/unicode/reports/tr15> and
559sections on case mapping in the L<Unicode Standard|http://www.unicode.org>.
ba62762e 560
1bfb14c4 561As of Perl 5.8.0, the "Full" case-folding of I<Case
afba1538
KW
562Mappings/SpecialCasing> is implemented, but bugs remain in C<qr//i> with them,
563mostly fixed by 5.14.
ba62762e
JH
564
565=item *
566
567String Collation
568
376d9008 569People like to see their strings nicely sorted--or as Unicode
ba62762e
JH
570parlance goes, collated. But again, what do you mean by collate?
571
07698885
RGS
572(Does C<LATIN CAPITAL LETTER A WITH ACUTE> come before or after
573C<LATIN CAPITAL LETTER A WITH GRAVE>?)
ba62762e 574
58c274a1 575The short answer is that by default, Perl compares strings (C<lt>,
ba62762e 576C<le>, C<cmp>, C<ge>, C<gt>) based only on the code points of the
1bfb14c4 577characters. In the above case, the answer is "after", since
da76a1f4 578C<0x00C1> > C<0x00C0>.
ba62762e
JH
579
580The long answer is that "it depends", and a good answer cannot be
581given without knowing (at the very least) the language context.
582See L<Unicode::Collate>, and I<Unicode Collation Algorithm>
2bbc8d55 583L<http://www.unicode.org/unicode/reports/tr10/>
ba62762e
JH
584
585=back
586
587=head2 Miscellaneous
588
589=over 4
590
591=item *
592
3ff56b75 593Character Ranges and Classes
ba62762e 594
0eb9ada1
KW
595Character ranges in regular expression bracketed character classes ( e.g.,
596C</[a-z]/>) and in the C<tr///> (also known as C<y///>) operator are not
597magically Unicode-aware. What this means is that C<[A-Za-z]> will not
598magically start to mean "all alphabetic letters" (not that it does mean that
599even for 8-bit characters; for those, if you are using locales (L<perllocale>),
600use C</[[:alpha:]]/>; and if not, use the 8-bit-aware property C<\p{alpha}>).
601
602All the properties that begin with C<\p> (and its inverse C<\P>) are actually
603character classes that are Unicode-aware. There are dozens of them, see
604L<perluniprops>.
605
606You can use Unicode code points as the end points of character ranges, and the
607range will include all Unicode code points that lie between those end points.
ba62762e
JH
608
609=item *
610
611String-To-Number Conversions
612
376d9008
JB
613Unicode does define several other decimal--and numeric--characters
614besides the familiar 0 to 9, such as the Arabic and Indic digits.
ba62762e 615Perl does not support string-to-number conversion for digits other
58c274a1 616than ASCII 0 to 9 (and ASCII a to f for hexadecimal).
c8695642
KW
617To get safe conversions from any Unicode string, use
618L<Unicode::UCD/num()>.
ba62762e
JH
619
620=back
621
622=head2 Questions With Answers
623
624=over 4
625
818c4caa 626=item *
5cb3728c
RB
627
628Will My Old Scripts Break?
ba62762e
JH
629
630Very probably not. Unless you are generating Unicode characters
1bfb14c4
JH
631somehow, old behaviour should be preserved. About the only behaviour
632that has changed and which could start generating Unicode is the old
633behaviour of C<chr()> where supplying an argument more than 255
634produced a character modulo 255. C<chr(300)>, for example, was equal
635to C<chr(45)> or "-" (in ASCII), now it is LATIN CAPITAL LETTER I WITH
636BREVE.
ba62762e 637
818c4caa 638=item *
5cb3728c
RB
639
640How Do I Make My Scripts Work With Unicode?
ba62762e
JH
641
642Very little work should be needed since nothing changes until you
1bfb14c4
JH
643generate Unicode data. The most important thing is getting input as
644Unicode; for that, see the earlier I/O discussion.
c8695642
KW
645To get full seamless Unicode support, add
646C<use feature 'unicode_strings'> (or C<use 5.012> or higher) to your
647script.
ba62762e 648
818c4caa 649=item *
5cb3728c
RB
650
651How Do I Know Whether My String Is In Unicode?
ba62762e 652
c8695642
KW
653You shouldn't have to care. But you may if your Perl is before 5.14.0
654or you haven't specified C<use feature 'unicode_strings'> or C<use
6555.012> (or higher) because otherwise the semantics of the code points
656in the range 128 to 255 are different depending on
2bbc8d55 657whether the string they are contained within is in Unicode or not.
e1b711da 658(See L<perlunicode/When Unicode Does Not Happen>.)
ba62762e 659
2bbc8d55 660To determine if a string is in Unicode, use:
ba62762e 661
8800c35a 662 print utf8::is_utf8($string) ? 1 : 0, "\n";
ba62762e
JH
663
664But note that this doesn't mean that any of the characters in the
665string are necessary UTF-8 encoded, or that any of the characters have
666code points greater than 0xFF (255) or even 0x80 (128), or that the
667string has any characters at all. All the C<is_utf8()> does is to
668return the value of the internal "utf8ness" flag attached to the
376d9008 669C<$string>. If the flag is off, the bytes in the scalar are interpreted
3c1c8017 670as a single byte encoding. If the flag is on, the bytes in the scalar
0eb9ada1
KW
671are interpreted as the (variable-length, potentially multi-byte) UTF-8 encoded
672code points of the characters. Bytes added to a UTF-8 encoded string are
1e54db1a 673automatically upgraded to UTF-8. If mixed non-UTF-8 and UTF-8 scalars
c8695642 674are merged (double-quoted interpolation, explicit concatenation, or
3c1c8017
AT
675printf/sprintf parameter substitution), the result will be UTF-8 encoded
676as if copies of the byte strings were upgraded to UTF-8: for example,
677
678 $a = "ab\x80c";
679 $b = "\x{100}";
680 print "$a = $b\n";
681
a02b5feb
JH
682the output string will be UTF-8-encoded C<ab\x80c = \x{100}\n>, but
683C<$a> will stay byte-encoded.
ba62762e
JH
684
685Sometimes you might really need to know the byte length of a string
ce7675db 686instead of the character length. For that use either the
00d823b9 687C<Encode::encode_utf8()> function or the C<bytes> pragma
c8695642 688and the C<length()> function:
ba62762e
JH
689
690 my $unicode = chr(0x100);
691 print length($unicode), "\n"; # will print 1
ce7675db
A
692 require Encode;
693 print length(Encode::encode_utf8($unicode)), "\n"; # will print 2
00d823b9 694 use bytes;
1bfb14c4
JH
695 print length($unicode), "\n"; # will also print 2
696 # (the 0xC4 0x80 of the UTF-8)
0eb9ada1 697 no bytes;
ba62762e 698
818c4caa 699=item *
5cb3728c 700
c8695642
KW
701How Do I Find Out What Encoding a File Has?
702
968ee499 703You might try L<Encode::Guess>, but it has a number of limitations.
c8695642
KW
704
705=item *
706
5cb3728c 707How Do I Detect Data That's Not Valid In a Particular Encoding?
ba62762e 708
8baee566
JF
709Use the C<Encode> package to try converting it.
710For example,
ba62762e 711
bb2f379c 712 use Encode 'decode_utf8';
2bbc8d55 713
228ee848 714 if (eval { decode_utf8($string, Encode::FB_CROAK); 1 }) {
a365f2ce 715 # $string is valid utf8
ba62762e 716 } else {
a365f2ce 717 # $string is not valid utf8
ba62762e
JH
718 }
719
f337b084 720Or use C<unpack> to try decoding it:
ba62762e
JH
721
722 use warnings;
f337b084 723 @chars = unpack("C0U*", $string_of_bytes_that_I_think_is_utf8);
ba62762e 724
ae5648b3
RGS
725If invalid, a C<Malformed UTF-8 character> warning is produced. The "C0" means
726"process the string character per character". Without that, the
727C<unpack("U*", ...)> would work in C<U0> mode (the default if the format
728string starts with C<U>) and it would return the bytes making up the UTF-8
f337b084 729encoding of the target string, something that will always work.
ba62762e 730
818c4caa 731=item *
5cb3728c
RB
732
733How Do I Convert Binary Data Into a Particular Encoding, Or Vice Versa?
ba62762e 734
8baee566
JF
735This probably isn't as useful as you might think.
736Normally, you shouldn't need to.
ba62762e 737
1bfb14c4 738In one sense, what you are asking doesn't make much sense: encodings
376d9008 739are for characters, and binary data are not "characters", so converting
a5f0baef
JH
740"data" into some encoding isn't meaningful unless you know in what
741character set and encoding the binary data is in, in which case it's
376d9008 742not just binary data, now is it?
8baee566 743
1bfb14c4
JH
744If you have a raw sequence of bytes that you know should be
745interpreted via a particular encoding, you can use C<Encode>:
ba62762e
JH
746
747 use Encode 'from_to';
748 from_to($data, "iso-8859-1", "utf-8"); # from latin-1 to utf-8
749
1bfb14c4
JH
750The call to C<from_to()> changes the bytes in C<$data>, but nothing
751material about the nature of the string has changed as far as Perl is
752concerned. Both before and after the call, the string C<$data>
753contains just a bunch of 8-bit bytes. As far as Perl is concerned,
754the encoding of the string remains as "system-native 8-bit bytes".
8baee566
JF
755
756You might relate this to a fictional 'Translate' module:
757
758 use Translate;
759 my $phrase = "Yes";
760 Translate::from_to($phrase, 'english', 'deutsch');
761 ## phrase now contains "Ja"
ba62762e 762
8baee566 763The contents of the string changes, but not the nature of the string.
1bfb14c4
JH
764Perl doesn't know any more after the call than before that the
765contents of the string indicates the affirmative.
ba62762e 766
376d9008 767Back to converting data. If you have (or want) data in your system's
a5f0baef
JH
768native 8-bit encoding (e.g. Latin-1, EBCDIC, etc.), you can use
769pack/unpack to convert to/from Unicode.
ba62762e 770
f337b084
TH
771 $native_string = pack("W*", unpack("U*", $Unicode_string));
772 $Unicode_string = pack("U*", unpack("W*", $native_string));
ba62762e
JH
773
774If you have a sequence of bytes you B<know> is valid UTF-8,
775but Perl doesn't know it yet, you can make Perl a believer, too:
776
777 use Encode 'decode_utf8';
8baee566 778 $Unicode = decode_utf8($bytes);
ba62762e 779
f337b084
TH
780or:
781
782 $Unicode = pack("U0a*", $bytes);
ae5648b3 783
2bbc8d55
SP
784You can find the bytes that make up a UTF-8 sequence with
785
9e5bbba0 786 @bytes = unpack("C*", $Unicode_string)
2bbc8d55
SP
787
788and you can create well-formed Unicode with
789
9e5bbba0 790 $Unicode_string = pack("U*", 0xff, ...)
ba62762e 791
818c4caa 792=item *
5cb3728c
RB
793
794How Do I Display Unicode? How Do I Input Unicode?
ba62762e 795
2bbc8d55
SP
796See L<http://www.alanwood.net/unicode/> and
797L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e 798
818c4caa 799=item *
5cb3728c
RB
800
801How Does Unicode Work With Traditional Locales?
ba62762e 802
afba1538
KW
803Perl tries to keep the two separated. Code points that are above 255
804are treated as Unicode; those below 256, generally as locale. This
805works reasonably well except in some case-insensitive regular expression
806pattern matches that in Unicode would cross the 255/256 boundary. These
807are disallowed.
808Also, the C<\p{}> and C<\N{}> constructs silently assume Unicode values
809even for code points below 256.
810See also L<perlrun> for the
4c496f0c
JH
811description of the C<-C> switch and its environment counterpart,
812C<$ENV{PERL_UNICODE}> to see how to enable various Unicode features,
813for example by using locale settings.
ba62762e
JH
814
815=back
816
817=head2 Hexadecimal Notation
818
376d9008
JB
819The Unicode standard prefers using hexadecimal notation because
820that more clearly shows the division of Unicode into blocks of 256 characters.
ba62762e
JH
821Hexadecimal is also simply shorter than decimal. You can use decimal
822notation, too, but learning to use hexadecimal just makes life easier
1bfb14c4 823with the Unicode standard. The C<U+HHHH> notation uses hexadecimal,
076d825e 824for example.
ba62762e
JH
825
826The C<0x> prefix means a hexadecimal number, the digits are 0-9 I<and>
827a-f (or A-F, case doesn't matter). Each hexadecimal digit represents
828four bits, or half a byte. C<print 0x..., "\n"> will show a
829hexadecimal number in decimal, and C<printf "%x\n", $decimal> will
830show a decimal number in hexadecimal. If you have just the
376d9008 831"hex digits" of a hexadecimal number, you can use the C<hex()> function.
ba62762e
JH
832
833 print 0x0009, "\n"; # 9
834 print 0x000a, "\n"; # 10
835 print 0x000f, "\n"; # 15
836 print 0x0010, "\n"; # 16
837 print 0x0011, "\n"; # 17
838 print 0x0100, "\n"; # 256
839
840 print 0x0041, "\n"; # 65
841
842 printf "%x\n", 65; # 41
843 printf "%#x\n", 65; # 0x41
844
845 print hex("41"), "\n"; # 65
846
847=head2 Further Resources
848
849=over 4
850
851=item *
852
853Unicode Consortium
854
2bbc8d55 855L<http://www.unicode.org/>
ba62762e
JH
856
857=item *
858
859Unicode FAQ
860
2bbc8d55 861L<http://www.unicode.org/unicode/faq/>
ba62762e
JH
862
863=item *
864
865Unicode Glossary
866
2bbc8d55 867L<http://www.unicode.org/glossary/>
ba62762e
JH
868
869=item *
870
c8695642
KW
871Unicode Recommended Reading List
872
873The Unicode Consortium has a list of articles and books, some of which
874give a much more in depth treatment of Unicode:
875L<http://unicode.org/resources/readinglist.html>
876
877=item *
878
ba62762e
JH
879Unicode Useful Resources
880
2bbc8d55 881L<http://www.unicode.org/unicode/onlinedat/resources.html>
ba62762e
JH
882
883=item *
884
885Unicode and Multilingual Support in HTML, Fonts, Web Browsers and Other Applications
886
2bbc8d55 887L<http://www.alanwood.net/unicode/>
ba62762e
JH
888
889=item *
890
891UTF-8 and Unicode FAQ for Unix/Linux
892
2bbc8d55 893L<http://www.cl.cam.ac.uk/~mgk25/unicode.html>
ba62762e
JH
894
895=item *
896
897Legacy Character Sets
898
2bbc8d55
SP
899L<http://www.czyborra.com/>
900L<http://www.eki.ee/letter/>
ba62762e
JH
901
902=item *
903
ba62762e
JH
904You can explore various information from the Unicode data files using
905the C<Unicode::UCD> module.
906
907=back
908
f6edf83b
JH
909=head1 UNICODE IN OLDER PERLS
910
911If you cannot upgrade your Perl to 5.8.0 or later, you can still
912do some Unicode processing by using the modules C<Unicode::String>,
913C<Unicode::Map8>, and C<Unicode::Map>, available from CPAN.
914If you have the GNU recode installed, you can also use the
376d9008 915Perl front-end C<Convert::Recode> for character conversions.
f6edf83b 916
aaef10c5 917The following are fast conversions from ISO 8859-1 (Latin-1) bytes
63de3cb2 918to UTF-8 bytes and back, the code works even with older Perl 5 versions.
aaef10c5
JH
919
920 # ISO 8859-1 to UTF-8
921 s/([\x80-\xFF])/chr(0xC0|ord($1)>>6).chr(0x80|ord($1)&0x3F)/eg;
922
923 # UTF-8 to ISO 8859-1
924 s/([\xC2\xC3])([\x80-\xBF])/chr(ord($1)<<6&0xC0|ord($2)&0x3F)/eg;
925
ba62762e
JH
926=head1 SEE ALSO
927
2575c402 928L<perlunitut>, L<perlunicode>, L<Encode>, L<open>, L<utf8>, L<bytes>,
4c496f0c
JH
929L<perlretut>, L<perlrun>, L<Unicode::Collate>, L<Unicode::Normalize>,
930L<Unicode::UCD>
ba62762e 931
376d9008 932=head1 ACKNOWLEDGMENTS
ba62762e
JH
933
934Thanks to the kind readers of the perl5-porters@perl.org,
935perl-unicode@perl.org, linux-utf8@nl.linux.org, and unicore@unicode.org
936mailing lists for their valuable feedback.
937
938=head1 AUTHOR, COPYRIGHT, AND LICENSE
939
c8695642 940Copyright 2001-2011 Jarkko Hietaniemi E<lt>jhi@iki.fiE<gt>
ba62762e
JH
941
942This document may be distributed under the same terms as Perl itself.