This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
perlunicode: Fix typo
[perl5.git] / pod / perlunicode.pod
CommitLineData
393fec97
GS
1=head1 NAME
2
3perlunicode - Unicode support in Perl
4
5=head1 DESCRIPTION
6
a6a7eedc
KW
7If you haven't already, before reading this document, you should become
8familiar with both L<perlunitut> and L<perluniintro>.
9
10Unicode aims to B<UNI>-fy the en-B<CODE>-ings of all the world's
11character sets into a single Standard. For quite a few of the various
12coding standards that existed when Unicode was first created, converting
13from each to Unicode essentially meant adding a constant to each code
14point in the original standard, and converting back meant just
15subtracting that same constant. For ASCII and ISO-8859-1, the constant
16is 0. For ISO-8859-5, (Cyrillic) the constant is 864; for Hebrew
17(ISO-8859-8), it's 1488; Thai (ISO-8859-11), 3424; and so forth. This
18made it easy to do the conversions, and facilitated the adoption of
19Unicode.
20
21And it worked; nowadays, those legacy standards are rarely used. Most
22everyone uses Unicode.
23
24Unicode is a comprehensive standard. It specifies many things outside
25the scope of Perl, such as how to display sequences of characters. For
26a full discussion of all aspects of Unicode, see
71c89d21 27L<https://www.unicode.org>.
a6a7eedc 28
0a1f2d14 29=head2 Important Caveats
21bad921 30
a6a7eedc
KW
31Even though some of this section may not be understandable to you on
32first reading, we think it's important enough to highlight some of the
33gotchas before delving further, so here goes:
34
376d9008 35Unicode support is an extensive requirement. While Perl does not
c349b1b9
JH
36implement the Unicode standard or the accompanying technical reports
37from cover to cover, Perl does support many Unicode features.
21bad921 38
f57d8456 39Also, the use of Unicode may present security issues that aren't
526f2ca9 40obvious, see L</Security Implications of Unicode> below.
9d1c51c1 41
13a2d996 42=over 4
21bad921 43
a9130ea9 44=item Safest if you C<use feature 'unicode_strings'>
42581d5d
KW
45
46In order to preserve backward compatibility, Perl does not turn
47on full internal Unicode support unless the pragma
b65e6125
KW
48L<S<C<use feature 'unicode_strings'>>|feature/The 'unicode_strings' feature>
49is specified. (This is automatically
50selected if you S<C<use 5.012>> or higher.) Failure to do this can
42581d5d
KW
51trigger unexpected surprises. See L</The "Unicode Bug"> below.
52
2269d15c
KW
53This pragma doesn't affect I/O. Nor does it change the internal
54representation of strings, only their interpretation. There are still
55several places where Unicode isn't fully supported, such as in
56filenames.
42581d5d 57
fae2c0fb 58=item Input and Output Layers
21bad921 59
a6a7eedc
KW
60Use the C<:encoding(...)> layer to read from and write to
61filehandles using the specified encoding. (See L<open>.)
c349b1b9 62
01c3fbbc 63=item You must convert your non-ASCII, non-UTF-8 Perl scripts to be
a6a7eedc 64UTF-8.
21bad921 65
01c3fbbc
TC
66The L<encoding> module has been deprecated since perl 5.18 and the
67perl internals it requires have been removed with perl 5.26.
21bad921 68
a6a7eedc 69=item C<use utf8> still needed to enable L<UTF-8|/Unicode Encodings> in scripts
21bad921 70
a6a7eedc
KW
71If your Perl script is itself encoded in L<UTF-8|/Unicode Encodings>,
72the S<C<use utf8>> pragma must be explicitly included to enable
73recognition of that (in string or regular expression literals, or in
74identifier names). B<This is the only time when an explicit S<C<use
75utf8>> is needed.> (See L<utf8>).
7aa207d6 76
27c74dfd
KW
77If a Perl script begins with the bytes that form the UTF-8 encoding of
78the Unicode BYTE ORDER MARK (C<BOM>, see L</Unicode Encodings>), those
79bytes are completely ignored.
80
81=item L<UTF-16|/Unicode Encodings> scripts autodetected
7aa207d6 82
fea12a3e 83If a Perl script begins with the Unicode C<BOM> (UTF-16LE,
27c74dfd 84UTF16-BE), or if the script looks like non-C<BOM>-marked
a6a7eedc 85UTF-16 of either endianness, Perl will correctly read in the script as
27c74dfd 86the appropriate Unicode encoding.
990e18f7 87
21bad921
GS
88=back
89
376d9008 90=head2 Byte and Character Semantics
393fec97 91
a6a7eedc
KW
92Before Unicode, most encodings used 8 bits (a single byte) to encode
93each character. Thus a character was a byte, and a byte was a
94character, and there could be only 256 or fewer possible characters.
95"Byte Semantics" in the title of this section refers to
96this behavior. There was no need to distinguish between "Byte" and
97"Character".
98
99Then along comes Unicode which has room for over a million characters
100(and Perl allows for even more). This means that a character may
101require more than a single byte to represent it, and so the two terms
102are no longer equivalent. What matter are the characters as whole
103entities, and not usually the bytes that comprise them. That's what the
104term "Character Semantics" in the title of this section refers to.
105
106Perl had to change internally to decouple "bytes" from "characters".
107It is important that you too change your ideas, if you haven't already,
108so that "byte" and "character" no longer mean the same thing in your
109mind.
110
111The basic building block of Perl strings has always been a "character".
112The changes basically come down to that the implementation no longer
113thinks that a character is always just a single byte.
114
115There are various things to note:
393fec97
GS
116
117=over 4
118
119=item *
120
a6a7eedc
KW
121String handling functions, for the most part, continue to operate in
122terms of characters. C<length()>, for example, returns the number of
123characters in a string, just as before. But that number no longer is
124necessarily the same as the number of bytes in the string (there may be
125more bytes than characters). The other such functions include
126C<chop()>, C<chomp()>, C<substr()>, C<pos()>, C<index()>, C<rindex()>,
127C<sort()>, C<sprintf()>, and C<write()>.
128
129The exceptions are:
130
131=over 4
132
133=item *
134
135the bit-oriented C<vec>
136
137E<nbsp>
138
139=item *
140
141the byte-oriented C<pack>/C<unpack> C<"C"> format
142
143However, the C<W> specifier does operate on whole characters, as does the
144C<U> specifier.
145
146=item *
147
148some operators that interact with the platform's operating system
149
150Operators dealing with filenames are examples.
151
152=item *
153
154when the functions are called from within the scope of the
155S<C<L<use bytes|bytes>>> pragma
156
157Likely, you should use this only for debugging anyway.
158
159=back
160
161=item *
162
376d9008 163Strings--including hash keys--and regular expression patterns may
b65e6125 164contain characters that have ordinal values larger than 255.
393fec97 165
2575c402
JW
166If you use a Unicode editor to edit your program, Unicode characters may
167occur directly within the literal strings in UTF-8 encoding, or UTF-16.
27c74dfd 168(The former requires a C<use utf8>, the latter may require a C<BOM>.)
3e4dbfed 169
a6a7eedc
KW
170L<perluniintro/Creating Unicode> gives other ways to place non-ASCII
171characters in your strings.
6f335b04 172
a6a7eedc 173=item *
fbb93542 174
a6a7eedc 175The C<chr()> and C<ord()> functions work on whole characters.
376d9008 176
393fec97
GS
177=item *
178
a6a7eedc
KW
179Regular expressions match whole characters. For example, C<"."> matches
180a whole character instead of only a single byte.
393fec97 181
393fec97
GS
182=item *
183
a6a7eedc
KW
184The C<tr///> operator translates whole characters. (Note that the
185C<tr///CU> functionality has been removed. For similar functionality to
186that, see C<pack('U0', ...)> and C<pack('C0', ...)>).
393fec97 187
393fec97
GS
188=item *
189
a6a7eedc 190C<scalar reverse()> reverses by character rather than by byte.
393fec97 191
393fec97
GS
192=item *
193
a6a7eedc 194The bit string operators, C<& | ^ ~> and (starting in v5.22)
fac71630
KW
195C<&. |. ^. ~.> can operate on bit strings encoded in UTF-8, but this
196can give unexpected results if any of the strings contain code points
197above 0xFF. Starting in v5.28, it is a fatal error to have such an
198operand. Otherwise, the operation is performed on a non-UTF-8 copy of
199the operand. If you're not sure about the encoding of a string,
200downgrade it before using any of these operators; you can use
a6a7eedc 201L<C<utf8::utf8_downgrade()>|utf8/Utility functions>.
822502e5 202
a6a7eedc 203=back
822502e5 204
a6a7eedc
KW
205The bottom line is that Perl has always practiced "Character Semantics",
206but with the advent of Unicode, that is now different than "Byte
207Semantics".
208
209=head2 ASCII Rules versus Unicode Rules
210
211Before Unicode, when a character was a byte was a character,
212Perl knew only about the 128 characters defined by ASCII, code points 0
b57dd509
KW
213through 127 (except for under L<S<C<use locale>>|perllocale>). That
214left the code
a6a7eedc
KW
215points 128 to 255 as unassigned, and available for whatever use a
216program might want. The only semantics they have is their ordinal
217numbers, and that they are members of none of the non-negative character
218classes. None are considered to match C<\w> for example, but all match
219C<\W>.
822502e5 220
a6a7eedc
KW
221Unicode, of course, assigns each of those code points a particular
222meaning (along with ones above 255). To preserve backward
223compatibility, Perl only uses the Unicode meanings when there is some
224indication that Unicode is what is intended; otherwise the non-ASCII
225code points remain treated as if they are unassigned.
226
227Here are the ways that Perl knows that a string should be treated as
228Unicode:
229
230=over
822502e5
ST
231
232=item *
233
a6a7eedc
KW
234Within the scope of S<C<use utf8>>
235
236If the whole program is Unicode (signified by using 8-bit B<U>nicode
e423fa83 237B<T>ransformation B<F>ormat), then all literal strings within it must be
a6a7eedc 238Unicode.
822502e5
ST
239
240=item *
241
a6a7eedc
KW
242Within the scope of
243L<S<C<use feature 'unicode_strings'>>|feature/The 'unicode_strings' feature>
244
245This pragma was created so you can explicitly tell Perl that operations
246executed within its scope are to use Unicode rules. More operations are
247affected with newer perls. See L</The "Unicode Bug">.
822502e5
ST
248
249=item *
250
a6a7eedc
KW
251Within the scope of S<C<use 5.012>> or higher
252
253This implicitly turns on S<C<use feature 'unicode_strings'>>.
822502e5
ST
254
255=item *
256
a6a7eedc
KW
257Within the scope of
258L<S<C<use locale 'not_characters'>>|perllocale/Unicode and UTF-8>,
259or L<S<C<use locale>>|perllocale> and the current
260locale is a UTF-8 locale.
822502e5 261
a6a7eedc
KW
262The former is defined to imply Unicode handling; and the latter
263indicates a Unicode locale, hence a Unicode interpretation of all
264strings within it.
822502e5
ST
265
266=item *
267
a6a7eedc
KW
268When the string contains a Unicode-only code point
269
270Perl has never accepted code points above 255 without them being
271Unicode, so their use implies Unicode for the whole string.
822502e5
ST
272
273=item *
274
a6a7eedc
KW
275When the string contains a Unicode named code point C<\N{...}>
276
277The C<\N{...}> construct explicitly refers to a Unicode code point,
278even if it is one that is also in ASCII. Therefore the string
279containing it must be Unicode.
822502e5
ST
280
281=item *
282
a6a7eedc
KW
283When the string has come from an external source marked as
284Unicode
285
286The L<C<-C>|perlrun/-C [numberE<sol>list]> command line option can
287specify that certain inputs to the program are Unicode, and the values
288of this can be read by your Perl code, see L<perlvar/"${^UNICODE}">.
289
290=item * When the string has been upgraded to UTF-8
291
292The function L<C<utf8::utf8_upgrade()>|utf8/Utility functions>
293can be explicitly used to permanently (unless a subsequent
294C<utf8::utf8_downgrade()> is called) cause a string to be treated as
295Unicode.
296
297=item * There are additional methods for regular expression patterns
298
f6cf4627
KW
299A pattern that is compiled with the C<< /u >> or C<< /a >> modifiers is
300treated as Unicode (though there are some restrictions with C<< /a >>).
301Under the C<< /d >> and C<< /l >> modifiers, there are several other
302indications for Unicode; see L<perlre/Character set modifiers>.
822502e5
ST
303
304=back
305
a6a7eedc
KW
306Note that all of the above are overridden within the scope of
307C<L<use bytes|bytes>>; but you should be using this pragma only for
308debugging.
309
310Note also that some interactions with the platform's operating system
311never use Unicode rules.
312
313When Unicode rules are in effect:
314
822502e5
ST
315=over 4
316
317=item *
318
a6a7eedc
KW
319Case translation operators use the Unicode case translation tables.
320
321Note that C<uc()>, or C<\U> in interpolated strings, translates to
322uppercase, while C<ucfirst>, or C<\u> in interpolated strings,
323translates to titlecase in languages that make the distinction (which is
324equivalent to uppercase in languages without the distinction).
325
326There is a CPAN module, C<L<Unicode::Casing>>, which allows you to
327define your own mappings to be used in C<lc()>, C<lcfirst()>, C<uc()>,
328C<ucfirst()>, and C<fc> (or their double-quoted string inlined versions
329such as C<\U>). (Prior to Perl 5.16, this functionality was partially
330provided in the Perl core, but suffered from a number of insurmountable
331drawbacks, so the CPAN module was written instead.)
332
333=item *
334
335Character classes in regular expressions match based on the character
336properties specified in the Unicode properties database.
337
338C<\w> can be used to match a Japanese ideograph, for instance; and
339C<[[:digit:]]> a Bengali number.
340
341=item *
342
343Named Unicode properties, scripts, and block ranges may be used (like
344bracketed character classes) by using the C<\p{}> "matches property"
345construct and the C<\P{}> negation, "doesn't match property".
346
347See L</"Unicode Character Properties"> for more details.
348
349You can define your own character properties and use them
350in the regular expression with the C<\p{}> or C<\P{}> construct.
351See L</"User-Defined Character Properties"> for more details.
822502e5
ST
352
353=back
354
a6a7eedc
KW
355=head2 Extended Grapheme Clusters (Logical characters)
356
357Consider a character, say C<H>. It could appear with various marks around it,
358such as an acute accent, or a circumflex, or various hooks, circles, arrows,
359I<etc.>, above, below, to one side or the other, I<etc>. There are many
360possibilities among the world's languages. The number of combinations is
361astronomical, and if there were a character for each combination, it would
362soon exhaust Unicode's more than a million possible characters. So Unicode
363took a different approach: there is a character for the base C<H>, and a
364character for each of the possible marks, and these can be variously combined
365to get a final logical character. So a logical character--what appears to be a
366single character--can be a sequence of more than one individual characters.
367The Unicode standard calls these "extended grapheme clusters" (which
368is an improved version of the no-longer much used "grapheme cluster");
369Perl furnishes the C<\X> regular expression construct to match such
370sequences in their entirety.
371
372But Unicode's intent is to unify the existing character set standards and
373practices, and several pre-existing standards have single characters that
374mean the same thing as some of these combinations, like ISO-8859-1,
375which has quite a few of them. For example, C<"LATIN CAPITAL LETTER E
376WITH ACUTE"> was already in this standard when Unicode came along.
377Unicode therefore added it to its repertoire as that single character.
378But this character is considered by Unicode to be equivalent to the
379sequence consisting of the character C<"LATIN CAPITAL LETTER E">
380followed by the character C<"COMBINING ACUTE ACCENT">.
381
382C<"LATIN CAPITAL LETTER E WITH ACUTE"> is called a "pre-composed"
383character, and its equivalence with the "E" and the "COMBINING ACCENT"
384sequence is called canonical equivalence. All pre-composed characters
385are said to have a decomposition (into the equivalent sequence), and the
386decomposition type is also called canonical. A string may be comprised
387as much as possible of precomposed characters, or it may be comprised of
388entirely decomposed characters. Unicode calls these respectively,
389"Normalization Form Composed" (NFC) and "Normalization Form Decomposed".
390The C<L<Unicode::Normalize>> module contains functions that convert
391between the two. A string may also have both composed characters and
392decomposed characters; this module can be used to make it all one or the
393other.
394
395You may be presented with strings in any of these equivalent forms.
396There is currently nothing in Perl 5 that ignores the differences. So
dabde021 397you'll have to specially handle it. The usual advice is to convert your
a6a7eedc
KW
398inputs to C<NFD> before processing further.
399
400For more detailed information, see L<http://unicode.org/reports/tr15/>.
401
822502e5
ST
402=head2 Unicode Character Properties
403
ee88f7b6 404(The only time that Perl considers a sequence of individual code
9d1c51c1
KW
405points as a single logical character is in the C<\X> construct, already
406mentioned above. Therefore "character" in this discussion means a single
ee88f7b6
KW
407Unicode code point.)
408
409Very nearly all Unicode character properties are accessible through
410regular expressions by using the C<\p{}> "matches property" construct
411and the C<\P{}> "doesn't match property" for its negation.
51f494cc 412
9d1c51c1 413For instance, C<\p{Uppercase}> matches any single character with the Unicode
a9130ea9
KW
414C<"Uppercase"> property, while C<\p{L}> matches any character with a
415C<General_Category> of C<"L"> (letter) property (see
416L</General_Category> below). Brackets are not
9d1c51c1 417required for single letter property names, so C<\p{L}> is equivalent to C<\pL>.
51f494cc 418
9d1c51c1 419More formally, C<\p{Uppercase}> matches any single character whose Unicode
a9130ea9
KW
420C<Uppercase> property value is C<True>, and C<\P{Uppercase}> matches any character
421whose C<Uppercase> property value is C<False>, and they could have been written as
9d1c51c1 422C<\p{Uppercase=True}> and C<\p{Uppercase=False}>, respectively.
51f494cc 423
b19eb496 424This formality is needed when properties are not binary; that is, if they can
a9130ea9
KW
425take on more values than just C<True> and C<False>. For example, the
426C<Bidi_Class> property (see L</"Bidirectional Character Types"> below),
427can take on several different
428values, such as C<Left>, C<Right>, C<Whitespace>, and others. To match these, one needs
429to specify both the property name (C<Bidi_Class>), AND the value being
5bff2035 430matched against
b65e6125 431(C<Left>, C<Right>, I<etc.>). This is done, as in the examples above, by having the
9f815e24 432two components separated by an equal sign (or interchangeably, a colon), like
51f494cc
KW
433C<\p{Bidi_Class: Left}>.
434
435All Unicode-defined character properties may be written in these compound forms
a9130ea9 436of C<\p{I<property>=I<value>}> or C<\p{I<property>:I<value>}>, but Perl provides some
51f494cc
KW
437additional properties that are written only in the single form, as well as
438single-form short-cuts for all binary properties and certain others described
439below, in which you may omit the property name and the equals or colon
440separator.
441
442Most Unicode character properties have at least two synonyms (or aliases if you
b19eb496 443prefer): a short one that is easier to type and a longer one that is more
a9130ea9
KW
444descriptive and hence easier to understand. Thus the C<"L"> and
445C<"Letter"> properties above are equivalent and can be used
446interchangeably. Likewise, C<"Upper"> is a synonym for C<"Uppercase">,
447and we could have written C<\p{Uppercase}> equivalently as C<\p{Upper}>.
448Also, there are typically various synonyms for the values the property
449can be. For binary properties, C<"True"> has 3 synonyms: C<"T">,
450C<"Yes">, and C<"Y">; and C<"False"> has correspondingly C<"F">,
451C<"No">, and C<"N">. But be careful. A short form of a value for one
d130120f
KW
452property may not mean the same thing as the short form spelled the same
453for another.
a9130ea9
KW
454Thus, for the C<L</General_Category>> property, C<"L"> means
455C<"Letter">, but for the L<C<Bidi_Class>|/Bidirectional Character Types>
456property, C<"L"> means C<"Left">. A complete list of properties and
457synonyms is in L<perluniprops>.
51f494cc 458
b19eb496 459Upper/lower case differences in property names and values are irrelevant;
51f494cc
KW
460thus C<\p{Upper}> means the same thing as C<\p{upper}> or even C<\p{UpPeR}>.
461Similarly, you can add or subtract underscores anywhere in the middle of a
462word, so that these are also equivalent to C<\p{U_p_p_e_r}>. And white space
673c254b
KW
463is generally irrelevant adjacent to non-word characters, such as the
464braces and the equals or colon separators, so C<\p{ Upper }> and
465C<\p{ Upper_case : Y }> are equivalent to these as well. In fact, white
466space and even hyphens can usually be added or deleted anywhere. So
467even C<\p{ Up-per case = Yes}> is equivalent. All this is called
468"loose-matching" by Unicode. The "name" property has some restrictions
469on this due to a few outlier names. Full details are given in
470L<https://www.unicode.org/reports/tr44/tr44-24.html#UAX44-LM2>.
471
472The few places where stricter matching is
473used is in the middle of numbers, the "name" property, and in the Perl
474extension properties that begin or end with an underscore. Stricter
475matching cares about white space (except adjacent to non-word
476characters), hyphens, and non-interior underscores.
4193bef7 477
376d9008 478You can also use negation in both C<\p{}> and C<\P{}> by introducing a caret
a9130ea9 479(C<^>) between the first brace and the property name: C<\p{^Tamil}> is
eb0cc9e3 480equal to C<\P{Tamil}>.
4193bef7 481
56ca34ca
KW
482Almost all properties are immune to case-insensitive matching. That is,
483adding a C</i> regular expression modifier does not change what they
484match. There are two sets that are affected.
485The first set is
486C<Uppercase_Letter>,
487C<Lowercase_Letter>,
488and C<Titlecase_Letter>,
489all of which match C<Cased_Letter> under C</i> matching.
490And the second set is
491C<Uppercase>,
492C<Lowercase>,
493and C<Titlecase>,
494all of which match C<Cased> under C</i> matching.
495This set also includes its subsets C<PosixUpper> and C<PosixLower> both
a9130ea9 496of which under C</i> match C<PosixAlpha>.
56ca34ca 497(The difference between these sets is that some things, such as Roman
b65e6125
KW
498numerals, come in both upper and lower case so they are C<Cased>, but
499aren't considered letters, so they aren't C<Cased_Letter>'s.)
56ca34ca 500
2d88a86a
KW
501See L</Beyond Unicode code points> for special considerations when
502matching Unicode properties against non-Unicode code points.
94b42e47 503
51f494cc 504=head3 B<General_Category>
14bb0a9a 505
51f494cc
KW
506Every Unicode character is assigned a general category, which is the "most
507usual categorization of a character" (from
71c89d21 508L<https://www.unicode.org/reports/tr44>).
822502e5 509
9f815e24 510The compound way of writing these is like C<\p{General_Category=Number}>
b65e6125 511(short: C<\p{gc:n}>). But Perl furnishes shortcuts in which everything up
51f494cc
KW
512through the equal or colon separator is omitted. So you can instead just write
513C<\pN>.
822502e5 514
a9130ea9
KW
515Here are the short and long forms of the values the C<General Category> property
516can have:
393fec97 517
d73e5302
JH
518 Short Long
519
520 L Letter
51f494cc
KW
521 LC, L& Cased_Letter (that is: [\p{Ll}\p{Lu}\p{Lt}])
522 Lu Uppercase_Letter
523 Ll Lowercase_Letter
524 Lt Titlecase_Letter
525 Lm Modifier_Letter
526 Lo Other_Letter
d73e5302
JH
527
528 M Mark
51f494cc
KW
529 Mn Nonspacing_Mark
530 Mc Spacing_Mark
531 Me Enclosing_Mark
d73e5302
JH
532
533 N Number
51f494cc
KW
534 Nd Decimal_Number (also Digit)
535 Nl Letter_Number
536 No Other_Number
537
538 P Punctuation (also Punct)
539 Pc Connector_Punctuation
540 Pd Dash_Punctuation
541 Ps Open_Punctuation
542 Pe Close_Punctuation
543 Pi Initial_Punctuation
d73e5302 544 (may behave like Ps or Pe depending on usage)
51f494cc 545 Pf Final_Punctuation
d73e5302 546 (may behave like Ps or Pe depending on usage)
51f494cc 547 Po Other_Punctuation
d73e5302
JH
548
549 S Symbol
51f494cc
KW
550 Sm Math_Symbol
551 Sc Currency_Symbol
552 Sk Modifier_Symbol
553 So Other_Symbol
d73e5302
JH
554
555 Z Separator
51f494cc
KW
556 Zs Space_Separator
557 Zl Line_Separator
558 Zp Paragraph_Separator
d73e5302
JH
559
560 C Other
d88362ca 561 Cc Control (also Cntrl)
e150c829 562 Cf Format
6d4f9cf2 563 Cs Surrogate
51f494cc 564 Co Private_Use
e150c829 565 Cn Unassigned
1ac13f9a 566
376d9008 567Single-letter properties match all characters in any of the
3e4dbfed 568two-letter sub-properties starting with the same letter.
b19eb496 569C<LC> and C<L&> are special: both are aliases for the set consisting of everything matched by C<Ll>, C<Lu>, and C<Lt>.
32293815 570
51f494cc 571=head3 B<Bidirectional Character Types>
822502e5 572
b19eb496 573Because scripts differ in their directionality (Hebrew and Arabic are
a9130ea9 574written right to left, for example) Unicode supplies a C<Bidi_Class> property.
1850f57f 575Some of the values this property can have are:
32293815 576
88af3b93 577 Value Meaning
92e830a9 578
12ac2576
JP
579 L Left-to-Right
580 LRE Left-to-Right Embedding
581 LRO Left-to-Right Override
582 R Right-to-Left
51f494cc 583 AL Arabic Letter
12ac2576
JP
584 RLE Right-to-Left Embedding
585 RLO Right-to-Left Override
586 PDF Pop Directional Format
587 EN European Number
51f494cc
KW
588 ES European Separator
589 ET European Terminator
12ac2576 590 AN Arabic Number
51f494cc 591 CS Common Separator
12ac2576
JP
592 NSM Non-Spacing Mark
593 BN Boundary Neutral
594 B Paragraph Separator
595 S Segment Separator
596 WS Whitespace
597 ON Other Neutrals
598
51f494cc
KW
599This property is always written in the compound form.
600For example, C<\p{Bidi_Class:R}> matches characters that are normally
1850f57f 601written right to left. Unlike the
a9130ea9 602C<L</General_Category>> property, this
1850f57f
KW
603property can have more values added in a future Unicode release. Those
604listed above comprised the complete set for many Unicode releases, but
605others were added in Unicode 6.3; you can always find what the
20ada7da 606current ones are in L<perluniprops>. And
71c89d21 607L<https://www.unicode.org/reports/tr9/> describes how to use them.
eb0cc9e3 608
51f494cc
KW
609=head3 B<Scripts>
610
b19eb496 611The world's languages are written in many different scripts. This sentence
e1b711da 612(unless you're reading it in translation) is written in Latin, while Russian is
c69ca1d4 613written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in
e1b711da 614Hiragana or Katakana. There are many more.
51f494cc 615
48791bf1
KW
616The Unicode C<Script> and C<Script_Extensions> properties give what
617script a given character is in. The C<Script_Extensions> property is an
618improved version of C<Script>, as demonstrated below. Either property
619can be specified with the compound form like
82aed44a
KW
620C<\p{Script=Hebrew}> (short: C<\p{sc=hebr}>), or
621C<\p{Script_Extensions=Javanese}> (short: C<\p{scx=java}>).
622In addition, Perl furnishes shortcuts for all
48791bf1
KW
623C<Script_Extensions> property names. You can omit everything up through
624the equals (or colon), and simply write C<\p{Latin}> or C<\P{Cyrillic}>.
625(This is not true for C<Script>, which is required to be
626written in the compound form. Prior to Perl v5.26, the single form
627returned the plain old C<Script> version, but was changed because
628C<Script_Extensions> gives better results.)
82aed44a
KW
629
630The difference between these two properties involves characters that are
631used in multiple scripts. For example the digits '0' through '9' are
632used in many parts of the world. These are placed in a script named
633C<Common>. Other characters are used in just a few scripts. For
a9130ea9 634example, the C<"KATAKANA-HIRAGANA DOUBLE HYPHEN"> is used in both Japanese
82aed44a
KW
635scripts, Katakana and Hiragana, but nowhere else. The C<Script>
636property places all characters that are used in multiple scripts in the
637C<Common> script, while the C<Script_Extensions> property places those
638that are used in only a few scripts into each of those scripts; while
639still using C<Common> for those used in many scripts. Thus both these
640match:
641
642 "0" =~ /\p{sc=Common}/ # Matches
643 "0" =~ /\p{scx=Common}/ # Matches
644
645and only the first of these match:
646
647 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Common} # Matches
648 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Common} # No match
649
650And only the last two of these match:
651
652 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Hiragana} # No match
653 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Katakana} # No match
654 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Hiragana} # Matches
655 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Katakana} # Matches
656
657C<Script_Extensions> is thus an improved C<Script>, in which there are
658fewer characters in the C<Common> script, and correspondingly more in
659other scripts. It is new in Unicode version 6.0, and its data are likely
660to change significantly in later releases, as things get sorted out.
b65e6125 661New code should probably be using C<Script_Extensions> and not plain
48791bf1
KW
662C<Script>. If you compile perl with a Unicode release that doesn't have
663C<Script_Extensions>, the single form Perl extensions will instead refer
664to the plain C<Script> property. If you compile with a version of
665Unicode that doesn't have the C<Script> property, these extensions will
666not be defined at all.
82aed44a
KW
667
668(Actually, besides C<Common>, the C<Inherited> script, contains
669characters that are used in multiple scripts. These are modifier
b65e6125 670characters which inherit the script value
82aed44a
KW
671of the controlling character. Some of these are used in many scripts,
672and so go into C<Inherited> in both C<Script> and C<Script_Extensions>.
673Others are used in just a few scripts, so are in C<Inherited> in
674C<Script>, but not in C<Script_Extensions>.)
675
676It is worth stressing that there are several different sets of digits in
677Unicode that are equivalent to 0-9 and are matchable by C<\d> in a
678regular expression. If they are used in a single language only, they
48791bf1 679are in that language's C<Script> and C<Script_Extensions>. If they are
82aed44a
KW
680used in more than one script, they will be in C<sc=Common>, but only
681if they are used in many scripts should they be in C<scx=Common>.
51f494cc 682
48791bf1 683The explanation above has omitted some detail; refer to UAX#24 "Unicode
71c89d21 684Script Property": L<https://www.unicode.org/reports/tr24>.
48791bf1 685
51f494cc
KW
686A complete list of scripts and their shortcuts is in L<perluniprops>.
687
a9130ea9 688=head3 B<Use of the C<"Is"> Prefix>
822502e5 689
7b0ac457 690For backward compatibility (with ancient Perl 5.6), all properties writable
b65e6125 691without using the compound form mentioned
51f494cc
KW
692so far may have C<Is> or C<Is_> prepended to their name, so C<\P{Is_Lu}>, for
693example, is equal to C<\P{Lu}>, and C<\p{IsScript:Arabic}> is equal to
694C<\p{Arabic}>.
eb0cc9e3 695
51f494cc 696=head3 B<Blocks>
2796c109 697
1bfb14c4
JH
698In addition to B<scripts>, Unicode also defines B<blocks> of
699characters. The difference between scripts and blocks is that the
700concept of scripts is closer to natural languages, while the concept
51f494cc 701of blocks is more of an artificial grouping based on groups of Unicode
a9130ea9 702characters with consecutive ordinal values. For example, the C<"Basic Latin">
b65e6125 703block is all the characters whose ordinals are between 0 and 127, inclusive; in
a9130ea9
KW
704other words, the ASCII characters. The C<"Latin"> script contains some letters
705from this as well as several other blocks, like C<"Latin-1 Supplement">,
b65e6125 706C<"Latin Extended-A">, I<etc.>, but it does not contain all the characters from
7be67b37 707those blocks. It does not, for example, contain the digits 0-9, because
82aed44a
KW
708those digits are shared across many scripts, and hence are in the
709C<Common> script.
51f494cc
KW
710
711For more about scripts versus blocks, see UAX#24 "Unicode Script Property":
71c89d21 712L<https://www.unicode.org/reports/tr24>
51f494cc 713
48791bf1 714The C<Script_Extensions> or C<Script> properties are likely to be the
82aed44a 715ones you want to use when processing
a9130ea9 716natural language; the C<Block> property may occasionally be useful in working
b19eb496 717with the nuts and bolts of Unicode.
51f494cc
KW
718
719Block names are matched in the compound form, like C<\p{Block: Arrows}> or
b19eb496 720C<\p{Blk=Hebrew}>. Unlike most other properties, only a few block names have a
6b5cf123
KW
721Unicode-defined short name.
722
723Perl also defines single form synonyms for the block property in cases
724where these do not conflict with something else. But don't use any of
725these, because they are unstable. Since these are Perl extensions, they
726are subordinate to official Unicode property names; Unicode doesn't know
727nor care about Perl's extensions. It may happen that a name that
728currently means the Perl extension will later be changed without warning
729to mean a different Unicode property in a future version of the perl
730interpreter that uses a later Unicode release, and your code would no
731longer work. The extensions are mentioned here for completeness: Take
732the block name and prefix it with one of: C<In> (for example
733C<\p{Blk=Arrows}> can currently be written as C<\p{In_Arrows}>); or
734sometimes C<Is> (like C<\p{Is_Arrows}>); or sometimes no prefix at all
48791bf1 735(C<\p{Arrows}>). As of this writing (Unicode 9.0) there are no
6b5cf123
KW
736conflicts with using the C<In_> prefix, but there are plenty with the
737other two forms. For example, C<\p{Is_Hebrew}> and C<\p{Hebrew}> mean
48791bf1
KW
738C<\p{Script_Extensions=Hebrew}> which is NOT the same thing as
739C<\p{Blk=Hebrew}>. Our
6b5cf123
KW
740advice used to be to use the C<In_> prefix as a single form way of
741specifying a block. But Unicode 8.0 added properties whose names begin
742with C<In>, and it's now clear that it's only luck that's so far
743prevented a conflict. Using C<In> is only marginally less typing than
744C<Blk:>, and the latter's meaning is clearer anyway, and guaranteed to
745never conflict. So don't take chances. Use C<\p{Blk=foo}> for new
746code. And be sure that block is what you really really want to do. In
747most cases scripts are what you want instead.
748
749A complete list of blocks is in L<perluniprops>.
51f494cc 750
9f815e24
KW
751=head3 B<Other Properties>
752
753There are many more properties than the very basic ones described here.
754A complete list is in L<perluniprops>.
755
756Unicode defines all its properties in the compound form, so all single-form
b19eb496
TC
757properties are Perl extensions. Most of these are just synonyms for the
758Unicode ones, but some are genuine extensions, including several that are in
9f815e24 759the compound form. And quite a few of these are actually recommended by Unicode
71c89d21 760(in L<https://www.unicode.org/reports/tr18>).
9f815e24 761
5bff2035
KW
762This section gives some details on all extensions that aren't just
763synonyms for compound-form Unicode properties
764(for those properties, you'll have to refer to the
71c89d21 765L<Unicode Standard|https://www.unicode.org/reports/tr44>.
9f815e24
KW
766
767=over
768
769=item B<C<\p{All}>>
770
2d88a86a
KW
771This matches every possible code point. It is equivalent to C<qr/./s>.
772Unlike all the other non-user-defined C<\p{}> property matches, no
773warning is ever generated if this is property is matched against a
774non-Unicode code point (see L</Beyond Unicode code points> below).
9f815e24
KW
775
776=item B<C<\p{Alnum}>>
777
778This matches any C<\p{Alphabetic}> or C<\p{Decimal_Number}> character.
779
780=item B<C<\p{Any}>>
781
2d88a86a
KW
782This matches any of the 1_114_112 Unicode code points. It is a synonym
783for C<\p{Unicode}>.
9f815e24 784
42581d5d
KW
785=item B<C<\p{ASCII}>>
786
787This matches any of the 128 characters in the US-ASCII character set,
788which is a subset of Unicode.
789
9f815e24
KW
790=item B<C<\p{Assigned}>>
791
a9130ea9
KW
792This matches any assigned code point; that is, any code point whose L<general
793category|/General_Category> is not C<Unassigned> (or equivalently, not C<Cn>).
9f815e24
KW
794
795=item B<C<\p{Blank}>>
796
797This is the same as C<\h> and C<\p{HorizSpace}>: A character that changes the
798spacing horizontally.
799
800=item B<C<\p{Decomposition_Type: Non_Canonical}>> (Short: C<\p{Dt=NonCanon}>)
801
802Matches a character that has a non-canonical decomposition.
803
a6a7eedc
KW
804The L</Extended Grapheme Clusters (Logical characters)> section above
805talked about canonical decompositions. However, many more characters
806have a different type of decomposition, a "compatible" or
807"non-canonical" decomposition. The sequences that form these
808decompositions are not considered canonically equivalent to the
809pre-composed character. An example is the C<"SUPERSCRIPT ONE">. It is
810somewhat like a regular digit 1, but not exactly; its decomposition into
811the digit 1 is called a "compatible" decomposition, specifically a
9f815e24 812"super" decomposition. There are several such compatibility
71c89d21 813decompositions (see L<https://www.unicode.org/reports/tr44>), including
b65e6125
KW
814one called "compat", which means some miscellaneous type of
815decomposition that doesn't fit into the other decomposition categories
816that Unicode has chosen.
9f815e24
KW
817
818Note that most Unicode characters don't have a decomposition, so their
a9130ea9 819decomposition type is C<"None">.
9f815e24 820
b19eb496
TC
821For your convenience, Perl has added the C<Non_Canonical> decomposition
822type to mean any of the several compatibility decompositions.
9f815e24
KW
823
824=item B<C<\p{Graph}>>
825
826Matches any character that is graphic. Theoretically, this means a character
827that on a printer would cause ink to be used.
828
829=item B<C<\p{HorizSpace}>>
830
b19eb496 831This is the same as C<\h> and C<\p{Blank}>: a character that changes the
9f815e24
KW
832spacing horizontally.
833
42581d5d 834=item B<C<\p{In=*}>>
9f815e24
KW
835
836This is a synonym for C<\p{Present_In=*}>
837
838=item B<C<\p{PerlSpace}>>
839
d28d8023 840This is the same as C<\s>, restricted to ASCII, namely C<S<[ \f\n\r\t]>>
779cf272 841and starting in Perl v5.18, a vertical tab.
9f815e24
KW
842
843Mnemonic: Perl's (original) space
844
845=item B<C<\p{PerlWord}>>
846
847This is the same as C<\w>, restricted to ASCII, namely C<[A-Za-z0-9_]>
848
849Mnemonic: Perl's (original) word.
850
42581d5d 851=item B<C<\p{Posix...}>>
9f815e24 852
b65e6125
KW
853There are several of these, which are equivalents, using the C<\p{}>
854notation, for Posix classes and are described in
42581d5d 855L<perlrecharclass/POSIX Character Classes>.
9f815e24
KW
856
857=item B<C<\p{Present_In: *}>> (Short: C<\p{In=*}>)
858
859This property is used when you need to know in what Unicode version(s) a
860character is.
861
526f2ca9
KW
862The "*" above stands for some Unicode version number, such as
863C<1.1> or C<12.0>; or the "*" can also be C<Unassigned>. This property will
9f815e24
KW
864match the code points whose final disposition has been settled as of the
865Unicode release given by the version number; C<\p{Present_In: Unassigned}>
866will match those code points whose meaning has yet to be assigned.
867
a9130ea9 868For example, C<U+0041> C<"LATIN CAPITAL LETTER A"> was present in the very first
9f815e24
KW
869Unicode release available, which is C<1.1>, so this property is true for all
870valid "*" versions. On the other hand, C<U+1EFF> was not assigned until version
a9130ea9 8715.1 when it became C<"LATIN SMALL LETTER Y WITH LOOP">, so the only "*" that
9f815e24
KW
872would match it are 5.1, 5.2, and later.
873
874Unicode furnishes the C<Age> property from which this is derived. The problem
875with Age is that a strict interpretation of it (which Perl takes) has it
876matching the precise release a code point's meaning is introduced in. Thus
877C<U+0041> would match only 1.1; and C<U+1EFF> only 5.1. This is not usually what
878you want.
879
880Some non-Perl implementations of the Age property may change its meaning to be
a9130ea9 881the same as the Perl C<Present_In> property; just be aware of that.
9f815e24
KW
882
883Another confusion with both these properties is that the definition is not
b19eb496
TC
884that the code point has been I<assigned>, but that the meaning of the code point
885has been I<determined>. This is because 66 code points will always be
a9130ea9 886unassigned, and so the C<Age> for them is the Unicode version in which the decision
b19eb496 887to make them so was made. For example, C<U+FDD0> is to be permanently
9f815e24 888unassigned to a character, and the decision to do that was made in version 3.1,
b19eb496 889so C<\p{Age=3.1}> matches this character, as also does C<\p{Present_In: 3.1}> and up.
9f815e24
KW
890
891=item B<C<\p{Print}>>
892
ae5b72c8 893This matches any character that is graphical or blank, except controls.
9f815e24
KW
894
895=item B<C<\p{SpacePerl}>>
896
897This is the same as C<\s>, including beyond ASCII.
898
4d4acfba 899Mnemonic: Space, as modified by Perl. (It doesn't include the vertical tab
779cf272 900until v5.18, which both the Posix standard and Unicode consider white space.)
9f815e24 901
4364919a
KW
902=item B<C<\p{Title}>> and B<C<\p{Titlecase}>>
903
904Under case-sensitive matching, these both match the same code points as
905C<\p{General Category=Titlecase_Letter}> (C<\p{gc=lt}>). The difference
906is that under C</i> caseless matching, these match the same as
907C<\p{Cased}>, whereas C<\p{gc=lt}> matches C<\p{Cased_Letter>).
908
2d88a86a
KW
909=item B<C<\p{Unicode}>>
910
911This matches any of the 1_114_112 Unicode code points.
912C<\p{Any}>.
913
9f815e24
KW
914=item B<C<\p{VertSpace}>>
915
916This is the same as C<\v>: A character that changes the spacing vertically.
917
918=item B<C<\p{Word}>>
919
b19eb496 920This is the same as C<\w>, including over 100_000 characters beyond ASCII.
9f815e24 921
42581d5d
KW
922=item B<C<\p{XPosix...}>>
923
b19eb496 924There are several of these, which are the standard Posix classes
42581d5d
KW
925extended to the full Unicode range. They are described in
926L<perlrecharclass/POSIX Character Classes>.
927
9f815e24
KW
928=back
929
673c254b
KW
930=head2 Comparison of C<\N{...}> and C<\p{name=...}>
931
932Starting in Perl 5.32, you can specify a character by its name in
933regular expression patterns using C<\p{name=...}>. This is in addition
934to the longstanding method of using C<\N{...}>. The following
935summarizes the differences between these two:
936
937 \N{...} \p{Name=...}
938 can interpolate only with eval yes [1]
939 custom names yes no [2]
940 name aliases yes yes [3]
941 named sequences yes not yet [4]
942 name value parsing exact Unicode loose [5]
943
944=over
945
946=item [1]
947
948The ability to interpolate means you can do something like
949
950 qr/\p{na=latin capital letter $which}/
951
952and specify C<$which> elsewhere.
953
954=item [2]
955
956You can create your own names for characters, and override official
957ones when using C<\N{...}>. See L<charnames/CUSTOM ALIASES>.
958
959=item [3]
960
961Some characters have multiple names (synonyms).
962
963=item [4]
964
965Some particular sequences of characters are given a single name, in
966addition to their individual ones.
967
968It is planned to add support for named sequences to the C<\p{...}> form
969before 5.32; in the meantime, an accurate but not fully informative
970message is generated if use of one of these is attempted.
971
972=item [5]
973
974Exact name value matching means you have to specify case, hyphens,
975underscores, and spaces precisely in the name you want. Loose matching
976follows the Unicode rules
977L<https://www.unicode.org/reports/tr44/tr44-24.html#UAX44-LM2>,
978where these are mostly irrelevant. Except for a few outlier character
979names, these are the same rules as are already used for any other
980C<\p{...}> property.
981
982=back
983
1532347b
KW
984=head2 Wildcards in Property Values
985
986Starting in Perl 5.30, it is possible to do do something like this:
987
988 qr!\p{numeric_value=/\A[0-5]\z/}!
989
990or, by abbreviating and adding C</x>,
991
992 qr! \p{nv= /(?x) \A [0-5] \z / }!
993
994This matches all code points whose numeric value is one of 0, 1, 2, 3,
9954, or 5. This particular example could instead have been written as
996
997 qr! \A [ \p{nv=0}\p{nv=1}\p{nv=2}\p{nv=3}\p{nv=4}\p{nv=5} ] \z !xx
998
999in earlier perls, so in this case this feature just makes things easier
1000and shorter to write. If we hadn't included the C<\A> and C<\z>, these
1001would have matched things like C<1E<sol>2> because that contains a 1 (as
1002well as a 2). As written, it matches things like subscripts that have
1003these numeric values. If we only wanted the decimal digits with those
1004numeric values, we could say,
1005
1006 qr! (?[ \d & \p{nv=/[0-5]/ ]) }!x
1007
1008The C<\d> gets rid of needing to anchor the pattern, since it forces the
1009result to only match C<[0-9]>, and the C<[0-5]> further restricts it.
1010
1011The text in the above examples enclosed between the C<"E<sol>">
1012characters can be just about any regular expression. It is independent
1013of the main pattern, so doesn't share any capturing groups, I<etc>. The
1014delimiters for it must be ASCII punctuation, but it may NOT be
1015delimited by C<"{">, nor C<"}"> nor contain a literal C<"}">, as that
1016delimits the end of the enclosing C<\p{}>. Like any pattern, certain
1017other delimiters are terminated by their mirror images. These are
1018C<"(">, C<"[>", and C<"E<lt>">. If the delimiter is any of C<"-">,
1019C<"_">, C<"+">, or C<"\">, or is the same delimiter as is used for the
1020enclosing pattern, it must be be preceded by a backslash escape, both
1021fore and aft.
1022
1023Beware of using C<"$"> to indicate to match the end of the string. It
1024can too easily be interpreted as being a punctuation variable, like
1025C<$/>.
1026
1027No modifiers may follow the final delimiter. Instead, use
1028L<perlre/(?adlupimnsx-imnsx)> and/or
1029L<perlre/(?adluimnsx-imnsx:pattern)> to specify modifiers.
4829f32d
KW
1030However, certain modifiers are illegal in your wildcard subpattern.
1031The only character set modifier specifiable is C</aa>;
1032any other character set, and C<-m>, and C<p>, and C<s> are all illegal.
1033Specifying modifiers like C<qr/.../gc> that aren't legal in the
1034C<(?...)> notation normally raise a warning, but with wildcard
1035subpatterns, their use is an error. The C<m> modifier is ineffective;
1036everything that matches will be a single line.
1037
1038By default, your pattern is matched case-insensitively, as if C</i> had
1039been specified. You can change this by saying C<(?-i)> in your pattern.
1040
1041There are also certain operations that are illegal. You can't nest
1042C<\p{...}> and C<\P{...}> calls within a wildcard subpattern, and C<\G>
1043doesn't make sense, so is also prohibited.
1044
1045And the C<*> quantifier (or its equivalent C<(0,}>) is illegal.
1532347b
KW
1046
1047This feature is not available when the left-hand side is prefixed by
1048C<Is_>, nor for any form that is marked as "Discouraged" in
1049L<perluniprops/Discouraged>.
1050
1532347b
KW
1051This experimental feature has been added to begin to implement
1052L<https://www.unicode.org/reports/tr18/#Wildcard_Properties>. Using it
1053will raise a (default-on) warning in the
1054C<experimental::uniprop_wildcards> category. We reserve the right to
1055change its operation as we gain experience.
1056
1057Your subpattern can be just about anything, but for it to have some
1058utility, it should match when called with either or both of
1059a) the full name of the property value with underscores (and/or spaces
1060in the Block property) and some things uppercase; or b) the property
1061value in all lowercase with spaces and underscores squeezed out. For
1062example,
1063
1064 qr!\p{Blk=/Old I.*/}!
1065 qr!\p{Blk=/oldi.*/}!
1066
1067would match the same things.
1068
1069A warning is issued if none of the legal values for a property are
1070matched by your pattern. It's likely that a future release will raise a
1071warning if your pattern ends up causing every possible code point to
1072match.
1073
1074Another example that shows that within C<\p{...}>, C</x> isn't needed to
1075have spaces:
1076
1077 qr!\p{scx= /Hebrew|Greek/ }!
1078
1079To be safe, we should have anchored the above example, to prevent
01d49772 1080matches for something like C<Hebrew_Braille>, but there aren't
1532347b
KW
1081any script names like that.
1082
1083There are certain properties that it doesn't currently work with. These
1084are:
1085
1086 Bidi Mirroring Glyph
1087 Bidi Paired Bracket
1088 Case Folding
1089 Decomposition Mapping
1090 Equivalent Unified Ideograph
1091 Name
1092 Name Alias
1093 Lowercase Mapping
1094 NFKC Case Fold
1095 Titlecase Mapping
1096 Uppercase Mapping
1097
1098Nor is the C<@I<unicode_property>@> form implemented.
1099
1100Here's a complete example of matching IPV4 internet protocol addresses
1101in any (single) script
1102
1532347b
KW
1103 no warnings 'experimental::regex_sets';
1104 no warnings 'experimental::uniprop_wildcards';
1105
1106 # Can match a substring, so this intermediate regex needs to have
1107 # context or anchoring in its final use. Using nt=de yields decimal
1108 # digits. When specifying a subset of these, we must include \d to
1109 # prevent things like U+00B2 SUPERSCRIPT TWO from matching
1110 my $zero_through_255 =
1111 qr/ \b (*sr: # All from same sript
1112 (?[ \p{nv=0} & \d ])* # Optional leading zeros
1113 ( # Then one of:
1114 \d{1,2} # 0 - 99
1115 | (?[ \p{nv=1} & \d ]) \d{2} # 100 - 199
1116 | (?[ \p{nv=2} & \d ])
1117 ( (?[ \p{nv=:[0-4]:} & \d ]) \d # 200 - 249
1118 | (?[ \p{nv=5} & \d ])
1119 (?[ \p{nv=:[0-5]:} & \d ]) # 250 - 255
1120 )
1121 )
1122 )
1123 \b
1124 /x;
1125
1126 my $ipv4 = qr/ \A (*sr: $zero_through_255
1127 (?: [.] $zero_through_255 ) {3}
1128 )
1129 \z
1130 /x;
a9130ea9 1131
376d9008 1132=head2 User-Defined Character Properties
491fd90a 1133
51f494cc 1134You can define your own binary character properties by defining subroutines
a9130ea9 1135whose names begin with C<"In"> or C<"Is">. (The experimental feature
9d1a5160
KW
1136L<perlre/(?[ ])> provides an alternative which allows more complex
1137definitions.) The subroutines can be defined in any
1c2f3d7a
KW
1138package. They override any Unicode properties expressed as the same
1139names. The user-defined properties can be used in the regular
1140expression
a9130ea9 1141C<\p{}> and C<\P{}> constructs; if you are using a user-defined property from a
51f494cc 1142package other than the one you are in, you must specify its package in the
a9130ea9 1143C<\p{}> or C<\P{}> construct.
bac0b425 1144
51f494cc 1145 # assuming property Is_Foreign defined in Lang::
bac0b425
JP
1146 package main; # property package name required
1147 if ($txt =~ /\p{Lang::IsForeign}+/) { ... }
1148
1149 package Lang; # property package name not required
1150 if ($txt =~ /\p{IsForeign}+/) { ... }
1151
1152
1153Note that the effect is compile-time and immutable once defined.
b19eb496
TC
1154However, the subroutines are passed a single parameter, which is 0 if
1155case-sensitive matching is in effect and non-zero if caseless matching
56ca34ca
KW
1156is in effect. The subroutine may return different values depending on
1157the value of the flag, and one set of values will immutably be in effect
b19eb496 1158for all case-sensitive matches, and the other set for all case-insensitive
56ca34ca 1159matches.
491fd90a 1160
b19eb496 1161Note that if the regular expression is tainted, then Perl will die rather
a9130ea9 1162than calling the subroutine when the name of the subroutine is
0e9be77f
DM
1163determined by the tainted data.
1164
376d9008
JB
1165The subroutines must return a specially-formatted string, with one
1166or more newline-separated lines. Each line must be one of the following:
491fd90a
JH
1167
1168=over 4
1169
1170=item *
1171
df9e1087 1172A single hexadecimal number denoting a code point to include.
510254c9
A
1173
1174=item *
1175
99a6b1f0 1176Two hexadecimal numbers separated by horizontal whitespace (space or
73b95840
KW
1177tabular characters) denoting a range of code points to include. The
1178second number must not be smaller than the first.
491fd90a
JH
1179
1180=item *
1181
a9130ea9
KW
1182Something to include, prefixed by C<"+">: a built-in character
1183property (prefixed by C<"utf8::">) or a fully qualified (including package
830137a2 1184name) user-defined character property,
bac0b425
JP
1185to represent all the characters in that property; two hexadecimal code
1186points for a range; or a single hexadecimal code point.
491fd90a
JH
1187
1188=item *
1189
a9130ea9
KW
1190Something to exclude, prefixed by C<"-">: an existing character
1191property (prefixed by C<"utf8::">) or a fully qualified (including package
830137a2 1192name) user-defined character property,
bac0b425
JP
1193to represent all the characters in that property; two hexadecimal code
1194points for a range; or a single hexadecimal code point.
491fd90a
JH
1195
1196=item *
1197
a9130ea9
KW
1198Something to negate, prefixed C<"!">: an existing character
1199property (prefixed by C<"utf8::">) or a fully qualified (including package
830137a2 1200name) user-defined character property,
bac0b425
JP
1201to represent all the characters in that property; two hexadecimal code
1202points for a range; or a single hexadecimal code point.
1203
1204=item *
1205
a9130ea9
KW
1206Something to intersect with, prefixed by C<"&">: an existing character
1207property (prefixed by C<"utf8::">) or a fully qualified (including package
830137a2 1208name) user-defined character property,
bac0b425
JP
1209for all the characters except the characters in the property; two
1210hexadecimal code points for a range; or a single hexadecimal code point.
491fd90a
JH
1211
1212=back
1213
1214For example, to define a property that covers both the Japanese
1215syllabaries (hiragana and katakana), you can define
1216
1217 sub InKana {
d88362ca 1218 return <<END;
d5822f25
A
1219 3040\t309F
1220 30A0\t30FF
491fd90a
JH
1221 END
1222 }
1223
d5822f25
A
1224Imagine that the here-doc end marker is at the beginning of the line.
1225Now you can use C<\p{InKana}> and C<\P{InKana}>.
491fd90a
JH
1226
1227You could also have used the existing block property names:
1228
1229 sub InKana {
d88362ca 1230 return <<'END';
491fd90a
JH
1231 +utf8::InHiragana
1232 +utf8::InKatakana
1233 END
1234 }
1235
1236Suppose you wanted to match only the allocated characters,
d5822f25 1237not the raw block ranges: in other words, you want to remove
b65e6125 1238the unassigned characters:
491fd90a
JH
1239
1240 sub InKana {
d88362ca 1241 return <<'END';
491fd90a
JH
1242 +utf8::InHiragana
1243 +utf8::InKatakana
1244 -utf8::IsCn
1245 END
1246 }
1247
1248The negation is useful for defining (surprise!) negated classes.
1249
1250 sub InNotKana {
d88362ca 1251 return <<'END';
491fd90a
JH
1252 !utf8::InHiragana
1253 -utf8::InKatakana
1254 +utf8::IsCn
1255 END
1256 }
1257
461020ad
KW
1258This will match all non-Unicode code points, since every one of them is
1259not in Kana. You can use intersection to exclude these, if desired, as
1260this modified example shows:
bac0b425 1261
461020ad 1262 sub InNotKana {
bac0b425 1263 return <<'END';
461020ad
KW
1264 !utf8::InHiragana
1265 -utf8::InKatakana
1266 +utf8::IsCn
1267 &utf8::Any
bac0b425
JP
1268 END
1269 }
1270
461020ad
KW
1271C<&utf8::Any> must be the last line in the definition.
1272
1273Intersection is used generally for getting the common characters matched
a9130ea9 1274by two (or more) classes. It's important to remember not to use C<"&"> for
461020ad 1275the first set; that would be intersecting with nothing, resulting in an
5acbde07 1276empty set. (Similarly using C<"-"> for the first set does nothing).
461020ad 1277
2d88a86a
KW
1278Unlike non-user-defined C<\p{}> property matches, no warning is ever
1279generated if these properties are matched against a non-Unicode code
1280point (see L</Beyond Unicode code points> below).
bac0b425 1281
68585b5e 1282=head2 User-Defined Case Mappings (for serious hackers only)
822502e5 1283
5d1892be 1284B<This feature has been removed as of Perl 5.16.>
a9130ea9 1285The CPAN module C<L<Unicode::Casing>> provides better functionality without
5d1892be
KW
1286the drawbacks that this feature had. If you are using a Perl earlier
1287than 5.16, this feature was most fully documented in the 5.14 version of
1288this pod:
1289L<http://perldoc.perl.org/5.14.0/perlunicode.html#User-Defined-Case-Mappings-%28for-serious-hackers-only%29>
3a2263fe 1290
376d9008 1291=head2 Character Encodings for Input and Output
8cbd9a7a 1292
7221edc9 1293See L<Encode>.
8cbd9a7a 1294
c29a771d 1295=head2 Unicode Regular Expression Support Level
776f8809 1296
b19eb496 1297The following list of Unicode supported features for regular expressions describes
fea12a3e
KW
1298all features currently directly supported by core Perl. The references
1299to "Level I<N>" and the section numbers refer to
71c89d21 1300L<UTS#18 "Unicode Regular Expressions"|https://www.unicode.org/reports/tr18>,
526f2ca9 1301version 18, October 2016.
fea12a3e
KW
1302
1303=head3 Level 1 - Basic Unicode Support
1304
1305 RL1.1 Hex Notation - Done [1]
1306 RL1.2 Properties - Done [2]
1307 RL1.2a Compatibility Properties - Done [3]
1308 RL1.3 Subtraction and Intersection - Experimental [4]
1309 RL1.4 Simple Word Boundaries - Done [5]
1310 RL1.5 Simple Loose Matches - Done [6]
1311 RL1.6 Line Boundaries - Partial [7]
1312 RL1.7 Supplementary Code Points - Done [8]
755789c0 1313
6f33e417
KW
1314=over 4
1315
a6a7eedc 1316=item [1] C<\N{U+...}> and C<\x{...}>
6f33e417 1317
fea12a3e
KW
1318=item [2]
1319C<\p{...}> C<\P{...}>. This requirement is for a minimal list of
58f92e50 1320properties. Perl supports these. See R2.7 for other properties.
6f33e417 1321
fea12a3e
KW
1322=item [3]
1323Perl has C<\d> C<\D> C<\s> C<\S> C<\w> C<\W> C<\X> C<[:I<prop>:]>
1324C<[:^I<prop>:]>, plus all the properties specified by
71c89d21 1325L<https://www.unicode.org/reports/tr18/#Compatibility_Properties>. These
fea12a3e 1326are described above in L</Other Properties>
6f33e417 1327
fea12a3e 1328=item [4]
6f33e417 1329
fea12a3e 1330The experimental feature C<"(?[...])"> starting in v5.18 accomplishes
a6a7eedc 1331this.
6f33e417 1332
a6a7eedc
KW
1333See L<perlre/(?[ ])>. If you don't want to use an experimental
1334feature, you can use one of the following:
6f33e417
KW
1335
1336=over 4
1337
a6a7eedc 1338=item *
f67a5002 1339Regular expression lookahead
6f33e417
KW
1340
1341You can mimic class subtraction using lookahead.
8158862b 1342For example, what UTS#18 might write as
29bdacb8 1343
209c9685 1344 [{Block=Greek}-[{UNASSIGNED}]]
dbe420b4
JH
1345
1346in Perl can be written as:
1347
209c9685
KW
1348 (?!\p{Unassigned})\p{Block=Greek}
1349 (?=\p{Assigned})\p{Block=Greek}
dbe420b4
JH
1350
1351But in this particular example, you probably really want
1352
209c9685 1353 \p{Greek}
dbe420b4
JH
1354
1355which will match assigned characters known to be part of the Greek script.
29bdacb8 1356
a6a7eedc
KW
1357=item *
1358
1359CPAN module C<L<Unicode::Regex::Set>>
8158862b 1360
6f33e417
KW
1361It does implement the full UTS#18 grouping, intersection, union, and
1362removal (subtraction) syntax.
8158862b 1363
a6a7eedc
KW
1364=item *
1365
1366L</"User-Defined Character Properties">
6f33e417 1367
a9130ea9 1368C<"+"> for union, C<"-"> for removal (set-difference), C<"&"> for intersection
6f33e417
KW
1369
1370=back
1371
fea12a3e
KW
1372=item [5]
1373C<\b> C<\B> meet most, but not all, the details of this requirement, but
1374C<\b{wb}> and C<\B{wb}> do, as well as the stricter R2.3.
1375
1376=item [6]
6f33e417 1377
a6a7eedc 1378Note that Perl does Full case-folding in matching, not Simple:
6f33e417 1379
a6a7eedc
KW
1380For example C<U+1F88> is equivalent to C<U+1F00 U+03B9>, instead of just
1381C<U+1F80>. This difference matters mainly for certain Greek capital
a9130ea9
KW
1382letters with certain modifiers: the Full case-folding decomposes the
1383letter, while the Simple case-folding would map it to a single
1384character.
6f33e417 1385
fea12a3e
KW
1386=item [7]
1387
1388The reason this is considered to be only partially implemented is that
1389Perl has L<C<qrE<sol>\b{lb}E<sol>>|perlrebackslash/\b{lb}> and
1390C<L<Unicode::LineBreak>> that are conformant with
71c89d21 1391L<UAX#14 "Unicode Line Breaking Algorithm"|https://www.unicode.org/reports/tr14>.
fea12a3e
KW
1392The regular expression construct provides default behavior, while the
1393heavier-weight module provides customizable line breaking.
1394
1395But Perl treats C<\n> as the start- and end-line
1396delimiter, whereas Unicode specifies more characters that should be
1397so-interpreted.
6f33e417 1398
a6a7eedc 1399These are:
6f33e417 1400
a6a7eedc
KW
1401 VT U+000B (\v in C)
1402 FF U+000C (\f)
1403 CR U+000D (\r)
1404 NEL U+0085
1405 LS U+2028
1406 PS U+2029
6f33e417 1407
a6a7eedc
KW
1408C<^> and C<$> in regular expression patterns are supposed to match all
1409these, but don't.
1410These characters also don't, but should, affect C<< <> >> C<$.>, and
1411script line numbers.
6f33e417 1412
a6a7eedc
KW
1413Also, lines should not be split within C<CRLF> (i.e. there is no
1414empty line between C<\r> and C<\n>). For C<CRLF>, try the C<:crlf>
1415layer (see L<PerlIO>).
1416
fea12a3e 1417=item [8]
a9130ea9
KW
1418UTF-8/UTF-EBDDIC used in Perl allows not only C<U+10000> to
1419C<U+10FFFF> but also beyond C<U+10FFFF>
6f33e417
KW
1420
1421=back
5ca1ac52 1422
fea12a3e 1423=head3 Level 2 - Extended Unicode Support
776f8809 1424
fea12a3e
KW
1425 RL2.1 Canonical Equivalents - Retracted [9]
1426 by Unicode
58f92e50
KW
1427 RL2.2 Extended Grapheme Clusters and - Partial [10]
1428 Character Classes with Strings
fea12a3e
KW
1429 RL2.3 Default Word Boundaries - Done [11]
1430 RL2.4 Default Case Conversion - Done
1431 RL2.5 Name Properties - Done
1532347b 1432 RL2.6 Wildcards in Property Values - Partial [12]
58f92e50
KW
1433 RL2.7 Full Properties - Partial [13]
1434 RL2.8 Optional Properties - Partial [14]
776f8809 1435
fea12a3e 1436=over 4
8158862b 1437
fea12a3e
KW
1438=item [9]
1439Unicode has rewritten this portion of UTS#18 to say that getting
1440canonical equivalence (see UAX#15
71c89d21 1441L<"Unicode Normalization Forms"|https://www.unicode.org/reports/tr15>)
fea12a3e
KW
1442is basically to be done at the programmer level. Use NFD to write
1443both your regular expressions and text to match them against (you
1444can use L<Unicode::Normalize>).
776f8809 1445
fea12a3e 1446=item [10]
58f92e50
KW
1447Perl has C<\X> and C<\b{gcb}>. Unicode has retracted their "Grapheme
1448Cluster Mode", and recently added string properties, which Perl does not
1449yet support.
fea12a3e
KW
1450
1451=item [11] see
71c89d21 1452L<UAX#29 "Unicode Text Segmentation"|https://www.unicode.org/reports/tr29>,
fea12a3e 1453
1532347b
KW
1454=item [12] see
1455L</Wildcards in Property Values> above.
1456
526f2ca9 1457=item [13]
58f92e50
KW
1458Perl supports all the properties in the Unicode Character Database
1459(UCD). It does not yet support the listed properties that come from
1460other Unicode sources.
776f8809 1461
526f2ca9 1462=item [14]
58f92e50
KW
1463The only optional property that Perl supports is Named Sequence. None
1464of these properties are in the UCD.
776f8809
JH
1465
1466=back
1467
58f92e50
KW
1468=head3 Level 3 - Tailored Support
1469
1470This has been retracted by Unicode.
1471
c349b1b9
JH
1472=head2 Unicode Encodings
1473
376d9008
JB
1474Unicode characters are assigned to I<code points>, which are abstract
1475numbers. To use these numbers, various encodings are needed.
c349b1b9
JH
1476
1477=over 4
1478
c29a771d 1479=item *
5cb3728c
RB
1480
1481UTF-8
c349b1b9 1482
6d4f9cf2 1483UTF-8 is a variable-length (1 to 4 bytes), byte-order independent
a6a7eedc
KW
1484encoding. In most of Perl's documentation, including elsewhere in this
1485document, the term "UTF-8" means also "UTF-EBCDIC". But in this section,
1486"UTF-8" refers only to the encoding used on ASCII platforms. It is a
1487superset of 7-bit US-ASCII, so anything encoded in ASCII has the
1488identical representation when encoded in UTF-8.
c349b1b9 1489
8c007b5a 1490The following table is from Unicode 3.2.
05632f9a 1491
755789c0 1492 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
05632f9a 1493
d88362ca 1494 U+0000..U+007F 00..7F
e1b711da 1495 U+0080..U+07FF * C2..DF 80..BF
d88362ca 1496 U+0800..U+0FFF E0 * A0..BF 80..BF
ec90690f
ST
1497 U+1000..U+CFFF E1..EC 80..BF 80..BF
1498 U+D000..U+D7FF ED 80..9F 80..BF
755789c0 1499 U+D800..U+DFFF +++++ utf16 surrogates, not legal utf8 +++++
ec90690f 1500 U+E000..U+FFFF EE..EF 80..BF 80..BF
d88362ca
KW
1501 U+10000..U+3FFFF F0 * 90..BF 80..BF 80..BF
1502 U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
1503 U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
e1b711da 1504
b19eb496 1505Note the gaps marked by "*" before several of the byte entries above. These are
e1b711da
KW
1506caused by legal UTF-8 avoiding non-shortest encodings: it is technically
1507possible to UTF-8-encode a single code point in different ways, but that is
1508explicitly forbidden, and the shortest possible encoding should always be used
1509(and that is what Perl does).
37361303 1510
376d9008 1511Another way to look at it is via bits:
05632f9a 1512
755789c0 1513 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
05632f9a 1514
755789c0
KW
1515 0aaaaaaa 0aaaaaaa
1516 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
1517 ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
1518 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
05632f9a 1519
a9130ea9 1520As you can see, the continuation bytes all begin with C<"10">, and the
e1b711da 1521leading bits of the start byte tell how many bytes there are in the
05632f9a
JH
1522encoded character.
1523
6d4f9cf2 1524The original UTF-8 specification allowed up to 6 bytes, to allow
a9130ea9 1525encoding of numbers up to C<0x7FFF_FFFF>. Perl continues to allow those,
6d4f9cf2
KW
1526and has extended that up to 13 bytes to encode code points up to what
1527can fit in a 64-bit word. However, Perl will warn if you output any of
b19eb496 1528these as being non-portable; and under strict UTF-8 input protocols,
526f2ca9 1529they are forbidden. In addition, it is now illegal to use a code point
760c7c2f
KW
1530larger than what a signed integer variable on your system can hold. On
153132-bit ASCII systems, this means C<0x7FFF_FFFF> is the legal maximum
526f2ca9 1532(much higher on 64-bit systems).
6d4f9cf2 1533
c29a771d 1534=item *
5cb3728c
RB
1535
1536UTF-EBCDIC
dbe420b4 1537
b65e6125 1538Like UTF-8, but EBCDIC-safe, in the way that UTF-8 is ASCII-safe.
a6a7eedc
KW
1539This means that all the basic characters (which includes all
1540those that have ASCII equivalents (like C<"A">, C<"0">, C<"%">, I<etc.>)
1541are the same in both EBCDIC and UTF-EBCDIC.)
1542
c0236afe
KW
1543UTF-EBCDIC is used on EBCDIC platforms. It generally requires more
1544bytes to represent a given code point than UTF-8 does; the largest
1545Unicode code points take 5 bytes to represent (instead of 4 in UTF-8),
1546and, extended for 64-bit words, it uses 14 bytes instead of 13 bytes in
1547UTF-8.
dbe420b4 1548
c29a771d 1549=item *
5cb3728c 1550
b65e6125 1551UTF-16, UTF-16BE, UTF-16LE, Surrogates, and C<BOM>'s (Byte Order Marks)
c349b1b9 1552
1bfb14c4
JH
1553The followings items are mostly for reference and general Unicode
1554knowledge, Perl doesn't use these constructs internally.
dbe420b4 1555
b19eb496
TC
1556Like UTF-8, UTF-16 is a variable-width encoding, but where
1557UTF-8 uses 8-bit code units, UTF-16 uses 16-bit code units.
1558All code points occupy either 2 or 4 bytes in UTF-16: code points
1559C<U+0000..U+FFFF> are stored in a single 16-bit unit, and code
1bfb14c4 1560points C<U+10000..U+10FFFF> in two 16-bit units. The latter case is
c349b1b9
JH
1561using I<surrogates>, the first 16-bit unit being the I<high
1562surrogate>, and the second being the I<low surrogate>.
1563
376d9008 1564Surrogates are code points set aside to encode the C<U+10000..U+10FFFF>
c349b1b9 1565range of Unicode code points in pairs of 16-bit units. The I<high
9f815e24 1566surrogates> are the range C<U+D800..U+DBFF> and the I<low surrogates>
376d9008 1567are the range C<U+DC00..U+DFFF>. The surrogate encoding is
c349b1b9 1568
d88362ca
KW
1569 $hi = ($uni - 0x10000) / 0x400 + 0xD800;
1570 $lo = ($uni - 0x10000) % 0x400 + 0xDC00;
c349b1b9
JH
1571
1572and the decoding is
1573
d88362ca 1574 $uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00);
c349b1b9 1575
376d9008 1576Because of the 16-bitness, UTF-16 is byte-order dependent. UTF-16
c349b1b9 1577itself can be used for in-memory computations, but if storage or
376d9008
JB
1578transfer is required either UTF-16BE (big-endian) or UTF-16LE
1579(little-endian) encodings must be chosen.
c349b1b9
JH
1580
1581This introduces another problem: what if you just know that your data
376d9008 1582is UTF-16, but you don't know which endianness? Byte Order Marks, or
b65e6125 1583C<BOM>'s, are a solution to this. A special character has been reserved
86bbd6d1 1584in Unicode to function as a byte order marker: the character with the
a9130ea9 1585code point C<U+FEFF> is the C<BOM>.
042da322 1586
a9130ea9 1587The trick is that if you read a C<BOM>, you will know the byte order,
376d9008
JB
1588since if it was written on a big-endian platform, you will read the
1589bytes C<0xFE 0xFF>, but if it was written on a little-endian platform,
1590you will read the bytes C<0xFF 0xFE>. (And if the originating platform
b65e6125
KW
1591was writing in ASCII platform UTF-8, you will read the bytes
1592C<0xEF 0xBB 0xBF>.)
042da322 1593
86bbd6d1 1594The way this trick works is that the character with the code point
6d4f9cf2 1595C<U+FFFE> is not supposed to be in input streams, so the
a9130ea9 1596sequence of bytes C<0xFF 0xFE> is unambiguously "C<BOM>, represented in
1bfb14c4 1597little-endian format" and cannot be C<U+FFFE>, represented in big-endian
6d4f9cf2
KW
1598format".
1599
1600Surrogates have no meaning in Unicode outside their use in pairs to
1601represent other code points. However, Perl allows them to be
1602represented individually internally, for example by saying
f651977e
TC
1603C<chr(0xD801)>, so that all code points, not just those valid for open
1604interchange, are
6d4f9cf2 1605representable. Unicode does define semantics for them, such as their
a9130ea9
KW
1606C<L</General_Category>> is C<"Cs">. But because their use is somewhat dangerous,
1607Perl will warn (using the warning category C<"surrogate">, which is a
1608sub-category of C<"utf8">) if an attempt is made
6d4f9cf2
KW
1609to do things like take the lower case of one, or match
1610case-insensitively, or to output them. (But don't try this on Perls
1611before 5.14.)
c349b1b9 1612
c29a771d 1613=item *
5cb3728c 1614
1e54db1a 1615UTF-32, UTF-32BE, UTF-32LE
c349b1b9 1616
b65e6125 1617The UTF-32 family is pretty much like the UTF-16 family, except that
042da322 1618the units are 32-bit, and therefore the surrogate scheme is not
a9130ea9 1619needed. UTF-32 is a fixed-width encoding. The C<BOM> signatures are
b19eb496 1620C<0x00 0x00 0xFE 0xFF> for BE and C<0xFF 0xFE 0x00 0x00> for LE.
c349b1b9 1621
c29a771d 1622=item *
5cb3728c
RB
1623
1624UCS-2, UCS-4
c349b1b9 1625
b19eb496 1626Legacy, fixed-width encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit
376d9008 1627encoding. Unlike UTF-16, UCS-2 is not extensible beyond C<U+FFFF>,
339cfa0e 1628because it does not use surrogates. UCS-4 is a 32-bit encoding,
b19eb496 1629functionally identical to UTF-32 (the difference being that
a9130ea9 1630UCS-4 forbids neither surrogates nor code points larger than C<0x10_FFFF>).
c349b1b9 1631
c29a771d 1632=item *
5cb3728c
RB
1633
1634UTF-7
c349b1b9 1635
376d9008
JB
1636A seven-bit safe (non-eight-bit) encoding, which is useful if the
1637transport or storage is not eight-bit safe. Defined by RFC 2152.
c349b1b9 1638
95a1a48b
JH
1639=back
1640
57e88091 1641=head2 Noncharacter code points
6d4f9cf2 1642
57e88091 164366 code points are set aside in Unicode as "noncharacter code points".
a9130ea9 1644These all have the C<Unassigned> (C<Cn>) C<L</General_Category>>, and
57e88091
KW
1645no character will ever be assigned to any of them. They are the 32 code
1646points between C<U+FDD0> and C<U+FDEF> inclusive, and the 34 code
1647points:
1648
1649 U+FFFE U+FFFF
1650 U+1FFFE U+1FFFF
1651 U+2FFFE U+2FFFF
1652 ...
1653 U+EFFFE U+EFFFF
1654 U+FFFFE U+FFFFF
1655 U+10FFFE U+10FFFF
1656
1657Until Unicode 7.0, the noncharacters were "B<forbidden> for use in open
1658interchange of Unicode text data", so that code that processed those
1659streams could use these code points as sentinels that could be mixed in
1660with character data, and would always be distinguishable from that data.
1661(Emphasis above and in the next paragraph are added in this document.)
1662
1663Unicode 7.0 changed the wording so that they are "B<not recommended> for
1664use in open interchange of Unicode text data". The 7.0 Standard goes on
1665to say:
1666
1667=over 4
1668
1669"If a noncharacter is received in open interchange, an application is
1670not required to interpret it in any way. It is good practice, however,
1671to recognize it as a noncharacter and to take appropriate action, such
1672as replacing it with C<U+FFFD> replacement character, to indicate the
1673problem in the text. It is not recommended to simply delete
1674noncharacter code points from such text, because of the potential
1675security issues caused by deleting uninterpreted characters. (See
1676conformance clause C7 in Section 3.2, Conformance Requirements, and
1677L<Unicode Technical Report #36, "Unicode Security
71c89d21 1678Considerations"|https://www.unicode.org/reports/tr36/#Substituting_for_Ill_Formed_Subsequences>)."
57e88091
KW
1679
1680=back
1681
1682This change was made because it was found that various commercial tools
1683like editors, or for things like source code control, had been written
1684so that they would not handle program files that used these code points,
1685effectively precluding their use almost entirely! And that was never
1686the intent. They've always been meant to be usable within an
1687application, or cooperating set of applications, at will.
1688
1689If you're writing code, such as an editor, that is supposed to be able
1690to handle any Unicode text data, then you shouldn't be using these code
1691points yourself, and instead allow them in the input. If you need
1692sentinels, they should instead be something that isn't legal Unicode.
1693For UTF-8 data, you can use the bytes 0xC1 and 0xC2 as sentinels, as
1694they never appear in well-formed UTF-8. (There are equivalents for
1695UTF-EBCDIC). You can also store your Unicode code points in integer
1696variables and use negative values as sentinels.
1697
1698If you're not writing such a tool, then whether you accept noncharacters
1699as input is up to you (though the Standard recommends that you not). If
1700you do strict input stream checking with Perl, these code points
1701continue to be forbidden. This is to maintain backward compatibility
1702(otherwise potential security holes could open up, as an unsuspecting
1703application that was written assuming the noncharacters would be
1704filtered out before getting to it, could now, without warning, start
1705getting them). To do strict checking, you can use the layer
1706C<:encoding('UTF-8')>.
1707
1708Perl continues to warn (using the warning category C<"nonchar">, which
1709is a sub-category of C<"utf8">) if an attempt is made to output
1710noncharacters.
42581d5d
KW
1711
1712=head2 Beyond Unicode code points
1713
a9130ea9
KW
1714The maximum Unicode code point is C<U+10FFFF>, and Unicode only defines
1715operations on code points up through that. But Perl works on code
526f2ca9 1716points up to the maximum permissible signed number available on the
42581d5d
KW
1717platform. However, Perl will not accept these from input streams unless
1718lax rules are being used, and will warn (using the warning category
2d88a86a
KW
1719C<"non_unicode">, which is a sub-category of C<"utf8">) if any are output.
1720
1721Since Unicode rules are not defined on these code points, if a
1722Unicode-defined operation is done on them, Perl uses what we believe are
1723sensible rules, while generally warning, using the C<"non_unicode">
1724category. For example, C<uc("\x{11_0000}")> will generate such a
1725warning, returning the input parameter as its result, since Perl defines
1726the uppercase of every non-Unicode code point to be the code point
b65e6125
KW
1727itself. (All the case changing operations, not just uppercasing, work
1728this way.)
2d88a86a
KW
1729
1730The situation with matching Unicode properties in regular expressions,
1731the C<\p{}> and C<\P{}> constructs, against these code points is not as
1732clear cut, and how these are handled has changed as we've gained
1733experience.
1734
1735One possibility is to treat any match against these code points as
1736undefined. But since Perl doesn't have the concept of a match being
1737undefined, it converts this to failing or C<FALSE>. This is almost, but
1738not quite, what Perl did from v5.14 (when use of these code points
1739became generally reliable) through v5.18. The difference is that Perl
1740treated all C<\p{}> matches as failing, but all C<\P{}> matches as
1741succeeding.
1742
f66ccb6c 1743One problem with this is that it leads to unexpected, and confusing
2d88a86a
KW
1744results in some cases:
1745
1746 chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Failed on <= v5.18
1747 chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Failed! on <= v5.18
1748
1749That is, it treated both matches as undefined, and converted that to
1750false (raising a warning on each). The first case is the expected
1751result, but the second is likely counterintuitive: "How could both be
1752false when they are complements?" Another problem was that the
1753implementation optimized many Unicode property matches down to already
1754existing simpler, faster operations, which don't raise the warning. We
1755chose to not forgo those optimizations, which help the vast majority of
1756matches, just to generate a warning for the unlikely event that an
1757above-Unicode code point is being matched against.
1758
1759As a result of these problems, starting in v5.20, what Perl does is
1760to treat non-Unicode code points as just typical unassigned Unicode
1761characters, and matches accordingly. (Note: Unicode has atypical
57e88091 1762unassigned code points. For example, it has noncharacter code points,
2d88a86a
KW
1763and ones that, when they do get assigned, are destined to be written
1764Right-to-left, as Arabic and Hebrew are. Perl assumes that no
1765non-Unicode code point has any atypical properties.)
1766
1767Perl, in most cases, will raise a warning when matching an above-Unicode
1768code point against a Unicode property when the result is C<TRUE> for
1769C<\p{}>, and C<FALSE> for C<\P{}>. For example:
1770
1771 chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Fails, no warning
1772 chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Succeeds, with warning
1773
1774In both these examples, the character being matched is non-Unicode, so
1775Unicode doesn't define how it should match. It clearly isn't an ASCII
1776hex digit, so the first example clearly should fail, and so it does,
1777with no warning. But it is arguable that the second example should have
1778an undefined, hence C<FALSE>, result. So a warning is raised for it.
1779
1780Thus the warning is raised for many fewer cases than in earlier Perls,
1781and only when what the result is could be arguable. It turns out that
1782none of the optimizations made by Perl (or are ever likely to be made)
1783cause the warning to be skipped, so it solves both problems of Perl's
1784earlier approach. The most commonly used property that is affected by
1785this change is C<\p{Unassigned}> which is a short form for
1786C<\p{General_Category=Unassigned}>. Starting in v5.20, all non-Unicode
1787code points are considered C<Unassigned>. In earlier releases the
1788matches failed because the result was considered undefined.
1789
1790The only place where the warning is not raised when it might ought to
1791have been is if optimizations cause the whole pattern match to not even
1792be attempted. For example, Perl may figure out that for a string to
1793match a certain regular expression pattern, the string has to contain
1794the substring C<"foobar">. Before attempting the match, Perl may look
1795for that substring, and if not found, immediately fail the match without
1796actually trying it; so no warning gets generated even if the string
1797contains an above-Unicode code point.
1798
1799This behavior is more "Do what I mean" than in earlier Perls for most
1800applications. But it catches fewer issues for code that needs to be
1801strictly Unicode compliant. Therefore there is an additional mode of
1802operation available to accommodate such code. This mode is enabled if a
1803regular expression pattern is compiled within the lexical scope where
1804the C<"non_unicode"> warning class has been made fatal, say by:
1805
1806 use warnings FATAL => "non_unicode"
1807
44ecbbd8 1808(see L<warnings>). In this mode of operation, Perl will raise the
2d88a86a
KW
1809warning for all matches against a non-Unicode code point (not just the
1810arguable ones), and it skips the optimizations that might cause the
1811warning to not be output. (It currently still won't warn if the match
1812isn't even attempted, like in the C<"foobar"> example above.)
1813
1814In summary, Perl now normally treats non-Unicode code points as typical
1815Unicode unassigned code points for regular expression matches, raising a
1816warning only when it is arguable what the result should be. However, if
1817this warning has been made fatal, it isn't skipped.
1818
1819There is one exception to all this. C<\p{All}> looks like a Unicode
1820property, but it is a Perl extension that is defined to be true for all
1821possible code points, Unicode or not, so no warning is ever generated
1822when matching this against a non-Unicode code point. (Prior to v5.20,
1823it was an exact synonym for C<\p{Any}>, matching code points C<0>
1824through C<0x10FFFF>.)
6d4f9cf2 1825
0d7c09bb
JH
1826=head2 Security Implications of Unicode
1827
b65e6125 1828First, read
71c89d21 1829L<Unicode Security Considerations|https://www.unicode.org/reports/tr36>.
b65e6125 1830
e1b711da
KW
1831Also, note the following:
1832
0d7c09bb
JH
1833=over 4
1834
1835=item *
1836
1837Malformed UTF-8
bf0fa0b2 1838
f57d8456
KW
1839UTF-8 is very structured, so many combinations of bytes are invalid. In
1840the past, Perl tried to soldier on and make some sense of invalid
1841combinations, but this can lead to security holes, so now, if the Perl
1842core needs to process an invalid combination, it will either raise a
1843fatal error, or will replace those bytes by the sequence that forms the
1844Unicode REPLACEMENT CHARACTER, for which purpose Unicode created it.
1845
1846Every code point can be represented by more than one possible
1847syntactically valid UTF-8 sequence. Early on, both Unicode and Perl
1848considered any of these to be valid, but now, all sequences longer
1849than the shortest possible one are considered to be malformed.
1850
1851Unicode considers many code points to be illegal, or to be avoided.
1852Perl generally accepts them, once they have passed through any input
1853filters that may try to exclude them. These have been discussed above
1854(see "Surrogates" under UTF-16 in L</Unicode Encodings>,
1855L</Noncharacter code points>, and L</Beyond Unicode code points>).
bf0fa0b2 1856
0d7c09bb
JH
1857=item *
1858
68693f9e 1859Regular expression pattern matching may surprise you if you're not
b19eb496
TC
1860accustomed to Unicode. Starting in Perl 5.14, several pattern
1861modifiers are available to control this, called the character set
42581d5d
KW
1862modifiers. Details are given in L<perlre/Character set modifiers>.
1863
1864=back
0d7c09bb 1865
376d9008 1866As discussed elsewhere, Perl has one foot (two hooves?) planted in
a6a7eedc
KW
1867each of two worlds: the old world of ASCII and single-byte locales, and
1868the new world of Unicode, upgrading when necessary.
376d9008 1869If your legacy code does not explicitly use Unicode, no automatic
a6a7eedc 1870switch-over to Unicode should happen.
0d7c09bb 1871
c349b1b9
JH
1872=head2 Unicode in Perl on EBCDIC
1873
a6a7eedc
KW
1874Unicode is supported on EBCDIC platforms. See L<perlebcdic>.
1875
1876Unless ASCII vs. EBCDIC issues are specifically being discussed,
1877references to UTF-8 encoding in this document and elsewhere should be
1878read as meaning UTF-EBCDIC on EBCDIC platforms.
1879See L<perlebcdic/Unicode and UTF>.
1880
1881Because UTF-EBCDIC is so similar to UTF-8, the differences are mostly
1882hidden from you; S<C<use utf8>> (and NOT something like
dabde021 1883S<C<use utfebcdic>>) declares the script is in the platform's
a6a7eedc
KW
1884"native" 8-bit encoding of Unicode. (Similarly for the C<":utf8">
1885layer.)
c349b1b9 1886
b310b053
JH
1887=head2 Locales
1888
42581d5d 1889See L<perllocale/Unicode and UTF-8>
b310b053 1890
1aad1664
JH
1891=head2 When Unicode Does Not Happen
1892
b65e6125
KW
1893There are still many places where Unicode (in some encoding or
1894another) could be given as arguments or received as results, or both in
1895Perl, but it is not, in spite of Perl having extensive ways to input and
1896output in Unicode, and a few other "entry points" like the C<@ARGV>
1897array (which can sometimes be interpreted as UTF-8).
1aad1664 1898
e1b711da
KW
1899The following are such interfaces. Also, see L</The "Unicode Bug">.
1900For all of these interfaces Perl
b9cedb1b 1901currently (as of v5.16.0) simply assumes byte strings both as arguments
b65e6125 1902and results, or UTF-8 strings if the (deprecated) C<encoding> pragma has been used.
1aad1664 1903
b19eb496
TC
1904One reason that Perl does not attempt to resolve the role of Unicode in
1905these situations is that the answers are highly dependent on the operating
1aad1664 1906system and the file system(s). For example, whether filenames can be
b19eb496
TC
1907in Unicode and in exactly what kind of encoding, is not exactly a
1908portable concept. Similarly for C<qx> and C<system>: how well will the
1909"command-line interface" (and which of them?) handle Unicode?
1aad1664
JH
1910
1911=over 4
1912
557a2462
RB
1913=item *
1914
a9130ea9
KW
1915C<chdir>, C<chmod>, C<chown>, C<chroot>, C<exec>, C<link>, C<lstat>, C<mkdir>,
1916C<rename>, C<rmdir>, C<stat>, C<symlink>, C<truncate>, C<unlink>, C<utime>, C<-X>
557a2462
RB
1917
1918=item *
1919
a9130ea9 1920C<%ENV>
557a2462
RB
1921
1922=item *
1923
a9130ea9 1924C<glob> (aka the C<E<lt>*E<gt>>)
557a2462
RB
1925
1926=item *
1aad1664 1927
a9130ea9 1928C<open>, C<opendir>, C<sysopen>
1aad1664 1929
557a2462 1930=item *
1aad1664 1931
a9130ea9 1932C<qx> (aka the backtick operator), C<system>
1aad1664 1933
557a2462 1934=item *
1aad1664 1935
a9130ea9 1936C<readdir>, C<readlink>
1aad1664
JH
1937
1938=back
1939
e1b711da
KW
1940=head2 The "Unicode Bug"
1941
a6a7eedc
KW
1942The term, "Unicode bug" has been applied to an inconsistency with the
1943code points in the C<Latin-1 Supplement> block, that is, between
1944128 and 255. Without a locale specified, unlike all other characters or
1945code points, these characters can have very different semantics
1946depending on the rules in effect. (Characters whose code points are
1947above 255 force Unicode rules; whereas the rules for ASCII characters
1948are the same under both ASCII and Unicode rules.)
1949
1950Under Unicode rules, these upper-Latin1 characters are interpreted as
1951Unicode code points, which means they have the same semantics as Latin-1
1952(ISO-8859-1) and C1 controls.
1953
1954As explained in L</ASCII Rules versus Unicode Rules>, under ASCII rules,
1955they are considered to be unassigned characters.
1956
1957This can lead to unexpected results. For example, a string's
1958semantics can suddenly change if a code point above 255 is appended to
1959it, which changes the rules from ASCII to Unicode. As an
1960example, consider the following program and its output:
1961
1962 $ perl -le'
f434f357 1963 no feature "unicode_strings";
a6a7eedc
KW
1964 $s1 = "\xC2";
1965 $s2 = "\x{2660}";
1966 for ($s1, $s2, $s1.$s2) {
1967 print /\w/ || 0;
1968 }
1969 '
1970 0
1971 0
1972 1
1973
1974If there's no C<\w> in C<s1> nor in C<s2>, why does their concatenation
1975have one?
1976
1977This anomaly stems from Perl's attempt to not disturb older programs that
1978didn't use Unicode, along with Perl's desire to add Unicode support
1979seamlessly. But the result turned out to not be seamless. (By the way,
1980you can choose to be warned when things like this happen. See
1981C<L<encoding::warnings>>.)
1982
1983L<S<C<use feature 'unicode_strings'>>|feature/The 'unicode_strings' feature>
1984was added, starting in Perl v5.12, to address this problem. It affects
1985these things:
e1b711da
KW
1986
1987=over 4
1988
1989=item *
1990
1991Changing the case of a scalar, that is, using C<uc()>, C<ucfirst()>, C<lc()>,
2e2b2571
KW
1992and C<lcfirst()>, or C<\L>, C<\U>, C<\u> and C<\l> in double-quotish
1993contexts, such as regular expression substitutions.
a6a7eedc
KW
1994
1995Under C<unicode_strings> starting in Perl 5.12.0, Unicode rules are
2e2b2571
KW
1996generally used. See L<perlfunc/lc> for details on how this works
1997in combination with various other pragmas.
e1b711da
KW
1998
1999=item *
2000
2e2b2571 2001Using caseless (C</i>) regular expression matching.
a6a7eedc 2002
2e2b2571 2003Starting in Perl 5.14.0, regular expressions compiled within
a6a7eedc 2004the scope of C<unicode_strings> use Unicode rules
2e2b2571
KW
2005even when executed or compiled into larger
2006regular expressions outside the scope.
e1b711da
KW
2007
2008=item *
2009
a6a7eedc
KW
2010Matching any of several properties in regular expressions.
2011
2012These properties are C<\b> (without braces), C<\B> (without braces),
2013C<\s>, C<\S>, C<\w>, C<\W>, and all the Posix character classes
630d17dc 2014I<except> C<[[:ascii:]]>.
a6a7eedc 2015
2e2b2571 2016Starting in Perl 5.14.0, regular expressions compiled within
a6a7eedc 2017the scope of C<unicode_strings> use Unicode rules
2e2b2571
KW
2018even when executed or compiled into larger
2019regular expressions outside the scope.
e1b711da
KW
2020
2021=item *
2022
a6a7eedc
KW
2023In C<quotemeta> or its inline equivalent C<\Q>.
2024
2e2b2571
KW
2025Starting in Perl 5.16.0, consistent quoting rules are used within the
2026scope of C<unicode_strings>, as described in L<perlfunc/quotemeta>.
a6a7eedc
KW
2027Prior to that, or outside its scope, no code points above 127 are quoted
2028in UTF-8 encoded strings, but in byte encoded strings, code points
2029between 128-255 are always quoted.
eb88ed9e 2030
d6c970c7
AC
2031=item *
2032
2033In the C<..> or L<range|perlop/Range Operators> operator.
2034
2035Starting in Perl 5.26.0, the range operator on strings treats their lengths
2036consistently within the scope of C<unicode_strings>. Prior to that, or
2037outside its scope, it could produce strings whose length in characters
2038exceeded that of the right-hand side, where the right-hand side took up more
2039bytes than the correct range endpoint.
2040
20ae58f7
AC
2041=item *
2042
2043In L<< C<split>'s special-case whitespace splitting|perlfunc/split >>.
2044
2045Starting in Perl 5.28.0, the C<split> function with a pattern specified as
2046a string containing a single space handles whitespace characters consistently
2047within the scope of of C<unicode_strings>. Prior to that, or outside its scope,
2048characters that are whitespace according to Unicode rules but not according to
2049ASCII rules were treated as field contents rather than field separators when
2050they appear in byte-encoded strings.
2051
e1b711da
KW
2052=back
2053
a6a7eedc
KW
2054You can see from the above that the effect of C<unicode_strings>
2055increased over several Perl releases. (And Perl's support for Unicode
2056continues to improve; it's best to use the latest available release in
2057order to get the most complete and accurate results possible.) Note that
2058C<unicode_strings> is automatically chosen if you S<C<use 5.012>> or
2059higher.
e1b711da 2060
2e2b2571 2061For Perls earlier than those described above, or when a string is passed
a6a7eedc 2062to a function outside the scope of C<unicode_strings>, see the next section.
e1b711da 2063
1aad1664
JH
2064=head2 Forcing Unicode in Perl (Or Unforcing Unicode in Perl)
2065
e1b711da
KW
2066Sometimes (see L</"When Unicode Does Not Happen"> or L</The "Unicode Bug">)
2067there are situations where you simply need to force a byte
a6a7eedc
KW
2068string into UTF-8, or vice versa. The standard module L<Encode> can be
2069used for this, or the low-level calls
a9130ea9 2070L<C<utf8::upgrade($bytestring)>|utf8/Utility functions> and
a6a7eedc 2071L<C<utf8::downgrade($utf8string[, FAIL_OK])>|utf8/Utility functions>.
1aad1664 2072
a9130ea9 2073Note that C<utf8::downgrade()> can fail if the string contains characters
2bbc8d55 2074that don't fit into a byte.
1aad1664 2075
e1b711da
KW
2076Calling either function on a string that already is in the desired state is a
2077no-op.
2078
a6a7eedc
KW
2079L</ASCII Rules versus Unicode Rules> gives all the ways that a string is
2080made to use Unicode rules.
95a1a48b 2081
37b3b608 2082=head2 Using Unicode in XS
c349b1b9 2083
37b3b608
KW
2084See L<perlguts/"Unicode Support"> for an introduction to Unicode at
2085the XS level, and L<perlapi/Unicode Support> for the API details.
95a1a48b 2086
e1b711da
KW
2087=head2 Hacking Perl to work on earlier Unicode versions (for very serious hackers only)
2088
a6a7eedc
KW
2089Perl by default comes with the latest supported Unicode version built-in, but
2090the goal is to allow you to change to use any earlier one. In Perls
2091v5.20 and v5.22, however, the earliest usable version is Unicode 5.1.
c55dd03d 2092Perl v5.18 and v5.24 are able to handle all earlier versions.
e1b711da 2093
42581d5d 2094Download the files in the desired version of Unicode from the Unicode web
71c89d21 2095site L<https://www.unicode.org>). These should replace the existing files in
b19eb496 2096F<lib/unicore> in the Perl source tree. Follow the instructions in
116693e8 2097F<README.perl> in that directory to change some of their names, and then build
26e391dd 2098perl (see L<INSTALL>).
116693e8 2099
c8d992ba
A
2100=head2 Porting code from perl-5.6.X
2101
a6a7eedc
KW
2102Perls starting in 5.8 have a different Unicode model from 5.6. In 5.6 the
2103programmer was required to use the C<utf8> pragma to declare that a
2104given scope expected to deal with Unicode data and had to make sure that
2105only Unicode data were reaching that scope. If you have code that is
c8d992ba 2106working with 5.6, you will need some of the following adjustments to
a6a7eedc
KW
2107your code. The examples are written such that the code will continue to
2108work under 5.6, so you should be safe to try them out.
c8d992ba 2109
755789c0 2110=over 3
c8d992ba
A
2111
2112=item *
2113
2114A filehandle that should read or write UTF-8
2115
b9cedb1b 2116 if ($] > 5.008) {
6d8e7450 2117 binmode $fh, ":encoding(UTF-8)";
c8d992ba
A
2118 }
2119
2120=item *
2121
2122A scalar that is going to be passed to some extension
2123
a9130ea9 2124Be it C<Compress::Zlib>, C<Apache::Request> or any extension that has no
c8d992ba 2125mention of Unicode in the manpage, you need to make sure that the
2575c402 2126UTF8 flag is stripped off. Note that at the time of this writing
b9cedb1b 2127(January 2012) the mentioned modules are not UTF-8-aware. Please
c8d992ba
A
2128check the documentation to verify if this is still true.
2129
b9cedb1b 2130 if ($] > 5.008) {
c8d992ba 2131 require Encode;
8e179dd8 2132 $val = Encode::encode("UTF-8", $val); # make octets
c8d992ba
A
2133 }
2134
2135=item *
2136
2137A scalar we got back from an extension
2138
2139If you believe the scalar comes back as UTF-8, you will most likely
2575c402 2140want the UTF8 flag restored:
c8d992ba 2141
b9cedb1b 2142 if ($] > 5.008) {
c8d992ba 2143 require Encode;
8e179dd8 2144 $val = Encode::decode("UTF-8", $val);
c8d992ba
A
2145 }
2146
2147=item *
2148
2149Same thing, if you are really sure it is UTF-8
2150
b9cedb1b 2151 if ($] > 5.008) {
c8d992ba
A
2152 require Encode;
2153 Encode::_utf8_on($val);
2154 }
2155
2156=item *
2157
a9130ea9 2158A wrapper for L<DBI> C<fetchrow_array> and C<fetchrow_hashref>
c8d992ba
A
2159
2160When the database contains only UTF-8, a wrapper function or method is
a9130ea9
KW
2161a convenient way to replace all your C<fetchrow_array> and
2162C<fetchrow_hashref> calls. A wrapper function will also make it easier to
c8d992ba 2163adapt to future enhancements in your database driver. Note that at the
b9cedb1b 2164time of this writing (January 2012), the DBI has no standardized way
a9130ea9 2165to deal with UTF-8 data. Please check the L<DBI documentation|DBI> to verify if
c8d992ba
A
2166that is still true.
2167
2168 sub fetchrow {
d88362ca
KW
2169 # $what is one of fetchrow_{array,hashref}
2170 my($self, $sth, $what) = @_;
b9cedb1b 2171 if ($] < 5.008) {
c8d992ba
A
2172 return $sth->$what;
2173 } else {
2174 require Encode;
2175 if (wantarray) {
2176 my @arr = $sth->$what;
2177 for (@arr) {
2178 defined && /[^\000-\177]/ && Encode::_utf8_on($_);
2179 }
2180 return @arr;
2181 } else {
2182 my $ret = $sth->$what;
2183 if (ref $ret) {
2184 for my $k (keys %$ret) {
d88362ca
KW
2185 defined
2186 && /[^\000-\177]/
2187 && Encode::_utf8_on($_) for $ret->{$k};
c8d992ba
A
2188 }
2189 return $ret;
2190 } else {
2191 defined && /[^\000-\177]/ && Encode::_utf8_on($_) for $ret;
2192 return $ret;
2193 }
2194 }
2195 }
2196 }
2197
2198
2199=item *
2200
2201A large scalar that you know can only contain ASCII
2202
2203Scalars that contain only ASCII and are marked as UTF-8 are sometimes
2204a drag to your program. If you recognize such a situation, just remove
2575c402 2205the UTF8 flag:
c8d992ba 2206
b9cedb1b 2207 utf8::downgrade($val) if $] > 5.008;
c8d992ba
A
2208
2209=back
2210
a6a7eedc
KW
2211=head1 BUGS
2212
2213See also L</The "Unicode Bug"> above.
2214
2215=head2 Interaction with Extensions
2216
2217When Perl exchanges data with an extension, the extension should be
2218able to understand the UTF8 flag and act accordingly. If the
2219extension doesn't recognize that flag, it's likely that the extension
2220will return incorrectly-flagged data.
2221
2222So if you're working with Unicode data, consult the documentation of
2223every module you're using if there are any issues with Unicode data
2224exchange. If the documentation does not talk about Unicode at all,
2225suspect the worst and probably look at the source to learn how the
2226module is implemented. Modules written completely in Perl shouldn't
2227cause problems. Modules that directly or indirectly access code written
2228in other programming languages are at risk.
2229
2230For affected functions, the simple strategy to avoid data corruption is
2231to always make the encoding of the exchanged data explicit. Choose an
2232encoding that you know the extension can handle. Convert arguments passed
2233to the extensions to that encoding and convert results back from that
2234encoding. Write wrapper functions that do the conversions for you, so
2235you can later change the functions when the extension catches up.
2236
2237To provide an example, let's say the popular C<Foo::Bar::escape_html>
2238function doesn't deal with Unicode data yet. The wrapper function
2239would convert the argument to raw UTF-8 and convert the result back to
2240Perl's internal representation like so:
2241
2242 sub my_escape_html ($) {
2243 my($what) = shift;
2244 return unless defined $what;
8e179dd8
P
2245 Encode::decode("UTF-8", Foo::Bar::escape_html(
2246 Encode::encode("UTF-8", $what)));
a6a7eedc
KW
2247 }
2248
2249Sometimes, when the extension does not convert data but just stores
2250and retrieves it, you will be able to use the otherwise
2251dangerous L<C<Encode::_utf8_on()>|Encode/_utf8_on> function. Let's say
2252the popular C<Foo::Bar> extension, written in C, provides a C<param>
2253method that lets you store and retrieve data according to these prototypes:
2254
2255 $self->param($name, $value); # set a scalar
2256 $value = $self->param($name); # retrieve a scalar
2257
2258If it does not yet provide support for any encoding, one could write a
2259derived class with such a C<param> method:
2260
2261 sub param {
2262 my($self,$name,$value) = @_;
2263 utf8::upgrade($name); # make sure it is UTF-8 encoded
2264 if (defined $value) {
2265 utf8::upgrade($value); # make sure it is UTF-8 encoded
2266 return $self->SUPER::param($name,$value);
2267 } else {
2268 my $ret = $self->SUPER::param($name);
2269 Encode::_utf8_on($ret); # we know, it is UTF-8 encoded
2270 return $ret;
2271 }
2272 }
2273
2274Some extensions provide filters on data entry/exit points, such as
2275C<DB_File::filter_store_key> and family. Look out for such filters in
2276the documentation of your extensions; they can make the transition to
2277Unicode data much easier.
2278
2279=head2 Speed
2280
2281Some functions are slower when working on UTF-8 encoded strings than
2282on byte encoded strings. All functions that need to hop over
2283characters such as C<length()>, C<substr()> or C<index()>, or matching
2284regular expressions can work B<much> faster when the underlying data are
2285byte-encoded.
2286
2287In Perl 5.8.0 the slowness was often quite spectacular; in Perl 5.8.1
2288a caching scheme was introduced which improved the situation. In general,
2289operations with UTF-8 encoded strings are still slower. As an example,
2290the Unicode properties (character classes) like C<\p{Nd}> are known to
2291be quite a bit slower (5-20 times) than their simpler counterparts
2292like C<[0-9]> (then again, there are hundreds of Unicode characters matching
2293C<Nd> compared with the 10 ASCII characters matching C<[0-9]>).
2294
393fec97
GS
2295=head1 SEE ALSO
2296
51f494cc 2297L<perlunitut>, L<perluniintro>, L<perluniprops>, L<Encode>, L<open>, L<utf8>, L<bytes>,
b65e6125 2298L<perlretut>, L<perlvar/"${^UNICODE}">,
71c89d21 2299L<https://www.unicode.org/reports/tr44>).
393fec97
GS
2300
2301=cut