| 1 | =head1 NAME |
| 2 | |
| 3 | perlunicode - Unicode support in Perl |
| 4 | |
| 5 | =head1 DESCRIPTION |
| 6 | |
| 7 | If you haven't already, before reading this document, you should become |
| 8 | familiar with both L<perlunitut> and L<perluniintro>. |
| 9 | |
| 10 | Unicode aims to B<UNI>-fy the en-B<CODE>-ings of all the world's |
| 11 | character sets into a single Standard. For quite a few of the various |
| 12 | coding standards that existed when Unicode was first created, converting |
| 13 | from each to Unicode essentially meant adding a constant to each code |
| 14 | point in the original standard, and converting back meant just |
| 15 | subtracting that same constant. For ASCII and ISO-8859-1, the constant |
| 16 | is 0. For ISO-8859-5, (Cyrillic) the constant is 864; for Hebrew |
| 17 | (ISO-8859-8), it's 1488; Thai (ISO-8859-11), 3424; and so forth. This |
| 18 | made it easy to do the conversions, and facilitated the adoption of |
| 19 | Unicode. |
| 20 | |
| 21 | And it worked; nowadays, those legacy standards are rarely used. Most |
| 22 | everyone uses Unicode. |
| 23 | |
| 24 | Unicode is a comprehensive standard. It specifies many things outside |
| 25 | the scope of Perl, such as how to display sequences of characters. For |
| 26 | a full discussion of all aspects of Unicode, see |
| 27 | L<http://www.unicode.org>. |
| 28 | |
| 29 | =head2 Important Caveats |
| 30 | |
| 31 | Even though some of this section may not be understandable to you on |
| 32 | first reading, we think it's important enough to highlight some of the |
| 33 | gotchas before delving further, so here goes: |
| 34 | |
| 35 | Unicode support is an extensive requirement. While Perl does not |
| 36 | implement the Unicode standard or the accompanying technical reports |
| 37 | from cover to cover, Perl does support many Unicode features. |
| 38 | |
| 39 | Also, the use of Unicode may present security issues that aren't obvious. |
| 40 | Read L<Unicode Security Considerations|http://www.unicode.org/reports/tr36>. |
| 41 | |
| 42 | =over 4 |
| 43 | |
| 44 | =item Safest if you C<use feature 'unicode_strings'> |
| 45 | |
| 46 | In order to preserve backward compatibility, Perl does not turn |
| 47 | on full internal Unicode support unless the pragma |
| 48 | L<S<C<use feature 'unicode_strings'>>|feature/The 'unicode_strings' feature> |
| 49 | is specified. (This is automatically |
| 50 | selected if you S<C<use 5.012>> or higher.) Failure to do this can |
| 51 | trigger unexpected surprises. See L</The "Unicode Bug"> below. |
| 52 | |
| 53 | This pragma doesn't affect I/O. Nor does it change the internal |
| 54 | representation of strings, only their interpretation. There are still |
| 55 | several places where Unicode isn't fully supported, such as in |
| 56 | filenames. |
| 57 | |
| 58 | =item Input and Output Layers |
| 59 | |
| 60 | Use the C<:encoding(...)> layer to read from and write to |
| 61 | filehandles using the specified encoding. (See L<open>.) |
| 62 | |
| 63 | =item You should convert your non-ASCII, non-UTF-8 Perl scripts to be |
| 64 | UTF-8. |
| 65 | |
| 66 | See L<encoding>. |
| 67 | |
| 68 | =item C<use utf8> still needed to enable L<UTF-8|/Unicode Encodings> in scripts |
| 69 | |
| 70 | If your Perl script is itself encoded in L<UTF-8|/Unicode Encodings>, |
| 71 | the S<C<use utf8>> pragma must be explicitly included to enable |
| 72 | recognition of that (in string or regular expression literals, or in |
| 73 | identifier names). B<This is the only time when an explicit S<C<use |
| 74 | utf8>> is needed.> (See L<utf8>). |
| 75 | |
| 76 | =item C<BOM>-marked scripts and L<UTF-16|/Unicode Encodings> scripts autodetected |
| 77 | |
| 78 | However, if a Perl script begins with the Unicode C<BOM> (UTF-16LE, |
| 79 | UTF16-BE, or UTF-8), or if the script looks like non-C<BOM>-marked |
| 80 | UTF-16 of either endianness, Perl will correctly read in the script as |
| 81 | the appropriate Unicode encoding. (C<BOM>-less UTF-8 cannot be |
| 82 | effectively recognized or differentiated from ISO 8859-1 or other |
| 83 | eight-bit encodings.) |
| 84 | |
| 85 | =back |
| 86 | |
| 87 | =head2 Byte and Character Semantics |
| 88 | |
| 89 | Before Unicode, most encodings used 8 bits (a single byte) to encode |
| 90 | each character. Thus a character was a byte, and a byte was a |
| 91 | character, and there could be only 256 or fewer possible characters. |
| 92 | "Byte Semantics" in the title of this section refers to |
| 93 | this behavior. There was no need to distinguish between "Byte" and |
| 94 | "Character". |
| 95 | |
| 96 | Then along comes Unicode which has room for over a million characters |
| 97 | (and Perl allows for even more). This means that a character may |
| 98 | require more than a single byte to represent it, and so the two terms |
| 99 | are no longer equivalent. What matter are the characters as whole |
| 100 | entities, and not usually the bytes that comprise them. That's what the |
| 101 | term "Character Semantics" in the title of this section refers to. |
| 102 | |
| 103 | Perl had to change internally to decouple "bytes" from "characters". |
| 104 | It is important that you too change your ideas, if you haven't already, |
| 105 | so that "byte" and "character" no longer mean the same thing in your |
| 106 | mind. |
| 107 | |
| 108 | The basic building block of Perl strings has always been a "character". |
| 109 | The changes basically come down to that the implementation no longer |
| 110 | thinks that a character is always just a single byte. |
| 111 | |
| 112 | There are various things to note: |
| 113 | |
| 114 | =over 4 |
| 115 | |
| 116 | =item * |
| 117 | |
| 118 | String handling functions, for the most part, continue to operate in |
| 119 | terms of characters. C<length()>, for example, returns the number of |
| 120 | characters in a string, just as before. But that number no longer is |
| 121 | necessarily the same as the number of bytes in the string (there may be |
| 122 | more bytes than characters). The other such functions include |
| 123 | C<chop()>, C<chomp()>, C<substr()>, C<pos()>, C<index()>, C<rindex()>, |
| 124 | C<sort()>, C<sprintf()>, and C<write()>. |
| 125 | |
| 126 | The exceptions are: |
| 127 | |
| 128 | =over 4 |
| 129 | |
| 130 | =item * |
| 131 | |
| 132 | the bit-oriented C<vec> |
| 133 | |
| 134 | E<nbsp> |
| 135 | |
| 136 | =item * |
| 137 | |
| 138 | the byte-oriented C<pack>/C<unpack> C<"C"> format |
| 139 | |
| 140 | However, the C<W> specifier does operate on whole characters, as does the |
| 141 | C<U> specifier. |
| 142 | |
| 143 | =item * |
| 144 | |
| 145 | some operators that interact with the platform's operating system |
| 146 | |
| 147 | Operators dealing with filenames are examples. |
| 148 | |
| 149 | =item * |
| 150 | |
| 151 | when the functions are called from within the scope of the |
| 152 | S<C<L<use bytes|bytes>>> pragma |
| 153 | |
| 154 | Likely, you should use this only for debugging anyway. |
| 155 | |
| 156 | =back |
| 157 | |
| 158 | =item * |
| 159 | |
| 160 | Strings--including hash keys--and regular expression patterns may |
| 161 | contain characters that have ordinal values larger than 255. |
| 162 | |
| 163 | If you use a Unicode editor to edit your program, Unicode characters may |
| 164 | occur directly within the literal strings in UTF-8 encoding, or UTF-16. |
| 165 | (The former requires a C<BOM> or C<use utf8>, the latter requires a C<BOM>.) |
| 166 | |
| 167 | L<perluniintro/Creating Unicode> gives other ways to place non-ASCII |
| 168 | characters in your strings. |
| 169 | |
| 170 | =item * |
| 171 | |
| 172 | The C<chr()> and C<ord()> functions work on whole characters. |
| 173 | |
| 174 | =item * |
| 175 | |
| 176 | Regular expressions match whole characters. For example, C<"."> matches |
| 177 | a whole character instead of only a single byte. |
| 178 | |
| 179 | =item * |
| 180 | |
| 181 | The C<tr///> operator translates whole characters. (Note that the |
| 182 | C<tr///CU> functionality has been removed. For similar functionality to |
| 183 | that, see C<pack('U0', ...)> and C<pack('C0', ...)>). |
| 184 | |
| 185 | =item * |
| 186 | |
| 187 | C<scalar reverse()> reverses by character rather than by byte. |
| 188 | |
| 189 | =item * |
| 190 | |
| 191 | The bit string operators, C<& | ^ ~> and (starting in v5.22) |
| 192 | C<&. |. ^. ~.> can operate on characters that don't fit into a byte. |
| 193 | However, the current behavior is likely to change. You should not use |
| 194 | these operators on strings that are encoded in UTF-8. If you're not |
| 195 | sure about the encoding of a string, downgrade it before using any of |
| 196 | these operators; you can use |
| 197 | L<C<utf8::utf8_downgrade()>|utf8/Utility functions>. |
| 198 | |
| 199 | =back |
| 200 | |
| 201 | The bottom line is that Perl has always practiced "Character Semantics", |
| 202 | but with the advent of Unicode, that is now different than "Byte |
| 203 | Semantics". |
| 204 | |
| 205 | =head2 ASCII Rules versus Unicode Rules |
| 206 | |
| 207 | Before Unicode, when a character was a byte was a character, |
| 208 | Perl knew only about the 128 characters defined by ASCII, code points 0 |
| 209 | through 127 (except for under S<C<use locale>>). That left the code |
| 210 | points 128 to 255 as unassigned, and available for whatever use a |
| 211 | program might want. The only semantics they have is their ordinal |
| 212 | numbers, and that they are members of none of the non-negative character |
| 213 | classes. None are considered to match C<\w> for example, but all match |
| 214 | C<\W>. |
| 215 | |
| 216 | Unicode, of course, assigns each of those code points a particular |
| 217 | meaning (along with ones above 255). To preserve backward |
| 218 | compatibility, Perl only uses the Unicode meanings when there is some |
| 219 | indication that Unicode is what is intended; otherwise the non-ASCII |
| 220 | code points remain treated as if they are unassigned. |
| 221 | |
| 222 | Here are the ways that Perl knows that a string should be treated as |
| 223 | Unicode: |
| 224 | |
| 225 | =over |
| 226 | |
| 227 | =item * |
| 228 | |
| 229 | Within the scope of S<C<use utf8>> |
| 230 | |
| 231 | If the whole program is Unicode (signified by using 8-bit B<U>nicode |
| 232 | B<T>ransformation B<F>ormat), then all strings within it must be |
| 233 | Unicode. |
| 234 | |
| 235 | =item * |
| 236 | |
| 237 | Within the scope of |
| 238 | L<S<C<use feature 'unicode_strings'>>|feature/The 'unicode_strings' feature> |
| 239 | |
| 240 | This pragma was created so you can explicitly tell Perl that operations |
| 241 | executed within its scope are to use Unicode rules. More operations are |
| 242 | affected with newer perls. See L</The "Unicode Bug">. |
| 243 | |
| 244 | =item * |
| 245 | |
| 246 | Within the scope of S<C<use 5.012>> or higher |
| 247 | |
| 248 | This implicitly turns on S<C<use feature 'unicode_strings'>>. |
| 249 | |
| 250 | =item * |
| 251 | |
| 252 | Within the scope of |
| 253 | L<S<C<use locale 'not_characters'>>|perllocale/Unicode and UTF-8>, |
| 254 | or L<S<C<use locale>>|perllocale> and the current |
| 255 | locale is a UTF-8 locale. |
| 256 | |
| 257 | The former is defined to imply Unicode handling; and the latter |
| 258 | indicates a Unicode locale, hence a Unicode interpretation of all |
| 259 | strings within it. |
| 260 | |
| 261 | =item * |
| 262 | |
| 263 | When the string contains a Unicode-only code point |
| 264 | |
| 265 | Perl has never accepted code points above 255 without them being |
| 266 | Unicode, so their use implies Unicode for the whole string. |
| 267 | |
| 268 | =item * |
| 269 | |
| 270 | When the string contains a Unicode named code point C<\N{...}> |
| 271 | |
| 272 | The C<\N{...}> construct explicitly refers to a Unicode code point, |
| 273 | even if it is one that is also in ASCII. Therefore the string |
| 274 | containing it must be Unicode. |
| 275 | |
| 276 | =item * |
| 277 | |
| 278 | When the string has come from an external source marked as |
| 279 | Unicode |
| 280 | |
| 281 | The L<C<-C>|perlrun/-C [numberE<sol>list]> command line option can |
| 282 | specify that certain inputs to the program are Unicode, and the values |
| 283 | of this can be read by your Perl code, see L<perlvar/"${^UNICODE}">. |
| 284 | |
| 285 | =item * When the string has been upgraded to UTF-8 |
| 286 | |
| 287 | The function L<C<utf8::utf8_upgrade()>|utf8/Utility functions> |
| 288 | can be explicitly used to permanently (unless a subsequent |
| 289 | C<utf8::utf8_downgrade()> is called) cause a string to be treated as |
| 290 | Unicode. |
| 291 | |
| 292 | =item * There are additional methods for regular expression patterns |
| 293 | |
| 294 | A pattern that is compiled with the C<< /u >> or C<< /a >> modifiers is |
| 295 | treated as Unicode (though there are some restrictions with C<< /a >>). |
| 296 | Under the C<< /d >> and C<< /l >> modifiers, there are several other |
| 297 | indications for Unicode; see L<perlre/Character set modifiers>. |
| 298 | |
| 299 | =back |
| 300 | |
| 301 | Note that all of the above are overridden within the scope of |
| 302 | C<L<use bytes|bytes>>; but you should be using this pragma only for |
| 303 | debugging. |
| 304 | |
| 305 | Note also that some interactions with the platform's operating system |
| 306 | never use Unicode rules. |
| 307 | |
| 308 | When Unicode rules are in effect: |
| 309 | |
| 310 | =over 4 |
| 311 | |
| 312 | =item * |
| 313 | |
| 314 | Case translation operators use the Unicode case translation tables. |
| 315 | |
| 316 | Note that C<uc()>, or C<\U> in interpolated strings, translates to |
| 317 | uppercase, while C<ucfirst>, or C<\u> in interpolated strings, |
| 318 | translates to titlecase in languages that make the distinction (which is |
| 319 | equivalent to uppercase in languages without the distinction). |
| 320 | |
| 321 | There is a CPAN module, C<L<Unicode::Casing>>, which allows you to |
| 322 | define your own mappings to be used in C<lc()>, C<lcfirst()>, C<uc()>, |
| 323 | C<ucfirst()>, and C<fc> (or their double-quoted string inlined versions |
| 324 | such as C<\U>). (Prior to Perl 5.16, this functionality was partially |
| 325 | provided in the Perl core, but suffered from a number of insurmountable |
| 326 | drawbacks, so the CPAN module was written instead.) |
| 327 | |
| 328 | =item * |
| 329 | |
| 330 | Character classes in regular expressions match based on the character |
| 331 | properties specified in the Unicode properties database. |
| 332 | |
| 333 | C<\w> can be used to match a Japanese ideograph, for instance; and |
| 334 | C<[[:digit:]]> a Bengali number. |
| 335 | |
| 336 | =item * |
| 337 | |
| 338 | Named Unicode properties, scripts, and block ranges may be used (like |
| 339 | bracketed character classes) by using the C<\p{}> "matches property" |
| 340 | construct and the C<\P{}> negation, "doesn't match property". |
| 341 | |
| 342 | See L</"Unicode Character Properties"> for more details. |
| 343 | |
| 344 | You can define your own character properties and use them |
| 345 | in the regular expression with the C<\p{}> or C<\P{}> construct. |
| 346 | See L</"User-Defined Character Properties"> for more details. |
| 347 | |
| 348 | =back |
| 349 | |
| 350 | =head2 Extended Grapheme Clusters (Logical characters) |
| 351 | |
| 352 | Consider a character, say C<H>. It could appear with various marks around it, |
| 353 | such as an acute accent, or a circumflex, or various hooks, circles, arrows, |
| 354 | I<etc.>, above, below, to one side or the other, I<etc>. There are many |
| 355 | possibilities among the world's languages. The number of combinations is |
| 356 | astronomical, and if there were a character for each combination, it would |
| 357 | soon exhaust Unicode's more than a million possible characters. So Unicode |
| 358 | took a different approach: there is a character for the base C<H>, and a |
| 359 | character for each of the possible marks, and these can be variously combined |
| 360 | to get a final logical character. So a logical character--what appears to be a |
| 361 | single character--can be a sequence of more than one individual characters. |
| 362 | The Unicode standard calls these "extended grapheme clusters" (which |
| 363 | is an improved version of the no-longer much used "grapheme cluster"); |
| 364 | Perl furnishes the C<\X> regular expression construct to match such |
| 365 | sequences in their entirety. |
| 366 | |
| 367 | But Unicode's intent is to unify the existing character set standards and |
| 368 | practices, and several pre-existing standards have single characters that |
| 369 | mean the same thing as some of these combinations, like ISO-8859-1, |
| 370 | which has quite a few of them. For example, C<"LATIN CAPITAL LETTER E |
| 371 | WITH ACUTE"> was already in this standard when Unicode came along. |
| 372 | Unicode therefore added it to its repertoire as that single character. |
| 373 | But this character is considered by Unicode to be equivalent to the |
| 374 | sequence consisting of the character C<"LATIN CAPITAL LETTER E"> |
| 375 | followed by the character C<"COMBINING ACUTE ACCENT">. |
| 376 | |
| 377 | C<"LATIN CAPITAL LETTER E WITH ACUTE"> is called a "pre-composed" |
| 378 | character, and its equivalence with the "E" and the "COMBINING ACCENT" |
| 379 | sequence is called canonical equivalence. All pre-composed characters |
| 380 | are said to have a decomposition (into the equivalent sequence), and the |
| 381 | decomposition type is also called canonical. A string may be comprised |
| 382 | as much as possible of precomposed characters, or it may be comprised of |
| 383 | entirely decomposed characters. Unicode calls these respectively, |
| 384 | "Normalization Form Composed" (NFC) and "Normalization Form Decomposed". |
| 385 | The C<L<Unicode::Normalize>> module contains functions that convert |
| 386 | between the two. A string may also have both composed characters and |
| 387 | decomposed characters; this module can be used to make it all one or the |
| 388 | other. |
| 389 | |
| 390 | You may be presented with strings in any of these equivalent forms. |
| 391 | There is currently nothing in Perl 5 that ignores the differences. So |
| 392 | you'll have to specially hanlde it. The usual advice is to convert your |
| 393 | inputs to C<NFD> before processing further. |
| 394 | |
| 395 | For more detailed information, see L<http://unicode.org/reports/tr15/>. |
| 396 | |
| 397 | =head2 Unicode Character Properties |
| 398 | |
| 399 | (The only time that Perl considers a sequence of individual code |
| 400 | points as a single logical character is in the C<\X> construct, already |
| 401 | mentioned above. Therefore "character" in this discussion means a single |
| 402 | Unicode code point.) |
| 403 | |
| 404 | Very nearly all Unicode character properties are accessible through |
| 405 | regular expressions by using the C<\p{}> "matches property" construct |
| 406 | and the C<\P{}> "doesn't match property" for its negation. |
| 407 | |
| 408 | For instance, C<\p{Uppercase}> matches any single character with the Unicode |
| 409 | C<"Uppercase"> property, while C<\p{L}> matches any character with a |
| 410 | C<General_Category> of C<"L"> (letter) property (see |
| 411 | L</General_Category> below). Brackets are not |
| 412 | required for single letter property names, so C<\p{L}> is equivalent to C<\pL>. |
| 413 | |
| 414 | More formally, C<\p{Uppercase}> matches any single character whose Unicode |
| 415 | C<Uppercase> property value is C<True>, and C<\P{Uppercase}> matches any character |
| 416 | whose C<Uppercase> property value is C<False>, and they could have been written as |
| 417 | C<\p{Uppercase=True}> and C<\p{Uppercase=False}>, respectively. |
| 418 | |
| 419 | This formality is needed when properties are not binary; that is, if they can |
| 420 | take on more values than just C<True> and C<False>. For example, the |
| 421 | C<Bidi_Class> property (see L</"Bidirectional Character Types"> below), |
| 422 | can take on several different |
| 423 | values, such as C<Left>, C<Right>, C<Whitespace>, and others. To match these, one needs |
| 424 | to specify both the property name (C<Bidi_Class>), AND the value being |
| 425 | matched against |
| 426 | (C<Left>, C<Right>, I<etc.>). This is done, as in the examples above, by having the |
| 427 | two components separated by an equal sign (or interchangeably, a colon), like |
| 428 | C<\p{Bidi_Class: Left}>. |
| 429 | |
| 430 | All Unicode-defined character properties may be written in these compound forms |
| 431 | of C<\p{I<property>=I<value>}> or C<\p{I<property>:I<value>}>, but Perl provides some |
| 432 | additional properties that are written only in the single form, as well as |
| 433 | single-form short-cuts for all binary properties and certain others described |
| 434 | below, in which you may omit the property name and the equals or colon |
| 435 | separator. |
| 436 | |
| 437 | Most Unicode character properties have at least two synonyms (or aliases if you |
| 438 | prefer): a short one that is easier to type and a longer one that is more |
| 439 | descriptive and hence easier to understand. Thus the C<"L"> and |
| 440 | C<"Letter"> properties above are equivalent and can be used |
| 441 | interchangeably. Likewise, C<"Upper"> is a synonym for C<"Uppercase">, |
| 442 | and we could have written C<\p{Uppercase}> equivalently as C<\p{Upper}>. |
| 443 | Also, there are typically various synonyms for the values the property |
| 444 | can be. For binary properties, C<"True"> has 3 synonyms: C<"T">, |
| 445 | C<"Yes">, and C<"Y">; and C<"False"> has correspondingly C<"F">, |
| 446 | C<"No">, and C<"N">. But be careful. A short form of a value for one |
| 447 | property may not mean the same thing as the same short form for another. |
| 448 | Thus, for the C<L</General_Category>> property, C<"L"> means |
| 449 | C<"Letter">, but for the L<C<Bidi_Class>|/Bidirectional Character Types> |
| 450 | property, C<"L"> means C<"Left">. A complete list of properties and |
| 451 | synonyms is in L<perluniprops>. |
| 452 | |
| 453 | Upper/lower case differences in property names and values are irrelevant; |
| 454 | thus C<\p{Upper}> means the same thing as C<\p{upper}> or even C<\p{UpPeR}>. |
| 455 | Similarly, you can add or subtract underscores anywhere in the middle of a |
| 456 | word, so that these are also equivalent to C<\p{U_p_p_e_r}>. And white space |
| 457 | is irrelevant adjacent to non-word characters, such as the braces and the equals |
| 458 | or colon separators, so C<\p{ Upper }> and C<\p{ Upper_case : Y }> are |
| 459 | equivalent to these as well. In fact, white space and even |
| 460 | hyphens can usually be added or deleted anywhere. So even C<\p{ Up-per case = Yes}> is |
| 461 | equivalent. All this is called "loose-matching" by Unicode. The few places |
| 462 | where stricter matching is used is in the middle of numbers, and in the Perl |
| 463 | extension properties that begin or end with an underscore. Stricter matching |
| 464 | cares about white space (except adjacent to non-word characters), |
| 465 | hyphens, and non-interior underscores. |
| 466 | |
| 467 | You can also use negation in both C<\p{}> and C<\P{}> by introducing a caret |
| 468 | (C<^>) between the first brace and the property name: C<\p{^Tamil}> is |
| 469 | equal to C<\P{Tamil}>. |
| 470 | |
| 471 | Almost all properties are immune to case-insensitive matching. That is, |
| 472 | adding a C</i> regular expression modifier does not change what they |
| 473 | match. There are two sets that are affected. |
| 474 | The first set is |
| 475 | C<Uppercase_Letter>, |
| 476 | C<Lowercase_Letter>, |
| 477 | and C<Titlecase_Letter>, |
| 478 | all of which match C<Cased_Letter> under C</i> matching. |
| 479 | And the second set is |
| 480 | C<Uppercase>, |
| 481 | C<Lowercase>, |
| 482 | and C<Titlecase>, |
| 483 | all of which match C<Cased> under C</i> matching. |
| 484 | This set also includes its subsets C<PosixUpper> and C<PosixLower> both |
| 485 | of which under C</i> match C<PosixAlpha>. |
| 486 | (The difference between these sets is that some things, such as Roman |
| 487 | numerals, come in both upper and lower case so they are C<Cased>, but |
| 488 | aren't considered letters, so they aren't C<Cased_Letter>'s.) |
| 489 | |
| 490 | See L</Beyond Unicode code points> for special considerations when |
| 491 | matching Unicode properties against non-Unicode code points. |
| 492 | |
| 493 | =head3 B<General_Category> |
| 494 | |
| 495 | Every Unicode character is assigned a general category, which is the "most |
| 496 | usual categorization of a character" (from |
| 497 | L<http://www.unicode.org/reports/tr44>). |
| 498 | |
| 499 | The compound way of writing these is like C<\p{General_Category=Number}> |
| 500 | (short: C<\p{gc:n}>). But Perl furnishes shortcuts in which everything up |
| 501 | through the equal or colon separator is omitted. So you can instead just write |
| 502 | C<\pN>. |
| 503 | |
| 504 | Here are the short and long forms of the values the C<General Category> property |
| 505 | can have: |
| 506 | |
| 507 | Short Long |
| 508 | |
| 509 | L Letter |
| 510 | LC, L& Cased_Letter (that is: [\p{Ll}\p{Lu}\p{Lt}]) |
| 511 | Lu Uppercase_Letter |
| 512 | Ll Lowercase_Letter |
| 513 | Lt Titlecase_Letter |
| 514 | Lm Modifier_Letter |
| 515 | Lo Other_Letter |
| 516 | |
| 517 | M Mark |
| 518 | Mn Nonspacing_Mark |
| 519 | Mc Spacing_Mark |
| 520 | Me Enclosing_Mark |
| 521 | |
| 522 | N Number |
| 523 | Nd Decimal_Number (also Digit) |
| 524 | Nl Letter_Number |
| 525 | No Other_Number |
| 526 | |
| 527 | P Punctuation (also Punct) |
| 528 | Pc Connector_Punctuation |
| 529 | Pd Dash_Punctuation |
| 530 | Ps Open_Punctuation |
| 531 | Pe Close_Punctuation |
| 532 | Pi Initial_Punctuation |
| 533 | (may behave like Ps or Pe depending on usage) |
| 534 | Pf Final_Punctuation |
| 535 | (may behave like Ps or Pe depending on usage) |
| 536 | Po Other_Punctuation |
| 537 | |
| 538 | S Symbol |
| 539 | Sm Math_Symbol |
| 540 | Sc Currency_Symbol |
| 541 | Sk Modifier_Symbol |
| 542 | So Other_Symbol |
| 543 | |
| 544 | Z Separator |
| 545 | Zs Space_Separator |
| 546 | Zl Line_Separator |
| 547 | Zp Paragraph_Separator |
| 548 | |
| 549 | C Other |
| 550 | Cc Control (also Cntrl) |
| 551 | Cf Format |
| 552 | Cs Surrogate |
| 553 | Co Private_Use |
| 554 | Cn Unassigned |
| 555 | |
| 556 | Single-letter properties match all characters in any of the |
| 557 | two-letter sub-properties starting with the same letter. |
| 558 | C<LC> and C<L&> are special: both are aliases for the set consisting of everything matched by C<Ll>, C<Lu>, and C<Lt>. |
| 559 | |
| 560 | =head3 B<Bidirectional Character Types> |
| 561 | |
| 562 | Because scripts differ in their directionality (Hebrew and Arabic are |
| 563 | written right to left, for example) Unicode supplies a C<Bidi_Class> property. |
| 564 | Some of the values this property can have are: |
| 565 | |
| 566 | Value Meaning |
| 567 | |
| 568 | L Left-to-Right |
| 569 | LRE Left-to-Right Embedding |
| 570 | LRO Left-to-Right Override |
| 571 | R Right-to-Left |
| 572 | AL Arabic Letter |
| 573 | RLE Right-to-Left Embedding |
| 574 | RLO Right-to-Left Override |
| 575 | PDF Pop Directional Format |
| 576 | EN European Number |
| 577 | ES European Separator |
| 578 | ET European Terminator |
| 579 | AN Arabic Number |
| 580 | CS Common Separator |
| 581 | NSM Non-Spacing Mark |
| 582 | BN Boundary Neutral |
| 583 | B Paragraph Separator |
| 584 | S Segment Separator |
| 585 | WS Whitespace |
| 586 | ON Other Neutrals |
| 587 | |
| 588 | This property is always written in the compound form. |
| 589 | For example, C<\p{Bidi_Class:R}> matches characters that are normally |
| 590 | written right to left. Unlike the |
| 591 | C<L</General_Category>> property, this |
| 592 | property can have more values added in a future Unicode release. Those |
| 593 | listed above comprised the complete set for many Unicode releases, but |
| 594 | others were added in Unicode 6.3; you can always find what the |
| 595 | current ones are in L<perluniprops>. And |
| 596 | L<http://www.unicode.org/reports/tr9/> describes how to use them. |
| 597 | |
| 598 | =head3 B<Scripts> |
| 599 | |
| 600 | The world's languages are written in many different scripts. This sentence |
| 601 | (unless you're reading it in translation) is written in Latin, while Russian is |
| 602 | written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in |
| 603 | Hiragana or Katakana. There are many more. |
| 604 | |
| 605 | The Unicode C<Script> and C<Script_Extensions> properties give what script a |
| 606 | given character is in. Either property can be specified with the |
| 607 | compound form like |
| 608 | C<\p{Script=Hebrew}> (short: C<\p{sc=hebr}>), or |
| 609 | C<\p{Script_Extensions=Javanese}> (short: C<\p{scx=java}>). |
| 610 | In addition, Perl furnishes shortcuts for all |
| 611 | C<Script> property names. You can omit everything up through the equals |
| 612 | (or colon), and simply write C<\p{Latin}> or C<\P{Cyrillic}>. |
| 613 | (This is not true for C<Script_Extensions>, which is required to be |
| 614 | written in the compound form.) |
| 615 | |
| 616 | The difference between these two properties involves characters that are |
| 617 | used in multiple scripts. For example the digits '0' through '9' are |
| 618 | used in many parts of the world. These are placed in a script named |
| 619 | C<Common>. Other characters are used in just a few scripts. For |
| 620 | example, the C<"KATAKANA-HIRAGANA DOUBLE HYPHEN"> is used in both Japanese |
| 621 | scripts, Katakana and Hiragana, but nowhere else. The C<Script> |
| 622 | property places all characters that are used in multiple scripts in the |
| 623 | C<Common> script, while the C<Script_Extensions> property places those |
| 624 | that are used in only a few scripts into each of those scripts; while |
| 625 | still using C<Common> for those used in many scripts. Thus both these |
| 626 | match: |
| 627 | |
| 628 | "0" =~ /\p{sc=Common}/ # Matches |
| 629 | "0" =~ /\p{scx=Common}/ # Matches |
| 630 | |
| 631 | and only the first of these match: |
| 632 | |
| 633 | "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Common} # Matches |
| 634 | "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Common} # No match |
| 635 | |
| 636 | And only the last two of these match: |
| 637 | |
| 638 | "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Hiragana} # No match |
| 639 | "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Katakana} # No match |
| 640 | "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Hiragana} # Matches |
| 641 | "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Katakana} # Matches |
| 642 | |
| 643 | C<Script_Extensions> is thus an improved C<Script>, in which there are |
| 644 | fewer characters in the C<Common> script, and correspondingly more in |
| 645 | other scripts. It is new in Unicode version 6.0, and its data are likely |
| 646 | to change significantly in later releases, as things get sorted out. |
| 647 | New code should probably be using C<Script_Extensions> and not plain |
| 648 | C<Script>. |
| 649 | |
| 650 | (Actually, besides C<Common>, the C<Inherited> script, contains |
| 651 | characters that are used in multiple scripts. These are modifier |
| 652 | characters which inherit the script value |
| 653 | of the controlling character. Some of these are used in many scripts, |
| 654 | and so go into C<Inherited> in both C<Script> and C<Script_Extensions>. |
| 655 | Others are used in just a few scripts, so are in C<Inherited> in |
| 656 | C<Script>, but not in C<Script_Extensions>.) |
| 657 | |
| 658 | It is worth stressing that there are several different sets of digits in |
| 659 | Unicode that are equivalent to 0-9 and are matchable by C<\d> in a |
| 660 | regular expression. If they are used in a single language only, they |
| 661 | are in that language's C<Script> and C<Script_Extension>. If they are |
| 662 | used in more than one script, they will be in C<sc=Common>, but only |
| 663 | if they are used in many scripts should they be in C<scx=Common>. |
| 664 | |
| 665 | A complete list of scripts and their shortcuts is in L<perluniprops>. |
| 666 | |
| 667 | =head3 B<Use of the C<"Is"> Prefix> |
| 668 | |
| 669 | For backward compatibility (with Perl 5.6), all properties writable |
| 670 | without using the compound form mentioned |
| 671 | so far may have C<Is> or C<Is_> prepended to their name, so C<\P{Is_Lu}>, for |
| 672 | example, is equal to C<\P{Lu}>, and C<\p{IsScript:Arabic}> is equal to |
| 673 | C<\p{Arabic}>. |
| 674 | |
| 675 | =head3 B<Blocks> |
| 676 | |
| 677 | In addition to B<scripts>, Unicode also defines B<blocks> of |
| 678 | characters. The difference between scripts and blocks is that the |
| 679 | concept of scripts is closer to natural languages, while the concept |
| 680 | of blocks is more of an artificial grouping based on groups of Unicode |
| 681 | characters with consecutive ordinal values. For example, the C<"Basic Latin"> |
| 682 | block is all the characters whose ordinals are between 0 and 127, inclusive; in |
| 683 | other words, the ASCII characters. The C<"Latin"> script contains some letters |
| 684 | from this as well as several other blocks, like C<"Latin-1 Supplement">, |
| 685 | C<"Latin Extended-A">, I<etc.>, but it does not contain all the characters from |
| 686 | those blocks. It does not, for example, contain the digits 0-9, because |
| 687 | those digits are shared across many scripts, and hence are in the |
| 688 | C<Common> script. |
| 689 | |
| 690 | For more about scripts versus blocks, see UAX#24 "Unicode Script Property": |
| 691 | L<http://www.unicode.org/reports/tr24> |
| 692 | |
| 693 | The C<Script> or C<Script_Extensions> properties are likely to be the |
| 694 | ones you want to use when processing |
| 695 | natural language; the C<Block> property may occasionally be useful in working |
| 696 | with the nuts and bolts of Unicode. |
| 697 | |
| 698 | Block names are matched in the compound form, like C<\p{Block: Arrows}> or |
| 699 | C<\p{Blk=Hebrew}>. Unlike most other properties, only a few block names have a |
| 700 | Unicode-defined short name. |
| 701 | |
| 702 | Perl also defines single form synonyms for the block property in cases |
| 703 | where these do not conflict with something else. But don't use any of |
| 704 | these, because they are unstable. Since these are Perl extensions, they |
| 705 | are subordinate to official Unicode property names; Unicode doesn't know |
| 706 | nor care about Perl's extensions. It may happen that a name that |
| 707 | currently means the Perl extension will later be changed without warning |
| 708 | to mean a different Unicode property in a future version of the perl |
| 709 | interpreter that uses a later Unicode release, and your code would no |
| 710 | longer work. The extensions are mentioned here for completeness: Take |
| 711 | the block name and prefix it with one of: C<In> (for example |
| 712 | C<\p{Blk=Arrows}> can currently be written as C<\p{In_Arrows}>); or |
| 713 | sometimes C<Is> (like C<\p{Is_Arrows}>); or sometimes no prefix at all |
| 714 | (C<\p{Arrows}>). As of this writing (Unicode 8.0) there are no |
| 715 | conflicts with using the C<In_> prefix, but there are plenty with the |
| 716 | other two forms. For example, C<\p{Is_Hebrew}> and C<\p{Hebrew}> mean |
| 717 | C<\p{Script=Hebrew}> which is NOT the same thing as C<\p{Blk=Hebrew}>. Our |
| 718 | advice used to be to use the C<In_> prefix as a single form way of |
| 719 | specifying a block. But Unicode 8.0 added properties whose names begin |
| 720 | with C<In>, and it's now clear that it's only luck that's so far |
| 721 | prevented a conflict. Using C<In> is only marginally less typing than |
| 722 | C<Blk:>, and the latter's meaning is clearer anyway, and guaranteed to |
| 723 | never conflict. So don't take chances. Use C<\p{Blk=foo}> for new |
| 724 | code. And be sure that block is what you really really want to do. In |
| 725 | most cases scripts are what you want instead. |
| 726 | |
| 727 | A complete list of blocks is in L<perluniprops>. |
| 728 | |
| 729 | =head3 B<Other Properties> |
| 730 | |
| 731 | There are many more properties than the very basic ones described here. |
| 732 | A complete list is in L<perluniprops>. |
| 733 | |
| 734 | Unicode defines all its properties in the compound form, so all single-form |
| 735 | properties are Perl extensions. Most of these are just synonyms for the |
| 736 | Unicode ones, but some are genuine extensions, including several that are in |
| 737 | the compound form. And quite a few of these are actually recommended by Unicode |
| 738 | (in L<http://www.unicode.org/reports/tr18>). |
| 739 | |
| 740 | This section gives some details on all extensions that aren't just |
| 741 | synonyms for compound-form Unicode properties |
| 742 | (for those properties, you'll have to refer to the |
| 743 | L<Unicode Standard|http://www.unicode.org/reports/tr44>. |
| 744 | |
| 745 | =over |
| 746 | |
| 747 | =item B<C<\p{All}>> |
| 748 | |
| 749 | This matches every possible code point. It is equivalent to C<qr/./s>. |
| 750 | Unlike all the other non-user-defined C<\p{}> property matches, no |
| 751 | warning is ever generated if this is property is matched against a |
| 752 | non-Unicode code point (see L</Beyond Unicode code points> below). |
| 753 | |
| 754 | =item B<C<\p{Alnum}>> |
| 755 | |
| 756 | This matches any C<\p{Alphabetic}> or C<\p{Decimal_Number}> character. |
| 757 | |
| 758 | =item B<C<\p{Any}>> |
| 759 | |
| 760 | This matches any of the 1_114_112 Unicode code points. It is a synonym |
| 761 | for C<\p{Unicode}>. |
| 762 | |
| 763 | =item B<C<\p{ASCII}>> |
| 764 | |
| 765 | This matches any of the 128 characters in the US-ASCII character set, |
| 766 | which is a subset of Unicode. |
| 767 | |
| 768 | =item B<C<\p{Assigned}>> |
| 769 | |
| 770 | This matches any assigned code point; that is, any code point whose L<general |
| 771 | category|/General_Category> is not C<Unassigned> (or equivalently, not C<Cn>). |
| 772 | |
| 773 | =item B<C<\p{Blank}>> |
| 774 | |
| 775 | This is the same as C<\h> and C<\p{HorizSpace}>: A character that changes the |
| 776 | spacing horizontally. |
| 777 | |
| 778 | =item B<C<\p{Decomposition_Type: Non_Canonical}>> (Short: C<\p{Dt=NonCanon}>) |
| 779 | |
| 780 | Matches a character that has a non-canonical decomposition. |
| 781 | |
| 782 | The L</Extended Grapheme Clusters (Logical characters)> section above |
| 783 | talked about canonical decompositions. However, many more characters |
| 784 | have a different type of decomposition, a "compatible" or |
| 785 | "non-canonical" decomposition. The sequences that form these |
| 786 | decompositions are not considered canonically equivalent to the |
| 787 | pre-composed character. An example is the C<"SUPERSCRIPT ONE">. It is |
| 788 | somewhat like a regular digit 1, but not exactly; its decomposition into |
| 789 | the digit 1 is called a "compatible" decomposition, specifically a |
| 790 | "super" decomposition. There are several such compatibility |
| 791 | decompositions (see L<http://www.unicode.org/reports/tr44>), including |
| 792 | one called "compat", which means some miscellaneous type of |
| 793 | decomposition that doesn't fit into the other decomposition categories |
| 794 | that Unicode has chosen. |
| 795 | |
| 796 | Note that most Unicode characters don't have a decomposition, so their |
| 797 | decomposition type is C<"None">. |
| 798 | |
| 799 | For your convenience, Perl has added the C<Non_Canonical> decomposition |
| 800 | type to mean any of the several compatibility decompositions. |
| 801 | |
| 802 | =item B<C<\p{Graph}>> |
| 803 | |
| 804 | Matches any character that is graphic. Theoretically, this means a character |
| 805 | that on a printer would cause ink to be used. |
| 806 | |
| 807 | =item B<C<\p{HorizSpace}>> |
| 808 | |
| 809 | This is the same as C<\h> and C<\p{Blank}>: a character that changes the |
| 810 | spacing horizontally. |
| 811 | |
| 812 | =item B<C<\p{In=*}>> |
| 813 | |
| 814 | This is a synonym for C<\p{Present_In=*}> |
| 815 | |
| 816 | =item B<C<\p{PerlSpace}>> |
| 817 | |
| 818 | This is the same as C<\s>, restricted to ASCII, namely C<S<[ \f\n\r\t]>> |
| 819 | and starting in Perl v5.18, a vertical tab. |
| 820 | |
| 821 | Mnemonic: Perl's (original) space |
| 822 | |
| 823 | =item B<C<\p{PerlWord}>> |
| 824 | |
| 825 | This is the same as C<\w>, restricted to ASCII, namely C<[A-Za-z0-9_]> |
| 826 | |
| 827 | Mnemonic: Perl's (original) word. |
| 828 | |
| 829 | =item B<C<\p{Posix...}>> |
| 830 | |
| 831 | There are several of these, which are equivalents, using the C<\p{}> |
| 832 | notation, for Posix classes and are described in |
| 833 | L<perlrecharclass/POSIX Character Classes>. |
| 834 | |
| 835 | =item B<C<\p{Present_In: *}>> (Short: C<\p{In=*}>) |
| 836 | |
| 837 | This property is used when you need to know in what Unicode version(s) a |
| 838 | character is. |
| 839 | |
| 840 | The "*" above stands for some two digit Unicode version number, such as |
| 841 | C<1.1> or C<4.0>; or the "*" can also be C<Unassigned>. This property will |
| 842 | match the code points whose final disposition has been settled as of the |
| 843 | Unicode release given by the version number; C<\p{Present_In: Unassigned}> |
| 844 | will match those code points whose meaning has yet to be assigned. |
| 845 | |
| 846 | For example, C<U+0041> C<"LATIN CAPITAL LETTER A"> was present in the very first |
| 847 | Unicode release available, which is C<1.1>, so this property is true for all |
| 848 | valid "*" versions. On the other hand, C<U+1EFF> was not assigned until version |
| 849 | 5.1 when it became C<"LATIN SMALL LETTER Y WITH LOOP">, so the only "*" that |
| 850 | would match it are 5.1, 5.2, and later. |
| 851 | |
| 852 | Unicode furnishes the C<Age> property from which this is derived. The problem |
| 853 | with Age is that a strict interpretation of it (which Perl takes) has it |
| 854 | matching the precise release a code point's meaning is introduced in. Thus |
| 855 | C<U+0041> would match only 1.1; and C<U+1EFF> only 5.1. This is not usually what |
| 856 | you want. |
| 857 | |
| 858 | Some non-Perl implementations of the Age property may change its meaning to be |
| 859 | the same as the Perl C<Present_In> property; just be aware of that. |
| 860 | |
| 861 | Another confusion with both these properties is that the definition is not |
| 862 | that the code point has been I<assigned>, but that the meaning of the code point |
| 863 | has been I<determined>. This is because 66 code points will always be |
| 864 | unassigned, and so the C<Age> for them is the Unicode version in which the decision |
| 865 | to make them so was made. For example, C<U+FDD0> is to be permanently |
| 866 | unassigned to a character, and the decision to do that was made in version 3.1, |
| 867 | so C<\p{Age=3.1}> matches this character, as also does C<\p{Present_In: 3.1}> and up. |
| 868 | |
| 869 | =item B<C<\p{Print}>> |
| 870 | |
| 871 | This matches any character that is graphical or blank, except controls. |
| 872 | |
| 873 | =item B<C<\p{SpacePerl}>> |
| 874 | |
| 875 | This is the same as C<\s>, including beyond ASCII. |
| 876 | |
| 877 | Mnemonic: Space, as modified by Perl. (It doesn't include the vertical tab |
| 878 | until v5.18, which both the Posix standard and Unicode consider white space.) |
| 879 | |
| 880 | =item B<C<\p{Title}>> and B<C<\p{Titlecase}>> |
| 881 | |
| 882 | Under case-sensitive matching, these both match the same code points as |
| 883 | C<\p{General Category=Titlecase_Letter}> (C<\p{gc=lt}>). The difference |
| 884 | is that under C</i> caseless matching, these match the same as |
| 885 | C<\p{Cased}>, whereas C<\p{gc=lt}> matches C<\p{Cased_Letter>). |
| 886 | |
| 887 | =item B<C<\p{Unicode}>> |
| 888 | |
| 889 | This matches any of the 1_114_112 Unicode code points. |
| 890 | C<\p{Any}>. |
| 891 | |
| 892 | =item B<C<\p{VertSpace}>> |
| 893 | |
| 894 | This is the same as C<\v>: A character that changes the spacing vertically. |
| 895 | |
| 896 | =item B<C<\p{Word}>> |
| 897 | |
| 898 | This is the same as C<\w>, including over 100_000 characters beyond ASCII. |
| 899 | |
| 900 | =item B<C<\p{XPosix...}>> |
| 901 | |
| 902 | There are several of these, which are the standard Posix classes |
| 903 | extended to the full Unicode range. They are described in |
| 904 | L<perlrecharclass/POSIX Character Classes>. |
| 905 | |
| 906 | =back |
| 907 | |
| 908 | |
| 909 | =head2 User-Defined Character Properties |
| 910 | |
| 911 | You can define your own binary character properties by defining subroutines |
| 912 | whose names begin with C<"In"> or C<"Is">. (The experimental feature |
| 913 | L<perlre/(?[ ])> provides an alternative which allows more complex |
| 914 | definitions.) The subroutines can be defined in any |
| 915 | package. The user-defined properties can be used in the regular expression |
| 916 | C<\p{}> and C<\P{}> constructs; if you are using a user-defined property from a |
| 917 | package other than the one you are in, you must specify its package in the |
| 918 | C<\p{}> or C<\P{}> construct. |
| 919 | |
| 920 | # assuming property Is_Foreign defined in Lang:: |
| 921 | package main; # property package name required |
| 922 | if ($txt =~ /\p{Lang::IsForeign}+/) { ... } |
| 923 | |
| 924 | package Lang; # property package name not required |
| 925 | if ($txt =~ /\p{IsForeign}+/) { ... } |
| 926 | |
| 927 | |
| 928 | Note that the effect is compile-time and immutable once defined. |
| 929 | However, the subroutines are passed a single parameter, which is 0 if |
| 930 | case-sensitive matching is in effect and non-zero if caseless matching |
| 931 | is in effect. The subroutine may return different values depending on |
| 932 | the value of the flag, and one set of values will immutably be in effect |
| 933 | for all case-sensitive matches, and the other set for all case-insensitive |
| 934 | matches. |
| 935 | |
| 936 | Note that if the regular expression is tainted, then Perl will die rather |
| 937 | than calling the subroutine when the name of the subroutine is |
| 938 | determined by the tainted data. |
| 939 | |
| 940 | The subroutines must return a specially-formatted string, with one |
| 941 | or more newline-separated lines. Each line must be one of the following: |
| 942 | |
| 943 | =over 4 |
| 944 | |
| 945 | =item * |
| 946 | |
| 947 | A single hexadecimal number denoting a code point to include. |
| 948 | |
| 949 | =item * |
| 950 | |
| 951 | Two hexadecimal numbers separated by horizontal whitespace (space or |
| 952 | tabular characters) denoting a range of code points to include. |
| 953 | |
| 954 | =item * |
| 955 | |
| 956 | Something to include, prefixed by C<"+">: a built-in character |
| 957 | property (prefixed by C<"utf8::">) or a fully qualified (including package |
| 958 | name) user-defined character property, |
| 959 | to represent all the characters in that property; two hexadecimal code |
| 960 | points for a range; or a single hexadecimal code point. |
| 961 | |
| 962 | =item * |
| 963 | |
| 964 | Something to exclude, prefixed by C<"-">: an existing character |
| 965 | property (prefixed by C<"utf8::">) or a fully qualified (including package |
| 966 | name) user-defined character property, |
| 967 | to represent all the characters in that property; two hexadecimal code |
| 968 | points for a range; or a single hexadecimal code point. |
| 969 | |
| 970 | =item * |
| 971 | |
| 972 | Something to negate, prefixed C<"!">: an existing character |
| 973 | property (prefixed by C<"utf8::">) or a fully qualified (including package |
| 974 | name) user-defined character property, |
| 975 | to represent all the characters in that property; two hexadecimal code |
| 976 | points for a range; or a single hexadecimal code point. |
| 977 | |
| 978 | =item * |
| 979 | |
| 980 | Something to intersect with, prefixed by C<"&">: an existing character |
| 981 | property (prefixed by C<"utf8::">) or a fully qualified (including package |
| 982 | name) user-defined character property, |
| 983 | for all the characters except the characters in the property; two |
| 984 | hexadecimal code points for a range; or a single hexadecimal code point. |
| 985 | |
| 986 | =back |
| 987 | |
| 988 | For example, to define a property that covers both the Japanese |
| 989 | syllabaries (hiragana and katakana), you can define |
| 990 | |
| 991 | sub InKana { |
| 992 | return <<END; |
| 993 | 3040\t309F |
| 994 | 30A0\t30FF |
| 995 | END |
| 996 | } |
| 997 | |
| 998 | Imagine that the here-doc end marker is at the beginning of the line. |
| 999 | Now you can use C<\p{InKana}> and C<\P{InKana}>. |
| 1000 | |
| 1001 | You could also have used the existing block property names: |
| 1002 | |
| 1003 | sub InKana { |
| 1004 | return <<'END'; |
| 1005 | +utf8::InHiragana |
| 1006 | +utf8::InKatakana |
| 1007 | END |
| 1008 | } |
| 1009 | |
| 1010 | Suppose you wanted to match only the allocated characters, |
| 1011 | not the raw block ranges: in other words, you want to remove |
| 1012 | the unassigned characters: |
| 1013 | |
| 1014 | sub InKana { |
| 1015 | return <<'END'; |
| 1016 | +utf8::InHiragana |
| 1017 | +utf8::InKatakana |
| 1018 | -utf8::IsCn |
| 1019 | END |
| 1020 | } |
| 1021 | |
| 1022 | The negation is useful for defining (surprise!) negated classes. |
| 1023 | |
| 1024 | sub InNotKana { |
| 1025 | return <<'END'; |
| 1026 | !utf8::InHiragana |
| 1027 | -utf8::InKatakana |
| 1028 | +utf8::IsCn |
| 1029 | END |
| 1030 | } |
| 1031 | |
| 1032 | This will match all non-Unicode code points, since every one of them is |
| 1033 | not in Kana. You can use intersection to exclude these, if desired, as |
| 1034 | this modified example shows: |
| 1035 | |
| 1036 | sub InNotKana { |
| 1037 | return <<'END'; |
| 1038 | !utf8::InHiragana |
| 1039 | -utf8::InKatakana |
| 1040 | +utf8::IsCn |
| 1041 | &utf8::Any |
| 1042 | END |
| 1043 | } |
| 1044 | |
| 1045 | C<&utf8::Any> must be the last line in the definition. |
| 1046 | |
| 1047 | Intersection is used generally for getting the common characters matched |
| 1048 | by two (or more) classes. It's important to remember not to use C<"&"> for |
| 1049 | the first set; that would be intersecting with nothing, resulting in an |
| 1050 | empty set. |
| 1051 | |
| 1052 | Unlike non-user-defined C<\p{}> property matches, no warning is ever |
| 1053 | generated if these properties are matched against a non-Unicode code |
| 1054 | point (see L</Beyond Unicode code points> below). |
| 1055 | |
| 1056 | =head2 User-Defined Case Mappings (for serious hackers only) |
| 1057 | |
| 1058 | B<This feature has been removed as of Perl 5.16.> |
| 1059 | The CPAN module C<L<Unicode::Casing>> provides better functionality without |
| 1060 | the drawbacks that this feature had. If you are using a Perl earlier |
| 1061 | than 5.16, this feature was most fully documented in the 5.14 version of |
| 1062 | this pod: |
| 1063 | L<http://perldoc.perl.org/5.14.0/perlunicode.html#User-Defined-Case-Mappings-%28for-serious-hackers-only%29> |
| 1064 | |
| 1065 | =head2 Character Encodings for Input and Output |
| 1066 | |
| 1067 | See L<Encode>. |
| 1068 | |
| 1069 | =head2 Unicode Regular Expression Support Level |
| 1070 | |
| 1071 | The following list of Unicode supported features for regular expressions describes |
| 1072 | all features currently directly supported by core Perl. The references to "Level N" |
| 1073 | and the section numbers refer to the Unicode Technical Standard #18, |
| 1074 | "Unicode Regular Expressions", version 13, from August 2008. |
| 1075 | |
| 1076 | =over 4 |
| 1077 | |
| 1078 | =item * |
| 1079 | |
| 1080 | Level 1 - Basic Unicode Support |
| 1081 | |
| 1082 | RL1.1 Hex Notation - done [1] |
| 1083 | RL1.2 Properties - done [2][3] |
| 1084 | RL1.2a Compatibility Properties - done [4] |
| 1085 | RL1.3 Subtraction and Intersection - experimental [5] |
| 1086 | RL1.4 Simple Word Boundaries - done [6] |
| 1087 | RL1.5 Simple Loose Matches - done [7] |
| 1088 | RL1.6 Line Boundaries - MISSING [8][9] |
| 1089 | RL1.7 Supplementary Code Points - done [10] |
| 1090 | |
| 1091 | =over 4 |
| 1092 | |
| 1093 | =item [1] C<\N{U+...}> and C<\x{...}> |
| 1094 | |
| 1095 | =item [2] C<\p{...}> C<\P{...}> |
| 1096 | |
| 1097 | =item [3] supports not only minimal list, but all Unicode character |
| 1098 | properties (see Unicode Character Properties above) |
| 1099 | |
| 1100 | =item [4] C<\d> C<\D> C<\s> C<\S> C<\w> C<\W> C<\X> C<[:I<prop>:]> |
| 1101 | C<[:^I<prop>:]> |
| 1102 | |
| 1103 | =item [5] The experimental feature starting in v5.18 C<"(?[...])"> accomplishes |
| 1104 | this. |
| 1105 | |
| 1106 | See L<perlre/(?[ ])>. If you don't want to use an experimental |
| 1107 | feature, you can use one of the following: |
| 1108 | |
| 1109 | =over 4 |
| 1110 | |
| 1111 | =item * |
| 1112 | |
| 1113 | Regular expression lookahead |
| 1114 | |
| 1115 | You can mimic class subtraction using lookahead. |
| 1116 | For example, what UTS#18 might write as |
| 1117 | |
| 1118 | [{Block=Greek}-[{UNASSIGNED}]] |
| 1119 | |
| 1120 | in Perl can be written as: |
| 1121 | |
| 1122 | (?!\p{Unassigned})\p{Block=Greek} |
| 1123 | (?=\p{Assigned})\p{Block=Greek} |
| 1124 | |
| 1125 | But in this particular example, you probably really want |
| 1126 | |
| 1127 | \p{Greek} |
| 1128 | |
| 1129 | which will match assigned characters known to be part of the Greek script. |
| 1130 | |
| 1131 | =item * |
| 1132 | |
| 1133 | CPAN module C<L<Unicode::Regex::Set>> |
| 1134 | |
| 1135 | It does implement the full UTS#18 grouping, intersection, union, and |
| 1136 | removal (subtraction) syntax. |
| 1137 | |
| 1138 | =item * |
| 1139 | |
| 1140 | L</"User-Defined Character Properties"> |
| 1141 | |
| 1142 | C<"+"> for union, C<"-"> for removal (set-difference), C<"&"> for intersection |
| 1143 | |
| 1144 | =back |
| 1145 | |
| 1146 | =item [6] C<\b> C<\B> |
| 1147 | |
| 1148 | =item [7] |
| 1149 | Note that Perl does Full case-folding in matching, not Simple: |
| 1150 | |
| 1151 | For example C<U+1F88> is equivalent to C<U+1F00 U+03B9>, instead of just |
| 1152 | C<U+1F80>. This difference matters mainly for certain Greek capital |
| 1153 | letters with certain modifiers: the Full case-folding decomposes the |
| 1154 | letter, while the Simple case-folding would map it to a single |
| 1155 | character. |
| 1156 | |
| 1157 | =item [8] |
| 1158 | Perl treats C<\n> as the start- and end-line delimiter. Unicode |
| 1159 | specifies more characters that should be so-interpreted. |
| 1160 | |
| 1161 | These are: |
| 1162 | |
| 1163 | VT U+000B (\v in C) |
| 1164 | FF U+000C (\f) |
| 1165 | CR U+000D (\r) |
| 1166 | NEL U+0085 |
| 1167 | LS U+2028 |
| 1168 | PS U+2029 |
| 1169 | |
| 1170 | C<^> and C<$> in regular expression patterns are supposed to match all |
| 1171 | these, but don't. |
| 1172 | These characters also don't, but should, affect C<< <> >> C<$.>, and |
| 1173 | script line numbers. |
| 1174 | |
| 1175 | Also, lines should not be split within C<CRLF> (i.e. there is no |
| 1176 | empty line between C<\r> and C<\n>). For C<CRLF>, try the C<:crlf> |
| 1177 | layer (see L<PerlIO>). |
| 1178 | |
| 1179 | =item [9] But C<qr/\b{lb}/> and C<L<Unicode::LineBreak>> are available. |
| 1180 | |
| 1181 | L<C<qrE<sol>\b{lb}E<sol>>|perlrebackslash/\b{lb}> supplies default line |
| 1182 | breaking conformant with |
| 1183 | L<UAX#14 "Unicode Line Breaking Algorithm"|http://www.unicode.org/reports/tr14>. |
| 1184 | |
| 1185 | And, the module C<L<Unicode::LineBreak>> also conformant with UAX#14, |
| 1186 | provides customizable line breaking. |
| 1187 | |
| 1188 | =item [10] |
| 1189 | UTF-8/UTF-EBDDIC used in Perl allows not only C<U+10000> to |
| 1190 | C<U+10FFFF> but also beyond C<U+10FFFF> |
| 1191 | |
| 1192 | =back |
| 1193 | |
| 1194 | =item * |
| 1195 | |
| 1196 | Level 2 - Extended Unicode Support |
| 1197 | |
| 1198 | RL2.1 Canonical Equivalents - MISSING [10][11] |
| 1199 | RL2.2 Default Grapheme Clusters - MISSING [12] |
| 1200 | RL2.3 Default Word Boundaries - DONE [14] |
| 1201 | RL2.4 Default Loose Matches - MISSING [15] |
| 1202 | RL2.5 Name Properties - DONE |
| 1203 | RL2.6 Wildcard Properties - MISSING |
| 1204 | |
| 1205 | [10] see UAX#15 "Unicode Normalization Forms" |
| 1206 | [11] have Unicode::Normalize but not integrated to regexes |
| 1207 | [12] have \X and \b{gcb} but we don't have a "Grapheme Cluster |
| 1208 | Mode" |
| 1209 | [14] see UAX#29, Word Boundaries |
| 1210 | [15] This is covered in Chapter 3.13 (in Unicode 6.0) |
| 1211 | |
| 1212 | =item * |
| 1213 | |
| 1214 | Level 3 - Tailored Support |
| 1215 | |
| 1216 | RL3.1 Tailored Punctuation - MISSING |
| 1217 | RL3.2 Tailored Grapheme Clusters - MISSING [17][18] |
| 1218 | RL3.3 Tailored Word Boundaries - MISSING |
| 1219 | RL3.4 Tailored Loose Matches - MISSING |
| 1220 | RL3.5 Tailored Ranges - MISSING |
| 1221 | RL3.6 Context Matching - MISSING [19] |
| 1222 | RL3.7 Incremental Matches - MISSING |
| 1223 | ( RL3.8 Unicode Set Sharing ) |
| 1224 | RL3.9 Possible Match Sets - MISSING |
| 1225 | RL3.10 Folded Matching - MISSING [20] |
| 1226 | RL3.11 Submatchers - MISSING |
| 1227 | |
| 1228 | [17] see UAX#10 "Unicode Collation Algorithms" |
| 1229 | [18] have Unicode::Collate but not integrated to regexes |
| 1230 | [19] have (?<=x) and (?=x), but lookaheads or lookbehinds |
| 1231 | should see outside of the target substring |
| 1232 | [20] need insensitive matching for linguistic features other |
| 1233 | than case; for example, hiragana to katakana, wide and |
| 1234 | narrow, simplified Han to traditional Han (see UTR#30 |
| 1235 | "Character Foldings") |
| 1236 | |
| 1237 | =back |
| 1238 | |
| 1239 | =head2 Unicode Encodings |
| 1240 | |
| 1241 | Unicode characters are assigned to I<code points>, which are abstract |
| 1242 | numbers. To use these numbers, various encodings are needed. |
| 1243 | |
| 1244 | =over 4 |
| 1245 | |
| 1246 | =item * |
| 1247 | |
| 1248 | UTF-8 |
| 1249 | |
| 1250 | UTF-8 is a variable-length (1 to 4 bytes), byte-order independent |
| 1251 | encoding. In most of Perl's documentation, including elsewhere in this |
| 1252 | document, the term "UTF-8" means also "UTF-EBCDIC". But in this section, |
| 1253 | "UTF-8" refers only to the encoding used on ASCII platforms. It is a |
| 1254 | superset of 7-bit US-ASCII, so anything encoded in ASCII has the |
| 1255 | identical representation when encoded in UTF-8. |
| 1256 | |
| 1257 | The following table is from Unicode 3.2. |
| 1258 | |
| 1259 | Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte |
| 1260 | |
| 1261 | U+0000..U+007F 00..7F |
| 1262 | U+0080..U+07FF * C2..DF 80..BF |
| 1263 | U+0800..U+0FFF E0 * A0..BF 80..BF |
| 1264 | U+1000..U+CFFF E1..EC 80..BF 80..BF |
| 1265 | U+D000..U+D7FF ED 80..9F 80..BF |
| 1266 | U+D800..U+DFFF +++++ utf16 surrogates, not legal utf8 +++++ |
| 1267 | U+E000..U+FFFF EE..EF 80..BF 80..BF |
| 1268 | U+10000..U+3FFFF F0 * 90..BF 80..BF 80..BF |
| 1269 | U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF |
| 1270 | U+100000..U+10FFFF F4 80..8F 80..BF 80..BF |
| 1271 | |
| 1272 | Note the gaps marked by "*" before several of the byte entries above. These are |
| 1273 | caused by legal UTF-8 avoiding non-shortest encodings: it is technically |
| 1274 | possible to UTF-8-encode a single code point in different ways, but that is |
| 1275 | explicitly forbidden, and the shortest possible encoding should always be used |
| 1276 | (and that is what Perl does). |
| 1277 | |
| 1278 | Another way to look at it is via bits: |
| 1279 | |
| 1280 | Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte |
| 1281 | |
| 1282 | 0aaaaaaa 0aaaaaaa |
| 1283 | 00000bbbbbaaaaaa 110bbbbb 10aaaaaa |
| 1284 | ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa |
| 1285 | 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa |
| 1286 | |
| 1287 | As you can see, the continuation bytes all begin with C<"10">, and the |
| 1288 | leading bits of the start byte tell how many bytes there are in the |
| 1289 | encoded character. |
| 1290 | |
| 1291 | The original UTF-8 specification allowed up to 6 bytes, to allow |
| 1292 | encoding of numbers up to C<0x7FFF_FFFF>. Perl continues to allow those, |
| 1293 | and has extended that up to 13 bytes to encode code points up to what |
| 1294 | can fit in a 64-bit word. However, Perl will warn if you output any of |
| 1295 | these as being non-portable; and under strict UTF-8 input protocols, |
| 1296 | they are forbidden. In addition, it is deprecated to use a code point |
| 1297 | larger than what a signed integer variable on your system can hold. On |
| 1298 | 32-bit ASCII systems, this means C<0x7FFF_FFFF> is the legal maximum |
| 1299 | going forward (much higher on 64-bit systems). |
| 1300 | |
| 1301 | =item * |
| 1302 | |
| 1303 | UTF-EBCDIC |
| 1304 | |
| 1305 | Like UTF-8, but EBCDIC-safe, in the way that UTF-8 is ASCII-safe. |
| 1306 | This means that all the basic characters (which includes all |
| 1307 | those that have ASCII equivalents (like C<"A">, C<"0">, C<"%">, I<etc.>) |
| 1308 | are the same in both EBCDIC and UTF-EBCDIC.) |
| 1309 | |
| 1310 | UTF-EBCDIC is used on EBCDIC platforms. It generally requires more |
| 1311 | bytes to represent a given code point than UTF-8 does; the largest |
| 1312 | Unicode code points take 5 bytes to represent (instead of 4 in UTF-8), |
| 1313 | and, extended for 64-bit words, it uses 14 bytes instead of 13 bytes in |
| 1314 | UTF-8. |
| 1315 | |
| 1316 | =item * |
| 1317 | |
| 1318 | UTF-16, UTF-16BE, UTF-16LE, Surrogates, and C<BOM>'s (Byte Order Marks) |
| 1319 | |
| 1320 | The followings items are mostly for reference and general Unicode |
| 1321 | knowledge, Perl doesn't use these constructs internally. |
| 1322 | |
| 1323 | Like UTF-8, UTF-16 is a variable-width encoding, but where |
| 1324 | UTF-8 uses 8-bit code units, UTF-16 uses 16-bit code units. |
| 1325 | All code points occupy either 2 or 4 bytes in UTF-16: code points |
| 1326 | C<U+0000..U+FFFF> are stored in a single 16-bit unit, and code |
| 1327 | points C<U+10000..U+10FFFF> in two 16-bit units. The latter case is |
| 1328 | using I<surrogates>, the first 16-bit unit being the I<high |
| 1329 | surrogate>, and the second being the I<low surrogate>. |
| 1330 | |
| 1331 | Surrogates are code points set aside to encode the C<U+10000..U+10FFFF> |
| 1332 | range of Unicode code points in pairs of 16-bit units. The I<high |
| 1333 | surrogates> are the range C<U+D800..U+DBFF> and the I<low surrogates> |
| 1334 | are the range C<U+DC00..U+DFFF>. The surrogate encoding is |
| 1335 | |
| 1336 | $hi = ($uni - 0x10000) / 0x400 + 0xD800; |
| 1337 | $lo = ($uni - 0x10000) % 0x400 + 0xDC00; |
| 1338 | |
| 1339 | and the decoding is |
| 1340 | |
| 1341 | $uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00); |
| 1342 | |
| 1343 | Because of the 16-bitness, UTF-16 is byte-order dependent. UTF-16 |
| 1344 | itself can be used for in-memory computations, but if storage or |
| 1345 | transfer is required either UTF-16BE (big-endian) or UTF-16LE |
| 1346 | (little-endian) encodings must be chosen. |
| 1347 | |
| 1348 | This introduces another problem: what if you just know that your data |
| 1349 | is UTF-16, but you don't know which endianness? Byte Order Marks, or |
| 1350 | C<BOM>'s, are a solution to this. A special character has been reserved |
| 1351 | in Unicode to function as a byte order marker: the character with the |
| 1352 | code point C<U+FEFF> is the C<BOM>. |
| 1353 | |
| 1354 | The trick is that if you read a C<BOM>, you will know the byte order, |
| 1355 | since if it was written on a big-endian platform, you will read the |
| 1356 | bytes C<0xFE 0xFF>, but if it was written on a little-endian platform, |
| 1357 | you will read the bytes C<0xFF 0xFE>. (And if the originating platform |
| 1358 | was writing in ASCII platform UTF-8, you will read the bytes |
| 1359 | C<0xEF 0xBB 0xBF>.) |
| 1360 | |
| 1361 | The way this trick works is that the character with the code point |
| 1362 | C<U+FFFE> is not supposed to be in input streams, so the |
| 1363 | sequence of bytes C<0xFF 0xFE> is unambiguously "C<BOM>, represented in |
| 1364 | little-endian format" and cannot be C<U+FFFE>, represented in big-endian |
| 1365 | format". |
| 1366 | |
| 1367 | Surrogates have no meaning in Unicode outside their use in pairs to |
| 1368 | represent other code points. However, Perl allows them to be |
| 1369 | represented individually internally, for example by saying |
| 1370 | C<chr(0xD801)>, so that all code points, not just those valid for open |
| 1371 | interchange, are |
| 1372 | representable. Unicode does define semantics for them, such as their |
| 1373 | C<L</General_Category>> is C<"Cs">. But because their use is somewhat dangerous, |
| 1374 | Perl will warn (using the warning category C<"surrogate">, which is a |
| 1375 | sub-category of C<"utf8">) if an attempt is made |
| 1376 | to do things like take the lower case of one, or match |
| 1377 | case-insensitively, or to output them. (But don't try this on Perls |
| 1378 | before 5.14.) |
| 1379 | |
| 1380 | =item * |
| 1381 | |
| 1382 | UTF-32, UTF-32BE, UTF-32LE |
| 1383 | |
| 1384 | The UTF-32 family is pretty much like the UTF-16 family, except that |
| 1385 | the units are 32-bit, and therefore the surrogate scheme is not |
| 1386 | needed. UTF-32 is a fixed-width encoding. The C<BOM> signatures are |
| 1387 | C<0x00 0x00 0xFE 0xFF> for BE and C<0xFF 0xFE 0x00 0x00> for LE. |
| 1388 | |
| 1389 | =item * |
| 1390 | |
| 1391 | UCS-2, UCS-4 |
| 1392 | |
| 1393 | Legacy, fixed-width encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit |
| 1394 | encoding. Unlike UTF-16, UCS-2 is not extensible beyond C<U+FFFF>, |
| 1395 | because it does not use surrogates. UCS-4 is a 32-bit encoding, |
| 1396 | functionally identical to UTF-32 (the difference being that |
| 1397 | UCS-4 forbids neither surrogates nor code points larger than C<0x10_FFFF>). |
| 1398 | |
| 1399 | =item * |
| 1400 | |
| 1401 | UTF-7 |
| 1402 | |
| 1403 | A seven-bit safe (non-eight-bit) encoding, which is useful if the |
| 1404 | transport or storage is not eight-bit safe. Defined by RFC 2152. |
| 1405 | |
| 1406 | =back |
| 1407 | |
| 1408 | =head2 Noncharacter code points |
| 1409 | |
| 1410 | 66 code points are set aside in Unicode as "noncharacter code points". |
| 1411 | These all have the C<Unassigned> (C<Cn>) C<L</General_Category>>, and |
| 1412 | no character will ever be assigned to any of them. They are the 32 code |
| 1413 | points between C<U+FDD0> and C<U+FDEF> inclusive, and the 34 code |
| 1414 | points: |
| 1415 | |
| 1416 | U+FFFE U+FFFF |
| 1417 | U+1FFFE U+1FFFF |
| 1418 | U+2FFFE U+2FFFF |
| 1419 | ... |
| 1420 | U+EFFFE U+EFFFF |
| 1421 | U+FFFFE U+FFFFF |
| 1422 | U+10FFFE U+10FFFF |
| 1423 | |
| 1424 | Until Unicode 7.0, the noncharacters were "B<forbidden> for use in open |
| 1425 | interchange of Unicode text data", so that code that processed those |
| 1426 | streams could use these code points as sentinels that could be mixed in |
| 1427 | with character data, and would always be distinguishable from that data. |
| 1428 | (Emphasis above and in the next paragraph are added in this document.) |
| 1429 | |
| 1430 | Unicode 7.0 changed the wording so that they are "B<not recommended> for |
| 1431 | use in open interchange of Unicode text data". The 7.0 Standard goes on |
| 1432 | to say: |
| 1433 | |
| 1434 | =over 4 |
| 1435 | |
| 1436 | "If a noncharacter is received in open interchange, an application is |
| 1437 | not required to interpret it in any way. It is good practice, however, |
| 1438 | to recognize it as a noncharacter and to take appropriate action, such |
| 1439 | as replacing it with C<U+FFFD> replacement character, to indicate the |
| 1440 | problem in the text. It is not recommended to simply delete |
| 1441 | noncharacter code points from such text, because of the potential |
| 1442 | security issues caused by deleting uninterpreted characters. (See |
| 1443 | conformance clause C7 in Section 3.2, Conformance Requirements, and |
| 1444 | L<Unicode Technical Report #36, "Unicode Security |
| 1445 | Considerations"|http://www.unicode.org/reports/tr36/#Substituting_for_Ill_Formed_Subsequences>)." |
| 1446 | |
| 1447 | =back |
| 1448 | |
| 1449 | This change was made because it was found that various commercial tools |
| 1450 | like editors, or for things like source code control, had been written |
| 1451 | so that they would not handle program files that used these code points, |
| 1452 | effectively precluding their use almost entirely! And that was never |
| 1453 | the intent. They've always been meant to be usable within an |
| 1454 | application, or cooperating set of applications, at will. |
| 1455 | |
| 1456 | If you're writing code, such as an editor, that is supposed to be able |
| 1457 | to handle any Unicode text data, then you shouldn't be using these code |
| 1458 | points yourself, and instead allow them in the input. If you need |
| 1459 | sentinels, they should instead be something that isn't legal Unicode. |
| 1460 | For UTF-8 data, you can use the bytes 0xC1 and 0xC2 as sentinels, as |
| 1461 | they never appear in well-formed UTF-8. (There are equivalents for |
| 1462 | UTF-EBCDIC). You can also store your Unicode code points in integer |
| 1463 | variables and use negative values as sentinels. |
| 1464 | |
| 1465 | If you're not writing such a tool, then whether you accept noncharacters |
| 1466 | as input is up to you (though the Standard recommends that you not). If |
| 1467 | you do strict input stream checking with Perl, these code points |
| 1468 | continue to be forbidden. This is to maintain backward compatibility |
| 1469 | (otherwise potential security holes could open up, as an unsuspecting |
| 1470 | application that was written assuming the noncharacters would be |
| 1471 | filtered out before getting to it, could now, without warning, start |
| 1472 | getting them). To do strict checking, you can use the layer |
| 1473 | C<:encoding('UTF-8')>. |
| 1474 | |
| 1475 | Perl continues to warn (using the warning category C<"nonchar">, which |
| 1476 | is a sub-category of C<"utf8">) if an attempt is made to output |
| 1477 | noncharacters. |
| 1478 | |
| 1479 | =head2 Beyond Unicode code points |
| 1480 | |
| 1481 | The maximum Unicode code point is C<U+10FFFF>, and Unicode only defines |
| 1482 | operations on code points up through that. But Perl works on code |
| 1483 | points up to the maximum permissible unsigned number available on the |
| 1484 | platform. However, Perl will not accept these from input streams unless |
| 1485 | lax rules are being used, and will warn (using the warning category |
| 1486 | C<"non_unicode">, which is a sub-category of C<"utf8">) if any are output. |
| 1487 | |
| 1488 | Since Unicode rules are not defined on these code points, if a |
| 1489 | Unicode-defined operation is done on them, Perl uses what we believe are |
| 1490 | sensible rules, while generally warning, using the C<"non_unicode"> |
| 1491 | category. For example, C<uc("\x{11_0000}")> will generate such a |
| 1492 | warning, returning the input parameter as its result, since Perl defines |
| 1493 | the uppercase of every non-Unicode code point to be the code point |
| 1494 | itself. (All the case changing operations, not just uppercasing, work |
| 1495 | this way.) |
| 1496 | |
| 1497 | The situation with matching Unicode properties in regular expressions, |
| 1498 | the C<\p{}> and C<\P{}> constructs, against these code points is not as |
| 1499 | clear cut, and how these are handled has changed as we've gained |
| 1500 | experience. |
| 1501 | |
| 1502 | One possibility is to treat any match against these code points as |
| 1503 | undefined. But since Perl doesn't have the concept of a match being |
| 1504 | undefined, it converts this to failing or C<FALSE>. This is almost, but |
| 1505 | not quite, what Perl did from v5.14 (when use of these code points |
| 1506 | became generally reliable) through v5.18. The difference is that Perl |
| 1507 | treated all C<\p{}> matches as failing, but all C<\P{}> matches as |
| 1508 | succeeding. |
| 1509 | |
| 1510 | One problem with this is that it leads to unexpected, and confusting |
| 1511 | results in some cases: |
| 1512 | |
| 1513 | chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Failed on <= v5.18 |
| 1514 | chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Failed! on <= v5.18 |
| 1515 | |
| 1516 | That is, it treated both matches as undefined, and converted that to |
| 1517 | false (raising a warning on each). The first case is the expected |
| 1518 | result, but the second is likely counterintuitive: "How could both be |
| 1519 | false when they are complements?" Another problem was that the |
| 1520 | implementation optimized many Unicode property matches down to already |
| 1521 | existing simpler, faster operations, which don't raise the warning. We |
| 1522 | chose to not forgo those optimizations, which help the vast majority of |
| 1523 | matches, just to generate a warning for the unlikely event that an |
| 1524 | above-Unicode code point is being matched against. |
| 1525 | |
| 1526 | As a result of these problems, starting in v5.20, what Perl does is |
| 1527 | to treat non-Unicode code points as just typical unassigned Unicode |
| 1528 | characters, and matches accordingly. (Note: Unicode has atypical |
| 1529 | unassigned code points. For example, it has noncharacter code points, |
| 1530 | and ones that, when they do get assigned, are destined to be written |
| 1531 | Right-to-left, as Arabic and Hebrew are. Perl assumes that no |
| 1532 | non-Unicode code point has any atypical properties.) |
| 1533 | |
| 1534 | Perl, in most cases, will raise a warning when matching an above-Unicode |
| 1535 | code point against a Unicode property when the result is C<TRUE> for |
| 1536 | C<\p{}>, and C<FALSE> for C<\P{}>. For example: |
| 1537 | |
| 1538 | chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Fails, no warning |
| 1539 | chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Succeeds, with warning |
| 1540 | |
| 1541 | In both these examples, the character being matched is non-Unicode, so |
| 1542 | Unicode doesn't define how it should match. It clearly isn't an ASCII |
| 1543 | hex digit, so the first example clearly should fail, and so it does, |
| 1544 | with no warning. But it is arguable that the second example should have |
| 1545 | an undefined, hence C<FALSE>, result. So a warning is raised for it. |
| 1546 | |
| 1547 | Thus the warning is raised for many fewer cases than in earlier Perls, |
| 1548 | and only when what the result is could be arguable. It turns out that |
| 1549 | none of the optimizations made by Perl (or are ever likely to be made) |
| 1550 | cause the warning to be skipped, so it solves both problems of Perl's |
| 1551 | earlier approach. The most commonly used property that is affected by |
| 1552 | this change is C<\p{Unassigned}> which is a short form for |
| 1553 | C<\p{General_Category=Unassigned}>. Starting in v5.20, all non-Unicode |
| 1554 | code points are considered C<Unassigned>. In earlier releases the |
| 1555 | matches failed because the result was considered undefined. |
| 1556 | |
| 1557 | The only place where the warning is not raised when it might ought to |
| 1558 | have been is if optimizations cause the whole pattern match to not even |
| 1559 | be attempted. For example, Perl may figure out that for a string to |
| 1560 | match a certain regular expression pattern, the string has to contain |
| 1561 | the substring C<"foobar">. Before attempting the match, Perl may look |
| 1562 | for that substring, and if not found, immediately fail the match without |
| 1563 | actually trying it; so no warning gets generated even if the string |
| 1564 | contains an above-Unicode code point. |
| 1565 | |
| 1566 | This behavior is more "Do what I mean" than in earlier Perls for most |
| 1567 | applications. But it catches fewer issues for code that needs to be |
| 1568 | strictly Unicode compliant. Therefore there is an additional mode of |
| 1569 | operation available to accommodate such code. This mode is enabled if a |
| 1570 | regular expression pattern is compiled within the lexical scope where |
| 1571 | the C<"non_unicode"> warning class has been made fatal, say by: |
| 1572 | |
| 1573 | use warnings FATAL => "non_unicode" |
| 1574 | |
| 1575 | (see L<warnings>). In this mode of operation, Perl will raise the |
| 1576 | warning for all matches against a non-Unicode code point (not just the |
| 1577 | arguable ones), and it skips the optimizations that might cause the |
| 1578 | warning to not be output. (It currently still won't warn if the match |
| 1579 | isn't even attempted, like in the C<"foobar"> example above.) |
| 1580 | |
| 1581 | In summary, Perl now normally treats non-Unicode code points as typical |
| 1582 | Unicode unassigned code points for regular expression matches, raising a |
| 1583 | warning only when it is arguable what the result should be. However, if |
| 1584 | this warning has been made fatal, it isn't skipped. |
| 1585 | |
| 1586 | There is one exception to all this. C<\p{All}> looks like a Unicode |
| 1587 | property, but it is a Perl extension that is defined to be true for all |
| 1588 | possible code points, Unicode or not, so no warning is ever generated |
| 1589 | when matching this against a non-Unicode code point. (Prior to v5.20, |
| 1590 | it was an exact synonym for C<\p{Any}>, matching code points C<0> |
| 1591 | through C<0x10FFFF>.) |
| 1592 | |
| 1593 | =head2 Security Implications of Unicode |
| 1594 | |
| 1595 | First, read |
| 1596 | L<Unicode Security Considerations|http://www.unicode.org/reports/tr36>. |
| 1597 | |
| 1598 | Also, note the following: |
| 1599 | |
| 1600 | =over 4 |
| 1601 | |
| 1602 | =item * |
| 1603 | |
| 1604 | Malformed UTF-8 |
| 1605 | |
| 1606 | Unfortunately, the original specification of UTF-8 leaves some room for |
| 1607 | interpretation of how many bytes of encoded output one should generate |
| 1608 | from one input Unicode character. Strictly speaking, the shortest |
| 1609 | possible sequence of UTF-8 bytes should be generated, |
| 1610 | because otherwise there is potential for an input buffer overflow at |
| 1611 | the receiving end of a UTF-8 connection. Perl always generates the |
| 1612 | shortest length UTF-8, and with warnings on, Perl will warn about |
| 1613 | non-shortest length UTF-8 along with other malformations, such as the |
| 1614 | surrogates, which are not Unicode code points valid for interchange. |
| 1615 | |
| 1616 | =item * |
| 1617 | |
| 1618 | Regular expression pattern matching may surprise you if you're not |
| 1619 | accustomed to Unicode. Starting in Perl 5.14, several pattern |
| 1620 | modifiers are available to control this, called the character set |
| 1621 | modifiers. Details are given in L<perlre/Character set modifiers>. |
| 1622 | |
| 1623 | =back |
| 1624 | |
| 1625 | As discussed elsewhere, Perl has one foot (two hooves?) planted in |
| 1626 | each of two worlds: the old world of ASCII and single-byte locales, and |
| 1627 | the new world of Unicode, upgrading when necessary. |
| 1628 | If your legacy code does not explicitly use Unicode, no automatic |
| 1629 | switch-over to Unicode should happen. |
| 1630 | |
| 1631 | =head2 Unicode in Perl on EBCDIC |
| 1632 | |
| 1633 | Unicode is supported on EBCDIC platforms. See L<perlebcdic>. |
| 1634 | |
| 1635 | Unless ASCII vs. EBCDIC issues are specifically being discussed, |
| 1636 | references to UTF-8 encoding in this document and elsewhere should be |
| 1637 | read as meaning UTF-EBCDIC on EBCDIC platforms. |
| 1638 | See L<perlebcdic/Unicode and UTF>. |
| 1639 | |
| 1640 | Because UTF-EBCDIC is so similar to UTF-8, the differences are mostly |
| 1641 | hidden from you; S<C<use utf8>> (and NOT something like |
| 1642 | S<C<use utfebcdic>>) declares the the script is in the platform's |
| 1643 | "native" 8-bit encoding of Unicode. (Similarly for the C<":utf8"> |
| 1644 | layer.) |
| 1645 | |
| 1646 | =head2 Locales |
| 1647 | |
| 1648 | See L<perllocale/Unicode and UTF-8> |
| 1649 | |
| 1650 | =head2 When Unicode Does Not Happen |
| 1651 | |
| 1652 | There are still many places where Unicode (in some encoding or |
| 1653 | another) could be given as arguments or received as results, or both in |
| 1654 | Perl, but it is not, in spite of Perl having extensive ways to input and |
| 1655 | output in Unicode, and a few other "entry points" like the C<@ARGV> |
| 1656 | array (which can sometimes be interpreted as UTF-8). |
| 1657 | |
| 1658 | The following are such interfaces. Also, see L</The "Unicode Bug">. |
| 1659 | For all of these interfaces Perl |
| 1660 | currently (as of v5.16.0) simply assumes byte strings both as arguments |
| 1661 | and results, or UTF-8 strings if the (deprecated) C<encoding> pragma has been used. |
| 1662 | |
| 1663 | One reason that Perl does not attempt to resolve the role of Unicode in |
| 1664 | these situations is that the answers are highly dependent on the operating |
| 1665 | system and the file system(s). For example, whether filenames can be |
| 1666 | in Unicode and in exactly what kind of encoding, is not exactly a |
| 1667 | portable concept. Similarly for C<qx> and C<system>: how well will the |
| 1668 | "command-line interface" (and which of them?) handle Unicode? |
| 1669 | |
| 1670 | =over 4 |
| 1671 | |
| 1672 | =item * |
| 1673 | |
| 1674 | C<chdir>, C<chmod>, C<chown>, C<chroot>, C<exec>, C<link>, C<lstat>, C<mkdir>, |
| 1675 | C<rename>, C<rmdir>, C<stat>, C<symlink>, C<truncate>, C<unlink>, C<utime>, C<-X> |
| 1676 | |
| 1677 | =item * |
| 1678 | |
| 1679 | C<%ENV> |
| 1680 | |
| 1681 | =item * |
| 1682 | |
| 1683 | C<glob> (aka the C<E<lt>*E<gt>>) |
| 1684 | |
| 1685 | =item * |
| 1686 | |
| 1687 | C<open>, C<opendir>, C<sysopen> |
| 1688 | |
| 1689 | =item * |
| 1690 | |
| 1691 | C<qx> (aka the backtick operator), C<system> |
| 1692 | |
| 1693 | =item * |
| 1694 | |
| 1695 | C<readdir>, C<readlink> |
| 1696 | |
| 1697 | =back |
| 1698 | |
| 1699 | =head2 The "Unicode Bug" |
| 1700 | |
| 1701 | The term, "Unicode bug" has been applied to an inconsistency with the |
| 1702 | code points in the C<Latin-1 Supplement> block, that is, between |
| 1703 | 128 and 255. Without a locale specified, unlike all other characters or |
| 1704 | code points, these characters can have very different semantics |
| 1705 | depending on the rules in effect. (Characters whose code points are |
| 1706 | above 255 force Unicode rules; whereas the rules for ASCII characters |
| 1707 | are the same under both ASCII and Unicode rules.) |
| 1708 | |
| 1709 | Under Unicode rules, these upper-Latin1 characters are interpreted as |
| 1710 | Unicode code points, which means they have the same semantics as Latin-1 |
| 1711 | (ISO-8859-1) and C1 controls. |
| 1712 | |
| 1713 | As explained in L</ASCII Rules versus Unicode Rules>, under ASCII rules, |
| 1714 | they are considered to be unassigned characters. |
| 1715 | |
| 1716 | This can lead to unexpected results. For example, a string's |
| 1717 | semantics can suddenly change if a code point above 255 is appended to |
| 1718 | it, which changes the rules from ASCII to Unicode. As an |
| 1719 | example, consider the following program and its output: |
| 1720 | |
| 1721 | $ perl -le' |
| 1722 | no feature 'unicode_strings'; |
| 1723 | $s1 = "\xC2"; |
| 1724 | $s2 = "\x{2660}"; |
| 1725 | for ($s1, $s2, $s1.$s2) { |
| 1726 | print /\w/ || 0; |
| 1727 | } |
| 1728 | ' |
| 1729 | 0 |
| 1730 | 0 |
| 1731 | 1 |
| 1732 | |
| 1733 | If there's no C<\w> in C<s1> nor in C<s2>, why does their concatenation |
| 1734 | have one? |
| 1735 | |
| 1736 | This anomaly stems from Perl's attempt to not disturb older programs that |
| 1737 | didn't use Unicode, along with Perl's desire to add Unicode support |
| 1738 | seamlessly. But the result turned out to not be seamless. (By the way, |
| 1739 | you can choose to be warned when things like this happen. See |
| 1740 | C<L<encoding::warnings>>.) |
| 1741 | |
| 1742 | L<S<C<use feature 'unicode_strings'>>|feature/The 'unicode_strings' feature> |
| 1743 | was added, starting in Perl v5.12, to address this problem. It affects |
| 1744 | these things: |
| 1745 | |
| 1746 | =over 4 |
| 1747 | |
| 1748 | =item * |
| 1749 | |
| 1750 | Changing the case of a scalar, that is, using C<uc()>, C<ucfirst()>, C<lc()>, |
| 1751 | and C<lcfirst()>, or C<\L>, C<\U>, C<\u> and C<\l> in double-quotish |
| 1752 | contexts, such as regular expression substitutions. |
| 1753 | |
| 1754 | Under C<unicode_strings> starting in Perl 5.12.0, Unicode rules are |
| 1755 | generally used. See L<perlfunc/lc> for details on how this works |
| 1756 | in combination with various other pragmas. |
| 1757 | |
| 1758 | =item * |
| 1759 | |
| 1760 | Using caseless (C</i>) regular expression matching. |
| 1761 | |
| 1762 | Starting in Perl 5.14.0, regular expressions compiled within |
| 1763 | the scope of C<unicode_strings> use Unicode rules |
| 1764 | even when executed or compiled into larger |
| 1765 | regular expressions outside the scope. |
| 1766 | |
| 1767 | =item * |
| 1768 | |
| 1769 | Matching any of several properties in regular expressions. |
| 1770 | |
| 1771 | These properties are C<\b> (without braces), C<\B> (without braces), |
| 1772 | C<\s>, C<\S>, C<\w>, C<\W>, and all the Posix character classes |
| 1773 | I<except> C<[[:ascii:]]>. |
| 1774 | |
| 1775 | Starting in Perl 5.14.0, regular expressions compiled within |
| 1776 | the scope of C<unicode_strings> use Unicode rules |
| 1777 | even when executed or compiled into larger |
| 1778 | regular expressions outside the scope. |
| 1779 | |
| 1780 | =item * |
| 1781 | |
| 1782 | In C<quotemeta> or its inline equivalent C<\Q>. |
| 1783 | |
| 1784 | Starting in Perl 5.16.0, consistent quoting rules are used within the |
| 1785 | scope of C<unicode_strings>, as described in L<perlfunc/quotemeta>. |
| 1786 | Prior to that, or outside its scope, no code points above 127 are quoted |
| 1787 | in UTF-8 encoded strings, but in byte encoded strings, code points |
| 1788 | between 128-255 are always quoted. |
| 1789 | |
| 1790 | =back |
| 1791 | |
| 1792 | You can see from the above that the effect of C<unicode_strings> |
| 1793 | increased over several Perl releases. (And Perl's support for Unicode |
| 1794 | continues to improve; it's best to use the latest available release in |
| 1795 | order to get the most complete and accurate results possible.) Note that |
| 1796 | C<unicode_strings> is automatically chosen if you S<C<use 5.012>> or |
| 1797 | higher. |
| 1798 | |
| 1799 | For Perls earlier than those described above, or when a string is passed |
| 1800 | to a function outside the scope of C<unicode_strings>, see the next section. |
| 1801 | |
| 1802 | =head2 Forcing Unicode in Perl (Or Unforcing Unicode in Perl) |
| 1803 | |
| 1804 | Sometimes (see L</"When Unicode Does Not Happen"> or L</The "Unicode Bug">) |
| 1805 | there are situations where you simply need to force a byte |
| 1806 | string into UTF-8, or vice versa. The standard module L<Encode> can be |
| 1807 | used for this, or the low-level calls |
| 1808 | L<C<utf8::upgrade($bytestring)>|utf8/Utility functions> and |
| 1809 | L<C<utf8::downgrade($utf8string[, FAIL_OK])>|utf8/Utility functions>. |
| 1810 | |
| 1811 | Note that C<utf8::downgrade()> can fail if the string contains characters |
| 1812 | that don't fit into a byte. |
| 1813 | |
| 1814 | Calling either function on a string that already is in the desired state is a |
| 1815 | no-op. |
| 1816 | |
| 1817 | L</ASCII Rules versus Unicode Rules> gives all the ways that a string is |
| 1818 | made to use Unicode rules. |
| 1819 | |
| 1820 | =head2 Using Unicode in XS |
| 1821 | |
| 1822 | See L<perlguts/"Unicode Support"> for an introduction to Unicode at |
| 1823 | the XS level, and L<perlapi/Unicode Support> for the API details. |
| 1824 | |
| 1825 | =head2 Hacking Perl to work on earlier Unicode versions (for very serious hackers only) |
| 1826 | |
| 1827 | Perl by default comes with the latest supported Unicode version built-in, but |
| 1828 | the goal is to allow you to change to use any earlier one. In Perls |
| 1829 | v5.20 and v5.22, however, the earliest usable version is Unicode 5.1. |
| 1830 | Perl v5.18 is able to handle all earlier versions. |
| 1831 | |
| 1832 | Download the files in the desired version of Unicode from the Unicode web |
| 1833 | site L<http://www.unicode.org>). These should replace the existing files in |
| 1834 | F<lib/unicore> in the Perl source tree. Follow the instructions in |
| 1835 | F<README.perl> in that directory to change some of their names, and then build |
| 1836 | perl (see L<INSTALL>). |
| 1837 | |
| 1838 | =head2 Porting code from perl-5.6.X |
| 1839 | |
| 1840 | Perls starting in 5.8 have a different Unicode model from 5.6. In 5.6 the |
| 1841 | programmer was required to use the C<utf8> pragma to declare that a |
| 1842 | given scope expected to deal with Unicode data and had to make sure that |
| 1843 | only Unicode data were reaching that scope. If you have code that is |
| 1844 | working with 5.6, you will need some of the following adjustments to |
| 1845 | your code. The examples are written such that the code will continue to |
| 1846 | work under 5.6, so you should be safe to try them out. |
| 1847 | |
| 1848 | =over 3 |
| 1849 | |
| 1850 | =item * |
| 1851 | |
| 1852 | A filehandle that should read or write UTF-8 |
| 1853 | |
| 1854 | if ($] > 5.008) { |
| 1855 | binmode $fh, ":encoding(utf8)"; |
| 1856 | } |
| 1857 | |
| 1858 | =item * |
| 1859 | |
| 1860 | A scalar that is going to be passed to some extension |
| 1861 | |
| 1862 | Be it C<Compress::Zlib>, C<Apache::Request> or any extension that has no |
| 1863 | mention of Unicode in the manpage, you need to make sure that the |
| 1864 | UTF8 flag is stripped off. Note that at the time of this writing |
| 1865 | (January 2012) the mentioned modules are not UTF-8-aware. Please |
| 1866 | check the documentation to verify if this is still true. |
| 1867 | |
| 1868 | if ($] > 5.008) { |
| 1869 | require Encode; |
| 1870 | $val = Encode::encode_utf8($val); # make octets |
| 1871 | } |
| 1872 | |
| 1873 | =item * |
| 1874 | |
| 1875 | A scalar we got back from an extension |
| 1876 | |
| 1877 | If you believe the scalar comes back as UTF-8, you will most likely |
| 1878 | want the UTF8 flag restored: |
| 1879 | |
| 1880 | if ($] > 5.008) { |
| 1881 | require Encode; |
| 1882 | $val = Encode::decode_utf8($val); |
| 1883 | } |
| 1884 | |
| 1885 | =item * |
| 1886 | |
| 1887 | Same thing, if you are really sure it is UTF-8 |
| 1888 | |
| 1889 | if ($] > 5.008) { |
| 1890 | require Encode; |
| 1891 | Encode::_utf8_on($val); |
| 1892 | } |
| 1893 | |
| 1894 | =item * |
| 1895 | |
| 1896 | A wrapper for L<DBI> C<fetchrow_array> and C<fetchrow_hashref> |
| 1897 | |
| 1898 | When the database contains only UTF-8, a wrapper function or method is |
| 1899 | a convenient way to replace all your C<fetchrow_array> and |
| 1900 | C<fetchrow_hashref> calls. A wrapper function will also make it easier to |
| 1901 | adapt to future enhancements in your database driver. Note that at the |
| 1902 | time of this writing (January 2012), the DBI has no standardized way |
| 1903 | to deal with UTF-8 data. Please check the L<DBI documentation|DBI> to verify if |
| 1904 | that is still true. |
| 1905 | |
| 1906 | sub fetchrow { |
| 1907 | # $what is one of fetchrow_{array,hashref} |
| 1908 | my($self, $sth, $what) = @_; |
| 1909 | if ($] < 5.008) { |
| 1910 | return $sth->$what; |
| 1911 | } else { |
| 1912 | require Encode; |
| 1913 | if (wantarray) { |
| 1914 | my @arr = $sth->$what; |
| 1915 | for (@arr) { |
| 1916 | defined && /[^\000-\177]/ && Encode::_utf8_on($_); |
| 1917 | } |
| 1918 | return @arr; |
| 1919 | } else { |
| 1920 | my $ret = $sth->$what; |
| 1921 | if (ref $ret) { |
| 1922 | for my $k (keys %$ret) { |
| 1923 | defined |
| 1924 | && /[^\000-\177]/ |
| 1925 | && Encode::_utf8_on($_) for $ret->{$k}; |
| 1926 | } |
| 1927 | return $ret; |
| 1928 | } else { |
| 1929 | defined && /[^\000-\177]/ && Encode::_utf8_on($_) for $ret; |
| 1930 | return $ret; |
| 1931 | } |
| 1932 | } |
| 1933 | } |
| 1934 | } |
| 1935 | |
| 1936 | |
| 1937 | =item * |
| 1938 | |
| 1939 | A large scalar that you know can only contain ASCII |
| 1940 | |
| 1941 | Scalars that contain only ASCII and are marked as UTF-8 are sometimes |
| 1942 | a drag to your program. If you recognize such a situation, just remove |
| 1943 | the UTF8 flag: |
| 1944 | |
| 1945 | utf8::downgrade($val) if $] > 5.008; |
| 1946 | |
| 1947 | =back |
| 1948 | |
| 1949 | =head1 BUGS |
| 1950 | |
| 1951 | See also L</The "Unicode Bug"> above. |
| 1952 | |
| 1953 | =head2 Interaction with Extensions |
| 1954 | |
| 1955 | When Perl exchanges data with an extension, the extension should be |
| 1956 | able to understand the UTF8 flag and act accordingly. If the |
| 1957 | extension doesn't recognize that flag, it's likely that the extension |
| 1958 | will return incorrectly-flagged data. |
| 1959 | |
| 1960 | So if you're working with Unicode data, consult the documentation of |
| 1961 | every module you're using if there are any issues with Unicode data |
| 1962 | exchange. If the documentation does not talk about Unicode at all, |
| 1963 | suspect the worst and probably look at the source to learn how the |
| 1964 | module is implemented. Modules written completely in Perl shouldn't |
| 1965 | cause problems. Modules that directly or indirectly access code written |
| 1966 | in other programming languages are at risk. |
| 1967 | |
| 1968 | For affected functions, the simple strategy to avoid data corruption is |
| 1969 | to always make the encoding of the exchanged data explicit. Choose an |
| 1970 | encoding that you know the extension can handle. Convert arguments passed |
| 1971 | to the extensions to that encoding and convert results back from that |
| 1972 | encoding. Write wrapper functions that do the conversions for you, so |
| 1973 | you can later change the functions when the extension catches up. |
| 1974 | |
| 1975 | To provide an example, let's say the popular C<Foo::Bar::escape_html> |
| 1976 | function doesn't deal with Unicode data yet. The wrapper function |
| 1977 | would convert the argument to raw UTF-8 and convert the result back to |
| 1978 | Perl's internal representation like so: |
| 1979 | |
| 1980 | sub my_escape_html ($) { |
| 1981 | my($what) = shift; |
| 1982 | return unless defined $what; |
| 1983 | Encode::decode_utf8(Foo::Bar::escape_html( |
| 1984 | Encode::encode_utf8($what))); |
| 1985 | } |
| 1986 | |
| 1987 | Sometimes, when the extension does not convert data but just stores |
| 1988 | and retrieves it, you will be able to use the otherwise |
| 1989 | dangerous L<C<Encode::_utf8_on()>|Encode/_utf8_on> function. Let's say |
| 1990 | the popular C<Foo::Bar> extension, written in C, provides a C<param> |
| 1991 | method that lets you store and retrieve data according to these prototypes: |
| 1992 | |
| 1993 | $self->param($name, $value); # set a scalar |
| 1994 | $value = $self->param($name); # retrieve a scalar |
| 1995 | |
| 1996 | If it does not yet provide support for any encoding, one could write a |
| 1997 | derived class with such a C<param> method: |
| 1998 | |
| 1999 | sub param { |
| 2000 | my($self,$name,$value) = @_; |
| 2001 | utf8::upgrade($name); # make sure it is UTF-8 encoded |
| 2002 | if (defined $value) { |
| 2003 | utf8::upgrade($value); # make sure it is UTF-8 encoded |
| 2004 | return $self->SUPER::param($name,$value); |
| 2005 | } else { |
| 2006 | my $ret = $self->SUPER::param($name); |
| 2007 | Encode::_utf8_on($ret); # we know, it is UTF-8 encoded |
| 2008 | return $ret; |
| 2009 | } |
| 2010 | } |
| 2011 | |
| 2012 | Some extensions provide filters on data entry/exit points, such as |
| 2013 | C<DB_File::filter_store_key> and family. Look out for such filters in |
| 2014 | the documentation of your extensions; they can make the transition to |
| 2015 | Unicode data much easier. |
| 2016 | |
| 2017 | =head2 Speed |
| 2018 | |
| 2019 | Some functions are slower when working on UTF-8 encoded strings than |
| 2020 | on byte encoded strings. All functions that need to hop over |
| 2021 | characters such as C<length()>, C<substr()> or C<index()>, or matching |
| 2022 | regular expressions can work B<much> faster when the underlying data are |
| 2023 | byte-encoded. |
| 2024 | |
| 2025 | In Perl 5.8.0 the slowness was often quite spectacular; in Perl 5.8.1 |
| 2026 | a caching scheme was introduced which improved the situation. In general, |
| 2027 | operations with UTF-8 encoded strings are still slower. As an example, |
| 2028 | the Unicode properties (character classes) like C<\p{Nd}> are known to |
| 2029 | be quite a bit slower (5-20 times) than their simpler counterparts |
| 2030 | like C<[0-9]> (then again, there are hundreds of Unicode characters matching |
| 2031 | C<Nd> compared with the 10 ASCII characters matching C<[0-9]>). |
| 2032 | |
| 2033 | =head1 SEE ALSO |
| 2034 | |
| 2035 | L<perlunitut>, L<perluniintro>, L<perluniprops>, L<Encode>, L<open>, L<utf8>, L<bytes>, |
| 2036 | L<perlretut>, L<perlvar/"${^UNICODE}">, |
| 2037 | L<http://www.unicode.org/reports/tr44>). |
| 2038 | |
| 2039 | =cut |