3 perlunicode - Unicode support in Perl
7 =head2 Important Caveats
9 Unicode support is an extensive requirement. While Perl does not
10 implement the Unicode standard or the accompanying technical reports
11 from cover to cover, Perl does support many Unicode features.
13 People who want to learn to use Unicode in Perl, should probably read
14 the L<Perl Unicode tutorial, perlunitut|perlunitut> and
15 L<perluniintro>, before reading
16 this reference document.
18 Also, the use of Unicode may present security issues that aren't obvious.
19 Read L<Unicode Security Considerations|http://www.unicode.org/reports/tr36>.
23 =item Safest if you "use feature 'unicode_strings'"
25 In order to preserve backward compatibility, Perl does not turn
26 on full internal Unicode support unless the pragma
27 C<use feature 'unicode_strings'> is specified. (This is automatically
28 selected if you use C<use 5.012> or higher.) Failure to do this can
29 trigger unexpected surprises. See L</The "Unicode Bug"> below.
31 This pragma doesn't affect I/O. Nor does it change the internal
32 representation of strings, only their interpretation. There are still
33 several places where Unicode isn't fully supported, such as in
36 =item Input and Output Layers
38 Perl knows when a filehandle uses Perl's internal Unicode encodings
39 (UTF-8, or UTF-EBCDIC if in EBCDIC) if the filehandle is opened with
40 the ":encoding(utf8)" layer. Other encodings can be converted to Perl's
41 encoding on input or from Perl's encoding on output by use of the
42 ":encoding(...)" layer. See L<open>.
44 To indicate that Perl source itself is in UTF-8, use C<use utf8;>.
46 =item C<use utf8> still needed to enable UTF-8/UTF-EBCDIC in scripts
48 As a compatibility measure, the C<use utf8> pragma must be explicitly
49 included to enable recognition of UTF-8 in the Perl scripts themselves
50 (in string or regular expression literals, or in identifier names) on
51 ASCII-based machines or to recognize UTF-EBCDIC on EBCDIC-based
52 machines. B<These are the only times when an explicit C<use utf8>
53 is needed.> See L<utf8>.
55 =item BOM-marked scripts and UTF-16 scripts autodetected
57 If a Perl script begins marked with the Unicode BOM (UTF-16LE, UTF16-BE,
58 or UTF-8), or if the script looks like non-BOM-marked UTF-16 of either
59 endianness, Perl will correctly read in the script as Unicode.
60 (BOMless UTF-8 cannot be effectively recognized or differentiated from
61 ISO 8859-1 or other eight-bit encodings.)
63 =item C<use encoding> needed to upgrade non-Latin-1 byte strings
65 By default, there is a fundamental asymmetry in Perl's Unicode model:
66 implicit upgrading from byte strings to Unicode strings assumes that
67 they were encoded in I<ISO 8859-1 (Latin-1)>, but Unicode strings are
68 downgraded with UTF-8 encoding. This happens because the first 256
69 codepoints in Unicode happens to agree with Latin-1.
71 See L</"Byte and Character Semantics"> for more details.
75 =head2 Byte and Character Semantics
77 Perl uses logically-wide characters to represent strings internally.
79 Starting in Perl 5.14, Perl-level operations work with
80 characters rather than bytes within the scope of a
81 C<L<use feature 'unicode_strings'|feature>> (or equivalently
82 C<use 5.012> or higher). (This is not true if bytes have been
83 explicitly requested by C<L<use bytes|bytes>>, nor necessarily true
84 for interactions with the platform's operating system.)
86 For earlier Perls, and when C<unicode_strings> is not in effect, Perl
87 provides a fairly safe environment that can handle both types of
88 semantics in programs. For operations where Perl can unambiguously
89 decide that the input data are characters, Perl switches to character
90 semantics. For operations where this determination cannot be made
91 without additional information from the user, Perl decides in favor of
92 compatibility and chooses to use byte semantics.
94 When C<use locale> (but not C<use locale ':not_characters'>) is in
95 effect, Perl uses the semantics associated with the current locale.
96 (C<use locale> overrides C<use feature 'unicode_strings'> in the same scope;
97 while C<use locale ':not_characters'> effectively also selects
98 C<use feature 'unicode_strings'> in its scope; see L<perllocale>.)
99 Otherwise, Perl uses the platform's native
100 byte semantics for characters whose code points are less than 256, and
101 Unicode semantics for those greater than 255. On EBCDIC platforms, this
102 is almost seamless, as the EBCDIC code pages that Perl handles are
103 equivalent to Unicode's first 256 code points. (The exception is that
104 EBCDIC regular expression case-insensitive matching rules are not as
105 as robust as Unicode's.) But on ASCII platforms, Perl uses US-ASCII
106 (or Basic Latin in Unicode terminology) byte semantics, meaning that characters
107 whose ordinal numbers are in the range 128 - 255 are undefined except for their
108 ordinal numbers. This means that none have case (upper and lower), nor are any
109 a member of character classes, like C<[:alpha:]> or C<\w>. (But all do belong
110 to the C<\W> class or the Perl regular expression extension C<[:^alpha:]>.)
112 This behavior preserves compatibility with earlier versions of Perl,
113 which allowed byte semantics in Perl operations only if
114 none of the program's inputs were marked as being a source of Unicode
115 character data. Such data may come from filehandles, from calls to
116 external programs, from information provided by the system (such as %ENV),
117 or from literals and constants in the source text.
119 The C<utf8> pragma is primarily a compatibility device that enables
120 recognition of UTF-(8|EBCDIC) in literals encountered by the parser.
121 Note that this pragma is only required while Perl defaults to byte
122 semantics; when character semantics become the default, this pragma
123 may become a no-op. See L<utf8>.
125 If strings operating under byte semantics and strings with Unicode
126 character data are concatenated, the new string will have
127 character semantics. This can cause surprises: See L</BUGS>, below.
128 You can choose to be warned when this happens. See L<encoding::warnings>.
130 Under character semantics, many operations that formerly operated on
131 bytes now operate on characters. A character in Perl is
132 logically just a number ranging from 0 to 2**31 or so. Larger
133 characters may encode into longer sequences of bytes internally, but
134 this internal detail is mostly hidden for Perl code.
135 See L<perluniintro> for more.
137 =head2 Effects of Character Semantics
139 Character semantics have the following effects:
145 Strings--including hash keys--and regular expression patterns may
146 contain characters that have an ordinal value larger than 255.
148 If you use a Unicode editor to edit your program, Unicode characters may
149 occur directly within the literal strings in UTF-8 encoding, or UTF-16.
150 (The former requires a BOM or C<use utf8>, the latter requires a BOM.)
152 Unicode characters can also be added to a string by using the C<\N{U+...}>
153 notation. The Unicode code for the desired character, in hexadecimal,
154 should be placed in the braces, after the C<U>. For instance, a smiley face is
157 Alternatively, you can use the C<\x{...}> notation for characters 0x100 and
158 above. For characters below 0x100 you may get byte semantics instead of
159 character semantics; see L</The "Unicode Bug">. On EBCDIC machines there is
160 the additional problem that the value for such characters gives the EBCDIC
161 character rather than the Unicode one, thus it is more portable to use
162 C<\N{U+...}> instead.
164 Additionally, you can use the C<\N{...}> notation and put the official
165 Unicode character name within the braces, such as
166 C<\N{WHITE SMILING FACE}>. This automatically loads the L<charnames>
167 module with the C<:full> and C<:short> options. If you prefer different
168 options for this module, you can instead, before the C<\N{...}>,
169 explicitly load it with your desired options; for example,
171 use charnames ':loose';
175 If an appropriate L<encoding> is specified, identifiers within the
176 Perl script may contain Unicode alphanumeric characters, including
177 ideographs. Perl does not currently attempt to canonicalize variable
182 Regular expressions match characters instead of bytes. "." matches
183 a character instead of a byte.
187 Bracketed character classes in regular expressions match characters instead of
188 bytes and match against the character properties specified in the
189 Unicode properties database. C<\w> can be used to match a Japanese
190 ideograph, for instance.
194 Named Unicode properties, scripts, and block ranges may be used (like bracketed
195 character classes) by using the C<\p{}> "matches property" construct and
196 the C<\P{}> negation, "doesn't match property".
197 See L</"Unicode Character Properties"> for more details.
199 You can define your own character properties and use them
200 in the regular expression with the C<\p{}> or C<\P{}> construct.
201 See L</"User-Defined Character Properties"> for more details.
205 The special pattern C<\X> matches a logical character, an "extended grapheme
206 cluster" in Standardese. In Unicode what appears to the user to be a single
207 character, for example an accented C<G>, may in fact be composed of a sequence
208 of characters, in this case a C<G> followed by an accent character. C<\X>
209 will match the entire sequence.
213 The C<tr///> operator translates characters instead of bytes. Note
214 that the C<tr///CU> functionality has been removed. For similar
215 functionality see pack('U0', ...) and pack('C0', ...).
219 Case translation operators use the Unicode case translation tables
220 when character input is provided. Note that C<uc()>, or C<\U> in
221 interpolated strings, translates to uppercase, while C<ucfirst>,
222 or C<\u> in interpolated strings, translates to titlecase in languages
223 that make the distinction (which is equivalent to uppercase in languages
224 without the distinction).
228 Most operators that deal with positions or lengths in a string will
229 automatically switch to using character positions, including
230 C<chop()>, C<chomp()>, C<substr()>, C<pos()>, C<index()>, C<rindex()>,
231 C<sprintf()>, C<write()>, and C<length()>. An operator that
232 specifically does not switch is C<vec()>. Operators that really don't
233 care include operators that treat strings as a bucket of bits such as
234 C<sort()>, and operators dealing with filenames.
238 The C<pack()>/C<unpack()> letter C<C> does I<not> change, since it is often
239 used for byte-oriented formats. Again, think C<char> in the C language.
241 There is a new C<U> specifier that converts between Unicode characters
242 and code points. There is also a C<W> specifier that is the equivalent of
243 C<chr>/C<ord> and properly handles character values even if they are above 255.
247 The C<chr()> and C<ord()> functions work on characters, similar to
248 C<pack("W")> and C<unpack("W")>, I<not> C<pack("C")> and
249 C<unpack("C")>. C<pack("C")> and C<unpack("C")> are methods for
250 emulating byte-oriented C<chr()> and C<ord()> on Unicode strings.
251 While these methods reveal the internal encoding of Unicode strings,
252 that is not something one normally needs to care about at all.
256 The bit string operators, C<& | ^ ~>, can operate on character data.
257 However, for backward compatibility, such as when using bit string
258 operations when characters are all less than 256 in ordinal value, one
259 should not use C<~> (the bit complement) with characters of both
260 values less than 256 and values greater than 256. Most importantly,
261 DeMorgan's laws (C<~($x|$y) eq ~$x&~$y> and C<~($x&$y) eq ~$x|~$y>)
262 will not hold. The reason for this mathematical I<faux pas> is that
263 the complement cannot return B<both> the 8-bit (byte-wide) bit
264 complement B<and> the full character-wide bit complement.
268 There is a CPAN module, L<Unicode::Casing>, which allows you to define
269 your own mappings to be used in C<lc()>, C<lcfirst()>, C<uc()>,
270 C<ucfirst()>, and C<fc> (or their double-quoted string inlined
271 versions such as C<\U>).
272 (Prior to Perl 5.16, this functionality was partially provided
273 in the Perl core, but suffered from a number of insurmountable
274 drawbacks, so the CPAN module was written instead.)
282 And finally, C<scalar reverse()> reverses by character rather than by byte.
286 =head2 Unicode Character Properties
288 (The only time that Perl considers a sequence of individual code
289 points as a single logical character is in the C<\X> construct, already
290 mentioned above. Therefore "character" in this discussion means a single
293 Very nearly all Unicode character properties are accessible through
294 regular expressions by using the C<\p{}> "matches property" construct
295 and the C<\P{}> "doesn't match property" for its negation.
297 For instance, C<\p{Uppercase}> matches any single character with the Unicode
298 "Uppercase" property, while C<\p{L}> matches any character with a
299 General_Category of "L" (letter) property. Brackets are not
300 required for single letter property names, so C<\p{L}> is equivalent to C<\pL>.
302 More formally, C<\p{Uppercase}> matches any single character whose Unicode
303 Uppercase property value is True, and C<\P{Uppercase}> matches any character
304 whose Uppercase property value is False, and they could have been written as
305 C<\p{Uppercase=True}> and C<\p{Uppercase=False}>, respectively.
307 This formality is needed when properties are not binary; that is, if they can
308 take on more values than just True and False. For example, the Bidi_Class (see
309 L</"Bidirectional Character Types"> below), can take on several different
310 values, such as Left, Right, Whitespace, and others. To match these, one needs
311 to specify both the property name (Bidi_Class), AND the value being
313 (Left, Right, etc.). This is done, as in the examples above, by having the
314 two components separated by an equal sign (or interchangeably, a colon), like
315 C<\p{Bidi_Class: Left}>.
317 All Unicode-defined character properties may be written in these compound forms
318 of C<\p{property=value}> or C<\p{property:value}>, but Perl provides some
319 additional properties that are written only in the single form, as well as
320 single-form short-cuts for all binary properties and certain others described
321 below, in which you may omit the property name and the equals or colon
324 Most Unicode character properties have at least two synonyms (or aliases if you
325 prefer): a short one that is easier to type and a longer one that is more
326 descriptive and hence easier to understand. Thus the "L" and "Letter" properties
327 above are equivalent and can be used interchangeably. Likewise,
328 "Upper" is a synonym for "Uppercase", and we could have written
329 C<\p{Uppercase}> equivalently as C<\p{Upper}>. Also, there are typically
330 various synonyms for the values the property can be. For binary properties,
331 "True" has 3 synonyms: "T", "Yes", and "Y"; and "False has correspondingly "F",
332 "No", and "N". But be careful. A short form of a value for one property may
333 not mean the same thing as the same short form for another. Thus, for the
334 General_Category property, "L" means "Letter", but for the Bidi_Class property,
335 "L" means "Left". A complete list of properties and synonyms is in
338 Upper/lower case differences in property names and values are irrelevant;
339 thus C<\p{Upper}> means the same thing as C<\p{upper}> or even C<\p{UpPeR}>.
340 Similarly, you can add or subtract underscores anywhere in the middle of a
341 word, so that these are also equivalent to C<\p{U_p_p_e_r}>. And white space
342 is irrelevant adjacent to non-word characters, such as the braces and the equals
343 or colon separators, so C<\p{ Upper }> and C<\p{ Upper_case : Y }> are
344 equivalent to these as well. In fact, white space and even
345 hyphens can usually be added or deleted anywhere. So even C<\p{ Up-per case = Yes}> is
346 equivalent. All this is called "loose-matching" by Unicode. The few places
347 where stricter matching is used is in the middle of numbers, and in the Perl
348 extension properties that begin or end with an underscore. Stricter matching
349 cares about white space (except adjacent to non-word characters),
350 hyphens, and non-interior underscores.
352 You can also use negation in both C<\p{}> and C<\P{}> by introducing a caret
353 (^) between the first brace and the property name: C<\p{^Tamil}> is
354 equal to C<\P{Tamil}>.
356 Almost all properties are immune to case-insensitive matching. That is,
357 adding a C</i> regular expression modifier does not change what they
358 match. There are two sets that are affected.
362 and C<Titlecase_Letter>,
363 all of which match C<Cased_Letter> under C</i> matching.
364 And the second set is
368 all of which match C<Cased> under C</i> matching.
369 This set also includes its subsets C<PosixUpper> and C<PosixLower> both
370 of which under C</i> matching match C<PosixAlpha>.
371 (The difference between these sets is that some things, such as Roman
372 numerals, come in both upper and lower case so they are C<Cased>, but aren't considered
373 letters, so they aren't C<Cased_Letter>s.)
375 The result is undefined if you try to match a non-Unicode code point
376 (that is, one above 0x10FFFF) against a Unicode property. Currently, a
377 warning is raised, and the match will fail. In some cases, this is
378 counterintuitive, as both these fail:
380 chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Fails.
381 chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Fails!
383 =head3 B<General_Category>
385 Every Unicode character is assigned a general category, which is the "most
386 usual categorization of a character" (from
387 L<http://www.unicode.org/reports/tr44>).
389 The compound way of writing these is like C<\p{General_Category=Number}>
390 (short, C<\p{gc:n}>). But Perl furnishes shortcuts in which everything up
391 through the equal or colon separator is omitted. So you can instead just write
394 Here are the short and long forms of the General Category properties:
399 LC, L& Cased_Letter (that is: [\p{Ll}\p{Lu}\p{Lt}])
412 Nd Decimal_Number (also Digit)
416 P Punctuation (also Punct)
417 Pc Connector_Punctuation
421 Pi Initial_Punctuation
422 (may behave like Ps or Pe depending on usage)
424 (may behave like Ps or Pe depending on usage)
436 Zp Paragraph_Separator
439 Cc Control (also Cntrl)
445 Single-letter properties match all characters in any of the
446 two-letter sub-properties starting with the same letter.
447 C<LC> and C<L&> are special: both are aliases for the set consisting of everything matched by C<Ll>, C<Lu>, and C<Lt>.
449 =head3 B<Bidirectional Character Types>
451 Because scripts differ in their directionality (Hebrew and Arabic are
452 written right to left, for example) Unicode supplies these properties in
453 the Bidi_Class class:
458 LRE Left-to-Right Embedding
459 LRO Left-to-Right Override
462 RLE Right-to-Left Embedding
463 RLO Right-to-Left Override
464 PDF Pop Directional Format
466 ES European Separator
467 ET European Terminator
472 B Paragraph Separator
477 This property is always written in the compound form.
478 For example, C<\p{Bidi_Class:R}> matches characters that are normally
479 written right to left.
483 The world's languages are written in many different scripts. This sentence
484 (unless you're reading it in translation) is written in Latin, while Russian is
485 written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in
486 Hiragana or Katakana. There are many more.
488 The Unicode Script and Script_Extensions properties give what script a
489 given character is in. Either property can be specified with the
491 C<\p{Script=Hebrew}> (short: C<\p{sc=hebr}>), or
492 C<\p{Script_Extensions=Javanese}> (short: C<\p{scx=java}>).
493 In addition, Perl furnishes shortcuts for all
494 C<Script> property names. You can omit everything up through the equals
495 (or colon), and simply write C<\p{Latin}> or C<\P{Cyrillic}>.
496 (This is not true for C<Script_Extensions>, which is required to be
497 written in the compound form.)
499 The difference between these two properties involves characters that are
500 used in multiple scripts. For example the digits '0' through '9' are
501 used in many parts of the world. These are placed in a script named
502 C<Common>. Other characters are used in just a few scripts. For
503 example, the "KATAKANA-HIRAGANA DOUBLE HYPHEN" is used in both Japanese
504 scripts, Katakana and Hiragana, but nowhere else. The C<Script>
505 property places all characters that are used in multiple scripts in the
506 C<Common> script, while the C<Script_Extensions> property places those
507 that are used in only a few scripts into each of those scripts; while
508 still using C<Common> for those used in many scripts. Thus both these
511 "0" =~ /\p{sc=Common}/ # Matches
512 "0" =~ /\p{scx=Common}/ # Matches
514 and only the first of these match:
516 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Common} # Matches
517 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Common} # No match
519 And only the last two of these match:
521 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Hiragana} # No match
522 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Katakana} # No match
523 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Hiragana} # Matches
524 "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Katakana} # Matches
526 C<Script_Extensions> is thus an improved C<Script>, in which there are
527 fewer characters in the C<Common> script, and correspondingly more in
528 other scripts. It is new in Unicode version 6.0, and its data are likely
529 to change significantly in later releases, as things get sorted out.
531 (Actually, besides C<Common>, the C<Inherited> script, contains
532 characters that are used in multiple scripts. These are modifier
533 characters which modify other characters, and inherit the script value
534 of the controlling character. Some of these are used in many scripts,
535 and so go into C<Inherited> in both C<Script> and C<Script_Extensions>.
536 Others are used in just a few scripts, so are in C<Inherited> in
537 C<Script>, but not in C<Script_Extensions>.)
539 It is worth stressing that there are several different sets of digits in
540 Unicode that are equivalent to 0-9 and are matchable by C<\d> in a
541 regular expression. If they are used in a single language only, they
542 are in that language's C<Script> and C<Script_Extension>. If they are
543 used in more than one script, they will be in C<sc=Common>, but only
544 if they are used in many scripts should they be in C<scx=Common>.
546 A complete list of scripts and their shortcuts is in L<perluniprops>.
548 =head3 B<Use of "Is" Prefix>
550 For backward compatibility (with Perl 5.6), all properties mentioned
551 so far may have C<Is> or C<Is_> prepended to their name, so C<\P{Is_Lu}>, for
552 example, is equal to C<\P{Lu}>, and C<\p{IsScript:Arabic}> is equal to
557 In addition to B<scripts>, Unicode also defines B<blocks> of
558 characters. The difference between scripts and blocks is that the
559 concept of scripts is closer to natural languages, while the concept
560 of blocks is more of an artificial grouping based on groups of Unicode
561 characters with consecutive ordinal values. For example, the "Basic Latin"
562 block is all characters whose ordinals are between 0 and 127, inclusive; in
563 other words, the ASCII characters. The "Latin" script contains some letters
564 from this as well as several other blocks, like "Latin-1 Supplement",
565 "Latin Extended-A", etc., but it does not contain all the characters from
566 those blocks. It does not, for example, contain the digits 0-9, because
567 those digits are shared across many scripts, and hence are in the
570 For more about scripts versus blocks, see UAX#24 "Unicode Script Property":
571 L<http://www.unicode.org/reports/tr24>
573 The C<Script> or C<Script_Extensions> properties are likely to be the
574 ones you want to use when processing
575 natural language; the Block property may occasionally be useful in working
576 with the nuts and bolts of Unicode.
578 Block names are matched in the compound form, like C<\p{Block: Arrows}> or
579 C<\p{Blk=Hebrew}>. Unlike most other properties, only a few block names have a
580 Unicode-defined short name. But Perl does provide a (slight) shortcut: You
581 can say, for example C<\p{In_Arrows}> or C<\p{In_Hebrew}>. For backwards
582 compatibility, the C<In> prefix may be omitted if there is no naming conflict
583 with a script or any other property, and you can even use an C<Is> prefix
584 instead in those cases. But it is not a good idea to do this, for a couple
591 It is confusing. There are many naming conflicts, and you may forget some.
592 For example, C<\p{Hebrew}> means the I<script> Hebrew, and NOT the I<block>
593 Hebrew. But would you remember that 6 months from now?
597 It is unstable. A new version of Unicode may pre-empt the current meaning by
598 creating a property with the same name. There was a time in very early Unicode
599 releases when C<\p{Hebrew}> would have matched the I<block> Hebrew; now it
604 Some people prefer to always use C<\p{Block: foo}> and C<\p{Script: bar}>
605 instead of the shortcuts, whether for clarity, because they can't remember the
606 difference between 'In' and 'Is' anyway, or they aren't confident that those who
607 eventually will read their code will know that difference.
609 A complete list of blocks and their shortcuts is in L<perluniprops>.
611 =head3 B<Other Properties>
613 There are many more properties than the very basic ones described here.
614 A complete list is in L<perluniprops>.
616 Unicode defines all its properties in the compound form, so all single-form
617 properties are Perl extensions. Most of these are just synonyms for the
618 Unicode ones, but some are genuine extensions, including several that are in
619 the compound form. And quite a few of these are actually recommended by Unicode
620 (in L<http://www.unicode.org/reports/tr18>).
622 This section gives some details on all extensions that aren't just
623 synonyms for compound-form Unicode properties
624 (for those properties, you'll have to refer to the
625 L<Unicode Standard|http://www.unicode.org/reports/tr44>.
631 This matches any of the 1_114_112 Unicode code points. It is a synonym for
634 =item B<C<\p{Alnum}>>
636 This matches any C<\p{Alphabetic}> or C<\p{Decimal_Number}> character.
640 This matches any of the 1_114_112 Unicode code points. It is a synonym for
643 =item B<C<\p{ASCII}>>
645 This matches any of the 128 characters in the US-ASCII character set,
646 which is a subset of Unicode.
648 =item B<C<\p{Assigned}>>
650 This matches any assigned code point; that is, any code point whose general
651 category is not Unassigned (or equivalently, not Cn).
653 =item B<C<\p{Blank}>>
655 This is the same as C<\h> and C<\p{HorizSpace}>: A character that changes the
656 spacing horizontally.
658 =item B<C<\p{Decomposition_Type: Non_Canonical}>> (Short: C<\p{Dt=NonCanon}>)
660 Matches a character that has a non-canonical decomposition.
662 To understand the use of this rarely used property=value combination, it is
663 necessary to know some basics about decomposition.
664 Consider a character, say H. It could appear with various marks around it,
665 such as an acute accent, or a circumflex, or various hooks, circles, arrows,
666 I<etc.>, above, below, to one side or the other, etc. There are many
667 possibilities among the world's languages. The number of combinations is
668 astronomical, and if there were a character for each combination, it would
669 soon exhaust Unicode's more than a million possible characters. So Unicode
670 took a different approach: there is a character for the base H, and a
671 character for each of the possible marks, and these can be variously combined
672 to get a final logical character. So a logical character--what appears to be a
673 single character--can be a sequence of more than one individual characters.
674 This is called an "extended grapheme cluster"; Perl furnishes the C<\X>
675 regular expression construct to match such sequences.
677 But Unicode's intent is to unify the existing character set standards and
678 practices, and several pre-existing standards have single characters that
679 mean the same thing as some of these combinations. An example is ISO-8859-1,
680 which has quite a few of these in the Latin-1 range, an example being "LATIN
681 CAPITAL LETTER E WITH ACUTE". Because this character was in this pre-existing
682 standard, Unicode added it to its repertoire. But this character is considered
683 by Unicode to be equivalent to the sequence consisting of the character
684 "LATIN CAPITAL LETTER E" followed by the character "COMBINING ACUTE ACCENT".
686 "LATIN CAPITAL LETTER E WITH ACUTE" is called a "pre-composed" character, and
687 its equivalence with the sequence is called canonical equivalence. All
688 pre-composed characters are said to have a decomposition (into the equivalent
689 sequence), and the decomposition type is also called canonical.
691 However, many more characters have a different type of decomposition, a
692 "compatible" or "non-canonical" decomposition. The sequences that form these
693 decompositions are not considered canonically equivalent to the pre-composed
694 character. An example, again in the Latin-1 range, is the "SUPERSCRIPT ONE".
695 It is somewhat like a regular digit 1, but not exactly; its decomposition
696 into the digit 1 is called a "compatible" decomposition, specifically a
697 "super" decomposition. There are several such compatibility
698 decompositions (see L<http://www.unicode.org/reports/tr44>), including one
699 called "compat", which means some miscellaneous type of decomposition
700 that doesn't fit into the decomposition categories that Unicode has chosen.
702 Note that most Unicode characters don't have a decomposition, so their
703 decomposition type is "None".
705 For your convenience, Perl has added the C<Non_Canonical> decomposition
706 type to mean any of the several compatibility decompositions.
708 =item B<C<\p{Graph}>>
710 Matches any character that is graphic. Theoretically, this means a character
711 that on a printer would cause ink to be used.
713 =item B<C<\p{HorizSpace}>>
715 This is the same as C<\h> and C<\p{Blank}>: a character that changes the
716 spacing horizontally.
720 This is a synonym for C<\p{Present_In=*}>
722 =item B<C<\p{PerlSpace}>>
724 This is the same as C<\s>, restricted to ASCII, namely C<S<[ \f\n\r\t]>>.
726 Mnemonic: Perl's (original) space
728 =item B<C<\p{PerlWord}>>
730 This is the same as C<\w>, restricted to ASCII, namely C<[A-Za-z0-9_]>
732 Mnemonic: Perl's (original) word.
734 =item B<C<\p{Posix...}>>
736 There are several of these, which are equivalents using the C<\p>
737 notation for Posix classes and are described in
738 L<perlrecharclass/POSIX Character Classes>.
740 =item B<C<\p{Present_In: *}>> (Short: C<\p{In=*}>)
742 This property is used when you need to know in what Unicode version(s) a
745 The "*" above stands for some two digit Unicode version number, such as
746 C<1.1> or C<4.0>; or the "*" can also be C<Unassigned>. This property will
747 match the code points whose final disposition has been settled as of the
748 Unicode release given by the version number; C<\p{Present_In: Unassigned}>
749 will match those code points whose meaning has yet to be assigned.
751 For example, C<U+0041> "LATIN CAPITAL LETTER A" was present in the very first
752 Unicode release available, which is C<1.1>, so this property is true for all
753 valid "*" versions. On the other hand, C<U+1EFF> was not assigned until version
754 5.1 when it became "LATIN SMALL LETTER Y WITH LOOP", so the only "*" that
755 would match it are 5.1, 5.2, and later.
757 Unicode furnishes the C<Age> property from which this is derived. The problem
758 with Age is that a strict interpretation of it (which Perl takes) has it
759 matching the precise release a code point's meaning is introduced in. Thus
760 C<U+0041> would match only 1.1; and C<U+1EFF> only 5.1. This is not usually what
763 Some non-Perl implementations of the Age property may change its meaning to be
764 the same as the Perl Present_In property; just be aware of that.
766 Another confusion with both these properties is that the definition is not
767 that the code point has been I<assigned>, but that the meaning of the code point
768 has been I<determined>. This is because 66 code points will always be
769 unassigned, and so the Age for them is the Unicode version in which the decision
770 to make them so was made. For example, C<U+FDD0> is to be permanently
771 unassigned to a character, and the decision to do that was made in version 3.1,
772 so C<\p{Age=3.1}> matches this character, as also does C<\p{Present_In: 3.1}> and up.
774 =item B<C<\p{Print}>>
776 This matches any character that is graphical or blank, except controls.
778 =item B<C<\p{SpacePerl}>>
780 This is the same as C<\s>, including beyond ASCII.
782 Mnemonic: Space, as modified by Perl. (It doesn't include the vertical tab
783 which both the Posix standard and Unicode consider white space.)
785 =item B<C<\p{Title}>> and B<C<\p{Titlecase}>>
787 Under case-sensitive matching, these both match the same code points as
788 C<\p{General Category=Titlecase_Letter}> (C<\p{gc=lt}>). The difference
789 is that under C</i> caseless matching, these match the same as
790 C<\p{Cased}>, whereas C<\p{gc=lt}> matches C<\p{Cased_Letter>).
792 =item B<C<\p{VertSpace}>>
794 This is the same as C<\v>: A character that changes the spacing vertically.
798 This is the same as C<\w>, including over 100_000 characters beyond ASCII.
800 =item B<C<\p{XPosix...}>>
802 There are several of these, which are the standard Posix classes
803 extended to the full Unicode range. They are described in
804 L<perlrecharclass/POSIX Character Classes>.
808 =head2 User-Defined Character Properties
810 You can define your own binary character properties by defining subroutines
811 whose names begin with "In" or "Is". The subroutines can be defined in any
812 package. The user-defined properties can be used in the regular expression
813 C<\p> and C<\P> constructs; if you are using a user-defined property from a
814 package other than the one you are in, you must specify its package in the
815 C<\p> or C<\P> construct.
817 # assuming property Is_Foreign defined in Lang::
818 package main; # property package name required
819 if ($txt =~ /\p{Lang::IsForeign}+/) { ... }
821 package Lang; # property package name not required
822 if ($txt =~ /\p{IsForeign}+/) { ... }
825 Note that the effect is compile-time and immutable once defined.
826 However, the subroutines are passed a single parameter, which is 0 if
827 case-sensitive matching is in effect and non-zero if caseless matching
828 is in effect. The subroutine may return different values depending on
829 the value of the flag, and one set of values will immutably be in effect
830 for all case-sensitive matches, and the other set for all case-insensitive
833 Note that if the regular expression is tainted, then Perl will die rather
834 than calling the subroutine, where the name of the subroutine is
835 determined by the tainted data.
837 The subroutines must return a specially-formatted string, with one
838 or more newline-separated lines. Each line must be one of the following:
844 A single hexadecimal number denoting a Unicode code point to include.
848 Two hexadecimal numbers separated by horizontal whitespace (space or
849 tabular characters) denoting a range of Unicode code points to include.
853 Something to include, prefixed by "+": a built-in character
854 property (prefixed by "utf8::") or a fully qualified (including package
855 name) user-defined character property,
856 to represent all the characters in that property; two hexadecimal code
857 points for a range; or a single hexadecimal code point.
861 Something to exclude, prefixed by "-": an existing character
862 property (prefixed by "utf8::") or a fully qualified (including package
863 name) user-defined character property,
864 to represent all the characters in that property; two hexadecimal code
865 points for a range; or a single hexadecimal code point.
869 Something to negate, prefixed "!": an existing character
870 property (prefixed by "utf8::") or a fully qualified (including package
871 name) user-defined character property,
872 to represent all the characters in that property; two hexadecimal code
873 points for a range; or a single hexadecimal code point.
877 Something to intersect with, prefixed by "&": an existing character
878 property (prefixed by "utf8::") or a fully qualified (including package
879 name) user-defined character property,
880 for all the characters except the characters in the property; two
881 hexadecimal code points for a range; or a single hexadecimal code point.
885 For example, to define a property that covers both the Japanese
886 syllabaries (hiragana and katakana), you can define
895 Imagine that the here-doc end marker is at the beginning of the line.
896 Now you can use C<\p{InKana}> and C<\P{InKana}>.
898 You could also have used the existing block property names:
907 Suppose you wanted to match only the allocated characters,
908 not the raw block ranges: in other words, you want to remove
919 The negation is useful for defining (surprise!) negated classes.
929 This will match all non-Unicode code points, since every one of them is
930 not in Kana. You can use intersection to exclude these, if desired, as
931 this modified example shows:
942 C<&utf8::Any> must be the last line in the definition.
944 Intersection is used generally for getting the common characters matched
945 by two (or more) classes. It's important to remember not to use "&" for
946 the first set; that would be intersecting with nothing, resulting in an
949 (Note that official Unicode properties differ from these in that they
950 automatically exclude non-Unicode code points and a warning is raised if
951 a match is attempted on one of those.)
953 =head2 User-Defined Case Mappings (for serious hackers only)
955 B<This feature has been removed as of Perl 5.16.>
956 The CPAN module L<Unicode::Casing> provides better functionality without
957 the drawbacks that this feature had. If you are using a Perl earlier
958 than 5.16, this feature was most fully documented in the 5.14 version of
960 L<http://perldoc.perl.org/5.14.0/perlunicode.html#User-Defined-Case-Mappings-%28for-serious-hackers-only%29>
962 =head2 Character Encodings for Input and Output
966 =head2 Unicode Regular Expression Support Level
968 The following list of Unicode supported features for regular expressions describes
969 all features currently directly supported by core Perl. The references to "Level N"
970 and the section numbers refer to the Unicode Technical Standard #18,
971 "Unicode Regular Expressions", version 13, from August 2008.
977 Level 1 - Basic Unicode Support
979 RL1.1 Hex Notation - done [1]
980 RL1.2 Properties - done [2][3]
981 RL1.2a Compatibility Properties - done [4]
982 RL1.3 Subtraction and Intersection - MISSING [5]
983 RL1.4 Simple Word Boundaries - done [6]
984 RL1.5 Simple Loose Matches - done [7]
985 RL1.6 Line Boundaries - MISSING [8][9]
986 RL1.7 Supplementary Code Points - done [10]
990 [3] supports not only minimal list, but all Unicode character
991 properties (see Unicode Character Properties above)
992 [4] \d \D \s \S \w \W \X [:prop:] [:^prop:]
993 [5] can use regular expression look-ahead [a] or
994 user-defined character properties [b] to emulate set
997 [7] note that Perl does Full case-folding in matching (but with
998 bugs), not Simple: for example U+1F88 is equivalent to
999 U+1F00 U+03B9, instead of just U+1F80. This difference
1000 matters mainly for certain Greek capital letters with certain
1001 modifiers: the Full case-folding decomposes the letter,
1002 while the Simple case-folding would map it to a single
1004 [8] should do ^ and $ also on U+000B (\v in C), FF (\f), CR
1005 (\r), CRLF (\r\n), NEL (U+0085), LS (U+2028), and PS
1006 (U+2029); should also affect <>, $., and script line
1007 numbers; should not split lines within CRLF [c] (i.e. there
1008 is no empty line between \r and \n)
1009 [9] Linebreaking conformant with UAX#14 "Unicode Line Breaking
1010 Algorithm" is available through the Unicode::LineBreaking
1012 [10] UTF-8/UTF-EBDDIC used in Perl allows not only U+10000 to
1013 U+10FFFF but also beyond U+10FFFF
1015 [a] You can mimic class subtraction using lookahead.
1016 For example, what UTS#18 might write as
1018 [{Greek}-[{UNASSIGNED}]]
1020 in Perl can be written as:
1022 (?!\p{Unassigned})\p{InGreekAndCoptic}
1023 (?=\p{Assigned})\p{InGreekAndCoptic}
1025 But in this particular example, you probably really want
1029 which will match assigned characters known to be part of the Greek script.
1031 Also see the L<Unicode::Regex::Set> module; it does implement the full
1032 UTS#18 grouping, intersection, union, and removal (subtraction) syntax.
1034 [b] '+' for union, '-' for removal (set-difference), '&' for intersection
1035 (see L</"User-Defined Character Properties">)
1037 [c] Try the C<:crlf> layer (see L<PerlIO>).
1041 Level 2 - Extended Unicode Support
1043 RL2.1 Canonical Equivalents - MISSING [10][11]
1044 RL2.2 Default Grapheme Clusters - MISSING [12]
1045 RL2.3 Default Word Boundaries - MISSING [14]
1046 RL2.4 Default Loose Matches - MISSING [15]
1047 RL2.5 Name Properties - DONE
1048 RL2.6 Wildcard Properties - MISSING
1050 [10] see UAX#15 "Unicode Normalization Forms"
1051 [11] have Unicode::Normalize but not integrated to regexes
1052 [12] have \X but we don't have a "Grapheme Cluster Mode"
1053 [14] see UAX#29, Word Boundaries
1054 [15] This is covered in Chapter 3.13 (in Unicode 6.0)
1058 Level 3 - Tailored Support
1060 RL3.1 Tailored Punctuation - MISSING
1061 RL3.2 Tailored Grapheme Clusters - MISSING [17][18]
1062 RL3.3 Tailored Word Boundaries - MISSING
1063 RL3.4 Tailored Loose Matches - MISSING
1064 RL3.5 Tailored Ranges - MISSING
1065 RL3.6 Context Matching - MISSING [19]
1066 RL3.7 Incremental Matches - MISSING
1067 ( RL3.8 Unicode Set Sharing )
1068 RL3.9 Possible Match Sets - MISSING
1069 RL3.10 Folded Matching - MISSING [20]
1070 RL3.11 Submatchers - MISSING
1072 [17] see UAX#10 "Unicode Collation Algorithms"
1073 [18] have Unicode::Collate but not integrated to regexes
1074 [19] have (?<=x) and (?=x), but look-aheads or look-behinds
1075 should see outside of the target substring
1076 [20] need insensitive matching for linguistic features other
1077 than case; for example, hiragana to katakana, wide and
1078 narrow, simplified Han to traditional Han (see UTR#30
1079 "Character Foldings")
1083 =head2 Unicode Encodings
1085 Unicode characters are assigned to I<code points>, which are abstract
1086 numbers. To use these numbers, various encodings are needed.
1094 UTF-8 is a variable-length (1 to 4 bytes), byte-order independent
1095 encoding. For ASCII (and we really do mean 7-bit ASCII, not another
1096 8-bit encoding), UTF-8 is transparent.
1098 The following table is from Unicode 3.2.
1100 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
1102 U+0000..U+007F 00..7F
1103 U+0080..U+07FF * C2..DF 80..BF
1104 U+0800..U+0FFF E0 * A0..BF 80..BF
1105 U+1000..U+CFFF E1..EC 80..BF 80..BF
1106 U+D000..U+D7FF ED 80..9F 80..BF
1107 U+D800..U+DFFF +++++ utf16 surrogates, not legal utf8 +++++
1108 U+E000..U+FFFF EE..EF 80..BF 80..BF
1109 U+10000..U+3FFFF F0 * 90..BF 80..BF 80..BF
1110 U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
1111 U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
1113 Note the gaps marked by "*" before several of the byte entries above. These are
1114 caused by legal UTF-8 avoiding non-shortest encodings: it is technically
1115 possible to UTF-8-encode a single code point in different ways, but that is
1116 explicitly forbidden, and the shortest possible encoding should always be used
1117 (and that is what Perl does).
1119 Another way to look at it is via bits:
1121 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
1124 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
1125 ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
1126 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
1128 As you can see, the continuation bytes all begin with "10", and the
1129 leading bits of the start byte tell how many bytes there are in the
1132 The original UTF-8 specification allowed up to 6 bytes, to allow
1133 encoding of numbers up to 0x7FFF_FFFF. Perl continues to allow those,
1134 and has extended that up to 13 bytes to encode code points up to what
1135 can fit in a 64-bit word. However, Perl will warn if you output any of
1136 these as being non-portable; and under strict UTF-8 input protocols,
1139 The Unicode non-character code points are also disallowed in UTF-8 in
1140 "open interchange". See L</Non-character code points>.
1146 Like UTF-8 but EBCDIC-safe, in the way that UTF-8 is ASCII-safe.
1150 UTF-16, UTF-16BE, UTF-16LE, Surrogates, and BOMs (Byte Order Marks)
1152 The followings items are mostly for reference and general Unicode
1153 knowledge, Perl doesn't use these constructs internally.
1155 Like UTF-8, UTF-16 is a variable-width encoding, but where
1156 UTF-8 uses 8-bit code units, UTF-16 uses 16-bit code units.
1157 All code points occupy either 2 or 4 bytes in UTF-16: code points
1158 C<U+0000..U+FFFF> are stored in a single 16-bit unit, and code
1159 points C<U+10000..U+10FFFF> in two 16-bit units. The latter case is
1160 using I<surrogates>, the first 16-bit unit being the I<high
1161 surrogate>, and the second being the I<low surrogate>.
1163 Surrogates are code points set aside to encode the C<U+10000..U+10FFFF>
1164 range of Unicode code points in pairs of 16-bit units. The I<high
1165 surrogates> are the range C<U+D800..U+DBFF> and the I<low surrogates>
1166 are the range C<U+DC00..U+DFFF>. The surrogate encoding is
1168 $hi = ($uni - 0x10000) / 0x400 + 0xD800;
1169 $lo = ($uni - 0x10000) % 0x400 + 0xDC00;
1173 $uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00);
1175 Because of the 16-bitness, UTF-16 is byte-order dependent. UTF-16
1176 itself can be used for in-memory computations, but if storage or
1177 transfer is required either UTF-16BE (big-endian) or UTF-16LE
1178 (little-endian) encodings must be chosen.
1180 This introduces another problem: what if you just know that your data
1181 is UTF-16, but you don't know which endianness? Byte Order Marks, or
1182 BOMs, are a solution to this. A special character has been reserved
1183 in Unicode to function as a byte order marker: the character with the
1184 code point C<U+FEFF> is the BOM.
1186 The trick is that if you read a BOM, you will know the byte order,
1187 since if it was written on a big-endian platform, you will read the
1188 bytes C<0xFE 0xFF>, but if it was written on a little-endian platform,
1189 you will read the bytes C<0xFF 0xFE>. (And if the originating platform
1190 was writing in UTF-8, you will read the bytes C<0xEF 0xBB 0xBF>.)
1192 The way this trick works is that the character with the code point
1193 C<U+FFFE> is not supposed to be in input streams, so the
1194 sequence of bytes C<0xFF 0xFE> is unambiguously "BOM, represented in
1195 little-endian format" and cannot be C<U+FFFE>, represented in big-endian
1198 Surrogates have no meaning in Unicode outside their use in pairs to
1199 represent other code points. However, Perl allows them to be
1200 represented individually internally, for example by saying
1201 C<chr(0xD801)>, so that all code points, not just those valid for open
1203 representable. Unicode does define semantics for them, such as their
1204 General Category is "Cs". But because their use is somewhat dangerous,
1205 Perl will warn (using the warning category "surrogate", which is a
1206 sub-category of "utf8") if an attempt is made
1207 to do things like take the lower case of one, or match
1208 case-insensitively, or to output them. (But don't try this on Perls
1213 UTF-32, UTF-32BE, UTF-32LE
1215 The UTF-32 family is pretty much like the UTF-16 family, expect that
1216 the units are 32-bit, and therefore the surrogate scheme is not
1217 needed. UTF-32 is a fixed-width encoding. The BOM signatures are
1218 C<0x00 0x00 0xFE 0xFF> for BE and C<0xFF 0xFE 0x00 0x00> for LE.
1224 Legacy, fixed-width encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit
1225 encoding. Unlike UTF-16, UCS-2 is not extensible beyond C<U+FFFF>,
1226 because it does not use surrogates. UCS-4 is a 32-bit encoding,
1227 functionally identical to UTF-32 (the difference being that
1228 UCS-4 forbids neither surrogates nor code points larger than 0x10_FFFF).
1234 A seven-bit safe (non-eight-bit) encoding, which is useful if the
1235 transport or storage is not eight-bit safe. Defined by RFC 2152.
1239 =head2 Non-character code points
1241 66 code points are set aside in Unicode as "non-character code points".
1242 These all have the Unassigned (Cn) General Category, and they never will
1243 be assigned. These are never supposed to be in legal Unicode input
1244 streams, so that code can use them as sentinels that can be mixed in
1245 with character data, and they always will be distinguishable from that data.
1246 To keep them out of Perl input streams, strict UTF-8 should be
1247 specified, such as by using the layer C<:encoding('UTF-8')>. The
1248 non-character code points are the 32 between U+FDD0 and U+FDEF, and the
1249 34 code points U+FFFE, U+FFFF, U+1FFFE, U+1FFFF, ... U+10FFFE, U+10FFFF.
1250 Some people are under the mistaken impression that these are "illegal",
1251 but that is not true. An application or cooperating set of applications
1252 can legally use them at will internally; but these code points are
1253 "illegal for open interchange". Therefore, Perl will not accept these
1254 from input streams unless lax rules are being used, and will warn
1255 (using the warning category "nonchar", which is a sub-category of "utf8") if
1256 an attempt is made to output them.
1258 =head2 Beyond Unicode code points
1260 The maximum Unicode code point is U+10FFFF. But Perl accepts code
1261 points up to the maximum permissible unsigned number available on the
1262 platform. However, Perl will not accept these from input streams unless
1263 lax rules are being used, and will warn (using the warning category
1264 "non_unicode", which is a sub-category of "utf8") if an attempt is made to
1265 operate on or output them. For example, C<uc(0x11_0000)> will generate
1266 this warning, returning the input parameter as its result, as the upper
1267 case of every non-Unicode code point is the code point itself.
1269 =head2 Security Implications of Unicode
1271 Read L<Unicode Security Considerations|http://www.unicode.org/reports/tr36>.
1272 Also, note the following:
1280 Unfortunately, the original specification of UTF-8 leaves some room for
1281 interpretation of how many bytes of encoded output one should generate
1282 from one input Unicode character. Strictly speaking, the shortest
1283 possible sequence of UTF-8 bytes should be generated,
1284 because otherwise there is potential for an input buffer overflow at
1285 the receiving end of a UTF-8 connection. Perl always generates the
1286 shortest length UTF-8, and with warnings on, Perl will warn about
1287 non-shortest length UTF-8 along with other malformations, such as the
1288 surrogates, which are not Unicode code points valid for interchange.
1292 Regular expression pattern matching may surprise you if you're not
1293 accustomed to Unicode. Starting in Perl 5.14, several pattern
1294 modifiers are available to control this, called the character set
1295 modifiers. Details are given in L<perlre/Character set modifiers>.
1299 As discussed elsewhere, Perl has one foot (two hooves?) planted in
1300 each of two worlds: the old world of bytes and the new world of
1301 characters, upgrading from bytes to characters when necessary.
1302 If your legacy code does not explicitly use Unicode, no automatic
1303 switch-over to characters should happen. Characters shouldn't get
1304 downgraded to bytes, either. It is possible to accidentally mix bytes
1305 and characters, however (see L<perluniintro>), in which case C<\w> in
1306 regular expressions might start behaving differently (unless the C</a>
1307 modifier is in effect). Review your code. Use warnings and the C<strict> pragma.
1309 =head2 Unicode in Perl on EBCDIC
1311 The way Unicode is handled on EBCDIC platforms is still
1312 experimental. On such platforms, references to UTF-8 encoding in this
1313 document and elsewhere should be read as meaning the UTF-EBCDIC
1314 specified in Unicode Technical Report 16, unless ASCII vs. EBCDIC issues
1315 are specifically discussed. There is no C<utfebcdic> pragma or
1316 ":utfebcdic" layer; rather, "utf8" and ":utf8" are reused to mean
1317 the platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic>
1318 for more discussion of the issues.
1322 See L<perllocale/Unicode and UTF-8>
1324 =head2 When Unicode Does Not Happen
1326 While Perl does have extensive ways to input and output in Unicode,
1327 and a few other "entry points" like the @ARGV array (which can sometimes be
1328 interpreted as UTF-8), there are still many places where Unicode
1329 (in some encoding or another) could be given as arguments or received as
1330 results, or both, but it is not.
1332 The following are such interfaces. Also, see L</The "Unicode Bug">.
1333 For all of these interfaces Perl
1334 currently (as of v5.16.0) simply assumes byte strings both as arguments
1335 and results, or UTF-8 strings if the (problematic) C<encoding> pragma has been used.
1337 One reason that Perl does not attempt to resolve the role of Unicode in
1338 these situations is that the answers are highly dependent on the operating
1339 system and the file system(s). For example, whether filenames can be
1340 in Unicode and in exactly what kind of encoding, is not exactly a
1341 portable concept. Similarly for C<qx> and C<system>: how well will the
1342 "command-line interface" (and which of them?) handle Unicode?
1348 chdir, chmod, chown, chroot, exec, link, lstat, mkdir,
1349 rename, rmdir, stat, symlink, truncate, unlink, utime, -X
1361 open, opendir, sysopen
1365 qx (aka the backtick operator), system
1373 =head2 The "Unicode Bug"
1375 The term, "Unicode bug" has been applied to an inconsistency
1376 on ASCII platforms with the
1377 Unicode code points in the Latin-1 Supplement block, that
1378 is, between 128 and 255. Without a locale specified, unlike all other
1379 characters or code points, these characters have very different semantics in
1380 byte semantics versus character semantics, unless
1381 C<use feature 'unicode_strings'> is specified, directly or indirectly.
1382 (It is indirectly specified by a C<use v5.12> or higher.)
1384 In character semantics these upper-Latin1 characters are interpreted as
1385 Unicode code points, which means
1386 they have the same semantics as Latin-1 (ISO-8859-1).
1388 In byte semantics (without C<unicode_strings>), they are considered to
1389 be unassigned characters, meaning that the only semantics they have is
1390 their ordinal numbers, and that they are
1391 not members of various character classes. None are considered to match C<\w>
1392 for example, but all match C<\W>.
1394 Perl 5.12.0 added C<unicode_strings> to force character semantics on
1395 these code points in some circumstances, which fixed portions of the
1396 bug; Perl 5.14.0 fixed almost all of it; and Perl 5.16.0 fixed the
1397 remainder (so far as we know, anyway). The lesson here is to enable
1398 C<unicode_strings> to avoid the headaches described below.
1400 The old, problematic behavior affects these areas:
1406 Changing the case of a scalar, that is, using C<uc()>, C<ucfirst()>, C<lc()>,
1407 and C<lcfirst()>, or C<\L>, C<\U>, C<\u> and C<\l> in double-quotish
1408 contexts, such as regular expression substitutions.
1409 Under C<unicode_strings> starting in Perl 5.12.0, character semantics are
1410 generally used. See L<perlfunc/lc> for details on how this works
1411 in combination with various other pragmas.
1415 Using caseless (C</i>) regular expression matching.
1416 Starting in Perl 5.14.0, regular expressions compiled within
1417 the scope of C<unicode_strings> use character semantics
1418 even when executed or compiled into larger
1419 regular expressions outside the scope.
1423 Matching any of several properties in regular expressions, namely C<\b>,
1424 C<\B>, C<\s>, C<\S>, C<\w>, C<\W>, and all the Posix character classes
1425 I<except> C<[[:ascii:]]>.
1426 Starting in Perl 5.14.0, regular expressions compiled within
1427 the scope of C<unicode_strings> use character semantics
1428 even when executed or compiled into larger
1429 regular expressions outside the scope.
1433 In C<quotemeta> or its inline equivalent C<\Q>, no code points above 127
1434 are quoted in UTF-8 encoded strings, but in byte encoded strings, code
1435 points between 128-255 are always quoted.
1436 Starting in Perl 5.16.0, consistent quoting rules are used within the
1437 scope of C<unicode_strings>, as described in L<perlfunc/quotemeta>.
1441 This behavior can lead to unexpected results in which a string's semantics
1442 suddenly change if a code point above 255 is appended to or removed from it,
1443 which changes the string's semantics from byte to character or vice versa. As
1444 an example, consider the following program and its output:
1447 no feature 'unicode_strings';
1450 for ($s1, $s2, $s1.$s2) {
1458 If there's no C<\w> in C<s1> or in C<s2>, why does their concatenation have one?
1460 This anomaly stems from Perl's attempt to not disturb older programs that
1461 didn't use Unicode, and hence had no semantics for characters outside of the
1462 ASCII range (except in a locale), along with Perl's desire to add Unicode
1463 support seamlessly. The result wasn't seamless: these characters were
1466 For Perls earlier than those described above, or when a string is passed
1467 to a function outside the subpragma's scope, a workaround is to always
1468 call C<utf8::upgrade($string)>,
1469 or to use the standard module L<Encode>. Also, a scalar that has any characters
1470 whose ordinal is above 0x100, or which were specified using either of the
1471 C<\N{...}> notations, will automatically have character semantics.
1473 =head2 Forcing Unicode in Perl (Or Unforcing Unicode in Perl)
1475 Sometimes (see L</"When Unicode Does Not Happen"> or L</The "Unicode Bug">)
1476 there are situations where you simply need to force a byte
1477 string into UTF-8, or vice versa. The low-level calls
1478 utf8::upgrade($bytestring) and utf8::downgrade($utf8string[, FAIL_OK]) are
1481 Note that utf8::downgrade() can fail if the string contains characters
1482 that don't fit into a byte.
1484 Calling either function on a string that already is in the desired state is a
1487 =head2 Using Unicode in XS
1489 If you want to handle Perl Unicode in XS extensions, you may find the
1490 following C APIs useful. See also L<perlguts/"Unicode Support"> for an
1491 explanation about Unicode at the XS level, and L<perlapi> for the API
1498 C<DO_UTF8(sv)> returns true if the C<UTF8> flag is on and the bytes
1499 pragma is not in effect. C<SvUTF8(sv)> returns true if the C<UTF8>
1500 flag is on; the bytes pragma is ignored. The C<UTF8> flag being on
1501 does B<not> mean that there are any characters of code points greater
1502 than 255 (or 127) in the scalar or that there are even any characters
1503 in the scalar. What the C<UTF8> flag means is that the sequence of
1504 octets in the representation of the scalar is the sequence of UTF-8
1505 encoded code points of the characters of a string. The C<UTF8> flag
1506 being off means that each octet in this representation encodes a
1507 single character with code point 0..255 within the string. Perl's
1508 Unicode model is not to use UTF-8 until it is absolutely necessary.
1512 C<uvchr_to_utf8(buf, chr)> writes a Unicode character code point into
1513 a buffer encoding the code point as UTF-8, and returns a pointer
1514 pointing after the UTF-8 bytes. It works appropriately on EBCDIC machines.
1518 C<utf8_to_uvchr_buf(buf, bufend, lenp)> reads UTF-8 encoded bytes from a
1520 returns the Unicode character code point and, optionally, the length of
1521 the UTF-8 byte sequence. It works appropriately on EBCDIC machines.
1525 C<utf8_length(start, end)> returns the length of the UTF-8 encoded buffer
1526 in characters. C<sv_len_utf8(sv)> returns the length of the UTF-8 encoded
1531 C<sv_utf8_upgrade(sv)> converts the string of the scalar to its UTF-8
1532 encoded form. C<sv_utf8_downgrade(sv)> does the opposite, if
1533 possible. C<sv_utf8_encode(sv)> is like sv_utf8_upgrade except that
1534 it does not set the C<UTF8> flag. C<sv_utf8_decode()> does the
1535 opposite of C<sv_utf8_encode()>. Note that none of these are to be
1536 used as general-purpose encoding or decoding interfaces: C<use Encode>
1537 for that. C<sv_utf8_upgrade()> is affected by the encoding pragma
1538 but C<sv_utf8_downgrade()> is not (since the encoding pragma is
1539 designed to be a one-way street).
1543 C<is_utf8_string(buf, len)> returns true if C<len> bytes of the buffer
1548 C<is_utf8_char_buf(buf, buf_end)> returns true if the pointer points to
1549 a valid UTF-8 character.
1553 C<UTF8SKIP(buf)> will return the number of bytes in the UTF-8 encoded
1554 character in the buffer. C<UNISKIP(chr)> will return the number of bytes
1555 required to UTF-8-encode the Unicode character code point. C<UTF8SKIP()>
1556 is useful for example for iterating over the characters of a UTF-8
1557 encoded buffer; C<UNISKIP()> is useful, for example, in computing
1558 the size required for a UTF-8 encoded buffer.
1562 C<utf8_distance(a, b)> will tell the distance in characters between the
1563 two pointers pointing to the same UTF-8 encoded buffer.
1567 C<utf8_hop(s, off)> will return a pointer to a UTF-8 encoded buffer
1568 that is C<off> (positive or negative) Unicode characters displaced
1569 from the UTF-8 buffer C<s>. Be careful not to overstep the buffer:
1570 C<utf8_hop()> will merrily run off the end or the beginning of the
1571 buffer if told to do so.
1575 C<pv_uni_display(dsv, spv, len, pvlim, flags)> and
1576 C<sv_uni_display(dsv, ssv, pvlim, flags)> are useful for debugging the
1577 output of Unicode strings and scalars. By default they are useful
1578 only for debugging--they display B<all> characters as hexadecimal code
1579 points--but with the flags C<UNI_DISPLAY_ISPRINT>,
1580 C<UNI_DISPLAY_BACKSLASH>, and C<UNI_DISPLAY_QQ> you can make the
1581 output more readable.
1585 C<foldEQ_utf8(s1, pe1, l1, u1, s2, pe2, l2, u2)> can be used to
1586 compare two strings case-insensitively in Unicode. For case-sensitive
1587 comparisons you can just use C<memEQ()> and C<memNE()> as usual, except
1588 if one string is in utf8 and the other isn't.
1592 For more information, see L<perlapi>, and F<utf8.c> and F<utf8.h>
1593 in the Perl source code distribution.
1595 =head2 Hacking Perl to work on earlier Unicode versions (for very serious hackers only)
1597 Perl by default comes with the latest supported Unicode version built in, but
1598 you can change to use any earlier one.
1600 Download the files in the desired version of Unicode from the Unicode web
1601 site L<http://www.unicode.org>). These should replace the existing files in
1602 F<lib/unicore> in the Perl source tree. Follow the instructions in
1603 F<README.perl> in that directory to change some of their names, and then build
1604 perl (see L<INSTALL>).
1608 =head2 Interaction with Locales
1610 See L<perllocale/Unicode and UTF-8>
1612 =head2 Problems with characters in the Latin-1 Supplement range
1614 See L</The "Unicode Bug">
1616 =head2 Interaction with Extensions
1618 When Perl exchanges data with an extension, the extension should be
1619 able to understand the UTF8 flag and act accordingly. If the
1620 extension doesn't recognize that flag, it's likely that the extension
1621 will return incorrectly-flagged data.
1623 So if you're working with Unicode data, consult the documentation of
1624 every module you're using if there are any issues with Unicode data
1625 exchange. If the documentation does not talk about Unicode at all,
1626 suspect the worst and probably look at the source to learn how the
1627 module is implemented. Modules written completely in Perl shouldn't
1628 cause problems. Modules that directly or indirectly access code written
1629 in other programming languages are at risk.
1631 For affected functions, the simple strategy to avoid data corruption is
1632 to always make the encoding of the exchanged data explicit. Choose an
1633 encoding that you know the extension can handle. Convert arguments passed
1634 to the extensions to that encoding and convert results back from that
1635 encoding. Write wrapper functions that do the conversions for you, so
1636 you can later change the functions when the extension catches up.
1638 To provide an example, let's say the popular Foo::Bar::escape_html
1639 function doesn't deal with Unicode data yet. The wrapper function
1640 would convert the argument to raw UTF-8 and convert the result back to
1641 Perl's internal representation like so:
1643 sub my_escape_html ($) {
1645 return unless defined $what;
1646 Encode::decode_utf8(Foo::Bar::escape_html(
1647 Encode::encode_utf8($what)));
1650 Sometimes, when the extension does not convert data but just stores
1651 and retrieves them, you will be able to use the otherwise
1652 dangerous Encode::_utf8_on() function. Let's say the popular
1653 C<Foo::Bar> extension, written in C, provides a C<param> method that
1654 lets you store and retrieve data according to these prototypes:
1656 $self->param($name, $value); # set a scalar
1657 $value = $self->param($name); # retrieve a scalar
1659 If it does not yet provide support for any encoding, one could write a
1660 derived class with such a C<param> method:
1663 my($self,$name,$value) = @_;
1664 utf8::upgrade($name); # make sure it is UTF-8 encoded
1665 if (defined $value) {
1666 utf8::upgrade($value); # make sure it is UTF-8 encoded
1667 return $self->SUPER::param($name,$value);
1669 my $ret = $self->SUPER::param($name);
1670 Encode::_utf8_on($ret); # we know, it is UTF-8 encoded
1675 Some extensions provide filters on data entry/exit points, such as
1676 DB_File::filter_store_key and family. Look out for such filters in
1677 the documentation of your extensions, they can make the transition to
1678 Unicode data much easier.
1682 Some functions are slower when working on UTF-8 encoded strings than
1683 on byte encoded strings. All functions that need to hop over
1684 characters such as length(), substr() or index(), or matching regular
1685 expressions can work B<much> faster when the underlying data are
1688 In Perl 5.8.0 the slowness was often quite spectacular; in Perl 5.8.1
1689 a caching scheme was introduced which will hopefully make the slowness
1690 somewhat less spectacular, at least for some operations. In general,
1691 operations with UTF-8 encoded strings are still slower. As an example,
1692 the Unicode properties (character classes) like C<\p{Nd}> are known to
1693 be quite a bit slower (5-20 times) than their simpler counterparts
1694 like C<\d> (then again, there are hundreds of Unicode characters matching C<Nd>
1695 compared with the 10 ASCII characters matching C<d>).
1697 =head2 Problems on EBCDIC platforms
1699 There are several known problems with Perl on EBCDIC platforms. If you
1700 want to use Perl there, send email to perlbug@perl.org.
1702 In earlier versions, when byte and character data were concatenated,
1703 the new string was sometimes created by
1704 decoding the byte strings as I<ISO 8859-1 (Latin-1)>, even if the
1705 old Unicode string used EBCDIC.
1707 If you find any of these, please report them as bugs.
1709 =head2 Porting code from perl-5.6.X
1711 Perl 5.8 has a different Unicode model from 5.6. In 5.6 the programmer
1712 was required to use the C<utf8> pragma to declare that a given scope
1713 expected to deal with Unicode data and had to make sure that only
1714 Unicode data were reaching that scope. If you have code that is
1715 working with 5.6, you will need some of the following adjustments to
1716 your code. The examples are written such that the code will continue
1717 to work under 5.6, so you should be safe to try them out.
1723 A filehandle that should read or write UTF-8
1726 binmode $fh, ":encoding(utf8)";
1731 A scalar that is going to be passed to some extension
1733 Be it Compress::Zlib, Apache::Request or any extension that has no
1734 mention of Unicode in the manpage, you need to make sure that the
1735 UTF8 flag is stripped off. Note that at the time of this writing
1736 (January 2012) the mentioned modules are not UTF-8-aware. Please
1737 check the documentation to verify if this is still true.
1741 $val = Encode::encode_utf8($val); # make octets
1746 A scalar we got back from an extension
1748 If you believe the scalar comes back as UTF-8, you will most likely
1749 want the UTF8 flag restored:
1753 $val = Encode::decode_utf8($val);
1758 Same thing, if you are really sure it is UTF-8
1762 Encode::_utf8_on($val);
1767 A wrapper for fetchrow_array and fetchrow_hashref
1769 When the database contains only UTF-8, a wrapper function or method is
1770 a convenient way to replace all your fetchrow_array and
1771 fetchrow_hashref calls. A wrapper function will also make it easier to
1772 adapt to future enhancements in your database driver. Note that at the
1773 time of this writing (January 2012), the DBI has no standardized way
1774 to deal with UTF-8 data. Please check the documentation to verify if
1778 # $what is one of fetchrow_{array,hashref}
1779 my($self, $sth, $what) = @_;
1785 my @arr = $sth->$what;
1787 defined && /[^\000-\177]/ && Encode::_utf8_on($_);
1791 my $ret = $sth->$what;
1793 for my $k (keys %$ret) {
1796 && Encode::_utf8_on($_) for $ret->{$k};
1800 defined && /[^\000-\177]/ && Encode::_utf8_on($_) for $ret;
1810 A large scalar that you know can only contain ASCII
1812 Scalars that contain only ASCII and are marked as UTF-8 are sometimes
1813 a drag to your program. If you recognize such a situation, just remove
1816 utf8::downgrade($val) if $] > 5.008;
1822 L<perlunitut>, L<perluniintro>, L<perluniprops>, L<Encode>, L<open>, L<utf8>, L<bytes>,
1823 L<perlretut>, L<perlvar/"${^UNICODE}">
1824 L<http://www.unicode.org/reports/tr44>).