Commit | Line | Data |
---|---|---|
393fec97 GS |
1 | =head1 NAME |
2 | ||
3 | perlunicode - Unicode support in Perl | |
4 | ||
5 | =head1 DESCRIPTION | |
6 | ||
0a1f2d14 | 7 | =head2 Important Caveats |
21bad921 | 8 | |
376d9008 | 9 | Unicode support is an extensive requirement. While Perl does not |
c349b1b9 JH |
10 | implement the Unicode standard or the accompanying technical reports |
11 | from cover to cover, Perl does support many Unicode features. | |
21bad921 | 12 | |
2575c402 | 13 | People who want to learn to use Unicode in Perl, should probably read |
0314f483 KW |
14 | the L<Perl Unicode tutorial, perlunitut|perlunitut> and |
15 | L<perluniintro>, before reading | |
e4911a48 | 16 | this reference document. |
2575c402 | 17 | |
9d1c51c1 KW |
18 | Also, the use of Unicode may present security issues that aren't obvious. |
19 | Read L<Unicode Security Considerations|http://www.unicode.org/reports/tr36>. | |
20 | ||
13a2d996 | 21 | =over 4 |
21bad921 | 22 | |
42581d5d KW |
23 | =item Safest if you "use feature 'unicode_strings'" |
24 | ||
25 | In order to preserve backward compatibility, Perl does not turn | |
26 | on full internal Unicode support unless the pragma | |
27 | C<use feature 'unicode_strings'> is specified. (This is automatically | |
28 | selected if you use C<use 5.012> or higher.) Failure to do this can | |
29 | trigger unexpected surprises. See L</The "Unicode Bug"> below. | |
30 | ||
2269d15c KW |
31 | This pragma doesn't affect I/O. Nor does it change the internal |
32 | representation of strings, only their interpretation. There are still | |
33 | several places where Unicode isn't fully supported, such as in | |
34 | filenames. | |
42581d5d | 35 | |
fae2c0fb | 36 | =item Input and Output Layers |
21bad921 | 37 | |
376d9008 | 38 | Perl knows when a filehandle uses Perl's internal Unicode encodings |
1bfb14c4 | 39 | (UTF-8, or UTF-EBCDIC if in EBCDIC) if the filehandle is opened with |
4ee7c0ea | 40 | the ":encoding(utf8)" layer. Other encodings can be converted to Perl's |
1bfb14c4 JH |
41 | encoding on input or from Perl's encoding on output by use of the |
42 | ":encoding(...)" layer. See L<open>. | |
c349b1b9 | 43 | |
2575c402 | 44 | To indicate that Perl source itself is in UTF-8, use C<use utf8;>. |
21bad921 | 45 | |
ad0029c4 | 46 | =item C<use utf8> still needed to enable UTF-8/UTF-EBCDIC in scripts |
21bad921 | 47 | |
376d9008 JB |
48 | As a compatibility measure, the C<use utf8> pragma must be explicitly |
49 | included to enable recognition of UTF-8 in the Perl scripts themselves | |
1bfb14c4 JH |
50 | (in string or regular expression literals, or in identifier names) on |
51 | ASCII-based machines or to recognize UTF-EBCDIC on EBCDIC-based | |
376d9008 | 52 | machines. B<These are the only times when an explicit C<use utf8> |
8f8cf39c | 53 | is needed.> See L<utf8>. |
21bad921 | 54 | |
7aa207d6 JH |
55 | =item BOM-marked scripts and UTF-16 scripts autodetected |
56 | ||
57 | If a Perl script begins marked with the Unicode BOM (UTF-16LE, UTF16-BE, | |
58 | or UTF-8), or if the script looks like non-BOM-marked UTF-16 of either | |
59 | endianness, Perl will correctly read in the script as Unicode. | |
60 | (BOMless UTF-8 cannot be effectively recognized or differentiated from | |
61 | ISO 8859-1 or other eight-bit encodings.) | |
62 | ||
990e18f7 AT |
63 | =item C<use encoding> needed to upgrade non-Latin-1 byte strings |
64 | ||
38a44b82 | 65 | By default, there is a fundamental asymmetry in Perl's Unicode model: |
990e18f7 AT |
66 | implicit upgrading from byte strings to Unicode strings assumes that |
67 | they were encoded in I<ISO 8859-1 (Latin-1)>, but Unicode strings are | |
68 | downgraded with UTF-8 encoding. This happens because the first 256 | |
51f494cc | 69 | codepoints in Unicode happens to agree with Latin-1. |
990e18f7 | 70 | |
990e18f7 AT |
71 | See L</"Byte and Character Semantics"> for more details. |
72 | ||
21bad921 GS |
73 | =back |
74 | ||
376d9008 | 75 | =head2 Byte and Character Semantics |
393fec97 | 76 | |
b9cedb1b | 77 | Perl uses logically-wide characters to represent strings internally. |
393fec97 | 78 | |
42581d5d KW |
79 | Starting in Perl 5.14, Perl-level operations work with |
80 | characters rather than bytes within the scope of a | |
81 | C<L<use feature 'unicode_strings'|feature>> (or equivalently | |
82 | C<use 5.012> or higher). (This is not true if bytes have been | |
b19eb496 | 83 | explicitly requested by C<L<use bytes|bytes>>, nor necessarily true |
42581d5d KW |
84 | for interactions with the platform's operating system.) |
85 | ||
86 | For earlier Perls, and when C<unicode_strings> is not in effect, Perl | |
87 | provides a fairly safe environment that can handle both types of | |
88 | semantics in programs. For operations where Perl can unambiguously | |
89 | decide that the input data are characters, Perl switches to character | |
90 | semantics. For operations where this determination cannot be made | |
91 | without additional information from the user, Perl decides in favor of | |
92 | compatibility and chooses to use byte semantics. | |
93 | ||
66cbab2c KW |
94 | When C<use locale> (but not C<use locale ':not_characters'>) is in |
95 | effect, Perl uses the semantics associated with the current locale. | |
96 | (C<use locale> overrides C<use feature 'unicode_strings'> in the same scope; | |
97 | while C<use locale ':not_characters'> effectively also selects | |
98 | C<use feature 'unicode_strings'> in its scope; see L<perllocale>.) | |
99 | Otherwise, Perl uses the platform's native | |
42581d5d | 100 | byte semantics for characters whose code points are less than 256, and |
4b9734bf KW |
101 | Unicode semantics for those greater than 255. That means that non-ASCII |
102 | characters are undefined except for their | |
e1b711da KW |
103 | ordinal numbers. This means that none have case (upper and lower), nor are any |
104 | a member of character classes, like C<[:alpha:]> or C<\w>. (But all do belong | |
105 | to the C<\W> class or the Perl regular expression extension C<[:^alpha:]>.) | |
2bbc8d55 | 106 | |
8cbd9a7a | 107 | This behavior preserves compatibility with earlier versions of Perl, |
376d9008 | 108 | which allowed byte semantics in Perl operations only if |
e1b711da | 109 | none of the program's inputs were marked as being a source of Unicode |
8cbd9a7a GS |
110 | character data. Such data may come from filehandles, from calls to |
111 | external programs, from information provided by the system (such as %ENV), | |
21bad921 | 112 | or from literals and constants in the source text. |
8cbd9a7a | 113 | |
8cbd9a7a | 114 | The C<utf8> pragma is primarily a compatibility device that enables |
75daf61c | 115 | recognition of UTF-(8|EBCDIC) in literals encountered by the parser. |
376d9008 JB |
116 | Note that this pragma is only required while Perl defaults to byte |
117 | semantics; when character semantics become the default, this pragma | |
118 | may become a no-op. See L<utf8>. | |
119 | ||
376d9008 | 120 | If strings operating under byte semantics and strings with Unicode |
51f494cc | 121 | character data are concatenated, the new string will have |
d9b01026 KW |
122 | character semantics. This can cause surprises: See L</BUGS>, below. |
123 | You can choose to be warned when this happens. See L<encoding::warnings>. | |
7dedd01f | 124 | |
feda178f | 125 | Under character semantics, many operations that formerly operated on |
376d9008 | 126 | bytes now operate on characters. A character in Perl is |
feda178f | 127 | logically just a number ranging from 0 to 2**31 or so. Larger |
376d9008 JB |
128 | characters may encode into longer sequences of bytes internally, but |
129 | this internal detail is mostly hidden for Perl code. | |
130 | See L<perluniintro> for more. | |
393fec97 | 131 | |
376d9008 | 132 | =head2 Effects of Character Semantics |
393fec97 GS |
133 | |
134 | Character semantics have the following effects: | |
135 | ||
136 | =over 4 | |
137 | ||
138 | =item * | |
139 | ||
376d9008 | 140 | Strings--including hash keys--and regular expression patterns may |
574c8022 | 141 | contain characters that have an ordinal value larger than 255. |
393fec97 | 142 | |
2575c402 JW |
143 | If you use a Unicode editor to edit your program, Unicode characters may |
144 | occur directly within the literal strings in UTF-8 encoding, or UTF-16. | |
145 | (The former requires a BOM or C<use utf8>, the latter requires a BOM.) | |
3e4dbfed | 146 | |
195e542a KW |
147 | Unicode characters can also be added to a string by using the C<\N{U+...}> |
148 | notation. The Unicode code for the desired character, in hexadecimal, | |
149 | should be placed in the braces, after the C<U>. For instance, a smiley face is | |
6f335b04 KW |
150 | C<\N{U+263A}>. |
151 | ||
195e542a KW |
152 | Alternatively, you can use the C<\x{...}> notation for characters 0x100 and |
153 | above. For characters below 0x100 you may get byte semantics instead of | |
6f335b04 | 154 | character semantics; see L</The "Unicode Bug">. On EBCDIC machines there is |
195e542a | 155 | the additional problem that the value for such characters gives the EBCDIC |
0bd42786 KW |
156 | character rather than the Unicode one, thus it is more portable to use |
157 | C<\N{U+...}> instead. | |
3e4dbfed | 158 | |
fbb93542 KW |
159 | Additionally, you can use the C<\N{...}> notation and put the official |
160 | Unicode character name within the braces, such as | |
161 | C<\N{WHITE SMILING FACE}>. This automatically loads the L<charnames> | |
162 | module with the C<:full> and C<:short> options. If you prefer different | |
163 | options for this module, you can instead, before the C<\N{...}>, | |
164 | explicitly load it with your desired options; for example, | |
165 | ||
166 | use charnames ':loose'; | |
376d9008 | 167 | |
393fec97 GS |
168 | =item * |
169 | ||
574c8022 JH |
170 | If an appropriate L<encoding> is specified, identifiers within the |
171 | Perl script may contain Unicode alphanumeric characters, including | |
376d9008 JB |
172 | ideographs. Perl does not currently attempt to canonicalize variable |
173 | names. | |
393fec97 | 174 | |
393fec97 GS |
175 | =item * |
176 | ||
1bfb14c4 | 177 | Regular expressions match characters instead of bytes. "." matches |
2575c402 | 178 | a character instead of a byte. |
393fec97 | 179 | |
393fec97 GS |
180 | =item * |
181 | ||
9d1c51c1 | 182 | Bracketed character classes in regular expressions match characters instead of |
376d9008 | 183 | bytes and match against the character properties specified in the |
1bfb14c4 | 184 | Unicode properties database. C<\w> can be used to match a Japanese |
75daf61c | 185 | ideograph, for instance. |
393fec97 | 186 | |
393fec97 GS |
187 | =item * |
188 | ||
9d1c51c1 KW |
189 | Named Unicode properties, scripts, and block ranges may be used (like bracketed |
190 | character classes) by using the C<\p{}> "matches property" construct and | |
822502e5 | 191 | the C<\P{}> negation, "doesn't match property". |
2575c402 | 192 | See L</"Unicode Character Properties"> for more details. |
822502e5 TS |
193 | |
194 | You can define your own character properties and use them | |
195 | in the regular expression with the C<\p{}> or C<\P{}> construct. | |
822502e5 TS |
196 | See L</"User-Defined Character Properties"> for more details. |
197 | ||
198 | =item * | |
199 | ||
9f815e24 KW |
200 | The special pattern C<\X> matches a logical character, an "extended grapheme |
201 | cluster" in Standardese. In Unicode what appears to the user to be a single | |
51f494cc KW |
202 | character, for example an accented C<G>, may in fact be composed of a sequence |
203 | of characters, in this case a C<G> followed by an accent character. C<\X> | |
204 | will match the entire sequence. | |
822502e5 TS |
205 | |
206 | =item * | |
207 | ||
208 | The C<tr///> operator translates characters instead of bytes. Note | |
209 | that the C<tr///CU> functionality has been removed. For similar | |
210 | functionality see pack('U0', ...) and pack('C0', ...). | |
211 | ||
212 | =item * | |
213 | ||
214 | Case translation operators use the Unicode case translation tables | |
215 | when character input is provided. Note that C<uc()>, or C<\U> in | |
216 | interpolated strings, translates to uppercase, while C<ucfirst>, | |
217 | or C<\u> in interpolated strings, translates to titlecase in languages | |
e1b711da KW |
218 | that make the distinction (which is equivalent to uppercase in languages |
219 | without the distinction). | |
822502e5 TS |
220 | |
221 | =item * | |
222 | ||
223 | Most operators that deal with positions or lengths in a string will | |
224 | automatically switch to using character positions, including | |
225 | C<chop()>, C<chomp()>, C<substr()>, C<pos()>, C<index()>, C<rindex()>, | |
226 | C<sprintf()>, C<write()>, and C<length()>. An operator that | |
51f494cc KW |
227 | specifically does not switch is C<vec()>. Operators that really don't |
228 | care include operators that treat strings as a bucket of bits such as | |
822502e5 TS |
229 | C<sort()>, and operators dealing with filenames. |
230 | ||
231 | =item * | |
232 | ||
51f494cc | 233 | The C<pack()>/C<unpack()> letter C<C> does I<not> change, since it is often |
822502e5 TS |
234 | used for byte-oriented formats. Again, think C<char> in the C language. |
235 | ||
236 | There is a new C<U> specifier that converts between Unicode characters | |
237 | and code points. There is also a C<W> specifier that is the equivalent of | |
238 | C<chr>/C<ord> and properly handles character values even if they are above 255. | |
239 | ||
240 | =item * | |
241 | ||
242 | The C<chr()> and C<ord()> functions work on characters, similar to | |
243 | C<pack("W")> and C<unpack("W")>, I<not> C<pack("C")> and | |
244 | C<unpack("C")>. C<pack("C")> and C<unpack("C")> are methods for | |
245 | emulating byte-oriented C<chr()> and C<ord()> on Unicode strings. | |
246 | While these methods reveal the internal encoding of Unicode strings, | |
247 | that is not something one normally needs to care about at all. | |
248 | ||
249 | =item * | |
250 | ||
251 | The bit string operators, C<& | ^ ~>, can operate on character data. | |
252 | However, for backward compatibility, such as when using bit string | |
253 | operations when characters are all less than 256 in ordinal value, one | |
254 | should not use C<~> (the bit complement) with characters of both | |
255 | values less than 256 and values greater than 256. Most importantly, | |
256 | DeMorgan's laws (C<~($x|$y) eq ~$x&~$y> and C<~($x&$y) eq ~$x|~$y>) | |
257 | will not hold. The reason for this mathematical I<faux pas> is that | |
258 | the complement cannot return B<both> the 8-bit (byte-wide) bit | |
259 | complement B<and> the full character-wide bit complement. | |
260 | ||
261 | =item * | |
262 | ||
5d1892be | 263 | There is a CPAN module, L<Unicode::Casing>, which allows you to define |
628253b8 BF |
264 | your own mappings to be used in C<lc()>, C<lcfirst()>, C<uc()>, |
265 | C<ucfirst()>, and C<fc> (or their double-quoted string inlined | |
266 | versions such as C<\U>). | |
267 | (Prior to Perl 5.16, this functionality was partially provided | |
5d1892be KW |
268 | in the Perl core, but suffered from a number of insurmountable |
269 | drawbacks, so the CPAN module was written instead.) | |
822502e5 TS |
270 | |
271 | =back | |
272 | ||
273 | =over 4 | |
274 | ||
275 | =item * | |
276 | ||
277 | And finally, C<scalar reverse()> reverses by character rather than by byte. | |
278 | ||
279 | =back | |
280 | ||
281 | =head2 Unicode Character Properties | |
282 | ||
ee88f7b6 | 283 | (The only time that Perl considers a sequence of individual code |
9d1c51c1 KW |
284 | points as a single logical character is in the C<\X> construct, already |
285 | mentioned above. Therefore "character" in this discussion means a single | |
ee88f7b6 KW |
286 | Unicode code point.) |
287 | ||
288 | Very nearly all Unicode character properties are accessible through | |
289 | regular expressions by using the C<\p{}> "matches property" construct | |
290 | and the C<\P{}> "doesn't match property" for its negation. | |
51f494cc | 291 | |
9d1c51c1 | 292 | For instance, C<\p{Uppercase}> matches any single character with the Unicode |
51f494cc KW |
293 | "Uppercase" property, while C<\p{L}> matches any character with a |
294 | General_Category of "L" (letter) property. Brackets are not | |
9d1c51c1 | 295 | required for single letter property names, so C<\p{L}> is equivalent to C<\pL>. |
51f494cc | 296 | |
9d1c51c1 KW |
297 | More formally, C<\p{Uppercase}> matches any single character whose Unicode |
298 | Uppercase property value is True, and C<\P{Uppercase}> matches any character | |
299 | whose Uppercase property value is False, and they could have been written as | |
300 | C<\p{Uppercase=True}> and C<\p{Uppercase=False}>, respectively. | |
51f494cc | 301 | |
b19eb496 | 302 | This formality is needed when properties are not binary; that is, if they can |
51f494cc | 303 | take on more values than just True and False. For example, the Bidi_Class (see |
b19eb496 | 304 | L</"Bidirectional Character Types"> below), can take on several different |
51f494cc | 305 | values, such as Left, Right, Whitespace, and others. To match these, one needs |
5bff2035 KW |
306 | to specify both the property name (Bidi_Class), AND the value being |
307 | matched against | |
9d1c51c1 | 308 | (Left, Right, etc.). This is done, as in the examples above, by having the |
9f815e24 | 309 | two components separated by an equal sign (or interchangeably, a colon), like |
51f494cc KW |
310 | C<\p{Bidi_Class: Left}>. |
311 | ||
312 | All Unicode-defined character properties may be written in these compound forms | |
313 | of C<\p{property=value}> or C<\p{property:value}>, but Perl provides some | |
314 | additional properties that are written only in the single form, as well as | |
315 | single-form short-cuts for all binary properties and certain others described | |
316 | below, in which you may omit the property name and the equals or colon | |
317 | separator. | |
318 | ||
319 | Most Unicode character properties have at least two synonyms (or aliases if you | |
b19eb496 TC |
320 | prefer): a short one that is easier to type and a longer one that is more |
321 | descriptive and hence easier to understand. Thus the "L" and "Letter" properties | |
322 | above are equivalent and can be used interchangeably. Likewise, | |
51f494cc KW |
323 | "Upper" is a synonym for "Uppercase", and we could have written |
324 | C<\p{Uppercase}> equivalently as C<\p{Upper}>. Also, there are typically | |
325 | various synonyms for the values the property can be. For binary properties, | |
326 | "True" has 3 synonyms: "T", "Yes", and "Y"; and "False has correspondingly "F", | |
327 | "No", and "N". But be careful. A short form of a value for one property may | |
e1b711da | 328 | not mean the same thing as the same short form for another. Thus, for the |
51f494cc KW |
329 | General_Category property, "L" means "Letter", but for the Bidi_Class property, |
330 | "L" means "Left". A complete list of properties and synonyms is in | |
331 | L<perluniprops>. | |
332 | ||
b19eb496 | 333 | Upper/lower case differences in property names and values are irrelevant; |
51f494cc KW |
334 | thus C<\p{Upper}> means the same thing as C<\p{upper}> or even C<\p{UpPeR}>. |
335 | Similarly, you can add or subtract underscores anywhere in the middle of a | |
336 | word, so that these are also equivalent to C<\p{U_p_p_e_r}>. And white space | |
337 | is irrelevant adjacent to non-word characters, such as the braces and the equals | |
b19eb496 TC |
338 | or colon separators, so C<\p{ Upper }> and C<\p{ Upper_case : Y }> are |
339 | equivalent to these as well. In fact, white space and even | |
340 | hyphens can usually be added or deleted anywhere. So even C<\p{ Up-per case = Yes}> is | |
51f494cc | 341 | equivalent. All this is called "loose-matching" by Unicode. The few places |
b19eb496 | 342 | where stricter matching is used is in the middle of numbers, and in the Perl |
51f494cc | 343 | extension properties that begin or end with an underscore. Stricter matching |
b19eb496 | 344 | cares about white space (except adjacent to non-word characters), |
51f494cc | 345 | hyphens, and non-interior underscores. |
4193bef7 | 346 | |
376d9008 JB |
347 | You can also use negation in both C<\p{}> and C<\P{}> by introducing a caret |
348 | (^) between the first brace and the property name: C<\p{^Tamil}> is | |
eb0cc9e3 | 349 | equal to C<\P{Tamil}>. |
4193bef7 | 350 | |
56ca34ca KW |
351 | Almost all properties are immune to case-insensitive matching. That is, |
352 | adding a C</i> regular expression modifier does not change what they | |
353 | match. There are two sets that are affected. | |
354 | The first set is | |
355 | C<Uppercase_Letter>, | |
356 | C<Lowercase_Letter>, | |
357 | and C<Titlecase_Letter>, | |
358 | all of which match C<Cased_Letter> under C</i> matching. | |
359 | And the second set is | |
360 | C<Uppercase>, | |
361 | C<Lowercase>, | |
362 | and C<Titlecase>, | |
363 | all of which match C<Cased> under C</i> matching. | |
364 | This set also includes its subsets C<PosixUpper> and C<PosixLower> both | |
365 | of which under C</i> matching match C<PosixAlpha>. | |
366 | (The difference between these sets is that some things, such as Roman | |
b19eb496 | 367 | numerals, come in both upper and lower case so they are C<Cased>, but aren't considered |
56ca34ca | 368 | letters, so they aren't C<Cased_Letter>s.) |
56ca34ca | 369 | |
94b42e47 KW |
370 | The result is undefined if you try to match a non-Unicode code point |
371 | (that is, one above 0x10FFFF) against a Unicode property. Currently, a | |
372 | warning is raised, and the match will fail. In some cases, this is | |
373 | counterintuitive, as both these fail: | |
374 | ||
375 | chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Fails. | |
376 | chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Fails! | |
377 | ||
51f494cc | 378 | =head3 B<General_Category> |
14bb0a9a | 379 | |
51f494cc KW |
380 | Every Unicode character is assigned a general category, which is the "most |
381 | usual categorization of a character" (from | |
382 | L<http://www.unicode.org/reports/tr44>). | |
822502e5 | 383 | |
9f815e24 | 384 | The compound way of writing these is like C<\p{General_Category=Number}> |
51f494cc KW |
385 | (short, C<\p{gc:n}>). But Perl furnishes shortcuts in which everything up |
386 | through the equal or colon separator is omitted. So you can instead just write | |
387 | C<\pN>. | |
822502e5 | 388 | |
51f494cc | 389 | Here are the short and long forms of the General Category properties: |
393fec97 | 390 | |
d73e5302 JH |
391 | Short Long |
392 | ||
393 | L Letter | |
51f494cc KW |
394 | LC, L& Cased_Letter (that is: [\p{Ll}\p{Lu}\p{Lt}]) |
395 | Lu Uppercase_Letter | |
396 | Ll Lowercase_Letter | |
397 | Lt Titlecase_Letter | |
398 | Lm Modifier_Letter | |
399 | Lo Other_Letter | |
d73e5302 JH |
400 | |
401 | M Mark | |
51f494cc KW |
402 | Mn Nonspacing_Mark |
403 | Mc Spacing_Mark | |
404 | Me Enclosing_Mark | |
d73e5302 JH |
405 | |
406 | N Number | |
51f494cc KW |
407 | Nd Decimal_Number (also Digit) |
408 | Nl Letter_Number | |
409 | No Other_Number | |
410 | ||
411 | P Punctuation (also Punct) | |
412 | Pc Connector_Punctuation | |
413 | Pd Dash_Punctuation | |
414 | Ps Open_Punctuation | |
415 | Pe Close_Punctuation | |
416 | Pi Initial_Punctuation | |
d73e5302 | 417 | (may behave like Ps or Pe depending on usage) |
51f494cc | 418 | Pf Final_Punctuation |
d73e5302 | 419 | (may behave like Ps or Pe depending on usage) |
51f494cc | 420 | Po Other_Punctuation |
d73e5302 JH |
421 | |
422 | S Symbol | |
51f494cc KW |
423 | Sm Math_Symbol |
424 | Sc Currency_Symbol | |
425 | Sk Modifier_Symbol | |
426 | So Other_Symbol | |
d73e5302 JH |
427 | |
428 | Z Separator | |
51f494cc KW |
429 | Zs Space_Separator |
430 | Zl Line_Separator | |
431 | Zp Paragraph_Separator | |
d73e5302 JH |
432 | |
433 | C Other | |
d88362ca | 434 | Cc Control (also Cntrl) |
e150c829 | 435 | Cf Format |
6d4f9cf2 | 436 | Cs Surrogate |
51f494cc | 437 | Co Private_Use |
e150c829 | 438 | Cn Unassigned |
1ac13f9a | 439 | |
376d9008 | 440 | Single-letter properties match all characters in any of the |
3e4dbfed | 441 | two-letter sub-properties starting with the same letter. |
b19eb496 | 442 | C<LC> and C<L&> are special: both are aliases for the set consisting of everything matched by C<Ll>, C<Lu>, and C<Lt>. |
32293815 | 443 | |
51f494cc | 444 | =head3 B<Bidirectional Character Types> |
822502e5 | 445 | |
b19eb496 | 446 | Because scripts differ in their directionality (Hebrew and Arabic are |
9d1c51c1 | 447 | written right to left, for example) Unicode supplies these properties in |
51f494cc | 448 | the Bidi_Class class: |
32293815 | 449 | |
eb0cc9e3 | 450 | Property Meaning |
92e830a9 | 451 | |
12ac2576 JP |
452 | L Left-to-Right |
453 | LRE Left-to-Right Embedding | |
454 | LRO Left-to-Right Override | |
455 | R Right-to-Left | |
51f494cc | 456 | AL Arabic Letter |
12ac2576 JP |
457 | RLE Right-to-Left Embedding |
458 | RLO Right-to-Left Override | |
459 | PDF Pop Directional Format | |
460 | EN European Number | |
51f494cc KW |
461 | ES European Separator |
462 | ET European Terminator | |
12ac2576 | 463 | AN Arabic Number |
51f494cc | 464 | CS Common Separator |
12ac2576 JP |
465 | NSM Non-Spacing Mark |
466 | BN Boundary Neutral | |
467 | B Paragraph Separator | |
468 | S Segment Separator | |
469 | WS Whitespace | |
470 | ON Other Neutrals | |
471 | ||
51f494cc KW |
472 | This property is always written in the compound form. |
473 | For example, C<\p{Bidi_Class:R}> matches characters that are normally | |
eb0cc9e3 JH |
474 | written right to left. |
475 | ||
51f494cc KW |
476 | =head3 B<Scripts> |
477 | ||
b19eb496 | 478 | The world's languages are written in many different scripts. This sentence |
e1b711da | 479 | (unless you're reading it in translation) is written in Latin, while Russian is |
c69ca1d4 | 480 | written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in |
e1b711da | 481 | Hiragana or Katakana. There are many more. |
51f494cc | 482 | |
82aed44a KW |
483 | The Unicode Script and Script_Extensions properties give what script a |
484 | given character is in. Either property can be specified with the | |
485 | compound form like | |
486 | C<\p{Script=Hebrew}> (short: C<\p{sc=hebr}>), or | |
487 | C<\p{Script_Extensions=Javanese}> (short: C<\p{scx=java}>). | |
488 | In addition, Perl furnishes shortcuts for all | |
489 | C<Script> property names. You can omit everything up through the equals | |
490 | (or colon), and simply write C<\p{Latin}> or C<\P{Cyrillic}>. | |
491 | (This is not true for C<Script_Extensions>, which is required to be | |
492 | written in the compound form.) | |
493 | ||
494 | The difference between these two properties involves characters that are | |
495 | used in multiple scripts. For example the digits '0' through '9' are | |
496 | used in many parts of the world. These are placed in a script named | |
497 | C<Common>. Other characters are used in just a few scripts. For | |
498 | example, the "KATAKANA-HIRAGANA DOUBLE HYPHEN" is used in both Japanese | |
499 | scripts, Katakana and Hiragana, but nowhere else. The C<Script> | |
500 | property places all characters that are used in multiple scripts in the | |
501 | C<Common> script, while the C<Script_Extensions> property places those | |
502 | that are used in only a few scripts into each of those scripts; while | |
503 | still using C<Common> for those used in many scripts. Thus both these | |
504 | match: | |
505 | ||
506 | "0" =~ /\p{sc=Common}/ # Matches | |
507 | "0" =~ /\p{scx=Common}/ # Matches | |
508 | ||
509 | and only the first of these match: | |
510 | ||
511 | "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Common} # Matches | |
512 | "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Common} # No match | |
513 | ||
514 | And only the last two of these match: | |
515 | ||
516 | "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Hiragana} # No match | |
517 | "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Katakana} # No match | |
518 | "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Hiragana} # Matches | |
519 | "\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Katakana} # Matches | |
520 | ||
521 | C<Script_Extensions> is thus an improved C<Script>, in which there are | |
522 | fewer characters in the C<Common> script, and correspondingly more in | |
523 | other scripts. It is new in Unicode version 6.0, and its data are likely | |
524 | to change significantly in later releases, as things get sorted out. | |
525 | ||
526 | (Actually, besides C<Common>, the C<Inherited> script, contains | |
527 | characters that are used in multiple scripts. These are modifier | |
528 | characters which modify other characters, and inherit the script value | |
529 | of the controlling character. Some of these are used in many scripts, | |
530 | and so go into C<Inherited> in both C<Script> and C<Script_Extensions>. | |
531 | Others are used in just a few scripts, so are in C<Inherited> in | |
532 | C<Script>, but not in C<Script_Extensions>.) | |
533 | ||
534 | It is worth stressing that there are several different sets of digits in | |
535 | Unicode that are equivalent to 0-9 and are matchable by C<\d> in a | |
536 | regular expression. If they are used in a single language only, they | |
537 | are in that language's C<Script> and C<Script_Extension>. If they are | |
538 | used in more than one script, they will be in C<sc=Common>, but only | |
539 | if they are used in many scripts should they be in C<scx=Common>. | |
51f494cc KW |
540 | |
541 | A complete list of scripts and their shortcuts is in L<perluniprops>. | |
542 | ||
51f494cc | 543 | =head3 B<Use of "Is" Prefix> |
822502e5 | 544 | |
1bfb14c4 | 545 | For backward compatibility (with Perl 5.6), all properties mentioned |
51f494cc KW |
546 | so far may have C<Is> or C<Is_> prepended to their name, so C<\P{Is_Lu}>, for |
547 | example, is equal to C<\P{Lu}>, and C<\p{IsScript:Arabic}> is equal to | |
548 | C<\p{Arabic}>. | |
eb0cc9e3 | 549 | |
51f494cc | 550 | =head3 B<Blocks> |
2796c109 | 551 | |
1bfb14c4 JH |
552 | In addition to B<scripts>, Unicode also defines B<blocks> of |
553 | characters. The difference between scripts and blocks is that the | |
554 | concept of scripts is closer to natural languages, while the concept | |
51f494cc | 555 | of blocks is more of an artificial grouping based on groups of Unicode |
9f815e24 | 556 | characters with consecutive ordinal values. For example, the "Basic Latin" |
b19eb496 | 557 | block is all characters whose ordinals are between 0 and 127, inclusive; in |
9f815e24 | 558 | other words, the ASCII characters. The "Latin" script contains some letters |
b19eb496 | 559 | from this as well as several other blocks, like "Latin-1 Supplement", |
9d1c51c1 | 560 | "Latin Extended-A", etc., but it does not contain all the characters from |
7be67b37 | 561 | those blocks. It does not, for example, contain the digits 0-9, because |
82aed44a KW |
562 | those digits are shared across many scripts, and hence are in the |
563 | C<Common> script. | |
51f494cc KW |
564 | |
565 | For more about scripts versus blocks, see UAX#24 "Unicode Script Property": | |
566 | L<http://www.unicode.org/reports/tr24> | |
567 | ||
82aed44a KW |
568 | The C<Script> or C<Script_Extensions> properties are likely to be the |
569 | ones you want to use when processing | |
b19eb496 TC |
570 | natural language; the Block property may occasionally be useful in working |
571 | with the nuts and bolts of Unicode. | |
51f494cc KW |
572 | |
573 | Block names are matched in the compound form, like C<\p{Block: Arrows}> or | |
b19eb496 | 574 | C<\p{Blk=Hebrew}>. Unlike most other properties, only a few block names have a |
51f494cc KW |
575 | Unicode-defined short name. But Perl does provide a (slight) shortcut: You |
576 | can say, for example C<\p{In_Arrows}> or C<\p{In_Hebrew}>. For backwards | |
577 | compatibility, the C<In> prefix may be omitted if there is no naming conflict | |
578 | with a script or any other property, and you can even use an C<Is> prefix | |
579 | instead in those cases. But it is not a good idea to do this, for a couple | |
580 | reasons: | |
581 | ||
582 | =over 4 | |
583 | ||
584 | =item 1 | |
585 | ||
586 | It is confusing. There are many naming conflicts, and you may forget some. | |
9f815e24 | 587 | For example, C<\p{Hebrew}> means the I<script> Hebrew, and NOT the I<block> |
51f494cc KW |
588 | Hebrew. But would you remember that 6 months from now? |
589 | ||
590 | =item 2 | |
591 | ||
3e2dd9ee | 592 | It is unstable. A new version of Unicode may preempt the current meaning by |
51f494cc | 593 | creating a property with the same name. There was a time in very early Unicode |
9f815e24 | 594 | releases when C<\p{Hebrew}> would have matched the I<block> Hebrew; now it |
51f494cc | 595 | doesn't. |
32293815 | 596 | |
393fec97 GS |
597 | =back |
598 | ||
b19eb496 TC |
599 | Some people prefer to always use C<\p{Block: foo}> and C<\p{Script: bar}> |
600 | instead of the shortcuts, whether for clarity, because they can't remember the | |
601 | difference between 'In' and 'Is' anyway, or they aren't confident that those who | |
602 | eventually will read their code will know that difference. | |
51f494cc KW |
603 | |
604 | A complete list of blocks and their shortcuts is in L<perluniprops>. | |
605 | ||
9f815e24 KW |
606 | =head3 B<Other Properties> |
607 | ||
608 | There are many more properties than the very basic ones described here. | |
609 | A complete list is in L<perluniprops>. | |
610 | ||
611 | Unicode defines all its properties in the compound form, so all single-form | |
b19eb496 TC |
612 | properties are Perl extensions. Most of these are just synonyms for the |
613 | Unicode ones, but some are genuine extensions, including several that are in | |
9f815e24 KW |
614 | the compound form. And quite a few of these are actually recommended by Unicode |
615 | (in L<http://www.unicode.org/reports/tr18>). | |
616 | ||
5bff2035 KW |
617 | This section gives some details on all extensions that aren't just |
618 | synonyms for compound-form Unicode properties | |
619 | (for those properties, you'll have to refer to the | |
9f815e24 KW |
620 | L<Unicode Standard|http://www.unicode.org/reports/tr44>. |
621 | ||
622 | =over | |
623 | ||
624 | =item B<C<\p{All}>> | |
625 | ||
626 | This matches any of the 1_114_112 Unicode code points. It is a synonym for | |
627 | C<\p{Any}>. | |
628 | ||
629 | =item B<C<\p{Alnum}>> | |
630 | ||
631 | This matches any C<\p{Alphabetic}> or C<\p{Decimal_Number}> character. | |
632 | ||
633 | =item B<C<\p{Any}>> | |
634 | ||
635 | This matches any of the 1_114_112 Unicode code points. It is a synonym for | |
636 | C<\p{All}>. | |
637 | ||
42581d5d KW |
638 | =item B<C<\p{ASCII}>> |
639 | ||
640 | This matches any of the 128 characters in the US-ASCII character set, | |
641 | which is a subset of Unicode. | |
642 | ||
9f815e24 KW |
643 | =item B<C<\p{Assigned}>> |
644 | ||
645 | This matches any assigned code point; that is, any code point whose general | |
646 | category is not Unassigned (or equivalently, not Cn). | |
647 | ||
648 | =item B<C<\p{Blank}>> | |
649 | ||
650 | This is the same as C<\h> and C<\p{HorizSpace}>: A character that changes the | |
651 | spacing horizontally. | |
652 | ||
653 | =item B<C<\p{Decomposition_Type: Non_Canonical}>> (Short: C<\p{Dt=NonCanon}>) | |
654 | ||
655 | Matches a character that has a non-canonical decomposition. | |
656 | ||
657 | To understand the use of this rarely used property=value combination, it is | |
658 | necessary to know some basics about decomposition. | |
659 | Consider a character, say H. It could appear with various marks around it, | |
660 | such as an acute accent, or a circumflex, or various hooks, circles, arrows, | |
b19eb496 | 661 | I<etc.>, above, below, to one side or the other, etc. There are many |
9f815e24 KW |
662 | possibilities among the world's languages. The number of combinations is |
663 | astronomical, and if there were a character for each combination, it would | |
664 | soon exhaust Unicode's more than a million possible characters. So Unicode | |
665 | took a different approach: there is a character for the base H, and a | |
b19eb496 | 666 | character for each of the possible marks, and these can be variously combined |
9f815e24 KW |
667 | to get a final logical character. So a logical character--what appears to be a |
668 | single character--can be a sequence of more than one individual characters. | |
b19eb496 TC |
669 | This is called an "extended grapheme cluster"; Perl furnishes the C<\X> |
670 | regular expression construct to match such sequences. | |
9f815e24 KW |
671 | |
672 | But Unicode's intent is to unify the existing character set standards and | |
b19eb496 | 673 | practices, and several pre-existing standards have single characters that |
9f815e24 KW |
674 | mean the same thing as some of these combinations. An example is ISO-8859-1, |
675 | which has quite a few of these in the Latin-1 range, an example being "LATIN | |
676 | CAPITAL LETTER E WITH ACUTE". Because this character was in this pre-existing | |
677 | standard, Unicode added it to its repertoire. But this character is considered | |
b19eb496 TC |
678 | by Unicode to be equivalent to the sequence consisting of the character |
679 | "LATIN CAPITAL LETTER E" followed by the character "COMBINING ACUTE ACCENT". | |
9f815e24 KW |
680 | |
681 | "LATIN CAPITAL LETTER E WITH ACUTE" is called a "pre-composed" character, and | |
b19eb496 | 682 | its equivalence with the sequence is called canonical equivalence. All |
9f815e24 | 683 | pre-composed characters are said to have a decomposition (into the equivalent |
b19eb496 | 684 | sequence), and the decomposition type is also called canonical. |
9f815e24 KW |
685 | |
686 | However, many more characters have a different type of decomposition, a | |
687 | "compatible" or "non-canonical" decomposition. The sequences that form these | |
688 | decompositions are not considered canonically equivalent to the pre-composed | |
689 | character. An example, again in the Latin-1 range, is the "SUPERSCRIPT ONE". | |
b19eb496 | 690 | It is somewhat like a regular digit 1, but not exactly; its decomposition |
9f815e24 KW |
691 | into the digit 1 is called a "compatible" decomposition, specifically a |
692 | "super" decomposition. There are several such compatibility | |
693 | decompositions (see L<http://www.unicode.org/reports/tr44>), including one | |
b19eb496 | 694 | called "compat", which means some miscellaneous type of decomposition |
42581d5d | 695 | that doesn't fit into the decomposition categories that Unicode has chosen. |
9f815e24 KW |
696 | |
697 | Note that most Unicode characters don't have a decomposition, so their | |
698 | decomposition type is "None". | |
699 | ||
b19eb496 TC |
700 | For your convenience, Perl has added the C<Non_Canonical> decomposition |
701 | type to mean any of the several compatibility decompositions. | |
9f815e24 KW |
702 | |
703 | =item B<C<\p{Graph}>> | |
704 | ||
705 | Matches any character that is graphic. Theoretically, this means a character | |
706 | that on a printer would cause ink to be used. | |
707 | ||
708 | =item B<C<\p{HorizSpace}>> | |
709 | ||
b19eb496 | 710 | This is the same as C<\h> and C<\p{Blank}>: a character that changes the |
9f815e24 KW |
711 | spacing horizontally. |
712 | ||
42581d5d | 713 | =item B<C<\p{In=*}>> |
9f815e24 KW |
714 | |
715 | This is a synonym for C<\p{Present_In=*}> | |
716 | ||
717 | =item B<C<\p{PerlSpace}>> | |
718 | ||
d28d8023 KW |
719 | This is the same as C<\s>, restricted to ASCII, namely C<S<[ \f\n\r\t]>> |
720 | and starting in Perl v5.18, experimentally, a vertical tab. | |
9f815e24 KW |
721 | |
722 | Mnemonic: Perl's (original) space | |
723 | ||
724 | =item B<C<\p{PerlWord}>> | |
725 | ||
726 | This is the same as C<\w>, restricted to ASCII, namely C<[A-Za-z0-9_]> | |
727 | ||
728 | Mnemonic: Perl's (original) word. | |
729 | ||
42581d5d | 730 | =item B<C<\p{Posix...}>> |
9f815e24 | 731 | |
b19eb496 TC |
732 | There are several of these, which are equivalents using the C<\p> |
733 | notation for Posix classes and are described in | |
42581d5d | 734 | L<perlrecharclass/POSIX Character Classes>. |
9f815e24 KW |
735 | |
736 | =item B<C<\p{Present_In: *}>> (Short: C<\p{In=*}>) | |
737 | ||
738 | This property is used when you need to know in what Unicode version(s) a | |
739 | character is. | |
740 | ||
741 | The "*" above stands for some two digit Unicode version number, such as | |
742 | C<1.1> or C<4.0>; or the "*" can also be C<Unassigned>. This property will | |
743 | match the code points whose final disposition has been settled as of the | |
744 | Unicode release given by the version number; C<\p{Present_In: Unassigned}> | |
745 | will match those code points whose meaning has yet to be assigned. | |
746 | ||
747 | For example, C<U+0041> "LATIN CAPITAL LETTER A" was present in the very first | |
748 | Unicode release available, which is C<1.1>, so this property is true for all | |
749 | valid "*" versions. On the other hand, C<U+1EFF> was not assigned until version | |
750 | 5.1 when it became "LATIN SMALL LETTER Y WITH LOOP", so the only "*" that | |
751 | would match it are 5.1, 5.2, and later. | |
752 | ||
753 | Unicode furnishes the C<Age> property from which this is derived. The problem | |
754 | with Age is that a strict interpretation of it (which Perl takes) has it | |
755 | matching the precise release a code point's meaning is introduced in. Thus | |
756 | C<U+0041> would match only 1.1; and C<U+1EFF> only 5.1. This is not usually what | |
757 | you want. | |
758 | ||
759 | Some non-Perl implementations of the Age property may change its meaning to be | |
760 | the same as the Perl Present_In property; just be aware of that. | |
761 | ||
762 | Another confusion with both these properties is that the definition is not | |
b19eb496 TC |
763 | that the code point has been I<assigned>, but that the meaning of the code point |
764 | has been I<determined>. This is because 66 code points will always be | |
765 | unassigned, and so the Age for them is the Unicode version in which the decision | |
766 | to make them so was made. For example, C<U+FDD0> is to be permanently | |
9f815e24 | 767 | unassigned to a character, and the decision to do that was made in version 3.1, |
b19eb496 | 768 | so C<\p{Age=3.1}> matches this character, as also does C<\p{Present_In: 3.1}> and up. |
9f815e24 KW |
769 | |
770 | =item B<C<\p{Print}>> | |
771 | ||
ae5b72c8 | 772 | This matches any character that is graphical or blank, except controls. |
9f815e24 KW |
773 | |
774 | =item B<C<\p{SpacePerl}>> | |
775 | ||
776 | This is the same as C<\s>, including beyond ASCII. | |
777 | ||
4d4acfba | 778 | Mnemonic: Space, as modified by Perl. (It doesn't include the vertical tab |
b19eb496 | 779 | which both the Posix standard and Unicode consider white space.) |
9f815e24 | 780 | |
4364919a KW |
781 | =item B<C<\p{Title}>> and B<C<\p{Titlecase}>> |
782 | ||
783 | Under case-sensitive matching, these both match the same code points as | |
784 | C<\p{General Category=Titlecase_Letter}> (C<\p{gc=lt}>). The difference | |
785 | is that under C</i> caseless matching, these match the same as | |
786 | C<\p{Cased}>, whereas C<\p{gc=lt}> matches C<\p{Cased_Letter>). | |
787 | ||
9f815e24 KW |
788 | =item B<C<\p{VertSpace}>> |
789 | ||
790 | This is the same as C<\v>: A character that changes the spacing vertically. | |
791 | ||
792 | =item B<C<\p{Word}>> | |
793 | ||
b19eb496 | 794 | This is the same as C<\w>, including over 100_000 characters beyond ASCII. |
9f815e24 | 795 | |
42581d5d KW |
796 | =item B<C<\p{XPosix...}>> |
797 | ||
b19eb496 | 798 | There are several of these, which are the standard Posix classes |
42581d5d KW |
799 | extended to the full Unicode range. They are described in |
800 | L<perlrecharclass/POSIX Character Classes>. | |
801 | ||
9f815e24 KW |
802 | =back |
803 | ||
376d9008 | 804 | =head2 User-Defined Character Properties |
491fd90a | 805 | |
51f494cc | 806 | You can define your own binary character properties by defining subroutines |
9d1a5160 KW |
807 | whose names begin with "In" or "Is". (The experimental feature |
808 | L<perlre/(?[ ])> provides an alternative which allows more complex | |
809 | definitions.) The subroutines can be defined in any | |
51f494cc KW |
810 | package. The user-defined properties can be used in the regular expression |
811 | C<\p> and C<\P> constructs; if you are using a user-defined property from a | |
812 | package other than the one you are in, you must specify its package in the | |
813 | C<\p> or C<\P> construct. | |
bac0b425 | 814 | |
51f494cc | 815 | # assuming property Is_Foreign defined in Lang:: |
bac0b425 JP |
816 | package main; # property package name required |
817 | if ($txt =~ /\p{Lang::IsForeign}+/) { ... } | |
818 | ||
819 | package Lang; # property package name not required | |
820 | if ($txt =~ /\p{IsForeign}+/) { ... } | |
821 | ||
822 | ||
823 | Note that the effect is compile-time and immutable once defined. | |
b19eb496 TC |
824 | However, the subroutines are passed a single parameter, which is 0 if |
825 | case-sensitive matching is in effect and non-zero if caseless matching | |
56ca34ca KW |
826 | is in effect. The subroutine may return different values depending on |
827 | the value of the flag, and one set of values will immutably be in effect | |
b19eb496 | 828 | for all case-sensitive matches, and the other set for all case-insensitive |
56ca34ca | 829 | matches. |
491fd90a | 830 | |
b19eb496 | 831 | Note that if the regular expression is tainted, then Perl will die rather |
0e9be77f DM |
832 | than calling the subroutine, where the name of the subroutine is |
833 | determined by the tainted data. | |
834 | ||
376d9008 JB |
835 | The subroutines must return a specially-formatted string, with one |
836 | or more newline-separated lines. Each line must be one of the following: | |
491fd90a JH |
837 | |
838 | =over 4 | |
839 | ||
840 | =item * | |
841 | ||
510254c9 A |
842 | A single hexadecimal number denoting a Unicode code point to include. |
843 | ||
844 | =item * | |
845 | ||
99a6b1f0 | 846 | Two hexadecimal numbers separated by horizontal whitespace (space or |
376d9008 | 847 | tabular characters) denoting a range of Unicode code points to include. |
491fd90a JH |
848 | |
849 | =item * | |
850 | ||
376d9008 | 851 | Something to include, prefixed by "+": a built-in character |
830137a2 KW |
852 | property (prefixed by "utf8::") or a fully qualified (including package |
853 | name) user-defined character property, | |
bac0b425 JP |
854 | to represent all the characters in that property; two hexadecimal code |
855 | points for a range; or a single hexadecimal code point. | |
491fd90a JH |
856 | |
857 | =item * | |
858 | ||
376d9008 | 859 | Something to exclude, prefixed by "-": an existing character |
830137a2 KW |
860 | property (prefixed by "utf8::") or a fully qualified (including package |
861 | name) user-defined character property, | |
bac0b425 JP |
862 | to represent all the characters in that property; two hexadecimal code |
863 | points for a range; or a single hexadecimal code point. | |
491fd90a JH |
864 | |
865 | =item * | |
866 | ||
376d9008 | 867 | Something to negate, prefixed "!": an existing character |
830137a2 KW |
868 | property (prefixed by "utf8::") or a fully qualified (including package |
869 | name) user-defined character property, | |
bac0b425 JP |
870 | to represent all the characters in that property; two hexadecimal code |
871 | points for a range; or a single hexadecimal code point. | |
872 | ||
873 | =item * | |
874 | ||
875 | Something to intersect with, prefixed by "&": an existing character | |
830137a2 KW |
876 | property (prefixed by "utf8::") or a fully qualified (including package |
877 | name) user-defined character property, | |
bac0b425 JP |
878 | for all the characters except the characters in the property; two |
879 | hexadecimal code points for a range; or a single hexadecimal code point. | |
491fd90a JH |
880 | |
881 | =back | |
882 | ||
883 | For example, to define a property that covers both the Japanese | |
884 | syllabaries (hiragana and katakana), you can define | |
885 | ||
886 | sub InKana { | |
d88362ca | 887 | return <<END; |
d5822f25 A |
888 | 3040\t309F |
889 | 30A0\t30FF | |
491fd90a JH |
890 | END |
891 | } | |
892 | ||
d5822f25 A |
893 | Imagine that the here-doc end marker is at the beginning of the line. |
894 | Now you can use C<\p{InKana}> and C<\P{InKana}>. | |
491fd90a JH |
895 | |
896 | You could also have used the existing block property names: | |
897 | ||
898 | sub InKana { | |
d88362ca | 899 | return <<'END'; |
491fd90a JH |
900 | +utf8::InHiragana |
901 | +utf8::InKatakana | |
902 | END | |
903 | } | |
904 | ||
905 | Suppose you wanted to match only the allocated characters, | |
d5822f25 | 906 | not the raw block ranges: in other words, you want to remove |
491fd90a JH |
907 | the non-characters: |
908 | ||
909 | sub InKana { | |
d88362ca | 910 | return <<'END'; |
491fd90a JH |
911 | +utf8::InHiragana |
912 | +utf8::InKatakana | |
913 | -utf8::IsCn | |
914 | END | |
915 | } | |
916 | ||
917 | The negation is useful for defining (surprise!) negated classes. | |
918 | ||
919 | sub InNotKana { | |
d88362ca | 920 | return <<'END'; |
491fd90a JH |
921 | !utf8::InHiragana |
922 | -utf8::InKatakana | |
923 | +utf8::IsCn | |
924 | END | |
925 | } | |
926 | ||
461020ad KW |
927 | This will match all non-Unicode code points, since every one of them is |
928 | not in Kana. You can use intersection to exclude these, if desired, as | |
929 | this modified example shows: | |
bac0b425 | 930 | |
461020ad | 931 | sub InNotKana { |
bac0b425 | 932 | return <<'END'; |
461020ad KW |
933 | !utf8::InHiragana |
934 | -utf8::InKatakana | |
935 | +utf8::IsCn | |
936 | &utf8::Any | |
bac0b425 JP |
937 | END |
938 | } | |
939 | ||
461020ad KW |
940 | C<&utf8::Any> must be the last line in the definition. |
941 | ||
942 | Intersection is used generally for getting the common characters matched | |
943 | by two (or more) classes. It's important to remember not to use "&" for | |
944 | the first set; that would be intersecting with nothing, resulting in an | |
945 | empty set. | |
946 | ||
947 | (Note that official Unicode properties differ from these in that they | |
948 | automatically exclude non-Unicode code points and a warning is raised if | |
949 | a match is attempted on one of those.) | |
bac0b425 | 950 | |
68585b5e | 951 | =head2 User-Defined Case Mappings (for serious hackers only) |
822502e5 | 952 | |
5d1892be KW |
953 | B<This feature has been removed as of Perl 5.16.> |
954 | The CPAN module L<Unicode::Casing> provides better functionality without | |
955 | the drawbacks that this feature had. If you are using a Perl earlier | |
956 | than 5.16, this feature was most fully documented in the 5.14 version of | |
957 | this pod: | |
958 | L<http://perldoc.perl.org/5.14.0/perlunicode.html#User-Defined-Case-Mappings-%28for-serious-hackers-only%29> | |
3a2263fe | 959 | |
376d9008 | 960 | =head2 Character Encodings for Input and Output |
8cbd9a7a | 961 | |
7221edc9 | 962 | See L<Encode>. |
8cbd9a7a | 963 | |
c29a771d | 964 | =head2 Unicode Regular Expression Support Level |
776f8809 | 965 | |
b19eb496 TC |
966 | The following list of Unicode supported features for regular expressions describes |
967 | all features currently directly supported by core Perl. The references to "Level N" | |
8158862b | 968 | and the section numbers refer to the Unicode Technical Standard #18, |
b19eb496 | 969 | "Unicode Regular Expressions", version 13, from August 2008. |
776f8809 JH |
970 | |
971 | =over 4 | |
972 | ||
973 | =item * | |
974 | ||
975 | Level 1 - Basic Unicode Support | |
976 | ||
755789c0 KW |
977 | RL1.1 Hex Notation - done [1] |
978 | RL1.2 Properties - done [2][3] | |
979 | RL1.2a Compatibility Properties - done [4] | |
9d1a5160 | 980 | RL1.3 Subtraction and Intersection - experimental [5] |
755789c0 KW |
981 | RL1.4 Simple Word Boundaries - done [6] |
982 | RL1.5 Simple Loose Matches - done [7] | |
983 | RL1.6 Line Boundaries - MISSING [8][9] | |
984 | RL1.7 Supplementary Code Points - done [10] | |
985 | ||
6f33e417 KW |
986 | =over 4 |
987 | ||
988 | =item [1] | |
989 | ||
990 | \x{...} | |
991 | ||
992 | =item [2] | |
993 | ||
994 | \p{...} \P{...} | |
995 | ||
996 | =item [3] | |
997 | ||
998 | supports not only minimal list, but all Unicode character properties (see Unicode Character Properties above) | |
999 | ||
1000 | =item [4] | |
1001 | ||
1002 | \d \D \s \S \w \W \X [:prop:] [:^prop:] | |
1003 | ||
1004 | =item [5] | |
1005 | ||
9d1a5160 KW |
1006 | The experimental feature in v5.18 "(?[...])" accomplishes this. See |
1007 | L<perlre/(?[ ])>. If you don't want to use an experimental feature, | |
1008 | you can use one of the following: | |
6f33e417 KW |
1009 | |
1010 | =over 4 | |
1011 | ||
1012 | =item * Regular expression look-ahead | |
1013 | ||
1014 | You can mimic class subtraction using lookahead. | |
8158862b | 1015 | For example, what UTS#18 might write as |
29bdacb8 | 1016 | |
209c9685 | 1017 | [{Block=Greek}-[{UNASSIGNED}]] |
dbe420b4 JH |
1018 | |
1019 | in Perl can be written as: | |
1020 | ||
209c9685 KW |
1021 | (?!\p{Unassigned})\p{Block=Greek} |
1022 | (?=\p{Assigned})\p{Block=Greek} | |
dbe420b4 JH |
1023 | |
1024 | But in this particular example, you probably really want | |
1025 | ||
209c9685 | 1026 | \p{Greek} |
dbe420b4 JH |
1027 | |
1028 | which will match assigned characters known to be part of the Greek script. | |
29bdacb8 | 1029 | |
6f33e417 | 1030 | =item * CPAN module L<Unicode::Regex::Set> |
8158862b | 1031 | |
6f33e417 KW |
1032 | It does implement the full UTS#18 grouping, intersection, union, and |
1033 | removal (subtraction) syntax. | |
8158862b | 1034 | |
6f33e417 KW |
1035 | =item * L</"User-Defined Character Properties"> |
1036 | ||
1037 | '+' for union, '-' for removal (set-difference), '&' for intersection | |
1038 | ||
1039 | =back | |
1040 | ||
1041 | =item [6] | |
1042 | ||
1043 | \b \B | |
1044 | ||
1045 | =item [7] | |
1046 | ||
1047 | Note that Perl does Full case-folding in matching (but with bugs), not Simple: for example U+1F88 is equivalent to U+1F00 U+03B9, instead of just U+1F80. This difference matters mainly for certain Greek capital letters with certain modifiers: the Full case-folding decomposes the letter, while the Simple case-folding would map it to a single character. | |
1048 | ||
1049 | =item [8] | |
1050 | ||
1051 | Should do ^ and $ also on U+000B (\v in C), FF (\f), CR (\r), CRLF | |
1052 | (\r\n), NEL (U+0085), LS (U+2028), and PS (U+2029); should also affect | |
1053 | <>, $., and script line numbers; should not split lines within CRLF | |
1054 | (i.e. there is no empty line between \r and \n). For CRLF, try the | |
1055 | C<:crlf> layer (see L<PerlIO>). | |
1056 | ||
1057 | =item [9] | |
1058 | ||
1059 | Linebreaking conformant with UAX#14 "Unicode Line Breaking Algorithm" is available through the Unicode::LineBreaking module. | |
1060 | ||
1061 | =item [10] | |
1062 | ||
1063 | UTF-8/UTF-EBDDIC used in Perl allows not only U+10000 to | |
1064 | U+10FFFF but also beyond U+10FFFF | |
1065 | ||
1066 | =back | |
5ca1ac52 | 1067 | |
776f8809 JH |
1068 | =item * |
1069 | ||
1070 | Level 2 - Extended Unicode Support | |
1071 | ||
755789c0 KW |
1072 | RL2.1 Canonical Equivalents - MISSING [10][11] |
1073 | RL2.2 Default Grapheme Clusters - MISSING [12] | |
1074 | RL2.3 Default Word Boundaries - MISSING [14] | |
1075 | RL2.4 Default Loose Matches - MISSING [15] | |
1076 | RL2.5 Name Properties - DONE | |
1077 | RL2.6 Wildcard Properties - MISSING | |
8158862b | 1078 | |
755789c0 KW |
1079 | [10] see UAX#15 "Unicode Normalization Forms" |
1080 | [11] have Unicode::Normalize but not integrated to regexes | |
1081 | [12] have \X but we don't have a "Grapheme Cluster Mode" | |
1082 | [14] see UAX#29, Word Boundaries | |
902b08d0 | 1083 | [15] This is covered in Chapter 3.13 (in Unicode 6.0) |
776f8809 JH |
1084 | |
1085 | =item * | |
1086 | ||
8158862b TS |
1087 | Level 3 - Tailored Support |
1088 | ||
755789c0 KW |
1089 | RL3.1 Tailored Punctuation - MISSING |
1090 | RL3.2 Tailored Grapheme Clusters - MISSING [17][18] | |
1091 | RL3.3 Tailored Word Boundaries - MISSING | |
1092 | RL3.4 Tailored Loose Matches - MISSING | |
1093 | RL3.5 Tailored Ranges - MISSING | |
1094 | RL3.6 Context Matching - MISSING [19] | |
1095 | RL3.7 Incremental Matches - MISSING | |
8158862b | 1096 | ( RL3.8 Unicode Set Sharing ) |
755789c0 KW |
1097 | RL3.9 Possible Match Sets - MISSING |
1098 | RL3.10 Folded Matching - MISSING [20] | |
1099 | RL3.11 Submatchers - MISSING | |
1100 | ||
1101 | [17] see UAX#10 "Unicode Collation Algorithms" | |
1102 | [18] have Unicode::Collate but not integrated to regexes | |
1103 | [19] have (?<=x) and (?=x), but look-aheads or look-behinds | |
1104 | should see outside of the target substring | |
1105 | [20] need insensitive matching for linguistic features other | |
1106 | than case; for example, hiragana to katakana, wide and | |
1107 | narrow, simplified Han to traditional Han (see UTR#30 | |
1108 | "Character Foldings") | |
776f8809 JH |
1109 | |
1110 | =back | |
1111 | ||
c349b1b9 JH |
1112 | =head2 Unicode Encodings |
1113 | ||
376d9008 JB |
1114 | Unicode characters are assigned to I<code points>, which are abstract |
1115 | numbers. To use these numbers, various encodings are needed. | |
c349b1b9 JH |
1116 | |
1117 | =over 4 | |
1118 | ||
c29a771d | 1119 | =item * |
5cb3728c RB |
1120 | |
1121 | UTF-8 | |
c349b1b9 | 1122 | |
6d4f9cf2 KW |
1123 | UTF-8 is a variable-length (1 to 4 bytes), byte-order independent |
1124 | encoding. For ASCII (and we really do mean 7-bit ASCII, not another | |
1125 | 8-bit encoding), UTF-8 is transparent. | |
c349b1b9 | 1126 | |
8c007b5a | 1127 | The following table is from Unicode 3.2. |
05632f9a | 1128 | |
755789c0 | 1129 | Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte |
05632f9a | 1130 | |
d88362ca | 1131 | U+0000..U+007F 00..7F |
e1b711da | 1132 | U+0080..U+07FF * C2..DF 80..BF |
d88362ca | 1133 | U+0800..U+0FFF E0 * A0..BF 80..BF |
ec90690f TS |
1134 | U+1000..U+CFFF E1..EC 80..BF 80..BF |
1135 | U+D000..U+D7FF ED 80..9F 80..BF | |
755789c0 | 1136 | U+D800..U+DFFF +++++ utf16 surrogates, not legal utf8 +++++ |
ec90690f | 1137 | U+E000..U+FFFF EE..EF 80..BF 80..BF |
d88362ca KW |
1138 | U+10000..U+3FFFF F0 * 90..BF 80..BF 80..BF |
1139 | U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF | |
1140 | U+100000..U+10FFFF F4 80..8F 80..BF 80..BF | |
e1b711da | 1141 | |
b19eb496 | 1142 | Note the gaps marked by "*" before several of the byte entries above. These are |
e1b711da KW |
1143 | caused by legal UTF-8 avoiding non-shortest encodings: it is technically |
1144 | possible to UTF-8-encode a single code point in different ways, but that is | |
1145 | explicitly forbidden, and the shortest possible encoding should always be used | |
1146 | (and that is what Perl does). | |
37361303 | 1147 | |
376d9008 | 1148 | Another way to look at it is via bits: |
05632f9a | 1149 | |
755789c0 | 1150 | Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte |
05632f9a | 1151 | |
755789c0 KW |
1152 | 0aaaaaaa 0aaaaaaa |
1153 | 00000bbbbbaaaaaa 110bbbbb 10aaaaaa | |
1154 | ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa | |
1155 | 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa | |
05632f9a | 1156 | |
9f815e24 | 1157 | As you can see, the continuation bytes all begin with "10", and the |
e1b711da | 1158 | leading bits of the start byte tell how many bytes there are in the |
05632f9a JH |
1159 | encoded character. |
1160 | ||
6d4f9cf2 KW |
1161 | The original UTF-8 specification allowed up to 6 bytes, to allow |
1162 | encoding of numbers up to 0x7FFF_FFFF. Perl continues to allow those, | |
1163 | and has extended that up to 13 bytes to encode code points up to what | |
1164 | can fit in a 64-bit word. However, Perl will warn if you output any of | |
b19eb496 | 1165 | these as being non-portable; and under strict UTF-8 input protocols, |
6d4f9cf2 KW |
1166 | they are forbidden. |
1167 | ||
1168 | The Unicode non-character code points are also disallowed in UTF-8 in | |
1169 | "open interchange". See L</Non-character code points>. | |
1170 | ||
c29a771d | 1171 | =item * |
5cb3728c RB |
1172 | |
1173 | UTF-EBCDIC | |
dbe420b4 | 1174 | |
376d9008 | 1175 | Like UTF-8 but EBCDIC-safe, in the way that UTF-8 is ASCII-safe. |
dbe420b4 | 1176 | |
c29a771d | 1177 | =item * |
5cb3728c | 1178 | |
1e54db1a | 1179 | UTF-16, UTF-16BE, UTF-16LE, Surrogates, and BOMs (Byte Order Marks) |
c349b1b9 | 1180 | |
1bfb14c4 JH |
1181 | The followings items are mostly for reference and general Unicode |
1182 | knowledge, Perl doesn't use these constructs internally. | |
dbe420b4 | 1183 | |
b19eb496 TC |
1184 | Like UTF-8, UTF-16 is a variable-width encoding, but where |
1185 | UTF-8 uses 8-bit code units, UTF-16 uses 16-bit code units. | |
1186 | All code points occupy either 2 or 4 bytes in UTF-16: code points | |
1187 | C<U+0000..U+FFFF> are stored in a single 16-bit unit, and code | |
1bfb14c4 | 1188 | points C<U+10000..U+10FFFF> in two 16-bit units. The latter case is |
c349b1b9 JH |
1189 | using I<surrogates>, the first 16-bit unit being the I<high |
1190 | surrogate>, and the second being the I<low surrogate>. | |
1191 | ||
376d9008 | 1192 | Surrogates are code points set aside to encode the C<U+10000..U+10FFFF> |
c349b1b9 | 1193 | range of Unicode code points in pairs of 16-bit units. The I<high |
9f815e24 | 1194 | surrogates> are the range C<U+D800..U+DBFF> and the I<low surrogates> |
376d9008 | 1195 | are the range C<U+DC00..U+DFFF>. The surrogate encoding is |
c349b1b9 | 1196 | |
d88362ca KW |
1197 | $hi = ($uni - 0x10000) / 0x400 + 0xD800; |
1198 | $lo = ($uni - 0x10000) % 0x400 + 0xDC00; | |
c349b1b9 JH |
1199 | |
1200 | and the decoding is | |
1201 | ||
d88362ca | 1202 | $uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00); |
c349b1b9 | 1203 | |
376d9008 | 1204 | Because of the 16-bitness, UTF-16 is byte-order dependent. UTF-16 |
c349b1b9 | 1205 | itself can be used for in-memory computations, but if storage or |
376d9008 JB |
1206 | transfer is required either UTF-16BE (big-endian) or UTF-16LE |
1207 | (little-endian) encodings must be chosen. | |
c349b1b9 JH |
1208 | |
1209 | This introduces another problem: what if you just know that your data | |
376d9008 JB |
1210 | is UTF-16, but you don't know which endianness? Byte Order Marks, or |
1211 | BOMs, are a solution to this. A special character has been reserved | |
86bbd6d1 | 1212 | in Unicode to function as a byte order marker: the character with the |
376d9008 | 1213 | code point C<U+FEFF> is the BOM. |
042da322 | 1214 | |
c349b1b9 | 1215 | The trick is that if you read a BOM, you will know the byte order, |
376d9008 JB |
1216 | since if it was written on a big-endian platform, you will read the |
1217 | bytes C<0xFE 0xFF>, but if it was written on a little-endian platform, | |
1218 | you will read the bytes C<0xFF 0xFE>. (And if the originating platform | |
1219 | was writing in UTF-8, you will read the bytes C<0xEF 0xBB 0xBF>.) | |
042da322 | 1220 | |
86bbd6d1 | 1221 | The way this trick works is that the character with the code point |
6d4f9cf2 | 1222 | C<U+FFFE> is not supposed to be in input streams, so the |
376d9008 | 1223 | sequence of bytes C<0xFF 0xFE> is unambiguously "BOM, represented in |
1bfb14c4 | 1224 | little-endian format" and cannot be C<U+FFFE>, represented in big-endian |
6d4f9cf2 KW |
1225 | format". |
1226 | ||
1227 | Surrogates have no meaning in Unicode outside their use in pairs to | |
1228 | represent other code points. However, Perl allows them to be | |
1229 | represented individually internally, for example by saying | |
f651977e TC |
1230 | C<chr(0xD801)>, so that all code points, not just those valid for open |
1231 | interchange, are | |
6d4f9cf2 KW |
1232 | representable. Unicode does define semantics for them, such as their |
1233 | General Category is "Cs". But because their use is somewhat dangerous, | |
b19eb496 TC |
1234 | Perl will warn (using the warning category "surrogate", which is a |
1235 | sub-category of "utf8") if an attempt is made | |
6d4f9cf2 KW |
1236 | to do things like take the lower case of one, or match |
1237 | case-insensitively, or to output them. (But don't try this on Perls | |
1238 | before 5.14.) | |
c349b1b9 | 1239 | |
c29a771d | 1240 | =item * |
5cb3728c | 1241 | |
1e54db1a | 1242 | UTF-32, UTF-32BE, UTF-32LE |
c349b1b9 JH |
1243 | |
1244 | The UTF-32 family is pretty much like the UTF-16 family, expect that | |
042da322 | 1245 | the units are 32-bit, and therefore the surrogate scheme is not |
f651977e | 1246 | needed. UTF-32 is a fixed-width encoding. The BOM signatures are |
b19eb496 | 1247 | C<0x00 0x00 0xFE 0xFF> for BE and C<0xFF 0xFE 0x00 0x00> for LE. |
c349b1b9 | 1248 | |
c29a771d | 1249 | =item * |
5cb3728c RB |
1250 | |
1251 | UCS-2, UCS-4 | |
c349b1b9 | 1252 | |
b19eb496 | 1253 | Legacy, fixed-width encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit |
376d9008 | 1254 | encoding. Unlike UTF-16, UCS-2 is not extensible beyond C<U+FFFF>, |
339cfa0e | 1255 | because it does not use surrogates. UCS-4 is a 32-bit encoding, |
b19eb496 | 1256 | functionally identical to UTF-32 (the difference being that |
ee88f7b6 | 1257 | UCS-4 forbids neither surrogates nor code points larger than 0x10_FFFF). |
c349b1b9 | 1258 | |
c29a771d | 1259 | =item * |
5cb3728c RB |
1260 | |
1261 | UTF-7 | |
c349b1b9 | 1262 | |
376d9008 JB |
1263 | A seven-bit safe (non-eight-bit) encoding, which is useful if the |
1264 | transport or storage is not eight-bit safe. Defined by RFC 2152. | |
c349b1b9 | 1265 | |
95a1a48b JH |
1266 | =back |
1267 | ||
6d4f9cf2 KW |
1268 | =head2 Non-character code points |
1269 | ||
1270 | 66 code points are set aside in Unicode as "non-character code points". | |
1271 | These all have the Unassigned (Cn) General Category, and they never will | |
1272 | be assigned. These are never supposed to be in legal Unicode input | |
1273 | streams, so that code can use them as sentinels that can be mixed in | |
1274 | with character data, and they always will be distinguishable from that data. | |
1275 | To keep them out of Perl input streams, strict UTF-8 should be | |
1276 | specified, such as by using the layer C<:encoding('UTF-8')>. The | |
1277 | non-character code points are the 32 between U+FDD0 and U+FDEF, and the | |
1278 | 34 code points U+FFFE, U+FFFF, U+1FFFE, U+1FFFF, ... U+10FFFE, U+10FFFF. | |
1279 | Some people are under the mistaken impression that these are "illegal", | |
1280 | but that is not true. An application or cooperating set of applications | |
1281 | can legally use them at will internally; but these code points are | |
42581d5d KW |
1282 | "illegal for open interchange". Therefore, Perl will not accept these |
1283 | from input streams unless lax rules are being used, and will warn | |
b19eb496 | 1284 | (using the warning category "nonchar", which is a sub-category of "utf8") if |
42581d5d KW |
1285 | an attempt is made to output them. |
1286 | ||
1287 | =head2 Beyond Unicode code points | |
1288 | ||
1289 | The maximum Unicode code point is U+10FFFF. But Perl accepts code | |
1290 | points up to the maximum permissible unsigned number available on the | |
1291 | platform. However, Perl will not accept these from input streams unless | |
1292 | lax rules are being used, and will warn (using the warning category | |
b19eb496 | 1293 | "non_unicode", which is a sub-category of "utf8") if an attempt is made to |
42581d5d KW |
1294 | operate on or output them. For example, C<uc(0x11_0000)> will generate |
1295 | this warning, returning the input parameter as its result, as the upper | |
ee88f7b6 | 1296 | case of every non-Unicode code point is the code point itself. |
6d4f9cf2 | 1297 | |
0d7c09bb JH |
1298 | =head2 Security Implications of Unicode |
1299 | ||
e1b711da KW |
1300 | Read L<Unicode Security Considerations|http://www.unicode.org/reports/tr36>. |
1301 | Also, note the following: | |
1302 | ||
0d7c09bb JH |
1303 | =over 4 |
1304 | ||
1305 | =item * | |
1306 | ||
1307 | Malformed UTF-8 | |
bf0fa0b2 | 1308 | |
42581d5d | 1309 | Unfortunately, the original specification of UTF-8 leaves some room for |
bf0fa0b2 | 1310 | interpretation of how many bytes of encoded output one should generate |
376d9008 JB |
1311 | from one input Unicode character. Strictly speaking, the shortest |
1312 | possible sequence of UTF-8 bytes should be generated, | |
1313 | because otherwise there is potential for an input buffer overflow at | |
feda178f | 1314 | the receiving end of a UTF-8 connection. Perl always generates the |
e1b711da | 1315 | shortest length UTF-8, and with warnings on, Perl will warn about |
376d9008 | 1316 | non-shortest length UTF-8 along with other malformations, such as the |
b19eb496 | 1317 | surrogates, which are not Unicode code points valid for interchange. |
bf0fa0b2 | 1318 | |
0d7c09bb JH |
1319 | =item * |
1320 | ||
68693f9e | 1321 | Regular expression pattern matching may surprise you if you're not |
b19eb496 TC |
1322 | accustomed to Unicode. Starting in Perl 5.14, several pattern |
1323 | modifiers are available to control this, called the character set | |
42581d5d KW |
1324 | modifiers. Details are given in L<perlre/Character set modifiers>. |
1325 | ||
1326 | =back | |
0d7c09bb | 1327 | |
376d9008 | 1328 | As discussed elsewhere, Perl has one foot (two hooves?) planted in |
1bfb14c4 JH |
1329 | each of two worlds: the old world of bytes and the new world of |
1330 | characters, upgrading from bytes to characters when necessary. | |
376d9008 JB |
1331 | If your legacy code does not explicitly use Unicode, no automatic |
1332 | switch-over to characters should happen. Characters shouldn't get | |
1bfb14c4 JH |
1333 | downgraded to bytes, either. It is possible to accidentally mix bytes |
1334 | and characters, however (see L<perluniintro>), in which case C<\w> in | |
42581d5d | 1335 | regular expressions might start behaving differently (unless the C</a> |
b19eb496 | 1336 | modifier is in effect). Review your code. Use warnings and the C<strict> pragma. |
0d7c09bb | 1337 | |
c349b1b9 JH |
1338 | =head2 Unicode in Perl on EBCDIC |
1339 | ||
376d9008 JB |
1340 | The way Unicode is handled on EBCDIC platforms is still |
1341 | experimental. On such platforms, references to UTF-8 encoding in this | |
1342 | document and elsewhere should be read as meaning the UTF-EBCDIC | |
1343 | specified in Unicode Technical Report 16, unless ASCII vs. EBCDIC issues | |
c349b1b9 | 1344 | are specifically discussed. There is no C<utfebcdic> pragma or |
376d9008 | 1345 | ":utfebcdic" layer; rather, "utf8" and ":utf8" are reused to mean |
86bbd6d1 PN |
1346 | the platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic> |
1347 | for more discussion of the issues. | |
c349b1b9 | 1348 | |
b310b053 JH |
1349 | =head2 Locales |
1350 | ||
42581d5d | 1351 | See L<perllocale/Unicode and UTF-8> |
b310b053 | 1352 | |
1aad1664 JH |
1353 | =head2 When Unicode Does Not Happen |
1354 | ||
1355 | While Perl does have extensive ways to input and output in Unicode, | |
b19eb496 TC |
1356 | and a few other "entry points" like the @ARGV array (which can sometimes be |
1357 | interpreted as UTF-8), there are still many places where Unicode | |
1358 | (in some encoding or another) could be given as arguments or received as | |
1aad1664 JH |
1359 | results, or both, but it is not. |
1360 | ||
e1b711da KW |
1361 | The following are such interfaces. Also, see L</The "Unicode Bug">. |
1362 | For all of these interfaces Perl | |
b9cedb1b | 1363 | currently (as of v5.16.0) simply assumes byte strings both as arguments |
b19eb496 | 1364 | and results, or UTF-8 strings if the (problematic) C<encoding> pragma has been used. |
1aad1664 | 1365 | |
b19eb496 TC |
1366 | One reason that Perl does not attempt to resolve the role of Unicode in |
1367 | these situations is that the answers are highly dependent on the operating | |
1aad1664 | 1368 | system and the file system(s). For example, whether filenames can be |
b19eb496 TC |
1369 | in Unicode and in exactly what kind of encoding, is not exactly a |
1370 | portable concept. Similarly for C<qx> and C<system>: how well will the | |
1371 | "command-line interface" (and which of them?) handle Unicode? | |
1aad1664 JH |
1372 | |
1373 | =over 4 | |
1374 | ||
557a2462 RB |
1375 | =item * |
1376 | ||
51f494cc | 1377 | chdir, chmod, chown, chroot, exec, link, lstat, mkdir, |
1e8e8236 | 1378 | rename, rmdir, stat, symlink, truncate, unlink, utime, -X |
557a2462 RB |
1379 | |
1380 | =item * | |
1381 | ||
1382 | %ENV | |
1383 | ||
1384 | =item * | |
1385 | ||
1386 | glob (aka the <*>) | |
1387 | ||
1388 | =item * | |
1aad1664 | 1389 | |
557a2462 | 1390 | open, opendir, sysopen |
1aad1664 | 1391 | |
557a2462 | 1392 | =item * |
1aad1664 | 1393 | |
557a2462 | 1394 | qx (aka the backtick operator), system |
1aad1664 | 1395 | |
557a2462 | 1396 | =item * |
1aad1664 | 1397 | |
557a2462 | 1398 | readdir, readlink |
1aad1664 JH |
1399 | |
1400 | =back | |
1401 | ||
e1b711da KW |
1402 | =head2 The "Unicode Bug" |
1403 | ||
2e2b2571 | 1404 | The term, "Unicode bug" has been applied to an inconsistency |
42581d5d KW |
1405 | on ASCII platforms with the |
1406 | Unicode code points in the Latin-1 Supplement block, that | |
e1b711da KW |
1407 | is, between 128 and 255. Without a locale specified, unlike all other |
1408 | characters or code points, these characters have very different semantics in | |
20db7501 | 1409 | byte semantics versus character semantics, unless |
2e2b2571 KW |
1410 | C<use feature 'unicode_strings'> is specified, directly or indirectly. |
1411 | (It is indirectly specified by a C<use v5.12> or higher.) | |
e1b711da | 1412 | |
2e2b2571 KW |
1413 | In character semantics these upper-Latin1 characters are interpreted as |
1414 | Unicode code points, which means | |
e1b711da KW |
1415 | they have the same semantics as Latin-1 (ISO-8859-1). |
1416 | ||
2e2b2571 KW |
1417 | In byte semantics (without C<unicode_strings>), they are considered to |
1418 | be unassigned characters, meaning that the only semantics they have is | |
1419 | their ordinal numbers, and that they are | |
e1b711da | 1420 | not members of various character classes. None are considered to match C<\w> |
42581d5d | 1421 | for example, but all match C<\W>. |
e1b711da | 1422 | |
2e2b2571 KW |
1423 | Perl 5.12.0 added C<unicode_strings> to force character semantics on |
1424 | these code points in some circumstances, which fixed portions of the | |
1425 | bug; Perl 5.14.0 fixed almost all of it; and Perl 5.16.0 fixed the | |
1426 | remainder (so far as we know, anyway). The lesson here is to enable | |
1427 | C<unicode_strings> to avoid the headaches described below. | |
1428 | ||
1429 | The old, problematic behavior affects these areas: | |
e1b711da KW |
1430 | |
1431 | =over 4 | |
1432 | ||
1433 | =item * | |
1434 | ||
1435 | Changing the case of a scalar, that is, using C<uc()>, C<ucfirst()>, C<lc()>, | |
2e2b2571 KW |
1436 | and C<lcfirst()>, or C<\L>, C<\U>, C<\u> and C<\l> in double-quotish |
1437 | contexts, such as regular expression substitutions. | |
1438 | Under C<unicode_strings> starting in Perl 5.12.0, character semantics are | |
1439 | generally used. See L<perlfunc/lc> for details on how this works | |
1440 | in combination with various other pragmas. | |
e1b711da KW |
1441 | |
1442 | =item * | |
1443 | ||
2e2b2571 KW |
1444 | Using caseless (C</i>) regular expression matching. |
1445 | Starting in Perl 5.14.0, regular expressions compiled within | |
c43ca372 | 1446 | the scope of C<unicode_strings> use character semantics |
2e2b2571 KW |
1447 | even when executed or compiled into larger |
1448 | regular expressions outside the scope. | |
e1b711da KW |
1449 | |
1450 | =item * | |
1451 | ||
b19eb496 | 1452 | Matching any of several properties in regular expressions, namely C<\b>, |
630d17dc KW |
1453 | C<\B>, C<\s>, C<\S>, C<\w>, C<\W>, and all the Posix character classes |
1454 | I<except> C<[[:ascii:]]>. | |
2e2b2571 | 1455 | Starting in Perl 5.14.0, regular expressions compiled within |
c43ca372 | 1456 | the scope of C<unicode_strings> use character semantics |
2e2b2571 KW |
1457 | even when executed or compiled into larger |
1458 | regular expressions outside the scope. | |
e1b711da KW |
1459 | |
1460 | =item * | |
1461 | ||
91faff93 KW |
1462 | In C<quotemeta> or its inline equivalent C<\Q>, no code points above 127 |
1463 | are quoted in UTF-8 encoded strings, but in byte encoded strings, code | |
1464 | points between 128-255 are always quoted. | |
2e2b2571 KW |
1465 | Starting in Perl 5.16.0, consistent quoting rules are used within the |
1466 | scope of C<unicode_strings>, as described in L<perlfunc/quotemeta>. | |
eb88ed9e | 1467 | |
e1b711da KW |
1468 | =back |
1469 | ||
1470 | This behavior can lead to unexpected results in which a string's semantics | |
1471 | suddenly change if a code point above 255 is appended to or removed from it, | |
1472 | which changes the string's semantics from byte to character or vice versa. As | |
1473 | an example, consider the following program and its output: | |
1474 | ||
1475 | $ perl -le' | |
42581d5d | 1476 | no feature 'unicode_strings'; |
e1b711da KW |
1477 | $s1 = "\xC2"; |
1478 | $s2 = "\x{2660}"; | |
1479 | for ($s1, $s2, $s1.$s2) { | |
1480 | print /\w/ || 0; | |
1481 | } | |
1482 | ' | |
1483 | 0 | |
1484 | 0 | |
1485 | 1 | |
1486 | ||
9f815e24 | 1487 | If there's no C<\w> in C<s1> or in C<s2>, why does their concatenation have one? |
e1b711da KW |
1488 | |
1489 | This anomaly stems from Perl's attempt to not disturb older programs that | |
1490 | didn't use Unicode, and hence had no semantics for characters outside of the | |
1491 | ASCII range (except in a locale), along with Perl's desire to add Unicode | |
1492 | support seamlessly. The result wasn't seamless: these characters were | |
1493 | orphaned. | |
1494 | ||
2e2b2571 KW |
1495 | For Perls earlier than those described above, or when a string is passed |
1496 | to a function outside the subpragma's scope, a workaround is to always | |
1497 | call C<utf8::upgrade($string)>, | |
20db7501 | 1498 | or to use the standard module L<Encode>. Also, a scalar that has any characters |
6f335b04 | 1499 | whose ordinal is above 0x100, or which were specified using either of the |
b19eb496 | 1500 | C<\N{...}> notations, will automatically have character semantics. |
e1b711da | 1501 | |
1aad1664 JH |
1502 | =head2 Forcing Unicode in Perl (Or Unforcing Unicode in Perl) |
1503 | ||
e1b711da KW |
1504 | Sometimes (see L</"When Unicode Does Not Happen"> or L</The "Unicode Bug">) |
1505 | there are situations where you simply need to force a byte | |
2bbc8d55 SP |
1506 | string into UTF-8, or vice versa. The low-level calls |
1507 | utf8::upgrade($bytestring) and utf8::downgrade($utf8string[, FAIL_OK]) are | |
1aad1664 JH |
1508 | the answers. |
1509 | ||
2bbc8d55 SP |
1510 | Note that utf8::downgrade() can fail if the string contains characters |
1511 | that don't fit into a byte. | |
1aad1664 | 1512 | |
e1b711da KW |
1513 | Calling either function on a string that already is in the desired state is a |
1514 | no-op. | |
1515 | ||
95a1a48b JH |
1516 | =head2 Using Unicode in XS |
1517 | ||
3a2263fe RGS |
1518 | If you want to handle Perl Unicode in XS extensions, you may find the |
1519 | following C APIs useful. See also L<perlguts/"Unicode Support"> for an | |
1520 | explanation about Unicode at the XS level, and L<perlapi> for the API | |
1521 | details. | |
95a1a48b JH |
1522 | |
1523 | =over 4 | |
1524 | ||
1525 | =item * | |
1526 | ||
1bfb14c4 | 1527 | C<DO_UTF8(sv)> returns true if the C<UTF8> flag is on and the bytes |
2bbc8d55 | 1528 | pragma is not in effect. C<SvUTF8(sv)> returns true if the C<UTF8> |
1bfb14c4 JH |
1529 | flag is on; the bytes pragma is ignored. The C<UTF8> flag being on |
1530 | does B<not> mean that there are any characters of code points greater | |
1531 | than 255 (or 127) in the scalar or that there are even any characters | |
1532 | in the scalar. What the C<UTF8> flag means is that the sequence of | |
1533 | octets in the representation of the scalar is the sequence of UTF-8 | |
1534 | encoded code points of the characters of a string. The C<UTF8> flag | |
1535 | being off means that each octet in this representation encodes a | |
1536 | single character with code point 0..255 within the string. Perl's | |
1537 | Unicode model is not to use UTF-8 until it is absolutely necessary. | |
95a1a48b JH |
1538 | |
1539 | =item * | |
1540 | ||
2bbc8d55 | 1541 | C<uvchr_to_utf8(buf, chr)> writes a Unicode character code point into |
1bfb14c4 | 1542 | a buffer encoding the code point as UTF-8, and returns a pointer |
2bbc8d55 | 1543 | pointing after the UTF-8 bytes. It works appropriately on EBCDIC machines. |
95a1a48b JH |
1544 | |
1545 | =item * | |
1546 | ||
4b88fb76 KW |
1547 | C<utf8_to_uvchr_buf(buf, bufend, lenp)> reads UTF-8 encoded bytes from a |
1548 | buffer and | |
376d9008 | 1549 | returns the Unicode character code point and, optionally, the length of |
2bbc8d55 | 1550 | the UTF-8 byte sequence. It works appropriately on EBCDIC machines. |
95a1a48b JH |
1551 | |
1552 | =item * | |
1553 | ||
376d9008 JB |
1554 | C<utf8_length(start, end)> returns the length of the UTF-8 encoded buffer |
1555 | in characters. C<sv_len_utf8(sv)> returns the length of the UTF-8 encoded | |
95a1a48b JH |
1556 | scalar. |
1557 | ||
1558 | =item * | |
1559 | ||
376d9008 JB |
1560 | C<sv_utf8_upgrade(sv)> converts the string of the scalar to its UTF-8 |
1561 | encoded form. C<sv_utf8_downgrade(sv)> does the opposite, if | |
1562 | possible. C<sv_utf8_encode(sv)> is like sv_utf8_upgrade except that | |
1563 | it does not set the C<UTF8> flag. C<sv_utf8_decode()> does the | |
1564 | opposite of C<sv_utf8_encode()>. Note that none of these are to be | |
1565 | used as general-purpose encoding or decoding interfaces: C<use Encode> | |
1566 | for that. C<sv_utf8_upgrade()> is affected by the encoding pragma | |
1567 | but C<sv_utf8_downgrade()> is not (since the encoding pragma is | |
1568 | designed to be a one-way street). | |
95a1a48b JH |
1569 | |
1570 | =item * | |
1571 | ||
dbfbbfa1 KW |
1572 | C<is_utf8_string(buf, len)> returns true if C<len> bytes of the buffer |
1573 | are valid UTF-8. | |
95a1a48b JH |
1574 | |
1575 | =item * | |
1576 | ||
49f4c4e4 KW |
1577 | C<is_utf8_char_buf(buf, buf_end)> returns true if the pointer points to |
1578 | a valid UTF-8 character. | |
95a1a48b JH |
1579 | |
1580 | =item * | |
1581 | ||
376d9008 JB |
1582 | C<UTF8SKIP(buf)> will return the number of bytes in the UTF-8 encoded |
1583 | character in the buffer. C<UNISKIP(chr)> will return the number of bytes | |
1584 | required to UTF-8-encode the Unicode character code point. C<UTF8SKIP()> | |
90f968e0 | 1585 | is useful for example for iterating over the characters of a UTF-8 |
376d9008 | 1586 | encoded buffer; C<UNISKIP()> is useful, for example, in computing |
90f968e0 | 1587 | the size required for a UTF-8 encoded buffer. |
95a1a48b JH |
1588 | |
1589 | =item * | |
1590 | ||
376d9008 | 1591 | C<utf8_distance(a, b)> will tell the distance in characters between the |
95a1a48b JH |
1592 | two pointers pointing to the same UTF-8 encoded buffer. |
1593 | ||
1594 | =item * | |
1595 | ||
2bbc8d55 | 1596 | C<utf8_hop(s, off)> will return a pointer to a UTF-8 encoded buffer |
376d9008 JB |
1597 | that is C<off> (positive or negative) Unicode characters displaced |
1598 | from the UTF-8 buffer C<s>. Be careful not to overstep the buffer: | |
1599 | C<utf8_hop()> will merrily run off the end or the beginning of the | |
1600 | buffer if told to do so. | |
95a1a48b | 1601 | |
d2cc3551 JH |
1602 | =item * |
1603 | ||
376d9008 JB |
1604 | C<pv_uni_display(dsv, spv, len, pvlim, flags)> and |
1605 | C<sv_uni_display(dsv, ssv, pvlim, flags)> are useful for debugging the | |
1606 | output of Unicode strings and scalars. By default they are useful | |
1607 | only for debugging--they display B<all> characters as hexadecimal code | |
1bfb14c4 JH |
1608 | points--but with the flags C<UNI_DISPLAY_ISPRINT>, |
1609 | C<UNI_DISPLAY_BACKSLASH>, and C<UNI_DISPLAY_QQ> you can make the | |
1610 | output more readable. | |
d2cc3551 JH |
1611 | |
1612 | =item * | |
1613 | ||
66615a54 | 1614 | C<foldEQ_utf8(s1, pe1, l1, u1, s2, pe2, l2, u2)> can be used to |
376d9008 | 1615 | compare two strings case-insensitively in Unicode. For case-sensitive |
66615a54 KW |
1616 | comparisons you can just use C<memEQ()> and C<memNE()> as usual, except |
1617 | if one string is in utf8 and the other isn't. | |
d2cc3551 | 1618 | |
c349b1b9 JH |
1619 | =back |
1620 | ||
95a1a48b JH |
1621 | For more information, see L<perlapi>, and F<utf8.c> and F<utf8.h> |
1622 | in the Perl source code distribution. | |
1623 | ||
e1b711da KW |
1624 | =head2 Hacking Perl to work on earlier Unicode versions (for very serious hackers only) |
1625 | ||
1626 | Perl by default comes with the latest supported Unicode version built in, but | |
1627 | you can change to use any earlier one. | |
1628 | ||
42581d5d | 1629 | Download the files in the desired version of Unicode from the Unicode web |
e1b711da | 1630 | site L<http://www.unicode.org>). These should replace the existing files in |
b19eb496 | 1631 | F<lib/unicore> in the Perl source tree. Follow the instructions in |
116693e8 | 1632 | F<README.perl> in that directory to change some of their names, and then build |
26e391dd | 1633 | perl (see L<INSTALL>). |
116693e8 | 1634 | |
c29a771d JH |
1635 | =head1 BUGS |
1636 | ||
376d9008 | 1637 | =head2 Interaction with Locales |
7eabb34d | 1638 | |
42581d5d | 1639 | See L<perllocale/Unicode and UTF-8> |
c29a771d | 1640 | |
9f815e24 | 1641 | =head2 Problems with characters in the Latin-1 Supplement range |
2bbc8d55 | 1642 | |
e1b711da KW |
1643 | See L</The "Unicode Bug"> |
1644 | ||
376d9008 | 1645 | =head2 Interaction with Extensions |
7eabb34d | 1646 | |
376d9008 | 1647 | When Perl exchanges data with an extension, the extension should be |
2575c402 | 1648 | able to understand the UTF8 flag and act accordingly. If the |
b19eb496 | 1649 | extension doesn't recognize that flag, it's likely that the extension |
376d9008 | 1650 | will return incorrectly-flagged data. |
7eabb34d A |
1651 | |
1652 | So if you're working with Unicode data, consult the documentation of | |
1653 | every module you're using if there are any issues with Unicode data | |
1654 | exchange. If the documentation does not talk about Unicode at all, | |
a73d23f6 | 1655 | suspect the worst and probably look at the source to learn how the |
376d9008 | 1656 | module is implemented. Modules written completely in Perl shouldn't |
a73d23f6 RGS |
1657 | cause problems. Modules that directly or indirectly access code written |
1658 | in other programming languages are at risk. | |
7eabb34d | 1659 | |
376d9008 | 1660 | For affected functions, the simple strategy to avoid data corruption is |
7eabb34d | 1661 | to always make the encoding of the exchanged data explicit. Choose an |
376d9008 | 1662 | encoding that you know the extension can handle. Convert arguments passed |
7eabb34d A |
1663 | to the extensions to that encoding and convert results back from that |
1664 | encoding. Write wrapper functions that do the conversions for you, so | |
1665 | you can later change the functions when the extension catches up. | |
1666 | ||
376d9008 | 1667 | To provide an example, let's say the popular Foo::Bar::escape_html |
7eabb34d A |
1668 | function doesn't deal with Unicode data yet. The wrapper function |
1669 | would convert the argument to raw UTF-8 and convert the result back to | |
376d9008 | 1670 | Perl's internal representation like so: |
7eabb34d A |
1671 | |
1672 | sub my_escape_html ($) { | |
d88362ca KW |
1673 | my($what) = shift; |
1674 | return unless defined $what; | |
1675 | Encode::decode_utf8(Foo::Bar::escape_html( | |
1676 | Encode::encode_utf8($what))); | |
7eabb34d A |
1677 | } |
1678 | ||
1679 | Sometimes, when the extension does not convert data but just stores | |
b19eb496 | 1680 | and retrieves them, you will be able to use the otherwise |
7eabb34d | 1681 | dangerous Encode::_utf8_on() function. Let's say the popular |
66b79f27 | 1682 | C<Foo::Bar> extension, written in C, provides a C<param> method that |
7eabb34d A |
1683 | lets you store and retrieve data according to these prototypes: |
1684 | ||
1685 | $self->param($name, $value); # set a scalar | |
1686 | $value = $self->param($name); # retrieve a scalar | |
1687 | ||
1688 | If it does not yet provide support for any encoding, one could write a | |
1689 | derived class with such a C<param> method: | |
1690 | ||
1691 | sub param { | |
1692 | my($self,$name,$value) = @_; | |
1693 | utf8::upgrade($name); # make sure it is UTF-8 encoded | |
af55fc6a | 1694 | if (defined $value) { |
7eabb34d A |
1695 | utf8::upgrade($value); # make sure it is UTF-8 encoded |
1696 | return $self->SUPER::param($name,$value); | |
1697 | } else { | |
1698 | my $ret = $self->SUPER::param($name); | |
1699 | Encode::_utf8_on($ret); # we know, it is UTF-8 encoded | |
1700 | return $ret; | |
1701 | } | |
1702 | } | |
1703 | ||
a73d23f6 RGS |
1704 | Some extensions provide filters on data entry/exit points, such as |
1705 | DB_File::filter_store_key and family. Look out for such filters in | |
66b79f27 | 1706 | the documentation of your extensions, they can make the transition to |
7eabb34d A |
1707 | Unicode data much easier. |
1708 | ||
376d9008 | 1709 | =head2 Speed |
7eabb34d | 1710 | |
c29a771d | 1711 | Some functions are slower when working on UTF-8 encoded strings than |
574c8022 | 1712 | on byte encoded strings. All functions that need to hop over |
7c17141f JH |
1713 | characters such as length(), substr() or index(), or matching regular |
1714 | expressions can work B<much> faster when the underlying data are | |
1715 | byte-encoded. | |
1716 | ||
1717 | In Perl 5.8.0 the slowness was often quite spectacular; in Perl 5.8.1 | |
1718 | a caching scheme was introduced which will hopefully make the slowness | |
a104b433 JH |
1719 | somewhat less spectacular, at least for some operations. In general, |
1720 | operations with UTF-8 encoded strings are still slower. As an example, | |
1721 | the Unicode properties (character classes) like C<\p{Nd}> are known to | |
1722 | be quite a bit slower (5-20 times) than their simpler counterparts | |
42581d5d | 1723 | like C<\d> (then again, there are hundreds of Unicode characters matching C<Nd> |
a104b433 | 1724 | compared with the 10 ASCII characters matching C<d>). |
666f95b9 | 1725 | |
e1b711da KW |
1726 | =head2 Problems on EBCDIC platforms |
1727 | ||
f651977e | 1728 | There are several known problems with Perl on EBCDIC platforms. If you |
e1b711da | 1729 | want to use Perl there, send email to perlbug@perl.org. |
fe749c9a KW |
1730 | |
1731 | In earlier versions, when byte and character data were concatenated, | |
1732 | the new string was sometimes created by | |
1733 | decoding the byte strings as I<ISO 8859-1 (Latin-1)>, even if the | |
1734 | old Unicode string used EBCDIC. | |
1735 | ||
1736 | If you find any of these, please report them as bugs. | |
1737 | ||
c8d992ba A |
1738 | =head2 Porting code from perl-5.6.X |
1739 | ||
1740 | Perl 5.8 has a different Unicode model from 5.6. In 5.6 the programmer | |
1741 | was required to use the C<utf8> pragma to declare that a given scope | |
1742 | expected to deal with Unicode data and had to make sure that only | |
1743 | Unicode data were reaching that scope. If you have code that is | |
1744 | working with 5.6, you will need some of the following adjustments to | |
1745 | your code. The examples are written such that the code will continue | |
1746 | to work under 5.6, so you should be safe to try them out. | |
1747 | ||
755789c0 | 1748 | =over 3 |
c8d992ba A |
1749 | |
1750 | =item * | |
1751 | ||
1752 | A filehandle that should read or write UTF-8 | |
1753 | ||
b9cedb1b | 1754 | if ($] > 5.008) { |
740d4bb2 | 1755 | binmode $fh, ":encoding(utf8)"; |
c8d992ba A |
1756 | } |
1757 | ||
1758 | =item * | |
1759 | ||
1760 | A scalar that is going to be passed to some extension | |
1761 | ||
1762 | Be it Compress::Zlib, Apache::Request or any extension that has no | |
1763 | mention of Unicode in the manpage, you need to make sure that the | |
2575c402 | 1764 | UTF8 flag is stripped off. Note that at the time of this writing |
b9cedb1b | 1765 | (January 2012) the mentioned modules are not UTF-8-aware. Please |
c8d992ba A |
1766 | check the documentation to verify if this is still true. |
1767 | ||
b9cedb1b | 1768 | if ($] > 5.008) { |
c8d992ba A |
1769 | require Encode; |
1770 | $val = Encode::encode_utf8($val); # make octets | |
1771 | } | |
1772 | ||
1773 | =item * | |
1774 | ||
1775 | A scalar we got back from an extension | |
1776 | ||
1777 | If you believe the scalar comes back as UTF-8, you will most likely | |
2575c402 | 1778 | want the UTF8 flag restored: |
c8d992ba | 1779 | |
b9cedb1b | 1780 | if ($] > 5.008) { |
c8d992ba A |
1781 | require Encode; |
1782 | $val = Encode::decode_utf8($val); | |
1783 | } | |
1784 | ||
1785 | =item * | |
1786 | ||
1787 | Same thing, if you are really sure it is UTF-8 | |
1788 | ||
b9cedb1b | 1789 | if ($] > 5.008) { |
c8d992ba A |
1790 | require Encode; |
1791 | Encode::_utf8_on($val); | |
1792 | } | |
1793 | ||
1794 | =item * | |
1795 | ||
1796 | A wrapper for fetchrow_array and fetchrow_hashref | |
1797 | ||
1798 | When the database contains only UTF-8, a wrapper function or method is | |
1799 | a convenient way to replace all your fetchrow_array and | |
1800 | fetchrow_hashref calls. A wrapper function will also make it easier to | |
1801 | adapt to future enhancements in your database driver. Note that at the | |
b9cedb1b | 1802 | time of this writing (January 2012), the DBI has no standardized way |
c8d992ba A |
1803 | to deal with UTF-8 data. Please check the documentation to verify if |
1804 | that is still true. | |
1805 | ||
1806 | sub fetchrow { | |
d88362ca KW |
1807 | # $what is one of fetchrow_{array,hashref} |
1808 | my($self, $sth, $what) = @_; | |
b9cedb1b | 1809 | if ($] < 5.008) { |
c8d992ba A |
1810 | return $sth->$what; |
1811 | } else { | |
1812 | require Encode; | |
1813 | if (wantarray) { | |
1814 | my @arr = $sth->$what; | |
1815 | for (@arr) { | |
1816 | defined && /[^\000-\177]/ && Encode::_utf8_on($_); | |
1817 | } | |
1818 | return @arr; | |
1819 | } else { | |
1820 | my $ret = $sth->$what; | |
1821 | if (ref $ret) { | |
1822 | for my $k (keys %$ret) { | |
d88362ca KW |
1823 | defined |
1824 | && /[^\000-\177]/ | |
1825 | && Encode::_utf8_on($_) for $ret->{$k}; | |
c8d992ba A |
1826 | } |
1827 | return $ret; | |
1828 | } else { | |
1829 | defined && /[^\000-\177]/ && Encode::_utf8_on($_) for $ret; | |
1830 | return $ret; | |
1831 | } | |
1832 | } | |
1833 | } | |
1834 | } | |
1835 | ||
1836 | ||
1837 | =item * | |
1838 | ||
1839 | A large scalar that you know can only contain ASCII | |
1840 | ||
1841 | Scalars that contain only ASCII and are marked as UTF-8 are sometimes | |
1842 | a drag to your program. If you recognize such a situation, just remove | |
2575c402 | 1843 | the UTF8 flag: |
c8d992ba | 1844 | |
b9cedb1b | 1845 | utf8::downgrade($val) if $] > 5.008; |
c8d992ba A |
1846 | |
1847 | =back | |
1848 | ||
393fec97 GS |
1849 | =head1 SEE ALSO |
1850 | ||
51f494cc | 1851 | L<perlunitut>, L<perluniintro>, L<perluniprops>, L<Encode>, L<open>, L<utf8>, L<bytes>, |
a05d7ebb | 1852 | L<perlretut>, L<perlvar/"${^UNICODE}"> |
51f494cc | 1853 | L<http://www.unicode.org/reports/tr44>). |
393fec97 GS |
1854 | |
1855 | =cut |