This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Doc tweaks.
[perl5.git] / pod / perlunicode.pod
CommitLineData
393fec97
GS
1=head1 NAME
2
3perlunicode - Unicode support in Perl
4
5=head1 DESCRIPTION
6
0a1f2d14 7=head2 Important Caveats
21bad921 8
c349b1b9
JH
9Unicode support is an extensive requirement. While perl does not
10implement the Unicode standard or the accompanying technical reports
11from cover to cover, Perl does support many Unicode features.
21bad921 12
13a2d996 13=over 4
21bad921
GS
14
15=item Input and Output Disciplines
16
75daf61c
JH
17A filehandle can be marked as containing perl's internal Unicode
18encoding (UTF-8 or UTF-EBCDIC) by opening it with the ":utf8" layer.
0a1f2d14 19Other encodings can be converted to perl's encoding on input, or from
c349b1b9
JH
20perl's encoding on output by use of the ":encoding(...)" layer.
21See L<open>.
22
d1be9408 23To mark the Perl source itself as being in a particular encoding,
c349b1b9 24see L<encoding>.
21bad921
GS
25
26=item Regular Expressions
27
c349b1b9
JH
28The regular expression compiler produces polymorphic opcodes. That is,
29the pattern adapts to the data and automatically switch to the Unicode
30character scheme when presented with Unicode data, or a traditional
31byte scheme when presented with byte data.
21bad921 32
ad0029c4 33=item C<use utf8> still needed to enable UTF-8/UTF-EBCDIC in scripts
21bad921 34
c349b1b9
JH
35As a compatibility measure, this pragma must be explicitly used to
36enable recognition of UTF-8 in the Perl scripts themselves on ASCII
3e4dbfed 37based machines, or to recognize UTF-EBCDIC on EBCDIC based machines.
c349b1b9
JH
38B<NOTE: this should be the only place where an explicit C<use utf8>
39is needed>.
21bad921 40
1768d7eb 41You can also use the C<encoding> pragma to change the default encoding
6ec9efec 42of the data in your script; see L<encoding>.
1768d7eb 43
21bad921
GS
44=back
45
46=head2 Byte and Character semantics
393fec97
GS
47
48Beginning with version 5.6, Perl uses logically wide characters to
3e4dbfed 49represent strings internally.
393fec97 50
75daf61c
JH
51In future, Perl-level operations can be expected to work with
52characters rather than bytes, in general.
393fec97 53
75daf61c
JH
54However, as strictly an interim compatibility measure, Perl aims to
55provide a safe migration path from byte semantics to character
56semantics for programs. For operations where Perl can unambiguously
57decide that the input data is characters, Perl now switches to
58character semantics. For operations where this determination cannot
59be made without additional information from the user, Perl decides in
60favor of compatibility, and chooses to use byte semantics.
8cbd9a7a
GS
61
62This behavior preserves compatibility with earlier versions of Perl,
63which allowed byte semantics in Perl operations, but only as long as
64none of the program's inputs are marked as being as source of Unicode
65character data. Such data may come from filehandles, from calls to
66external programs, from information provided by the system (such as %ENV),
21bad921 67or from literals and constants in the source text.
8cbd9a7a 68
c349b1b9 69On Windows platforms, if the C<-C> command line switch is used, (or the
75daf61c
JH
70${^WIDE_SYSTEM_CALLS} global flag is set to C<1>), all system calls
71will use the corresponding wide character APIs. Note that this is
c349b1b9
JH
72currently only implemented on Windows since other platforms lack an
73API standard on this area.
8cbd9a7a 74
75daf61c
JH
75Regardless of the above, the C<bytes> pragma can always be used to
76force byte semantics in a particular lexical scope. See L<bytes>.
8cbd9a7a
GS
77
78The C<utf8> pragma is primarily a compatibility device that enables
75daf61c 79recognition of UTF-(8|EBCDIC) in literals encountered by the parser.
7dedd01f
JH
80Note that this pragma is only required until a future version of Perl
81in which character semantics will become the default. This pragma may
82then become a no-op. See L<utf8>.
8cbd9a7a
GS
83
84Unless mentioned otherwise, Perl operators will use character semantics
85when they are dealing with Unicode data, and byte semantics otherwise.
86Thus, character semantics for these operations apply transparently; if
87the input data came from a Unicode source (for example, by adding a
88character encoding discipline to the filehandle whence it came, or a
3e4dbfed 89literal Unicode string constant in the program), character semantics
8cbd9a7a 90apply; otherwise, byte semantics are in effect. To force byte semantics
8058d7ab 91on Unicode data, the C<bytes> pragma should be used.
393fec97 92
0a378802
JH
93Notice that if you concatenate strings with byte semantics and strings
94with Unicode character data, the bytes will by default be upgraded
95I<as if they were ISO 8859-1 (Latin-1)> (or if in EBCDIC, after a
3e4dbfed
JF
96translation to ISO 8859-1). This is done without regard to the
97system's native 8-bit encoding, so to change this for systems with
98non-Latin-1 (or non-EBCDIC) native encodings, use the C<encoding>
0a378802 99pragma, see L<encoding>.
7dedd01f 100
feda178f
JH
101Under character semantics, many operations that formerly operated on
102bytes change to operating on characters. A character in Perl is
103logically just a number ranging from 0 to 2**31 or so. Larger
104characters may encode to longer sequences of bytes internally, but
105this is just an internal detail which is hidden at the Perl level.
106See L<perluniintro> for more on this.
393fec97 107
8cbd9a7a 108=head2 Effects of character semantics
393fec97
GS
109
110Character semantics have the following effects:
111
112=over 4
113
114=item *
115
116Strings and patterns may contain characters that have an ordinal value
21bad921 117larger than 255.
393fec97 118
feda178f
JH
119If you use a Unicode editor to edit your program, Unicode characters
120may occur directly within the literal strings in one of the various
121Unicode encodings (UTF-8, UTF-EBCDIC, UCS-2, etc.), but are recognized
122as such (and converted to Perl's internal representation) only if the
123appropriate L<encoding> is specified.
3e4dbfed
JF
124
125You can also get Unicode characters into a string by using the C<\x{...}>
126notation, putting the Unicode code for the desired character, in
127hexadecimal, into the curlies. For instance, a smiley face is C<\x{263A}>.
128This works only for characters with a code 0x100 and above.
129
130Additionally, if you
131 use charnames ':full';
132you can use the C<\N{...}> notation, putting the official Unicode character
133name within the curlies. For example, C<\N{WHITE SMILING FACE}>.
134This works for all characters that have names.
393fec97
GS
135
136=item *
137
3e4dbfed
JF
138If an appropriate L<encoding> is specified,
139identifiers within the Perl script may contain Unicode alphanumeric
393fec97 140characters, including ideographs. (You are currently on your own when
75daf61c
JH
141it comes to using the canonical forms of characters--Perl doesn't
142(yet) attempt to canonicalize variable names for you.)
393fec97 143
393fec97
GS
144=item *
145
146Regular expressions match characters instead of bytes. For instance,
147"." matches a character instead of a byte. (However, the C<\C> pattern
75daf61c 148is provided to force a match a single byte ("C<char>" in C, hence C<\C>).)
393fec97 149
393fec97
GS
150=item *
151
152Character classes in regular expressions match characters instead of
153bytes, and match against the character properties specified in the
75daf61c
JH
154Unicode properties database. So C<\w> can be used to match an
155ideograph, for instance.
393fec97 156
393fec97
GS
157=item *
158
cfc01aea 159Named Unicode properties and block ranges may be used as character
393fec97
GS
160classes via the new C<\p{}> (matches property) and C<\P{}> (doesn't
161match property) constructs. For instance, C<\p{Lu}> matches any
feda178f
JH
162character with the Unicode "Lu" (Letter, uppercase) property, while
163C<\p{M}> matches any character with a "M" (mark -- accents and such)
164property. Single letter properties may omit the brackets, so that can
165be written C<\pM> also. Many predefined character classes are
166available, such as C<\p{IsMirrored}> and C<\p{InTibetan}>.
4193bef7
JH
167
168The C<\p{Is...}> test for "general properties" such as "letter",
169"digit", while the C<\p{In...}> test for Unicode scripts and blocks.
170
cfc01aea 171The official Unicode script and block names have spaces and dashes as
e150c829
JH
172separators, but for convenience you can have dashes, spaces, and
173underbars at every word division, and you need not care about correct
174casing. It is recommended, however, that for consistency you use the
175following naming: the official Unicode script, block, or property name
feda178f
JH
176(see below for the additional rules that apply to block names), with
177whitespace and dashes replaced with underbar, and the words
e150c829
JH
178"uppercase-first-lowercase-rest". That is, "Latin-1 Supplement"
179becomes "Latin_1_Supplement".
4193bef7 180
a1cc1cb1 181You can also negate both C<\p{}> and C<\P{}> by introducing a caret
e150c829
JH
182(^) between the first curly and the property name: C<\p{^In_Tamil}> is
183equal to C<\P{In_Tamil}>.
4193bef7 184
61247495 185The C<In> and C<Is> can be left out: C<\p{Greek}> is equal to
e150c829 186C<\p{In_Greek}>, C<\P{Pd}> is equal to C<\P{Pd}>.
393fec97 187
d73e5302
JH
188 Short Long
189
190 L Letter
e150c829
JH
191 Lu Uppercase_Letter
192 Ll Lowercase_Letter
193 Lt Titlecase_Letter
194 Lm Modifier_Letter
195 Lo Other_Letter
d73e5302
JH
196
197 M Mark
e150c829
JH
198 Mn Nonspacing_Mark
199 Mc Spacing_Mark
200 Me Enclosing_Mark
d73e5302
JH
201
202 N Number
e150c829
JH
203 Nd Decimal_Number
204 Nl Letter_Number
205 No Other_Number
d73e5302
JH
206
207 P Punctuation
e150c829
JH
208 Pc Connector_Punctuation
209 Pd Dash_Punctuation
210 Ps Open_Punctuation
211 Pe Close_Punctuation
212 Pi Initial_Punctuation
d73e5302 213 (may behave like Ps or Pe depending on usage)
e150c829 214 Pf Final_Punctuation
d73e5302 215 (may behave like Ps or Pe depending on usage)
e150c829 216 Po Other_Punctuation
d73e5302
JH
217
218 S Symbol
e150c829
JH
219 Sm Math_Symbol
220 Sc Currency_Symbol
221 Sk Modifier_Symbol
222 So Other_Symbol
d73e5302
JH
223
224 Z Separator
e150c829
JH
225 Zs Space_Separator
226 Zl Line_Separator
227 Zp Paragraph_Separator
d73e5302
JH
228
229 C Other
e150c829
JH
230 Cc Control
231 Cf Format
232 Cs Surrogate
233 Co Private_Use
234 Cn Unassigned
1ac13f9a 235
3e4dbfed
JF
236The single-letter properties match all characters in any of the
237two-letter sub-properties starting with the same letter.
1ac13f9a 238There's also C<L&> which is an alias for C<Ll>, C<Lu>, and C<Lt>.
32293815 239
d73e5302
JH
240The following reserved ranges have C<In> tests:
241
e150c829
JH
242 CJK_Ideograph_Extension_A
243 CJK_Ideograph
244 Hangul_Syllable
245 Non_Private_Use_High_Surrogate
246 Private_Use_High_Surrogate
247 Low_Surrogate
248 Private_Surrogate
249 CJK_Ideograph_Extension_B
250 Plane_15_Private_Use
251 Plane_16_Private_Use
d73e5302
JH
252
253For example C<"\x{AC00}" =~ \p{HangulSyllable}> will test true.
e9ad1727 254(Handling of surrogates is not implemented yet, because Perl
0675fa52
JH
255uses UTF-8 and not UTF-16 internally to represent Unicode.
256So you really can't use the "Cs" category.)
d73e5302 257
32293815
JH
258Additionally, because scripts differ in their directionality
259(for example Hebrew is written right to left), all characters
260have their directionality defined:
261
d73e5302
JH
262 BidiL Left-to-Right
263 BidiLRE Left-to-Right Embedding
264 BidiLRO Left-to-Right Override
265 BidiR Right-to-Left
266 BidiAL Right-to-Left Arabic
267 BidiRLE Right-to-Left Embedding
268 BidiRLO Right-to-Left Override
269 BidiPDF Pop Directional Format
270 BidiEN European Number
271 BidiES European Number Separator
272 BidiET European Number Terminator
273 BidiAN Arabic Number
274 BidiCS Common Number Separator
275 BidiNSM Non-Spacing Mark
276 BidiBN Boundary Neutral
277 BidiB Paragraph Separator
278 BidiS Segment Separator
279 BidiWS Whitespace
280 BidiON Other Neutrals
32293815 281
210b36aa
AMS
282=back
283
2796c109
JH
284=head2 Scripts
285
75daf61c 286The scripts available for C<\p{In...}> and C<\P{In...}>, for example
cfc01aea 287C<\p{InLatin}> or \p{InCyrillic>, are as follows:
2796c109 288
1ac13f9a 289 Arabic
e9ad1727 290 Armenian
1ac13f9a 291 Bengali
e9ad1727
JH
292 Bopomofo
293 Canadian-Aboriginal
294 Cherokee
295 Cyrillic
296 Deseret
297 Devanagari
298 Ethiopic
299 Georgian
300 Gothic
301 Greek
1ac13f9a 302 Gujarati
e9ad1727
JH
303 Gurmukhi
304 Han
305 Hangul
306 Hebrew
307 Hiragana
308 Inherited
1ac13f9a 309 Kannada
e9ad1727
JH
310 Katakana
311 Khmer
1ac13f9a 312 Lao
e9ad1727
JH
313 Latin
314 Malayalam
315 Mongolian
1ac13f9a 316 Myanmar
1ac13f9a 317 Ogham
e9ad1727
JH
318 Old-Italic
319 Oriya
1ac13f9a 320 Runic
e9ad1727
JH
321 Sinhala
322 Syriac
323 Tamil
324 Telugu
325 Thaana
326 Thai
327 Tibetan
1ac13f9a 328 Yi
1ac13f9a
JH
329
330There are also extended property classes that supplement the basic
331properties, defined by the F<PropList> Unicode database:
332
e9ad1727 333 ASCII_Hex_Digit
1ac13f9a 334 Bidi_Control
1ac13f9a 335 Dash
1ac13f9a
JH
336 Diacritic
337 Extender
e9ad1727
JH
338 Hex_Digit
339 Hyphen
340 Ideographic
341 Join_Control
342 Noncharacter_Code_Point
343 Other_Alphabetic
1ac13f9a 344 Other_Lowercase
e9ad1727 345 Other_Math
1ac13f9a 346 Other_Uppercase
e9ad1727 347 Quotation_Mark
e150c829 348 White_Space
1ac13f9a
JH
349
350and further derived properties:
351
352 Alphabetic Lu + Ll + Lt + Lm + Lo + Other_Alphabetic
353 Lowercase Ll + Other_Lowercase
354 Uppercase Lu + Other_Uppercase
355 Math Sm + Other_Math
356
357 ID_Start Lu + Ll + Lt + Lm + Lo + Nl
358 ID_Continue ID_Start + Mn + Mc + Nd + Pc
359
360 Any Any character
361 Assigned Any non-Cn character
362 Common Any character (or unassigned code point)
e150c829 363 not explicitly assigned to a script
2796c109
JH
364
365=head2 Blocks
366
367In addition to B<scripts>, Unicode also defines B<blocks> of
368characters. The difference between scripts and blocks is that the
e9ad1727 369scripts concept is closer to natural languages, while the blocks
2796c109
JH
370concept is more an artificial grouping based on groups of 256 Unicode
371characters. For example, the C<Latin> script contains letters from
e9ad1727 372many blocks. On the other hand, the C<Latin> script does not contain
feda178f
JH
373all the characters from those blocks. It does not, for example,
374contain digits because digits are shared across many scripts. Digits
375and other similar groups, like punctuation, are in a category called
e9ad1727 376C<Common>.
2796c109 377
cfc01aea
JF
378For more about scripts, see the UTR #24:
379
380 http://www.unicode.org/unicode/reports/tr24/
381
382For more about blocks, see:
383
384 http://www.unicode.org/Public/UNIDATA/Blocks.txt
2796c109
JH
385
386Because there are overlaps in naming (there are, for example, both
387a script called C<Katakana> and a block called C<Katakana>, the block
388version has C<Block> appended to its name, C<\p{InKatakanaBlock}>.
389
390Notice that this definition was introduced in Perl 5.8.0: in Perl
e150c829 3915.6 only the blocks were used; in Perl 5.8.0 scripts became the
feda178f
JH
392preferential Unicode character class definition (prompted by
393recommendations from the Unicode consortium); this meant that
394the definitions of some character classes changed (the ones in
395the below list that have the C<Block> appended).
2796c109 396
e9ad1727
JH
397 Alphabetic Presentation Forms
398 Arabic Block
399 Arabic Presentation Forms-A
400 Arabic Presentation Forms-B
401 Armenian Block
402 Arrows
71d929cb 403 Basic Latin
e9ad1727
JH
404 Bengali Block
405 Block Elements
406 Bopomofo Block
407 Bopomofo Extended
408 Box Drawing
409 Braille Patterns
410 Byzantine Musical Symbols
411 CJK Compatibility
412 CJK Compatibility Forms
413 CJK Compatibility Ideographs
414 CJK Compatibility Ideographs Supplement
415 CJK Radicals Supplement
416 CJK Symbols and Punctuation
417 CJK Unified Ideographs
418 CJK Unified Ideographs Extension A
419 CJK Unified Ideographs Extension B
420 Cherokee Block
71d929cb 421 Combining Diacritical Marks
e9ad1727
JH
422 Combining Half Marks
423 Combining Marks for Symbols
424 Control Pictures
425 Currency Symbols
71d929cb 426 Cyrillic Block
e9ad1727 427 Deseret Block
71d929cb 428 Devanagari Block
e9ad1727
JH
429 Dingbats
430 Enclosed Alphanumerics
431 Enclosed CJK Letters and Months
432 Ethiopic Block
433 General Punctuation
434 Geometric Shapes
71d929cb 435 Georgian Block
e9ad1727
JH
436 Gothic Block
437 Greek Block
438 Greek Extended
439 Gujarati Block
440 Gurmukhi Block
441 Halfwidth and Fullwidth Forms
442 Hangul Compatibility Jamo
71d929cb 443 Hangul Jamo
e9ad1727
JH
444 Hangul Syllables
445 Hebrew Block
446 High Private Use Surrogates
447 High Surrogates
448 Hiragana Block
449 IPA Extensions
450 Ideographic Description Characters
451 Kanbun
452 Kangxi Radicals
453 Kannada Block
454 Katakana Block
71d929cb 455 Khmer Block
e9ad1727
JH
456 Lao Block
457 Latin 1 Supplement
71d929cb 458 Latin Extended Additional
e9ad1727
JH
459 Latin Extended-A
460 Latin Extended-B
71d929cb 461 Letterlike Symbols
e9ad1727
JH
462 Low Surrogates
463 Malayalam Block
464 Mathematical Alphanumeric Symbols
71d929cb 465 Mathematical Operators
e9ad1727 466 Miscellaneous Symbols
71d929cb 467 Miscellaneous Technical
e9ad1727
JH
468 Mongolian Block
469 Musical Symbols
470 Myanmar Block
471 Number Forms
472 Ogham Block
473 Old Italic Block
71d929cb 474 Optical Character Recognition
e9ad1727 475 Oriya Block
71d929cb 476 Private Use
e9ad1727
JH
477 Runic Block
478 Sinhala Block
71d929cb 479 Small Form Variants
e9ad1727 480 Spacing Modifier Letters
2796c109 481 Specials
e9ad1727
JH
482 Superscripts and Subscripts
483 Syriac Block
2796c109 484 Tags
e9ad1727
JH
485 Tamil Block
486 Telugu Block
487 Thaana Block
488 Thai Block
489 Tibetan Block
490 Unified Canadian Aboriginal Syllabics
491 Yi Radicals
492 Yi Syllables
32293815 493
210b36aa
AMS
494=over 4
495
393fec97
GS
496=item *
497
498The special pattern C<\X> match matches any extended Unicode sequence
499(a "combining character sequence" in Standardese), where the first
500character is a base character and subsequent characters are mark
501characters that apply to the base character. It is equivalent to
502C<(?:\PM\pM*)>.
503
393fec97
GS
504=item *
505
383e7cdd
JH
506The C<tr///> operator translates characters instead of bytes. Note
507that the C<tr///CU> functionality has been removed, as the interface
508was a mistake. For similar functionality see pack('U0', ...) and
509pack('C0', ...).
393fec97 510
393fec97
GS
511=item *
512
513Case translation operators use the Unicode case translation tables
44bc797b
JH
514when provided character input. Note that C<uc()> (also known as C<\U>
515in doublequoted strings) translates to uppercase, while C<ucfirst>
516(also known as C<\u> in doublequoted strings) translates to titlecase
517(for languages that make the distinction). Naturally the
518corresponding backslash sequences have the same semantics.
393fec97
GS
519
520=item *
521
522Most operators that deal with positions or lengths in the string will
75daf61c
JH
523automatically switch to using character positions, including
524C<chop()>, C<substr()>, C<pos()>, C<index()>, C<rindex()>,
525C<sprintf()>, C<write()>, and C<length()>. Operators that
526specifically don't switch include C<vec()>, C<pack()>, and
527C<unpack()>. Operators that really don't care include C<chomp()>, as
528well as any other operator that treats a string as a bucket of bits,
529such as C<sort()>, and the operators dealing with filenames.
393fec97
GS
530
531=item *
532
533The C<pack()>/C<unpack()> letters "C<c>" and "C<C>" do I<not> change,
534since they're often used for byte-oriented formats. (Again, think
535"C<char>" in the C language.) However, there is a new "C<U>" specifier
3e4dbfed 536that will convert between Unicode characters and integers.
393fec97
GS
537
538=item *
539
540The C<chr()> and C<ord()> functions work on characters. This is like
541C<pack("U")> and C<unpack("U")>, not like C<pack("C")> and
542C<unpack("C")>. In fact, the latter are how you now emulate
35bcd338 543byte-oriented C<chr()> and C<ord()> for Unicode strings.
3e4dbfed
JF
544(Note that this reveals the internal encoding of Unicode strings,
545which is not something one normally needs to care about at all.)
393fec97
GS
546
547=item *
548
a1ca4561
YST
549The bit string operators C<& | ^ ~> can operate on character data.
550However, for backward compatibility reasons (bit string operations
75daf61c
JH
551when the characters all are less than 256 in ordinal value) one should
552not mix C<~> (the bit complement) and characters both less than 256 and
a1ca4561
YST
553equal or greater than 256. Most importantly, the DeMorgan's laws
554(C<~($x|$y) eq ~$x&~$y>, C<~($x&$y) eq ~$x|~$y>) won't hold.
555Another way to look at this is that the complement cannot return
75daf61c 556B<both> the 8-bit (byte) wide bit complement B<and> the full character
a1ca4561
YST
557wide bit complement.
558
559=item *
560
983ffd37
JH
561lc(), uc(), lcfirst(), and ucfirst() work for the following cases:
562
563=over 8
564
565=item *
566
567the case mapping is from a single Unicode character to another
568single Unicode character
569
570=item *
571
572the case mapping is from a single Unicode character to more
573than one Unicode character
574
575=back
576
210b36aa 577What doesn't yet work are the following cases:
983ffd37
JH
578
579=over 8
580
581=item *
582
583the "final sigma" (Greek)
584
585=item *
586
587anything to with locales (Lithuanian, Turkish, Azeri)
588
589=back
590
591See the Unicode Technical Report #21, Case Mappings, for more details.
ac1256e8
JH
592
593=item *
594
393fec97
GS
595And finally, C<scalar reverse()> reverses by character rather than by byte.
596
597=back
598
8cbd9a7a
GS
599=head2 Character encodings for input and output
600
7221edc9 601See L<Encode>.
8cbd9a7a 602
393fec97
GS
603=head1 CAVEATS
604
8cbd9a7a
GS
605Whether an arbitrary piece of data will be treated as "characters" or
606"bytes" by internal operations cannot be divined at the current time.
393fec97 607
feda178f
JH
608Use of locales with Unicode data may lead to odd results. Currently
609there is some attempt to apply 8-bit locale info to characters in the
610range 0..255, but this is demonstrably incorrect for locales that use
3e4dbfed 611characters above that range when mapped into Unicode. It will also
393fec97
GS
612tend to run slower. Avoidance of locales is strongly encouraged.
613
776f8809
JH
614=head1 UNICODE REGULAR EXPRESSION SUPPORT LEVEL
615
616The following list of Unicode regular expression support describes
617feature by feature the Unicode support implemented in Perl as of Perl
6185.8.0. The "Level N" and the section numbers refer to the Unicode
619Technical Report 18, "Unicode Regular Expression Guidelines".
620
621=over 4
622
623=item *
624
625Level 1 - Basic Unicode Support
626
627 2.1 Hex Notation - done [1]
3bfdc84c 628 Named Notation - done [2]
776f8809
JH
629 2.2 Categories - done [3][4]
630 2.3 Subtraction - MISSING [5][6]
631 2.4 Simple Word Boundaries - done [7]
78d3e1bf 632 2.5 Simple Loose Matches - done [8]
776f8809
JH
633 2.6 End of Line - MISSING [9][10]
634
635 [ 1] \x{...}
636 [ 2] \N{...}
637 [ 3] . \p{Is...} \P{Is...}
29bdacb8 638 [ 4] now scripts (see UTR#24 Script Names) in addition to blocks
776f8809 639 [ 5] have negation
29bdacb8 640 [ 6] can use look-ahead to emulate subtraction (*)
776f8809 641 [ 7] include Letters in word characters
78d3e1bf
JH
642 [ 8] some cases of "ss"/"SS" matching U+00DF in a character
643 class are missing, but that is allowed according to the TR18.
776f8809 644 [ 9] see UTR#13 Unicode Newline Guidelines
ec83e909
JH
645 [10] should do ^ and $ also on \x{85}, \x{2028} and \x{2029})
646 (should also affect <>, $., and script line numbers)
3bfdc84c 647 (the \x{85}, \x{2028} and \x{2029} do match \s)
7207e29d 648
dbe420b4
JH
649(*) You can mimic class subtraction using lookahead.
650For example, what TR18 might write as
29bdacb8 651
dbe420b4
JH
652 [{Greek}-[{UNASSIGNED}]]
653
654in Perl can be written as:
655
656 (?!\p{UNASSIGNED})\p{GreekBlock}
657 (?=\p{ASSIGNED})\p{GreekBlock}
658
659But in this particular example, you probably really want
660
661 \p{Greek}
662
663which will match assigned characters known to be part of the Greek script.
29bdacb8 664
776f8809
JH
665=item *
666
667Level 2 - Extended Unicode Support
668
669 3.1 Surrogates - MISSING
670 3.2 Canonical Equivalents - MISSING [11][12]
671 3.3 Locale-Independent Graphemes - MISSING [13]
672 3.4 Locale-Independent Words - MISSING [14]
673 3.5 Locale-Independent Loose Matches - MISSING [15]
674
675 [11] see UTR#15 Unicode Normalization
676 [12] have Unicode::Normalize but not integrated to regexes
677 [13] have \X but at this level . should equal that
678 [14] need three classes, not just \w and \W
679 [15] see UTR#21 Case Mappings
680
681=item *
682
683Level 3 - Locale-Sensitive Support
684
685 4.1 Locale-Dependent Categories - MISSING
686 4.2 Locale-Dependent Graphemes - MISSING [16][17]
687 4.3 Locale-Dependent Words - MISSING
688 4.4 Locale-Dependent Loose Matches - MISSING
689 4.5 Locale-Dependent Ranges - MISSING
690
691 [16] see UTR#10 Unicode Collation Algorithms
692 [17] have Unicode::Collate but not integrated to regexes
693
694=back
695
c349b1b9
JH
696=head2 Unicode Encodings
697
698Unicode characters are assigned to I<code points> which are abstract
86bbd6d1 699numbers. To use these numbers various encodings are needed.
c349b1b9
JH
700
701=over 4
702
703=item UTF-8
704
3e4dbfed
JF
705UTF-8 is a variable-length (1 to 6 bytes, current character allocations
706require 4 bytes), byteorder independent encoding. For ASCII, UTF-8 is
707transparent (and we really do mean 7-bit ASCII, not another 8-bit encoding).
c349b1b9 708
05632f9a
JH
709The following table is from Unicode 3.1.
710
711 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
712
713 U+0000..U+007F 00..7F   
714 U+0080..U+07FF C2..DF 80..BF   
715 U+0800..U+0FFF E0 A0..BF 80..BF  
716 U+1000..U+FFFF E1..EF 80..BF 80..BF  
717 U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
718 U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
719 U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
720
721Or, another way to look at it, as bits:
722
723 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
724
725 0aaaaaaa 0aaaaaaa
726 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
727 ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
728 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
729
730As you can see, the continuation bytes all begin with C<10>, and the
731leading bits of the start byte tells how many bytes the are in the
732encoded character.
733
fe854a6f 734=item UTF-EBCDIC
dbe420b4 735
fe854a6f 736Like UTF-8, but EBCDIC-safe, as UTF-8 is ASCII-safe.
dbe420b4 737
c349b1b9
JH
738=item UTF-16, UTF-16BE, UTF16-LE, Surrogates, and BOMs (Byte Order Marks)
739
dbe420b4
JH
740(The followings items are mostly for reference, Perl doesn't
741use them internally.)
742
c349b1b9
JH
743UTF-16 is a 2 or 4 byte encoding. The Unicode code points
7440x0000..0xFFFF are stored in two 16-bit units, and the code points
dbe420b4 7450x010000..0x10FFFF in two 16-bit units. The latter case is
c349b1b9
JH
746using I<surrogates>, the first 16-bit unit being the I<high
747surrogate>, and the second being the I<low surrogate>.
748
749Surrogates are code points set aside to encode the 0x01000..0x10FFFF
750range of Unicode code points in pairs of 16-bit units. The I<high
751surrogates> are the range 0xD800..0xDBFF, and the I<low surrogates>
752are the range 0xDC00..0xDFFFF. The surrogate encoding is
753
754 $hi = ($uni - 0x10000) / 0x400 + 0xD800;
755 $lo = ($uni - 0x10000) % 0x400 + 0xDC00;
756
757and the decoding is
758
759 $uni = 0x10000 + ($hi - 0xD8000) * 0x400 + ($lo - 0xDC00);
760
feda178f
JH
761If you try to generate surrogates (for example by using chr()), you
762will get a warning if warnings are turned on (C<-w> or C<use
763warnings;>) because those code points are not valid for a Unicode
764character.
9466bab6 765
86bbd6d1 766Because of the 16-bitness, UTF-16 is byteorder dependent. UTF-16
c349b1b9 767itself can be used for in-memory computations, but if storage or
86bbd6d1 768transfer is required, either UTF-16BE (Big Endian) or UTF-16LE
c349b1b9
JH
769(Little Endian) must be chosen.
770
771This introduces another problem: what if you just know that your data
772is UTF-16, but you don't know which endianness? Byte Order Marks
773(BOMs) are a solution to this. A special character has been reserved
86bbd6d1
PN
774in Unicode to function as a byte order marker: the character with the
775code point 0xFEFF is the BOM.
042da322 776
c349b1b9
JH
777The trick is that if you read a BOM, you will know the byte order,
778since if it was written on a big endian platform, you will read the
86bbd6d1
PN
779bytes 0xFE 0xFF, but if it was written on a little endian platform,
780you will read the bytes 0xFF 0xFE. (And if the originating platform
781was writing in UTF-8, you will read the bytes 0xEF 0xBB 0xBF.)
042da322 782
86bbd6d1
PN
783The way this trick works is that the character with the code point
7840xFFFE is guaranteed not to be a valid Unicode character, so the
785sequence of bytes 0xFF 0xFE is unambiguously "BOM, represented in
042da322
JH
786little-endian format" and cannot be "0xFFFE, represented in big-endian
787format".
c349b1b9
JH
788
789=item UTF-32, UTF-32BE, UTF32-LE
790
791The UTF-32 family is pretty much like the UTF-16 family, expect that
042da322
JH
792the units are 32-bit, and therefore the surrogate scheme is not
793needed. The BOM signatures will be 0x00 0x00 0xFE 0xFF for BE and
7940xFF 0xFE 0x00 0x00 for LE.
c349b1b9
JH
795
796=item UCS-2, UCS-4
797
86bbd6d1
PN
798Encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit
799encoding, UCS-4 is a 32-bit encoding. Unlike UTF-16, UCS-2
800is not extensible beyond 0xFFFF, because it does not use surrogates.
c349b1b9
JH
801
802=item UTF-7
803
804A seven-bit safe (non-eight-bit) encoding, useful if the
805transport/storage is not eight-bit safe. Defined by RFC 2152.
806
95a1a48b
JH
807=back
808
bf0fa0b2
JH
809=head2 Security Implications of Malformed UTF-8
810
811Unfortunately, the specification of UTF-8 leaves some room for
812interpretation of how many bytes of encoded output one should generate
813from one input Unicode character. Strictly speaking, one is supposed
814to always generate the shortest possible sequence of UTF-8 bytes,
feda178f
JH
815because otherwise there is potential for input buffer overflow at
816the receiving end of a UTF-8 connection. Perl always generates the
817shortest length UTF-8, and with warnings on (C<-w> or C<use
818warnings;>) Perl will warn about non-shortest length UTF-8 (and other
819malformations, too, such as the surrogates, which are not real
820Unicode code points.)
bf0fa0b2 821
c349b1b9
JH
822=head2 Unicode in Perl on EBCDIC
823
824The way Unicode is handled on EBCDIC platforms is still rather
86bbd6d1 825experimental. On such a platform, references to UTF-8 encoding in this
c349b1b9
JH
826document and elsewhere should be read as meaning UTF-EBCDIC as
827specified in Unicode Technical Report 16 unless ASCII vs EBCDIC issues
828are specifically discussed. There is no C<utfebcdic> pragma or
86bbd6d1
PN
829":utfebcdic" layer, rather, "utf8" and ":utf8" are re-used to mean
830the platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic>
831for more discussion of the issues.
c349b1b9 832
95a1a48b
JH
833=head2 Using Unicode in XS
834
835If you want to handle Perl Unicode in XS extensions, you may find
836the following C APIs useful:
837
838=over 4
839
840=item *
841
842DO_UTF8(sv) returns true if the UTF8 flag is on and the bytes
843pragma is not in effect. SvUTF8(sv) returns true is the UTF8
844flag is on, the bytes pragma is ignored. Remember that UTF8
845flag being on does not mean that there would be any characters
846of code points greater than 255 or 127 in the scalar, or that
847there even are any characters in the scalar. The UTF8 flag
848means that any characters added to the string will be encoded
849in UTF8 if the code points of the characters are greater than
850255. Not "if greater than 127", since Perl's Unicode model
851is not to use UTF-8 until it's really necessary.
852
853=item *
854
855uvuni_to_utf8(buf, chr) writes a Unicode character code point into a
cfc01aea 856buffer encoding the code point as UTF-8, and returns a pointer
95a1a48b
JH
857pointing after the UTF-8 bytes.
858
859=item *
860
861utf8_to_uvuni(buf, lenp) reads UTF-8 encoded bytes from a buffer and
862returns the Unicode character code point (and optionally the length of
863the UTF-8 byte sequence).
864
865=item *
866
867utf8_length(s, len) returns the length of the UTF-8 encoded buffer in
868characters. sv_len_utf8(sv) returns the length of the UTF-8 encoded
869scalar.
870
871=item *
872
873sv_utf8_upgrade(sv) converts the string of the scalar to its UTF-8
874encoded form. sv_utf8_downgrade(sv) does the opposite (if possible).
875sv_utf8_encode(sv) is like sv_utf8_upgrade but the UTF8 flag does not
876get turned on. sv_utf8_decode() does the opposite of sv_utf8_encode().
877
878=item *
879
880is_utf8_char(buf) returns true if the buffer points to valid UTF-8.
881
882=item *
883
884is_utf8_string(buf, len) returns true if the len bytes of the buffer
885are valid UTF-8.
886
887=item *
888
889UTF8SKIP(buf) will return the number of bytes in the UTF-8 encoded
890character in the buffer. UNISKIP(chr) will return the number of bytes
891required to UTF-8-encode the Unicode character code point.
892
893=item *
894
895utf8_distance(a, b) will tell the distance in characters between the
896two pointers pointing to the same UTF-8 encoded buffer.
897
898=item *
899
900utf8_hop(s, off) will return a pointer to an UTF-8 encoded buffer that
901is C<off> (positive or negative) Unicode characters displaced from the
902UTF-8 buffer C<s>.
903
d2cc3551
JH
904=item *
905
906pv_uni_display(dsv, spv, len, pvlim, flags) and sv_uni_display(dsv,
907ssv, pvlim, flags) are useful for debug output of Unicode strings and
908scalars (only for debug: they display B<all> characters as hexadecimal
909code points).
910
911=item *
912
332ddc25
JH
913ibcmp_utf8(s1, u1, len1, s2, u2, len2) can be used to compare two
914strings case-insensitively in Unicode. (For case-sensitive
915comparisons you can just use memEQ() and memNE() as usual.)
d2cc3551 916
c349b1b9
JH
917=back
918
95a1a48b
JH
919For more information, see L<perlapi>, and F<utf8.c> and F<utf8.h>
920in the Perl source code distribution.
921
393fec97
GS
922=head1 SEE ALSO
923
72ff2908
JH
924L<perluniintro>, L<encoding>, L<Encode>, L<open>, L<utf8>, L<bytes>,
925L<perlretut>, L<perlvar/"${^WIDE_SYSTEM_CALLS}">
393fec97
GS
926
927=cut