This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
ext/POSIX/t/posix.t -- neuter final test on VOS
[perl5.git] / pod / perlunicode.pod
CommitLineData
393fec97
GS
1=head1 NAME
2
3perlunicode - Unicode support in Perl
4
5=head1 DESCRIPTION
6
0a1f2d14 7=head2 Important Caveats
21bad921 8
c349b1b9
JH
9Unicode support is an extensive requirement. While perl does not
10implement the Unicode standard or the accompanying technical reports
11from cover to cover, Perl does support many Unicode features.
21bad921 12
13a2d996 13=over 4
21bad921
GS
14
15=item Input and Output Disciplines
16
75daf61c
JH
17A filehandle can be marked as containing perl's internal Unicode
18encoding (UTF-8 or UTF-EBCDIC) by opening it with the ":utf8" layer.
0a1f2d14 19Other encodings can be converted to perl's encoding on input, or from
c349b1b9
JH
20perl's encoding on output by use of the ":encoding(...)" layer.
21See L<open>.
22
d1be9408 23To mark the Perl source itself as being in a particular encoding,
c349b1b9 24see L<encoding>.
21bad921
GS
25
26=item Regular Expressions
27
c349b1b9
JH
28The regular expression compiler produces polymorphic opcodes. That is,
29the pattern adapts to the data and automatically switch to the Unicode
30character scheme when presented with Unicode data, or a traditional
31byte scheme when presented with byte data.
21bad921 32
ad0029c4 33=item C<use utf8> still needed to enable UTF-8/UTF-EBCDIC in scripts
21bad921 34
c349b1b9
JH
35As a compatibility measure, this pragma must be explicitly used to
36enable recognition of UTF-8 in the Perl scripts themselves on ASCII
3e4dbfed 37based machines, or to recognize UTF-EBCDIC on EBCDIC based machines.
c349b1b9
JH
38B<NOTE: this should be the only place where an explicit C<use utf8>
39is needed>.
21bad921 40
1768d7eb 41You can also use the C<encoding> pragma to change the default encoding
6ec9efec 42of the data in your script; see L<encoding>.
1768d7eb 43
21bad921
GS
44=back
45
46=head2 Byte and Character semantics
393fec97
GS
47
48Beginning with version 5.6, Perl uses logically wide characters to
3e4dbfed 49represent strings internally.
393fec97 50
75daf61c
JH
51In future, Perl-level operations can be expected to work with
52characters rather than bytes, in general.
393fec97 53
75daf61c
JH
54However, as strictly an interim compatibility measure, Perl aims to
55provide a safe migration path from byte semantics to character
56semantics for programs. For operations where Perl can unambiguously
57decide that the input data is characters, Perl now switches to
58character semantics. For operations where this determination cannot
59be made without additional information from the user, Perl decides in
60favor of compatibility, and chooses to use byte semantics.
8cbd9a7a
GS
61
62This behavior preserves compatibility with earlier versions of Perl,
63which allowed byte semantics in Perl operations, but only as long as
64none of the program's inputs are marked as being as source of Unicode
65character data. Such data may come from filehandles, from calls to
66external programs, from information provided by the system (such as %ENV),
21bad921 67or from literals and constants in the source text.
8cbd9a7a 68
c349b1b9 69On Windows platforms, if the C<-C> command line switch is used, (or the
75daf61c
JH
70${^WIDE_SYSTEM_CALLS} global flag is set to C<1>), all system calls
71will use the corresponding wide character APIs. Note that this is
c349b1b9
JH
72currently only implemented on Windows since other platforms lack an
73API standard on this area.
8cbd9a7a 74
75daf61c
JH
75Regardless of the above, the C<bytes> pragma can always be used to
76force byte semantics in a particular lexical scope. See L<bytes>.
8cbd9a7a
GS
77
78The C<utf8> pragma is primarily a compatibility device that enables
75daf61c 79recognition of UTF-(8|EBCDIC) in literals encountered by the parser.
7dedd01f
JH
80Note that this pragma is only required until a future version of Perl
81in which character semantics will become the default. This pragma may
82then become a no-op. See L<utf8>.
8cbd9a7a
GS
83
84Unless mentioned otherwise, Perl operators will use character semantics
85when they are dealing with Unicode data, and byte semantics otherwise.
86Thus, character semantics for these operations apply transparently; if
87the input data came from a Unicode source (for example, by adding a
88character encoding discipline to the filehandle whence it came, or a
3e4dbfed 89literal Unicode string constant in the program), character semantics
8cbd9a7a 90apply; otherwise, byte semantics are in effect. To force byte semantics
8058d7ab 91on Unicode data, the C<bytes> pragma should be used.
393fec97 92
0a378802
JH
93Notice that if you concatenate strings with byte semantics and strings
94with Unicode character data, the bytes will by default be upgraded
95I<as if they were ISO 8859-1 (Latin-1)> (or if in EBCDIC, after a
3e4dbfed
JF
96translation to ISO 8859-1). This is done without regard to the
97system's native 8-bit encoding, so to change this for systems with
98non-Latin-1 (or non-EBCDIC) native encodings, use the C<encoding>
0a378802 99pragma, see L<encoding>.
7dedd01f 100
feda178f
JH
101Under character semantics, many operations that formerly operated on
102bytes change to operating on characters. A character in Perl is
103logically just a number ranging from 0 to 2**31 or so. Larger
104characters may encode to longer sequences of bytes internally, but
105this is just an internal detail which is hidden at the Perl level.
106See L<perluniintro> for more on this.
393fec97 107
8cbd9a7a 108=head2 Effects of character semantics
393fec97
GS
109
110Character semantics have the following effects:
111
112=over 4
113
114=item *
115
574c8022
JH
116Strings (including hash keys) and regular expression patterns may
117contain characters that have an ordinal value larger than 255.
393fec97 118
feda178f
JH
119If you use a Unicode editor to edit your program, Unicode characters
120may occur directly within the literal strings in one of the various
121Unicode encodings (UTF-8, UTF-EBCDIC, UCS-2, etc.), but are recognized
122as such (and converted to Perl's internal representation) only if the
123appropriate L<encoding> is specified.
3e4dbfed
JF
124
125You can also get Unicode characters into a string by using the C<\x{...}>
126notation, putting the Unicode code for the desired character, in
127hexadecimal, into the curlies. For instance, a smiley face is C<\x{263A}>.
128This works only for characters with a code 0x100 and above.
129
130Additionally, if you
574c8022 131
3e4dbfed 132 use charnames ':full';
574c8022 133
3e4dbfed
JF
134you can use the C<\N{...}> notation, putting the official Unicode character
135name within the curlies. For example, C<\N{WHITE SMILING FACE}>.
136This works for all characters that have names.
393fec97
GS
137
138=item *
139
3990cdf5
JH
140If Unicode is used in hash keys, there is a subtle effect on the hashes.
141The hash becomes "Unicode-sticky" so that keys retrieved from the hash
142(either by %hash, each %hash, or keys %hash) will be in Unicode, not
143in bytes, even when the keys were bytes went they "went in". This
144"stickiness" persists unless the hash is completely emptied, either by
145using delete() or clearing the with undef() or assigning an empty list
146to the hash. Most of the time this difference is negligible, but
147there are few places where it matters: for example the regular
148expression character classes like C<\w> behave differently for
149bytes and characters.
150
151=item *
152
574c8022
JH
153If an appropriate L<encoding> is specified, identifiers within the
154Perl script may contain Unicode alphanumeric characters, including
155ideographs. (You are currently on your own when it comes to using the
156canonical forms of characters--Perl doesn't (yet) attempt to
157canonicalize variable names for you.)
393fec97 158
393fec97
GS
159=item *
160
161Regular expressions match characters instead of bytes. For instance,
162"." matches a character instead of a byte. (However, the C<\C> pattern
75daf61c 163is provided to force a match a single byte ("C<char>" in C, hence C<\C>).)
393fec97 164
393fec97
GS
165=item *
166
167Character classes in regular expressions match characters instead of
168bytes, and match against the character properties specified in the
75daf61c
JH
169Unicode properties database. So C<\w> can be used to match an
170ideograph, for instance.
393fec97 171
393fec97
GS
172=item *
173
eb0cc9e3
JH
174Named Unicode properties, scripts, and block ranges may be used like
175character classes via the new C<\p{}> (matches property) and C<\P{}>
176(doesn't match property) constructs. For instance, C<\p{Lu}> matches any
feda178f
JH
177character with the Unicode "Lu" (Letter, uppercase) property, while
178C<\p{M}> matches any character with a "M" (mark -- accents and such)
eb0cc9e3
JH
179property. Single letter properties may omit the brackets, so that can be
180written C<\pM> also. Many predefined properties are available, such
181as C<\p{Mirrored}> and C<\p{Tibetan}>.
4193bef7 182
cfc01aea 183The official Unicode script and block names have spaces and dashes as
eb0cc9e3
JH
184separators, but for convenience you can have dashes, spaces, and underbars
185at every word division, and you need not care about correct casing. It is
186recommended, however, that for consistency you use the following naming:
187the official Unicode script, block, or property name (see below for the
188additional rules that apply to block names), with whitespace and dashes
189removed, and the words "uppercase-first-lowercase-rest". That is, "Latin-1
190Supplement" becomes "Latin1Supplement".
4193bef7 191
a1cc1cb1 192You can also negate both C<\p{}> and C<\P{}> by introducing a caret
eb0cc9e3
JH
193(^) between the first curly and the property name: C<\p{^Tamil}> is
194equal to C<\P{Tamil}>.
4193bef7 195
eb0cc9e3
JH
196Here are the basic Unicode General Category properties, followed by their
197long form (you can use either, e.g. C<\p{Lu}> and C<\p{LowercaseLetter}>
198are identical).
393fec97 199
d73e5302
JH
200 Short Long
201
202 L Letter
eb0cc9e3
JH
203 Lu UppercaseLetter
204 Ll LowercaseLetter
205 Lt TitlecaseLetter
206 Lm ModifierLetter
207 Lo OtherLetter
d73e5302
JH
208
209 M Mark
eb0cc9e3
JH
210 Mn NonspacingMark
211 Mc SpacingMark
212 Me EnclosingMark
d73e5302
JH
213
214 N Number
eb0cc9e3
JH
215 Nd DecimalNumber
216 Nl LetterNumber
217 No OtherNumber
d73e5302
JH
218
219 P Punctuation
eb0cc9e3
JH
220 Pc ConnectorPunctuation
221 Pd DashPunctuation
222 Ps OpenPunctuation
223 Pe ClosePunctuation
224 Pi InitialPunctuation
d73e5302 225 (may behave like Ps or Pe depending on usage)
eb0cc9e3 226 Pf FinalPunctuation
d73e5302 227 (may behave like Ps or Pe depending on usage)
eb0cc9e3 228 Po OtherPunctuation
d73e5302
JH
229
230 S Symbol
eb0cc9e3
JH
231 Sm MathSymbol
232 Sc CurrencySymbol
233 Sk ModifierSymbol
234 So OtherSymbol
d73e5302
JH
235
236 Z Separator
eb0cc9e3
JH
237 Zs SpaceSeparator
238 Zl LineSeparator
239 Zp ParagraphSeparator
d73e5302
JH
240
241 C Other
e150c829
JH
242 Cc Control
243 Cf Format
eb0cc9e3
JH
244 Cs Surrogate (not usable)
245 Co PrivateUse
e150c829 246 Cn Unassigned
1ac13f9a 247
3e4dbfed
JF
248The single-letter properties match all characters in any of the
249two-letter sub-properties starting with the same letter.
1ac13f9a 250There's also C<L&> which is an alias for C<Ll>, C<Lu>, and C<Lt>.
32293815 251
eb0cc9e3
JH
252Because Perl hides the need for the user to understand the internal
253representation of Unicode characters, it has no need to support the
254somewhat messy concept of surrogates. Therefore, the C<Cs> property is not
255supported.
d73e5302 256
eb0cc9e3
JH
257Because scripts differ in their directionality (for example Hebrew is
258written right to left), Unicode supplies these properties:
32293815 259
eb0cc9e3 260 Property Meaning
92e830a9 261
d73e5302
JH
262 BidiL Left-to-Right
263 BidiLRE Left-to-Right Embedding
264 BidiLRO Left-to-Right Override
265 BidiR Right-to-Left
266 BidiAL Right-to-Left Arabic
267 BidiRLE Right-to-Left Embedding
268 BidiRLO Right-to-Left Override
269 BidiPDF Pop Directional Format
270 BidiEN European Number
271 BidiES European Number Separator
272 BidiET European Number Terminator
273 BidiAN Arabic Number
274 BidiCS Common Number Separator
275 BidiNSM Non-Spacing Mark
276 BidiBN Boundary Neutral
277 BidiB Paragraph Separator
278 BidiS Segment Separator
279 BidiWS Whitespace
280 BidiON Other Neutrals
32293815 281
eb0cc9e3
JH
282For example, C<\p{BidiR}> matches all characters that are normally
283written right to left.
284
210b36aa
AMS
285=back
286
2796c109
JH
287=head2 Scripts
288
eb0cc9e3
JH
289The scripts available via C<\p{...}> and C<\P{...}>, for example
290C<\p{Latin}> or \p{Cyrillic>, are as follows:
2796c109 291
1ac13f9a 292 Arabic
e9ad1727 293 Armenian
1ac13f9a 294 Bengali
e9ad1727 295 Bopomofo
eb0cc9e3 296 CanadianAboriginal
e9ad1727
JH
297 Cherokee
298 Cyrillic
299 Deseret
300 Devanagari
301 Ethiopic
302 Georgian
303 Gothic
304 Greek
1ac13f9a 305 Gujarati
e9ad1727
JH
306 Gurmukhi
307 Han
308 Hangul
309 Hebrew
310 Hiragana
311 Inherited
1ac13f9a 312 Kannada
e9ad1727
JH
313 Katakana
314 Khmer
1ac13f9a 315 Lao
e9ad1727
JH
316 Latin
317 Malayalam
318 Mongolian
1ac13f9a 319 Myanmar
1ac13f9a 320 Ogham
eb0cc9e3 321 OldItalic
e9ad1727 322 Oriya
1ac13f9a 323 Runic
e9ad1727
JH
324 Sinhala
325 Syriac
326 Tamil
327 Telugu
328 Thaana
329 Thai
330 Tibetan
1ac13f9a 331 Yi
1ac13f9a
JH
332
333There are also extended property classes that supplement the basic
334properties, defined by the F<PropList> Unicode database:
335
e9ad1727 336 ASCII_Hex_Digit
eb0cc9e3 337 BidiControl
1ac13f9a 338 Dash
1ac13f9a
JH
339 Diacritic
340 Extender
eb0cc9e3 341 HexDigit
e9ad1727
JH
342 Hyphen
343 Ideographic
eb0cc9e3
JH
344 JoinControl
345 NoncharacterCodePoint
346 OtherAlphabetic
347 OtherLowercase
348 OtherMath
349 OtherUppercase
350 QuotationMark
351 WhiteSpace
1ac13f9a
JH
352
353and further derived properties:
354
eb0cc9e3
JH
355 Alphabetic Lu + Ll + Lt + Lm + Lo + OtherAlphabetic
356 Lowercase Ll + OtherLowercase
357 Uppercase Lu + OtherUppercase
358 Math Sm + OtherMath
1ac13f9a
JH
359
360 ID_Start Lu + Ll + Lt + Lm + Lo + Nl
361 ID_Continue ID_Start + Mn + Mc + Nd + Pc
362
363 Any Any character
eb0cc9e3
JH
364 Assigned Any non-Cn character (i.e. synonym for C<\P{Cn}>)
365 Unassigned Synonym for C<\p{Cn}>
1ac13f9a 366 Common Any character (or unassigned code point)
e150c829 367 not explicitly assigned to a script
2796c109 368
eb0cc9e3
JH
369For backward compatability, all properties mentioned so far may have C<Is>
370prepended to their name (e.g. C<\P{IsLu}> is equal to C<\P{Lu}>).
371
2796c109
JH
372=head2 Blocks
373
eb0cc9e3
JH
374In addition to B<scripts>, Unicode also defines B<blocks> of characters.
375The difference between scripts and blocks is that the scripts concept is
376closer to natural languages, while the blocks concept is more an artificial
377grouping based on groups of mostly 256 Unicode characters. For example, the
378C<Latin> script contains letters from many blocks. On the other hand, the
379C<Latin> script does not contain all the characters from those blocks. It
380does not, for example, contain digits because digits are shared across many
381scripts. Digits and other similar groups, like punctuation, are in a
382category called C<Common>.
2796c109 383
cfc01aea
JF
384For more about scripts, see the UTR #24:
385
386 http://www.unicode.org/unicode/reports/tr24/
387
388For more about blocks, see:
389
390 http://www.unicode.org/Public/UNIDATA/Blocks.txt
2796c109 391
eb0cc9e3 392Blocks names are given with the C<In> prefix. For example, the
92e830a9 393Katakana block is referenced via C<\p{InKatakana}>. The C<In>
eb0cc9e3
JH
394prefix may be omitted if there is no nameing conflict with a script
395or any other property, but it is recommended that C<In> always be used
396to avoid confusion.
397
398These block names are supported:
399
400 InAlphabeticPresentationForms
401 InArabicBlock
402 InArabicPresentationFormsA
403 InArabicPresentationFormsB
404 InArmenianBlock
405 InArrows
406 InBasicLatin
407 InBengaliBlock
408 InBlockElements
409 InBopomofoBlock
410 InBopomofoExtended
411 InBoxDrawing
412 InBraillePatterns
413 InByzantineMusicalSymbols
414 InCJKCompatibility
415 InCJKCompatibilityForms
416 InCJKCompatibilityIdeographs
417 InCJKCompatibilityIdeographsSupplement
418 InCJKRadicalsSupplement
419 InCJKSymbolsAndPunctuation
420 InCJKUnifiedIdeographs
421 InCJKUnifiedIdeographsExtensionA
422 InCJKUnifiedIdeographsExtensionB
423 InCherokeeBlock
424 InCombiningDiacriticalMarks
425 InCombiningHalfMarks
426 InCombiningMarksForSymbols
427 InControlPictures
428 InCurrencySymbols
429 InCyrillicBlock
430 InDeseretBlock
431 InDevanagariBlock
432 InDingbats
433 InEnclosedAlphanumerics
434 InEnclosedCJKLettersAndMonths
435 InEthiopicBlock
436 InGeneralPunctuation
437 InGeometricShapes
438 InGeorgianBlock
439 InGothicBlock
440 InGreekBlock
441 InGreekExtended
442 InGujaratiBlock
443 InGurmukhiBlock
444 InHalfwidthAndFullwidthForms
445 InHangulCompatibilityJamo
446 InHangulJamo
447 InHangulSyllables
448 InHebrewBlock
449 InHighPrivateUseSurrogates
450 InHighSurrogates
451 InHiraganaBlock
452 InIPAExtensions
453 InIdeographicDescriptionCharacters
454 InKanbun
455 InKangxiRadicals
456 InKannadaBlock
457 InKatakanaBlock
458 InKhmerBlock
459 InLaoBlock
460 InLatin1Supplement
461 InLatinExtendedAdditional
462 InLatinExtended-A
463 InLatinExtended-B
464 InLetterlikeSymbols
465 InLowSurrogates
466 InMalayalamBlock
467 InMathematicalAlphanumericSymbols
468 InMathematicalOperators
469 InMiscellaneousSymbols
470 InMiscellaneousTechnical
471 InMongolianBlock
472 InMusicalSymbols
473 InMyanmarBlock
474 InNumberForms
475 InOghamBlock
476 InOldItalicBlock
477 InOpticalCharacterRecognition
478 InOriyaBlock
479 InPrivateUse
480 InRunicBlock
481 InSinhalaBlock
482 InSmallFormVariants
483 InSpacingModifierLetters
484 InSpecials
485 InSuperscriptsAndSubscripts
486 InSyriacBlock
487 InTags
488 InTamilBlock
489 InTeluguBlock
490 InThaanaBlock
491 InThaiBlock
492 InTibetanBlock
493 InUnifiedCanadianAboriginalSyllabics
494 InYiRadicals
495 InYiSyllables
32293815 496
210b36aa
AMS
497=over 4
498
393fec97
GS
499=item *
500
c29a771d 501The special pattern C<\X> matches any extended Unicode sequence
393fec97
GS
502(a "combining character sequence" in Standardese), where the first
503character is a base character and subsequent characters are mark
504characters that apply to the base character. It is equivalent to
505C<(?:\PM\pM*)>.
506
393fec97
GS
507=item *
508
383e7cdd
JH
509The C<tr///> operator translates characters instead of bytes. Note
510that the C<tr///CU> functionality has been removed, as the interface
511was a mistake. For similar functionality see pack('U0', ...) and
512pack('C0', ...).
393fec97 513
393fec97
GS
514=item *
515
516Case translation operators use the Unicode case translation tables
44bc797b
JH
517when provided character input. Note that C<uc()> (also known as C<\U>
518in doublequoted strings) translates to uppercase, while C<ucfirst>
519(also known as C<\u> in doublequoted strings) translates to titlecase
520(for languages that make the distinction). Naturally the
521corresponding backslash sequences have the same semantics.
393fec97
GS
522
523=item *
524
525Most operators that deal with positions or lengths in the string will
75daf61c
JH
526automatically switch to using character positions, including
527C<chop()>, C<substr()>, C<pos()>, C<index()>, C<rindex()>,
528C<sprintf()>, C<write()>, and C<length()>. Operators that
529specifically don't switch include C<vec()>, C<pack()>, and
530C<unpack()>. Operators that really don't care include C<chomp()>, as
531well as any other operator that treats a string as a bucket of bits,
532such as C<sort()>, and the operators dealing with filenames.
393fec97
GS
533
534=item *
535
536The C<pack()>/C<unpack()> letters "C<c>" and "C<C>" do I<not> change,
537since they're often used for byte-oriented formats. (Again, think
538"C<char>" in the C language.) However, there is a new "C<U>" specifier
3e4dbfed 539that will convert between Unicode characters and integers.
393fec97
GS
540
541=item *
542
543The C<chr()> and C<ord()> functions work on characters. This is like
544C<pack("U")> and C<unpack("U")>, not like C<pack("C")> and
545C<unpack("C")>. In fact, the latter are how you now emulate
35bcd338 546byte-oriented C<chr()> and C<ord()> for Unicode strings.
3e4dbfed
JF
547(Note that this reveals the internal encoding of Unicode strings,
548which is not something one normally needs to care about at all.)
393fec97
GS
549
550=item *
551
a1ca4561
YST
552The bit string operators C<& | ^ ~> can operate on character data.
553However, for backward compatibility reasons (bit string operations
75daf61c
JH
554when the characters all are less than 256 in ordinal value) one should
555not mix C<~> (the bit complement) and characters both less than 256 and
a1ca4561
YST
556equal or greater than 256. Most importantly, the DeMorgan's laws
557(C<~($x|$y) eq ~$x&~$y>, C<~($x&$y) eq ~$x|~$y>) won't hold.
558Another way to look at this is that the complement cannot return
75daf61c 559B<both> the 8-bit (byte) wide bit complement B<and> the full character
a1ca4561
YST
560wide bit complement.
561
562=item *
563
983ffd37
JH
564lc(), uc(), lcfirst(), and ucfirst() work for the following cases:
565
566=over 8
567
568=item *
569
570the case mapping is from a single Unicode character to another
571single Unicode character
572
573=item *
574
575the case mapping is from a single Unicode character to more
576than one Unicode character
577
578=back
579
210b36aa 580What doesn't yet work are the following cases:
983ffd37
JH
581
582=over 8
583
584=item *
585
586the "final sigma" (Greek)
587
588=item *
589
590anything to with locales (Lithuanian, Turkish, Azeri)
591
592=back
593
594See the Unicode Technical Report #21, Case Mappings, for more details.
ac1256e8
JH
595
596=item *
597
393fec97
GS
598And finally, C<scalar reverse()> reverses by character rather than by byte.
599
600=back
601
8cbd9a7a
GS
602=head2 Character encodings for input and output
603
7221edc9 604See L<Encode>.
8cbd9a7a 605
c29a771d 606=head2 Unicode Regular Expression Support Level
776f8809
JH
607
608The following list of Unicode regular expression support describes
609feature by feature the Unicode support implemented in Perl as of Perl
6105.8.0. The "Level N" and the section numbers refer to the Unicode
611Technical Report 18, "Unicode Regular Expression Guidelines".
612
613=over 4
614
615=item *
616
617Level 1 - Basic Unicode Support
618
619 2.1 Hex Notation - done [1]
3bfdc84c 620 Named Notation - done [2]
776f8809
JH
621 2.2 Categories - done [3][4]
622 2.3 Subtraction - MISSING [5][6]
623 2.4 Simple Word Boundaries - done [7]
78d3e1bf 624 2.5 Simple Loose Matches - done [8]
776f8809
JH
625 2.6 End of Line - MISSING [9][10]
626
627 [ 1] \x{...}
628 [ 2] \N{...}
eb0cc9e3 629 [ 3] . \p{...} \P{...}
29bdacb8 630 [ 4] now scripts (see UTR#24 Script Names) in addition to blocks
776f8809 631 [ 5] have negation
29bdacb8 632 [ 6] can use look-ahead to emulate subtraction (*)
776f8809 633 [ 7] include Letters in word characters
e0f9d4a8
JH
634 [ 8] note that perl does Full casefolding in matching, not Simple:
635 for example U+1F88 is equivalent with U+1F000 U+03B9,
636 not with 1F80. This difference matters for certain Greek
637 capital letters with certain modifiers: the Full casefolding
638 decomposes the letter, while the Simple casefolding would map
639 it to a single character.
776f8809 640 [ 9] see UTR#13 Unicode Newline Guidelines
ec83e909
JH
641 [10] should do ^ and $ also on \x{85}, \x{2028} and \x{2029})
642 (should also affect <>, $., and script line numbers)
3bfdc84c 643 (the \x{85}, \x{2028} and \x{2029} do match \s)
7207e29d 644
dbe420b4
JH
645(*) You can mimic class subtraction using lookahead.
646For example, what TR18 might write as
29bdacb8 647
dbe420b4
JH
648 [{Greek}-[{UNASSIGNED}]]
649
650in Perl can be written as:
651
eb0cc9e3
JH
652 (?!\p{Unassigned})\p{InGreek}
653 (?=\p{Assigned})\p{InGreek}
dbe420b4
JH
654
655But in this particular example, you probably really want
656
657 \p{Greek}
658
659which will match assigned characters known to be part of the Greek script.
29bdacb8 660
776f8809
JH
661=item *
662
663Level 2 - Extended Unicode Support
664
665 3.1 Surrogates - MISSING
666 3.2 Canonical Equivalents - MISSING [11][12]
667 3.3 Locale-Independent Graphemes - MISSING [13]
668 3.4 Locale-Independent Words - MISSING [14]
669 3.5 Locale-Independent Loose Matches - MISSING [15]
670
671 [11] see UTR#15 Unicode Normalization
672 [12] have Unicode::Normalize but not integrated to regexes
673 [13] have \X but at this level . should equal that
674 [14] need three classes, not just \w and \W
675 [15] see UTR#21 Case Mappings
676
677=item *
678
679Level 3 - Locale-Sensitive Support
680
681 4.1 Locale-Dependent Categories - MISSING
682 4.2 Locale-Dependent Graphemes - MISSING [16][17]
683 4.3 Locale-Dependent Words - MISSING
684 4.4 Locale-Dependent Loose Matches - MISSING
685 4.5 Locale-Dependent Ranges - MISSING
686
687 [16] see UTR#10 Unicode Collation Algorithms
688 [17] have Unicode::Collate but not integrated to regexes
689
690=back
691
c349b1b9
JH
692=head2 Unicode Encodings
693
694Unicode characters are assigned to I<code points> which are abstract
86bbd6d1 695numbers. To use these numbers various encodings are needed.
c349b1b9
JH
696
697=over 4
698
c29a771d 699=item *
5cb3728c
RB
700
701UTF-8
c349b1b9 702
3e4dbfed
JF
703UTF-8 is a variable-length (1 to 6 bytes, current character allocations
704require 4 bytes), byteorder independent encoding. For ASCII, UTF-8 is
705transparent (and we really do mean 7-bit ASCII, not another 8-bit encoding).
c349b1b9 706
8c007b5a 707The following table is from Unicode 3.2.
05632f9a
JH
708
709 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
710
8c007b5a
JH
711 U+0000..U+007F 00..7F
712 U+0080..U+07FF C2..DF 80..BF
05632f9a 713 U+0800..U+0FFF E0 A0..BF 80..BF  
8c007b5a
JH
714 U+1000..U+CFFF E1..EC 80..BF 80..BF  
715 U+D000..U+D7FF ED 80..9F 80..BF  
716 U+D800..U+DFFF ******* ill-formed *******
717 U+E000..U+FFFF EE..EF 80..BF 80..BF  
05632f9a
JH
718 U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
719 U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
720 U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
721
8c007b5a
JH
722Note the A0..BF in U+0800..U+0FFF, the 80..9F in U+D000...U+D7FF,
723the 90..BF in U+10000..U+3FFFF, and the 80...8F in U+100000..U+10FFFF.
05632f9a
JH
724Or, another way to look at it, as bits:
725
726 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
727
728 0aaaaaaa 0aaaaaaa
729 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
730 ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
731 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
732
733As you can see, the continuation bytes all begin with C<10>, and the
8c007b5a 734leading bits of the start byte tell how many bytes the are in the
05632f9a
JH
735encoded character.
736
c29a771d 737=item *
5cb3728c
RB
738
739UTF-EBCDIC
dbe420b4 740
fe854a6f 741Like UTF-8, but EBCDIC-safe, as UTF-8 is ASCII-safe.
dbe420b4 742
c29a771d 743=item *
5cb3728c
RB
744
745UTF-16, UTF-16BE, UTF16-LE, Surrogates, and BOMs (Byte Order Marks)
c349b1b9 746
dbe420b4
JH
747(The followings items are mostly for reference, Perl doesn't
748use them internally.)
749
c349b1b9
JH
750UTF-16 is a 2 or 4 byte encoding. The Unicode code points
7510x0000..0xFFFF are stored in two 16-bit units, and the code points
dbe420b4 7520x010000..0x10FFFF in two 16-bit units. The latter case is
c349b1b9
JH
753using I<surrogates>, the first 16-bit unit being the I<high
754surrogate>, and the second being the I<low surrogate>.
755
756Surrogates are code points set aside to encode the 0x01000..0x10FFFF
757range of Unicode code points in pairs of 16-bit units. The I<high
758surrogates> are the range 0xD800..0xDBFF, and the I<low surrogates>
759are the range 0xDC00..0xDFFFF. The surrogate encoding is
760
761 $hi = ($uni - 0x10000) / 0x400 + 0xD800;
762 $lo = ($uni - 0x10000) % 0x400 + 0xDC00;
763
764and the decoding is
765
766 $uni = 0x10000 + ($hi - 0xD8000) * 0x400 + ($lo - 0xDC00);
767
feda178f
JH
768If you try to generate surrogates (for example by using chr()), you
769will get a warning if warnings are turned on (C<-w> or C<use
770warnings;>) because those code points are not valid for a Unicode
771character.
9466bab6 772
86bbd6d1 773Because of the 16-bitness, UTF-16 is byteorder dependent. UTF-16
c349b1b9 774itself can be used for in-memory computations, but if storage or
86bbd6d1 775transfer is required, either UTF-16BE (Big Endian) or UTF-16LE
c349b1b9
JH
776(Little Endian) must be chosen.
777
778This introduces another problem: what if you just know that your data
779is UTF-16, but you don't know which endianness? Byte Order Marks
780(BOMs) are a solution to this. A special character has been reserved
86bbd6d1
PN
781in Unicode to function as a byte order marker: the character with the
782code point 0xFEFF is the BOM.
042da322 783
c349b1b9
JH
784The trick is that if you read a BOM, you will know the byte order,
785since if it was written on a big endian platform, you will read the
86bbd6d1
PN
786bytes 0xFE 0xFF, but if it was written on a little endian platform,
787you will read the bytes 0xFF 0xFE. (And if the originating platform
788was writing in UTF-8, you will read the bytes 0xEF 0xBB 0xBF.)
042da322 789
86bbd6d1
PN
790The way this trick works is that the character with the code point
7910xFFFE is guaranteed not to be a valid Unicode character, so the
792sequence of bytes 0xFF 0xFE is unambiguously "BOM, represented in
042da322
JH
793little-endian format" and cannot be "0xFFFE, represented in big-endian
794format".
c349b1b9 795
c29a771d 796=item *
5cb3728c
RB
797
798UTF-32, UTF-32BE, UTF32-LE
c349b1b9
JH
799
800The UTF-32 family is pretty much like the UTF-16 family, expect that
042da322
JH
801the units are 32-bit, and therefore the surrogate scheme is not
802needed. The BOM signatures will be 0x00 0x00 0xFE 0xFF for BE and
8030xFF 0xFE 0x00 0x00 for LE.
c349b1b9 804
c29a771d 805=item *
5cb3728c
RB
806
807UCS-2, UCS-4
c349b1b9 808
86bbd6d1
PN
809Encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit
810encoding, UCS-4 is a 32-bit encoding. Unlike UTF-16, UCS-2
811is not extensible beyond 0xFFFF, because it does not use surrogates.
c349b1b9 812
c29a771d 813=item *
5cb3728c
RB
814
815UTF-7
c349b1b9
JH
816
817A seven-bit safe (non-eight-bit) encoding, useful if the
818transport/storage is not eight-bit safe. Defined by RFC 2152.
819
95a1a48b
JH
820=back
821
bf0fa0b2
JH
822=head2 Security Implications of Malformed UTF-8
823
824Unfortunately, the specification of UTF-8 leaves some room for
825interpretation of how many bytes of encoded output one should generate
826from one input Unicode character. Strictly speaking, one is supposed
827to always generate the shortest possible sequence of UTF-8 bytes,
feda178f
JH
828because otherwise there is potential for input buffer overflow at
829the receiving end of a UTF-8 connection. Perl always generates the
830shortest length UTF-8, and with warnings on (C<-w> or C<use
831warnings;>) Perl will warn about non-shortest length UTF-8 (and other
832malformations, too, such as the surrogates, which are not real
833Unicode code points.)
bf0fa0b2 834
c349b1b9
JH
835=head2 Unicode in Perl on EBCDIC
836
837The way Unicode is handled on EBCDIC platforms is still rather
86bbd6d1 838experimental. On such a platform, references to UTF-8 encoding in this
c349b1b9
JH
839document and elsewhere should be read as meaning UTF-EBCDIC as
840specified in Unicode Technical Report 16 unless ASCII vs EBCDIC issues
841are specifically discussed. There is no C<utfebcdic> pragma or
86bbd6d1
PN
842":utfebcdic" layer, rather, "utf8" and ":utf8" are re-used to mean
843the platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic>
844for more discussion of the issues.
c349b1b9 845
b310b053
JH
846=head2 Locales
847
4616122b 848Usually locale settings and Unicode do not affect each other, but
b310b053
JH
849there are a couple of exceptions:
850
851=over 4
852
853=item *
854
855If your locale environment variables (LANGUAGE, LC_ALL, LC_CTYPE, LANG)
856contain the strings 'UTF-8' or 'UTF8' (case-insensitive matching),
857the default encoding of your STDIN, STDOUT, and STDERR, and of
858B<any subsequent file open>, is UTF-8.
859
860=item *
861
862Perl tries really hard to work both with Unicode and the old byte
863oriented world: most often this is nice, but sometimes this causes
574c8022 864problems.
b310b053
JH
865
866=back
867
95a1a48b
JH
868=head2 Using Unicode in XS
869
870If you want to handle Perl Unicode in XS extensions, you may find
90f968e0 871the following C APIs useful (see perlapi for details):
95a1a48b
JH
872
873=over 4
874
875=item *
876
f1e62f77
AT
877DO_UTF8(sv) returns true if the UTF8 flag is on and the bytes pragma
878is not in effect. SvUTF8(sv) returns true is the UTF8 flag is on, the
879bytes pragma is ignored. The UTF8 flag being on does B<not> mean that
b31c5e31
AT
880there are any characters of code points greater than 255 (or 127) in
881the scalar, or that there even are any characters in the scalar.
882What the UTF8 flag means is that the sequence of octets in the
883representation of the scalar is the sequence of UTF-8 encoded
884code points of the characters of a string. The UTF8 flag being
885off means that each octet in this representation encodes a single
886character with codepoint 0..255 within the string. Perl's Unicode
887model is not to use UTF-8 until it's really necessary.
95a1a48b
JH
888
889=item *
890
891uvuni_to_utf8(buf, chr) writes a Unicode character code point into a
cfc01aea 892buffer encoding the code point as UTF-8, and returns a pointer
95a1a48b
JH
893pointing after the UTF-8 bytes.
894
895=item *
896
897utf8_to_uvuni(buf, lenp) reads UTF-8 encoded bytes from a buffer and
898returns the Unicode character code point (and optionally the length of
899the UTF-8 byte sequence).
900
901=item *
902
90f968e0
JH
903utf8_length(start, end) returns the length of the UTF-8 encoded buffer
904in characters. sv_len_utf8(sv) returns the length of the UTF-8 encoded
95a1a48b
JH
905scalar.
906
907=item *
908
909sv_utf8_upgrade(sv) converts the string of the scalar to its UTF-8
910encoded form. sv_utf8_downgrade(sv) does the opposite (if possible).
911sv_utf8_encode(sv) is like sv_utf8_upgrade but the UTF8 flag does not
912get turned on. sv_utf8_decode() does the opposite of sv_utf8_encode().
13a6c0e0
JH
913Note that none of these are to be used as general purpose encoding/decoding
914interfaces: use Encode for that. sv_utf8_upgrade() is affected by the
915encoding pragma, but sv_utf8_downgrade() is not (since the encoding
916pragma is designed to be a one-way street).
95a1a48b
JH
917
918=item *
919
90f968e0
JH
920is_utf8_char(s) returns true if the pointer points to a valid UTF-8
921character.
95a1a48b
JH
922
923=item *
924
925is_utf8_string(buf, len) returns true if the len bytes of the buffer
926are valid UTF-8.
927
928=item *
929
930UTF8SKIP(buf) will return the number of bytes in the UTF-8 encoded
931character in the buffer. UNISKIP(chr) will return the number of bytes
90f968e0
JH
932required to UTF-8-encode the Unicode character code point. UTF8SKIP()
933is useful for example for iterating over the characters of a UTF-8
934encoded buffer; UNISKIP() is useful for example in computing
935the size required for a UTF-8 encoded buffer.
95a1a48b
JH
936
937=item *
938
939utf8_distance(a, b) will tell the distance in characters between the
940two pointers pointing to the same UTF-8 encoded buffer.
941
942=item *
943
944utf8_hop(s, off) will return a pointer to an UTF-8 encoded buffer that
945is C<off> (positive or negative) Unicode characters displaced from the
90f968e0
JH
946UTF-8 buffer C<s>. Be careful not to overstep the buffer: utf8_hop()
947will merrily run off the end or the beginning if told to do so.
95a1a48b 948
d2cc3551
JH
949=item *
950
951pv_uni_display(dsv, spv, len, pvlim, flags) and sv_uni_display(dsv,
952ssv, pvlim, flags) are useful for debug output of Unicode strings and
90f968e0
JH
953scalars. By default they are useful only for debug: they display
954B<all> characters as hexadecimal code points, but with the flags
955UNI_DISPLAY_ISPRINT and UNI_DISPLAY_BACKSLASH you can make the output
956more readable.
d2cc3551
JH
957
958=item *
959
90f968e0
JH
960ibcmp_utf8(s1, pe1, u1, l1, u1, s2, pe2, l2, u2) can be used to
961compare two strings case-insensitively in Unicode.
962(For case-sensitive comparisons you can just use memEQ() and memNE()
963as usual.)
d2cc3551 964
c349b1b9
JH
965=back
966
95a1a48b
JH
967For more information, see L<perlapi>, and F<utf8.c> and F<utf8.h>
968in the Perl source code distribution.
969
c29a771d
JH
970=head1 BUGS
971
972Use of locales with Unicode data may lead to odd results. Currently
973there is some attempt to apply 8-bit locale info to characters in the
974range 0..255, but this is demonstrably incorrect for locales that use
975characters above that range when mapped into Unicode. It will also
574c8022 976tend to run slower. Use of locales with Unicode is discouraged.
c29a771d
JH
977
978Some functions are slower when working on UTF-8 encoded strings than
574c8022 979on byte encoded strings. All functions that need to hop over
c29a771d
JH
980characters such as length(), substr() or index() can work B<much>
981faster when the underlying data are byte-encoded. Witness the
982following benchmark:
666f95b9 983
c29a771d
JH
984 % perl -e '
985 use Benchmark;
986 use strict;
987 our $l = 10000;
988 our $u = our $b = "x" x $l;
989 substr($u,0,1) = "\x{100}";
990 timethese(-2,{
991 LENGTH_B => q{ length($b) },
992 LENGTH_U => q{ length($u) },
993 SUBSTR_B => q{ substr($b, $l/4, $l/2) },
994 SUBSTR_U => q{ substr($u, $l/4, $l/2) },
995 });
996 '
997 Benchmark: running LENGTH_B, LENGTH_U, SUBSTR_B, SUBSTR_U for at least 2 CPU seconds...
998 LENGTH_B: 2 wallclock secs ( 2.36 usr + 0.00 sys = 2.36 CPU) @ 5649983.05/s (n=13333960)
999 LENGTH_U: 2 wallclock secs ( 2.11 usr + 0.00 sys = 2.11 CPU) @ 12155.45/s (n=25648)
1000 SUBSTR_B: 3 wallclock secs ( 2.16 usr + 0.00 sys = 2.16 CPU) @ 374480.09/s (n=808877)
1001 SUBSTR_U: 2 wallclock secs ( 2.11 usr + 0.00 sys = 2.11 CPU) @ 6791.00/s (n=14329)
666f95b9 1002
c29a771d
JH
1003The numbers show an incredible slowness on long UTF-8 strings and you
1004should carefully avoid to use these functions within tight loops. For
1005example if you want to iterate over characters, it is infinitely
1006better to split into an array than to use substr, as the following
1007benchmark shows:
1008
1009 % perl -e '
1010 use Benchmark;
1011 use strict;
1012 our $l = 10000;
1013 our $u = our $b = "x" x $l;
1014 substr($u,0,1) = "\x{100}";
1015 timethese(-5,{
1016 SPLIT_B => q{ for my $c (split //, $b){} },
1017 SPLIT_U => q{ for my $c (split //, $u){} },
1018 SUBSTR_B => q{ for my $i (0..length($b)-1){my $c = substr($b,$i,1);} },
1019 SUBSTR_U => q{ for my $i (0..length($u)-1){my $c = substr($u,$i,1);} },
1020 });
1021 '
1022 Benchmark: running SPLIT_B, SPLIT_U, SUBSTR_B, SUBSTR_U for at least 5 CPU seconds...
1023 SPLIT_B: 6 wallclock secs ( 5.29 usr + 0.00 sys = 5.29 CPU) @ 56.14/s (n=297)
1024 SPLIT_U: 5 wallclock secs ( 5.17 usr + 0.01 sys = 5.18 CPU) @ 55.21/s (n=286)
1025 SUBSTR_B: 5 wallclock secs ( 5.34 usr + 0.00 sys = 5.34 CPU) @ 123.22/s (n=658)
1026 SUBSTR_U: 7 wallclock secs ( 6.20 usr + 0.00 sys = 6.20 CPU) @ 0.81/s (n=5)
1027
1028You see, the algorithm based on substr() was faster with byte encoded
1029data but it is pathologically slow with UTF-8 data.
666f95b9 1030
393fec97
GS
1031=head1 SEE ALSO
1032
72ff2908
JH
1033L<perluniintro>, L<encoding>, L<Encode>, L<open>, L<utf8>, L<bytes>,
1034L<perlretut>, L<perlvar/"${^WIDE_SYSTEM_CALLS}">
393fec97
GS
1035
1036=cut