This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
mg.c: using #ifdef inside a macro call ENONPORTABLE (causes e.g. AIX to barf)
[perl5.git] / pod / perlunicode.pod
CommitLineData
393fec97
GS
1=head1 NAME
2
3perlunicode - Unicode support in Perl
4
5=head1 DESCRIPTION
6
0a1f2d14 7=head2 Important Caveats
21bad921 8
376d9008 9Unicode support is an extensive requirement. While Perl does not
c349b1b9
JH
10implement the Unicode standard or the accompanying technical reports
11from cover to cover, Perl does support many Unicode features.
21bad921 12
13a2d996 13=over 4
21bad921 14
fae2c0fb 15=item Input and Output Layers
21bad921 16
376d9008 17Perl knows when a filehandle uses Perl's internal Unicode encodings
1bfb14c4
JH
18(UTF-8, or UTF-EBCDIC if in EBCDIC) if the filehandle is opened with
19the ":utf8" layer. Other encodings can be converted to Perl's
20encoding on input or from Perl's encoding on output by use of the
21":encoding(...)" layer. See L<open>.
c349b1b9 22
376d9008 23To indicate that Perl source itself is using a particular encoding,
c349b1b9 24see L<encoding>.
21bad921
GS
25
26=item Regular Expressions
27
c349b1b9 28The regular expression compiler produces polymorphic opcodes. That is,
376d9008
JB
29the pattern adapts to the data and automatically switches to the Unicode
30character scheme when presented with Unicode data--or instead uses
31a traditional byte scheme when presented with byte data.
21bad921 32
ad0029c4 33=item C<use utf8> still needed to enable UTF-8/UTF-EBCDIC in scripts
21bad921 34
376d9008
JB
35As a compatibility measure, the C<use utf8> pragma must be explicitly
36included to enable recognition of UTF-8 in the Perl scripts themselves
1bfb14c4
JH
37(in string or regular expression literals, or in identifier names) on
38ASCII-based machines or to recognize UTF-EBCDIC on EBCDIC-based
376d9008 39machines. B<These are the only times when an explicit C<use utf8>
8f8cf39c 40is needed.> See L<utf8>.
21bad921 41
1768d7eb 42You can also use the C<encoding> pragma to change the default encoding
6ec9efec 43of the data in your script; see L<encoding>.
1768d7eb 44
7aa207d6
JH
45=item BOM-marked scripts and UTF-16 scripts autodetected
46
47If a Perl script begins marked with the Unicode BOM (UTF-16LE, UTF16-BE,
48or UTF-8), or if the script looks like non-BOM-marked UTF-16 of either
49endianness, Perl will correctly read in the script as Unicode.
50(BOMless UTF-8 cannot be effectively recognized or differentiated from
51ISO 8859-1 or other eight-bit encodings.)
52
990e18f7
AT
53=item C<use encoding> needed to upgrade non-Latin-1 byte strings
54
55By default, there is a fundamental asymmetry in Perl's unicode model:
56implicit upgrading from byte strings to Unicode strings assumes that
57they were encoded in I<ISO 8859-1 (Latin-1)>, but Unicode strings are
58downgraded with UTF-8 encoding. This happens because the first 256
59codepoints in Unicode happens to agree with Latin-1.
60
61If you wish to interpret byte strings as UTF-8 instead, use the
62C<encoding> pragma:
63
64 use encoding 'utf8';
65
66See L</"Byte and Character Semantics"> for more details.
67
21bad921
GS
68=back
69
376d9008 70=head2 Byte and Character Semantics
393fec97 71
376d9008 72Beginning with version 5.6, Perl uses logically-wide characters to
3e4dbfed 73represent strings internally.
393fec97 74
376d9008
JB
75In future, Perl-level operations will be expected to work with
76characters rather than bytes.
393fec97 77
376d9008 78However, as an interim compatibility measure, Perl aims to
75daf61c
JH
79provide a safe migration path from byte semantics to character
80semantics for programs. For operations where Perl can unambiguously
376d9008 81decide that the input data are characters, Perl switches to
75daf61c
JH
82character semantics. For operations where this determination cannot
83be made without additional information from the user, Perl decides in
376d9008 84favor of compatibility and chooses to use byte semantics.
8cbd9a7a
GS
85
86This behavior preserves compatibility with earlier versions of Perl,
376d9008
JB
87which allowed byte semantics in Perl operations only if
88none of the program's inputs were marked as being as source of Unicode
8cbd9a7a
GS
89character data. Such data may come from filehandles, from calls to
90external programs, from information provided by the system (such as %ENV),
21bad921 91or from literals and constants in the source text.
8cbd9a7a 92
376d9008
JB
93The C<bytes> pragma will always, regardless of platform, force byte
94semantics in a particular lexical scope. See L<bytes>.
8cbd9a7a
GS
95
96The C<utf8> pragma is primarily a compatibility device that enables
75daf61c 97recognition of UTF-(8|EBCDIC) in literals encountered by the parser.
376d9008
JB
98Note that this pragma is only required while Perl defaults to byte
99semantics; when character semantics become the default, this pragma
100may become a no-op. See L<utf8>.
101
102Unless explicitly stated, Perl operators use character semantics
103for Unicode data and byte semantics for non-Unicode data.
104The decision to use character semantics is made transparently. If
105input data comes from a Unicode source--for example, if a character
fae2c0fb 106encoding layer is added to a filehandle or a literal Unicode
376d9008
JB
107string constant appears in a program--character semantics apply.
108Otherwise, byte semantics are in effect. The C<bytes> pragma should
109be used to force byte semantics on Unicode data.
110
111If strings operating under byte semantics and strings with Unicode
990e18f7
AT
112character data are concatenated, the new string will be created by
113decoding the byte strings as I<ISO 8859-1 (Latin-1)>, even if the
114old Unicode string used EBCDIC. This translation is done without
115regard to the system's native 8-bit encoding. To change this for
116systems with non-Latin-1 and non-EBCDIC native encodings, use the
117C<encoding> pragma. See L<encoding>.
7dedd01f 118
feda178f 119Under character semantics, many operations that formerly operated on
376d9008 120bytes now operate on characters. A character in Perl is
feda178f 121logically just a number ranging from 0 to 2**31 or so. Larger
376d9008
JB
122characters may encode into longer sequences of bytes internally, but
123this internal detail is mostly hidden for Perl code.
124See L<perluniintro> for more.
393fec97 125
376d9008 126=head2 Effects of Character Semantics
393fec97
GS
127
128Character semantics have the following effects:
129
130=over 4
131
132=item *
133
376d9008 134Strings--including hash keys--and regular expression patterns may
574c8022 135contain characters that have an ordinal value larger than 255.
393fec97 136
feda178f
JH
137If you use a Unicode editor to edit your program, Unicode characters
138may occur directly within the literal strings in one of the various
376d9008
JB
139Unicode encodings (UTF-8, UTF-EBCDIC, UCS-2, etc.), but will be recognized
140as such and converted to Perl's internal representation only if the
feda178f 141appropriate L<encoding> is specified.
3e4dbfed 142
1bfb14c4
JH
143Unicode characters can also be added to a string by using the
144C<\x{...}> notation. The Unicode code for the desired character, in
376d9008
JB
145hexadecimal, should be placed in the braces. For instance, a smiley
146face is C<\x{263A}>. This encoding scheme only works for characters
147with a code of 0x100 or above.
3e4dbfed
JF
148
149Additionally, if you
574c8022 150
3e4dbfed 151 use charnames ':full';
574c8022 152
1bfb14c4
JH
153you can use the C<\N{...}> notation and put the official Unicode
154character name within the braces, such as C<\N{WHITE SMILING FACE}>.
376d9008 155
393fec97
GS
156
157=item *
158
574c8022
JH
159If an appropriate L<encoding> is specified, identifiers within the
160Perl script may contain Unicode alphanumeric characters, including
376d9008
JB
161ideographs. Perl does not currently attempt to canonicalize variable
162names.
393fec97 163
393fec97
GS
164=item *
165
1bfb14c4
JH
166Regular expressions match characters instead of bytes. "." matches
167a character instead of a byte. The C<\C> pattern is provided to force
168a match a single byte--a C<char> in C, hence C<\C>.
393fec97 169
393fec97
GS
170=item *
171
172Character classes in regular expressions match characters instead of
376d9008 173bytes and match against the character properties specified in the
1bfb14c4 174Unicode properties database. C<\w> can be used to match a Japanese
75daf61c 175ideograph, for instance.
393fec97 176
b08eb2a8
RGS
177(However, and as a limitation of the current implementation, using
178C<\w> or C<\W> I<inside> a C<[...]> character class will still match
179with byte semantics.)
180
393fec97
GS
181=item *
182
eb0cc9e3 183Named Unicode properties, scripts, and block ranges may be used like
376d9008
JB
184character classes via the C<\p{}> "matches property" construct and
185the C<\P{}> negation, "doesn't match property".
1bfb14c4
JH
186
187For instance, C<\p{Lu}> matches any character with the Unicode "Lu"
188(Letter, uppercase) property, while C<\p{M}> matches any character
189with an "M" (mark--accents and such) property. Brackets are not
190required for single letter properties, so C<\p{M}> is equivalent to
191C<\pM>. Many predefined properties are available, such as
192C<\p{Mirrored}> and C<\p{Tibetan}>.
4193bef7 193
cfc01aea 194The official Unicode script and block names have spaces and dashes as
376d9008 195separators, but for convenience you can use dashes, spaces, or
1bfb14c4
JH
196underbars, and case is unimportant. It is recommended, however, that
197for consistency you use the following naming: the official Unicode
198script, property, or block name (see below for the additional rules
199that apply to block names) with whitespace and dashes removed, and the
200words "uppercase-first-lowercase-rest". C<Latin-1 Supplement> thus
201becomes C<Latin1Supplement>.
4193bef7 202
376d9008
JB
203You can also use negation in both C<\p{}> and C<\P{}> by introducing a caret
204(^) between the first brace and the property name: C<\p{^Tamil}> is
eb0cc9e3 205equal to C<\P{Tamil}>.
4193bef7 206
14bb0a9a
JH
207B<NOTE: the properties, scripts, and blocks listed here are as of
208Unicode 3.2.0, March 2002, or Perl 5.8.0, July 2002. Unicode 4.0.0
209came out in April 2003, and Perl 5.8.1 in September 2003.>
210
eb0cc9e3 211Here are the basic Unicode General Category properties, followed by their
68cd2d32 212long form. You can use either; C<\p{Lu}> and C<\p{UppercaseLetter}>,
376d9008 213for instance, are identical.
393fec97 214
d73e5302
JH
215 Short Long
216
217 L Letter
12ac2576 218 LC CasedLetter
eb0cc9e3
JH
219 Lu UppercaseLetter
220 Ll LowercaseLetter
221 Lt TitlecaseLetter
222 Lm ModifierLetter
223 Lo OtherLetter
d73e5302
JH
224
225 M Mark
eb0cc9e3
JH
226 Mn NonspacingMark
227 Mc SpacingMark
228 Me EnclosingMark
d73e5302
JH
229
230 N Number
eb0cc9e3
JH
231 Nd DecimalNumber
232 Nl LetterNumber
233 No OtherNumber
d73e5302
JH
234
235 P Punctuation
eb0cc9e3
JH
236 Pc ConnectorPunctuation
237 Pd DashPunctuation
238 Ps OpenPunctuation
239 Pe ClosePunctuation
240 Pi InitialPunctuation
d73e5302 241 (may behave like Ps or Pe depending on usage)
eb0cc9e3 242 Pf FinalPunctuation
d73e5302 243 (may behave like Ps or Pe depending on usage)
eb0cc9e3 244 Po OtherPunctuation
d73e5302
JH
245
246 S Symbol
eb0cc9e3
JH
247 Sm MathSymbol
248 Sc CurrencySymbol
249 Sk ModifierSymbol
250 So OtherSymbol
d73e5302
JH
251
252 Z Separator
eb0cc9e3
JH
253 Zs SpaceSeparator
254 Zl LineSeparator
255 Zp ParagraphSeparator
d73e5302
JH
256
257 C Other
e150c829
JH
258 Cc Control
259 Cf Format
eb0cc9e3
JH
260 Cs Surrogate (not usable)
261 Co PrivateUse
e150c829 262 Cn Unassigned
1ac13f9a 263
376d9008 264Single-letter properties match all characters in any of the
3e4dbfed 265two-letter sub-properties starting with the same letter.
12ac2576
JP
266C<LC> and C<L&> are special cases, which are aliases for the set of
267C<Ll>, C<Lu>, and C<Lt>.
32293815 268
eb0cc9e3 269Because Perl hides the need for the user to understand the internal
1bfb14c4
JH
270representation of Unicode characters, there is no need to implement
271the somewhat messy concept of surrogates. C<Cs> is therefore not
eb0cc9e3 272supported.
d73e5302 273
376d9008 274Because scripts differ in their directionality--Hebrew is
12ac2576
JP
275written right to left, for example--Unicode supplies these properties in
276the BidiClass class:
32293815 277
eb0cc9e3 278 Property Meaning
92e830a9 279
12ac2576
JP
280 L Left-to-Right
281 LRE Left-to-Right Embedding
282 LRO Left-to-Right Override
283 R Right-to-Left
284 AL Right-to-Left Arabic
285 RLE Right-to-Left Embedding
286 RLO Right-to-Left Override
287 PDF Pop Directional Format
288 EN European Number
289 ES European Number Separator
290 ET European Number Terminator
291 AN Arabic Number
292 CS Common Number Separator
293 NSM Non-Spacing Mark
294 BN Boundary Neutral
295 B Paragraph Separator
296 S Segment Separator
297 WS Whitespace
298 ON Other Neutrals
299
300For example, C<\p{BidiClass:R}> matches characters that are normally
eb0cc9e3
JH
301written right to left.
302
210b36aa
AMS
303=back
304
2796c109
JH
305=head2 Scripts
306
376d9008
JB
307The script names which can be used by C<\p{...}> and C<\P{...}>,
308such as in C<\p{Latin}> or C<\p{Cyrillic}>, are as follows:
2796c109 309
1ac13f9a 310 Arabic
e9ad1727 311 Armenian
1ac13f9a 312 Bengali
e9ad1727 313 Bopomofo
1d81abf3 314 Buhid
eb0cc9e3 315 CanadianAboriginal
e9ad1727
JH
316 Cherokee
317 Cyrillic
318 Deseret
319 Devanagari
320 Ethiopic
321 Georgian
322 Gothic
323 Greek
1ac13f9a 324 Gujarati
e9ad1727
JH
325 Gurmukhi
326 Han
327 Hangul
1d81abf3 328 Hanunoo
e9ad1727
JH
329 Hebrew
330 Hiragana
331 Inherited
1ac13f9a 332 Kannada
e9ad1727
JH
333 Katakana
334 Khmer
1ac13f9a 335 Lao
e9ad1727
JH
336 Latin
337 Malayalam
338 Mongolian
1ac13f9a 339 Myanmar
1ac13f9a 340 Ogham
eb0cc9e3 341 OldItalic
e9ad1727 342 Oriya
1ac13f9a 343 Runic
e9ad1727
JH
344 Sinhala
345 Syriac
1d81abf3
JH
346 Tagalog
347 Tagbanwa
e9ad1727
JH
348 Tamil
349 Telugu
350 Thaana
351 Thai
352 Tibetan
1ac13f9a 353 Yi
1ac13f9a 354
376d9008 355Extended property classes can supplement the basic
1ac13f9a
JH
356properties, defined by the F<PropList> Unicode database:
357
1d81abf3 358 ASCIIHexDigit
eb0cc9e3 359 BidiControl
1ac13f9a 360 Dash
1d81abf3 361 Deprecated
1ac13f9a
JH
362 Diacritic
363 Extender
1d81abf3 364 GraphemeLink
eb0cc9e3 365 HexDigit
e9ad1727
JH
366 Hyphen
367 Ideographic
1d81abf3
JH
368 IDSBinaryOperator
369 IDSTrinaryOperator
eb0cc9e3 370 JoinControl
1d81abf3 371 LogicalOrderException
eb0cc9e3
JH
372 NoncharacterCodePoint
373 OtherAlphabetic
1d81abf3
JH
374 OtherDefaultIgnorableCodePoint
375 OtherGraphemeExtend
eb0cc9e3
JH
376 OtherLowercase
377 OtherMath
378 OtherUppercase
379 QuotationMark
1d81abf3
JH
380 Radical
381 SoftDotted
382 TerminalPunctuation
383 UnifiedIdeograph
eb0cc9e3 384 WhiteSpace
1ac13f9a 385
376d9008 386and there are further derived properties:
1ac13f9a 387
eb0cc9e3
JH
388 Alphabetic Lu + Ll + Lt + Lm + Lo + OtherAlphabetic
389 Lowercase Ll + OtherLowercase
390 Uppercase Lu + OtherUppercase
391 Math Sm + OtherMath
1ac13f9a
JH
392
393 ID_Start Lu + Ll + Lt + Lm + Lo + Nl
394 ID_Continue ID_Start + Mn + Mc + Nd + Pc
395
396 Any Any character
66b79f27
RGS
397 Assigned Any non-Cn character (i.e. synonym for \P{Cn})
398 Unassigned Synonym for \p{Cn}
1ac13f9a 399 Common Any character (or unassigned code point)
e150c829 400 not explicitly assigned to a script
2796c109 401
1bfb14c4
JH
402For backward compatibility (with Perl 5.6), all properties mentioned
403so far may have C<Is> prepended to their name, so C<\P{IsLu}>, for
404example, is equal to C<\P{Lu}>.
eb0cc9e3 405
2796c109
JH
406=head2 Blocks
407
1bfb14c4
JH
408In addition to B<scripts>, Unicode also defines B<blocks> of
409characters. The difference between scripts and blocks is that the
410concept of scripts is closer to natural languages, while the concept
411of blocks is more of an artificial grouping based on groups of 256
376d9008 412Unicode characters. For example, the C<Latin> script contains letters
1bfb14c4 413from many blocks but does not contain all the characters from those
376d9008
JB
414blocks. It does not, for example, contain digits, because digits are
415shared across many scripts. Digits and similar groups, like
416punctuation, are in a category called C<Common>.
2796c109 417
cfc01aea
JF
418For more about scripts, see the UTR #24:
419
420 http://www.unicode.org/unicode/reports/tr24/
421
422For more about blocks, see:
423
424 http://www.unicode.org/Public/UNIDATA/Blocks.txt
2796c109 425
376d9008
JB
426Block names are given with the C<In> prefix. For example, the
427Katakana block is referenced via C<\p{InKatakana}>. The C<In>
7eabb34d 428prefix may be omitted if there is no naming conflict with a script
eb0cc9e3 429or any other property, but it is recommended that C<In> always be used
1bfb14c4 430for block tests to avoid confusion.
eb0cc9e3
JH
431
432These block names are supported:
433
1d81abf3
JH
434 InAlphabeticPresentationForms
435 InArabic
436 InArabicPresentationFormsA
437 InArabicPresentationFormsB
438 InArmenian
439 InArrows
440 InBasicLatin
441 InBengali
442 InBlockElements
443 InBopomofo
444 InBopomofoExtended
445 InBoxDrawing
446 InBraillePatterns
447 InBuhid
448 InByzantineMusicalSymbols
449 InCJKCompatibility
450 InCJKCompatibilityForms
451 InCJKCompatibilityIdeographs
452 InCJKCompatibilityIdeographsSupplement
453 InCJKRadicalsSupplement
454 InCJKSymbolsAndPunctuation
455 InCJKUnifiedIdeographs
456 InCJKUnifiedIdeographsExtensionA
457 InCJKUnifiedIdeographsExtensionB
458 InCherokee
459 InCombiningDiacriticalMarks
460 InCombiningDiacriticalMarksforSymbols
461 InCombiningHalfMarks
462 InControlPictures
463 InCurrencySymbols
464 InCyrillic
465 InCyrillicSupplementary
466 InDeseret
467 InDevanagari
468 InDingbats
469 InEnclosedAlphanumerics
470 InEnclosedCJKLettersAndMonths
471 InEthiopic
472 InGeneralPunctuation
473 InGeometricShapes
474 InGeorgian
475 InGothic
476 InGreekExtended
477 InGreekAndCoptic
478 InGujarati
479 InGurmukhi
480 InHalfwidthAndFullwidthForms
481 InHangulCompatibilityJamo
482 InHangulJamo
483 InHangulSyllables
484 InHanunoo
485 InHebrew
486 InHighPrivateUseSurrogates
487 InHighSurrogates
488 InHiragana
489 InIPAExtensions
490 InIdeographicDescriptionCharacters
491 InKanbun
492 InKangxiRadicals
493 InKannada
494 InKatakana
495 InKatakanaPhoneticExtensions
496 InKhmer
497 InLao
498 InLatin1Supplement
499 InLatinExtendedA
500 InLatinExtendedAdditional
501 InLatinExtendedB
502 InLetterlikeSymbols
503 InLowSurrogates
504 InMalayalam
505 InMathematicalAlphanumericSymbols
506 InMathematicalOperators
507 InMiscellaneousMathematicalSymbolsA
508 InMiscellaneousMathematicalSymbolsB
509 InMiscellaneousSymbols
510 InMiscellaneousTechnical
511 InMongolian
512 InMusicalSymbols
513 InMyanmar
514 InNumberForms
515 InOgham
516 InOldItalic
517 InOpticalCharacterRecognition
518 InOriya
519 InPrivateUseArea
520 InRunic
521 InSinhala
522 InSmallFormVariants
523 InSpacingModifierLetters
524 InSpecials
525 InSuperscriptsAndSubscripts
526 InSupplementalArrowsA
527 InSupplementalArrowsB
528 InSupplementalMathematicalOperators
529 InSupplementaryPrivateUseAreaA
530 InSupplementaryPrivateUseAreaB
531 InSyriac
532 InTagalog
533 InTagbanwa
534 InTags
535 InTamil
536 InTelugu
537 InThaana
538 InThai
539 InTibetan
540 InUnifiedCanadianAboriginalSyllabics
541 InVariationSelectors
542 InYiRadicals
543 InYiSyllables
32293815 544
210b36aa
AMS
545=over 4
546
393fec97
GS
547=item *
548
376d9008
JB
549The special pattern C<\X> matches any extended Unicode
550sequence--"a combining character sequence" in Standardese--where the
551first character is a base character and subsequent characters are mark
552characters that apply to the base character. C<\X> is equivalent to
393fec97
GS
553C<(?:\PM\pM*)>.
554
393fec97
GS
555=item *
556
383e7cdd 557The C<tr///> operator translates characters instead of bytes. Note
376d9008
JB
558that the C<tr///CU> functionality has been removed. For similar
559functionality see pack('U0', ...) and pack('C0', ...).
393fec97 560
393fec97
GS
561=item *
562
563Case translation operators use the Unicode case translation tables
376d9008
JB
564when character input is provided. Note that C<uc()>, or C<\U> in
565interpolated strings, translates to uppercase, while C<ucfirst>,
566or C<\u> in interpolated strings, translates to titlecase in languages
567that make the distinction.
393fec97
GS
568
569=item *
570
376d9008 571Most operators that deal with positions or lengths in a string will
75daf61c 572automatically switch to using character positions, including
f5b005ca 573C<chop()>, C<chomp()>, C<substr()>, C<pos()>, C<index()>, C<rindex()>,
f337b084
TH
574C<sprintf()>, C<write()>, and C<length()>. An operator that
575specifically does not switch is C<vec()>. Operators that really don't
576care include operators that treat strings as a bucket of bits such as
577C<sort()>, and operators dealing with filenames.
393fec97
GS
578
579=item *
580
f337b084
TH
581The C<pack()>/C<unpack()> letter C<C> does I<not> change, since it is often
582used for byte-oriented formats. Again, think C<char> in the C language.
1bfb14c4
JH
583
584There is a new C<U> specifier that converts between Unicode characters
f337b084
TH
585and code points. There is also a C<W> specifier that is the equivalent of
586C<chr>/C<ord> and properly handles character values even if they are above 255.
393fec97
GS
587
588=item *
589
376d9008 590The C<chr()> and C<ord()> functions work on characters, similar to
f337b084 591C<pack("W")> and C<unpack("W")>, I<not> C<pack("C")> and
376d9008
JB
592C<unpack("C")>. C<pack("C")> and C<unpack("C")> are methods for
593emulating byte-oriented C<chr()> and C<ord()> on Unicode strings.
594While these methods reveal the internal encoding of Unicode strings,
595that is not something one normally needs to care about at all.
393fec97
GS
596
597=item *
598
376d9008
JB
599The bit string operators, C<& | ^ ~>, can operate on character data.
600However, for backward compatibility, such as when using bit string
601operations when characters are all less than 256 in ordinal value, one
602should not use C<~> (the bit complement) with characters of both
603values less than 256 and values greater than 256. Most importantly,
604DeMorgan's laws (C<~($x|$y) eq ~$x&~$y> and C<~($x&$y) eq ~$x|~$y>)
605will not hold. The reason for this mathematical I<faux pas> is that
606the complement cannot return B<both> the 8-bit (byte-wide) bit
607complement B<and> the full character-wide bit complement.
a1ca4561
YST
608
609=item *
610
983ffd37
JH
611lc(), uc(), lcfirst(), and ucfirst() work for the following cases:
612
613=over 8
614
615=item *
616
617the case mapping is from a single Unicode character to another
376d9008 618single Unicode character, or
983ffd37
JH
619
620=item *
621
622the case mapping is from a single Unicode character to more
376d9008 623than one Unicode character.
983ffd37
JH
624
625=back
626
63de3cb2
JH
627Things to do with locales (Lithuanian, Turkish, Azeri) do B<not> work
628since Perl does not understand the concept of Unicode locales.
983ffd37 629
dc33ebcf
RGS
630See the Unicode Technical Report #21, Case Mappings, for more details.
631
983ffd37
JH
632=back
633
dc33ebcf 634=over 4
ac1256e8
JH
635
636=item *
637
393fec97
GS
638And finally, C<scalar reverse()> reverses by character rather than by byte.
639
640=back
641
376d9008 642=head2 User-Defined Character Properties
491fd90a
JH
643
644You can define your own character properties by defining subroutines
bac0b425
JP
645whose names begin with "In" or "Is". The subroutines can be defined in
646any package. The user-defined properties can be used in the regular
647expression C<\p> and C<\P> constructs; if you are using a user-defined
648property from a package other than the one you are in, you must specify
649its package in the C<\p> or C<\P> construct.
650
651 # assuming property IsForeign defined in Lang::
652 package main; # property package name required
653 if ($txt =~ /\p{Lang::IsForeign}+/) { ... }
654
655 package Lang; # property package name not required
656 if ($txt =~ /\p{IsForeign}+/) { ... }
657
658
659Note that the effect is compile-time and immutable once defined.
491fd90a 660
376d9008
JB
661The subroutines must return a specially-formatted string, with one
662or more newline-separated lines. Each line must be one of the following:
491fd90a
JH
663
664=over 4
665
666=item *
667
99a6b1f0 668Two hexadecimal numbers separated by horizontal whitespace (space or
376d9008 669tabular characters) denoting a range of Unicode code points to include.
491fd90a
JH
670
671=item *
672
376d9008 673Something to include, prefixed by "+": a built-in character
bac0b425
JP
674property (prefixed by "utf8::") or a user-defined character property,
675to represent all the characters in that property; two hexadecimal code
676points for a range; or a single hexadecimal code point.
491fd90a
JH
677
678=item *
679
376d9008 680Something to exclude, prefixed by "-": an existing character
bac0b425
JP
681property (prefixed by "utf8::") or a user-defined character property,
682to represent all the characters in that property; two hexadecimal code
683points for a range; or a single hexadecimal code point.
491fd90a
JH
684
685=item *
686
376d9008 687Something to negate, prefixed "!": an existing character
bac0b425
JP
688property (prefixed by "utf8::") or a user-defined character property,
689to represent all the characters in that property; two hexadecimal code
690points for a range; or a single hexadecimal code point.
691
692=item *
693
694Something to intersect with, prefixed by "&": an existing character
695property (prefixed by "utf8::") or a user-defined character property,
696for all the characters except the characters in the property; two
697hexadecimal code points for a range; or a single hexadecimal code point.
491fd90a
JH
698
699=back
700
701For example, to define a property that covers both the Japanese
702syllabaries (hiragana and katakana), you can define
703
704 sub InKana {
d5822f25
A
705 return <<END;
706 3040\t309F
707 30A0\t30FF
491fd90a
JH
708 END
709 }
710
d5822f25
A
711Imagine that the here-doc end marker is at the beginning of the line.
712Now you can use C<\p{InKana}> and C<\P{InKana}>.
491fd90a
JH
713
714You could also have used the existing block property names:
715
716 sub InKana {
717 return <<'END';
718 +utf8::InHiragana
719 +utf8::InKatakana
720 END
721 }
722
723Suppose you wanted to match only the allocated characters,
d5822f25 724not the raw block ranges: in other words, you want to remove
491fd90a
JH
725the non-characters:
726
727 sub InKana {
728 return <<'END';
729 +utf8::InHiragana
730 +utf8::InKatakana
731 -utf8::IsCn
732 END
733 }
734
735The negation is useful for defining (surprise!) negated classes.
736
737 sub InNotKana {
738 return <<'END';
739 !utf8::InHiragana
740 -utf8::InKatakana
741 +utf8::IsCn
742 END
743 }
744
bac0b425
JP
745Intersection is useful for getting the common characters matched by
746two (or more) classes.
747
748 sub InFooAndBar {
749 return <<'END';
750 +main::Foo
751 &main::Bar
752 END
753 }
754
755It's important to remember not to use "&" for the first set -- that
756would be intersecting with nothing (resulting in an empty set).
757
3a2263fe
RGS
758You can also define your own mappings to be used in the lc(),
759lcfirst(), uc(), and ucfirst() (or their string-inlined versions).
760The principle is the same: define subroutines in the C<main> package
761with names like C<ToLower> (for lc() and lcfirst()), C<ToTitle> (for
762the first character in ucfirst()), and C<ToUpper> (for uc(), and the
763rest of the characters in ucfirst()).
764
765The string returned by the subroutines needs now to be three
766hexadecimal numbers separated by tabulators: start of the source
767range, end of the source range, and start of the destination range.
768For example:
769
770 sub ToUpper {
771 return <<END;
772 0061\t0063\t0041
773 END
774 }
775
776defines an uc() mapping that causes only the characters "a", "b", and
777"c" to be mapped to "A", "B", "C", all other characters will remain
778unchanged.
779
780If there is no source range to speak of, that is, the mapping is from
781a single character to another single character, leave the end of the
782source range empty, but the two tabulator characters are still needed.
783For example:
784
785 sub ToLower {
786 return <<END;
787 0041\t\t0061
788 END
789 }
790
791defines a lc() mapping that causes only "A" to be mapped to "a", all
792other characters will remain unchanged.
793
794(For serious hackers only) If you want to introspect the default
795mappings, you can find the data in the directory
796C<$Config{privlib}>/F<unicore/To/>. The mapping data is returned as
797the here-document, and the C<utf8::ToSpecFoo> are special exception
798mappings derived from <$Config{privlib}>/F<unicore/SpecialCasing.txt>.
799The C<Digit> and C<Fold> mappings that one can see in the directory
800are not directly user-accessible, one can use either the
801C<Unicode::UCD> module, or just match case-insensitively (that's when
802the C<Fold> mapping is used).
803
804A final note on the user-defined property tests and mappings: they
805will be used only if the scalar has been marked as having Unicode
806characters. Old byte-style strings will not be affected.
807
376d9008 808=head2 Character Encodings for Input and Output
8cbd9a7a 809
7221edc9 810See L<Encode>.
8cbd9a7a 811
c29a771d 812=head2 Unicode Regular Expression Support Level
776f8809 813
376d9008
JB
814The following list of Unicode support for regular expressions describes
815all the features currently supported. The references to "Level N"
816and the section numbers refer to the Unicode Technical Report 18,
965cd703
JH
817"Unicode Regular Expression Guidelines", version 6 (Unicode 3.2.0,
818Perl 5.8.0).
776f8809
JH
819
820=over 4
821
822=item *
823
824Level 1 - Basic Unicode Support
825
826 2.1 Hex Notation - done [1]
3bfdc84c 827 Named Notation - done [2]
776f8809
JH
828 2.2 Categories - done [3][4]
829 2.3 Subtraction - MISSING [5][6]
830 2.4 Simple Word Boundaries - done [7]
78d3e1bf 831 2.5 Simple Loose Matches - done [8]
776f8809
JH
832 2.6 End of Line - MISSING [9][10]
833
834 [ 1] \x{...}
835 [ 2] \N{...}
eb0cc9e3 836 [ 3] . \p{...} \P{...}
12ac2576
JP
837 [ 4] support for scripts (see UTR#24 Script Names), blocks,
838 binary properties, enumerated non-binary properties, and
839 numeric properties (as listed in UTR#18 Other Properties)
776f8809 840 [ 5] have negation
237bad5b
JH
841 [ 6] can use regular expression look-ahead [a]
842 or user-defined character properties [b] to emulate subtraction
776f8809 843 [ 7] include Letters in word characters
376d9008 844 [ 8] note that Perl does Full case-folding in matching, not Simple:
835863de 845 for example U+1F88 is equivalent with U+1F00 U+03B9,
e0f9d4a8 846 not with 1F80. This difference matters for certain Greek
376d9008
JB
847 capital letters with certain modifiers: the Full case-folding
848 decomposes the letter, while the Simple case-folding would map
e0f9d4a8 849 it to a single character.
5ca1ac52 850 [ 9] see UTR #13 Unicode Newline Guidelines
835863de 851 [10] should do ^ and $ also on \x{85}, \x{2028} and \x{2029}
ec83e909 852 (should also affect <>, $., and script line numbers)
3bfdc84c 853 (the \x{85}, \x{2028} and \x{2029} do match \s)
7207e29d 854
237bad5b 855[a] You can mimic class subtraction using lookahead.
5ca1ac52 856For example, what UTR #18 might write as
29bdacb8 857
dbe420b4
JH
858 [{Greek}-[{UNASSIGNED}]]
859
860in Perl can be written as:
861
1d81abf3
JH
862 (?!\p{Unassigned})\p{InGreekAndCoptic}
863 (?=\p{Assigned})\p{InGreekAndCoptic}
dbe420b4
JH
864
865But in this particular example, you probably really want
866
1bfb14c4 867 \p{GreekAndCoptic}
dbe420b4
JH
868
869which will match assigned characters known to be part of the Greek script.
29bdacb8 870
5ca1ac52
JH
871Also see the Unicode::Regex::Set module, it does implement the full
872UTR #18 grouping, intersection, union, and removal (subtraction) syntax.
873
818c4caa 874[b] See L</"User-Defined Character Properties">.
237bad5b 875
776f8809
JH
876=item *
877
878Level 2 - Extended Unicode Support
879
63de3cb2
JH
880 3.1 Surrogates - MISSING [11]
881 3.2 Canonical Equivalents - MISSING [12][13]
882 3.3 Locale-Independent Graphemes - MISSING [14]
883 3.4 Locale-Independent Words - MISSING [15]
884 3.5 Locale-Independent Loose Matches - MISSING [16]
885
886 [11] Surrogates are solely a UTF-16 concept and Perl's internal
887 representation is UTF-8. The Encode module does UTF-16, though.
888 [12] see UTR#15 Unicode Normalization
889 [13] have Unicode::Normalize but not integrated to regexes
890 [14] have \X but at this level . should equal that
891 [15] need three classes, not just \w and \W
892 [16] see UTR#21 Case Mappings
776f8809
JH
893
894=item *
895
896Level 3 - Locale-Sensitive Support
897
898 4.1 Locale-Dependent Categories - MISSING
899 4.2 Locale-Dependent Graphemes - MISSING [16][17]
900 4.3 Locale-Dependent Words - MISSING
901 4.4 Locale-Dependent Loose Matches - MISSING
902 4.5 Locale-Dependent Ranges - MISSING
903
904 [16] see UTR#10 Unicode Collation Algorithms
905 [17] have Unicode::Collate but not integrated to regexes
906
907=back
908
c349b1b9
JH
909=head2 Unicode Encodings
910
376d9008
JB
911Unicode characters are assigned to I<code points>, which are abstract
912numbers. To use these numbers, various encodings are needed.
c349b1b9
JH
913
914=over 4
915
c29a771d 916=item *
5cb3728c
RB
917
918UTF-8
c349b1b9 919
3e4dbfed 920UTF-8 is a variable-length (1 to 6 bytes, current character allocations
376d9008
JB
921require 4 bytes), byte-order independent encoding. For ASCII (and we
922really do mean 7-bit ASCII, not another 8-bit encoding), UTF-8 is
923transparent.
c349b1b9 924
8c007b5a 925The following table is from Unicode 3.2.
05632f9a
JH
926
927 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
928
8c007b5a
JH
929 U+0000..U+007F 00..7F
930 U+0080..U+07FF C2..DF 80..BF
ec90690f
ST
931 U+0800..U+0FFF E0 A0..BF 80..BF
932 U+1000..U+CFFF E1..EC 80..BF 80..BF
933 U+D000..U+D7FF ED 80..9F 80..BF
8c007b5a 934 U+D800..U+DFFF ******* ill-formed *******
ec90690f 935 U+E000..U+FFFF EE..EF 80..BF 80..BF
05632f9a
JH
936 U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
937 U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
938 U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
939
376d9008
JB
940Note the C<A0..BF> in C<U+0800..U+0FFF>, the C<80..9F> in
941C<U+D000...U+D7FF>, the C<90..B>F in C<U+10000..U+3FFFF>, and the
942C<80...8F> in C<U+100000..U+10FFFF>. The "gaps" are caused by legal
943UTF-8 avoiding non-shortest encodings: it is technically possible to
944UTF-8-encode a single code point in different ways, but that is
945explicitly forbidden, and the shortest possible encoding should always
946be used. So that's what Perl does.
37361303 947
376d9008 948Another way to look at it is via bits:
05632f9a
JH
949
950 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
951
952 0aaaaaaa 0aaaaaaa
953 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
954 ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
955 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
956
957As you can see, the continuation bytes all begin with C<10>, and the
8c007b5a 958leading bits of the start byte tell how many bytes the are in the
05632f9a
JH
959encoded character.
960
c29a771d 961=item *
5cb3728c
RB
962
963UTF-EBCDIC
dbe420b4 964
376d9008 965Like UTF-8 but EBCDIC-safe, in the way that UTF-8 is ASCII-safe.
dbe420b4 966
c29a771d 967=item *
5cb3728c 968
1e54db1a 969UTF-16, UTF-16BE, UTF-16LE, Surrogates, and BOMs (Byte Order Marks)
c349b1b9 970
1bfb14c4
JH
971The followings items are mostly for reference and general Unicode
972knowledge, Perl doesn't use these constructs internally.
dbe420b4 973
c349b1b9 974UTF-16 is a 2 or 4 byte encoding. The Unicode code points
1bfb14c4
JH
975C<U+0000..U+FFFF> are stored in a single 16-bit unit, and the code
976points C<U+10000..U+10FFFF> in two 16-bit units. The latter case is
c349b1b9
JH
977using I<surrogates>, the first 16-bit unit being the I<high
978surrogate>, and the second being the I<low surrogate>.
979
376d9008 980Surrogates are code points set aside to encode the C<U+10000..U+10FFFF>
c349b1b9 981range of Unicode code points in pairs of 16-bit units. The I<high
376d9008
JB
982surrogates> are the range C<U+D800..U+DBFF>, and the I<low surrogates>
983are the range C<U+DC00..U+DFFF>. The surrogate encoding is
c349b1b9
JH
984
985 $hi = ($uni - 0x10000) / 0x400 + 0xD800;
986 $lo = ($uni - 0x10000) % 0x400 + 0xDC00;
987
988and the decoding is
989
1a3fa709 990 $uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00);
c349b1b9 991
feda178f 992If you try to generate surrogates (for example by using chr()), you
376d9008
JB
993will get a warning if warnings are turned on, because those code
994points are not valid for a Unicode character.
9466bab6 995
376d9008 996Because of the 16-bitness, UTF-16 is byte-order dependent. UTF-16
c349b1b9 997itself can be used for in-memory computations, but if storage or
376d9008
JB
998transfer is required either UTF-16BE (big-endian) or UTF-16LE
999(little-endian) encodings must be chosen.
c349b1b9
JH
1000
1001This introduces another problem: what if you just know that your data
376d9008
JB
1002is UTF-16, but you don't know which endianness? Byte Order Marks, or
1003BOMs, are a solution to this. A special character has been reserved
86bbd6d1 1004in Unicode to function as a byte order marker: the character with the
376d9008 1005code point C<U+FEFF> is the BOM.
042da322 1006
c349b1b9 1007The trick is that if you read a BOM, you will know the byte order,
376d9008
JB
1008since if it was written on a big-endian platform, you will read the
1009bytes C<0xFE 0xFF>, but if it was written on a little-endian platform,
1010you will read the bytes C<0xFF 0xFE>. (And if the originating platform
1011was writing in UTF-8, you will read the bytes C<0xEF 0xBB 0xBF>.)
042da322 1012
86bbd6d1 1013The way this trick works is that the character with the code point
376d9008
JB
1014C<U+FFFE> is guaranteed not to be a valid Unicode character, so the
1015sequence of bytes C<0xFF 0xFE> is unambiguously "BOM, represented in
1bfb14c4 1016little-endian format" and cannot be C<U+FFFE>, represented in big-endian
042da322 1017format".
c349b1b9 1018
c29a771d 1019=item *
5cb3728c 1020
1e54db1a 1021UTF-32, UTF-32BE, UTF-32LE
c349b1b9
JH
1022
1023The UTF-32 family is pretty much like the UTF-16 family, expect that
042da322 1024the units are 32-bit, and therefore the surrogate scheme is not
376d9008
JB
1025needed. The BOM signatures will be C<0x00 0x00 0xFE 0xFF> for BE and
1026C<0xFF 0xFE 0x00 0x00> for LE.
c349b1b9 1027
c29a771d 1028=item *
5cb3728c
RB
1029
1030UCS-2, UCS-4
c349b1b9 1031
86bbd6d1 1032Encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit
376d9008 1033encoding. Unlike UTF-16, UCS-2 is not extensible beyond C<U+FFFF>,
339cfa0e
JH
1034because it does not use surrogates. UCS-4 is a 32-bit encoding,
1035functionally identical to UTF-32.
c349b1b9 1036
c29a771d 1037=item *
5cb3728c
RB
1038
1039UTF-7
c349b1b9 1040
376d9008
JB
1041A seven-bit safe (non-eight-bit) encoding, which is useful if the
1042transport or storage is not eight-bit safe. Defined by RFC 2152.
c349b1b9 1043
95a1a48b
JH
1044=back
1045
0d7c09bb
JH
1046=head2 Security Implications of Unicode
1047
1048=over 4
1049
1050=item *
1051
1052Malformed UTF-8
bf0fa0b2
JH
1053
1054Unfortunately, the specification of UTF-8 leaves some room for
1055interpretation of how many bytes of encoded output one should generate
376d9008
JB
1056from one input Unicode character. Strictly speaking, the shortest
1057possible sequence of UTF-8 bytes should be generated,
1058because otherwise there is potential for an input buffer overflow at
feda178f 1059the receiving end of a UTF-8 connection. Perl always generates the
376d9008
JB
1060shortest length UTF-8, and with warnings on Perl will warn about
1061non-shortest length UTF-8 along with other malformations, such as the
1062surrogates, which are not real Unicode code points.
bf0fa0b2 1063
0d7c09bb
JH
1064=item *
1065
1066Regular expressions behave slightly differently between byte data and
376d9008
JB
1067character (Unicode) data. For example, the "word character" character
1068class C<\w> will work differently depending on if data is eight-bit bytes
1069or Unicode.
0d7c09bb 1070
376d9008
JB
1071In the first case, the set of C<\w> characters is either small--the
1072default set of alphabetic characters, digits, and the "_"--or, if you
0d7c09bb
JH
1073are using a locale (see L<perllocale>), the C<\w> might contain a few
1074more letters according to your language and country.
1075
376d9008 1076In the second case, the C<\w> set of characters is much, much larger.
1bfb14c4
JH
1077Most importantly, even in the set of the first 256 characters, it will
1078probably match different characters: unlike most locales, which are
1079specific to a language and country pair, Unicode classifies all the
1080characters that are letters I<somewhere> as C<\w>. For example, your
1081locale might not think that LATIN SMALL LETTER ETH is a letter (unless
1082you happen to speak Icelandic), but Unicode does.
0d7c09bb 1083
376d9008 1084As discussed elsewhere, Perl has one foot (two hooves?) planted in
1bfb14c4
JH
1085each of two worlds: the old world of bytes and the new world of
1086characters, upgrading from bytes to characters when necessary.
376d9008
JB
1087If your legacy code does not explicitly use Unicode, no automatic
1088switch-over to characters should happen. Characters shouldn't get
1bfb14c4
JH
1089downgraded to bytes, either. It is possible to accidentally mix bytes
1090and characters, however (see L<perluniintro>), in which case C<\w> in
1091regular expressions might start behaving differently. Review your
1092code. Use warnings and the C<strict> pragma.
0d7c09bb
JH
1093
1094=back
1095
c349b1b9
JH
1096=head2 Unicode in Perl on EBCDIC
1097
376d9008
JB
1098The way Unicode is handled on EBCDIC platforms is still
1099experimental. On such platforms, references to UTF-8 encoding in this
1100document and elsewhere should be read as meaning the UTF-EBCDIC
1101specified in Unicode Technical Report 16, unless ASCII vs. EBCDIC issues
c349b1b9 1102are specifically discussed. There is no C<utfebcdic> pragma or
376d9008 1103":utfebcdic" layer; rather, "utf8" and ":utf8" are reused to mean
86bbd6d1
PN
1104the platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic>
1105for more discussion of the issues.
c349b1b9 1106
b310b053
JH
1107=head2 Locales
1108
4616122b 1109Usually locale settings and Unicode do not affect each other, but
b310b053
JH
1110there are a couple of exceptions:
1111
1112=over 4
1113
1114=item *
1115
8aa8f774
JH
1116You can enable automatic UTF-8-ification of your standard file
1117handles, default C<open()> layer, and C<@ARGV> by using either
1118the C<-C> command line switch or the C<PERL_UNICODE> environment
1119variable, see L<perlrun> for the documentation of the C<-C> switch.
b310b053
JH
1120
1121=item *
1122
376d9008
JB
1123Perl tries really hard to work both with Unicode and the old
1124byte-oriented world. Most often this is nice, but sometimes Perl's
1125straddling of the proverbial fence causes problems.
b310b053
JH
1126
1127=back
1128
1aad1664
JH
1129=head2 When Unicode Does Not Happen
1130
1131While Perl does have extensive ways to input and output in Unicode,
1132and few other 'entry points' like the @ARGV which can be interpreted
1133as Unicode (UTF-8), there still are many places where Unicode (in some
1134encoding or another) could be given as arguments or received as
1135results, or both, but it is not.
1136
6cd4dd6c
JH
1137The following are such interfaces. For all of these interfaces Perl
1138currently (as of 5.8.3) simply assumes byte strings both as arguments
1139and results, or UTF-8 strings if the C<encoding> pragma has been used.
1aad1664
JH
1140
1141One reason why Perl does not attempt to resolve the role of Unicode in
1142this cases is that the answers are highly dependent on the operating
1143system and the file system(s). For example, whether filenames can be
1144in Unicode, and in exactly what kind of encoding, is not exactly a
1145portable concept. Similarly for the qx and system: how well will the
1146'command line interface' (and which of them?) handle Unicode?
1147
1148=over 4
1149
557a2462
RB
1150=item *
1151
254c2b64 1152chdir, chmod, chown, chroot, exec, link, lstat, mkdir,
1e8e8236 1153rename, rmdir, stat, symlink, truncate, unlink, utime, -X
557a2462
RB
1154
1155=item *
1156
1157%ENV
1158
1159=item *
1160
1161glob (aka the <*>)
1162
1163=item *
1aad1664 1164
557a2462 1165open, opendir, sysopen
1aad1664 1166
557a2462 1167=item *
1aad1664 1168
557a2462 1169qx (aka the backtick operator), system
1aad1664 1170
557a2462 1171=item *
1aad1664 1172
557a2462 1173readdir, readlink
1aad1664
JH
1174
1175=back
1176
1177=head2 Forcing Unicode in Perl (Or Unforcing Unicode in Perl)
1178
1179Sometimes (see L</"When Unicode Does Not Happen">) there are
1180situations where you simply need to force Perl to believe that a byte
1181string is UTF-8, or vice versa. The low-level calls
1182utf8::upgrade($bytestring) and utf8::downgrade($utf8string) are
1183the answers.
1184
1185Do not use them without careful thought, though: Perl may easily get
1186very confused, angry, or even crash, if you suddenly change the 'nature'
1187of scalar like that. Especially careful you have to be if you use the
1188utf8::upgrade(): any random byte string is not valid UTF-8.
1189
95a1a48b
JH
1190=head2 Using Unicode in XS
1191
3a2263fe
RGS
1192If you want to handle Perl Unicode in XS extensions, you may find the
1193following C APIs useful. See also L<perlguts/"Unicode Support"> for an
1194explanation about Unicode at the XS level, and L<perlapi> for the API
1195details.
95a1a48b
JH
1196
1197=over 4
1198
1199=item *
1200
1bfb14c4
JH
1201C<DO_UTF8(sv)> returns true if the C<UTF8> flag is on and the bytes
1202pragma is not in effect. C<SvUTF8(sv)> returns true is the C<UTF8>
1203flag is on; the bytes pragma is ignored. The C<UTF8> flag being on
1204does B<not> mean that there are any characters of code points greater
1205than 255 (or 127) in the scalar or that there are even any characters
1206in the scalar. What the C<UTF8> flag means is that the sequence of
1207octets in the representation of the scalar is the sequence of UTF-8
1208encoded code points of the characters of a string. The C<UTF8> flag
1209being off means that each octet in this representation encodes a
1210single character with code point 0..255 within the string. Perl's
1211Unicode model is not to use UTF-8 until it is absolutely necessary.
95a1a48b
JH
1212
1213=item *
1214
fb9cc174 1215C<uvuni_to_utf8(buf, chr)> writes a Unicode character code point into
1bfb14c4 1216a buffer encoding the code point as UTF-8, and returns a pointer
95a1a48b
JH
1217pointing after the UTF-8 bytes.
1218
1219=item *
1220
376d9008
JB
1221C<utf8_to_uvuni(buf, lenp)> reads UTF-8 encoded bytes from a buffer and
1222returns the Unicode character code point and, optionally, the length of
1223the UTF-8 byte sequence.
95a1a48b
JH
1224
1225=item *
1226
376d9008
JB
1227C<utf8_length(start, end)> returns the length of the UTF-8 encoded buffer
1228in characters. C<sv_len_utf8(sv)> returns the length of the UTF-8 encoded
95a1a48b
JH
1229scalar.
1230
1231=item *
1232
376d9008
JB
1233C<sv_utf8_upgrade(sv)> converts the string of the scalar to its UTF-8
1234encoded form. C<sv_utf8_downgrade(sv)> does the opposite, if
1235possible. C<sv_utf8_encode(sv)> is like sv_utf8_upgrade except that
1236it does not set the C<UTF8> flag. C<sv_utf8_decode()> does the
1237opposite of C<sv_utf8_encode()>. Note that none of these are to be
1238used as general-purpose encoding or decoding interfaces: C<use Encode>
1239for that. C<sv_utf8_upgrade()> is affected by the encoding pragma
1240but C<sv_utf8_downgrade()> is not (since the encoding pragma is
1241designed to be a one-way street).
95a1a48b
JH
1242
1243=item *
1244
376d9008 1245C<is_utf8_char(s)> returns true if the pointer points to a valid UTF-8
90f968e0 1246character.
95a1a48b
JH
1247
1248=item *
1249
376d9008 1250C<is_utf8_string(buf, len)> returns true if C<len> bytes of the buffer
95a1a48b
JH
1251are valid UTF-8.
1252
1253=item *
1254
376d9008
JB
1255C<UTF8SKIP(buf)> will return the number of bytes in the UTF-8 encoded
1256character in the buffer. C<UNISKIP(chr)> will return the number of bytes
1257required to UTF-8-encode the Unicode character code point. C<UTF8SKIP()>
90f968e0 1258is useful for example for iterating over the characters of a UTF-8
376d9008 1259encoded buffer; C<UNISKIP()> is useful, for example, in computing
90f968e0 1260the size required for a UTF-8 encoded buffer.
95a1a48b
JH
1261
1262=item *
1263
376d9008 1264C<utf8_distance(a, b)> will tell the distance in characters between the
95a1a48b
JH
1265two pointers pointing to the same UTF-8 encoded buffer.
1266
1267=item *
1268
376d9008
JB
1269C<utf8_hop(s, off)> will return a pointer to an UTF-8 encoded buffer
1270that is C<off> (positive or negative) Unicode characters displaced
1271from the UTF-8 buffer C<s>. Be careful not to overstep the buffer:
1272C<utf8_hop()> will merrily run off the end or the beginning of the
1273buffer if told to do so.
95a1a48b 1274
d2cc3551
JH
1275=item *
1276
376d9008
JB
1277C<pv_uni_display(dsv, spv, len, pvlim, flags)> and
1278C<sv_uni_display(dsv, ssv, pvlim, flags)> are useful for debugging the
1279output of Unicode strings and scalars. By default they are useful
1280only for debugging--they display B<all> characters as hexadecimal code
1bfb14c4
JH
1281points--but with the flags C<UNI_DISPLAY_ISPRINT>,
1282C<UNI_DISPLAY_BACKSLASH>, and C<UNI_DISPLAY_QQ> you can make the
1283output more readable.
d2cc3551
JH
1284
1285=item *
1286
376d9008
JB
1287C<ibcmp_utf8(s1, pe1, u1, l1, u1, s2, pe2, l2, u2)> can be used to
1288compare two strings case-insensitively in Unicode. For case-sensitive
1289comparisons you can just use C<memEQ()> and C<memNE()> as usual.
d2cc3551 1290
c349b1b9
JH
1291=back
1292
95a1a48b
JH
1293For more information, see L<perlapi>, and F<utf8.c> and F<utf8.h>
1294in the Perl source code distribution.
1295
c29a771d
JH
1296=head1 BUGS
1297
376d9008 1298=head2 Interaction with Locales
7eabb34d 1299
376d9008
JB
1300Use of locales with Unicode data may lead to odd results. Currently,
1301Perl attempts to attach 8-bit locale info to characters in the range
13020..255, but this technique is demonstrably incorrect for locales that
1303use characters above that range when mapped into Unicode. Perl's
1304Unicode support will also tend to run slower. Use of locales with
1305Unicode is discouraged.
c29a771d 1306
376d9008 1307=head2 Interaction with Extensions
7eabb34d 1308
376d9008 1309When Perl exchanges data with an extension, the extension should be
7eabb34d 1310able to understand the UTF-8 flag and act accordingly. If the
376d9008
JB
1311extension doesn't know about the flag, it's likely that the extension
1312will return incorrectly-flagged data.
7eabb34d
A
1313
1314So if you're working with Unicode data, consult the documentation of
1315every module you're using if there are any issues with Unicode data
1316exchange. If the documentation does not talk about Unicode at all,
a73d23f6 1317suspect the worst and probably look at the source to learn how the
376d9008 1318module is implemented. Modules written completely in Perl shouldn't
a73d23f6
RGS
1319cause problems. Modules that directly or indirectly access code written
1320in other programming languages are at risk.
7eabb34d 1321
376d9008 1322For affected functions, the simple strategy to avoid data corruption is
7eabb34d 1323to always make the encoding of the exchanged data explicit. Choose an
376d9008 1324encoding that you know the extension can handle. Convert arguments passed
7eabb34d
A
1325to the extensions to that encoding and convert results back from that
1326encoding. Write wrapper functions that do the conversions for you, so
1327you can later change the functions when the extension catches up.
1328
376d9008 1329To provide an example, let's say the popular Foo::Bar::escape_html
7eabb34d
A
1330function doesn't deal with Unicode data yet. The wrapper function
1331would convert the argument to raw UTF-8 and convert the result back to
376d9008 1332Perl's internal representation like so:
7eabb34d
A
1333
1334 sub my_escape_html ($) {
1335 my($what) = shift;
1336 return unless defined $what;
1337 Encode::decode_utf8(Foo::Bar::escape_html(Encode::encode_utf8($what)));
1338 }
1339
1340Sometimes, when the extension does not convert data but just stores
1341and retrieves them, you will be in a position to use the otherwise
1342dangerous Encode::_utf8_on() function. Let's say the popular
66b79f27 1343C<Foo::Bar> extension, written in C, provides a C<param> method that
7eabb34d
A
1344lets you store and retrieve data according to these prototypes:
1345
1346 $self->param($name, $value); # set a scalar
1347 $value = $self->param($name); # retrieve a scalar
1348
1349If it does not yet provide support for any encoding, one could write a
1350derived class with such a C<param> method:
1351
1352 sub param {
1353 my($self,$name,$value) = @_;
1354 utf8::upgrade($name); # make sure it is UTF-8 encoded
1355 if (defined $value)
1356 utf8::upgrade($value); # make sure it is UTF-8 encoded
1357 return $self->SUPER::param($name,$value);
1358 } else {
1359 my $ret = $self->SUPER::param($name);
1360 Encode::_utf8_on($ret); # we know, it is UTF-8 encoded
1361 return $ret;
1362 }
1363 }
1364
a73d23f6
RGS
1365Some extensions provide filters on data entry/exit points, such as
1366DB_File::filter_store_key and family. Look out for such filters in
66b79f27 1367the documentation of your extensions, they can make the transition to
7eabb34d
A
1368Unicode data much easier.
1369
376d9008 1370=head2 Speed
7eabb34d 1371
c29a771d 1372Some functions are slower when working on UTF-8 encoded strings than
574c8022 1373on byte encoded strings. All functions that need to hop over
7c17141f
JH
1374characters such as length(), substr() or index(), or matching regular
1375expressions can work B<much> faster when the underlying data are
1376byte-encoded.
1377
1378In Perl 5.8.0 the slowness was often quite spectacular; in Perl 5.8.1
1379a caching scheme was introduced which will hopefully make the slowness
a104b433
JH
1380somewhat less spectacular, at least for some operations. In general,
1381operations with UTF-8 encoded strings are still slower. As an example,
1382the Unicode properties (character classes) like C<\p{Nd}> are known to
1383be quite a bit slower (5-20 times) than their simpler counterparts
1384like C<\d> (then again, there 268 Unicode characters matching C<Nd>
1385compared with the 10 ASCII characters matching C<d>).
666f95b9 1386
c8d992ba
A
1387=head2 Porting code from perl-5.6.X
1388
1389Perl 5.8 has a different Unicode model from 5.6. In 5.6 the programmer
1390was required to use the C<utf8> pragma to declare that a given scope
1391expected to deal with Unicode data and had to make sure that only
1392Unicode data were reaching that scope. If you have code that is
1393working with 5.6, you will need some of the following adjustments to
1394your code. The examples are written such that the code will continue
1395to work under 5.6, so you should be safe to try them out.
1396
1397=over 4
1398
1399=item *
1400
1401A filehandle that should read or write UTF-8
1402
1403 if ($] > 5.007) {
1404 binmode $fh, ":utf8";
1405 }
1406
1407=item *
1408
1409A scalar that is going to be passed to some extension
1410
1411Be it Compress::Zlib, Apache::Request or any extension that has no
1412mention of Unicode in the manpage, you need to make sure that the
1413UTF-8 flag is stripped off. Note that at the time of this writing
1414(October 2002) the mentioned modules are not UTF-8-aware. Please
1415check the documentation to verify if this is still true.
1416
1417 if ($] > 5.007) {
1418 require Encode;
1419 $val = Encode::encode_utf8($val); # make octets
1420 }
1421
1422=item *
1423
1424A scalar we got back from an extension
1425
1426If you believe the scalar comes back as UTF-8, you will most likely
1427want the UTF-8 flag restored:
1428
1429 if ($] > 5.007) {
1430 require Encode;
1431 $val = Encode::decode_utf8($val);
1432 }
1433
1434=item *
1435
1436Same thing, if you are really sure it is UTF-8
1437
1438 if ($] > 5.007) {
1439 require Encode;
1440 Encode::_utf8_on($val);
1441 }
1442
1443=item *
1444
1445A wrapper for fetchrow_array and fetchrow_hashref
1446
1447When the database contains only UTF-8, a wrapper function or method is
1448a convenient way to replace all your fetchrow_array and
1449fetchrow_hashref calls. A wrapper function will also make it easier to
1450adapt to future enhancements in your database driver. Note that at the
1451time of this writing (October 2002), the DBI has no standardized way
1452to deal with UTF-8 data. Please check the documentation to verify if
1453that is still true.
1454
1455 sub fetchrow {
1456 my($self, $sth, $what) = @_; # $what is one of fetchrow_{array,hashref}
1457 if ($] < 5.007) {
1458 return $sth->$what;
1459 } else {
1460 require Encode;
1461 if (wantarray) {
1462 my @arr = $sth->$what;
1463 for (@arr) {
1464 defined && /[^\000-\177]/ && Encode::_utf8_on($_);
1465 }
1466 return @arr;
1467 } else {
1468 my $ret = $sth->$what;
1469 if (ref $ret) {
1470 for my $k (keys %$ret) {
1471 defined && /[^\000-\177]/ && Encode::_utf8_on($_) for $ret->{$k};
1472 }
1473 return $ret;
1474 } else {
1475 defined && /[^\000-\177]/ && Encode::_utf8_on($_) for $ret;
1476 return $ret;
1477 }
1478 }
1479 }
1480 }
1481
1482
1483=item *
1484
1485A large scalar that you know can only contain ASCII
1486
1487Scalars that contain only ASCII and are marked as UTF-8 are sometimes
1488a drag to your program. If you recognize such a situation, just remove
1489the UTF-8 flag:
1490
1491 utf8::downgrade($val) if $] > 5.007;
1492
1493=back
1494
393fec97
GS
1495=head1 SEE ALSO
1496
72ff2908 1497L<perluniintro>, L<encoding>, L<Encode>, L<open>, L<utf8>, L<bytes>,
a05d7ebb 1498L<perlretut>, L<perlvar/"${^UNICODE}">
393fec97
GS
1499
1500=cut