This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
[perl #8063] perlfilter doesn't mention potential DATA clobber
[perl5.git] / pod / perlunicode.pod
CommitLineData
393fec97
GS
1=head1 NAME
2
3perlunicode - Unicode support in Perl
4
5=head1 DESCRIPTION
6
0a1f2d14 7=head2 Important Caveats
21bad921 8
376d9008 9Unicode support is an extensive requirement. While Perl does not
c349b1b9
JH
10implement the Unicode standard or the accompanying technical reports
11from cover to cover, Perl does support many Unicode features.
21bad921 12
13a2d996 13=over 4
21bad921 14
fae2c0fb 15=item Input and Output Layers
21bad921 16
376d9008 17Perl knows when a filehandle uses Perl's internal Unicode encodings
1bfb14c4
JH
18(UTF-8, or UTF-EBCDIC if in EBCDIC) if the filehandle is opened with
19the ":utf8" layer. Other encodings can be converted to Perl's
20encoding on input or from Perl's encoding on output by use of the
21":encoding(...)" layer. See L<open>.
c349b1b9 22
376d9008 23To indicate that Perl source itself is using a particular encoding,
c349b1b9 24see L<encoding>.
21bad921
GS
25
26=item Regular Expressions
27
c349b1b9 28The regular expression compiler produces polymorphic opcodes. That is,
376d9008
JB
29the pattern adapts to the data and automatically switches to the Unicode
30character scheme when presented with Unicode data--or instead uses
31a traditional byte scheme when presented with byte data.
21bad921 32
ad0029c4 33=item C<use utf8> still needed to enable UTF-8/UTF-EBCDIC in scripts
21bad921 34
376d9008
JB
35As a compatibility measure, the C<use utf8> pragma must be explicitly
36included to enable recognition of UTF-8 in the Perl scripts themselves
1bfb14c4
JH
37(in string or regular expression literals, or in identifier names) on
38ASCII-based machines or to recognize UTF-EBCDIC on EBCDIC-based
376d9008 39machines. B<These are the only times when an explicit C<use utf8>
8f8cf39c 40is needed.> See L<utf8>.
21bad921 41
1768d7eb 42You can also use the C<encoding> pragma to change the default encoding
6ec9efec 43of the data in your script; see L<encoding>.
1768d7eb 44
21bad921
GS
45=back
46
376d9008 47=head2 Byte and Character Semantics
393fec97 48
376d9008 49Beginning with version 5.6, Perl uses logically-wide characters to
3e4dbfed 50represent strings internally.
393fec97 51
376d9008
JB
52In future, Perl-level operations will be expected to work with
53characters rather than bytes.
393fec97 54
376d9008 55However, as an interim compatibility measure, Perl aims to
75daf61c
JH
56provide a safe migration path from byte semantics to character
57semantics for programs. For operations where Perl can unambiguously
376d9008 58decide that the input data are characters, Perl switches to
75daf61c
JH
59character semantics. For operations where this determination cannot
60be made without additional information from the user, Perl decides in
376d9008 61favor of compatibility and chooses to use byte semantics.
8cbd9a7a
GS
62
63This behavior preserves compatibility with earlier versions of Perl,
376d9008
JB
64which allowed byte semantics in Perl operations only if
65none of the program's inputs were marked as being as source of Unicode
8cbd9a7a
GS
66character data. Such data may come from filehandles, from calls to
67external programs, from information provided by the system (such as %ENV),
21bad921 68or from literals and constants in the source text.
8cbd9a7a 69
376d9008
JB
70The C<bytes> pragma will always, regardless of platform, force byte
71semantics in a particular lexical scope. See L<bytes>.
8cbd9a7a
GS
72
73The C<utf8> pragma is primarily a compatibility device that enables
75daf61c 74recognition of UTF-(8|EBCDIC) in literals encountered by the parser.
376d9008
JB
75Note that this pragma is only required while Perl defaults to byte
76semantics; when character semantics become the default, this pragma
77may become a no-op. See L<utf8>.
78
79Unless explicitly stated, Perl operators use character semantics
80for Unicode data and byte semantics for non-Unicode data.
81The decision to use character semantics is made transparently. If
82input data comes from a Unicode source--for example, if a character
fae2c0fb 83encoding layer is added to a filehandle or a literal Unicode
376d9008
JB
84string constant appears in a program--character semantics apply.
85Otherwise, byte semantics are in effect. The C<bytes> pragma should
86be used to force byte semantics on Unicode data.
87
88If strings operating under byte semantics and strings with Unicode
89character data are concatenated, the new string will be upgraded to
90I<ISO 8859-1 (Latin-1)>, even if the old Unicode string used EBCDIC.
91This translation is done without regard to the system's native 8-bit
92encoding, so to change this for systems with non-Latin-1 and
93non-EBCDIC native encodings use the C<encoding> pragma. See
94L<encoding>.
7dedd01f 95
feda178f 96Under character semantics, many operations that formerly operated on
376d9008 97bytes now operate on characters. A character in Perl is
feda178f 98logically just a number ranging from 0 to 2**31 or so. Larger
376d9008
JB
99characters may encode into longer sequences of bytes internally, but
100this internal detail is mostly hidden for Perl code.
101See L<perluniintro> for more.
393fec97 102
376d9008 103=head2 Effects of Character Semantics
393fec97
GS
104
105Character semantics have the following effects:
106
107=over 4
108
109=item *
110
376d9008 111Strings--including hash keys--and regular expression patterns may
574c8022 112contain characters that have an ordinal value larger than 255.
393fec97 113
feda178f
JH
114If you use a Unicode editor to edit your program, Unicode characters
115may occur directly within the literal strings in one of the various
376d9008
JB
116Unicode encodings (UTF-8, UTF-EBCDIC, UCS-2, etc.), but will be recognized
117as such and converted to Perl's internal representation only if the
feda178f 118appropriate L<encoding> is specified.
3e4dbfed 119
1bfb14c4
JH
120Unicode characters can also be added to a string by using the
121C<\x{...}> notation. The Unicode code for the desired character, in
376d9008
JB
122hexadecimal, should be placed in the braces. For instance, a smiley
123face is C<\x{263A}>. This encoding scheme only works for characters
124with a code of 0x100 or above.
3e4dbfed
JF
125
126Additionally, if you
574c8022 127
3e4dbfed 128 use charnames ':full';
574c8022 129
1bfb14c4
JH
130you can use the C<\N{...}> notation and put the official Unicode
131character name within the braces, such as C<\N{WHITE SMILING FACE}>.
376d9008 132
393fec97
GS
133
134=item *
135
574c8022
JH
136If an appropriate L<encoding> is specified, identifiers within the
137Perl script may contain Unicode alphanumeric characters, including
376d9008
JB
138ideographs. Perl does not currently attempt to canonicalize variable
139names.
393fec97 140
393fec97
GS
141=item *
142
1bfb14c4
JH
143Regular expressions match characters instead of bytes. "." matches
144a character instead of a byte. The C<\C> pattern is provided to force
145a match a single byte--a C<char> in C, hence C<\C>.
393fec97 146
393fec97
GS
147=item *
148
149Character classes in regular expressions match characters instead of
376d9008 150bytes and match against the character properties specified in the
1bfb14c4 151Unicode properties database. C<\w> can be used to match a Japanese
75daf61c 152ideograph, for instance.
393fec97 153
393fec97
GS
154=item *
155
eb0cc9e3 156Named Unicode properties, scripts, and block ranges may be used like
376d9008
JB
157character classes via the C<\p{}> "matches property" construct and
158the C<\P{}> negation, "doesn't match property".
1bfb14c4
JH
159
160For instance, C<\p{Lu}> matches any character with the Unicode "Lu"
161(Letter, uppercase) property, while C<\p{M}> matches any character
162with an "M" (mark--accents and such) property. Brackets are not
163required for single letter properties, so C<\p{M}> is equivalent to
164C<\pM>. Many predefined properties are available, such as
165C<\p{Mirrored}> and C<\p{Tibetan}>.
4193bef7 166
cfc01aea 167The official Unicode script and block names have spaces and dashes as
376d9008 168separators, but for convenience you can use dashes, spaces, or
1bfb14c4
JH
169underbars, and case is unimportant. It is recommended, however, that
170for consistency you use the following naming: the official Unicode
171script, property, or block name (see below for the additional rules
172that apply to block names) with whitespace and dashes removed, and the
173words "uppercase-first-lowercase-rest". C<Latin-1 Supplement> thus
174becomes C<Latin1Supplement>.
4193bef7 175
376d9008
JB
176You can also use negation in both C<\p{}> and C<\P{}> by introducing a caret
177(^) between the first brace and the property name: C<\p{^Tamil}> is
eb0cc9e3 178equal to C<\P{Tamil}>.
4193bef7 179
eb0cc9e3 180Here are the basic Unicode General Category properties, followed by their
68cd2d32 181long form. You can use either; C<\p{Lu}> and C<\p{UppercaseLetter}>,
376d9008 182for instance, are identical.
393fec97 183
d73e5302
JH
184 Short Long
185
186 L Letter
eb0cc9e3
JH
187 Lu UppercaseLetter
188 Ll LowercaseLetter
189 Lt TitlecaseLetter
190 Lm ModifierLetter
191 Lo OtherLetter
d73e5302
JH
192
193 M Mark
eb0cc9e3
JH
194 Mn NonspacingMark
195 Mc SpacingMark
196 Me EnclosingMark
d73e5302
JH
197
198 N Number
eb0cc9e3
JH
199 Nd DecimalNumber
200 Nl LetterNumber
201 No OtherNumber
d73e5302
JH
202
203 P Punctuation
eb0cc9e3
JH
204 Pc ConnectorPunctuation
205 Pd DashPunctuation
206 Ps OpenPunctuation
207 Pe ClosePunctuation
208 Pi InitialPunctuation
d73e5302 209 (may behave like Ps or Pe depending on usage)
eb0cc9e3 210 Pf FinalPunctuation
d73e5302 211 (may behave like Ps or Pe depending on usage)
eb0cc9e3 212 Po OtherPunctuation
d73e5302
JH
213
214 S Symbol
eb0cc9e3
JH
215 Sm MathSymbol
216 Sc CurrencySymbol
217 Sk ModifierSymbol
218 So OtherSymbol
d73e5302
JH
219
220 Z Separator
eb0cc9e3
JH
221 Zs SpaceSeparator
222 Zl LineSeparator
223 Zp ParagraphSeparator
d73e5302
JH
224
225 C Other
e150c829
JH
226 Cc Control
227 Cf Format
eb0cc9e3
JH
228 Cs Surrogate (not usable)
229 Co PrivateUse
e150c829 230 Cn Unassigned
1ac13f9a 231
376d9008 232Single-letter properties match all characters in any of the
3e4dbfed 233two-letter sub-properties starting with the same letter.
376d9008 234C<L&> is a special case, which is an alias for C<Ll>, C<Lu>, and C<Lt>.
32293815 235
eb0cc9e3 236Because Perl hides the need for the user to understand the internal
1bfb14c4
JH
237representation of Unicode characters, there is no need to implement
238the somewhat messy concept of surrogates. C<Cs> is therefore not
eb0cc9e3 239supported.
d73e5302 240
376d9008
JB
241Because scripts differ in their directionality--Hebrew is
242written right to left, for example--Unicode supplies these properties:
32293815 243
eb0cc9e3 244 Property Meaning
92e830a9 245
d73e5302
JH
246 BidiL Left-to-Right
247 BidiLRE Left-to-Right Embedding
248 BidiLRO Left-to-Right Override
249 BidiR Right-to-Left
250 BidiAL Right-to-Left Arabic
251 BidiRLE Right-to-Left Embedding
252 BidiRLO Right-to-Left Override
253 BidiPDF Pop Directional Format
254 BidiEN European Number
255 BidiES European Number Separator
256 BidiET European Number Terminator
257 BidiAN Arabic Number
258 BidiCS Common Number Separator
259 BidiNSM Non-Spacing Mark
260 BidiBN Boundary Neutral
261 BidiB Paragraph Separator
262 BidiS Segment Separator
263 BidiWS Whitespace
264 BidiON Other Neutrals
32293815 265
376d9008 266For example, C<\p{BidiR}> matches characters that are normally
eb0cc9e3
JH
267written right to left.
268
210b36aa
AMS
269=back
270
2796c109
JH
271=head2 Scripts
272
376d9008
JB
273The script names which can be used by C<\p{...}> and C<\P{...}>,
274such as in C<\p{Latin}> or C<\p{Cyrillic}>, are as follows:
2796c109 275
1ac13f9a 276 Arabic
e9ad1727 277 Armenian
1ac13f9a 278 Bengali
e9ad1727 279 Bopomofo
1d81abf3 280 Buhid
eb0cc9e3 281 CanadianAboriginal
e9ad1727
JH
282 Cherokee
283 Cyrillic
284 Deseret
285 Devanagari
286 Ethiopic
287 Georgian
288 Gothic
289 Greek
1ac13f9a 290 Gujarati
e9ad1727
JH
291 Gurmukhi
292 Han
293 Hangul
1d81abf3 294 Hanunoo
e9ad1727
JH
295 Hebrew
296 Hiragana
297 Inherited
1ac13f9a 298 Kannada
e9ad1727
JH
299 Katakana
300 Khmer
1ac13f9a 301 Lao
e9ad1727
JH
302 Latin
303 Malayalam
304 Mongolian
1ac13f9a 305 Myanmar
1ac13f9a 306 Ogham
eb0cc9e3 307 OldItalic
e9ad1727 308 Oriya
1ac13f9a 309 Runic
e9ad1727
JH
310 Sinhala
311 Syriac
1d81abf3
JH
312 Tagalog
313 Tagbanwa
e9ad1727
JH
314 Tamil
315 Telugu
316 Thaana
317 Thai
318 Tibetan
1ac13f9a 319 Yi
1ac13f9a 320
376d9008 321Extended property classes can supplement the basic
1ac13f9a
JH
322properties, defined by the F<PropList> Unicode database:
323
1d81abf3 324 ASCIIHexDigit
eb0cc9e3 325 BidiControl
1ac13f9a 326 Dash
1d81abf3 327 Deprecated
1ac13f9a
JH
328 Diacritic
329 Extender
1d81abf3 330 GraphemeLink
eb0cc9e3 331 HexDigit
e9ad1727
JH
332 Hyphen
333 Ideographic
1d81abf3
JH
334 IDSBinaryOperator
335 IDSTrinaryOperator
eb0cc9e3 336 JoinControl
1d81abf3 337 LogicalOrderException
eb0cc9e3
JH
338 NoncharacterCodePoint
339 OtherAlphabetic
1d81abf3
JH
340 OtherDefaultIgnorableCodePoint
341 OtherGraphemeExtend
eb0cc9e3
JH
342 OtherLowercase
343 OtherMath
344 OtherUppercase
345 QuotationMark
1d81abf3
JH
346 Radical
347 SoftDotted
348 TerminalPunctuation
349 UnifiedIdeograph
eb0cc9e3 350 WhiteSpace
1ac13f9a 351
376d9008 352and there are further derived properties:
1ac13f9a 353
eb0cc9e3
JH
354 Alphabetic Lu + Ll + Lt + Lm + Lo + OtherAlphabetic
355 Lowercase Ll + OtherLowercase
356 Uppercase Lu + OtherUppercase
357 Math Sm + OtherMath
1ac13f9a
JH
358
359 ID_Start Lu + Ll + Lt + Lm + Lo + Nl
360 ID_Continue ID_Start + Mn + Mc + Nd + Pc
361
362 Any Any character
66b79f27
RGS
363 Assigned Any non-Cn character (i.e. synonym for \P{Cn})
364 Unassigned Synonym for \p{Cn}
1ac13f9a 365 Common Any character (or unassigned code point)
e150c829 366 not explicitly assigned to a script
2796c109 367
1bfb14c4
JH
368For backward compatibility (with Perl 5.6), all properties mentioned
369so far may have C<Is> prepended to their name, so C<\P{IsLu}>, for
370example, is equal to C<\P{Lu}>.
eb0cc9e3 371
2796c109
JH
372=head2 Blocks
373
1bfb14c4
JH
374In addition to B<scripts>, Unicode also defines B<blocks> of
375characters. The difference between scripts and blocks is that the
376concept of scripts is closer to natural languages, while the concept
377of blocks is more of an artificial grouping based on groups of 256
376d9008 378Unicode characters. For example, the C<Latin> script contains letters
1bfb14c4 379from many blocks but does not contain all the characters from those
376d9008
JB
380blocks. It does not, for example, contain digits, because digits are
381shared across many scripts. Digits and similar groups, like
382punctuation, are in a category called C<Common>.
2796c109 383
cfc01aea
JF
384For more about scripts, see the UTR #24:
385
386 http://www.unicode.org/unicode/reports/tr24/
387
388For more about blocks, see:
389
390 http://www.unicode.org/Public/UNIDATA/Blocks.txt
2796c109 391
376d9008
JB
392Block names are given with the C<In> prefix. For example, the
393Katakana block is referenced via C<\p{InKatakana}>. The C<In>
7eabb34d 394prefix may be omitted if there is no naming conflict with a script
eb0cc9e3 395or any other property, but it is recommended that C<In> always be used
1bfb14c4 396for block tests to avoid confusion.
eb0cc9e3
JH
397
398These block names are supported:
399
1d81abf3
JH
400 InAlphabeticPresentationForms
401 InArabic
402 InArabicPresentationFormsA
403 InArabicPresentationFormsB
404 InArmenian
405 InArrows
406 InBasicLatin
407 InBengali
408 InBlockElements
409 InBopomofo
410 InBopomofoExtended
411 InBoxDrawing
412 InBraillePatterns
413 InBuhid
414 InByzantineMusicalSymbols
415 InCJKCompatibility
416 InCJKCompatibilityForms
417 InCJKCompatibilityIdeographs
418 InCJKCompatibilityIdeographsSupplement
419 InCJKRadicalsSupplement
420 InCJKSymbolsAndPunctuation
421 InCJKUnifiedIdeographs
422 InCJKUnifiedIdeographsExtensionA
423 InCJKUnifiedIdeographsExtensionB
424 InCherokee
425 InCombiningDiacriticalMarks
426 InCombiningDiacriticalMarksforSymbols
427 InCombiningHalfMarks
428 InControlPictures
429 InCurrencySymbols
430 InCyrillic
431 InCyrillicSupplementary
432 InDeseret
433 InDevanagari
434 InDingbats
435 InEnclosedAlphanumerics
436 InEnclosedCJKLettersAndMonths
437 InEthiopic
438 InGeneralPunctuation
439 InGeometricShapes
440 InGeorgian
441 InGothic
442 InGreekExtended
443 InGreekAndCoptic
444 InGujarati
445 InGurmukhi
446 InHalfwidthAndFullwidthForms
447 InHangulCompatibilityJamo
448 InHangulJamo
449 InHangulSyllables
450 InHanunoo
451 InHebrew
452 InHighPrivateUseSurrogates
453 InHighSurrogates
454 InHiragana
455 InIPAExtensions
456 InIdeographicDescriptionCharacters
457 InKanbun
458 InKangxiRadicals
459 InKannada
460 InKatakana
461 InKatakanaPhoneticExtensions
462 InKhmer
463 InLao
464 InLatin1Supplement
465 InLatinExtendedA
466 InLatinExtendedAdditional
467 InLatinExtendedB
468 InLetterlikeSymbols
469 InLowSurrogates
470 InMalayalam
471 InMathematicalAlphanumericSymbols
472 InMathematicalOperators
473 InMiscellaneousMathematicalSymbolsA
474 InMiscellaneousMathematicalSymbolsB
475 InMiscellaneousSymbols
476 InMiscellaneousTechnical
477 InMongolian
478 InMusicalSymbols
479 InMyanmar
480 InNumberForms
481 InOgham
482 InOldItalic
483 InOpticalCharacterRecognition
484 InOriya
485 InPrivateUseArea
486 InRunic
487 InSinhala
488 InSmallFormVariants
489 InSpacingModifierLetters
490 InSpecials
491 InSuperscriptsAndSubscripts
492 InSupplementalArrowsA
493 InSupplementalArrowsB
494 InSupplementalMathematicalOperators
495 InSupplementaryPrivateUseAreaA
496 InSupplementaryPrivateUseAreaB
497 InSyriac
498 InTagalog
499 InTagbanwa
500 InTags
501 InTamil
502 InTelugu
503 InThaana
504 InThai
505 InTibetan
506 InUnifiedCanadianAboriginalSyllabics
507 InVariationSelectors
508 InYiRadicals
509 InYiSyllables
32293815 510
210b36aa
AMS
511=over 4
512
393fec97
GS
513=item *
514
376d9008
JB
515The special pattern C<\X> matches any extended Unicode
516sequence--"a combining character sequence" in Standardese--where the
517first character is a base character and subsequent characters are mark
518characters that apply to the base character. C<\X> is equivalent to
393fec97
GS
519C<(?:\PM\pM*)>.
520
393fec97
GS
521=item *
522
383e7cdd 523The C<tr///> operator translates characters instead of bytes. Note
376d9008
JB
524that the C<tr///CU> functionality has been removed. For similar
525functionality see pack('U0', ...) and pack('C0', ...).
393fec97 526
393fec97
GS
527=item *
528
529Case translation operators use the Unicode case translation tables
376d9008
JB
530when character input is provided. Note that C<uc()>, or C<\U> in
531interpolated strings, translates to uppercase, while C<ucfirst>,
532or C<\u> in interpolated strings, translates to titlecase in languages
533that make the distinction.
393fec97
GS
534
535=item *
536
376d9008 537Most operators that deal with positions or lengths in a string will
75daf61c
JH
538automatically switch to using character positions, including
539C<chop()>, C<substr()>, C<pos()>, C<index()>, C<rindex()>,
540C<sprintf()>, C<write()>, and C<length()>. Operators that
376d9008
JB
541specifically do not switch include C<vec()>, C<pack()>, and
542C<unpack()>. Operators that really don't care include C<chomp()>,
543operators that treats strings as a bucket of bits such as C<sort()>,
544and operators dealing with filenames.
393fec97
GS
545
546=item *
547
1bfb14c4 548The C<pack()>/C<unpack()> letters C<c> and C<C> do I<not> change,
376d9008 549since they are often used for byte-oriented formats. Again, think
1bfb14c4
JH
550C<char> in the C language.
551
552There is a new C<U> specifier that converts between Unicode characters
553and code points.
393fec97
GS
554
555=item *
556
376d9008
JB
557The C<chr()> and C<ord()> functions work on characters, similar to
558C<pack("U")> and C<unpack("U")>, I<not> C<pack("C")> and
559C<unpack("C")>. C<pack("C")> and C<unpack("C")> are methods for
560emulating byte-oriented C<chr()> and C<ord()> on Unicode strings.
561While these methods reveal the internal encoding of Unicode strings,
562that is not something one normally needs to care about at all.
393fec97
GS
563
564=item *
565
376d9008
JB
566The bit string operators, C<& | ^ ~>, can operate on character data.
567However, for backward compatibility, such as when using bit string
568operations when characters are all less than 256 in ordinal value, one
569should not use C<~> (the bit complement) with characters of both
570values less than 256 and values greater than 256. Most importantly,
571DeMorgan's laws (C<~($x|$y) eq ~$x&~$y> and C<~($x&$y) eq ~$x|~$y>)
572will not hold. The reason for this mathematical I<faux pas> is that
573the complement cannot return B<both> the 8-bit (byte-wide) bit
574complement B<and> the full character-wide bit complement.
a1ca4561
YST
575
576=item *
577
983ffd37
JH
578lc(), uc(), lcfirst(), and ucfirst() work for the following cases:
579
580=over 8
581
582=item *
583
584the case mapping is from a single Unicode character to another
376d9008 585single Unicode character, or
983ffd37
JH
586
587=item *
588
589the case mapping is from a single Unicode character to more
376d9008 590than one Unicode character.
983ffd37
JH
591
592=back
593
63de3cb2
JH
594Things to do with locales (Lithuanian, Turkish, Azeri) do B<not> work
595since Perl does not understand the concept of Unicode locales.
983ffd37 596
dc33ebcf
RGS
597See the Unicode Technical Report #21, Case Mappings, for more details.
598
983ffd37
JH
599=back
600
dc33ebcf 601=over 4
ac1256e8
JH
602
603=item *
604
393fec97
GS
605And finally, C<scalar reverse()> reverses by character rather than by byte.
606
607=back
608
376d9008 609=head2 User-Defined Character Properties
491fd90a
JH
610
611You can define your own character properties by defining subroutines
3a2263fe
RGS
612whose names begin with "In" or "Is". The subroutines must be defined
613in the C<main> package. The user-defined properties can be used in the
614regular expression C<\p> and C<\P> constructs. Note that the effect
615is compile-time and immutable once defined.
491fd90a 616
376d9008
JB
617The subroutines must return a specially-formatted string, with one
618or more newline-separated lines. Each line must be one of the following:
491fd90a
JH
619
620=over 4
621
622=item *
623
99a6b1f0 624Two hexadecimal numbers separated by horizontal whitespace (space or
376d9008 625tabular characters) denoting a range of Unicode code points to include.
491fd90a
JH
626
627=item *
628
376d9008
JB
629Something to include, prefixed by "+": a built-in character
630property (prefixed by "utf8::"), to represent all the characters in that
631property; two hexadecimal code points for a range; or a single
632hexadecimal code point.
491fd90a
JH
633
634=item *
635
376d9008 636Something to exclude, prefixed by "-": an existing character
11ef8fdd 637property (prefixed by "utf8::"), for all the characters in that
376d9008
JB
638property; two hexadecimal code points for a range; or a single
639hexadecimal code point.
491fd90a
JH
640
641=item *
642
376d9008 643Something to negate, prefixed "!": an existing character
11ef8fdd 644property (prefixed by "utf8::") for all the characters except the
376d9008
JB
645characters in the property; two hexadecimal code points for a range;
646or a single hexadecimal code point.
491fd90a
JH
647
648=back
649
650For example, to define a property that covers both the Japanese
651syllabaries (hiragana and katakana), you can define
652
653 sub InKana {
d5822f25
A
654 return <<END;
655 3040\t309F
656 30A0\t30FF
491fd90a
JH
657 END
658 }
659
d5822f25
A
660Imagine that the here-doc end marker is at the beginning of the line.
661Now you can use C<\p{InKana}> and C<\P{InKana}>.
491fd90a
JH
662
663You could also have used the existing block property names:
664
665 sub InKana {
666 return <<'END';
667 +utf8::InHiragana
668 +utf8::InKatakana
669 END
670 }
671
672Suppose you wanted to match only the allocated characters,
d5822f25 673not the raw block ranges: in other words, you want to remove
491fd90a
JH
674the non-characters:
675
676 sub InKana {
677 return <<'END';
678 +utf8::InHiragana
679 +utf8::InKatakana
680 -utf8::IsCn
681 END
682 }
683
684The negation is useful for defining (surprise!) negated classes.
685
686 sub InNotKana {
687 return <<'END';
688 !utf8::InHiragana
689 -utf8::InKatakana
690 +utf8::IsCn
691 END
692 }
693
3a2263fe
RGS
694You can also define your own mappings to be used in the lc(),
695lcfirst(), uc(), and ucfirst() (or their string-inlined versions).
696The principle is the same: define subroutines in the C<main> package
697with names like C<ToLower> (for lc() and lcfirst()), C<ToTitle> (for
698the first character in ucfirst()), and C<ToUpper> (for uc(), and the
699rest of the characters in ucfirst()).
700
701The string returned by the subroutines needs now to be three
702hexadecimal numbers separated by tabulators: start of the source
703range, end of the source range, and start of the destination range.
704For example:
705
706 sub ToUpper {
707 return <<END;
708 0061\t0063\t0041
709 END
710 }
711
712defines an uc() mapping that causes only the characters "a", "b", and
713"c" to be mapped to "A", "B", "C", all other characters will remain
714unchanged.
715
716If there is no source range to speak of, that is, the mapping is from
717a single character to another single character, leave the end of the
718source range empty, but the two tabulator characters are still needed.
719For example:
720
721 sub ToLower {
722 return <<END;
723 0041\t\t0061
724 END
725 }
726
727defines a lc() mapping that causes only "A" to be mapped to "a", all
728other characters will remain unchanged.
729
730(For serious hackers only) If you want to introspect the default
731mappings, you can find the data in the directory
732C<$Config{privlib}>/F<unicore/To/>. The mapping data is returned as
733the here-document, and the C<utf8::ToSpecFoo> are special exception
734mappings derived from <$Config{privlib}>/F<unicore/SpecialCasing.txt>.
735The C<Digit> and C<Fold> mappings that one can see in the directory
736are not directly user-accessible, one can use either the
737C<Unicode::UCD> module, or just match case-insensitively (that's when
738the C<Fold> mapping is used).
739
740A final note on the user-defined property tests and mappings: they
741will be used only if the scalar has been marked as having Unicode
742characters. Old byte-style strings will not be affected.
743
376d9008 744=head2 Character Encodings for Input and Output
8cbd9a7a 745
7221edc9 746See L<Encode>.
8cbd9a7a 747
c29a771d 748=head2 Unicode Regular Expression Support Level
776f8809 749
376d9008
JB
750The following list of Unicode support for regular expressions describes
751all the features currently supported. The references to "Level N"
752and the section numbers refer to the Unicode Technical Report 18,
753"Unicode Regular Expression Guidelines".
776f8809
JH
754
755=over 4
756
757=item *
758
759Level 1 - Basic Unicode Support
760
761 2.1 Hex Notation - done [1]
3bfdc84c 762 Named Notation - done [2]
776f8809
JH
763 2.2 Categories - done [3][4]
764 2.3 Subtraction - MISSING [5][6]
765 2.4 Simple Word Boundaries - done [7]
78d3e1bf 766 2.5 Simple Loose Matches - done [8]
776f8809
JH
767 2.6 End of Line - MISSING [9][10]
768
769 [ 1] \x{...}
770 [ 2] \N{...}
eb0cc9e3 771 [ 3] . \p{...} \P{...}
29bdacb8 772 [ 4] now scripts (see UTR#24 Script Names) in addition to blocks
776f8809 773 [ 5] have negation
237bad5b
JH
774 [ 6] can use regular expression look-ahead [a]
775 or user-defined character properties [b] to emulate subtraction
776f8809 776 [ 7] include Letters in word characters
376d9008 777 [ 8] note that Perl does Full case-folding in matching, not Simple:
835863de 778 for example U+1F88 is equivalent with U+1F00 U+03B9,
e0f9d4a8 779 not with 1F80. This difference matters for certain Greek
376d9008
JB
780 capital letters with certain modifiers: the Full case-folding
781 decomposes the letter, while the Simple case-folding would map
e0f9d4a8 782 it to a single character.
776f8809 783 [ 9] see UTR#13 Unicode Newline Guidelines
835863de 784 [10] should do ^ and $ also on \x{85}, \x{2028} and \x{2029}
ec83e909 785 (should also affect <>, $., and script line numbers)
3bfdc84c 786 (the \x{85}, \x{2028} and \x{2029} do match \s)
7207e29d 787
237bad5b 788[a] You can mimic class subtraction using lookahead.
dbe420b4 789For example, what TR18 might write as
29bdacb8 790
dbe420b4
JH
791 [{Greek}-[{UNASSIGNED}]]
792
793in Perl can be written as:
794
1d81abf3
JH
795 (?!\p{Unassigned})\p{InGreekAndCoptic}
796 (?=\p{Assigned})\p{InGreekAndCoptic}
dbe420b4
JH
797
798But in this particular example, you probably really want
799
1bfb14c4 800 \p{GreekAndCoptic}
dbe420b4
JH
801
802which will match assigned characters known to be part of the Greek script.
29bdacb8 803
818c4caa 804[b] See L</"User-Defined Character Properties">.
237bad5b 805
776f8809
JH
806=item *
807
808Level 2 - Extended Unicode Support
809
63de3cb2
JH
810 3.1 Surrogates - MISSING [11]
811 3.2 Canonical Equivalents - MISSING [12][13]
812 3.3 Locale-Independent Graphemes - MISSING [14]
813 3.4 Locale-Independent Words - MISSING [15]
814 3.5 Locale-Independent Loose Matches - MISSING [16]
815
816 [11] Surrogates are solely a UTF-16 concept and Perl's internal
817 representation is UTF-8. The Encode module does UTF-16, though.
818 [12] see UTR#15 Unicode Normalization
819 [13] have Unicode::Normalize but not integrated to regexes
820 [14] have \X but at this level . should equal that
821 [15] need three classes, not just \w and \W
822 [16] see UTR#21 Case Mappings
776f8809
JH
823
824=item *
825
826Level 3 - Locale-Sensitive Support
827
828 4.1 Locale-Dependent Categories - MISSING
829 4.2 Locale-Dependent Graphemes - MISSING [16][17]
830 4.3 Locale-Dependent Words - MISSING
831 4.4 Locale-Dependent Loose Matches - MISSING
832 4.5 Locale-Dependent Ranges - MISSING
833
834 [16] see UTR#10 Unicode Collation Algorithms
835 [17] have Unicode::Collate but not integrated to regexes
836
837=back
838
c349b1b9
JH
839=head2 Unicode Encodings
840
376d9008
JB
841Unicode characters are assigned to I<code points>, which are abstract
842numbers. To use these numbers, various encodings are needed.
c349b1b9
JH
843
844=over 4
845
c29a771d 846=item *
5cb3728c
RB
847
848UTF-8
c349b1b9 849
3e4dbfed 850UTF-8 is a variable-length (1 to 6 bytes, current character allocations
376d9008
JB
851require 4 bytes), byte-order independent encoding. For ASCII (and we
852really do mean 7-bit ASCII, not another 8-bit encoding), UTF-8 is
853transparent.
c349b1b9 854
8c007b5a 855The following table is from Unicode 3.2.
05632f9a
JH
856
857 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
858
8c007b5a
JH
859 U+0000..U+007F 00..7F
860 U+0080..U+07FF C2..DF 80..BF
ec90690f
ST
861 U+0800..U+0FFF E0 A0..BF 80..BF
862 U+1000..U+CFFF E1..EC 80..BF 80..BF
863 U+D000..U+D7FF ED 80..9F 80..BF
8c007b5a 864 U+D800..U+DFFF ******* ill-formed *******
ec90690f 865 U+E000..U+FFFF EE..EF 80..BF 80..BF
05632f9a
JH
866 U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
867 U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
868 U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
869
376d9008
JB
870Note the C<A0..BF> in C<U+0800..U+0FFF>, the C<80..9F> in
871C<U+D000...U+D7FF>, the C<90..B>F in C<U+10000..U+3FFFF>, and the
872C<80...8F> in C<U+100000..U+10FFFF>. The "gaps" are caused by legal
873UTF-8 avoiding non-shortest encodings: it is technically possible to
874UTF-8-encode a single code point in different ways, but that is
875explicitly forbidden, and the shortest possible encoding should always
876be used. So that's what Perl does.
37361303 877
376d9008 878Another way to look at it is via bits:
05632f9a
JH
879
880 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
881
882 0aaaaaaa 0aaaaaaa
883 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
884 ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
885 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
886
887As you can see, the continuation bytes all begin with C<10>, and the
8c007b5a 888leading bits of the start byte tell how many bytes the are in the
05632f9a
JH
889encoded character.
890
c29a771d 891=item *
5cb3728c
RB
892
893UTF-EBCDIC
dbe420b4 894
376d9008 895Like UTF-8 but EBCDIC-safe, in the way that UTF-8 is ASCII-safe.
dbe420b4 896
c29a771d 897=item *
5cb3728c
RB
898
899UTF-16, UTF-16BE, UTF16-LE, Surrogates, and BOMs (Byte Order Marks)
c349b1b9 900
1bfb14c4
JH
901The followings items are mostly for reference and general Unicode
902knowledge, Perl doesn't use these constructs internally.
dbe420b4 903
c349b1b9 904UTF-16 is a 2 or 4 byte encoding. The Unicode code points
1bfb14c4
JH
905C<U+0000..U+FFFF> are stored in a single 16-bit unit, and the code
906points C<U+10000..U+10FFFF> in two 16-bit units. The latter case is
c349b1b9
JH
907using I<surrogates>, the first 16-bit unit being the I<high
908surrogate>, and the second being the I<low surrogate>.
909
376d9008 910Surrogates are code points set aside to encode the C<U+10000..U+10FFFF>
c349b1b9 911range of Unicode code points in pairs of 16-bit units. The I<high
376d9008
JB
912surrogates> are the range C<U+D800..U+DBFF>, and the I<low surrogates>
913are the range C<U+DC00..U+DFFF>. The surrogate encoding is
c349b1b9
JH
914
915 $hi = ($uni - 0x10000) / 0x400 + 0xD800;
916 $lo = ($uni - 0x10000) % 0x400 + 0xDC00;
917
918and the decoding is
919
1a3fa709 920 $uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00);
c349b1b9 921
feda178f 922If you try to generate surrogates (for example by using chr()), you
376d9008
JB
923will get a warning if warnings are turned on, because those code
924points are not valid for a Unicode character.
9466bab6 925
376d9008 926Because of the 16-bitness, UTF-16 is byte-order dependent. UTF-16
c349b1b9 927itself can be used for in-memory computations, but if storage or
376d9008
JB
928transfer is required either UTF-16BE (big-endian) or UTF-16LE
929(little-endian) encodings must be chosen.
c349b1b9
JH
930
931This introduces another problem: what if you just know that your data
376d9008
JB
932is UTF-16, but you don't know which endianness? Byte Order Marks, or
933BOMs, are a solution to this. A special character has been reserved
86bbd6d1 934in Unicode to function as a byte order marker: the character with the
376d9008 935code point C<U+FEFF> is the BOM.
042da322 936
c349b1b9 937The trick is that if you read a BOM, you will know the byte order,
376d9008
JB
938since if it was written on a big-endian platform, you will read the
939bytes C<0xFE 0xFF>, but if it was written on a little-endian platform,
940you will read the bytes C<0xFF 0xFE>. (And if the originating platform
941was writing in UTF-8, you will read the bytes C<0xEF 0xBB 0xBF>.)
042da322 942
86bbd6d1 943The way this trick works is that the character with the code point
376d9008
JB
944C<U+FFFE> is guaranteed not to be a valid Unicode character, so the
945sequence of bytes C<0xFF 0xFE> is unambiguously "BOM, represented in
1bfb14c4 946little-endian format" and cannot be C<U+FFFE>, represented in big-endian
042da322 947format".
c349b1b9 948
c29a771d 949=item *
5cb3728c
RB
950
951UTF-32, UTF-32BE, UTF32-LE
c349b1b9
JH
952
953The UTF-32 family is pretty much like the UTF-16 family, expect that
042da322 954the units are 32-bit, and therefore the surrogate scheme is not
376d9008
JB
955needed. The BOM signatures will be C<0x00 0x00 0xFE 0xFF> for BE and
956C<0xFF 0xFE 0x00 0x00> for LE.
c349b1b9 957
c29a771d 958=item *
5cb3728c
RB
959
960UCS-2, UCS-4
c349b1b9 961
86bbd6d1 962Encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit
376d9008 963encoding. Unlike UTF-16, UCS-2 is not extensible beyond C<U+FFFF>,
339cfa0e
JH
964because it does not use surrogates. UCS-4 is a 32-bit encoding,
965functionally identical to UTF-32.
c349b1b9 966
c29a771d 967=item *
5cb3728c
RB
968
969UTF-7
c349b1b9 970
376d9008
JB
971A seven-bit safe (non-eight-bit) encoding, which is useful if the
972transport or storage is not eight-bit safe. Defined by RFC 2152.
c349b1b9 973
95a1a48b
JH
974=back
975
0d7c09bb
JH
976=head2 Security Implications of Unicode
977
978=over 4
979
980=item *
981
982Malformed UTF-8
bf0fa0b2
JH
983
984Unfortunately, the specification of UTF-8 leaves some room for
985interpretation of how many bytes of encoded output one should generate
376d9008
JB
986from one input Unicode character. Strictly speaking, the shortest
987possible sequence of UTF-8 bytes should be generated,
988because otherwise there is potential for an input buffer overflow at
feda178f 989the receiving end of a UTF-8 connection. Perl always generates the
376d9008
JB
990shortest length UTF-8, and with warnings on Perl will warn about
991non-shortest length UTF-8 along with other malformations, such as the
992surrogates, which are not real Unicode code points.
bf0fa0b2 993
0d7c09bb
JH
994=item *
995
996Regular expressions behave slightly differently between byte data and
376d9008
JB
997character (Unicode) data. For example, the "word character" character
998class C<\w> will work differently depending on if data is eight-bit bytes
999or Unicode.
0d7c09bb 1000
376d9008
JB
1001In the first case, the set of C<\w> characters is either small--the
1002default set of alphabetic characters, digits, and the "_"--or, if you
0d7c09bb
JH
1003are using a locale (see L<perllocale>), the C<\w> might contain a few
1004more letters according to your language and country.
1005
376d9008 1006In the second case, the C<\w> set of characters is much, much larger.
1bfb14c4
JH
1007Most importantly, even in the set of the first 256 characters, it will
1008probably match different characters: unlike most locales, which are
1009specific to a language and country pair, Unicode classifies all the
1010characters that are letters I<somewhere> as C<\w>. For example, your
1011locale might not think that LATIN SMALL LETTER ETH is a letter (unless
1012you happen to speak Icelandic), but Unicode does.
0d7c09bb 1013
376d9008 1014As discussed elsewhere, Perl has one foot (two hooves?) planted in
1bfb14c4
JH
1015each of two worlds: the old world of bytes and the new world of
1016characters, upgrading from bytes to characters when necessary.
376d9008
JB
1017If your legacy code does not explicitly use Unicode, no automatic
1018switch-over to characters should happen. Characters shouldn't get
1bfb14c4
JH
1019downgraded to bytes, either. It is possible to accidentally mix bytes
1020and characters, however (see L<perluniintro>), in which case C<\w> in
1021regular expressions might start behaving differently. Review your
1022code. Use warnings and the C<strict> pragma.
0d7c09bb
JH
1023
1024=back
1025
c349b1b9
JH
1026=head2 Unicode in Perl on EBCDIC
1027
376d9008
JB
1028The way Unicode is handled on EBCDIC platforms is still
1029experimental. On such platforms, references to UTF-8 encoding in this
1030document and elsewhere should be read as meaning the UTF-EBCDIC
1031specified in Unicode Technical Report 16, unless ASCII vs. EBCDIC issues
c349b1b9 1032are specifically discussed. There is no C<utfebcdic> pragma or
376d9008 1033":utfebcdic" layer; rather, "utf8" and ":utf8" are reused to mean
86bbd6d1
PN
1034the platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic>
1035for more discussion of the issues.
c349b1b9 1036
b310b053
JH
1037=head2 Locales
1038
4616122b 1039Usually locale settings and Unicode do not affect each other, but
b310b053
JH
1040there are a couple of exceptions:
1041
1042=over 4
1043
1044=item *
1045
8aa8f774
JH
1046You can enable automatic UTF-8-ification of your standard file
1047handles, default C<open()> layer, and C<@ARGV> by using either
1048the C<-C> command line switch or the C<PERL_UNICODE> environment
1049variable, see L<perlrun> for the documentation of the C<-C> switch.
b310b053
JH
1050
1051=item *
1052
376d9008
JB
1053Perl tries really hard to work both with Unicode and the old
1054byte-oriented world. Most often this is nice, but sometimes Perl's
1055straddling of the proverbial fence causes problems.
b310b053
JH
1056
1057=back
1058
95a1a48b
JH
1059=head2 Using Unicode in XS
1060
3a2263fe
RGS
1061If you want to handle Perl Unicode in XS extensions, you may find the
1062following C APIs useful. See also L<perlguts/"Unicode Support"> for an
1063explanation about Unicode at the XS level, and L<perlapi> for the API
1064details.
95a1a48b
JH
1065
1066=over 4
1067
1068=item *
1069
1bfb14c4
JH
1070C<DO_UTF8(sv)> returns true if the C<UTF8> flag is on and the bytes
1071pragma is not in effect. C<SvUTF8(sv)> returns true is the C<UTF8>
1072flag is on; the bytes pragma is ignored. The C<UTF8> flag being on
1073does B<not> mean that there are any characters of code points greater
1074than 255 (or 127) in the scalar or that there are even any characters
1075in the scalar. What the C<UTF8> flag means is that the sequence of
1076octets in the representation of the scalar is the sequence of UTF-8
1077encoded code points of the characters of a string. The C<UTF8> flag
1078being off means that each octet in this representation encodes a
1079single character with code point 0..255 within the string. Perl's
1080Unicode model is not to use UTF-8 until it is absolutely necessary.
95a1a48b
JH
1081
1082=item *
1083
1bfb14c4
JH
1084C<uvuni_to_utf8(buf, chr>) writes a Unicode character code point into
1085a buffer encoding the code point as UTF-8, and returns a pointer
95a1a48b
JH
1086pointing after the UTF-8 bytes.
1087
1088=item *
1089
376d9008
JB
1090C<utf8_to_uvuni(buf, lenp)> reads UTF-8 encoded bytes from a buffer and
1091returns the Unicode character code point and, optionally, the length of
1092the UTF-8 byte sequence.
95a1a48b
JH
1093
1094=item *
1095
376d9008
JB
1096C<utf8_length(start, end)> returns the length of the UTF-8 encoded buffer
1097in characters. C<sv_len_utf8(sv)> returns the length of the UTF-8 encoded
95a1a48b
JH
1098scalar.
1099
1100=item *
1101
376d9008
JB
1102C<sv_utf8_upgrade(sv)> converts the string of the scalar to its UTF-8
1103encoded form. C<sv_utf8_downgrade(sv)> does the opposite, if
1104possible. C<sv_utf8_encode(sv)> is like sv_utf8_upgrade except that
1105it does not set the C<UTF8> flag. C<sv_utf8_decode()> does the
1106opposite of C<sv_utf8_encode()>. Note that none of these are to be
1107used as general-purpose encoding or decoding interfaces: C<use Encode>
1108for that. C<sv_utf8_upgrade()> is affected by the encoding pragma
1109but C<sv_utf8_downgrade()> is not (since the encoding pragma is
1110designed to be a one-way street).
95a1a48b
JH
1111
1112=item *
1113
376d9008 1114C<is_utf8_char(s)> returns true if the pointer points to a valid UTF-8
90f968e0 1115character.
95a1a48b
JH
1116
1117=item *
1118
376d9008 1119C<is_utf8_string(buf, len)> returns true if C<len> bytes of the buffer
95a1a48b
JH
1120are valid UTF-8.
1121
1122=item *
1123
376d9008
JB
1124C<UTF8SKIP(buf)> will return the number of bytes in the UTF-8 encoded
1125character in the buffer. C<UNISKIP(chr)> will return the number of bytes
1126required to UTF-8-encode the Unicode character code point. C<UTF8SKIP()>
90f968e0 1127is useful for example for iterating over the characters of a UTF-8
376d9008 1128encoded buffer; C<UNISKIP()> is useful, for example, in computing
90f968e0 1129the size required for a UTF-8 encoded buffer.
95a1a48b
JH
1130
1131=item *
1132
376d9008 1133C<utf8_distance(a, b)> will tell the distance in characters between the
95a1a48b
JH
1134two pointers pointing to the same UTF-8 encoded buffer.
1135
1136=item *
1137
376d9008
JB
1138C<utf8_hop(s, off)> will return a pointer to an UTF-8 encoded buffer
1139that is C<off> (positive or negative) Unicode characters displaced
1140from the UTF-8 buffer C<s>. Be careful not to overstep the buffer:
1141C<utf8_hop()> will merrily run off the end or the beginning of the
1142buffer if told to do so.
95a1a48b 1143
d2cc3551
JH
1144=item *
1145
376d9008
JB
1146C<pv_uni_display(dsv, spv, len, pvlim, flags)> and
1147C<sv_uni_display(dsv, ssv, pvlim, flags)> are useful for debugging the
1148output of Unicode strings and scalars. By default they are useful
1149only for debugging--they display B<all> characters as hexadecimal code
1bfb14c4
JH
1150points--but with the flags C<UNI_DISPLAY_ISPRINT>,
1151C<UNI_DISPLAY_BACKSLASH>, and C<UNI_DISPLAY_QQ> you can make the
1152output more readable.
d2cc3551
JH
1153
1154=item *
1155
376d9008
JB
1156C<ibcmp_utf8(s1, pe1, u1, l1, u1, s2, pe2, l2, u2)> can be used to
1157compare two strings case-insensitively in Unicode. For case-sensitive
1158comparisons you can just use C<memEQ()> and C<memNE()> as usual.
d2cc3551 1159
c349b1b9
JH
1160=back
1161
95a1a48b
JH
1162For more information, see L<perlapi>, and F<utf8.c> and F<utf8.h>
1163in the Perl source code distribution.
1164
c29a771d
JH
1165=head1 BUGS
1166
376d9008 1167=head2 Interaction with Locales
7eabb34d 1168
376d9008
JB
1169Use of locales with Unicode data may lead to odd results. Currently,
1170Perl attempts to attach 8-bit locale info to characters in the range
11710..255, but this technique is demonstrably incorrect for locales that
1172use characters above that range when mapped into Unicode. Perl's
1173Unicode support will also tend to run slower. Use of locales with
1174Unicode is discouraged.
c29a771d 1175
376d9008 1176=head2 Interaction with Extensions
7eabb34d 1177
376d9008 1178When Perl exchanges data with an extension, the extension should be
7eabb34d 1179able to understand the UTF-8 flag and act accordingly. If the
376d9008
JB
1180extension doesn't know about the flag, it's likely that the extension
1181will return incorrectly-flagged data.
7eabb34d
A
1182
1183So if you're working with Unicode data, consult the documentation of
1184every module you're using if there are any issues with Unicode data
1185exchange. If the documentation does not talk about Unicode at all,
a73d23f6 1186suspect the worst and probably look at the source to learn how the
376d9008 1187module is implemented. Modules written completely in Perl shouldn't
a73d23f6
RGS
1188cause problems. Modules that directly or indirectly access code written
1189in other programming languages are at risk.
7eabb34d 1190
376d9008 1191For affected functions, the simple strategy to avoid data corruption is
7eabb34d 1192to always make the encoding of the exchanged data explicit. Choose an
376d9008 1193encoding that you know the extension can handle. Convert arguments passed
7eabb34d
A
1194to the extensions to that encoding and convert results back from that
1195encoding. Write wrapper functions that do the conversions for you, so
1196you can later change the functions when the extension catches up.
1197
376d9008 1198To provide an example, let's say the popular Foo::Bar::escape_html
7eabb34d
A
1199function doesn't deal with Unicode data yet. The wrapper function
1200would convert the argument to raw UTF-8 and convert the result back to
376d9008 1201Perl's internal representation like so:
7eabb34d
A
1202
1203 sub my_escape_html ($) {
1204 my($what) = shift;
1205 return unless defined $what;
1206 Encode::decode_utf8(Foo::Bar::escape_html(Encode::encode_utf8($what)));
1207 }
1208
1209Sometimes, when the extension does not convert data but just stores
1210and retrieves them, you will be in a position to use the otherwise
1211dangerous Encode::_utf8_on() function. Let's say the popular
66b79f27 1212C<Foo::Bar> extension, written in C, provides a C<param> method that
7eabb34d
A
1213lets you store and retrieve data according to these prototypes:
1214
1215 $self->param($name, $value); # set a scalar
1216 $value = $self->param($name); # retrieve a scalar
1217
1218If it does not yet provide support for any encoding, one could write a
1219derived class with such a C<param> method:
1220
1221 sub param {
1222 my($self,$name,$value) = @_;
1223 utf8::upgrade($name); # make sure it is UTF-8 encoded
1224 if (defined $value)
1225 utf8::upgrade($value); # make sure it is UTF-8 encoded
1226 return $self->SUPER::param($name,$value);
1227 } else {
1228 my $ret = $self->SUPER::param($name);
1229 Encode::_utf8_on($ret); # we know, it is UTF-8 encoded
1230 return $ret;
1231 }
1232 }
1233
a73d23f6
RGS
1234Some extensions provide filters on data entry/exit points, such as
1235DB_File::filter_store_key and family. Look out for such filters in
66b79f27 1236the documentation of your extensions, they can make the transition to
7eabb34d
A
1237Unicode data much easier.
1238
376d9008 1239=head2 Speed
7eabb34d 1240
c29a771d 1241Some functions are slower when working on UTF-8 encoded strings than
574c8022 1242on byte encoded strings. All functions that need to hop over
7c17141f
JH
1243characters such as length(), substr() or index(), or matching regular
1244expressions can work B<much> faster when the underlying data are
1245byte-encoded.
1246
1247In Perl 5.8.0 the slowness was often quite spectacular; in Perl 5.8.1
1248a caching scheme was introduced which will hopefully make the slowness
1249somewhat less spectacular. Operations with UTF-8 encoded strings are
1250still slower, though.
666f95b9 1251
c8d992ba
A
1252=head2 Porting code from perl-5.6.X
1253
1254Perl 5.8 has a different Unicode model from 5.6. In 5.6 the programmer
1255was required to use the C<utf8> pragma to declare that a given scope
1256expected to deal with Unicode data and had to make sure that only
1257Unicode data were reaching that scope. If you have code that is
1258working with 5.6, you will need some of the following adjustments to
1259your code. The examples are written such that the code will continue
1260to work under 5.6, so you should be safe to try them out.
1261
1262=over 4
1263
1264=item *
1265
1266A filehandle that should read or write UTF-8
1267
1268 if ($] > 5.007) {
1269 binmode $fh, ":utf8";
1270 }
1271
1272=item *
1273
1274A scalar that is going to be passed to some extension
1275
1276Be it Compress::Zlib, Apache::Request or any extension that has no
1277mention of Unicode in the manpage, you need to make sure that the
1278UTF-8 flag is stripped off. Note that at the time of this writing
1279(October 2002) the mentioned modules are not UTF-8-aware. Please
1280check the documentation to verify if this is still true.
1281
1282 if ($] > 5.007) {
1283 require Encode;
1284 $val = Encode::encode_utf8($val); # make octets
1285 }
1286
1287=item *
1288
1289A scalar we got back from an extension
1290
1291If you believe the scalar comes back as UTF-8, you will most likely
1292want the UTF-8 flag restored:
1293
1294 if ($] > 5.007) {
1295 require Encode;
1296 $val = Encode::decode_utf8($val);
1297 }
1298
1299=item *
1300
1301Same thing, if you are really sure it is UTF-8
1302
1303 if ($] > 5.007) {
1304 require Encode;
1305 Encode::_utf8_on($val);
1306 }
1307
1308=item *
1309
1310A wrapper for fetchrow_array and fetchrow_hashref
1311
1312When the database contains only UTF-8, a wrapper function or method is
1313a convenient way to replace all your fetchrow_array and
1314fetchrow_hashref calls. A wrapper function will also make it easier to
1315adapt to future enhancements in your database driver. Note that at the
1316time of this writing (October 2002), the DBI has no standardized way
1317to deal with UTF-8 data. Please check the documentation to verify if
1318that is still true.
1319
1320 sub fetchrow {
1321 my($self, $sth, $what) = @_; # $what is one of fetchrow_{array,hashref}
1322 if ($] < 5.007) {
1323 return $sth->$what;
1324 } else {
1325 require Encode;
1326 if (wantarray) {
1327 my @arr = $sth->$what;
1328 for (@arr) {
1329 defined && /[^\000-\177]/ && Encode::_utf8_on($_);
1330 }
1331 return @arr;
1332 } else {
1333 my $ret = $sth->$what;
1334 if (ref $ret) {
1335 for my $k (keys %$ret) {
1336 defined && /[^\000-\177]/ && Encode::_utf8_on($_) for $ret->{$k};
1337 }
1338 return $ret;
1339 } else {
1340 defined && /[^\000-\177]/ && Encode::_utf8_on($_) for $ret;
1341 return $ret;
1342 }
1343 }
1344 }
1345 }
1346
1347
1348=item *
1349
1350A large scalar that you know can only contain ASCII
1351
1352Scalars that contain only ASCII and are marked as UTF-8 are sometimes
1353a drag to your program. If you recognize such a situation, just remove
1354the UTF-8 flag:
1355
1356 utf8::downgrade($val) if $] > 5.007;
1357
1358=back
1359
393fec97
GS
1360=head1 SEE ALSO
1361
72ff2908 1362L<perluniintro>, L<encoding>, L<Encode>, L<open>, L<utf8>, L<bytes>,
a05d7ebb 1363L<perlretut>, L<perlvar/"${^UNICODE}">
393fec97
GS
1364
1365=cut