This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
missing quote from #17957
[perl5.git] / pod / perlunicode.pod
CommitLineData
393fec97
GS
1=head1 NAME
2
3perlunicode - Unicode support in Perl
4
5=head1 DESCRIPTION
6
0a1f2d14 7=head2 Important Caveats
21bad921 8
376d9008 9Unicode support is an extensive requirement. While Perl does not
c349b1b9
JH
10implement the Unicode standard or the accompanying technical reports
11from cover to cover, Perl does support many Unicode features.
21bad921 12
13a2d996 13=over 4
21bad921 14
fae2c0fb 15=item Input and Output Layers
21bad921 16
376d9008 17Perl knows when a filehandle uses Perl's internal Unicode encodings
1bfb14c4
JH
18(UTF-8, or UTF-EBCDIC if in EBCDIC) if the filehandle is opened with
19the ":utf8" layer. Other encodings can be converted to Perl's
20encoding on input or from Perl's encoding on output by use of the
21":encoding(...)" layer. See L<open>.
c349b1b9 22
376d9008 23To indicate that Perl source itself is using a particular encoding,
c349b1b9 24see L<encoding>.
21bad921
GS
25
26=item Regular Expressions
27
c349b1b9 28The regular expression compiler produces polymorphic opcodes. That is,
376d9008
JB
29the pattern adapts to the data and automatically switches to the Unicode
30character scheme when presented with Unicode data--or instead uses
31a traditional byte scheme when presented with byte data.
21bad921 32
ad0029c4 33=item C<use utf8> still needed to enable UTF-8/UTF-EBCDIC in scripts
21bad921 34
376d9008
JB
35As a compatibility measure, the C<use utf8> pragma must be explicitly
36included to enable recognition of UTF-8 in the Perl scripts themselves
1bfb14c4
JH
37(in string or regular expression literals, or in identifier names) on
38ASCII-based machines or to recognize UTF-EBCDIC on EBCDIC-based
376d9008 39machines. B<These are the only times when an explicit C<use utf8>
8f8cf39c 40is needed.> See L<utf8>.
21bad921 41
1768d7eb 42You can also use the C<encoding> pragma to change the default encoding
6ec9efec 43of the data in your script; see L<encoding>.
1768d7eb 44
21bad921
GS
45=back
46
376d9008 47=head2 Byte and Character Semantics
393fec97 48
376d9008 49Beginning with version 5.6, Perl uses logically-wide characters to
3e4dbfed 50represent strings internally.
393fec97 51
376d9008
JB
52In future, Perl-level operations will be expected to work with
53characters rather than bytes.
393fec97 54
376d9008 55However, as an interim compatibility measure, Perl aims to
75daf61c
JH
56provide a safe migration path from byte semantics to character
57semantics for programs. For operations where Perl can unambiguously
376d9008 58decide that the input data are characters, Perl switches to
75daf61c
JH
59character semantics. For operations where this determination cannot
60be made without additional information from the user, Perl decides in
376d9008 61favor of compatibility and chooses to use byte semantics.
8cbd9a7a
GS
62
63This behavior preserves compatibility with earlier versions of Perl,
376d9008
JB
64which allowed byte semantics in Perl operations only if
65none of the program's inputs were marked as being as source of Unicode
8cbd9a7a
GS
66character data. Such data may come from filehandles, from calls to
67external programs, from information provided by the system (such as %ENV),
21bad921 68or from literals and constants in the source text.
8cbd9a7a 69
376d9008
JB
70On Windows platforms, if the C<-C> command line switch is used or the
71${^WIDE_SYSTEM_CALLS} global flag is set to C<1>, all system calls
72will use the corresponding wide-character APIs. This feature is
73available only on Windows to conform to the API standard already
1bfb14c4
JH
74established for that platform--and there are very few non-Windows
75platforms that have Unicode-aware APIs.
8cbd9a7a 76
376d9008
JB
77The C<bytes> pragma will always, regardless of platform, force byte
78semantics in a particular lexical scope. See L<bytes>.
8cbd9a7a
GS
79
80The C<utf8> pragma is primarily a compatibility device that enables
75daf61c 81recognition of UTF-(8|EBCDIC) in literals encountered by the parser.
376d9008
JB
82Note that this pragma is only required while Perl defaults to byte
83semantics; when character semantics become the default, this pragma
84may become a no-op. See L<utf8>.
85
86Unless explicitly stated, Perl operators use character semantics
87for Unicode data and byte semantics for non-Unicode data.
88The decision to use character semantics is made transparently. If
89input data comes from a Unicode source--for example, if a character
fae2c0fb 90encoding layer is added to a filehandle or a literal Unicode
376d9008
JB
91string constant appears in a program--character semantics apply.
92Otherwise, byte semantics are in effect. The C<bytes> pragma should
93be used to force byte semantics on Unicode data.
94
95If strings operating under byte semantics and strings with Unicode
96character data are concatenated, the new string will be upgraded to
97I<ISO 8859-1 (Latin-1)>, even if the old Unicode string used EBCDIC.
98This translation is done without regard to the system's native 8-bit
99encoding, so to change this for systems with non-Latin-1 and
100non-EBCDIC native encodings use the C<encoding> pragma. See
101L<encoding>.
7dedd01f 102
feda178f 103Under character semantics, many operations that formerly operated on
376d9008 104bytes now operate on characters. A character in Perl is
feda178f 105logically just a number ranging from 0 to 2**31 or so. Larger
376d9008
JB
106characters may encode into longer sequences of bytes internally, but
107this internal detail is mostly hidden for Perl code.
108See L<perluniintro> for more.
393fec97 109
376d9008 110=head2 Effects of Character Semantics
393fec97
GS
111
112Character semantics have the following effects:
113
114=over 4
115
116=item *
117
376d9008 118Strings--including hash keys--and regular expression patterns may
574c8022 119contain characters that have an ordinal value larger than 255.
393fec97 120
feda178f
JH
121If you use a Unicode editor to edit your program, Unicode characters
122may occur directly within the literal strings in one of the various
376d9008
JB
123Unicode encodings (UTF-8, UTF-EBCDIC, UCS-2, etc.), but will be recognized
124as such and converted to Perl's internal representation only if the
feda178f 125appropriate L<encoding> is specified.
3e4dbfed 126
1bfb14c4
JH
127Unicode characters can also be added to a string by using the
128C<\x{...}> notation. The Unicode code for the desired character, in
376d9008
JB
129hexadecimal, should be placed in the braces. For instance, a smiley
130face is C<\x{263A}>. This encoding scheme only works for characters
131with a code of 0x100 or above.
3e4dbfed
JF
132
133Additionally, if you
574c8022 134
3e4dbfed 135 use charnames ':full';
574c8022 136
1bfb14c4
JH
137you can use the C<\N{...}> notation and put the official Unicode
138character name within the braces, such as C<\N{WHITE SMILING FACE}>.
376d9008 139
393fec97
GS
140
141=item *
142
574c8022
JH
143If an appropriate L<encoding> is specified, identifiers within the
144Perl script may contain Unicode alphanumeric characters, including
376d9008
JB
145ideographs. Perl does not currently attempt to canonicalize variable
146names.
393fec97 147
393fec97
GS
148=item *
149
1bfb14c4
JH
150Regular expressions match characters instead of bytes. "." matches
151a character instead of a byte. The C<\C> pattern is provided to force
152a match a single byte--a C<char> in C, hence C<\C>.
393fec97 153
393fec97
GS
154=item *
155
156Character classes in regular expressions match characters instead of
376d9008 157bytes and match against the character properties specified in the
1bfb14c4 158Unicode properties database. C<\w> can be used to match a Japanese
75daf61c 159ideograph, for instance.
393fec97 160
393fec97
GS
161=item *
162
eb0cc9e3 163Named Unicode properties, scripts, and block ranges may be used like
376d9008
JB
164character classes via the C<\p{}> "matches property" construct and
165the C<\P{}> negation, "doesn't match property".
1bfb14c4
JH
166
167For instance, C<\p{Lu}> matches any character with the Unicode "Lu"
168(Letter, uppercase) property, while C<\p{M}> matches any character
169with an "M" (mark--accents and such) property. Brackets are not
170required for single letter properties, so C<\p{M}> is equivalent to
171C<\pM>. Many predefined properties are available, such as
172C<\p{Mirrored}> and C<\p{Tibetan}>.
4193bef7 173
cfc01aea 174The official Unicode script and block names have spaces and dashes as
376d9008 175separators, but for convenience you can use dashes, spaces, or
1bfb14c4
JH
176underbars, and case is unimportant. It is recommended, however, that
177for consistency you use the following naming: the official Unicode
178script, property, or block name (see below for the additional rules
179that apply to block names) with whitespace and dashes removed, and the
180words "uppercase-first-lowercase-rest". C<Latin-1 Supplement> thus
181becomes C<Latin1Supplement>.
4193bef7 182
376d9008
JB
183You can also use negation in both C<\p{}> and C<\P{}> by introducing a caret
184(^) between the first brace and the property name: C<\p{^Tamil}> is
eb0cc9e3 185equal to C<\P{Tamil}>.
4193bef7 186
eb0cc9e3 187Here are the basic Unicode General Category properties, followed by their
376d9008
JB
188long form. You can use either; C<\p{Lu}> and C<\p{LowercaseLetter}>,
189for instance, are identical.
393fec97 190
d73e5302
JH
191 Short Long
192
193 L Letter
eb0cc9e3
JH
194 Lu UppercaseLetter
195 Ll LowercaseLetter
196 Lt TitlecaseLetter
197 Lm ModifierLetter
198 Lo OtherLetter
d73e5302
JH
199
200 M Mark
eb0cc9e3
JH
201 Mn NonspacingMark
202 Mc SpacingMark
203 Me EnclosingMark
d73e5302
JH
204
205 N Number
eb0cc9e3
JH
206 Nd DecimalNumber
207 Nl LetterNumber
208 No OtherNumber
d73e5302
JH
209
210 P Punctuation
eb0cc9e3
JH
211 Pc ConnectorPunctuation
212 Pd DashPunctuation
213 Ps OpenPunctuation
214 Pe ClosePunctuation
215 Pi InitialPunctuation
d73e5302 216 (may behave like Ps or Pe depending on usage)
eb0cc9e3 217 Pf FinalPunctuation
d73e5302 218 (may behave like Ps or Pe depending on usage)
eb0cc9e3 219 Po OtherPunctuation
d73e5302
JH
220
221 S Symbol
eb0cc9e3
JH
222 Sm MathSymbol
223 Sc CurrencySymbol
224 Sk ModifierSymbol
225 So OtherSymbol
d73e5302
JH
226
227 Z Separator
eb0cc9e3
JH
228 Zs SpaceSeparator
229 Zl LineSeparator
230 Zp ParagraphSeparator
d73e5302
JH
231
232 C Other
e150c829
JH
233 Cc Control
234 Cf Format
eb0cc9e3
JH
235 Cs Surrogate (not usable)
236 Co PrivateUse
e150c829 237 Cn Unassigned
1ac13f9a 238
376d9008 239Single-letter properties match all characters in any of the
3e4dbfed 240two-letter sub-properties starting with the same letter.
376d9008 241C<L&> is a special case, which is an alias for C<Ll>, C<Lu>, and C<Lt>.
32293815 242
eb0cc9e3 243Because Perl hides the need for the user to understand the internal
1bfb14c4
JH
244representation of Unicode characters, there is no need to implement
245the somewhat messy concept of surrogates. C<Cs> is therefore not
eb0cc9e3 246supported.
d73e5302 247
376d9008
JB
248Because scripts differ in their directionality--Hebrew is
249written right to left, for example--Unicode supplies these properties:
32293815 250
eb0cc9e3 251 Property Meaning
92e830a9 252
d73e5302
JH
253 BidiL Left-to-Right
254 BidiLRE Left-to-Right Embedding
255 BidiLRO Left-to-Right Override
256 BidiR Right-to-Left
257 BidiAL Right-to-Left Arabic
258 BidiRLE Right-to-Left Embedding
259 BidiRLO Right-to-Left Override
260 BidiPDF Pop Directional Format
261 BidiEN European Number
262 BidiES European Number Separator
263 BidiET European Number Terminator
264 BidiAN Arabic Number
265 BidiCS Common Number Separator
266 BidiNSM Non-Spacing Mark
267 BidiBN Boundary Neutral
268 BidiB Paragraph Separator
269 BidiS Segment Separator
270 BidiWS Whitespace
271 BidiON Other Neutrals
32293815 272
376d9008 273For example, C<\p{BidiR}> matches characters that are normally
eb0cc9e3
JH
274written right to left.
275
210b36aa
AMS
276=back
277
2796c109
JH
278=head2 Scripts
279
376d9008
JB
280The script names which can be used by C<\p{...}> and C<\P{...}>,
281such as in C<\p{Latin}> or C<\p{Cyrillic}>, are as follows:
2796c109 282
1ac13f9a 283 Arabic
e9ad1727 284 Armenian
1ac13f9a 285 Bengali
e9ad1727 286 Bopomofo
1d81abf3 287 Buhid
eb0cc9e3 288 CanadianAboriginal
e9ad1727
JH
289 Cherokee
290 Cyrillic
291 Deseret
292 Devanagari
293 Ethiopic
294 Georgian
295 Gothic
296 Greek
1ac13f9a 297 Gujarati
e9ad1727
JH
298 Gurmukhi
299 Han
300 Hangul
1d81abf3 301 Hanunoo
e9ad1727
JH
302 Hebrew
303 Hiragana
304 Inherited
1ac13f9a 305 Kannada
e9ad1727
JH
306 Katakana
307 Khmer
1ac13f9a 308 Lao
e9ad1727
JH
309 Latin
310 Malayalam
311 Mongolian
1ac13f9a 312 Myanmar
1ac13f9a 313 Ogham
eb0cc9e3 314 OldItalic
e9ad1727 315 Oriya
1ac13f9a 316 Runic
e9ad1727
JH
317 Sinhala
318 Syriac
1d81abf3
JH
319 Tagalog
320 Tagbanwa
e9ad1727
JH
321 Tamil
322 Telugu
323 Thaana
324 Thai
325 Tibetan
1ac13f9a 326 Yi
1ac13f9a 327
376d9008 328Extended property classes can supplement the basic
1ac13f9a
JH
329properties, defined by the F<PropList> Unicode database:
330
1d81abf3 331 ASCIIHexDigit
eb0cc9e3 332 BidiControl
1ac13f9a 333 Dash
1d81abf3 334 Deprecated
1ac13f9a
JH
335 Diacritic
336 Extender
1d81abf3 337 GraphemeLink
eb0cc9e3 338 HexDigit
e9ad1727
JH
339 Hyphen
340 Ideographic
1d81abf3
JH
341 IDSBinaryOperator
342 IDSTrinaryOperator
eb0cc9e3 343 JoinControl
1d81abf3 344 LogicalOrderException
eb0cc9e3
JH
345 NoncharacterCodePoint
346 OtherAlphabetic
1d81abf3
JH
347 OtherDefaultIgnorableCodePoint
348 OtherGraphemeExtend
eb0cc9e3
JH
349 OtherLowercase
350 OtherMath
351 OtherUppercase
352 QuotationMark
1d81abf3
JH
353 Radical
354 SoftDotted
355 TerminalPunctuation
356 UnifiedIdeograph
eb0cc9e3 357 WhiteSpace
1ac13f9a 358
376d9008 359and there are further derived properties:
1ac13f9a 360
eb0cc9e3
JH
361 Alphabetic Lu + Ll + Lt + Lm + Lo + OtherAlphabetic
362 Lowercase Ll + OtherLowercase
363 Uppercase Lu + OtherUppercase
364 Math Sm + OtherMath
1ac13f9a
JH
365
366 ID_Start Lu + Ll + Lt + Lm + Lo + Nl
367 ID_Continue ID_Start + Mn + Mc + Nd + Pc
368
369 Any Any character
66b79f27
RGS
370 Assigned Any non-Cn character (i.e. synonym for \P{Cn})
371 Unassigned Synonym for \p{Cn}
1ac13f9a 372 Common Any character (or unassigned code point)
e150c829 373 not explicitly assigned to a script
2796c109 374
1bfb14c4
JH
375For backward compatibility (with Perl 5.6), all properties mentioned
376so far may have C<Is> prepended to their name, so C<\P{IsLu}>, for
377example, is equal to C<\P{Lu}>.
eb0cc9e3 378
2796c109
JH
379=head2 Blocks
380
1bfb14c4
JH
381In addition to B<scripts>, Unicode also defines B<blocks> of
382characters. The difference between scripts and blocks is that the
383concept of scripts is closer to natural languages, while the concept
384of blocks is more of an artificial grouping based on groups of 256
376d9008 385Unicode characters. For example, the C<Latin> script contains letters
1bfb14c4 386from many blocks but does not contain all the characters from those
376d9008
JB
387blocks. It does not, for example, contain digits, because digits are
388shared across many scripts. Digits and similar groups, like
389punctuation, are in a category called C<Common>.
2796c109 390
cfc01aea
JF
391For more about scripts, see the UTR #24:
392
393 http://www.unicode.org/unicode/reports/tr24/
394
395For more about blocks, see:
396
397 http://www.unicode.org/Public/UNIDATA/Blocks.txt
2796c109 398
376d9008
JB
399Block names are given with the C<In> prefix. For example, the
400Katakana block is referenced via C<\p{InKatakana}>. The C<In>
7eabb34d 401prefix may be omitted if there is no naming conflict with a script
eb0cc9e3 402or any other property, but it is recommended that C<In> always be used
1bfb14c4 403for block tests to avoid confusion.
eb0cc9e3
JH
404
405These block names are supported:
406
1d81abf3
JH
407 InAlphabeticPresentationForms
408 InArabic
409 InArabicPresentationFormsA
410 InArabicPresentationFormsB
411 InArmenian
412 InArrows
413 InBasicLatin
414 InBengali
415 InBlockElements
416 InBopomofo
417 InBopomofoExtended
418 InBoxDrawing
419 InBraillePatterns
420 InBuhid
421 InByzantineMusicalSymbols
422 InCJKCompatibility
423 InCJKCompatibilityForms
424 InCJKCompatibilityIdeographs
425 InCJKCompatibilityIdeographsSupplement
426 InCJKRadicalsSupplement
427 InCJKSymbolsAndPunctuation
428 InCJKUnifiedIdeographs
429 InCJKUnifiedIdeographsExtensionA
430 InCJKUnifiedIdeographsExtensionB
431 InCherokee
432 InCombiningDiacriticalMarks
433 InCombiningDiacriticalMarksforSymbols
434 InCombiningHalfMarks
435 InControlPictures
436 InCurrencySymbols
437 InCyrillic
438 InCyrillicSupplementary
439 InDeseret
440 InDevanagari
441 InDingbats
442 InEnclosedAlphanumerics
443 InEnclosedCJKLettersAndMonths
444 InEthiopic
445 InGeneralPunctuation
446 InGeometricShapes
447 InGeorgian
448 InGothic
449 InGreekExtended
450 InGreekAndCoptic
451 InGujarati
452 InGurmukhi
453 InHalfwidthAndFullwidthForms
454 InHangulCompatibilityJamo
455 InHangulJamo
456 InHangulSyllables
457 InHanunoo
458 InHebrew
459 InHighPrivateUseSurrogates
460 InHighSurrogates
461 InHiragana
462 InIPAExtensions
463 InIdeographicDescriptionCharacters
464 InKanbun
465 InKangxiRadicals
466 InKannada
467 InKatakana
468 InKatakanaPhoneticExtensions
469 InKhmer
470 InLao
471 InLatin1Supplement
472 InLatinExtendedA
473 InLatinExtendedAdditional
474 InLatinExtendedB
475 InLetterlikeSymbols
476 InLowSurrogates
477 InMalayalam
478 InMathematicalAlphanumericSymbols
479 InMathematicalOperators
480 InMiscellaneousMathematicalSymbolsA
481 InMiscellaneousMathematicalSymbolsB
482 InMiscellaneousSymbols
483 InMiscellaneousTechnical
484 InMongolian
485 InMusicalSymbols
486 InMyanmar
487 InNumberForms
488 InOgham
489 InOldItalic
490 InOpticalCharacterRecognition
491 InOriya
492 InPrivateUseArea
493 InRunic
494 InSinhala
495 InSmallFormVariants
496 InSpacingModifierLetters
497 InSpecials
498 InSuperscriptsAndSubscripts
499 InSupplementalArrowsA
500 InSupplementalArrowsB
501 InSupplementalMathematicalOperators
502 InSupplementaryPrivateUseAreaA
503 InSupplementaryPrivateUseAreaB
504 InSyriac
505 InTagalog
506 InTagbanwa
507 InTags
508 InTamil
509 InTelugu
510 InThaana
511 InThai
512 InTibetan
513 InUnifiedCanadianAboriginalSyllabics
514 InVariationSelectors
515 InYiRadicals
516 InYiSyllables
32293815 517
210b36aa
AMS
518=over 4
519
393fec97
GS
520=item *
521
376d9008
JB
522The special pattern C<\X> matches any extended Unicode
523sequence--"a combining character sequence" in Standardese--where the
524first character is a base character and subsequent characters are mark
525characters that apply to the base character. C<\X> is equivalent to
393fec97
GS
526C<(?:\PM\pM*)>.
527
393fec97
GS
528=item *
529
383e7cdd 530The C<tr///> operator translates characters instead of bytes. Note
376d9008
JB
531that the C<tr///CU> functionality has been removed. For similar
532functionality see pack('U0', ...) and pack('C0', ...).
393fec97 533
393fec97
GS
534=item *
535
536Case translation operators use the Unicode case translation tables
376d9008
JB
537when character input is provided. Note that C<uc()>, or C<\U> in
538interpolated strings, translates to uppercase, while C<ucfirst>,
539or C<\u> in interpolated strings, translates to titlecase in languages
540that make the distinction.
393fec97
GS
541
542=item *
543
376d9008 544Most operators that deal with positions or lengths in a string will
75daf61c
JH
545automatically switch to using character positions, including
546C<chop()>, C<substr()>, C<pos()>, C<index()>, C<rindex()>,
547C<sprintf()>, C<write()>, and C<length()>. Operators that
376d9008
JB
548specifically do not switch include C<vec()>, C<pack()>, and
549C<unpack()>. Operators that really don't care include C<chomp()>,
550operators that treats strings as a bucket of bits such as C<sort()>,
551and operators dealing with filenames.
393fec97
GS
552
553=item *
554
1bfb14c4 555The C<pack()>/C<unpack()> letters C<c> and C<C> do I<not> change,
376d9008 556since they are often used for byte-oriented formats. Again, think
1bfb14c4
JH
557C<char> in the C language.
558
559There is a new C<U> specifier that converts between Unicode characters
560and code points.
393fec97
GS
561
562=item *
563
376d9008
JB
564The C<chr()> and C<ord()> functions work on characters, similar to
565C<pack("U")> and C<unpack("U")>, I<not> C<pack("C")> and
566C<unpack("C")>. C<pack("C")> and C<unpack("C")> are methods for
567emulating byte-oriented C<chr()> and C<ord()> on Unicode strings.
568While these methods reveal the internal encoding of Unicode strings,
569that is not something one normally needs to care about at all.
393fec97
GS
570
571=item *
572
376d9008
JB
573The bit string operators, C<& | ^ ~>, can operate on character data.
574However, for backward compatibility, such as when using bit string
575operations when characters are all less than 256 in ordinal value, one
576should not use C<~> (the bit complement) with characters of both
577values less than 256 and values greater than 256. Most importantly,
578DeMorgan's laws (C<~($x|$y) eq ~$x&~$y> and C<~($x&$y) eq ~$x|~$y>)
579will not hold. The reason for this mathematical I<faux pas> is that
580the complement cannot return B<both> the 8-bit (byte-wide) bit
581complement B<and> the full character-wide bit complement.
a1ca4561
YST
582
583=item *
584
983ffd37
JH
585lc(), uc(), lcfirst(), and ucfirst() work for the following cases:
586
587=over 8
588
589=item *
590
591the case mapping is from a single Unicode character to another
376d9008 592single Unicode character, or
983ffd37
JH
593
594=item *
595
596the case mapping is from a single Unicode character to more
376d9008 597than one Unicode character.
983ffd37
JH
598
599=back
600
63de3cb2
JH
601Things to do with locales (Lithuanian, Turkish, Azeri) do B<not> work
602since Perl does not understand the concept of Unicode locales.
983ffd37 603
dc33ebcf
RGS
604See the Unicode Technical Report #21, Case Mappings, for more details.
605
983ffd37
JH
606=back
607
dc33ebcf 608=over 4
ac1256e8
JH
609
610=item *
611
393fec97
GS
612And finally, C<scalar reverse()> reverses by character rather than by byte.
613
614=back
615
376d9008 616=head2 User-Defined Character Properties
491fd90a
JH
617
618You can define your own character properties by defining subroutines
376d9008 619whose names begin with "In" or "Is". The subroutines must be
491fd90a
JH
620visible in the package that uses the properties. The user-defined
621properties can be used in the regular expression C<\p> and C<\P>
622constructs.
623
376d9008
JB
624The subroutines must return a specially-formatted string, with one
625or more newline-separated lines. Each line must be one of the following:
491fd90a
JH
626
627=over 4
628
629=item *
630
99a6b1f0 631Two hexadecimal numbers separated by horizontal whitespace (space or
376d9008 632tabular characters) denoting a range of Unicode code points to include.
491fd90a
JH
633
634=item *
635
376d9008
JB
636Something to include, prefixed by "+": a built-in character
637property (prefixed by "utf8::"), to represent all the characters in that
638property; two hexadecimal code points for a range; or a single
639hexadecimal code point.
491fd90a
JH
640
641=item *
642
376d9008 643Something to exclude, prefixed by "-": an existing character
11ef8fdd 644property (prefixed by "utf8::"), for all the characters in that
376d9008
JB
645property; two hexadecimal code points for a range; or a single
646hexadecimal code point.
491fd90a
JH
647
648=item *
649
376d9008 650Something to negate, prefixed "!": an existing character
11ef8fdd 651property (prefixed by "utf8::") for all the characters except the
376d9008
JB
652characters in the property; two hexadecimal code points for a range;
653or a single hexadecimal code point.
491fd90a
JH
654
655=back
656
657For example, to define a property that covers both the Japanese
658syllabaries (hiragana and katakana), you can define
659
660 sub InKana {
d5822f25
A
661 return <<END;
662 3040\t309F
663 30A0\t30FF
491fd90a
JH
664 END
665 }
666
d5822f25
A
667Imagine that the here-doc end marker is at the beginning of the line.
668Now you can use C<\p{InKana}> and C<\P{InKana}>.
491fd90a
JH
669
670You could also have used the existing block property names:
671
672 sub InKana {
673 return <<'END';
674 +utf8::InHiragana
675 +utf8::InKatakana
676 END
677 }
678
679Suppose you wanted to match only the allocated characters,
d5822f25 680not the raw block ranges: in other words, you want to remove
491fd90a
JH
681the non-characters:
682
683 sub InKana {
684 return <<'END';
685 +utf8::InHiragana
686 +utf8::InKatakana
687 -utf8::IsCn
688 END
689 }
690
691The negation is useful for defining (surprise!) negated classes.
692
693 sub InNotKana {
694 return <<'END';
695 !utf8::InHiragana
696 -utf8::InKatakana
697 +utf8::IsCn
698 END
699 }
700
376d9008 701=head2 Character Encodings for Input and Output
8cbd9a7a 702
7221edc9 703See L<Encode>.
8cbd9a7a 704
c29a771d 705=head2 Unicode Regular Expression Support Level
776f8809 706
376d9008
JB
707The following list of Unicode support for regular expressions describes
708all the features currently supported. The references to "Level N"
709and the section numbers refer to the Unicode Technical Report 18,
710"Unicode Regular Expression Guidelines".
776f8809
JH
711
712=over 4
713
714=item *
715
716Level 1 - Basic Unicode Support
717
718 2.1 Hex Notation - done [1]
3bfdc84c 719 Named Notation - done [2]
776f8809
JH
720 2.2 Categories - done [3][4]
721 2.3 Subtraction - MISSING [5][6]
722 2.4 Simple Word Boundaries - done [7]
78d3e1bf 723 2.5 Simple Loose Matches - done [8]
776f8809
JH
724 2.6 End of Line - MISSING [9][10]
725
726 [ 1] \x{...}
727 [ 2] \N{...}
eb0cc9e3 728 [ 3] . \p{...} \P{...}
29bdacb8 729 [ 4] now scripts (see UTR#24 Script Names) in addition to blocks
776f8809 730 [ 5] have negation
237bad5b
JH
731 [ 6] can use regular expression look-ahead [a]
732 or user-defined character properties [b] to emulate subtraction
776f8809 733 [ 7] include Letters in word characters
376d9008 734 [ 8] note that Perl does Full case-folding in matching, not Simple:
e0f9d4a8
JH
735 for example U+1F88 is equivalent with U+1F000 U+03B9,
736 not with 1F80. This difference matters for certain Greek
376d9008
JB
737 capital letters with certain modifiers: the Full case-folding
738 decomposes the letter, while the Simple case-folding would map
e0f9d4a8 739 it to a single character.
776f8809 740 [ 9] see UTR#13 Unicode Newline Guidelines
ec83e909
JH
741 [10] should do ^ and $ also on \x{85}, \x{2028} and \x{2029})
742 (should also affect <>, $., and script line numbers)
3bfdc84c 743 (the \x{85}, \x{2028} and \x{2029} do match \s)
7207e29d 744
237bad5b 745[a] You can mimic class subtraction using lookahead.
dbe420b4 746For example, what TR18 might write as
29bdacb8 747
dbe420b4
JH
748 [{Greek}-[{UNASSIGNED}]]
749
750in Perl can be written as:
751
1d81abf3
JH
752 (?!\p{Unassigned})\p{InGreekAndCoptic}
753 (?=\p{Assigned})\p{InGreekAndCoptic}
dbe420b4
JH
754
755But in this particular example, you probably really want
756
1bfb14c4 757 \p{GreekAndCoptic}
dbe420b4
JH
758
759which will match assigned characters known to be part of the Greek script.
29bdacb8 760
818c4caa 761[b] See L</"User-Defined Character Properties">.
237bad5b 762
776f8809
JH
763=item *
764
765Level 2 - Extended Unicode Support
766
63de3cb2
JH
767 3.1 Surrogates - MISSING [11]
768 3.2 Canonical Equivalents - MISSING [12][13]
769 3.3 Locale-Independent Graphemes - MISSING [14]
770 3.4 Locale-Independent Words - MISSING [15]
771 3.5 Locale-Independent Loose Matches - MISSING [16]
772
773 [11] Surrogates are solely a UTF-16 concept and Perl's internal
774 representation is UTF-8. The Encode module does UTF-16, though.
775 [12] see UTR#15 Unicode Normalization
776 [13] have Unicode::Normalize but not integrated to regexes
777 [14] have \X but at this level . should equal that
778 [15] need three classes, not just \w and \W
779 [16] see UTR#21 Case Mappings
776f8809
JH
780
781=item *
782
783Level 3 - Locale-Sensitive Support
784
785 4.1 Locale-Dependent Categories - MISSING
786 4.2 Locale-Dependent Graphemes - MISSING [16][17]
787 4.3 Locale-Dependent Words - MISSING
788 4.4 Locale-Dependent Loose Matches - MISSING
789 4.5 Locale-Dependent Ranges - MISSING
790
791 [16] see UTR#10 Unicode Collation Algorithms
792 [17] have Unicode::Collate but not integrated to regexes
793
794=back
795
c349b1b9
JH
796=head2 Unicode Encodings
797
376d9008
JB
798Unicode characters are assigned to I<code points>, which are abstract
799numbers. To use these numbers, various encodings are needed.
c349b1b9
JH
800
801=over 4
802
c29a771d 803=item *
5cb3728c
RB
804
805UTF-8
c349b1b9 806
3e4dbfed 807UTF-8 is a variable-length (1 to 6 bytes, current character allocations
376d9008
JB
808require 4 bytes), byte-order independent encoding. For ASCII (and we
809really do mean 7-bit ASCII, not another 8-bit encoding), UTF-8 is
810transparent.
c349b1b9 811
8c007b5a 812The following table is from Unicode 3.2.
05632f9a
JH
813
814 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
815
8c007b5a
JH
816 U+0000..U+007F 00..7F
817 U+0080..U+07FF C2..DF 80..BF
ec90690f
ST
818 U+0800..U+0FFF E0 A0..BF 80..BF
819 U+1000..U+CFFF E1..EC 80..BF 80..BF
820 U+D000..U+D7FF ED 80..9F 80..BF
8c007b5a 821 U+D800..U+DFFF ******* ill-formed *******
ec90690f 822 U+E000..U+FFFF EE..EF 80..BF 80..BF
05632f9a
JH
823 U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
824 U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
825 U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
826
376d9008
JB
827Note the C<A0..BF> in C<U+0800..U+0FFF>, the C<80..9F> in
828C<U+D000...U+D7FF>, the C<90..B>F in C<U+10000..U+3FFFF>, and the
829C<80...8F> in C<U+100000..U+10FFFF>. The "gaps" are caused by legal
830UTF-8 avoiding non-shortest encodings: it is technically possible to
831UTF-8-encode a single code point in different ways, but that is
832explicitly forbidden, and the shortest possible encoding should always
833be used. So that's what Perl does.
37361303 834
376d9008 835Another way to look at it is via bits:
05632f9a
JH
836
837 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
838
839 0aaaaaaa 0aaaaaaa
840 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
841 ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
842 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
843
844As you can see, the continuation bytes all begin with C<10>, and the
8c007b5a 845leading bits of the start byte tell how many bytes the are in the
05632f9a
JH
846encoded character.
847
c29a771d 848=item *
5cb3728c
RB
849
850UTF-EBCDIC
dbe420b4 851
376d9008 852Like UTF-8 but EBCDIC-safe, in the way that UTF-8 is ASCII-safe.
dbe420b4 853
c29a771d 854=item *
5cb3728c
RB
855
856UTF-16, UTF-16BE, UTF16-LE, Surrogates, and BOMs (Byte Order Marks)
c349b1b9 857
1bfb14c4
JH
858The followings items are mostly for reference and general Unicode
859knowledge, Perl doesn't use these constructs internally.
dbe420b4 860
c349b1b9 861UTF-16 is a 2 or 4 byte encoding. The Unicode code points
1bfb14c4
JH
862C<U+0000..U+FFFF> are stored in a single 16-bit unit, and the code
863points C<U+10000..U+10FFFF> in two 16-bit units. The latter case is
c349b1b9
JH
864using I<surrogates>, the first 16-bit unit being the I<high
865surrogate>, and the second being the I<low surrogate>.
866
376d9008 867Surrogates are code points set aside to encode the C<U+10000..U+10FFFF>
c349b1b9 868range of Unicode code points in pairs of 16-bit units. The I<high
376d9008
JB
869surrogates> are the range C<U+D800..U+DBFF>, and the I<low surrogates>
870are the range C<U+DC00..U+DFFF>. The surrogate encoding is
c349b1b9
JH
871
872 $hi = ($uni - 0x10000) / 0x400 + 0xD800;
873 $lo = ($uni - 0x10000) % 0x400 + 0xDC00;
874
875and the decoding is
876
1a3fa709 877 $uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00);
c349b1b9 878
feda178f 879If you try to generate surrogates (for example by using chr()), you
376d9008
JB
880will get a warning if warnings are turned on, because those code
881points are not valid for a Unicode character.
9466bab6 882
376d9008 883Because of the 16-bitness, UTF-16 is byte-order dependent. UTF-16
c349b1b9 884itself can be used for in-memory computations, but if storage or
376d9008
JB
885transfer is required either UTF-16BE (big-endian) or UTF-16LE
886(little-endian) encodings must be chosen.
c349b1b9
JH
887
888This introduces another problem: what if you just know that your data
376d9008
JB
889is UTF-16, but you don't know which endianness? Byte Order Marks, or
890BOMs, are a solution to this. A special character has been reserved
86bbd6d1 891in Unicode to function as a byte order marker: the character with the
376d9008 892code point C<U+FEFF> is the BOM.
042da322 893
c349b1b9 894The trick is that if you read a BOM, you will know the byte order,
376d9008
JB
895since if it was written on a big-endian platform, you will read the
896bytes C<0xFE 0xFF>, but if it was written on a little-endian platform,
897you will read the bytes C<0xFF 0xFE>. (And if the originating platform
898was writing in UTF-8, you will read the bytes C<0xEF 0xBB 0xBF>.)
042da322 899
86bbd6d1 900The way this trick works is that the character with the code point
376d9008
JB
901C<U+FFFE> is guaranteed not to be a valid Unicode character, so the
902sequence of bytes C<0xFF 0xFE> is unambiguously "BOM, represented in
1bfb14c4 903little-endian format" and cannot be C<U+FFFE>, represented in big-endian
042da322 904format".
c349b1b9 905
c29a771d 906=item *
5cb3728c
RB
907
908UTF-32, UTF-32BE, UTF32-LE
c349b1b9
JH
909
910The UTF-32 family is pretty much like the UTF-16 family, expect that
042da322 911the units are 32-bit, and therefore the surrogate scheme is not
376d9008
JB
912needed. The BOM signatures will be C<0x00 0x00 0xFE 0xFF> for BE and
913C<0xFF 0xFE 0x00 0x00> for LE.
c349b1b9 914
c29a771d 915=item *
5cb3728c
RB
916
917UCS-2, UCS-4
c349b1b9 918
86bbd6d1 919Encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit
376d9008 920encoding. Unlike UTF-16, UCS-2 is not extensible beyond C<U+FFFF>,
339cfa0e
JH
921because it does not use surrogates. UCS-4 is a 32-bit encoding,
922functionally identical to UTF-32.
c349b1b9 923
c29a771d 924=item *
5cb3728c
RB
925
926UTF-7
c349b1b9 927
376d9008
JB
928A seven-bit safe (non-eight-bit) encoding, which is useful if the
929transport or storage is not eight-bit safe. Defined by RFC 2152.
c349b1b9 930
95a1a48b
JH
931=back
932
0d7c09bb
JH
933=head2 Security Implications of Unicode
934
935=over 4
936
937=item *
938
939Malformed UTF-8
bf0fa0b2
JH
940
941Unfortunately, the specification of UTF-8 leaves some room for
942interpretation of how many bytes of encoded output one should generate
376d9008
JB
943from one input Unicode character. Strictly speaking, the shortest
944possible sequence of UTF-8 bytes should be generated,
945because otherwise there is potential for an input buffer overflow at
feda178f 946the receiving end of a UTF-8 connection. Perl always generates the
376d9008
JB
947shortest length UTF-8, and with warnings on Perl will warn about
948non-shortest length UTF-8 along with other malformations, such as the
949surrogates, which are not real Unicode code points.
bf0fa0b2 950
0d7c09bb
JH
951=item *
952
953Regular expressions behave slightly differently between byte data and
376d9008
JB
954character (Unicode) data. For example, the "word character" character
955class C<\w> will work differently depending on if data is eight-bit bytes
956or Unicode.
0d7c09bb 957
376d9008
JB
958In the first case, the set of C<\w> characters is either small--the
959default set of alphabetic characters, digits, and the "_"--or, if you
0d7c09bb
JH
960are using a locale (see L<perllocale>), the C<\w> might contain a few
961more letters according to your language and country.
962
376d9008 963In the second case, the C<\w> set of characters is much, much larger.
1bfb14c4
JH
964Most importantly, even in the set of the first 256 characters, it will
965probably match different characters: unlike most locales, which are
966specific to a language and country pair, Unicode classifies all the
967characters that are letters I<somewhere> as C<\w>. For example, your
968locale might not think that LATIN SMALL LETTER ETH is a letter (unless
969you happen to speak Icelandic), but Unicode does.
0d7c09bb 970
376d9008 971As discussed elsewhere, Perl has one foot (two hooves?) planted in
1bfb14c4
JH
972each of two worlds: the old world of bytes and the new world of
973characters, upgrading from bytes to characters when necessary.
376d9008
JB
974If your legacy code does not explicitly use Unicode, no automatic
975switch-over to characters should happen. Characters shouldn't get
1bfb14c4
JH
976downgraded to bytes, either. It is possible to accidentally mix bytes
977and characters, however (see L<perluniintro>), in which case C<\w> in
978regular expressions might start behaving differently. Review your
979code. Use warnings and the C<strict> pragma.
0d7c09bb
JH
980
981=back
982
c349b1b9
JH
983=head2 Unicode in Perl on EBCDIC
984
376d9008
JB
985The way Unicode is handled on EBCDIC platforms is still
986experimental. On such platforms, references to UTF-8 encoding in this
987document and elsewhere should be read as meaning the UTF-EBCDIC
988specified in Unicode Technical Report 16, unless ASCII vs. EBCDIC issues
c349b1b9 989are specifically discussed. There is no C<utfebcdic> pragma or
376d9008 990":utfebcdic" layer; rather, "utf8" and ":utf8" are reused to mean
86bbd6d1
PN
991the platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic>
992for more discussion of the issues.
c349b1b9 993
b310b053
JH
994=head2 Locales
995
4616122b 996Usually locale settings and Unicode do not affect each other, but
b310b053
JH
997there are a couple of exceptions:
998
999=over 4
1000
1001=item *
1002
1003If your locale environment variables (LANGUAGE, LC_ALL, LC_CTYPE, LANG)
1004contain the strings 'UTF-8' or 'UTF8' (case-insensitive matching),
376d9008
JB
1005the default encodings of your STDIN, STDOUT, and STDERR, and of
1006B<any subsequent file open>, are considered to be UTF-8.
b310b053
JH
1007
1008=item *
1009
376d9008
JB
1010Perl tries really hard to work both with Unicode and the old
1011byte-oriented world. Most often this is nice, but sometimes Perl's
1012straddling of the proverbial fence causes problems.
b310b053
JH
1013
1014=back
1015
95a1a48b
JH
1016=head2 Using Unicode in XS
1017
1018If you want to handle Perl Unicode in XS extensions, you may find
376d9008 1019the following C APIs useful. See L<perlapi> for details.
95a1a48b
JH
1020
1021=over 4
1022
1023=item *
1024
1bfb14c4
JH
1025C<DO_UTF8(sv)> returns true if the C<UTF8> flag is on and the bytes
1026pragma is not in effect. C<SvUTF8(sv)> returns true is the C<UTF8>
1027flag is on; the bytes pragma is ignored. The C<UTF8> flag being on
1028does B<not> mean that there are any characters of code points greater
1029than 255 (or 127) in the scalar or that there are even any characters
1030in the scalar. What the C<UTF8> flag means is that the sequence of
1031octets in the representation of the scalar is the sequence of UTF-8
1032encoded code points of the characters of a string. The C<UTF8> flag
1033being off means that each octet in this representation encodes a
1034single character with code point 0..255 within the string. Perl's
1035Unicode model is not to use UTF-8 until it is absolutely necessary.
95a1a48b
JH
1036
1037=item *
1038
1bfb14c4
JH
1039C<uvuni_to_utf8(buf, chr>) writes a Unicode character code point into
1040a buffer encoding the code point as UTF-8, and returns a pointer
95a1a48b
JH
1041pointing after the UTF-8 bytes.
1042
1043=item *
1044
376d9008
JB
1045C<utf8_to_uvuni(buf, lenp)> reads UTF-8 encoded bytes from a buffer and
1046returns the Unicode character code point and, optionally, the length of
1047the UTF-8 byte sequence.
95a1a48b
JH
1048
1049=item *
1050
376d9008
JB
1051C<utf8_length(start, end)> returns the length of the UTF-8 encoded buffer
1052in characters. C<sv_len_utf8(sv)> returns the length of the UTF-8 encoded
95a1a48b
JH
1053scalar.
1054
1055=item *
1056
376d9008
JB
1057C<sv_utf8_upgrade(sv)> converts the string of the scalar to its UTF-8
1058encoded form. C<sv_utf8_downgrade(sv)> does the opposite, if
1059possible. C<sv_utf8_encode(sv)> is like sv_utf8_upgrade except that
1060it does not set the C<UTF8> flag. C<sv_utf8_decode()> does the
1061opposite of C<sv_utf8_encode()>. Note that none of these are to be
1062used as general-purpose encoding or decoding interfaces: C<use Encode>
1063for that. C<sv_utf8_upgrade()> is affected by the encoding pragma
1064but C<sv_utf8_downgrade()> is not (since the encoding pragma is
1065designed to be a one-way street).
95a1a48b
JH
1066
1067=item *
1068
376d9008 1069C<is_utf8_char(s)> returns true if the pointer points to a valid UTF-8
90f968e0 1070character.
95a1a48b
JH
1071
1072=item *
1073
376d9008 1074C<is_utf8_string(buf, len)> returns true if C<len> bytes of the buffer
95a1a48b
JH
1075are valid UTF-8.
1076
1077=item *
1078
376d9008
JB
1079C<UTF8SKIP(buf)> will return the number of bytes in the UTF-8 encoded
1080character in the buffer. C<UNISKIP(chr)> will return the number of bytes
1081required to UTF-8-encode the Unicode character code point. C<UTF8SKIP()>
90f968e0 1082is useful for example for iterating over the characters of a UTF-8
376d9008 1083encoded buffer; C<UNISKIP()> is useful, for example, in computing
90f968e0 1084the size required for a UTF-8 encoded buffer.
95a1a48b
JH
1085
1086=item *
1087
376d9008 1088C<utf8_distance(a, b)> will tell the distance in characters between the
95a1a48b
JH
1089two pointers pointing to the same UTF-8 encoded buffer.
1090
1091=item *
1092
376d9008
JB
1093C<utf8_hop(s, off)> will return a pointer to an UTF-8 encoded buffer
1094that is C<off> (positive or negative) Unicode characters displaced
1095from the UTF-8 buffer C<s>. Be careful not to overstep the buffer:
1096C<utf8_hop()> will merrily run off the end or the beginning of the
1097buffer if told to do so.
95a1a48b 1098
d2cc3551
JH
1099=item *
1100
376d9008
JB
1101C<pv_uni_display(dsv, spv, len, pvlim, flags)> and
1102C<sv_uni_display(dsv, ssv, pvlim, flags)> are useful for debugging the
1103output of Unicode strings and scalars. By default they are useful
1104only for debugging--they display B<all> characters as hexadecimal code
1bfb14c4
JH
1105points--but with the flags C<UNI_DISPLAY_ISPRINT>,
1106C<UNI_DISPLAY_BACKSLASH>, and C<UNI_DISPLAY_QQ> you can make the
1107output more readable.
d2cc3551
JH
1108
1109=item *
1110
376d9008
JB
1111C<ibcmp_utf8(s1, pe1, u1, l1, u1, s2, pe2, l2, u2)> can be used to
1112compare two strings case-insensitively in Unicode. For case-sensitive
1113comparisons you can just use C<memEQ()> and C<memNE()> as usual.
d2cc3551 1114
c349b1b9
JH
1115=back
1116
95a1a48b
JH
1117For more information, see L<perlapi>, and F<utf8.c> and F<utf8.h>
1118in the Perl source code distribution.
1119
c29a771d
JH
1120=head1 BUGS
1121
376d9008 1122=head2 Interaction with Locales
7eabb34d 1123
376d9008
JB
1124Use of locales with Unicode data may lead to odd results. Currently,
1125Perl attempts to attach 8-bit locale info to characters in the range
11260..255, but this technique is demonstrably incorrect for locales that
1127use characters above that range when mapped into Unicode. Perl's
1128Unicode support will also tend to run slower. Use of locales with
1129Unicode is discouraged.
c29a771d 1130
376d9008 1131=head2 Interaction with Extensions
7eabb34d 1132
376d9008 1133When Perl exchanges data with an extension, the extension should be
7eabb34d 1134able to understand the UTF-8 flag and act accordingly. If the
376d9008
JB
1135extension doesn't know about the flag, it's likely that the extension
1136will return incorrectly-flagged data.
7eabb34d
A
1137
1138So if you're working with Unicode data, consult the documentation of
1139every module you're using if there are any issues with Unicode data
1140exchange. If the documentation does not talk about Unicode at all,
a73d23f6 1141suspect the worst and probably look at the source to learn how the
376d9008 1142module is implemented. Modules written completely in Perl shouldn't
a73d23f6
RGS
1143cause problems. Modules that directly or indirectly access code written
1144in other programming languages are at risk.
7eabb34d 1145
376d9008 1146For affected functions, the simple strategy to avoid data corruption is
7eabb34d 1147to always make the encoding of the exchanged data explicit. Choose an
376d9008 1148encoding that you know the extension can handle. Convert arguments passed
7eabb34d
A
1149to the extensions to that encoding and convert results back from that
1150encoding. Write wrapper functions that do the conversions for you, so
1151you can later change the functions when the extension catches up.
1152
376d9008 1153To provide an example, let's say the popular Foo::Bar::escape_html
7eabb34d
A
1154function doesn't deal with Unicode data yet. The wrapper function
1155would convert the argument to raw UTF-8 and convert the result back to
376d9008 1156Perl's internal representation like so:
7eabb34d
A
1157
1158 sub my_escape_html ($) {
1159 my($what) = shift;
1160 return unless defined $what;
1161 Encode::decode_utf8(Foo::Bar::escape_html(Encode::encode_utf8($what)));
1162 }
1163
1164Sometimes, when the extension does not convert data but just stores
1165and retrieves them, you will be in a position to use the otherwise
1166dangerous Encode::_utf8_on() function. Let's say the popular
66b79f27 1167C<Foo::Bar> extension, written in C, provides a C<param> method that
7eabb34d
A
1168lets you store and retrieve data according to these prototypes:
1169
1170 $self->param($name, $value); # set a scalar
1171 $value = $self->param($name); # retrieve a scalar
1172
1173If it does not yet provide support for any encoding, one could write a
1174derived class with such a C<param> method:
1175
1176 sub param {
1177 my($self,$name,$value) = @_;
1178 utf8::upgrade($name); # make sure it is UTF-8 encoded
1179 if (defined $value)
1180 utf8::upgrade($value); # make sure it is UTF-8 encoded
1181 return $self->SUPER::param($name,$value);
1182 } else {
1183 my $ret = $self->SUPER::param($name);
1184 Encode::_utf8_on($ret); # we know, it is UTF-8 encoded
1185 return $ret;
1186 }
1187 }
1188
a73d23f6
RGS
1189Some extensions provide filters on data entry/exit points, such as
1190DB_File::filter_store_key and family. Look out for such filters in
66b79f27 1191the documentation of your extensions, they can make the transition to
7eabb34d
A
1192Unicode data much easier.
1193
376d9008 1194=head2 Speed
7eabb34d 1195
c29a771d 1196Some functions are slower when working on UTF-8 encoded strings than
574c8022 1197on byte encoded strings. All functions that need to hop over
c29a771d
JH
1198characters such as length(), substr() or index() can work B<much>
1199faster when the underlying data are byte-encoded. Witness the
1200following benchmark:
666f95b9 1201
c29a771d
JH
1202 % perl -e '
1203 use Benchmark;
1204 use strict;
1205 our $l = 10000;
1206 our $u = our $b = "x" x $l;
1207 substr($u,0,1) = "\x{100}";
1208 timethese(-2,{
1209 LENGTH_B => q{ length($b) },
1210 LENGTH_U => q{ length($u) },
1211 SUBSTR_B => q{ substr($b, $l/4, $l/2) },
1212 SUBSTR_U => q{ substr($u, $l/4, $l/2) },
1213 });
1214 '
1215 Benchmark: running LENGTH_B, LENGTH_U, SUBSTR_B, SUBSTR_U for at least 2 CPU seconds...
1216 LENGTH_B: 2 wallclock secs ( 2.36 usr + 0.00 sys = 2.36 CPU) @ 5649983.05/s (n=13333960)
1217 LENGTH_U: 2 wallclock secs ( 2.11 usr + 0.00 sys = 2.11 CPU) @ 12155.45/s (n=25648)
1218 SUBSTR_B: 3 wallclock secs ( 2.16 usr + 0.00 sys = 2.16 CPU) @ 374480.09/s (n=808877)
1219 SUBSTR_U: 2 wallclock secs ( 2.11 usr + 0.00 sys = 2.11 CPU) @ 6791.00/s (n=14329)
666f95b9 1220
376d9008
JB
1221The numbers show an incredible slowness on long UTF-8 strings. You
1222should carefully avoid using these functions in tight loops. If you
1223want to iterate over characters, the superior coding technique would
1224split the characters into an array instead of using substr, as the following
c29a771d
JH
1225benchmark shows:
1226
1227 % perl -e '
1228 use Benchmark;
1229 use strict;
1230 our $l = 10000;
1231 our $u = our $b = "x" x $l;
1232 substr($u,0,1) = "\x{100}";
1233 timethese(-5,{
1234 SPLIT_B => q{ for my $c (split //, $b){} },
1235 SPLIT_U => q{ for my $c (split //, $u){} },
1236 SUBSTR_B => q{ for my $i (0..length($b)-1){my $c = substr($b,$i,1);} },
1237 SUBSTR_U => q{ for my $i (0..length($u)-1){my $c = substr($u,$i,1);} },
1238 });
1239 '
1240 Benchmark: running SPLIT_B, SPLIT_U, SUBSTR_B, SUBSTR_U for at least 5 CPU seconds...
1241 SPLIT_B: 6 wallclock secs ( 5.29 usr + 0.00 sys = 5.29 CPU) @ 56.14/s (n=297)
1242 SPLIT_U: 5 wallclock secs ( 5.17 usr + 0.01 sys = 5.18 CPU) @ 55.21/s (n=286)
1243 SUBSTR_B: 5 wallclock secs ( 5.34 usr + 0.00 sys = 5.34 CPU) @ 123.22/s (n=658)
1244 SUBSTR_U: 7 wallclock secs ( 6.20 usr + 0.00 sys = 6.20 CPU) @ 0.81/s (n=5)
1245
376d9008
JB
1246Even though the algorithm based on C<substr()> is faster than
1247C<split()> for byte-encoded data, it pales in comparison to the speed
1248of C<split()> when used with UTF-8 data.
666f95b9 1249
393fec97
GS
1250=head1 SEE ALSO
1251
72ff2908
JH
1252L<perluniintro>, L<encoding>, L<Encode>, L<open>, L<utf8>, L<bytes>,
1253L<perlretut>, L<perlvar/"${^WIDE_SYSTEM_CALLS}">
393fec97
GS
1254
1255=cut