This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Phrasing fixes by Nicholas Clark
[perl5.git] / pod / perlunicode.pod
CommitLineData
393fec97
GS
1=head1 NAME
2
3perlunicode - Unicode support in Perl
4
5=head1 DESCRIPTION
6
0a1f2d14 7=head2 Important Caveats
21bad921 8
c349b1b9
JH
9Unicode support is an extensive requirement. While perl does not
10implement the Unicode standard or the accompanying technical reports
11from cover to cover, Perl does support many Unicode features.
21bad921 12
13a2d996 13=over 4
21bad921
GS
14
15=item Input and Output Disciplines
16
75daf61c
JH
17A filehandle can be marked as containing perl's internal Unicode
18encoding (UTF-8 or UTF-EBCDIC) by opening it with the ":utf8" layer.
0a1f2d14 19Other encodings can be converted to perl's encoding on input, or from
c349b1b9
JH
20perl's encoding on output by use of the ":encoding(...)" layer.
21See L<open>.
22
d1be9408 23To mark the Perl source itself as being in a particular encoding,
c349b1b9 24see L<encoding>.
21bad921
GS
25
26=item Regular Expressions
27
c349b1b9
JH
28The regular expression compiler produces polymorphic opcodes. That is,
29the pattern adapts to the data and automatically switch to the Unicode
30character scheme when presented with Unicode data, or a traditional
31byte scheme when presented with byte data.
21bad921 32
ad0029c4 33=item C<use utf8> still needed to enable UTF-8/UTF-EBCDIC in scripts
21bad921 34
c349b1b9
JH
35As a compatibility measure, this pragma must be explicitly used to
36enable recognition of UTF-8 in the Perl scripts themselves on ASCII
3e4dbfed 37based machines, or to recognize UTF-EBCDIC on EBCDIC based machines.
c349b1b9
JH
38B<NOTE: this should be the only place where an explicit C<use utf8>
39is needed>.
21bad921 40
1768d7eb 41You can also use the C<encoding> pragma to change the default encoding
6ec9efec 42of the data in your script; see L<encoding>.
1768d7eb 43
21bad921
GS
44=back
45
46=head2 Byte and Character semantics
393fec97
GS
47
48Beginning with version 5.6, Perl uses logically wide characters to
3e4dbfed 49represent strings internally.
393fec97 50
75daf61c
JH
51In future, Perl-level operations can be expected to work with
52characters rather than bytes, in general.
393fec97 53
75daf61c
JH
54However, as strictly an interim compatibility measure, Perl aims to
55provide a safe migration path from byte semantics to character
56semantics for programs. For operations where Perl can unambiguously
57decide that the input data is characters, Perl now switches to
58character semantics. For operations where this determination cannot
59be made without additional information from the user, Perl decides in
60favor of compatibility, and chooses to use byte semantics.
8cbd9a7a
GS
61
62This behavior preserves compatibility with earlier versions of Perl,
63which allowed byte semantics in Perl operations, but only as long as
64none of the program's inputs are marked as being as source of Unicode
65character data. Such data may come from filehandles, from calls to
66external programs, from information provided by the system (such as %ENV),
21bad921 67or from literals and constants in the source text.
8cbd9a7a 68
c349b1b9 69On Windows platforms, if the C<-C> command line switch is used, (or the
75daf61c
JH
70${^WIDE_SYSTEM_CALLS} global flag is set to C<1>), all system calls
71will use the corresponding wide character APIs. Note that this is
c349b1b9
JH
72currently only implemented on Windows since other platforms lack an
73API standard on this area.
8cbd9a7a 74
75daf61c
JH
75Regardless of the above, the C<bytes> pragma can always be used to
76force byte semantics in a particular lexical scope. See L<bytes>.
8cbd9a7a
GS
77
78The C<utf8> pragma is primarily a compatibility device that enables
75daf61c 79recognition of UTF-(8|EBCDIC) in literals encountered by the parser.
7dedd01f
JH
80Note that this pragma is only required until a future version of Perl
81in which character semantics will become the default. This pragma may
82then become a no-op. See L<utf8>.
8cbd9a7a
GS
83
84Unless mentioned otherwise, Perl operators will use character semantics
85when they are dealing with Unicode data, and byte semantics otherwise.
86Thus, character semantics for these operations apply transparently; if
87the input data came from a Unicode source (for example, by adding a
88character encoding discipline to the filehandle whence it came, or a
3e4dbfed 89literal Unicode string constant in the program), character semantics
8cbd9a7a 90apply; otherwise, byte semantics are in effect. To force byte semantics
8058d7ab 91on Unicode data, the C<bytes> pragma should be used.
393fec97 92
0a378802
JH
93Notice that if you concatenate strings with byte semantics and strings
94with Unicode character data, the bytes will by default be upgraded
95I<as if they were ISO 8859-1 (Latin-1)> (or if in EBCDIC, after a
3e4dbfed
JF
96translation to ISO 8859-1). This is done without regard to the
97system's native 8-bit encoding, so to change this for systems with
98non-Latin-1 (or non-EBCDIC) native encodings, use the C<encoding>
0a378802 99pragma, see L<encoding>.
7dedd01f 100
feda178f
JH
101Under character semantics, many operations that formerly operated on
102bytes change to operating on characters. A character in Perl is
103logically just a number ranging from 0 to 2**31 or so. Larger
104characters may encode to longer sequences of bytes internally, but
105this is just an internal detail which is hidden at the Perl level.
106See L<perluniintro> for more on this.
393fec97 107
8cbd9a7a 108=head2 Effects of character semantics
393fec97
GS
109
110Character semantics have the following effects:
111
112=over 4
113
114=item *
115
574c8022
JH
116Strings (including hash keys) and regular expression patterns may
117contain characters that have an ordinal value larger than 255.
393fec97 118
feda178f
JH
119If you use a Unicode editor to edit your program, Unicode characters
120may occur directly within the literal strings in one of the various
121Unicode encodings (UTF-8, UTF-EBCDIC, UCS-2, etc.), but are recognized
122as such (and converted to Perl's internal representation) only if the
123appropriate L<encoding> is specified.
3e4dbfed
JF
124
125You can also get Unicode characters into a string by using the C<\x{...}>
126notation, putting the Unicode code for the desired character, in
127hexadecimal, into the curlies. For instance, a smiley face is C<\x{263A}>.
128This works only for characters with a code 0x100 and above.
129
130Additionally, if you
574c8022 131
3e4dbfed 132 use charnames ':full';
574c8022 133
3e4dbfed
JF
134you can use the C<\N{...}> notation, putting the official Unicode character
135name within the curlies. For example, C<\N{WHITE SMILING FACE}>.
136This works for all characters that have names.
393fec97
GS
137
138=item *
139
574c8022
JH
140If an appropriate L<encoding> is specified, identifiers within the
141Perl script may contain Unicode alphanumeric characters, including
142ideographs. (You are currently on your own when it comes to using the
143canonical forms of characters--Perl doesn't (yet) attempt to
144canonicalize variable names for you.)
393fec97 145
393fec97
GS
146=item *
147
148Regular expressions match characters instead of bytes. For instance,
149"." matches a character instead of a byte. (However, the C<\C> pattern
75daf61c 150is provided to force a match a single byte ("C<char>" in C, hence C<\C>).)
393fec97 151
393fec97
GS
152=item *
153
154Character classes in regular expressions match characters instead of
155bytes, and match against the character properties specified in the
75daf61c
JH
156Unicode properties database. So C<\w> can be used to match an
157ideograph, for instance.
393fec97 158
393fec97
GS
159=item *
160
eb0cc9e3
JH
161Named Unicode properties, scripts, and block ranges may be used like
162character classes via the new C<\p{}> (matches property) and C<\P{}>
163(doesn't match property) constructs. For instance, C<\p{Lu}> matches any
feda178f
JH
164character with the Unicode "Lu" (Letter, uppercase) property, while
165C<\p{M}> matches any character with a "M" (mark -- accents and such)
eb0cc9e3
JH
166property. Single letter properties may omit the brackets, so that can be
167written C<\pM> also. Many predefined properties are available, such
168as C<\p{Mirrored}> and C<\p{Tibetan}>.
4193bef7 169
cfc01aea 170The official Unicode script and block names have spaces and dashes as
eb0cc9e3
JH
171separators, but for convenience you can have dashes, spaces, and underbars
172at every word division, and you need not care about correct casing. It is
173recommended, however, that for consistency you use the following naming:
174the official Unicode script, block, or property name (see below for the
175additional rules that apply to block names), with whitespace and dashes
176removed, and the words "uppercase-first-lowercase-rest". That is, "Latin-1
177Supplement" becomes "Latin1Supplement".
4193bef7 178
a1cc1cb1 179You can also negate both C<\p{}> and C<\P{}> by introducing a caret
eb0cc9e3
JH
180(^) between the first curly and the property name: C<\p{^Tamil}> is
181equal to C<\P{Tamil}>.
4193bef7 182
eb0cc9e3
JH
183Here are the basic Unicode General Category properties, followed by their
184long form (you can use either, e.g. C<\p{Lu}> and C<\p{LowercaseLetter}>
185are identical).
393fec97 186
d73e5302
JH
187 Short Long
188
189 L Letter
eb0cc9e3
JH
190 Lu UppercaseLetter
191 Ll LowercaseLetter
192 Lt TitlecaseLetter
193 Lm ModifierLetter
194 Lo OtherLetter
d73e5302
JH
195
196 M Mark
eb0cc9e3
JH
197 Mn NonspacingMark
198 Mc SpacingMark
199 Me EnclosingMark
d73e5302
JH
200
201 N Number
eb0cc9e3
JH
202 Nd DecimalNumber
203 Nl LetterNumber
204 No OtherNumber
d73e5302
JH
205
206 P Punctuation
eb0cc9e3
JH
207 Pc ConnectorPunctuation
208 Pd DashPunctuation
209 Ps OpenPunctuation
210 Pe ClosePunctuation
211 Pi InitialPunctuation
d73e5302 212 (may behave like Ps or Pe depending on usage)
eb0cc9e3 213 Pf FinalPunctuation
d73e5302 214 (may behave like Ps or Pe depending on usage)
eb0cc9e3 215 Po OtherPunctuation
d73e5302
JH
216
217 S Symbol
eb0cc9e3
JH
218 Sm MathSymbol
219 Sc CurrencySymbol
220 Sk ModifierSymbol
221 So OtherSymbol
d73e5302
JH
222
223 Z Separator
eb0cc9e3
JH
224 Zs SpaceSeparator
225 Zl LineSeparator
226 Zp ParagraphSeparator
d73e5302
JH
227
228 C Other
e150c829
JH
229 Cc Control
230 Cf Format
eb0cc9e3
JH
231 Cs Surrogate (not usable)
232 Co PrivateUse
e150c829 233 Cn Unassigned
1ac13f9a 234
3e4dbfed
JF
235The single-letter properties match all characters in any of the
236two-letter sub-properties starting with the same letter.
1ac13f9a 237There's also C<L&> which is an alias for C<Ll>, C<Lu>, and C<Lt>.
32293815 238
eb0cc9e3
JH
239Because Perl hides the need for the user to understand the internal
240representation of Unicode characters, it has no need to support the
241somewhat messy concept of surrogates. Therefore, the C<Cs> property is not
242supported.
d73e5302 243
eb0cc9e3
JH
244Because scripts differ in their directionality (for example Hebrew is
245written right to left), Unicode supplies these properties:
32293815 246
eb0cc9e3 247 Property Meaning
92e830a9 248
d73e5302
JH
249 BidiL Left-to-Right
250 BidiLRE Left-to-Right Embedding
251 BidiLRO Left-to-Right Override
252 BidiR Right-to-Left
253 BidiAL Right-to-Left Arabic
254 BidiRLE Right-to-Left Embedding
255 BidiRLO Right-to-Left Override
256 BidiPDF Pop Directional Format
257 BidiEN European Number
258 BidiES European Number Separator
259 BidiET European Number Terminator
260 BidiAN Arabic Number
261 BidiCS Common Number Separator
262 BidiNSM Non-Spacing Mark
263 BidiBN Boundary Neutral
264 BidiB Paragraph Separator
265 BidiS Segment Separator
266 BidiWS Whitespace
267 BidiON Other Neutrals
32293815 268
eb0cc9e3
JH
269For example, C<\p{BidiR}> matches all characters that are normally
270written right to left.
271
210b36aa
AMS
272=back
273
2796c109
JH
274=head2 Scripts
275
eb0cc9e3 276The scripts available via C<\p{...}> and C<\P{...}>, for example
66b79f27 277C<\p{Latin}> or C<\p{Cyrillic}>, are as follows:
2796c109 278
1ac13f9a 279 Arabic
e9ad1727 280 Armenian
1ac13f9a 281 Bengali
e9ad1727 282 Bopomofo
1d81abf3 283 Buhid
eb0cc9e3 284 CanadianAboriginal
e9ad1727
JH
285 Cherokee
286 Cyrillic
287 Deseret
288 Devanagari
289 Ethiopic
290 Georgian
291 Gothic
292 Greek
1ac13f9a 293 Gujarati
e9ad1727
JH
294 Gurmukhi
295 Han
296 Hangul
1d81abf3 297 Hanunoo
e9ad1727
JH
298 Hebrew
299 Hiragana
300 Inherited
1ac13f9a 301 Kannada
e9ad1727
JH
302 Katakana
303 Khmer
1ac13f9a 304 Lao
e9ad1727
JH
305 Latin
306 Malayalam
307 Mongolian
1ac13f9a 308 Myanmar
1ac13f9a 309 Ogham
eb0cc9e3 310 OldItalic
e9ad1727 311 Oriya
1ac13f9a 312 Runic
e9ad1727
JH
313 Sinhala
314 Syriac
1d81abf3
JH
315 Tagalog
316 Tagbanwa
e9ad1727
JH
317 Tamil
318 Telugu
319 Thaana
320 Thai
321 Tibetan
1ac13f9a 322 Yi
1ac13f9a
JH
323
324There are also extended property classes that supplement the basic
325properties, defined by the F<PropList> Unicode database:
326
1d81abf3 327 ASCIIHexDigit
eb0cc9e3 328 BidiControl
1ac13f9a 329 Dash
1d81abf3 330 Deprecated
1ac13f9a
JH
331 Diacritic
332 Extender
1d81abf3 333 GraphemeLink
eb0cc9e3 334 HexDigit
e9ad1727
JH
335 Hyphen
336 Ideographic
1d81abf3
JH
337 IDSBinaryOperator
338 IDSTrinaryOperator
eb0cc9e3 339 JoinControl
1d81abf3 340 LogicalOrderException
eb0cc9e3
JH
341 NoncharacterCodePoint
342 OtherAlphabetic
1d81abf3
JH
343 OtherDefaultIgnorableCodePoint
344 OtherGraphemeExtend
eb0cc9e3
JH
345 OtherLowercase
346 OtherMath
347 OtherUppercase
348 QuotationMark
1d81abf3
JH
349 Radical
350 SoftDotted
351 TerminalPunctuation
352 UnifiedIdeograph
eb0cc9e3 353 WhiteSpace
1ac13f9a
JH
354
355and further derived properties:
356
eb0cc9e3
JH
357 Alphabetic Lu + Ll + Lt + Lm + Lo + OtherAlphabetic
358 Lowercase Ll + OtherLowercase
359 Uppercase Lu + OtherUppercase
360 Math Sm + OtherMath
1ac13f9a
JH
361
362 ID_Start Lu + Ll + Lt + Lm + Lo + Nl
363 ID_Continue ID_Start + Mn + Mc + Nd + Pc
364
365 Any Any character
66b79f27
RGS
366 Assigned Any non-Cn character (i.e. synonym for \P{Cn})
367 Unassigned Synonym for \p{Cn}
1ac13f9a 368 Common Any character (or unassigned code point)
e150c829 369 not explicitly assigned to a script
2796c109 370
7eabb34d 371For backward compatibility, all properties mentioned so far may have C<Is>
eb0cc9e3
JH
372prepended to their name (e.g. C<\P{IsLu}> is equal to C<\P{Lu}>).
373
2796c109
JH
374=head2 Blocks
375
eb0cc9e3
JH
376In addition to B<scripts>, Unicode also defines B<blocks> of characters.
377The difference between scripts and blocks is that the scripts concept is
378closer to natural languages, while the blocks concept is more an artificial
379grouping based on groups of mostly 256 Unicode characters. For example, the
380C<Latin> script contains letters from many blocks. On the other hand, the
381C<Latin> script does not contain all the characters from those blocks. It
382does not, for example, contain digits because digits are shared across many
383scripts. Digits and other similar groups, like punctuation, are in a
384category called C<Common>.
2796c109 385
cfc01aea
JF
386For more about scripts, see the UTR #24:
387
388 http://www.unicode.org/unicode/reports/tr24/
389
390For more about blocks, see:
391
392 http://www.unicode.org/Public/UNIDATA/Blocks.txt
2796c109 393
eb0cc9e3 394Blocks names are given with the C<In> prefix. For example, the
92e830a9 395Katakana block is referenced via C<\p{InKatakana}>. The C<In>
7eabb34d 396prefix may be omitted if there is no naming conflict with a script
eb0cc9e3
JH
397or any other property, but it is recommended that C<In> always be used
398to avoid confusion.
399
400These block names are supported:
401
1d81abf3
JH
402 InAlphabeticPresentationForms
403 InArabic
404 InArabicPresentationFormsA
405 InArabicPresentationFormsB
406 InArmenian
407 InArrows
408 InBasicLatin
409 InBengali
410 InBlockElements
411 InBopomofo
412 InBopomofoExtended
413 InBoxDrawing
414 InBraillePatterns
415 InBuhid
416 InByzantineMusicalSymbols
417 InCJKCompatibility
418 InCJKCompatibilityForms
419 InCJKCompatibilityIdeographs
420 InCJKCompatibilityIdeographsSupplement
421 InCJKRadicalsSupplement
422 InCJKSymbolsAndPunctuation
423 InCJKUnifiedIdeographs
424 InCJKUnifiedIdeographsExtensionA
425 InCJKUnifiedIdeographsExtensionB
426 InCherokee
427 InCombiningDiacriticalMarks
428 InCombiningDiacriticalMarksforSymbols
429 InCombiningHalfMarks
430 InControlPictures
431 InCurrencySymbols
432 InCyrillic
433 InCyrillicSupplementary
434 InDeseret
435 InDevanagari
436 InDingbats
437 InEnclosedAlphanumerics
438 InEnclosedCJKLettersAndMonths
439 InEthiopic
440 InGeneralPunctuation
441 InGeometricShapes
442 InGeorgian
443 InGothic
444 InGreekExtended
445 InGreekAndCoptic
446 InGujarati
447 InGurmukhi
448 InHalfwidthAndFullwidthForms
449 InHangulCompatibilityJamo
450 InHangulJamo
451 InHangulSyllables
452 InHanunoo
453 InHebrew
454 InHighPrivateUseSurrogates
455 InHighSurrogates
456 InHiragana
457 InIPAExtensions
458 InIdeographicDescriptionCharacters
459 InKanbun
460 InKangxiRadicals
461 InKannada
462 InKatakana
463 InKatakanaPhoneticExtensions
464 InKhmer
465 InLao
466 InLatin1Supplement
467 InLatinExtendedA
468 InLatinExtendedAdditional
469 InLatinExtendedB
470 InLetterlikeSymbols
471 InLowSurrogates
472 InMalayalam
473 InMathematicalAlphanumericSymbols
474 InMathematicalOperators
475 InMiscellaneousMathematicalSymbolsA
476 InMiscellaneousMathematicalSymbolsB
477 InMiscellaneousSymbols
478 InMiscellaneousTechnical
479 InMongolian
480 InMusicalSymbols
481 InMyanmar
482 InNumberForms
483 InOgham
484 InOldItalic
485 InOpticalCharacterRecognition
486 InOriya
487 InPrivateUseArea
488 InRunic
489 InSinhala
490 InSmallFormVariants
491 InSpacingModifierLetters
492 InSpecials
493 InSuperscriptsAndSubscripts
494 InSupplementalArrowsA
495 InSupplementalArrowsB
496 InSupplementalMathematicalOperators
497 InSupplementaryPrivateUseAreaA
498 InSupplementaryPrivateUseAreaB
499 InSyriac
500 InTagalog
501 InTagbanwa
502 InTags
503 InTamil
504 InTelugu
505 InThaana
506 InThai
507 InTibetan
508 InUnifiedCanadianAboriginalSyllabics
509 InVariationSelectors
510 InYiRadicals
511 InYiSyllables
32293815 512
210b36aa
AMS
513=over 4
514
393fec97
GS
515=item *
516
c29a771d 517The special pattern C<\X> matches any extended Unicode sequence
393fec97
GS
518(a "combining character sequence" in Standardese), where the first
519character is a base character and subsequent characters are mark
520characters that apply to the base character. It is equivalent to
521C<(?:\PM\pM*)>.
522
393fec97
GS
523=item *
524
383e7cdd
JH
525The C<tr///> operator translates characters instead of bytes. Note
526that the C<tr///CU> functionality has been removed, as the interface
527was a mistake. For similar functionality see pack('U0', ...) and
528pack('C0', ...).
393fec97 529
393fec97
GS
530=item *
531
532Case translation operators use the Unicode case translation tables
44bc797b
JH
533when provided character input. Note that C<uc()> (also known as C<\U>
534in doublequoted strings) translates to uppercase, while C<ucfirst>
535(also known as C<\u> in doublequoted strings) translates to titlecase
536(for languages that make the distinction). Naturally the
537corresponding backslash sequences have the same semantics.
393fec97
GS
538
539=item *
540
541Most operators that deal with positions or lengths in the string will
75daf61c
JH
542automatically switch to using character positions, including
543C<chop()>, C<substr()>, C<pos()>, C<index()>, C<rindex()>,
544C<sprintf()>, C<write()>, and C<length()>. Operators that
545specifically don't switch include C<vec()>, C<pack()>, and
546C<unpack()>. Operators that really don't care include C<chomp()>, as
547well as any other operator that treats a string as a bucket of bits,
548such as C<sort()>, and the operators dealing with filenames.
393fec97
GS
549
550=item *
551
552The C<pack()>/C<unpack()> letters "C<c>" and "C<C>" do I<not> change,
553since they're often used for byte-oriented formats. (Again, think
554"C<char>" in the C language.) However, there is a new "C<U>" specifier
3e4dbfed 555that will convert between Unicode characters and integers.
393fec97
GS
556
557=item *
558
559The C<chr()> and C<ord()> functions work on characters. This is like
560C<pack("U")> and C<unpack("U")>, not like C<pack("C")> and
561C<unpack("C")>. In fact, the latter are how you now emulate
35bcd338 562byte-oriented C<chr()> and C<ord()> for Unicode strings.
3e4dbfed
JF
563(Note that this reveals the internal encoding of Unicode strings,
564which is not something one normally needs to care about at all.)
393fec97
GS
565
566=item *
567
a1ca4561
YST
568The bit string operators C<& | ^ ~> can operate on character data.
569However, for backward compatibility reasons (bit string operations
75daf61c
JH
570when the characters all are less than 256 in ordinal value) one should
571not mix C<~> (the bit complement) and characters both less than 256 and
a1ca4561
YST
572equal or greater than 256. Most importantly, the DeMorgan's laws
573(C<~($x|$y) eq ~$x&~$y>, C<~($x&$y) eq ~$x|~$y>) won't hold.
574Another way to look at this is that the complement cannot return
75daf61c 575B<both> the 8-bit (byte) wide bit complement B<and> the full character
a1ca4561
YST
576wide bit complement.
577
578=item *
579
983ffd37
JH
580lc(), uc(), lcfirst(), and ucfirst() work for the following cases:
581
582=over 8
583
584=item *
585
586the case mapping is from a single Unicode character to another
587single Unicode character
588
589=item *
590
591the case mapping is from a single Unicode character to more
592than one Unicode character
593
594=back
595
210b36aa 596What doesn't yet work are the following cases:
983ffd37
JH
597
598=over 8
599
600=item *
601
602the "final sigma" (Greek)
603
604=item *
605
606anything to with locales (Lithuanian, Turkish, Azeri)
607
608=back
609
610See the Unicode Technical Report #21, Case Mappings, for more details.
ac1256e8
JH
611
612=item *
613
393fec97
GS
614And finally, C<scalar reverse()> reverses by character rather than by byte.
615
616=back
617
8cbd9a7a
GS
618=head2 Character encodings for input and output
619
7221edc9 620See L<Encode>.
8cbd9a7a 621
c29a771d 622=head2 Unicode Regular Expression Support Level
776f8809
JH
623
624The following list of Unicode regular expression support describes
625feature by feature the Unicode support implemented in Perl as of Perl
6265.8.0. The "Level N" and the section numbers refer to the Unicode
627Technical Report 18, "Unicode Regular Expression Guidelines".
628
629=over 4
630
631=item *
632
633Level 1 - Basic Unicode Support
634
635 2.1 Hex Notation - done [1]
3bfdc84c 636 Named Notation - done [2]
776f8809
JH
637 2.2 Categories - done [3][4]
638 2.3 Subtraction - MISSING [5][6]
639 2.4 Simple Word Boundaries - done [7]
78d3e1bf 640 2.5 Simple Loose Matches - done [8]
776f8809
JH
641 2.6 End of Line - MISSING [9][10]
642
643 [ 1] \x{...}
644 [ 2] \N{...}
eb0cc9e3 645 [ 3] . \p{...} \P{...}
29bdacb8 646 [ 4] now scripts (see UTR#24 Script Names) in addition to blocks
776f8809 647 [ 5] have negation
29bdacb8 648 [ 6] can use look-ahead to emulate subtraction (*)
776f8809 649 [ 7] include Letters in word characters
e0f9d4a8
JH
650 [ 8] note that perl does Full casefolding in matching, not Simple:
651 for example U+1F88 is equivalent with U+1F000 U+03B9,
652 not with 1F80. This difference matters for certain Greek
653 capital letters with certain modifiers: the Full casefolding
654 decomposes the letter, while the Simple casefolding would map
655 it to a single character.
776f8809 656 [ 9] see UTR#13 Unicode Newline Guidelines
ec83e909
JH
657 [10] should do ^ and $ also on \x{85}, \x{2028} and \x{2029})
658 (should also affect <>, $., and script line numbers)
3bfdc84c 659 (the \x{85}, \x{2028} and \x{2029} do match \s)
7207e29d 660
dbe420b4
JH
661(*) You can mimic class subtraction using lookahead.
662For example, what TR18 might write as
29bdacb8 663
dbe420b4
JH
664 [{Greek}-[{UNASSIGNED}]]
665
666in Perl can be written as:
667
1d81abf3
JH
668 (?!\p{Unassigned})\p{InGreekAndCoptic}
669 (?=\p{Assigned})\p{InGreekAndCoptic}
dbe420b4
JH
670
671But in this particular example, you probably really want
672
673 \p{Greek}
674
675which will match assigned characters known to be part of the Greek script.
29bdacb8 676
776f8809
JH
677=item *
678
679Level 2 - Extended Unicode Support
680
681 3.1 Surrogates - MISSING
682 3.2 Canonical Equivalents - MISSING [11][12]
683 3.3 Locale-Independent Graphemes - MISSING [13]
684 3.4 Locale-Independent Words - MISSING [14]
685 3.5 Locale-Independent Loose Matches - MISSING [15]
686
687 [11] see UTR#15 Unicode Normalization
688 [12] have Unicode::Normalize but not integrated to regexes
689 [13] have \X but at this level . should equal that
690 [14] need three classes, not just \w and \W
691 [15] see UTR#21 Case Mappings
692
693=item *
694
695Level 3 - Locale-Sensitive Support
696
697 4.1 Locale-Dependent Categories - MISSING
698 4.2 Locale-Dependent Graphemes - MISSING [16][17]
699 4.3 Locale-Dependent Words - MISSING
700 4.4 Locale-Dependent Loose Matches - MISSING
701 4.5 Locale-Dependent Ranges - MISSING
702
703 [16] see UTR#10 Unicode Collation Algorithms
704 [17] have Unicode::Collate but not integrated to regexes
705
706=back
707
c349b1b9
JH
708=head2 Unicode Encodings
709
710Unicode characters are assigned to I<code points> which are abstract
86bbd6d1 711numbers. To use these numbers various encodings are needed.
c349b1b9
JH
712
713=over 4
714
c29a771d 715=item *
5cb3728c
RB
716
717UTF-8
c349b1b9 718
3e4dbfed
JF
719UTF-8 is a variable-length (1 to 6 bytes, current character allocations
720require 4 bytes), byteorder independent encoding. For ASCII, UTF-8 is
721transparent (and we really do mean 7-bit ASCII, not another 8-bit encoding).
c349b1b9 722
8c007b5a 723The following table is from Unicode 3.2.
05632f9a
JH
724
725 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
726
8c007b5a
JH
727 U+0000..U+007F 00..7F
728 U+0080..U+07FF C2..DF 80..BF
05632f9a 729 U+0800..U+0FFF E0 A0..BF 80..BF  
8c007b5a
JH
730 U+1000..U+CFFF E1..EC 80..BF 80..BF  
731 U+D000..U+D7FF ED 80..9F 80..BF  
732 U+D800..U+DFFF ******* ill-formed *******
733 U+E000..U+FFFF EE..EF 80..BF 80..BF  
05632f9a
JH
734 U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
735 U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
736 U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
737
8c007b5a
JH
738Note the A0..BF in U+0800..U+0FFF, the 80..9F in U+D000...U+D7FF,
739the 90..BF in U+10000..U+3FFFF, and the 80...8F in U+100000..U+10FFFF.
37361303
JH
740The "gaps" are caused by legal UTF-8 avoiding non-shortest encodings:
741it is technically possible to UTF-8-encode a single code point in different
742ways, but that is explicitly forbidden, and the shortest possible encoding
743should always be used (and that is what Perl does).
744
05632f9a
JH
745Or, another way to look at it, as bits:
746
747 Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
748
749 0aaaaaaa 0aaaaaaa
750 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
751 ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
752 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
753
754As you can see, the continuation bytes all begin with C<10>, and the
8c007b5a 755leading bits of the start byte tell how many bytes the are in the
05632f9a
JH
756encoded character.
757
c29a771d 758=item *
5cb3728c
RB
759
760UTF-EBCDIC
dbe420b4 761
fe854a6f 762Like UTF-8, but EBCDIC-safe, as UTF-8 is ASCII-safe.
dbe420b4 763
c29a771d 764=item *
5cb3728c
RB
765
766UTF-16, UTF-16BE, UTF16-LE, Surrogates, and BOMs (Byte Order Marks)
c349b1b9 767
dbe420b4
JH
768(The followings items are mostly for reference, Perl doesn't
769use them internally.)
770
c349b1b9
JH
771UTF-16 is a 2 or 4 byte encoding. The Unicode code points
7720x0000..0xFFFF are stored in two 16-bit units, and the code points
dbe420b4 7730x010000..0x10FFFF in two 16-bit units. The latter case is
c349b1b9
JH
774using I<surrogates>, the first 16-bit unit being the I<high
775surrogate>, and the second being the I<low surrogate>.
776
777Surrogates are code points set aside to encode the 0x01000..0x10FFFF
778range of Unicode code points in pairs of 16-bit units. The I<high
779surrogates> are the range 0xD800..0xDBFF, and the I<low surrogates>
780are the range 0xDC00..0xDFFFF. The surrogate encoding is
781
782 $hi = ($uni - 0x10000) / 0x400 + 0xD800;
783 $lo = ($uni - 0x10000) % 0x400 + 0xDC00;
784
785and the decoding is
786
1a3fa709 787 $uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00);
c349b1b9 788
feda178f
JH
789If you try to generate surrogates (for example by using chr()), you
790will get a warning if warnings are turned on (C<-w> or C<use
791warnings;>) because those code points are not valid for a Unicode
792character.
9466bab6 793
86bbd6d1 794Because of the 16-bitness, UTF-16 is byteorder dependent. UTF-16
c349b1b9 795itself can be used for in-memory computations, but if storage or
86bbd6d1 796transfer is required, either UTF-16BE (Big Endian) or UTF-16LE
c349b1b9
JH
797(Little Endian) must be chosen.
798
799This introduces another problem: what if you just know that your data
800is UTF-16, but you don't know which endianness? Byte Order Marks
801(BOMs) are a solution to this. A special character has been reserved
86bbd6d1
PN
802in Unicode to function as a byte order marker: the character with the
803code point 0xFEFF is the BOM.
042da322 804
c349b1b9
JH
805The trick is that if you read a BOM, you will know the byte order,
806since if it was written on a big endian platform, you will read the
86bbd6d1
PN
807bytes 0xFE 0xFF, but if it was written on a little endian platform,
808you will read the bytes 0xFF 0xFE. (And if the originating platform
809was writing in UTF-8, you will read the bytes 0xEF 0xBB 0xBF.)
042da322 810
86bbd6d1
PN
811The way this trick works is that the character with the code point
8120xFFFE is guaranteed not to be a valid Unicode character, so the
813sequence of bytes 0xFF 0xFE is unambiguously "BOM, represented in
042da322
JH
814little-endian format" and cannot be "0xFFFE, represented in big-endian
815format".
c349b1b9 816
c29a771d 817=item *
5cb3728c
RB
818
819UTF-32, UTF-32BE, UTF32-LE
c349b1b9
JH
820
821The UTF-32 family is pretty much like the UTF-16 family, expect that
042da322
JH
822the units are 32-bit, and therefore the surrogate scheme is not
823needed. The BOM signatures will be 0x00 0x00 0xFE 0xFF for BE and
8240xFF 0xFE 0x00 0x00 for LE.
c349b1b9 825
c29a771d 826=item *
5cb3728c
RB
827
828UCS-2, UCS-4
c349b1b9 829
86bbd6d1
PN
830Encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit
831encoding, UCS-4 is a 32-bit encoding. Unlike UTF-16, UCS-2
832is not extensible beyond 0xFFFF, because it does not use surrogates.
c349b1b9 833
c29a771d 834=item *
5cb3728c
RB
835
836UTF-7
c349b1b9
JH
837
838A seven-bit safe (non-eight-bit) encoding, useful if the
839transport/storage is not eight-bit safe. Defined by RFC 2152.
840
95a1a48b
JH
841=back
842
0d7c09bb
JH
843=head2 Security Implications of Unicode
844
845=over 4
846
847=item *
848
849Malformed UTF-8
bf0fa0b2
JH
850
851Unfortunately, the specification of UTF-8 leaves some room for
852interpretation of how many bytes of encoded output one should generate
853from one input Unicode character. Strictly speaking, one is supposed
854to always generate the shortest possible sequence of UTF-8 bytes,
feda178f
JH
855because otherwise there is potential for input buffer overflow at
856the receiving end of a UTF-8 connection. Perl always generates the
857shortest length UTF-8, and with warnings on (C<-w> or C<use
858warnings;>) Perl will warn about non-shortest length UTF-8 (and other
859malformations, too, such as the surrogates, which are not real
860Unicode code points.)
bf0fa0b2 861
0d7c09bb
JH
862=item *
863
864Regular expressions behave slightly differently between byte data and
865character (Unicode data). For example, the "word character" character
866class C<\w> will work differently when the data is all eight-bit bytes
867or when the data is Unicode.
868
869In the first case, the set of C<\w> characters is either small (the
870default set of alphabetic characters, digits, and the "_"), or, if you
871are using a locale (see L<perllocale>), the C<\w> might contain a few
872more letters according to your language and country.
873
874In the second case, the C<\w> set of characters is much, much larger,
875and most importantly, even in the set of the first 256 characters, it
876will most probably be different: as opposed to most locales (which are
877specific to a language and country pair) Unicode classifies all the
878characters that are letters as C<\w>. For example: your locale might
879not think that LATIN SMALL LETTER ETH is a letter (unless you happen
880to speak Icelandic), but Unicode does.
881
a73d23f6
RGS
882As discussed elsewhere, Perl tries to stand one leg (two legs, as
883camels are quadrupeds?) in two worlds: the old world of byte and the new
0d7c09bb
JH
884world of characters, upgrading from bytes to characters when necessary.
885If your legacy code is not explicitly using Unicode, no automatic
886switchover to characters should happen, and characters shouldn't get
887downgraded back to bytes, either. It is possible to accidentally mix
888bytes and characters, however (see L<perluniintro>), in which case the
889C<\w> might start behaving differently. Review your code.
890
891=back
892
c349b1b9
JH
893=head2 Unicode in Perl on EBCDIC
894
895The way Unicode is handled on EBCDIC platforms is still rather
86bbd6d1 896experimental. On such a platform, references to UTF-8 encoding in this
c349b1b9
JH
897document and elsewhere should be read as meaning UTF-EBCDIC as
898specified in Unicode Technical Report 16 unless ASCII vs EBCDIC issues
899are specifically discussed. There is no C<utfebcdic> pragma or
86bbd6d1
PN
900":utfebcdic" layer, rather, "utf8" and ":utf8" are re-used to mean
901the platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic>
902for more discussion of the issues.
c349b1b9 903
b310b053
JH
904=head2 Locales
905
4616122b 906Usually locale settings and Unicode do not affect each other, but
b310b053
JH
907there are a couple of exceptions:
908
909=over 4
910
911=item *
912
913If your locale environment variables (LANGUAGE, LC_ALL, LC_CTYPE, LANG)
914contain the strings 'UTF-8' or 'UTF8' (case-insensitive matching),
915the default encoding of your STDIN, STDOUT, and STDERR, and of
916B<any subsequent file open>, is UTF-8.
917
918=item *
919
920Perl tries really hard to work both with Unicode and the old byte
921oriented world: most often this is nice, but sometimes this causes
574c8022 922problems.
b310b053
JH
923
924=back
925
95a1a48b
JH
926=head2 Using Unicode in XS
927
928If you want to handle Perl Unicode in XS extensions, you may find
90f968e0 929the following C APIs useful (see perlapi for details):
95a1a48b
JH
930
931=over 4
932
933=item *
934
f1e62f77
AT
935DO_UTF8(sv) returns true if the UTF8 flag is on and the bytes pragma
936is not in effect. SvUTF8(sv) returns true is the UTF8 flag is on, the
937bytes pragma is ignored. The UTF8 flag being on does B<not> mean that
b31c5e31
AT
938there are any characters of code points greater than 255 (or 127) in
939the scalar, or that there even are any characters in the scalar.
940What the UTF8 flag means is that the sequence of octets in the
941representation of the scalar is the sequence of UTF-8 encoded
942code points of the characters of a string. The UTF8 flag being
943off means that each octet in this representation encodes a single
944character with codepoint 0..255 within the string. Perl's Unicode
945model is not to use UTF-8 until it's really necessary.
95a1a48b
JH
946
947=item *
948
949uvuni_to_utf8(buf, chr) writes a Unicode character code point into a
cfc01aea 950buffer encoding the code point as UTF-8, and returns a pointer
95a1a48b
JH
951pointing after the UTF-8 bytes.
952
953=item *
954
955utf8_to_uvuni(buf, lenp) reads UTF-8 encoded bytes from a buffer and
956returns the Unicode character code point (and optionally the length of
957the UTF-8 byte sequence).
958
959=item *
960
90f968e0
JH
961utf8_length(start, end) returns the length of the UTF-8 encoded buffer
962in characters. sv_len_utf8(sv) returns the length of the UTF-8 encoded
95a1a48b
JH
963scalar.
964
965=item *
966
967sv_utf8_upgrade(sv) converts the string of the scalar to its UTF-8
968encoded form. sv_utf8_downgrade(sv) does the opposite (if possible).
969sv_utf8_encode(sv) is like sv_utf8_upgrade but the UTF8 flag does not
970get turned on. sv_utf8_decode() does the opposite of sv_utf8_encode().
13a6c0e0
JH
971Note that none of these are to be used as general purpose encoding/decoding
972interfaces: use Encode for that. sv_utf8_upgrade() is affected by the
973encoding pragma, but sv_utf8_downgrade() is not (since the encoding
974pragma is designed to be a one-way street).
95a1a48b
JH
975
976=item *
977
90f968e0
JH
978is_utf8_char(s) returns true if the pointer points to a valid UTF-8
979character.
95a1a48b
JH
980
981=item *
982
983is_utf8_string(buf, len) returns true if the len bytes of the buffer
984are valid UTF-8.
985
986=item *
987
988UTF8SKIP(buf) will return the number of bytes in the UTF-8 encoded
989character in the buffer. UNISKIP(chr) will return the number of bytes
90f968e0
JH
990required to UTF-8-encode the Unicode character code point. UTF8SKIP()
991is useful for example for iterating over the characters of a UTF-8
992encoded buffer; UNISKIP() is useful for example in computing
993the size required for a UTF-8 encoded buffer.
95a1a48b
JH
994
995=item *
996
997utf8_distance(a, b) will tell the distance in characters between the
998two pointers pointing to the same UTF-8 encoded buffer.
999
1000=item *
1001
1002utf8_hop(s, off) will return a pointer to an UTF-8 encoded buffer that
1003is C<off> (positive or negative) Unicode characters displaced from the
90f968e0
JH
1004UTF-8 buffer C<s>. Be careful not to overstep the buffer: utf8_hop()
1005will merrily run off the end or the beginning if told to do so.
95a1a48b 1006
d2cc3551
JH
1007=item *
1008
1009pv_uni_display(dsv, spv, len, pvlim, flags) and sv_uni_display(dsv,
1010ssv, pvlim, flags) are useful for debug output of Unicode strings and
90f968e0
JH
1011scalars. By default they are useful only for debug: they display
1012B<all> characters as hexadecimal code points, but with the flags
1013UNI_DISPLAY_ISPRINT and UNI_DISPLAY_BACKSLASH you can make the output
1014more readable.
d2cc3551
JH
1015
1016=item *
1017
90f968e0
JH
1018ibcmp_utf8(s1, pe1, u1, l1, u1, s2, pe2, l2, u2) can be used to
1019compare two strings case-insensitively in Unicode.
1020(For case-sensitive comparisons you can just use memEQ() and memNE()
1021as usual.)
d2cc3551 1022
c349b1b9
JH
1023=back
1024
95a1a48b
JH
1025For more information, see L<perlapi>, and F<utf8.c> and F<utf8.h>
1026in the Perl source code distribution.
1027
c29a771d
JH
1028=head1 BUGS
1029
7eabb34d
A
1030=head2 Interaction with locales
1031
c29a771d
JH
1032Use of locales with Unicode data may lead to odd results. Currently
1033there is some attempt to apply 8-bit locale info to characters in the
1034range 0..255, but this is demonstrably incorrect for locales that use
1035characters above that range when mapped into Unicode. It will also
574c8022 1036tend to run slower. Use of locales with Unicode is discouraged.
c29a771d 1037
7eabb34d
A
1038=head2 Interaction with extensions
1039
1040When perl exchanges data with an extension, the extension should be
1041able to understand the UTF-8 flag and act accordingly. If the
1042extension doesn't know about the flag, the risk is high that it will
1043return data that are incorrectly flagged.
1044
1045So if you're working with Unicode data, consult the documentation of
1046every module you're using if there are any issues with Unicode data
1047exchange. If the documentation does not talk about Unicode at all,
a73d23f6
RGS
1048suspect the worst and probably look at the source to learn how the
1049module is implemented. Modules written completely in perl shouldn't
1050cause problems. Modules that directly or indirectly access code written
1051in other programming languages are at risk.
7eabb34d
A
1052
1053For affected functions the simple strategy to avoid data corruption is
1054to always make the encoding of the exchanged data explicit. Choose an
1055encoding you know the extension can handle. Convert arguments passed
1056to the extensions to that encoding and convert results back from that
1057encoding. Write wrapper functions that do the conversions for you, so
1058you can later change the functions when the extension catches up.
1059
1060To provide an example let's say the popular Foo::Bar::escape_html
1061function doesn't deal with Unicode data yet. The wrapper function
1062would convert the argument to raw UTF-8 and convert the result back to
1063perl's internal representation like so:
1064
1065 sub my_escape_html ($) {
1066 my($what) = shift;
1067 return unless defined $what;
1068 Encode::decode_utf8(Foo::Bar::escape_html(Encode::encode_utf8($what)));
1069 }
1070
1071Sometimes, when the extension does not convert data but just stores
1072and retrieves them, you will be in a position to use the otherwise
1073dangerous Encode::_utf8_on() function. Let's say the popular
66b79f27 1074C<Foo::Bar> extension, written in C, provides a C<param> method that
7eabb34d
A
1075lets you store and retrieve data according to these prototypes:
1076
1077 $self->param($name, $value); # set a scalar
1078 $value = $self->param($name); # retrieve a scalar
1079
1080If it does not yet provide support for any encoding, one could write a
1081derived class with such a C<param> method:
1082
1083 sub param {
1084 my($self,$name,$value) = @_;
1085 utf8::upgrade($name); # make sure it is UTF-8 encoded
1086 if (defined $value)
1087 utf8::upgrade($value); # make sure it is UTF-8 encoded
1088 return $self->SUPER::param($name,$value);
1089 } else {
1090 my $ret = $self->SUPER::param($name);
1091 Encode::_utf8_on($ret); # we know, it is UTF-8 encoded
1092 return $ret;
1093 }
1094 }
1095
a73d23f6
RGS
1096Some extensions provide filters on data entry/exit points, such as
1097DB_File::filter_store_key and family. Look out for such filters in
66b79f27 1098the documentation of your extensions, they can make the transition to
7eabb34d
A
1099Unicode data much easier.
1100
1101=head2 speed
1102
c29a771d 1103Some functions are slower when working on UTF-8 encoded strings than
574c8022 1104on byte encoded strings. All functions that need to hop over
c29a771d
JH
1105characters such as length(), substr() or index() can work B<much>
1106faster when the underlying data are byte-encoded. Witness the
1107following benchmark:
666f95b9 1108
c29a771d
JH
1109 % perl -e '
1110 use Benchmark;
1111 use strict;
1112 our $l = 10000;
1113 our $u = our $b = "x" x $l;
1114 substr($u,0,1) = "\x{100}";
1115 timethese(-2,{
1116 LENGTH_B => q{ length($b) },
1117 LENGTH_U => q{ length($u) },
1118 SUBSTR_B => q{ substr($b, $l/4, $l/2) },
1119 SUBSTR_U => q{ substr($u, $l/4, $l/2) },
1120 });
1121 '
1122 Benchmark: running LENGTH_B, LENGTH_U, SUBSTR_B, SUBSTR_U for at least 2 CPU seconds...
1123 LENGTH_B: 2 wallclock secs ( 2.36 usr + 0.00 sys = 2.36 CPU) @ 5649983.05/s (n=13333960)
1124 LENGTH_U: 2 wallclock secs ( 2.11 usr + 0.00 sys = 2.11 CPU) @ 12155.45/s (n=25648)
1125 SUBSTR_B: 3 wallclock secs ( 2.16 usr + 0.00 sys = 2.16 CPU) @ 374480.09/s (n=808877)
1126 SUBSTR_U: 2 wallclock secs ( 2.11 usr + 0.00 sys = 2.11 CPU) @ 6791.00/s (n=14329)
666f95b9 1127
c29a771d
JH
1128The numbers show an incredible slowness on long UTF-8 strings and you
1129should carefully avoid to use these functions within tight loops. For
1130example if you want to iterate over characters, it is infinitely
1131better to split into an array than to use substr, as the following
1132benchmark shows:
1133
1134 % perl -e '
1135 use Benchmark;
1136 use strict;
1137 our $l = 10000;
1138 our $u = our $b = "x" x $l;
1139 substr($u,0,1) = "\x{100}";
1140 timethese(-5,{
1141 SPLIT_B => q{ for my $c (split //, $b){} },
1142 SPLIT_U => q{ for my $c (split //, $u){} },
1143 SUBSTR_B => q{ for my $i (0..length($b)-1){my $c = substr($b,$i,1);} },
1144 SUBSTR_U => q{ for my $i (0..length($u)-1){my $c = substr($u,$i,1);} },
1145 });
1146 '
1147 Benchmark: running SPLIT_B, SPLIT_U, SUBSTR_B, SUBSTR_U for at least 5 CPU seconds...
1148 SPLIT_B: 6 wallclock secs ( 5.29 usr + 0.00 sys = 5.29 CPU) @ 56.14/s (n=297)
1149 SPLIT_U: 5 wallclock secs ( 5.17 usr + 0.01 sys = 5.18 CPU) @ 55.21/s (n=286)
1150 SUBSTR_B: 5 wallclock secs ( 5.34 usr + 0.00 sys = 5.34 CPU) @ 123.22/s (n=658)
1151 SUBSTR_U: 7 wallclock secs ( 6.20 usr + 0.00 sys = 6.20 CPU) @ 0.81/s (n=5)
1152
1153You see, the algorithm based on substr() was faster with byte encoded
1154data but it is pathologically slow with UTF-8 data.
666f95b9 1155
393fec97
GS
1156=head1 SEE ALSO
1157
72ff2908
JH
1158L<perluniintro>, L<encoding>, L<Encode>, L<open>, L<utf8>, L<bytes>,
1159L<perlretut>, L<perlvar/"${^WIDE_SYSTEM_CALLS}">
393fec97
GS
1160
1161=cut