=head2 Important Caveats
-WARNING: While the implementation of Unicode support in Perl is now
-fairly complete it is still evolving to some extent.
-
-In particular the way Unicode is handled on EBCDIC platforms is still
-rather experimental. On such a platform references to UTF-8 encoding
-in this document and elsewhere should be read as meaning UTF-EBCDIC as
-specified in Unicode Technical Report 16 unless ASCII vs EBCDIC issues
-are specifically discussed. There is no C<utfebcdic> pragma or
-":utfebcdic" layer, rather "utf8" and ":utf8" are re-used to mean
-platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic> for
-more discussion of the issues.
-
-The following areas are still under development.
+Unicode support is an extensive requirement. While perl does not
+implement the Unicode standard or the accompanying technical reports
+from cover to cover, Perl does support many Unicode features.
=over 4
A filehandle can be marked as containing perl's internal Unicode
encoding (UTF-8 or UTF-EBCDIC) by opening it with the ":utf8" layer.
Other encodings can be converted to perl's encoding on input, or from
-perl's encoding on output by use of the ":encoding()" layer. There is
-not yet a clean way to mark the Perl source itself as being in an
-particular encoding.
+perl's encoding on output by use of the ":encoding(...)" layer.
+See L<open>.
+
+To mark the Perl source itself as being in a particular encoding,
+see L<encoding>.
=item Regular Expressions
-The regular expression compiler does now attempt to produce
-polymorphic opcodes. That is the pattern should now adapt to the data
-and automatically switch to the Unicode character scheme when
-presented with Unicode data, or a traditional byte scheme when
-presented with byte data. The implementation is still new and
-(particularly on EBCDIC platforms) may need further work.
+The regular expression compiler produces polymorphic opcodes. That is,
+the pattern adapts to the data and automatically switch to the Unicode
+character scheme when presented with Unicode data, or a traditional
+byte scheme when presented with byte data.
=item C<use utf8> still needed to enable UTF-8/UTF-EBCDIC in scripts
-The C<utf8> pragma implements the tables used for Unicode support.
-These tables are automatically loaded on demand, so the C<utf8> pragma
-need not normally be used.
+As a compatibility measure, this pragma must be explicitly used to
+enable recognition of UTF-8 in the Perl scripts themselves on ASCII
+based machines, or to recognize UTF-EBCDIC on EBCDIC based machines.
+B<NOTE: this should be the only place where an explicit C<use utf8>
+is needed>.
-However, as a compatibility measure, this pragma must be explicitly
-used to enable recognition of UTF-8 in the Perl scripts themselves on
-ASCII based machines or recognize UTF-EBCDIC on EBCDIC based machines.
-B<NOTE: this should be the only place where an explicit C<use utf8> is
-needed>.
+You can also use the C<encoding> pragma to change the default encoding
+of the data in your script; see L<encoding>.
=back
=head2 Byte and Character semantics
Beginning with version 5.6, Perl uses logically wide characters to
-represent strings internally. This internal representation of strings
-uses either the UTF-8 or the UTF-EBCDIC encoding.
+represent strings internally.
In future, Perl-level operations can be expected to work with
characters rather than bytes, in general.
external programs, from information provided by the system (such as %ENV),
or from literals and constants in the source text.
-If the C<-C> command line switch is used, (or the
+On Windows platforms, if the C<-C> command line switch is used, (or the
${^WIDE_SYSTEM_CALLS} global flag is set to C<1>), all system calls
will use the corresponding wide character APIs. Note that this is
-currently only implemented on Windows since other platforms API
-standard on this area.
+currently only implemented on Windows since other platforms lack an
+API standard on this area.
Regardless of the above, the C<bytes> pragma can always be used to
force byte semantics in a particular lexical scope. See L<bytes>.
Thus, character semantics for these operations apply transparently; if
the input data came from a Unicode source (for example, by adding a
character encoding discipline to the filehandle whence it came, or a
-literal UTF-8 string constant in the program), character semantics
+literal Unicode string constant in the program), character semantics
apply; otherwise, byte semantics are in effect. To force byte semantics
on Unicode data, the C<bytes> pragma should be used.
-Notice that if you have a string with byte semantics and you then
-add character data into it, the bytes will be upgraded I<as if they
-were ISO 8859-1 (Latin-1)> (or if in EBCDIC, after a translation
-to ISO 8859-1).
+Notice that if you concatenate strings with byte semantics and strings
+with Unicode character data, the bytes will by default be upgraded
+I<as if they were ISO 8859-1 (Latin-1)> (or if in EBCDIC, after a
+translation to ISO 8859-1). This is done without regard to the
+system's native 8-bit encoding, so to change this for systems with
+non-Latin-1 (or non-EBCDIC) native encodings, use the C<encoding>
+pragma, see L<encoding>.
Under character semantics, many operations that formerly operated on
-bytes change to operating on characters. For ASCII data this makes no
-difference, because UTF-8 stores ASCII in single bytes, but for any
-character greater than C<chr(127)>, the character B<may> be stored in
-a sequence of two or more bytes, all of which have the high bit set.
-
-For C1 controls or Latin 1 characters on an EBCDIC platform the
-character may be stored in a UTF-EBCDIC multi byte sequence. But by
-and large, the user need not worry about this, because Perl hides it
-from the user. A character in Perl is logically just a number ranging
-from 0 to 2**32 or so. Larger characters encode to longer sequences
-of bytes internally, but again, this is just an internal detail which
-is hidden at the Perl level.
+bytes change to operating on characters. A character in Perl is
+logically just a number ranging from 0 to 2**31 or so. Larger
+characters may encode to longer sequences of bytes internally, but
+this is just an internal detail which is hidden at the Perl level.
+See L<perluniintro> for more on this.
=head2 Effects of character semantics
=item *
-Strings and patterns may contain characters that have an ordinal value
-larger than 255.
+Strings (including hash keys) and regular expression patterns may
+contain characters that have an ordinal value larger than 255.
-Presuming you use a Unicode editor to edit your program, such
-characters will typically occur directly within the literal strings as
-UTF-8 (or UTF-EBCDIC on EBCDIC platforms) characters, but you can also
-specify a particular character with an extension of the C<\x>
-notation. UTF-X characters are specified by putting the hexadecimal
-code within curlies after the C<\x>. For instance, a Unicode smiley
-face is C<\x{263A}>.
+If you use a Unicode editor to edit your program, Unicode characters
+may occur directly within the literal strings in one of the various
+Unicode encodings (UTF-8, UTF-EBCDIC, UCS-2, etc.), but are recognized
+as such (and converted to Perl's internal representation) only if the
+appropriate L<encoding> is specified.
+
+You can also get Unicode characters into a string by using the C<\x{...}>
+notation, putting the Unicode code for the desired character, in
+hexadecimal, into the curlies. For instance, a smiley face is C<\x{263A}>.
+This works only for characters with a code 0x100 and above.
+
+Additionally, if you
+
+ use charnames ':full';
+
+you can use the C<\N{...}> notation, putting the official Unicode character
+name within the curlies. For example, C<\N{WHITE SMILING FACE}>.
+This works for all characters that have names.
=item *
-Identifiers within the Perl script may contain Unicode alphanumeric
-characters, including ideographs. (You are currently on your own when
-it comes to using the canonical forms of characters--Perl doesn't
-(yet) attempt to canonicalize variable names for you.)
+If an appropriate L<encoding> is specified, identifiers within the
+Perl script may contain Unicode alphanumeric characters, including
+ideographs. (You are currently on your own when it comes to using the
+canonical forms of characters--Perl doesn't (yet) attempt to
+canonicalize variable names for you.)
=item *
=item *
-Named Unicode properties and block ranges make be used as character
-classes via the new C<\p{}> (matches property) and C<\P{}> (doesn't
-match property) constructs. For instance, C<\p{Lu}> matches any
-character with the Unicode uppercase property, while C<\p{M}> matches
-any mark character. Single letter properties may omit the brackets,
-so that can be written C<\pM> also. Many predefined character classes
-are available, such as C<\p{IsMirrored}> and C<\p{InTibetan}>.
-
-The C<\p{Is...}> test for "general properties" such as "letter",
-"digit", while the C<\p{In...}> test for Unicode scripts and blocks.
-
-The official Unicode script and block names have spaces and dashes and
-separators, but for convenience you can have dashes, spaces, and
-underbars at every word division, and you need not care about correct
-casing. It is recommended, however, that for consistency you use the
-following naming: the official Unicode script, block, or property name
-(see below for the additional rules that apply to block names),
-with whitespace and dashes replaced with underbar, and the words
-"uppercase-first-lowercase-rest". That is, "Latin-1 Supplement"
-becomes "Latin_1_Supplement".
+Named Unicode properties, scripts, and block ranges may be used like
+character classes via the new C<\p{}> (matches property) and C<\P{}>
+(doesn't match property) constructs. For instance, C<\p{Lu}> matches any
+character with the Unicode "Lu" (Letter, uppercase) property, while
+C<\p{M}> matches any character with a "M" (mark -- accents and such)
+property. Single letter properties may omit the brackets, so that can be
+written C<\pM> also. Many predefined properties are available, such
+as C<\p{Mirrored}> and C<\p{Tibetan}>.
+
+The official Unicode script and block names have spaces and dashes as
+separators, but for convenience you can have dashes, spaces, and underbars
+at every word division, and you need not care about correct casing. It is
+recommended, however, that for consistency you use the following naming:
+the official Unicode script, block, or property name (see below for the
+additional rules that apply to block names), with whitespace and dashes
+removed, and the words "uppercase-first-lowercase-rest". That is, "Latin-1
+Supplement" becomes "Latin1Supplement".
You can also negate both C<\p{}> and C<\P{}> by introducing a caret
-(^) between the first curly and the property name: C<\p{^In_Tamil}> is
-equal to C<\P{In_Tamil}>.
+(^) between the first curly and the property name: C<\p{^Tamil}> is
+equal to C<\P{Tamil}>.
-The C<In> and C<Is> can be left out: C<\p{Greek}> is equal to
-C<\p{In_Greek}>, C<\P{Pd}> is equal to C<\P{Pd}>.
+Here are the basic Unicode General Category properties, followed by their
+long form (you can use either, e.g. C<\p{Lu}> and C<\p{LowercaseLetter}>
+are identical).
Short Long
L Letter
- Lu Uppercase_Letter
- Ll Lowercase_Letter
- Lt Titlecase_Letter
- Lm Modifier_Letter
- Lo Other_Letter
+ Lu UppercaseLetter
+ Ll LowercaseLetter
+ Lt TitlecaseLetter
+ Lm ModifierLetter
+ Lo OtherLetter
M Mark
- Mn Nonspacing_Mark
- Mc Spacing_Mark
- Me Enclosing_Mark
+ Mn NonspacingMark
+ Mc SpacingMark
+ Me EnclosingMark
N Number
- Nd Decimal_Number
- Nl Letter_Number
- No Other_Number
+ Nd DecimalNumber
+ Nl LetterNumber
+ No OtherNumber
P Punctuation
- Pc Connector_Punctuation
- Pd Dash_Punctuation
- Ps Open_Punctuation
- Pe Close_Punctuation
- Pi Initial_Punctuation
+ Pc ConnectorPunctuation
+ Pd DashPunctuation
+ Ps OpenPunctuation
+ Pe ClosePunctuation
+ Pi InitialPunctuation
(may behave like Ps or Pe depending on usage)
- Pf Final_Punctuation
+ Pf FinalPunctuation
(may behave like Ps or Pe depending on usage)
- Po Other_Punctuation
+ Po OtherPunctuation
S Symbol
- Sm Math_Symbol
- Sc Currency_Symbol
- Sk Modifier_Symbol
- So Other_Symbol
+ Sm MathSymbol
+ Sc CurrencySymbol
+ Sk ModifierSymbol
+ So OtherSymbol
Z Separator
- Zs Space_Separator
- Zl Line_Separator
- Zp Paragraph_Separator
+ Zs SpaceSeparator
+ Zl LineSeparator
+ Zp ParagraphSeparator
C Other
Cc Control
Cf Format
- Cs Surrogate
- Co Private_Use
+ Cs Surrogate (not usable)
+ Co PrivateUse
Cn Unassigned
+The single-letter properties match all characters in any of the
+two-letter sub-properties starting with the same letter.
There's also C<L&> which is an alias for C<Ll>, C<Lu>, and C<Lt>.
-The following reserved ranges have C<In> tests:
-
- CJK_Ideograph_Extension_A
- CJK_Ideograph
- Hangul_Syllable
- Non_Private_Use_High_Surrogate
- Private_Use_High_Surrogate
- Low_Surrogate
- Private_Surrogate
- CJK_Ideograph_Extension_B
- Plane_15_Private_Use
- Plane_16_Private_Use
+Because Perl hides the need for the user to understand the internal
+representation of Unicode characters, it has no need to support the
+somewhat messy concept of surrogates. Therefore, the C<Cs> property is not
+supported.
-For example C<"\x{AC00}" =~ \p{HangulSyllable}> will test true.
-(Handling of surrogates is not implemented yet, because Perl
-uses UTF-8 and not UTF-16 internally to represent Unicode.)
+Because scripts differ in their directionality (for example Hebrew is
+written right to left), Unicode supplies these properties:
-Additionally, because scripts differ in their directionality
-(for example Hebrew is written right to left), all characters
-have their directionality defined:
+ Property Meaning
BidiL Left-to-Right
BidiLRE Left-to-Right Embedding
BidiWS Whitespace
BidiON Other Neutrals
+For example, C<\p{BidiR}> matches all characters that are normally
+written right to left.
+
+=back
+
=head2 Scripts
-The scripts available for C<\p{In...}> and C<\P{In...}>, for example
-\p{InCyrillic>, are as follows, for example C<\p{InLatin}> or C<\P{InHan}>:
+The scripts available via C<\p{...}> and C<\P{...}>, for example
+C<\p{Latin}> or C<\p{Cyrillic}>, are as follows:
Arabic
Armenian
Bengali
Bopomofo
- Canadian-Aboriginal
+ Buhid
+ CanadianAboriginal
Cherokee
Cyrillic
Deseret
Gurmukhi
Han
Hangul
+ Hanunoo
Hebrew
Hiragana
Inherited
Mongolian
Myanmar
Ogham
- Old-Italic
+ OldItalic
Oriya
Runic
Sinhala
Syriac
+ Tagalog
+ Tagbanwa
Tamil
Telugu
Thaana
There are also extended property classes that supplement the basic
properties, defined by the F<PropList> Unicode database:
- ASCII_Hex_Digit
- Bidi_Control
+ ASCIIHexDigit
+ BidiControl
Dash
+ Deprecated
Diacritic
Extender
- Hex_Digit
+ GraphemeLink
+ HexDigit
Hyphen
Ideographic
- Join_Control
- Noncharacter_Code_Point
- Other_Alphabetic
- Other_Lowercase
- Other_Math
- Other_Uppercase
- Quotation_Mark
- White_Space
+ IDSBinaryOperator
+ IDSTrinaryOperator
+ JoinControl
+ LogicalOrderException
+ NoncharacterCodePoint
+ OtherAlphabetic
+ OtherDefaultIgnorableCodePoint
+ OtherGraphemeExtend
+ OtherLowercase
+ OtherMath
+ OtherUppercase
+ QuotationMark
+ Radical
+ SoftDotted
+ TerminalPunctuation
+ UnifiedIdeograph
+ WhiteSpace
and further derived properties:
- Alphabetic Lu + Ll + Lt + Lm + Lo + Other_Alphabetic
- Lowercase Ll + Other_Lowercase
- Uppercase Lu + Other_Uppercase
- Math Sm + Other_Math
+ Alphabetic Lu + Ll + Lt + Lm + Lo + OtherAlphabetic
+ Lowercase Ll + OtherLowercase
+ Uppercase Lu + OtherUppercase
+ Math Sm + OtherMath
ID_Start Lu + Ll + Lt + Lm + Lo + Nl
ID_Continue ID_Start + Mn + Mc + Nd + Pc
Any Any character
- Assigned Any non-Cn character
+ Assigned Any non-Cn character (i.e. synonym for \P{Cn})
+ Unassigned Synonym for \p{Cn}
Common Any character (or unassigned code point)
not explicitly assigned to a script
+For backward compatibility, all properties mentioned so far may have C<Is>
+prepended to their name (e.g. C<\P{IsLu}> is equal to C<\P{Lu}>).
+
=head2 Blocks
-In addition to B<scripts>, Unicode also defines B<blocks> of
-characters. The difference between scripts and blocks is that the
-scripts concept is closer to natural languages, while the blocks
-concept is more an artificial grouping based on groups of 256 Unicode
-characters. For example, the C<Latin> script contains letters from
-many blocks. On the other hand, the C<Latin> script does not contain
-all the characters from those blocks, it does not for example contain
-digits because digits are shared across many scripts. Digits and
-other similar groups, like punctuation, are in a category called
-C<Common>.
-
-For more about scripts see the UTR #24:
-http://www.unicode.org/unicode/reports/tr24/
-For more about blocks see
-http://www.unicode.org/Public/UNIDATA/Blocks.txt
-
-Because there are overlaps in naming (there are, for example, both
-a script called C<Katakana> and a block called C<Katakana>, the block
-version has C<Block> appended to its name, C<\p{InKatakanaBlock}>.
-
-Notice that this definition was introduced in Perl 5.8.0: in Perl
-5.6 only the blocks were used; in Perl 5.8.0 scripts became the
-preferential Unicode character class definition; this meant that
-the definitions of some character classes changed (the ones in the
-below list that have the C<Block> appended).
-
- Alphabetic Presentation Forms
- Arabic Block
- Arabic Presentation Forms-A
- Arabic Presentation Forms-B
- Armenian Block
- Arrows
- Basic Latin
- Bengali Block
- Block Elements
- Bopomofo Block
- Bopomofo Extended
- Box Drawing
- Braille Patterns
- Byzantine Musical Symbols
- CJK Compatibility
- CJK Compatibility Forms
- CJK Compatibility Ideographs
- CJK Compatibility Ideographs Supplement
- CJK Radicals Supplement
- CJK Symbols and Punctuation
- CJK Unified Ideographs
- CJK Unified Ideographs Extension A
- CJK Unified Ideographs Extension B
- Cherokee Block
- Combining Diacritical Marks
- Combining Half Marks
- Combining Marks for Symbols
- Control Pictures
- Currency Symbols
- Cyrillic Block
- Deseret Block
- Devanagari Block
- Dingbats
- Enclosed Alphanumerics
- Enclosed CJK Letters and Months
- Ethiopic Block
- General Punctuation
- Geometric Shapes
- Georgian Block
- Gothic Block
- Greek Block
- Greek Extended
- Gujarati Block
- Gurmukhi Block
- Halfwidth and Fullwidth Forms
- Hangul Compatibility Jamo
- Hangul Jamo
- Hangul Syllables
- Hebrew Block
- High Private Use Surrogates
- High Surrogates
- Hiragana Block
- IPA Extensions
- Ideographic Description Characters
- Kanbun
- Kangxi Radicals
- Kannada Block
- Katakana Block
- Khmer Block
- Lao Block
- Latin 1 Supplement
- Latin Extended Additional
- Latin Extended-A
- Latin Extended-B
- Letterlike Symbols
- Low Surrogates
- Malayalam Block
- Mathematical Alphanumeric Symbols
- Mathematical Operators
- Miscellaneous Symbols
- Miscellaneous Technical
- Mongolian Block
- Musical Symbols
- Myanmar Block
- Number Forms
- Ogham Block
- Old Italic Block
- Optical Character Recognition
- Oriya Block
- Private Use
- Runic Block
- Sinhala Block
- Small Form Variants
- Spacing Modifier Letters
- Specials
- Superscripts and Subscripts
- Syriac Block
- Tags
- Tamil Block
- Telugu Block
- Thaana Block
- Thai Block
- Tibetan Block
- Unified Canadian Aboriginal Syllabics
- Yi Radicals
- Yi Syllables
-
-=item *
-
-The special pattern C<\X> match matches any extended Unicode sequence
+In addition to B<scripts>, Unicode also defines B<blocks> of characters.
+The difference between scripts and blocks is that the scripts concept is
+closer to natural languages, while the blocks concept is more an artificial
+grouping based on groups of mostly 256 Unicode characters. For example, the
+C<Latin> script contains letters from many blocks. On the other hand, the
+C<Latin> script does not contain all the characters from those blocks. It
+does not, for example, contain digits because digits are shared across many
+scripts. Digits and other similar groups, like punctuation, are in a
+category called C<Common>.
+
+For more about scripts, see the UTR #24:
+
+ http://www.unicode.org/unicode/reports/tr24/
+
+For more about blocks, see:
+
+ http://www.unicode.org/Public/UNIDATA/Blocks.txt
+
+Blocks names are given with the C<In> prefix. For example, the
+Katakana block is referenced via C<\p{InKatakana}>. The C<In>
+prefix may be omitted if there is no naming conflict with a script
+or any other property, but it is recommended that C<In> always be used
+to avoid confusion.
+
+These block names are supported:
+
+ InAlphabeticPresentationForms
+ InArabic
+ InArabicPresentationFormsA
+ InArabicPresentationFormsB
+ InArmenian
+ InArrows
+ InBasicLatin
+ InBengali
+ InBlockElements
+ InBopomofo
+ InBopomofoExtended
+ InBoxDrawing
+ InBraillePatterns
+ InBuhid
+ InByzantineMusicalSymbols
+ InCJKCompatibility
+ InCJKCompatibilityForms
+ InCJKCompatibilityIdeographs
+ InCJKCompatibilityIdeographsSupplement
+ InCJKRadicalsSupplement
+ InCJKSymbolsAndPunctuation
+ InCJKUnifiedIdeographs
+ InCJKUnifiedIdeographsExtensionA
+ InCJKUnifiedIdeographsExtensionB
+ InCherokee
+ InCombiningDiacriticalMarks
+ InCombiningDiacriticalMarksforSymbols
+ InCombiningHalfMarks
+ InControlPictures
+ InCurrencySymbols
+ InCyrillic
+ InCyrillicSupplementary
+ InDeseret
+ InDevanagari
+ InDingbats
+ InEnclosedAlphanumerics
+ InEnclosedCJKLettersAndMonths
+ InEthiopic
+ InGeneralPunctuation
+ InGeometricShapes
+ InGeorgian
+ InGothic
+ InGreekExtended
+ InGreekAndCoptic
+ InGujarati
+ InGurmukhi
+ InHalfwidthAndFullwidthForms
+ InHangulCompatibilityJamo
+ InHangulJamo
+ InHangulSyllables
+ InHanunoo
+ InHebrew
+ InHighPrivateUseSurrogates
+ InHighSurrogates
+ InHiragana
+ InIPAExtensions
+ InIdeographicDescriptionCharacters
+ InKanbun
+ InKangxiRadicals
+ InKannada
+ InKatakana
+ InKatakanaPhoneticExtensions
+ InKhmer
+ InLao
+ InLatin1Supplement
+ InLatinExtendedA
+ InLatinExtendedAdditional
+ InLatinExtendedB
+ InLetterlikeSymbols
+ InLowSurrogates
+ InMalayalam
+ InMathematicalAlphanumericSymbols
+ InMathematicalOperators
+ InMiscellaneousMathematicalSymbolsA
+ InMiscellaneousMathematicalSymbolsB
+ InMiscellaneousSymbols
+ InMiscellaneousTechnical
+ InMongolian
+ InMusicalSymbols
+ InMyanmar
+ InNumberForms
+ InOgham
+ InOldItalic
+ InOpticalCharacterRecognition
+ InOriya
+ InPrivateUseArea
+ InRunic
+ InSinhala
+ InSmallFormVariants
+ InSpacingModifierLetters
+ InSpecials
+ InSuperscriptsAndSubscripts
+ InSupplementalArrowsA
+ InSupplementalArrowsB
+ InSupplementalMathematicalOperators
+ InSupplementaryPrivateUseAreaA
+ InSupplementaryPrivateUseAreaB
+ InSyriac
+ InTagalog
+ InTagbanwa
+ InTags
+ InTamil
+ InTelugu
+ InThaana
+ InThai
+ InTibetan
+ InUnifiedCanadianAboriginalSyllabics
+ InVariationSelectors
+ InYiRadicals
+ InYiSyllables
+
+=over 4
+
+=item *
+
+The special pattern C<\X> matches any extended Unicode sequence
(a "combining character sequence" in Standardese), where the first
character is a base character and subsequent characters are mark
characters that apply to the base character. It is equivalent to
The C<pack()>/C<unpack()> letters "C<c>" and "C<C>" do I<not> change,
since they're often used for byte-oriented formats. (Again, think
"C<char>" in the C language.) However, there is a new "C<U>" specifier
-that will convert between UTF-8 characters and integers. (It works
-outside of the utf8 pragma too.)
+that will convert between Unicode characters and integers.
=item *
C<pack("U")> and C<unpack("U")>, not like C<pack("C")> and
C<unpack("C")>. In fact, the latter are how you now emulate
byte-oriented C<chr()> and C<ord()> for Unicode strings.
-(Note that this reveals the internal UTF-8 encoding of strings and
-you are not supposed to do that unless you know what you are doing.)
+(Note that this reveals the internal encoding of Unicode strings,
+which is not something one normally needs to care about at all.)
=item *
=back
-What doesn't yet work are the followng cases:
+What doesn't yet work are the following cases:
=over 8
See L<Encode>.
-=head1 CAVEATS
-
-As of yet, there is no method for automatically coercing input and
-output to some encoding other than UTF-8 or UTF-EBCDIC. This is planned
-in the near future, however.
-
-Whether an arbitrary piece of data will be treated as "characters" or
-"bytes" by internal operations cannot be divined at the current time.
-
-Use of locales with utf8 may lead to odd results. Currently there is
-some attempt to apply 8-bit locale info to characters in the range
-0..255, but this is demonstrably incorrect for locales that use
-characters above that range (when mapped into Unicode). It will also
-tend to run slower. Avoidance of locales is strongly encouraged.
-
-=head1 UNICODE REGULAR EXPRESSION SUPPORT LEVEL
+=head2 Unicode Regular Expression Support Level
The following list of Unicode regular expression support describes
feature by feature the Unicode support implemented in Perl as of Perl
Level 1 - Basic Unicode Support
2.1 Hex Notation - done [1]
- Named Notation - done [2]
+ Named Notation - done [2]
2.2 Categories - done [3][4]
2.3 Subtraction - MISSING [5][6]
2.4 Simple Word Boundaries - done [7]
- 2.5 Simple Loose Matches - MISSING [8]
+ 2.5 Simple Loose Matches - done [8]
2.6 End of Line - MISSING [9][10]
[ 1] \x{...}
[ 2] \N{...}
- [ 3] . \p{Is...} \P{Is...}
- [ 4] now scripts (see UTR#24 Script Names) in addition to blocks
+ [ 3] . \p{...} \P{...}
+ [ 4] now scripts (see UTR#24 Script Names) in addition to blocks
[ 5] have negation
- [ 6] can use look-ahead to emulate subtracion
+ [ 6] can use look-ahead to emulate subtraction (*)
[ 7] include Letters in word characters
- [ 8] see UTR#21 Case Mappings
+ [ 8] note that perl does Full casefolding in matching, not Simple:
+ for example U+1F88 is equivalent with U+1F000 U+03B9,
+ not with 1F80. This difference matters for certain Greek
+ capital letters with certain modifiers: the Full casefolding
+ decomposes the letter, while the Simple casefolding would map
+ it to a single character.
[ 9] see UTR#13 Unicode Newline Guidelines
- [10] should do ^ and $ also on \x{2028} and \x{2029}
+ [10] should do ^ and $ also on \x{85}, \x{2028} and \x{2029})
+ (should also affect <>, $., and script line numbers)
+ (the \x{85}, \x{2028} and \x{2029} do match \s)
+
+(*) You can mimic class subtraction using lookahead.
+For example, what TR18 might write as
+
+ [{Greek}-[{UNASSIGNED}]]
+
+in Perl can be written as:
+
+ (?!\p{Unassigned})\p{InGreekAndCoptic}
+ (?=\p{Assigned})\p{InGreekAndCoptic}
+
+But in this particular example, you probably really want
+
+ \p{Greek}
+
+which will match assigned characters known to be part of the Greek script.
=item *
=back
+=head2 Unicode Encodings
+
+Unicode characters are assigned to I<code points> which are abstract
+numbers. To use these numbers various encodings are needed.
+
+=over 4
+
+=item *
+
+UTF-8
+
+UTF-8 is a variable-length (1 to 6 bytes, current character allocations
+require 4 bytes), byteorder independent encoding. For ASCII, UTF-8 is
+transparent (and we really do mean 7-bit ASCII, not another 8-bit encoding).
+
+The following table is from Unicode 3.2.
+
+ Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
+
+ U+0000..U+007F 00..7F
+ U+0080..U+07FF C2..DF 80..BF
+ U+0800..U+0FFF E0 A0..BF 80..BF
+ U+1000..U+CFFF E1..EC 80..BF 80..BF
+ U+D000..U+D7FF ED 80..9F 80..BF
+ U+D800..U+DFFF ******* ill-formed *******
+ U+E000..U+FFFF EE..EF 80..BF 80..BF
+ U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
+ U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
+ U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
+
+Note the A0..BF in U+0800..U+0FFF, the 80..9F in U+D000...U+D7FF,
+the 90..BF in U+10000..U+3FFFF, and the 80...8F in U+100000..U+10FFFF.
+The "gaps" are caused by legal UTF-8 avoiding non-shortest encodings:
+it is technically possible to UTF-8-encode a single code point in different
+ways, but that is explicitly forbidden, and the shortest possible encoding
+should always be used (and that is what Perl does).
+
+Or, another way to look at it, as bits:
+
+ Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
+
+ 0aaaaaaa 0aaaaaaa
+ 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
+ ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
+ 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
+
+As you can see, the continuation bytes all begin with C<10>, and the
+leading bits of the start byte tell how many bytes the are in the
+encoded character.
+
+=item *
+
+UTF-EBCDIC
+
+Like UTF-8, but EBCDIC-safe, as UTF-8 is ASCII-safe.
+
+=item *
+
+UTF-16, UTF-16BE, UTF16-LE, Surrogates, and BOMs (Byte Order Marks)
+
+(The followings items are mostly for reference, Perl doesn't
+use them internally.)
+
+UTF-16 is a 2 or 4 byte encoding. The Unicode code points
+0x0000..0xFFFF are stored in two 16-bit units, and the code points
+0x010000..0x10FFFF in two 16-bit units. The latter case is
+using I<surrogates>, the first 16-bit unit being the I<high
+surrogate>, and the second being the I<low surrogate>.
+
+Surrogates are code points set aside to encode the 0x01000..0x10FFFF
+range of Unicode code points in pairs of 16-bit units. The I<high
+surrogates> are the range 0xD800..0xDBFF, and the I<low surrogates>
+are the range 0xDC00..0xDFFFF. The surrogate encoding is
+
+ $hi = ($uni - 0x10000) / 0x400 + 0xD800;
+ $lo = ($uni - 0x10000) % 0x400 + 0xDC00;
+
+and the decoding is
+
+ $uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00);
+
+If you try to generate surrogates (for example by using chr()), you
+will get a warning if warnings are turned on (C<-w> or C<use
+warnings;>) because those code points are not valid for a Unicode
+character.
+
+Because of the 16-bitness, UTF-16 is byteorder dependent. UTF-16
+itself can be used for in-memory computations, but if storage or
+transfer is required, either UTF-16BE (Big Endian) or UTF-16LE
+(Little Endian) must be chosen.
+
+This introduces another problem: what if you just know that your data
+is UTF-16, but you don't know which endianness? Byte Order Marks
+(BOMs) are a solution to this. A special character has been reserved
+in Unicode to function as a byte order marker: the character with the
+code point 0xFEFF is the BOM.
+
+The trick is that if you read a BOM, you will know the byte order,
+since if it was written on a big endian platform, you will read the
+bytes 0xFE 0xFF, but if it was written on a little endian platform,
+you will read the bytes 0xFF 0xFE. (And if the originating platform
+was writing in UTF-8, you will read the bytes 0xEF 0xBB 0xBF.)
+
+The way this trick works is that the character with the code point
+0xFFFE is guaranteed not to be a valid Unicode character, so the
+sequence of bytes 0xFF 0xFE is unambiguously "BOM, represented in
+little-endian format" and cannot be "0xFFFE, represented in big-endian
+format".
+
+=item *
+
+UTF-32, UTF-32BE, UTF32-LE
+
+The UTF-32 family is pretty much like the UTF-16 family, expect that
+the units are 32-bit, and therefore the surrogate scheme is not
+needed. The BOM signatures will be 0x00 0x00 0xFE 0xFF for BE and
+0xFF 0xFE 0x00 0x00 for LE.
+
+=item *
+
+UCS-2, UCS-4
+
+Encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit
+encoding, UCS-4 is a 32-bit encoding. Unlike UTF-16, UCS-2
+is not extensible beyond 0xFFFF, because it does not use surrogates.
+
+=item *
+
+UTF-7
+
+A seven-bit safe (non-eight-bit) encoding, useful if the
+transport/storage is not eight-bit safe. Defined by RFC 2152.
+
+=back
+
+=head2 Security Implications of Unicode
+
+=over 4
+
+=item *
+
+Malformed UTF-8
+
+Unfortunately, the specification of UTF-8 leaves some room for
+interpretation of how many bytes of encoded output one should generate
+from one input Unicode character. Strictly speaking, one is supposed
+to always generate the shortest possible sequence of UTF-8 bytes,
+because otherwise there is potential for input buffer overflow at
+the receiving end of a UTF-8 connection. Perl always generates the
+shortest length UTF-8, and with warnings on (C<-w> or C<use
+warnings;>) Perl will warn about non-shortest length UTF-8 (and other
+malformations, too, such as the surrogates, which are not real
+Unicode code points.)
+
+=item *
+
+Regular expressions behave slightly differently between byte data and
+character (Unicode data). For example, the "word character" character
+class C<\w> will work differently when the data is all eight-bit bytes
+or when the data is Unicode.
+
+In the first case, the set of C<\w> characters is either small (the
+default set of alphabetic characters, digits, and the "_"), or, if you
+are using a locale (see L<perllocale>), the C<\w> might contain a few
+more letters according to your language and country.
+
+In the second case, the C<\w> set of characters is much, much larger,
+and most importantly, even in the set of the first 256 characters, it
+will most probably be different: as opposed to most locales (which are
+specific to a language and country pair) Unicode classifies all the
+characters that are letters as C<\w>. For example: your locale might
+not think that LATIN SMALL LETTER ETH is a letter (unless you happen
+to speak Icelandic), but Unicode does.
+
+As discussed elsewhere, Perl tries to stand one leg (two legs, as
+camels are quadrupeds?) in two worlds: the old world of bytes and the new
+world of characters, upgrading from bytes to characters when necessary.
+If your legacy code is not explicitly using Unicode, no automatic
+switchover to characters should happen, and characters shouldn't get
+downgraded back to bytes, either. It is possible to accidentally mix
+bytes and characters, however (see L<perluniintro>), in which case the
+C<\w> might start behaving differently. Review your code.
+
+=back
+
+=head2 Unicode in Perl on EBCDIC
+
+The way Unicode is handled on EBCDIC platforms is still rather
+experimental. On such a platform, references to UTF-8 encoding in this
+document and elsewhere should be read as meaning UTF-EBCDIC as
+specified in Unicode Technical Report 16 unless ASCII vs EBCDIC issues
+are specifically discussed. There is no C<utfebcdic> pragma or
+":utfebcdic" layer, rather, "utf8" and ":utf8" are re-used to mean
+the platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic>
+for more discussion of the issues.
+
+=head2 Locales
+
+Usually locale settings and Unicode do not affect each other, but
+there are a couple of exceptions:
+
+=over 4
+
+=item *
+
+If your locale environment variables (LANGUAGE, LC_ALL, LC_CTYPE, LANG)
+contain the strings 'UTF-8' or 'UTF8' (case-insensitive matching),
+the default encoding of your STDIN, STDOUT, and STDERR, and of
+B<any subsequent file open>, is UTF-8.
+
+=item *
+
+Perl tries really hard to work both with Unicode and the old byte
+oriented world: most often this is nice, but sometimes this causes
+problems.
+
+=back
+
+=head2 Using Unicode in XS
+
+If you want to handle Perl Unicode in XS extensions, you may find
+the following C APIs useful (see perlapi for details):
+
+=over 4
+
+=item *
+
+DO_UTF8(sv) returns true if the UTF8 flag is on and the bytes pragma
+is not in effect. SvUTF8(sv) returns true is the UTF8 flag is on, the
+bytes pragma is ignored. The UTF8 flag being on does B<not> mean that
+there are any characters of code points greater than 255 (or 127) in
+the scalar, or that there even are any characters in the scalar.
+What the UTF8 flag means is that the sequence of octets in the
+representation of the scalar is the sequence of UTF-8 encoded
+code points of the characters of a string. The UTF8 flag being
+off means that each octet in this representation encodes a single
+character with codepoint 0..255 within the string. Perl's Unicode
+model is not to use UTF-8 until it's really necessary.
+
+=item *
+
+uvuni_to_utf8(buf, chr) writes a Unicode character code point into a
+buffer encoding the code point as UTF-8, and returns a pointer
+pointing after the UTF-8 bytes.
+
+=item *
+
+utf8_to_uvuni(buf, lenp) reads UTF-8 encoded bytes from a buffer and
+returns the Unicode character code point (and optionally the length of
+the UTF-8 byte sequence).
+
+=item *
+
+utf8_length(start, end) returns the length of the UTF-8 encoded buffer
+in characters. sv_len_utf8(sv) returns the length of the UTF-8 encoded
+scalar.
+
+=item *
+
+sv_utf8_upgrade(sv) converts the string of the scalar to its UTF-8
+encoded form. sv_utf8_downgrade(sv) does the opposite (if possible).
+sv_utf8_encode(sv) is like sv_utf8_upgrade but the UTF8 flag does not
+get turned on. sv_utf8_decode() does the opposite of sv_utf8_encode().
+Note that none of these are to be used as general purpose encoding/decoding
+interfaces: use Encode for that. sv_utf8_upgrade() is affected by the
+encoding pragma, but sv_utf8_downgrade() is not (since the encoding
+pragma is designed to be a one-way street).
+
+=item *
+
+is_utf8_char(s) returns true if the pointer points to a valid UTF-8
+character.
+
+=item *
+
+is_utf8_string(buf, len) returns true if the len bytes of the buffer
+are valid UTF-8.
+
+=item *
+
+UTF8SKIP(buf) will return the number of bytes in the UTF-8 encoded
+character in the buffer. UNISKIP(chr) will return the number of bytes
+required to UTF-8-encode the Unicode character code point. UTF8SKIP()
+is useful for example for iterating over the characters of a UTF-8
+encoded buffer; UNISKIP() is useful for example in computing
+the size required for a UTF-8 encoded buffer.
+
+=item *
+
+utf8_distance(a, b) will tell the distance in characters between the
+two pointers pointing to the same UTF-8 encoded buffer.
+
+=item *
+
+utf8_hop(s, off) will return a pointer to an UTF-8 encoded buffer that
+is C<off> (positive or negative) Unicode characters displaced from the
+UTF-8 buffer C<s>. Be careful not to overstep the buffer: utf8_hop()
+will merrily run off the end or the beginning if told to do so.
+
+=item *
+
+pv_uni_display(dsv, spv, len, pvlim, flags) and sv_uni_display(dsv,
+ssv, pvlim, flags) are useful for debug output of Unicode strings and
+scalars. By default they are useful only for debug: they display
+B<all> characters as hexadecimal code points, but with the flags
+UNI_DISPLAY_ISPRINT and UNI_DISPLAY_BACKSLASH you can make the output
+more readable.
+
+=item *
+
+ibcmp_utf8(s1, pe1, u1, l1, u1, s2, pe2, l2, u2) can be used to
+compare two strings case-insensitively in Unicode.
+(For case-sensitive comparisons you can just use memEQ() and memNE()
+as usual.)
+
+=back
+
+For more information, see L<perlapi>, and F<utf8.c> and F<utf8.h>
+in the Perl source code distribution.
+
+=head1 BUGS
+
+=head2 Interaction with locales
+
+Use of locales with Unicode data may lead to odd results. Currently
+there is some attempt to apply 8-bit locale info to characters in the
+range 0..255, but this is demonstrably incorrect for locales that use
+characters above that range when mapped into Unicode. It will also
+tend to run slower. Use of locales with Unicode is discouraged.
+
+=head2 Interaction with extensions
+
+When perl exchanges data with an extension, the extension should be
+able to understand the UTF-8 flag and act accordingly. If the
+extension doesn't know about the flag, the risk is high that it will
+return data that are incorrectly flagged.
+
+So if you're working with Unicode data, consult the documentation of
+every module you're using if there are any issues with Unicode data
+exchange. If the documentation does not talk about Unicode at all,
+suspect the worst and probably look at the source to learn how the
+module is implemented. Modules written completely in perl shouldn't
+cause problems. Modules that directly or indirectly access code written
+in other programming languages are at risk.
+
+For affected functions the simple strategy to avoid data corruption is
+to always make the encoding of the exchanged data explicit. Choose an
+encoding you know the extension can handle. Convert arguments passed
+to the extensions to that encoding and convert results back from that
+encoding. Write wrapper functions that do the conversions for you, so
+you can later change the functions when the extension catches up.
+
+To provide an example let's say the popular Foo::Bar::escape_html
+function doesn't deal with Unicode data yet. The wrapper function
+would convert the argument to raw UTF-8 and convert the result back to
+perl's internal representation like so:
+
+ sub my_escape_html ($) {
+ my($what) = shift;
+ return unless defined $what;
+ Encode::decode_utf8(Foo::Bar::escape_html(Encode::encode_utf8($what)));
+ }
+
+Sometimes, when the extension does not convert data but just stores
+and retrieves them, you will be in a position to use the otherwise
+dangerous Encode::_utf8_on() function. Let's say the popular
+C<Foo::Bar> extension, written in C, provides a C<param> method that
+lets you store and retrieve data according to these prototypes:
+
+ $self->param($name, $value); # set a scalar
+ $value = $self->param($name); # retrieve a scalar
+
+If it does not yet provide support for any encoding, one could write a
+derived class with such a C<param> method:
+
+ sub param {
+ my($self,$name,$value) = @_;
+ utf8::upgrade($name); # make sure it is UTF-8 encoded
+ if (defined $value)
+ utf8::upgrade($value); # make sure it is UTF-8 encoded
+ return $self->SUPER::param($name,$value);
+ } else {
+ my $ret = $self->SUPER::param($name);
+ Encode::_utf8_on($ret); # we know, it is UTF-8 encoded
+ return $ret;
+ }
+ }
+
+Some extensions provide filters on data entry/exit points, such as
+DB_File::filter_store_key and family. Look out for such filters in
+the documentation of your extensions, they can make the transition to
+Unicode data much easier.
+
+=head2 speed
+
+Some functions are slower when working on UTF-8 encoded strings than
+on byte encoded strings. All functions that need to hop over
+characters such as length(), substr() or index() can work B<much>
+faster when the underlying data are byte-encoded. Witness the
+following benchmark:
+
+ % perl -e '
+ use Benchmark;
+ use strict;
+ our $l = 10000;
+ our $u = our $b = "x" x $l;
+ substr($u,0,1) = "\x{100}";
+ timethese(-2,{
+ LENGTH_B => q{ length($b) },
+ LENGTH_U => q{ length($u) },
+ SUBSTR_B => q{ substr($b, $l/4, $l/2) },
+ SUBSTR_U => q{ substr($u, $l/4, $l/2) },
+ });
+ '
+ Benchmark: running LENGTH_B, LENGTH_U, SUBSTR_B, SUBSTR_U for at least 2 CPU seconds...
+ LENGTH_B: 2 wallclock secs ( 2.36 usr + 0.00 sys = 2.36 CPU) @ 5649983.05/s (n=13333960)
+ LENGTH_U: 2 wallclock secs ( 2.11 usr + 0.00 sys = 2.11 CPU) @ 12155.45/s (n=25648)
+ SUBSTR_B: 3 wallclock secs ( 2.16 usr + 0.00 sys = 2.16 CPU) @ 374480.09/s (n=808877)
+ SUBSTR_U: 2 wallclock secs ( 2.11 usr + 0.00 sys = 2.11 CPU) @ 6791.00/s (n=14329)
+
+The numbers show an incredible slowness on long UTF-8 strings and you
+should carefully avoid to use these functions within tight loops. For
+example if you want to iterate over characters, it is infinitely
+better to split into an array than to use substr, as the following
+benchmark shows:
+
+ % perl -e '
+ use Benchmark;
+ use strict;
+ our $l = 10000;
+ our $u = our $b = "x" x $l;
+ substr($u,0,1) = "\x{100}";
+ timethese(-5,{
+ SPLIT_B => q{ for my $c (split //, $b){} },
+ SPLIT_U => q{ for my $c (split //, $u){} },
+ SUBSTR_B => q{ for my $i (0..length($b)-1){my $c = substr($b,$i,1);} },
+ SUBSTR_U => q{ for my $i (0..length($u)-1){my $c = substr($u,$i,1);} },
+ });
+ '
+ Benchmark: running SPLIT_B, SPLIT_U, SUBSTR_B, SUBSTR_U for at least 5 CPU seconds...
+ SPLIT_B: 6 wallclock secs ( 5.29 usr + 0.00 sys = 5.29 CPU) @ 56.14/s (n=297)
+ SPLIT_U: 5 wallclock secs ( 5.17 usr + 0.01 sys = 5.18 CPU) @ 55.21/s (n=286)
+ SUBSTR_B: 5 wallclock secs ( 5.34 usr + 0.00 sys = 5.34 CPU) @ 123.22/s (n=658)
+ SUBSTR_U: 7 wallclock secs ( 6.20 usr + 0.00 sys = 6.20 CPU) @ 0.81/s (n=5)
+
+You see, the algorithm based on substr() was faster with byte encoded
+data but it is pathologically slow with UTF-8 data.
+
=head1 SEE ALSO
-L<bytes>, L<utf8>, L<perlretut>, L<perlvar/"${^WIDE_SYSTEM_CALLS}">
+L<perluniintro>, L<encoding>, L<Encode>, L<open>, L<utf8>, L<bytes>,
+L<perlretut>, L<perlvar/"${^WIDE_SYSTEM_CALLS}">
=cut
https://perl5.git.perl.org / perl5.git / blobdiff