This is a live mirror of the Perl 5 development currently hosted at
HINT_UTF8 is not propagated to the op tree anymore; add a
[perl5.git] / pod / perlunicode.pod
1=head1 NAME
3perlunicode - Unicode support in Perl
7WARNING: The implementation of Unicode support in Perl is incomplete.
8Expect sudden and unannounced changes!
10Beginning with version 5.6, Perl uses logically wide characters to
11represent strings internally. This internal representation of strings
12uses the UTF-8 encoding.
14In future, Perl-level operations will expect to work with characters
15rather than bytes, in general.
17However, Perl v5.6 aims to provide a safe migration path from byte
18semantics to character semantics for programs. To preserve compatibility
19with earlier versions of Perl which allowed byte semantics in Perl
20operations (owing to the fact that the internal representation for
21characters was in bytes) byte semantics will continue to be in effect
22until a the C<utf8> pragma is used in the C<main> package, or the C<$^U>
23global flag is explicitly set.
25Under character semantics, many operations that formerly operated on
26bytes change to operating on characters. For ASCII data this makes
27no difference, because UTF-8 stores ASCII in single bytes, but for
28any character greater than C<chr(127)>, the character is stored in
29a sequence of two or more bytes, all of which have the high bit set.
30But by and large, the user need not worry about this, because Perl
31hides it from the user. A character in Perl is logically just a number
32ranging from 0 to 2**32 or so. Larger characters encode to longer
33sequences of bytes internally, but again, this is just an internal
34detail which is hidden at the Perl level.
36The C<byte> pragma can be used to force byte semantics in a particular
37lexical scope. See L<byte>.
39The C<utf8> pragma is a compatibility device to enables recognition
40of UTF-8 in literals encountered by the parser. It is also used
41for enabling some experimental Unicode support features. Note that
42this pragma is only required until a future version of Perl in which
43character semantics will become the default. This pragma may then
44become a no-op. See L<utf8>.
46Character semantics have the following effects:
48=over 4
50=item *
52Strings and patterns may contain characters that have an ordinal value
53larger than 255. In Perl v5.6, this is only enabled if the lexical
54scope has a C<use utf8> declaration (due to compatibility needs) but
55future versions may enable this by default.
57Presuming you use a Unicode editor to edit your program, such characters
58will typically occur directly within the literal strings as UTF-8
59characters, but you can also specify a particular character with an
60extension of the C<\x> notation. UTF-8 characters are specified by
61putting the hexadecimal code within curlies after the C<\x>. For instance,
62a Unicode smiley face is C<\x{263A}>. A character in the Latin-1 range
63(128..255) should be written C<\x{ab}> rather than C<\xab>, since the
64former will turn into a two-byte UTF-8 code, while the latter will
65continue to be interpreted as generating a 8-bit byte rather than a
66character. In fact, if C<-w> is turned on, it will produce a warning
67that you might be generating invalid UTF-8.
69=item *
71Identifiers within the Perl script may contain Unicode alphanumeric
72characters, including ideographs. (You are currently on your own when
73it comes to using the canonical forms of characters--Perl doesn't (yet)
74attempt to canonicalize variable names for you.)
76This also needs C<use utf8> currently. [XXX: Why? High-bit chars were
77syntax errors when they occurred within identifiers in previous versions,
78so this should be enabled by default.]
80=item *
82Regular expressions match characters instead of bytes. For instance,
83"." matches a character instead of a byte. (However, the C<\C> pattern
84is provided to force a match a single byte ("C<char>" in C, hence
87Unicode support in regular expressions needs C<use utf8> currently.
88[XXX: Because the SWASH routines need to be loaded. And the RE engine
89appears to need an overhaul to Unicode by default anyway.]
91=item *
93Character classes in regular expressions match characters instead of
94bytes, and match against the character properties specified in the
95Unicode properties database. So C<\w> can be used to match an ideograph,
96for instance.
98C<use utf8> is needed to enable this. See above.
100=item *
102Named Unicode properties and block ranges make be used as character
103classes via the new C<\p{}> (matches property) and C<\P{}> (doesn't
104match property) constructs. For instance, C<\p{Lu}> matches any
105character with the Unicode uppercase property, while C<\p{M}> matches
106any mark character. Single letter properties may omit the brackets, so
107that can be written C<\pM> also. Many predefined character classes are
108available, such as C<\p{IsMirrored}> and C<\p{InTibetan}>.
110C<use utf8> is needed to enable this. See above.
112=item *
114The special pattern C<\X> match matches any extended Unicode sequence
115(a "combining character sequence" in Standardese), where the first
116character is a base character and subsequent characters are mark
117characters that apply to the base character. It is equivalent to
120C<use utf8> is needed to enable this. See above.
122=item *
124The C<tr///> operator translates characters instead of bytes. It can also
125be forced to translate between 8-bit codes and UTF-8 regardless of the
126surrounding utf8 state. For instance, if you know your input in Latin-1,
127you can say:
129 use utf8;
130 while (<>) {
131 tr/\0-\xff//CU; # latin1 char to utf8
132 ...
133 }
135Similarly you could translate your output with
137 tr/\0-\x{ff}//UC; # utf8 to latin1 char
139No, C<s///> doesn't take /U or /C (yet?).
141C<use utf8> is needed to enable this. See above.
143=item *
145Case translation operators use the Unicode case translation tables
146when provided character input. Note that C<uc()> translates to
147uppercase, while C<ucfirst> translates to titlecase (for languages
148that make the distinction). Naturally the corresponding backslash
149sequences have the same semantics.
151=item *
153Most operators that deal with positions or lengths in the string will
154automatically switch to using character positions, including C<chop()>,
155C<substr()>, C<pos()>, C<index()>, C<rindex()>, C<sprintf()>,
156C<write()>, and C<length()>. Operators that specifically don't switch
157include C<vec()>, C<pack()>, and C<unpack()>. Operators that really
158don't care include C<chomp()>, as well as any other operator that
159treats a string as a bucket of bits, such as C<sort()>, and the
160operators dealing with filenames.
162=item *
164The C<pack()>/C<unpack()> letters "C<c>" and "C<C>" do I<not> change,
165since they're often used for byte-oriented formats. (Again, think
166"C<char>" in the C language.) However, there is a new "C<U>" specifier
167that will convert between UTF-8 characters and integers. (It works
168outside of the utf8 pragma too.)
170=item *
172The C<chr()> and C<ord()> functions work on characters. This is like
173C<pack("U")> and C<unpack("U")>, not like C<pack("C")> and
174C<unpack("C")>. In fact, the latter are how you now emulate
175byte-oriented C<chr()> and C<ord()> under utf8.
177=item *
179And finally, C<scalar reverse()> reverses by character rather than by byte.
183=head1 CAVEATS
185As of yet, there is no method for automatically coercing input and
186output to some encoding other than UTF-8. This is planned in the near
187future, however.
189Whether a piece of data will be treated as "characters" or "bytes"
190by internal operations cannot be divined at the current time.
192Use of locales with utf8 may lead to odd results. Currently there is
193some attempt to apply 8-bit locale info to characters in the range
1940..255, but this is demonstrably incorrect for locales that use
195characters above that range (when mapped into Unicode). It will also
196tend to run slower. Avoidance of locales is strongly encouraged.
198=head1 SEE ALSO
200L<byte>, L<utf8>, L<perlvar/"$^U">