This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
merge branch zefram/cloexec
[perl5.git] / lib / utf8.pm
CommitLineData
a0ed51b3
LW
1package utf8;
2
d5448623
GS
3$utf8::hint_bits = 0x00800000;
4
0397beb0 5our $VERSION = '1.20';
b75c8c73 6
a0ed51b3 7sub import {
d5448623 8 $^H |= $utf8::hint_bits;
a0ed51b3
LW
9}
10
11sub unimport {
d5448623 12 $^H &= ~$utf8::hint_bits;
a0ed51b3
LW
13}
14
15sub AUTOLOAD {
16 require "utf8_heavy.pl";
daf4d4ea 17 goto &$AUTOLOAD if defined &$AUTOLOAD;
bd7017d3 18 require Carp;
daf4d4ea 19 Carp::croak("Undefined subroutine $AUTOLOAD called");
a0ed51b3
LW
20}
21
221;
23__END__
24
25=head1 NAME
26
b3419ed8 27utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code
a0ed51b3
LW
28
29=head1 SYNOPSIS
30
291cc134
KW
31 use utf8;
32 no utf8;
a0ed51b3 33
291cc134 34 # Convert the internal representation of a Perl scalar to/from UTF-8.
836ccc8e 35
291cc134 36 $num_octets = utf8::upgrade($string);
98695e13 37 $success = utf8::downgrade($string[, $fail_ok]);
973655a8 38
291cc134
KW
39 # Change each character of a Perl scalar to/from a series of
40 # characters that represent the UTF-8 bytes of each original character.
836ccc8e 41
291cc134
KW
42 utf8::encode($string); # "\x{100}" becomes "\xc4\x80"
43 utf8::decode($string); # "\xc4\x80" becomes "\x{100}"
973655a8 44
ca3d51ba
KW
45 # Convert a code point from the platform native character set to
46 # Unicode, and vice-versa.
47 $unicode = utf8::native_to_unicode(ord('A')); # returns 65 on both
48 # ASCII and EBCDIC
49 # platforms
a04477f8
KW
50 $native = utf8::unicode_to_native(65); # returns 65 on ASCII
51 # platforms; 193 on
52 # EBCDIC
ca3d51ba 53
ac8b87d7
EB
54 $flag = utf8::is_utf8($string); # since Perl 5.8.1
55 $flag = utf8::valid($string);
973655a8 56
a0ed51b3
LW
57=head1 DESCRIPTION
58
393fec97 59The C<use utf8> pragma tells the Perl parser to allow UTF-8 in the
a04477f8
KW
60program text in the current lexical scope. The C<no utf8> pragma tells Perl
61to switch back to treating the source text as literal bytes in the current
62lexical scope. (On EBCDIC platforms, technically it is allowing UTF-EBCDIC,
63and not UTF-8, but this distinction is academic, so in this document the term
64UTF-8 is used to mean both).
a0ed51b3 65
19b49582
JH
66B<Do not use this pragma for anything else than telling Perl that your
67script is written in UTF-8.> The utility functions described below are
2575c402
JW
68directly usable without C<use utf8;>.
69
70Because it is not possible to reliably tell UTF-8 from native 8 bit
71encodings, you need either a Byte Order Mark at the beginning of your
72source code, or C<use utf8;>, to instruct perl.
19b49582 73
2575c402 74When UTF-8 becomes the standard source format, this pragma will
a04477f8 75effectively become a no-op.
a0ed51b3 76
a74e8b45 77See also the effects of the C<-C> switch and its cousin, the
127161e0 78C<PERL_UNICODE> environment variable, in L<perlrun>.
a74e8b45 79
ad0029c4 80Enabling the C<utf8> pragma has the following effect:
a0ed51b3 81
4ac9195f 82=over 4
a0ed51b3
LW
83
84=item *
85
a04477f8
KW
86Bytes in the source text that are not in the ASCII character set will be
87treated as being part of a literal UTF-8 sequence. This includes most
c20e2abd 88literals such as identifier names, string constants, and constant
8f8cf39c
JH
89regular expression patterns.
90
4ac9195f
MS
91=back
92
a04477f8
KW
93Note that if you have non-ASCII, non-UTF-8 bytes in your script (for example
94embedded Latin-1 in your string literals), C<use utf8> will be unhappy. If
95you want to have such bytes under C<use utf8>, you can disable this pragma
96until the end the block (or file, if at top level) by C<no utf8;>.
ae90e350 97
1b026014
NIS
98=head2 Utility functions
99
8800c35a
JH
100The following functions are defined in the C<utf8::> package by the
101Perl core. You do not need to say C<use utf8> to use these and in fact
2f7e5073 102you should not say that unless you really want to have UTF-8 source code.
1b026014
NIS
103
104=over 4
105
308a4ae1 106=item * C<$num_octets = utf8::upgrade($string)>
1b026014 107
a04477f8 108(Since Perl v5.8.0)
836ccc8e 109Converts in-place the internal representation of the string from an octet
a04477f8 110sequence in the native encoding (Latin-1 or EBCDIC) to UTF-8. The
836ccc8e 111logical character sequence itself is unchanged. If I<$string> is already
0397beb0
TC
112upgraded, then this is a no-op. Returns the
113number of octets necessary to represent the string as UTF-8.
114
115If your code needs to be compatible with versions of perl without
116C<use feature 'unicode_strings';>, you can force Unicode semantics on
117a given string:
118
119 # force unicode semantics for $string without the
120 # "unicode_strings" feature
121 utf8::upgrade($string);
122
123For example:
124
125 # without explicit or implicit use feature 'unicode_strings'
126 my $x = "\xDF"; # LATIN SMALL LETTER SHARP S
127 $x =~ /ss/i; # won't match
128 my $y = uc($x); # won't convert
129 utf8::upgrade($x);
130 $x =~ /ss/i; # matches
131 my $z = uc($x); # converts to "SS"
78ea37eb 132
a04477f8
KW
133B<Note that this function does not handle arbitrary encodings>;
134use L<Encode> instead.
1b026014 135
308a4ae1 136=item * C<$success = utf8::downgrade($string[, $fail_ok])>
1b026014 137
a04477f8 138(Since Perl v5.8.0)
730d7228 139Converts in-place the internal representation of the string from
a04477f8 140UTF-8 to the equivalent octet sequence in the native encoding (Latin-1
836ccc8e
DM
141or EBCDIC). The logical character sequence itself is unchanged. If
142I<$string> is already stored as native 8 bit, then this is a no-op. Can
143be used to
2575c402
JW
144make sure that the UTF-8 flag is off, e.g. when you want to make sure
145that the substr() or length() function works with the usually faster
146byte algorithm.
78ea37eb 147
a04477f8 148Fails if the original UTF-8 sequence cannot be represented in the
ac8b87d7 149native 8 bit encoding. On failure dies or, if the value of I<$fail_ok> is
2575c402 150true, returns false.
78ea37eb 151
2575c402
JW
152Returns true on success.
153
0397beb0
TC
154If your code expects an octet sequence this can be used to validate
155that you've received one:
156
157 # throw an exception if not representable as octets
158 utf8::downgrade($string)
159
160 # or do your own error handling
161 utf8::downgrade($string, 1) or die "string must be octets";
162
a04477f8
KW
163B<Note that this function does not handle arbitrary encodings>;
164use L<Encode> instead.
78ea37eb 165
308a4ae1 166=item * C<utf8::encode($string)>
1b026014 167
a04477f8 168(Since Perl v5.8.0)
2575c402 169Converts in-place the character sequence to the corresponding octet
a04477f8 170sequence in UTF-8. That is, every (possibly wide) character gets
836ccc8e 171replaced with a sequence of one or more characters that represent the
a04477f8 172individual UTF-8 bytes of the character. The UTF8 flag is turned off.
836ccc8e
DM
173Returns nothing.
174
0397beb0
TC
175 my $x = "\x{100}"; # $x contains one character, with ord 0x100
176 utf8::encode($x); # $x contains two characters, with ords (on
a04477f8
KW
177 # ASCII platforms) 0xc4 and 0x80. On EBCDIC
178 # 1047, this would instead be 0x8C and 0x41.
78ea37eb 179
0397beb0
TC
180Similar to:
181
182 use Encode;
183 $x = Encode::encode("utf8", $x);
184
a04477f8
KW
185B<Note that this function does not handle arbitrary encodings>;
186use L<Encode> instead.
094ce63c 187
308a4ae1 188=item * C<$success = utf8::decode($string)>
1b026014 189
a04477f8
KW
190(Since Perl v5.8.0)
191Attempts to convert in-place the octet sequence encoded as UTF-8 to the
836ccc8e 192corresponding character sequence. That is, it replaces each sequence of
a04477f8 193characters in the string whose ords represent a valid UTF-8 byte
836ccc8e 194sequence, with the corresponding single character. The UTF-8 flag is
a04477f8
KW
195turned on only if the source string contains multiple-byte UTF-8
196characters. If I<$string> is invalid as UTF-8, returns false;
836ccc8e
DM
197otherwise returns true.
198
0397beb0 199 my $x = "\xc4\x80"; # $x contains two characters, with ords
ca3d51ba 200 # 0xc4 and 0x80
0397beb0 201 utf8::decode($x); # On ASCII platforms, $x contains one char,
a04477f8 202 # with ord 0x100. Since these bytes aren't
0397beb0 203 # legal UTF-EBCDIC, on EBCDIC platforms, $x is
a04477f8 204 # unchanged and the function returns FALSE.
78ea37eb 205
a04477f8
KW
206B<Note that this function does not handle arbitrary encodings>;
207use L<Encode> instead.
78ea37eb 208
ca3d51ba
KW
209=item * C<$unicode = utf8::native_to_unicode($code_point)>
210
273e254d 211(Since Perl v5.8.0)
ca3d51ba
KW
212This takes an unsigned integer (which represents the ordinal number of a
213character (or a code point) on the platform the program is being run on) and
214returns its Unicode equivalent value. Since ASCII platforms natively use the
215Unicode code points, this function returns its input on them. On EBCDIC
bc1767aa 216platforms it converts from EBCDIC to Unicode.
ca3d51ba
KW
217
218A meaningless value will currently be returned if the input is not an unsigned
219integer.
220
273e254d
KW
221Since Perl v5.22.0, calls to this function are optimized out on ASCII
222platforms, so there is no performance hit in using it there.
223
ca3d51ba
KW
224=item * C<$native = utf8::unicode_to_native($code_point)>
225
273e254d 226(Since Perl v5.8.0)
ca3d51ba
KW
227This is the inverse of C<utf8::native_to_unicode()>, converting the other
228direction. Again, on ASCII platforms, this returns its input, but on EBCDIC
229platforms it will find the native platform code point, given any Unicode one.
230
231A meaningless value will currently be returned if the input is not an unsigned
232integer.
233
273e254d
KW
234Since Perl v5.22.0, calls to this function are optimized out on ASCII
235platforms, so there is no performance hit in using it there.
236
308a4ae1 237=item * C<$flag = utf8::is_utf8($string)>
8800c35a 238
ac8b87d7 239(Since Perl 5.8.1) Test whether I<$string> is marked internally as encoded in
0397beb0
TC
240UTF-8. Functionally the same as C<Encode::is_utf8($string)>.
241
242Typically only necessary for debugging and testing, if you need to
243dump the internals of an SV, L<Devel::Peek's|Devel::Peek> Dump()
244provides more detail in a compact form.
245
246If you still think you need this outside of debugging, testing or
247dealing with filenames, you should probably read L<perlunitut> and
248L<perlunifaq/What is "the UTF8 flag"?>.
249
250Don't use this flag as a marker to distinguish character and binary
251data, that should be decided for each variable when you write your
252code.
253
254To force unicode semantics in code portable to perl 5.8 and 5.10, call
255C<utf8::upgrade($string)> unconditionally.
8800c35a 256
308a4ae1 257=item * C<$flag = utf8::valid($string)>
70122e76 258
ac8b87d7 259[INTERNAL] Test whether I<$string> is in a consistent state regarding
9a54da5c 260UTF-8. Will return true if it is well-formed UTF-8 and has the UTF-8 flag
ac8b87d7 261on B<or> if I<$string> is held as bytes (both these states are 'consistent').
637ec54e 262Main reason for this routine is to allow Perl's test suite to check
0397beb0 263that operations have left strings in a consistent state.
70122e76 264
1b026014
NIS
265=back
266
7d865a91 267C<utf8::encode> is like C<utf8::upgrade>, but the UTF8 flag is
a04477f8
KW
268cleared. See L<perlunicode>, and the C API
269functions C<L<sv_utf8_upgrade|perlapi/sv_utf8_upgrade>>,
270C<L<perlapi/sv_utf8_downgrade>>, C<L<perlapi/sv_utf8_encode>>,
271and C<L<perlapi/sv_utf8_decode>>, which are wrapped by the Perl functions
094ce63c 272C<utf8::upgrade>, C<utf8::downgrade>, C<utf8::encode> and
a04477f8
KW
273C<utf8::decode>. Also, the functions C<utf8::is_utf8>, C<utf8::valid>,
274C<utf8::encode>, C<utf8::decode>, C<utf8::upgrade>, and C<utf8::downgrade> are
7edb8f2b
RGS
275actually internal, and thus always available, without a C<require utf8>
276statement.
f1e62f77 277
8f8cf39c
JH
278=head1 BUGS
279
a04477f8
KW
280Some filesystems may not support UTF-8 file names, or they may be supported
281incompatibly with Perl. Therefore UTF-8 names that are visible to the
282filesystem, such as module names may not work.
8f8cf39c 283
393fec97 284=head1 SEE ALSO
a0ed51b3 285
2575c402 286L<perlunitut>, L<perluniintro>, L<perlrun>, L<bytes>, L<perlunicode>
a0ed51b3
LW
287
288=cut