This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Remove unused AUTOGEN_FILES Makefile variable
[perl5.git] / lib / utf8.pm
CommitLineData
a0ed51b3
LW
1package utf8;
2
d5448623
GS
3$utf8::hint_bits = 0x00800000;
4
50a85cfe 5our $VERSION = '1.21';
b75c8c73 6
a0ed51b3 7sub import {
d5448623 8 $^H |= $utf8::hint_bits;
a0ed51b3
LW
9}
10
11sub unimport {
d5448623 12 $^H &= ~$utf8::hint_bits;
a0ed51b3
LW
13}
14
15sub AUTOLOAD {
16 require "utf8_heavy.pl";
daf4d4ea 17 goto &$AUTOLOAD if defined &$AUTOLOAD;
bd7017d3 18 require Carp;
daf4d4ea 19 Carp::croak("Undefined subroutine $AUTOLOAD called");
a0ed51b3
LW
20}
21
221;
23__END__
24
25=head1 NAME
26
b3419ed8 27utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code
a0ed51b3
LW
28
29=head1 SYNOPSIS
30
291cc134
KW
31 use utf8;
32 no utf8;
a0ed51b3 33
291cc134 34 # Convert the internal representation of a Perl scalar to/from UTF-8.
836ccc8e 35
291cc134 36 $num_octets = utf8::upgrade($string);
98695e13 37 $success = utf8::downgrade($string[, $fail_ok]);
973655a8 38
291cc134
KW
39 # Change each character of a Perl scalar to/from a series of
40 # characters that represent the UTF-8 bytes of each original character.
836ccc8e 41
291cc134
KW
42 utf8::encode($string); # "\x{100}" becomes "\xc4\x80"
43 utf8::decode($string); # "\xc4\x80" becomes "\x{100}"
973655a8 44
ca3d51ba
KW
45 # Convert a code point from the platform native character set to
46 # Unicode, and vice-versa.
47 $unicode = utf8::native_to_unicode(ord('A')); # returns 65 on both
48 # ASCII and EBCDIC
49 # platforms
a04477f8
KW
50 $native = utf8::unicode_to_native(65); # returns 65 on ASCII
51 # platforms; 193 on
52 # EBCDIC
ca3d51ba 53
ac8b87d7
EB
54 $flag = utf8::is_utf8($string); # since Perl 5.8.1
55 $flag = utf8::valid($string);
973655a8 56
a0ed51b3
LW
57=head1 DESCRIPTION
58
393fec97 59The C<use utf8> pragma tells the Perl parser to allow UTF-8 in the
a04477f8
KW
60program text in the current lexical scope. The C<no utf8> pragma tells Perl
61to switch back to treating the source text as literal bytes in the current
62lexical scope. (On EBCDIC platforms, technically it is allowing UTF-EBCDIC,
63and not UTF-8, but this distinction is academic, so in this document the term
64UTF-8 is used to mean both).
a0ed51b3 65
19b49582
JH
66B<Do not use this pragma for anything else than telling Perl that your
67script is written in UTF-8.> The utility functions described below are
2575c402
JW
68directly usable without C<use utf8;>.
69
70Because it is not possible to reliably tell UTF-8 from native 8 bit
71encodings, you need either a Byte Order Mark at the beginning of your
72source code, or C<use utf8;>, to instruct perl.
19b49582 73
2575c402 74When UTF-8 becomes the standard source format, this pragma will
a04477f8 75effectively become a no-op.
a0ed51b3 76
a74e8b45 77See also the effects of the C<-C> switch and its cousin, the
127161e0 78C<PERL_UNICODE> environment variable, in L<perlrun>.
a74e8b45 79
ad0029c4 80Enabling the C<utf8> pragma has the following effect:
a0ed51b3 81
4ac9195f 82=over 4
a0ed51b3
LW
83
84=item *
85
a04477f8
KW
86Bytes in the source text that are not in the ASCII character set will be
87treated as being part of a literal UTF-8 sequence. This includes most
c20e2abd 88literals such as identifier names, string constants, and constant
8f8cf39c
JH
89regular expression patterns.
90
4ac9195f
MS
91=back
92
a04477f8
KW
93Note that if you have non-ASCII, non-UTF-8 bytes in your script (for example
94embedded Latin-1 in your string literals), C<use utf8> will be unhappy. If
95you want to have such bytes under C<use utf8>, you can disable this pragma
96until the end the block (or file, if at top level) by C<no utf8;>.
ae90e350 97
1b026014
NIS
98=head2 Utility functions
99
8800c35a
JH
100The following functions are defined in the C<utf8::> package by the
101Perl core. You do not need to say C<use utf8> to use these and in fact
2f7e5073 102you should not say that unless you really want to have UTF-8 source code.
1b026014
NIS
103
104=over 4
105
308a4ae1 106=item * C<$num_octets = utf8::upgrade($string)>
1b026014 107
a04477f8 108(Since Perl v5.8.0)
836ccc8e 109Converts in-place the internal representation of the string from an octet
a04477f8 110sequence in the native encoding (Latin-1 or EBCDIC) to UTF-8. The
836ccc8e 111logical character sequence itself is unchanged. If I<$string> is already
0397beb0
TC
112upgraded, then this is a no-op. Returns the
113number of octets necessary to represent the string as UTF-8.
114
115If your code needs to be compatible with versions of perl without
116C<use feature 'unicode_strings';>, you can force Unicode semantics on
117a given string:
118
119 # force unicode semantics for $string without the
120 # "unicode_strings" feature
121 utf8::upgrade($string);
122
123For example:
124
125 # without explicit or implicit use feature 'unicode_strings'
126 my $x = "\xDF"; # LATIN SMALL LETTER SHARP S
127 $x =~ /ss/i; # won't match
128 my $y = uc($x); # won't convert
129 utf8::upgrade($x);
130 $x =~ /ss/i; # matches
131 my $z = uc($x); # converts to "SS"
78ea37eb 132
a04477f8
KW
133B<Note that this function does not handle arbitrary encodings>;
134use L<Encode> instead.
1b026014 135
308a4ae1 136=item * C<$success = utf8::downgrade($string[, $fail_ok])>
1b026014 137
a04477f8 138(Since Perl v5.8.0)
50a85cfe
KW
139Converts in-place the internal representation of the string from UTF-8 to the
140equivalent octet sequence in the native encoding (Latin-1 or EBCDIC). The
141logical character sequence itself is unchanged. If I<$string> is already
142stored as native 8 bit, then this is a no-op. Can be used to make sure that
143the UTF-8 flag is off, e.g. when you want to make sure that the substr() or
144length() function works with the usually faster byte algorithm.
78ea37eb 145
a04477f8 146Fails if the original UTF-8 sequence cannot be represented in the
ac8b87d7 147native 8 bit encoding. On failure dies or, if the value of I<$fail_ok> is
2575c402 148true, returns false.
78ea37eb 149
2575c402
JW
150Returns true on success.
151
0397beb0
TC
152If your code expects an octet sequence this can be used to validate
153that you've received one:
154
155 # throw an exception if not representable as octets
156 utf8::downgrade($string)
157
158 # or do your own error handling
159 utf8::downgrade($string, 1) or die "string must be octets";
160
a04477f8
KW
161B<Note that this function does not handle arbitrary encodings>;
162use L<Encode> instead.
78ea37eb 163
308a4ae1 164=item * C<utf8::encode($string)>
1b026014 165
a04477f8 166(Since Perl v5.8.0)
2575c402 167Converts in-place the character sequence to the corresponding octet
50a85cfe
KW
168sequence in Perl's extended UTF-8. That is, every (possibly wide) character
169gets replaced with a sequence of one or more characters that represent the
a04477f8 170individual UTF-8 bytes of the character. The UTF8 flag is turned off.
836ccc8e
DM
171Returns nothing.
172
0397beb0
TC
173 my $x = "\x{100}"; # $x contains one character, with ord 0x100
174 utf8::encode($x); # $x contains two characters, with ords (on
a04477f8
KW
175 # ASCII platforms) 0xc4 and 0x80. On EBCDIC
176 # 1047, this would instead be 0x8C and 0x41.
78ea37eb 177
0397beb0
TC
178Similar to:
179
180 use Encode;
181 $x = Encode::encode("utf8", $x);
182
a04477f8
KW
183B<Note that this function does not handle arbitrary encodings>;
184use L<Encode> instead.
094ce63c 185
308a4ae1 186=item * C<$success = utf8::decode($string)>
1b026014 187
a04477f8 188(Since Perl v5.8.0)
50a85cfe
KW
189Attempts to convert in-place the octet sequence encoded in Perl's extended
190UTF-8 to the corresponding character sequence. That is, it replaces each
191sequence of characters in the string whose ords represent a valid (extended)
192UTF-8 byte sequence, with the corresponding single character. The UTF-8 flag
193is turned on only if the source string contains multiple-byte UTF-8
194characters. If I<$string> is invalid as extended UTF-8, returns false;
836ccc8e
DM
195otherwise returns true.
196
0397beb0 197 my $x = "\xc4\x80"; # $x contains two characters, with ords
ca3d51ba 198 # 0xc4 and 0x80
0397beb0 199 utf8::decode($x); # On ASCII platforms, $x contains one char,
a04477f8 200 # with ord 0x100. Since these bytes aren't
0397beb0 201 # legal UTF-EBCDIC, on EBCDIC platforms, $x is
a04477f8 202 # unchanged and the function returns FALSE.
78ea37eb 203
a04477f8
KW
204B<Note that this function does not handle arbitrary encodings>;
205use L<Encode> instead.
78ea37eb 206
ca3d51ba
KW
207=item * C<$unicode = utf8::native_to_unicode($code_point)>
208
273e254d 209(Since Perl v5.8.0)
ca3d51ba
KW
210This takes an unsigned integer (which represents the ordinal number of a
211character (or a code point) on the platform the program is being run on) and
212returns its Unicode equivalent value. Since ASCII platforms natively use the
213Unicode code points, this function returns its input on them. On EBCDIC
bc1767aa 214platforms it converts from EBCDIC to Unicode.
ca3d51ba
KW
215
216A meaningless value will currently be returned if the input is not an unsigned
217integer.
218
273e254d
KW
219Since Perl v5.22.0, calls to this function are optimized out on ASCII
220platforms, so there is no performance hit in using it there.
221
ca3d51ba
KW
222=item * C<$native = utf8::unicode_to_native($code_point)>
223
273e254d 224(Since Perl v5.8.0)
ca3d51ba
KW
225This is the inverse of C<utf8::native_to_unicode()>, converting the other
226direction. Again, on ASCII platforms, this returns its input, but on EBCDIC
227platforms it will find the native platform code point, given any Unicode one.
228
229A meaningless value will currently be returned if the input is not an unsigned
230integer.
231
273e254d
KW
232Since Perl v5.22.0, calls to this function are optimized out on ASCII
233platforms, so there is no performance hit in using it there.
234
308a4ae1 235=item * C<$flag = utf8::is_utf8($string)>
8800c35a 236
ac8b87d7 237(Since Perl 5.8.1) Test whether I<$string> is marked internally as encoded in
0397beb0
TC
238UTF-8. Functionally the same as C<Encode::is_utf8($string)>.
239
240Typically only necessary for debugging and testing, if you need to
241dump the internals of an SV, L<Devel::Peek's|Devel::Peek> Dump()
242provides more detail in a compact form.
243
244If you still think you need this outside of debugging, testing or
245dealing with filenames, you should probably read L<perlunitut> and
246L<perlunifaq/What is "the UTF8 flag"?>.
247
248Don't use this flag as a marker to distinguish character and binary
249data, that should be decided for each variable when you write your
250code.
251
252To force unicode semantics in code portable to perl 5.8 and 5.10, call
253C<utf8::upgrade($string)> unconditionally.
8800c35a 254
308a4ae1 255=item * C<$flag = utf8::valid($string)>
70122e76 256
ac8b87d7 257[INTERNAL] Test whether I<$string> is in a consistent state regarding
8b4b6c86
KW
258UTF-8. Will return true if it is well-formed Perl extended UTF-8 and has the
259UTF-8 flag
ac8b87d7 260on B<or> if I<$string> is held as bytes (both these states are 'consistent').
637ec54e 261Main reason for this routine is to allow Perl's test suite to check
0397beb0 262that operations have left strings in a consistent state.
70122e76 263
1b026014
NIS
264=back
265
7d865a91 266C<utf8::encode> is like C<utf8::upgrade>, but the UTF8 flag is
a04477f8
KW
267cleared. See L<perlunicode>, and the C API
268functions C<L<sv_utf8_upgrade|perlapi/sv_utf8_upgrade>>,
269C<L<perlapi/sv_utf8_downgrade>>, C<L<perlapi/sv_utf8_encode>>,
270and C<L<perlapi/sv_utf8_decode>>, which are wrapped by the Perl functions
094ce63c 271C<utf8::upgrade>, C<utf8::downgrade>, C<utf8::encode> and
a04477f8
KW
272C<utf8::decode>. Also, the functions C<utf8::is_utf8>, C<utf8::valid>,
273C<utf8::encode>, C<utf8::decode>, C<utf8::upgrade>, and C<utf8::downgrade> are
7edb8f2b
RGS
274actually internal, and thus always available, without a C<require utf8>
275statement.
f1e62f77 276
8f8cf39c
JH
277=head1 BUGS
278
a04477f8
KW
279Some filesystems may not support UTF-8 file names, or they may be supported
280incompatibly with Perl. Therefore UTF-8 names that are visible to the
281filesystem, such as module names may not work.
8f8cf39c 282
393fec97 283=head1 SEE ALSO
a0ed51b3 284
2575c402 285L<perlunitut>, L<perluniintro>, L<perlrun>, L<bytes>, L<perlunicode>
a0ed51b3
LW
286
287=cut