This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Weak spots in the utf8 manpage
[perl5.git] / lib / utf8.pm
CommitLineData
a0ed51b3
LW
1package utf8;
2
d5448623
GS
3$utf8::hint_bits = 0x00800000;
4
b75c8c73
MS
5our $VERSION = '1.00';
6
a0ed51b3 7sub import {
d5448623 8 $^H |= $utf8::hint_bits;
a0ed51b3
LW
9 $enc{caller()} = $_[1] if $_[1];
10}
11
12sub unimport {
d5448623 13 $^H &= ~$utf8::hint_bits;
a0ed51b3
LW
14}
15
16sub AUTOLOAD {
17 require "utf8_heavy.pl";
daf4d4ea
SC
18 goto &$AUTOLOAD if defined &$AUTOLOAD;
19 Carp::croak("Undefined subroutine $AUTOLOAD called");
a0ed51b3
LW
20}
21
221;
23__END__
24
25=head1 NAME
26
b3419ed8 27utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code
a0ed51b3
LW
28
29=head1 SYNOPSIS
30
31 use utf8;
32 no utf8;
33
34=head1 DESCRIPTION
35
393fec97 36The C<use utf8> pragma tells the Perl parser to allow UTF-8 in the
b3419ed8 37program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based
70122e76 38platforms). The C<no utf8> pragma tells Perl to switch back to treating
b3419ed8 39the source text as literal bytes in the current lexical scope.
a0ed51b3 40
393fec97
GS
41This pragma is primarily a compatibility device. Perl versions
42earlier than 5.6 allowed arbitrary bytes in source code, whereas
43in future we would like to standardize on the UTF-8 encoding for
44source text. Until UTF-8 becomes the default format for source
45text, this pragma should be used to recognize UTF-8 in the source.
46When UTF-8 becomes the standard source format, this pragma will
b3419ed8 47effectively become a no-op. For convenience in what follows the
ad0029c4 48term I<UTF-X> is used to refer to UTF-8 on ASCII and ISO Latin based
b3419ed8 49platforms and UTF-EBCDIC on EBCDIC based platforms.
a0ed51b3 50
ad0029c4 51Enabling the C<utf8> pragma has the following effect:
a0ed51b3 52
4ac9195f 53=over 4
a0ed51b3
LW
54
55=item *
56
393fec97 57Bytes in the source text that have their high-bit set will be treated
ad0029c4
JH
58as being part of a literal UTF-8 character. This includes most
59literals such as identifiers, string constants, constant regular
60expression patterns and package names. On EBCDIC platforms characters
61in the Latin 1 character set are treated as being part of a literal
62UTF-EBCDIC character.
a0ed51b3 63
4ac9195f
MS
64=back
65
ae90e350
JH
66Note that if you have bytes with the eighth bit on in your script
67(for example embedded Latin-1 in your string literals), C<use utf8>
68will be unhappy since the bytes are most probably not well-formed
69UTF-8. If you want to have such bytes and use utf8, you can disable
70utf8 until the end the block (or file, if at top level) by C<no utf8;>.
71
1b026014
NIS
72=head2 Utility functions
73
74The following functions are defined in the C<utf8::> package by the perl core.
75
76=over 4
77
78=item * $num_octets = utf8::upgrade($string);
79
8dd9dd9f 80Converts (in-place) internal representation of string to Perl's internal
ad0029c4 81I<UTF-X> form. Returns the number of octets necessary to represent
8dd9dd9f
A
82the string as I<UTF-X>. Can be used to make sure that the
83UTF-8 flag is on, so that C<\w> or C<lc()> work as expected on strings
84containing characters in the range 0x80-0xFF. Note that this should
85not be used to convert
13a6c0e0
JH
86a legacy byte encoding to Unicode: use Encode for that. Affected
87by the encoding pragma.
1b026014 88
8dd9dd9f 89=item * utf8::downgrade($string[, FAIL_OK])
1b026014 90
8dd9dd9f
A
91Converts (in-place) internal representation of string to be un-encoded bytes.
92Returns true on success. On failure dies or, if the value of
93FAIL_OK is true, returns false. Can be used to make sure that the
94UTF-8 flag is off, e.g. when you want to make sure that the substr()
95or length() function works with the usually faster byte algorithm.
13a6c0e0
JH
96Note that this should not be used to convert Unicode back to a legacy
97byte encoding: use Encode for that. B<Not> affected by the encoding
98pragma.
1b026014
NIS
99
100=item * utf8::encode($string)
101
13a6c0e0 102Converts (in-place) I<$string> from logical characters to octet
8dd9dd9f
A
103sequence representing it in Perl's I<UTF-X> encoding. Same as
104Encode::encode_utf8(). Note that this
13a6c0e0 105should not be used to convert a legacy byte encoding to Unicode: use
094ce63c
AT
106Encode for that.
107
108=item * $flag = utf8::decode($string)
1b026014 109
ad0029c4 110Attempts to convert I<$string> in-place from Perl's I<UTF-X> encoding
8dd9dd9f 111into logical characters. Same as Encode::decode_utf8(). Note that this should not be used to convert
13a6c0e0 112Unicode back to a legacy byte encoding: use Encode for that.
1b026014 113
70122e76
JH
114=item * $flag = utf8::valid(STRING)
115
116[INTERNAL] Test whether STRING is in a consistent state. Will return
117true if string is held as bytes, or is well-formed UTF-8 and has the
118UTF-8 flag on. Main reason for this routine is to allow Perl's
119testsuite to check that operations have left strings in a consistent
120state.
121
1b026014
NIS
122=back
123
094ce63c
AT
124C<utf8::encode> is like C<utf8::upgrade>, but the UTF8 flag is cleared.
125See L<perlunicode> for more on the UTF8 flag and the C API functions
126C<sv_utf8_upgrade>, C<sv_utf8_downgrade>, C<sv_utf8_encode>,
127and C<sv_utf8_decode>, which are wrapped by the Perl functions
128C<utf8::upgrade>, C<utf8::downgrade>, C<utf8::encode> and
8dd9dd9f
A
129C<utf8::decode>. The functions utf8::valid, utf8::encode,
130utf8::decode, utf8::upgrade, and utf8::downgrade are always available,
131without a C<require utf8> statement.
f1e62f77 132
393fec97 133=head1 SEE ALSO
a0ed51b3 134
8058d7ab 135L<perlunicode>, L<bytes>
a0ed51b3
LW
136
137=cut