This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Further clarification about safe pipe opens.
[perl5.git] / lib / utf8.pm
CommitLineData
a0ed51b3
LW
1package utf8;
2
d5448623
GS
3$utf8::hint_bits = 0x00800000;
4
b75c8c73
MS
5our $VERSION = '1.00';
6
a0ed51b3 7sub import {
d5448623 8 $^H |= $utf8::hint_bits;
a0ed51b3
LW
9 $enc{caller()} = $_[1] if $_[1];
10}
11
12sub unimport {
d5448623 13 $^H &= ~$utf8::hint_bits;
a0ed51b3
LW
14}
15
16sub AUTOLOAD {
17 require "utf8_heavy.pl";
daf4d4ea
SC
18 goto &$AUTOLOAD if defined &$AUTOLOAD;
19 Carp::croak("Undefined subroutine $AUTOLOAD called");
a0ed51b3
LW
20}
21
221;
23__END__
24
25=head1 NAME
26
b3419ed8 27utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code
a0ed51b3
LW
28
29=head1 SYNOPSIS
30
31 use utf8;
32 no utf8;
33
34=head1 DESCRIPTION
35
393fec97 36The C<use utf8> pragma tells the Perl parser to allow UTF-8 in the
b3419ed8 37program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based
70122e76 38platforms). The C<no utf8> pragma tells Perl to switch back to treating
b3419ed8 39the source text as literal bytes in the current lexical scope.
a0ed51b3 40
393fec97
GS
41This pragma is primarily a compatibility device. Perl versions
42earlier than 5.6 allowed arbitrary bytes in source code, whereas
43in future we would like to standardize on the UTF-8 encoding for
44source text. Until UTF-8 becomes the default format for source
45text, this pragma should be used to recognize UTF-8 in the source.
46When UTF-8 becomes the standard source format, this pragma will
b3419ed8 47effectively become a no-op. For convenience in what follows the
ad0029c4 48term I<UTF-X> is used to refer to UTF-8 on ASCII and ISO Latin based
b3419ed8 49platforms and UTF-EBCDIC on EBCDIC based platforms.
a0ed51b3 50
ad0029c4 51Enabling the C<utf8> pragma has the following effect:
a0ed51b3 52
4ac9195f 53=over 4
a0ed51b3
LW
54
55=item *
56
393fec97 57Bytes in the source text that have their high-bit set will be treated
ad0029c4 58as being part of a literal UTF-8 character. This includes most
c20e2abd
JH
59literals such as identifier names, string constants, and constant
60regular expression patterns. On EBCDIC platforms characters in
61the Latin 1 character set are treated as being part of a literal
ad0029c4 62UTF-EBCDIC character.
a0ed51b3 63
4ac9195f
MS
64=back
65
ae90e350
JH
66Note that if you have bytes with the eighth bit on in your script
67(for example embedded Latin-1 in your string literals), C<use utf8>
68will be unhappy since the bytes are most probably not well-formed
69UTF-8. If you want to have such bytes and use utf8, you can disable
70utf8 until the end the block (or file, if at top level) by C<no utf8;>.
71
1b026014
NIS
72=head2 Utility functions
73
74The following functions are defined in the C<utf8::> package by the perl core.
75
76=over 4
77
78=item * $num_octets = utf8::upgrade($string);
79
8dd9dd9f 80Converts (in-place) internal representation of string to Perl's internal
ad0029c4 81I<UTF-X> form. Returns the number of octets necessary to represent
8dd9dd9f
A
82the string as I<UTF-X>. Can be used to make sure that the
83UTF-8 flag is on, so that C<\w> or C<lc()> work as expected on strings
84containing characters in the range 0x80-0xFF. Note that this should
85not be used to convert
13a6c0e0
JH
86a legacy byte encoding to Unicode: use Encode for that. Affected
87by the encoding pragma.
1b026014 88
8dd9dd9f 89=item * utf8::downgrade($string[, FAIL_OK])
1b026014 90
7d865a91
JH
91Converts (in-place) internal representation of string to be un-encoded
92bytes. Returns true on success. On failure dies or, if the value of
93FAIL_OK is true, returns false. Can be used to make sure that the
8dd9dd9f
A
94UTF-8 flag is off, e.g. when you want to make sure that the substr()
95or length() function works with the usually faster byte algorithm.
13a6c0e0
JH
96Note that this should not be used to convert Unicode back to a legacy
97byte encoding: use Encode for that. B<Not> affected by the encoding
98pragma.
1b026014
NIS
99
100=item * utf8::encode($string)
101
13a6c0e0 102Converts (in-place) I<$string> from logical characters to octet
8dd9dd9f 103sequence representing it in Perl's I<UTF-X> encoding. Same as
7d865a91
JH
104Encode::encode_utf8(). Note that this should not be used to convert
105a legacy byte encoding to Unicode: use Encode for that.
094ce63c
AT
106
107=item * $flag = utf8::decode($string)
1b026014 108
ad0029c4 109Attempts to convert I<$string> in-place from Perl's I<UTF-X> encoding
7d865a91
JH
110into logical characters. Same as Encode::decode_utf8(). Note that this
111should not be used to convert Unicode back to a legacy byte encoding:
112use Encode for that.
1b026014 113
70122e76
JH
114=item * $flag = utf8::valid(STRING)
115
116[INTERNAL] Test whether STRING is in a consistent state. Will return
117true if string is held as bytes, or is well-formed UTF-8 and has the
118UTF-8 flag on. Main reason for this routine is to allow Perl's
119testsuite to check that operations have left strings in a consistent
120state.
121
1b026014
NIS
122=back
123
7d865a91
JH
124C<utf8::encode> is like C<utf8::upgrade>, but the UTF8 flag is
125cleared. See L<perlunicode> for more on the UTF8 flag and the C API
126functions C<sv_utf8_upgrade>, C<sv_utf8_downgrade>, C<sv_utf8_encode>,
094ce63c
AT
127and C<sv_utf8_decode>, which are wrapped by the Perl functions
128C<utf8::upgrade>, C<utf8::downgrade>, C<utf8::encode> and
7d865a91
JH
129C<utf8::decode>. Note that in the Perl 5.8.0 implementation the
130functions utf8::valid, utf8::encode, utf8::decode, utf8::upgrade,
131and utf8::downgrade are always available, without a C<require utf8>
132statement-- this may change in future releases.
f1e62f77 133
393fec97 134=head1 SEE ALSO
a0ed51b3 135
8058d7ab 136L<perlunicode>, L<bytes>
a0ed51b3
LW
137
138=cut