This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Slight tweaks on the length() and chr() entries,
[perl5.git] / lib / utf8.pm
CommitLineData
a0ed51b3
LW
1package utf8;
2
d5448623
GS
3$utf8::hint_bits = 0x00800000;
4
b75c8c73
MS
5our $VERSION = '1.00';
6
a0ed51b3 7sub import {
d5448623 8 $^H |= $utf8::hint_bits;
a0ed51b3
LW
9 $enc{caller()} = $_[1] if $_[1];
10}
11
12sub unimport {
d5448623 13 $^H &= ~$utf8::hint_bits;
a0ed51b3
LW
14}
15
16sub AUTOLOAD {
17 require "utf8_heavy.pl";
daf4d4ea
SC
18 goto &$AUTOLOAD if defined &$AUTOLOAD;
19 Carp::croak("Undefined subroutine $AUTOLOAD called");
a0ed51b3
LW
20}
21
221;
23__END__
24
25=head1 NAME
26
b3419ed8 27utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code
a0ed51b3
LW
28
29=head1 SYNOPSIS
30
31 use utf8;
32 no utf8;
33
34=head1 DESCRIPTION
35
393fec97 36The C<use utf8> pragma tells the Perl parser to allow UTF-8 in the
b3419ed8 37program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based
70122e76 38platforms). The C<no utf8> pragma tells Perl to switch back to treating
b3419ed8 39the source text as literal bytes in the current lexical scope.
a0ed51b3 40
393fec97
GS
41This pragma is primarily a compatibility device. Perl versions
42earlier than 5.6 allowed arbitrary bytes in source code, whereas
43in future we would like to standardize on the UTF-8 encoding for
44source text. Until UTF-8 becomes the default format for source
45text, this pragma should be used to recognize UTF-8 in the source.
46When UTF-8 becomes the standard source format, this pragma will
b3419ed8 47effectively become a no-op. For convenience in what follows the
ad0029c4 48term I<UTF-X> is used to refer to UTF-8 on ASCII and ISO Latin based
b3419ed8 49platforms and UTF-EBCDIC on EBCDIC based platforms.
a0ed51b3 50
ad0029c4 51Enabling the C<utf8> pragma has the following effect:
a0ed51b3 52
4ac9195f 53=over 4
a0ed51b3
LW
54
55=item *
56
393fec97 57Bytes in the source text that have their high-bit set will be treated
ad0029c4 58as being part of a literal UTF-8 character. This includes most
c20e2abd 59literals such as identifier names, string constants, and constant
8f8cf39c
JH
60regular expression patterns.
61
62On EBCDIC platforms characters in the Latin 1 character set are
63treated as being part of a literal UTF-EBCDIC character.
a0ed51b3 64
4ac9195f
MS
65=back
66
ae90e350
JH
67Note that if you have bytes with the eighth bit on in your script
68(for example embedded Latin-1 in your string literals), C<use utf8>
69will be unhappy since the bytes are most probably not well-formed
70UTF-8. If you want to have such bytes and use utf8, you can disable
71utf8 until the end the block (or file, if at top level) by C<no utf8;>.
72
1b026014
NIS
73=head2 Utility functions
74
75The following functions are defined in the C<utf8::> package by the perl core.
76
77=over 4
78
79=item * $num_octets = utf8::upgrade($string);
80
8dd9dd9f 81Converts (in-place) internal representation of string to Perl's internal
ad0029c4 82I<UTF-X> form. Returns the number of octets necessary to represent
8dd9dd9f
A
83the string as I<UTF-X>. Can be used to make sure that the
84UTF-8 flag is on, so that C<\w> or C<lc()> work as expected on strings
85containing characters in the range 0x80-0xFF. Note that this should
86not be used to convert
13a6c0e0
JH
87a legacy byte encoding to Unicode: use Encode for that. Affected
88by the encoding pragma.
1b026014 89
8dd9dd9f 90=item * utf8::downgrade($string[, FAIL_OK])
1b026014 91
7d865a91
JH
92Converts (in-place) internal representation of string to be un-encoded
93bytes. Returns true on success. On failure dies or, if the value of
94FAIL_OK is true, returns false. Can be used to make sure that the
8dd9dd9f
A
95UTF-8 flag is off, e.g. when you want to make sure that the substr()
96or length() function works with the usually faster byte algorithm.
13a6c0e0
JH
97Note that this should not be used to convert Unicode back to a legacy
98byte encoding: use Encode for that. B<Not> affected by the encoding
99pragma.
1b026014
NIS
100
101=item * utf8::encode($string)
102
13a6c0e0 103Converts (in-place) I<$string> from logical characters to octet
8dd9dd9f 104sequence representing it in Perl's I<UTF-X> encoding. Same as
7d865a91
JH
105Encode::encode_utf8(). Note that this should not be used to convert
106a legacy byte encoding to Unicode: use Encode for that.
094ce63c
AT
107
108=item * $flag = utf8::decode($string)
1b026014 109
ad0029c4 110Attempts to convert I<$string> in-place from Perl's I<UTF-X> encoding
7d865a91
JH
111into logical characters. Same as Encode::decode_utf8(). Note that this
112should not be used to convert Unicode back to a legacy byte encoding:
113use Encode for that.
1b026014 114
70122e76
JH
115=item * $flag = utf8::valid(STRING)
116
117[INTERNAL] Test whether STRING is in a consistent state. Will return
118true if string is held as bytes, or is well-formed UTF-8 and has the
119UTF-8 flag on. Main reason for this routine is to allow Perl's
120testsuite to check that operations have left strings in a consistent
121state.
122
1b026014
NIS
123=back
124
7d865a91
JH
125C<utf8::encode> is like C<utf8::upgrade>, but the UTF8 flag is
126cleared. See L<perlunicode> for more on the UTF8 flag and the C API
127functions C<sv_utf8_upgrade>, C<sv_utf8_downgrade>, C<sv_utf8_encode>,
094ce63c
AT
128and C<sv_utf8_decode>, which are wrapped by the Perl functions
129C<utf8::upgrade>, C<utf8::downgrade>, C<utf8::encode> and
7d865a91
JH
130C<utf8::decode>. Note that in the Perl 5.8.0 implementation the
131functions utf8::valid, utf8::encode, utf8::decode, utf8::upgrade,
132and utf8::downgrade are always available, without a C<require utf8>
133statement-- this may change in future releases.
f1e62f77 134
8f8cf39c
JH
135=head1 BUGS
136
137One can have Unicode in identifier names, but not in package/class or
138subroutine names. While some limited functionality towards this does
139exist as of Perl 5.8.0, that is more accidental than designed; use of
140Unicode for the said purposes is unsupported.
141
142One reason of this unfinishedness is its (currently) inherent
143unportability: since both package names and subroutine names may need
144to be mapped to file and directory names, the Unicode capability of
145the filesystem becomes important-- and there unfortunately aren't
146portable answers.
147
393fec97 148=head1 SEE ALSO
a0ed51b3 149
8058d7ab 150L<perlunicode>, L<bytes>
a0ed51b3
LW
151
152=cut