This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Pod::Html test clean-up (was Re: maint @ 20617 (on VMS))
[perl5.git] / lib / utf8.pm
CommitLineData
a0ed51b3
LW
1package utf8;
2
d5448623
GS
3$utf8::hint_bits = 0x00800000;
4
973655a8 5our $VERSION = '1.02';
b75c8c73 6
a0ed51b3 7sub import {
d5448623 8 $^H |= $utf8::hint_bits;
a0ed51b3
LW
9 $enc{caller()} = $_[1] if $_[1];
10}
11
12sub unimport {
d5448623 13 $^H &= ~$utf8::hint_bits;
a0ed51b3
LW
14}
15
16sub AUTOLOAD {
17 require "utf8_heavy.pl";
daf4d4ea
SC
18 goto &$AUTOLOAD if defined &$AUTOLOAD;
19 Carp::croak("Undefined subroutine $AUTOLOAD called");
a0ed51b3
LW
20}
21
221;
23__END__
24
25=head1 NAME
26
b3419ed8 27utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code
a0ed51b3
LW
28
29=head1 SYNOPSIS
30
31 use utf8;
32 no utf8;
33
973655a8
SB
34 $num_octets = utf8::upgrade($string);
35 $success = utf8::downgrade($string[, FAIL_OK]);
36
37 utf8::encode($string);
38 utf8::decode($string);
39
40 $flag = utf8::is_utf8(STRING);
41 $flag = utf8::valid(STRING);
42
a0ed51b3
LW
43=head1 DESCRIPTION
44
393fec97 45The C<use utf8> pragma tells the Perl parser to allow UTF-8 in the
b3419ed8 46program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based
70122e76 47platforms). The C<no utf8> pragma tells Perl to switch back to treating
b3419ed8 48the source text as literal bytes in the current lexical scope.
a0ed51b3 49
393fec97
GS
50This pragma is primarily a compatibility device. Perl versions
51earlier than 5.6 allowed arbitrary bytes in source code, whereas
52in future we would like to standardize on the UTF-8 encoding for
53source text. Until UTF-8 becomes the default format for source
54text, this pragma should be used to recognize UTF-8 in the source.
55When UTF-8 becomes the standard source format, this pragma will
b3419ed8 56effectively become a no-op. For convenience in what follows the
ad0029c4 57term I<UTF-X> is used to refer to UTF-8 on ASCII and ISO Latin based
b3419ed8 58platforms and UTF-EBCDIC on EBCDIC based platforms.
a0ed51b3 59
ad0029c4 60Enabling the C<utf8> pragma has the following effect:
a0ed51b3 61
4ac9195f 62=over 4
a0ed51b3
LW
63
64=item *
65
393fec97 66Bytes in the source text that have their high-bit set will be treated
ad0029c4 67as being part of a literal UTF-8 character. This includes most
c20e2abd 68literals such as identifier names, string constants, and constant
8f8cf39c
JH
69regular expression patterns.
70
71On EBCDIC platforms characters in the Latin 1 character set are
72treated as being part of a literal UTF-EBCDIC character.
a0ed51b3 73
4ac9195f
MS
74=back
75
ae90e350
JH
76Note that if you have bytes with the eighth bit on in your script
77(for example embedded Latin-1 in your string literals), C<use utf8>
78will be unhappy since the bytes are most probably not well-formed
79UTF-8. If you want to have such bytes and use utf8, you can disable
80utf8 until the end the block (or file, if at top level) by C<no utf8;>.
81
1b026014
NIS
82=head2 Utility functions
83
8800c35a
JH
84The following functions are defined in the C<utf8::> package by the
85Perl core. You do not need to say C<use utf8> to use these and in fact
86you should not unless you really want to have UTF-8 source code.
1b026014
NIS
87
88=over 4
89
973655a8 90=item * $num_octets = utf8::upgrade($string)
1b026014 91
6e37fd2a
JH
92Converts (in-place) internal representation of string to Perl's
93internal I<UTF-X> form. Returns the number of octets necessary to
94represent the string as I<UTF-X>. Can be used to make sure that the
8dd9dd9f 95UTF-8 flag is on, so that C<\w> or C<lc()> work as expected on strings
6e37fd2a
JH
96containing characters in the range 0x80-0xFF (oon ASCII and
97derivatives). Note that this should not be used to convert a legacy
98byte encoding to Unicode: use Encode for that. Affected by the
99encoding pragma.
1b026014 100
973655a8 101=item * $success = utf8::downgrade($string[, FAIL_OK])
1b026014 102
7d865a91
JH
103Converts (in-place) internal representation of string to be un-encoded
104bytes. Returns true on success. On failure dies or, if the value of
105FAIL_OK is true, returns false. Can be used to make sure that the
8dd9dd9f
A
106UTF-8 flag is off, e.g. when you want to make sure that the substr()
107or length() function works with the usually faster byte algorithm.
13a6c0e0
JH
108Note that this should not be used to convert Unicode back to a legacy
109byte encoding: use Encode for that. B<Not> affected by the encoding
110pragma.
1b026014
NIS
111
112=item * utf8::encode($string)
113
13a6c0e0 114Converts (in-place) I<$string> from logical characters to octet
6e37fd2a
JH
115sequence representing it in Perl's I<UTF-X> encoding. Returns
116nothing. Same as Encode::encode_utf8(). Note that this should not be
117used to convert a legacy byte encoding to Unicode: use Encode for
118that.
094ce63c 119
973655a8 120=item * utf8::decode($string)
1b026014 121
ad0029c4 122Attempts to convert I<$string> in-place from Perl's I<UTF-X> encoding
6e37fd2a
JH
123into logical characters. Returns nothing. Same as Encode::decode_utf8().
124Note that this should not be used to convert Unicode back to a legacy
125byte encoding: use Encode for that.
1b026014 126
8800c35a
JH
127=item * $flag = utf8::is_utf8(STRING)
128
6e37fd2a 129Test whether STRING is in UTF-8. Same as Encode::is_utf8().
8800c35a 130
70122e76
JH
131=item * $flag = utf8::valid(STRING)
132
8800c35a
JH
133[INTERNAL] Test whether STRING is in a consistent state regarding
134UTF-8. Will return true is well-formed UTF-8 and has the UTF-8 flag
135on B<or> if string is held as bytes (both these states are 'consistent').
136Main reason for this routine is to allow Perl's testsuite to check
137that operations have left strings in a consistent state. You most
138probably want to use utf8::is_utf8() instead.
70122e76 139
1b026014
NIS
140=back
141
7d865a91
JH
142C<utf8::encode> is like C<utf8::upgrade>, but the UTF8 flag is
143cleared. See L<perlunicode> for more on the UTF8 flag and the C API
144functions C<sv_utf8_upgrade>, C<sv_utf8_downgrade>, C<sv_utf8_encode>,
094ce63c
AT
145and C<sv_utf8_decode>, which are wrapped by the Perl functions
146C<utf8::upgrade>, C<utf8::downgrade>, C<utf8::encode> and
7d865a91 147C<utf8::decode>. Note that in the Perl 5.8.0 implementation the
6e37fd2a
JH
148functions utf8::is_utf8, utf8::valid, utf8::encode, utf8::decode,
149utf8::upgrade, and utf8::downgrade are always available, without a
150C<require utf8> statement-- this may change in future releases.
f1e62f77 151
8f8cf39c
JH
152=head1 BUGS
153
154One can have Unicode in identifier names, but not in package/class or
155subroutine names. While some limited functionality towards this does
156exist as of Perl 5.8.0, that is more accidental than designed; use of
157Unicode for the said purposes is unsupported.
158
159One reason of this unfinishedness is its (currently) inherent
160unportability: since both package names and subroutine names may need
161to be mapped to file and directory names, the Unicode capability of
162the filesystem becomes important-- and there unfortunately aren't
163portable answers.
164
393fec97 165=head1 SEE ALSO
a0ed51b3 166
8058d7ab 167L<perlunicode>, L<bytes>
a0ed51b3
LW
168
169=cut