This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Have win32/makefile.mk default to gcc, and update docs
[perl5.git] / lib / utf8.pm
CommitLineData
a0ed51b3
LW
1package utf8;
2
d5448623
GS
3$utf8::hint_bits = 0x00800000;
4
20a6717b 5our $VERSION = '1.03';
b75c8c73 6
a0ed51b3 7sub import {
d5448623 8 $^H |= $utf8::hint_bits;
a0ed51b3
LW
9 $enc{caller()} = $_[1] if $_[1];
10}
11
12sub unimport {
d5448623 13 $^H &= ~$utf8::hint_bits;
a0ed51b3
LW
14}
15
16sub AUTOLOAD {
17 require "utf8_heavy.pl";
daf4d4ea
SC
18 goto &$AUTOLOAD if defined &$AUTOLOAD;
19 Carp::croak("Undefined subroutine $AUTOLOAD called");
a0ed51b3
LW
20}
21
221;
23__END__
24
25=head1 NAME
26
b3419ed8 27utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code
a0ed51b3
LW
28
29=head1 SYNOPSIS
30
31 use utf8;
32 no utf8;
33
2c9359a2 34 # Convert a Perl scalar to/from UTF-8.
973655a8
SB
35 $num_octets = utf8::upgrade($string);
36 $success = utf8::downgrade($string[, FAIL_OK]);
37
2c9359a2 38 # Change the native bytes of a Perl scalar to/from UTF-8 bytes.
973655a8
SB
39 utf8::encode($string);
40 utf8::decode($string);
41
786c9463 42 $flag = utf8::is_utf8(STRING); # since Perl 5.8.1
973655a8
SB
43 $flag = utf8::valid(STRING);
44
a0ed51b3
LW
45=head1 DESCRIPTION
46
393fec97 47The C<use utf8> pragma tells the Perl parser to allow UTF-8 in the
b3419ed8 48program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based
70122e76 49platforms). The C<no utf8> pragma tells Perl to switch back to treating
b3419ed8 50the source text as literal bytes in the current lexical scope.
a0ed51b3 51
393fec97
GS
52This pragma is primarily a compatibility device. Perl versions
53earlier than 5.6 allowed arbitrary bytes in source code, whereas
54in future we would like to standardize on the UTF-8 encoding for
63708e74
JH
55source text.
56
19b49582
JH
57B<Do not use this pragma for anything else than telling Perl that your
58script is written in UTF-8.> The utility functions described below are
59useful for their own purposes, but they are not really part of the
60"pragmatic" effect.
61
63708e74
JH
62Until UTF-8 becomes the default format for source text, either this
63pragma or the L</encoding> pragma should be used to recognize UTF-8
64in the source. When UTF-8 becomes the standard source format, this
65pragma will effectively become a no-op. For convenience in what
66follows the term I<UTF-X> is used to refer to UTF-8 on ASCII and ISO
67Latin based platforms and UTF-EBCDIC on EBCDIC based platforms.
a0ed51b3 68
a74e8b45
JH
69See also the effects of the C<-C> switch and its cousin, the
70C<$ENV{PERL_UNICODE}>, in L<perlrun>.
71
ad0029c4 72Enabling the C<utf8> pragma has the following effect:
a0ed51b3 73
4ac9195f 74=over 4
a0ed51b3
LW
75
76=item *
77
393fec97 78Bytes in the source text that have their high-bit set will be treated
ad0029c4 79as being part of a literal UTF-8 character. This includes most
c20e2abd 80literals such as identifier names, string constants, and constant
8f8cf39c
JH
81regular expression patterns.
82
83On EBCDIC platforms characters in the Latin 1 character set are
84treated as being part of a literal UTF-EBCDIC character.
a0ed51b3 85
4ac9195f
MS
86=back
87
ae90e350
JH
88Note that if you have bytes with the eighth bit on in your script
89(for example embedded Latin-1 in your string literals), C<use utf8>
90will be unhappy since the bytes are most probably not well-formed
91UTF-8. If you want to have such bytes and use utf8, you can disable
92utf8 until the end the block (or file, if at top level) by C<no utf8;>.
93
63708e74
JH
94If you want to automatically upgrade your 8-bit legacy bytes to UTF-8,
95use the L</encoding> pragma instead of this pragma. For example, if
96you want to implicitly upgrade your ISO 8859-1 (Latin-1) bytes to UTF-8
97as used in e.g. C<chr()> and C<\x{...}>, try this:
98
99 use encoding "latin-1";
100 my $c = chr(0xc4);
101 my $x = "\x{c5}";
102
103In case you are wondering: yes, C<use encoding 'utf8';> works much
104the same as C<use utf8;>.
105
1b026014
NIS
106=head2 Utility functions
107
8800c35a
JH
108The following functions are defined in the C<utf8::> package by the
109Perl core. You do not need to say C<use utf8> to use these and in fact
19b49582 110you should not say that unless you really want to have UTF-8 source code.
1b026014
NIS
111
112=over 4
113
973655a8 114=item * $num_octets = utf8::upgrade($string)
1b026014 115
6e37fd2a
JH
116Converts (in-place) internal representation of string to Perl's
117internal I<UTF-X> form. Returns the number of octets necessary to
118represent the string as I<UTF-X>. Can be used to make sure that the
8dd9dd9f 119UTF-8 flag is on, so that C<\w> or C<lc()> work as expected on strings
6e37fd2a
JH
120containing characters in the range 0x80-0xFF (oon ASCII and
121derivatives). Note that this should not be used to convert a legacy
122byte encoding to Unicode: use Encode for that. Affected by the
123encoding pragma.
1b026014 124
973655a8 125=item * $success = utf8::downgrade($string[, FAIL_OK])
1b026014 126
7d865a91
JH
127Converts (in-place) internal representation of string to be un-encoded
128bytes. Returns true on success. On failure dies or, if the value of
129FAIL_OK is true, returns false. Can be used to make sure that the
8dd9dd9f
A
130UTF-8 flag is off, e.g. when you want to make sure that the substr()
131or length() function works with the usually faster byte algorithm.
13a6c0e0
JH
132Note that this should not be used to convert Unicode back to a legacy
133byte encoding: use Encode for that. B<Not> affected by the encoding
134pragma.
1b026014
NIS
135
136=item * utf8::encode($string)
137
2c9359a2
RGS
138Converts in-place the octets of the I<$string> to the octet sequence
139in Perl's I<UTF-X> encoding. Returns nothing. B<Note that this does
140not change the "type" of I<$string> to UTF-8>, and that this handles
141only ISO 8859-1 (or EBCDIC) as the source character set. Therefore
142this should not be used to convert a legacy 8-bit encoding to Unicode:
143use Encode::decode() for that. In the very limited case of wanting to
144handle just ISO 8859-1 (or EBCDIC), you could use utf8::upgrade().
094ce63c 145
973655a8 146=item * utf8::decode($string)
1b026014 147
ad0029c4 148Attempts to convert I<$string> in-place from Perl's I<UTF-X> encoding
2c9359a2
RGS
149into octets. Returns nothing. B<Note that this does not change the
150"type" of <$string> from UTF-8>, and that this handles only ISO 8859-1
151(or EBCDIC) as the destination character set. Therefore this should
152not be used to convert Unicode back to a legacy 8-bit encoding:
153use Encode::encode() for that. In the very limited case of wanting
154to handle just ISO 8859-1 (or EBCDIC), you could use utf8::downgrade().
1b026014 155
8800c35a
JH
156=item * $flag = utf8::is_utf8(STRING)
157
786c9463
JH
158(Since Perl 5.8.1) Test whether STRING is in UTF-8. Functionally
159the same as Encode::is_utf8().
8800c35a 160
70122e76
JH
161=item * $flag = utf8::valid(STRING)
162
8800c35a
JH
163[INTERNAL] Test whether STRING is in a consistent state regarding
164UTF-8. Will return true is well-formed UTF-8 and has the UTF-8 flag
165on B<or> if string is held as bytes (both these states are 'consistent').
166Main reason for this routine is to allow Perl's testsuite to check
167that operations have left strings in a consistent state. You most
168probably want to use utf8::is_utf8() instead.
70122e76 169
1b026014
NIS
170=back
171
7d865a91
JH
172C<utf8::encode> is like C<utf8::upgrade>, but the UTF8 flag is
173cleared. See L<perlunicode> for more on the UTF8 flag and the C API
174functions C<sv_utf8_upgrade>, C<sv_utf8_downgrade>, C<sv_utf8_encode>,
094ce63c
AT
175and C<sv_utf8_decode>, which are wrapped by the Perl functions
176C<utf8::upgrade>, C<utf8::downgrade>, C<utf8::encode> and
786c9463
JH
177C<utf8::decode>. Note that in the Perl 5.8.0 and 5.8.1 implementation
178the functions utf8::is_utf8, utf8::valid, utf8::encode, utf8::decode,
6e37fd2a
JH
179utf8::upgrade, and utf8::downgrade are always available, without a
180C<require utf8> statement-- this may change in future releases.
f1e62f77 181
8f8cf39c
JH
182=head1 BUGS
183
184One can have Unicode in identifier names, but not in package/class or
185subroutine names. While some limited functionality towards this does
186exist as of Perl 5.8.0, that is more accidental than designed; use of
187Unicode for the said purposes is unsupported.
188
189One reason of this unfinishedness is its (currently) inherent
190unportability: since both package names and subroutine names may need
191to be mapped to file and directory names, the Unicode capability of
192the filesystem becomes important-- and there unfortunately aren't
193portable answers.
194
393fec97 195=head1 SEE ALSO
a0ed51b3 196
a74e8b45 197L<perluniintro>, L<encoding>, L<perlrun>, L<bytes>, L<perlunicode>
a0ed51b3
LW
198
199=cut