Promote v5.36 usage and feature bundles doc

[perl5.git] / lib / utf8.pm

diff --git a/lib/utf8.pm b/lib/utf8.pm
index 34930a0..823193b 100644 (file)
--- a/lib/utf8.pm
+++ b/lib/utf8.pm
@@ -1,19 +1,22 @@
 package utf8;
 
-$utf8::hint_bits = 0x00800000;
+use strict;
+use warnings;
 
-our $VERSION = '1.20';
+our $hint_bits = 0x00800000;
+
+our $VERSION = '1.24';
+our $AUTOLOAD;
 
 sub import {
-    $^H |= $utf8::hint_bits;
+    $^H |= $hint_bits;
 }
 
 sub unimport {
-    $^H &= ~$utf8::hint_bits;
+    $^H &= ~$hint_bits;
 }
 
 sub AUTOLOAD {
-    require "utf8_heavy.pl";
     goto &$AUTOLOAD if defined &$AUTOLOAD;
     require Carp;
     Carp::croak("Undefined subroutine $AUTOLOAD called");
@@ -136,14 +139,12 @@ use L<Encode> instead.
 =item * C<$success = utf8::downgrade($string[, $fail_ok])>
 
 (Since Perl v5.8.0)
-Converts in-place the internal representation of the string from
-UTF-8 to the equivalent octet sequence in the native encoding (Latin-1
-or EBCDIC). The logical character sequence itself is unchanged. If
-I<$string> is already stored as native 8 bit, then this is a no-op.  Can
-be used to
-make sure that the UTF-8 flag is off, e.g. when you want to make sure
-that the substr() or length() function works with the usually faster
-byte algorithm.
+Converts in-place the internal representation of the string from UTF-8 to the
+equivalent octet sequence in the native encoding (Latin-1 or EBCDIC). The
+logical character sequence itself is unchanged. If I<$string> is already
+stored as native 8 bit, then this is a no-op.  Can be used to make sure that
+the UTF-8 flag is off, e.g. when you want to make sure that the substr() or
+length() function works with the usually faster byte algorithm.
 
 Fails if the original UTF-8 sequence cannot be represented in the
 native 8 bit encoding. On failure dies or, if the value of I<$fail_ok> is
@@ -167,8 +168,8 @@ use L<Encode> instead.
 
 (Since Perl v5.8.0)
 Converts in-place the character sequence to the corresponding octet
-sequence in UTF-8. That is, every (possibly wide) character gets
-replaced with a sequence of one or more characters that represent the
+sequence in Perl's extended UTF-8. That is, every (possibly wide) character
+gets replaced with a sequence of one or more characters that represent the
 individual UTF-8 bytes of the character.  The UTF8 flag is turned off.
 Returns nothing.
 
@@ -188,12 +189,12 @@ use L<Encode> instead.
 =item * C<$success = utf8::decode($string)>
 
 (Since Perl v5.8.0)
-Attempts to convert in-place the octet sequence encoded as UTF-8 to the
-corresponding character sequence. That is, it replaces each sequence of
-characters in the string whose ords represent a valid UTF-8 byte
-sequence, with the corresponding single character.  The UTF-8 flag is
-turned on only if the source string contains multiple-byte UTF-8
-characters.  If I<$string> is invalid as UTF-8, returns false;
+Attempts to convert in-place the octet sequence encoded in Perl's extended
+UTF-8 to the corresponding character sequence. That is, it replaces each
+sequence of characters in the string whose ords represent a valid (extended)
+UTF-8 byte sequence, with the corresponding single character.  The UTF-8 flag
+is turned on only if the source string contains multiple-byte UTF-8
+characters.  If I<$string> is invalid as extended UTF-8, returns false;
 otherwise returns true.
 
  my $x = "\xc4\x80"; # $x contains two characters, with ords
@@ -202,6 +203,11 @@ otherwise returns true.
                      # with ord 0x100.   Since these bytes aren't
                      # legal UTF-EBCDIC, on EBCDIC platforms, $x is
                      # unchanged and the function returns FALSE.
+ my $y = "\xc3\x83\xc2\xab"; This has been encoded twice; this
+                     # example is only for ASCII platforms
+ utf8::decode($y);   # Converts $y to \xc3\xab, returns TRUE;
+ utf8::decode($y);   # Further converts to \xeb, returns TRUE;
+ utf8::decode($y);   # Returns FALSE, leaves $y unchanged
 
 B<Note that this function does not handle arbitrary encodings>;
 use L<Encode> instead.
@@ -248,7 +254,7 @@ dealing with filenames, you should probably read L<perlunitut> and
 L<perlunifaq/What is "the UTF8 flag"?>.
 
 Don't use this flag as a marker to distinguish character and binary
-data, that should be decided for each variable when you write your
+data: that should be decided for each variable when you write your
 code.
 
 To force unicode semantics in code portable to perl 5.8 and 5.10, call
@@ -257,9 +263,10 @@ C<utf8::upgrade($string)> unconditionally.
 =item * C<$flag = utf8::valid($string)>
 
 [INTERNAL] Test whether I<$string> is in a consistent state regarding
-UTF-8.  Will return true if it is well-formed UTF-8 and has the UTF-8 flag
+UTF-8.  Will return true if it is well-formed Perl extended UTF-8 and has the
+UTF-8 flag
 on B<or> if I<$string> is held as bytes (both these states are 'consistent').
-Main reason for this routine is to allow Perl's test suite to check
+The main reason for this routine is to allow Perl's test suite to check
 that operations have left strings in a consistent state.
 
 =back