+=head1 BUGS
+
+=head2 Interaction with locales
+
+Use of locales with Unicode data may lead to odd results. Currently
+there is some attempt to apply 8-bit locale info to characters in the
+range 0..255, but this is demonstrably incorrect for locales that use
+characters above that range when mapped into Unicode. It will also
+tend to run slower. Use of locales with Unicode is discouraged.
+
+=head2 Interaction with extensions
+
+When perl exchanges data with an extension, the extension should be
+able to understand the UTF-8 flag and act accordingly. If the
+extension doesn't know about the flag, the risk is high that it will
+return data that are incorrectly flagged.
+
+So if you're working with Unicode data, consult the documentation of
+every module you're using if there are any issues with Unicode data
+exchange. If the documentation does not talk about Unicode at all,
+suspect the worst and probably look at the source to learn how the
+module is implemented. Modules written completely in perl shouldn't
+cause problems. Modules that directly or indirectly access code written
+in other programming languages are at risk.
+
+For affected functions the simple strategy to avoid data corruption is
+to always make the encoding of the exchanged data explicit. Choose an
+encoding you know the extension can handle. Convert arguments passed
+to the extensions to that encoding and convert results back from that
+encoding. Write wrapper functions that do the conversions for you, so
+you can later change the functions when the extension catches up.
+
+To provide an example let's say the popular Foo::Bar::escape_html
+function doesn't deal with Unicode data yet. The wrapper function
+would convert the argument to raw UTF-8 and convert the result back to
+perl's internal representation like so:
+
+ sub my_escape_html ($) {
+ my($what) = shift;
+ return unless defined $what;
+ Encode::decode_utf8(Foo::Bar::escape_html(Encode::encode_utf8($what)));
+ }
+
+Sometimes, when the extension does not convert data but just stores
+and retrieves them, you will be in a position to use the otherwise
+dangerous Encode::_utf8_on() function. Let's say the popular
+C<Foo::Bar> extension, written in C, provides a C<param> method that
+lets you store and retrieve data according to these prototypes:
+
+ $self->param($name, $value); # set a scalar
+ $value = $self->param($name); # retrieve a scalar
+
+If it does not yet provide support for any encoding, one could write a
+derived class with such a C<param> method:
+
+ sub param {
+ my($self,$name,$value) = @_;
+ utf8::upgrade($name); # make sure it is UTF-8 encoded
+ if (defined $value)
+ utf8::upgrade($value); # make sure it is UTF-8 encoded
+ return $self->SUPER::param($name,$value);
+ } else {
+ my $ret = $self->SUPER::param($name);
+ Encode::_utf8_on($ret); # we know, it is UTF-8 encoded
+ return $ret;
+ }
+ }
+
+Some extensions provide filters on data entry/exit points, such as
+DB_File::filter_store_key and family. Look out for such filters in
+the documentation of your extensions, they can make the transition to
+Unicode data much easier.
+
+=head2 speed
+
+Some functions are slower when working on UTF-8 encoded strings than
+on byte encoded strings. All functions that need to hop over
+characters such as length(), substr() or index() can work B<much>
+faster when the underlying data are byte-encoded. Witness the
+following benchmark:
+
+ % perl -e '
+ use Benchmark;
+ use strict;
+ our $l = 10000;
+ our $u = our $b = "x" x $l;
+ substr($u,0,1) = "\x{100}";
+ timethese(-2,{
+ LENGTH_B => q{ length($b) },
+ LENGTH_U => q{ length($u) },
+ SUBSTR_B => q{ substr($b, $l/4, $l/2) },
+ SUBSTR_U => q{ substr($u, $l/4, $l/2) },
+ });
+ '
+ Benchmark: running LENGTH_B, LENGTH_U, SUBSTR_B, SUBSTR_U for at least 2 CPU seconds...
+ LENGTH_B: 2 wallclock secs ( 2.36 usr + 0.00 sys = 2.36 CPU) @ 5649983.05/s (n=13333960)
+ LENGTH_U: 2 wallclock secs ( 2.11 usr + 0.00 sys = 2.11 CPU) @ 12155.45/s (n=25648)
+ SUBSTR_B: 3 wallclock secs ( 2.16 usr + 0.00 sys = 2.16 CPU) @ 374480.09/s (n=808877)
+ SUBSTR_U: 2 wallclock secs ( 2.11 usr + 0.00 sys = 2.11 CPU) @ 6791.00/s (n=14329)
+
+The numbers show an incredible slowness on long UTF-8 strings and you
+should carefully avoid to use these functions within tight loops. For
+example if you want to iterate over characters, it is infinitely
+better to split into an array than to use substr, as the following
+benchmark shows:
+
+ % perl -e '
+ use Benchmark;
+ use strict;
+ our $l = 10000;
+ our $u = our $b = "x" x $l;
+ substr($u,0,1) = "\x{100}";
+ timethese(-5,{
+ SPLIT_B => q{ for my $c (split //, $b){} },
+ SPLIT_U => q{ for my $c (split //, $u){} },
+ SUBSTR_B => q{ for my $i (0..length($b)-1){my $c = substr($b,$i,1);} },
+ SUBSTR_U => q{ for my $i (0..length($u)-1){my $c = substr($u,$i,1);} },
+ });
+ '
+ Benchmark: running SPLIT_B, SPLIT_U, SUBSTR_B, SUBSTR_U for at least 5 CPU seconds...
+ SPLIT_B: 6 wallclock secs ( 5.29 usr + 0.00 sys = 5.29 CPU) @ 56.14/s (n=297)
+ SPLIT_U: 5 wallclock secs ( 5.17 usr + 0.01 sys = 5.18 CPU) @ 55.21/s (n=286)
+ SUBSTR_B: 5 wallclock secs ( 5.34 usr + 0.00 sys = 5.34 CPU) @ 123.22/s (n=658)
+ SUBSTR_U: 7 wallclock secs ( 6.20 usr + 0.00 sys = 6.20 CPU) @ 0.81/s (n=5)
+
+You see, the algorithm based on substr() was faster with byte encoded
+data but it is pathologically slow with UTF-8 data.
+
https://perl5.git.perl.org / perl5.git / blobdiff