perldelta: Add entry for new XS macro isUTF8_CHAR

[perl5.git] / pod / perldelta.pod

diff --git a/pod/perldelta.pod b/pod/perldelta.pod
index dcd656d..43a577b 100644 (file)
--- a/pod/perldelta.pod
+++ b/pod/perldelta.pod
@@ -27,6 +27,24 @@ here, but most should go in the L</Performance Enhancements> section.
 
 [ List each enhancement as a =head2 entry ]
 
 
 [ List each enhancement as a =head2 entry ]
 

+=head2 Experimental C Backtrace API
+
+Starting from Perl 5.21.1, on some platforms Perl supports retrieving
+the C level backtrace (similar to what symbolic debuggers like gdb do).
+
+The backtrace returns the stack trace of the C call frames,
+with the symbol names (function names), the object names (like "perl"),
+and if it can, also the source code locations (file:line).
+
+The supported platforms are Linux and OS X (some *BSD might work at
+least partly, but they have not yet been tested).
+
+The feature needs to be enabled with C<Configure -Dusecbacktrace>.
+
+Also included is a C API to retrieve backtraces.
+
+See L<perlhacktips/"C backtrace"> for more information.
+

 =head2 C<qr/foo/x> now ignores any Unicode pattern white space
 
 The C</x> regular expression modifier allows the pattern to contain
 =head2 C<qr/foo/x> now ignores any Unicode pattern white space
 
 The C</x> regular expression modifier allows the pattern to contain


@@ -41,6 +59,12 @@ U+2028 LINE SEPARATOR,
 and
 U+2029 PARAGRAPH SEPARATOR.
 
 and
 U+2029 PARAGRAPH SEPARATOR.
 

+=head2 S<C<use locale>> can restrict which locale categories are affected
+
+It is now possible to pass a parameter to S<C<use locale>> to specify
+a subset of locale categories to be locale-aware, with the remaining
+ones unaffected.  See L<perllocale/The "use locale" pragma> for details.
+

 =head1 Security
 
 XXX Any security-related notices go here.  In particular, any security
 =head1 Security
 
 XXX Any security-related notices go here.  In particular, any security


@@ -57,6 +81,10 @@ XXX For a release on a stable branch, this section aspires to be:
     If any exist, they are bugs, and we request that you submit a
     report.  See L</Reporting Bugs> below.
 
     If any exist, they are bugs, and we request that you submit a
     report.  See L</Reporting Bugs> below.
 

+=head2 C<\N{}> with a sequence of multiple spaces is now a fatal error.
+
+This has been deprecated since v5.18.
+

 =head2 In double-quotish C<\cI<X>>, I<X> must now be a printable ASCII character
 
 In prior releases, failure to do this raised a deprecation warning.
 =head2 In double-quotish C<\cI<X>>, I<X> must now be a printable ASCII character
 
 In prior releases, failure to do this raised a deprecation warning.


@@ -90,6 +118,27 @@ Really old Perl let you omit the @ on array names and the % on hash
 names in some spots.  This has issued a deprecation warning since Perl
 5.0, and is no longer permitted.
 
 names in some spots.  This has issued a deprecation warning since Perl
 5.0, and is no longer permitted.
 

+=head2 C<"$!"> text is now in English outside C<"use locale"> scope
+
+Previously, the text, unlike almost everything else, always came out
+based on the current underlying locale of the program.  (Also affected
+on some systems is C<"$^E>".)  For programs that are unprepared to
+handle locale, this can cause garbage text to be displayed.  It's better
+to display text that is translatable via some tool than garbage text
+which is much harder to figure out.
+
+=head2 C<"$!"> text will be returned in UTF-8 when appropriate
+
+The stringification of C<$!> and C<$^E> will have the UTF-8 flag set
+when the text is actually non-ASCII UTF-8.  This will enable programs
+that are set up to be locale-aware to properly output messages in the
+user's native language.  Code that needs to continue the 5.20 and
+earlier behavior can do the stringification within the scopes of both
+'use bytes' and 'use locale ":messages".  No other Perl operations will
+be affected by locale; only C<$!> and C<$^E> stringification.  The
+'bytes' pragma causes the UTF-8 flag to not be set, just as in previous
+Perl releases.  This resolves [perl #112208].
+

 =head1 Deprecations
 
 XXX Any deprecated features, syntax, modules etc. should be listed here.
 =head1 Deprecations
 
 XXX Any deprecated features, syntax, modules etc. should be listed here.


@@ -101,6 +150,15 @@ This non-graphic character is essentially indistinguishable from a
 regular space, and so should not be allowed.  See
 L<charnames/CUSTOM ALIASES>.
 
 regular space, and so should not be allowed.  See
 L<charnames/CUSTOM ALIASES>.
 

+=head2 A literal C<"{"> should now be escaped in a pattern
+
+If you want a literal left curly bracket (also called a left brace) in a
+regular expression pattern, you should now escape it by either
+preceding it with a backslash (C<"\{">) or enclosing it within square
+brackets C<"[{]">, or by using C<\Q>; otherwise a deprecation warning
+will be raised.  This was first announced as forthcoming in the v5.16
+release; it will allow future extensions to the language to happen.
+

 =head2 Module removals
 
 XXX Remove this section if inapplicable.
 =head2 Module removals
 
 XXX Remove this section if inapplicable.


@@ -141,7 +199,18 @@ There may well be none in a stable release.
 
 =item *
 
 
 =item *
 

-XXX
+Many internal functions have been refactored to improve performance and reduce
+their memory footprints.
+
+L<[perl #121436]|https://rt.perl.org/Ticket/Display.html?id=121436>
+L<[perl #121906]|https://rt.perl.org/Ticket/Display.html?id=121906>
+L<[perl #121969]|https://rt.perl.org/Ticket/Display.html?id=121969>
+
+=item *
+
+C<-T> and C<-B> filetests will return sooner when an empty file is detected.
+
+L<perl #121489|https://rt.perl.org/Ticket/Display.html?id=121489>

 
 =back
 
 
 =back
 


@@ -180,6 +249,16 @@ L<[perl #121574]|https://rt.perl.org/Ticket/Display.html?id=121574>
 
 =item *
 
 
 =item *
 

+L<Data::Dumper> has been upgraded from version 2.151 to 2.152.
+
+Changes to resolve Coverity issues.
+
+XS dumps incorrectly stored the name of code references stored in a
+GLOB.
+L<[perl #122070]|https://rt.perl.org/Ticket/Display.html?id=122070>
+
+=item *
+

 L<Encode> has been upgraded from version 2.60_01 to 2.62.
 
 B<piconv> now has better error handling when the encoding name is nonexistent,
 L<Encode> has been upgraded from version 2.60_01 to 2.62.
 
 B<piconv> now has better error handling when the encoding name is nonexistent,


@@ -188,6 +267,12 @@ been fixed.
 
 =item *
 
 
 =item *
 

+L<Hash::Util> has been upgraded from version 0.16 to 0.17.
+
+Minor bug fixes and documentation fixes to Hash::Util::hash_stats()
+
+=item *
+

 The libnet collection of modules has been upgraded from version 1.25 to 1.27.
 
 There are only whitespace changes to the installed files.
 The libnet collection of modules has been upgraded from version 1.25 to 1.27.
 
 There are only whitespace changes to the installed files.


@@ -227,16 +312,11 @@ than 6.63_03
 
 =item *
 
 
 =item *
 

-L<perl5db.pl> has been upgraded from version 1.44 to 1.45.
+L<OS2::Process> has been upgraded from version 1.09 to 1.10.

 
 =item *
 
 
 =item *
 

-A mismatch between the documentation and the code in utf8::downgrade()
-was fixed in favour of the documentation. The optional second argument
-is now correctly treated as a perl boolean (true/false semantics) and
-not as an integer.
-
-=item *
+L<perl5db.pl> has been upgraded from version 1.44 to 1.45.

 
 fork() in the debugger under C<tmux> will now create a new window for
 the forked process. L<[perl
 
 fork() in the debugger under C<tmux> will now create a new window for
 the forked process. L<[perl


@@ -246,6 +326,26 @@ The debugger now saves the current working directory on startup and
 restores it when you restart your program with C<R> or <rerun>. L<[perl
 #121509]|https://rt.perl.org/Ticket/Display.html?id=121509>
 
 restores it when you restart your program with C<R> or <rerun>. L<[perl
 #121509]|https://rt.perl.org/Ticket/Display.html?id=121509>
 

+=item *
+
+L<PerlIO::encoding> has been upgraded from version 0.18 to 0.19.
+
+No changes in behaviour.
+
+=item *
+
+L<PerlIO::mmap> has been upgraded from version 0.012 to 0.013.
+
+No changes in behaviour.
+
+=item *
+
+L<PerlIO::scalar> has been upgraded from version 0.18 to 0.19.
+
+No changes in behaviour.
+
+=item *
+

 L<Unicode::Collate> has been upgraded from version 1.04 to 1.07.
 
 Version 0.67's improved discontiguous contractions is invalidated by default
 L<Unicode::Collate> has been upgraded from version 1.04 to 1.07.
 
 Version 0.67's improved discontiguous contractions is invalidated by default


@@ -259,10 +359,10 @@ The XSUB implementation has been removed in favour of pure Perl.
 
 =item *
 
 
 =item *
 

-L<Hash::Util> has been upgraded from version 0.16 to 0.17.
-
-Minor bug fixes and documentation fixes to Hash::Util::hash_stats()
-
+A mismatch between the documentation and the code in utf8::downgrade()
+was fixed in favour of the documentation. The optional second argument
+is now correctly treated as a perl boolean (true/false semantics) and
+not as an integer.

 
 =back
 
 
 =back
 


@@ -295,13 +395,89 @@ XXX Changes which significantly change existing files in F<pod/> go here.
 However, any changes to F<pod/perldiag.pod> should go in the L</Diagnostics>
 section.
 
 However, any changes to F<pod/perldiag.pod> should go in the L</Diagnostics>
 section.
 

-=head3 L<XXX>
+=head3 L<perlfunc>

 
 =over 4
 
 =item *
 
 
 =over 4
 
 =item *
 

-XXX Description of the change here
+C<-l> now notes that it will return false if symlinks aren't supported by the 
+file system.
+
+L<[perl #121523]|https://rt.perl.org/Ticket/Display.html?id=121523>
+
+=item *
+
+Note that C<exec LIST> and C<system LIST> may fall back to the shell on
+Win32. Only C<exec PROGRAM LIST> and C<system PROGRAM LIST> indirect object
+syntax will reliably avoid using the shell.
+
+This has also been noted in L<perlport>.
+
+L<[perl #122046]|https://rt.perl.org/Ticket/Display.html?id=122046>
+
+=back
+
+=head3 L<perlapi>
+
+=over 4
+
+=item *
+
+C<sv_usepvn_flags> - Fix documentation to mention the use of C<NewX> instead of 
+C<malloc>.
+
+L<[perl #121869]|https://rt.perl.org/Ticket/Display.html?id=121869>
+
+=item *
+
+Clarify where C<NUL> may be embedded or is required to terminate a string.
+
+=item *
+
+Previously missing documentation due to formatting errors are now included.
+
+=item *
+
+Entries are now organized into groups rather than by file where they are found.
+
+=item *
+
+Alphabetical sorting of entries is now handled by the POD generator to make 
+entries easier to find when scanning.
+
+=back
+
+=head3 L<perlhacktips>
+
+=over 4
+
+=item *
+
+Updated documentation for the C<test.valgrind> C<make> target.
+
+L<[perl #121431]|https://rt.perl.org/Ticket/Display.html?id=121431>
+
+=back
+
+=head3 L<perlre>
+
+=over 4
+
+=item *
+
+The C</x> modifier has been clarified to note that comments cannot be continued 
+onto the next line by escaping them.
+
+=back
+
+=head3 L<Unicode::UCD>
+
+=over 4
+
+=item *
+
+The documentation includes many clarifications and fixes.

 
 =back
 
 
 =back
 


@@ -335,7 +511,13 @@ XXX L<message|perldiag/"message">
 
 =item *
 
 
 =item *
 

-XXX L<message|perldiag/"message">
+L<PerlIO layer ':win32' is experimental|perldiag/"PerlIO layer ':win32' is experimental">:
+
+(S experimental::win32_perlio) The C<:win32> PerlIO layer is
+experimental.  If you want to take the risk of using this layer,
+simply disable this warning:
+
+    no warnings "experimental::win32_perlio";

 
 =back
 
 
 =back
 


@@ -347,6 +529,14 @@ XXX Changes (i.e. rewording) of diagnostic messages go here
 
 =item *
 
 
 =item *
 

+<> should be quotes
+
+This warning has been changed to
+L<< <> at require-statement should be quotes|perldiag/"<> at require-statement should be quotes" >>
+to make the issue more identifiable.
+
+=item *
+

 L<Unsuccessful %s on filename containing newline|perldiag/"Unsuccessful %s on filename containing newline">
 
 This warning is now only produced when the newline is at the end of
 L<Unsuccessful %s on filename containing newline|perldiag/"Unsuccessful %s on filename containing newline">
 
 This warning is now only produced when the newline is at the end of


@@ -357,19 +547,22 @@ the filename.
 =head1 Utility Changes
 
 XXX Changes to installed programs such as F<perlbug> and F<xsubpp> go here.
 =head1 Utility Changes
 
 XXX Changes to installed programs such as F<perlbug> and F<xsubpp> go here.

-Most of these are built within the directories F<utils> and F<x2p>.
+Most of these are built within the directory F<utils>.

 
 [ List utility changes as a =head2 entry for each utility and =item
 entries for each change
 Use L<XXX> with program names to get proper documentation linking. ]
 
 
 [ List utility changes as a =head2 entry for each utility and =item
 entries for each change
 Use L<XXX> with program names to get proper documentation linking. ]
 

-=head2 L<XXX>
+=head2 F<x2p/>

 
 =over 4
 
 =item *
 
 
 =over 4
 
 =item *
 

-XXX
+The F<x2p/> directory has been removed from the Perl core. 
+
+This removes find2perl, s2p and a2p. They have all been released to CPAN as 
+separate distributions (App::find2perl, App::s2p, App::a2p).

 
 =back
 
 
 =back
 


@@ -386,7 +579,15 @@ L</Platform Support> section, instead.
 
 =item *
 
 
 =item *
 

-XXX
+C<make test.valgrind> now supports parallel testing.
+
+For example:
+
+    TEST_JOBS=9 make test.valgrind
+
+See L<perlhacktips/valgrind> for more information.
+
+L<[perl #121431]|https://rt.perl.org/Ticket/Display.html?id=121431>

 
 =back
 
 
 =back
 


@@ -436,9 +637,12 @@ XXX List any platforms that this version of perl no longer compiles on.
 
 =over 4
 
 
 =over 4
 

-=item XXX-some-platform
+=item NeXTSTEP/OPENSTEP

 
 

-XXX
+NeXTSTEP was proprietary OS bundled with NeXT's workstations in the early
+to mid 90's; OPENSTEP was an API specification that provided a NeXTSTEP-like 
+environment on a non-NeXTSTEP system.  Both are now long dead, so support
+for building Perl on them has been removed.

 
 =back
 
 
 =back
 


@@ -455,6 +659,16 @@ L</Modules and Pragmata> section.
 
 XXX
 
 
 XXX
 

+=item OpenBSD
+
+On OpenBSD, Perl will now default to using the system C<malloc> due to the
+security features it provides. Perl's own malloc wrapper has been in use
+since v5.14 due to performance reasons, but the OpenBSD project believes
+the tradeoff is worth it and would prefer that users who need the speed
+specifically ask for it.
+
+L<[perl #122000]|https://rt.perl.org/Ticket/Display.html?id=122000>.
+

 =back
 
 =head1 Internal Changes
 =back
 
 =head1 Internal Changes


@@ -469,6 +683,27 @@ well.
 
 The deprecated variable C<PL_sv_objcount> has been removed.
 
 
 The deprecated variable C<PL_sv_objcount> has been removed.
 

+=item *
+
+Perl now tries to keep the locale category C<LC_NUMERIC> set to "C"
+except around operations that need it to be set to the program's
+underlying locale.  This protects the many XS modules that cannot cope
+with the decimal radix character not being a dot.  Prior to this
+release, Perl initialized this category to "C", but a call to
+C<POSIX::setlocale()> would change it.  Now such a call will change the
+underlying locale of the C<LC_NUMERIC> category for the program, but the
+locale exposed to XS code will remain "C".  There is an API under
+development for those relatively few modules that need to use the
+underlying locale.  This API will be nailed down during the course of
+developing v5.21.  Send email to L<mailto:perl5-porters@perl.org> for
+guidance.
+
+=item *
+
+A new macro L<C<isUTF8_CHAR>|perlapi/isUTF8_CHAR> has been written which
+efficiently determines if the string given by its parameters begins
+with a well-formed UTF-8 encoded character.
+

 =back
 
 =head1 Selected Bug Fixes
 =back
 
 =head1 Selected Bug Fixes


@@ -497,6 +732,69 @@ destroy perl engines within the same process.
 C<POSIX::localeconv()> now returns the data for the program's underlying
 locale even when called from outside the scope of S<C<use locale>>.
 
 C<POSIX::localeconv()> now returns the data for the program's underlying
 locale even when called from outside the scope of S<C<use locale>>.
 

+=item *
+
+C<POSIX::localeconv()> now works properly on platforms which don't have
+C<LC_NUMERIC> and/or C<LC_MONETARY>, or for which Perl has been compiled
+to disregard either or both of these locale categories.  In such
+circumstances, there are now no entries for the corresponding values in
+the hash returned by C<localeconv()>.
+
+=item *
+
+C<POSIX::localeconv()> now marks appropriately the values it returns as
+UTF-8 or not.  Previously they were always returned as a bytes, even if
+they were supposed to be encoded as UTF-8.
+
+=item *
+
+On Microsoft Windows, within the scope of C<S<use locale>>, the following
+POSIX character classes gave results for many locales that did not
+conform to the POSIX standard:
+C<[[:alnum:]]>,
+C<[[:alpha:]]>,
+C<[[:blank:]]>,
+C<[[:digit:]]>,
+C<[[:graph:]]>,
+C<[[:lower:]]>,
+C<[[:print:]]>,
+C<[[:punct:]]>,
+C<[[:upper:]]>,
+C<[[:word:]]>,
+and
+C<[[:xdigit:]]>.
+These are because the underlying Microsoft implementation does not
+follow the standard.  Perl now takes special precautions to correct for
+this.
+
+=item *
+
+Many issues have been detected by L<Coverity|http://www.coverity.com/> and 
+fixed.
+
+=item *
+
+system() and friends should now work properly on more Android builds.
+
+Due to an oversight, the value specified through -Dtargetsh to Configure
+would end up being ignored by some of the build process.  This caused perls
+cross-compiled for Android to end up with defective versions of system(),
+exec() and backticks: the commands would end up looking for C</bin/sh>
+instead of C</system/bin/sh>, and so would fail for the vast majority
+of devices, leaving C<$!> as C<ENOENT>.
+
+=item *
+
+C<qr(...\(...\)...)>,
+C<qr[...\[...\]...]>,
+and
+C<qr{...\{...\}...}>
+now work.  Previously it was impossible to escape these three
+left-characters with a backslash within a regular expression pattern
+where otherwise they would be considered metacharacters, and the pattern
+opening delimiter was the character, and the closing delimiter was its
+mirror character.
+

 =back
 
 =head1 Known Problems
 =back
 
 =head1 Known Problems