lexer API fixes

[perl5.git] / pod / perl593delta.pod

=head1 NAME

perl593delta - what is new for perl v5.9.3

=head1 DESCRIPTION

This document describes differences between the 5.9.2 and the 5.9.3
development releases. See L<perl590delta>, L<perl591delta> and
L<perl592delta> for the differences between 5.8.0 and 5.9.2.

=head1 Incompatible Changes

=head2 Parsing of C<-f _>

The identifier C<_> is now forced to be a bareword after a filetest
operator. This solves a number of misparsing issues when a global C<_>
subroutine is defined.

=head2 C<mkdir()>

C<mkdir()> without arguments now defaults to C<$_>.

=head2 Magic goto and eval

The construct C<eval { goto &foo }> is now disallowed. (Note that the
similar construct, but with C<eval("")> instead, was already forbidden.)

=head2 C<$#> has been removed

The deprecated C<$#> variable (output format for numbers) has been
removed. A new warning, C<$# is no longer supported>, has been added.

=head2 C<:unique>

The C<:unique> attribute has been made a no-op, since its current
implementation was fundamentally flawed and not threadsafe.

=head2 Scoping of the C<sort> pragma

The C<sort> pragma is now lexically scoped. Its effect used to be global.

=head1 Core Enhancements

=head2 The C<feature> pragma

The C<feature> pragma is used to enable new syntax that would break Perl's
backwards-compatibility with older releases of the language. It's a lexical
pragma, like C<strict> or C<warnings>.

Currently the following new features are available: C<switch> (adds a
switch statement), C<~~> (adds a Perl 6-like smart match operator), C<say>
(adds a C<say> built-in function), and C<err> (adds an C<err> keyword).
Those features are described below.

Note that C<err> low-precedence defined-or operator used to be enabled by
default (although as a weak keyword, meaning that any function would
override it). It's now only recognized when explicitly turned on (and is
then a regular keyword).

Those features, and the C<feature> pragma itself, have been contributed by
Robin Houston.

=head2 Switch and Smart Match operator

Perl 5 now has a switch statement. It's available when C<use feature
'switch'> is in effect. This feature introduces three new keywords,
C<given>, C<when>, and C<default>:

    given ($foo) {
	when (/^abc/) { $abc = 1; }
	when (/^def/) { $def = 1; }
	when (/^xyz/) { $xyz = 1; }
	default { $nothing = 1; }
    }

A more complete description of how Perl matches the switch variable
against the C<when> conditions is given in L<perlsyn/"Switch statements">.

This kind of match is called I<smart match>, and it's also possible to use
it outside of switch statements, via the new C<~~> operator (enabled via
the C<use feature '~~'> directive). See L<perlsyn/"Smart matching in
detail">.

=head2 C<say()>

say() is a new built-in, only available when C<use feature 'say'> is in
effect, that is similar to print(), but that implicitly appends a newline
to the printed string. See L<perlfunc/say>.

=head2 C<CLONE_SKIP()>

Perl has now support for the C<CLONE_SKIP> special subroutine. Like
C<CLONE>, C<CLONE_SKIP> is called once per package; however, it is called
just before cloning starts, and in the context of the parent thread. If it
returns a true value, then no objects of that class will be cloned. See
L<perlmod> for details. (Contributed by Dave Mitchell.)

=head2 C<${^CHILD_ERROR_NATIVE}>

A new internal variable, C<${^CHILD_ERROR_NATIVE}>, gives the native
status returned by the last pipe close, backtick command, successful call
to wait() or waitpid(), or from the system() operator. See L<perlrun> for
details. (Contributed by Gisle Aas.)

=head2 Assertions

The support for assertions, introduced in perl 5.9.0, has been improved.
The syntax for the C<-A> command-line switch has changed; it now accepts
an optional module name, defaulting to C<assertions::activate>. See
L<assertions> and L<perlrun>. (Contributed by Salvador Fandiño García.)

=head2 Unicode Character Database 4.1.0

The copy of the Unicode Character Database included in Perl 5.9 has
been updated to 4.1.0.

=head2 C<no VERSION>

You can now use C<no> followed by a version number to specify that you
want to use a version of perl older than the specified one.

=head2 Recursive sort subs

You can now use recursive subroutines with sort(), thanks to Robin Houston.

=head2 Effect of pragmas in eval

The compile-time value of the C<%^H> hint variable can now propagate into
eval("")uated code. This makes it more useful to implement lexical
pragmas.

As a side-effect of this, the overloaded-ness of constants now propagates
into eval("").

=head2 New B<-E> command-line switch

B<-E> is equivalent to B<-e>, but it implicitly enables all
optional features (like C<use feature ":5.10">).

=head2 C<chdir>, C<chmod> and C<chown> on filehandles

C<chdir>, C<chmod> and C<chown> can now work on filehandles as well as
filenames, if the system supports respectively C<fchdir>, C<fchmod> and
C<fchown>, thanks to a patch provided by Gisle Aas.

=head2 OS groups

C<$(> and C<$)> now return groups in the order where the OS returns them,
thanks to Gisle Aas. This wasn't previously the case.

=head1 Modules and Pragmata

=head2 New Core Modules

=over 4

=item *

A new pragma, C<feature>, has been added; see above in L</"Core
Enhancements">.

=item *

C<assertions::compat>, also available on CPAN, allows the use of assertions on
perl versions prior to 5.9.0 (that is the first one to natively support
them).

=item *

C<Math::BigInt::FastCalc> is an XS-enabled, and thus faster, version of
C<Math::BigInt::Calc>.

=item *

C<Compress::Zlib> is an interface to the zlib compression library. It
comes with a bundled version of zlib, so having a working zlib is not a
prerequisite to install it. It's used by C<Archive::Tar> (see below).

=item *

C<IO::Zlib> is an C<IO::>-style interface to C<Compress::Zlib>.

=item *

C<Archive::Tar> is a module to manipulate C<tar> archives.

=item *

C<Digest::SHA> is a module used to calculate many types of SHA digests,
has been included for SHA support in the CPAN module.

=item *

C<ExtUtils::CBuilder> and C<ExtUtils::ParseXS> have been added.

=back

=head1 Utility Changes

=head2 C<ptar>

C<ptar> is a pure perl implementation of C<tar>, that comes with
C<Archive::Tar>.

=head2 C<ptardiff>

C<ptardiff> is a small script used to generate a diff between the contents
of a tar archive and a directory tree. Like C<ptar>, it comes with
C<Archive::Tar>.

=head2 C<shasum>

This command-line utility, used to print or to check SHA digests, comes
with the new C<Digest::SHA> module.

=head2 C<h2xs> enhancements

C<h2xs> implements a new option C<--use-xsloader> to force use of
C<XSLoader> even in backwards compatible modules.

The handling of authors' names that had apostrophes has been fixed.

Any enums with negative values are now skipped.

=head2 C<perlivp> enhancements

C<perlivp> no longer checks for F<*.ph> files by default.  Use the new C<-a>
option to run I<all> tests.

=head1 Documentation

=head2 Perl Glossary

The L<perlglossary> manpage is a glossary of terms used in the Perl
documentation, technical and otherwise, kindly provided by O'Reilly Media,
Inc.

L<perltodo> now lists a rough roadmap to Perl 5.10.

=head1 Performance Enhancements

=head2 XS-assisted SWASHGET

Some pure-perl code that perl was using to retrieve Unicode properties and
transliteration mappings has been reimplemented in XS.

=head2 Constant subroutines

The interpreter internals now support a far more memory efficient form of
inlineable constants. Storing a reference to a constant value in a symbol
table is equivalent to a full typeglob referencing a constant subroutine,
but using about 400 bytes less memory. This proxy constant subroutine is
automatically upgraded to a real typeglob with subroutine if necessary.
The approach taken is analogous to the existing space optimisation for
subroutine stub declarations, which are stored as plain scalars in place
of the full typeglob.

Several of the core modules have been converted to use this feature for
their system dependent constants - as a result C<use POSIX;> now takes about
200K less memory.

=head2 C<PERL_DONT_CREATE_GVSV>

The new compilation flag C<PERL_DONT_CREATE_GVSV>, introduced as an option
in perl 5.8.8, is turned on by default in perl 5.9.3. It prevents perl
from creating an empty scalar with every new typeglob. See L<perl589delta>
for details.

=head2 Weak references are cheaper

Weak reference creation is now I<O(1)> rather than I<O(n)>, courtesy of
Nicholas Clark. Weak reference deletion remains I<O(n)>, but if deletion only
happens at program exit, it may be skipped completely.

=head2 sort() enhancements

Salvador Fandiño provided improvements to reduce the memory usage of C<sort>
and to speed up some cases.

=head1 Installation and Configuration Improvements

=head2 Compilation improvements

Parallel makes should work properly now, although there may still be problems
if C<make test> is instructed to run in parallel.

Building with Borland's compilers on Win32 should work more smoothly. In
particular Steve Hay has worked to side step many warnings emitted by their
compilers and at least one C compiler internal error.

Perl extensions on Windows now can be statically built into the Perl DLL,
thanks to a work by Vadim Konovalov.

=head2 New Or Improved Platforms

Perl is being ported to Symbian OS. See L<perlsymbian> for more
information.

The VMS port has been improved. See L<perlvms>.

DynaLoader::dl_unload_file() now works on Windows.

Portability of Perl on various recent compilers on Windows has been
improved (Borland C++, Visual C++ 7.0).

=head2 New probes

C<Configure> will now detect C<clearenv> and C<unsetenv>, thanks to a
patch from Alan Burlison. It will also probe for C<futimes> (and use it
internally if available), and whether C<sprintf> correctly returns the
length of the formatted string.

=head2 Module auxiliary files

README files and changelogs for CPAN modules bundled with perl are no
longer installed.

=head1 Selected Bug Fixes

=head2 C<defined $$x>

C<use strict "refs"> was ignoring taking a hard reference in an argument
to defined(), as in :

    use strict "refs";
    my $x = "foo";
    if (defined $$x) {...}

This now correctly produces the run-time error C<Can't use string as a
SCALAR ref while "strict refs" in use>. (However, C<defined @$foo> and
C<defined %$foo> are still allowed. Those constructs are discouraged
anyway.)

=head2 Calling CORE::require()

CORE::require() and CORE::do() were always parsed as require() and do()
when they were overridden. This is now fixed.

=head2 Subscripts of slices

You can now use a non-arrowed form for chained subscripts after a list
slice, like in:

    ({foo => "bar"})[0]{foo}

This used to be a syntax error; a C<< -> >> was required.

=head2 Remove over-optimisation

Perl 5.9.2 introduced a change so that assignments of C<undef> to a
scalar, or of an empty list to an array or a hash, were optimised out. As
this could cause problems when C<goto> jumps were involved, this change
was backed out.

=head2 sprintf() fixes

Using the sprintf() function with some formats could lead to a buffer
overflow in some specific cases. This has been fixed, along with several
other bugs, notably in bounds checking.

In related fixes, it was possible for badly written code that did not follow
the documentation of C<Sys::Syslog> to have formatting vulnerabilities.
C<Sys::Syslog> has been changed to protect people from poor quality third
party code.

=head2 no warnings 'category' works correctly with -w

Previously when running with warnings enabled globally via C<-w>, selective
disabling of specific warning categories would actually turn off all warnings.
This is now fixed; now C<no warnings 'io';> will only turn off warnings in the
C<io> class. Previously it would erroneously turn off all warnings.

=head2 Smaller fixes

=over 4

=item *

C<FindBin> now works better with directories where access rights are more
restrictive than usual.

=item *

Several memory leaks in ithreads were closed. Also, ithreads were made
less memory-intensive.

=item *

Trailing spaces are now trimmed from C<$!> and C<$^E>.

=item *

Operations that require perl to read a process' list of groups, such as reads
of C<$(> and C<$)>, now dynamically allocate memory rather than using a
fixed sized array. The fixed size array could cause C stack exhaustion on
systems configured to use large numbers of groups.

=item *

C<PerlIO::scalar> now works better with non-default C<$/> settings.

=item *

The C<x> repetition operator is now able to operate on C<qw//> lists. This
used to raise a syntax error.

=item *

The debugger now traces correctly execution in eval("")uated code that
contains #line directives.

=item *

The value of the C<open> pragma is no longer ignored for three-argument
opens.

=item *

Perl will now use the C library calls C<unsetenv> and C<clearenv> if present
to delete keys from C<%ENV> and delete C<%ENV> entirely, thanks to a patch
from Alan Burlison.

=back

=head2 More Unicode Fixes

=over 4

=item *

chr() on a negative value now gives C<\x{FFFD}>, the Unicode replacement
character, unless when the C<bytes> pragma is in effect, where the low
eight bytes of the value are used.

=item *

Some case insensitive matches between UTF-8 encoded data and 8 bit regexps,
and vice versa, could give malformed character warnings. These have been
fixed by Dave Mitchell and Yves Orton.

=item *

C<lcfirst> and C<ucfirst> could corrupt the string for certain cases where
the length UTF-8 encoding of the string in lower case, upper case or title
case differed. This was fixed by Nicholas Clark.

=back

=head1 New or Changed Diagnostics

=head2 Attempt to set length of freed array

This is a new warning, produced in situations like the following one:

    $r = do {my @a; \$#a};
    $$r = 503;

=head2 Non-string passed as bitmask

This is a new warning, produced when number has been passed as a argument to
select(), instead of a bitmask.

    # Wrong, will now warn
    $rin = fileno(STDIN);
    ($nfound,$timeleft) = select($rout=$rin, undef, undef, $timeout);
    
    # Should be
    $rin = '';
    vec($rin,fileno(STDIN),1) = 1;
    ($nfound,$timeleft) = select($rout=$rin, undef, undef, $timeout);

=head2 Search pattern not terminated or ternary operator parsed as search pattern

This syntax error indicates that the lexer couldn't find the final
delimiter of a C<?PATTERN?> construct. Mentioning the ternary operator in
this error message makes syntax diagnostic easier.

=head2 "%s" variable %s masks earlier declaration

This warning is now emitted in more consistent cases; in short, when one
of the declarations involved is a C<my> variable:

    my $x;   my $x;	# warns
    my $x;  our $x;	# warns
    our $x;  my $x;	# warns

On the other hand, the following:

    our $x; our $x;

now gives a C<"our" variable %s redeclared> warning.

=head2 readdir()/closedir()/etc. attempted on invalid dirhandle

These new warnings are now emitted when a dirhandle is used but is
either closed or not really a dirhandle.

=head1 Changed Internals

In general, the source code of perl has been refactored, tied up, and
optimized in many places. Also, memory management and allocation has been
improved in a couple of points.

Andy Lester supplied many improvements to determine which function
parameters and local variables could actually be declared C<const> to the C
compiler. Steve Peters provided new C<*_set> macros and reworked the core to
use these rather than assigning to macros in LVALUE context.

Dave Mitchell improved the lexer debugging output under C<-DT>.

A new file, F<mathoms.c>, has been added. It contains functions that are
no longer used in the perl core, but that remain available for binary or
source compatibility reasons. However, those functions will not be
compiled in if you add C<-DNO_MATHOMS> in the compiler flags.

The C<AvFLAGS> macro has been removed.

The C<av_*()> functions, used to manipulate arrays, no longer accept null
C<AV*> parameters.

=head2 B:: modules inheritance changed

The inheritance hierarchy of C<B::> modules has changed; C<B::NV> now
inherits from C<B::SV> (it used to inherit from C<B::IV>).

=head1 Reporting Bugs

If you find what you think is a bug, you might check the articles
recently posted to the comp.lang.perl.misc newsgroup and the perl
bug database at http://bugs.perl.org/ .  There may also be
information at http://www.perl.org/ , the Perl Home Page.

If you believe you have an unreported bug, please run the B<perlbug>
program included with your release.  Be sure to trim your bug down
to a tiny but sufficient test case.  Your bug report, along with the
output of C<perl -V>, will be sent off to perlbug@perl.org to be
analysed by the Perl porting team.

=head1 SEE ALSO

The F<Changes> file for exhaustive details on what changed.

The F<INSTALL> file for how to build Perl.

The F<README> file for general stuff.

The F<Artistic> and F<Copying> files for copyright information.

=cut