This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
perl5.git
3 years agoPOSIX.xs: Fix localeconv_l()
Karl Williamson [Thu, 15 Feb 2018 11:22:19 +0000 (04:22 -0700)]
POSIX.xs: Fix localeconv_l()

Commit 7d5966ae3c3b8f0e43fa09554a46cb8de9c98f5f added use of
localeconv_l().  But this commit is buggy in some locales.  It needs to
find the current locale AFTER any changes have been made due to the
LC_NUMERIC locale toggling.

3 years agoregcomp.c: Add cast to avoid compiler warning
Karl Williamson [Sun, 18 Feb 2018 15:46:56 +0000 (08:46 -0700)]
regcomp.c: Add cast to avoid compiler warning

3 years agoUpdate dist/if/Changes to current version in blead.
James E Keenan [Sun, 18 Feb 2018 00:19:58 +0000 (19:19 -0500)]
Update dist/if/Changes to current version in blead.

3 years agoAdd MANIFEST to PathTools
Sawyer X [Sat, 17 Feb 2018 20:22:04 +0000 (22:22 +0200)]
Add MANIFEST to PathTools

3 years agoReflect what I think are the correct changesets for PathTools:
Sawyer X [Sat, 17 Feb 2018 20:18:38 +0000 (22:18 +0200)]
Reflect what I think are the correct changesets for PathTools:

Building towards an upcoming release of PathTools, I'm documenting
based on the commits what I think is the correct Changeset.

Considering that at the time of this commit, CPAN only has
PathTools 3.62, I think we should be fine.

Correct if you see a mistake. :)

3 years agodefine NEED_croak_xs_usage in PathTools
Zefram [Sat, 17 Feb 2018 18:10:48 +0000 (18:10 +0000)]
define NEED_croak_xs_usage in PathTools

ppport.h nowadays defines croak_xs_usage(), which is of course used
in the XS code, but its definition is broken unless the flag macro
NEED_croak_xs_usage was defined first.  Define the flag macro to avoid
breakage.

3 years agoAdd ABSTRACT, LICENSE, AUTHOR, etc. elements.
James E Keenan [Sun, 11 Feb 2018 16:49:19 +0000 (11:49 -0500)]
Add ABSTRACT, LICENSE, AUTHOR, etc. elements.

Add them conditionally so that CPAN PathTools can accommodate different
versions of ExtUtils::MakeMaker and CPAN metadata protocols.

Styled after Module-CoreList's Makefile.PL for better readability.

See: RT 132853

3 years agoEnsure that all tests in t/cwd.t have descriptions.
James E Keenan [Sun, 11 Feb 2018 16:44:21 +0000 (11:44 -0500)]
Ensure that all tests in t/cwd.t have descriptions.

3 years agoProvide .gitignore for PathTools distribution.
James E Keenan [Sun, 11 Feb 2018 16:36:45 +0000 (11:36 -0500)]
Provide .gitignore for PathTools distribution.

3 years agoUpdate ExtUtils-MakeMaker to CPAN version 7.32
Chris 'BinGOs' Williams [Sat, 17 Feb 2018 10:42:32 +0000 (10:42 +0000)]
Update ExtUtils-MakeMaker to CPAN version 7.32

  [DELTA]

7.32  Fri Feb 16 20:10:58 GMT 2018

    No changes since v7.31_08

7.31_08 Mon Feb 12 12:19:11 GMT 2018

    VMS fixes:
    - Fix 8-space line prefixes in MM_VMS.pm

7.31_07 Tue Jan 16 16:09:04 GMT 2018

    Bug fixes:
    - Change warning text for libscan() README.pod

7.31_06 Tue Jan 16 10:28:11 GMT 2018

    Bug fixes:
    - Teach libscan() to exclude README.pod

    Doc fixes:
    - Documented when XSMULTI and XSBUILD are available
    - Documented more version requirements for attributes

7.31_05 Sat Nov 25 09:22:51 GMT 2017 (LPW Edition)

    Doc fixes:
    - Spelling fixes
    - clarify behaviour if VENDORLIB and no VENDORARCH

7.31_04 Thu Oct  5 12:22:24 BST 2017

    Darwin/NeXT fixes:
    - Escape ld -framework flags

7.31_03 Mon Jul 10 09:34:26 BST 2017

    Bug fixes:
    - Filter out non-XS .a files for static builds

7.31_02 Mon Jun 26 13:53:15 BST 2017

    VMS fixes:
    - Fixes for .PL tests
    - VMS override for static_lib_pure_cmd
    - Make the new subdirsstatic test portable

7.31_01 Wed Jun 14 15:57:52 BST 2017

    Test fixes:
    - Skip static test unless in perl core or in a .git directory

3 years agoChange name of regnode for clarity
Karl Williamson [Sat, 17 Feb 2018 04:33:24 +0000 (21:33 -0700)]
Change name of regnode for clarity

The EXACTFA nodes are in fact not generated by /a, but by /aa.  Change
the name to EXACTFAA to correspond.

I found myself getting confused by this.

3 years agoAPItest: Add U8* typemap, and use it
Karl Williamson [Mon, 12 Feb 2018 16:03:09 +0000 (09:03 -0700)]
APItest: Add U8* typemap, and use it

This missiing typemap has slowed me down on numerous occasions, as I
keep forgetting it's missing

3 years agoNo list open on VMS as yet.
Craig A. Berry [Fri, 16 Feb 2018 20:12:11 +0000 (14:12 -0600)]
No list open on VMS as yet.

So do things the old way in openpid.t on VMS only.

3 years agodon't clobber file bytes in :encoding layer
Zefram [Fri, 16 Feb 2018 17:20:34 +0000 (17:20 +0000)]
don't clobber file bytes in :encoding layer

The PerlIO::encoding layer, when used on input, was creating an SvLEN==0
scalar pointing into the byte buffer, to pass to the ->decode method
of the encoding object.  Since the method mutates this scalar, for some
encodings this led to mutating the byte buffer, and depending on where
it came from that might be something visible elsewhere that should not
be mutated.  Remove the code for the SvLEN==0 scalar, instead always
using the alternate code that would copy the bytes into a separate buffer
owned by the scalar.  Fixes [perl #132833].

3 years agoFix gcc/dmake build on Windows
Steve Hay [Fri, 16 Feb 2018 08:59:04 +0000 (08:59 +0000)]
Fix gcc/dmake build on Windows

Remove circular dependency in makefile.mk, following the same pattern as in
GNUmakefile.

3 years agoutf8.c: Silence compiler warnings
Karl Williamson [Thu, 15 Feb 2018 18:07:16 +0000 (11:07 -0700)]
utf8.c: Silence compiler warnings

These are spurious warnings, from netbsd

3 years agoavoid vivifying UNIVERSAL::isa:: in Carp
Zefram [Thu, 15 Feb 2018 18:21:14 +0000 (18:21 +0000)]
avoid vivifying UNIVERSAL::isa:: in Carp

The test added to Carp by commit 915a6810d3e3198d759f025f85d1fd6f3171dd27
for UNIVERSAL::isa being loaded had the side effect of vivifying the
UNIVERSAL::isa stash.  Take more care about checking for UNIVERSAL::isa
to avoid vivifying it, as for the other checks for things in optional
modules.  Fixes [perl #132788].

3 years agoExtUtils::ParseXS::Eval: Fix variable name ALIAS in documentation
Pali [Fri, 9 Feb 2018 11:02:34 +0000 (12:02 +0100)]
ExtUtils::ParseXS::Eval: Fix variable name ALIAS in documentation

Correct name of variable is ALIAS, not Alias.

Committer: Increment all $VERSION to 3.38 for consistency (per demerphq).

3 years agoCorrect typo: agreement in tense.
James E Keenan [Thu, 15 Feb 2018 14:16:14 +0000 (09:16 -0500)]
Correct typo: agreement in tense.

3 years agoperldelta for b6811f8d3a5c
Tony Cook [Thu, 15 Feb 2018 03:52:33 +0000 (14:52 +1100)]
perldelta for b6811f8d3a5c

3 years ago[perl #121028] avoid creating a shell process
Tony Cook [Tue, 28 Jan 2014 04:52:22 +0000 (15:52 +1100)]
[perl #121028] avoid creating a shell process

Win32 now has list form open, so this can now be supplied.

3 years agoUpdate IPC-Cmd to CPAN version 1.00
Chris 'BinGOs' Williams [Wed, 14 Feb 2018 16:51:31 +0000 (16:51 +0000)]
Update IPC-Cmd to CPAN version 1.00

  [DELTA]

1.00 Wed Feb 14 16:14:01 GMT 2018

  Bug fixes:
  - Resolve RT #91784  'run_forked("/usr/bin/echo test") captures no output'
  - Resolve RT #124415 'The 03_run_forked.t test fails (sometimes) on HPUX'

3 years agoUpdate link in documentation.
James E Keenan [Wed, 14 Feb 2018 13:47:12 +0000 (08:47 -0500)]
Update link in documentation.

Original link was reported dead and replacement link suggested by freethunk.

For: RT # 123859

3 years agoConstruct Limit.pm dependency with File::Spec.
Craig A. Berry [Sun, 11 Feb 2018 22:33:29 +0000 (16:33 -0600)]
Construct Limit.pm dependency with File::Spec.

Paths passed to WriteMakefile() will get converted to whatever
format is necessary for the make or make-like utility being used,
but paths in hard-coded extra dependencies will not.  So construct
the target name with File::Spec->catfile().

3 years agoVMS does have wchar.h.
Craig A. Berry [Sun, 11 Feb 2018 22:29:32 +0000 (16:29 -0600)]
VMS does have wchar.h.

And we now need it for prototypes of functions that locale.c is
using.

3 years agoregcomp.c: Under/i segregate folding vs non-folding characters
Karl Williamson [Mon, 5 Feb 2018 02:43:00 +0000 (19:43 -0700)]
regcomp.c: Under/i segregate folding vs non-folding characters

For matching sequences of characters, the regex compiler generates
various flavors of EXACT-type nodes.  The optimizer uses those nodes to
look for sequences that must be in the matched string.  In this way, the
pattern matching engine may be able to quickly rule out any possible
match altogether, or to narrow down the places in the target string that
might match.

Under /i matching, this generally has not been possible, because there
is no fixed string that the optimizer can grab onto, as something can
match, say, either 'A' or 'a', etc.  However, in many patterns that
contain text, there are characters that are fixed even under /i.  Things
like tabs, space, and punctuation, for example.

This commit segregates such folding vs non-folding characters into
separate nodes.  I proposed this 7 months ago:

http://nntp.perl.org/group/perl.perl5.porters/245342

and in talking with Yves recently, decided to go ahead with it.

In the proposal of July, I suggested that a new node type be used to
mark those nodes which are under /i but contain no characters that match
other than themselves under /i.  These nodes could be joined with either
a plain EXACT node, or an EXACTFish one to create longer nodes.

The reason for joining is that there is overhead in the engine whenever
we switch to the next node.  But the reason for not doing it is that it
is slower to match /i nodes than ones that an memEQ can be used on.

I suppose we could join short nodes, and leave longer ones separate, but
that decision can be deferred based on real-world experience.

This patch also consolidates into one place the handling of the Latin
Sharp S, in order to avoid extra #ifdefs, and cause the logic to be
linearly shown.

3 years agoUpgrade Encode from version 2.94 to 2.96
Steve Hay [Mon, 12 Feb 2018 20:37:36 +0000 (20:37 +0000)]
Upgrade Encode from version 2.94 to 2.96

3 years ago(perl #127743) remove the no longer needed stacksize_in.h
Tony Cook [Mon, 12 Feb 2018 00:02:26 +0000 (11:02 +1100)]
(perl #127743) remove the no longer needed stacksize_in.h

I should have removed this when I changed the limits from being baked
into the binary.

3 years ago(perl #132849) try to disable core files when deliberatly segfaulting
Tony Cook [Sun, 11 Feb 2018 23:50:24 +0000 (10:50 +1100)]
(perl #132849) try to disable core files when deliberatly segfaulting

The dist/Storable/stacksize tool probes for recursion limits for
Storable, in an attempt to prevent stack overflows, and causes
segfaults when the probes exceed the available stack size.

This was leaving a core file for each failure (whether that ended
up as a single file or one for each segfault depended on the
system settings).

Try to probe for the ulimit -c option and if it doesn't cause an
error, prefix the command used to probe for recursion limits.

Ideally I could use setrlimit() for this, but we don't expose it in
core, nor in POSIX.pm, and adding it wouldn't help for the older
perls that Storable needs to build for.

3 years agopodcheck.t: Recognize more forms of core files
Karl Williamson [Mon, 12 Feb 2018 00:39:29 +0000 (17:39 -0700)]
podcheck.t: Recognize more forms of core files

This extends 199fc8cde4ec30a56626c0765b3b0efacb327664 to include more
common forms of core files.

.*\.core.*
core
core\..*

should all now be ignored.

3 years agobump $warnings::VERSION
Tony Cook [Mon, 12 Feb 2018 00:07:14 +0000 (11:07 +1100)]
bump $warnings::VERSION

since warnings.(pm|pl) was updated in 25ebbc2270

3 years agoTest for _at_level functions with fh
Father Chrysostomos [Sun, 11 Feb 2018 20:40:09 +0000 (12:40 -0800)]
Test for _at_level functions with fh

3 years agowarnings: Omit handle when $. is 0
Slaven Rezić [Sun, 11 Feb 2018 20:36:07 +0000 (12:36 -0800)]
warnings: Omit handle when $. is 0

otherwise the new _at_level functions end up including
‘at <handle> line 0’.

[Commit message by the committer.]

3 years agoDon't export PL_underlying_numeric_obj if it doesn't exist.
Craig A. Berry [Sat, 10 Feb 2018 23:05:07 +0000 (17:05 -0600)]
Don't export PL_underlying_numeric_obj if it doesn't exist.

This broke the build on VMS and probably would on anything else
with strict linking where HAS_NEW_LOCALE is not defined.

3 years agopodcheck.t: Skip core files in all directories
Karl Williamson [Sat, 10 Feb 2018 22:17:29 +0000 (15:17 -0700)]
podcheck.t: Skip core files in all directories

Commit 446f12a860e9ab9b234cd6796e937fbff85adf5c only skipped files named
'core' in the top directory.  This should skip them in any directory.

3 years agopodcheck.t: Skip core files
Karl Williamson [Sat, 10 Feb 2018 20:50:54 +0000 (13:50 -0700)]
podcheck.t: Skip core files

This is the likely cause of
https://rt.perl.org/Ticket/Display.html?id=132843

3 years agoAPItest: Fix C++ compiles
Karl Williamson [Fri, 9 Feb 2018 17:10:47 +0000 (10:10 -0700)]
APItest: Fix C++ compiles

0a9f8f95112ff40e13406b3e5aab49c01487f045 introduced failures on C++
compilations.  This is a better patch, suggested by ilmari.

The issue was in cases where the pointer size is 32 bits and the word
size was 64, a (STRLEN) -1 returned as an error was getting turned into
0xFFFFFFFF instead of -1.

3 years agoAPItest: Correct parameter sign
Karl Williamson [Thu, 8 Feb 2018 20:51:48 +0000 (13:51 -0700)]
APItest: Correct parameter sign

Commit ae315a0a3c51e68887704d4907bb6a502a6d4e3f added tests for
utf8_to_bytes().  For testing purposes, this parameter should be signed
when it comes from perl space.

3 years agoUpgrade Time::HiRes from version 1.9752 to 1.9753
Steve Hay [Thu, 8 Feb 2018 13:40:09 +0000 (13:40 +0000)]
Upgrade Time::HiRes from version 1.9752 to 1.9753

3 years agoUpgrade Test-Simple from version 1.302120 to 1.302122
Steve Hay [Thu, 8 Feb 2018 13:37:16 +0000 (13:37 +0000)]
Upgrade Test-Simple from version 1.302120 to 1.302122

3 years agoUpgrade bignum from version 0.47 to 0.49
Steve Hay [Thu, 8 Feb 2018 13:30:27 +0000 (13:30 +0000)]
Upgrade bignum from version 0.47 to 0.49

3 years agoutil.c: Fix typo in comment
Karl Williamson [Thu, 8 Feb 2018 05:40:23 +0000 (22:40 -0700)]
util.c: Fix typo in comment

3 years ago(perl #127743) merge Storable work to date
Tony Cook [Thu, 8 Feb 2018 03:23:43 +0000 (14:23 +1100)]
(perl #127743) merge Storable work to date

This merges a large number of changes from cperl Storable, largely for
64-bit object support, and then further extends 64-bit object support
and fixes a number of other issues.

3 years agobump $Storable::VERSION to 3.06
Tony Cook [Tue, 16 Jan 2018 04:50:24 +0000 (15:50 +1100)]
bump $Storable::VERSION to 3.06

3 years ago(perl #127743) update Changes
Tony Cook [Wed, 7 Feb 2018 04:56:24 +0000 (15:56 +1100)]
(perl #127743) update Changes

3 years ago(perl #127743) re-work recursion limit handling
Tony Cook [Wed, 10 Jan 2018 00:32:43 +0000 (11:32 +1100)]
(perl #127743) re-work recursion limit handling

Previously this worked by calculating the limits and then rebuilding
Storable.so, this meant Storable.so was built twice meaning a new make
would then rebuild its dependencies.

This was also a hard limit, so if Storable's user knew the available
stack space changed they would need to rebuild Storable to adapt.

This also allows for successful static Storable builds.

I also updated the stacksize tool to prevent popping up segfault error
messages at the user on Win32.

Since I was still seeing test failures in recurse.t on a variety of
platforms, I made the default limit calculations even more
conservative.

3 years ago(perl #50608) add experimental regular expression support
Tony Cook [Tue, 19 Dec 2017 00:06:31 +0000 (01:06 +0100)]
(perl #50608) add experimental regular expression support

This doesn't work on 5.6, but I'm not sure why.

For a bad regular expression _make_re() throws an exception, and it's
propagated back to the original caller, and execution continues.

But after the program is complete retrieve_regexp() continues
execution from the call_pv(), typically with count == -1 and dies
horribly.

3 years ago(perl #127743) fix dependencies for Storable.pm
Tony Cook [Mon, 18 Dec 2017 05:06:10 +0000 (06:06 +0100)]
(perl #127743) fix dependencies for Storable.pm

Since ad2ec6b5 Storable.pm has been generated by Storable.pm.PL from
__Storable__pm.

Unfortunately the PL_FILES EU::MM key doesn't let you specify any
dependencies beyond the .PL file, so modifications to __Storable__.pm
weren't causing Storable.pm to rebuilt.

So I've replaced PL_FILES with an empty hash (to prevent EU::MM
generating once) and added my own rule.

3 years ago(perl #25933) always rethrow exceptions thrown through or by the XS implementation
Tony Cook [Mon, 11 Dec 2017 22:02:47 +0000 (23:02 +0100)]
(perl #25933) always rethrow exceptions thrown through or by the XS implementation

Also, preserve any references thrown.

Such references could be thrown by STORABLE_freeze, STORABLE_thaw or
STORABLE_attach implementations.

Several of the wrappers in Storable.pm had code similar to:

  eval { ... };
  logcroak $@ if $@ =~ s/\.?\n$/,/;

with $@ discarded if the condition failed.

This lead to two problems:

- exceptions not ending in "\n" (which is just references without
  string overloading, or with overloading but that didn't return a
  string ending in "\n") would not be rethrown.

- thrown references that did happen to have overloading that returned
  "\n" would be converted into strings.

This behaviour appears to have been present since the initial release
of Storable.

3 years ago(perl #127743) improve performance in -DDEBUGGING builds
Tony Cook [Sun, 10 Dec 2017 23:53:35 +0000 (00:53 +0100)]
(perl #127743) improve performance in -DDEBUGGING builds

The changes imported from cperl included a change to enable Storable's debugging
output mechanism for -DDEBUGGING builds.

When built with debugging output, Storable's TRACEME() macro fetched
the $Storable::DEBUGME global and checked its truth, this
significantly slowed down Storable, especially noticable for dumps
with large numbers of objects.

I added a cached traceme value to the Storable context object and
modified TRACEME() to used that instead.  A few TRACEME()'s that are
called before the context object is available have been replaced with
calls to TRACEMD(), a new macro that does what TRACEME() did before.

3 years ago(perl #131136) clear padding before writing long doubles
Tony Cook [Thu, 7 Dec 2017 00:47:47 +0000 (01:47 +0100)]
(perl #131136) clear padding before writing long doubles

This means:

- valgrind won't complain about uninitialized bytes when writing long doubles

- we get consistent output when writing equivalent data structures

3 years ago(perl #118551) an empty string from _freeze() now supplies the same to _thaw()/_attach()
Tony Cook [Wed, 6 Dec 2017 00:52:20 +0000 (01:52 +0100)]
(perl #118551) an empty string from _freeze() now supplies the same to _thaw()/_attach()

The retrieve_hook() code would simply pass the string length to
NEWSV(), so if the length was zero, no PV would be allocated, and the
SV would not be upgraded.

The following code would then set POK on an SV that wasn't SVt_PV (or
better), resulting in fun later down the line.

Change to always supply at least 1 as the buffer size for NEWSV(), and
always set CUR and NUL terminate the buffer.

3 years ago(perl #127743) ensure gzip/gzip are available before we use them
Tony Cook [Tue, 5 Dec 2017 05:53:16 +0000 (06:53 +0100)]
(perl #127743) ensure gzip/gzip are available before we use them

3 years ago(perl #127743) support for >= 4GB data from hooks
Tony Cook [Tue, 5 Dec 2017 04:24:20 +0000 (05:24 +0100)]
(perl #127743) support for >= 4GB data from hooks

Though we use this mechanism for >= 2GB.

This emits an SX_LOBJECT op and writes the string length as a 64-bit
value if STORABLE_attach() returns more than 2GB of data.

The boundary is set at 2GB since older versions of Storable handle
sizes between 2GB and 4GB badly, resulting in a memory overwrite.  By
using the alternate op codes an older Storable will reject the data
instead.

We still accept such data that might have been written by an older
Storable.

3 years ago(perl #131999) treat a 32-bit hook data size as unsigned
Tony Cook [Mon, 4 Dec 2017 23:42:48 +0000 (00:42 +0100)]
(perl #131999) treat a 32-bit hook data size as unsigned

Previously retrieve_hook() treated the 32-bit hook data size as a signed value, but
the parameter to NEWSV() is treated as unsigned.  Due to promotion rules this negative
signed value is converted to a very large unsigned value running us out of memory.

Fixing this bug will let us accept previously frozen hook data in the
2GB to 4GB rannge.  The next commit will hopefully allow us to write
and read 4GB+ data correctly.

3 years ago(perl #131990) treat the length for LSCALAR, LUTF8STR as unsigned
Tony Cook [Mon, 4 Dec 2017 04:49:27 +0000 (15:49 +1100)]
(perl #131990) treat the length for LSCALAR, LUTF8STR as unsigned

Older Storable without 64-bit object might write such objects.

A malicious file might also be provided with such a large length
to cause a resource consumption attack (allocating large amounts of
memory), but Storable explicitly doesn't protect against attacks.

3 years agowe only need 64-bit object support with 64-bit pointer builds
Tony Cook [Mon, 4 Dec 2017 03:41:45 +0000 (14:41 +1100)]
we only need 64-bit object support with 64-bit pointer builds

Previously this enabled 64-bit object support with 64-bit int builds,
such as a 32-bit build with -Duse64bitint supplied to Configure,
but a 32-bit system can't realistically have more than 2**31 array
elements or hash elements.

With an operating system built or configured especially to allow it
you might be able to find a 32-bit system that supported a PV with more
than 2**31 bytes, but that seems unlikely and not a case to optimize for.

3 years agofix various 32-bit issues:
Tony Cook [Mon, 4 Dec 2017 03:41:19 +0000 (14:41 +1100)]
fix various 32-bit issues:

- the comparison against where_is_undef caused signed vs unsigned
  comparison warnings

- don't define get_larray() on 32-bit builds

3 years ago(perl #127743) read the hash's flags for large SX_FLAG_HASH ops
Tony Cook [Wed, 22 Nov 2017 04:24:29 +0000 (05:24 +0100)]
(perl #127743) read the hash's flags for large SX_FLAG_HASH ops

Unfortunately testing this requires more memory than any machine I
have available has.

3 years ago(perl #127743) we should only handle SX_LOBJECT ops on 64-bit platforms
Tony Cook [Tue, 21 Nov 2017 23:17:33 +0000 (00:17 +0100)]
(perl #127743) we should only handle SX_LOBJECT ops on 64-bit platforms

retrieve_lobject() handled such large objects I'm assuming to handle
smaller sizes stored in a 64-bit length/count, but that can only
happen if someone isn't using Storable to generate the data.

3 years ago(perl #127743) fix network order read/write of large values
Tony Cook [Tue, 21 Nov 2017 04:55:32 +0000 (05:55 +0100)]
(perl #127743) fix network order read/write of large values

This had a couple of bugs:

- W64LEN() used a union where it should have used an array or struct,
  so a network order 64-bit length was written as 4 bytes.  Oops.

  Changed to use an array.

- retrieve_lobject() always read the length in host order, so it sort
  of tried to read the length in network order on 32-bit platforms,
  but then didn't fix the order of the length it read.

  Now uses READ_U64() (which I originally messed up myself, oops.)

3 years ago(perl #127743) fixes for READ_U64()
Tony Cook [Tue, 21 Nov 2017 04:51:07 +0000 (05:51 +0100)]
(perl #127743) fixes for READ_U64()

- ntohl() the low word too, doh

- if ntohl() isn't provided, then provide it, with appropriate
  wrapping.  Should be exteneded to other uses.

- only available for 64-bit builds

3 years ago(perl #127743) fix some -DDEBUGGING build issues
Tony Cook [Tue, 21 Nov 2017 04:12:16 +0000 (05:12 +0100)]
(perl #127743) fix some -DDEBUGGING build issues

3 years ago(perl #127743) limit the number of extra refs returned by STORABLE_freeze
Tony Cook [Mon, 20 Nov 2017 23:05:28 +0000 (00:05 +0100)]
(perl #127743) limit the number of extra refs returned by STORABLE_freeze

Ensure some over enthusiastic STORABLE_freeze() doesn't cause us to
emit a negative reference count for SX_HOOK.

This fails to be parsed by older versions and flags 64-bit ids for new
versions of Storable.

3 years ago(perl #127743) fix large object ids in hook sequences
Tony Cook [Thu, 2 Nov 2017 05:55:45 +0000 (06:55 +0100)]
(perl #127743) fix large object ids in hook sequences

As with SX_OBJECT, if there are a large number of SVs being frozen the
object ids can exceed the 32-bit limit.

The problem here is how to indicate that we have 64-bit ids without
breaking older versions parsing our results for smaller object trees?

We don't have any room in the flags byte, currently used as:

 bits  use
 ----  ---
 0,1   type
 2     class length is > 255 bytes
 3     frozen string > 255 bytes
 4     more than 255 extra references
 5     class name has been seen before and is represented as an index
 6     an extra reference is stored next
 7     has a list of extra references

and the extra byte is only used for tied SVs.  We can't repurpose the
bits 6, 7 since it would break older readers.

I considered adding SX_LARGE_HOOK, or something similar, but we find
out that large ids are needed well after the op code has been emitted.

So how is the handling of the length + ids handled in retrieve_hook()?

    I32 len3 = 0;
    ...
        if (flags & SHF_LARGE_LISTLEN)
            RLEN(len3);
else
            GETMARK(len3);
        if (len3) {
            av = newAV();
            av_extend(av, len3 + 1); /* Leave room for [0] */
            AvFILLp(av) = len3; /* About to be filled anyway */
        }

For SHF_LARGE_LISTLEN this treats the len3 as signed - so if the
writer had more than 2**31-2 objects after the "frozen" string,
decoding is going to break anyway, as av_extend_guts() panics on "key"
values less than -1.

The code that actually reads the ids will fail to read any ids when
len3 is negative, since the for loop does a i <= len3 check.

So rather than trying to fix this, I used a negative len3 to indicate
that the object ids are 64-bit.

This means we get backward compatibility in the cases where 32-bit ids
work, and older versions of Storable will reject it.

If an older version of Storable wrote a negative len3 (due to
overflow) we'll be attempting to read 32-bit ids as typically very
large 64-bit ids (much larger than 2**32 in most cases) which won't be
found in the seen array, failing the thaw.

3 years ago(perl #127743) update PERL_TEST_MEMORY requirements for the older tests
Tony Cook [Wed, 1 Nov 2017 22:58:49 +0000 (23:58 +0100)]
(perl #127743) update PERL_TEST_MEMORY requirements for the older tests

PERL_TEST_MEMORY includes any swap, it's not the minimum resident set
for the test.

3 years ago(perl #127743) fix two problems with large object ids
Tony Cook [Wed, 1 Nov 2017 22:31:39 +0000 (23:31 +0100)]
(perl #127743) fix two problems with large object ids

Storable assigns an object id to every scalar it freezes, including to
unused elements in arrays.

There were two problems here:

a) in retrieve(), object ids over 2**31-1 but less than 2**32 were
   treated as signed, so the wrong object was produced in the resulting
   data structure.  Two changes we made to fix this:

  i)  retrieve() now treats object ids in the problem range as
      unsigned, so data written by older Storables is now treated
      correctly.

  ii) store() now writes object ids in the problem range as 64-bit
      ids, so that older Storables will fail rather than producing an
      incorrect result data structure.

b) once over 2**32 scalars had been output, the code still produced
   32-bit object ids when referring to previous scalars.  Fixed by
   adding support for 64-bit object ids.

There's still an issue with object ids in hook produced data.

Testing these changes requires ridiculous amounts of memory - ~32GB
for a) and ~66GB for b), and the tests take a long time to run, hence
for those tests to run you need the following in the environment;

  PERL_TEST_MEMORY >= 70
  PERL_RUN_SLOW_TESTS != 0

3 years ago(perl #127743) correct some types for Win32
Tony Cook [Wed, 25 Oct 2017 02:55:52 +0000 (13:55 +1100)]
(perl #127743) correct some types for Win32

In a couple of places the code uses unsigned long to represent values
possibly larger than 2**32, but long is 32-bits on x64 Win32, so
use STRLEN or Ssize_t instead.

3 years agoStorable 3.05_13: safe defaults
Reini Urban [Mon, 24 Jul 2017 11:58:37 +0000 (13:58 +0200)]
Storable 3.05_13: safe defaults

mingw fix: use safe defaults, not segfaulting defaults.
mingw fails on the stacksize binary search, leaving it empty.

(cherry picked from commit 27c613e5f8dfab7b8b43a57e93a478aa962dd9ca)

Conflicts:
dist/Module-CoreList/lib/Module/CoreList.pm
dist/Storable/Storable.pm
pod/perlcdelta.pod
t/porting/customized.dat

3 years agoStorable: Analyze #304
Reini Urban [Fri, 14 Jul 2017 14:15:40 +0000 (16:15 +0200)]
Storable: Analyze #304

Add a better error message on one of the possible failure causes
when the system cannot catch the stackoverflow.

(cherry picked from commit 3f655dc6444cd56d09d4a99a4be5a92c1119506b)

3 years agoStorable: cygwin64 MAX_DEPTH /2
Reini Urban [Wed, 17 May 2017 20:40:59 +0000 (22:40 +0200)]
Storable: cygwin64 MAX_DEPTH /2

(cherry picked from commit 5c3259bf9a54c29b526cb36cdf155005d8f93236)

3 years agoStorable: Need more stack reserve on cygwin also
Reini Urban [Sun, 14 May 2017 20:28:21 +0000 (22:28 +0200)]
Storable: Need more stack reserve on cygwin also

Same stack quirks as on MSWin32.
cygwin64 cannot even do 32, try 40.

(cherry picked from commit c663e2126c5060f80eb973828a341606f210f671)

3 years agoStorable: fix cygwin stack probing
Reini Urban [Fri, 12 May 2017 07:49:38 +0000 (09:49 +0200)]
Storable: fix cygwin stack probing

don't set the PATH needed for other DLLs also.

(cherry picked from commit d0f8b62c3f9ffee9fa8397d4acdadbd695cc4839)

3 years agoStorable: Update to 3.05_12
Reini Urban [Wed, 19 Apr 2017 07:18:49 +0000 (09:18 +0200)]
Storable: Update to 3.05_12

enhance stack reserve from 8 to 16

(cherry picked from commit e344ff0826555342fe207a4a52bcdda4b00e2c44)

Conflicts:
Porting/Maintainers.pl
dist/Module-CoreList/lib/Module/CoreList.pm
dist/Storable/Storable.pm
pod/perlcdelta.pod
t/porting/customized.dat

3 years agoStorable: fix stacksize with shared libcperl
Reini Urban [Tue, 18 Apr 2017 08:39:29 +0000 (10:39 +0200)]
Storable: fix stacksize with shared libcperl

LD_LIBRARY_PATH=dist/Storable/../..
Interestingly only detected on Solaris, not elsewhere.

(cherry picked from commit 4fc1e35393d951b249e36e02f89ec8d0f9cd7537)

3 years agoStorable: fix wrong fake_tag check
Reini Urban [Thu, 30 Mar 2017 09:22:31 +0000 (11:22 +0200)]
Storable: fix wrong fake_tag check

coverity CID 165506. if (!sv) is always true. check fake_tag instead.

(cherry picked from commit 3cb71a68201936293a78ed93ca8049ee4cba5497)

3 years agoStorable: memory leak
Reini Urban [Thu, 30 Mar 2017 07:04:07 +0000 (09:04 +0200)]
Storable: memory leak

detected by coverity

(cherry picked from commit b60e26ff913f3ada34f57e79347e11d3872876c9)

3 years agoStorable 3.05_11: croak on sizes read > I32_MAX
Reini Urban [Wed, 29 Mar 2017 19:04:28 +0000 (21:04 +0200)]
Storable 3.05_11: croak on sizes read > I32_MAX

detected by coverity:
CID 165606 (#2 of 2): Untrusted value as argument (TAINTED_SCALAR)
146. tainted_data: Passing tainted variable size to a tainted sink.

(cherry picked from commit 735d9229a540d05f2d34f46ffea1e68aea9fbbc1)

Conflicts:
dist/Storable/Storable.pm

3 years agoStorable: simplify last_op_in_netorder
Reini Urban [Wed, 29 Mar 2017 19:52:30 +0000 (21:52 +0200)]
Storable: simplify last_op_in_netorder

remove dead code, detected by coverity

(cherry picked from commit f1e1f909cfd3e474a0b11f0200a3889b8ff2cc6f)

3 years agoStorable: protect from empty retrieve_vstring
Reini Urban [Wed, 29 Mar 2017 19:32:57 +0000 (21:32 +0200)]
Storable: protect from empty retrieve_vstring

empty sv detected by coverity

(cherry picked from commit 96c0b32bf6ab37f5e82494b79149680012dde49f)

3 years agoStorable: protect store_other error buf
Reini Urban [Wed, 29 Mar 2017 19:29:46 +0000 (21:29 +0200)]
Storable: protect store_other error buf

coverity warns about static buffer overflow.

(cherry picked from commit 74804347b8da42bc5b471646032c2e8d9903106f)

3 years agoStorable: Bump to 3.05_10
Reini Urban [Tue, 14 Mar 2017 09:05:24 +0000 (10:05 +0100)]
Storable: Bump to 3.05_10

No changes, synced cpan with CORE

(cherry picked from commit 0edca17dff085908bd7c3286ffabb3c0e9371706)

Conflicts:
Porting/Maintainers.pl
dist/Module-CoreList/lib/Module/CoreList.pm
dist/Storable/Storable.pm
pod/perlcdelta.pod
t/porting/customized.dat

3 years agoStorable: various core fixups: $PERL,libpth,getcwd
Reini Urban [Sun, 12 Mar 2017 11:28:31 +0000 (12:28 +0100)]
Storable: various core fixups: $PERL,libpth,getcwd

stacksize needs to run perl, not miniperl,
but the chdir broke $^X (a relative path).
Quote $^X only with whitespace.
Set LD_LIBRARY_PATH with a shared libperl.
Support a --core argument to stacksize.
Probe it twice.

(cherry picked from commit 0df5d8e463b9b5cfe14f9c235191fd3e5e4013a4)

3 years agoStorable: core Makefile.SH support
Reini Urban [Fri, 10 Mar 2017 09:59:53 +0000 (10:59 +0100)]
Storable: core Makefile.SH support

redo building Storable after perl was built,
with the probed stacksize.h values.
We can only produce stacksize.h with a PERL_EXE
so we have to do it twice.

TonyC:

 - this fixes the stack overflow iff the rebuild_storable target runs
   after storable is built, but there's nothing preventing
   lib/auto/Storable/Storable.so being built *after* rebuild_storable
   in a parallel build at this point.

 - It's also possible for both the lib/...Storable.so target and the
   rebuild_storable target to run at the same time, possibly
   corrupting the generated binary (check 4d106cc5 for ext/SDBM_File
   for example)

Conflicts:
.git-rr-cache
pod/perlcdelta.pod
win32/Makefile.ce

3 years agoStorable 3.05_09: compute stacksize
Reini Urban [Thu, 9 Mar 2017 08:49:31 +0000 (09:49 +0100)]
Storable 3.05_09: compute stacksize

(cherry picked from commit a10c78df8c03018ce50f0885bb6b6a5509d9e58f)

TonyC:
 - dist/Storable/t/recurse.t is still crashing on a stack overflow

Conflicts:
Porting/Maintainers.pl
dist/.gitignore
dist/Module-CoreList/lib/Module/CoreList.pm
dist/Storable/Storable.pm
pod/perlcdelta.pod
t/porting/customized.dat

3 years agoStorable 3.05_08: adjust MAX_DEPTH
Reini Urban [Wed, 8 Mar 2017 15:18:32 +0000 (16:18 +0100)]
Storable 3.05_08: adjust MAX_DEPTH

2000 is too large for DEBUGGING with clang++. It runs into stack overflow.
Seperate between c++ and c, debugging and non, 32bit and 64bit.

Tune the stack sizes for various variants.
New stack recursion limits:
64: c++ dbg: 1200   DEBUGGING + DEBUG_LEAKING_SCALAR non-threaded
    c   dbg: 3000
    c -O3:   34000 (tested: 34500)
32: c -O3:   30000 (tested: 32600)
    c   dbg: 750   (tested: 1500)
asan: 254

Note that the result has to be int, as the value is then right shifted by the
pre-processor, >> 1, for hashes (i.e. divide by 2).

TonyC:
 - Storable.o fails to build, but the offending code is removed in the
   next commit

(cherry picked from commit a7aff4921e16004c42e40914914e5f3320238e12)

Conflicts:
dist/Module-CoreList/lib/Module/CoreList.pm
dist/Storable/Storable.pm
dist/Storable/t/blessed.t
pod/perlcdelta.pod
t/porting/customized.dat

3 years agoStorable 3.05_07: update documentation from CPAN
Reini Urban [Sun, 5 Mar 2017 10:29:54 +0000 (11:29 +0100)]
Storable 3.05_07: update documentation from CPAN

from the CPAN version.

(cherry picked from commit 81259118dd8e16b10dfbdcfe1e2355bae507ac51)

Conflicts:
.git-rr-cache
dist/Storable/Storable.pm
pod/perlcdelta.pod

3 years agoStorable: improve recursion depth check
Reini Urban [Sun, 5 Mar 2017 10:09:01 +0000 (11:09 +0100)]
Storable: improve recursion depth check

Store each recursive SV and only ++depth when recursing
into this same SV, not nested into 2 levels, for performance reasons.
(We'd really need to store a hash, but this would be slow.)
So it only protects from the simpliest stack exhaustion attacks,
others still will segfault.
Decrease the max depth numbers: 2000 for RV and AV, 1000 for HV,
experimentally tested for normal stack sizes.

At least it doesnt falsely error with an arrayref nested into another
big array, as with the CPAN write_metadata_cache.
Check now also refs recursion into each other.

Closes #257.

(cherry picked from commit cd2b11a42ef9c6829960039382a06622d7248755)

Conflicts:
dist/Storable/Storable.pm
pod/perlcdelta.pod

3 years agoStorable: add testcase for #257
Reini Urban [Sat, 4 Mar 2017 11:42:36 +0000 (12:42 +0100)]
Storable: add testcase for #257

detected by CPAN,
wrong recursion warning with CPAN write_metadata_cache.
There's no recursion involved, just nesting a hash into a big
array will fail.

(cherry picked from commit 2804828cb8c402649cc89f1c40fda63932ccae44)

3 years agoFix rurban email address
Reini Urban [Tue, 28 Feb 2017 22:34:45 +0000 (23:34 +0100)]
Fix rurban email address

use the generic cpan.org

(cherry picked from commit 90875988463bbf4b10af59633e23a6a2b2a2cfbe)

TonyC:
 - at this point dist/Storable/t/recurse.t crashes with a stack
   overflow, presumably fixed by the later "tuning" commits

Conflicts:
.travis.yml
AUTHORS
Porting/do-conf-cperl-release
Porting/do-make-cperl-release
Porting/release_cperl.pod
appveyor.yml
cpan/CPAN-Meta-Requirements/lib/CPAN/Meta/Requirements.pm
cpan/CPAN-Meta/lib/CPAN/Meta.pm
cpan/Config-Perl-V/t/28_plv522c.t
cpan/Cpanel-JSON-XS/XS.pm
cpan/Scalar-List-Utils/lib/Sub/Util.pm
cpan/YAML-LibYAML/lib/YAML/XS.pod
dist/CPAN-Meta/lib/Parse/CPAN/Meta.pm
ext/Config/Makefile.PL
pod/cperl.pod
pod/perl.pod
t/porting/customized.dat

3 years agoStorable: Update to 3.05_03
Reini Urban [Mon, 30 Jan 2017 21:16:44 +0000 (22:16 +0100)]
Storable: Update to 3.05_03

Appveyor win64 fails to create nested hash.
Need to repro it on a failing machine. Maybe Win64
uses a much bigger stack for its threads. But 5000 should
still be doable, as other machines can do 22_000.

(cherry picked from commit 81786bf3f6be1f6326b588eca9ae3ccc4824b5a8)

Conflicts:
.git-rr-cache
Porting/Maintainers.pl
dist/Storable/Storable.pm
pod/perlcdelta.pod

3 years agoStorable: Update to 3.05_02
Reini Urban [Mon, 30 Jan 2017 16:52:24 +0000 (17:52 +0100)]
Storable: Update to 3.05_02

Protect against stack overflows with nested arrays and hashes
[cpan #97526]. This imposes a new limit to your nested structures,
but JSON::XS has a limit of 512. We use a max_depth of 3000 for the
typical stack limit of 8k.
You can still overflow the call stack with nested scalar refs.

(cherry picked from commit 63a5b7f922c2900e328ce97b57d0a54fbc763a98)

Conflicts:
.git-rr-cache
Porting/Maintainers.pl
dist/Storable/Storable.pm
pod/perlcdelta.pod

3 years agoStorable: Upgrade to 3.05 from CPAN
Reini Urban [Sun, 29 Jan 2017 11:43:18 +0000 (12:43 +0100)]
Storable: Upgrade to 3.05 from CPAN

See https://github.com/rurban/Storable/

(cherry picked from commit 17a1797c711ed6de48985b0746ae59282f634b12)

Conflicts:
Porting/Maintainers.pl
dist/Storable/Storable.pm
dist/Storable/Storable.xs
pod/perlcdelta.pod
t/porting/customized.dat

3 years agoStorable: re-indent from 8 hard tabs to soft 4
Reini Urban [Fri, 27 Jan 2017 23:10:17 +0000 (17:10 -0600)]
Storable: re-indent from 8 hard tabs to soft 4

(cherry picked from commit a5b8bb11df1a50b585a1b07965c860c234877ce3)

Conflicts:
dist/Storable/Storable.xs
t/porting/customized.dat

3 years agoStorable: skip LOBJECT on 32bit
Reini Urban [Fri, 27 Jan 2017 16:45:53 +0000 (10:45 -0600)]
Storable: skip LOBJECT on 32bit

croak there.  There is no way to store 64bit objects, and when reading them
we need to croak.  We check the len though (at least on little-endian, on
big-endian this is broken.)

(cherry picked from commit d6e38371a2997e9546cbf340148a41439091b3c2)

Conflicts:
.git-rr-cache
pod/perlcdelta.pod
t/porting/customized.dat

3 years agoStorable: fixup huge
Reini Urban [Thu, 26 Jan 2017 23:31:35 +0000 (17:31 -0600)]
Storable: fixup huge

followup to 67a5186ac1aee4b63 with 3.00c.
MBUF_XTEND and more internal sizes, esp. the dclone size,
needs to use long, not int.
Improve TRACEME to cut off printing overlong strings.

(cherry picked from commit d987f810b5f285b075103f58ce43e54d181643bc)

Conflicts:
pod/perlcdelta.pod
pod/perlcperl.pod
t/porting/customized.dat

3 years agoStorable: protect against classname len overflow
Reini Urban [Wed, 25 Jan 2017 17:27:07 +0000 (11:27 -0600)]
Storable: protect against classname len overflow

name lenghts can be max. I32 (hek_len), but are read from
system size integers. e.g. -1 you can cause a malloc fail => exit.

Even worse len > LG_BLESS didn't detect that so you could overwrite
the stack by malcrafted Storable files.
Found out by JD. RT #130635. No CVE since p5p believes local Storable
files are not exploitable.

(cherry picked from commit c82e80f6d88891738e2b5329723606b48347fa31)

Conflicts:
.git-rr-cache
Porting/Maintainers.pl
dist/Storable/Storable.xs
pod/perlcdelta.pod
t/porting/customized.dat

3 years agoStorable 3.04c: fix printf types
Reini Urban [Sat, 7 Jan 2017 08:01:29 +0000 (09:01 +0100)]
Storable 3.04c: fix printf types

esp. for 32bit use64bitint:
Storable.xs:2348:4: warning: shift count >= width of type [-Wshift-count-overflow]
                        STORE_PV_LEN((const char *)mg->mg_ptr, len

cast printf args to the right type.

(cherry picked from commit c08d0a10791ec59b04354a1fbd2ed07b3681c703)

Conflicts:
.git-rr-cache
dist/Storable/Storable.pm
t/porting/customized.dat

3 years agoStorable: Update to 3.02c
Reini Urban [Sun, 20 Nov 2016 17:06:45 +0000 (18:06 +0100)]
Storable:  Update to 3.02c

Fix -Wc++11-compat warnings

(cherry picked from commit 852424577630848b6ed6faef3f3a0396f2e91bb8)

Conflicts:
dist/Storable/Storable.pm
dist/Storable/Storable.xs
pod/perlcdelta.pod
t/porting/customized.dat

3 years agoStorable: fix -Wchar-subscripts
Reini Urban [Mon, 10 Oct 2016 10:14:49 +0000 (12:14 +0200)]
Storable: fix -Wchar-subscripts

array subscript is of type char. in DEBUGGING code only.

(cherry picked from commit 07af5b27a799f0b88a5368d81fd57c7643e2edff)

Conflicts:
t/porting/customized.dat