[perl #123677] Crash with token stack overflow

[perl #123677] Crash with token stack overflow

In this naughty code snippet:

s)$0{0h());qx(@0);qx(@0);qx(@0)

the s)...)) is treated as a substition, with $0{0h( for the left part.
When the lexer reaches the h( it tries to emit two tokens at once, '&'
and a WORD token representing the h.  To do that it pushes the WORD on
to the pending token stack and then emits '&'.  The next call to yylex
will usually pop the token off the pending stack and use that, because
the lexing state (PL_lex_state) is LEX_KNOWNEXT.

However, when the parser sees '&', it immediately reports it as
a syntax error, and tries to pop tokens to make sense of what it
has, popping scopes in the process.  Inside a quote-like operator,
PL_lex_state is localised, so the value after this scope-popping is
no longer LEX_KNOWNEXT, so the next call to yylex continues parsing
‘;qx...’ and ignores the pending token.

When it reaches the @0 inside the qx, it tries to push five pending
tokens on to the stack at once, because that’s how the implicit join
works.  But the stack only has room for five items.  Since it already
has one, the last item overflows, corrupting the parser state.
Crashes ensue.

If we check for the number of pending tokens and always emit any
regardless of the lexing state, then we avoid the crash.  This is
arguably how it should have been written to begin with.

This makes LEX_KNOWNEXT, and probably PL_lex_defer, redundant, but I
will wait till after perl 5.22 before removing those, as the removal
may break CPAN modules, and this is a little late in the dev cycle.