Change filter of problematic code points for EBCDIC

There are three classes of problematic Unicode code points that may
require special handling.  Which code points are problematic is fairly
complicated, requiring lots of branches.  However, the smallest of them
is 0xD800, which means that most code points in modern use are below
them all, and a single test can be used to exclude just about everything
likely to be encountered.  The problem was that the way this test was
done on EBCDIC caused way too many things to pass and have to be checked
with the more complicated branches.  The digits 0-9 and some capital
letters were not filtered out, for example.  This commit changes the
EBCDIC test to transform into I8 (an array lookup), and this fixes it to
exclude things that shouldn't have passed before.

diff --git a/utf8.c b/utf8.c
index 2a9d20e..2a44d75 100644 (file)
--- a/utf8.c
+++ b/utf8.c
@@ -3840,7 +3840,7 @@ Perl_check_utf8_print(pTHX_ const U8* s, const STRLEN len)
                           "%s in %s", unees, PL_op ? OP_DESC(PL_op) : "print");
            return FALSE;
        }
                           "%s in %s", unees, PL_op ? OP_DESC(PL_op) : "print");
            return FALSE;
        }

-       if (UNLIKELY(*s >= UTF8_FIRST_PROBLEMATIC_CODE_POINT_FIRST_BYTE)) {
+       if (UNLIKELY(isUTF8_POSSIBLY_PROBLEMATIC(*s))) {

            STRLEN char_len;
            if (UTF8_IS_SUPER(s, e)) {
                if (ckWARN_d(WARN_NON_UNICODE)) {
            STRLEN char_len;
            if (UTF8_IS_SUPER(s, e)) {
                if (ckWARN_d(WARN_NON_UNICODE)) {

diff --git a/utf8.h b/utf8.h
index 85bf590..d8769b5 100644 (file)
--- a/utf8.h
+++ b/utf8.h
@@ -552,8 +552,11 @@ case any call to string overloading updates the internal UTF-8 encoding flag.
  * problematic in some contexts.  This allows code that needs to check for
  * those to to quickly exclude the vast majority of code points it will
  * encounter */
  * problematic in some contexts.  This allows code that needs to check for
  * those to to quickly exclude the vast majority of code points it will
  * encounter */

-#define UTF8_FIRST_PROBLEMATIC_CODE_POINT_FIRST_BYTE \
-                                    FIRST_SURROGATE_UTF8_FIRST_BYTE
+#ifdef EBCDIC
+#   define isUTF8_POSSIBLY_PROBLEMATIC(c) (NATIVE_UTF8_TO_I8(c) >= 0xF1)
+#else
+#   define isUTF8_POSSIBLY_PROBLEMATIC(c) ((U8) c >= 0xED)
+#endif

 
 /* Several of the macros below have a second parameter that is currently
  * unused; but could be used in the future to make sure that the input is
 
 /* Several of the macros below have a second parameter that is currently
  * unused; but could be used in the future to make sure that the input is