utf8.h: Don't allow zero length malformation unless requested

The bottom level Perl routine that decodes UTF-8 into a code point has
long accepted inputs where the length is specified to be 0, returning a
NUL.  It considers this a malformation, which is accepted in some
scenarios, but not others.  In consultation with Tony Cook, we decided
this really isn't a malformation, but is a bug in the calling program.
Rather than call the decode routine when it has nothing to decode, it
should just not call it.

This commit removes the acceptance of a zero length string from any of
the canned flag combinations passed to the decode function.  One can
convert to specify this flag explicitly, if necessary.  However the next
commit will cause this to fail under DEBUGGING builds, as a step towards
removing the capability altogether.

diff --git a/utf8.h b/utf8.h
index 3da10e0..a4cae09 100644 (file)
--- a/utf8.h
+++ b/utf8.h
@@ -787,12 +787,16 @@ case any call to string overloading updates the internal UTF-8 encoding flag.
 #define UTF8_WARN_ILLEGAL_INTERCHANGE \
                           (UTF8_WARN_ILLEGAL_C9_INTERCHANGE|UTF8_WARN_NONCHAR)
 
-#define UTF8_ALLOW_ANY                                                          \
-           (~( UTF8_DISALLOW_ILLEGAL_INTERCHANGE|UTF8_DISALLOW_ABOVE_31_BIT    \
-               |UTF8_WARN_ILLEGAL_INTERCHANGE|UTF8_WARN_ABOVE_31_BIT))
-#define UTF8_ALLOW_ANYUV  UTF8_ALLOW_EMPTY
-#define UTF8_ALLOW_DEFAULT             (ckWARN(WARN_UTF8) ? 0 : \
-                                        UTF8_ALLOW_ANYUV)
+#define UTF8_ALLOW_ANY ( UTF8_ALLOW_CONTINUATION                                \
+                        |UTF8_ALLOW_NON_CONTINUATION                            \
+                        |UTF8_ALLOW_SHORT                                       \
+                        |UTF8_ALLOW_LONG)
+
+/* Accept any Perl-extended UTF-8 that evaluates to any UV on the platform, but
+ * not any malformed.  This is the default.  (Note that UVs above IV_MAX are
+ * deprecated. */
+#define UTF8_ALLOW_ANYUV   0
+#define UTF8_ALLOW_DEFAULT UTF8_ALLOW_ANYUV
 
 /*
 =for apidoc Am|bool|UTF8_IS_SURROGATE|const U8 *s|const U8 *e