This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Nicholas Clark [Sat, 2 Jun 2012 15:32:44 +0000 (16:32 +0100)]
mktables memory reduction
Does the attached patch make sense? It lowers RAM and CPU usage by about 10%
on Linux, and 6% on FreeBSD.
Nicholas Clark
>From
fe46bd796c282f6a6e4793afaf847e04d3be3524 Mon Sep 17 00:00:00 2001
From: Nicholas Clark <nick@ccl4.org>
Date: Mon, 7 May 2012 09:58:13 +0200
Subject: [PATCH] In mktables, lazily compute the 'standard_form' for Ranges.
Instead of calculating the standard form up front, calculate it only when
needed and cache the result. There are 368676 non-special objects, but
the standard form is only requested for 22047 of them. For the systems I
tested on, this reduces RAM and CPU usage by about 10% on Linux, and 6% on
FreeBSD.
This is more significant than it may first seem, because mktables is the
largest RAM user of anything run during the build process, so this reduces
the build process peak RAM requirement.
Karl Williamson [Fri, 1 Jun 2012 19:31:31 +0000 (13:31 -0600)]
podcheck.t: white-space only: indent properly
Karl Williamson [Sat, 2 Jun 2012 15:54:14 +0000 (09:54 -0600)]
perldelta entry for Perl working on all Unicode releases
Craig A. Berry [Sat, 2 Jun 2012 15:15:57 +0000 (10:15 -0500)]
VMS --> __VMS in a2p.h.
C++ does not define VMS, only __VMS. We define VMS in config.h
for C++, but here we're trying to figure out how to include config.h,
so we don't have it yet.
The only compiler that defined only VMS and not __VMS was VAX C,
which isn't C89-compliant and can't build Perls from the current
century.
Karl Williamson [Sat, 2 Jun 2012 14:37:18 +0000 (08:37 -0600)]
Allow Perl to use older Unicode releases
We have claimed for several releases now that Perl can handle any
release of Unicode so far. It turns out that this was never entirely
true, and became less true in 5.16 due to the changes necessitated by
Unicode's publishing names for the control characters. Enough work was
done in 5.16 so that Perl could continue to handle Unicode 6.0, which
meant that someone using Perl 5.14 and Unicode 6.0, and who didn't like
6.1 could still upgrade to Perl 5.16 while retaining 6.0. By having
mktables able to compile old releases of Unicode, we can continue to
evolve Perl without having to write special code in the core for
backwards Unicode compatibility.
This series of commits is mostly about allowing Perl to work on any
Unicode release. And, I have done a make test using these patches on
recent bleads for every Unicode release. Tests fail which use code
points that have yet to be assigned in those earlier releases, as well
as tests that expect certain values that Unicode has since changed, but
Perl works as well as can be expected on all releases. I changed a few
.t's so that they adapt more to the Unicode release they are run with.
The biggest outage is Unicode::Normalize. Prior to Unicode 3.0, there
could be decompositions containing more than 2 code points.
Unicode::Normalize will not compile with those early versions. However,
a trivial change to it (documented now in mktables) to use carp instead
of croak will cause it to compile and run, though all 3-or-more
character decompositions will be unknown to it. I think that the
earliest release of Unicode that it makes sense to do serious work on is
3.2.
There is the issue as to how to prevent future changes to mktables from
breaking backwards compatibility. I don't know how to fix that, except
that as long as I'm involved, I will test it on old releases some time
before code freeze.
Karl Williamson [Fri, 1 Jun 2012 02:22:26 +0000 (20:22 -0600)]
mktables: Use for loop instead of each
I think the 'for' is easier to understand
Karl Williamson [Fri, 1 Jun 2012 02:21:13 +0000 (20:21 -0600)]
mktables: Allow easy generation of Unicode-deprecated files
Sometimes in debugging, etc, it is useful to have these files; this adds
a single scalar to control if they get generated.
Karl Williamson [Fri, 1 Jun 2012 02:20:20 +0000 (20:20 -0600)]
mktables, README.perl: Add, fix comments
Karl Williamson [Fri, 1 Jun 2012 01:06:31 +0000 (19:06 -0600)]
perl.pod: Remove reference to specific Unicode version
I forgot to update this from 6.1, and it really isn't necessary. Also
update the instructions for new releases to omit reference to this.
Karl Williamson [Wed, 16 May 2012 17:31:50 +0000 (11:31 -0600)]
utf8.c: Use new internal properties for \X
These new properties are generated for all Unicode releases, and so \X
can now work on all Unicodes, not just the ones where Unicode has
defined them.
Karl Williamson [Wed, 16 May 2012 17:30:53 +0000 (11:30 -0600)]
mktables: White-space only
This outdents code since the surrounding blocks have been removed
Karl Williamson [Wed, 16 May 2012 16:38:49 +0000 (10:38 -0600)]
mktables: Use modern \X definition for early Unicodes
\X matches an extended grapheme cluster, and requires the Grapheme
Cluster Break Property, and the Hangul Syllable Type property for
complete proper functioning. This concept was not introduced until
Unicode 5.1. Prior to that was the basic "grapheme cluster", which was
introduced in Unicode series 3. The extended version is only a slight
modification of the basic one, and so the basic one can be used for
adequate processing of most scripts. That is, if we use the basic
definition in Unicode version that lack the extra information for the
extended version, things are degraded cleanly. It would be possible to
add intelligence to mktables to generate the extended version for all
releases, but since the difference between the two isn't great, the
payoff isn't worth it.
Until commit
5edb45fb0350d27b109ca489724513b86f9cb7bc for Perl v5.12,
Perl used a simpler definition for \X: a non-mark character followed by
any number of marks. Until the current commit, that definition was
clung to when run on early Unicodes that didn't furnish the definitions
of the properties needed to generate the grapheme cluster definition.
However, it turns out that it is fairly easy to generate those property
definitions, and this commit does that. And, once those are done, the
special cases for early versions can be removed (except for the
distinction between extended and non- grapheme clusters).
This commit generates the needed tables into Perl internal properties.
A future commit will change the code that processes \X to use these; in
the meantime it uses the existing properties that are generated anyway
for Unicodes where they don't have to be calculated, and which are
identical to the new internal ones. (Code elsewhere in mktables assures
that identical properties share the same file.) By using internal
properties, application code only gains access to the official
properties in the releases they are officially defined. For example, it
won't work for an application to use the Hangul Syllable Type property
unless it is in a release where Unicode has defined it. To allow access
in earlier releases would be misrepresenting what the Standard says.
Karl Williamson [Wed, 16 May 2012 16:27:36 +0000 (10:27 -0600)]
perluniprops: Work correctly with PLACEHOLDER entries
mktables generates perluniprops.pod, but until this commit, PLACEHOLDER
entries could confuse it.
Karl Williamson [Wed, 16 May 2012 16:22:18 +0000 (10:22 -0600)]
mktables: Create get accessor for object
A future commit will want to get at the value of this object.
Karl Williamson [Wed, 16 May 2012 04:19:51 +0000 (22:19 -0600)]
mktables: Make sure a \p{ID_Continue} is also \p{Word}
This is for consistency with ID_Start. This property has yet to be used
in the Perl core.
Karl Williamson [Wed, 16 May 2012 04:16:47 +0000 (22:16 -0600)]
mktables: Improve defn of xidc for early Unicodes
Karl Williamson [Wed, 16 May 2012 04:11:16 +0000 (22:11 -0600)]
mktables: Improve \p{xids} defn for early Unicodes
Karl Williamson [Wed, 16 May 2012 04:07:48 +0000 (22:07 -0600)]
mktables: Defined \p{AHex} and \p{ASCII_Hex_Digit} for early Unicodes
A number of tests presume these are always defined; and probably a bunch
of real code as well.
Karl Williamson [Wed, 16 May 2012 03:59:55 +0000 (21:59 -0600)]
mktables: Improve Lower, Upper defns in early Unicodes
This adds the missing code points to the existing definitions to make
them better for early Unicode releases
Karl Williamson [Wed, 16 May 2012 03:45:56 +0000 (21:45 -0600)]
mktables: Remove early Unicode defective \p{Alpha=Y}
The \p{Alphabetic=y} property was not defined in all Unicode releases;
however in some of those early ones, there was a data file that
contained a definition for it, and prior to this patch, mktables used
that definition to construct a \p{Alphabetic=y} table. However, it
turns out that the definition is quite defective in many of the releases
it occurred in. So rather than mislead code into thinking there is a
good definition of that property for the early releases, this just
doesn't generate a table for it.
But, prior commits have created a good definition for the Perl
single-form extensions \p{Alpha} and \p{Alphabetic}, and most code uses
those anyway.
Karl Williamson [Wed, 16 May 2012 03:39:54 +0000 (21:39 -0600)]
mktables: Improve \p{Alphabetic} definition for early Unicodes
Prior to this patch, the definition of this property was admittedly
approximate. This cleans that up.
Karl Williamson [Wed, 16 May 2012 03:26:23 +0000 (21:26 -0600)]
perluniprops: Omit mention of zero-length special tables
Some tables generated by mktables are special, not for external
consumption. These should not be called out when they turn out to be
zero length, as they are in some Unicode versions
Karl Williamson [Wed, 16 May 2012 02:49:33 +0000 (20:49 -0600)]
mktables: Reorder enum
This will enable future commits to use '<' and '>' for some tests
Karl Williamson [Wed, 16 May 2012 02:01:31 +0000 (20:01 -0600)]
mktables: White-space only change
Karl Williamson [Wed, 16 May 2012 01:18:30 +0000 (19:18 -0600)]
Shorten a $variable name
Karl Williamson [Wed, 16 May 2012 01:11:32 +0000 (19:11 -0600)]
add period to end of sentence in comment
Karl Williamson [Sun, 29 Apr 2012 16:01:02 +0000 (10:01 -0600)]
fold_grind.t: DEBUG mode should automatically give full output results
We're debugging, so don't want abbreviated results
Karl Williamson [Thu, 19 Apr 2012 18:27:32 +0000 (12:27 -0600)]
mktables: slight efficiency improvement
Instead of generating all 128 code points individually in this range, we
can just add a single range that includes all of them.
Karl Williamson [Mon, 16 Apr 2012 17:25:45 +0000 (11:25 -0600)]
mktables: Improve warning messages and handling for overloads
This commit gives clearer messages when an overloaded op has swapped
arguments and we can't handle them. undef should be returned in this
case, as otherwise we are returning the swapped argument.
Karl Williamson [Mon, 16 Apr 2012 16:32:28 +0000 (10:32 -0600)]
mktables: '.=' should be just '.'
This fixes an unnecessary .=
Karl Williamson [Sat, 7 Apr 2012 16:13:46 +0000 (10:13 -0600)]
mktables: Don't use turkic casefold rules for early Unicodes
This isn't done for later Unicodes version; shouldn't be done for early
ones either, as leads to wrong results. The early ones used a different
marker than the later ones; this now recognizes both.
Karl Williamson [Sat, 7 Apr 2012 15:45:21 +0000 (09:45 -0600)]
mktables: correct, clarify comments
Karl Williamson [Sat, 7 Apr 2012 15:30:24 +0000 (09:30 -0600)]
mktables: fix warning message
The dot operators show up as periods because this message is now a HERE
document
Karl Williamson [Fri, 6 Apr 2012 15:32:48 +0000 (09:32 -0600)]
mktables: improve a tracing msg
Karl Williamson [Thu, 5 Apr 2012 16:42:44 +0000 (10:42 -0600)]
Unicode::UCD: Cope with early Unicodes for casespec()
Karl Williamson [Thu, 5 Apr 2012 16:19:11 +0000 (10:19 -0600)]
uni/fold.t: Improve some handling of older Unicodes
In this loop, if a code point is unassigned, it means that the test is
being run on an early Unicode version which doesn't have this character
yet, or something is very wrong. Instead of persisting with the tests
that aren't going to succeed, fail with an appropriate message.
This means that the .t will not pass, but it gives fewer and better
messages. We want to mark the failure for the case where the problem
isn't an early Unicode version.
Karl Williamson [Thu, 5 Apr 2012 16:18:15 +0000 (10:18 -0600)]
uni/fold.t: Add details to some test names
Karl Williamson [Thu, 5 Apr 2012 16:07:08 +0000 (10:07 -0600)]
fold_grind.t: white-space, comments only
This adds and revises some commented out debugging code
Karl Williamson [Wed, 4 Apr 2012 19:51:02 +0000 (13:51 -0600)]
mktables: white-space only
This indents a newly-formed block, and reflows to fit into 80 columns
Karl Williamson [Wed, 4 Apr 2012 18:35:54 +0000 (12:35 -0600)]
mktables: Enable -annotate arg on early Unicodes
On early Unicode releases, there is no NChar property, and on V1.1.5, no
surrogates. The code for -annotate previously assumed these existed,
so failed. This moves the surrogates testing later, after new code that
specially handles the situation in 1.1.5, and has special handling for
the non-character code points, in that they may not exist in the current
version.
Karl Williamson [Sat, 31 Mar 2012 18:55:29 +0000 (12:55 -0600)]
mktables: Move some code around for proximity
These variable definitions are moved closer to where they are used.
Karl Williamson [Sat, 31 Mar 2012 18:51:52 +0000 (12:51 -0600)]
mktables: Rmv unnecessary $temporary
Karl Williamson [Fri, 30 Mar 2012 03:59:15 +0000 (21:59 -0600)]
mktables: Give correct \X defn for earlier Unicode
The comment removed in this commit was wrong. It is possible to get
grapheme cluster boundaries without the extended ones. And that is what
this commit does, for Unicode releases that have the GCB property, but
not the extended version, back to Unicode 4.0.
Karl Williamson [Thu, 29 Mar 2012 16:16:46 +0000 (10:16 -0600)]
mktables: suppress placeholder in gen'd pod
Placeholder tables should not be mentioned in the pod. In particular,
a future commit will make early releases of Unicode have a placeholder
table generated that is the complement of Cased. This would otherwise
be mentioned as what lowercase and uppercase are equivalent to under /i.
Karl Williamson [Thu, 29 Mar 2012 03:06:30 +0000 (21:06 -0600)]
UCD.t: Allow to test earlier Unicodes
In Unicode 6.1, the only property that is stored in hex format that
wasn't handled elsewhere is the bmg property, but earlier Unicodes had
some of the Unihan (if they are being compiled) ones stored that way
too. So make it more general.
Karl Williamson [Tue, 27 Mar 2012 16:03:30 +0000 (10:03 -0600)]
reg_fold.t: Allow to work on early Unicodes
If the version of Unicode being compiled doesn't have the modern
casefolding .txt file, get the values from Unicode::UCD. Also for
EBCDIC, where otherwise the file would have to be translated.
Karl Williamson [Thu, 29 Mar 2012 03:04:34 +0000 (21:04 -0600)]
UCD.pm: Fix grammar in comment
Karl Williamson [Wed, 28 Mar 2012 14:00:10 +0000 (08:00 -0600)]
uni/fold.t: Use Unicode::UCD::casefolds() for inputs
This allows this .t to work on early Unicodes.
Karl Williamson [Wed, 28 Mar 2012 13:50:12 +0000 (07:50 -0600)]
Add all_casefolds()
This function returns the entire structure that casefold() builds. It
is useful for a .t.
Karl Williamson [Wed, 28 Mar 2012 02:51:09 +0000 (20:51 -0600)]
t/uni/case.pl: Allow to work on early Unicodes
This changes case.pl to use Unicode::UCD instead of directly reading
the casing files. This allows it to be used on Unicode releases that
don't have those files, as Unicode::UCD has the intelligence to cope
with that. The EBCDIC code in it can be removed as Unicode::UCD should
cope with that as well.
As a result, the .t's that call it have a slightly different API.
Karl Williamson [Wed, 28 Mar 2012 02:31:12 +0000 (20:31 -0600)]
t/uni/case.pl: Use warnings and strict
Karl Williamson [Fri, 1 Jun 2012 16:47:47 +0000 (10:47 -0600)]
Unicode::UCD: Allow some fncs to work under minitest
Some of the functions defined in this module are needed for minitest,
where dclone is not available. This defines and uses a substitute
dclone when Storable::dclone is not available.
It also conditionally loads Unicode::Normalize. The function that uses
that module is not executed in minitest.
Karl Williamson [Tue, 27 Mar 2012 16:09:01 +0000 (10:09 -0600)]
reg_fold.t: Fix test to work on EBCDIC platform
I noticed these constants won't work on EBCDIC.
Karl Williamson [Tue, 27 Mar 2012 16:03:30 +0000 (10:03 -0600)]
reg_fold.t: Allow to work on early Unicodes
If the version of Unicode being compiled doesn't have the modern
casefolding .txt file, get the values from Unicode::UCD. Also for
EBCDIC, where otherwise the file would have to be translated.
Karl Williamson [Tue, 27 Mar 2012 15:54:53 +0000 (09:54 -0600)]
mk_PL_charclass.pl: Allow to work on early Unicodes
If the version of Unicode being compiled doesn't have the modern
casefolding .txt file, get the values from Unicode::UCD. Also for
EBCDIC, where otherwise the file would have to be translated.
Karl Williamson [Tue, 27 Mar 2012 14:52:02 +0000 (08:52 -0600)]
fold_grind.t: Allow to work on early Unicodes
If the version of Unicode being tested doesn't have the modern
casefolding .txt file, get the values from Unicode::UCD. Also for
EBCDIC, where otherwise the file would have to be translated.
Karl Williamson [Tue, 27 Mar 2012 13:26:47 +0000 (07:26 -0600)]
fold_grind.t: read environment variable just once
Setting a lexical variable to the environment variable made it easier
for me to temporarily override things during a sequence of tests that
took place on many different occasions where I needed this set and kept
forgetting to set the environment variable.
Karl Williamson [Sun, 25 Mar 2012 04:01:35 +0000 (22:01 -0600)]
mktables: Extend Name_Alias property to early Unicode releases
This uses this property so that charnames can work on early Unicode
releases, by including all the aliases that were previously defined in
charnames until Unicode 6.1 started defining them itself.
Karl Williamson [Thu, 5 Apr 2012 15:36:52 +0000 (09:36 -0600)]
charnames.t: White-space only
Indent newly formed block
Karl Williamson [Tue, 27 Mar 2012 03:37:31 +0000 (21:37 -0600)]
charnames.t: Fix to work on Unicodes without NameAliases
This is a recent addition. Use alternate means if the file doesn't
exist in the Unicode release, or is for a non-ASCII platform (as the
alternate means should take care of the translation in that case).
Karl Williamson [Tue, 27 Mar 2012 03:23:45 +0000 (21:23 -0600)]
charnames.t: Skip hangul syllable testing for early Unicodes
If the Unicode release doesn't contain hangul syllables, just skip those
tests
Karl Williamson [Tue, 27 Mar 2012 03:22:38 +0000 (21:22 -0600)]
charnames.t: Indent newly formed block
Karl Williamson [Tue, 27 Mar 2012 03:19:14 +0000 (21:19 -0600)]
charnames.t: Skip testing named sequences if don't exist
Instead of dying when applied to a Unicode version that doesn't have
named sequences, skip them.
Karl Williamson [Mon, 26 Mar 2012 18:35:04 +0000 (12:35 -0600)]
installperl: CaseFolding.txt no longer needed
A previous commit has removed the dependency on this file.
Karl Williamson [Mon, 26 Mar 2012 18:31:20 +0000 (12:31 -0600)]
Unicode::UCD::casefold(): Don't use .txt file for source
This converts this function to using the outputs of prop_invmap() to get
its casefolding definitions. This allows it to work on versions of
Unicode which don't have this file, allows the file to not have to be
installed, and removes this function from having to be different on
EBCDIC platforms (which wasn't coded anyway).
Karl Williamson [Mon, 26 Mar 2012 18:16:08 +0000 (12:16 -0600)]
UCD.t: Don't use BEL for $/
This causes failures on early Unicode releases, and is not necessary
Karl Williamson [Mon, 26 Mar 2012 18:15:06 +0000 (12:15 -0600)]
UCD.t: Skip PropValueAliases tests on early Unicodes
Karl Williamson [Mon, 26 Mar 2012 18:14:27 +0000 (12:14 -0600)]
UCD.t: Skip tests for PropertyAlias on early Unicodes
Karl Williamson [Mon, 26 Mar 2012 18:13:26 +0000 (12:13 -0600)]
UCD.t: Use v-string for easier version comparison
Karl Williamson [Mon, 26 Mar 2012 18:09:10 +0000 (12:09 -0600)]
UCD.t: white-space only
Indent because a previous commit surrounded this with an 'if'
Karl Williamson [Mon, 26 Mar 2012 18:04:22 +0000 (12:04 -0600)]
Unicode::UCD: Fix blocks to work on early Unicodes
Not all Unicode releases supported blocks
Karl Williamson [Mon, 26 Mar 2012 17:58:26 +0000 (11:58 -0600)]
Unicode::UCD: Fix to work on Unicodes without script property
Karl Williamson [Mon, 26 Mar 2012 17:53:04 +0000 (11:53 -0600)]
Unicode::UCD::compexcl(): Fix to work on early Unicodes
Karl Williamson [Mon, 26 Mar 2012 17:46:44 +0000 (11:46 -0600)]
Unicode::UCD::charinfo(): Fix to handle decomps in early Unicode releases
There are no hangul syllables in early releases.
Karl Williamson [Mon, 26 Mar 2012 17:43:55 +0000 (11:43 -0600)]
Unicode::UCD::prop_invmap(): Fix so handles dm in earlier Unicodes
Some versions of Unicode did not have hangul syllables; and there is a
bug in handling them that doesn't show up in the latest versions.
Karl Williamson [Mon, 26 Mar 2012 17:39:51 +0000 (11:39 -0600)]
Unicode::UCD::prop_invmap: Fix so works on very early Unicode
Some versions of Unicode don't have the AHex property. Instead use
[:xdigit:] which is defined in all versions.
Karl Williamson [Mon, 26 Mar 2012 17:36:58 +0000 (11:36 -0600)]
Unicode::UCD::prop_invmap(): Fix to work on early Unicodes
The scf property was originally known as the sfc property. This handles
both possibilities.
Karl Williamson [Mon, 26 Mar 2012 17:34:52 +0000 (11:34 -0600)]
Unicode::UCD::num(): Fix so works on early Unicode releases
This has to do extra work for releases prior to 6.0.
Karl Williamson [Mon, 26 Mar 2012 17:30:03 +0000 (11:30 -0600)]
Unicode::UCD::charinfo(): get ISO comment for earlier Unicodes
This field had meaning in earlier Unicode versions.
Karl Williamson [Mon, 26 Mar 2012 17:24:05 +0000 (11:24 -0600)]
Unicode::UCD: Store v-string Unicode version.
This value will be used in future commits to make version comparisons
easier.
Karl Williamson [Mon, 26 Mar 2012 17:17:48 +0000 (11:17 -0600)]
mktables: Fix up Unicod 2.18 special casing file
which had duplicate entries.
Karl Williamson [Mon, 26 Mar 2012 17:15:53 +0000 (11:15 -0600)]
mktables: Fix-up for Unicode version 1
The Unicode data files for version 1 have a number of errors and
anomalies in them.
Karl Williamson [Mon, 26 Mar 2012 17:13:25 +0000 (11:13 -0600)]
mktables: Add definitions for missing property value aliases
This is for early Unicode releases where things were omitted.
Karl Williamson [Mon, 26 Mar 2012 17:11:55 +0000 (11:11 -0600)]
mktables: Add Unicode_1_names if missing from early Unicode releases
Karl Williamson [Mon, 26 Mar 2012 17:07:03 +0000 (11:07 -0600)]
mktables: Allow simple_mapfoo tables to work in early Unicode releases
Not all Unicode releases have the special casing file; a previous commit
has already added this necessary processing to the mainline code
execution; no need to do it twice.
Karl Williamson [Mon, 26 Mar 2012 17:03:56 +0000 (11:03 -0600)]
mktables: Work around bug in early Unicode databases
Fix up the erroneous fields
Karl Williamson [Mon, 26 Mar 2012 16:50:27 +0000 (10:50 -0600)]
mktables: Fix to enable XIDS property in early Unicode releases
Construct this property if isn't in the Unicode version being compiled,
as it is needed by the Perl core to function
Karl Williamson [Mon, 26 Mar 2012 16:37:03 +0000 (10:37 -0600)]
mktables: Compile \w correctly on early Unicode versions
Early Unicode versions did not have the Connector_Punctuation property.
In those cases the underscore should still be part of \p{Word}.
Karl Williamson [Mon, 26 Mar 2012 16:29:43 +0000 (10:29 -0600)]
mktables: Consolidate handling of Cased property
This moves all the handling dealing with Cased to one place, and fixes
it so it works properly when compiling Unicode versions without the
official Cased property.
Karl Williamson [Mon, 26 Mar 2012 16:11:51 +0000 (10:11 -0600)]
mktables: Fix for Unicode versions without fold, special casing
This code was supposed to work on early Unicode releases which did not
have the special casing and casefold .txt files in them, but was broken.
Processing is moved earlier to set these tables up before the loop that
populates the default mapping for all Unicode tables, so that that loop
will automatically process these as well.
Karl Williamson [Mon, 26 Mar 2012 16:01:57 +0000 (10:01 -0600)]
mktables: Suppress invalid warning on early Unicode releases
When a Unicode release is compile which doesn't have titlecasing the
table is created anyway, but is empty. This suppresses the empty table
sanity check warning, since it's sane for this to be empty in those
versions
Karl Williamson [Mon, 26 Mar 2012 15:53:29 +0000 (09:53 -0600)]
mktables: Move assignment out of loop
This assignment only needs to be done once.
Karl Williamson [Mon, 26 Mar 2012 15:51:31 +0000 (09:51 -0600)]
mktables: White-space only
A previous commit removed the block this code was contained in, so can
outdent and reflow the comments to take up the full width available
Karl Williamson [Mon, 26 Mar 2012 15:48:43 +0000 (09:48 -0600)]
mktables: remove now-always-true test
A previous commit causes the Name_Alias property to be defined in all
Unicode versions, so no longer have to test it.
Karl Williamson [Sun, 25 Mar 2012 04:01:35 +0000 (22:01 -0600)]
mktables: Extend Name_Alias property to early Unicode releases
This uses this property so that charnames can work on early Unicode
releases, by including all the aliases that were previously defined in
charnames until Unicode 6.1 started defining them itself.
Karl Williamson [Mon, 26 Mar 2012 15:16:17 +0000 (09:16 -0600)]
mktables: Loop should begin at 0
This loop is supposed to flush all the input buffering for the fields of
UCD.txt, but skipped field 0, the name field. This is not a problem
except for compiling very early Unicode versions where the final entry in
the file could be a real name. Subsequent versions have the final entry
be for nameless private use code points, so this bug is not triggered
for them.
Karl Williamson [Mon, 26 Mar 2012 15:11:10 +0000 (09:11 -0600)]
mktables: Add input error check and warning
Karl Williamson [Mon, 26 Mar 2012 15:07:34 +0000 (09:07 -0600)]
mktables: Add overload for '&=' on property objects
Karl Williamson [Mon, 26 Mar 2012 15:00:37 +0000 (09:00 -0600)]
mk_invlists.pl: Add error check and die if fails
Karl Williamson [Mon, 26 Mar 2012 03:47:42 +0000 (21:47 -0600)]
mktables: Move dummy table generation earlier
These two tables are not furnished by Unicode in early releases, but if
they are not present, Perl's regex code will fail very ungracefully when
trying to deal with \X. Therefore, we create placeholders for these so
that the code will degrade gracefully when Perl is run with early
Unicode versions.
This patch moves the generation of these placeholder tables to a
function that gets executed earlier in the process. This allows the
loop that follows the new placement to create the proper match tables
and fill in the defaults, like it does for all other Unicode properties.
This allows this commit to remove special tests for the existence of
these properties as well.
This is an interim patch. Future commits will actually fill in these
tables, making them non placeholders.