-The *.txt files were copied from
+# Perl should compile and reasonably run any version of Unicode. That doesn't
+# mean that the test suite will run without showing errors. A few of the
+# very-Unicode specific test files have been modified to account for different
+# versions, but most have not. For example, some tests use characters that
+# aren't encoded in all Unicode versions; others have hard-coded the General
+# Categories that were correct at the time the test was written. Perl itself
+# will not compile under Unicode releases prior to 3.0 without a simple change to
+# Unicode::Normalize. mktables contains instructions for this, as well as other
+# hints for using older Unicode versions.
- http://www.unicode.org/Public/5.1.0/ucd
+# The *.txt files were copied from
-and subdirectories 'extracted' and 'auxiliary' as of Unicode 5.1.0 (March 2008).
+# ftp://www.unicode.org/Public/UNIDATA
-The big file, Unihan.txt (28 MB, 5.8 MB zip) was not included due to space
-considerations. Also NOT included were any *.html files and *Test.txt files.
+# (which always points to the latest version) with subdirectories 'extracted' and
+# 'auxiliary'. Older versions are located under Public with an appropriate name.
-To be 8.3 filesystem friendly, the names of some of the input files have been
-changed from the values that are in the Unicode DB:
+# The Unihan files were not included due to space considerations. Also NOT
+# included were any *.html files. It is possible to add the Unihan files, and
+# edit mktables (see instructions near its beginning) to look at them.
+
+# The file named 'version' should exist and be a single line with the Unicode
+# version, like:
+# 5.2.0
+
+# To be 8.3 filesystem friendly, the names of some of the input files have been
+# changed from the values that are in the Unicode DB. Not all of the Test
+# files are currently used, so may not be present, so some of the mv's can
+# fail. The .html Test files are not touched.
mv PropertyValueAliases.txt PropValueAliases.txt
mv NamedSequencesProv.txt NamedSqProv.txt
+mv NormalizationTest.txt NormTest.txt
mv DerivedAge.txt DAge.txt
mv DerivedCoreProperties.txt DCoreProperties.txt
mv DerivedNormalizationProps.txt DNormalizationProps.txt
+
+# Some early releases don't have the extracted directory, and hence these files
+# should be moved to it.
+mkdir extracted 2>/dev/null
+mv DerivedBidiClass.txt DerivedBinaryProperties.txt extracted 2>/dev/null
+mv DerivedCombiningClass.txt DerivedDecompositionType.txt extracted 2>/dev/null
+mv DerivedEastAsianWidth.txt DerivedGeneralCategory.txt extracted 2>/dev/null
+mv DerivedJoiningGroup.txt DerivedJoiningType.txt extracted 2>/dev/null
+mv DerivedLineBreak.txt DerivedNumericType.txt DerivedNumericValues.txt extracted 2>/dev/null
+
mv extracted/DerivedBidiClass.txt extracted/DBidiClass.txt
mv extracted/DerivedBinaryProperties.txt extracted/DBinaryProperties.txt
mv extracted/DerivedCombiningClass.txt extracted/DCombiningClass.txt
mv extracted/DerivedNumericType.txt extracted/DNumType.txt
mv extracted/DerivedNumericValues.txt extracted/DNumValues.txt
-The names of files, such as test files, that are not used by mktables are not
-changed, and will not work correctly on 8.3 filesystems.
-
-The file 'version' should exist and be a single line with the Unicode version,
-like 5.1.0
-
-NOTE: If you modify the input file set you should also run
-
- mktables -makelist
-
-which will recreate the mktables.lst file which is used to speed up
-the build process.
-
-FOR PUMPKINS
-
-The files are inter-related. If you take the latest UnicodeData.txt, for example,
-but leave the older versions of other files, there can be subtle problems.
-
-The *.pl files are generated from the *.txt files by the mktables script,
-more recently done during the Perl build process, but if you want to try
-the old manual way:
-
- cd lib/unicore
- cp .../UnicodeOriginal/*.txt .
- rm NormalizationTest.txt Unihan.txt Derived*.txt
- p4 edit Properties *.pl */*.pl
- perl ./mktables
- p4 revert -a
- cd ../..
- perl Porting/manicheck
-
-You need to update version by hand
-
- p4 edit version
- ...
-
-If any new (or deleted, unlikely but not impossible) *.pl files are indicated:
-
- cd lib/unicore
- p4 add ...
- p4 delete ...
- cd ../...
- p4 edit MANIFEST
- ...
-
-And finally:
-
- p4 submit
-
---
-jhi@iki.fi; updated by nick@ccl4.org
-
-mktables is used to generate the tables used by the rest of Perl. It will warn
-you about any *.txt files in the directory substructure that it doesn't know
-about. You should remove any so-identified, or edit mktables to add them to
-its lists. You can run
-
- mktables -globlist
-
-to have it try to process these tables generically.
-
-Also, for each new release, you should run
-
- mktables -v -makelist
-
-to see all warnings and to recreate the mktables.lst file which is used to
-speed up the build process.
-
-The files are inter-related. If you take the latest UnicodeData.txt, for example,
-but leave the older versions of other files, there can be subtle problems.
-
-jhi@iki.fi; updated by nick@ccl4.org, public@khwilliamson.com
+mv auxiliary/GraphemeBreakTest.txt auxiliary/GCBTest.txt
+mv auxiliary/LineBreakTest.txt auxiliary/LBTest.txt
+mv auxiliary/SentenceBreakTest.txt auxiliary/SBTest.txt
+mv auxiliary/WordBreakTest.txt auxiliary/WBTest.txt
+
+# If you have the Unihan database (5.2 and above), you should also do the
+# following:
+
+mv Unihan_DictionaryIndices.txt UnihanIndicesDictionary.txt
+mv Unihan_DictionaryLikeData.txt UnihanDataDictionaryLike.txt
+mv Unihan_IRGSources.txt UnihanIRGSources.txt
+mv Unihan_NumericValues.txt UnihanNumericValues.txt
+mv Unihan_OtherMappings.txt UnihanOtherMappings.txt
+mv Unihan_RadicalStrokeCounts.txt UnihanRadicalStrokeCounts.txt
+mv Unihan_Readings.txt UnihanReadings.txt
+mv Unihan_Variants.txt UnihanVariants.txt
+
+# If you download everything, the names of files that are not used by mktables
+# are not changed by the above, and hence may not work correctly as-is on 8.3
+# filesystems.
+
+# mktables is used to generate the tables used by the rest of Perl. It will
+# warn you about any *.txt files in the directory substructure that it doesn't
+# know about. You should remove any so-identified, or edit mktables to add
+# them to its lists to process. You can run
+#
+# mktables -globlist
+#
+#to have it try to process these tables generically.
+#
+# FOR PUMPKINS
+#
+# The files are inter-related. If you take the latest UnicodeData.txt, for
+# example, but leave the older versions of other files, there can be subtle
+# problems. So get everything available from Unicode, and delete those which
+# aren't needed.
+#
+# When moving to a new version of Unicode, you need to update 'version' by hand
+#
+# p4 edit version
+# ...
+#
+# You should look in the Unicode release notes (which are probably towards the
+# bottom of http://www.unicode.org/reports/tr44/) to see if any properties have
+# newly been moved to be Obsolete, Deprecated, or Stabilized. The full names
+# for these should be added to the respective lists near the beginning of
+# mktables, using an 'if' to add them for just this Unicode version going
+# forward, so that mktables can continue to be used for earlier Unicode
+# versions.
+#
+# When putting out a new Perl release, think about if any of the Deprecated
+# properties should be moved to Suppressed.
+#
+# perlrecharclass.pod has a list of all the characters that are white space,
+# which needs to be updated if there are changes. A quick way to check if
+# there have been changes would be to see if the number of such characters
+# listed in perluniprops.pod (generated by running mktables) for the property
+# \p{White_Space} is no longer 26. Further investigation would then be
+# necessary to classify the new characters as horizontal and vertical.
+#
+# The code in regexec.c for the \X match construct is intimately tied to the
+# regular expression in UAX #29 (http://www.unicode.org/reports/tr29/). You
+# should see if it has changed, and if so regexec.c should be modified. The
+# current one is
+# ( CRLF
+# | Prepend* ( Hangul-syllable | !Control )
+# ( Grapheme_Extend | Spacing_Mark)*
+# | . )
+#
+# mktables has many checks to warn you if there are unexpected or novel things
+# that it doesn't know how to handle.
+#
+# Module::CoreList should be changed to include the new release
+#
+# Also, you should regen l1_char_class_tab.h, by
+#
+# perl regen/mk_L_charclass.pl
+#
+# and, regen charclass_invlists.h by
+#
+# perl regen/mk_invlists.pl
+#
+# Finally:
+#
+# p4 submit
+#
+# --
+# jhi@iki.fi; updated by nick@ccl4.org, public@khwilliamson.com