-The *.txt files were copied from
+# Perl should compile and reasonably run any version of Unicode. That doesn't
+# mean that the test suite will run without showing errors. A few of the
+# very-Unicode specific test files have been modified to account for different
+# versions, but most have not. For example, some tests use characters that
+# aren't encoded in all Unicode versions; others have hard-coded the General
+# Categories that were correct at the time the test was written. Perl itself
+# will not compile under Unicode releases prior to 3.0 without a simple change to
+# Unicode::Normalize. mktables contains instructions for this, as well as other
+# hints for using older Unicode versions.
- http://www.unicode.org/Public/5.0.0/ucd
+# The *.txt files were copied from
-as of Unicode 5.0.0 (July 2006).
+# ftp://www.unicode.org/Public/UNIDATA
-The two big files, NormalizationTest.txt (2 MB) and Unihan.txt (28 MB,
-5.8 MB zip) were not included due to space considerations. Also NOT
-included were any *.html files and the Derived*.txt files
+# (which always points to the latest version) with subdirectories 'extracted' and
+# 'auxiliary'. Older versions are located under Public with an appropriate name.
- DerivedAge.txt
- DerivedCoreProperties.txt
- DerivedNormalizationProps.txt
+# The Unihan files were not included due to space considerations. Also NOT
+# included were any *.html files. It is possible to add the Unihan files, and
+# edit mktables (see instructions near its beginning) to look at them.
-or any files from subdirectories.
+# The file named 'version' should exist and be a single line with the Unicode
+# version, like:
+# 5.2.0
-To be 8.3-friendly, the lib/unicore/PropertyValueAliases.txt was
-renamed to be lib/unicore/PropValueAliases.txt and the
-lib/unicore/NamedSequencesProv.txt was renamed to be
-lib/unicore/NamedSqProv.txt, since otherwise they would have
-conflicted with lib/unicore/PropertyAliases.txt and
-lib/unicore/NamedSequences.txt.
+# To be 8.3 filesystem friendly, the names of some of the input files have been
+# changed from the values that are in the Unicode DB. Not all of the Test
+# files are currently used, so may not be present, so some of the mv's can
+# fail. The .html Test files are not touched.
-NOTE: If you modify the input file set you should also run
-
- mktables -makelist
-
-which will recreate the mktables.lst file which is used to speed up
-the build process.
+mv PropertyValueAliases.txt PropValueAliases.txt
+mv NamedSequencesProv.txt NamedSqProv.txt
+mv NormalizationTest.txt NormTest.txt
+mv DerivedAge.txt DAge.txt
+mv DerivedCoreProperties.txt DCoreProperties.txt
+mv DerivedNormalizationProps.txt DNormalizationProps.txt
-FOR PUMPKINS
+# Some early releases don't have the extracted directory, and hence these files
+# should be moved to it.
+mkdir extracted 2>/dev/null
+mv DerivedBidiClass.txt DerivedBinaryProperties.txt extracted 2>/dev/null
+mv DerivedCombiningClass.txt DerivedDecompositionType.txt extracted 2>/dev/null
+mv DerivedEastAsianWidth.txt DerivedGeneralCategory.txt extracted 2>/dev/null
+mv DerivedJoiningGroup.txt DerivedJoiningType.txt extracted 2>/dev/null
+mv DerivedLineBreak.txt DerivedNumericType.txt DerivedNumericValues.txt extracted 2>/dev/null
-The *.pl files are generated from the *.txt files by the mktables script,
-more recently done during the Perl build process, but if you want to try
-the old manual way:
-
- cd lib/unicore
- cp .../UnicodeOriginal/*.txt .
- rm NormalizationTest.txt Unihan.txt Derived*.txt
- p4 edit Properties *.pl */*.pl
- perl ./mktables
- p4 revert -a
- cd ../..
- perl Porting/manicheck
+mv extracted/DerivedBidiClass.txt extracted/DBidiClass.txt
+mv extracted/DerivedBinaryProperties.txt extracted/DBinaryProperties.txt
+mv extracted/DerivedCombiningClass.txt extracted/DCombiningClass.txt
+mv extracted/DerivedDecompositionType.txt extracted/DDecompositionType.txt
+mv extracted/DerivedEastAsianWidth.txt extracted/DEastAsianWidth.txt
+mv extracted/DerivedGeneralCategory.txt extracted/DGeneralCategory.txt
+mv extracted/DerivedJoiningGroup.txt extracted/DJoinGroup.txt
+mv extracted/DerivedJoiningType.txt extracted/DJoinType.txt
+mv extracted/DerivedLineBreak.txt extracted/DLineBreak.txt
+mv extracted/DerivedNumericType.txt extracted/DNumType.txt
+mv extracted/DerivedNumericValues.txt extracted/DNumValues.txt
-You need to update version by hand
+mv auxiliary/GraphemeBreakTest.txt auxiliary/GCBTest.txt
+mv auxiliary/LineBreakTest.txt auxiliary/LBTest.txt
+mv auxiliary/SentenceBreakTest.txt auxiliary/SBTest.txt
+mv auxiliary/WordBreakTest.txt auxiliary/WBTest.txt
- p4 edit version
- ...
-
-If any new (or deleted, unlikely but not impossible) *.pl files are indicated:
+# If you have the Unihan database (5.2 and above), you should also do the
+# following:
- cd lib/unicore
- p4 add ...
- p4 delete ...
- cd ../...
- p4 edit MANIFEST
- ...
+mv Unihan_DictionaryIndices.txt UnihanIndicesDictionary.txt
+mv Unihan_DictionaryLikeData.txt UnihanDataDictionaryLike.txt
+mv Unihan_IRGSources.txt UnihanIRGSources.txt
+mv Unihan_NumericValues.txt UnihanNumericValues.txt
+mv Unihan_OtherMappings.txt UnihanOtherMappings.txt
+mv Unihan_RadicalStrokeCounts.txt UnihanRadicalStrokeCounts.txt
+mv Unihan_Readings.txt UnihanReadings.txt
+mv Unihan_Variants.txt UnihanVariants.txt
-And finally:
+# If you download everything, the names of files that are not used by mktables
+# are not changed by the above, and hence may not work correctly as-is on 8.3
+# filesystems.
- p4 submit
-
---
-jhi@iki.fi; updated by nick@ccl4.org
+# mktables is used to generate the tables used by the rest of Perl. It will
+# warn you about any *.txt files in the directory substructure that it doesn't
+# know about. You should remove any so-identified, or edit mktables to add
+# them to its lists to process. You can run
+#
+# mktables -globlist
+#
+#to have it try to process these tables generically.
+#
+# FOR PUMPKINS
+#
+# The files are inter-related. If you take the latest UnicodeData.txt, for
+# example, but leave the older versions of other files, there can be subtle
+# problems. So get everything available from Unicode, and delete those which
+# aren't needed.
+#
+# When moving to a new version of Unicode, you need to update 'version' by hand
+#
+# p4 edit version
+# ...
+#
+# You should look in the Unicode release notes (which are probably towards the
+# bottom of http://www.unicode.org/reports/tr44/) to see if any properties have
+# newly been moved to be Obsolete, Deprecated, or Stabilized. The full names
+# for these should be added to the respective lists near the beginning of
+# mktables, using an 'if' to add them for just this Unicode version going
+# forward, so that mktables can continue to be used for earlier Unicode
+# versions.
+#
+# When putting out a new Perl release, think about if any of the Deprecated
+# properties should be moved to Suppressed.
+#
+# perlrecharclass.pod has a list of all the characters that are white space,
+# which needs to be updated if there are changes. A quick way to check if
+# there have been changes would be to see if the number of such characters
+# listed in perluniprops.pod (generated by running mktables) for the property
+# \p{White_Space} is no longer 26. Further investigation would then be
+# necessary to classify the new characters as horizontal and vertical.
+#
+# The code in regexec.c for the \X match construct is intimately tied to the
+# regular expression in UAX #29 (http://www.unicode.org/reports/tr29/). You
+# should see if it has changed, and if so regexec.c should be modified. The
+# current one is
+# ( CRLF
+# | Prepend* ( Hangul-syllable | !Control )
+# ( Grapheme_Extend | Spacing_Mark)*
+# | . )
+#
+# mktables has many checks to warn you if there are unexpected or novel things
+# that it doesn't know how to handle.
+#
+# Module::CoreList should be changed to include the new release
+#
+# Also, you should regen l1_char_class_tab.h, by
+#
+# perl regen/mk_L_charclass.pl
+#
+# and, regen charclass_invlists.h by
+#
+# perl regen/mk_invlists.pl
+#
+# Finally:
+#
+# p4 submit
+#
+# --
+# jhi@iki.fi; updated by nick@ccl4.org, public@khwilliamson.com
https://perl5.git.perl.org / perl5.git / blobdiff