This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Tweak change 27457 - make PL_lasttoke MAD only, but teach makedef.pl
[perl5.git] / pod / perltodo.pod
CommitLineData
7711098a
GS
1=head1 NAME
2
3perltodo - Perl TO-DO List
4
5=head1 DESCRIPTION
e50bb9a1 6
0bdfc961
NC
7This is a list of wishes for Perl. The tasks we think are smaller or easier
8are listed first. Anyone is welcome to work on any of these, but it's a good
9idea to first contact I<perl5-porters@perl.org> to avoid duplication of
10effort. By all means contact a pumpking privately first if you prefer.
e50bb9a1 11
0bdfc961
NC
12Whilst patches to make the list shorter are most welcome, ideas to add to
13the list are also encouraged. Check the perl5-porters archives for past
14ideas, and any discussion about them. One set of archives may be found at:
e50bb9a1 15
0bdfc961 16 http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/
938c8732 17
617eabfa
NC
18What can we offer you in return? Fame, fortune, and everlasting glory? Maybe
19not, but if your patch is incorporated, then we'll add your name to the
20F<AUTHORS> file, which ships in the official distribution. How many other
21programming languages offer you 1 line of immortality?
938c8732 22
4e577f8b 23=head1 The roadmap to 5.10
938c8732 24
4e577f8b
NC
25The roadmap to 5.10 envisages feature based releases, as various items in this
26TODO are completed.
27
4e577f8b
NC
28=head2 Needed for a 5.9.4 release
29
30=over
31
32=item *
78ef48ad
RGS
33
34Review assertions. Review syntax to combine assertions. Assertions could take
35advantage of the lexical pragmas work. L</What hooks would assertions need?>
4e577f8b 36
860f190d
RGS
37=item *
38
39C<encoding::warnings> should be turned into a lexical pragma.
0d720714 40C<encoding> should, too (probably).
860f190d 41
4e577f8b
NC
42=back
43
44=head2 Needed for a 5.9.5 release
45
46=over
47
48=item *
49Implement L</_ prototype character>
50
51=item *
52Implement L</state variables>
53
54=back
55
56=head2 Needed for a 5.9.6 release
57
58Stabilisation. If all goes well, this will be the equivalent of a 5.10-beta.
e50bb9a1 59
0bdfc961 60=head1 Tasks that only need Perl knowledge
e50bb9a1 61
0bdfc961 62=head2 common test code for timed bail out
e50bb9a1 63
0bdfc961
NC
64Write portable self destruct code for tests to stop them burning CPU in
65infinite loops. This needs to avoid using alarm, as some of the tests are
66testing alarm/sleep or timers.
e50bb9a1 67
0bdfc961 68=head2 POD -> HTML conversion in the core still sucks
e50bb9a1 69
938c8732 70Which is crazy given just how simple POD purports to be, and how simple HTML
adebf063
NC
71can be. It's not actually I<as> simple as it sounds, particularly with the
72flexibility POD allows for C<=item>, but it would be good to improve the
73visual appeal of the HTML generated, and to avoid it having any validation
74errors. See also L</make HTML install work>, as the layout of installation tree
75is needed to improve the cross-linking.
938c8732 76
dc0fb092
SP
77The addition of C<Pod::Simple> and its related modules may make this task
78easier to complete.
79
aa237293
NC
80=head2 Parallel testing
81
82The core regression test suite is getting ever more comprehensive, which has
83the side effect that it takes longer to run. This isn't so good. Investigate
84whether it would be feasible to give the harness script the B<option> of
85running sets of tests in parallel. This would be useful for tests in
86F<t/op/*.t> and F<t/uni/*.t> and maybe some sets of tests in F<lib/>.
87
88Questions to answer
89
90=over 4
91
92=item 1
93
94How does screen layout work when you're running more than one test?
95
96=item 2
97
98How does the caller of test specify how many tests to run in parallel?
99
100=item 3
101
102How do setup/teardown tests identify themselves?
103
104=back
105
106Pugs already does parallel testing - can their approach be re-used?
107
0bdfc961 108=head2 Make Schwern poorer
e50bb9a1 109
0bdfc961
NC
110We should have for everything. When all the core's modules are tested,
111Schwern has promised to donate to $500 to TPF. We may need volunteers to
112hold him upside down and shake vigorously in order to actually extract the
113cash.
3958b146 114
0bdfc961 115See F<t/lib/1_compile.t> for the 3 remaining modules that need tests.
e50bb9a1 116
0bdfc961 117=head2 Improve the coverage of the core tests
e50bb9a1 118
0bdfc961
NC
119Use Devel::Cover to ascertain the core's test coverage, then add tests that
120are currently missing.
30222c0f 121
0bdfc961 122=head2 test B
e50bb9a1 123
0bdfc961 124A full test suite for the B module would be nice.
e50bb9a1 125
0bdfc961 126=head2 A decent benchmark
e50bb9a1 127
617eabfa 128C<perlbench> seems impervious to any recent changes made to the perl core. It
0bdfc961
NC
129would be useful to have a reasonable general benchmarking suite that roughly
130represented what current perl programs do, and measurably reported whether
131tweaks to the core improve, degrade or don't really affect performance, to
132guide people attempting to optimise the guts of perl. Gisle would welcome
133new tests for perlbench.
6168cf99 134
0bdfc961 135=head2 fix tainting bugs
6168cf99 136
0bdfc961
NC
137Fix the bugs revealed by running the test suite with the C<-t> switch (via
138C<make test.taintwarn>).
e50bb9a1 139
0bdfc961 140=head2 Dual life everything
e50bb9a1 141
0bdfc961
NC
142As part of the "dists" plan, anything that doesn't belong in the smallest perl
143distribution needs to be dual lifed. Anything else can be too. Figure out what
144changes would be needed to package that module and its tests up for CPAN, and
145do so. Test it with older perl releases, and fix the problems you find.
e50bb9a1 146
0bdfc961 147=head2 Improving C<threads::shared>
722d2a37 148
0bdfc961
NC
149Investigate whether C<threads::shared> could share aggregates properly with
150only Perl level changes to shared.pm
722d2a37 151
0bdfc961 152=head2 POSIX memory footprint
e50bb9a1 153
0bdfc961
NC
154Ilya observed that use POSIX; eats memory like there's no tomorrow, and at
155various times worked to cut it down. There is probably still fat to cut out -
156for example POSIX passes Exporter some very memory hungry data structures.
e50bb9a1 157
e50bb9a1 158
e50bb9a1 159
e50bb9a1 160
e50bb9a1 161
adebf063 162
adebf063 163
0bdfc961 164=head1 Tasks that need a little sysadmin-type knowledge
e50bb9a1 165
0bdfc961
NC
166Or if you prefer, tasks that you would learn from, and broaden your skills
167base...
e50bb9a1 168
617eabfa
NC
169=head2 Relocatable perl
170
171The C level patches needed to create a relocatable perl binary are done, as
172is the work on F<Config.pm>. All that's left to do is the C<Configure> tweaking
173to let people specify how they want to do the install.
174
cd793d32 175=head2 make HTML install work
e50bb9a1 176
adebf063
NC
177There is an C<installhtml> target in the Makefile. It's marked as
178"experimental". It would be good to get this tested, make it work reliably, and
179remove the "experimental" tag. This would include
180
181=over 4
182
183=item 1
184
185Checking that cross linking between various parts of the documentation works.
186In particular that links work between the modules (files with POD in F<lib/>)
187and the core documentation (files in F<pod/>)
188
189=item 2
190
617eabfa
NC
191Work out how to split C<perlfunc> into chunks, preferably one per function
192group, preferably with general case code that could be used elsewhere.
193Challenges here are correctly identifying the groups of functions that go
194together, and making the right named external cross-links point to the right
195page. Things to be aware of are C<-X>, groups such as C<getpwnam> to
196C<endservent>, two or more C<=items> giving the different parameter lists, such
197as
adebf063
NC
198
199 =item substr EXPR,OFFSET,LENGTH,REPLACEMENT
200
201 =item substr EXPR,OFFSET,LENGTH
202
203 =item substr EXPR,OFFSET
204
205and different parameter lists having different meanings. (eg C<select>)
206
207=back
3a89a73c 208
0bdfc961
NC
209=head2 compressed man pages
210
211Be able to install them. This would probably need a configure test to see how
212the system does compressed man pages (same directory/different directory?
213same filename/different filename), as well as tweaking the F<installman> script
214to compress as necessary.
215
30222c0f
NC
216=head2 Add a code coverage target to the Makefile
217
218Make it easy for anyone to run Devel::Cover on the core's tests. The steps
219to do this manually are roughly
220
221=over 4
222
223=item *
224
225do a normal C<Configure>, but include Devel::Cover as a module to install
226(see F<INSTALL> for how to do this)
227
228=item *
229
230 make perl
231
232=item *
233
234 cd t; HARNESS_PERL_SWITCHES=-MDevel::Cover ./perl -I../lib harness
235
236=item *
237
238Process the resulting Devel::Cover database
239
240=back
241
242This just give you the coverage of the F<.pm>s. To also get the C level
243coverage you need to
244
245=over 4
246
247=item *
248
249Additionally tell C<Configure> to use the appropriate C compiler flags for
250C<gcov>
251
252=item *
253
254 make perl.gcov
255
256(instead of C<make perl>)
257
258=item *
259
260After running the tests run C<gcov> to generate all the F<.gcov> files.
261(Including down in the subdirectories of F<ext/>
262
263=item *
264
265(From the top level perl directory) run C<gcov2perl> on all the C<.gcov> files
266to get their stats into the cover_db directory.
267
268=item *
269
270Then process the Devel::Cover database
271
272=back
273
274It would be good to add a single switch to C<Configure> to specify that you
275wanted to perform perl level coverage, and another to specify C level
276coverage, and have C<Configure> and the F<Makefile> do all the right things
277automatically.
278
0bdfc961
NC
279=head2 Make Config.pm cope with differences between build and installed perl
280
281Quite often vendors ship a perl binary compiled with their (pay-for)
282compilers. People install a free compiler, such as gcc. To work out how to
283build extensions, Perl interrogates C<%Config>, so in this situation
284C<%Config> describes compilers that aren't there, and extension building
285fails. This forces people into choosing between re-compiling perl themselves
286using the compiler they have, or only using modules that the vendor ships.
287
288It would be good to find a way teach C<Config.pm> about the installation setup,
289possibly involving probing at install time or later, so that the C<%Config> in
290a binary distribution better describes the installed machine, when the
291installed machine differs from the build machine in some significant way.
292
46925299 293=head2 make parallel builds work
0bdfc961 294
46925299
NC
295Currently parallel builds (such as C<make -j3>) don't work reliably. We believe
296that this is due to incomplete dependency specification in the F<Makefile>.
297It would be good if someone were able to track down the causes of these
298problems, so that parallel builds worked properly.
0bdfc961 299
728f4ecd
NC
300=head2 linker specification files
301
302Some platforms mandate that you provide a list of a shared library's external
303symbols to the linker, so the core already has the infrastructure in place to
304do this for generating shared perl libraries. My understanding is that the
305GNU toolchain can accept an optional linker specification file, and restrict
306visibility just to symbols declared in that file. It would be good to extend
307F<makedef.pl> to support this format, and to provide a means within
308C<Configure> to enable it. This would allow Unix users to test that the
309export list is correct, and to build a perl that does not pollute the global
310namespace with private symbols.
311
8523e164 312
0bdfc961
NC
313
314
315=head1 Tasks that need a little C knowledge
316
317These tasks would need a little C knowledge, but don't need any specific
318background or experience with XS, or how the Perl interpreter works
319
320=head2 Make it clear from -v if this is the exact official release
89007cb3 321
617eabfa
NC
322Currently perl from C<p4>/C<rsync> ships with a F<patchlevel.h> file that
323usually defines one local patch, of the form "MAINT12345" or "RC1". The output
324of perl -v doesn't report that a perl isn't an official release, and this
89007cb3 325information can get lost in bugs reports. Because of this, the minor version
fa11829f 326isn't bumped up until RC time, to minimise the possibility of versions of perl
89007cb3
NC
327escaping that believe themselves to be newer than they actually are.
328
329It would be useful to find an elegant way to have the "this is an interim
330maintenance release" or "this is a release candidate" in the terse -v output,
331and have it so that it's easy for the pumpking to remove this just as the
332release tarball is rolled up. This way the version pulled out of rsync would
333always say "I'm a development release" and it would be safe to bump the
334reported minor version as soon as a release ships, which would aid perl
335developers.
336
0bdfc961
NC
337This task is really about thinking of an elegant way to arrange the C source
338such that it's trivial for the Pumpking to flag "this is an official release"
339when making a tarball, yet leave the default source saying "I'm not the
340official release".
341
0f788cd2
NC
342=head2 Ordering of "global" variables.
343
344F<thrdvar.h> and F<intrpvarh> define the "global" variables that need to be
345per-thread under ithreads, where the variables are actually elements in a
346structure. As C dictates, the variables must be laid out in order of
347declaration. There is a comment
348C</* Important ones in the first cache line (if alignment is done right) */>
349which implies that at some point in the past the ordering was carefully chosen
350(at least in part). However, it's clear that the ordering is less than perfect,
351as currently there are things such as 7 C<bool>s in a row, then something
352typically requiring 4 byte alignment, and then an odd C<bool> later on.
353(C<bool>s are typically defined as C<char>s). So it would be good for someone
354to review the ordering of the variables, to see how much alignment padding can
355be removed.
356
d7939546
NC
357It's also worth checking that all variables are actually used. Perl 5.8.0
358shipped with C<PL_nrs> still defined in F<thrdvar.h>, despite it being unused
359since a change over a year earlier. Had this been spotted before release, it
360could have been removed, but now it has to remain in the 5.8.x releases to
361keep the structure the same size, to retain binary compatibility.
362
62403a3c
NC
363=head2 am I hot or not?
364
365The idea of F<pp_hot.c> is that it contains the I<hot> ops, the ops that are
366most commonly used. The idea is that by grouping them, their object code will
367be adjacent in the executable, so they have a greater chance of already being
368in the CPU cache (or swapped in) due to being near another op already in use.
369
370Except that it's not clear if these really are the most commonly used ops. So
371anyone feeling like exercising their skill with coverage and profiling tools
372might want to determine what ops I<really> are the most commonly used. And in
373turn suggest evictions and promotions to achieve a better F<pp_hot.c>.
374
98fed0ad
NC
375=head2 Shrink struct context
376
377In F<cop.h>, we have
378
379 struct context {
380 U32 cx_type; /* what kind of context this is */
381 union {
382 struct block cx_blk;
383 struct subst cx_subst;
384 } cx_u;
385 };
386
387There are less than 256 values for C<cx_type>, and the constituent parts
388C<struct block> and C<struct subst> both contain some C<U8> and C<U16> fields,
389so it should be possible to move them to the first word, and share space with
390a C<U8> C<cx_type>, saving 1 word.
391
392=head2 Allocate OPs from arenas
393
394Currently all new OP structures are individually malloc()ed and free()d.
395All C<malloc> implementations have space overheads, and are now as fast as
396custom allocates so it would both use less memory and less CPU to allocate
397the various OP structures from arenas. The SV arena code can probably be
398re-used for this.
399
0bdfc961
NC
400
401
402
0bdfc961 403=head1 Tasks that need a knowledge of XS
e50bb9a1 404
0bdfc961
NC
405These tasks would need C knowledge, and roughly the level of knowledge of
406the perl API that comes from writing modules that use XS to interface to
407C.
408
409=head2 IPv6
410
411Clean this up. Check everything in core works
412
35b64ab6 413=head2 shrink C<IO>s
4a750395 414
35b64ab6
NC
415By removing unused elements and careful re-ordering, the structures for C<AV>s,
416C<HV>s, C<CV>s and C<GV>s have recently been shrunk considerably. C<PVIO>s and
417C<PVBM>s might have some savings to win.
4a750395 418
0bdfc961
NC
419=head2 UTF8 caching code
420
421The string position/offset cache is not optional. It should be.
422
423=head2 Implicit Latin 1 => Unicode translation
424
425Conversions from byte strings to UTF-8 currently map high bit characters
426to Unicode without translation (or, depending on how you look at it, by
427implicitly assuming that the byte strings are in Latin-1). As perl assumes
428the C locale by default, upgrading a string to UTF-8 may change the
429meaning of its contents regarding character classes, case mapping, etc.
430This should probably emit a warning (at least).
431
432This task is incremental - even a little bit of work on it will help.
e50bb9a1 433
cd793d32 434=head2 autovivification
e50bb9a1 435
cd793d32 436Make all autovivification consistent w.r.t LVALUE/RVALUE and strict/no strict;
e50bb9a1 437
0bdfc961 438This task is incremental - even a little bit of work on it will help.
e50bb9a1 439
0bdfc961 440=head2 Unicode in Filenames
e50bb9a1 441
0bdfc961
NC
442chdir, chmod, chown, chroot, exec, glob, link, lstat, mkdir, open,
443opendir, qx, readdir, readlink, rename, rmdir, stat, symlink, sysopen,
444system, truncate, unlink, utime, -X. All these could potentially accept
445Unicode filenames either as input or output (and in the case of system
446and qx Unicode in general, as input or output to/from the shell).
447Whether a filesystem - an operating system pair understands Unicode in
448filenames varies.
e50bb9a1 449
0bdfc961
NC
450Known combinations that have some level of understanding include
451Microsoft NTFS, Apple HFS+ (In Mac OS 9 and X) and Apple UFS (in Mac
452OS X), NFS v4 is rumored to be Unicode, and of course Plan 9. How to
453create Unicode filenames, what forms of Unicode are accepted and used
454(UCS-2, UTF-16, UTF-8), what (if any) is the normalization form used,
455and so on, varies. Finding the right level of interfacing to Perl
456requires some thought. Remember that an OS does not implicate a
457filesystem.
e50bb9a1 458
0bdfc961
NC
459(The Windows -C command flag "wide API support" has been at least
460temporarily retired in 5.8.1, and the -C has been repurposed, see
461L<perlrun>.)
969e704b 462
0bdfc961 463=head2 Unicode in %ENV
969e704b 464
0bdfc961 465Currently the %ENV entries are always byte strings.
e50bb9a1 466
0bdfc961 467=head2 use less 'memory'
e50bb9a1 468
0bdfc961
NC
469Investigate trade offs to switch out perl's choices on memory usage.
470Particularly perl should be able to give memory back.
e50bb9a1 471
0bdfc961 472This task is incremental - even a little bit of work on it will help.
0abe3f7c 473
0bdfc961 474=head2 Re-implement C<:unique> in a way that is actually thread-safe
0abe3f7c 475
0bdfc961
NC
476The old implementation made bad assumptions on several levels. A good 90%
477solution might be just to make C<:unique> work to share the string buffer
478of SvPVs. That way large constant strings can be shared between ithreads,
479such as the configuration information in F<Config>.
0abe3f7c 480
0bdfc961 481=head2 Make tainting consistent
0abe3f7c 482
0bdfc961
NC
483Tainting would be easier to use if it didn't take documented shortcuts and
484allow taint to "leak" everywhere within an expression.
0abe3f7c 485
0bdfc961 486=head2 readpipe(LIST)
0abe3f7c 487
0bdfc961
NC
488system() accepts a LIST syntax (and a PROGRAM LIST syntax) to avoid
489running a shell. readpipe() (the function behind qx//) could be similarly
490extended.
0abe3f7c 491
e50bb9a1 492
e50bb9a1 493
e50bb9a1 494
f86a8bc5 495
0bdfc961 496=head1 Tasks that need a knowledge of the interpreter
3298bd4d 497
0bdfc961
NC
498These tasks would need C knowledge, and knowledge of how the interpreter works,
499or a willingness to learn.
3298bd4d 500
0bdfc961
NC
501=head2 lexical pragmas
502
78ef48ad
RGS
503Document the new support for lexical pragmas in 5.9.3 and how %^H works.
504Maybe C<re>, C<encoding>, maybe other pragmas could be made lexical.
0562c0e3 505
d10fc472 506=head2 Attach/detach debugger from running program
1626a787 507
cd793d32
NC
508The old perltodo notes "With C<gdb>, you can attach the debugger to a running
509program if you pass the process ID. It would be good to do this with the Perl
0bdfc961
NC
510debugger on a running Perl program, although I'm not sure how it would be
511done." ssh and screen do this with named pipes in /tmp. Maybe we can too.
1626a787 512
0bdfc961 513=head2 Constant folding
80b46460 514
0bdfc961
NC
515The peephole optimiser should trap errors during constant folding, and give
516up on the folding, rather than bailing out at compile time. It is quite
517possible that the unfoldable constant is in unreachable code, eg something
518akin to C<$a = 0/0 if 0;>
519
520=head2 LVALUE functions for lists
521
522The old perltodo notes that lvalue functions don't work for list or hash
523slices. This would be good to fix.
524
525=head2 LVALUE functions in the debugger
526
527The old perltodo notes that lvalue functions don't work in the debugger. This
528would be good to fix.
529
530=head2 _ prototype character
531
532Study the possibility of adding a new prototype character, C<_>, meaning
533"this argument defaults to $_".
534
4e577f8b
NC
535=head2 state variables
536
537C<my $foo if 0;> is deprecated, and should be replaced with
538C<state $x = "initial value\n";> the syntax from Perl 6.
539
0bdfc961
NC
540=head2 @INC source filter to Filter::Simple
541
542The second return value from a sub in @INC can be a source filter. This isn't
543documented. It should be changed to use Filter::Simple, tested and documented.
544
545=head2 regexp optimiser optional
546
547The regexp optimiser is not optional. It should configurable to be, to allow
548its performance to be measured, and its bugs to be easily demonstrated.
549
550=head2 UNITCHECK
551
552Introduce a new special block, UNITCHECK, which is run at the end of a
553compilation unit (module, file, eval(STRING) block). This will correspond to
554the Perl 6 CHECK. Perl 5's CHECK cannot be changed or removed because the
555O.pm/B.pm backend framework depends on it.
556
557=head2 optional optimizer
558
559Make the peephole optimizer optional. Currently it performs two tasks as
560it walks the optree - genuine peephole optimisations, and necessary fixups of
561ops. It would be good to find an efficient way to switch out the
562optimisations whilst keeping the fixups.
563
564=head2 You WANT *how* many
565
566Currently contexts are void, scalar and list. split has a special mechanism in
567place to pass in the number of return values wanted. It would be useful to
568have a general mechanism for this, backwards compatible and little speed hit.
569This would allow proposals such as short circuiting sort to be implemented
570as a module on CPAN.
571
572=head2 lexical aliases
573
574Allow lexical aliases (maybe via the syntax C<my \$alias = \$foo>.
575
576=head2 entersub XS vs Perl
577
578At the moment pp_entersub is huge, and has code to deal with entering both
579perl and XS subroutines. Subroutine implementations rarely change between
580perl and XS at run time, so investigate using 2 ops to enter subs (one for
581XS, one for perl) and swap between if a sub is redefined.
2810d901
NC
582
583=head2 Self ties
584
585self ties are currently illegal because they caused too many segfaults. Maybe
586the causes of these could be tracked down and self-ties on all types re-
587instated.
0bdfc961
NC
588
589=head2 Optimize away @_
590
591The old perltodo notes "Look at the "reification" code in C<av.c>".
592
0bdfc961
NC
593=head2 What hooks would assertions need?
594
595Assertions are in the core, and work. However, assertions needed to be added
596as a core patch, rather than an XS module in ext, or a CPAN module, because
597the core has no hooks in the necessary places. It would be useful to
598investigate what hooks would need to be added to make it possible to provide
599the full assertion support from a CPAN module, so that we aren't constraining
600the imagination of future CPAN authors.
601
602
603
604
605
0bdfc961
NC
606=head1 Big projects
607
608Tasks that will get your name mentioned in the description of the "Highlights
609of 5.10"
610
611=head2 make ithreads more robust
612
4e577f8b 613Generally make ithreads more robust. See also L</iCOW>
0bdfc961
NC
614
615This task is incremental - even a little bit of work on it will help, and
616will be greatly appreciated.
617
6c047da7
YST
618One bit would be to write the missing code in sv.c:Perl_dirp_dup.
619
0bdfc961
NC
620=head2 iCOW
621
622Sarathy and Arthur have a proposal for an improved Copy On Write which
623specifically will be able to COW new ithreads. If this can be implemented
624it would be a good thing.
625
626=head2 (?{...}) closures in regexps
627
628Fix (or rewrite) the implementation of the C</(?{...})/> closures.
629
630=head2 A re-entrant regexp engine
631
632This will allow the use of a regex from inside (?{ }), (??{ }) and
633(?(?{ })|) constructs.