This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Remove the #define - ARENASETS are always on now.
[perl5.git] / pod / perltodo.pod
CommitLineData
7711098a
GS
1=head1 NAME
2
3perltodo - Perl TO-DO List
4
5=head1 DESCRIPTION
e50bb9a1 6
0bdfc961
NC
7This is a list of wishes for Perl. The tasks we think are smaller or easier
8are listed first. Anyone is welcome to work on any of these, but it's a good
9idea to first contact I<perl5-porters@perl.org> to avoid duplication of
10effort. By all means contact a pumpking privately first if you prefer.
e50bb9a1 11
0bdfc961
NC
12Whilst patches to make the list shorter are most welcome, ideas to add to
13the list are also encouraged. Check the perl5-porters archives for past
14ideas, and any discussion about them. One set of archives may be found at:
e50bb9a1 15
0bdfc961 16 http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/
938c8732 17
617eabfa
NC
18What can we offer you in return? Fame, fortune, and everlasting glory? Maybe
19not, but if your patch is incorporated, then we'll add your name to the
20F<AUTHORS> file, which ships in the official distribution. How many other
21programming languages offer you 1 line of immortality?
938c8732 22
4e577f8b 23=head1 The roadmap to 5.10
938c8732 24
4e577f8b
NC
25The roadmap to 5.10 envisages feature based releases, as various items in this
26TODO are completed.
27
4e577f8b
NC
28=head2 Needed for a 5.9.4 release
29
30=over
31
32=item *
78ef48ad
RGS
33
34Review assertions. Review syntax to combine assertions. Assertions could take
35advantage of the lexical pragmas work. L</What hooks would assertions need?>
4e577f8b 36
860f190d
RGS
37=item *
38
39C<encoding::warnings> should be turned into a lexical pragma.
0d720714 40C<encoding> should, too (probably).
860f190d 41
4e577f8b
NC
42=back
43
44=head2 Needed for a 5.9.5 release
45
46=over
47
48=item *
49Implement L</_ prototype character>
50
51=item *
52Implement L</state variables>
53
54=back
55
56=head2 Needed for a 5.9.6 release
57
58Stabilisation. If all goes well, this will be the equivalent of a 5.10-beta.
e50bb9a1 59
0bdfc961 60=head1 Tasks that only need Perl knowledge
e50bb9a1 61
0bdfc961 62=head2 common test code for timed bail out
e50bb9a1 63
0bdfc961
NC
64Write portable self destruct code for tests to stop them burning CPU in
65infinite loops. This needs to avoid using alarm, as some of the tests are
66testing alarm/sleep or timers.
e50bb9a1 67
0bdfc961 68=head2 POD -> HTML conversion in the core still sucks
e50bb9a1 69
938c8732 70Which is crazy given just how simple POD purports to be, and how simple HTML
adebf063
NC
71can be. It's not actually I<as> simple as it sounds, particularly with the
72flexibility POD allows for C<=item>, but it would be good to improve the
73visual appeal of the HTML generated, and to avoid it having any validation
74errors. See also L</make HTML install work>, as the layout of installation tree
75is needed to improve the cross-linking.
938c8732 76
dc0fb092
SP
77The addition of C<Pod::Simple> and its related modules may make this task
78easier to complete.
79
aa237293
NC
80=head2 Parallel testing
81
82The core regression test suite is getting ever more comprehensive, which has
83the side effect that it takes longer to run. This isn't so good. Investigate
84whether it would be feasible to give the harness script the B<option> of
85running sets of tests in parallel. This would be useful for tests in
86F<t/op/*.t> and F<t/uni/*.t> and maybe some sets of tests in F<lib/>.
87
88Questions to answer
89
90=over 4
91
92=item 1
93
94How does screen layout work when you're running more than one test?
95
96=item 2
97
98How does the caller of test specify how many tests to run in parallel?
99
100=item 3
101
102How do setup/teardown tests identify themselves?
103
104=back
105
106Pugs already does parallel testing - can their approach be re-used?
107
0bdfc961 108=head2 Make Schwern poorer
e50bb9a1 109
0bdfc961
NC
110We should have for everything. When all the core's modules are tested,
111Schwern has promised to donate to $500 to TPF. We may need volunteers to
112hold him upside down and shake vigorously in order to actually extract the
113cash.
3958b146 114
0bdfc961 115See F<t/lib/1_compile.t> for the 3 remaining modules that need tests.
e50bb9a1 116
0bdfc961 117=head2 Improve the coverage of the core tests
e50bb9a1 118
0bdfc961
NC
119Use Devel::Cover to ascertain the core's test coverage, then add tests that
120are currently missing.
30222c0f 121
0bdfc961 122=head2 test B
e50bb9a1 123
0bdfc961 124A full test suite for the B module would be nice.
e50bb9a1 125
0bdfc961 126=head2 A decent benchmark
e50bb9a1 127
617eabfa 128C<perlbench> seems impervious to any recent changes made to the perl core. It
0bdfc961
NC
129would be useful to have a reasonable general benchmarking suite that roughly
130represented what current perl programs do, and measurably reported whether
131tweaks to the core improve, degrade or don't really affect performance, to
132guide people attempting to optimise the guts of perl. Gisle would welcome
133new tests for perlbench.
6168cf99 134
0bdfc961 135=head2 fix tainting bugs
6168cf99 136
0bdfc961
NC
137Fix the bugs revealed by running the test suite with the C<-t> switch (via
138C<make test.taintwarn>).
e50bb9a1 139
0bdfc961 140=head2 Dual life everything
e50bb9a1 141
0bdfc961
NC
142As part of the "dists" plan, anything that doesn't belong in the smallest perl
143distribution needs to be dual lifed. Anything else can be too. Figure out what
144changes would be needed to package that module and its tests up for CPAN, and
145do so. Test it with older perl releases, and fix the problems you find.
e50bb9a1 146
0bdfc961 147=head2 Improving C<threads::shared>
722d2a37 148
0bdfc961
NC
149Investigate whether C<threads::shared> could share aggregates properly with
150only Perl level changes to shared.pm
722d2a37 151
0bdfc961 152=head2 POSIX memory footprint
e50bb9a1 153
0bdfc961
NC
154Ilya observed that use POSIX; eats memory like there's no tomorrow, and at
155various times worked to cut it down. There is probably still fat to cut out -
156for example POSIX passes Exporter some very memory hungry data structures.
e50bb9a1 157
e50bb9a1 158
e50bb9a1 159
e50bb9a1 160
e50bb9a1 161
adebf063 162
adebf063 163
0bdfc961 164=head1 Tasks that need a little sysadmin-type knowledge
e50bb9a1 165
0bdfc961
NC
166Or if you prefer, tasks that you would learn from, and broaden your skills
167base...
e50bb9a1 168
617eabfa
NC
169=head2 Relocatable perl
170
171The C level patches needed to create a relocatable perl binary are done, as
172is the work on F<Config.pm>. All that's left to do is the C<Configure> tweaking
173to let people specify how they want to do the install.
174
cd793d32 175=head2 make HTML install work
e50bb9a1 176
adebf063
NC
177There is an C<installhtml> target in the Makefile. It's marked as
178"experimental". It would be good to get this tested, make it work reliably, and
179remove the "experimental" tag. This would include
180
181=over 4
182
183=item 1
184
185Checking that cross linking between various parts of the documentation works.
186In particular that links work between the modules (files with POD in F<lib/>)
187and the core documentation (files in F<pod/>)
188
189=item 2
190
617eabfa
NC
191Work out how to split C<perlfunc> into chunks, preferably one per function
192group, preferably with general case code that could be used elsewhere.
193Challenges here are correctly identifying the groups of functions that go
194together, and making the right named external cross-links point to the right
195page. Things to be aware of are C<-X>, groups such as C<getpwnam> to
196C<endservent>, two or more C<=items> giving the different parameter lists, such
197as
adebf063
NC
198
199 =item substr EXPR,OFFSET,LENGTH,REPLACEMENT
200
201 =item substr EXPR,OFFSET,LENGTH
202
203 =item substr EXPR,OFFSET
204
205and different parameter lists having different meanings. (eg C<select>)
206
207=back
3a89a73c 208
0bdfc961
NC
209=head2 compressed man pages
210
211Be able to install them. This would probably need a configure test to see how
212the system does compressed man pages (same directory/different directory?
213same filename/different filename), as well as tweaking the F<installman> script
214to compress as necessary.
215
30222c0f
NC
216=head2 Add a code coverage target to the Makefile
217
218Make it easy for anyone to run Devel::Cover on the core's tests. The steps
219to do this manually are roughly
220
221=over 4
222
223=item *
224
225do a normal C<Configure>, but include Devel::Cover as a module to install
226(see F<INSTALL> for how to do this)
227
228=item *
229
230 make perl
231
232=item *
233
234 cd t; HARNESS_PERL_SWITCHES=-MDevel::Cover ./perl -I../lib harness
235
236=item *
237
238Process the resulting Devel::Cover database
239
240=back
241
242This just give you the coverage of the F<.pm>s. To also get the C level
243coverage you need to
244
245=over 4
246
247=item *
248
249Additionally tell C<Configure> to use the appropriate C compiler flags for
250C<gcov>
251
252=item *
253
254 make perl.gcov
255
256(instead of C<make perl>)
257
258=item *
259
260After running the tests run C<gcov> to generate all the F<.gcov> files.
261(Including down in the subdirectories of F<ext/>
262
263=item *
264
265(From the top level perl directory) run C<gcov2perl> on all the C<.gcov> files
266to get their stats into the cover_db directory.
267
268=item *
269
270Then process the Devel::Cover database
271
272=back
273
274It would be good to add a single switch to C<Configure> to specify that you
275wanted to perform perl level coverage, and another to specify C level
276coverage, and have C<Configure> and the F<Makefile> do all the right things
277automatically.
278
0bdfc961
NC
279=head2 Make Config.pm cope with differences between build and installed perl
280
281Quite often vendors ship a perl binary compiled with their (pay-for)
282compilers. People install a free compiler, such as gcc. To work out how to
283build extensions, Perl interrogates C<%Config>, so in this situation
284C<%Config> describes compilers that aren't there, and extension building
285fails. This forces people into choosing between re-compiling perl themselves
286using the compiler they have, or only using modules that the vendor ships.
287
288It would be good to find a way teach C<Config.pm> about the installation setup,
289possibly involving probing at install time or later, so that the C<%Config> in
290a binary distribution better describes the installed machine, when the
291installed machine differs from the build machine in some significant way.
292
46925299 293=head2 make parallel builds work
0bdfc961 294
46925299
NC
295Currently parallel builds (such as C<make -j3>) don't work reliably. We believe
296that this is due to incomplete dependency specification in the F<Makefile>.
297It would be good if someone were able to track down the causes of these
298problems, so that parallel builds worked properly.
0bdfc961 299
728f4ecd
NC
300=head2 linker specification files
301
302Some platforms mandate that you provide a list of a shared library's external
303symbols to the linker, so the core already has the infrastructure in place to
304do this for generating shared perl libraries. My understanding is that the
305GNU toolchain can accept an optional linker specification file, and restrict
306visibility just to symbols declared in that file. It would be good to extend
307F<makedef.pl> to support this format, and to provide a means within
308C<Configure> to enable it. This would allow Unix users to test that the
309export list is correct, and to build a perl that does not pollute the global
310namespace with private symbols.
311
8523e164 312
0bdfc961
NC
313
314
315=head1 Tasks that need a little C knowledge
316
317These tasks would need a little C knowledge, but don't need any specific
318background or experience with XS, or how the Perl interpreter works
319
320=head2 Make it clear from -v if this is the exact official release
89007cb3 321
617eabfa
NC
322Currently perl from C<p4>/C<rsync> ships with a F<patchlevel.h> file that
323usually defines one local patch, of the form "MAINT12345" or "RC1". The output
324of perl -v doesn't report that a perl isn't an official release, and this
89007cb3 325information can get lost in bugs reports. Because of this, the minor version
fa11829f 326isn't bumped up until RC time, to minimise the possibility of versions of perl
89007cb3
NC
327escaping that believe themselves to be newer than they actually are.
328
329It would be useful to find an elegant way to have the "this is an interim
330maintenance release" or "this is a release candidate" in the terse -v output,
331and have it so that it's easy for the pumpking to remove this just as the
332release tarball is rolled up. This way the version pulled out of rsync would
333always say "I'm a development release" and it would be safe to bump the
334reported minor version as soon as a release ships, which would aid perl
335developers.
336
0bdfc961
NC
337This task is really about thinking of an elegant way to arrange the C source
338such that it's trivial for the Pumpking to flag "this is an official release"
339when making a tarball, yet leave the default source saying "I'm not the
340official release".
341
0f788cd2
NC
342=head2 Ordering of "global" variables.
343
344F<thrdvar.h> and F<intrpvarh> define the "global" variables that need to be
345per-thread under ithreads, where the variables are actually elements in a
346structure. As C dictates, the variables must be laid out in order of
347declaration. There is a comment
348C</* Important ones in the first cache line (if alignment is done right) */>
349which implies that at some point in the past the ordering was carefully chosen
350(at least in part). However, it's clear that the ordering is less than perfect,
351as currently there are things such as 7 C<bool>s in a row, then something
352typically requiring 4 byte alignment, and then an odd C<bool> later on.
353(C<bool>s are typically defined as C<char>s). So it would be good for someone
354to review the ordering of the variables, to see how much alignment padding can
355be removed.
356
d7939546
NC
357It's also worth checking that all variables are actually used. Perl 5.8.0
358shipped with C<PL_nrs> still defined in F<thrdvar.h>, despite it being unused
359since a change over a year earlier. Had this been spotted before release, it
360could have been removed, but now it has to remain in the 5.8.x releases to
361keep the structure the same size, to retain binary compatibility.
362
62403a3c
NC
363=head2 am I hot or not?
364
365The idea of F<pp_hot.c> is that it contains the I<hot> ops, the ops that are
366most commonly used. The idea is that by grouping them, their object code will
367be adjacent in the executable, so they have a greater chance of already being
368in the CPU cache (or swapped in) due to being near another op already in use.
369
370Except that it's not clear if these really are the most commonly used ops. So
371anyone feeling like exercising their skill with coverage and profiling tools
372might want to determine what ops I<really> are the most commonly used. And in
373turn suggest evictions and promotions to achieve a better F<pp_hot.c>.
374
0bdfc961
NC
375
376
377
0bdfc961 378=head1 Tasks that need a knowledge of XS
e50bb9a1 379
0bdfc961
NC
380These tasks would need C knowledge, and roughly the level of knowledge of
381the perl API that comes from writing modules that use XS to interface to
382C.
383
384=head2 IPv6
385
386Clean this up. Check everything in core works
387
35b64ab6 388=head2 shrink C<IO>s
4a750395 389
35b64ab6
NC
390By removing unused elements and careful re-ordering, the structures for C<AV>s,
391C<HV>s, C<CV>s and C<GV>s have recently been shrunk considerably. C<PVIO>s and
392C<PVBM>s might have some savings to win.
4a750395 393
0bdfc961
NC
394=head2 UTF8 caching code
395
396The string position/offset cache is not optional. It should be.
397
398=head2 Implicit Latin 1 => Unicode translation
399
400Conversions from byte strings to UTF-8 currently map high bit characters
401to Unicode without translation (or, depending on how you look at it, by
402implicitly assuming that the byte strings are in Latin-1). As perl assumes
403the C locale by default, upgrading a string to UTF-8 may change the
404meaning of its contents regarding character classes, case mapping, etc.
405This should probably emit a warning (at least).
406
407This task is incremental - even a little bit of work on it will help.
e50bb9a1 408
cd793d32 409=head2 autovivification
e50bb9a1 410
cd793d32 411Make all autovivification consistent w.r.t LVALUE/RVALUE and strict/no strict;
e50bb9a1 412
0bdfc961 413This task is incremental - even a little bit of work on it will help.
e50bb9a1 414
0bdfc961 415=head2 Unicode in Filenames
e50bb9a1 416
0bdfc961
NC
417chdir, chmod, chown, chroot, exec, glob, link, lstat, mkdir, open,
418opendir, qx, readdir, readlink, rename, rmdir, stat, symlink, sysopen,
419system, truncate, unlink, utime, -X. All these could potentially accept
420Unicode filenames either as input or output (and in the case of system
421and qx Unicode in general, as input or output to/from the shell).
422Whether a filesystem - an operating system pair understands Unicode in
423filenames varies.
e50bb9a1 424
0bdfc961
NC
425Known combinations that have some level of understanding include
426Microsoft NTFS, Apple HFS+ (In Mac OS 9 and X) and Apple UFS (in Mac
427OS X), NFS v4 is rumored to be Unicode, and of course Plan 9. How to
428create Unicode filenames, what forms of Unicode are accepted and used
429(UCS-2, UTF-16, UTF-8), what (if any) is the normalization form used,
430and so on, varies. Finding the right level of interfacing to Perl
431requires some thought. Remember that an OS does not implicate a
432filesystem.
e50bb9a1 433
0bdfc961
NC
434(The Windows -C command flag "wide API support" has been at least
435temporarily retired in 5.8.1, and the -C has been repurposed, see
436L<perlrun>.)
969e704b 437
0bdfc961 438=head2 Unicode in %ENV
969e704b 439
0bdfc961 440Currently the %ENV entries are always byte strings.
e50bb9a1 441
0bdfc961 442=head2 use less 'memory'
e50bb9a1 443
0bdfc961
NC
444Investigate trade offs to switch out perl's choices on memory usage.
445Particularly perl should be able to give memory back.
e50bb9a1 446
0bdfc961 447This task is incremental - even a little bit of work on it will help.
0abe3f7c 448
0bdfc961 449=head2 Re-implement C<:unique> in a way that is actually thread-safe
0abe3f7c 450
0bdfc961
NC
451The old implementation made bad assumptions on several levels. A good 90%
452solution might be just to make C<:unique> work to share the string buffer
453of SvPVs. That way large constant strings can be shared between ithreads,
454such as the configuration information in F<Config>.
0abe3f7c 455
0bdfc961 456=head2 Make tainting consistent
0abe3f7c 457
0bdfc961
NC
458Tainting would be easier to use if it didn't take documented shortcuts and
459allow taint to "leak" everywhere within an expression.
0abe3f7c 460
0bdfc961 461=head2 readpipe(LIST)
0abe3f7c 462
0bdfc961
NC
463system() accepts a LIST syntax (and a PROGRAM LIST syntax) to avoid
464running a shell. readpipe() (the function behind qx//) could be similarly
465extended.
0abe3f7c 466
e50bb9a1 467
e50bb9a1 468
e50bb9a1 469
f86a8bc5 470
0bdfc961 471=head1 Tasks that need a knowledge of the interpreter
3298bd4d 472
0bdfc961
NC
473These tasks would need C knowledge, and knowledge of how the interpreter works,
474or a willingness to learn.
3298bd4d 475
0bdfc961
NC
476=head2 lexical pragmas
477
78ef48ad
RGS
478Document the new support for lexical pragmas in 5.9.3 and how %^H works.
479Maybe C<re>, C<encoding>, maybe other pragmas could be made lexical.
0562c0e3 480
d10fc472 481=head2 Attach/detach debugger from running program
1626a787 482
cd793d32
NC
483The old perltodo notes "With C<gdb>, you can attach the debugger to a running
484program if you pass the process ID. It would be good to do this with the Perl
0bdfc961
NC
485debugger on a running Perl program, although I'm not sure how it would be
486done." ssh and screen do this with named pipes in /tmp. Maybe we can too.
1626a787 487
0bdfc961 488=head2 Constant folding
80b46460 489
0bdfc961
NC
490The peephole optimiser should trap errors during constant folding, and give
491up on the folding, rather than bailing out at compile time. It is quite
492possible that the unfoldable constant is in unreachable code, eg something
493akin to C<$a = 0/0 if 0;>
494
495=head2 LVALUE functions for lists
496
497The old perltodo notes that lvalue functions don't work for list or hash
498slices. This would be good to fix.
499
500=head2 LVALUE functions in the debugger
501
502The old perltodo notes that lvalue functions don't work in the debugger. This
503would be good to fix.
504
505=head2 _ prototype character
506
507Study the possibility of adding a new prototype character, C<_>, meaning
508"this argument defaults to $_".
509
4e577f8b
NC
510=head2 state variables
511
512C<my $foo if 0;> is deprecated, and should be replaced with
513C<state $x = "initial value\n";> the syntax from Perl 6.
514
0bdfc961
NC
515=head2 @INC source filter to Filter::Simple
516
517The second return value from a sub in @INC can be a source filter. This isn't
518documented. It should be changed to use Filter::Simple, tested and documented.
519
520=head2 regexp optimiser optional
521
522The regexp optimiser is not optional. It should configurable to be, to allow
523its performance to be measured, and its bugs to be easily demonstrated.
524
525=head2 UNITCHECK
526
527Introduce a new special block, UNITCHECK, which is run at the end of a
528compilation unit (module, file, eval(STRING) block). This will correspond to
529the Perl 6 CHECK. Perl 5's CHECK cannot be changed or removed because the
530O.pm/B.pm backend framework depends on it.
531
532=head2 optional optimizer
533
534Make the peephole optimizer optional. Currently it performs two tasks as
535it walks the optree - genuine peephole optimisations, and necessary fixups of
536ops. It would be good to find an efficient way to switch out the
537optimisations whilst keeping the fixups.
538
539=head2 You WANT *how* many
540
541Currently contexts are void, scalar and list. split has a special mechanism in
542place to pass in the number of return values wanted. It would be useful to
543have a general mechanism for this, backwards compatible and little speed hit.
544This would allow proposals such as short circuiting sort to be implemented
545as a module on CPAN.
546
547=head2 lexical aliases
548
549Allow lexical aliases (maybe via the syntax C<my \$alias = \$foo>.
550
551=head2 entersub XS vs Perl
552
553At the moment pp_entersub is huge, and has code to deal with entering both
554perl and XS subroutines. Subroutine implementations rarely change between
555perl and XS at run time, so investigate using 2 ops to enter subs (one for
556XS, one for perl) and swap between if a sub is redefined.
2810d901
NC
557
558=head2 Self ties
559
560self ties are currently illegal because they caused too many segfaults. Maybe
561the causes of these could be tracked down and self-ties on all types re-
562instated.
0bdfc961
NC
563
564=head2 Optimize away @_
565
566The old perltodo notes "Look at the "reification" code in C<av.c>".
567
0bdfc961
NC
568=head2 What hooks would assertions need?
569
570Assertions are in the core, and work. However, assertions needed to be added
571as a core patch, rather than an XS module in ext, or a CPAN module, because
572the core has no hooks in the necessary places. It would be useful to
573investigate what hooks would need to be added to make it possible to provide
574the full assertion support from a CPAN module, so that we aren't constraining
575the imagination of future CPAN authors.
576
577
578
579
580
0bdfc961
NC
581=head1 Big projects
582
583Tasks that will get your name mentioned in the description of the "Highlights
584of 5.10"
585
586=head2 make ithreads more robust
587
4e577f8b 588Generally make ithreads more robust. See also L</iCOW>
0bdfc961
NC
589
590This task is incremental - even a little bit of work on it will help, and
591will be greatly appreciated.
592
6c047da7
YST
593One bit would be to write the missing code in sv.c:Perl_dirp_dup.
594
0bdfc961
NC
595=head2 iCOW
596
597Sarathy and Arthur have a proposal for an improved Copy On Write which
598specifically will be able to COW new ithreads. If this can be implemented
599it would be a good thing.
600
601=head2 (?{...}) closures in regexps
602
603Fix (or rewrite) the implementation of the C</(?{...})/> closures.
604
605=head2 A re-entrant regexp engine
606
607This will allow the use of a regex from inside (?{ }), (??{ }) and
608(?(?{ })|) constructs.