This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
todo.pod: Clean up entries related to op slabs
[perl5.git] / Porting / todo.pod
CommitLineData
7711098a
GS
1=head1 NAME
2
c3143508 3todo - Perl TO-DO list
7711098a
GS
4
5=head1 DESCRIPTION
e50bb9a1 6
049aabcb 7This is a list of wishes for Perl. The most up to date version of this file
c3143508 8is at L<http://perl5.git.perl.org/perl.git/blob_plain/HEAD:/Porting/todo.pod>
049aabcb
NC
9
10The tasks we think are smaller or easier are listed first. Anyone is welcome
11to work on any of these, but it's a good idea to first contact
12I<perl5-porters@perl.org> to avoid duplication of effort, and to learn from
13any previous attempts. By all means contact a pumpking privately first if you
14prefer.
e50bb9a1 15
0bdfc961
NC
16Whilst patches to make the list shorter are most welcome, ideas to add to
17the list are also encouraged. Check the perl5-porters archives for past
b4af8972
RB
18ideas, and any discussion about them. One set of archives may be found at
19L<http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/>
938c8732 20
617eabfa
NC
21What can we offer you in return? Fame, fortune, and everlasting glory? Maybe
22not, but if your patch is incorporated, then we'll add your name to the
23F<AUTHORS> file, which ships in the official distribution. How many other
24programming languages offer you 1 line of immortality?
938c8732 25
0bdfc961 26=head1 Tasks that only need Perl knowledge
e50bb9a1 27
de2b17d8
NC
28=head2 Migrate t/ from custom TAP generation
29
30Many tests below F<t/> still generate TAP by "hand", rather than using library
96090e4f 31functions. As explained in L<perlhack/TESTING>, tests in F<t/> are
de2b17d8
NC
32written in a particular way to test that more complex constructions actually
33work before using them routinely. Hence they don't use C<Test::More>, but
34instead there is an intentionally simpler library, F<t/test.pl>. However,
35quite a few tests in F<t/> have not been refactored to use it. Refactoring
36any of these tests, one at a time, is a useful thing TODO.
37
0d8e5a42
RGS
38The subdirectories F<base>, F<cmd> and F<comp>, that contain the most
39basic tests, should be excluded from this task.
40
0be987a2
NC
41=head2 Automate perldelta generation
42
43The perldelta file accompanying each release summaries the major changes.
44It's mostly manually generated currently, but some of that could be
45automated with a bit of perl, specifically the generation of
46
47=over
48
49=item Modules and Pragmata
50
51=item New Documentation
52
53=item New Tests
54
55=back
56
57See F<Porting/how_to_write_a_perldelta.pod> for details.
58
0bdfc961 59=head2 Make Schwern poorer
e50bb9a1 60
613bd4f7 61We should have tests for everything. When all the core's modules are tested,
0bdfc961
NC
62Schwern has promised to donate to $500 to TPF. We may need volunteers to
63hold him upside down and shake vigorously in order to actually extract the
64cash.
3958b146 65
0bdfc961 66=head2 Improve the coverage of the core tests
e50bb9a1 67
e1020413 68Use Devel::Cover to ascertain the core modules' test coverage, then add
02f21748 69tests that are currently missing.
30222c0f 70
0bdfc961 71=head2 test B
e50bb9a1 72
0bdfc961 73A full test suite for the B module would be nice.
e50bb9a1 74
0bdfc961 75=head2 A decent benchmark
e50bb9a1 76
617eabfa 77C<perlbench> seems impervious to any recent changes made to the perl core. It
0bdfc961
NC
78would be useful to have a reasonable general benchmarking suite that roughly
79represented what current perl programs do, and measurably reported whether
80tweaks to the core improve, degrade or don't really affect performance, to
81guide people attempting to optimise the guts of perl. Gisle would welcome
4e1c9055
NC
82new tests for perlbench. Steffen Schwingon would welcome help with
83L<Benchmark::Perl::Formance>
6168cf99 84
0bdfc961 85=head2 fix tainting bugs
6168cf99 86
0bdfc961
NC
87Fix the bugs revealed by running the test suite with the C<-t> switch (via
88C<make test.taintwarn>).
e50bb9a1 89
0bdfc961 90=head2 Dual life everything
e50bb9a1 91
0bdfc961
NC
92As part of the "dists" plan, anything that doesn't belong in the smallest perl
93distribution needs to be dual lifed. Anything else can be too. Figure out what
94changes would be needed to package that module and its tests up for CPAN, and
95do so. Test it with older perl releases, and fix the problems you find.
e50bb9a1 96
a393eb28
RGS
97To make a minimal perl distribution, it's useful to look at
98F<t/lib/commonsense.t>.
99
0bdfc961 100=head2 POSIX memory footprint
e50bb9a1 101
0bdfc961
NC
102Ilya observed that use POSIX; eats memory like there's no tomorrow, and at
103various times worked to cut it down. There is probably still fat to cut out -
104for example POSIX passes Exporter some very memory hungry data structures.
e50bb9a1 105
8c422da5
NC
106=head2 makedef.pl and conditional compilation
107
108The script F<makedef.pl> that generates the list of exported symbols on
109platforms which need this. Functions are declared in F<embed.fnc>, variables
110in F<intrpvar.h>. Quite a few of the functions and variables are conditionally
111declared there, using C<#ifdef>. However, F<makedef.pl> doesn't understand the
112C macros, so the rules about which symbols are present when is duplicated in
113the Perl code. Writing things twice is bad, m'kay. It would be good to teach
114F<.pl> to understand the conditional compilation, and hence remove the
115duplication, and the mistakes it has caused.
e50bb9a1 116
801de10e
NC
117=head2 use strict; and AutoLoad
118
119Currently if you write
120
121 package Whack;
122 use AutoLoader 'AUTOLOAD';
123 use strict;
124 1;
125 __END__
126 sub bloop {
127 print join (' ', No, strict, here), "!\n";
128 }
129
130then C<use strict;> isn't in force within the autoloaded subroutines. It would
131be more consistent (and less surprising) to arrange for all lexical pragmas
132in force at the __END__ block to be in force within each autoloaded subroutine.
133
773b3597
RGS
134There's a similar problem with SelfLoader.
135
91d0cbf6
NC
136=head2 profile installman
137
138The F<installman> script is slow. All it is doing text processing, which we're
139told is something Perl is good at. So it would be nice to know what it is doing
140that is taking so much CPU, and where possible address it.
141
c69ca1d4 142=head2 enable lexical enabling/disabling of individual warnings
a9ed9b74
JV
143
144Currently, warnings can only be enabled or disabled by category. There
145are times when it would be useful to quash a single warning, not a
146whole category.
91d0cbf6 147
85234543
KW
148=head2 document diagnostics
149
150Many diagnostic messages are not currently documented. The list is at the end
151of t/porting/diag.t.
152
0bdfc961 153=head1 Tasks that need a little sysadmin-type knowledge
e50bb9a1 154
0bdfc961
NC
155Or if you prefer, tasks that you would learn from, and broaden your skills
156base...
e50bb9a1 157
cd793d32 158=head2 make HTML install work
e50bb9a1 159
78b489b0 160There is an C<install.html> target in the Makefile. It's marked as
adebf063
NC
161"experimental". It would be good to get this tested, make it work reliably, and
162remove the "experimental" tag. This would include
163
164=over 4
165
166=item 1
167
168Checking that cross linking between various parts of the documentation works.
169In particular that links work between the modules (files with POD in F<lib/>)
170and the core documentation (files in F<pod/>)
171
172=item 2
173
78b489b0
NC
174Improving the code that split C<perlfunc> into chunks, preferably with
175general case code added to L<Pod::Functions> that could be used elsewhere.
176
617eabfa
NC
177Challenges here are correctly identifying the groups of functions that go
178together, and making the right named external cross-links point to the right
78b489b0
NC
179page. Currently this works reasonably well in the general case, and correctly
180parses two or more C<=items> giving the different parameter lists for the
181same function, such used by C<substr>. However it fails completely where
182I<different> functions are listed as a sequence of C<=items> but share the
183same description. All the functions from C<getpwnam> to C<endprotoent> have
184individual stub pages, with only the page for C<endservent> holding the
185description common to all. Likewise C<q>, C<qq> and C<qw> have stub pages,
186instead of sharing the body of C<qx>.
187
188Note also the current code isn't ideal with the two forms of C<select>, mushing
189them both into one F<select.html> with the two descriptions run together.
190Fixing this may well be a special case.
adebf063
NC
191
192=back
3a89a73c 193
0bdfc961
NC
194=head2 compressed man pages
195
196Be able to install them. This would probably need a configure test to see how
197the system does compressed man pages (same directory/different directory?
198same filename/different filename), as well as tweaking the F<installman> script
199to compress as necessary.
200
30222c0f
NC
201=head2 Add a code coverage target to the Makefile
202
203Make it easy for anyone to run Devel::Cover on the core's tests. The steps
204to do this manually are roughly
205
206=over 4
207
208=item *
209
210do a normal C<Configure>, but include Devel::Cover as a module to install
f11a3063 211(see L<INSTALL> for how to do this)
30222c0f
NC
212
213=item *
214
215 make perl
216
217=item *
218
219 cd t; HARNESS_PERL_SWITCHES=-MDevel::Cover ./perl -I../lib harness
220
221=item *
222
223Process the resulting Devel::Cover database
224
225=back
226
227This just give you the coverage of the F<.pm>s. To also get the C level
228coverage you need to
229
230=over 4
231
232=item *
233
234Additionally tell C<Configure> to use the appropriate C compiler flags for
235C<gcov>
236
237=item *
238
239 make perl.gcov
240
241(instead of C<make perl>)
242
243=item *
244
245After running the tests run C<gcov> to generate all the F<.gcov> files.
246(Including down in the subdirectories of F<ext/>
247
248=item *
249
250(From the top level perl directory) run C<gcov2perl> on all the C<.gcov> files
251to get their stats into the cover_db directory.
252
253=item *
254
255Then process the Devel::Cover database
256
257=back
258
259It would be good to add a single switch to C<Configure> to specify that you
260wanted to perform perl level coverage, and another to specify C level
261coverage, and have C<Configure> and the F<Makefile> do all the right things
262automatically.
263
02f21748 264=head2 Make Config.pm cope with differences between built and installed perl
0bdfc961
NC
265
266Quite often vendors ship a perl binary compiled with their (pay-for)
267compilers. People install a free compiler, such as gcc. To work out how to
268build extensions, Perl interrogates C<%Config>, so in this situation
269C<%Config> describes compilers that aren't there, and extension building
270fails. This forces people into choosing between re-compiling perl themselves
271using the compiler they have, or only using modules that the vendor ships.
272
273It would be good to find a way teach C<Config.pm> about the installation setup,
274possibly involving probing at install time or later, so that the C<%Config> in
275a binary distribution better describes the installed machine, when the
276installed machine differs from the build machine in some significant way.
277
728f4ecd
NC
278=head2 linker specification files
279
280Some platforms mandate that you provide a list of a shared library's external
281symbols to the linker, so the core already has the infrastructure in place to
4e1c9055
NC
282do this for generating shared perl libraries. Florian Ragwitz has been working
283to offer this for the GNU toolchain, to allow Unix users to test that the
728f4ecd 284export list is correct, and to build a perl that does not pollute the global
32d539f5 285namespace with private symbols, and will fail in the same way as msvc or mingw
4e1c9055 286builds or when using PERL_DL_NONLAZY=1. See the branch smoke-me/rafl/ld_export
728f4ecd 287
a229ae3b
RGS
288=head2 Cross-compile support
289
4e1c9055
NC
290We get requests for "how to cross compile Perl". The vast majority of these
291seem to be for a couple of scenarios:
292
293=over 4
294
295=item *
296
297Platforms that could build natively using F<./Configure> (I<e.g.> Linux or
298NetBSD on MIPS or ARM) but people want to use a beefier machine (and on the
299same OS) to build more easily.
300
301=item *
302
303Platforms that can't build natively, but no (significant) porting changes
304are needed to our current source code. Prime example of this is Android.
305
306=back
307
308There are several scripts and tools for cross-compiling perl for other
309platforms. However, these are somewhat inconsistent and scattered across the
310codebase, none are documented well, none are clearly flexible enough to
311be confident that they can support any TARGET/HOST plaform pair other than
312that which they were developed on, and it's not clear how bitrotted they are.
313
314For example, C<Configure> understands C<-Dusecrosscompile> option. This option
a229ae3b 315arranges for building C<miniperl> for TARGET machine, so this C<miniperl> is
4e1c9055
NC
316assumed then to be copied to TARGET machine and used as a replacement of
317full C<perl> executable. This code is almost 10 years old. Meanwhile, the
318F<Cross/> directory contains two different approaches for cross compiling to
319ARM Linux targets, relying on hand curated F<config.sh> files, but that code
320is getting on for 5 years old, and requires insider knowledge of perl's
321build system to draft a F<config.sh> for a new platform.
322
323Jess Robinson has sumbitted a grant to TPF to work on cleaning this up.
0bdfc961 324
98fca0e8
NC
325=head2 Split "linker" from "compiler"
326
327Right now, Configure probes for two commands, and sets two variables:
328
329=over 4
330
b91dd380 331=item * C<cc> (in F<cc.U>)
98fca0e8
NC
332
333This variable holds the name of a command to execute a C compiler which
334can resolve multiple global references that happen to have the same
335name. Usual values are F<cc> and F<gcc>.
336Fervent ANSI compilers may be called F<c89>. AIX has F<xlc>.
337
b91dd380 338=item * C<ld> (in F<dlsrc.U>)
98fca0e8
NC
339
340This variable indicates the program to be used to link
341libraries for dynamic loading. On some systems, it is F<ld>.
342On ELF systems, it should be C<$cc>. Mostly, we'll try to respect
343the hint file setting.
344
345=back
346
8d159ec1
NC
347There is an implicit historical assumption from around Perl5.000alpha
348something, that C<$cc> is also the correct command for linking object files
349together to make an executable. This may be true on Unix, but it's not true
350on other platforms, and there are a maze of work arounds in other places (such
351as F<Makefile.SH>) to cope with this.
98fca0e8
NC
352
353Ideally, we should create a new variable to hold the name of the executable
354linker program, probe for it in F<Configure>, and centralise all the special
355case logic there or in hints files.
356
357A small bikeshed issue remains - what to call it, given that C<$ld> is already
8d159ec1
NC
358taken (arguably for the wrong thing now, but on SunOS 4.1 it is the command
359for creating dynamically-loadable modules) and C<$link> could be confused with
360the Unix command line executable of the same name, which does something
361completely different. Andy Dougherty makes the counter argument "In parrot, I
362tried to call the command used to link object files and libraries into an
363executable F<link>, since that's what my vaguely-remembered DOS and VMS
364experience suggested. I don't think any real confusion has ensued, so it's
365probably a reasonable name for perl5 to use."
98fca0e8
NC
366
367"Alas, I've always worried that introducing it would make things worse,
368since now the module building utilities would have to look for
369C<$Config{link}> and institute a fall-back plan if it weren't found."
8d159ec1
NC
370Although I can see that as confusing, given that C<$Config{d_link}> is true
371when (hard) links are available.
98fca0e8 372
75585ce3
SP
373=head2 Configure Windows using PowerShell
374
375Currently, Windows uses hard-coded config files based to build the
376config.h for compiling Perl. Makefiles are also hard-coded and need to be
377hand edited prior to building Perl. While this makes it easy to create a perl.exe
378that works across multiple Windows versions, being able to accurately
379configure a perl.exe for a specific Windows versions and VS C++ would be
380a nice enhancement. With PowerShell available on Windows XP and up, this
381may now be possible. Step 1 might be to investigate whether this is possible
382and use this to clean up our current makefile situation. Step 2 would be to
383see if there would be a way to use our existing metaconfig units to configure a
384Windows Perl or whether we go in a separate direction and make it so. Of
385course, we all know what step 3 is.
386
0bdfc961
NC
387=head1 Tasks that need a little C knowledge
388
389These tasks would need a little C knowledge, but don't need any specific
390background or experience with XS, or how the Perl interpreter works
391
3d826b29
NC
392=head2 Weed out needless PERL_UNUSED_ARG
393
394The C code uses the macro C<PERL_UNUSED_ARG> to stop compilers warning about
395unused arguments. Often the arguments can't be removed, as there is an
396external constraint that determines the prototype of the function, so this
397approach is valid. However, there are some cases where C<PERL_UNUSED_ARG>
398could be removed. Specifically
399
400=over 4
401
402=item *
403
404The prototypes of (nearly all) static functions can be changed
405
406=item *
407
408Unused arguments generated by short cut macros are wasteful - the short cut
409macro used can be changed.
410
411=back
412
bcbaa2d5
RGS
413=head2 -Duse32bit*
414
415Natively 64-bit systems need neither -Duse64bitint nor -Duse64bitall.
416On these systems, it might be the default compilation mode, and there
417is currently no guarantee that passing no use64bitall option to the
418Configure process will build a 32bit perl. Implementing -Duse32bit*
e12cb30b 419options would be nice for perl 5.18.0.
bcbaa2d5 420
fee0a0f7 421=head2 Profile Perl - am I hot or not?
62403a3c 422
fee0a0f7
NC
423The Perl source code is stable enough that it makes sense to profile it,
424identify and optimise the hotspots. It would be good to measure the
425performance of the Perl interpreter using free tools such as cachegrind,
426gprof, and dtrace, and work to reduce the bottlenecks they reveal.
427
428As part of this, the idea of F<pp_hot.c> is that it contains the I<hot> ops,
429the ops that are most commonly used. The idea is that by grouping them, their
430object code will be adjacent in the executable, so they have a greater chance
431of already being in the CPU cache (or swapped in) due to being near another op
432already in use.
62403a3c
NC
433
434Except that it's not clear if these really are the most commonly used ops. So
fee0a0f7
NC
435as part of exercising your skills with coverage and profiling tools you might
436want to determine what ops I<really> are the most commonly used. And in turn
437suggest evictions and promotions to achieve a better F<pp_hot.c>.
62403a3c 438
91d0cbf6
NC
439One piece of Perl code that might make a good testbed is F<installman>.
440
a229ae3b 441=head2 Improve win32/wince.c
0bdfc961 442
a229ae3b 443Currently, numerous functions look virtually, if not completely,
c23989d1 444identical in both F<win32/wince.c> and F<win32/win32.c> files, which can't
6d71adcd
NC
445be good.
446
c5b31784
SH
447=head2 Use secure CRT functions when building with VC8 on Win32
448
449Visual C++ 2005 (VC++ 8.x) deprecated a number of CRT functions on the basis
450that they were "unsafe" and introduced differently named secure versions of
451them as replacements, e.g. instead of writing
452
453 FILE* f = fopen(__FILE__, "r");
454
455one should now write
456
457 FILE* f;
458 errno_t err = fopen_s(&f, __FILE__, "r");
459
460Currently, the warnings about these deprecations have been disabled by adding
461-D_CRT_SECURE_NO_DEPRECATE to the CFLAGS. It would be nice to remove that
462warning suppressant and actually make use of the new secure CRT functions.
463
464There is also a similar issue with POSIX CRT function names like fileno having
465been deprecated in favour of ISO C++ conformant names like _fileno. These
26a6faa8 466warnings are also currently suppressed by adding -D_CRT_NONSTDC_NO_DEPRECATE. It
c5b31784
SH
467might be nice to do as Microsoft suggest here too, although, unlike the secure
468functions issue, there is presumably little or no benefit in this case.
469
038ae9a4
SH
470=head2 Fix POSIX::access() and chdir() on Win32
471
472These functions currently take no account of DACLs and therefore do not behave
473correctly in situations where access is restricted by DACLs (as opposed to the
474read-only attribute).
475
476Furthermore, POSIX::access() behaves differently for directories having the
477read-only attribute set depending on what CRT library is being used. For
478example, the _access() function in the VC6 and VC7 CRTs (wrongly) claim that
479such directories are not writable, whereas in fact all directories are writable
480unless access is denied by DACLs. (In the case of directories, the read-only
481attribute actually only means that the directory cannot be deleted.) This CRT
482bug is fixed in the VC8 and VC9 CRTs (but, of course, the directory may still
483not actually be writable if access is indeed denied by DACLs).
484
485For the chdir() issue, see ActiveState bug #74552:
b4af8972 486L<http://bugs.activestate.com/show_bug.cgi?id=74552>
038ae9a4
SH
487
488Therefore, DACLs should be checked both for consistency across CRTs and for
489the correct answer.
490
491(Note that perl's -w operator should not be modified to check DACLs. It has
492been written so that it reflects the state of the read-only attribute, even
493for directories (whatever CRT is being used), for symmetry with chmod().)
494
16815324
NC
495=head2 strcat(), strcpy(), strncat(), strncpy(), sprintf(), vsprintf()
496
497Maybe create a utility that checks after each libperl.a creation that
498none of the above (nor sprintf(), vsprintf(), or *SHUDDER* gets())
499ever creep back to libperl.a.
500
501 nm libperl.a | ./miniperl -alne '$o = $F[0] if /:$/; print "$o $F[1]" if $F[0] eq "U" && $F[1] =~ /^(?:strn?c(?:at|py)|v?sprintf|gets)$/'
502
503Note, of course, that this will only tell whether B<your> platform
504is using those naughty interfaces.
505
2a930eea 506=head2 -D_FORTIFY_SOURCE=2
de96509d 507
2a930eea 508Recent glibcs support C<-D_FORTIFY_SOURCE=2> which gives
de96509d 509protection against various kinds of buffer overflow problems.
2a930eea 510It should probably be used for compiling Perl whenever available,
de96509d 511Configure and/or hints files should be adjusted to probe for the
2a930eea 512availability of these feature and enable it as appropriate.
16815324 513
8964cfe0
NC
514=head2 Arenas for GPs? For MAGIC?
515
516C<struct gp> and C<struct magic> are both currently allocated by C<malloc>.
517It might be a speed or memory saving to change to using arenas. Or it might
518not. It would need some suitable benchmarking first. In particular, C<GP>s
519can probably be changed with minimal compatibility impact (probably nothing
520outside of the core, or even outside of F<gv.c> allocates them), but they
521probably aren't allocated/deallocated often enough for a speed saving. Whereas
522C<MAGIC> is allocated/deallocated more often, but in turn, is also something
523more externally visible, so changing the rules here may bite external code.
524
3880c8ec
NC
525=head2 Shared arenas
526
527Several SV body structs are now the same size, notably PVMG and PVGV, PVAV and
528PVHV, and PVCV and PVFM. It should be possible to allocate and return same
529sized bodies from the same actual arena, rather than maintaining one arena for
530each. This could save 4-6K per thread, of memory no longer tied up in the
531not-yet-allocated part of an arena.
532
8964cfe0 533
6d71adcd
NC
534=head1 Tasks that need a knowledge of XS
535
536These tasks would need C knowledge, and roughly the level of knowledge of
537the perl API that comes from writing modules that use XS to interface to
538C.
539
e851c105
DG
540=head2 Write an XS cookbook
541
542Create pod/perlxscookbook.pod with short, task-focused 'recipes' in XS that
543demonstrate common tasks and good practices. (Some of these might be
544extracted from perlguts.) The target audience should be XS novices, who need
545more examples than perlguts but something less overwhelming than perlapi.
546Recipes should provide "one pretty good way to do it" instead of TIMTOWTDI.
547
5b7d14ff
DG
548Rather than focusing on interfacing Perl to C libraries, such a cookbook
549should probably focus on how to optimize Perl routines by re-writing them
550in XS. This will likely be more motivating to those who mostly work in
551Perl but are looking to take the next step into XS.
552
553Deconstructing and explaining some simpler XS modules could be one way to
554bootstrap a cookbook. (List::Util? Class::XSAccessor? Tree::Ternary_XS?)
555Another option could be deconstructing the implementation of some simpler
556functions in op.c.
557
0b162fb0 558=head2 Document how XSUBs can use C<cv_set_call_checker> to inline themselves as OPs
05fb4e20
NC
559
560For a simple XSUB, often the subroutine dispatch takes more time than the
0b162fb0
NC
561XSUB itself. v5.14.0 now allows XSUBs to register a function which will be
562called when the parser is finished building an C<entersub> op which calls
563them.
564
565Registration is done with C<Perl_cv_set_call_checker>, is documented at the
566API level in L<perlapi>, and L<perl5140delta/Custom per-subroutine check hooks>
567notes that it can be used to inline a subroutine, by replacing it with a
568custom op. However there is no further detail of the code needed to do this.
569It would be useful to add one or more annotated examples of how to create
570XSUBs that inline.
571
572This should provide a measurable speed up to simple XSUBs inside
05fb4e20
NC
573tight loops. Initially one would have to write the OP alternative
574implementation by hand, but it's likely that this should be reasonably
575straightforward for the type of XSUB that would benefit the most. Longer
576term, once the run-time implementation is proven, it should be possible to
577progressively update ExtUtils::ParseXS to generate OP implementations for
578some XSUBs.
579
318bf708
NC
580=head2 Remove the use of SVs as temporaries in dump.c
581
582F<dump.c> contains debugging routines to dump out the contains of perl data
583structures, such as C<SV>s, C<AV>s and C<HV>s. Currently, the dumping code
584B<uses> C<SV>s for its temporary buffers, which was a logical initial
585implementation choice, as they provide ready made memory handling.
586
587However, they also lead to a lot of confusion when it happens that what you're
588trying to debug is seen by the code in F<dump.c>, correctly or incorrectly, as
589a temporary scalar it can use for a temporary buffer. It's also not possible
590to dump scalars before the interpreter is properly set up, such as during
591ithreads cloning. It would be good to progressively replace the use of scalars
592as string accumulation buffers with something much simpler, directly allocated
593by C<malloc>. The F<dump.c> code is (or should be) only producing 7 bit
594US-ASCII, so output character sets are not an issue.
595
596Producing and proving an internal simple buffer allocation would make it easier
597to re-write the internals of the PerlIO subsystem to avoid using C<SV>s for
598B<its> buffers, use of which can cause problems similar to those of F<dump.c>,
599at similar times.
600
5d96f598
NC
601=head2 safely supporting POSIX SA_SIGINFO
602
603Some years ago Jarkko supplied patches to provide support for the POSIX
604SA_SIGINFO feature in Perl, passing the extra data to the Perl signal handler.
605
606Unfortunately, it only works with "unsafe" signals, because under safe
607signals, by the time Perl gets to run the signal handler, the extra
608information has been lost. Moreover, it's not easy to store it somewhere,
609as you can't call mutexs, or do anything else fancy, from inside a signal
610handler.
611
612So it strikes me that we could provide safe SA_SIGINFO support
613
614=over 4
615
616=item 1
617
618Provide global variables for two file descriptors
619
620=item 2
621
622When the first request is made via C<sigaction> for C<SA_SIGINFO>, create a
623pipe, store the reader in one, the writer in the other
624
625=item 3
626
627In the "safe" signal handler (C<Perl_csighandler()>/C<S_raise_signal()>), if
628the C<siginfo_t> pointer non-C<NULL>, and the writer file handle is open,
629
630=over 8
631
632=item 1
633
634serialise signal number, C<struct siginfo_t> (or at least the parts we care
635about) into a small auto char buff
636
637=item 2
638
639C<write()> that (non-blocking) to the writer fd
640
641=over 12
642
643=item 1
644
645if it writes 100%, flag the signal in a counter of "signals on the pipe" akin
646to the current per-signal-number counts
647
648=item 2
649
650if it writes 0%, assume the pipe is full. Flag the data as lost?
651
652=item 3
653
654if it writes partially, croak a panic, as your OS is broken.
655
656=back
657
658=back
659
660=item 4
661
662in the regular C<PERL_ASYNC_CHECK()> processing, if there are "signals on
663the pipe", read the data out, deserialise, build the Perl structures on
664the stack (code in C<Perl_sighandler()>, the "unsafe" handler), and call as
665usual.
666
667=back
668
669I think that this gets us decent C<SA_SIGINFO> support, without the current risk
670of running Perl code inside the signal handler context. (With all the dangers
671of things like C<malloc> corruption that that currently offers us)
672
673For more information see the thread starting with this message:
b4af8972 674L<http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2008-03/msg00305.html>
5d96f598 675
6d71adcd
NC
676=head2 autovivification
677
678Make all autovivification consistent w.r.t LVALUE/RVALUE and strict/no strict;
679
680This task is incremental - even a little bit of work on it will help.
681
682=head2 Unicode in Filenames
683
684chdir, chmod, chown, chroot, exec, glob, link, lstat, mkdir, open,
685opendir, qx, readdir, readlink, rename, rmdir, stat, symlink, sysopen,
686system, truncate, unlink, utime, -X. All these could potentially accept
687Unicode filenames either as input or output (and in the case of system
688and qx Unicode in general, as input or output to/from the shell).
689Whether a filesystem - an operating system pair understands Unicode in
690filenames varies.
691
692Known combinations that have some level of understanding include
693Microsoft NTFS, Apple HFS+ (In Mac OS 9 and X) and Apple UFS (in Mac
694OS X), NFS v4 is rumored to be Unicode, and of course Plan 9. How to
695create Unicode filenames, what forms of Unicode are accepted and used
696(UCS-2, UTF-16, UTF-8), what (if any) is the normalization form used,
697and so on, varies. Finding the right level of interfacing to Perl
698requires some thought. Remember that an OS does not implicate a
699filesystem.
700
701(The Windows -C command flag "wide API support" has been at least
702temporarily retired in 5.8.1, and the -C has been repurposed, see
703L<perlrun>.)
704
87a942b1
JH
705Most probably the right way to do this would be this:
706L</"Virtualize operating system access">.
707
6d71adcd
NC
708=head2 Unicode in %ENV
709
710Currently the %ENV entries are always byte strings.
87a942b1 711See L</"Virtualize operating system access">.
6d71adcd 712
799c141b
SH
713(See RT ticket #113536 for information on Win32's handling of %ENV,
714which was fixed to work with native ANSI codepage characters in the
715environment, but still doesn't work with other characters outside of
716that codepage present in the environment.)
717
1f2e7916
JD
718=head2 Unicode and glob()
719
720Currently glob patterns and filenames returned from File::Glob::glob()
87a942b1 721are always byte strings. See L</"Virtualize operating system access">.
1f2e7916 722
6d71adcd
NC
723=head2 use less 'memory'
724
725Investigate trade offs to switch out perl's choices on memory usage.
726Particularly perl should be able to give memory back.
727
728This task is incremental - even a little bit of work on it will help.
729
730=head2 Re-implement C<:unique> in a way that is actually thread-safe
731
732The old implementation made bad assumptions on several levels. A good 90%
733solution might be just to make C<:unique> work to share the string buffer
734of SvPVs. That way large constant strings can be shared between ithreads,
735such as the configuration information in F<Config>.
736
737=head2 Make tainting consistent
738
739Tainting would be easier to use if it didn't take documented shortcuts and
740allow taint to "leak" everywhere within an expression.
741
742=head2 readpipe(LIST)
743
744system() accepts a LIST syntax (and a PROGRAM LIST syntax) to avoid
745running a shell. readpipe() (the function behind qx//) could be similarly
746extended.
747
6d71adcd
NC
748=head2 Audit the code for destruction ordering assumptions
749
750Change 25773 notes
751
752 /* Need to check SvMAGICAL, as during global destruction it may be that
753 AvARYLEN(av) has been freed before av, and hence the SvANY() pointer
754 is now part of the linked list of SV heads, rather than pointing to
755 the original body. */
756 /* FIXME - audit the code for other bugs like this one. */
757
758adding the C<SvMAGICAL> check to
759
760 if (AvARYLEN(av) && SvMAGICAL(AvARYLEN(av))) {
761 MAGIC *mg = mg_find (AvARYLEN(av), PERL_MAGIC_arylen);
762
763Go through the core and look for similar assumptions that SVs have particular
764types, as all bets are off during global destruction.
765
749904bf
JH
766=head2 Extend PerlIO and PerlIO::Scalar
767
768PerlIO::Scalar doesn't know how to truncate(). Implementing this
769would require extending the PerlIO vtable.
770
771Similarly the PerlIO vtable doesn't know about formats (write()), or
772about stat(), or chmod()/chown(), utime(), or flock().
773
774(For PerlIO::Scalar it's hard to see what e.g. mode bits or ownership
775would mean.)
776
777PerlIO doesn't do directories or symlinks, either: mkdir(), rmdir(),
778opendir(), closedir(), seekdir(), rewinddir(), glob(); symlink(),
779readlink().
780
94da6c29
JH
781See also L</"Virtualize operating system access">.
782
d6c1e11f
JH
783=head2 Organize error messages
784
785Perl's diagnostics (error messages, see L<perldiag>) could use
a8d0aeb9 786reorganizing and formalizing so that each error message has its
d6c1e11f
JH
787stable-for-all-eternity unique id, categorized by severity, type, and
788subsystem. (The error messages would be listed in a datafile outside
c4bd451b
CB
789of the Perl source code, and the source code would only refer to the
790messages by the id.) This clean-up and regularizing should apply
d6c1e11f
JH
791for all croak() messages.
792
793This would enable all sorts of things: easier translation/localization
794of the messages (though please do keep in mind the caveats of
795L<Locale::Maketext> about too straightforward approaches to
796translation), filtering by severity, and instead of grepping for a
797particular error message one could look for a stable error id. (Of
798course, changing the error messages by default would break all the
799existing software depending on some particular error message...)
800
801This kind of functionality is known as I<message catalogs>. Look for
802inspiration for example in the catgets() system, possibly even use it
803if available-- but B<only> if available, all platforms will B<not>
de96509d 804have catgets().
d6c1e11f
JH
805
806For the really pure at heart, consider extending this item to cover
807also the warning messages (see L<perllexwarn>, C<warnings.pl>).
3236f110 808
0bdfc961 809=head1 Tasks that need a knowledge of the interpreter
3298bd4d 810
0bdfc961
NC
811These tasks would need C knowledge, and knowledge of how the interpreter works,
812or a willingness to learn.
3298bd4d 813
10517af5
JD
814=head2 forbid labels with keyword names
815
816Currently C<goto keyword> "computes" the label value:
817
818 $ perl -e 'goto print'
819 Can't find label 1 at -e line 1.
820
343c8006
JD
821It is controversial if the right way to avoid the confusion is to forbid
822labels with keyword names, or if it would be better to always treat
823bareword expressions after a "goto" as a label and never as a keyword.
10517af5 824
de6375e3
RGS
825=head2 truncate() prototype
826
827The prototype of truncate() is currently C<$$>. It should probably
828be C<*$> instead. (This is changed in F<opcode.pl>)
829
565590b5
NC
830=head2 error reporting of [$a ; $b]
831
832Using C<;> inside brackets is a syntax error, and we don't propose to change
833that by giving it any meaning. However, it's not reported very helpfully:
834
835 $ perl -e '$a = [$b; $c];'
836 syntax error at -e line 1, near "$b;"
837 syntax error at -e line 1, near "$c]"
838 Execution of -e aborted due to compilation errors.
839
840It should be possible to hook into the tokeniser or the lexer, so that when a
841C<;> is parsed where it is not legal as a statement terminator (ie inside
842C<{}> used as a hashref, C<[]> or C<()>) it issues an error something like
843I<';' isn't legal inside an expression - if you need multiple statements use a
844do {...} block>. See the thread starting at
b4af8972 845L<http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2008-09/msg00573.html>
565590b5 846
718140ec
NC
847=head2 lexicals used only once
848
849This warns:
850
851 $ perl -we '$pie = 42'
852 Name "main::pie" used only once: possible typo at -e line 1.
853
854This does not:
855
856 $ perl -we 'my $pie = 42'
857
858Logically all lexicals used only once should warn, if the user asks for
d6f4ea2e
SP
859warnings. An unworked RT ticket (#5087) has been open for almost seven
860years for this discrepancy.
718140ec 861
a3d15f9a
RGS
862=head2 UTF-8 revamp
863
85c006b6
KW
864The handling of Unicode is unclean in many places. In the regex engine
865there are especially many problems. The swash data structure could be
866replaced my something better. Inversion lists and maps are likely
867candidates. The whole Unicode database could be placed in-core for a
868huge speed-up. Only minimal work was done on the optimizer when utf8
869was added, with the result that the synthetic start class often will
870fail to narrow down the possible choices when given non-Latin1 input.
4e1c9055 871Karl Williamson has been working on this - talk to him.
a3d15f9a 872
636e63cb
NC
873=head2 state variable initialization in list context
874
875Currently this is illegal:
876
877 state ($a, $b) = foo();
878
a2874905 879In Perl 6, C<state ($a) = foo();> and C<(state $a) = foo();> have different
a8d0aeb9 880semantics, which is tricky to implement in Perl 5 as currently they produce
a2874905 881the same opcode trees. The Perl 6 design is firm, so it would be good to
a8d0aeb9 882implement the necessary code in Perl 5. There are comments in
a2874905
NC
883C<Perl_newASSIGNOP()> that show the code paths taken by various assignment
884constructions involving state variables.
636e63cb 885
a393eb28
RGS
886=head2 A does() built-in
887
888Like ref(), only useful. It would call the C<DOES> method on objects; it
889would also tell whether something can be dereferenced as an
890array/hash/etc., or used as a regexp, etc.
891L<http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2007-03/msg00481.html>
892
893=head2 Tied filehandles and write() don't mix
894
895There is no method on tied filehandles to allow them to be called back by
896formats.
4fedb12c 897
53967bb9
RGS
898=head2 Propagate compilation hints to the debugger
899
900Currently a debugger started with -dE on the command-line doesn't see the
901features enabled by -E. More generally hints (C<$^H> and C<%^H>) aren't
902propagated to the debugger. Probably it would be a good thing to propagate
903hints from the innermost non-C<DB::> scope: this would make code eval'ed
904in the debugger see the features (and strictures, etc.) currently in
905scope.
906
d10fc472 907=head2 Attach/detach debugger from running program
1626a787 908
cd793d32
NC
909The old perltodo notes "With C<gdb>, you can attach the debugger to a running
910program if you pass the process ID. It would be good to do this with the Perl
0bdfc961
NC
911debugger on a running Perl program, although I'm not sure how it would be
912done." ssh and screen do this with named pipes in /tmp. Maybe we can too.
1626a787 913
0bdfc961
NC
914=head2 LVALUE functions for lists
915
916The old perltodo notes that lvalue functions don't work for list or hash
917slices. This would be good to fix.
918
0bdfc961
NC
919=head2 regexp optimiser optional
920
921The regexp optimiser is not optional. It should configurable to be, to allow
922its performance to be measured, and its bugs to be easily demonstrated.
923
ef36c6a7
RGS
924=head2 C</w> regex modifier
925
926That flag would enable to match whole words, and also to interpolate
927arrays as alternations. With it, C</P/w> would be roughly equivalent to:
928
929 do { local $"='|'; /\b(?:P)\b/ }
930
b4af8972
RB
931See
932L<http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2007-01/msg00400.html>
ef36c6a7
RGS
933for the discussion.
934
0bdfc961
NC
935=head2 optional optimizer
936
937Make the peephole optimizer optional. Currently it performs two tasks as
938it walks the optree - genuine peephole optimisations, and necessary fixups of
939ops. It would be good to find an efficient way to switch out the
940optimisations whilst keeping the fixups.
941
942=head2 You WANT *how* many
943
944Currently contexts are void, scalar and list. split has a special mechanism in
945place to pass in the number of return values wanted. It would be useful to
946have a general mechanism for this, backwards compatible and little speed hit.
947This would allow proposals such as short circuiting sort to be implemented
948as a module on CPAN.
949
950=head2 lexical aliases
951
e12cb30b 952Allow lexical aliases (maybe via the syntax C<my \$alias = \$foo>).
0bdfc961 953
de535794 954=head2 Self-ties
2810d901 955
de535794 956Self-ties are currently illegal because they caused too many segfaults. Maybe
a8d0aeb9 957the causes of these could be tracked down and self-ties on all types
de535794 958reinstated.
0bdfc961
NC
959
960=head2 Optimize away @_
961
962The old perltodo notes "Look at the "reification" code in C<av.c>".
963
87a942b1
JH
964=head2 Virtualize operating system access
965
966Implement a set of "vtables" that virtualizes operating system access
967(open(), mkdir(), unlink(), readdir(), getenv(), etc.) At the very
968least these interfaces should take SVs as "name" arguments instead of
969bare char pointers; probably the most flexible and extensible way
e1a3d5d1
JH
970would be for the Perl-facing interfaces to accept HVs. The system
971needs to be per-operating-system and per-file-system
972hookable/filterable, preferably both from XS and Perl level
87a942b1
JH
973(L<perlport/"Files and Filesystems"> is good reading at this point,
974in fact, all of L<perlport> is.)
975
e1a3d5d1
JH
976This has actually already been implemented (but only for Win32),
977take a look at F<iperlsys.h> and F<win32/perlhost.h>. While all Win32
978variants go through a set of "vtables" for operating system access,
e1020413 979non-Win32 systems currently go straight for the POSIX/Unix-style
e1a3d5d1
JH
980system/library call. Similar system as for Win32 should be
981implemented for all platforms. The existing Win32 implementation
982probably does not need to survive alongside this proposed new
983implementation, the approaches could be merged.
87a942b1
JH
984
985What would this give us? One often-asked-for feature this would
94da6c29
JH
986enable is using Unicode for filenames, and other "names" like %ENV,
987usernames, hostnames, and so forth.
988(See L<perlunicode/"When Unicode Does Not Happen">.)
989
990But this kind of virtualization would also allow for things like
991virtual filesystems, virtual networks, and "sandboxes" (though as long
992as dynamic loading of random object code is allowed, not very safe
993sandboxes since external code of course know not of Perl's vtables).
994An example of a smaller "sandbox" is that this feature can be used to
995implement per-thread working directories: Win32 already does this.
996
997See also L</"Extend PerlIO and PerlIO::Scalar">.
87a942b1 998
52960e22
JC
999=head2 repack the optree
1000
1001Repacking the optree after execution order is determined could allow
057163d7 1002removal of NULL ops, and optimal ordering of OPs with respect to cache-line
2723c0fb 1003filling. I think that
057163d7
NC
1004the best way to do this is to make it an optional step just before the
1005completed optree is attached to anything else, and to use the slab allocator
2723c0fb
FC
1006unchanged--but allocate a single slab the right size, avoiding partial
1007slabs--, so that freeing ops is identical whether or not this step runs.
057163d7
NC
1008Note that the slab allocator allocates ops downwards in memory, so one would
1009have to actually "allocate" the ops in reverse-execution order to get them
1010contiguous in memory in execution order.
1011
b4af8972
RB
1012See
1013L<http://www.nntp.perl.org/group/perl.perl5.porters/2007/12/msg131975.html>
057163d7
NC
1014
1015Note that running this copy, and then freeing all the old location ops would
1016cause their slabs to be freed, which would eliminate possible memory wastage if
1017the previous suggestion is implemented, and we swap slabs more frequently.
52960e22 1018
12e06b6f
NC
1019=head2 eliminate incorrect line numbers in warnings
1020
1021This code
1022
1023 use warnings;
1024 my $undef;
1025
1026 if ($undef == 3) {
1027 } elsif ($undef == 0) {
1028 }
1029
18a16cc5 1030used to produce this output:
12e06b6f
NC
1031
1032 Use of uninitialized value in numeric eq (==) at wrong.pl line 4.
1033 Use of uninitialized value in numeric eq (==) at wrong.pl line 4.
1034
18a16cc5
NC
1035where the line of the second warning was misreported - it should be line 5.
1036Rafael fixed this - the problem arose because there was no nextstate OP
1037between the execution of the C<if> and the C<elsif>, hence C<PL_curcop> still
1038reports that the currently executing line is line 4. The solution was to inject
1039a nextstate OPs for each C<elsif>, although it turned out that the nextstate
1040OP needed to be a nulled OP, rather than a live nextstate OP, else other line
1041numbers became misreported. (Jenga!)
12e06b6f
NC
1042
1043The problem is more general than C<elsif> (although the C<elsif> case is the
1044most common and the most confusing). Ideally this code
1045
1046 use warnings;
1047 my $undef;
1048
1049 my $a = $undef + 1;
1050 my $b
1051 = $undef
1052 + 1;
1053
1054would produce this output
1055
1056 Use of uninitialized value $undef in addition (+) at wrong.pl line 4.
1057 Use of uninitialized value $undef in addition (+) at wrong.pl line 7.
1058
1059(rather than lines 4 and 5), but this would seem to require every OP to carry
1060(at least) line number information.
1061
1062What might work is to have an optional line number in memory just before the
1063BASEOP structure, with a flag bit in the op to say whether it's present.
1064Initially during compile every OP would carry its line number. Then add a late
1065pass to the optimiser (potentially combined with L</repack the optree>) which
1066looks at the two ops on every edge of the graph of the execution path. If
1067the line number changes, flags the destination OP with this information.
1068Once all paths are traced, replace every op with the flag with a
1069nextstate-light op (that just updates C<PL_curcop>), which in turn then passes
1070control on to the true op. All ops would then be replaced by variants that
1071do not store the line number. (Which, logically, why it would work best in
1072conjunction with L</repack the optree>, as that is already copying/reallocating
1073all the OPs)
1074
18a16cc5
NC
1075(Although I should note that we're not certain that doing this for the general
1076case is worth it)
1077
52960e22
JC
1078=head2 optimize tail-calls
1079
1080Tail-calls present an opportunity for broadly applicable optimization;
1081anywhere that C<< return foo(...) >> is called, the outer return can
1082be replaced by a goto, and foo will return directly to the outer
1083caller, saving (conservatively) 25% of perl's call&return cost, which
1084is relatively higher than in C. The scheme language is known to do
1085this heavily. B::Concise provides good insight into where this
1086optimization is possible, ie anywhere entersub,leavesub op-sequence
1087occurs.
1088
1089 perl -MO=Concise,-exec,a,b,-main -e 'sub a{ 1 }; sub b {a()}; b(2)'
1090
1091Bottom line on this is probably a new pp_tailcall function which
1092combines the code in pp_entersub, pp_leavesub. This should probably
1093be done 1st in XS, and using B::Generate to patch the new OP into the
1094optrees.
1095
e12cb30b 1096=head2 Add C<0odddd>
0c397127
KW
1097
1098It has been proposed that octal constants be specifiable through the syntax
1099C<0oddddd>, parallel to the existing construct to specify hex constants
1100C<0xddddd>
1101
0bdfc961
NC
1102=head1 Big projects
1103
1104Tasks that will get your name mentioned in the description of the "Highlights
e12cb30b 1105of 5.18.0"
0bdfc961
NC
1106
1107=head2 make ithreads more robust
1108
45a81a90 1109Generally make ithreads more robust.
0bdfc961
NC
1110
1111This task is incremental - even a little bit of work on it will help, and
1112will be greatly appreciated.
1113
07577ec1
FC
1114One bit would be to determine how to clone directory handles on systems
1115without a C<fchdir> function (in sv.c:Perl_dirp_dup).
6c047da7 1116
59c7f7d5
RGS
1117Fix Perl_sv_dup, et al so that threads can return objects.
1118
6bda09f9
YO
1119=head2 Add class set operations to regexp engine
1120
1121Apparently these are quite useful. Anyway, Jeffery Friedl wants them.
1122
1123demerphq has this on his todo list, but right at the bottom.
44a7a252
JV
1124
1125
1126=head1 Tasks for microperl
1127
1128
1129[ Each and every one of these may be obsolete, but they were listed
1130 in the old Todo.micro file]
1131
44a7a252
JV
1132=head2 do away with fork/exec/wait?
1133
1134(system, popen should be enough?)
1135
1136=head2 some of the uconfig.sh really needs to be probed (using cc) in buildtime:
1137
1138(uConfigure? :-) native datatype widths and endianness come to mind
1139