This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
blead doesn't include cpan/perlfaq/inc/perlfaq.tt
[perl5.git] / cpan / Test-Simple / lib / Test / FAQ.pod
CommitLineData
18864292
CG
1=pod
2
3=encoding UTF-8
4
518760d9
CG
5=head1 NAME
6
7Test::FAQ - Frequently Asked Questions about testing with Perl
8
9=head1 DESCRIPTION
10
11Frequently Asked Questions about testing in general and specific
12issues with Perl.
13
14=head2 Is there any tutorial on testing?
15
16L<Test::Tutorial>
17
18=head2 Are there any modules for testing?
19
20A whole bunch. Start with L<Test::Simple> then move onto Test::More.
21
22Then go onto L<http://search.cpan.org> and search for "Test".
23
24Also check out L<Fennec>.
25
26=head2 Are there any modules for testing web pages/CGI programs?
27
28L<Test::WWW::Mechanize>, L<Test::WWW::Selenium>
29
30=head2 Are there any modules for testing external programs?
31
32L<Test::Cmd>
33
34=head2 Can you do xUnit/JUnit style testing in Perl?
35
36Yes, L<Test::Class> allows you to write test methods while continuing to
37use all the usual CPAN testing modules. It is the best and most
38perlish way to do xUnit style testing.
39
40L<Test::Unit> is a more direct port of XUnit to Perl, but it does not use
41the Perl conventions and does not play well with other CPAN testing
42modules. As of this writing, it is abandoned. B<Do not use>.
43
44The L<Test::Inline> (aka L<Pod::Tests>) is worth mentioning as it allows you to
45put tests into the POD in the same file as the code.
46
47
48=head2 How do I test my module is backwards/forwards compatible?
49
50First, install a bunch of perls of commonly used versions. At the
51moment, you could try these
52
53 5.7.2
54 5.6.1
55 5.005_03
56 5.004_05
57
58if you're feeling brave, you might want to have on hand these
59
60 bleadperl
61 5.6.0
62 5.004_04
63 5.004
64
65going back beyond 5.003 is probably beyond the call of duty.
66
67You can then add something like this to your F<Makefile.PL>. It
68overrides the L<ExtUtils::MakeMaker> C<test_via_harness()> method to run the tests
69against several different versions of Perl.
70
71 # If PERL_TEST_ALL is set, run "make test" against
72 # other perls as well as the current perl.
73 {
74 package MY;
75
76 sub test_via_harness {
77 my($self, $orig_perl, $tests) = @_;
78
79 # names of your other perl binaries.
80 my @other_perls = qw(perl5.004_05 perl5.005_03 perl5.7.2);
81
82 my @perls = ($orig_perl);
83 push @perls, @other_perls if $ENV{PERL_TEST_ALL};
84
85 my $out;
86 foreach my $perl (@perls) {
87 $out .= $self->SUPER::test_via_harness($perl, $tests);
88 }
89
90 return $out;
91 }
92 }
93
94and re-run your F<Makefile.PL> with the C<PERL_TEST_ALL> environment
95variable set
96
97 PERL_TEST_ALL=1 perl Makefile.PL
98
99now C<make test> will run against each of your other perls.
100
101
102=head2 If I'm testing Foo::Bar, where do I put tests for Foo::Bar::Baz?
103
104=head2 How do I know when my tests are good enough?
105
106A: Use tools for measuring the code coverage of your tests, e.g. how many of
107your source code lines/subs/expressions/paths are executed (aka covered) by
108the test suite. The more, the better, of course, although you may not
109be able achieve 100%. If your testsuite covers under 100%, then
110the rest of your code is, basically, untested. Which means it may work in
111surprising ways (e.g. doesn't do things like they are intended or
112documented), have bugs (e.g. return wrong results) or it may not work at
113all.
114
115=head2 How do I measure the coverage of my test suite?
116
117L<Devel::Cover>
118
119=head2 How do I get tests to run in a certain order?
120
121Tests run in alphabetical order, so simply name your test files in the order
122you want them to run. Numbering your test files works, too.
123
124 t/00_compile.t
125 t/01_config.t
126 t/zz_teardown.t
127
1280 runs first, z runs last.
129
130To achieve a specific order, try L<Test::Manifest>.
131
132Typically you do B<not> want your tests to require being run in a
133certain order, but it can be useful to do a compile check first or to
134run the tests on a very basic module before everything else. This
135gives you early information if a basic module fails which will bring
136everything else down.
137
138Another use is if you have a suite wide setup/teardown, such as
139creating and delete a large test database, which may be too
140expensive to do for every test.
141
142We recommend B<against> numbering every test file. For most files
143this ordering will be arbitrary and the leading number obscures the
144real name of the file. See L<What should I name my test files?> for
145more information.
146
147
148=head2 What should I name my tests?
149
150=head2 What should I name my test files?
151
152A test filename serves three purposes:
153
154Most importantly, it serves to identify what is being tested. Each
155test file should test a clear piece of functionality. This could be
156at single class, a single method, even a single bug.
157
158The order in which tests are run is usually dictated by the filename.
159See L<How do I get tests to run in a certain order?> for details.
160
161Finally, the grouping of tests into common bits of functionality can
162be achieved by directory and filenames. For example, all the tests
163for L<Test::Builder> are in the F<t/Builder/> directory.
164
165As an example, F<t/Builder/reset.t> contains the tests for
166C<< Test::Builder->reset >>. F<t/00compile.t> checks that everything
167compiles, and it will run first. F<t/dont_overwrite_die_handler.t>
168checks that we don't overwrite the C<< $SIG{__DIE__} >> handler.
169
170
171=head2 How do I deal with tests that sometimes pass and sometimes fail?
172
173=head2 How do I test with a database/network/server that the user may or may not have?
174
175=head2 What's a good way to test lists?
176
177C<is_deeply()> from L<Test::More> as well as L<Test::Deep>.
178
179=head2 Is there such a thing as untestable code?
180
181There's always compile/export checks.
182
183Code must be written with testability in mind. Separation of form and
184functionality.
185
186=head2 What do I do when I can't make the code do the same thing twice?
187
188Force it to do the same thing twice.
189
190Even a random number generator can be tested.
191
192=head2 How do I test a GUI?
193
194=head2 How do I test an image generator?
195
196=head2 How do I test that my code handles failures gracefully?
197
198=head2 How do I check the right warnings are issued?
199
200L<Test::Warn>
201
202=head2 How do I test code that prints?
203
204L<Test::Output>
205
206=head2 I want to test that my code dies when I do X
207
208L<Test::Exception>
209
210=head2 I want to print out more diagnostic info on failure.
211
212C<ok(...) || diag "...";>
213
214=head2 How can I simulate failures to make sure that my code does the Right Thing in the face of them?
215
216
217=head2 Why use an ok() function?
218
219On Tue, Aug 28, 2001 at 02:12:46PM +0100, Robin Houston wrote:
220> Michael Schwern wrote:
221> > Ah HA! I've been wondering why nobody ever thinks to write a simple
222> > ok() function for their tests! perlhack has bad testing advice.
223>
224> Could you explain the advantage of having a "simple ok() function"?
225
226Because writing:
227
228 print "not " unless some thing worked;
229 print "ok $test\n"; $test++;
230
231gets rapidly annoying. This is why we made up subroutines in the
232first place. It also looks like hell and obscures the real purpose.
233
234Besides, that will cause problems on VMS.
235
236
237> As somebody who has spent many painful hours debugging test failures,
238> I'm intimately familiar with the _disadvantages_. When you run the
239> test, you know that "test 113 failed". That's all you know, in general.
240
241Second advantage is you can easily upgrade the C<ok()> function to fix
242this, either by slapping this line in:
243
244 printf "# Failed test at line %d\n", (caller)[2];
245
246or simply junking the whole thing and switching to L<Test::Simple> or
247L<Test::More>, which does all sorts of nice diagnostics-on-failure for
248you. Its C<ok()> function is backwards compatible with the above.
249
250There's some issues with using L<Test::Simple> to test really basic Perl
251functionality, you have to choose on a per test basis. Since
252L<Test::Simple> doesn't use C<pack()> it's safe for F<t/op/pack.t> to use
253L<Test::Simple>. I just didn't want to make the perlhack patching
254example too complicated.
255
256
257=head2 Dummy Mode
258
259> One compromise would be to use a test-generating script, which allows
260> the tests to be structured simply and _generates_ the actual test
261> code. One could then grep the generated test script to locate the
262> failing code.
263
264This is a very interesting, and very common, response to the problem.
265I'm going to make some observations about reactions to testing,
266they're not specific to you.
267
268If you've ever read the Bastard Operator From Hell series, you'll
269recall the Dummy Mode.
270
271 The words "power surging" and "drivers" have got her. People hear
272 words like that and go into Dummy Mode and do ANYTHING you say. I
273 could tell her to run naked across campus with a powercord rammed
274 up her backside and she'd probably do it... Hmmm...
275
276There seems to be a Dummy Mode WRT testing. An otherwise competent
277person goes to write a test and they suddenly forget all basic
278programming practice.
279
280
281The reasons for using an C<ok()> function above are the same reasons to
282use functions in general, we should all know them. We'd laugh our
283heads off at code that repeated as much as your average test does.
284These are newbie mistakes.
285
286And the normal 'can do' flair seems to disappear. I know Robin. I
287*know* that in any other situation he would have come up with the
288C<caller()> trick in about 15 seconds flat. Instead weird, elaborate,
289inelegant hacks are thought up to solve the simplest problems.
290
291
292I guess there are certain programming idioms that are foreign enough
293to throw your brain into reverse if you're not ready for them. Like
294trying to think in Lisp, for example. Or being presented with OO for
295the first time. I guess writing test is one of those.
296
297
298=head2 How do I use Test::More without depending on it?
299
300Install L<Test::More> into F<t/lib> under your source directory. Then in your tests
301say C<use lib 't/lib'>.
302
303=head2 How do I deal with threads and forking?
304
305 use Test::More qw/enable_forking/;
306
307or
308
309 use Test::More qw/modern/;
310
311=head2 Why do I need more than ok?
312
313Since every test can be reduced to checking if a statement is true,
314C<ok()> can test everything. But C<ok()> doesn't tell you why the test
315failed. For that you need to tell the test more... which is why
316you need L<Test::More>.
317
318 ok $pirate->name eq "Roberts", "person's name";
319
320 not ok 1 - person's name
321 # Failed test at pirates.t line 23.
322
323If the above fails, you don't know what C<< $person->name >> returned.
324You have to go in and add a C<diag> call. This is time consuming. If
325it's a heisenbug, it might not fail again! If it's a user reporting a
326test failure, they might not be bothered to hack the tests to give you
327more information.
328
329 is $person->name, "Roberts", "person's name";
330
331 not ok 1 - person's name
332 # Failed test at pirates.t line 23.
333 # got: 'Wesley'
334 # expected: 'Roberts'
335
336Using C<is> from L<Test::More> you now know what value you got and
337what value you expected.
338
339The most useful functions in L<Test::More> are C<is()>, C<like()> and C<is_deeply()>.
340
341
342=head2 What's wrong with C<print $test ? "ok" : "not ok">?
343
344=head2 How do I check for an infinite loop?
345
346On Mon, Mar 18, 2002 at 03:57:55AM -0500, Mark-Jason Dominus wrote:
347>
348> Michael The Schwern <schwern@pobox.com> says:
349> > Use alarm and skip the test if $Config{d_alarm} is false (see
350> > t/op/alarm.t for an example). If you think the infinite loop is due
351> > to a programming glitch, as opposed to a cross-platform issue, this
352> > will be enough.
353>
354> Thanks very much!
355>
356
357=head2 How can I check that flock works?
358
359=head2 How do I use the comparison functions of a testing module without it being a test?
360
361Any testing function based on L<Test::Builder>, most are, can be quieted so it does
362not do any testing. It simply returns true or false. Use the following code...
363
364 use Test::More; # or any testing module
365
366 use Test::Builder;
367 use File::Spec;
368
369 # Get the internal Test::Builder object
370 my $tb = Test::Builder->new;
371
372 $tb->plan("no_plan");
373
374 # Keep Test::Builder from displaying anything
375 $tb->no_diag(1);
376 $tb->no_ending(1);
377 $tb->no_header(1);
378 $tb->output( File::Spec->devnull );
379
380 # Now you can use the testing function.
381 print is_deeply( "foo", "bar" ) ? "Yes" : "No";
382
518760d9
CG
383=head1 SOURCE
384
385The source code repository for Test::More can be found at
386F<http://github.com/Test-More/test-more/>.
387
388=head1 MAINTAINER
389
390=over 4
391
392=item Chad Granum E<lt>exodist@cpan.orgE<gt>
393
394=back
395
396=head1 AUTHORS
397
398The following people have all contributed to the Test-More dist (sorted using
399VIM's sort function).
400
401=over 4
402
403=item Chad Granum E<lt>exodist@cpan.orgE<gt>
404
405=item Fergal Daly E<lt>fergal@esatclear.ie>E<gt>
406
407=item Mark Fowler E<lt>mark@twoshortplanks.comE<gt>
408
409=item Michael G Schwern E<lt>schwern@pobox.comE<gt>
410
411=item 唐鳳
412
413=back
414
415=head1 COPYRIGHT
416
417There has been a lot of code migration between modules,
418here are all the original copyrights together:
419
420=over 4
421
422=item Test::Stream
423
424=item Test::Stream::Tester
425
426Copyright 2014 Chad Granum E<lt>exodist7@gmail.comE<gt>.
427
428This program is free software; you can redistribute it and/or
429modify it under the same terms as Perl itself.
430
431See F<http://www.perl.com/perl/misc/Artistic.html>
432
433=item Test::Simple
434
435=item Test::More
436
437=item Test::Builder
438
439Originally authored by Michael G Schwern E<lt>schwern@pobox.comE<gt> with much
440inspiration from Joshua Pritikin's Test module and lots of help from Barrie
441Slaymaker, Tony Bowden, blackstar.co.uk, chromatic, Fergal Daly and the perl-qa
442gang.
443
444Idea by Tony Bowden and Paul Johnson, code by Michael G Schwern
445E<lt>schwern@pobox.comE<gt>, wardrobe by Calvin Klein.
446
447Copyright 2001-2008 by Michael G Schwern E<lt>schwern@pobox.comE<gt>.
448
449This program is free software; you can redistribute it and/or
450modify it under the same terms as Perl itself.
451
452See F<http://www.perl.com/perl/misc/Artistic.html>
453
454=item Test::use::ok
455
456To the extent possible under law, 唐鳳 has waived all copyright and related
457or neighboring rights to L<Test-use-ok>.
458
459This work is published from Taiwan.
460
461L<http://creativecommons.org/publicdomain/zero/1.0>
462
463=item Test::Tester
464
465This module is copyright 2005 Fergal Daly <fergal@esatclear.ie>, some parts
466are based on other people's work.
467
468Under the same license as Perl itself
469
470See http://www.perl.com/perl/misc/Artistic.html
471
472=item Test::Builder::Tester
473
474Copyright Mark Fowler E<lt>mark@twoshortplanks.comE<gt> 2002, 2004.
475
476This program is free software; you can redistribute it
477and/or modify it under the same terms as Perl itself.
478
479=back