This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
updated to v1.50 from CPAN
[perl5.git] / pod / perlfaq9.pod
CommitLineData
68dc0745
PP
1=head1 NAME
2
65acb1b1 3perlfaq9 - Networking ($Revision: 1.24 $, $Date: 1999/01/08 05:39:48 $)
68dc0745
PP
4
5=head1 DESCRIPTION
6
7This section deals with questions related to networking, the internet,
8and a few on the web.
9
c8db1d39 10=head2 My CGI script runs from the command line but not the browser. (500 Server Error)
68dc0745 11
c8db1d39
TC
12If you can demonstrate that you've read the following FAQs and that
13your problem isn't something simple that can be easily answered, you'll
14probably receive a courteous and useful reply to your question if you
15post it on comp.infosystems.www.authoring.cgi (if it's something to do
16with HTTP, HTML, or the CGI protocols). Questions that appear to be Perl
17questions but are really CGI ones that are posted to comp.lang.perl.misc
18may not be so well received.
68dc0745 19
c8db1d39 20The useful FAQs and related documents are:
68dc0745 21
c8db1d39 22 CGI FAQ
be94a901 23 http://www.webthing.com/tutorials/cgifaq.html
68dc0745 24
c8db1d39 25 Web FAQ
92c2ed05 26 http://www.boutell.com/faq/
68dc0745 27
c8db1d39
TC
28 WWW Security FAQ
29 http://www.w3.org/Security/Faq/
30
31 HTTP Spec
32 http://www.w3.org/pub/WWW/Protocols/HTTP/
33
34 HTML Spec
35 http://www.w3.org/TR/REC-html40/
36 http://www.w3.org/pub/WWW/MarkUp/
37
38 CGI Spec
39 http://www.w3.org/CGI/
40
41 CGI Security FAQ
42 http://www.go2net.com/people/paulp/cgi-security/safe-cgi.txt
43
44=head2 How can I get better error messages from a CGI program?
45
46Use the CGI::Carp module. It replaces C<warn> and C<die>, plus the
47normal Carp modules C<carp>, C<croak>, and C<confess> functions with
48more verbose and safer versions. It still sends them to the normal
49server error log.
50
51 use CGI::Carp;
52 warn "This is a complaint";
53 die "But this one is serious";
54
55The following use of CGI::Carp also redirects errors to a file of your choice,
56placed in a BEGIN block to catch compile-time warnings as well:
57
58 BEGIN {
59 use CGI::Carp qw(carpout);
60 open(LOG, ">>/var/local/cgi-logs/mycgi-log")
61 or die "Unable to append to mycgi-log: $!\n";
62 carpout(*LOG);
63 }
64
65You can even arrange for fatal errors to go back to the client browser,
66which is nice for your own debugging, but might confuse the end user.
67
68 use CGI::Carp qw(fatalsToBrowser);
69 die "Bad error here";
70
71Even if the error happens before you get the HTTP header out, the module
72will try to take care of this to avoid the dreaded server 500 errors.
73Normal warnings still go out to the server error log (or wherever
74you've sent them with C<carpout>) with the application name and date
75stamp prepended.
76
68dc0745
PP
77=head2 How do I remove HTML from a string?
78
79The most correct way (albeit not the fastest) is to use HTML::Parse
65acb1b1 80from CPAN (part of the HTML-Tree package on CPAN).
68dc0745
PP
81
82Many folks attempt a simple-minded regular expression approach, like
83C<s/E<lt>.*?E<gt>//g>, but that fails in many cases because the tags
84may continue over line breaks, they may contain quoted angle-brackets,
85or HTML comment may be present. Plus folks forget to convert
86entities, like C<&lt;> for example.
87
88Here's one "simple-minded" approach, that works for most files:
89
90 #!/usr/bin/perl -p0777
91 s/<(?:[^>'"]*|(['"]).*?\1)*>//gs
92
93If you want a more complete solution, see the 3-stage striphtml
94program in
95http://www.perl.com/CPAN/authors/Tom_Christiansen/scripts/striphtml.gz
96.
97
c8db1d39
TC
98Here are some tricky cases that you should think about when picking
99a solution:
100
101 <IMG SRC = "foo.gif" ALT = "A > B">
102
103 <IMG SRC = "foo.gif"
104 ALT = "A > B">
105
106 <!-- <A comment> -->
107
108 <script>if (a<b && a>c)</script>
109
110 <# Just data #>
111
112 <![INCLUDE CDATA [ >>>>>>>>>>>> ]]>
113
114If HTML comments include other tags, those solutions would also break
115on text like this:
116
117 <!-- This section commented out.
118 <B>You can't see me!</B>
119 -->
120
68dc0745
PP
121=head2 How do I extract URLs?
122
54310121 123A quick but imperfect approach is
68dc0745
PP
124
125 #!/usr/bin/perl -n00
126 # qxurl - tchrist@perl.com
127 print "$2\n" while m{
128 < \s*
129 A \s+ HREF \s* = \s* (["']) (.*?) \1
130 \s* >
131 }gsix;
132
133This version does not adjust relative URLs, understand alternate
46fc3d4c
PP
134bases, deal with HTML comments, deal with HREF and NAME attributes in
135the same tag, or accept URLs themselves as arguments. It also runs
136about 100x faster than a more "complete" solution using the LWP suite
137of modules, such as the
68dc0745
PP
138http://www.perl.com/CPAN/authors/Tom_Christiansen/scripts/xurl.gz
139program.
140
141=head2 How do I download a file from the user's machine? How do I open a file on another machine?
142
143In the context of an HTML form, you can use what's known as
144B<multipart/form-data> encoding. The CGI.pm module (available from
145CPAN) supports this in the start_multipart_form() method, which isn't
146the same as the startform() method.
147
148=head2 How do I make a pop-up menu in HTML?
149
150Use the B<E<lt>SELECTE<gt>> and B<E<lt>OPTIONE<gt>> tags. The CGI.pm
151module (available from CPAN) supports this widget, as well as many
152others, including some that it cleverly synthesizes on its own.
153
154=head2 How do I fetch an HTML file?
155
46fc3d4c
PP
156One approach, if you have the lynx text-based HTML browser installed
157on your system, is this:
68dc0745
PP
158
159 $html_code = `lynx -source $url`;
160 $text_data = `lynx -dump $url`;
161
46fc3d4c
PP
162The libwww-perl (LWP) modules from CPAN provide a more powerful way to
163do this. They work through proxies, and don't require lynx:
164
c8db1d39
TC
165 # simplest version
166 use LWP::Simple;
167 $content = get($URL);
168
169 # or print HTML from a URL
46fc3d4c
PP
170 use LWP::Simple;
171 getprint "http://www.sn.no/libwww-perl/";
172
c8db1d39 173 # or print ASCII from HTML from a URL
65acb1b1 174 # also need HTML-Tree package from CPAN
46fc3d4c
PP
175 use LWP::Simple;
176 use HTML::Parse;
177 use HTML::FormatText;
178 my ($html, $ascii);
179 $html = get("http://www.perl.com/");
180 defined $html
181 or die "Can't fetch HTML from http://www.perl.com/";
182 $ascii = HTML::FormatText->new->format(parse_html($html));
183 print $ascii;
184
c8db1d39
TC
185=head2 How do I automate an HTML form submission?
186
187If you're submitting values using the GET method, create a URL and encode
188the form using the C<query_form> method:
189
190 use LWP::Simple;
191 use URI::URL;
192
193 my $url = url('http://www.perl.com/cgi-bin/cpan_mod');
194 $url->query_form(module => 'DB_File', readme => 1);
195 $content = get($url);
196
197If you're using the POST method, create your own user agent and encode
198the content appropriately.
199
200 use HTTP::Request::Common qw(POST);
201 use LWP::UserAgent;
202
203 $ua = LWP::UserAgent->new();
204 my $req = POST 'http://www.perl.com/cgi-bin/cpan_mod',
205 [ module => 'DB_File', readme => 1 ];
206 $content = $ua->request($req)->as_string;
207
208=head2 How do I decode or create those %-encodings on the web?
68dc0745
PP
209
210Here's an example of decoding:
211
212 $string = "http://altavista.digital.com/cgi-bin/query?pg=q&what=news&fmt=.&q=%2Bcgi-bin+%2Bperl.exe";
213 $string =~ s/%([a-fA-F0-9]{2})/chr(hex($1))/ge;
214
215Encoding is a bit harder, because you can't just blindly change
c2611fb3 216all the non-alphanumeric characters (C<\W>) into their hex escapes.
68dc0745
PP
217It's important that characters with special meaning like C</> and C<?>
218I<not> be translated. Probably the easiest way to get this right is
219to avoid reinventing the wheel and just use the URI::Escape module,
220which is part of the libwww-perl package (LWP) available from CPAN.
221
222=head2 How do I redirect to another page?
223
224Instead of sending back a C<Content-Type> as the headers of your
225reply, send back a C<Location:> header. Officially this should be a
226C<URI:> header, so the CGI.pm module (available from CPAN) sends back
227both:
228
229 Location: http://www.domain.com/newpage
230 URI: http://www.domain.com/newpage
231
232Note that relative URLs in these headers can cause strange effects
233because of "optimizations" that servers do.
234
c8db1d39
TC
235 $url = "http://www.perl.com/CPAN/";
236 print "Location: $url\n\n";
237 exit;
238
239To be correct to the spec, each of those C<"\n">
240should really each be C<"\015\012">, but unless you're
241stuck on MacOS, you probably won't notice.
242
68dc0745
PP
243=head2 How do I put a password on my web pages?
244
245That depends. You'll need to read the documentation for your web
246server, or perhaps check some of the other FAQs referenced above.
247
248=head2 How do I edit my .htpasswd and .htgroup files with Perl?
249
250The HTTPD::UserAdmin and HTTPD::GroupAdmin modules provide a
251consistent OO interface to these files, regardless of how they're
46fc3d4c 252stored. Databases may be text, dbm, Berkley DB or any database with a
68dc0745
PP
253DBI compatible driver. HTTPD::UserAdmin supports files used by the
254`Basic' and `Digest' authentication schemes. Here's an example:
255
256 use HTTPD::UserAdmin ();
257 HTTPD::UserAdmin
258 ->new(DB => "/foo/.htpasswd")
259 ->add($username => $password);
260
46fc3d4c
PP
261=head2 How do I make sure users can't enter values into a form that cause my CGI script to do bad things?
262
263Read the CGI security FAQ, at
264http://www-genome.wi.mit.edu/WWW/faqs/www-security-faq.html, and the
265Perl/CGI FAQ at
266http://www.perl.com/CPAN/doc/FAQs/cgi/perl-cgi-faq.html.
267
268In brief: use tainting (see L<perlsec>), which makes sure that data
269from outside your script (eg, CGI parameters) are never used in
270C<eval> or C<system> calls. In addition to tainting, never use the
271single-argument form of system() or exec(). Instead, supply the
272command and arguments as a list, which prevents shell globbing.
273
5a964f20 274=head2 How do I parse a mail header?
68dc0745
PP
275
276For a quick-and-dirty solution, try this solution derived
277from page 222 of the 2nd edition of "Programming Perl":
278
279 $/ = '';
280 $header = <MSG>;
281 $header =~ s/\n\s+/ /g; # merge continuation lines
282 %head = ( UNIX_FROM_LINE, split /^([-\w]+):\s*/m, $header );
283
284That solution doesn't do well if, for example, you're trying to
285maintain all the Received lines. A more complete approach is to use
286the Mail::Header module from CPAN (part of the MailTools package).
287
288=head2 How do I decode a CGI form?
289
c8db1d39
TC
290You use a standard module, probably CGI.pm. Under no circumstances
291should you attempt to do so by hand!
292
293You'll see a lot of CGI programs that blindly read from STDIN the number
294of bytes equal to CONTENT_LENGTH for POSTs, or grab QUERY_STRING for
295decoding GETs. These programs are very poorly written. They only work
296sometimes. They typically forget to check the return value of the read()
297system call, which is a cardinal sin. They don't handle HEAD requests.
298They don't handle multipart forms used for file uploads. They don't deal
299with GET/POST combinations where query fields are in more than one place.
300They don't deal with keywords in the query string.
301
302In short, they're bad hacks. Resist them at all costs. Please do not be
303tempted to reinvent the wheel. Instead, use the CGI.pm or CGI_Lite.pm
304(available from CPAN), or if you're trapped in the module-free land
305of perl1 .. perl4, you might look into cgi-lib.pl (available from
65acb1b1 306http://cgi-lib.stanford.edu/cgi-lib/ ).
c8db1d39
TC
307
308Make sure you know whether to use a GET or a POST in your form.
309GETs should only be used for something that doesn't update the server.
310Otherwise you can get mangled databases and repeated feedback mail
311messages. The fancy word for this is ``idempotency''. This simply
312means that there should be no difference between making a GET request
313for a particular URL once or multiple times. This is because the
314HTTP protocol definition says that a GET request may be cached by the
315browser, or server, or an intervening proxy. POST requests cannot be
316cached, because each request is independent and matters. Typically,
317POST requests change or depend on state on the server (query or update
318a database, send mail, or purchase a computer).
68dc0745 319
5a964f20 320=head2 How do I check a valid mail address?
68dc0745 321
c8db1d39 322You can't, at least, not in real time. Bummer, eh?
68dc0745 323
c8db1d39
TC
324Without sending mail to the address and seeing whether there's a human
325on the other hand to answer you, you cannot determine whether a mail
326address is valid. Even if you apply the mail header standard, you
327can have problems, because there are deliverable addresses that aren't
328RFC-822 (the mail header standard) compliant, and addresses that aren't
329deliverable which are compliant.
68dc0745 330
c8db1d39
TC
331Many are tempted to try to eliminate many frequently-invalid
332mail addresses with a simple regexp, such as
333C</^[\w.-]+\@([\w.-]\.)+\w+$/>. It's a very bad idea. However,
334this also throws out many valid ones, and says nothing about
335potential deliverability, so is not suggested. Instead, see
68dc0745
PP
336http://www.perl.com/CPAN/authors/Tom_Christiansen/scripts/ckaddr.gz ,
337which actually checks against the full RFC spec (except for nested
5a964f20 338comments), looks for addresses you may not wish to accept mail to
68dc0745 339(say, Bill Clinton or your postmaster), and then makes sure that the
c8db1d39
TC
340hostname given can be looked up in the DNS MX records. It's not fast,
341but it works for what it tries to do.
342
343Our best advice for verifying a person's mail address is to have them
344enter their address twice, just as you normally do to change a password.
345This usually weeds out typos. If both versions match, send
346mail to that address with a personal message that looks somewhat like:
347
348 Dear someuser@host.com,
349
350 Please confirm the mail address you gave us Wed May 6 09:38:41
351 MDT 1998 by replying to this message. Include the string
352 "Rumpelstiltskin" in that reply, but spelled in reverse; that is,
353 start with "Nik...". Once this is done, your confirmed address will
354 be entered into our records.
355
356If you get the message back and they've followed your directions,
357you can be reasonably assured that it's real.
68dc0745 358
c8db1d39
TC
359A related strategy that's less open to forgery is to give them a PIN
360(personal ID number). Record the address and PIN (best that it be a
361random one) for later processing. In the mail you send, ask them to
362include the PIN in their reply. But if it bounces, or the message is
363included via a ``vacation'' script, it'll be there anyway. So it's
364best to ask them to mail back a slight alteration of the PIN, such as
365with the characters reversed, one added or subtracted to each digit, etc.
46fc3d4c 366
68dc0745
PP
367=head2 How do I decode a MIME/BASE64 string?
368
369The MIME-tools package (available from CPAN) handles this and a lot
370more. Decoding BASE64 becomes as simple as:
371
372 use MIME::base64;
373 $decoded = decode_base64($encoded);
374
375A more direct approach is to use the unpack() function's "u"
376format after minor transliterations:
377
378 tr#A-Za-z0-9+/##cd; # remove non-base64 chars
379 tr#A-Za-z0-9+/# -_#; # convert to uuencoded format
380 $len = pack("c", 32 + 0.75*length); # compute length byte
381 print unpack("u", $len . $_); # uudecode and print
382
5a964f20 383=head2 How do I return the user's mail address?
68dc0745
PP
384
385On systems that support getpwuid, the $E<lt> variable and the
386Sys::Hostname module (which is part of the standard perl distribution),
387you can probably try using something like this:
388
389 use Sys::Hostname;
65acb1b1 390 $address = sprintf('%s@%s', getpwuid($<), hostname);
68dc0745 391
5a964f20
TC
392Company policies on mail address can mean that this generates addresses
393that the company's mail system will not accept, so you should ask for
394users' mail addresses when this matters. Furthermore, not all systems
68dc0745
PP
395on which Perl runs are so forthcoming with this information as is Unix.
396
397The Mail::Util module from CPAN (part of the MailTools package) provides a
398mailaddress() function that tries to guess the mail address of the user.
399It makes a more intelligent guess than the code above, using information
400given when the module was installed, but it could still be incorrect.
401Again, the best way is often just to ask the user.
402
c8db1d39 403=head2 How do I send mail?
68dc0745 404
c8db1d39
TC
405Use the C<sendmail> program directly:
406
407 open(SENDMAIL, "|/usr/lib/sendmail -oi -t -odq")
408 or die "Can't fork for sendmail: $!\n";
409 print SENDMAIL <<"EOF";
410 From: User Originating Mail <me\@host>
411 To: Final Destination <you\@otherhost>
412 Subject: A relevant subject line
413
65acb1b1
TC
414 Body of the message goes here after the blank line
415 in as many lines as you like.
c8db1d39
TC
416 EOF
417 close(SENDMAIL) or warn "sendmail didn't close nicely";
418
419The B<-oi> option prevents sendmail from interpreting a line consisting
420of a single dot as "end of message". The B<-t> option says to use the
421headers to decide who to send the message to, and B<-odq> says to put
422the message into the queue. This last option means your message won't
423be immediately delivered, so leave it out if you want immediate
424delivery.
425
426Or use the CPAN module Mail::Mailer:
427
428 use Mail::Mailer;
429
430 $mailer = Mail::Mailer->new();
431 $mailer->open({ From => $from_address,
432 To => $to_address,
433 Subject => $subject,
434 })
435 or die "Can't open: $!\n";
436 print $mailer $body;
437 $mailer->close();
438
439The Mail::Internet module uses Net::SMTP which is less Unix-centric than
440Mail::Mailer, but less reliable. Avoid raw SMTP commands. There
441are many reasons to use a mail transport agent like sendmail. These
442include queueing, MX records, and security.
443
444=head2 How do I read mail?
445
65acb1b1
TC
446Use the Mail::Folder module from CPAN (part of the MailFolder package) or
447the Mail::Internet module from CPAN (also part of the MailTools package).
68dc0745 448
3fe9a6f1
PP
449 # sending mail
450 use Mail::Internet;
451 use Mail::Header;
452 # say which mail host to use
453 $ENV{SMTPHOSTS} = 'mail.frii.com';
454 # create headers
455 $header = new Mail::Header;
456 $header->add('From', 'gnat@frii.com');
457 $header->add('Subject', 'Testing');
458 $header->add('To', 'gnat@frii.com');
459 # create body
460 $body = 'This is a test, ignore';
461 # create mail object
462 $mail = new Mail::Internet(undef, Header => $header, Body => \[$body]);
463 # send it
464 $mail->smtpsend or die;
465
c8db1d39
TC
466Often a module is overkill, though. Here's a mail sorter.
467
468 #!/usr/bin/perl
469 # bysub1 - simple sort by subject
470 my(@msgs, @sub);
471 my $msgno = -1;
472 $/ = ''; # paragraph reads
473 while (<>) {
474 if (/^From/m) {
475 /^Subject:\s*(?:Re:\s*)*(.*)/mi;
476 $sub[++$msgno] = lc($1) || '';
477 }
478 $msgs[$msgno] .= $_;
479 }
480 for my $i (sort { $sub[$a] cmp $sub[$b] || $a <=> $b } (0 .. $#msgs)) {
481 print $msgs[$i];
482 }
483
484Or more succinctly,
485
486 #!/usr/bin/perl -n00
487 # bysub2 - awkish sort-by-subject
488 BEGIN { $msgno = -1 }
489 $sub[++$msgno] = (/^Subject:\s*(?:Re:\s*)*(.*)/mi)[0] if /^From/m;
490 $msg[$msgno] .= $_;
491 END { print @msg[ sort { $sub[$a] cmp $sub[$b] || $a <=> $b } (0 .. $#msg) ] }
492
68dc0745
PP
493=head2 How do I find out my hostname/domainname/IP address?
494
c8db1d39
TC
495The normal way to find your own hostname is to call the C<`hostname`>
496program. While sometimes expedient, this has some problems, such as
497not knowing whether you've got the canonical name or not. It's one of
498those tradeoffs of convenience versus portability.
68dc0745
PP
499
500The Sys::Hostname module (part of the standard perl distribution) will
501give you the hostname after which you can find out the IP address
502(assuming you have working DNS) with a gethostbyname() call.
503
504 use Socket;
505 use Sys::Hostname;
506 my $host = hostname();
65acb1b1 507 my $addr = inet_ntoa(scalar gethostbyname($host || 'localhost'));
68dc0745
PP
508
509Probably the simplest way to learn your DNS domain name is to grok
510it out of /etc/resolv.conf, at least under Unix. Of course, this
511assumes several things about your resolv.conf configuration, including
512that it exists.
513
514(We still need a good DNS domain name-learning method for non-Unix
515systems.)
516
517=head2 How do I fetch a news article or the active newsgroups?
518
519Use the Net::NNTP or News::NNTPClient modules, both available from CPAN.
520This can make tasks like fetching the newsgroup list as simple as:
521
522 perl -MNews::NNTPClient
523 -e 'print News::NNTPClient->new->list("newsgroups")'
524
525=head2 How do I fetch/put an FTP file?
526
527LWP::Simple (available from CPAN) can fetch but not put. Net::FTP (also
528available from CPAN) is more complex but can put as well as fetch.
529
530=head2 How can I do RPC in Perl?
531
532A DCE::RPC module is being developed (but is not yet available), and
533will be released as part of the DCE-Perl package (available from
65acb1b1
TC
534CPAN). The rpcgen suite, available from CPAN/authors/id/JAKE/, is
535an RPC stub generator and includes an RPC::ONC module.
68dc0745
PP
536
537=head1 AUTHOR AND COPYRIGHT
538
65acb1b1 539Copyright (c) 1997-1999 Tom Christiansen and Nathan Torkington.
5a964f20
TC
540All rights reserved.
541
542When included as part of the Standard Version of Perl, or as part of
543its complete documentation whether printed or otherwise, this work
c2611fb3 544may be distributed only under the terms of Perl's Artistic Licence.
5a964f20
TC
545Any distribution of this file or derivatives thereof I<outside>
546of that package require that special arrangements be made with
547copyright holder.
548
549Irrespective of its distribution, all code examples in this file
550are hereby placed into the public domain. You are permitted and
551encouraged to use this code in your own programs for fun
552or for profit as you see fit. A simple comment in the code giving
553credit would be courteous but is not required.
65acb1b1 554