1N/Aperlfaq9 - Networking ($Revision: 1.15 $, $Date: 2003/01/31 17:36:57 $)
1N/AThis section deals with questions related to networking, the internet,
1N/Aand a few on the web.
1N/A=head2 What is the correct form of response from a CGI script?
1N/A(Alan Flavell <flavell+www@a5.ph.gla.ac.uk> answers...)
1N/AThe Common Gateway Interface (CGI) specifies a software interface between
1N/Aa program ("CGI script") and a web server (HTTPD). It is not specific
1N/Ato Perl, and has its own FAQs and tutorials, and usenet group,
1N/AThese Perl FAQs very selectively cover some CGI issues. However, Perl
1N/Aprogrammers are strongly advised to use the
CGI.pm module, to take care
1N/Aof the details for them.
1N/AThe similarity between CGI response headers (defined in the CGI
1N/Aspecification) and HTTP response headers (defined in the HTTP
1N/Aspecification, RFC2616) is intentional, but can sometimes be confusing.
1N/AThe CGI specification defines two kinds of script: the "Parsed Header"
1N/Ascript, and the "Non Parsed Header" (NPH) script. Check your server
1N/Adocumentation to see what it supports. "Parsed Header" scripts are
1N/Asimpler in various respects. The CGI specification allows any of the
1N/Ausual newline representations in the CGI response (it's the server's
1N/Ajob to create an accurate HTTP response based on it). So "\n" written in
1N/Atext mode is technically correct, and recommended. NPH scripts are more
1N/Atricky: they must put out a complete and accurate set of HTTP
1N/Atransaction response headers; the HTTP specification calls for records
1N/Ato be terminated with carriage-return and line-feed,
i.e ASCII \015\012
1N/Awritten in binary mode.
1N/AUsing
CGI.pm gives excellent platform independence, including EBCDIC
1N/Asystems.
CGI.pm selects an appropriate newline representation
1N/A($CGI::CRLF) and sets binmode as appropriate.
1N/A=head2 My CGI script runs from the command line but not the browser. (500 Server Error)
1N/ASeveral things could be wrong. You can go through the "Troubleshooting
1N/APerl CGI scripts" guide at
1N/AIf, after that, you can demonstrate that you've read the FAQs and that
1N/Ayour problem isn't something simple that can be easily answered, you'll
1N/Aprobably receive a courteous and useful reply to your question if you
1N/Awith HTTP or the CGI protocols). Questions that appear to be Perl
1N/Aare not so well received.
1N/AThe useful FAQs, related documents, and troubleshooting guides are
1N/Alisted in the CGI Meta FAQ:
1N/A=head2 How can I get better error messages from a CGI program?
1N/AUse the CGI::Carp module. It replaces C<warn> and C<die>, plus the
1N/Anormal Carp modules C<carp>, C<croak>, and C<confess> functions with
1N/Amore verbose and safer versions. It still sends them to the normal
1N/A warn "This is a complaint";
1N/A die "But this one is serious";
1N/AThe following use of CGI::Carp also redirects errors to a file of your choice,
1N/Aplaced in a BEGIN block to catch compile-time warnings as well:
1N/A use CGI::Carp qw(carpout);
1N/A or die "Unable to append to mycgi-log: $!\n";
1N/AYou can even arrange for fatal errors to go back to the client browser,
1N/Awhich is nice for your own debugging, but might confuse the end user.
1N/A use CGI::Carp qw(fatalsToBrowser);
1N/A die "Bad error here";
1N/AEven if the error happens before you get the HTTP header out, the module
1N/Awill try to take care of this to avoid the dreaded server 500 errors.
1N/ANormal warnings still go out to the server error log (or wherever
1N/Ayou've sent them with C<carpout>) with the application name and date
1N/A=head2 How do I remove HTML from a string?
1N/AThe most correct way (albeit not the fastest) is to use HTML::Parser
1N/Afrom CPAN. Another mostly correct
1N/Away is to use HTML::FormatText which not only removes HTML but also
1N/Aattempts to do a little simple formatting of the resulting plain text.
1N/AMany folks attempt a simple-minded regular expression approach, like
1N/AC<< s/<.*?>//g >>, but that fails in many cases because the tags
1N/Amay continue over line breaks, they may contain quoted angle-brackets,
1N/Aor HTML comment may be present. Plus, folks forget to convert
1N/Aentities--like C<<> for example.
1N/AHere's one "simple-minded" approach, that works for most files:
1N/A s/<(?:[^>'"]*|(['"]).*?\1)*>//gs
1N/AIf you want a more complete solution, see the 3-stage striphtml
1N/AHere are some tricky cases that you should think about when picking
1N/A <!-- <A comment> -->
1N/A <script>if (a<b && a>c)</script>
1N/A <![INCLUDE CDATA [ >>>>>>>>>>>> ]]>
1N/AIf HTML comments include other tags, those solutions would also break
1N/A <!-- This section commented out.
1N/A <B>You can't see me!</B>
1N/A=head2 How do I extract URLs?
1N/AYou can easily extract all sorts of URLs from HTML with
1N/AC<HTML::SimpleLinkExtor> which handles anchors, images, objects,
1N/Aframes, and many other tags that can contain a URL. If you need
1N/Aanything more complex, you can create your own subclass of
1N/AC<HTML::LinkExtor> or C<HTML::Parser>. You might even use
1N/AC<HTML::SimpleLinkExtor> as an example for something specifically
1N/Asuited to your needs.
1N/AYou can use URI::Find to extract URLs from an arbitrary text document.
1N/ALess complete solutions involving regular expressions can save
1N/Ayou a lot of processing time if you know that the input is simple. One
1N/Asolution from Tom Christiansen runs 100 times faster than most
1N/Amodule based approaches but only extracts URLs from anchors where the first
1N/Aattribute is HREF and there are no other attributes.
1N/A # qxurl - tchrist@perl.com
1N/A print "$2\n" while m{
1N/A A \s+ HREF \s* = \s* (["']) (.*?) \1
1N/A=head2 How do I download a file from the user's machine? How do I open a file on another machine?
1N/AIn this case, download means to use the file upload feature of HTML
1N/Aforms. You allow the web surfer to specify a file to send to your web
1N/Aserver. To you it looks like a download, and to the user it looks
1N/Alike an upload. No matter what you call it, you do it with what's
1N/Acomes with Perl as part of the Standard Library) supports this in the
1N/Astart_multipart_form() method, which isn't the same as the startform()
1N/ASee the section in the
CGI.pm documentation on file uploads for code
1N/Aexamples and details.
1N/A=head2 How do I make a pop-up menu in HTML?
1N/AUse the B<< <SELECT> >> and B<< <OPTION> >> tags. The
CGI.pm 1N/Amodule (available from CPAN) supports this widget, as well as many
1N/Aothers, including some that it cleverly synthesizes on its own.
1N/A=head2 How do I fetch an HTML file?
1N/AOne approach, if you have the lynx text-based HTML browser installed
1N/Aon your system, is this:
1N/A $html_code = `lynx -source $url`;
1N/A $text_data = `lynx -dump $url`;
1N/AThe libwww-perl (LWP) modules from CPAN provide a more powerful way
1N/Ato do this. They don't require lynx, but like lynx, can still work
1N/A $content = get($URL);
1N/A # or print HTML from a URL
1N/A # or print ASCII from HTML from a URL
1N/A # also need HTML-Tree package from CPAN
1N/A use HTML::FormatText;
1N/A $ascii = HTML::FormatText->new->format(parse_html($html));
1N/A=head2 How do I automate an HTML form submission?
1N/AIf you're submitting values using the GET method, create a URL and encode
1N/Athe form using the C<query_form> method:
1N/A $url->query_form(module => 'DB_File', readme => 1);
1N/A $content = get($url);
1N/AIf you're using the POST method, create your own user agent and encode
1N/Athe content appropriately.
1N/A use HTTP::Request::Common qw(POST);
1N/A $ua = LWP::UserAgent->new();
1N/A [ module => 'DB_File', readme => 1 ];
1N/A $content = $ua->request($req)->as_string;
1N/A=head2 How do I decode or create those %-encodings on the web?
1N/AIf you are writing a CGI script, you should be using the
CGI.pm module
1N/Athat comes with perl, or some other equivalent module. The CGI module
1N/Aautomatically decodes queries for you, and provides an escape()
1N/Afunction to handle encoding.
1N/AThe best source of detailed information on URI encoding is RFC 2396.
1N/ABasically, the following substitutions do it:
1N/A s/([^\w()'*~!.-])/sprintf '%%%02x', ord $1/eg; # encode
1N/A s/%([A-Fa-f\d]{2})/chr hex $1/eg; # decode
1N/AHowever, you should only apply them to individual URI components, not
1N/Athe entire URI, otherwise you'll lose information and generally mess
1N/Athings up. If that didn't explain it, don't worry. Just go read
1N/Asection 2 of the RFC, it's probably the best explanation there is.
1N/ARFC 2396 also contains a lot of other useful information, including a
1N/Aregexp for breaking any arbitrary URI into components (Appendix B).
1N/A=head2 How do I redirect to another page?
1N/ASpecify the complete URL of the destination (even if it is on the same
1N/Aserver). This is one of the two different kinds of CGI "Location:"
1N/Aresponses which are defined in the CGI specification for a Parsed Headers
1N/Ascript. The other kind (an absolute URLpath) is resolved internally to
1N/Athe server without any HTTP redirection. The CGI specifications do not
1N/Aallow relative URLs in either case.
1N/AUse of
CGI.pm is strongly recommended. This example shows redirection
1N/Awith a complete URL. This redirection is handled by the web browser.
1N/A use CGI qw/:standard/;
1N/A print redirect($url);
1N/AThis example shows a redirection with an absolute URLpath. This
1N/Aredirection is handled by the local web server.
1N/A print redirect($url);
1N/ABut if coded directly, it could be as follows (the final "\n" is
1N/Ashown separately, for clarity), using either a complete URL or
1N/A print "Location: $url\n"; # CGI response header
1N/A print "\n"; # end of headers
1N/A=head2 How do I put a password on my web pages?
1N/ATo enable authentication for your web server, you need to configure
1N/Ayour web server. The configuration is different for different sorts
1N/Aof web servers---apache does it differently from iPlanet which does
1N/Ait differently from IIS. Check your web server documentation for
1N/Athe details for your particular server.
1N/A=head2 How do I edit my .htpasswd and .htgroup files with Perl?
1N/AThe HTTPD::UserAdmin and HTTPD::GroupAdmin modules provide a
1N/Aconsistent OO interface to these files, regardless of how they're
1N/Astored. Databases may be text, dbm, Berkeley DB or any database with
1N/Aa DBI compatible driver. HTTPD::UserAdmin supports files used by the
1N/A`Basic' and `Digest' authentication schemes. Here's an example:
1N/A use HTTPD::UserAdmin ();
1N/A ->new(DB => "/foo/.htpasswd")
1N/A ->add($username => $password);
1N/A=head2 How do I make sure users can't enter values into a form that cause my CGI script to do bad things?
1N/ASee the security references listed in the CGI Meta FAQ
1N/A=head2 How do I parse a mail header?
1N/AFor a quick-and-dirty solution, try this solution derived
1N/A $header =~ s/\n\s+/ /g; # merge continuation lines
1N/A %head = ( UNIX_FROM_LINE, split /^([-\w]+):\s*/m, $header );
1N/AThat solution doesn't do well if, for example, you're trying to
1N/Amaintain all the Received lines. A more complete approach is to use
1N/Athe Mail::Header module from CPAN (part of the MailTools package).
1N/A=head2 How do I decode a CGI form?
1N/AYou use a standard module, probably
CGI.pm. Under no circumstances
1N/Ashould you attempt to do so by hand!
1N/AYou'll see a lot of CGI programs that blindly read from STDIN the number
1N/Aof bytes equal to CONTENT_LENGTH for POSTs, or grab QUERY_STRING for
1N/Adecoding GETs. These programs are very poorly written. They only work
1N/Asometimes. They typically forget to check the return value of the read()
1N/Asystem call, which is a cardinal sin. They don't handle HEAD requests.
1N/AThey don't handle multipart forms used for file uploads. They don't deal
1N/Awith
GET/POST combinations where query fields are in more than one place.
1N/AThey don't deal with keywords in the query string.
1N/AIn short, they're bad hacks. Resist them at all costs. Please do not be
1N/A(available from CPAN), or if you're trapped in the module-free land
1N/AMake sure you know whether to use a GET or a POST in your form.
1N/AGETs should only be used for something that doesn't update the server.
1N/AOtherwise you can get mangled databases and repeated feedback mail
1N/Amessages. The fancy word for this is ``idempotency''. This simply
1N/Ameans that there should be no difference between making a GET request
1N/Afor a particular URL once or multiple times. This is because the
1N/AHTTP protocol definition says that a GET request may be cached by the
1N/Abrowser, or server, or an intervening proxy. POST requests cannot be
1N/Acached, because each request is independent and matters. Typically,
1N/APOST requests change or depend on state on the server (query or update
1N/Aa database, send mail, or purchase a computer).
1N/A=head2 How do I check a valid mail address?
1N/AYou can't, at least, not in real time. Bummer, eh?
1N/AWithout sending mail to the address and seeing whether there's a human
1N/Aon the other hand to answer you, you cannot determine whether a mail
1N/Aaddress is valid. Even if you apply the mail header standard, you
1N/Acan have problems, because there are deliverable addresses that aren't
1N/ARFC-822 (the mail header standard) compliant, and addresses that aren't
1N/Adeliverable which are compliant.
1N/AYou can use the Email::Valid or RFC::RFC822::Address which check
1N/Athe format of the address, although they cannot actually tell you
1N/Aif it is a deliverable address (
i.e. that mail to the address
1N/Awill not bounce). Modules like Mail::CheckUser and Mail::EXPN
1N/Atry to interact with the domain name system or particular
1N/Amail servers to learn even more, but their methods do not
1N/Awork everywhere---especially for security conscious administrators.
1N/AMany are tempted to try to eliminate many frequently-invalid
1N/Amail addresses with a simple regex, such as
1N/AC</^[\w.-]+\@(?:[\w-]+\.)+\w+$/>. It's a very bad idea. However,
1N/Athis also throws out many valid ones, and says nothing about
1N/Apotential deliverability, so it is not suggested. Instead, see
1N/Awhich actually checks against the full RFC spec (except for nested
1N/Acomments), looks for addresses you may not wish to accept mail to
1N/A(say, Bill Clinton or your postmaster), and then makes sure that the
1N/Ahostname given can be looked up in the DNS MX records. It's not fast,
1N/Abut it works for what it tries to do.
1N/AOur best advice for verifying a person's mail address is to have them
1N/Aenter their address twice, just as you normally do to change a password.
1N/AThis usually weeds out typos. If both versions match, send
1N/Amail to that address with a personal message that looks somewhat like:
1N/A Dear someuser@host.com,
1N/A Please confirm the mail address you gave us Wed May 6 09:38:41
1N/A MDT 1998 by replying to this message. Include the string
1N/A "Rumpelstiltskin" in that reply, but spelled in reverse; that is,
1N/A start with "Nik...". Once this is done, your confirmed address will
1N/A be entered into our records.
1N/AIf you get the message back and they've followed your directions,
1N/Ayou can be reasonably assured that it's real.
1N/AA related strategy that's less open to forgery is to give them a PIN
1N/A(personal ID number). Record the address and PIN (best that it be a
1N/Arandom one) for later processing. In the mail you send, ask them to
1N/Ainclude the PIN in their reply. But if it bounces, or the message is
1N/Aincluded via a ``vacation'' script, it'll be there anyway. So it's
1N/Abest to ask them to mail back a slight alteration of the PIN, such as
1N/Awith the characters reversed, one added or subtracted to each digit, etc.
1N/AThe MIME-Base64 package (available from CPAN) handles this as well as
1N/Athe
MIME/QP encoding. Decoding BASE64 becomes as simple as:
1N/A $decoded = decode_base64($encoded);
1N/AThe MIME-Tools package (available from CPAN) supports extraction with
1N/Adecoding of BASE64 encoded attachments and content directly from email
1N/AIf the string to decode is short (less than 84 bytes long)
1N/Aa more direct approach is to use the unpack() function's "u"
1N/Aformat after minor transliterations:
1N/A tr#A-Za-z0-9+/##cd; # remove non-base64 chars
1N/A tr#A-Za-z0-9+/# -_#; # convert to uuencoded format
1N/A $len = pack("c", 32 + 0.75*length); # compute length byte
1N/A print unpack("u", $len . $_); # uudecode and print
1N/A=head2 How do I return the user's mail address?
1N/AOn systems that support getpwuid, the $< variable, and the
1N/ASys::Hostname module (which is part of the standard perl distribution),
1N/Ayou can probably try using something like this:
1N/A $address = sprintf('%s@%s', scalar getpwuid($<), hostname);
1N/ACompany policies on mail address can mean that this generates addresses
1N/Athat the company's mail system will not accept, so you should ask for
1N/Ausers' mail addresses when this matters. Furthermore, not all systems
1N/Aon which Perl runs are so forthcoming with this information as is Unix.
1N/AThe Mail::Util module from CPAN (part of the MailTools package) provides a
1N/Amailaddress() function that tries to guess the mail address of the user.
1N/AIt makes a more intelligent guess than the code above, using information
1N/Agiven when the module was installed, but it could still be incorrect.
1N/AAgain, the best way is often just to ask the user.
1N/A=head2 How do I send mail?
1N/AUse the C<sendmail> program directly:
1N/A or die "Can't fork for sendmail: $!\n";
1N/A print SENDMAIL <<"EOF";
1N/A From: User Originating Mail <me\@host>
1N/A To: Final Destination <you\@otherhost>
1N/A Subject: A relevant subject line
1N/A Body of the message goes here after the blank line
1N/A in as many lines as you like.
1N/A close(SENDMAIL) or warn "sendmail didn't close nicely";
1N/AThe B<-oi> option prevents sendmail from interpreting a line consisting
1N/Aof a single dot as "end of message". The B<-t> option says to use the
1N/Aheaders to decide who to send the message to, and B<-odq> says to put
1N/Athe message into the queue. This last option means your message won't
1N/Abe immediately delivered, so leave it out if you want immediate
1N/AAlternate, less convenient approaches include calling mail (sometimes
1N/Acalled mailx) directly or simply opening up port 25 have having an
1N/Aintimate conversation between just you and the remote SMTP daemon,
1N/AOr you might be able use the CPAN module Mail::Mailer:
1N/A $mailer = Mail::Mailer->new();
1N/A $mailer->open({ From => $from_address,
1N/A Subject => $subject,
1N/A or die "Can't open: $!\n";
1N/A print $mailer $body;
1N/AThe Mail::Internet module uses Net::SMTP which is less Unix-centric than
1N/AMail::Mailer, but less reliable. Avoid raw SMTP commands. There
1N/Aare many reasons to use a mail transport agent like sendmail. These
1N/Ainclude queuing, MX records, and security.
1N/A=head2 How do I use MIME to make an attachment to a mail message?
1N/AThis answer is extracted directly from the MIME::Lite documentation.
1N/ACreate a multipart message (
i.e., one with attachments).
1N/A ### Create a new multipart message:
1N/A $msg = MIME::Lite->new(
1N/A From =>'me@myhost.com',
1N/A To =>'you@yourhost.com',
1N/A Cc =>'some@other.com, some@more.com',
1N/A Subject =>'A message with 2 parts...',
1N/A ### Add parts (each "attach" has same arguments as "new"):
1N/A $msg->attach(Type =>'TEXT',
1N/A Data =>"Here's the GIF file you wanted"
1N/A $text = $msg->as_string;
1N/AMIME::Lite also includes a method for sending these things.
1N/AThis defaults to using L<sendmail> but can be customized to use
1N/ASMTP via L<Net::SMTP>.
1N/A=head2 How do I read mail?
1N/AWhile you could use the Mail::Folder module from CPAN (part of the
1N/AMailFolder package) or the Mail::Internet module from CPAN (part
1N/Aof the MailTools package), often a module is overkill. Here's a
1N/A $/ = ''; # paragraph reads
1N/A /^Subject:\s*(?:Re:\s*)*(.*)/mi;
1N/A $sub[++$msgno] = lc($1) || '';
1N/A $msgs[$msgno] .= $_;
1N/A for my $i (sort { $sub[$a] cmp $sub[$b] || $a <=> $b } (0 .. $#msgs)) {
1N/A # bysub2 - awkish sort-by-subject
1N/A BEGIN { $msgno = -1 }
1N/A $sub[++$msgno] = (/^Subject:\s*(?:Re:\s*)*(.*)/mi)[0] if /^From/m;
1N/A END { print @msg[ sort { $sub[$a] cmp $sub[$b] || $a <=> $b } (0 .. $#msg) ] }
1N/AThe normal way to find your own hostname is to call the C<`hostname`>
1N/Aprogram. While sometimes expedient, this has some problems, such as
1N/Anot knowing whether you've got the canonical name or not. It's one of
1N/Athose tradeoffs of convenience versus portability.
1N/AThe Sys::Hostname module (part of the standard perl distribution) will
1N/Agive you the hostname after which you can find out the IP address
1N/A(assuming you have working DNS) with a gethostbyname() call.
1N/A my $host = hostname();
1N/A my $addr = inet_ntoa(scalar gethostbyname($host || 'localhost'));
1N/AProbably the simplest way to learn your DNS domain name is to grok
1N/A(We still need a good DNS domain name-learning method for non-Unix
1N/A=head2 How do I fetch a news article or the active newsgroups?
1N/AUse the Net::NNTP or News::NNTPClient modules, both available from CPAN.
1N/AThis can make tasks like fetching the newsgroup list as simple as
1N/A perl -MNews::NNTPClient
1N/A -e 'print News::NNTPClient->new->list("newsgroups")'
1N/ALWP::Simple (available from CPAN) can fetch but not put. Net::FTP (also
1N/Aavailable from CPAN) is more complex but can put as well as fetch.
1N/A=head2 How can I do RPC in Perl?
1N/AA DCE::RPC module is being developed (but is not yet available) and
1N/Awill be released as part of the DCE-Perl package (available from
1N/Aan RPC stub generator and includes an RPC::ONC module.
1N/A=head1 AUTHOR AND COPYRIGHT
1N/ACopyright (c) 1997-2002 Tom Christiansen and Nathan Torkington.
1N/AThis documentation is free; you can redistribute it
and/or modify it
1N/Aunder the same terms as Perl itself.
1N/AIrrespective of its distribution, all code examples in this file
1N/Aare hereby placed into the public domain. You are permitted and
1N/Aencouraged to use this code in your own programs for fun
1N/Aor for profit as you see fit. A simple comment in the code giving
1N/Acredit would be courteous but is not required.
