1N/Aperlfilter - Source Filters
1N/AThis article is about a little-known feature of Perl called
1N/AI<source filters>. Source filters alter the program text of a module
1N/Abefore Perl sees it, much as a C preprocessor alters the source text of
1N/Aa C program before the compiler sees it. This article tells you more
1N/Aabout what source filters are, how they work, and how to write your
1N/AThe original purpose of source filters was to let you encrypt your
1N/Aprogram source to prevent casual piracy. This isn't all they can do, as
1N/Ayou'll soon learn. But first, the basics.
1N/ABefore the Perl interpreter can execute a Perl script, it must first
1N/Aread it from a file into memory for parsing and compilation. If that
1N/Ascript itself includes other scripts with a C<use> or C<require>
1N/Astatement, then each of those scripts will have to be read from their
1N/Arespective files as well.
1N/ANow think of each logical connection between the Perl parser and an
1N/Aindividual file as a I<source stream>. A source stream is created when
1N/Athe Perl parser opens a file, it continues to exist as the source code
1N/Ais read into memory, and it is destroyed when Perl is finished parsing
1N/Athe file. If the parser encounters a C<require> or C<use> statement in
1N/Aa source stream, a new and distinct stream is created just for that
1N/AThe diagram below represents a single source stream, with the flow of
1N/Asource from a Perl script file on the left into the Perl parser on the
1N/Aright. This is how Perl normally operates.
1N/A file -------> parser
1N/AThere are two important points to remember:
1N/AAlthough there can be any number of source streams in existence at any
1N/Agiven time, only one will be active.
1N/AEvery source stream is associated with only one file.
1N/AA source filter is a special kind of Perl module that intercepts and
1N/Amodifies a source stream before it reaches the parser. A source filter
1N/Achanges our diagram like this:
1N/A file ----> filter ----> parser
1N/AIf that doesn't make much sense, consider the analogy of a command
1N/Apipeline. Say you have a shell script stored in the compressed file
1N/AI<
trial.gz>. The simple pipeline command below runs the script without
1N/Aneeding to create a temporary file to hold the uncompressed file.
1N/AIn this case, the data flow from the pipeline can be represented as follows:
1N/AWith source filters, you can store the text of your script compressed and use a source filter to uncompress it for Perl's parser:
1N/A Perl program ---> source filter ---> parser
1N/ASo how do you use a source filter in a Perl script? Above, I said that
1N/Aa source filter is just a special kind of module. Like all Perl
1N/Amodules, a source filter is invoked with a use statement.
1N/ASay you want to pass your Perl source through the C preprocessor before
1N/Aexecution. You could use the existing C<-P> command line option to do
1N/Athis, but as it happens, the source filters distribution comes with a C
1N/Apreprocessor filter module called Filter::cpp. Let's use that instead.
1N/ABelow is an example program, C<cpp_test>, which makes use of this filter.
1N/ALine numbers have been added to allow specific lines to be referenced
1N/A 1: use Filter::cpp ;
1N/A 4: print "a = $a\n" ;
1N/AWhen you execute this script, Perl creates a source stream for the
1N/Afile. Before the parser processes any of the lines from the file, the
1N/Asource stream looks like this:
1N/A cpp_test ---------> parser
1N/ALine 1, C<use Filter::cpp>, includes and installs the C<cpp> filter
1N/Amodule. All source filters work this way. The use statement is compiled
1N/Aand executed at compile time, before any more of the file is read, and
1N/Ait attaches the cpp filter to the source stream behind the scenes. Now
1N/Athe data flow looks like this:
1N/A cpp_test ----> cpp filter ----> parser
1N/AAs the parser reads the second and subsequent lines from the source
1N/Astream, it feeds those lines through the C<cpp> source filter before
1N/Aprocessing them. The C<cpp> filter simply passes each line through the
1N/Areal C preprocessor. The output from the C preprocessor is then
1N/Ainserted back into the source stream by the filter.
1N/A cpp_test ----> cpp filter ----> parser
1N/AThe parser then sees the following code:
1N/ALet's consider what happens when the filtered code includes another
1N/A 1: use Filter::cpp ;
1N/A 5: print "a = $a\n" ;
1N/AThe C<cpp> filter does not apply to the text of the Fred module, only
1N/Ato the text of the file that used it (C<cpp_test>). Although the use
1N/Astatement on line 3 will pass through the cpp filter, the module that
1N/Agets included (C<Fred>) will not. The source streams look like this
1N/Aafter line 3 has been parsed and before line 4 is parsed:
1N/A cpp_test ---> cpp filter ---> parser (INACTIVE)
1N/AAs you can see, a new stream has been created for reading the source
1N/Ahas been parsed. The source stream for C<cpp_test> will still exist,
1N/Abut is inactive. Once the parser has finished reading
Fred.pm, the
1N/Asource stream associated with it will be destroyed. The source stream
1N/Afor C<cpp_test> then becomes active again and the parser reads line 4
1N/Aand subsequent lines from C<cpp_test>.
1N/AYou can use more than one source filter on a single file. Similarly,
1N/Ayou can reuse the same filter in as many files as you like.
1N/AFor example, if you have a uuencoded and compressed source file, it is
1N/Apossible to stack a uudecode filter and an uncompression filter like
1N/A use Filter::uudecode ; use Filter::uncompress ;
1N/A M'XL(".H<US4''V9I;F%L')Q;>7/;1I;_>_I3=&E=%:F*I"T?22Q/
1N/A M6]9*<IQCO*XFT"0[PL%%'Y+IG?WN^ZYN-$'J.[.JE$,20/?K=_[>
1N/AOnce the first line has been processed, the flow will look like this:
1N/A file ---> uudecode ---> uncompress ---> parser
1N/AData flows through filters in the same order they appear in the source
1N/Afile. The uudecode filter appeared before the uncompress filter, so the
1N/Asource file will be uudecoded before it's uncompressed.
1N/A=head1 WRITING A SOURCE FILTER
1N/AThere are three ways to write your own source filter. You can write it
1N/Ain C, use an external program as a filter, or write the filter in Perl.
1N/AI won't cover the first two in any great detail, so I'll get them out
1N/Aof the way first. Writing the filter in Perl is most convenient, so
1N/AI'll devote the most space to it.
1N/A=head1 WRITING A SOURCE FILTER IN C
1N/AThe first of the three available techniques is to write the filter
1N/Acompletely in C. The external module you create interfaces directly
1N/Awith the source filter hooks provided by Perl.
1N/AThe advantage of this technique is that you have complete control over
1N/Athe implementation of your filter. The big disadvantage is the
1N/Aincreased complexity required to write the filter - not only do you
1N/Aneed to understand the source filter hooks, but you also need a
1N/Areasonable knowledge of Perl guts. One of the few times it is worth
1N/Agoing to this trouble is when writing a source scrambler. The
1N/AC<decrypt> filter (which unscrambles the source before Perl parses it)
1N/Aincluded with the source filter distribution is an example of a C
1N/Asource filter (see Decryption Filters, below).
1N/A=item B<Decryption Filters>
1N/AAll decryption filters work on the principle of "security through
1N/Aobscurity." Regardless of how well you write a decryption filter and
1N/Ahow strong your encryption algorithm, anyone determined enough can
1N/Aretrieve the original source code. The reason is quite simple - once
1N/Athe decryption filter has decrypted the source back to its original
1N/Aform, fragments of it will be stored in the computer's memory as Perl
1N/Aparses it. The source might only be in memory for a short period of
1N/Atime, but anyone possessing a debugger, skill, and lots of patience can
1N/Aeventually reconstruct your program.
1N/AThat said, there are a number of steps that can be taken to make life
1N/Adifficult for the potential cracker. The most important: Write your
1N/Adecryption filter in C and statically link the decryption module into
1N/Athe Perl binary. For further tips to make life difficult for the
1N/Apotential cracker, see the file I<
decrypt.pm> in the source filters
1N/A=head1 CREATING A SOURCE FILTER AS A SEPARATE EXECUTABLE
1N/AAn alternative to writing the filter in C is to create a separate
1N/Aexecutable in the language of your choice. The separate executable
1N/Areads from standard input, does whatever processing is necessary, and
1N/Awrites the filtered data to standard output. C<Filter:cpp> is an
1N/Aexample of a source filter implemented as a separate executable - the
1N/Aexecutable is the C preprocessor bundled with your C compiler.
1N/AThe source filter distribution includes two modules that simplify this
1N/Atask: C<Filter::exec> and C<Filter::sh>. Both allow you to run any
1N/Aexternal executable. Both use a coprocess to control the flow of data
1N/Ainto and out of the external executable. (For details on coprocesses,
1N/Asee Stephens,
W.R. "Advanced Programming in the UNIX Environment."
1N/AAddison-Wesley, ISBN 0-210-56317-7, pages 441-445.) The difference
1N/Abetween them is that C<Filter::exec> spawns the external command
1N/Adirectly, while C<Filter::sh> spawns a shell to execute the external
1N/Acommand. (Unix uses the Bourne shell; NT uses the cmd shell.) Spawning
1N/Aa shell allows you to make use of the shell metacharacters and
1N/Aredirection facilities.
1N/AHere is an example script that uses C<Filter::sh>:
1N/A use Filter::sh 'tr XYZ PQR' ;
1N/A print "XYZ a = $a\n" ;
1N/AThe output you'll get when the script is executed:
1N/AWriting a source filter as a separate executable works fine, but a
1N/Asmall performance penalty is incurred. For example, if you execute the
1N/Asmall example above, a separate subprocess will be created to run the
1N/AUnix C<tr> command. Each use of the filter requires its own subprocess.
1N/AIf creating subprocesses is expensive on your system, you might want to
1N/Aconsider one of the other options for creating source filters.
1N/A=head1 WRITING A SOURCE FILTER IN PERL
1N/AThe easiest and most portable option available for creating your own
1N/Asource filter is to write it completely in Perl. To distinguish this
1N/Afrom the previous two techniques, I'll call it a Perl source filter.
1N/ATo help understand how to write a Perl source filter we need an example
1N/Ato study. Here is a complete source filter that performs rot13
1N/Adecoding. (Rot13 is a very simple encryption scheme used in Usenet
1N/Apostings to hide the contents of offensive posts. It moves every letter
1N/Aforward thirteen places, so that A becomes N, B becomes O, and Z
1N/A use Filter::Util::Call ;
1N/A filter_add(bless $ref) ;
1N/A if ($status = filter_read()) > 0 ;
1N/AAll Perl source filters are implemented as Perl classes and have the
1N/Asame basic structure as the example above.
1N/AFirst, we include the C<Filter::Util::Call> module, which exports a
1N/Anumber of functions into your filter's namespace. The filter shown
1N/Aabove uses two of these functions, C<filter_add()> and
1N/ANext, we create the filter object and associate it with the source
1N/Astream by defining the C<import> function. If you know Perl well
1N/Aenough, you know that C<import> is called automatically every time a
1N/Amodule is included with a use statement. This makes C<import> the ideal
1N/Aplace to both create and install a filter object.
1N/AIn the example filter, the object (C<$ref>) is blessed just like any
1N/Aother Perl object. Our example uses an anonymous array, but this isn't
1N/Aa requirement. Because this example doesn't need to store any context
1N/Ainformation, we could have used a scalar or hash reference just as
1N/Awell. The next section demonstrates context data.
1N/AThe association between the filter object and the source stream is made
1N/Awith the C<filter_add()> function. This takes a filter object as a
1N/Aparameter (C<$ref> in this case) and installs it in the source stream.
1N/AFinally, there is the code that actually does the filtering. For this
1N/Atype of Perl source filter, all the filtering is done in a method
1N/Acalled C<filter()>. (It is also possible to write a Perl source filter
1N/Ausing a closure. See the C<Filter::Util::Call> manual page for more
1N/Adetails.) It's called every time the Perl parser needs another line of
1N/Asource to process. The C<filter()> method, in turn, reads lines from
1N/Athe source stream using the C<filter_read()> function.
1N/AIf a line was available from the source stream, C<filter_read()>
1N/Areturns a status value greater than zero and appends the line to C<$_>.
1N/AA status value of zero indicates end-of-file, less than zero means an
1N/Aerror. The filter function itself is expected to return its status in
1N/Athe same way, and put the filtered line it wants written to the source
1N/Astream in C<$_>. The use of C<$_> accounts for the brevity of most Perl
1N/AIn order to make use of the rot13 filter we need some way of encoding
1N/Athe source file in rot13 format. The script below, C<mkrot13>, does
1N/A die "usage mkrot13 filename\n" unless @ARGV ;
1N/A open(IN, "<$in") or die "Cannot open file $in: $!\n";
1N/A open(OUT, ">$out") or die "Cannot open file $out: $!\n";
1N/A print OUT "use Rot13;\n" ;
1N/AIf we encrypt this with C<mkrot13>:
1N/A print " hello fred \n" ;
1N/Athe result will be this:
1N/A cevag "uryyb serq\a" ;
1N/ARunning it produces this output:
1N/A=head1 USING CONTEXT: THE DEBUG FILTER
1N/AThe rot13 example was a trivial example. Here's another demonstration
1N/Athat shows off a few more features.
1N/ASay you wanted to include a lot of debugging code in your Perl script
1N/Aduring development, but you didn't want it available in the released
1N/Aproduct. Source filters offer a solution. In order to keep the example
1N/Asimple, let's say you wanted the debugging output to be controlled by
1N/Aan environment variable, C<DEBUG>. Debugging code is enabled if the
1N/Avariable exists, otherwise it is disabled.
1N/ATwo special marker lines will bracket debugging code, like this:
1N/A warn "Debug: millennium bug in year $year\n" ;
1N/AWhen the C<DEBUG> environment variable exists, the filter ensures that
1N/APerl parses only the code between the C<DEBUG_BEGIN> and C<DEBUG_END>
1N/Amarkers. That means that when C<DEBUG> does exist, the code above
1N/Ashould be passed through the filter unchanged. The marker lines can
1N/Aalso be passed through as-is, because the Perl parser will see them as
1N/Acomment lines. When C<DEBUG> isn't set, we need a way to disable the
1N/Adebug code. A simple way to achieve that is to convert the lines
1N/Abetween the two markers into comments:
1N/A #if ($year > 1999) {
1N/A # warn "Debug: millennium bug in year $year\n" ;
1N/AHere is the complete Debug filter:
1N/A use Filter::Util::Call ;
1N/A use constant TRUE => 1 ;
1N/A use constant FALSE => 0 ;
1N/A Enabled => defined $ENV{DEBUG},
1N/A InTraceBlock => FALSE,
1N/A Filename => (caller)[1],
1N/A filter_add(bless \%context) ;
1N/A my ($self) = shift ;
1N/A my ($message) = shift ;
1N/A my ($line_no) = shift || $self->{LastBegin} ;
1N/A die "$message at $self->{Filename} line $line_no.\n"
1N/A $status = filter_read() ;
1N/A ++ $self->{LineNo} ;
1N/A $self->Die("DEBUG_BEGIN has no DEBUG_END")
1N/A if $self->{InTraceBlock} ;
1N/A if ($self->{InTraceBlock}) {
1N/A if (/^\s*##\s*DEBUG_BEGIN/ ) {
1N/A $self->Die("Nested DEBUG_BEGIN", $self->{LineNo})
1N/A } elsif (/^\s*##\s*DEBUG_END/) {
1N/A $self->{InTraceBlock} = FALSE ;
1N/A # comment out the debug lines when the filter is disabled
1N/A s/^/#/ if ! $self->{Enabled} ;
1N/A } elsif ( /^\s*##\s*DEBUG_BEGIN/ ) {
1N/A $self->{InTraceBlock} = TRUE ;
1N/A $self->{LastBegin} = $self->{LineNo} ;
1N/A } elsif ( /^\s*##\s*DEBUG_END/ ) {
1N/A $self->Die("DEBUG_END has no DEBUG_BEGIN", $self->{LineNo});
1N/AThe big difference between this filter and the previous example is the
1N/Ause of context data in the filter object. The filter object is based on
1N/Aa hash reference, and is used to keep various pieces of context
1N/Ainformation between calls to the filter function. All but two of the
1N/Ahash fields are used for error reporting. The first of those two,
1N/AEnabled, is used by the filter to determine whether the debugging code
1N/Ashould be given to the Perl parser. The second, InTraceBlock, is true
1N/Awhen the filter has encountered a C<DEBUG_BEGIN> line, but has not yet
1N/Aencountered the following C<DEBUG_END> line.
1N/AIf you ignore all the error checking that most of the code does, the
1N/Aessence of the filter is as follows:
1N/A $status = filter_read() ;
1N/A return $status if $status <= 0 ;
1N/A if ($self->{InTraceBlock}) {
1N/A if (/^\s*##\s*DEBUG_END/) {
1N/A $self->{InTraceBlock} = FALSE
1N/A # comment out debug lines when the filter is disabled
1N/A s/^/#/ if ! $self->{Enabled} ;
1N/A } elsif ( /^\s*##\s*DEBUG_BEGIN/ ) {
1N/A $self->{InTraceBlock} = TRUE ;
1N/ABe warned: just as the C-preprocessor doesn't know C, the Debug filter
1N/Adoesn't know Perl. It can be fooled quite easily:
1N/ASuch things aside, you can see that a lot can be achieved with a modest
1N/AYou now have better understanding of what a source filter is, and you
1N/Amight even have a possible use for them. If you feel like playing with
1N/Asource filters but need a bit of inspiration, here are some extra
1N/Afeatures you could add to the Debug filter.
1N/AFirst, an easy one. Rather than having debugging code that is
1N/Aall-or-nothing, it would be much more useful to be able to control
1N/Awhich specific blocks of debugging code get included. Try extending the
1N/Asyntax for debug blocks to allow each to be identified. The contents of
1N/Athe C<DEBUG> environment variable can then be used to control which
1N/AOnce you can identify individual blocks, try allowing them to be
1N/Anested. That isn't difficult either.
1N/AHere is an interesting idea that doesn't involve the Debug filter.
1N/ACurrently Perl subroutines have fairly limited support for formal
1N/Aparameter lists. You can specify the number of parameters and their
1N/Atype, but you still have to manually take them out of the C<@_> array
1N/Ayourself. Write a source filter that allows you to have a named
1N/Aparameter list. Such a filter would turn this:
1N/A sub MySub ($first, $second, @rest) { ... }
1N/A my ($first) = shift ;
1N/A my ($second) = shift ;
1N/AFinally, if you feel like a real challenge, have a go at writing a
1N/Afull-blown Perl macro preprocessor as a source filter. Borrow the
1N/Auseful features from the C preprocessor and any other macro processors
1N/Ayou know. The tricky bit will be choosing how much knowledge of Perl's
1N/Asyntax you want your filter to have.
1N/A=head1 THINGS TO LOOK OUT FOR
1N/A=item Some Filters Clobber the C<DATA> Handle
1N/ASome source filters use the C<DATA> handle to read the calling program.
1N/AWhen using these source filters you cannot rely on this handle, nor expect
1N/Aany particular kind of behavior when operating on it. Filters based on
1N/AFilter::Util::Call (and therefore Filter::Simple) do not alter the C<DATA>
1N/AThe Source Filters distribution is available on CPAN, in
1N/AStarting from Perl 5.8 Filter::Util::Call (the core part of the
1N/ASource Filters distribution) is part of the standard Perl distribution.
1N/AAlso included is a friendlier interface called Filter::Simple, by
1N/APaul Marquess E<lt>Paul.Marquess@btinternet.comE<gt>
1N/AThis article originally appeared in The Perl Journal #11, and is
1N/Acopyright 1998 The Perl Journal. It appears courtesy of Jon Orwant and
1N/AThe Perl Journal. This document may be distributed under the same terms