APACHE 2.x ROADMAP

==================

Last modified at [$Date: 2002/10/17 17:16:04 $]

INTRODUCTION

------------

The Apache HTTP Server project must balance two competing and disjoint

objectives; maintain stable code for third party authors, distributors and

most importantly users so that bug and security fixes can be quickly adopted

without significant hardship due to API changes; and continue the development

process that requires ongoing redesign to work around earlier oversights in

the implementation of a fluid and flexible API.

The Apache HTTP Server versions, through 2.0, used the Module Magic Number

to reflect the relatively frequent API changes. This had the shortcoming

of often leaving binary download users hunting to replace their loaded third

party modules. This left the third party module authors searching through

the API change histories to determine the new declarations, APIs and side

effects of making the necessary code changes.

With the simultaneous release of Apache 2.1-stable and Apache 2.2-development,

the Apache HTTP Server project is moving to a more predictable stable code

branch, while opening the development to forward progress without concern

for breaking the stable branch. This document explains the rationale between

the two versions and their behavior, going forward.

STABLE RELEASES, 2.{odd}.{revision}

------------------------------------

All even numbered releases will be considered stable revisions. That means;

* Forward Compatibility; users are not required to find new downloads of

currently loaded modules to upgrade from other revisions of the same

version. To upgrade from 2.1.0 and 2.1.27 will require no new modules.

However, the third party modules may break from buggy code, or code that

used an undocumented side effect of an API call, which may be changed to

close bugs or security vulnerabilities. Modules should be retested.

Moreover, new APIs may be introduced within the lifespan of the release,

and it is up to the third party module author to call out what version

forward this module is compatible with (e.g. "Compatible with Apache

HTTP Server version 2.1.12 and foward.") The next stable release that

causes module incompatibility for 2.1.x users will be an upgrade to

either the current 2.2.x-development releases or the 2.3.0-stable release.

* No Deprecated modules; although new modules may be introduced within the

stable release, no loadable modules or their directives will be removed

within the lifetime of a given stable release version. The next release

that deprecates old modules for 2.1.x users will be an upgrade to either

the 2.2.x-development release or the 2.3.0-stable release.

* Warnings should be provided in the documentation to give users a heads up

that a given module or directive will disappear in the future release,

and advise developers that a given API will change. However, it is always

best to check the corresponding development release to determine the full

impact of such changes.

* No 'Experimental' modules; while it may be possible (based on API changes

required to support a given module) to load a 2.2-development module into

a 2.1-stable build of Apache, there are no guarantees. Experimental

modules will be introduced to the 2.2-development versions and either

added to 2.1-stable once they are proven and compatible, or deferred

to the 2.3-stable release if they cannot be incorporated in the current

stable release due to API change requirements.

* The stable CVS tree must not remain unstable at any time. Atomic commits

must be used to introduce code from the development version to the stable

tree. At any given time a security release may be in preparation,

unbeknownst to other contributors. At any given time, testers may be

checking out CVS head to confirm that a bug has been corrected. And as

all code was well-tested in development prior to committing to the stable

tree, there is really no reason for this tree to be broken for more than

a few minutes during a lengthy commit.

DEVELOPMENT RELEASES, 2.{even}.{revision}

-----------------------------------------

All even numbered releases designate the 'next' possible stable release,

therefore the current development version will always be one greater than

the stable release. Work proceeds on development releases, permitting

the modification of the MMN at any time in order to correct deficiencies

or shortcomings in the API. This means that third party modules from one

revision to another may not be binary compatible, and may not successfully

compile without modification to accomodate the API changes.

The only 'supported' development release at any time will be the most

recently released version. Developers will not be answering bug reports

of older development releases once a new release is available, it becomes

the resposibility of the reporter to use the latest development version

to confirm that the bug still exists.

Any new code, new API features or new ('experimental') modules may be

promoted at any time to the next stable release, by a vote of the project

contributors. This vote is based on the technical stability of the new

code and the stability of the interface. Once moved to stable, that feature

cannot change for the remainder of that lifetime of that stable verions,

so the vote must reflect that the final decisions on the behavior and naming

of that new feature were reached. Vetos continue to apply to this choice

of introducing the new work to the stable version.

At any given time, when the quality of changes to the development branch

is considered release quality, that version may become a candidate for the

next stable release. This includes some or all of the API changes, promoting

experimental modules to stable or deprecating and eliminating older modules

from the last stable release. All of these choices are considered by the

project as a group in the interests of promoting the stable release, so that

any given change may be 'deferred' for a future release by the group, rather

than introduce unacceptable risks to adopting the next stable release.

Third party module authors are strongly encouraged to test with the latest

development version. This assures that the module will be ready for the next

stable release, but more importantly, the author can react to shortcomings

in the API early enough to warn the dev@httpd.apache.org community of the

shortcomings so that they can be addressed before the stable release. The

entire onus is on the third party module author to anticipate the needs of

their module before the stable release is created, once it has been released

they will be stuck with that API for the lifetime of that stable release.

WORKS IN PROGRESS

-----------------

* Source code should follow style guidelines.

OK, we all agree pretty code is good. Probably best to clean this

up by hand immediately upon branching a 2.1 tree.

Status: Justin volunteers to hand-edit the entire source tree ;)

Justin says:

Recall when the release plan for 2.0 was written:

Absolute Enforcement of an "Apache Style" for code.

Watch this slip into 3.0.

David says:

The style guide needs to be reviewed before this can be done.

http://httpd.apache.org/dev/styleguide.html

The current file is dated April 20th 1998!

OtherBill offers:

It's survived since '98 because it's welldone :-) Suggest we

simply follow whatever is documented in styleguide.html as we

branch the next tree. Really sort of straightforward, if you

dislike a bit within that doc, bring it up on the dev@httpd

list prior to the next branch.

So Bill sums up ... let's get the code cleaned up in CVS head.

Remember, it just takes cvs diff -b (that is, --ignore-space-change)

to see the code changes and ignore that cruft. Get editing Justin :)

* revamp the input filter syntax to provide for ordering of

filters created with the Set{Input|Output}Filter and the

Add{Input|Output}Filter directives. A 'relative to filterx'

syntax is definately preferable.

* Platforms that do not support fork (primarily Win32 and AS/400)

Architect start-up code that avoids initializing all the modules

in the parent process on platforms that do not support fork.

. Better yet - not only inform the startup of which phase it's in,

but allow the parent 'process' to initialize shared memory, etc,

and create a module-by-module stream to pass to the child, so the

parent can actually arbitrate the important stuff.

* Replace stat [deferred open] with open/fstat in directory_walk.

Justin, Ian, OtherBill all interested in this. Implies setting up

the apr_file_t member in request_rec, and having all modules use

that file, and allow the cleanup to close it [if it isn't a shared,

cached file handle.]

* The Async Apache Server implemented in terms of APR.

[Bill Stoddard's pet project.]

Message-ID: <008301c17d42$9b446970$01000100@sashimi> (dev@apr)

OtherBill notes that this can proceed in two parts...

Async accept, setup, and tear-down of the request

e.g. dealing with the incoming request headers, prior to

dispatching the request to a thread for processing.

This doesn't need to wait for a 2.x/3.0 bump.

Async delegation of the entire request processing chain

Too many handlers use stack storage and presume it is

available for the life of the request, so a complete

async implementation would need to happen 3.0 release.

* Add a string "class" that combines a char* with a length

and a reference count. This will help reduce the number

of strlen and strdup operations during request processing.

Including both the length and allocation will save us a ton

of reallocation we do today, in terms of string manipulation.

OtherBill asks if this is really an APR issue, not an HTTPD issue?

MAKING APACHE REPOSITORY-AGNOSTIC

(or: remove knowledge of the filesystem)

[ 2002/10/01: discussion in progress on items below; this isn't

planned yet ]

* dav_resource concept for an HTTP resource ("ap_resource")

* r->filename, r->canonical_filename, r->finfo need to

disappear. All users need to use new APIs on the ap_resource

object.

(backwards compat: today, when this occurs with mod_dav and a

custom backend, the above items refer to the topmost directory

mapped by a location; e.g. docroot)

Need to preserve a 'filename'-like string for mime-by-name

sorts of operations. But this only needs to be the name itself

and not a full path.

Justin: Can we leverage the path info, or do we not trust the

user?

gstein: well, it isn't the "path info", but the actual URI of

the resource. And of course we trust the user... that is

the resource they requested.

dav_resource->uri is the field you want. path_info might

still exist, but that portion might be related to the

CGI concept of "path translated" or some other further

resolution.

To continue, I would suggest that "path translated" and

having *any* path info is Badness. It means that you did

not fully resolve a resource for the given URI. The

"abs_path" in a URI identifies a resource, and that

should get fully resolved. None of this "resolve to

<here> and then we have a magical second resolution

(inside the CGI script)" or somesuch.

Justin: Well, let's consider mod_mbox for a second. It is sort of

a virtual filesystem in its own right - as it introduces

it's own notion of a URI space, but it is intrinsically

tied to the filesystem to do the lookups. But, for the

portion that isn't resolved on the file system, it has

its own addressing scheme. Do we need the ability to

layer resolution?

* The translate_name hook goes away

Wrowe altogether disagrees. translate_name today even operates

on URIs ... this mechansim needs to be preserved.

* The doc for map_to_storage is totally opaque to me. It has

something to do with filesystems, but it also talks about

security and per_dir_config and other stuff. I presume something

needs to happen there -- at least better doc.

Wrowe agrees and will write it up.

* The directory_walk concept disappears. All configuration is

tagged to Locations. The "mod_filesystem" module might have some

internal concept of the same config appearing in multiple

places, but that is handled internally rather than by Apache

core.

Wrowe suggests this is wrong, instead it's private to filesystem

requests, and is already invoked from map_to_storage, not the core

handler. <Directory > and <Files > blocks are preserved as-is,

but <Directory > sections become specific to the filesystem handler

alone. Because alternate filesystem schemes could be loaded, this

should be exposed, from the core, for other file-based stores to

share. Consider an archive store where the layers become

<Directory path> -> <Archive store> -> <File name>

Justin: How do we map Directory entries to Locations?

* The "Location tree" is an in-memory representation of the URL

namespace. Nodes of the tree have configuration specific to that

location in the namespace.

Something like:

typedef struct {

const char *name; /* name of this node relative to parent */

struct ap_conf_vector_t *locn_config;

apr_hash_t *children; /* NULL if no child configs */

} ap_locn_node;

The following config:

<Location /server-status>

SetHandler server-status

Order deny,allow

Deny from all

Allow from 127.0.0.1

</Location>

Creates a node with name=="server_status", and the node is a

child of the "/" node. (hmm. node->name is redundant with the

hash key; maybe drop node->name)

In the config vector, mod_access has stored its Order, Deny, and

Allow configs. mod_core has stored the SetHandler.

During the Location walk, we merge the config vectors normally.

Note that an Alias simply associates a filesystem path (in

mod_filesystem) with that Location in the tree. Merging

continues with child locations, but a merge is never done

through filesystem locations. Config on a specific subdir needs

to be mapped back into the corresponding point in the Location

tree for proper merging.

* Config is parsed into a tree, as we did for the 2.0 timeframe,

but that tree is just a representation of the config (for

multiple runs and for in-memory manipulation and usage). It is

unrelated to the "Location tree".

* Calls to apr_file_io functions generally need to be replaced

with operations against the ap_resource. For example, rather

than calling apr_dir_open/read/close(), a caller uses

resource->repos->get_children() or somesuch.

Note that things like mod_dir, mod_autoindex, and mod_negotation

need to be converted to use these mechanisms so that their

functions will work on logical repositories rather than just

filesystems.

* How do we handle CGI scripts? Especially when the resource may

not be backed by a file? Ideally, we should be able to come up

with some mechanism to allow CGIs to work in a

repository-independent manner.

- Writing the virtual data as a file and then executing it?

- Can a shell be executed in a streamy manner? (Portably?)

- Have an 'execute_resource' hook/func that allows the

repository to choose its manner - be it exec() or whatever.

- Won't this approach lead to duplication of code? Helper fns?

gstein: PHP, Perl, and Python scripts are nominally executed by

a filter inserted by mod_php/perl/python. I'd suggest

that shell/batch scripts are similar.

But to ask further: what if it is an executable

*program* rather than just a script? Do we yank that out

of the repository, drop it onto the filesystem, and run

it? eeewwwww...

I'll vote -0.9 for CGIs as a filter. Keep 'em handlers.

Justin: So, do we give up executing CGIs from virtual repositories?

That seems like a sad tradeoff to make. I'd like to have

my CGI scripts under DAV (SVN) control.

* How do we handle overlaying of Location and Directory entries?

Right now, we have a problem when /cgi-bin/ is ScriptAlias'd and

mod_dav has control over /. Some people believe that /cgi-bin/

shouldn't be under DAV control, while others do believe it

should be. What's the right strategy?