49N/AThe contents of this file are subject to the terms of the
49N/ACommon Development and Distribution License (the "License").
49N/AYou may not use this file except in compliance with the License.
49N/ASee the License for the specific language governing permissions
49N/Aand limitations under the License.
49N/AWhen distributing Covered Code, include this CDDL HEADER in each
49N/AIf applicable, add the following below this CDDL HEADER, with the
49N/Afields enclosed by brackets "[]" replaced with your own identifying
49N/Ainformation: Portions Copyright [yyyy] [name of copyright owner]
49N/ACopyright 2007 Sun Microsystems, Inc. All rights reserved.
49N/AUse is subject to license terms.
2205N/AArchitectural Overview for the DHCP agent
49N/AThe Solaris DHCP agent (dhcpagent) is a DHCP client implementation
49N/Acompliant with RFCs 2131, 3315, and others. The major forces shaping
51N/A * Must be capable of managing multiple network interfaces.
289N/A * Must consume little CPU, since it will always be running.
941N/A * Must have a small memory footprint, since it will always be
1859N/A * Must not rely on any shared libraries outside of /lib, since
49N/A it must run before all filesystems have been mounted.
49N/AWhen a DHCP agent implementation is only required to control a single
49N/Ainterface on a machine, the problem is expressed well as a simple
1846N/Astate-machine, as shown in RFC2131. However, when a DHCP agent is
1846N/Aresponsible for managing more than one interface at a time, the
51N/Aproblem becomes much more complicated.
2205N/AThis can be resolved using threads or with an event-driven model.
2205N/AGiven that DHCP's behavior can be expressed concisely as a state
2205N/Amachine, the event-driven model is the closest match.
2639N/AWhile tried-and-true, that model is subtle and easy to get wrong.
51N/AIndeed, much of the agent's code is there to manage the complexity of
307N/Aprogramming in an asynchronous event-driven paradigm.
2205N/AThe DHCP agent consists of roughly 30 source files, most with a
2205N/Acompanion header file. While the largest source file is around 1700
2205N/Alines, most are much shorter. The source files can largely be broken
2205N/A * Source files that, along with their companion header files,
2205N/A define an abstract "object" that is used by other parts of
2205N/A an interface for managing asynchronous transactions within
307N/A * Source files that implement a given state of the agent; for
281N/A the procedural "work" which must be done while in the
3109N/A REQUESTING state of the agent. By encapsulating states in
3109N/A files, it becomes easier to debug errors in the
3109N/A constraints, since all the relevant code is in one place.
3109N/A * Source files, which along with their companion header files,
3109N/A encapsulate a given task or related set of tasks. The
3109N/A difference between this and the first group is that the
3109N/A interfaces exported from these files do not operate on
3109N/A an "object", but rather perform a specific task. Examples
3109N/A to DLPI-related i/o operations.
3109N/AHere we discuss the essential objects and subtle aspects of the
3109N/ADHCP agent implementation. Note that there is of course much more
3109N/Athat is not discussed here, but after this overview you should be able
3109N/Ato fend for yourself in the source code.
3109N/AFor details on the DHCPv6 aspects of the design, and how this relates
3109N/Ato the implementation present in previous releases of Solaris, see the
3109N/AEvent Handlers and Timer Queues
3109N/A-------------------------------
3109N/AThe most important object in the agent is the event handler, whose
3109N/Alibinetutil. The event handler is essentially an object-oriented
3109N/Awrapper around poll(2): other components of the agent can register to
3158N/Abe called back when specific events on file descriptors happen -- for
3109N/Ainstance, to wait for requests to arrive on its IPC socket, the agent
3109N/Aregisters a callback function (accept_event()) that will be called
3109N/Aback whenever a new connection arrives on the file descriptor
3109N/Aassociated with the IPC socket. When the agent initially begins in
3158N/Amain(), it registers a number of events with the event handler, and
3109N/Athen calls iu_handle_events(), which proceeds to wait for events to
3109N/Ahappen -- this function does not return until the agent is shutdown
3109N/AWhen the registered events occur, the callback functions are called
3109N/Aback, which in turn might lead to additional callbacks being
3109N/Aregistered -- this is the classic event-driven model. (As an aside,
3109N/Anote that programming in an event-driven model means that callbacks
3109N/Acannot block, or else the agent will become unresponsive.)
3109N/AA special kind of "event" is a timeout. Since there are many timers
3109N/Awhich must be maintained for each DHCP-controlled interface (such as a
3109N/Alease expiration timer, time-to-first-renewal (t1) timer, and so
3109N/Aforth), an object-oriented abstraction to timers called a "timer
3109N/Acorresponding implementation in libinetutil. The timer queue allows
136N/Acallback functions to be "scheduled" for callback after a certain
49N/Aamount of time has passed.
1755N/AThe event handler and timer queue objects work hand-in-hand: the event
1755N/Ahandler is passed a pointer to a timer queue in iu_handle_events() --
1755N/Afrom there, it can use the iu_earliest_timer() routine to find the
1755N/Atimer which will next fire, and use this to set its timeout value in
1755N/Aits call to poll(2). If poll(2) returns due to a timeout, the event
1755N/Ahandler calls iu_expire_timers() to expire all timers that expired
1755N/A(note that more than one may have expired if, for example, multiple
1755N/Atimers were set to expire at the same time).
1784N/AAlthough it is possible to instantiate more than one timer queue or
1784N/Aevent handler object, it doesn't make a lot of sense -- these objects
72N/Aare really "singletons". Accordingly, the agent has two global
1784N/Avariables, `eh' and `tq', which store pointers to the global event
1784N/AFor each network interface managed by the agent, there is a set of
1784N/Aassociated state that describes both its general properties (such as
72N/Athe maximum MTU) and its connections to DHCP-related state (the
3047N/Aprotocol state machines). This state is stored in a pair of
1859N/Astructures called `dhcp_pif_t' (the IP physical interface layer or
1859N/APIF) and `dhcp_lif_t' (the IP logical interface layer or LIF). Each
1859N/Adhcp_pif_t represents a single physical interface, such as "hme0," for
51N/Aa given IP protocol version (4 or 6), and has a list of dhcp_lif_t
95N/Astructures representing the logical interfaces (such as "hme0:1") in
95N/AThis split is important because of differences between IPv4 and IPv6.
72N/AFor IPv4, each DHCP state machine manages a single IP address and
3109N/Aassociated configuration data. This corresponds to a single logical
1859N/Ainterface, which must be specified by the user. For IPv6, however,
95N/Aeach DHCP state machine manages a group of addresses, and is
95N/Aassociated with DUID value rather than with just an interface.
95N/AThus, DHCPv6 behaves more like
in.ndpd in its creation of "ADDRCONF"
95N/Ainterfaces. The agent automatically plumbs logical interfaces when
95N/Aneeded and removes them when the addresses expire.
95N/AThe state for a given session is stored separately in `dhcp_smach_t'.
95N/AThis state machine then points to the main LIF used for I/O, and to a
95N/Alist of `dhcp_lease_t' structures representing individual leases, and
95N/Aeach of those points to a list of LIFs corresponding to the individual
72N/Aaddresses being managed.
3109N/AOne point that was brushed over in the preceding discussion of event
3109N/Ahandlers and timer queues was context. Recall that the event-driven
3109N/Anature of the agent requires that functions cannot block, lest they
3109N/Astarve out others and impact the observed responsiveness of the agent.
3109N/AAs an example, consider the process of extending a lease: the agent
3109N/Amust send a REQUEST packet and wait for an ACK or NAK packet in
3109N/Aresponse. This is done by sending a REQUEST and then returning to the
3109N/Aevent handler that waits for an ACK or NAK packet to arrive on the
3109N/Afile descriptor associated with the interface. Note however, that
49N/Awhen the ACK or NAK does arrive, and the callback function called
2818N/Aback, it must know which state machine this packet is for (it must get
2818N/Aback its context). This could be handled through an ad-hoc mapping of
2818N/Afile descriptors to state machines, but a cleaner approach is to have
2818N/Athe event handler's register function (iu_register_event()) take in an
2818N/Aopaque context pointer, which will then be passed back to the
2818N/Acallback. In the agent, the context pointer used depends on the
2818N/Anature of the event: events on LIFs use the dhcp_lif_t pointer, events
2818N/Aon the state machine use dhcp_smach_t, and so on.
2818N/ANote that there is nothing that guarantees the pointer passed into
2818N/Aiu_register_event() or iu_schedule_timer() will still be valid when
72N/Athe callback is called back (for instance, the memory may have been
72N/Afreed in the meantime). To solve this problem, all of the data
289N/Astructures used in this way are reference counted. For more details
72N/Aon how the reference count scheme is implemented, see the closing
1685N/AMany operations performed via DHCP must be performed in groups -- for
1685N/Ainstance, acquiring a lease requires several steps: sending a
1685N/ADISCOVER, collecting OFFERs, selecting an OFFER, sending a REQUEST,
222N/Aand receiving an ACK, assuming everything goes well. Note however
1685N/Athat due to the event-driven model the agent operates in, these
1281N/Aoperations are not inherently "grouped" -- instead, the agent sends a
1685N/ADISCOVER, goes back into the main event loop, waits for events
289N/A(perhaps even requests on the IPC channel to begin acquiring a lease
136N/Aon another state machine), eventually checks to see if an acceptable
3047N/AOFFER has come in, and so forth. To some degree, the notion of the
462N/Astate machine's current state (SELECTING, REQUESTING, etc) helps
462N/Acontrol the potential chaos of the event-driven model (for instance,
462N/Aif while the agent is waiting for an OFFER on a given state machine,
462N/Aan IPC event comes in requesting that the leases be RELEASED, the
462N/Aagent knows to send back an error since the state machine must be in
1859N/Aat least the BOUND state before a RELEASE can be performed.)
1859N/AHowever, states are not enough -- for instance, suppose that the agent
2293N/Abegins trying to renew a lease. This is done by sending a REQUEST
1859N/Apacket and waiting for an ACK or NAK, which might never come. If,
1859N/Awhile waiting for the ACK or NAK, the user sends a request to renew
1859N/Athe lease as well, then if the agent were to send another REQUEST,
1859N/Athings could get quite complicated (and this is only the beginning of
2293N/Athis rathole). To protect against this, two objects exist:
1859N/A`async_action' and `ipc_action'. These objects are related, but
1859N/Aindependent of one another; the more essential object is the
1859N/A`async_action', which we will discuss first.
3158N/AIn short, an `async_action' represents a pending transaction (aka
2987N/Aasynchronous action), of which each state machine can have at most
2987N/Aone. The `async_action' structure is embedded in the `dhcp_smach_t'
2987N/Astructure, which is fine since there can be at most one pending
3158N/Atransaction per state machine. Typical "asynchronous transactions"
3158N/Aare START, EXTEND, and INFORM, since each consists of a sequence of
2987N/Apackets that must be done without interruption. Note that not all
2987N/ADHCP operations are "asynchronous" -- for instance, a DHCPv4 RELEASE
2987N/Aoperation is synchronous (not asynchronous) since after the RELEASE is
1859N/Asent no reply is expected from the DHCP server, but DHCPv6 Release is
1859N/Aasynchronous, as all DHCPv6 messages are transactional. Some
1859N/Aoperations, such as status query, are synchronous and do not affect
1859N/Athe system state, and thus do not require sequencing.
3158N/AWhen the agent realizes it must perform an asynchronous transaction,
2987N/Ait calls async_async() to open the transaction. If one is already
3158N/Apending, then the new transaction must fail (the details of failure
3158N/Adepend on how the transaction was initiated, which is described in
2987N/Amore detail later when the `ipc_action' object is discussed). If
2987N/Athere is no pending asynchronous transaction, the operation succeeds.
462N/AWhen the transaction is complete, either async_finish() or
462N/Aasync_cancel() must be called to complete or cancel the asynchronous
140N/Aaction on that state machine. If the transaction is unable to
144N/Acomplete within a certain amount of time (more on this later), a timer
1100N/Ashould be used to cancel the operation.
1100N/AThe notion of asynchronous transactions is complicated by the fact
941N/Athat they may originate from both inside and outside of the agent.
2204N/AFor instance, a user initiates an asynchronous START transaction when
941N/Ahe performs an `ifconfig hme0 dhcp start', but the agent will
941N/Ainternally need to perform asynchronous EXTEND transactions to extend
2204N/Athe lease before it expires. Note that user-initiated actions always
941N/Ahave priority over internal actions: the former will cancel the
1755N/AThis leads us into the `ipc_action' object. An `ipc_action'
1755N/Arepresents the IPC-related pieces of an asynchronous transaction that
1755N/Awas started as a result of a user request, as well as the `BUSY' state
1755N/Aof the administrative interface. Only IPC-generated asynchronous
1755N/Atransactions have a valid `ipc_action' object. Note that since there
1755N/Acan be at most one asynchronous action per state machine, there can
1755N/Aalso be at most one `ipc_action' per state machine (this means it can
1755N/Aalso conveniently be embedded inside the `dhcp_smach_t' structure).
1755N/AOne of the main purposes of the `ipc_action' object is to timeout user
1755N/Aevents. When the user specifies a timeout value as an argument to
2476N/Aifconfig, he is specifying an `ipc_action' timeout; in other words,
2476N/Ahow long he is willing to wait for the command to complete. When this
2476N/Atime expires, the ipc_action is terminated, as well as the
2476N/AThe API provided for the `ipc_action' object is quite similar to the
one for the `async_action' object: when an IPC request comes in for an
operation requiring asynchronous operation, ipc_action_start() is
called. When the request completes, ipc_action_finish() is called.
If the user times out before the request completes, then
ipc_action_timeout() is called.
Another complicated area is packet management: building, manipulating,
sending and receiving packets. These operations are all encapsulated
behind a dozen or so interfaces (see
packet.h) that abstract the
unimportant details away from the rest of the agent code. In order to
send a DHCP packet, code first calls init_pkt(), which returns a
dhcp_pkt_t initialized suitably for transmission. Note that currently
init_pkt() returns a dhcp_pkt_t that is actually allocated as part of
the `dhcp_smach_t', but this may change in the future.. After calling
init_pkt(), the add_pkt_opt*() functions are used to add options to
the DHCP packet. Finally, send_pkt() and send_pkt_v6() can be used to
transmit the packet to a given IP address.
The send_pkt() function is actually quite complicated; for one, it
must internally use either DLPI or sockets depending on the machine
state; for another, it handles the details of packet timeout and
retransmission. The last argument to send_pkt() is a pointer to a
"stop function." If this argument is passed as NULL, then the packet
will only be sent once (it won't be retransmitted). Otherwise, before
each retransmission, the stop function will be called back prior to
retransmission. The callback may alter dsm_send_timeout if necessary
to place a cap on the next timeout; this is done for DHCPv6 in
stop_init_reboot() in order to implement the CNF_MAX_RD constraint.
The return value from this function indicates whether to continue
retransmission or not, which allows the send_pkt() caller to control
the retransmission policy without making it have to deal with the
retransmission mechanism. See
request.c for an example of this in
The recv_pkt() function is simpler but still complicated by the fact
that one may want to receive several different types of packets at
once and in different ways (DLPI or sockets). The caller registers an
event handler on the file descriptor, and then calls recv_pkt() to
read in the packet along with meta information about the message (the
sender and interface identifier).
For IPv6, packet reception is done with a single socket, using
IPV6_PKTINFO to determine the actual destination address and receiving
interface. Packets are then matched against the state machines on the
given interface through the transaction ID.
The same facility exists for inbound IPv4 packets, but because there's
no IP_PKTINFO processing on output yet in Solaris, and because IPv4
still relies on DLPI, DHCP packets are handled on a per-LIF (when
bound) and per-PIF (when unbound) basis. Eventually, when IP_PKTINFO
is available for IPv4, the per-LIF sockets can go away. If it ever
becomes possible to send and receive IP packets without having an IP
address configured on an interface, then the DLPI streams can go as
The notion of time is an exceptionally subtle area. You will notice
five ways that time is represented in the source: as lease_t's,
uint32_t's, time_t's, hrtime_t's, and monosec_t's. Each of these
types serves a slightly different function.
The `lease_t' type is the simplest to understand; it is the unit of
time in the CD_{LEASE,T1,T2}_TIME options in a DHCP packet, as defined
by RFC2131. This is defined as a positive number of seconds (relative
to some fixed point in time) or the value `-1' (DHCP_PERM) which
represents infinity (
i.e., a permanent lease). The lease_t should be
used either when dealing with actual DHCP packets that are sent on the
wire or for variables which follow the exact definition given in the
The `uint32_t' type is also used to represent a relative time in
seconds. However, here the value `-1' is not special and of course
this type is not tied to any definition given in RFC2131. Use this
for representing "offsets" from another point in time that are not
The `time_t' type is the natural Unix type for representing time since
the epoch. Unfortunately, it is affected by stime(2) or adjtime(2)
and since the DHCP client is used during system installation (and thus
when time is typically being configured), the time_t cannot be used in
general to represent an absolute time since the epoch. For instance,
if a time_t were used to keep track of when a lease began, and then a
minute later stime(2) was called to adjust the system clock forward a
year, then the lease would appeared to have expired a year ago even
though it has only been a minute. For this reason, time_t's should
only be used either when wall time must be displayed (such as in
DHCP_STATUS ipc transaction) or when a time meaningful across reboots
must be obtained (such as when caching an ACK packet at system
The `hrtime_t' type returned from gethrtime() works around the
limitations of the time_t in that it is not affected by stime(2) or
adjtime(2), with the disadvantage that it represents time from some
arbitrary time in the past and in nanoseconds. The timer queue code
deals with hrtime_t's directly since that particular piece of code is
meant to be fairly independent of the rest of the DHCP client.
However, dealing with nanoseconds is error-prone when all the other
time types are in seconds. As a result, yet another time type, the
`monosec_t' was created to represent a monotonically increasing time
in seconds, and is really no more than (hrtime_t / NANOSEC). Note
that this unit is typically used where time_t's would've traditionally
been used. The function monosec() in
util.c returns the current
monosec, and monosec_to_time() can convert a given monosec to wall
time, using the system's current notion of time.
One additional limitation of the `hrtime_t' and `monosec_t' types is
events (
e.g., those generated by sys-suspend(1M)). For example, if
gethrtime() returns time T, and then the machine is suspended for 2
hours, and then gethrtime() is called again, the time returned is not
T + (2 * 60 * 60 * NANOSEC), but rather approximately still T.
when a system is resumed, the DHCP client makes the pessimistic
assumption that all finite leases have expired while the machine was
suspended and must be obtained again. This is known as "refreshing"
the leases, and is handled by refresh_smachs().
Note that it appears like a more intelligent approach would be to
record the time(2) when the system is suspended, compare that against
the time(2) when the system is resumed, and use the delta between them
to decide which leases have expired. Sadly, this cannot be done since
through at least Solaris 10, it is not possible for userland programs
to be notified of system suspend events.
For the most part, the DHCP client only *retrieves* configuration data
from the DHCP server, leaving the configuration to scripts (such as
boot scripts), which themselves use dhcpinfo(1) to retrieve the data
from the DHCP client. This is desirable because it keeps the mechanism
of retrieving the configuration data decoupled from the policy of using
However, unless used in "inform" mode, the DHCP client *does*
configure each IP interface enough to allow it to communicate with
other hosts. Specifically, the DHCP client configures the interface's
IP address, netmask, and broadcast address using the information
provided by the server. Further, for IPv4 logical interface 0
("hme0"), any provided default routes are also configured.
For IPv6, only the IP addresses are set. The netmask (prefix) is then
set automatically by
in.ndpd, and routes are discovered in the usual
way by router discovery or routing protocols. DHCPv6 doesn't set
Since logical interfaces cannot be specified as output interfaces in
the kernel forwarding table, and in most cases, logical interfaces
share a default route with their associated physical interface, the
DHCP client does not automatically add or remove default routes when
IPv4 leases are acquired or expired on logical interfaces.
The DHCP client supports user program invocations on DHCP events. The
supported events are BOUND, EXTEND, EXPIRE, DROP, RELEASE, and INFORM
for DHCPv4, and BUILD6, EXTEND6, EXPIRE6, DROP6, LOSS6, RELEASE6, and
INFORM6 for DHCPv6. The user program runs asynchronous to the DHCP
client so that the main event loop stays active to process other
events, including events triggered by the user program (for example,
when it invokes dhcpinfo).
The user program execution is part of the transaction of a DHCP command.
For example, if the user program is not enabled, the transaction of the
DHCP command START is considered over when an ACK is received and the
interface is configured successfully. If the user program is enabled,
it is invoked after the interface is configured successfully, and the
transaction is considered over only when the user program exits. The
event scripting implementation makes use of the asynchronous operations
discussed in the "Transactions" section.
An upper bound of 58 seconds is imposed on how long the user program
can run. If the user program does not exit after 55 seconds, the signal
SIGTERM is sent to it. If it still does not exit after additional 3
seconds, the signal SIGKILL is sent to it. Since the event handler is
a wrapper around poll(), the DHCP client cannot directly observe the
completion of the user program. Instead, the DHCP client creates a
child "helper" process to synchronously monitor the user program (this
process is also used to send the aformentioned signals to the process,
if necessary). The DHCP client and the helper process share a pipe
which is included in the set of poll descriptors monitored by the DHCP
client's event handler. When the user program exits, the helper process
passes the user program exit status to the DHCP client through the pipe,
informing the DHCP client that the user program has finished. When the
DHCP client is asked to shut down, it will wait for any running instances
of the user program to complete.