14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * CDDL HEADER START
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * The contents of this file are subject to the terms of the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Common Development and Distribution License (the "License").
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * You may not use this file except in compliance with the License.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * See the License for the specific language governing permissions
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * and limitations under the License.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * When distributing Covered Code, include this CDDL HEADER in each
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If applicable, add the following below this CDDL HEADER, with the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * fields enclosed by brackets "[]" replaced with your own identifying
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * information: Portions Copyright [yyyy] [name of copyright owner]
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * CDDL HEADER END
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Copyright 2009 Sun Microsystems, Inc. All rights reserved.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Use is subject to license terms.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * User-Land Trap-Based Tracing
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * ----------------------------
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * The fasttrap provider allows DTrace consumers to instrument any user-level
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * instruction to gather data; this includes probes with semantic
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * signifigance like entry and return as well as simple offsets into the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * function. While the specific techniques used are very ISA specific, the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * methodology is generalizable to any architecture.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * The General Methodology
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * -----------------------
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * With the primary goal of tracing every user-land instruction and the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * limitation that we can't trust user space so don't want to rely on much
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * information there, we begin by replacing the instructions we want to trace
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * with trap instructions. Each instruction we overwrite is saved into a hash
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * table keyed by process ID and pc address. When we enter the kernel due to
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * this trap instruction, we need the effects of the replaced instruction to
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * appear to have occurred before we proceed with the user thread's
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * execution.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Each user level thread is represented by a ulwp_t structure which is
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * always easily accessible through a register. The most basic way to produce
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * the effects of the instruction we replaced is to copy that instruction out
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * to a bit of scratch space reserved in the user thread's ulwp_t structure
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * (a sort of kernel-private thread local storage), set the PC to that
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * scratch space and single step. When we reenter the kernel after single
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * stepping the instruction we must then adjust the PC to point to what would
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * normally be the next instruction. Of course, special care must be taken
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * for branches and jumps, but these represent such a small fraction of any
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * instruction set that writing the code to emulate these in the kernel is
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * not too difficult.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Return probes may require several tracepoints to trace every return site,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * and, conversely, each tracepoint may activate several probes (the entry
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * and offset 0 probes, for example). To solve this muliplexing problem,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * tracepoints contain lists of probes to activate and probes contain lists
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * of tracepoints to enable. If a probe is activated, it adds its ID to
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * existing tracepoints or creates new ones as necessary.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Most probes are activated _before_ the instruction is executed, but return
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * probes are activated _after_ the effects of the last instruction of the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * function are visible. Return probes must be fired _after_ we have
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * single-stepped the instruction whereas all other probes are fired
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * beforehand.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Lock Ordering
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * -------------
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * The lock ordering below -- both internally and with respect to the DTrace
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * framework -- is a little tricky and bears some explanation. Each provider
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * has a lock (ftp_mtx) that protects its members including reference counts
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * for enabled probes (ftp_rcount), consumers actively creating probes
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * (ftp_ccount) and USDT consumers (ftp_mcount); all three prevent a provider
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * from being freed. A provider is looked up by taking the bucket lock for the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * provider hash table, and is returned with its lock held. The provider lock
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * may be taken in functions invoked by the DTrace framework, but may not be
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * held while calling functions in the DTrace framework.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * To ensure consistency over multiple calls to the DTrace framework, the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * creation lock (ftp_cmtx) should be held. Naturally, the creation lock may
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * not be taken when holding the provider lock as that would create a cyclic
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * lock ordering. In situations where one would naturally take the provider
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * lock and then the creation lock, we instead up a reference count to prevent
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * the provider from disappearing, drop the provider lock, and acquire the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * creation lock.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * bucket lock before provider lock
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * DTrace before provider lock
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * creation lock before DTrace
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * never hold the provider lock and creation lock simultaneously
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Generation count on modifications to the global tracepoint lookup table.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * When the fasttrap provider is loaded, fasttrap_max is set to either
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * FASTTRAP_MAX_DEFAULT or the value for fasttrap-max-probes in the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * fasttrap.conf file. Each time a probe is created, fasttrap_total is
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * incremented by the number of tracepoints that may be associated with that
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * probe; fasttrap_total is capped at fasttrap_max.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncstatic uint64_t fasttrap_pid_count; /* pid ref count */
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncstatic kmutex_t fasttrap_count_mtx; /* lock on ref count */
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncstatic int fasttrap_tracepoint_enable(proc_t *, fasttrap_probe_t *, uint_t);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncstatic void fasttrap_tracepoint_disable(proc_t *, fasttrap_probe_t *, uint_t);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncstatic fasttrap_provider_t *fasttrap_provider_lookup(pid_t, const char *,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncstatic void fasttrap_provider_retire(pid_t, const char *, int);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncstatic void fasttrap_provider_free(fasttrap_provider_t *);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncstatic fasttrap_proc_t *fasttrap_proc_lookup(pid_t);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncstatic void fasttrap_proc_release(fasttrap_proc_t *);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync ((fasttrap_hash_str(name) + (pid)) & fasttrap_provs.fth_mask)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync#define FASTTRAP_PROCS_INDEX(pid) ((pid) & fasttrap_procs.fth_mask)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (i == 0)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync return (0);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (i & 0xffffffff00000000ul) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (i & 0xffff0000) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (i & 0xff00) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (i & 0xf0) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (i & 0xc) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (i & 0x2) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync return (h);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync unsigned int g;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync while (*p) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_sigtrap(proc_t *p, kthread_t *t, uintptr_t pc)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync sigqueue_t *sqp = kmem_zalloc(sizeof (sigqueue_t), KM_SLEEP);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * This function ensures that no threads are actively using the memory
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * associated with probes that were formerly live.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync for (i = 0; i < NCPU; i++) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * This is the timeout's callback for cleaning up the providers and their
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync/*ARGSUSED*/
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Iterate over all the providers trying to remove the marked
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * ones. If a provider is marked but not retired, we just
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * have to take a crack at removing it -- it's no big deal if
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * we can't.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If this provider has consumers actively
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * creating probes (ftp_ccount) or is a USDT
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * provider (ftp_mcount), we can't unregister
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * or even condense.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If we successfully unregister this
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * provider we can remove it from the hash
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * chain and free the memory. If our attempt
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * to unregister fails and this is a retired
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * provider, increment our flag to try again
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * pretty soon. If we've consumed more than
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * half of our total permitted number of
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * probes call dtrace_condense() to try to
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * clean out the unenabled probes.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If we were unable to remove a retired provider, try again after
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * a second. This situation can occur in certain circumstances where
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * providers cannot be unregistered even though they have no probes
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * enabled because of an execution of dtrace -l or something similar.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If the timeout has been disabled (set to 1 because we're trying
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * to detach), we set fasttrap_cleanup_work to ensure that we'll
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * get a chance to do that work if and when the timeout is reenabled
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * (if detach fails).
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (later > 0 && fasttrap_timeout != (timeout_id_t)1)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_timeout = timeout(&fasttrap_pid_cleanup_cb, NULL, hz);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync else if (later > 0)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Activates the asynchronous cleanup mechanism.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_timeout = timeout(&fasttrap_pid_cleanup_cb, NULL, 1);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * This is called from cfork() via dtrace_fasttrap_fork(). The child
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * process's address space is (roughly) a copy of the parent process's so
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * we have to remove all the instrumentation we had previously enabled in the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * This would be simpler and faster if we maintained per-process
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * hash tables of enabled tracepoints. It could, however, potentially
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * slow down execution of a tracepoint since we'd need to go
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * through two levels of indirection. In the future, we should
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * consider either maintaining per-process ancillary lists of
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * enabled tracepoints or hanging a pointer to a per-process hash
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * table of enabled tracepoints off the proc structure.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * We don't have to worry about the child process disappearing
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * because we're in fork().
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Iterate over every tracepoint looking for ones that belong to the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * parent process, and remove each from the child process.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_bucket_t *bucket = &fasttrap_tpoints.fth_table[i];
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync for (tp = bucket->ftb_data; tp != NULL; tp = tp->ftt_next) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * The count of active providers can only be
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * decremented (i.e. to zero) during exec,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * exit, and removal of a meta provider so it
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * should be impossible to drop the count
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * mid-fork.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * This is called from proc_exit() or from exec_common() if p_dtrace_probes
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * is set on the proc structure to indicate that there is a pid provider
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * associated with this process.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * We clean up the pid provider for this process here; user-land
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * static probes are handled by the meta-provider remove entry point.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_provider_retire(p->p_pid, FASTTRAP_PID_NAME, 0);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync/*ARGSUSED*/
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_pid_provide(void *arg, const dtrace_probedesc_t *desc)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * There are no "default" pid probes.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_tracepoint_enable(proc_t *p, fasttrap_probe_t *probe, uint_t index)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync ASSERT(probe->ftp_tps[index].fit_tp->ftt_pid == pid);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Before we make any modifications, make sure we've imposed a barrier
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * on the generation in which this probe was last modified.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync bucket = &fasttrap_tpoints.fth_table[FASTTRAP_TPOINTS_INDEX(pid, pc)];
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If the tracepoint has already been enabled, just add our id to the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * list of interested probes. This may be our second time through
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * this path in which case we'll have constructed the tracepoint we'd
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * like to install. If we can't find a match, and have an allocated
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * tracepoint ready to go, enable that one now.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * A tracepoint whose process is defunct is also considered defunct.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync for (tp = bucket->ftb_data; tp != NULL; tp = tp->ftt_next) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Note that it's safe to access the active count on the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * associated proc structure because we know that at least one
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * provider (this one) will still be around throughout this
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * operation.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Now that we've found a matching tracepoint, it would be
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * a decent idea to confirm that the tracepoint is still
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * enabled and the trap instruction hasn't been overwritten.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Since this is a little hairy, we'll punt for now.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * This can't be the first interested probe. We don't have
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * to worry about another thread being in the midst of
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * deleting this tracepoint (which would be the only valid
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * reason for a tracepoint to have no interested probes)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * since we're holding P_PR_LOCK for this process.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync ASSERT(tp->ftt_ids != NULL || tp->ftt_retids != NULL);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync return (0);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If we have a good tracepoint ready to go, install it now while
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * we have the lock held and no one can screw with us.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Activate the tracepoint in the ISA-specific manner.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If this fails, we need to report the failure, but
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * indicate that this tracepoint must still be disabled
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * by calling fasttrap_tracepoint_disable().
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Increment the count of the number of tracepoints active in
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * the victim process.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Initialize the tracepoint that's been preallocated with the probe.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync ASSERT(new_tp->ftt_proc == probe->ftp_prov->ftp_proc);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If the ISA-dependent initialization goes to plan, go back to the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * beginning and try to install this freshly made tracepoint.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (fasttrap_tracepoint_init(p, new_tp, pc, id->fti_ptype) == 0)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_tracepoint_disable(proc_t *p, fasttrap_probe_t *probe, uint_t index)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync ASSERT(probe->ftp_tps[index].fit_tp->ftt_pid == pid);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Find the tracepoint and make sure that our id is one of the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * ones registered with it.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync bucket = &fasttrap_tpoints.fth_table[FASTTRAP_TPOINTS_INDEX(pid, pc)];
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync for (tp = bucket->ftb_data; tp != NULL; tp = tp->ftt_next) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If we somehow lost this tracepoint, we're in a world of hurt.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If there are other registered enablings of this tracepoint, we're
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * all done, but if this was the last probe assocated with this
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * this tracepoint, we need to remove and free it.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (tp->ftt_ids != NULL || tp->ftt_retids != NULL) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If the current probe's tracepoint is in use, swap it
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * for an unused tracepoint.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync /* LINTED - alignment */
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync /* LINTED - alignment */
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Tag the modified probe with the generation in which it was
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * We can't safely remove the tracepoint from the set of active
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * tracepoints until we've actually removed the fasttrap instruction
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * from the process's text. We can, however, operate on this
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * tracepoint secure in the knowledge that no other thread is going to
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * be looking at it since we hold P_PR_LOCK on the process if it's
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * live or we hold the provider lock on the process if it's dead and
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * We only need to remove the actual instruction if we're looking
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * at an existing process
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (p != NULL) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If we fail to restore the instruction we need to kill
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * this process since it's in a completely unrecoverable
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Decrement the count of the number of tracepoints active
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * in the victim process.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Remove the probe from the hash table of active tracepoints.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Tag the modified probe with the generation in which it was changed.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * We don't have to play the rw lock game here because we're
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * providing something rather than taking something away --
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * we can be sure that no threads have tried to follow this
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * function pointer yet.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync ASSERT(dtrace_pid_probe_ptr == &fasttrap_pid_probe);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync ASSERT(dtrace_return_probe_ptr == &fasttrap_return_probe);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync/*ARGSUSED*/
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_pid_enable(void *arg, dtrace_id_t id, void *parg)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Increment the count of enabled probes on this probe's provider;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * the provider can't go away while the probe still exists. We
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * must increment this even if we aren't able to properly enable
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * this probe.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If this probe's provider is retired (meaning it was valid in a
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * previously exec'ed incarnation of this address space), bail out. The
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * provider can't go away while we're in this code path.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync return (0);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If we can't find the process, it may be that we're in the context of
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * a fork in which the traced process is being born and we're copying
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * USDT probes. Otherwise, the process is gone so bail.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync return (0);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Confirm that curproc is indeed forking the process in which
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * we're trying to enable probes.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * We have to enable the trap entry point before any user threads have
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * the chance to execute the trap instruction we're about to place
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * in their process's text.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Enable all the tracepoints and add this probe's id to each
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * tracepoint's list of active probes.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if ((rc = fasttrap_tracepoint_enable(p, probe, i)) != 0) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If enabling the tracepoint failed completely,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * we don't have to disable it; if the failure
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * was only partial we must disable it.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Back up and pull out all the tracepoints we've
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * created so far for this probe.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync while (i >= 0) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Since we're not actually enabling this probe,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * drop our reference on the trap table entry.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync return (0);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync return (0);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync/*ARGSUSED*/
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_pid_disable(void *arg, dtrace_id_t id, void *parg)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * We won't be able to acquire a /proc-esque lock on the process
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * iff the process is dead and gone. In this case, we rely on the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * provider lock as a point of mutual exclusion to prevent other
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * DTrace consumers from disabling this probe.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Disable all the associated tracepoints (for fully enabled probes).
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (p != NULL) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Even though we may not be able to remove it entirely, we
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * mark this retired provider to get a chance to remove some
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * of the associated probes.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (provider->ftp_retired && !provider->ftp_marked)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If the process is dead, we're just waiting for the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * last probe to be disabled to be able to free it.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (provider->ftp_rcount == 0 && !provider->ftp_marked)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync/*ARGSUSED*/
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_pid_getargdesc(void *arg, dtrace_id_t id, void *parg,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync probe->ftp_argmap[desc->dtargd_ndx] : desc->dtargd_ndx;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync for (i = 0; i < ndx; i++) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync ASSERT(strlen(str + 1) < sizeof (desc->dtargd_native));
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync ASSERT(strlen(str + 1) < sizeof (desc->dtargd_xlate));
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync/*ARGSUSED*/
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_pid_destroy(void *arg, dtrace_id_t id, void *parg)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync size = offsetof(fasttrap_probe_t, ftp_tps[probe->ftp_ntps]);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync{ DTRACE_STABILITY_EVOLVING, DTRACE_STABILITY_EVOLVING, DTRACE_CLASS_ISA },
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync{ DTRACE_STABILITY_PRIVATE, DTRACE_STABILITY_PRIVATE, DTRACE_CLASS_UNKNOWN },
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync{ DTRACE_STABILITY_PRIVATE, DTRACE_STABILITY_PRIVATE, DTRACE_CLASS_UNKNOWN },
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync{ DTRACE_STABILITY_EVOLVING, DTRACE_STABILITY_EVOLVING, DTRACE_CLASS_ISA },
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync{ DTRACE_STABILITY_PRIVATE, DTRACE_STABILITY_PRIVATE, DTRACE_CLASS_UNKNOWN },
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync bucket = &fasttrap_procs.fth_table[FASTTRAP_PROCS_INDEX(pid)];
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync for (fprc = bucket->ftb_data; fprc != NULL; fprc = fprc->ftpc_next) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (fprc->ftpc_pid == pid && fprc->ftpc_acount != 0) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Drop the bucket lock so we don't try to perform a sleeping
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * allocation under it.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync new_fprc = kmem_zalloc(sizeof (fasttrap_proc_t), KM_SLEEP);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Take another lap through the list to make sure a proc hasn't
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * been created for this pid while we weren't under the bucket lock.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync for (fprc = bucket->ftb_data; fprc != NULL; fprc = fprc->ftpc_next) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (fprc->ftpc_pid == pid && fprc->ftpc_acount != 0) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * There should definitely be no live providers associated with this
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * process at this point.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync bucket = &fasttrap_procs.fth_table[FASTTRAP_PROCS_INDEX(pid)];
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Something strange has happened if we can't find the proc.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Lookup a fasttrap-managed provider based on its name and associated pid.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If the pattr argument is non-NULL, this function instantiates the provider
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * if it doesn't exist otherwise it returns NULL. The provider is returned
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * with its lock held.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_provider_lookup(pid_t pid, const char *name,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync bucket = &fasttrap_provs.fth_table[FASTTRAP_PROVS_INDEX(pid, name)];
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Take a lap through the list and return the match if we find it.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync for (fp = bucket->ftb_data; fp != NULL; fp = fp->ftp_next) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (fp->ftp_pid == pid && strcmp(fp->ftp_name, name) == 0 &&
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Drop the bucket lock so we don't try to perform a sleeping
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * allocation under it.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Make sure the process exists, isn't a child created as the result
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * of a vfork(2), and isn't a zombie (but may be in fork).
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Increment p_dtrace_probes so that the process knows to inform us
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * when it exits or execs. fasttrap_provider_free() decrements this
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * when we're done with this provider.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Grab the credentials for this process so we have
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * something to pass to dtrace_register().
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync new_fp = kmem_zalloc(sizeof (fasttrap_provider_t), KM_SLEEP);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Take another lap through the list to make sure a provider hasn't
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * been created for this pid while we weren't under the bucket lock.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync for (fp = bucket->ftb_data; fp != NULL; fp = fp->ftp_next) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (fp->ftp_pid == pid && strcmp(fp->ftp_name, name) == 0 &&
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Fail and return NULL if either the provider name is too long
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * or we fail to register this new provider with the DTrace
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * framework. Note that this is the only place we ever construct
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * the full provider name -- we keep it in pieces in the provider
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * structure.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (snprintf(provname, sizeof (provname), "%s%u", name, (uint_t)pid) >=
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync DTRACE_PRIV_PROC | DTRACE_PRIV_OWNER | DTRACE_PRIV_ZONEOWNER, cred,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync pattr == &pid_attr ? &pid_pops : &usdt_pops, new_fp,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_provider_free(fasttrap_provider_t *provider)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * There need to be no associated enabled probes, no consumers
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * creating probes, and no meta providers referencing this provider.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If this provider hasn't been retired, we need to explicitly drop the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * count of active providers on the associated process structure.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync atomic_add_64(&provider->ftp_proc->ftpc_acount, -1);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Decrement p_dtrace_probes on the process whose provider we're
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * freeing. We don't have to worry about clobbering somone else's
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * modifications to it because we have locked the bucket that
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * corresponds to this process's hash chain in the provider hash
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * table. Don't sweat it if we can't find the process.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_provider_retire(pid_t pid, const char *name, int mprov)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync bucket = &fasttrap_provs.fth_table[FASTTRAP_PROVS_INDEX(pid, name)];
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync for (fp = bucket->ftb_data; fp != NULL; fp = fp->ftp_next) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (fp->ftp_pid == pid && strcmp(fp->ftp_name, name) == 0 &&
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Mark the provider to be removed in our post-processing step, mark it
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * retired, and drop the active count on its proc. Marking it indicates
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * that we should try to remove it; setting the retired flag indicates
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * that we're done with this provider; dropping the active the proc
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * releases our hold, and when this reaches zero (as it will during
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * exit or exec) the proc and associated providers become defunct.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * We obviously need to take the bucket lock before the provider lock
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * to perform the lookup, but we need to drop the provider lock
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * before calling into the DTrace framework since we acquire the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * provider lock in callbacks invoked from the DTrace framework. The
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * bucket lock therefore protects the integrity of the provider hash
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync ASSERT(fp->ftp_proc->ftpc_acount < fp->ftp_proc->ftpc_rcount);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * We don't have to worry about invalidating the same provider twice
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * since fasttrap_provider_lookup() will ignore provider that have
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * been marked as retired.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync return (*(const uint32_t *)ap - *(const uint32_t *)bp);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync return (*(const uint64_t *)ap - *(const uint64_t *)bp);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * There needs to be at least one desired trace point.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if ((provider = fasttrap_provider_lookup(pdata->ftps_pid,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Increment this reference count to indicate that a consumer is
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * actively adding a new probe associated with this provider. This
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * prevents the provider from being deleted -- we'll need to check
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * for pending deletions when we drop this reference count.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Grab the creation lock to ensure consistency between calls to
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * dtrace_probe_lookup() and dtrace_probe_create() in the face of
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * other threads creating probes. We must drop the provider lock
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * before taking this lock to avoid a three-way deadlock with the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * DTrace framework.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync pp = kmem_zalloc(sizeof (fasttrap_probe_t), KM_SLEEP);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync pp->ftp_tps[0].fit_id.fti_ptype = pdata->ftps_type;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync pp->ftp_id = dtrace_probe_create(provider->ftp_provid,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync } else if (dtrace_probe_lookup(provider->ftp_provid, pdata->ftps_mod,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync atomic_add_32(&fasttrap_total, -pdata->ftps_noffs);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Make sure all tracepoint program counter values are unique.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * We later assume that each probe has exactly one tracepoint
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * for a given pc.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync atomic_add_32(&fasttrap_total, -pdata->ftps_noffs);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync pp->ftp_tps[i].fit_id.fti_ptype = pdata->ftps_type;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync pp->ftp_id = dtrace_probe_create(provider->ftp_provid,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync pdata->ftps_mod, pdata->ftps_func, name, aframes, pp);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * We know that the provider is still valid since we incremented the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * creation reference count. If someone tried to clean up this provider
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * while we were using it (e.g. because the process called exec(2) or
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * exit(2)), take note of that and try to clean it up now.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync return (0);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If we've exhausted the allowable resources, we'll try to remove
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * this provider to free some up. This is to cover the case where
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * the user has accidentally created many more probes than was
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * intended (e.g. pid123:::).
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync/*ARGSUSED*/
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncstatic void *
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_meta_provide(void *arg, dtrace_helper_provdesc_t *dhpv, pid_t pid)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * A 32-bit unsigned integer (like a pid for example) can be
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * expressed in 10 or fewer decimal digits. Make sure that we'll
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * have enough space for the provider name.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync cmn_err(CE_WARN, "failed to instantiate provider %s: "
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync "name too long to accomodate pid", dhpv->dthpv_provname);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Don't let folks spoof the true pid provider.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (strcmp(dhpv->dthpv_provname, FASTTRAP_PID_NAME) == 0) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync cmn_err(CE_WARN, "failed to instantiate provider %s: "
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * The highest stability class that fasttrap supports is ISA; cap
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * the stability of the new provider accordingly.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (dhpv->dthpv_pattr.dtpa_provider.dtat_class > DTRACE_CLASS_ISA)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync dhpv->dthpv_pattr.dtpa_provider.dtat_class = DTRACE_CLASS_ISA;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (dhpv->dthpv_pattr.dtpa_mod.dtat_class > DTRACE_CLASS_ISA)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync dhpv->dthpv_pattr.dtpa_mod.dtat_class = DTRACE_CLASS_ISA;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (dhpv->dthpv_pattr.dtpa_func.dtat_class > DTRACE_CLASS_ISA)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync dhpv->dthpv_pattr.dtpa_func.dtat_class = DTRACE_CLASS_ISA;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (dhpv->dthpv_pattr.dtpa_name.dtat_class > DTRACE_CLASS_ISA)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync dhpv->dthpv_pattr.dtpa_name.dtat_class = DTRACE_CLASS_ISA;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (dhpv->dthpv_pattr.dtpa_args.dtat_class > DTRACE_CLASS_ISA)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync dhpv->dthpv_pattr.dtpa_args.dtat_class = DTRACE_CLASS_ISA;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if ((provider = fasttrap_provider_lookup(pid, dhpv->dthpv_provname,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync cmn_err(CE_WARN, "failed to instantiate provider %s for "
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Up the meta provider count so this provider isn't removed until
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * the meta provider has been told to remove it.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync/*ARGSUSED*/
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Since the meta provider count is non-zero we don't have to worry
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * about this provider disappearing.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * The offsets must be unique.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync qsort(dhpb->dthpb_offs, dhpb->dthpb_noffs, sizeof (uint32_t),
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync qsort(dhpb->dthpb_enoffs, dhpb->dthpb_nenoffs, sizeof (uint32_t),
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Grab the creation lock to ensure consistency between calls to
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * dtrace_probe_lookup() and dtrace_probe_create() in the face of
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * other threads creating probes.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (dtrace_probe_lookup(provider->ftp_provid, dhpb->dthpb_mod,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync pp = kmem_zalloc(offsetof(fasttrap_probe_t, ftp_tps[ntps]), KM_SLEEP);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * First create a tracepoint for each actual point of interest.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync tp = kmem_zalloc(sizeof (fasttrap_tracepoint_t), KM_SLEEP);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync tp->ftt_pc = dhpb->dthpb_base + dhpb->dthpb_offs[i];
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync pp->ftp_tps[i].fit_id.fti_ptype = DTFTP_POST_OFFSETS;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Then create a tracepoint for each is-enabled point.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync for (j = 0; i < ntps; i++, j++) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync tp = kmem_zalloc(sizeof (fasttrap_tracepoint_t), KM_SLEEP);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync tp->ftt_pc = dhpb->dthpb_base + dhpb->dthpb_enoffs[j];
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync pp->ftp_tps[i].fit_id.fti_ptype = DTFTP_IS_ENABLED;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If the arguments are shuffled around we set the argument remapping
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * table. Later, when the probe fires, we only remap the arguments
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * if the table is non-NULL.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * The probe is fully constructed -- register it with DTrace.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync pp->ftp_id = dtrace_probe_create(provider->ftp_provid, dhpb->dthpb_mod,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync dhpb->dthpb_func, dhpb->dthpb_name, FASTTRAP_OFFSET_AFRAMES, pp);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync/*ARGSUSED*/
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_meta_remove(void *arg, dtrace_helper_provdesc_t *dhpv, pid_t pid)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Clean up the USDT provider. There may be active consumers of the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * provider busy adding probes, no damage will actually befall the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * provider until that count has dropped to zero. This just puts
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * the provider on death row.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_provider_retire(pid, dhpv->dthpv_provname, 1);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync/*ARGSUSED*/
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_open(dev_t *devp, int flag, int otyp, cred_t *cred_p)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync return (0);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync/*ARGSUSED*/
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_ioctl(dev_t dev, int cmd, intptr_t arg, int md, cred_t *cr, int *rv)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Probes must have at least one tracepoint.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Verify that the function and module strings contain no
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * funny characters.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Report an error if the process doesn't exist
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * or is actively being birthed.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if ((p = prfind(pid)) == NULL || p->p_stat == SIDL) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (copyin((void *)arg, &instr, sizeof (instr)) != 0)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Report an error if the process doesn't exist
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * or is actively being birthed.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if ((p = prfind(pid)) == NULL || p->p_stat == SIDL) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync index = FASTTRAP_TPOINTS_INDEX(instr.ftiq_pid, instr.ftiq_pc);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync mutex_enter(&fasttrap_tpoints.fth_table[index].ftb_mtx);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync mutex_exit(&fasttrap_tpoints.fth_table[index].ftb_mtx);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync mutex_exit(&fasttrap_tpoints.fth_table[index].ftb_mtx);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (copyout(&instr, (void *)arg, sizeof (instr)) != 0)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync return (0);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync 0, /* streamtab */
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync/*ARGSUSED*/
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_info(dev_info_t *dip, ddi_info_cmd_t infocmd, void *arg, void **result)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync *result = (void *)0;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_attach(dev_info_t *devi, ddi_attach_cmd_t cmd)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (ddi_create_minor_node(devi, "fasttrap", S_IFCHR, 0,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Install our hooks into fork(2), exec(2), and exit(2).
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_max = ddi_getprop(DDI_DEV_T_ANY, devi, DDI_PROP_DONTPASS,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Conjure up the tracepoints hashtable...
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync nent = ddi_getprop(DDI_DEV_T_ANY, devi, DDI_PROP_DONTPASS,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync "fasttrap-hash-size", FASTTRAP_TPOINTS_DEFAULT_SIZE);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_tpoints.fth_nent = 1 << fasttrap_highbit(nent);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_tpoints.fth_mask = fasttrap_tpoints.fth_nent - 1;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_tpoints.fth_table = kmem_zalloc(fasttrap_tpoints.fth_nent *
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * ... and the providers hash table...
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_provs.fth_nent = 1 << fasttrap_highbit(nent);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_provs.fth_mask = fasttrap_provs.fth_nent - 1;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_provs.fth_table = kmem_zalloc(fasttrap_provs.fth_nent *
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * ... and the procs hash table.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_procs.fth_nent = 1 << fasttrap_highbit(nent);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_procs.fth_mask = fasttrap_procs.fth_nent - 1;
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_procs.fth_table = kmem_zalloc(fasttrap_procs.fth_nent *
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync (void) dtrace_meta_register("fasttrap", &fasttrap_mops, NULL,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsyncfasttrap_detach(dev_info_t *devi, ddi_detach_cmd_t cmd)
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Unregister the meta-provider to make sure no new fasttrap-
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * managed providers come along while we're trying to close up
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * shop. If we fail to detach, we'll need to re-register as a
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * meta-provider. We can fail to unregister as a meta-provider
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * if providers we manage still exist.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Prevent any new timeouts from running by setting fasttrap_timeout
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * to a non-zero value, and wait for the current timeout to complete.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync if (tmp != 0) {
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Iterate over all of our providers. If there's still a process
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * that corresponds to that pid, fail to detach.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_bucket_t *bucket = &fasttrap_provs.fth_table[i];
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Acquire and release the lock as a simple way of
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * waiting for any other consumer to finish with
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * this provider. A thread must first acquire the
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * bucket lock so there's no chance of another thread
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * blocking on the provider's lock.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * If we're failing to detach, we need to unblock timeouts
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * and start a new timeout if any work has accumulated while
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * we've been unsuccessfully trying to detach.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync (void) dtrace_meta_register("fasttrap", &fasttrap_mops, NULL,
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_tpoints.fth_nent * sizeof (fasttrap_bucket_t));
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_provs.fth_nent * sizeof (fasttrap_bucket_t));
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync fasttrap_procs.fth_nent * sizeof (fasttrap_bucket_t));
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * We know there are no tracepoints in any process anywhere in
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * the system so there is no process which has its p_dtrace_count
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * greater than zero, therefore we know that no thread can actively
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * be executing code in fasttrap_fork(). Similarly for p_dtrace_probes
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * and fasttrap_exec() and fasttrap_exit().
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync ASSERT(dtrace_fasttrap_fork_ptr == &fasttrap_fork);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync ASSERT(dtrace_fasttrap_exec_ptr == &fasttrap_exec_exit);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync ASSERT(dtrace_fasttrap_exit_ptr == &fasttrap_exec_exit);
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync 0, /* refcnt */
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync * Module linkage information for the kernel.
14ea49401f3c8c61422aefbda43809e275f60c6cvboxsync &mod_driverops, /* module type (this is a pseudo driver) */