b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * CDDL HEADER START
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * The contents of this file are subject to the terms of the
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Common Development and Distribution License (the "License").
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * You may not use this file except in compliance with the License.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * or http://www.opensolaris.org/os/licensing.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * See the License for the specific language governing permissions
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * and limitations under the License.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * When distributing Covered Code, include this CDDL HEADER in each
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * If applicable, add the following below this CDDL HEADER, with the
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * fields enclosed by brackets "[]" replaced with your own identifying
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * information: Portions Copyright [yyyy] [name of copyright owner]
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * CDDL HEADER END
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * Copyright 2010 Sun Microsystems, Inc. All rights reserved.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Use is subject to license terms.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * DTrace CPU Performance Counter Provider
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * ---------------------------------------
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * The DTrace cpc provider allows DTrace consumers to access the CPU
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * performance counter overflow mechanism of a CPU. The configuration
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * presented in a probe specification is programmed into the performance
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * counter hardware of all available CPUs on a system. Programming the
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * hardware causes a counter on each CPU to begin counting events of the
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * given type. When the specified number of events have occurred, an overflow
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * interrupt will be generated and the probe is fired.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * The required configuration for the performance counter is encoded into
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * the probe specification and this includes the performance counter event
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * name, processor mode, overflow rate and an optional unit mask.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Most processors provide several counters (PICs) which can count all or a
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * subset of the events available for a given CPU. However, when overflow
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * profiling is being used, not all CPUs can detect which counter generated the
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * overflow interrupt. In this case we cannot reliably determine which counter
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * overflowed and we therefore only allow such CPUs to configure one event at
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * a time. Processors that can determine the counter which overflowed are
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * allowed to program as many events at one time as possible (in theory up to
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * the number of instrumentation counters supported by that platform).
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Therefore, multiple consumers can enable multiple probes at the same time
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * on such platforms. Platforms which cannot determine the source of an
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * overflow interrupt are only allowed to program a single event at one time.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * The performance counter hardware is made available to consumers on a
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * first-come, first-served basis. Only a finite amount of hardware resource
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * is available and, while we make every attempt to accomodate requests from
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * consumers, we must deny requests when hardware resources have been exhausted.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * A consumer will fail to enable probes when resources are currently in use.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * The cpc provider contends for shared hardware resources along with other
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * consumers of the kernel CPU performance counter subsystem (e.g. cpustat(1M)).
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Only one such consumer can use the performance counters at any one time and
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * counters are made available on a first-come, first-served basis. As with
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * cpustat, the cpc provider has priority over per-LWP libcpc usage (e.g.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * cputrack(1)). Invoking the cpc provider will cause all existing per-LWP
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * counter contexts to be invalidated.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam int64_t dcpc_umask; /* umask/emask for this event */
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam int dcpc_picno; /* pic this event is programmed in */
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam int dcpc_enabled; /* probe is actually enabled? */
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam int dcpc_disabling; /* probe is currently being disabled */
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam dtrace_id_t dcpc_id; /* probeid this request is enabling */
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam int dcpc_actv_req_idx; /* idx into dcpc_actv_reqs[] */
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * When the dcpc provider is loaded, dcpc_min_overflow is set to either
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * DCPC_MIN_OVF_DEFAULT or the value that dcpc-min-overflow is set to in
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * the dcpc.conf file. Decrease this value to set probes with smaller
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * overflow values. Remember that very small values could render a system
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * unusable with frequently occurring events.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslamstatic int dcpc_aframes = 0; /* override for artificial frame setting */
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Called from the platform overflow interrupt handler. 'bitmap' is a mask
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * which contains the pic(s) that have overflowed.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * No counter was marked as overflowing. Shout about it and get out.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam cmn_err(CE_NOTE, "dcpc_fire: no counter overflow found\n");
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * This is the common case of a processor that doesn't support
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * multiple overflow events. Such systems are only allowed a single
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * enabling and therefore we just look for the first entry in
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * the active request array.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam for (i = 0; i < cpc_ncounters; i++) {
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * This is a processor capable of handling multiple overflow events.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Iterate over the array of active requests and locate the counters
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * that overflowed (note: it is possible for more than one counter to
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * have overflowed at the same time).
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam for (i = 0; i < cpc_ncounters; i++) {
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam (bitmap & (1ULL << dcpc_actv_reqs[i]->dcpc_picno))) {
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslamdcpc_create_probe(dtrace_provider_id_t id, const char *probename,
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam char *eventname, int64_t umask, uint32_t ovfval, char flag)
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam int nr_frames = DCPC_ARTIFICIAL_FRAMES + dtrace_mach_aframes();
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam if (dtrace_probe_lookup(id, NULL, NULL, probename) != 0)
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam pp = kmem_zalloc(sizeof (dcpc_probe_t), KM_SLEEP);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam (void) strncpy(pp->dcpc_event_name, eventname,
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam pp->dcpc_event_name[sizeof (pp->dcpc_event_name) - 1] = '\0';
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam pp->dcpc_actv_req_idx = pp->dcpc_picno = pp->dcpc_disabling = -1;
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam pp->dcpc_id = dtrace_probe_create(id, NULL, NULL, probename,
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslamdcpc_provide(void *arg, const dtrace_probedesc_t *desc)
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * The format of a probe is:
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * event_name-mode-{optional_umask}-overflow_rate
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * DC_refill_from_system-user-0x1e-50000, or,
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * DC_refill_from_system-all-10000
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * The 'cpc' provider offers no probes by default.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * We have a poor man's strtok() going on here. Replace any hyphens
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * in the the probe name with NULL characters in order to make it
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * easy to parse the string with regular string functions.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam for (i = 0; i < len; i++) {
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * The first part of the string must be either a platform event
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * name or a generic event name.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam (void) strncpy(event, p, CPC_MAX_EVENT_LEN - 1);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * The next part of the name is the mode specification. Valid
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * settings are "user", "kernel" or "all".
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Next we either have a mask specification followed by an overflow
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * rate or just an overflow rate on its own.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam if (p[0] == '0' && (p[1] == 'x' || p[1] == 'X')) {
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * A unit mask can only be specified if:
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * 1) this performance counter back end supports masks.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * 2) the specified event is platform specific.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * 3) a valid hex number is converted.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * 4) no extraneous characters follow the mask specification.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam if (dcpc_mask_type != 0 && strncmp(event, "PAPI", 4) != 0 &&
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * This final part must be an overflow value which has to be greater
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * than the minimum permissible overflow rate.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam if ((ddi_strtol(p, &end, 10, &val) != 0) || end != p + strlen(p) ||
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Validate the event and create the probe.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam for (i = 0; i < cpc_ncounters; i++) {
8cb74972a66bde0af7b1a957d01e0095b82a8b91Jonathan Haslam for (j = 0; j < llen; j++) {
8cb74972a66bde0af7b1a957d01e0095b82a8b91Jonathan Haslam dcpc_create_probe(dcpc_pid, desc->dtpd_name,
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslamdcpc_destroy(void *arg, dtrace_id_t id, void *parg)
7d5c9b5fdaad32a290692afde7867bd051784395Bryan Cantrilldcpc_mode(void *arg, dtrace_id_t id, void *parg)
7d5c9b5fdaad32a290692afde7867bd051784395Bryan Cantrill return (DTRACE_MODE_NOPRIV_DROP | DTRACE_MODE_USER);
7d5c9b5fdaad32a290692afde7867bd051784395Bryan Cantrill return (DTRACE_MODE_NOPRIV_DROP | DTRACE_MODE_KERNEL);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslamdcpc_populate_set(cpu_t *c, dcpc_probe_t *pp, kcpc_set_t *set, int reqno)
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam (void) strncpy(set->ks_req[reqno].kr_event, pp->dcpc_event_name,
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam set->ks_req[reqno].kr_flags = pp->dcpc_flag;
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * If a unit mask has been specified then detect which attribute
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * the platform needs. For now, it's either "umask" or "emask".
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam kmem_zalloc(sizeof (kcpc_attr_t), KM_SLEEP);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam (void) strncpy(set->ks_req[reqno].kr_attr->ka_name,
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam (void) strncpy(set->ks_req[reqno].kr_attr->ka_name,
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam set->ks_req[reqno].kr_attr->ka_val = pp->dcpc_umask;
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * If this probe is enabled, obtain its current countdown value
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * and use that. The CPUs cpc context might not exist yet if we
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * are dealing with a CPU that is just coming online.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam if (pp->dcpc_enabled && (c->cpu_cpc_ctx != NULL)) {
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam set->ks_req[reqno].kr_preset = UINT64_MAX - pp->dcpc_ovfval;
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Create a fresh request set for the enablings represented in the
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * 'dcpc_actv_reqs' array which contains the probes we want to be
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * in the set. This can be called for several reasons:
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * 1) We are on a single or multi overflow platform and we have no
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * current events so we can just create the set and initialize it.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * 2) We are on a multi-overflow platform and we already have one or
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * more existing events and we are adding a new enabling. Create a
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * new set and copy old requests in and then add the new request.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * 3) We are on a multi-overflow platform and we have just removed an
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * enabling but we still have enablings whch are valid. Create a new
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * set and copy in still valid requests.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * First get a count of the number of currently active requests.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Note that dcpc_actv_reqs[] should always reflect which requests
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * we want to be in the set that is to be created. It is the
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * responsibility of the caller of dcpc_create_set() to adjust that
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * array accordingly beforehand.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam for (i = 0; i < cpc_ncounters; i++) {
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam set = kmem_zalloc(sizeof (kcpc_set_t), KM_SLEEP);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam kmem_zalloc(sizeof (kcpc_request_t) * active_requests, KM_SLEEP);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam kmem_zalloc(active_requests * sizeof (uint64_t), KM_SLEEP);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Look for valid entries in the active requests array and populate
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * the request set for any entries found.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam for (i = 0; i < cpc_ncounters; i++) {
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam dcpc_populate_set(c, dcpc_actv_reqs[i], set, reqno);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov set->ks_ctx = ctx = kcpc_ctx_alloc(KM_SLEEP);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam if (kcpc_configure_reqs(ctx, set, &subcode) != 0)
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam for (j = 0; j < cpc_ncounters; j++) {
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * If we already have an active enabling then save the current cpc
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * context away.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam kmem_free(oset->ks_data, oset->ks_nreqs * sizeof (uint64_t));
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * We failed to configure this request up so free things up and
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam kmem_free(set->ks_data, set->ks_nreqs * sizeof (uint64_t));
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Leave this CPU alone if it's already offline.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Grab CPUs CPC context before kcpc_cpu_stop() stops counters and
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam kmem_free(set->ks_data, set->ks_nreqs * sizeof (uint64_t));
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * The dcpc_*_interrupts() routines are responsible for manipulating the
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * per-CPU dcpc interrupt state byte. The purpose of the state byte is to
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * synchronize processing of hardware overflow interrupts wth configuration
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * changes made to the CPU performance counter subsystem by the dcpc provider.
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * The dcpc provider claims ownership of the overflow interrupt mechanism
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * by transitioning the state byte from DCPC_INTR_INACTIVE (indicating the
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * dcpc provider is not in use) to DCPC_INTR_FREE (the dcpc provider owns the
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * overflow mechanism and interrupts may be processed). Before modifying
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * a CPUs configuration state the state byte is transitioned from
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * DCPC_INTR_FREE to DCPC_INTR_CONFIG ("configuration in process" state).
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * The hardware overflow handler, kcpc_hw_overflow_intr(), will only process
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * an interrupt when a configuration is not in process (i.e. the state is
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * marked as free). During interrupt processing the state is set to
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * DCPC_INTR_PROCESSING by the overflow handler. When the last dcpc based
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * enabling is removed, the state byte is set to DCPC_INTR_INACTIVE to indicate
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * the dcpc provider is no longer interested in overflow interrupts.
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam ASSERT(cpu_core[c->cpu_id].cpuc_dcpc_intr_state != DCPC_INTR_INACTIVE);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam state = &cpu_core[c->cpu_id].cpuc_dcpc_intr_state;
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Set all CPUs dcpc interrupt state to DCPC_INTR_FREE to indicate that
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * overflow interrupts can be processed safely.
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam ASSERT(cpu_core[c->cpu_id].cpuc_dcpc_intr_state != DCPC_INTR_INACTIVE);
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam cpu_core[c->cpu_id].cpuc_dcpc_intr_state = DCPC_INTR_FREE;
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * Transition all CPUs dcpc interrupt state from DCPC_INTR_INACTIVE to
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * to DCPC_INTR_FREE. This indicates that the dcpc provider is now
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * responsible for handling all overflow interrupt activity. Should only be
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * called before enabling the first dcpc based probe.
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam ASSERT(cpu_core[c->cpu_id].cpuc_dcpc_intr_state == DCPC_INTR_INACTIVE);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam cpu_core[c->cpu_id].cpuc_dcpc_intr_state = DCPC_INTR_FREE;
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * Set all CPUs dcpc interrupt state to DCPC_INTR_INACTIVE to indicate that
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * the dcpc provider is no longer processing overflow interrupts. Only called
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * during removal of the last dcpc based enabling.
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam ASSERT(cpu_core[c->cpu_id].cpuc_dcpc_intr_state != DCPC_INTR_INACTIVE);
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam cpu_core[c->cpu_id].cpuc_dcpc_intr_state = DCPC_INTR_INACTIVE;
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * dcpc_program_event() can be called owing to a new enabling or if a multi
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * overflow platform has disabled a request but needs to program the requests
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * that are still valid.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Every invocation of dcpc_program_event() will create a new kcpc_ctx_t
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * and a new request set which contains the new enabling and any old enablings
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * which are still valid (possible with multi-overflow platforms).
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Skip CPUs that are currently offline.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Stop counters but preserve existing DTrace CPC context
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * if there is one.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * If we come here when the first event is programmed for a CPU,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * there should be no DTrace CPC context installed. In this
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * case, kcpc_cpu_stop() will ensure that there is no other
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * context on the CPU.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * If we add new enabling to the original one, the CPU should
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * have the old DTrace CPC context which we need to keep around
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * since dcpc_program_event() will add to it.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * If this enabling is being removed (in the case of a multi event
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * capable system with more than one active enabling), we can now
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * update the active request array to reflect the enablings that need
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * to be reprogrammed.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam dcpc_actv_reqs[pp->dcpc_actv_req_idx] = NULL;
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Skip CPUs that are currently offline.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam } while ((c = c->cpu_next) != cpu_list && ret == 0);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * If dcpc_program_cpu_event() fails then it is because we couldn't
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * configure the requests in the set for the CPU and not because of
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * an error programming the hardware. If we have a failure here then
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * we assume no CPUs have been programmed in the above step as they
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * are all configured identically.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslamdcpc_enable(void *arg, dtrace_id_t id, void *parg)
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Bail out if the counters are being used by a libcpc consumer.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Locate this enabling in the first free entry of the active
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * request array.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam for (i = 0; i < cpc_ncounters; i++) {
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * If we couldn't find a slot for this probe then there is no
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * room at the inn.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * DTrace is taking over CPC contexts, so stop collecting
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * capacity/utilization data for all CPUs.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * The following must hold true if we are to (attempt to) enable
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * this request:
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * 1) No enablings currently exist. We allow all platforms to
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * proceed if this is true.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * 2) If the platform is multi overflow capable and there are
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * less valid enablings than there are counters. There is no
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * guarantee that a platform can accommodate as many events as
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * it has counters for but we will at least try to program
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * up to that many requests.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * The 'dcpc_enablings' variable is implictly protected by locking
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * provided by the DTrace framework and the cpu management framework.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam if (dcpc_enablings == 0 || (dcpc_mult_ovf_cap &&
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Before attempting to program the first enabling we need to
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * invalidate any lwp-based contexts and lay claim to the
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * overflow interrupt mechanism.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * If active enablings existed before we failed to enable this probe
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * on a multi event capable platform then we need to restart counters
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * as they will have been stopped in the attempted configuration. The
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * context should now just contain the request prior to this failed
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam if (dcpc_enablings > 0 && dcpc_mult_ovf_cap) {
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Skip CPUs that are currently offline.
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * Give up any claim to the overflow interrupt mechanism if no
af4595ed84fc11745bc4136f8510c528996355f6Jonathan Haslam * dcpc based enablings exist.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam dcpc_actv_reqs[pp->dcpc_actv_req_idx] = NULL;
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam pp->dcpc_actv_req_idx = pp->dcpc_picno = -1;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * If all probes are removed, enable capacity/utilization data
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * collection for every CPU.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * If only one enabling is active then remove the context and free
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * everything up. If there are multiple enablings active then remove this
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * one, its associated meta-data and re-program the hardware.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslamdcpc_disable(void *arg, dtrace_id_t id, void *parg)
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * This probe didn't actually make it as far as being fully enabled
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * so we needn't do anything with it.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * If we actually allocated this request a slot in the
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * request array but failed to enabled it then remove the
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * entry in the array.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam dcpc_actv_reqs[pp->dcpc_actv_req_idx] = NULL;
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * If this is the only enabling then stop all the counters and
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * free up the meta-data.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam dcpc_actv_reqs[pp->dcpc_actv_req_idx] = NULL;
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * This platform can support multiple overflow events and
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * the enabling being disabled is not the last one. Remove this
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * enabling and re-program the hardware with the new config.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam pp->dcpc_actv_req_idx = pp->dcpc_picno = pp->dcpc_disabling = -1;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * If all probes are removed, enable capacity/utilization data
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * collection for every CPU
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslamdcpc_cpu_setup(cpu_setup_t what, processorid_t cpu, void *arg)
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Offline CPUs are not allowed to take part so remove this
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * CPU if we are actively tracing.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam state = &cpu_core[c->cpu_id].cpuc_dcpc_intr_state;
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Indicate that a configuration is in process in
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * order to stop overflow interrupts being processed
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * on this CPU while we disable it.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Reset this CPUs interrupt state as the configuration
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * This CPU is being initialized or brought online so program
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * it with the current request set if we are actively tracing.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam{ DTRACE_STABILITY_EVOLVING, DTRACE_STABILITY_EVOLVING, DTRACE_CLASS_COMMON },
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam{ DTRACE_STABILITY_PRIVATE, DTRACE_STABILITY_PRIVATE, DTRACE_CLASS_UNKNOWN },
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam{ DTRACE_STABILITY_PRIVATE, DTRACE_STABILITY_PRIVATE, DTRACE_CLASS_UNKNOWN },
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam{ DTRACE_STABILITY_EVOLVING, DTRACE_STABILITY_EVOLVING, DTRACE_CLASS_CPU },
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam{ DTRACE_STABILITY_EVOLVING, DTRACE_STABILITY_EVOLVING, DTRACE_CLASS_COMMON },
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslamdcpc_open(dev_t *devp, int flag, int otyp, cred_t *cred_p)
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslamdcpc_info(dev_info_t *dip, ddi_info_cmd_t infocmd, void *arg, void **result)
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslamdcpc_detach(dev_info_t *devi, ddi_detach_cmd_t cmd)
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam unregister_cpu_setup_func(dcpc_cpu_setup, NULL);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam kmem_free(dcpc_actv_reqs, cpc_ncounters * sizeof (dcpc_probe_t *));
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslamdcpc_attach(dev_info_t *devi, ddi_attach_cmd_t cmd)
8cb74972a66bde0af7b1a957d01e0095b82a8b91Jonathan Haslam cmn_err(CE_NOTE, "!dcpc: Counter Overflow not supported"\
8cb74972a66bde0af7b1a957d01e0095b82a8b91Jonathan Haslam " on this processor");
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam if (ddi_create_minor_node(devi, "dcpc", S_IFCHR, 0,
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam dtrace_register("cpc", &dcpc_attr, DTRACE_PRIV_KERNEL,
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam register_cpu_setup_func(dcpc_cpu_setup, NULL);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Determine which, if any, mask attribute the back-end can use.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * The dcpc_actv_reqs array is used to store the requests that
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * we currently have programmed. The order of requests in this
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * array is not necessarily the order that the event appears in
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * the kcpc_request_t array. Once entered into a slot in the array
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * the entry is not moved until it's removed.
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam kmem_zalloc(cpc_ncounters * sizeof (dcpc_probe_t *), KM_SLEEP);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam dcpc_min_overflow = ddi_prop_get_int(DDI_DEV_T_ANY, devi,
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam DDI_PROP_DONTPASS, "dcpc-min-overflow", DCPC_MIN_OVF_DEFAULT);
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam 0, /* streamtab */
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam D_NEW | D_MP /* Driver compatibility flag */
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam 0, /* refcnt */
b9e93c10c0a2a4bb069d38bb311021a9478c4711Jonathan Haslam * Module linkage information for the kernel.