b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * CDDL HEADER START
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * The contents of this file are subject to the terms of the
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Common Development and Distribution License (the "License").
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * You may not use this file except in compliance with the License.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * or http://www.opensolaris.org/os/licensing.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * See the License for the specific language governing permissions
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * and limitations under the License.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * When distributing Covered Code, include this CDDL HEADER in each
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * If applicable, add the following below this CDDL HEADER, with the
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * fields enclosed by brackets "[]" replaced with your own identifying
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * information: Portions Copyright [yyyy] [name of copyright owner]
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * CDDL HEADER END
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov * Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Support for determining capacity and utilization of performance relevant
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * hardware components in a computer
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * The capacity and utilization of the performance relevant hardware components
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * is needed to be able to optimize performance while minimizing the amount of
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * power used on a system. The idea is to use hardware performance counters
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * and potentially other means to determine the capacity and utilization of
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * performance relevant hardware components (eg. execution pipeline, cache,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * memory, etc.) and attribute the utilization to the responsible CPU and the
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * thread running there.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * This will help characterize the utilization of performance relevant
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * components and how much is used by each CPU and each thread. With
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * that data, the utilization can be aggregated to all the CPUs sharing each
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * performance relevant hardware component to calculate the total utilization
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * of each component and compare that with the component's capacity to
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * essentially determine the actual hardware load of the component. The
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * hardware utilization attributed to each running thread can also be
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * aggregated to determine the total hardware utilization of each component to
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Once that is done, one can determine how much of each performance relevant
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * hardware component is needed by a given thread or set of threads (eg. a
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * workload) and size up exactly what hardware is needed by the threads and how
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * much. With this info, we can better place threads among CPUs to match their
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * exact hardware resource needs and potentially lower or raise the power based
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * on their utilization or pack threads onto the fewest hardware components
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * needed and power off any remaining unused components to minimize power
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * without sacrificing performance.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * IMPLEMENTATION
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * --------------
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * The code has been designed and implemented to make (un)programming and
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * reading the counters for a given CPU as lightweight and fast as possible.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * This is very important because we need to read and potentially (un)program
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * the counters very often and in performance sensitive code. Specifically,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * the counters may need to be (un)programmed during context switch and/or a
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cyclic handler when there are more counter events to count than existing
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Consequently, the code has been split up to allow allocating and
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * initializing everything needed to program and read the counters on a given
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * CPU once and make (un)programming and reading the counters for a given CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * not have to allocate/free memory or grab any locks. To do this, all the
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * state needed to (un)program and read the counters on a CPU is kept per CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * and is made lock free by forcing any code that reads or manipulates the
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * counters or the state needed to (un)program or read the counters to run on
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * the target CPU and disable preemption while running on the target CPU to
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * protect any critical sections. All counter manipulation on the target CPU is
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * happening either from a cross-call to the target CPU or at the same PIL as
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * used by the cross-call subsystem. This guarantees that counter manipulation
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * is not interrupted by cross-calls from other CPUs.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * The synchronization has been made lock free or as simple as possible for
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * performance and to avoid getting the locking all tangled up when we interpose
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * on the CPC routines that (un)program the counters to manage the counters
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * between the kernel and user on each CPU. When the user starts using the
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * counters on a given CPU, the kernel will unprogram the counters that it is
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * using on that CPU just before they are programmed for the user. Then the
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * kernel will program the counters on a given CPU for its own use when the user
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * stops using them.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * There is a special interaction with DTrace cpc provider (dcpc). Before dcpc
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * enables any probe, it requests to disable and unprogram all counters used for
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * capacity and utilizations. These counters are never re-programmed back until
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * dcpc completes. When all DTrace cpc probes are removed, dcpc notifies CU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * framework and it re-programs the counters.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * When a CPU is going offline, its CU counters are unprogrammed and disabled,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * so that they would not be re-programmed again by some other activity on the
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * CPU that is going offline.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * The counters are programmed during boot. However, a flag is available to
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * disable this if necessary (see cu_flag below). A handler is provided to
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * (un)program the counters during CPU on/offline. Basic routines are provided
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * to initialize and tear down this module, initialize and tear down any state
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * needed for a given CPU, and (un)program the counters for a given CPU.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Lastly, a handler is provided to read the counters and attribute the
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * utilization to the responsible CPU.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Default CPU hardware performance counter flags to use for measuring capacity
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * and utilization
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov (CPC_COUNT_USER|CPC_COUNT_SYSTEM|CPC_OVF_NOTIFY_EMT)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Possible Flags for controlling this module.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov#define CU_FLAG_ENABLE 1 /* Enable module */
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov#define CU_FLAG_READY 2 /* Ready to setup module */
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * pg_cpu kstats calculate utilization rate and maximum utilization rate for
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * some CPUs. The rate is calculated based on data from two subsequent
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * snapshots. When the time between such two snapshots is too small, the
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * resulting rate may have low accuracy, so we only consider snapshots which
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * are separated by SAMPLE_INTERVAL nanoseconds from one another. We do not
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * update the rate if the interval is smaller than that.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Use one tenth of a second as the minimum interval for utilization rate
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * calculation.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * NOTE: The CU_SAMPLE_INTERVAL_MIN should be higher than the scaling factor in
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * the CU_RATE() macro below to guarantee that we never divide by zero.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Rate is the number of events per second. The rate is the number of events
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * divided by time and multiplied by the number of nanoseconds in a second. We
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * do not want time to be too small since it will cause large errors in
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * We do not want to multiply two large numbers (the instruction count and
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * NANOSEC) either since it may cause integer overflow. So we divide both the
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * numerator and the denominator by the same value.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * NOTE: The scaling factor below should be less than CU_SAMPLE_INTERVAL_MIN
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * above to guarantee that time divided by this value is always non-zero.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov (((val) * (NANOSEC / CU_SCALE)) / ((time) / CU_SCALE))
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov#define CU_SAMPLE_INTERVAL_MIN (NANOSEC / 10)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov#define CU_SCALE (CU_SAMPLE_INTERVAL_MIN / 10000)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * When the time between two kstat reads for the same CPU is less than
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * CU_UPDATE_THRESHOLD use the old counter data and skip updating counter values
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * for the CPU. This helps reduce cross-calls when kstat consumers read data
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * very often or when they read PG utilization data and then CPU utilization
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * data quickly after that.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov#define CU_UPDATE_THRESHOLD (NANOSEC / 10)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * The IS_HIPIL() macro verifies that the code is executed either from a
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cross-call or from high-PIL interrupt
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov#define IS_HIPIL() (getpil() >= XCALL_PIL)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov#endif /* DEBUG */
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovtypedef void (*cu_cpu_func_t)(uintptr_t, int *);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Flags to use for programming CPU hardware performance counters to measure
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * capacity and utilization
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Initial value used for programming hardware counters
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * List of CPC event requests for capacity and utilization.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovstatic kcpc_request_list_t *cu_cpc_reqs = NULL;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * When a CPU is a member of PG with a sharing relationship that is supported
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * by the capacity/utilization framework, a kstat is created for that CPU and
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * sharing relationship.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * These kstats are updated one at a time, so we can have a single scratch
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * space to fill the data.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * CPU counter kstats fields:
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cu_cpu_id CPU ID for this kstat
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov * cu_pg_id PG ID for this kstat
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cu_generation Generation value that increases whenever any CPU goes
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * offline or online. Two kstat snapshots for the same
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * CPU may only be compared if they have the same
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cu_pg_id PG ID for the relationship described by this kstat
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cu_cpu_util Running value of CPU utilization for the sharing
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * relationship
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cu_cpu_time_running Total time spent collecting CU data. The time may be
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * less than wall time if CU counters were stopped for
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cu_cpu_time_stopped Total time the CU counters were stopped.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cu_cpu_rate Utilization rate, expressed in operations per second.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cu_cpu_rate_max Maximum observed value of utilization rate.
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov * cu_cpu_relationship Name of sharing relationship for the PG in this kstat
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov { "hw_util_time_running", KSTAT_DATA_UINT64 },
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov { "hw_util_time_stopped", KSTAT_DATA_UINT64 },
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov { "hw_util_rate_max", KSTAT_DATA_UINT64 },
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Flags for controlling this module
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Error return value for cu_init() since it can't return anything to be called
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * from mp_init_tbl[] (:-(
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovhrtime_t cu_sample_interval_min = CU_SAMPLE_INTERVAL_MIN;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovhrtime_t cu_update_threshold = CU_UPDATE_THRESHOLD;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Forward declaration of interface routines
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovvoid cu_cpc_unprogram(cpu_t *cp, int *err);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovint cu_cpu_update(struct cpu *cp, boolean_t move_to);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Forward declaration of private routines
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovstatic int cu_cpc_init(cpu_t *cp, kcpc_request_list_t *reqs, int nreqs);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovstatic void cu_cpc_program_xcall(uintptr_t arg, int *err);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovstatic int cu_cpc_req_add(char *event, kcpc_request_list_t *reqs,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov int nreqs, cu_cntr_stats_t *stats, int kmem_flags, int *nevents);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovstatic int cu_cpu_callback(cpu_setup_t what, int id, void *arg);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovstatic int cu_cpu_init(cpu_t *cp, kcpc_request_list_t *reqs);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovstatic void cu_cpu_kstat_create(pghw_t *pg, cu_cntr_info_t *cntr_info);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovstatic int cu_cpu_kstat_update(kstat_t *ksp, int rw);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovstatic int cu_cpu_run(cpu_t *cp, cu_cpu_func_t func, uintptr_t arg);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovstatic int cu_cpu_update_stats(cu_cntr_stats_t *stats,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovstatic void cu_cpu_info_detach_xcall(void);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Disable or enable Capacity Utilization counters on all CPUs.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov } while ((cp = cp->cpu_next_onln) != cpu_active);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov } while ((cp = cp->cpu_next_onln) != cpu_active);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Setup capacity and utilization support
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (!(cu_flags & CU_FLAG_ENABLE) || (cu_flags & CU_FLAG_ON)) {
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Can't measure hardware capacity and utilization without CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * hardware performance counters
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Setup CPC event request queue
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cu_cpc_reqs = kcpc_reqs_init(cpc_ncounters, KM_SLEEP);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Mark flags to say that module is ready to be setup
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Allocate and setup state needed to measure capacity and
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Reset list of counter event requests so its space can be
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * reused for a different set of requests for next CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Mark flags to say that module is on now and counters are ready to be
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * programmed on all active CPUs
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Program counters on currently active CPUs
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Register callback for CPU state changes to enable and disable
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * CPC counters as CPUs come on and offline
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov register_cpu_setup_func(cu_cpu_callback, NULL);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Return number of counter events needed to measure capacity and utilization
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * for specified CPU and fill in list of CPC requests with each counter event
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * needed if list where to add CPC requests is given
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * NOTE: Use KM_NOSLEEP for kmem_{,z}alloc() since cpu_lock is held and free
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * everything that has been successfully allocated if any memory
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * allocation fails
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovcu_cpc_init(cpu_t *cp, kcpc_request_list_t *reqs, int nreqs)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * There has to be a target CPU for this
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Return 0 when CPU doesn't belong to any group
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (cpu_pgs == NULL || GROUP_SIZE(&cpu_pgs->cmt_pgs) < 1)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Grab counter statistics and info
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (cu_cpu_info == NULL || cu_cpu_info->cu_cntr_stats == NULL)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cntr_info_array = cu_cpu_info->cu_cntr_info;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * See whether platform (or processor) specific code knows which CPC
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * events to request, etc. are needed to measure hardware capacity and
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * utilization on this machine
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov nevents = cu_plat_cpc_init(cp, reqs, nreqs);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Let common code decide which CPC events to request, etc. to measure
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * capacity and utilization since platform (or processor) specific does
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * not know....
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Walk CPU's PG lineage and do following:
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * - Setup CPC request, counter info, and stats needed for each counter
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * event to measure capacity and and utilization for each of CPU's PG
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * hardware sharing relationships
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * - Create PG CPU kstats to export capacity and utilization for each PG
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov while ((pg_cmt = group_iterate(cmt_pgs, &iter)) != NULL) {
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (cu_cpc_req_add("PAPI_tot_ins", reqs, nreqs, stats,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (cu_cpc_req_add("PAPI_fp_ins", reqs, nreqs, stats,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Don't measure capacity and utilization for this kind
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * of PG hardware relationship so skip to next PG in
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * CPU's PG lineage
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Nothing to measure for this hardware sharing relationship
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov kmem_free(cntr_info, sizeof (cu_cntr_info_t));
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Fill in counter info for this PG hardware relationship
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cntr_info = kmem_zalloc(sizeof (cu_cntr_info_t),
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cntr_info->ci_stats = &stats[nevents_save];
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Create PG CPU kstats for this hardware relationship
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Program counters for capacity and utilization on given CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * If any of the following conditions is true, the counters are not programmed:
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * - CU framework is disabled
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * - The cpu_cu_info field of the cpu structure is NULL
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * - DTrace is active
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * - Counters are programmed already
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * - Counters are disabled (by calls to cu_cpu_disable())
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Should be running on given CPU. We disable preemption to keep CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * from disappearing and make sure flags and CPC context don't change
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * from underneath us
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Module not ready to program counters
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * If DTrace CPC is active or counters turned on already or are
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * disabled, just return.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (dtrace_cpc_in_use || (cu_cpu_info->cu_flag & CU_CPU_CNTRS_ON) ||
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov !(CPU->cpu_cpc_ctx->kc_flags & KCPC_CTX_INVALID_STOPPED)) {
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Get CPU's CPC context needed for capacity and utilization
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov ASSERT(cpu_ctx->ctx_ptr_array == NULL || cpu_ctx->ctx_ptr_array_sz > 0);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov ASSERT(cpu_ctx->nctx <= cpu_ctx->ctx_ptr_array_sz);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (cpu_ctx->nctx <= 0 || cpu_ctx->ctx_ptr_array == NULL ||
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Increment index in CPU's CPC context info to point at next context
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * NOTE: Do this now instead of after programming counters to ensure
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * that index will always point at *current* context so we will
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * always be able to unprogram *current* context if necessary
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cpu_ctx->cur_index = (cpu_ctx->cur_index + 1) % cpu_ctx->nctx;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov ctx = cpu_ctx->ctx_ptr_array[cpu_ctx->cur_index];
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Clear KCPC_CTX_INVALID and KCPC_CTX_INVALID_STOPPED from CPU's CPC
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * context before programming counters
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Context is marked with KCPC_CTX_INVALID_STOPPED when context is
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * unprogrammed and may be marked with KCPC_CTX_INVALID when
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * kcpc_invalidate_all() is called by cpustat(1M) and dtrace CPC to
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * invalidate all CPC contexts before they take over all the counters.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * This isn't necessary since these flags are only used for thread bound
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * CPC contexts not CPU bound CPC contexts like ones used for capacity
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * and utilization.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * There is no need to protect the flag update since no one is using
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * this context now.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov ctx->kc_flags &= ~(KCPC_CTX_INVALID | KCPC_CTX_INVALID_STOPPED);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Program counters on this CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Set state in CPU structure to say that CPU's counters are programmed
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * for capacity and utilization now and that they are transitioning from
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * off to on state. This will cause cu_cpu_update to update stop times
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * for all programmed counters.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cu_cpu_info->cu_flag |= CU_CPU_CNTRS_ON | CU_CPU_CNTRS_OFF_ON;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Update counter statistics
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cu_cpu_info->cu_flag &= ~CU_CPU_CNTRS_OFF_ON;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Cross call wrapper routine for cu_cpc_program()
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Checks to make sure that counters on CPU aren't being used by someone else
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * before calling cu_cpc_program() since cu_cpc_program() needs to assert that
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * nobody else is using the counters to catch and prevent any broken code.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Also, this check needs to happen on the target CPU since the CPU's CPC
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * context can only be changed while running on the CPU.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * If the first argument is TRUE, cu_cpc_program_xcall also checks that there is
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * no valid thread bound cpc context. This is important to check to prevent
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * re-programming thread counters with CU counters when CPU is coming on-line.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovcu_cpc_program_xcall(uintptr_t arg, int *err)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov boolean_t avoid_thread_context = (boolean_t)arg;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov !(CPU->cpu_cpc_ctx->kc_flags & KCPC_CTX_INVALID_STOPPED)) {
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (avoid_thread_context && (curthread->t_cpc_ctx != NULL) &&
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov !(curthread->t_cpc_ctx->kc_flags & KCPC_CTX_INVALID_STOPPED)) {
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Unprogram counters for capacity and utilization on given CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * This function should be always executed on the target CPU at high PIL
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Should be running on given CPU with preemption disabled to keep CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * from disappearing and make sure flags and CPC context don't change
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * from underneath us
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Module not on
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Counters turned off already
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (!(cu_cpu_info->cu_flag & CU_CPU_CNTRS_ON)) {
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Update counter statistics
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Get CPU's CPC context needed for capacity and utilization
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (cpu_ctx->nctx <= 0 || cpu_ctx->ctx_ptr_array == NULL ||
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov ctx = cpu_ctx->ctx_ptr_array[cpu_ctx->cur_index];
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * CPU's CPC context should be current capacity and utilization CPC
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Unprogram counters on CPU.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov ASSERT(ctx->kc_flags & KCPC_CTX_INVALID_STOPPED);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Unset state in CPU structure saying that CPU's counters are
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cu_cpu_info->cu_flag &= ~CU_CPU_CNTRS_ON;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Add given counter event to list of CPC requests
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovcu_cpc_req_add(char *event, kcpc_request_list_t *reqs, int nreqs,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cu_cntr_stats_t *stats, int kmem_flags, int *nevents)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Return error when no counter event specified, counter event not
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * supported by CPC's PCBE, or number of events not given
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (event == NULL || kcpc_event_supported(event) == B_FALSE ||
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Only count number of counter events needed if list
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * where to add CPC requests not given
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Return error when stats not given or not enough room on list of CPC
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * requests for more counter events
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (stats == NULL || (nreqs <= 0 && n >= nreqs))
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Use flags in cu_cpc_flags to program counters and enable overflow
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * interrupts/traps (unless PCBE can't handle overflow interrupts) so
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * PCBE can catch counters before they wrap to hopefully give us an
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * accurate (64-bit) virtualized counter
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if ((kcpc_pcbe_capabilities() & CPC_CAP_OVERFLOW_INTERRUPT) == 0)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Add CPC request to list
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov retval = kcpc_reqs_add(reqs, event, cu_cpc_preset_value,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Enable or disable collection of capacity/utilization data for a current CPU.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Counters are enabled if 'on' argument is True and disabled if it is False.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * This function should be always executed at high PIL
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovcu_cpc_trigger(uintptr_t arg1, uintptr_t arg2)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov !(cu_cpu_info->cu_flag & CU_CPU_CNTRS_ON));
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Decrement the cu_disabled counter.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Once it drops to zero, call cu_cpc_program.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov } else if (cu_cpu_info->cu_disabled++ == 0) {
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * This is the first attempt to disable CU, so turn it off
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov ASSERT(!(cu_cpu_info->cu_flag & CU_CPU_CNTRS_ON));
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Callback for changes in CPU states
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Used to enable or disable hardware performance counters on CPUs that are
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * turned on or off
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * NOTE: cpc should be programmed/unprogrammed while running on the target CPU.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * We have to use thread_affinity_set to hop to the right CPU because these
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * routines expect cpu_lock held, so we can't cross-call other CPUs while
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * holding CPU lock.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov/* LINTED E_FUNC_ARG_UNUSED */
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovcu_cpu_callback(cpu_setup_t what, int id, void *arg)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Setup counters on CPU being turned on
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Reset list of counter event requests so its space can be
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * reused for a different set of requests for next CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Setup counters on CPU being turned on.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov retval = cu_cpu_run(cp, cu_cpc_program_xcall,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Disable counters on CPU being turned off. Counters will not
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * be re-enabled on this CPU until it comes back online.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Disable or enable Capacity Utilization counters on a given CPU. This function
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * can be called from any CPU to disable counters on the given CPU.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cpu_call(cp, cu_cpc_trigger, (uintptr_t)cp, (uintptr_t)B_FALSE);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cpu_call(cp, cu_cpc_trigger, (uintptr_t)cp, (uintptr_t)B_TRUE);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Setup capacity and utilization support for given CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * NOTE: Use KM_NOSLEEP for kmem_{,z}alloc() since cpu_lock is held and free
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * everything that has been successfully allocated including cpu_cu_info
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * if any memory allocation fails
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovcu_cpu_init(cpu_t *cp, kcpc_request_list_t *reqs)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cpu_lock should be held and protect against CPU going away and races
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * with cu_{init,fini,cpu_fini}()
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Return if not ready to setup counters yet
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cp->cpu_cu_info = kmem_zalloc(sizeof (cu_cpu_info_t),
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Get capacity and utilization CPC context for CPU and check to see
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * whether it has been setup already
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cu_cpu_info->cu_disabled = dtrace_cpc_in_use ? 1 : 0;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (cpu_ctx->nctx > 0 && cpu_ctx->ctx_ptr_array != NULL &&
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Should have no contexts since it hasn't been setup already
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov ASSERT(cpu_ctx->nctx == 0 && cpu_ctx->ctx_ptr_array == NULL &&
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Determine how many CPC events needed to measure capacity and
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * utilization for this CPU, allocate space for counter statistics for
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * each event, and fill in list of CPC event requests with corresponding
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * counter stats for each request to make attributing counter data
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * easier later....
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cu_cpu_info->cu_cntr_stats = kmem_zalloc(n * sizeof (cu_cntr_stats_t),
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (cu_cpu_info->cu_cntr_stats == NULL) {
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Create CPC context with given requests
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov n = kcpc_cpu_ctx_create(cp, reqs, KM_NOSLEEP, &ctx_ptr_array,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Should have contexts
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov ASSERT(n > 0 && ctx_ptr_array != NULL && ctx_ptr_array_sz > 0);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (ctx_ptr_array == NULL || ctx_ptr_array_sz <= 0) {
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Fill in CPC context info for CPU needed for capacity and utilization
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cpu_ctx->ctx_ptr_array_sz = ctx_ptr_array_sz;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Tear down capacity and utilization support for given CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cpu_lock should be held and protect against CPU going away and races
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * with cu_{init,fini,cpu_init}()
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Have to at least be ready to setup counters to have allocated
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * anything that needs to be deallocated now
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Nothing to do if CPU's capacity and utilization info doesn't exist
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Tear down any existing kstats and counter info for each hardware
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * sharing relationship
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov for (pg_hw_type = PGHW_START; pg_hw_type < PGHW_NUM_COMPONENTS;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cntr_info = cu_cpu_info->cu_cntr_info[pg_hw_type];
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov kmem_free(cntr_info, sizeof (cu_cntr_info_t));
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Free counter statistics for CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov ASSERT(cu_cpu_info->cu_cntr_stats == NULL ||
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (cu_cpu_info->cu_cntr_stats != NULL &&
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cu_cpu_info->cu_ncntr_stats * sizeof (cu_cntr_stats_t));
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Get capacity and utilization CPC contexts for given CPU and check to
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * see whether they have been freed already
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (cpu_ctx != NULL && cpu_ctx->ctx_ptr_array != NULL &&
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Free CPC contexts for given CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Free CPC context pointer array
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov kmem_free(cpu_ctx->ctx_ptr_array, cpu_ctx->ctx_ptr_array_sz);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Zero CPC info for CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Set cp->cpu_cu_info pointer to NULL. Go through cross-call to ensure
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * that no one is going to access the cpu_cu_info whicch we are going to
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cpu_call(cp, (cpu_call_func_t)cu_cpu_info_detach_xcall, 0, 0);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Free CPU's capacity and utilization info
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov kmem_free(cu_cpu_info, sizeof (cu_cpu_info_t));
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Create capacity & utilization kstats for given PG CPU hardware sharing
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * relationship
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovcu_cpu_kstat_create(pghw_t *pg, cu_cntr_info_t *cntr_info)
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov char *sharing = pghw_type_string(pg->pghw_hw);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Just return when no counter info or CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (cntr_info == NULL || cntr_info->ci_cpu == NULL)
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov * Canonify PG name to conform to kstat name rules
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov (void) strncpy(name, pghw_type_string(pg->pghw_hw), KSTAT_STRLEN + 1);
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov if ((ks = kstat_create_zone("pg_hw_perf_cpu",
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov name, "processor_group", KSTAT_TYPE_NAMED,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov sizeof (cu_cpu_kstat) / sizeof (kstat_named_t),
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov KSTAT_FLAG_VIRTUAL, GLOBAL_ZONEID)) == NULL)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Propagate values from CPU capacity & utilization stats to kstats
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cu_cntr_info_t *cntr_info = ksp->ks_private;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov struct cu_cpu_kstat *kstat = &cu_cpu_kstat;
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov kstat->cu_cpu_id.value.ui32 = cp->cpu_id;
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov kstat->cu_pg_id.value.i32 = ((pg_t *)pg)->pg_id;
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov * The caller should have priv_cpc_cpu privilege to get utilization
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov * data. Callers who do not have the privilege will see zeroes as the
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov if (secpolicy_cpc_cpu(crgetcred()) != 0) {
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov kstat->cu_generation.value.ui32 = cp->cpu_generation;
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov kstat_named_setstr(&kstat->cu_cpu_relationship,
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov kstat->cu_cpu_time_running.value.ui64 = 0;
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov kstat->cu_cpu_time_stopped.value.ui64 = 0;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Update capacity and utilization statistics needed for CPU's PG (CPU)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov kstat->cu_generation.value.ui32 = cp->cpu_generation;
d3c9722485327eb5b96de2f2108e9a84bd46096dAlexander Kolbasov kstat_named_setstr(&kstat->cu_cpu_relationship,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov kstat->cu_cpu_util.value.ui64 = stats->cs_value_total;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov kstat->cu_cpu_rate.value.ui64 = stats->cs_rate;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov kstat->cu_cpu_rate_max.value.ui64 = stats->cs_rate_max;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov kstat->cu_cpu_time_running.value.ui64 = stats->cs_time_running;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov kstat->cu_cpu_time_stopped.value.ui64 = stats->cs_time_stopped;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Counters are stopped now, so the cs_time_stopped was last
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * updated at cs_time_start time. Add the time passed since then
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * to the stopped time.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (!(cp->cpu_cu_info->cu_flag & CU_CPU_CNTRS_ON))
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Run specified function with specified argument on a given CPU and return
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * whatever the function returns
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovcu_cpu_run(cpu_t *cp, cu_cpu_func_t func, uintptr_t arg)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cpu_call() will call func on the CPU specified with given argument
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * and return func's return value in last argument
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cpu_call(cp, (cpu_call_func_t)func, arg, (uintptr_t)&error);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Update counter statistics on a given CPU.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * If move_to argument is True, execute the function on the CPU specified
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Otherwise, assume that it is already runninng on the right CPU
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * If move_to is specified, the caller should hold cpu_lock or have preemption
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * disabled. Otherwise it is up to the caller to guarantee that things do not
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * change in the process.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovcu_cpu_update(struct cpu *cp, boolean_t move_to)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cu_cpu_info_t *cu_cpu_info = cp->cpu_cu_info;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov ASSERT(!move_to || MUTEX_HELD(&cpu_lock) || curthread->t_preempt > 0);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Nothing to do if counters are not programmed
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov !(cu_cpu_info->cu_flag & CU_CPU_CNTRS_ON))
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Don't update CPU statistics if it was updated recently
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * and provide old results instead
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if ((time_snap - cu_cpu_info->cu_sample_time) < cu_update_threshold) {
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov DTRACE_PROBE1(cu__drop__sample, cpu_t *, cp);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * CPC counter should be read on the CPU that is running the counter. We
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * either have to move ourselves to the target CPU or insure that we
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * already run there.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * We use cross-call to the target CPU to execute kcpc_read() and
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cu_cpu_update_stats() there.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov (void) cu_cpu_run(cp, (cu_cpu_func_t)kcpc_read,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov retval = kcpc_read((kcpc_update_func_t)cu_cpu_update_stats);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Offset negative return value by -10 so we can distinguish it
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * from error return values of this routine vs kcpc_read()
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Update CPU counter statistics for current CPU.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * This function may be called from a cross-call
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasovcu_cpu_update_stats(cu_cntr_stats_t *stats, uint64_t cntr_value)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cu_cpu_info_t *cu_cpu_info = CPU->cpu_cu_info;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Nothing to do if counters are not programmed. This should not happen,
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * but we check just in case.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Take snapshot of high resolution timer
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * CU counters have just been programmed. We cannot assume that the new
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * cntr_value continues from where we left off, so use the cntr_value as
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * the new initial value.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Calculate delta in counter values between start of sampling period
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov delta = cntr_value - stats->cs_value_start;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Calculate time between start of sampling period and now
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (time_delta > 0) { /* wrap shouldn't happen */
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Update either running or stopped time based on the transition
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Update rest of counter statistics if counter value didn't wrap
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Update utilization rate if the interval between samples is
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov ASSERT(cu_sample_interval_min > CU_SCALE);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov stats->cs_rate = CU_RATE(delta, time_delta);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Update CMT PG utilization data.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * This routine computes the running total utilization and times for the
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * specified PG by adding up the total utilization and counter running and
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * stopped times of all CPUs in the PG and calculates the utilization rate and
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * maximum rate for all CPUs in the PG.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov uint64_t old_utilization = hw_util->pghw_util;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Initialize running total utilization and times for PG to 0
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Iterate over all CPUs in the PG and aggregate utilization, running
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * time and stopped time.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov while ((cpu = pg_cpu_next(&cpu_iter)) != NULL) {
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cu_cpu_info_t *cu_cpu_info = cpu->cpu_cu_info;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Update utilization data for the CPU and then
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * aggregate per CPU running totals for PG
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov cntr_info = cu_cpu_info->cu_cntr_info[pg_hwtype];
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (cntr_info == NULL || (stats = cntr_info->ci_stats) == NULL)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov hw_util->pghw_util += stats->cs_value_total;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov hw_util->pghw_time_running += stats->cs_time_running;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov hw_util->pghw_time_stopped += stats->cs_time_stopped;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * If counters are stopped now, the pg_time_stopped was last
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * updated at cs_time_start time. Add the time passed since then
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * to the stopped time.
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (!(cu_cpu_info->cu_flag & CU_CPU_CNTRS_ON))
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Compute per PG instruction rate and maximum rate
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov time_delta = now - hw_util->pghw_time_stamp;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Calculate change in utilization over sampling period and set this to
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * 0 if the delta would be 0 or negative which may happen if any CPUs go
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * offline during the sampling period
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (hw_util->pghw_util > old_utilization)
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov utilization_delta = hw_util->pghw_util - old_utilization;
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Update utilization rate if the interval between samples is
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov ASSERT(cu_sample_interval_min > CU_SCALE);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov hw_util->pghw_rate = CU_RATE(utilization_delta, time_delta);
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov * Update the maximum observed rate
b885580b43755ee4ea1e280b85428893d2ba9291Alexander Kolbasov if (hw_util->pghw_rate_max < hw_util->pghw_rate)