The following are demonstrations of the hotkernel DTrace program.

Here hotkernel is run for a couple of seconds then Ctrl-C is hit,

# ./hotkernel

Sampling... Hit Ctrl-C to end.

^C

FUNCTION COUNT PCNT

unix`swtch 1 0.1%

pcplusmp`apic_redistribute_compute 1 0.1%

genunix`strrput 1 0.1%

unix`sys_call 1 0.1%

genunix`fsflush_do_pages 1 0.1%

TS`ts_wakeup 1 0.1%

genunix`callout_schedule_1 1 0.1%

unix`page_create_putback 1 0.1%

unix`mutex_enter 4 0.3%

unix`cpu_halt 1575 99.2%

The output summarises which kernel-level function was sampled on the

CPU the most. This report shows that unix`cpu_halt was sampled 1575

times, which was 99.2% of the kernel-level samples.

As it turns out, unix`cpu_halt is called on this x86 server as part of the

kernel idle thread - explaining why it is so often found on the CPU,

# dtrace -n 'fbt::cpu_halt:entry { @[stack()] = count(); }'

dtrace: description 'fbt::cpu_halt:entry ' matched 1 probe

^C

unix`idle+0x3b

unix`thread_start+0x3

956

This kernel stack trace indicates that cpu_halt() is called by idle().

The following is a SPARC example,

# ./hotkernel

Sampling... Hit Ctrl-C to end.

^C

FUNCTION COUNT PCNT

genunix`fop_ioctl 1 0.1%

genunix`allocb_cred 1 0.1%

genunix`poll_common 1 0.1%

genunix`cv_block 1 0.1%

genunix`strioctl 1 0.1%

genunix`disp_lock_exit 1 0.1%

genunix`crfree 1 0.1%

ufs`ufs_getpage 1 0.1%

SUNW,UltraSPARC-IIi`copyin 1 0.1%

genunix`strmakedata 1 0.1%

genunix`cv_waituntil_sig 1 0.1%

SUNW,UltraSPARC-IIi`prefetch_page_r 1 0.1%

unix`set_freemem 1 0.1%

unix`page_trylock 1 0.1%

genunix`anon_get_ptr 1 0.1%

unix`page_hashin 1 0.1%

genunix`bt_getlowbit 1 0.1%

unix`pp_load_tlb 1 0.1%

unix`_resume_from_idle 1 0.1%

unix`hat_pageunload 1 0.1%

genunix`strrput 1 0.1%

genunix`strpoll 1 0.1%

unix`page_do_hashin 1 0.1%

unix`cpu_vm_stats_ks_update 1 0.1%

genunix`sleepq_wakeone_chan 1 0.1%

unix`lock_set_spl 1 0.1%

tl`tl_wput 1 0.1%

genunix`kstrgetmsg 1 0.1%

genunix`qbackenable 1 0.1%

genunix`releasef 1 0.1%

genunix`callout_execute 1 0.1%

uata`ata_hba_start 1 0.1%

genunix`pcacheset_cmp 1 0.1%

genunix`sleepq_insert 1 0.1%

genunix`syscall_mstate 1 0.1%

sockfs`sotpi_recvmsg 1 0.1%

genunix`strput 1 0.1%

genunix`timespectohz 1 0.1%

unix`lock_clear_splx 1 0.1%

genunix`read 1 0.1%

genunix`as_segcompar 1 0.1%

unix`atomic_cas_64 1 0.1%

unix`mutex_exit 1 0.1%

genunix`cv_unsleep 1 0.1%

unix`putnext 1 0.1%

unix`intr_thread 1 0.1%

genunix`hrt2tv 1 0.1%

sockfs`socktpi_poll 1 0.1%

unix`sfmmu_mlspl_enter 1 0.1%

SUNW,UltraSPARC-IIi`get_ecache_tag 1 0.1%

SUNW,UltraSPARC-IIi`gethrestime 1 0.1%

genunix`cv_timedwait_sig 1 0.1%

genunix`getq_noenab 1 0.1%

SUNW,UltraSPARC-IIi`flushecacheline 1 0.1%

unix`utl0 1 0.1%

genunix`anon_alloc 1 0.1%

unix`page_downgrade 1 0.1%

unix`setfrontdq 1 0.1%

genunix`timeout_common 1 0.1%

unix`bzero 1 0.1%

unix`ktl0 2 0.1%

genunix`canputnext 2 0.1%

genunix`clear_active_fd 2 0.1%

unix`sfmmu_tlb_demap 2 0.1%

unix`page_vpadd 2 0.1%

SUNW,UltraSPARC-IIi`check_ecache_line 2 0.1%

genunix`cyclic_softint 2 0.1%

genunix`restore_mstate 2 0.1%

genunix`anon_map_getpages 2 0.1%

genunix`putq 2 0.1%

unix`page_lookup_create 2 0.1%

dtrace`dtrace_dynvar_clean 2 0.1%

unix`sfmmu_pageunload 2 0.1%

genunix`cpu_decay 2 0.1%

genunix`kmem_cache_alloc 3 0.2%

unix`rw_exit 3 0.2%

tl`tl_wput_data_ser 3 0.2%

unix`page_get_replacement_page 3 0.2%

unix`page_sub 3 0.2%

genunix`clock 3 0.2%

SUNW,UltraSPARC-IIi`copyout 3 0.2%

unix`mutex_enter 4 0.2%

genunix`pcache_poll 5 0.3%

SUNW,UltraSPARC-IIi`scrub_ecache_line 5 0.3%

SUNW,UltraSPARC-IIi`hwblkpagecopy 22 1.2%

SUNW,UltraSPARC-IIi`hwblkclr 39 2.1%

unix`generic_idle_cpu 506 26.8%

unix`idle 1199 63.5%

Which shows the most common function is unix`idle.

Now the hotkernel tool is demonstrated with the -m option, to only print

out samples by module,

# ./hotkernel -m

Sampling... Hit Ctrl-C to end.

^C

MODULE COUNT PCNT

usbms 1 0.0%

specfs 1 0.0%

uhci 1 0.0%

sockfs 2 0.0%

genunix 28 0.6%

unix 4539 99.3%

Here, genunix and unix (the two core parts of the kernel) were the most

common module to be executing on-CPU.