1N/A**************************************************************************
1N/A* The following are notes regarding the overheads of running DTrace.
1N/A* COPYRIGHT: Copyright (c) 2007 Brendan Gregg.
1N/A**************************************************************************
1N/AThe following are notes regarding the overheads of running DTrace.
1N/A* What are the overheads of running DTrace?
1N/AIt depends what the DTrace script does, in particular, the frequency of
1N/Aevents that it is tracing.
1N/AThe following tips should explain what the overheads probably are,
1N/A- if your script traces less than 1000 events per second, then the overhead
1N/A is probably negligible. ie, less than 0.1% CPU.
1N/A- if your script traces more than 100,000 events per second, then the
1N/A overhead will start to be significant. If you are tracing kernel events,
1N/A then perhaps this could be 10% per CPU. If you are tracing user land
1N/A application events, then the overhead can be greater than 30% per CPU.
1N/A- if your script produes pages of output, then the CPU cost of drawing
1N/A this output to the screen and rendering the fonts is usually far greater
1N/A than DTrace itself. Redirect the output of DTrace to a file in /tmp
1N/A- a ballpark figure for the overhead of a DTrace probe would be 500 ns.
1N/A This can be much less (kernel only), or much more (many user to kerel
1N/A copyin()s); I've provided it to give you a very rough idea. Of course,
1N/A as CPUs become faster, this overhead will become smaller.
1N/AIf overheads are a concern - then perform tests to measure their magnitude
1N/Afor both your workload and the scripts applied, such as benchmarks with
1N/Aand without DTrace running. Also read the scripts you are using, and
1N/Aconsider how frequent the probes will fire, and if you can customise the
1N/Ascript to reduce the frequency of probes.
1N/AFor example, scripts that trace,
1N/Awould usually cause significant performance overhead, since they fire two
1N/Aprobes for every function called (and can easily reach 100,000 per second).
1N/AYou could reduce this by modifying the script to only trace the libraries
1N/Ayou are interested in. For example, if you were only interested in
1N/Alibsocket and libnsl, then change the above lines wherever they appeared to,
1N/A pid$target:libsocket::entry,
1N/A pid$target:libsocket::return,
1N/A pid$target:libnsl::entry,
1N/A pid$target:libnsl::return
1N/Aand you may notice the overheads are significantly reduced (especially anytime
1N/Ayou drop libc and libdl). To go further, only list functions of interest,
1N/A pid$target:libsocket:connect:entry,
1N/A pid$target:libsocket:connect:return,
1N/A pid$target:libsocket:listen:entry,
1N/A pid$target:libsocket:listen:return,
1N/AThere are additional notes in
Docs/Faq about the DTraceToolkit's scripts
1N/Aand performance overhead.
1N/A* When are the overheads a problem?
1N/AWhen they are significant (due to frequent events), and you are tracing
1N/Ain a production environment that is sensitive to additional CPU load.
1N/AOverheads should be considered if you are measuring times (delta, elapsed,
1N/Aon-CPU, etc) for performance analysis. In practise, overheads aren't
1N/Athat much of a problem -- the script will either identify your issues
1N/Acorrectly (great), or not (keep looking). Any it is usually easy to quickly
1N/Aconfirm what DTrace does find by using other tools, or by hacking quick
1N/Acode changes. You might be using DTrace output that you know has a
1N/Asignificant margin of error - but that becomes moot after you prove that
1N/Athe performance fix works through benchmarking a quick fix.
1N/AAt the end of the day, if DTrace helps find real measurable performance wins
1N/A(and it should), then it has been successful.
1N/A* When are overheads not a problem?
1N/AWhen the script is not tracing extreamly frequent events.
1N/AAlso, when you are in development and tracing events for troubleshooting
1N/Apurposes (args to functions, for example), DTrace overheads are usually