E
dit
A
ttach
P
rintable
r8 - 2013-05-16 - 21:30:45 - Main.harryabadi
You are here:
TWiki
>
Deployment Web
>
DeploymentTroubleshooting
>
PerformanceTroubleshooting
>
WhyIsMyCPUSpiking
>
AixCpuSpike
<div id="header-title" style="padding: 10px 15px; border-width:1px; border-style:solid; border-color:#FFD28C; background-image: url(<nop>https://jazz.net/wiki/pub/Deployment/WebPreferences/TLASE.jpg); background-size: cover; font-size:120%"> ---+!! <img src="https://jazz.net/wiki/pub/Deployment/WebPreferences/uc.png" alt="uc.png" width="50" height="50" align="right"/> AIX CPU spike %DKGRAY% Authors: Main.MichaelAfshar <br> Build basis: AIX 6.1 or later %ENDCOLOR%</div></sticky> <!-- Page contents top of page on right hand side in box --> <sticky><div style="float:right; border-width:1px; border-style:solid; border-color:#DFDFDF; background-color:#F6F6F6; margin:0 0 15px 15px; padding: 0 15px 0 15px;"> %TOC{title="Page contents"}% </div></sticky> <sticky><div style="margin:15px;"></sticky> Introductory paragraph: TBD ---++ AIX CPU monitoring 1. system overview 1. vmstat data with a 1 second granularity, timestamped 1. top output (with a 2 or 3 second granularity) 1. tprof data samples during the CPU spike 1. javacores collected with a (say) 30 second granularity during the CPU spike 1. snapshots of the "Active Services" running at the time of the CPU spike ---+++ System overview We start by trying to build up a picture of the CPU resources available to the system. The kinds of questions we may have are: 1. How many (virtual) processors are configured (or assigned to the LPAR)? 1. What SMT level is enabled? 1. If virtual, are the number of processors available to this LPAR capped, or uncapped? What is the entitlement for this Virtual Machine? 1. Depending on the entitlement, and if uncapped, how large is the pool, and how many other LPARs share processors in this pool, how overcommitted is the pool of processors. Running the AIX "lparstat -i" answers most of the questions we may have. <verbatim> % lparstat -i </verbatim> We refer the reader to the lparstat man page for field by field descriptions of this output. One may need to look at CEC data to build up a more complete picture of the LPARs sharing a CPU pool with this one. ---+++ vmstat To locate the time of the CPU spike, and to gain an insight into whether user or system CPU is being consumed at this time, the best tool to use is vmstat. The preferred flags to use for this are: <verbatim> vmstat -wtI 1 <iterations> </verbatim> This command should be issued in the background, wrapped with a nohup command, and redirecting the output to a file. This will collect data with a 1 second granularity. Please note that this level of granularity is necessary for a comprehensive insight into the consumption of this resource. A system administrator may well be running other tools that are also collecting data (e.g. topasrec, or sar) with a larger (e.g. 5 minute or 15 minute) granularity - but this output, though useful for capacity planning purposes, will not be helpful in the analysis of possibly short-lived CPU spikes. Check the vmstat output to identify the time of the CPU spike, its duration, its amplitude, and whether, when CPU consumption is high, whether this CPU is in kernel, user or wait I/O? Check also for excessive paging. ---+++ topas output Log topas output to a file. Analyze the top output to check if there one process consuming a lot of CPU, or several, or many? If it is just the java process that is consuming memory, see the next section. If processes other than java are responsible for the CPU peak, ascertain if these really need to run on this system. ---+++ top output for threads Follow the directions in [[http://publib.boulder.ibm.com/infocenter/javasdk/tools/index.jsp?topic=%2Fcom.ibm.java.doc.igaa%2F_1vg0001475cb4a-1190e2e0f74-8000_1006.html][AIX Java thread CPU monitoring]] to collect per thread data for the Java threads. ---+++ javacores Run the [[https://wait.ibm.com/][waittool]] to collect javacores at the time of the CPU spike with a 30 second frequency. Look at the stack traces of the java threads that (in the above output) are responsible for most of the CPU consumption. ---+++++!! Related topics: None ---+++++!! External links: * None ---+++++!! Additional contributors: None <sticky></div></sticky>
E
dit
|
A
ttach
|
P
rintable
|
V
iew topic
|
Backlinks:
We
b
,
A
l
l Webs
|
H
istory
: r8
<
r7
<
r6
<
r5
<
r4
|
M
ore topic actions
Deployment
Deployment web
Planning and design
Installing and upgrading
Migrating and evolving
Integrating
Administering
Monitoring
Troubleshooting
Community information and contribution guidelines
Create new topic
Topic list
Search
Advanced search
Notify
RSS
Atom
Changes
Statistics
Web preferences
NOTE: Please use the Sandbox web for testing
Status icon key:
To do
Under construction
New
Updated
Constant change
None - stable page
Smaller versions of status icons for inline text:
Copyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our
Terms of Use.
Please read the following
disclaimer
.
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more
here
.