Yesterday took place the project on-line test of the JVM Out of Memory error, leading off the system down and the server file system Cheng Bao. Check reason is excessive memory leaks, system availability and performance of memory decline; eventually lead to memory (OutOfMemory). We have developed with the IBM WebSphere platform, in websphere / AppServer next generation of large amounts javacore *, heapdump * like the file, resulting in websphere garbage collection function failed resulting in。 Among them, javacore file is on the cpu, heapdump file is about memory.
Production environment is ibm minicomputers, in accordance with the revised Handbook: Application Server> server1> Process Definition> Java Virtual Machine, the "maximum heap size (default 256)" to more than 768 or 1024. Delete the file system javacore *, heapdump * returned to normal after. Analyze the reasons, first, set JVM problem, the second (final cause) procedures are not sufficiently robust, a lot of resources are not immediately release the resources were Chengbao. In particular, database connection resource.Had suspended the process, and this time, the process hangs mainly SQL query can not be timely returns as a result, the page has been waiting for a resource returned by hang (suspend). Third, the large amount of data produced by the result set returns the result. Always suspected that we use data structures not reasonable, but did not attempt to modify.
Summarize:. .
On the framework you should use the sound mode, when you want to use resources in a timely release and recovery of waste;.
Distributed as far as possible the structure, similar to the MVC pattern;. .
Efforts to solve the root from the procedural issues.
Attachment: reproduced. .
WebSphere application server memory leak detection and diagnostic tools.
Level: Intermediate Xue-Chao (lixuec @ cn.. Ibm.. Com), Senior Software Engineer, IBM China Software Development Center. .
November 21, 2006.
This article describes how to implement WebSphere Application Server application memory leak detection, and for IBM provides a series of analysis and diagnostic tool, given the specific configuration steps and use of best practices. .
In the introduction.
Memory leaks are more common form of application performance problems, in the event, the system's available memory and performance continued to decline; eventually lead to memory (OutOfMemory), the system completely dawdle away, can not respond to any request, the harm is very serious. At the same time, Java heap (Heap) in a large number of objects and complex relationships between objects, resulting in memory leak detection and analysis are more difficult, with the corresponding support tools is necessary. .
WebSphere Application Server provides a series of targeted memory problems of detection and analysis of diagnostic tools, these tools can help users to detect memory issues, ensure the system is in place before the OOM, users can do not need to perform complex analysis of the conditions foreseen in the deployment of applications for memory leaks. If you do have a memory leak, WebSphere also provides tools to help users diagnose, and find the real cause of memory leaks.
1. . Memory leak detection and diagnosis steps. .
In practice, we can use the following steps to address a memory leak problem:.
(1) First, in WebSphere, we enable real-time memory leak detection tool, WebSphere Performance Diagnostics consultant will issue a warning message before the memory leak. .
(2) enable WebSphere's own Tivoli performance Viewer to monitor system usage, determine the JVM memory leak is occurring.
(3) The need to generate more memory recovery log, use the PMAT tool to analyze and determine the leak of the time, cycles. .
(4) to build a single or multiple selection of Heapdump file for analysis and diagnosis, MDD4J find the real cause of memory leaks.
(5) to the development sector code fix, then re-deployed to the WebSphere Application Server. .
The following sections, we target of each link configuration and the tools used to elaborate.
2. WebSphere application server memory leak detection tool. .
2. Introduction to .1 performance diagnosis consultants.
Performance Diagnostic Consultant (Performance and Diagnostic Advisor), in the WebSphere Application Server 6. .0. .2 Version is called before the run-time performance of consultants (Runtime Performance Advisor). The tool can periodically check the WebSphere settings and adjust the recommended values are given. Since WebSphere Application Server 6. .0. .2 version, the tool implements a lightweight memory detection mechanism, can be very easy to help users detect whether a system memory leak problems and advance through logs and management console for notification. This will give users enough time to take the necessary measures to prevent the system, while at the same time Bush can collect or generate related files for offline analysis, to find the root cause of the leak.
2. .2 Configuration. .
WebSphere application server in the management console to enable performance diagnostic consultancy.
(1) Access Management Console -> Servers -> Application Server. .
(2) select the server you want to configure.
(3) In the performance area, select properties and diagnostic consultant configuration. .
(4) as shown in the figure, there are two Tab, runtime and configuration. The difference is that inside the content at runtime without having to restart the server to take effect, but the next time you restart the server, the configuration will be lost. Configuration Tab content only when the server restarts after the entry into force, and the configuration of the content will always be present, unless the landing and removed again.
(5) in other property areas, click the Performance and diagnostic suggestions configuration rules to ensure that the memory leak is running (green arrow). .
2. view the information leakage warning .3.
WebSphere Performance Diagnostics Advisor output information can be displayed in the WebSphere administration console and recorded in the WebSphere Application Server SystemOut. . Log Log file inside. .
(1) SystemOut. .log log.
[8/31/06 13:21:43:545 CST] 00000010 TraceResponse W TUNE9001W: Heap. .
utilization patterns indicate that you may have a memory leak 。.
Additional explanatory data follows. .
Data values for free memory between 8/31/06 1:20 PM and 8/31/06 1:21 PM were。.
consistently below minimum required percentage. .
(2) management console.
- Log Management Console -> Diagnosis -> run-time message -> click on run-time warning. .
3. Java virtual machine profiling and verbose garbage collection.
Further test whether the occurrence of memory leaks, and leakage occurred at a time, period and speed, we can enable the Java virtual machine garbage collection in detail, then analyzes the corresponding log. WebSphere Application Server 6. .1 Use the Java SDK5. .0, In the Window, Linux, AIX, i5/OS, z / Linux and z / OS using the IBM's JAVA virtual machines, Solaris and HP-UX in use on Sun's JVM.Java virtual machine profiling tools interfaces (Java Virtual Machine Tool Interface, JVMTI) support from the running application server's Java virtual machine (JVM) to collect information (e.g., regarding garbage collection data, object and thread state) and supports more comprehensive performance analysis. Once enabled, you can use JVMTI PMI customization options to enable the selected statistics to gather specific data.
3. .1 Enable the Java virtual machine profiling and detailed garbage collection configuration steps. .
Configuration steps:.
1. . In the console navigation tree, click Servers> Application Server. .
2. Click to select the desired application server.
3. . In the "Server Infrastructure", click Java and Process Management -> Process Definition. .
4. In the additional properties, click the Java virtual machine.
5. . Select the Configuration Tab of the detailed options for garbage collection. .
6. In the generic JVM arguments field, enter-agentlib: pmiJvmtiProfiler.
Note: WebSphere6. .1 In, JVM profiling interface to a Java Virtual Machine Tool Interface (JVMTI). Previous versions are JVMPI. If you need JVMPI time, you can enter here-XrunpmiJvmpiProfiler. In addition, to enable a greater impact on performance JVMTI interface to avoid the use in a production environment. .
7. click apply or OK.
8. . Click the Save button. .
9. Restart WebSphere Application Server.
3. .2 GC data analysis tools PMAT. .
In the WebSphere application server's log directory native_stderr. .log file we need memory recovery analysis file. We recommend using the IBM Pattern Modeling and Analysis Tool for Java Garbage Collector tool, called PMAT. Tools for parsing JAVA SDK PMAT detailed memory collection (GC) log, and to provide statistics, charts, analysis and recommend Java heap allocation. PMAT provides rich graphical interface to display Java heap usage, and thus more easily determine whether there is memory problems. The tool can be downloaded from IBM's alphaWorks Web site, in English only. .
We can take the GC files downloaded from the server to the machine, and then PMAT under WebSphere platform choice to open the corresponding GC file for analysis. The following is a fragment, the GC log is quite laborious manual analysis, but need an in-depth understanding of the knowledge of the JVM.
PMAT GC log analysis, this paper gives a summary. The figure, for example, we can see the impact of GC on system performance, and the completion of the garbage collection frequency, and we can see that tool gives the recommended (Recommendations) display system using the Java heap is a continuous increase. .
Further, we can view the details of the GC, click on the Analysis menu, and then select View All Graph, we can select the curve that you want to view. As shown, the red curve represents the use of memory, the blue curve represents each garbage collection after the available memory. Has been a gradual increase in memory usage, available memory decreases over time shows that the system may be a memory leak.
4. . TPV JVM monitoring the situation. .
Another approach is to use TPV and PMI to real-time monitoring JVM, analyze performance curve to determine whether there is a memory leak condition occurs. WebSphere performance monitoring infrastructure (PMI) and the Java virtual machine profiling tools interface (JVMTI) can help us collect the performance of the system state data, use the Tivoli performance Viewer (TPV) graphically display the data (performance counter), you can confirm whether the system is experiencing a memory leak.
4. .1 PMI and TPV. .
PMI provides WebSphere runtime and application resources to conduct a comprehensive set of data. For example, PMI offers database connection pool size, the servlet response time, Enterprise JavaBeans (EJB) method of response time, the Java virtual machine (JVM) garbage collection time and CPU usage, and so on. Use PMI data, you can identify and fix application performance bottlenecks in the server, you can also use PMI data to monitor the application server's health. PMI data can be from the Tivoli Performance Viewer (TPV), the other Tivoli tools, your own applications or third-party tools to monitor and analyze. TPV is the WebSphere Application Server together with the PMI data provided by the graphics viewer. .
Tivoli Performance Viewer (TPV) allows us to view a chart or table, so interpretation of WebSphere performance monitoring infrastructure (PMI) data.
4. .2 PMI's configuration. .
By default, PMI is already turned on, the level is the default (Default). Configuration steps:.
1. . In the console navigation tree, click Monitoring & Adjustment -> Performance Monitoring Infrastructure (PMI). .
2. Select the name of the server you want to configure.
3. Click the Configuration tab, where you can monitor the content according to needs, to select the PMI of any of the statistical information set (none, basic, extended, all, custom). Here we select "Custom." .
Note: If the configure tab, when you restart the server to apply the setting. If you run time tab, and then immediately apply settings.
4. Click the Custom -> in a custom monitor levels of the tree, select the Configuration tab, and then opening the JVM run-time, as needed, to enable or disable the corresponding counter. .
5. Save and restart the WebSphere server.
4. .3 TPV to use. .
Real-time view of TPV performance module:.
(1) in the console navigation tree, click Monitoring and adjustment -> Performance Viewer -> Current Events -> Server Name) -> Performance Module. .
(2) select to view the properties of each module, such as the JVM is running.
(3) Click View Modules button. The right side of the page will display the selected performance data into the form of graphics or switch. Note: Each module has several counters associated with it. The counter displays the data in graphs or tables in the table below. You can select or deselect the box next to the counter, the counter added to the chart or table, or removing. .
TPV display used memory graphics ideally should be a jagged, graph each slope (down) corresponding to a memory garbage collection (Garbage collection), as has the use of the memory of the curve, displayed is not a memory leak condition.
If the test occurred during the following circumstances, there may be a memory leak:. .
-After every garbage collection of used memory value fluctuations.
-TPV corresponding graph of used memory closer to step (staircase), or serious irregular zigzag shape. .
-You can also view the number of objects allocated and freed the difference between the number of objects, if this number is growing, there are memory leaks (if you need to view the number of objects, you need to enable the JVMTI interface and enable the corresponding PMI JVM counter).
On the map, the red curve has been used on behalf of the memory, the overall trend, we can see that already use the memory has been growing. TPV can help find memory leaks, in order to get optimal results, we can repeat the experiment, and each can increase the test of time, such as test 1000,3000 or 5000 page request. .
5. Heap dump file is generated.
WAS6. .1, In using the IBM JDK's platform, can use the following methods, at any time produce the required heapdump file. If the performance of diagnostic consultancy configuration which selected the "Enable automatic heap dump collection, you can automatically located in the WebSphere profile path (eg / opt/IBM/WebSphere/WAS6. .1/profiles/AppSrv01) Files generated heapdump , prepare the user for analysis..
When using the IBM SDK platform, e.g., AIX, Linux and Windows, enabling performance diagnostics tool, if you detect a memory leak occurs, WebSphere automatically generates two heapdump file for subsequent analysis.
We at any time, at any time required to manually generate heap dump file. In WAS6. .1 Profile's bin directory, run the wsadmin scripting client first, and then can call generateHeapDump operation to complete. .
Key steps:.
1. Find the JVM object name. .
set objectName 。.
WebSphere: type = JVM, process =, node =. .
<节点名字>,*]。.
2. Call generateHeapDump operation on JVM MBean. .
$AdminControl invoke $objectName generateHeapDump。.
For example:. .
[root@csspvm bin]# pwd。.
/ Opt/IBM/WebSphere/WAS6. .1/profiles/AppSrv01/bin. .
[root@csspvm bin]# 。./wsadmin。.sh -username root -password demo4you。.
WASX7209I: Connected to process "server1" on node csspvmNode02 using SOAP. .
connector; The type of process is: UnManagedProcess。.
WASX8011W: AdminTask object is not available. .
WASX7029I: For help, enter: "$Help help"。.
wsadmin> set objectName [$ AdminControl queryNames. .
WebSphere:type=JVM,process=server1, node=csspvmNode02,*]。.
WebSphere: name = JVM, process = server1, platform = proxy, node = csspvmNode02,. .
j2eeType=JVM,J2EEServer=server1,。.
version = 6. .1. .0. .0, Type = JVM, mbeanIdentifier = JVM, cell = csspvmNode02Cell, spec = 1. .0. .
wsadmin>$AdminControl invoke $ objectName generateHeapDump。.
/ Opt/IBM/WebSphere/WAS6. .1/profiles/AppSrv01 /. . / Heapdump. .20060904. .075650. .3576. . Phd. .
wsadmin>quit。.
Ideally, when the detection of the problem as soon as generate an initial heap dump, and then closely monitor the situation of memory Shi Yong, Deng Dao had enough of the memory leak of the time, Zai generate another heap dump, Zhe Yang can be compared to a more accurate analysis find the leak. .
Note: the generated HeapDump file is quite expensive, so please only do so when required by the operation.
6 memory leak analysis and diagnosis tools-MDD4J. .
Once the system has a memory leak, and this generates a heap dump. Next, we can put these files from the WebSphere Application Server to offline analysis tools is located on the machine, for offline analysis.
6. .1 Tool description. .
MDD4J (Memory Dump Diagnostic for Java) is a memory leak analysis tool that is used to run WebSphere Application Server virtual machine (JVM) that are generated by the common memory dump (heap dump) format for analysis. A memory dump (Memory dump) analysis aims to determine the actual cause of Java heap memory leaks classes and packages (classes and packages), which would reduce the scope of memory leaks and find the real reason, this analysis also identified application program Java heap footprint of the main components and the relationships between them have. .
This tool supports the following formats: a memory dump format.
-IBM's PHD format (heapdump.. Phd). .
-IBM text heap dump (heapdump. .txt).
-HPROF heap dump format (hprof.. Txt, primarily for Solaris and HP-UX platform). .
-SVC dump (.bin dump., IBM's WebSphere z-Series).
The analysis tool provides two mechanisms: one dump analysis and comparative analysis of two dumps. .
A single dump analysis is most commonly used to automatically in the event of an OutOfMemoryException is triggered by a memory dump. This type of analysis to find suspicious data structure that can be relatively quickly to provide analysis of suspicious objects to leak.
Comparative analysis is used to run the memory leak during application (you can use the Java heap when memory loss) for the two memory dump for analysis. In the run early leak triggered the application is called the baseline memory dump memory dump, leak application runs for some time (to allow greater degree of leakage) after being called to trigger a memory dump main memory dump. Memory leak in the event of a situation, the main memory dump may contain a large number of objects, which object occupied Java heap space will be much larger than the baseline memory dump. .
In order to obtain better results of the analysis, it is recommended that the main memory dump trigger points and baseline memory dump trigger points in time to pull a certain distance, making the total consumption of heap size in points between two trigger substantial growth.
MDD4J analysis results is based on the WEB interface, has the following features:. .
-Lists the results of the analysis, the heap contents, size and growth rate of the summary.
- Lists of suspicious data structure, data types and packages, which are caused by the increasing use of the heap (for comparative analysis) and a larger heap size (for single-dump analysis) the main reason. .
-Ownership relations context view shows the major components of the footprint, as well as a set of summary the main consumption component contains important data type.
- In the heap dump the contents of the interactive tree view, browse function to display all the heap of any object into the reference (in the tree shows only a reference to the other references shown separately) and out references, and the child reaches the size of the object by sort. .
-Navigation feature allows you to select from a list of suspicious objects to the context of all relations, as well as from the content view to go to the Browse view.
- Provide a memory dump of all the objects and data types of table view, the view through a filter and sort the columns. .
6. the use of tools .2.
WebSphere Application Server v6. .1 Fringe CD IBM Support Assistant tool inside the installation file, run the appropriate installation file, MDD4J also be installed as a plug-in. .
Additionally, also available from the IBM technical support site http://www-306. .ibm. .com/software/support/isa/download Support Assistant tool, and then select Update stand-alone installation MDD4J plug-in.
Start Step:. .
(1) program-Support Assitant-> IBM > IBM Support Assistant v3.
(2) Support Assistant window, select Tools -> select WebSphere version. .
Click on the link, MDD4J can open MDD4J tool. In the interface, we can submit a single heap dump file for a single dump analysis, or both to compare two files. You can also analyze the results from a memory dump of the drop-down options before the results of the analysis, in order to view the previous analysis.
View of progress. .
Click upload and analysis "button, the heap dump MDD4J start analysis. In the analysis of the implementation process, the login page will automatically refresh to reflect the currently executing analysis steps, as well as the overall analysis of the progress. If the page for some reason does not refresh, you can click the Refresh button to see the current analysis of the State. If you want to stop profiling, you can click "stop" button, this will be the currently executing module complete termination analysis.
The submission of the heap dump file, MDD4J display of status. .
View the results of the analysis.
The analysis is complete, Mdd4J page will redirect to the "results" page. "Results" page contains four tabs:. .
"Analysis of the Summary tab displays the analysis results, and lists the next group is used to view the results of the analysis of the steps.
"Suspicious object" tab: it shows the four types of suspicious objects, that is, the greatest impact on the growth rate of the data structure, to reach the size of a significant loss of data structures, a large number of instances of object types and a large number of object instances of the Java package. .
"Look at the context and content of the tab displays the main memory dump Java heap footprint in the major components of the context diagram ownership relations, as well as map the contents of the selected node.
"Browse" tab: object reference map based on the implementation of the depth-first traversal, with the tree view shows all the contents of main memory dump. .
Other content, please refer to the Help MDD4J tools document that help documentation with the detailed instructions for use, instead.
7. Summary. .
IBM offers a range of tools to assist users to memory problems of monitoring and analysis, at the appropriate stage of selecting the right tools can help us fix a memory leak. The tools described here are incidental or WebSphere, IBM Tivoli free tool also provides more powerful monitoring and diagnostics capabilities, such as ITCAM (IBM Tivoli Composite Application Management), you can choose according to the actual situation.
Add to: Del. . Icio. . Us. .
Tag：work java 。.
Reference Address:. .
Astrocat was published on 10: 21 | editor | topics | forward | continue sharing 0.
