Featured post
java - Better ways to identify objects not getting garbage collected? -
in nutshell
i have program gradually using more , more memory on time. using jmap , jhat try , diagnose still not quite there.
background
the program long-running server backed hbase datastore providing thrift service bunch of other stuff. however, after running few days, hit allocated heap limit, , thrash , forth time spent in garbage collection. seem references getting kept lot of data somewhere
what i've done far
after fiddling jstat , jconsole ended taking heapdumps jmap of running process, , run through jhat, , numbers simple don't add anywhere near memory utilisation
jmap -f -dump:live,format=b,file=heap.dump 12765
jmap -f -dump:format=b,file=heap.all 12765
some stuff off top of histogram
class instance count total size class [b 7493 228042570 class java.util.hashmap$entry 2833152 79328256 class [ljava.util.hashmap$entry; 541 33647856 class [ljava.lang.object; 303698 29106440 class java.lang.long 2851889 22815112 class org.apache.hadoop.hbase.keyvalue 303593 13358092 class org.apache.hadoop.hbase.client.result 303592 9714944 class [i 14074 9146580 class java.util.linkedlist$entry 303841 7292184 class [lorg.apache.hadoop.hbase.keyvalue; 303592 7286208 class org.apache.hadoop.hbase.io.immutablebyteswritable 305097 4881552 class java.util.arraylist 302633 4842128 class [lorg.apache.hadoop.hbase.client.result; 297 2433488 class [c 5391 320190
while totals here don't add it, @ point heap dump taken process using on 1gb of memory.
the immediate apparent culprit seems i'm leaving hbase result , keyvalue entries on place. trying trace references, hit
object @ 0x2aab091e46d0 instance of org.apache.hadoop.hbase.ipc.hbaseclient$call@0x2aab091e46d0 (53 bytes) class: class org.apache.hadoop.hbase.ipc.hbaseclient$call instance data members: done (z) : true error (l) : <null> id (i) : 57316 param (l) : org.apache.hadoop.hbase.ipc.hbaserpc$invocation@0x2aab091e4678 (48 bytes) this$0 (l) : org.apache.hadoop.hbase.ipc.hbaseclient@0x2aaabfb78f30 (86 bytes) value (l) : org.apache.hadoop.hbase.io.hbaseobjectwritable@0x2aab092e31c0 (40 bytes) references object: other queries reference chains rootset exclude weak refs include weak refs objects reachable here
help needed:
there seems no references final hbaseclient$call object(or of others it, each hold thousand or keyvalues internal data). shouldn't getting gced? misunderstanding how gc works or extent jhat verify references? if further can track down "missing" memory? other steps can take track down?
check java memory monitoring article link
this may on come issue http://java.sun.com/developer/technicalarticles/j2se/monitoring/
- Get link
- X
- Other Apps
Comments
Post a Comment