再记录一次Full GC分析的案例

问题描述：

部分同事返回API无法正常访问，经过测试发现API响应非常慢，需要十几秒的时间——以前只需要30毫秒左右。第一反应会不会是服务器的内存、CPU的问题，先使用TOP命令，发现有一个API应用的CPU占用非常高

                                                                                                                           
	28674 root      20   0 3926768 478440  13316 S 140.6  2.9   1216:52 java

使用jstat命令查看GC的情况：

	jstat -gc 28674 5s

发现5秒钟就会进行30次FULL GC

	 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT   
	1024.0 1024.0  0.0    0.0   151552.0 151552.0  153600.0   153596.2  36696.0 34801.6 4352.0 4011.6  18800   72.465 102817 11787.917 11860.382
	1024.0 1024.0  0.0    0.0   151552.0 151552.0  153600.0   153596.2  36696.0 34801.6 4352.0 4011.6  18800   72.465 102847 11791.866 11864.33

接着查看堆内存使用情况

		jmap -heap 28674

可以看到分配的300M堆内存已经用完

		Attaching to process ID 28674, please wait...
		Debugger attached successfully.
		Server compiler detected.
		JVM version is 25.20-b23
		
		using thread-local object allocation.
		Parallel GC with 4 thread(s)
		
		Heap Configuration:
		   MinHeapFreeRatio         = 0
		   MaxHeapFreeRatio         = 100
		   MaxHeapSize              = 314572800 (300.0MB)
		   NewSize                  = 157286400 (150.0MB)
		   MaxNewSize               = 157286400 (150.0MB)
		   OldSize                  = 157286400 (150.0MB)
		   NewRatio                 = 1
		   SurvivorRatio            = 8
		   MetaspaceSize            = 21807104 (20.796875MB)
		   CompressedClassSpaceSize = 1073741824 (1024.0MB)
		   MaxMetaspaceSize         = 17592186044415 MB
		   G1HeapRegionSize         = 0 (0.0MB)
		
		Heap Usage:
		PS Young Generation
		Eden Space:
		   capacity = 155189248 (148.0MB)
		   used     = 155189248 (148.0MB)
		   free     = 0 (0.0MB)
		   100.0% used
		From Space:
		   capacity = 1048576 (1.0MB)
		   used     = 0 (0.0MB)
		   free     = 1048576 (1.0MB)
		   0.0% used
		To Space:
		   capacity = 1048576 (1.0MB)
		   used     = 0 (0.0MB)
		   free     = 1048576 (1.0MB)
		   0.0% used
		PS Old Generation
		   capacity = 157286400 (150.0MB)
		   used     = 157282464 (149.99624633789062MB)
		   free     = 3936 (0.003753662109375MB)
		   99.99749755859375% used
		
		11367 interned Strings occupying 1025048 bytes.

将堆内存dump后下载到本地（压缩后传输）

	jmap -dump:format=b,file=api_dump 28674

用jvisualvm工具打开dump文件，看不出什么内容，直接使用mat打开堆文件

根据引用链，找到可能因为内存溢出的对象，选择list objects -> with outgoing reference，可以看到这个对象的内部详情

根据上述的详情，初步判断是断路器的逻辑中出现了问题。

再记录一次Full GC分析的案例

最后更新：2020-04-24