Series Navigation - Part 1: What They Are and Why They Matter - Part 2: How Spring Helps You Avoid Them - Part 3: Built-in Detection in Major Connection Pools - Part 4: Building a Custom Connection Monitor - Part 5: Root Cause Analysis with Heap Dumps (current)

Slug: connection-pool-leak-heapdump-analysis

---

1. When Should You Use a Heap Dump?

The methods covered in previous parts have their limitations:

Method	Limitation
Built-in pool detection	Prints the stack trace at borrow time, but cannot show which object is currently holding the connection
Custom monitor	Same — only captures the borrow-time context

A heap dump analysis is the most powerful approach when: - Logs confirm a leak exists, but the root cause cannot be found in the code - The leaked connection is held by a complex object graph that stack traces alone cannot reveal - You need to know precisely which object is holding the leaked connection at this moment

---

2. Capturing a Heap Dump

# 1. Find the Java process PID
jps -l | grep your-service

# 2. Capture the heap dump
#
# With `live`: exports only live objects, triggers a Full GC
# Recommended for connection leak analysis — excludes already-reclaimable objects
jmap -dump:live,format=b,file=/tmp/heapdump.hprof <pid>
#
# Without `live`: exports all objects including reclaimable ones
# Does not necessarily trigger Full GC, but produces a larger file with more noise
# jmap -dump:format=b,file=/tmp/heapdump.hprof <pid>

# 3. Copy out of the container if needed
kubectl cp <pod-name>:/tmp/heapdump.hprof ./heapdump.hprof

Production notes: - jmap -dump:live triggers a Full GC and may cause a brief pause. Run during off-peak hours and assess the impact beforehand. - Consider setting -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/ to capture a dump automatically at OOM time, avoiding the need for manual intervention.

---

3. Analyzing with Eclipse MAT

Download Eclipse MAT and open the .hprof file.

Step 1: Dominator Tree

Open Window → Heap Dump Details → Dominator Tree. Sort by Retained Heap descending. Look for connection objects near the top.

Connection class names by pool:

Pool	Class Name in Heap Dump
Druid	com.alibaba.druid.pool.DruidPooledConnection
HikariCP	com.zaxxer.hikari.pool.HikariProxyConnection (generated by Javassist at compile time)
Tomcat JDBC	org.apache.tomcat.jdbc.pool.PooledConnection
DBCP2	org.apache.commons.dbcp2.PoolableConnection

Step 2: OQL Queries for Active Connections

Open the OQL console via File → OQL.

Druid — find non-abandoned connections:

SELECT s.@objectId AS id, s.borrowCount AS borrowCount
FROM com.alibaba.druid.pool.DruidPooledConnection s
WHERE s.abandoned = false

DBCP2 — find non-closed connections:

SELECT s.@objectId AS id
FROM org.apache.commons.dbcp2.PoolableConnection s
WHERE s._closed = false

HikariCP note: HikariProxyConnection is generated by Javassist at compile time. Whether a connection is closed is determined by checking whether the internal delegate field is the CLOSED_SENTINEL object — not a simple boolean isClosed field. Querying isClosed = false via OQL is likely to produce unexpected results. Instead, search for HikariProxyConnection directly in the Dominator Tree and manually inspect the delegate field value.

Step 3: Tracing the GC Root Path — Identify the Holder

This is the most important step: find out exactly which object is holding the leaked connection.

Right-click a connection object → Merge Shortest Paths to GC Roots → exclude all phantom/weak/soft etc. references

MAT displays the complete reference chain from the GC root to the connection object. For example:

GC Root: Thread "http-nio-8080-exec-3"
  └── OrderService$$EnhancerBySpringCGLIB
        └── conn (java.sql.Connection)  ← the leaked connection

From this chain you can identify: - Which thread is holding the connection - Which field on which business class is holding it

Step 4: Correlate with the Borrow-Time Stack Trace

Cross-reference the holder information from MAT (thread name, class name) with the borrow-time stack trace printed by the connection pool or the custom monitor. Together, they give a complete picture of both how the connection was borrowed and what is currently holding it.

---

4. Common Leak Patterns

Pattern	Likely Cause
Holder is the same business method across multiple connections	That method is missing close() or has an unhandled exception path
Holder is a Spring transaction-related object	Misconfigured transaction boundary; transaction never committed or rolled back
Holder is a worker thread in a thread pool	Async task manually acquired a connection without closing it, or an exception was swallowed
Holder varies across multiple different threads	Concurrent scenario with a connection leak in shared logic

---

5. Pros and Cons

Pros	Cons
Precisely identifies the object and method holding the connection	live dump triggers Full GC — use with caution in production
Can reveal complex leaks that other methods cannot locate	Heap dump files can be very large (several GB)
Can analyze connection state at any point in time	Higher skill requirement — familiarity with MAT is needed

---

6. Series Summary

This five-part series covers the full lifecycle of connection leak detection, prevention, and analysis.

Part	Core Topic	Target Audience
Part 1: What and Why	Concepts, symptoms, common causes	All developers
Part 2: Spring Prevention	JdbcTemplate, @Transactional, async thread scenarios	Spring developers
Part 3: Built-in Detection	Druid / HikariCP / Tomcat JDBC / DBCP2 configuration	All developers
Part 4: Custom Monitor	Decorator pattern, alerting system integration	Teams with custom monitoring needs
Part 5: Heap Dump Analysis	MAT usage, OQL queries, GC root tracing	Engineers debugging production incidents

Recommended defense-in-depth strategy:

Prevent (Part 2)
    Use JdbcTemplate / Spring Data / MyBatis
    Follow @Transactional best practices
    Use framework-managed connections in async threads
        ↓
Detect (Part 3)
    Enable built-in pool leak detection as a safety net
        ↓
Monitor (Part 4)
    Custom monitor integrated with alerting systems for continuous observation
        ↓
Analyze (Part 5)
    Use heap dumps to pinpoint root cause when a production incident occurs

---

This series is based on Druid 1.2.x, HikariCP 5.x (Java 11+; use HikariCP 3.x for Java 8), Tomcat JDBC 10.x, and DBCP2 2.9.x. All code examples are for reference only. Adjust them to your specific environment before using in production.