Series Navigation - Part 1: What They Are and Why They Matter - Part 2: How Spring Helps You Avoid Them - Part 3: Built-in Detection in Major Connection Pools - Part 4: Building a Custom Connection Monitor (current) - Part 5: Root Cause Analysis with Heap Dumps

---

1. Why Go Beyond Built-in Detection?

Built-in detection can identify problems, but it has limitations:

Limitation	Description
Single alerting channel	Can only print logs; cannot integrate with Prometheus, PagerDuty, or other systems
No historical trends	Cannot record the distribution of connection hold times over time, making slow degradation hard to spot
Coarse-grained configuration	Cannot set different thresholds for different business scenarios
Pool-specific configuration	Switching connection pools requires reconfiguration

A custom monitor wraps the DataSource using the decorator pattern, instrumenting the connection layer to enable: - Custom thresholds and alerting rules - Integration with any monitoring or alerting system - Pool-agnostic implementation that works with any connection pool

---

2. Why Not Spring AOP?

A common misconception is using Spring AOP to intercept Connection.close() to track connection returns:

// ❌ This does not work
@Around("execution(* java.sql.Connection.close())")
public Object aroundClose(ProceedingJoinPoint pjp) throws Throwable {
    // Connection is not a Spring Bean — Spring AOP cannot intercept its methods
}

Why: Spring AOP is proxy-based and can only intercept methods on Spring-managed beans. Connection objects are created internally by the connection pool and are not Spring beans, so Spring AOP cannot intercept Connection.close().

The correct approach is the decorator pattern: wrap the real connection in a proxy returned from getConnection(), and perform tracking logic inside the proxy's close().

---

3. Implementation

The overall design:

Business code
    │
    │ getConnection()
    ▼
TrackingDataSource          ← decorator: intercepts getConnection()
    │
    ├── records borrow info
    └── returns a proxy Connection
              │
              │ close() is called
              ▼
    TrackingConnection       ← proxy: intercepts close()
              │
              └── records return info
              ▼
    Real Connection (pool internal)

ConnectionTracker            ← background thread: periodically scans for timed-out connections

Step 1: Borrow Info Record

public class BorrowInfo {
    public final long borrowTime;
    public final StackTraceElement[] stackTrace;
    public final String threadName;

    public BorrowInfo() {
        this.borrowTime = System.currentTimeMillis();
        this.stackTrace = Thread.currentThread().getStackTrace();
        this.threadName = Thread.currentThread().getName();
    }
}

Step 2: Connection Tracker

public class ConnectionTracker {
    private static final Logger log = LoggerFactory.getLogger(ConnectionTracker.class);

    // keyed by identityHashCode to avoid issues if Connection implementations
    // override equals/hashCode
    private static final ConcurrentHashMap<Integer, BorrowInfo> activeConnections =
            new ConcurrentHashMap<>();

    private static final ScheduledExecutorService scheduler =
            Executors.newSingleThreadScheduledExecutor(r -> {
                Thread t = new Thread(r, "connection-leak-detector");
                t.setDaemon(true);
                return t;
            });

    private static final long LEAK_THRESHOLD_MS    = 300_000;    // 5 minutes
    private static final long CLEANUP_THRESHOLD_MS = 3_600_000;  // 1 hour

    static {
        scheduler.scheduleAtFixedRate(() -> {
            long now = System.currentTimeMillis();
            activeConnections.forEach((id, info) -> {
                long duration = now - info.borrowTime;
                if (duration > CLEANUP_THRESHOLD_MS) {
                    // forcibly remove from the tracker to prevent the tracker itself from leaking memory
                    activeConnections.remove(id);
                    log.error("Connection (id={}) held for {}ms, forcibly removed from tracker. thread: {}",
                            id, duration, info.threadName);
                } else if (duration > LEAK_THRESHOLD_MS) {
                    log.error("Potential connection leak! id={}, held for {}ms, thread: {}",
                            id, duration, info.threadName);
                    for (StackTraceElement ste : info.stackTrace) {
                        log.error("  at {}", ste);
                    }
                }
            });
        }, 60, 60, TimeUnit.SECONDS);
    }

    public static void onBorrow(int connectionId) {
        activeConnections.put(connectionId, new BorrowInfo());
    }

    public static void onReturn(int connectionId) {
        BorrowInfo info = activeConnections.remove(connectionId);
        if (info != null) {
            long duration = System.currentTimeMillis() - info.borrowTime;
            if (duration > 30_000) {
                log.warn("Connection (id={}) held for {}ms, thread: {}",
                        connectionId, duration, info.threadName);
            }
        }
    }
}

Step 3: Connection Proxy — Intercept close()

public class TrackingConnection implements InvocationHandler {
    private final Connection realConnection;
    private final int connectionId;

    public TrackingConnection(Connection realConnection) {
        this.realConnection = realConnection;
        this.connectionId = System.identityHashCode(realConnection);
    }

    public static Connection wrap(Connection realConnection) {
        return (Connection) Proxy.newProxyInstance(
                realConnection.getClass().getClassLoader(),
                new Class[]{Connection.class},
                new TrackingConnection(realConnection)
        );
    }

    @Override
    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        if ("close".equals(method.getName())) {
            try {
                return method.invoke(realConnection, args);
            } finally {
                // record the return even if close() throws an exception
                ConnectionTracker.onReturn(connectionId);
            }
        }
        return method.invoke(realConnection, args);
    }
}

Step 4: DataSource Decorator

public class TrackingDataSource implements DataSource {
    private final DataSource delegate;

    public TrackingDataSource(DataSource delegate) {
        this.delegate = delegate;
    }

    @Override
    public Connection getConnection() throws SQLException {
        Connection real = delegate.getConnection();
        int connectionId = System.identityHashCode(real);
        ConnectionTracker.onBorrow(connectionId);
        return TrackingConnection.wrap(real);
    }

    @Override
    public Connection getConnection(String username, String password) throws SQLException {
        Connection real = delegate.getConnection(username, password);
        int connectionId = System.identityHashCode(real);
        ConnectionTracker.onBorrow(connectionId);
        return TrackingConnection.wrap(real);
    }

    // delegate everything else
    @Override
    public PrintWriter getLogWriter() throws SQLException { return delegate.getLogWriter(); }
    @Override
    public void setLogWriter(PrintWriter out) throws SQLException { delegate.setLogWriter(out); }
    @Override
    public void setLoginTimeout(int seconds) throws SQLException { delegate.setLoginTimeout(seconds); }
    @Override
    public int getLoginTimeout() throws SQLException { return delegate.getLoginTimeout(); }
    @Override
    public Logger getParentLogger() throws SQLFeatureNotSupportedException { return delegate.getParentLogger(); }
    @Override
    public <T> T unwrap(Class<T> iface) throws SQLException { return delegate.unwrap(iface); }
    @Override
    public boolean isWrapperFor(Class<?> iface) throws SQLException { return delegate.isWrapperFor(iface); }
}

Step 5: Register in Spring

@Configuration
public class DataSourceConfig {

    @Bean
    @ConfigurationProperties(prefix = "spring.datasource.hikari")
    public DataSource hikariDataSource() {
        return DataSourceBuilder.create().type(HikariDataSource.class).build();
    }

    @Bean
    @Primary
    public DataSource dataSource(DataSource hikariDataSource) {
        // wrap the real pool with TrackingDataSource — fully transparent to business code
        return new TrackingDataSource(hikariDataSource);
    }
}

---

4. Extension: Integrating with an Alerting System

The alerting logic in ConnectionTracker can be extended to push to any system:

} else if (duration > LEAK_THRESHOLD_MS) {
    String message = String.format(
        "Potential connection leak! id=%d, held=%dms, thread=%s",
        id, duration, info.threadName
    );

    // integrate with Prometheus
    connectionLeakCounter.increment();

    // integrate with PagerDuty / Slack / DingTalk
    alertService.send(message);

    // keep the log as well
    log.error(message);
}

---

5. Pros and Cons

Pros	Cons
Fully customizable thresholds and alert rules	Requires custom development
Can track historical trends and integrate with monitoring systems	Minor overhead from JDK dynamic proxy
Pool-agnostic — works with any connection pool	Extremely rare hash collision risk with System.identityHashCode
Suitable for continuous production monitoring	Duplicate log entries if used alongside pool built-in detection

---

This series is based on Druid 1.2.x, HikariCP 5.x (Java 11+; use HikariCP 3.x for Java 8), Tomcat JDBC 10.x, and DBCP2 2.9.x. All code examples are for reference only. Adjust them to your specific environment before using in production.