Recently, I encountered a typical memory leak issue in a microservice based on a dynamic rules engine. The container had modest specifications (1 Core, 1GB RAM), and after running for a while, Kubernetes would forcibly restart it with an OOMKilled status. Ultimately, we traced it back to the script engine generating a massive number of uncollectable ClassLoaders at the underlying level, causing the Metaspace to grow continuously without ever shrinking. This post documents the troubleshooting process and compares the underlying differences between various script engines.

1. The Symptom: Normal Heap, Ever-Growing Metaspace

The business container was configured with 1 Core / 1GB RAM. The JVM parameters set the Heap to 512MB, while Metaspace was left without an explicit upper limit (defaulting to unlimited).

Monitoring data revealed: * Normal Heap Water Level: Daily usage hovered around 40%. Minor GCs were cleanly clearing out short-lived objects. * Continuous Metaspace Growth: Starting at a few tens of megabytes at boot, it grew linearly by 20~30MB per day, and never dropped once. * Rare Full GCs: Because Heap space was plentiful and Old Generation accumulation was slow, the JVM rarely triggered Full GCs. * The Result: Metaspace kept expanding, eating up more and more of the container's physical memory. When total memory usage approached 1GB, the Linux kernel's OOM Killer was triggered, directly killing and restarting the container.

2. Why Didn't Full GC Reclaim Metaspace?

Many developers have a misconception: they believe that triggering a Full GC will automatically clean up obsolete classes in the Metaspace. In reality, the conditions for Metaspace reclamation are extremely strict.

For a class and its metadata to be unloaded, three conditions must be met simultaneously: 1. All instances of that class have been GC'd. 2. The ClassLoader instance that loaded the class has been GC'd. 3. There are no references to the corresponding java.lang.Class object.

The core is point 2: The ClassLoader must die first before the classes it loaded can die.

In our scenario, because the Heap was large enough, short-lived business objects were cleaned up during Minor GCs. The Old Generation grew slowly, so the JVM lacked the pressure to trigger Full GCs. Even if an occasional Full GC occurred, some dynamically generated ClassLoaders were indirectly referenced by hidden GC Roots (like ThreadLocal or unclosed Context objects). This meant the reclamation conditions were still not met, causing Metaspace to only increase, never decrease.

For a small 1c1g container, this slow leak is fatal. It hits the physical memory ceiling in just a few days.

3. Step-by-Step: Pinpointing the Cause from NMT to Arthas

Suspecting a class loading leak, we didn't rush to pull a multi-GB Heap Dump (doing so on a 1c1g container easily causes the app to freeze or trigger a secondary OOM). Instead, we used a lightweight combination of tools to pinpoint the issue.

3.1 Macro Qualification: Confirming the Memory Growth Area with NMT

First, we enabled Native Memory Tracking in the JVM startup parameters (-XX:NativeMemoryTracking=summary). After the application started and stabilized, we ran the command to establish a baseline:

jcmd <pid> VM.native_memory baseline

After running for a while, we observed the memory diff:

jcmd <pid> VM.native_memory summary.diff

In the output, other areas (like Heap and Thread) showed only minimal fluctuation. However, the Class area showed significant positive growth (marked with a +):

-                    Class (reserved=1283311KB +13614KB, committed=279919KB +17070KB)
                            (classes #30595 +1610)
                              (malloc=19695KB +1326KB #133906 +6398) 
                              (mmap: reserved=1263616KB +12288KB, committed=260224KB +15744KB)

This NMT report directly qualifies the essence of the problem. We need to highlight three key metrics:

• (classes #30595 +1610): This is the core smoking gun! The app loaded 30,595 classes at startup, and during the observation window, 1,610 new classes were added. For a service running steady business logic, the class count should be constant. This continuous increase is absolutely abnormal.
• committed=279919KB +17070KB: This means the JVM actually requested about 16.6MB of physical memory from the OS to store these new class metadata. This memory belongs to Metaspace and is not managed by regular Heap GCs.
• malloc=... +1326KB #... +6398: The number of malloc calls increased by 6,398. At the JVM underlying level, loading a class requires not only mmap to allocate Metaspace for bytecode structures but also malloc to allocate C++ level metadata objects like InstanceKlass. The 6,398 new malloc allocations perfectly align with the 1,610 new classes.

NMT Conclusion: The root cause is that class loaders are continuously and frequently creating new classes that cannot be unloaded, causing Metaspace to grow indefinitely.

3.2 Micro Localization: Locking the Culprit ClassLoader with JFR

Knowing that classes are constantly increasing, the next step was to find out who was frantically loading them. We used JFR (Java Flight Recorder), which has a minimal performance footprint and is perfect for dynamic enabling in production.

# Start JFR recording, capturing ClassLoad events for 60 seconds
jcmd <pid> JFR.start name=classload settings=profile duration=60s filename=/tmp/classload.jfr

We downloaded the .jfr file locally and analyzed it using JDK Mission Control (JMC). Filtering for Class Load and Class Define events in the event browser revealed extremely dense loading records. The ClassLoader loading these dynamic classes was exclusively org.mozilla.javascript.DefiningClassLoader.

3.3 Catching the Culprit: Finding the Business Call Source with Arthas

We found the culprit component, but since it's an underlying library, we had to find out which business code was triggering these creations. Enter Arthas.

Using Arthas's stack command, we directly traced the constructor of DefiningClassLoader and printed the full call stack:

stack org.mozilla.javascript.DefiningClassLoader <init>

Soon, the console spat out the call chain, and the truth came to light:

[arthas@12345] stack org.mozilla.javascript.DefiningClassLoader <init>
... omitted internal stack frames ...
at org.mozilla.javascript.Context.compileString(Context.java)
at com.xxx.business.RuleEngineService.executeDynamicRule(RuleEngineService.java:45)
...

Following the call stack, we pinpointed line 45 in RuleEngineService.java:

Context cx = Context.enter();
try {
    // 🚨 Fatal flaw here: Enabled compilation optimization, compiling JS to Java bytecode
    cx.setOptimizationLevel(9); 
    // Dynamically compiles based on different rules for every request
    Script script = cx.compileString(dynamicRuleScript, "rule_" + ruleId, 1, null);
    script.exec(cx, scope);
} finally {
    Context.exit();
}

4. Root Cause Analysis: The Underlying Disaster of Rhino's Compiled Mode

The problem lies in the underlying mechanism when OptimizationLevel > -1:

When Rhino compiles JS code, it transforms the JS into Java bytecode. To support the independent unloading of individual scripts, Rhino's design strategy is: create a separate DefiningClassLoader for each compiled script class.

In our dynamic rules scenario, tens of thousands of distinct rule inputs daily meant tens of thousands of DefiningClassLoader instances and their corresponding Classes were injected into Metaspace every day. Coupled with complex Web container reference chains, these Loaders were extremely difficult for GC to collect, eventually turning into zombies in the Metaspace.

5. Engine Comparison: Rhino vs. Nashorn vs. GraalJS

A dynamic script engine's ClassLoader strategy directly dictates the health of the Metaspace. Let's do a horizontal comparison:

5.1 Rhino (Compiled Mode, Opt > -1)

• Execution Mechanism: JS code -> Generate Java bytecode -> Instantiate Class.
• ClassLoader Strategy: 1 Script = 1 DefiningClassLoader.
• Impact: Extreme fragmentation, highly prone to triggering Metaspace OOM. Should be absolutely forbidden on small-spec containers.

5.2 Rhino (Interpreted Mode, Opt = -1)

• Execution Mechanism: JS code -> Internal AST traversal interpreted execution -> No bytecode generated.
• ClassLoader Strategy: 0 Dynamic ClassLoaders. Scripts are uniformly mapped to a fixed InterpretedScript class inside the Jar, loaded by AppClassLoader.
• Impact: Metaspace is absolutely safe, zero leak risk. The tradeoff is slower pure interpretation speed and lack of support for modern ES6+ syntax.

5.3 Nashorn (Built-in JDK 8)

• Execution Mechanism: JS code -> Generate Java bytecode -> Instantiate Class.
• ClassLoader Strategy: All scripts share 1 DynamicClassLoader.
• Impact: Avoids fragmentation but makes unloading much harder. As long as 1 script class is in use, the entire massive Loader and all Classes it loaded cannot be GC'd. Long-term running still poses a Metaspace growth risk. Officially deprecated since JDK 15.

5.4 GraalJS (The Modern Ultimate Solution)

• Execution Mechanism: JS code -> Truffle AST -> JVM directly interprets and JIT-optimizes the AST itself -> Generates no Java bytecode.
• ClassLoader Strategy: 0 Dynamic ClassLoaders. Code logic is expressed in plain Java objects (AST nodes), with no dynamic class generation.
• Impact: Objects die and are GC'd directly in the normal Heap, fundamentally circumventing Metaspace leaks by design. Excellent performance, full support for modern ES6+ syntax.

Simple Test Verification: Loop compile 5 different scripts and observe the ClassLoader HashCode of underlying objects. * Rhino (Compiled): HashCode changes every time (constantly creating new Loaders) * Nashorn: HashCode remains consistent (sharing the same Loader) * GraalJS: HashCode remains consistent (reusing AppClassLoader, no dynamic class generation)

6. Pragmatic Recommendations (By JDK Version)

Addressing this OOM issue in a 1c1g container, and considering different tech stack realities, here are pragmatic resolution paths:

🥇 For JDK 8 / JDK 11 Projects: Rhino Interpreted Mode First

On older JDK versions, it is not recommended to forcefully introduce GraalJS. The last version of GraalJS supporting JDK 8/11 is 22.3.3; not only does it introduce a massive 20MB+ third-party dependency, but that version is also end-of-life, carrying unknown risks and technical debt.

The most stable, lowest-cost stopgap is: Stay on Rhino, but immediately switch to interpreted mode.

Context cx = Context.enter();
try {
    // Turn off bytecode generation, eliminate DefiningClassLoader creation
    cx.setOptimizationLevel(-1); 
    // ... execute script logic
} finally {
    Context.exit();
}

Pros: Only a one-line code change, the Rhino Jar is only 1MB, and Metaspace leakage drops to zero instantly. Cons: Slower execution speed, no ES6 syntax. However, for most backend lightweight rules engines (like simple condition checks, property mappings), the performance hit of interpreted mode is usually within acceptable limits.

🥈 For JDK 17 / JDK 21+ Projects: Embrace GraalJS

If you are already enjoying the dividends of modern JDKs, GraalJS is the only right path. It completely discards the old bytecode generation mechanism, eradicating class leaks at the architectural root while bringing top-tier execution performance and full modern syntax support.

Maven dependency config (for JDK 17+):

<dependency>
    <groupId>org.graalvm.polyglot</groupId>
    <artifactId>polyglot</artifactId>
    <version>23.1.1</version> <!-- Use latest version for JDK 17+ -->
</dependency>
<dependency>
    <groupId>org.graalvm.js</groupId>
    <artifactId>js</artifactId>
    <version>23.1.1</version>
</dependency>

🔧 Ops Safety Net: Limit Metaspace Upper Bound

Regardless of the solution chosen, for small containers like 1c1g, you must set an upper limit for Metaspace to prevent infinite expansion from eating up physical memory and getting killed by the kernel. Even if there is a leak, this causes the JVM to throw an OOM exception and restart early, rather than being silently killed by the kernel, leaving no traces for troubleshooting.

-XX:MaxMetaspaceSize=256m  # Recommended to limit to 256m or less for 1c1g containers, leaving ample room for Heap and Native

💡 Additional Performance Optimization: Cache Compilation Results

Whether using Rhino interpreted mode or GraalJS, the "compile/parse" action of dynamic scripts consumes CPU. For high-frequency rules with fixed content, make sure to parse them at app startup or first execution, and cache the Script / Value objects for reuse, avoiding re-parsing strings on every request.

Conclusion

In small-spec containers like 1c1g, any minor memory leak is rapidly amplified. When troubleshooting such issues, using NMT to qualify the area, JFR to qualify the component, and Arthas to find the call stack is a low-impact, high-efficiency practical combo.

Dynamically generating Java bytecode and relying on ClassLoader unloading to clean up Metaspace is an extremely fragile design in modern JVMs (especially given the G1/ZGC trend of reducing Full GCs). When you encounter Metaspace growing without dropping, check the ClassLoader source first. When selecting a fix, be sure to consider your JDK version: Old projects should use Rhino interpreted mode for stable stopgap measures; new projects should use GraalJS for a fundamental cure. Do not introduce heavy, EOL dependencies onto older JDKs just for theoretical engine performance—that will only snowball your technical debt.