Debugging a C2 JIT Compiler Infinite Loop on AArch64

2026-06-26 13:11:59 · 11 views

Debugging a C2 JIT Compiler Infinite Loop on AArch64

tl;dr: A production Java 8 service on AArch64 experienced 100% CPU on a single core caused by a C2 JIT compiler infinite loop. The root cause was a cycle in C2's Ideal Graph triggered by dead code elimination of MemBarCPUOrder nodes. We confirmed the cycle through raw memory analysis of a core dump and resolved the issue in production by switching to Alibaba Dragonwell 8, which includes a depth guard patch for this exact loop.

Prologue

A mysterious CPU spike appeared on a production Java 8 service running on OpenJDK 8u442 on AArch64 processors.

The symptom: one core was pinned at 100%, the entire application became sluggish, and top showed C2 CompilerThread0 as the culprit.

This issue was not immediately reproducible on demand. It did not manifest during startup or under light load. However, after the application had been running for a period of time — typically under sustained mixed workload — the hang would eventually and reliably surface. This made it particularly challenging to diagnose, as it depended on the JVM's internal compilation state, code cache occupancy, and the specific set of methods selected for C2 compilation over time.

The problem is rooted in the JVM's C2 compiler internals and cannot be triggered by a simple standalone Java test case. It emerges from complex interactions between method inlining, dead code elimination, and the memory barrier nodes generated for the AArch64 weak memory model.

This is the story of how we tracked down this JIT compiler infinite loop, identified the root cause, and explored potential fixes — a tale of CPU architectures, memory barriers, and a hidden cycle in the C2 Ideal Graph.

1. The Flame Graph Revelation

We started with a flame graph taken during the incident:

80% of samples landed inside MemNode::can_see_stored_value
21% were in MergeMemNode::memory_at

These two functions, deep in the OpenJDK C2 compiler, were burning cycles. The graph didn't show any progress — the threads were stuck in the same stack frame again and again. This was definitely an infinite loop inside the C2 compiler.

2. The Suspect Loop

A careful reading of memnode.cpp (from OpenJDK 8u442 source) revealed a potential while loop that could spin without exit:

while (current->is_Proj()) {
    int opc = current->in(0)->Opcode();
    if (opc == Op_MemBarRelease || opc == Op_StoreFence ||
        opc == Op_MemBarAcquire || opc == Op_MemBarCPUOrder ...) {
        Node* mem = current->in(0)->in(TypeFunc::Memory);
        if (mem->is_MergeMem()) {
            MergeMemNode* merge = mem->as_MergeMem();
            Node* new_st = merge->memory_at(alias_idx);
            if (new_st == merge->base_memory()) {
                current = new_st;    // ← keep searching
                continue;            // ← GO BACK TO WHILE — INFINITE LOOP RISK
            }
            result = new_st;
        }
    }
    break;
}

Note: In the actual OpenJDK source, the condition variable shown as final in some listings is named is_final or similar, since final is a reserved keyword in C++11 and later. This article uses simplified names for clarity.

The continue inside the while combined with current = new_st is the critical pattern. If new_st equals current, the loop never terminates.

3. The Platform Difference: Why Only AArch64?

We tried to reproduce on x86 – nothing. On AArch64 – the hang would eventually appear after sustained operation.

Why?

Memory Model.

x86 (TSO) is strongly ordered. C2 rarely inserts MemBarCPUOrder or MemBarRelease barriers for object allocation; they are often folded away.
AArch64 (weak memory model) requires explicit barriers for safe publication. Every new Object() inserted into a loop with a volatile field generates MemBarCPUOrder and MemBarRelease nodes in the Ideal Graph.

The bug only fires when:

There is a loop that creates objects and writes to a volatile field.
C2 performs dead code elimination on a path inside that loop.
The cleanup folds a MergeMem node and makes its base_memory point back to the loop's own Proj node.

This creates a cycle in the data-flow graph:

Proj → MemBarCPUOrder → MergeMem → base_memory → Proj (again)

The while loop then runs current = new_st; continue infinitely.

4. Getting the Core Dump — The Difficult Dance

In a container environment with no gdb or perf inside, we had to work from the host.

First attempt: gcore from host → failed with failed to read a valid object file image from memory.

Second attempt: nsenter → didn't have gdb.

Third attempt: direct kill -6 → core file was swallowed by systemd-coredump due to disk space limits.

Final solution:

Temporarily changed core_pattern on host to a directory with enough space: bash echo "/data/coredump/core.%e.%p.%t" > /proc/sys/kernel/core_pattern
Set prlimit --pid $PID --core=unlimited.
Sent SIGABRT (kill -6 $PID).
Core file appeared in /data/coredump/.

5. Loading Debug Symbols — And Hitting a Wall

The core dump was from a stripped libjvm.so. We obtained the matching debuginfo package and loaded it into GDB:

(gdb) set sysroot /proc/<PID>/root
(gdb) set debug-file-directory /path/to/debuginfo/usr/lib/debug
(gdb) add-symbol-file /path/to/libjvm.so.debug <load_address>

After loading symbols, info locals became partially available. We could see some variables:

current = 0xfffee901ce98
alias_idx = 36
result = 0x0

But the critical variables were optimized away:

merge = <optimized out>
new_st = <optimized out>
mem = <optimized out>
opc = <optimized out>

We tried calling member functions directly:

(gdb) print merge->base_memory()
cannot evaluate function -- may be inlined

(gdb) p *merge
value has been optimized out

(gdb) print ((Node*)0xfffee901ce98)->in(0)
inlined

GDB was effectively blind to the very data we needed. The release build's aggressive optimizations had erased the connection between variable names and their physical storage.

6. The Turning Point: Raw Memory Analysis

Since GDB's symbolic debugging was blocked by compiler optimizations, we switched to raw memory analysis — reading the binary layout of objects directly from the core dump, bypassing all DWARF symbol dependencies.

We knew two things: - The current node (ProjNode) address: 0xfffee901ce98 - A MergeMemNode object address found in the same compilation context: 0x0000fffee51fb9e0

If the cycle hypothesis was correct, the MergeMemNode's internal memory must contain a pointer back to the current node. We dumped the raw memory:

(gdb) x/32xg 0x0000fffee51fb9e0

Scanning through the 256 bytes of the MergeMemNode's memory layout, we found it:

0x0000fffee51fb...: ...  0x0000fffee901ce98  ...
                         ^^^^^^^^^^^^^^^^^^
                         Exact address of current!

The MergeMemNode physically contained the address of the current ProjNode in its internal data structure. This is the base_memory (or memory_at) slot pointing back to the loop's own Proj node.

Combined with the logical flow: - current (ProjNode at 0xfffee901ce98) → in(0) → MemBarCPUOrder - MemBarCPUOrder → in(TypeFunc::Memory) → MergeMemNode (at 0x0000fffee51fb9e0) - MergeMemNode → base_memory / memory_at(36) → current (0xfffee901ce98)

This physically confirmed a topologically closed loop in the Ideal Graph. The while loop was guaranteed to spin forever.

7. Identifying the Triggering Java Methods

Once we confirmed the cycle, the next question was: which Java methods were being compiled when C2 hung?

The methodology for inspecting the ciMethod object follows the approach described in Vladimir Sitnikov's 2018 article on analyzing stuck C2 compilations. The original technique was developed for x86; we adapted and validated it for AArch64 in this investigation.

Step 1: Locate the Compile Frame in GDB

From the C2 compiler thread's stack, we navigated to a frame where the Compile object was accessible:

(gdb) thread <C2_compiler_thread>
(gdb) bt
# Find a frame in Compile::Optimize or Compile::Compile
(gdb) frame <N>

Step 2: Extract the ciMethod from the Compile Object

Rather than brute-forcing registers and stack memory, we directly accessed the Compile object's _method field:

(gdb) p/x C._method

This gave us the address of the ciMethod object being compiled.

Step 3: CLHSDB — Resolve the Method Name

We then used CLHSDB (Command-line HotSpot Debugger) to inspect the ciMethod object and resolve its name. CLHSDB does not rely on DWARF debug symbols. Instead, it uses VMStructs, a built-in mechanism in HotSpot that exposes the layout of key JVM data structures even in release builds.

$JAVA_HOME/bin/java -classpath $JAVA_HOME/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB

hsdb> attach $JAVA_HOME/bin/java /path/to/core.dump

hsdb> inspect <ciMethod_address>
Type is ciMethod (size of ...)
...

hsdb> inspect <name_field_address>
Type is ciSymbol (size of ...)
Symbol* ciSymbol::_symbol: Symbol @ <symbol_address>

hsdb> symbol <symbol_address>
#getParameterType

hsdb> inspect <holder_field_address>
...
hsdb> symbol <holder_symbol_address>
#org/springframework/core/MethodParameter

Result

We traced compilation tasks that triggered the hang to at least the following Spring methods:

org.springframework.core.MethodParameter.getParameterType()
org.springframework.core.ResolvableType.forField()

These are not the only methods that can trigger the issue. The common triggering pattern is:

A loop that performs recursive or iterative type resolution
Object allocation inside the loop (e.g., creating wrapper objects for resolved types)
Writing the result to a volatile field (e.g., caching the resolved type for future use)

This pattern is common in Spring Framework's type resolution infrastructure, so other methods in ResolvableType, GenericTypeResolver, and similar classes could also trigger this bug under the right compilation conditions.

8. Root Cause Summary

Root cause:

In OpenJDK 8u442 on AArch64, C2's dead-code elimination can create a data-flow cycle when a loop containing both new Object() and volatile writes is partially optimized. The cycle is:

Proj → MemBarCPUOrder → MergeMem → base_memory → same Proj

The while loop in can_see_stored_value that walks through Proj → MemBar → MergeMem → base then spins forever because current == new_st.

Trigger:

Spring's type resolution methods (MethodParameter.getParameterType, ResolvableType.forField, etc.) use loops with generic type resolution that involve creating objects and writing to volatile cache fields.

Why 8u442:

A related upstream fix addressing C2 memory barrier handling on AArch64 has not been backported into this update. The specific fix targets the interaction between dead code elimination and MemBarCPUOrder nodes in loops with object allocation, a pattern that is unique to weak memory model architectures.

Why not trivially reproducible:

This issue does not manifest on every JVM invocation or under simple test cases. It depends on a specific confluence of C2 compilation events: the method must reach the C2 compilation threshold, inlining decisions must create the specific IR shape, and dead code elimination must remove the Phi node in a particular order. In production, these conditions are met after sustained operation, making the issue reproducible in practice but difficult to isolate in a standalone test.

9. Production Resolution

We resolved this issue in production by switching to Alibaba Dragonwell 8, which includes a patch that adds a cycle-detection guard to break out of the infinite loop in can_see_stored_value.

The patch is conceptually similar to adding a bounded counter to the while loop:

int cont_times = 0;

while (current->is_Proj()) {
    int opc = current->in(0)->Opcode();
    if (opc == Op_MemBarRelease || ... || opc == Op_MemBarCPUOrder) {
        Node* mem = current->in(0)->in(TypeFunc::Memory);
        if (mem->is_MergeMem()) {
            MergeMemNode* merge = mem->as_MergeMem();
            Node* new_st = merge->memory_at(alias_idx);

            if (new_st == merge->base_memory()) {
                if (opc == Op_MemBarCPUOrder) {
                    cont_times++;
                }
                if (cont_times > 1000) {
                    phase->C->record_method_not_compilable(
                        "Possible dead loop in can_see_stored_value");
                    return NULL;
                }
                current = new_st;
                continue;
            }
            result = new_st;
        }
    }
    break;
}

When the compiler detects it has followed the same memory-walk path too many times, it abandons compilation of the affected method and falls back to interpreter execution. This prevents the infinite loop at the cost of deoptimizing one method.

With this build, the affected methods are no longer able to trap the compiler thread in an endless loop.

10. Reporting to OpenJDK

We reported this issue to the OpenJDK HotSpot compiler team via the hotspot-compiler-dev mailing list, including:

The flame graph evidence
The raw memory analysis confirming the cycle
The triggering Java methods and their common pattern
The Dragonwell 8 patch as a reference mitigation

We hope this will lead to a similar fix being backported to the upstream OpenJDK 8u branch to protect all AArch64 users.

11. Full Debugging Command Sequence

For reference, here is the complete workflow used in this investigation:

# 1. Find container PID on host
docker inspect <container> --format '{{.State.Pid}}'

# 2. Allow ptrace (if needed)
echo 0 > /proc/sys/kernel/yama/ptrace_scope

# 3. Capture core dump
echo "/data/coredump/core.%e.%p.%t" > /proc/sys/kernel/core_pattern
prlimit --pid <PID> --core=unlimited
kill -6 <PID>

# 4. Load core dump with debug symbols
gdb -q \
  -ex "set sysroot /proc/<PID>/root" \
  -ex "set debug-file-directory /path/to/debuginfo/usr/lib/debug" \
  /proc/<PID>/root/<path-to-jdk>/bin/java \
  /data/coredump/core.java.<PID>.*

# 5. In GDB: find the C2 compiler thread
(gdb) thread apply all bt 3
# Look for MemNode::can_see_stored_value in output

# 6. In GDB: inspect available locals
(gdb) thread <N>
(gdb) frame <can_see_stored_value frame>
(gdb) info locals

# 7. In GDB: raw memory analysis (when locals are optimized out)
(gdb) x/32xg <MergeMemNode_address>
# Search for current node's address in the output

# 8. In GDB: extract ciMethod address
(gdb) frame <Compile frame>
(gdb) p/x C._method

# 9. In CLHSDB: resolve method name
$JAVA_HOME/bin/java -classpath $JAVA_HOME/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB
hsdb> attach $JAVA_HOME/bin/java /path/to/core.dump
hsdb> inspect <ciMethod_address>
hsdb> symbol <symbol_address>

Conclusion

Key Findings

Root cause: C2's Ideal Graph can form a cycle when dead code elimination removes nodes in the object allocation path, causing MemBarCPUOrder's MergeMem to point back to itself.
Architecture specificity: The bug only manifests on AArch64 because only weak memory models require the MemBarCPUOrder nodes that participate in the cycle.
Trigger: Spring Framework's type resolution methods (MethodParameter.getParameterType, ResolvableType.forField, etc.) — common methods that combine volatile accesses and object creation in loops.
Reproducibility: The issue is not immediately reproducible on demand, but emerges reliably after sustained production workload due to the complex interplay of C2 compilation conditions.
Production fix: Alibaba Dragonwell 8 includes a depth guard patch that effectively mitigates this issue.

Lessons Learned

Flame graphs are the first line of defense — the 80%/21% ratio immediately pointed to the right code.
GDB with debuginfo can still hit <optimized out> — compiler optimizations often strip away variable values, even when symbols are loaded.
Raw memory analysis is the ultimate fallback — when symbolic debugging fails, x/Nxg on known object addresses can reveal the truth directly from memory.
CLHSDB is the right tool for JVM internals — VMStructs provide direct access to memory without DWARF dependencies.
Container debugging requires creative tooling — host gcore + sysroot works where in-container debugging fails.
C2 hangs are notoriously difficult to reproduce synthetically — real workload profiling is essential.
When debugging intermittent JVM issues, capture core dumps early — you may not get a second chance.

Final Recommendation

If you run Spring Boot on AArch64 and see unexplained 100% CPU from C2 CompilerThread:

Profile with async-profiler to confirm it's in can_see_stored_value.
Consider switching to a JDK distribution that includes the depth guard patch (e.g., Alibaba Dragonwell 8).
Alternatively, exclude the triggering methods: -XX:CompileCommand=exclude,org/springframework/core/MethodParameter::getParameterType -XX:CompileCommand=exclude,org/springframework/core/ResolvableType::forField
Capture a core dump and verify the cycle using raw memory analysis.
Report to your JDK vendor with the evidence.

Acknowledgments

The method for extracting ciMethod information from core dumps was adapted from Vladimir Sitnikov's excellent 2018 article on analyzing stuck C2 compilations.

This investigation was conducted on OpenJDK 8u442 running on AArch64 processors.

Tags: JVM Internals, C2 Compiler, AArch64, Debugging, Spring Framework, Performance

This article is for informational purposes only. Always test JVM flags in a non-production environment first.