Metaspace OOM during HPM updates of a large app

I've hit java.lang.OutOfMemoryError: Metaspace several times now, most recently on a C-8 while HPM was updating my app (MCP Rule Server, about 2.3 MB of Groovy), and earlier on a C-7 where the OOM left a Rule Machine rule corrupted. HPM compile timeouts on the same app are a regular thing too. The 2021 thread says the compile spike normally comes back down after a few GC cycles, and the webCoRE update thread looks like the same problem with another large app. So I tried to see what an update actually costs using only what the hub exposes.

Procedure. C-7, 2.5.2.135, about 8 hours of uptime, my package installed through HPM. HPM, Repair, my package, three times in a row about 10 minutes apart. Each repair re-saves the parent and child app code exactly as an HPM update does. I watched the Free Memory history (Settings, Hub Details, or /hub/advanced/freeOSMemoryHistory, 5-minute samples) and HPM's own "Repairing" to "Repair complete" timing.

Sample Free OS memory Free Java heap 5-min CPU
before 702.7 MB 248.7 MB 0.24
during repair 1 663.4 MB 51.0 MB 0.63
start of repair 2 657.0 MB 27.7 MB 0.33
after repair 2 685.4 MB 231.8 MB 0.28
before repair 3 671.9 MB 31.8 MB 0.45
during repair 3 666.5 MB 64.9 MB 0.38
20 min after repair 3 687.0 MB 229.6 MB 0.15

Each repair took about 3 minutes of hub time from "Repairing Package" to "Repair complete" (the first one 5½ minutes including the download). A plain Apps Code save of the same source takes 2½ to 3½ minutes. Free OS memory dropped about 40 MB on the first repair and only partly recovered; 20 minutes after the third repair the hub sat about 16 MB below where it started, with nothing else changed.

What I think is going on

  • Saving app code appears to compile the source twice per save, so for a few minutes two or three full sets of the app's classes exist at once. For an app this size that is the spike that lands on top of whatever Metaspace is already in use.
  • After it settles, two full sets seem to stay resident rather than one, and they don't go away until a reboot. For a 2.3 MB app that looks like roughly 13 MB of Metaspace held per installed app code.
  • The running app itself seems cheap, with classes loaded on first use, so the cost is in the save/compile path, not in having the app installed.
  • I ran the same repeated-save procedure on Echo Speaks from the HPM catalog and saw free memory step down with every save and not come back even after deleting the app code, which makes me think some apps' superseded compiles are never released at all. I don't know yet whether anything on the C-8 that OOM'd behaves like that.

Things from the Groovy side that might be worth checking

  • GROOVY-8189 describes exactly this shape: GroovyClassLoader's classCache keeps the classes of a superseded InnerLoader reachable, so the loader never unloads and Metaspace grows until OOM. The fix on the embedding side is either GroovyClassLoader.clearCache() after a recompile or compiling each save in a fresh GroovyClassLoader so the old loader becomes unreachable.
  • Groovy's MetaClassRegistry keeps a MetaClassImpl per class, and those reference the classes and their methods. If the registry entries for the previous compile aren't removed (GroovySystem.getMetaClassRegistry().removeMetaClass(cls)), the old class set stays pinned.
  • If the second compile per save is a validation pass, reusing the first CompilationUnit's output would halve the spike on its own.
  • The JVM only frees a class set when its defining loader is unreachable, so any single strong reference from a static cache to one class of the old compile keeps the whole 1,700-class set in Metaspace. A heap dump during one of these updates would show which reference that is in a few minutes.

Questions

  1. Is a double compile per save expected (a validation pass plus an install pass?), and is keeping the previous compile resident afterwards by design?
  2. For the Echo Speaks pattern, what in app code keeps a superseded compile reachable? Is there any guidance for developers on constructs to avoid?
  3. Is there anything developers of large apps can do to shrink the compile-time spike, or anything on the roadmap for the Metaspace ceiling?

Happy to re-run anything on the C-7 and post the free-memory history.

2 Likes