Out of memory/garbage collection error in my embedding using 140ESR

SpiderMonkey
1

Hi,

I’m hoping that someone might be able to give me some guidance/advice on a garbage collection issue I’m having.
I’ve been using Spidermonkey in my embedding for many years, but have now had an issue reported to me where I am running out of memory. In my embedding this seems to happen when a user is creating an object which contains ~900,000 properties.
I’m struggling to understand why there isn’t an attempt to reclaim memory by doing garbage collection before the out of memory error occurs though.

For background, I declare a runtime/context with 50MB of memory. I then use JS_SetGCCallback to register a GC callback function. In this I use JS_GetGCParameter(cx, JSGC_MAX_BYTES) and JS_GetGCParameter(cx, JSGC_BYTES) to see what percentage of the garbage collection memory is being used at the beginning and end of garbage collection.

If at the end of garbage collection the GC memory usage is > 90%, or alternatively if at the start of garbage collection the usage is > 70% and at the end the usage has not dropped my more than 1%, then I double the GC memory by calling

JS_SetGCParameter(cx, JSGC_MAX_BYTES, new_size);

I’ve added some debugging to see what is going on and in a release build I see the following, just before the OOM error

Part 100220 (465/2791)
Called dj_GC_callback_func at 13:28:13.550 with status BEGIN reason 6
Max bytes=52428800, bytes=49512448 (94%)

Called dj_GC_callback_func at 13:28:13.573 with status END reason 6
Max bytes=52428800, bytes=32133120 (61%)

900888 elements
Component 326
Called dj_GC_callback_func at 13:28:13.593 with status BEGIN reason 1
Max bytes=52428800, bytes=40996864 (78%)

Called dj_GC_callback_func at 13:28:13.612 with status END reason 1
Max bytes=52428800, bytes=41005056 (78%)
GC memory use of 41005056 > 70% of 52428800 and not dropped
Increasing GC size to 104857600

Called dj_GC_callback_func at 13:28:13.994 with status BEGIN reason 4
Max bytes=104857600, bytes=104857600 (100%)

Called dj_GC_callback_func at 13:28:14.076 with status END reason 4
Max bytes=104857600, bytes=53841920 (51%)

Component 325
Called dj_GC_callback_func at 13:28:14.343 with status BEGIN reason 1
Max bytes=104857600, bytes=84996096 (81%)

Called dj_GC_callback_func at 13:28:14.377 with status END reason 1
Max bytes=104857600, bytes=45277184 (43%)

Component 7
OOM at 13:28:14.798 Max bytes=104857600, bytes=104857600)

From this I can see that by GC callback function is called, firstly with reason ‘ALLOC_TRIGGER’ (6) and the GC memory drops from 94% to 61%.
Then shortly after it is called again with reason ‘EAGER_ALLOC_TRIGGER’ (1) and the GC memory doesn’t drop so I expand the memory to 104857600 bytes.

Then it is called with reason ‘LAST_DITCH’ (4) because the GC memory gets to 100% but it then successfully drops to 51%.
It is then called again with EAGER_ALLOC_TRIGGER and manages to reclaim memory again (81% drops to 43%).

Then suddenly ~500ms later I get an OOM error.
Why doesn’t SM try to do garbage collection again before failing with the error?

I’ve tried with a debug build and it gets further, but fails in a similar issue creating an object with ~900,000 properties again. Interestingly though, in this case I do not get the ‘LAST_DITCH’ call.

Called dj_GC_callback_func at 13:55:52.078 with status BEGIN reason 1
Max bytes=104857600, bytes=40353792 (38%)

Called dj_GC_callback_func at 13:55:52.655 with status END reason 1
Max bytes=104857600, bytes=24379392 (23%)

Component 7
Component 9
Part 100432 (523/2791)
Called dj_GC_callback_func at 13:56: 2.025 with status BEGIN reason 6
Max bytes=104857600, bytes=42950656 (40%)

Called dj_GC_callback_func at 13:56: 4.712 with status END reason 6
Max bytes=104857600, bytes=40923136 (39%)

900888 elements
Component 326
OOM at 13:56:27.553 Max bytes=104857600, bytes=104857600)

I’m presuming that as my embedding is trying to make an object with so many properties, this is a very different use case to the normal usage in Firefox and so for some reason I don’t understand, in my case, garbage collection is not being triggered. Perhaps it is somehow because I am making so many objects/properties and collecting them over a small time duration (in ms)? Maybe it is too soon after GC was previously done or something? However I’m completely guessing and this could be garbage.

I have looked in GCAPI.h and I can see all sorts of parameters that can be changed in JSGCParamKey for altering the garbage collector. However, I’m at a bit of a loss to know what values I should be trying to change and what to change them to.
Garbage collection has always been a bit of a ‘black box’ for me. It just ‘worked’. However, it now seems I have ‘Pandora’s box’… :frowning:

Can anyone provide me with any guidance/advice on why garbage collection is not being triggered for me, causing the out of memory error?
Can anyone recommend which JSGCParamKey values I should be trying to change, or if there is something else that I’m missing because I’m being stupid…

I would be very grateful for any thoughts anyone might have.
Many thanks in advance.

Miles

2

Hi Miles!

The main issue is that JSGC_MAX_BYTES is a hard limit for the amount of GC heap you allow the JS engine to allocate. This should only be set once at the start of the program (or left at its default of 4GB). If the heap size exceeds this value then an OOM error can be generated on allocation.

GC is triggered automatically when the heap size grows. There are a ton of parameters that affect it but mostly these can be left at their default values.

If you don’t change the JSGC_MAX_BYTES parameter does that help things?

Feel free to get in touch on the SpiderMonkey channel on matrix.

Jon

3

Hi Jon,

Many thanks for getting back to me. Much appreciated.

I’m a bit confused by your answer though. Perhaps I’m being stupid and missing something.

In the years that I have been using SM, I have always had to define a size in the call to JS_NewContext. That is where I am giving my initial size of 50MB.

If I look at spidermonkey-embedding-examples/examples/boilerplate.cpp at esr115 · mozilla-spidermonkey/spidermonkey-embedding-examples · GitHub I see that all the examples use

JSContext* cx = JS_NewContext(JS::DefaultHeapMaxBytes);

and looking in the source I have for ESR140, JS::DefaultHeapMaxBytes is defined in HeapAPI.h as

const uint32_t DefaultHeapMaxBytes = 32 * 1024 * 1024;

which is 32MB. It is this value in the call to JS_NewContext which sets JSGC_MAX_BYTES.
So in all of the examples above the hard limit for the heap size is 32MB and not 4GB.
Surely, once the limit of 32MB is met, and no memory can be reclaimed, there will be an OOM error?
This is what I’m trying to avoid in my embedding. I’m trying to modify JSGC_MAX_BYTES in my function registered with JS_SetGCCallback when the memory use gets close to JSGC_MAX_BYTES and memory can’t be reclaimed.
As the size I gave in the JS_NewContext call was 50MB, for me JSGC_MAX_BYTES is 50MB initially.

Are you suggesting that I should always just call
JS_NewContext(4 * 1024 * 1024 * 1204)
when setting up the context then?
If so, how much memory does SM allocate initially? i.e. does it initally allocate all 4GB?

Thanks

Miles

4

Well JS::DefaultHeapMaxBytes does look pretty small at 32MB. I was mis-remembering - in Firefox we set this to 4BG after passing the default value of 32MB when creating the context. I don’t know why this is. Note that this doesn’t allocate anything - it’s just a limit on the max amount it can allocate.

For your use feel free to set the heap limit to whatever limit makes sense.

GC is triggered automatically when the heap grows beyond a threshold that is calculated from a bunch of the other GC parameters, not including the max heap limit (although it will also trigger GC when it reaches this). This is roughly calculated as max(bytes_remaining_after_last_collection, allocation_threshold) * growth_factor, where growth_factor starts at 1.5 and may be increased for various reasons.

It may be that the default heap limit and the default GC parameters don’t play well together. We don’t really test with the default heap limit in place.

In short I’d recommend setting the max heap bytes to something large and seeing if that fixes things.

Jon

5

OK. Understood. I’ll increase the size.
I was (clearly incorrectly) under the impression that setting it to a larger size would allocate more memory initially. I didn’t realise that it was just a maximum so many thanks for clearing that up.

I have had some success in modifying JSGC_MAX_BYTES in my GC callback function. I think if I always increase it if I get a LAST_DITCH GCReason it works. The only other thing that it took me a long time to work out is that I also had to set JSGC_MIN_LAST_DITCH_GC_PERIOD to 0 in case it still needed even more memory a bit later as that defaults to 60s, so I think I’ll leave the logic I have in there but start with a larger size

It might be worth someone changing the examples at GitHub - mozilla-spidermonkey/spidermonkey-embedding-examples: Documentation and examples for embedding the SpiderMonkey JavaScript / WebAssembly engine in their applications. to not use JS::DefaultHeapMaxBytes, or add a comment or something to stop other embedders encountering this issue.

Thanks again for the help

Miles

6

Great, hopefully that helps.

I’ve filed 2017847 - JS::DefaultMaxBytes is too small when used with default GC parameters to change the default to something more sensible.

Jon

1 Like