Growable Fiber Stacks with Guard Pages

Memory chips on a circuit board

Every fiber coroutine runs on its own stack, and that stack has to be sized before the fiber runs a single instruction. Pick it too small and a deep call chain overflows into whatever lies below. Pick it too large and a server handling a few hundred connections burns megabytes on stack space that is never touched.

Ioto 3.0 removes this choice on Linux, MacOS and Windows. Each fiber reserves a large virtual address range, commits only a small initial stack, and lets the MMU grow the stack on demand. An overflow past the configured maximum faults immediately instead of silently corrupting memory.

This post explains the problem fixed-size stacks cause, how the guard page design works, how to tune it with the limits.fiberStack properties in ioto.json5, and where the approach does not apply.

The Fixed Stack Problem

Ioto uses fiber coroutines for parallelism instead of threads or callbacks. Each HTTP request, MQTT message and database sync runs on a fiber, and a busy device may have dozens active at once.

A thread gets its stack from the operating system, which typically reserves 8MB of address space and pages it in as needed. A fiber gets its stack from the runtime, and before Ioto 3.0 that was a fixed block from the heap. The runtime had to guess a size that would hold the deepest call path a fiber might take.

That guess is hard to get right. A TLS handshake with a 4K certificate buffer, a getaddrinfo lookup, and a printf with floating point formatting each consume several kilobytes on their own. Stack them in one request path and a 32K stack is tight. Meanwhile, a fiber that only blinks an LED needs almost nothing, yet pays the same.

The traditional defense was a guard pattern: a 128-byte band of 0xFE bytes written below the stack and checked whenever the fiber yields. It catches overflows, but only after the fact, and only if the overflow touched the band. An overflow that jumped past it into the next heap block went undetected.

How Guard Page Stacks Work

A guard page is a page of virtual memory mapped with no access permission. Any read or write to it raises a hardware fault. Operating systems have used them below thread stacks for decades. Ioto 3.0 uses the same mechanism for fibers, with one addition: the fault handler grows the stack instead of killing the process.

When Ioto allocates a fiber on a platform with an MMU, it does three things:

  1. Reserves the full maximum stack range from the virtual memory system with no access permission. On 64-bit systems this defaults to 8MB. Reserving address space costs no physical memory.
  2. Commits the initial stack size at the top of that range as readable and writable. The default is 64K on 64-bit systems and 32K on 32-bit. This is the only part backed by real pages.
  3. Leaves everything below the committed region inaccessible. That whole region is the guard.

The layout looks like this, with the stack growing downward:

top       +------------------------+  highest address
          |  committed stack (64K) |  read/write
usable    +------------------------+
          |                        |
          |  reserved, PROT_NONE   |  guard region, grows into on fault
          |                        |
base      +------------------------+  lowest address

When a fiber’s call chain pushes the stack pointer below usable, the CPU faults. The runtime installs a handler for SIGSEGV and SIGBUS on Unix, since Linux raises the first and MacOS the second for a guard page hit. The handler runs on its own alternate signal stack, because the fiber’s stack is the thing that just overflowed. On Windows it installs a Vectored Exception Handler for EXCEPTION_ACCESS_VIOLATION.

The handler checks whether the faulting address lies between base and usable. If it does, this is a stack growth request, not a bug. The handler commits another 16K below the current usable boundary and returns. The faulting instruction re-executes, succeeds, and the fiber never knows anything happened.

if (faultAddr >= stack->base && faultAddr < stack->usable) {
    if (growFiberStack(currentFiber) == 0) {
        return;  // Resume execution - stack has been grown
    }
}

If the fault address is anywhere else, it is a genuine bad memory access. The handler falls through to Ioto’s normal fiber exception handling, which either unwinds to a fiber exception block if one is active or aborts the process.

The growth function itself does almost nothing. It checks that committing another increment would not exceed the maximum, changes the protection on those pages, and updates the bookkeeping. It runs inside a signal handler, so it is restricted to async-signal-safe calls. There is no logging, no allocation and no locking on this path.

What Happens at the Limit

Growth is bounded by the maximum reserved size. When a fiber has committed everything up to that limit and faults again, the growth attempt fails and the fault is treated as an overflow. The process aborts, or the exception block catches it, at the exact instruction that overran.

This is the second benefit of guard pages. A runaway recursion is caught by hardware at the moment it exceeds the limit, with the faulting address in hand. Compare the guard pattern approach, where the damage is done first and detected at the next yield, if at all.

Pool Reset

Ioto keeps a pool of idle fibers so a new request does not pay for allocation. A fiber that grew to 512K during one unusual request would otherwise hold that memory for every later request it serves.

When a fiber returns to the pool, the runtime checks its committed size against limits.fiberStackReset, which defaults to 64K. A stack above that threshold is trimmed back to the initial size by making the grown pages inaccessible again, which releases the physical memory. A stack at or below the threshold is left alone, since re-growing it on the next request would cost more faults than the memory saves.

Configuring Stack Limits

Four properties under limits in ioto.json5 control the behavior. Sizes accept the usual suffixes.

limits: {
    fiberStack: '32k',
    fiberStackMax: '256k',
    fiberStackGrow: '16k',
    fiberStackReset: '64k',
}
PropertyDefaultDescription
fiberStack64K on 64-bit, 32K on 32-bit, 16K on ESP32Initial committed size. Replaces the older limits.stack, which is still honored.
fiberStackMax8MB on 64-bit, 1MB on 32-bitMaximum size a stack may grow to. This is the reserved address range per fiber.
fiberStackGrow16KIncrement committed on each growth fault.
fiberStackReset64KStacks larger than this are trimmed to the initial size when returned to the pool.

A value of zero, or an omitted property, keeps the default. Every value is rounded up to the system page size. The initial size is clamped to a floor of 8K, which is the least the context switch code can work with. It is not enough for a TLS handshake, so treat 16K as the practical minimum.

The same settings are available at runtime through the R runtime API, which is what the configuration loader calls:

rSetFiberStackLimits(32 * 1024, 256 * 1024, 16 * 1024, 64 * 1024);

Pass zero for any argument to leave that setting unchanged. The matching rGetFiberStackLimits returns the current values, and returns zero for the growth parameters on platforms where guard pages are disabled.

Choosing Values

For most deployments the defaults are right. Two situations call for a change.

If memory is tight and connection counts are high, lower fiberStackMax to a few hundred kilobytes. The reserved range is address space rather than memory, but on 32-bit systems address space is itself a finite resource, and a lower ceiling also turns a runaway recursion into a fault sooner.

If your handlers run known stack-hungry code, such as a JSON parser on a large document or a recursive directory walk, raise fiberStack so the common case does not fault its way up 16K at a time. Each growth fault is a trip through the kernel and the signal handler. It is cheap, but it is not free.

Where It Does Not Apply

Guard pages need a memory management unit and a kernel that exposes mprotect or its equivalent. ESP32 and FreeRTOS targets have neither, so ME_FIBER_GROWABLE_STACK is compiled out there and fiber stacks are fixed at the initial size. The fiberStackMax, fiberStackGrow and fiberStackReset properties are accepted and ignored.

On those platforms the 128-byte guard pattern is still in place and is checked at each yield. The practical advice is the same as before 3.0: keep large buffers on the heap, avoid deep recursion in fiber code, and size fiberStack for the deepest path you expect. The ESP32 default is 16K, which covers the TLS handshake and request parsing with some margin.

One further limit applies everywhere. The growth handler only acts when the fault belongs to the current fiber’s reserved range. A fault from a foreign thread, or from the main fiber, which runs on the platform’s own stack, is passed on as an ordinary exception.

Summary

Ioto 3.0 fibers start with a small committed stack inside a much larger reserved range, and the MMU grows the stack on demand up to a configurable maximum. Memory use tracks what each fiber actually needs, overflow is caught by hardware at the offending instruction, and pooled fibers give back memory they no longer need. The limits.fiberStack properties tune it, and the defaults suit most devices.

Want More Now?

To learn more about EmbedThis Ioto, please read:

Comments

Make a Comment

Thank You!

Messages are moderated.

Your message will be posted shortly.

Sorry

Your message could not be processed at this time.

Error:

Please retry later.

OK