Your allocator function needs to handle the case where nsize is nonzero, but ptr is null. This is the case for a new allocation, rather than a resize of an existing block. In this case, osize represents something else (see below), and you don't want to do the "memsize +=" line.
From the manual:
"When ptr is NULL, osize encodes the kind of object that Lua is allocating. osize is any of LUA_TSTRING, LUA_TTABLE, LUA_TFUNCTION, LUA_TUSERDATA, or LUA_TTHREAD when (and only when) Lua is creating a new object of that type. When osize is some other value, Lua is allocating memory for something else."