As far as I know, this bug is known in older versions of Lua, and fixed in 5.3.4 and later versions by using unsigned integers for the pivot in `unbound_search` and `luaH_getn` .
As you can see there is an explicit overflow check in `unbound_search` before computing the pivot. For `auxsort`, the same check is performed in `sort` before calling it.
For `countlevels`, it doesn't exist anymore (replaced by `lastlevel` in 5.3.2) and I don't think this can possibly overflow in general since indices are bounded by the size of the stack, which is much less than `INT_MAX / 2`. *Maybe* it is possible to make it overflow in theory by changing `LUAI_MAXSTACK` in `luaconf.h` though?