LUA_32BITS: Use 16bit in _tt to allow 48bit Integers in tables?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

LUA_32BITS: Use 16bit in _tt to allow 48bit Integers in tables?

bil til
Hi,
I discovered Lua on last weekend, and I am completely fascinated. Thank you
for this great work (also thanks to Dave Poo for the great youtube videos).
I need this for normal windows software, but also for controllers mainly
running on ARM Cortex M4 (float support 32bit, int support 32bit).

Those Controllers are getting larger and larger concerning ROM and RAM, but
this of course will always be a bottleneck on such devices. I do not assume,
that 64bit support comes fast there, because 32bit is so versatile and ROM
typically still under 10MByte for this devices, so 32bit pointers perfectly
fine there. Numeric support with double (float 64bit) is supported, e.g. on
Cortex M7 - but this is availalbe only for larger and new devices.

When I use lua with this luaconf.h setting LUA_32BITS, it looks that I can
be perfectly happy with this (maybe stripping out string.h code to reduce
ROM...). Also this Integer support of Lua 5.3 of course is very nice and
important for controller applications.

Just unfortunately there are 2 very typical applications for Controllers,
where int 32bit (max value 2G) is too poor: Encoding systems for motor
stages, and timestamps:
- For exact motor drive control with 1um resolution, you typically will
internally need to use numbers which are about Factor 1000 better, so
resolution 1nm. For a number 2G would then mean 2 meters. This is sufficient
most, but it is a bit at the limit.
- For time counters like os.time, counting seconds, 2G means about 50 years,
this also is at the limt for date applications.

To extend this, you can use int64, then of course all fine. But if I skip
the LUA_32BITS for lua, then RAM need will bloat up by factor 2, and RAM is
anyway sparse... .

Looking in the way of number presentation in tables in lua lobject.h, I find
this structure
*#define TValuefields Value value_; int tt_
typedef struct lua_TValue {
  TValuefields;
} TValue;
*
The type Value has 32bit, and this typetag tt_ has also 32bit, but (as I see
it) only 1 byte used (3 bits for type, and 4 further bits for
BIT_ISCOLLECTABLE and further sub-typing (as float/integer, or function
sub-type...). This is extremely efficient, seen what lua can do, but there
are 24bits "free" in _tt.

Would it be ok/ quite "future-proof", to tweek out 2 bytes of this tt_
integer (e. g. the 2 high bytes), to allow 48bit integers? With 48 bit
integers in these 2 applications above I am perfectly fine, I end up with
about 100km with resolution of 1nm, or 100k years with resolution of 1 sec.
- this is fine for me (and I think also for VERY many other people).

In lua this also would not create too much overhead. Just in any integer
number operation, I first have to check, whether the number possibly has
more than 31 significant bits ... is is a very fast check with bit routines.
Up to 31bit I can use normal int32 math, and above I then use int64 math
(for those I would only support +, -, *, //(2-exp numbers 2,4,8,16... for
shifting) and possibly binary boolean ... any other division will create a
float result, so will strip the precision ... ). As integer type anyway very
new, makeing such fast checks for int48 would create a minimum overhead
(because lua_Integer as number type is mainly only used in the math lib).

And further I can then use such int48 numbers for motor and time control
very RAM-efficiently in indexed tables ... .

@Lua Internals Specialists : Would this be "quite ok and future proof", or
is this a completly silly idea, which will block any future lua interesting
extension of this type-tag _tt? Any recommendations or comments?





--
Sent from: http://lua.2524044.n2.nabble.com/Lua-l-f2524044.html

Reply | Threaded
Open this post in threaded view
|

Re: LUA_32BITS: Use 16bit in _tt to allow 48bit Integers in tables?

bil til
This post was updated on .
Sorry that I dare to extend this blog myself, although no answer yet.

Maybe I should ask more in detail, not so general.

I have the impression, that this int _tt in the TValue struct uses only max 8
bit of the 32 bits of int. So the uper 24bits (I need only the upper 16 bits)
seem to be out of use currently.

Is this correct, or is there any hidden usage of these 16 upper bits in
LUA_32BITS? In LUA_64BITS it would concern the upper 56 bits of _tt ... but
currently please let's concentrate on LUA_32BITS ... I am currently only
interested in this, and my main headache are date numbers with second
resolution... I just need some bits more for them, and as they appear quite
often I would not like to define a 2int metatable or userdata for them ... .
But the same later will apply for motor encoding signals in my applications,
I am quite sure ... . Linear scale encoders become better and better every
year, and for nice control applications many people need huge numbers.
Floats are no solution here - you cannot tell the user, that a linear
scale counts the ticks nicely only in the "inner range" but not at the edges,
the same as you cannot tell the users, that a clock will stop counting
with resolution of 1 second after year 2020 or even 2050 ... this really
would be not nice info for most users.

If such 16 bits are NOT in use currently, do you think they are scheduled to
be used for some new applications in the next time?

So now I reduced it to 2 short questions:
- are these 16 bits in use currently?
- if not, could they to be used for anything in some "next time"?






--
Sent from: http://lua.2524044.n2.nabble.com/Lua-l-f2524044.html

Reply | Threaded
Open this post in threaded view
|

Re: LUA_32BITS: Use 16bit in _tt to allow 48bit Integers in tables?

Roberto Ierusalimschy
> I have the impression, that this int _ in the TValue struct uses only < 8
> bit of the 32 bits. So the uper 24bits (I need only the upper 16 bits) are
> out of use currently.
>
> Is this correct, or is there any hidden usage of these 16 upper bits in
> LUA_32BITS? (In LUA_64BITS it would concern the upper 56 bits of _tt ... but
> currently please let's concentrate on LUA_32BITS ... I am currently only
> interested in this, and my main headache are date numbers with second
> resolution... I just need some bits more for them, and as they appear quite
> often I would not like to define a metatable for them ...).

A simple grep shows that the field 'tt_' is only mentioned in the
file 'lobject.h'. A careful examination of this file should give
the answer you are looking for.

For Lua 5.3, I guess those 24 bits are free. However, Lua 5.4 defines
tt_ as a lu_byte (an unsigned char), not as an int, and it explores this
smaller size to better pack key-value pairs in tables.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: LUA_32BITS: Use 16bit in _tt to allow 48bit Integers in tables?

bil til
This post was updated on .
Thank you, this is bad news then for my original plan.

So if values are placed packed in tables, before using some value of the
index part of a table, it has to be copied to stack / to some aligend memory
place I assume? ... But I possilbly can look this up myself in your 5.4
code, when I am lucky to find it ... I did NOT yet look at the 5.4 code I
must admit, only sticking with 5.3 currently.

PS: Do you have some "recommended way" to save such int64 numbers in
LUA_32BIT (for "more seldom" usage): e. g. combining the Low Int32 and
the High Int32 in some table? (so then I should rewrite os.time() such,
that in my case it would return a 2-value table?). In this case I also
would best create some math functions, at least add64 and sub64 to
handle such tables (or make a metatable with __add and __sub...).
Would this make sense, or do you see some more easy / straight forward
method?

PPS: So then the required space for the index part of a large  table in lua 5.4
will take about 40% less storage space than in lua 5.3? ... maybe you could
specify this in the "change list of 5.4", at least I did not see this there... .



--
Sent from: http://lua.2524044.n2.nabble.com/Lua-l-f2524044.html