# An operator syntax for bitfields using metatables

16 messages
Open this post in threaded view
|

## An operator syntax for bitfields using metatables

 I've been following the discussion on the bit32 library and share the desire for an operator rather than a functional syntax. However I realised that 99% of my use of this library is for cracking and assembling bitfields for communications protocols and register mapping in embedded systems. It is straightforward to add a syntax for this in a C library by providing a metatable for the number type with 'index' and 'newindex' metamethods. Then: b = n[3]        -- Extract bit 3 as a boolean. n[23] = true    -- Set bit 23. n[6] = not n[6] -- Toggle bit 6. If you really need to bitwise and two numbers: for i=0, 31 do nr[i] = n1[i] and n2[i] end Nowhere near as efficient as: nr = bit32.band(n1,n2) But more expressive and intuitive especially when you need to manipulate a few bits in relatively complex ways. This concept can be extended to extract and replace numeric fields within a larger field (like the bit32 functions with these names). This requires that the index encode two five bit integers which could be done using bitfields within a single number or by using a string index which gets parsed in the metamethod: bitrange = function(s,e) return ((e * 0xFF) + 1) + s end  -- Range testing omitted for clarity nr = n[bitrange(10,15)]  -- Mask out bits 10 through 15 and shift right 10. nr = n[bitrange(15,10)]  -- Mask out bits 10 through 15, shift and reverse the bit order. nr = n[bitrange(12,12)]  -- nr = 1 if bit 12 is set, else 0. b = n[bitrange(12)]      -- b = true only if bit 12 is set (as before). n[bitrange(4,6)] = 2     -- replace bits 4 through 6 with 010.
Open this post in threaded view
|

## Re: An operator syntax for bitfields using metatables

 FWIW. I like the syntax. On Sat, Jul 13, 2013 at 1:24 AM, John Hind <[hidden email]> wrote: > I've been following the discussion on the bit32 library and share the desire > for an operator rather than a functional syntax. However I realised that 99% > of my use of this library is for cracking and assembling bitfields for > communications protocols and register mapping in embedded systems. It is > straightforward to add a syntax for this in a C library by providing a > metatable for the number type with 'index' and 'newindex' metamethods. Then: > > b = n[3]        -- Extract bit 3 as a boolean. > n[23] = true    -- Set bit 23. > n[6] = not n[6] -- Toggle bit 6. > > If you really need to bitwise and two numbers: > > for i=0, 31 do nr[i] = n1[i] and n2[i] end > > Nowhere near as efficient as: > > nr = bit32.band(n1,n2) > > But more expressive and intuitive especially when you need to manipulate a > few bits in relatively complex ways. > > This concept can be extended to extract and replace numeric fields within a > larger field (like the bit32 functions with these names). This requires that > the index encode two five bit integers which could be done using bitfields > within a single number or by using a string index which gets parsed in the > metamethod: > > bitrange = function(s,e) return ((e * 0xFF) + 1) + s end  -- Range testing > omitted for clarity > > nr = n[bitrange(10,15)]  -- Mask out bits 10 through 15 and shift right 10. > nr = n[bitrange(15,10)]  -- Mask out bits 10 through 15, shift and reverse > the bit order. > nr = n[bitrange(12,12)]  -- nr = 1 if bit 12 is set, else 0. > b = n[bitrange(12)]      -- b = true only if bit 12 is set (as before). > n[bitrange(4,6)] = 2     -- replace bits 4 through 6 with 010. > > > > -- David Burgess
Open this post in threaded view
|

## Re: An operator syntax for bitfields using metatables

 2013/7/13 David Burgess <[hidden email]>: > FWIW. I like the syntax. > > On Sat, Jul 13, 2013 at 1:24 AM, John Hind <[hidden email]> wrote: >> I've been following the discussion on the bit32 library and share the desire >> for an operator rather than a functional syntax. However I realised that 99% >> of my use of this library is for cracking and assembling bitfields for >> communications protocols and register mapping in embedded systems. It is >> straightforward to add a syntax for this in a C library by providing a >> metatable for the number type with 'index' and 'newindex' metamethods. Then: >> >> b = n[3]        -- Extract bit 3 as a boolean. >> n[23] = true    -- Set bit 23. >> n[6] = not n[6] -- Toggle bit 6. >> Up to here I like it too. >> This concept can be extended to extract and replace numeric fields within a >> larger field (like the bit32 functions with these names). This requires that >> the index encode two five bit integers which could be done using bitfields >> within a single number or by using a string index which gets parsed in the >> metamethod: >> >> bitrange = function(s,e) return ((e * 0xFF) + 1) + s end  -- Range testing >> omitted for clarity >> >> nr = n[bitrange(10,15)]  -- Mask out bits 10 through 15 and shift right 10. >> nr = n[bitrange(15,10)]  -- Mask out bits 10 through 15, shift and reverse >> the bit order. >> nr = n[bitrange(12,12)]  -- nr = 1 if bit 12 is set, else 0. >> b = n[bitrange(12)]      -- b = true only if bit 12 is set (as before). >> n[bitrange(4,6)] = 2     -- replace bits 4 through 6 with 010. Here I would prefer string-valued indexing. nr = n["10:15"] n["4:6"] = 2 String-valued assignment can be handled too. n["4:6"] = "010" The available metamethods for numbers have not been exhausted yet. What about: __len  -- number of 1's in n __call -- iterator over the positions of the 1's in n
Open this post in threaded view
|

## RE: An operator syntax for bitfields using metatables

 In reply to this post by John Hind > -----Original Message----- > From: John Hind [mailto:[hidden email]] > Sent: 12 July 2013 16:25 > It is straightforward to add a syntax for this in > a C library by providing a metatable for the number type with 'index' > and 'newindex' metamethods. Then: > > b = n[3]        -- Extract bit 3 as a boolean. > n[23] = true    -- Set bit 23. > n[6] = not n[6] -- Toggle bit 6. > Arghh! This does not work! (But I still think it should.) (Phew! I feared David Burgess and/or my old nemesis Dick Laurie had spotted my idiocy, but no!) The problem is that Lua does not honour a return value from the 'newindex' metamethod so the second and third lines in the above example do not actually change n. This is fine for tables and userdata, but makes the 'newindex' metamethod pretty useless for any other type! So unless we patch Lua so this does work as (I, and apparently the other two) expected, or alternatively implement the bit selection syntax directly, this idea is a non-starter. Doing it directly as a syntax patch would have the advantage that we could do n[10,12] directly rather than by bitfield encoding or string parsing, but I still think the operation of the metamethod should be also changed to make it usable for all types (a non-breaking change if nil return is interpreted as 'silently do nothing' as at present). Dick: if using a string key, it is neat to have something like "bit_4_to_12" so dot syntax can be used. A lesson learned: test your code before trying to sell it!
Open this post in threaded view
|

## Re: An operator syntax for bitfields using metatables

 In reply to this post by John Hind On Jul 13, 2013, at 7:28 AM, John Hind wrote: >> From: John Hind [mailto:[hidden email]] > >> It is straightforward to add a syntax for this in >> a C library by providing a metatable for the number type with 'index' >> and 'newindex' metamethods. Then: >> >> b = n[3]        -- Extract bit 3 as a boolean. >> n[23] = true    -- Set bit 23. >> n[6] = not n[6] -- Toggle bit 6. >> > > Arghh! This does not work! (But I still think it should.) > > The problem is that Lua does not honour a return value from the 'newindex' > metamethod so the second and third lines in the above example do not > actually change n. This is fine for tables and userdata, but makes the > 'newindex' metamethod pretty useless for any other type! More broadly, this is an issue for any immutable type. Consider tuples:   tpl = Tuple(1, 2, 3)   orig_tpl = tpl   assert( tpl[2] == 2 )   tpl[2] = 0   ==> error("that's not really an lvalue")   s = "123"   s[2]="0"   ==> error("that's not an lvalue either") So for __newindex-like-behavior to have any meaning, the assignment had to be syntactic sugar for   tpl = tuple__copy_replacing_index(tpl, 2, 0)   s = string__copy_replacing_index(s, 2, "0") and note that definitionally tpl~=orig_tpl. I think this is a scary transformation to do conditionally at runtime. I suppose the answer is to turn *all* t[k]=v statements into sugar; the table implementation doesn't need to copy of course. I'm still scared of it though.... Jay
Open this post in threaded view
|

## Re: An operator syntax for bitfields using metatables

 In reply to this post by John Hind While operators for bit-fields are very useful I think a better way is a syntax which allows you describe/build the whole bitstring in one go. We have this in Erlang and it makes building/pulling apart protocol packets very much easier and clearer. Some simple examples of the top of my head: 64-bit IEEE floating point: <> IP datagram: <> The second one is a bit longer than 32 bits. :-) I think you get the idea. We can also use the same syntax for pulling apart the binary/bitstring. This really is powerful and simplifies a lot of things and I think that something similar would be useful for Lua if you seriously intend to work with bitfields. Robert ----- Original Message ----- > From: "John Hind" <[hidden email]> > To: [hidden email] > Sent: Saturday, 13 July, 2013 1:28:35 PM > Subject: RE: An operator syntax for bitfields using metatables > > > -----Original Message----- > > From: John Hind [mailto:[hidden email]] > > Sent: 12 July 2013 16:25 > > > > It is straightforward to add a syntax for this in > > a C library by providing a metatable for the number type with > > 'index' > > and 'newindex' metamethods. Then: > > > > b = n[3]        -- Extract bit 3 as a boolean. > > n[23] = true    -- Set bit 23. > > n[6] = not n[6] -- Toggle bit 6. > > > > Arghh! This does not work! (But I still think it should.) > > (Phew! I feared David Burgess and/or my old nemesis Dick Laurie had > spotted > my idiocy, but no!) > > The problem is that Lua does not honour a return value from the > 'newindex' > metamethod so the second and third lines in the above example do not > actually change n. This is fine for tables and userdata, but makes > the > 'newindex' metamethod pretty useless for any other type! > > So unless we patch Lua so this does work as (I, and apparently the > other > two) expected, or alternatively implement the bit selection syntax > directly, > this idea is a non-starter. Doing it directly as a syntax patch would > have > the advantage that we could do n[10,12] directly rather than by > bitfield > encoding or string parsing, but I still think the operation of the > metamethod should be also changed to make it usable for all types (a > non-breaking change if nil return is interpreted as 'silently do > nothing' as > at present). > > Dick: if using a string key, it is neat to have something like > "bit_4_to_12" > so dot syntax can be used. > > A lesson learned: test your code before trying to sell it! > > > >
Open this post in threaded view
|

## Re: An operator syntax for bitfields using metatables

 > 64-bit IEEE floating point: > <> > > IP datagram: > <       ID:16, Flgs:3, FragOff:13, >       TTL:8, Proto:8, HdrChkSum:16, >       SrcIP:32, DestIP:32, RestDgram/binary>> Given these strings it easy to parse them into a Lua table containing the names and starting and end points of the bitfields described there. That table can then be used to set index and newindex metamethods for get and put the bitfields using Lua syntax, or perhaps pack and unpack functions: template = "<>" name = "64-bit IEEE floating point" b = bitfelds.parse(name,template) s = io.read(file,b.size) t = b:unpack(s) print(t.Sign,t.Exponent,t.Mantissa) t.Exponent=0 s = b:pack(t)
Open this post in threaded view
|

## RE: An operator syntax for bitfields using metatables

 In reply to this post by Robert Virding Diverging from the original topic a bit... Robert wrote: > While operators for bit-fields are very useful I think a > better way is a syntax which allows you describe/build > the whole bitstring in one go. We have this in Erlang and > it makes building/pulling apart protocol packets very > much easier and clearer. ... > This really is powerful and simplifies a lot of things > and I think that something similar would be useful for Lua > if you seriously intend to work with bitfields. I agree - something like vstruct [1, 2] could do the job. I have used it for just that, packing/unpacking network protocols. Tom [1] https://github.com/ToxicFrog/vstruct[2] http://lua-users.org/lists/lua-l/2013-01/msg00433.html
Open this post in threaded view
|

## Re: An operator syntax for bitfields using metatables

 Thomas Buergel <[hidden email]> writes: > I agree - something like vstruct [1, 2] could do the job. I have used > it for just that, packing/unpacking network protocols. The last time I looked, vstruct seemed fairly slow and made a lot of garbage...  Roberto's struct or Luiz's lpack might be better bets... -miles -- Marriage, n. The state or condition of a community consisting of a master, a mistress and two slaves, making in all, two.
Open this post in threaded view
|

## Re: An operator syntax for bitfields using metatables

 In reply to this post by Luiz Henrique de Figueiredo On Mon, Jul 15, 2013, at 09:58 PM, Luiz Henrique de Figueiredo wrote: > > 64-bit IEEE floating point: > > <> > > > > IP datagram: > > < >       ID:16, Flgs:3, FragOff:13, > >       TTL:8, Proto:8, HdrChkSum:16, > >       SrcIP:32, DestIP:32, RestDgram/binary>> > > Given these strings it easy to parse them into a Lua table containing the > names and starting and end points of the bitfields described there. That > table can then be used to set index and newindex metamethods for get and > put the bitfields using Lua syntax, or perhaps pack and unpack functions: > > template = "<>" > name = "64-bit IEEE floating point" > b = bitfelds.parse(name,template) > s = io.read(file,b.size) > t = b:unpack(s) > print(t.Sign,t.Exponent,t.Mantissa) > t.Exponent=0 > s = b:pack(t) > Erlang's bit syntax is a joy to work with and I miss it every time I use another programming language to decode protocols. In particular, I like the inline binding to variables as opposed to the regex/scanf like approach taken by other languages. An efficient analogue (to Erlang's bit syntax) in Lua would be tremendously useful. Lua tables would work great for something like this.  :) /todd
Open this post in threaded view
|

## Re: An operator syntax for bitfields using metatables

 It was thus said that the Great Todd Coram once stated: > On Mon, Jul 15, 2013, at 09:58 PM, Luiz Henrique de Figueiredo wrote: > > > 64-bit IEEE floating point: > > > <> > > > > > > IP datagram: > > > < > >       ID:16, Flgs:3, FragOff:13, > > >       TTL:8, Proto:8, HdrChkSum:16, > > >       SrcIP:32, DestIP:32, RestDgram/binary>> > > > > Given these strings it easy to parse them into a Lua table containing the > > names and starting and end points of the bitfields described there. That > > table can then be used to set index and newindex metamethods for get and > > put the bitfields using Lua syntax, or perhaps pack and unpack functions: > > > > template = "<>" > > name = "64-bit IEEE floating point" > > b = bitfelds.parse(name,template) > > s = io.read(file,b.size) > > t = b:unpack(s) > > print(t.Sign,t.Exponent,t.Mantissa) > > t.Exponent=0 > > s = b:pack(t) > > > > Erlang's bit syntax is a joy to work with and I miss it every time I use > another programming language to decode protocols. > In particular, I like the inline binding to variables as opposed to the > regex/scanf like approach taken by other languages. > > An efficient analogue (to Erlang's bit syntax) in Lua would be > tremendously useful. > > Lua tables would work great for something like this.  :)   A good test for this would be DNS packets.  The header portion isn't that bad:         id:16,         query:1, -- boolean, 0=query, 1=response         opcode:4, -- type of query         aa:1, -- boolean, authoritative answer         tc:1, -- boolean, truncation         rd:1, -- boolean, recursion desired         ra:1, -- boolean, recursion available         z:1, -- not used, must be 0         ad:1, -- boolean, authentic bit         cd:1, -- boolean, checking disabled         rcode:4, -- return code         qcount:16, -- query count         ancount:16, -- answer count         nscount:16, -- nameserver count         adcount:16 -- additional count But the record types are variable length:         name:??, -- key for record, example: www.conman.org         type:16, -- type of record         class:16, -- class of record         ttl:16, -- Time-To-Live         rdlength:16, -- length of rdata         rdata:?? -- rest of data And the name portion isn't easy to skip---you *have* to parse it to find the end (check RFC-1035 for the gory details).     -spc
Open this post in threaded view
|

## Re: An operator syntax for bitfields using metatables

Open this post in threaded view
|

## Re: An operator syntax for bitfields using metatables

 In reply to this post by Robert Virding 2013/7/16 Robert Virding <[hidden email]>: > While operators for bit-fields are very useful I think a better way is a syntax which allows you describe/build the whole bitstring in one go. We have this in Erlang and it makes building/pulling apart protocol packets very much easier and clearer. Some simple examples of the top of my head: > > 64-bit IEEE floating point: > <> > > IP datagram: > <       ID:16, Flgs:3, FragOff:13, >       TTL:8, Proto:8, HdrChkSum:16, >       SrcIP:32, DestIP:32, RestDgram/binary>> > > The second one is a bit longer than 32 bits. :-) I think you get the idea. We can also use the same syntax for pulling apart the binary/bitstring. This really is powerful and simplifies a lot of things and I think that something similar would be useful for Lua if you seriously intend to work with bitfields. There are already plenty of libraries that simplify parsing such binary structures, some using the Lua flexible syntax as a kind of DSL to describe them. I don't see how another set of syntactic constructs could make it any easier or clearer. Can you elaborate on that claim?
Open this post in threaded view
|

## Re: An operator syntax for bitfields using metatables

 > From: "Jerome Vuarand" <[hidden email]> > > 2013/7/16 Robert Virding <[hidden email]>: > > While operators for bit-fields are very useful I think a better way > > is a syntax which allows you describe/build the whole bitstring in > > one go. We have this in Erlang and it makes building/pulling apart > > protocol packets very much easier and clearer. Some simple > > examples of the top of my head: > > > > 64-bit IEEE floating point: > > <> > > > > IP datagram: > > < >       ID:16, Flgs:3, FragOff:13, > >       TTL:8, Proto:8, HdrChkSum:16, > >       SrcIP:32, DestIP:32, RestDgram/binary>> > > > > The second one is a bit longer than 32 bits. :-) I think you get > > the idea. We can also use the same syntax for pulling apart the > > binary/bitstring. This really is powerful and simplifies a lot of > > things and I think that something similar would be useful for Lua > > if you seriously intend to work with bitfields. > > There are already plenty of libraries that simplify parsing such > binary structures, some using the Lua flexible syntax as a kind of > DSL > to describe them. I don't see how another set of syntactic constructs > could make it any easier or clearer. Can you elaborate on that claim? - I make no claims to be a Lua expert, so I am not surprised to hear that there are libraries that support this type of thing. - My comment was on the suggestion to add operator syntax for bitfields. I have done enough bitfiddling in both C and Erlang to appreciate Erlang's bitsyntax for building/pulling apart bitstrings. - If it is something that is to be done often and speed is critical then you need some for of "compiler" support. You can either add it to the language like in Erlang or maybe have it so that your library can compile a bit pattern into some descriptive data structure or code for processing the bitstring. Erlang can do something similar to this for regular expressions. - To be clear I wasn't suggesting using the Erlang syntax, it is to Erlangy and wouldn't fit into Lua. Although it does have a similar feel as Lua syntax for constructing tables and you could probably add some special syntax which builds a suitable table. Robert