binary data as string snag

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

binary data as string snag

John Belmonte-2
Maybe this is obvious but...

I started implementing Perl-style pack/unpack functions using Lua strings for binary data but
ran into a snag.  Using strings is fine if you're just looking to write data out to a file.
However if you want to pass the strings to C to be used directly then there are alignment
issues.

Because of the way string data is represented in Lua, it will not have the alignment in use by
malloc but instead malloc's alignment plus some constant offset depending on the position of the
"str" field in the TString structure of lobject.h.  (Based on first guess.)

-John



Reply | Threaded
Open this post in threaded view
|

Re: binary data as string snag

Roberto Ierusalimschy
> However if you want to pass the strings to C to be used directly then
> there are alignment issues.

Yes; Lua gives no garanties about alignment of its strings (except that
they are valid for strings ;-).


> I started implementing Perl-style pack/unpack functions [...]

Wouldn't you have to handle alignment details anyway? For instance, if
you pack with format "cf", the floating point will not be aligned.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: binary data as string snag

Edgar Toernig
In reply to this post by John Belmonte-2
Hi,

John Belmonte wrote:
> I started implementing Perl-style pack/unpack functions using Lua strings for
> binary data but ran into a snag.  Using strings is fine if you're just looking
> to write data out to a file.
> However if you want to pass the strings to C to be used directly then there are
> alignment issues.

You could try to stuff padding bytes at the head of the string (number of bytes
detected during libsetup).  But I would simply copy it like that:

struct bar {...};
extern void foo(struct bar *);

static int
call_foo(lua_State *L)
{
	struct bar tmp;

	foo(arg_check_struct(L, 1, &tmp, sizeof(tmp)));
	return 0;
}

static void *
arg_check_struct(lua_State *L, int argno, void *buf, size_t len)
{
	size_t l;
	char *str = luaL_check_lstr(L, argno, &l);

	luaL_arg_check(L, l == len, argno, "invalid structure size");
	memcpy(buf, str, l);
	return buf;
}

Compared to the creation of the packed strings the time to copy it once more
should be negligible.  Only 'P' will not work that way.  The last resort
would be a userdata...

Ciao, ET.


Reply | Threaded
Open this post in threaded view
|

Re: binary data as string snag

John Belmonte-2
In reply to this post by Roberto Ierusalimschy
Roberto wrote:

> Yes; Lua gives no garanties about alignment of its strings (except that
> they are valid for strings ;-).

Somehow this seems unfortunate.  Lua is supposed to interface with C well.  Yet it cannot load
binary data for C without converting it to a userdata type which requires a memory copy.  Then
if you want Lua to save the data back out, you need to convert back to a string with another
memory copy.  I guess the read/write functions can be rewritten for some userdata type, but is
it really such a stretch for Lua to align buffers as malloc does?  The argument of wasting space
cannot be used since TString already has padding to match malloc's alignment.

(By the way, according to ANSI C malloc's "allocated space is suitably aligned for storage of
any type of object".)

> Wouldn't you have to handle alignment details anyway? For instance, if
> you pack with format "cf", the floating point will not be aligned.

This is not related to the issue of the start alignment of the binary data.  Of course the
person writing the template needs to know exact the structure he's working with, including
alignment details.  There is 'x' for byte padding.  If floats are 4-byte aligned you would write
"cx3f".

-John