Speed over beauty - vstruct vs struct

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Speed over beauty - vstruct vs struct

David Crayford
I've recently been investigating Lua structure packing libraries.
vstruct is a thing of beauty as it allows me to layout a format string
that looks like a structure,
and it supports names, tables etc. The downside is that compared to
struct it's very slow. struct is orders of magnitude faster but it's
interface is not so nice. It brings
me to tears to choose a library for speed over beauty but performance is
important to my project.

How easy would it be to rewrite certain parts of vstruct in C or maybe
even front end struct with the same semantics as vstruct? The format
string below is what
I've knocked up for vstruct.

local fmtstr =
[[
   request:  s4
   key:{
     department: s1
     employee_no: s5
   }
   name: s20
   addr1: s20
   addr2: s20
   addr3: s20
   phone_no: s8
   timestamp: s8
   salary: s8
   start_date: s8
   remarks: s32
   msg: s60
   upd_ind: {
     name: s1
     addr1: s1
     addr2: s1
     addr3: s1
     phone_no: s1
     salary: s1
     start_date: s1
     remarks: s1
   }
   browseeq: s8
   update_ind: s1
   delete_ind: s1
   add_ind: s1
   error_ind: s1
   browse_ind: s1
   dept_ind: s1
   count: s2
   total: s2
   item_no: s2
   sbtot: s2
   sbnum: s2
   lineout: { 10 * {
     filler1: s6
     employee_no: s5
     filler2: s3
     employee_name: s20
     filler3: s3
     employee_phone: s8
     filler4: s34
   }}
]]




Reply | Threaded
Open this post in threaded view
|

Re: Speed over beauty - vstruct vs struct

Steve Litt
On Wed, 23 Jul 2014 20:21:42 +0800
David Crayford <[hidden email]> wrote:

> I've recently been investigating Lua structure packing libraries.
> vstruct is a thing of beauty as it allows me to layout a format
> string that looks like a structure,
> and it supports names, tables etc. The downside is that compared to
> struct it's very slow. struct is orders of magnitude faster but it's
> interface is not so nice. It brings
> me to tears to choose a library for speed over beauty but performance
> is important to my project.
>
> How easy would it be to rewrite certain parts of vstruct in C or
> maybe even front end struct with the same semantics as vstruct? The
> format string below is what
> I've knocked up for vstruct.
>
> local fmtstr =
> [[
>    request:  s4
>    key:{
>      department: s1
>      employee_no: s5
>    }
>    name: s20
>    addr1: s20
>    addr2: s20
>    addr3: s20
>    phone_no: s8
>    timestamp: s8
>    salary: s8
>    start_date: s8
>    remarks: s32
>    msg: s60
>    upd_ind: {
>      name: s1
>      addr1: s1
>      addr2: s1
>      addr3: s1
>      phone_no: s1
>      salary: s1
>      start_date: s1
>      remarks: s1
>    }
>    browseeq: s8
>    update_ind: s1
>    delete_ind: s1
>    add_ind: s1
>    error_ind: s1
>    browse_ind: s1
>    dept_ind: s1
>    count: s2
>    total: s2
>    item_no: s2
>    sbtot: s2
>    sbnum: s2
>    lineout: { 10 * {
>      filler1: s6
>      employee_no: s5
>      filler2: s3
>      employee_name: s20
>      filler3: s3
>      employee_phone: s8
>      filler4: s34
>    }}
> ]]

It's a process. You need to find some way of profiling vstruct to see
where it's burning up all that time,  and fix that one part. If it's
several order of magnitude slower, I'm betting that you'll find one
little tiny part that's written stupidly (hey, I've written some pretty
stupid code myself), and you can easily fix that one little spot to
perform well, and because it's the bottleneck, the throughput of the
whole will rise proportionally to the bottleneck you cleared.

Rewriting the whole thing in C won't help that much if you translate
the bad algorithm. I'd first find out where all that time is getting
consumed.

SteveT

Steve Litt                *  http://www.troubleshooters.com/
Troubleshooting Training  *  Human Performance


Reply | Threaded
Open this post in threaded view
|

Re: Speed over beauty - vstruct vs struct

Xavier Wang
In reply to this post by David Crayford
If you want a C implement, you can try my lbuffer[1], it's a mutable
string implement of Lua. it has a pack/unpack interface.

some examples (from README.rst):

-- read *.mo file
function read_mofile(b)
    local info = b:unpack [[ {
        magic = i,
        revision = i,
        nstrings = i,
        orig_tab_offset = i,
        trans_tab_offset = i,
        hash_tab_size = i,
        hash_tab_offset = i,
    } ]]

    local trans = {}
    for i = 0, info.nstrings-1 do
        local o_len, o_offset = b:unpack(info.orig_tab_offset+8*i+1, "<ii")
        local t_len, t_offset = b:unpack(info.trans_tab_offset+8*i+1, "<ii")
        local os = b:unpack(o_offset+1, "s")
        local ts = b:unpack(t_offset+1, "s")
        trans[os] = ts
    end
    return info, trans
end


[1]: https://github.com/starwing/lbuffer

2014-07-23 20:21 GMT+08:00 David Crayford <[hidden email]>:

> I've recently been investigating Lua structure packing libraries. vstruct is
> a thing of beauty as it allows me to layout a format string that looks like
> a structure,
> and it supports names, tables etc. The downside is that compared to struct
> it's very slow. struct is orders of magnitude faster but it's interface is
> not so nice. It brings
> me to tears to choose a library for speed over beauty but performance is
> important to my project.
>
> How easy would it be to rewrite certain parts of vstruct in C or maybe even
> front end struct with the same semantics as vstruct? The format string below
> is what
> I've knocked up for vstruct.
>
> local fmtstr =
> [[
>   request:  s4
>   key:{
>     department: s1
>     employee_no: s5
>   }
>   name: s20
>   addr1: s20
>   addr2: s20
>   addr3: s20
>   phone_no: s8
>   timestamp: s8
>   salary: s8
>   start_date: s8
>   remarks: s32
>   msg: s60
>   upd_ind: {
>     name: s1
>     addr1: s1
>     addr2: s1
>     addr3: s1
>     phone_no: s1
>     salary: s1
>     start_date: s1
>     remarks: s1
>   }
>   browseeq: s8
>   update_ind: s1
>   delete_ind: s1
>   add_ind: s1
>   error_ind: s1
>   browse_ind: s1
>   dept_ind: s1
>   count: s2
>   total: s2
>   item_no: s2
>   sbtot: s2
>   sbnum: s2
>   lineout: { 10 * {
>     filler1: s6
>     employee_no: s5
>     filler2: s3
>     employee_name: s20
>     filler3: s3
>     employee_phone: s8
>     filler4: s34
>   }}
> ]]
>
>
>
>



--
regards,
Xavier Wang.