Lua "amalgamation" for 5-10% speed improvement and ease of use

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua "amalgamation" for 5-10% speed improvement and ease of use

Aladdin LampÃ
Hi list!

I was reading the following page related to the SQLite project: http://www.sqlite.org/cvstrac/wiki?p=TheAmalgamation
and found this:
"In addition to making SQLite easier to incorporate into other projects, the amalgamation also makes it run faster. Many compilers are able to do additional optimizations on code when it is contained with in a single translation unit such as it is in the amalgamation. We have measured performance improvements of between 5 and 10% when we use the amalgamation to compile SQLite rather than individual source files."

So I thought it might be a good thing for Lua too to build it as an "amalgamation". I generated "luall.c" with files in the following order (based on modules dependencies): luaconf.h, lua.h, lauxlib.h, lualib.h, llimits.h, lbaselib.c, ldblib.c, lopcodes.h, loadlib.c, ltablib.c, lmathlib.c, lstrlib.c, lauxlib.c, lobject.h, loslib.c, lua.c, liolib.c, linit.c, lmem.h, lopcodes.c, lzio.h, ltm.h, ltable.h, lfunc.h, lgc.h, lapi.h, lstate.h, lparser.h, llex.h, lundump.h, lstring.h, lzio.c, lfunc.c, ldo.h, ldebug.h, ldump.c, lcode.h, lstate.c, lcode.c, print.c, lvm.h, ltm.c, ltable.c, lparser.c, lmem.c, llex.c, lgc.c, lundump.c, lstring.c, ldo.c, ldebug.c, lvm.c, lobject.c, lapi.c
and removed all the "#include" of lua source files inside them.

The generated file compiles almost fine with VS2008 but I had to turn on 2 flags: "LUA_CORE" and "luaall_c" and I get 2 warnings:
- warning C4005: 'LUA_CORE': macro redefinition
- warning C4005: 'VOID': macro redefinition

Note: For some reason, I also had to rename the function "LoadString" to something else because of a conflict with another "LoadStringW" function.

Now, what do you Lua gurus think of all that? Any comments appreciated:
1. Is it after all a good idea to do this amalgamation for Lua?
2. Is it a problem to turn on the 2 flags LUA_CORE and luaall_c?
3. Can I safely ignore the 2 warnings I get about macro redefinition?
4. How can I avoid to rename the "LoadString" function? (because this is unnecessary when compiling each file individually, there shoud be a better way to solve this)
5. Overall, did I do things the right way? or am I missing other obvious ways of doing this?
6. May I get the advertised 5-10% speed improvement with VS2008?
7. If this amalgamation is after all a good idea, is there a chance that it gets supported as a standard way of building Lua in the future?

Thank you for your help,
Aladdin

_________________________________________________________________
Plus besoin de clé usb ! Stockez votre musique en ligne gratuitement !
http://www.windowslive.fr/skydrive/

Reply | Threaded
Open this post in threaded view
|

Re: Lua "amalgamation" for 5-10% speed improvement and ease of use

Luiz Henrique de Figueiredo
> 1. Is it after all a good idea to do this amalgamation for Lua?

Yes. And we distribute one in etc/all.c.

> 2. Is it a problem to turn on the 2 flags LUA_CORE and luaall_c?

No, but I think you only need to turn on luaall_c, like etc/all.c does.

> 3. Can I safely ignore the 2 warnings I get about macro redefinition?

I don't get any compilation warnings with etc/all.c with gcc in Linux.

> 4. How can I avoid to rename the "LoadString" function?

Use a compiler that does not polutte the global namespace :-)

> 5. Overall, did I do things the right way? or am I missing other obvious ways of doing this?

You missed etc/all.c... Note that it uses the Lua sources unmodified.
No need to remove the "#include" from them.

> 7. If this amalgamation is after all a good idea, is there a chance that it gets supported as a standard way of building Lua in the future?

Ditto.

--lhf

Reply | Threaded
Open this post in threaded view
|

Re: Lua "amalgamation" for 5-10% speed improvement and ease of use

René Rebe
Hi,

7. If this amalgamation is after all a good idea, is there a chance that it gets supported as a standard way of building Lua in the future?
Although I did not took a look a this "amalgamation" I guess it's just including all sources in one big compiler invocation ala etc/all.c, such as the KDE project
also already did for a long time when build with --enable-final.

Note that all this is handled by modern compilers with link time optimizations, such as possible with LLVM and it's work is in progress for gcc. So hopefully this manually cat'ing together source files can soon be a relict of the past :-)

Yours,

--
 René Rebe - ExactCODE GmbH - Europe, Germany, Berlin
 http://exactcode.de | http://t2-project.org | http://rene.rebe.name


Reply | Threaded
Open this post in threaded view
|

Re: Lua "amalgamation" for 5-10% speed improvement and ease of use

Joshua Jensen
In reply to this post by Luiz Henrique de Figueiredo
----- Original Message -----
From: Luiz Henrique de Figueiredo
Date: 8/27/2008 5:10 AM
4. How can I avoid to rename the "LoadString" function?
    
Use a compiler that does not polutte the global namespace :-)
  
LoadString has a #define in the Windows headers.  It is very sad that Microsoft redefines common words behind your back.  You could either #undef it or use #pragma push_macro("LoadString") then a #undef LoadString and then a #pragma pop_macro("LoadString").
5. Overall, did I do things the right way? or am I missing other obvious ways of doing this?
    
You missed etc/all.c... Note that it uses the Lua sources unmodified.
No need to remove the "#include" from them
Why does the etc/all.c amalgamation #include "lua.c"?

Josh
Reply | Threaded
Open this post in threaded view
|

Re: Lua "amalgamation" for 5-10% speed improvement and ease of use

Joshua Jensen
In reply to this post by René Rebe
----- Original Message -----
From: René Rebe
Date: 8/27/2008 6:08 AM
Note that all this is handled by modern compilers with link time optimizations, such as possible with LLVM and it's work is in progress for gcc. So hopefully this manually cat'ing together source files can soon be a relict of the past :-)
The compile times are also seriously faster. In fact, for every modern compiler I've worked with across many platforms, creating an amalgamation file (we use the term 'lump') improves overall compilation times in a major way.

Josh

Reply | Threaded
Open this post in threaded view
|

Re: Lua "amalgamation" for 5-10% speed improvement and ease of use

Luiz Henrique de Figueiredo
In reply to this post by Joshua Jensen
> Why does the etc/all.c amalgamation #include "lua.c"?

Because it is meant to build a complete Lua interpreter, not just the library.
The version attached below has options to build the interpreter, the compiler,
or just the library. --lhf
/*
* all.c -- Lua core, libraries and interpreter in a single file
*/

/* default is to build the full interpreter */
#ifndef MAKE_LIB
#ifndef MAKE_LUAC
#undef  MAKE_LUA
#define MAKE_LUA
#endif
#endif

#define luaall_c

/* core -- used by all */
#include "lapi.c"
#include "lcode.c"
#include "ldebug.c"
#include "ldo.c"
#include "ldump.c"
#include "lfunc.c"
#include "lgc.c"
#include "llex.c"
#include "lmem.c"
#include "lobject.c"
#include "lopcodes.c"
#include "lparser.c"
#include "lstate.c"
#include "lstring.c"
#include "ltable.c"
#include "ltm.c"
#include "lundump.c"
#include "lvm.c"
#include "lzio.c"

/* auxiliary library -- used by all */
#include "lauxlib.c"

/* standard library  -- not used by luac */
#ifndef MAKE_LUAC
#include "lbaselib.c"
#include "ldblib.c"
#include "liolib.c"
#include "lmathlib.c"
#include "loadlib.c"
#include "loslib.c"
#include "lstrlib.c"
#include "ltablib.c"
#endif

/* lua */
#ifdef MAKE_LUA
#include "linit.c"
#include "lua.c"
#endif

/* luac */
#ifdef MAKE_LUAC
#include "print.c"
#include "luac.c"
#endif
Reply | Threaded
Open this post in threaded view
|

Re: Lua "amalgamation" for 5-10% speed improvement and ease of use

René Rebe
In reply to this post by Joshua Jensen
Joshua Jensen wrote:
----- Original Message -----
From: René Rebe
Date: 8/27/2008 6:08 AM
Note that all this is handled by modern compilers with link time optimizations, such as possible with LLVM and it's work is in progress for gcc. So hopefully this manually cat'ing together source files can soon be a relict of the past :-)
The compile times are also seriously faster. In fact, for every modern compiler I've worked with across many platforms, creating an amalgamation file (we use the term 'lump') improves overall compilation times in a major way.
I guess mostly C code then? Many mid-sized C++ files (making heavy use of
static polymorphisms) already bring quite some boxes to swap to disk, before
g++ is eventuallyed killed away Out Of Memory, ...

--
 René Rebe - ExactCODE GmbH - Europe, Germany, Berlin
 http://exactcode.de | http://t2-project.org | http://rene.rebe.name


Reply | Threaded
Open this post in threaded view
|

Re: Lua "amalgamation" for 5-10% speed improvement and ease of use

dcharno
In reply to this post by Aladdin LampÃ
I was reading the following page related to the SQLite project: http://www.sqlite.org/cvstrac/wiki?p=TheAmalgamation
and found this:
"In addition to making SQLite easier to incorporate into other projects, the amalgamation also makes it run faster. Many compilers are able to do additional optimizations on code when it is contained with in a single translation unit such as it is in the amalgamation. We have measured performance improvements of between 5 and 10% when we use the amalgamation to compile SQLite rather than individual source files."

This is an old thread, but ...

After using SQLite in a number of projects I really came to appreciate the convenience of the amalgamation approach. etc/all.c helps quite a bit, but it is nice dealing with fewer files.

The attached script makes amalgamated C and header files based upon all.c. To use them in a project, just build in am-lua.c and include am-lua.h. Maybe someone else finds it useful


--
-- Copy inputs to output, inlining any Lua #include once
--

local PREFIX = "am-"
local SRC_PATH = "../src/"

function CreateAmalgamation(ofilename, files)

  local out = assert(io.open(ofilename, "w"))
  local headers = { ["lua.c"]=true }

  function CopyWithInline(filename)
  if headers[filename] == nil then
    headers[filename] = true

    local inp = assert(io.open(SRC_PATH .. filename, "r"))
      for l in inp:lines() do
        local inc = l:match('#include "([^"]+)"')
        if inc and (inc ~= "luaconf.h") then
          CopyWithInline(inc)
        else
          out:write(l .. "\n")
        end
      end
    end
  end

  for _, file in ipairs(files) do
    CopyWithInline(file)
  end

end

CreateAmalgamation(PREFIX.."lua.c", {"../etc/all.c"})
CreateAmalgamation(PREFIX.."lua.h", { "lua.h", "lauxlib.h", "lualib.h"})



Reply | Threaded
Open this post in threaded view
|

Re: Lua "amalgamation" for 5-10% speed improvement and ease of use

Asko Kauppi

Why exactly are you merging together the headers as well? That is for "ease of use" (which is questionable), not for speed, right?

For just getting the compiler optimizations, simply concatenating the .c's together would be enough? Roberto mentioned to have a Makefile target for this in upcoming versions, that would indeed be fine.

-asko


dcharno kirjoitti 15.2.2009 kello 19:49:

I was reading the following page related to the SQLite project: http://www.sqlite.org/cvstrac/wiki?p=TheAmalgamation
and found this:
"In addition to making SQLite easier to incorporate into other projects, the amalgamation also makes it run faster. Many compilers are able to do additional optimizations on code when it is contained with in a single translation unit such as it is in the amalgamation. We have measured performance improvements of between 5 and 10% when we use the amalgamation to compile SQLite rather than individual source files."

This is an old thread, but ...

After using SQLite in a number of projects I really came to appreciate the convenience of the amalgamation approach. etc/all.c helps quite a bit, but it is nice dealing with fewer files.

The attached script makes amalgamated C and header files based upon all.c. To use them in a project, just build in am-lua.c and include am-lua.h. Maybe someone else finds it useful


--
-- Copy inputs to output, inlining any Lua #include once
--

local PREFIX = "am-"
local SRC_PATH = "../src/"

function CreateAmalgamation(ofilename, files)

 local out = assert(io.open(ofilename, "w"))
 local headers = { ["lua.c"]=true }

 function CopyWithInline(filename)
 if headers[filename] == nil then
   headers[filename] = true

   local inp = assert(io.open(SRC_PATH .. filename, "r"))
     for l in inp:lines() do
       local inc = l:match('#include "([^"]+)"')
       if inc and (inc ~= "luaconf.h") then
         CopyWithInline(inc)
       else
         out:write(l .. "\n")
       end
     end
   end
 end

 for _, file in ipairs(files) do
   CopyWithInline(file)
 end

end

CreateAmalgamation(PREFIX.."lua.c", {"../etc/all.c"})
CreateAmalgamation(PREFIX.."lua.h", { "lua.h", "lauxlib.h", "lualib.h"})




Reply | Threaded
Open this post in threaded view
|

Re: Lua "amalgamation" for 5-10% speed improvement and ease of use

dcharno
Asko Kauppi wrote:
Why exactly are you merging together the headers as well? That is for "ease of use" (which is questionable), not for speed, right?

You mean lua.h, lauxlib.h, and lualib.h?


Reply | Threaded
Open this post in threaded view
|

Re: Lua "amalgamation" for 5-10% speed improvement and ease of use

Asko Kauppi

Yes, the script had:

CreateAmalgamation(PREFIX.."lua.h", { "lua.h", "lauxlib.h", "lualib.h"})

I'm just double checking that there's no performance reason to make all three under the same file. Maybe this is vague and I should shut up?

-asko


dcharno kirjoitti 15.2.2009 kello 20:40:

Asko Kauppi wrote:
Why exactly are you merging together the headers as well? That is for "ease of use" (which is questionable), not for speed, right?

You mean lua.h, lauxlib.h, and lualib.h?



Reply | Threaded
Open this post in threaded view
|

Re: Lua "amalgamation" for 5-10% speed improvement and ease of use

dcharno
Asko Kauppi wrote:

Yes, the script had:

CreateAmalgamation(PREFIX.."lua.h", { "lua.h", "lauxlib.h", "lualib.h"})

I'm just double checking that there's no performance reason to make all three under the same file. Maybe this is vague and I should shut up?

Just for "convenience". Not a huge savings, but I've never emebedded Lua without using them anyways.