5.1 beta problem linking luac against .so

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

5.1 beta problem linking luac against .so

John Belmonte
I can link lua interpreter against the .so form of the Lua library, but
when linking luac the following undefined's come up:

  luac.o: In function `pmain':
  luac.c:(.text+0x1f2): undefined reference to `luaF_newproto'
  luac.c:(.text+0x235): undefined reference to `luaS_newlstr'
  luac.c:(.text+0x26d): undefined reference to `luaM_realloc_'
  luac.c:(.text+0x29c): undefined reference to `luaM_realloc_'
  luac.c:(.text+0x31c): undefined reference to `luaU_dump'
  luac.c:(.text+0x362): undefined reference to `luaM_toobig'
  luac.c:(.text+0x389): undefined reference to `luaM_toobig'
  luac.c:(.text+0x47a): undefined reference to `luaD_growstack'
  print.o: In function `luaU_print':
  print.c:(.text+0x418): undefined reference to `luaP_opnames'
  print.c:(.text+0x429): undefined reference to `luaP_opmodes'
  print.c:(.text+0x67c): undefined reference to `luaP_opmodes'

If I remove the "__attribute__((visibility("hidden")))" from LUAI_FUNC
and LUAI_DATA in luaconf.h then the link is OK.  I suspect that luac is
accessing some internals of the Lua library.  The functions and data
required by luac will have to be exported.

--John

Reply | Threaded
Open this post in threaded view
|

Re: 5.1 beta problem linking luac against .so

Mike Pall-60
Hi,

John Belmonte wrote:
> I can link lua interpreter against the .so form of the Lua library,

Ouch -- I knew it. That's for the Debian package, right?

PLEASE, PLEASE DON'T DO THIS!

It has been discussed a few times on the list. It's a deadly
sin to link the Lua interpreter against a shared library compiled
with -fpic on x86 platforms. You'll loose anywhere between 10-20%
of speed and this is clearly unacceptable.

The main loop of the interpreter (luaV_execute) is very sensitive
to the number of available registers. Compiling a shared library
with -fpic on x86 makes 1 out of 6 registers unavailable and
forces the compiler to produce really bad code.

The best idea is _not_ to compile a shared library at all. Just
link lua (the interpreter) statically. There is no downside to
it -- it's very small after all. And executables _are_ shared in
memory, too.

Any application incorporating Lua is better served by linking it
statically. It needs to be modified in many cases. Many apps are
very version specific, since the ABI changes on minor Lua
versions, too.

And if you really, really insist on making a shared library
available then by all means compile it twice and link the Lua
interpreter statically against the version _not_ compiled
with -fpic. That's exactly what is usually done with Perl
and Python (for the very same reason). Thank you!

> but when linking luac the following undefined's come up:
>   luac.c:(.text+0x1f2): undefined reference to `luaF_newproto'
>[...]

These are internal symbols after all. Not that luac needs to be
fast, but it's a lot easier to link it statically.

[
A better solution is to provide introspection capabilities in
the Lua core and then make luac a simple lua script. There is
such a script right now, but it has limited features.

FYI: LuaJIT has these introspection capabilities added and they
need less than 1.5K of code space. IMHO it would be entirely
reasonable to add this to the standard Lua core. Guess it's a bit
late now to discuss this for 5.1.0 ...
]

> If I remove the "__attribute__((visibility("hidden")))" from LUAI_FUNC
> and LUAI_DATA in luaconf.h then the link is OK.

This is a bad idea because then you'll force the compiler to
generate even worse code for shared libraries. The attribute was
introduced to avoid this (and to avoid pollution of the exported
symbol space).

Ok, you could remove the attribute only for the required symbols.
But your best bet is to link luac statically and avoid all the
hassle. We are talking about 150K -- this is 1/1.000.000 of the
average size harddisk sold nowadays.

[NB: This discussion is about desktop or server environments, not
about embedded environments.]

Bye,
     Mike

Reply | Threaded
Open this post in threaded view
|

Re: 5.1 beta problem linking luac against .so

Adam D. Moss
Mike Pall wrote:
But your best bet is to link luac statically and avoid all the
hassle. We are talking about 150K -- this is 1/1.000.000 of the
average size harddisk sold nowadays.

A minor datapoint - luac compiles to under 68K here.

On a tangent, Mike, are you waiting for the final 5.1 release
before issuing a LuaJIT update?  With the Coco/old-glibc problem
avoided I was very impressed at the stability of LuaJIT 1.0.3
and was hoping to make it my 'standard' Lua for developing
against before too long.

Cheers,
--Adam
--
Adam D. Moss   -   [hidden email]

Reply | Threaded
Open this post in threaded view
|

LuaJIT release plan (was: 5.1 beta problem linking luac against .so)

Mike Pall-60
Hi,

Adam D. Moss wrote:
> On a tangent, Mike, are you waiting for the final 5.1 release
> before issuing a LuaJIT update?  With the Coco/old-glibc problem
> avoided I was very impressed at the stability of LuaJIT 1.0.3
> and was hoping to make it my 'standard' Lua for developing
> against before too long.

I had to stop working on LuaJIT for the past weeks. I'm not sure
when I can continue working on 1.0.4. There's still a lot to be
done before I can release it (it's broken for many corner cases
right now). This is more or less independent of the Lua release
plan.

Here's the incomplete changelog to whet your appetite:

- Faster loads of FP constants. Remove narrow-to-wide store-to-load
  forwarding stalls.
- C stack is kept 16 byte aligned (mandatory for Mac OS X/x86).
  [Regression: slower for deeply recursive calls (ackermann, takfp).]
- Improved hinting for table keys derived from table values (t1[t2[x]]).
- Lookup hinting now works with arbitrary object types and
  supports chained lookups, too (__index).
- Use adaptive deoptimization for uncommon fallback cases. If
  runtime verification for a hint fails, the function is
  recompiled on-the-fly (instead of encoding the fallback case).
  Regular programs will hardly ever trigger deoptimization.
- The trace module prints a notice about deoptimization.
- Deoptimize calls with a type mismatch or a mismatch for an
  inlined function.
- Inlined OP_MOD (%). With better accuracy than the C variant, too.
- Inlined most math.* functions.
- Inlined string.len and string.sub.

[Some benchmarks (random, nsievebits, fasta, nbody, fannkuch)
 already run between 30%-100% faster than with 1.0.3.]

TODO list:

- Change function prologue to cope with 5.1-beta modifications.
- Fix all corner cases having to do with inlining.
- Inline pairs and ipairs iterators.
- Inline some of the base library functions.
- Inline table.insert and table.remove.
- Inline more string functions (main problem: libc NLS macros).
- Inline coroutine.yield and maybe coroutine.resume.
- Maybe inline OP_CONCAT for some common cases.
- Shortcut recursive calls to JIT compiled functions.
- Partially inline the C callgate. Maybe remove closure callgates.
- Stack frame reorganization.
- Combine MOVE/GETUPVAL with CALL/TAILCALL.
- Smoother fallback to partial deoptimization.
- Lots of doc updates.
- Update to new set of benchmarks.
- Plus many long term goals (new optimizer, better integer opts,
  profiling support etc.).

But you know ... I'd rather work full time on LuaJIT than do
other things ... I have so many more ideas ...

Bye,
     Mike

Reply | Threaded
Open this post in threaded view
|

Re: 5.1 beta problem linking luac against .so

Dušan Majkić-2
In reply to this post by Mike Pall-60
> The best idea is _not_ to compile a shared library at all. Just 
> link lua (the interpreter) statically. There is no downside to it

  (Somewhat off topic)
  
  Is there a way to statically link Lua in Delphi application?  
  
  I know that lua src should be compiled with Borland C++, and that 
  missing C RTL functions should be implemented. But I had no luck. 
  
  Docs say that dlls are preffered, and linking OBJs is possible with
  {$L some.obj} but nothing more. Zlib and jpeg are linked that way.
 




Reply | Threaded
Open this post in threaded view
|

Re: 5.1 beta problem linking luac against .so

Glenn Maynard
In reply to this post by Mike Pall-60
On Wed, Nov 23, 2005 at 08:48:14PM +0100, Mike Pall wrote:
> The best idea is _not_ to compile a shared library at all. Just
> link lua (the interpreter) statically. There is no downside to
> it -- it's very small after all. And executables _are_ shared in
> memory, too.

> Ok, you could remove the attribute only for the required symbols.
> But your best bet is to link luac statically and avoid all the
> hassle. We are talking about 150K -- this is 1/1.000.000 of the
> average size harddisk sold nowadays.

150k is a lot when it's repeated over a dozen or more binaries--for example,
a package containing a suite of small tools.  ("It's just 150k" is the
mindset that leads to the bloat of a lot of applications today--150k here
and 150k there adds up quickly.)

More importantly, there's the other major benefit of shared libraries:
updating them system-wide doesn't require updating every binary that
uses it.  This could be significant, for example, if a security-related
bug was fixed.

I statically link for other reasons (to avoid adding a runtime dependency),
but "there is no downside" is overstating the case.  (Many people would
call encouraging static linking to be a "deadly sin".  Caught between hells?)

-- 
Glenn Maynard

Reply | Threaded
Open this post in threaded view
|

Re: 5.1 beta problem linking luac against .so

Enrico Tassi-3
In reply to this post by Mike Pall-60
On Wed, Nov 23, 2005 at 08:48:14PM +0100, Mike Pall wrote:
> Hi,
> 
> John Belmonte wrote:
> > I can link lua interpreter against the .so form of the Lua library,
> 
> Ouch -- I knew it. That's for the Debian package, right?
> 
> PLEASE, PLEASE DON'T DO THIS!

I completely agree about your performance and disk-space considerations.
_but_ Debian has some "standards"...

I think we must provide both .a and .so for Debian. linking the
interpreter statically is not a big deal since it comes from the same
source package (as John already pointed out to me by provate mail).
The same for luac.

If you will need lua in applications that are not CPU-bound you should
prefer the .so, and this is because huge systems like Debian have to
consider maintainibility (read: updates should be easy). I know LUA is a really
stable software and patches are not so frequent... but I still think that 
we should prefer the easy-to-maintain way.

If you want all the CPU cicles you can have you can link lua statically
just changing the linker command line arguments, not a big problem. 

Currently on Debian there are few packages depending on liblua50, but I
hope they will grow:

  xmoto
  stratagus
  luasocket
  lua50
  liblualib50
  ion3
  ion2
  imapfilter
  elinks
  celestia

I'm interested in packaging at least luacurl, luaexpat and luafilesystem.

ciao
-- 
Enrico Tassi

Reply | Threaded
Open this post in threaded view
|

Re: 5.1 beta problem linking luac against .so

Klaus Ripke
On Thu, Nov 24, 2005 at 11:55:06PM +0100, Enrico Tassi wrote:
> I completely agree about your performance and disk-space considerations.
> _but_ Debian has some "standards"...
who are they to set "standards" how software is supposed to work?
everybody has their "standards", one more broken than the other.

> If you will need lua in applications that are not CPU-bound you should
> prefer the .so, and this is because huge systems like Debian have to
> consider maintainibility (read: updates should be easy).
IMHO this is a gross misconception.
>From practice, running a couple of dozen boxes using various
linux distros, *BSDs and others I know many admins which
rather stick with their known problems then introduce new ones.

> I know LUA is a really
> stable software and patches are not so frequent... but I still think that 
> we should prefer the easy-to-maintain way.
Easy to maintain?
Like just redirecting liblua.so.5 from liblua.so.5.0 to liblua.so.5.1?
In accordance with all those "standards" about major and minor and x,
but blowing up just plain everything.

> If you want all the CPU cicles you can have you can link lua statically
> just changing the linker command line arguments, not a big problem. 
for an "apt get" updating any number of binaries
is even less a problem. actually no problem at all.

> Currently on Debian there are few packages depending on liblua50, but I
> hope they will grow:
> 
>   xmoto
> ...
It's great if they are depending on liblua50,
so nothing is going to happen there.
However, any binary somehow dynamically linked to "Lua 5"
will blow up with 5.1, as lua major version numbers do not
imply API compatibility, at least not on link level
(as much is done by macros).
And that's the opposite of maintainability:
if you can not upgrade a library because it's likely to break something.

This versioning policy may be considered "bad style"
by Debian packagers, and actually I consider a
major-version-linkage-upwards-compatibility-policy
an interesting idea, and would not mind the next thing
being labeled Lua 6.
Yet, it's the packager's job to develop an overall
coherent distribution, not the Lua author's task
(considering all those platforms, many without
a concept of major, minor or patchlevel versions).

After all, as far as distributions are concerned,
they can easily re-compile hundreds of binaries for any new
lib version and any admin can just as easily update
hundreds of binaries with a single "apt" or "rpm"
or whatever command, wasting less bandwidth than one
single Firefox update.

OTOH, I'd really hate to see my app blow up just because
of some "update" and so I will always link it statically.
Using dietlibc or similar, it's not only self-contained
and much faster, it's even smaller.



cheers


Reply | Threaded
Open this post in threaded view
|

Re: 5.1 beta problem linking luac against .so

David Given
On Thursday 24 November 2005 23:49, Klaus Ripke wrote:
[...]
> who are they to set "standards" how software is supposed to work?

Um, they're the people who are designing the operating system. It's their job. 
Remember: Linux is not an OS, it's an OS *kernel*. Your build your operating 
system out of parts, and the kernel is just one part, as is Lua. Debian is 
one Linux-based OS. Red Hat is another Linux-based OS that just happens to be 
largely compatible. It's the people who put together the parts whose job it 
is to make sure that all the parts interoperate correctly.

[...]
> Easy to maintain?
> Like just redirecting liblua.so.5 from liblua.so.5.0 to liblua.so.5.1?
> In accordance with all those "standards" about major and minor and x,
> but blowing up just plain everything.

Well, yeah, which is why you don't do that. Because Lua 5.0 is a substantially 
different beast to Lua 5.1, Debian supplies them as different packages. It's 
the package maintainer's job to make sure that no update ever breaks things. 
Debian is incredibly strict about this, and it's one reason why Debian works 
so well.

[...]
> OTOH, I'd really hate to see my app blow up just because
> of some "update" and so I will always link it statically.

...and any Debian package maintainer worth their Debian membership will swear 
at you, because you've just prevented *any* security updates or bugfixes from 
*ever* getting automatically applied to your package! You'll end up having to 
make *your* package maintainer duplicate all the work that the real Lua 
package maintainer has already done, months ago...

I'd strongly recommend reading the relevant bits from the Debian policy 
document (http://www.us.debian.org/doc/debian-policy). They're not idiots, 
and they have rules and best practices in place to make all this work 
*reliably* and, above all, *maintainably*... because in the long run, it's 
better to have something that works now adequately and will continue to work 
adequatly over something that works now well but will break tomorrow.

-- 
+- David Given --McQ-+ 
|  [hidden email]    | "Those that repeat truisms, are also forced to
| ([hidden email]) | repeat them." --- Anonymous from Slashdot
+- www.cowlark.com --+ 

Attachment: pgpmUVrD7muUy.pgp
Description: PGP signature

Reply | Threaded
Open this post in threaded view
|

Re: 5.1 beta problem linking luac against .so

Daniel Silverstone
In reply to this post by Klaus Ripke
On Fri, 2005-11-25 at 00:49 +0100, Klaus Ripke wrote:
> Easy to maintain?
> Like just redirecting liblua.so.5 from liblua.so.5.0 to liblua.so.5.1?
> In accordance with all those "standards" about major and minor and x,
> but blowing up just plain everything.

No dear.

Easy to maintain as in liblua50.so.5.0 and thusly liblua51.so.5.1

There's a reason Debian's SONAME isn't "liblua" but is "liblua50" and
that's because the person who did the original packaging *THOUGHT* about
this problem and solved it.

> Yet, it's the packager's job to develop an overall
> coherent distribution, not the Lua author's task
> (considering all those platforms, many without
> a concept of major, minor or patchlevel versions).

Well, some would argue that it's upstream's job to actually follow the
rules about API/ABI compatibility in major/minor/revision versioning
systems, but Lua makes no claim to follow those rules and thus the
Debian packager's job is made somewhat easy by simply being careful
about the SONAMEs.

> OTOH, I'd really hate to see my app blow up just because
> of some "update" and so I will always link it statically.
> Using dietlibc or similar, it's not only self-contained
> and much faster, it's even smaller.

It's nice to know that you don't trust your distribution maintainers to
look after you. I deliberately don't statically link so that I can take
advantage of security updates.

Regards,

Daniel.
(The original Lua packager for Debian :-)

-- 
Daniel Silverstone                         http://www.digital-scurf.org/
PGP mail accepted and encouraged.            Key Id: 2BC8 4016 2068 7895



Reply | Threaded
Open this post in threaded view
|

Re: 5.1 beta problem linking luac against .so

Robert Raschke-2
In reply to this post by Glenn Maynard
[hidden email] wrote:
> 150k is a lot when it's repeated over a dozen or more binaries--for example,
> a package containing a suite of small tools.  ("It's just 150k" is the
> mindset that leads to the bloat of a lot of applications today--150k here
> and 150k there adds up quickly.)
> 
> More importantly, there's the other major benefit of shared libraries:
> updating them system-wide doesn't require updating every binary that
> uses it.  This could be significant, for example, if a security-related
> bug was fixed.
> 
> I statically link for other reasons (to avoid adding a runtime dependency),
> but "there is no downside" is overstating the case.  (Many people would
> call encouraging static linking to be a "deadly sin".  Caught between hells?)

I can understand the lots of 150K adds up to a lot, but ...  if I'm
running in an environment where disk and memory space is tight, I'll
be doing special things from the outset, if I'm running on any kind of
"normal" OS with disk space and memory in abundance (>20GB and >256MB,
respectively) I don't have a problem (apart from an ethical one).
Even if I have 100 Lua utilities with Lua statically linked, that's
only 15MB.  That's still less than a lot of single applications these
days.  Have you looked at the size of libraries (.so, .dll, .jar,
.lib, .a, whatever) recently?

And while shared objects seem to be great, i.e., "updating them
system-wide doesn't require updating every binary that uses it", in my
experience this leads to "some binaries mysteriously fail".

I'm probably preaching to the converted, so please excuse the fervour.

Robby


Reply | Threaded
Open this post in threaded view
|

Re: 5.1 beta problem linking luac against .so

Enrico Tassi-3
In reply to this post by David Given
On Fri, Nov 25, 2005 at 12:22:43AM +0000, David Given wrote:
> On Thursday 24 November 2005 23:49, Klaus Ripke wrote:
> > who are they to set "standards" how software is supposed to work?
> Um, they're the people who are designing the operating system. It's their job. 
Other people from the Debian world answered for me :)
thnks

There is a huge amount of work behind a linux distro. I believe Debian
is the best one, technically speaking, because it has standards that
are usually respected. And these standards are usually the result of
years spent in putting stuff together.

LUA is a special case since it has not much reverse dependencies in
Debian, but consider an ubiquitous software like Perl. A security bug
in Perl now means downloading few packages and the mountain of software
depending on Perl is then secure. In case Perl was linked statically
all the packages depending on it had to be relinked... 

In Debian we see something similar each C++ ABI changes... and at my
eyes it seems an hell.

If you don't like the way Debian works, you can just use one of the
other distros (that probably do not ship LUA at all).

ciao
-- 
Enrico Tassi

Reply | Threaded
Open this post in threaded view
|

Re: 5.1 beta problem linking luac against .so

Glenn Maynard
On Fri, Nov 25, 2005 at 11:10:44AM +0100, Enrico Tassi wrote:
> LUA is a special case since it has not much reverse dependencies in
> Debian, but consider an ubiquitous software like Perl. A security bug

That doesn't mean Lua is a special case; it just means Lua isn't as popular
yet, and there are hundreds of "not very heavily used libraries" in Debian.
(And I suspect Lua is much more likely to eventually end up widely-used than
many.)

-- 
Glenn Maynard