Re: Making LUA safe to execute untrusted scripts [replay]

classic Classic list List threaded Threaded
38 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Python and Lua

John Belmonte-2
I found that reading the paper "Lua-an extensible extension language" on the
documentation page of the Lua web site gave me good insight into what Lua is
all about.  It has perspective that you can't get from just reading the
manual or staring at code.  Maybe as programmers we tend to just jump to the
manual.  After all how many of us have read an academic paper on the C/C++?
The Lua paper is well written and I think deserves a read by anyone who may
have overlooked it.

I may be mistaken, but I think the source of the "3rd-party" library problem
is that Lua is a lightweight language.  Like Scheme it doesn't come with a
module system, object system, etc. but is expressive enough to implement
those features if you need them.  The problem is that everyone will
implement them differently, so that any code based on such an extension
cannot be easily used by others.

-John Belmonte


----- Original Message -----
From: "Ashley Fryer" <[hidden email]>
To: "Multiple recipients of list" <[hidden email]>
Sent: Thursday, April 20, 2000 8:25 PM
Subject: RE: Python and Lua


> I admit this is an area where I'm jealous of Python!  It has a very nice
> collection of libraries for image manipulation, ftp, www, tkinter, etc.  I
> think many people decide to use Python because of the libraries.
>
> So, why do those libraries exist for Python but not for Lua?
>
> Theories:
>
...


Reply | Threaded
Open this post in threaded view
|

Re: Python and Lua

Maciej Maczynski
In reply to this post by Ashley Fryer-2

>
> > -----Original Message-----
> > From: [hidden email] [[hidden email]
> > Behalf Of Nick Trout
> > Sent: Thursday, April 20, 2000 3:40 AM
>
> > You have more choice with Lua, that is its strength.
> > Python's is that the
> > functionality is there and users concentrate on modules and
> > extension libraries
> > hence further increasing its "usefulness".
>
> I admit this is an area where I'm jealous of Python!  It has a very nice
> collection of libraries for image manipulation, ftp, www, tkinter, etc.  I
> think many people decide to use Python because of the libraries.
>
> So, why do those libraries exist for Python but not for Lua?
>
> Theories:
>
> * Python supports "import" semantics, thus encouraging libraries.
>
> * From its early history, libraries were published for Python.  The early
> contributions encouraged later contributions.
>
> * Marketing... Lua is marketed mainly as an extension language, Python
> mainly as a standalone scripting language, so Python naturally attracts
more
> extension libraries.
>
> * User base.  Python has more users ( I think ), therefore it has more
> libraries.
>
> * Libraries are platform dependent, thus harder to propogate.  This
explains
> why Lua _doesn't_ have more librares, but Python has this problem too.
How
> come Python overcame it but Lua didn't?
>
> It would be nice to overcome this obstacle to broader acceptance of Lua.
>
> Thoughts?
>

I think this is the destiny of embedded language - I've choosen Lua because
it HAS NOT tons of libraries I don't need.
Looking for embedded language you mind its size and extension capabilities.
If it also has nice, clean Pascal-like syntax, as Lua does - that's all I
need.

Maciej


Reply | Threaded
Open this post in threaded view
|

Lua and Dylan

Michael T. Richter-2
In reply to this post by Nick Trout-2
> [ By the way, on a seperate note, and you may know. Is Dylan similar
> to Lua? I think that has language extension features. How does Lua
> compare? ]

Well, my favorite language for development is Dylan and Lua is one of two
favorites (Python is the other) for scripting.  I think I can help you out a
bit here.  :-)

Dylan and Lua are in different domains as languages.  There's some thematic
similarity, but not a lot.

Dylan is an impure, non-lazy functional programming language at its core.
It supplements this functional core with a hygenic macro system that permits
it to emulate other programming styles.  There is a stock set of such macros
in the language specification which permit it to support object-oriented[1]
programming and procedural programming out-of-the-box.  Looping constructs,
for example, are predefined macros.  (Note that these predefined macros tend
to be implemented in the compiler itself, but their semantics and behaviour
can be defined by macros.)

As an example of the power of this approach, consider DUIM, the GUI library
for the Functional Developer implementation of Dylan.  DUIM consists of a
whole bunch of classes and macros.  The macros permit large, complex GUI
applications to be built in a concise declarative (as opposed to functional,
procedural or object-oriented) style.  Anybody who has ever written a
Windows resource (.RC) file will recognize the approach and find it natural.

Dylan, in any of its current incarnations, isn't suitable as a scripting
language, however.  It is oriented towards compilation, not interpretation.
To make Dylan a scripting language would require, in essense, inclusion of
the full compiler into the scripting executable.  Dylan compilers are a bit
large for this, and they are a bit slow when they hit macro expansion
(macros "calls" can expand into macro calls which can expand into macro
calls and so on).

I suspect I don't have to introduce Lua to the members of this list.  :-)

The thematic similarity that Dylan and Lua share is the use of a
meta-mechanism to extend the semantics of the language.  Lua uses tags.
Dylan uses macros.  Dylan's approach is much more flexible, but for that
much more expensive at compilation time.  Lua's approach is simpler, faster
to interpret and occupies a whole lot less space.

I've currently got Lua built as a pair of Win32 DLLs with the C API
functions exposed and am working on moving the API into Dylan's C-FFI (C
Foreign Function Interface) library.  When I'm finished with the process
(currently somewhat stalled), I'll be able to write software in my favorite
development language and to plug in my favorite extension language on the
side.  I'm happy!  :-)


Reply | Threaded
Open this post in threaded view
|

RE: Python and Lua

Martin Dvorak
In reply to this post by Maciej Maczynski
> I think this is the destiny of embedded language - I've choosen
> Lua because
> it HAS NOT tons of libraries I don't need.
> Looking for embedded language you mind its size and extension
> capabilities.
> If it also has nice, clean Pascal-like syntax, as Lua does - that's all I
> need.
>
> Maciej

I agree with Maciej, but with one little remark:

For our project we needed a small and fast embedabble language - Lua seemed
to
be the best choice. The only objections against Lua were small things like
uneasy
implementation of OOP, absence of interger numeric type or problem with
global variables
being the default - that is things rising from the fact that Lua language
was designed
to be simple configuration language. If we've not needed great control over
the
interpreter and the speed of it, Python would be the best choice because the
design
of this language perfectly suits our needs, it's very clean and synoptic,
and it is
much faster to develop larger program in Python than in Lua.

So, in my opinion, it is no doubt that Python is better language than Lua.
But
only when speaking about the language itself, not the whole environment,
which
is very huge and is not ideal for embedding in application as a "helper"
language. We decided to use Lua because it's small, fast and the embedding
application has much more control over it than in case of Python. But if
there was
a language which is as great as Lua in embedding capabalities and as great
as Python in language desing, I would definitely go for it.

Martin

Reply | Threaded
Open this post in threaded view
|

Re: Lua and Dylan

Michael T. Richter-2
In reply to this post by Michael T. Richter-2
> There is a stock set of such macros in the language specification
> which permit it to support object-oriented[1] programming and
> procedural programming out-of-the-box.

Oops.  I forgot to amplify on the [1] above.

[1] Dylan's approach to object-oriented programming will shock and annoy
most C++, Java or Eiffel programmers.  Classes are not bundles of data and
operations, for example.  They just have the data (from the logical
perspective).  Further classes are not the unit of access control.  All
class members are public.  Methods are not dispatched from a single special
argument (the C++ "this" pointer for example) but are rather dispatched by
all arguments.  Because of this methods are not necessarily bound to
specific classes.


Reply | Threaded
Open this post in threaded view
|

Re: Making LUA safe to execute untrusted scripts [replay]

Fred Bertsch
In reply to this post by Luiz Henrique de Figueiredo
Sorry about the long delay replying to this one, but I've already tried to
solve the original problem.  (ie: making Lua safe for untrusted scripts.)

I do it by disabling many (most?) features of the Lua language.  Users are
only allowed to execute scripts which can be proven to take CPU time and
memory proportional to their length.  Thus, function definitions and loops
are disabled.  While this allows only very simple scripts to be written,
it is all my scripts require because they have access to a rich C++
library to manipulate the game world with.  I also disabled the ability to
save state in global variables.  

Oddly enough, I just recently created a webpage with this code (minus the
actual game world manipulation code) on it to give to potential employers.  
You're welcome to look at it:

http://mems.ee.cornell.edu/Fred/Resume/CodeSamples/

In particular, see KBrokenLua.h and Lua.h.  (Those are the C++ wrappers
for the Lua api that I use.)

Note that pretty much all the code there is copyrighted, but if some of
you want to use it, feel free to contact me, and I'll probably be willing
to let you use it.  The concept behind it is pretty simple, so it
shouldn't be hard to duplicate the code, either.  Here's the docs for it:

http://mems.ee.cornell.edu/Fred/Resume/CodeSamples/Conversations.html

Oh, I didn't actually prove that it's secure.  In fact, I think I left in
the ability to sort a table, so I suppose it's really O( N lgN ) or
whatever the sorting algorithm in Lua is, but at least you have to work
pretty hard to bypass the security in any more serious way.  Let me know
if you see any holes.

Finally, I'm still looking for a job...  :)

-Fred


Reply | Threaded
Open this post in threaded view
|

Re: Python and Lua

Nick Trout-2
In reply to this post by John Belmonte-2
Thanks to Michael for the Dylan review. Think I'll steer clear of Dylan for the
time being! Too many macros scare me, I like to see nice explicit code in front
of me. I suppose its for the same reason I havent got round to studying the C++
STL yet. How do you debug templates?!


> I found that reading the paper "Lua-an extensible extension language" on the
> documentation page of the Lua web site gave me good insight into what Lua is
> all about.  It has perspective that you can't get from just reading the
> manual or staring at code.

I agree! Very insightful. 5 good bullet points in the intro.

> I may be mistaken, but I think the source of the "3rd-party" library problem
> is that Lua is a lightweight language.  Like Scheme it doesn't come with a
> module system, object system, etc. but is expressive enough to implement
> those features if you need them.  The problem is that everyone will
> implement them differently, so that any code based on such an extension
> cannot be easily used by others.

Ahhh but that is exactly what I am interested in seeing!! I would like to see
lots of implementations of peoples ideas, applicable or not! A Life program is
completely useless to most people but it shows a coding style and ideas. That's
what make Open Source such a great thing. You have the ability browse and choose
the most suitable bit of code to butcher. Hope everybody's been to
www.sourceforge.net (Home of 10,000 incomplete RPG's  ;-D )

> > I admit this is an area where I'm jealous of Python!  It has a very nice
> > collection of libraries for image manipulation, ftp, www, tkinter, etc.  I
> > think many people decide to use Python because of the libraries.
> > So, why do those libraries exist for Python but not for Lua?

I'm not sure they need to exist. I agree with the "Lua is an embedded language
and should stay that way" train of thought. The whole idea is that you just
expose parts for the hosts libraries to allow customisation of the host
application. And if you need libraries tolua is great. (There's a SWIG binder as
well now bit havent tried that.). If I wanted to script something using all
those libraries I would just use Python, why reinvent the wheel?

And why has the Python user base grown so rapidly? Because its easy, they have a
very friendly, generous and collaborative development community and there's
absolutely loads of Python scripts all over the web. But, Python has different
goals its scripting language etc.

Right I'm off for Easter now, going to Silverstone for the British Grand Prix,
wahay!
Cheers,
Nick.



Reply | Threaded
Open this post in threaded view
|

Re: Python and Lua

Michael T. Richter-2
> Thanks to Michael for the Dylan review. Think I'll steer clear of Dylan
> for the time being! Too many macros scare me, I like to see nice explicit
> code in front of me. I suppose its for the same reason I havent got
> round to studying the C++ STL yet. How do you debug templates?!

Dylan macros are *NOTHING* *AT* *ALL* like the evil C/C++ macros.  They're
more like what are called macros in Common Lisp (from whence Dylan inherits
most of what it is outside of syntax) or the later versions of Scheme.
Think of them as special forms of subroutines with the ability to add new
keywords to the language instead.

Really, in actual practice, Dylan macros are no different from Lua tag
methods.  They allow you to change the semantics of the language to be more
natural for specific purposes.  Poorly used they can be brain-damaging (in
both Lua tags and Dylan macros).  Used judiciously they can be wondrous
(witness DUIM as a perfect example).

I've been hobby-programming Dylan for just under a year now.  I've never
actually written my own macros (except as learning experiences).  I've
*used* dozens of them.  Many times I'm not even aware I'm using them,
they're that native.

> Ahhh but that [libraries and modules] is exactly what I am interested
> in seeing!!

I must confess that this is part of Lua I find weakest as well.  Still, Lua
is a relatively young language that's only just now getting attention.
Python and Perl-<spit> have been around for a long time and have as a result
accumulated a lot of good tools.


Reply | Threaded
Open this post in threaded view
|

Re: Python and Lua

Russell Y. Webb
Where can those of us who would like to learn more about the macros of 
lisp and Dylan look for a simple explaination?

Russ

Reply | Threaded
Open this post in threaded view
|

Re: Python and Lua

John Belmonte-2
Russ wrote:

> Where can those of us who would like to learn more about the macros of
> lisp and Dylan look for a simple explaination?

The macro system of Scheme, which is a very clean dialect of Lisp, is
probably a good one to study.  Following is a link to the macro chapter of
"An Introduction to Scheme and its Implementation".  Possibly it isn't very
easy to undestand.  But the nice thing is that it begins by considering a
naive implementation (such as in Lisp) and works from there.  No doubt that
Scheme is a tough vitamin to swallow... but it will make you strong.  (I
still have a long way to go.)

    http://www.cs.utexas.edu/users/wilson/schintro/schintro_128.html#SEC180


The macro system in Dylan shares Scheme's important trait of being hygienic,
which means you never have to worry about name clashes.  All I found was a
brief example:

    http://www.gwydiondylan.org/fragments/macros.phtml

To me Dylan is basically a rendering of Scheme using infix ("C") syntax,
adding a great module and OO system, and the ability to do static typing and
match C's performance when necessary.  Unfortunately the most significant
implementation is commercial and windows-only.

-John


Reply | Threaded
Open this post in threaded view
|

Re: Making LUA safe to execute untrusted scripts [replay]

David Jeske-3
In reply to this post by Nick Trout-2
On Thu, Apr 20, 2000 at 07:38:24AM -0300, Nick Trout wrote:
> This is what I find nice about Python you dont have to go round the
> houses all the time to find solutions to programming styles ie. in
> Lua it seems you're constantly trying to emulate the behaviour of
> other languages because a feature you would like is lacking! Now you
> can see this as both a pain and great feature

I'd just like to quickly use your own words to prove the point of a
previous poster. 

All this stuff you're talking about in Python which you like is
definetly added complexity.

While I might agree with the statment that the Python language
distribution is more immediately useful for software development,
Python is definetly more complex than Lua.

My (very short) list of languages simpler than Lua is: SELF,
Scheme/LISP, Forth.

However, if you rank them by real-world usefulness, Lua ends up on
top. :)

-- 
David Jeske (N9LCA) + http://www.chat.net/~jeske/ + [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Python and Lua

Ashley Fryer-2
In reply to this post by Nick Trout-2
> -----Original Message-----
> From: [hidden email] [[hidden email]
> Behalf Of Nick Trout
> Sent: Thursday, April 20, 2000 9:31 AM

>
> And why has the Python user base grown so rapidly? Because its
> easy, they have a
> very friendly, generous and collaborative development community
> and there's
> absolutely loads of Python scripts all over the web. But, Python
> has different
> goals its scripting language etc.
>

I think you've identified the main issue here.  Lua is marketed as an
extension language, and thus it has attracted a different community than
Python.  We, myself included, would prefer to hide in our caves and develop
private, proprietary extensions rather than shared, public extensions.
Thus, Python will always have a wider range of 3rd party support, and a
consequently larger user base.

ashley


Reply | Threaded
Open this post in threaded view
|

Re: Making LUA safe to execute untrusted scripts [replay]

Renato Cerqueira
In reply to this post by Nick Trout-2
At 07:38 20/04/00 -0300, Nick Trout wrote:
What is the difference between a prototype based OO system and a class based
one? I thought a prototyped systems were just part of the design phase of OO?!

Hi Nick,

Besides the prototyping phase in software development processes,
there are prototype-based languages.
Basically, those languages are OO languages that use delegation,
instead of classes, to share behavior among objects.

You can find more information about this topic in the following references:

Author = "Lynn Andrea Stein and Henry Lieberman and David Ungar",
Title = "A Shared View of Sharing: The Treaty of Orlando",
BookTitle = "Object Oriented Concepts, Applications, and Databases",
Editor = "Won Kim and Fred Lochovsky",
Publisher = "ACM Press",
Year = 1988

Author = "Christophe Dony and Jacques Malenfant and Pierre Cointe",
Title = "Prototype-Based Languages: From a New Taxonomy to Constructive Proposals and Their Validation",
BookTitle= "{OOPSLA}'92 Proceedings",
Year = 1992,
Pages = "201--217"

Regards,
Renato

Reply | Threaded
Open this post in threaded view
|

RE: Python and Lua

Renato Cerqueira
In reply to this post by Martin Dvorak
At 11:17 20/04/00 -0300, Martin Dvorak wrote:
The only objections against Lua were small things like
uneasy
implementation of OOP, absence of interger numeric type or problem with
global variables
being the default - that is things rising from the fact that Lua language
was designed
to be simple configuration language. If we've not needed great control over
the
interpreter and the speed of it, Python would be the best choice because the
design
of this language perfectly suits our needs, it's very clean and synoptic,
and it is
much faster to develop larger program in Python than in Lua.
So, in my opinion, it is no doubt that Python is better language than Lua.

Hi Martin,

>From this point of view, Java is a "better language" than Lua and Python.
Just to emphasize some previous messages,
each of those languages is a "better language" for some specific job,
and they are designed with different goals.

I think the following paper could give a good insight into this subject:
  author = "John Ousterhout",
  title = "Scripting: Higher Level Programming for the 21st Century",
  journal = "IEEE Computer",
  year = "1998",
  month = mar,
  volume = "31",
  number = "3",
  pages = "23-30"

BTW, what do you suggest instead of "global variables being the default"?
I think Roberto gave a good explanation about this topic some months ago
in the mailing list:
http://www.egroups.com/message/lua-l/961

Regards,
Renato

Reply | Threaded
Open this post in threaded view
|

RE: Python and Lua

Martin Dvorak
>  From this point of view, Java is a "better language" than Lua and Python.
> Just to emphasize some previous messages,
> each of those languages is a "better language" for some specific job,
> and they are designed with different goals.

I agree. I just wanted to point out that when selecting the language to
embed, it's sometimes necessary to make compromise. That is
to realize what's more important to have (in my case): suitable language
design or suitable embedding capabalities. I care more for the second
and that is why I've chosen Lua. That does not mean that Lua perfectly
suits my needs. But from all languages I've evaluated, Lua is closest
to my requirements. So I rather use Lua's tag methods to tailor
its language (even though I loose a bit of performance).
If I tried to downsize and modify Python, I would never
make it so fast and excellent for embedding as Lua is.

 
> BTW, what do you suggest instead of "global variables being the default"?
> I think Roberto gave a good explanation about this topic some months ago
> in the mailing list:
> http://www.egroups.com/message/lua-l/961

Yes, he does. But I'd rather have to use tag methods to implement the
opposite (I still can't get rid of a feeling that more tag methods
means less performance).

Martin

Reply | Threaded
Open this post in threaded view
|

RE: Python and Lua

Luiz Henrique de Figueiredo
In reply to this post by Edgar Toernig
>From: "Martin Dvorak" <[hidden email]>
>
>I still can't get rid of a feeling that more tag methods
>means less performance

The number of tag methods does not affect performance.
The use of tag methods does, but only slightly (essentially one more C call
and one more Lua call, plus of course whatever you do in the tag method itself).

As I have said before in this list, I sympathize with the concerns about
performance, but it's notorious that we programmers are very bad at detecting
performance bottlenecks. This is even more true for interpreted languages.
So, the only way to know is to measure the time taken by your programs.
In any case, Lua is one of the fastest languages around, and 4.0 is even faster
than 3.2.
--lhf

Reply | Threaded
Open this post in threaded view
|

RE: Python and Lua

Luiz Henrique de Figueiredo
In reply to this post by Edgar Toernig
[I'm reposting this message of mine because it does not seem to have been
 distributed, even though it made into the archive! --lhf]

>From: "Martin Dvorak" <[hidden email]>
>
>I still can't get rid of a feeling that more tag methods
>means less performance

The number of tag methods does not affect performance.
The use of tag methods does, but only slightly (essentially one more C call
and one more Lua call, plus of course whatever you do in the tag method itself).

As I have said before in this list, I sympathize with the concerns about
performance, but it's notorious that we programmers are very bad at detecting
performance bottlenecks. This is even more true for interpreted languages.
So, the only way to know is to measure the time taken by your programs.
In any case, Lua is one of the fastest languages around, and 4.0 is even faster
than 3.2.
--lhf

Reply | Threaded
Open this post in threaded view
|

RE: Python and Lua

Luiz Carlos de Castro Silveira Filho

-----Original Message-----
From: [hidden email] [[hidden email]
Behalf Of Luiz Henrique de Figueiredo
Sent: Tuesday, April 25, 2000 11:38 AM
To: Multiple recipients of list
Subject: RE: Python and Lua


[I'm reposting this message of mine because it does not seem to have been
 distributed, even though it made into the archive! --lhf]

>From: "Martin Dvorak" <[hidden email]>
>
>I still can't get rid of a feeling that more tag methods
>means less performance

The number of tag methods does not affect performance.
The use of tag methods does, but only slightly (essentially one more C call
and one more Lua call, plus of course whatever you do in the tag method itself).

As I have said before in this list, I sympathize with the concerns about
performance, but it's notorious that we programmers are very bad at detecting
performance bottlenecks. This is even more true for interpreted languages.
So, the only way to know is to measure the time taken by your programs.
In any case, Lua is one of the fastest languages around, and 4.0 is even faster
than 3.2.
--lhf


I´m working on a profiler (to measure the execution time of each lua function). There is almost an alpha version of it, as soon as I
fix some implementation problems - which prevents you from analyzing the profiler log file of a very complex execution - and give a
better interface to the analyzer output - I´ll need to learn some javascript for that, since i´m creating an html file to show all
the information. Basically the profiler has two parts: one to create a file with the local time and the total time (time spent in
the function code and time spent among all subcalls it produces) of *each called function* (if a function is called 20 times, 20
lines will be created, no calls, no lines) which you can analyze in your favourite spreadsheet; the other part is the built-in
analyzer, which allows you to trace the execution of your program and choose which function you would like to see the times for,
giving you some statistics - this is useful if the times for a function depends mostly on their parameters, for example, a function
to make a query to a database (your bottleneck may not be your function, but their parameters, if the query is inefficient).
I don´t know yet if it compensates to have a log file that big (each line is about 60 chars), or I should elaborate more and make
something like gprof (the GNU profiler) which generates a log of about 500 times shorter but only give you average times. But I
guess I´m planning some "compensating" stuff for the analyzer ;)

thank you,
Luiz


12