Unicode

classic Classic list List threaded Threaded
52 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Unicode

Jakab
Strings in Lua are 8-bit. Has anyone tried using Unicode with Lua? What were
your results, and do you have source code (if necessary) available if they
were successful?

- Steve Jakab
Verant

Reply | Threaded
Open this post in threaded view
|

Re: Unicode

Reuben Thomas-3
> Strings in Lua are 8-bit. Has anyone tried using Unicode with Lua? What were
> your results, and do you have source code (if necessary) available if they
> were successful?

This reminds me that I meant to ask about Unicode too, and to suggest that
having a user-settable character type (much in the same way that the
number type can be redefined) should be a goal for Lua in the near future
(I don't think that the use of char is parametrised yet in the source
code, nor the functions which are used to act on strings).

A related topic that should also be addressed is that of
internationalisation: this should be quite straightforward, because the
number and range of messages output by the Lua system is relatively small.

-- 
http://sc3d.org/rrt/ | maxim, n.  wisdom for fools


Reply | Threaded
Open this post in threaded view
|

Windows binaries

Reuben Thomas-3
It's been quite a while since the release, and still no official Windows
binaries. What is the difficulty in producing them? It's a pity that I
can't just point people to something they can download for Windows when I
recommend Lua (as most of the world (sadly) use Windows). Compiling from
source is not most non-Unix users' idea of "just trying something out"...

Fortunately all my favourite platforms already have downloadable binaries
:-)

-- 
http://sc3d.org/rrt/ | maxim, n.  wisdom for fools


Reply | Threaded
Open this post in threaded view
|

Re: Unicode

Guy English
In reply to this post by Jakab
Hi,

On Monday, January 29, 2001, at 03:23 PM, Reuben Thomas wrote:
> > Strings in Lua are 8-bit. Has anyone tried using Unicode with Lua? What were 
> > your results, and do you have source code (if necessary) available if they 
> > were successful? 
> This reminds me that I meant to ask about Unicode too, and to suggest that 
> having a user-settable character type (much in the same way that the 
> number type can be redefined) should be a goal for Lua in the near future 
> (I don't think that the use of char is parametrised yet in the source 
> code, nor the functions which are used to act on strings). 

Proper UNICODE support would have to go beyond wchar_t substitution. Recognizing compound characters ( like O and umlaut in sequence ) and upper/lower case issues ( in some scripts there is no clear mapping ) would appear to be a non-trivial modification to Lua. On our game we basically use wide characters as 16bit index into a glyph array and I assume that's what Verant wants to do too. I agree that an easy way to set the character size is a good idea but full UNICODE support is a mine field.

Take care,
Guy English

ps. Built in co-routines would make me really, really happy.



--------------
Guy English
[hidden email]

Project Leader/Lead Programmer
StrategyFirst, Inc.

Reply | Threaded
Open this post in threaded view
|

co-routines [was Re: Unicode]

Roberto Ierusalimschy
> ps. Built in co-routines would make me really, really happy.

Lua 4.1 will have full support for co-routines (and multi-threads) 
implemented *outside* Lua. That means multiple stacks sharing the same 
global environment plus macros to allow lock/unlock accesses to the state. 
On most machines, some tricks with setjmp/longjmp or a few lines of 
assembler, in a library outside Lua, will be enough to implement 
co-routines. 

But Lua 4.1 will not be "stackless"; that would compromise its neat 
integration with C. 

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: co-routines [was Re: Unicode]

Reuben Thomas-3
> Lua 4.1 will have full support for co-routines (and multi-threads)
> implemented *outside* Lua.

Some sample implementations for the commonest platforms would really help.

-- 
http://sc3d.org/rrt/ | humour, n.  unexpected recognition


Reply | Threaded
Open this post in threaded view
|

Re: co-routines [was Re: Unicode]

Reuben Thomas-3
In reply to this post by Roberto Ierusalimschy
> But Lua 4.1 will not be "stackless"; that would compromise its neat
> integration with C.

Will it be internally stackless? Presumably the hard bit is making it stay
stackless when it calls C; not doing that would be forgiveable, and in the
common case (where a C program starts a Lua thread, which calls C routines,
but they do not themselves call back to Lua) the Lua part could still be
nicely stackless (with only one Lua "stack").

-- 
http://sc3d.org/rrt/
L'art des vers est de transformer en beautés les faiblesses (Aragon)


Reply | Threaded
Open this post in threaded view
|

Re: co-routines [was Re: Unicode]

Luiz Henrique de Figueiredo
In reply to this post by Roberto Ierusalimschy
>> Lua 4.1 will have full support for co-routines (and multi-threads)
>> implemented *outside* Lua.
>
>Some sample implementations for the commonest platforms would really help.

Yes, we plan to distribute (in lua/etc/) at least an implementation based on
pthreads, and possibily an implementation based on coroutines, either for a
setjmp hack that I wrote or for Edgar Toernig's package available at
http://themen01.exit.de/user/member/froese/ .
--lhf

Reply | Threaded
Open this post in threaded view
|

Re: co-routines [was Re: Unicode]

Roberto Ierusalimschy
In reply to this post by Reuben Thomas-3
> > But Lua 4.1 will not be "stackless"; that would compromise its neat
> > integration with C.
> 
> Will it be internally stackless?

No.


> Presumably the hard bit is making it stay stackless when it calls C;

The problem is that there are lots of places where Lua can make a 
call to C. Any tagmethod can be a C function, and so most basic operations 
may call C. Currently we handle calls to C and to Lua in the same way. 
To change that would break a nice "simetry" in the code, and would need
lots of changes in the code. (It could also compromise performance...)

We had three goals in mind: restricted co-rotines (only Lua), generic 
co-routines, and multi-thread. Lua stackless would demand lots of work, and 
would solve only the first goal; it would be useless for the other two. 
Multiple stacks solves the other two goals (with a small external help), it 
is quite simple to implement, and has no impact in performance. 

-- Roberto


Reply | Threaded
Open this post in threaded view
|

Re: co-routines [was Re: Unicode]

Jean-Claude Wippler
Roberto Ierusalimschy <[hidden email]> wrote:

[co-routines and stackless]
>We had three goals in mind: restricted co-rotines (only Lua), generic 
>co-routines, and multi-thread. Lua stackless would demand lots of work, and 
>would solve only the first goal; it would be useless for the other two. 
>Multiple stacks solves the other two goals (with a small external help), it 
>is quite simple to implement, and has no impact in performance. 

Please forgive me for addressing you directly, and for a somewhat off-
topic response.

I have been going through all the archived Lua messages and all html
documents, in an attempt to get up to speed with Lua as quick and in
depth as possible (my background is Tcl/Python/C++/Forth).  Having been
involved in open source for some time now (much of my own work is public
domain nowadays), I would like to contribute to Lua if possible, to help
it evolve in issues which matter to me.

After yesterday's "Son of Lua" post on lua-l, I am very concerned about
the direction and future of Lua.  As a smaller language, it seems that
the risk of people taking it in different directions is more acute than
with larger bodies of code.  This would be very unfortunate, since it
would prevent consolidation and growth of a common core and user base. 
Edgar Toernig's work is important IMO, because some changes appear to be
very valuable (though a source code reformatting I could have done without).

Before starting some serious work based on Lua, if have a few questions:
 - is there a page describing the short/mid-term plans with Lua?
 - do you accept patches with feature changes (if deemed useful)?
 - is lua-l a suitable place to start feature/language discussions?

Let me close off by describing my situation: I'm from the Netherlands,
but currently in Canada for a sabbatical year, to work on some new ideas
(self-employed, not in an academic setting).  I wrote a database/
persistence library called MetaKit, with bindings for Tcl and Python (and
C++).  This library is used in a growing number of projects, but the
deployment issues of C and especially C++ are stifling.  My interest in
Lua is as a "scripting in scripting" language: I have started turning Lua
into an extension for Tcl and plan to make bindings for Python and
several other contexts (e.g. Perl, Ruby, Java, Squeak, COM).  This gives
me the ability to write *my* code in Lua (+ C where needed), and deploy
the results in a range of languages without compile/build problems.  In
the past I have considered Forth for this role (as explored in my
Minotaur project), but this is somewhat low level.  My most recent
experiment is a vectorized/type-safe variant of Forth called Thrive, but
now it looks like much work could be avoided by adopting Lua instead.

The projects mentioned above are listed at http://www.equi4.com/jericho/

Again, my apologies for taking your time in this way.  Thank you and your
team for creating such an innovative language, with all the "right"
trade-offs.

Regards,
Jean-Claude Wippler

Reply | Threaded
Open this post in threaded view
|

Re: co-routines [was Re: Unicode] - OOPS!

Jean-Claude Wippler
Jean-Claude Wippler <[hidden email]> wrote:

.. something which should have been sent in private ...

Deeply embarrassed,
Jean-Claude

Reply | Threaded
Open this post in threaded view
|

Re: co-routines [was Re: Unicode]

Roberto Ierusalimschy
In reply to this post by Jean-Claude Wippler
> - is there a page describing the short/mid-term plans with Lua?

No. Currently, our short-term plans are only better support for co-routines 
and multi-threading, and "names for tags". Both involve small changes in 
the code, and are completely downward compatible with Lua 4.0.

Our most important mid-term plan is "consolidation and growth of a common 
core and user base" ;-) That means adding libraries and tools to Lua, 
instead of changing the language. 

I, personally, have a mid-term plan of finnishing the book about Lua;
for that I also need a stable language ;-)


> - do you accept patches with feature changes (if deemed useful)?

Yes (if deemed useful).

> - is lua-l a suitable place to start feature/language discussions?

Yes. But keep in mind that, despite the big change in its API for version 
4.0, Lua has been a quite stable language, and we plan to be more and
more "conservative". 

We really prefer when people (and ourselves ;-) direct their efforts to 
improving Lua, without changing it. Examples include porting it to
"difficult" platforms (such as EPOC and PalmOS), libraries, tools
(such as ToLua), etc. There is no shortage of things to be done.

One of the main goals of Lua is to be "an extensible extention language"; 
someone described it as a "kit for building domain specific languages". 
So, another interesting area for Lua is changing the language for
specific tasks without changing its implementation. A good example of
that is LuaOrb, that transforms Lua into a scripting language for CORBA.
It uses a "stock" version of Lua 3.2, modified through its "official"
ways (tag methods).

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: co-routines [was Re: Unicode]

Edgar Toernig
In reply to this post by Jean-Claude Wippler
Jean-Claude Wippler wrote:
> 
> After yesterday's "Son of Lua" post on lua-l, I am very concerned about
> the direction and future of Lua.

Don't bother about that.  My post are ignored most of the time.
And the Lua developers are very resistant to new ideas.  ;-)

> Having been
> involved in open source for some time now (much of my own work is public
> domain nowadays), I would like to contribute to Lua if possible, to help
> it evolve in issues which matter to me.

Lua is open source but closed development in its best.  The good point
is, they do a pretty good job at it.

> As a smaller language, it seems that
> the risk of people taking it in different directions is more acute than
> with larger bodies of code.  This would be very unfortunate, since it
> would prevent consolidation and growth of a common core and user base.

Sol is meant as an experimental platform.  I mainly made it to show
how some ideas could be implemented and the effects of them.  And I
hope that some of the ideas will flow back to Lua.  (If not, I at least
have a neat little language to play with *g*)

> Before starting some serious work based on Lua, if have a few questions:
>  - is there a page describing the short/mid-term plans with Lua?

Hmm... make it faster and stay compatible. ;)

>  - do you accept patches with feature changes (if deemed useful)?

Of course you can sent patches (I would like to see more of them).
But don't expect any response.  Wait a year and you'll see if it
has made into a new version.

>  - is lua-l a suitable place to start feature/language discussions?

Discussions are very difficult if you get no response...  I guess that
most subscribers are programmers but there have been not very much
discussions about new ideas and their implementation.  Pretty strange.

Ciao, ET.



Reply | Threaded
Open this post in threaded view
|

Re: co-routines [was Re: Unicode]

Edgar Toernig
In reply to this post by Roberto Ierusalimschy
Roberto Ierusalimschy wrote:
> 
> No. Currently, our short-term plans are only better support for co-routines
> and multi-threading, and "names for tags".

Could you be a little bit more specific?  Just some functions to save/set
L->stack & co and a registration for lock-callbacks on table-set/get?
Or something more sophisticated?  What about the GC?
 
> Our most important mid-term plan is "consolidation and growth of a common
> core and user base" ;-) That means adding libraries and tools to Lua,
> instead of changing the language.

Hmm...  IMHO the lack of libs and tools is that there's no sane
module concept.  Look at the hacks luasocket has to do to get its
own namespaces.  Compare this the clean way in Sol's siolib.c.

I think, you'ld have it much easier if you would make the "user base"
take part in the development of Lua.  At the moment it looks like
a company.  Every your you get a new version.  Take it or die.
What happens between releases is totally closed.  Why don't you discuss
supposed changes?  For example, the 4.0 API is much better than that
of 3.2.  But some parts are still really strange.  I guess some talk on
the list would have given a better API.  The same thing happens now
with coroutines.  I bet you'll get something together.  But then 4.1
is out and it takes another year to get the misconceptions cleared.

That's the reason I made Sol.  I want to try something out and want
to get some feedback.  This feedback could be useful for the Lua
developers too.  Maybe someone says "that's bullshit".  Ok, he may
be right.  But when he says "that's great" or "make it this or that
and it would be even better" you get a much better feeling of what
people want or need.  I tried this a couple of times with smaller
patches but got nearly null feedback.  So I said, ok, make an own
derivate that people can really use and hear what they think about
it.  The yesterdays snapshot is the first step.

Ciao, ET.


PS to the Lua team: Don't take these harsh words to personally.
Lua is a fine language.  But sometimes I simply get frustrated
by its development style :-)


Reply | Threaded
Open this post in threaded view
|

language experimentation [was: co-routines]

Tom Wrensch
In reply to this post by Edgar Toernig
> Jean-Claude Wippler wrote:
> 
> >  - is lua-l a suitable place to start feature/language discussions?
> 
> Discussions are very difficult if you get no response...  I guess that
> most subscribers are programmers but there have been not very much
> discussions about new ideas and their implementation.  Pretty strange.

We just don't have much imagination, and read in slack-jawed awe as others
discuss that which we cannot understand  ;-). 

Actually, I'd love to do some of the same kind of experimentation you're
doing. Unfortunately I can't even find time to look at what you did.

  - Tom


Reply | Threaded
Open this post in threaded view
|

Re: language experimentation [was: co-routines]

Nick Trout-2
> Jean-Claude Wippler wrote:
> >  - is lua-l a suitable place to start feature/language discussions?
> Discussions are very difficult if you get no response...  I guess that
> most subscribers are programmers but there have been not very much
> discussions about new ideas and their implementation.  Pretty strange.
>>We just don't have much imagination, and read in slack-jawed awe as others
discuss that which we cannot understand  ;-).
Actually, I'd love to do some of the same kind of experimentation you're
doing. Unfortunately I can't even find time to look at what you did.
  - Tom

I'd just like to second this :-) I see Lua as a fantastic time saver, i.e. a
great, practical, fast, embeddable, usable language which saves me writing
one. I'd love to contribute more but just dont have the time. I think for a
lot of users (including me!) it takes all their time to understand the
language as it is and use it in the way intended (thank God for toLua -
painless move from 3 to 4). I feel like I am only scratching the surface
with it.

Personally, I think the discussion forum would be improved by moving it to a
newsgroup (look at comp.lang.python as an example - you have a community
there). Different threads could be discussed more coherently. I know this
has been mentioned before, maybe there are enough people now? - Or maybe I
just need to get a decent mail browser that can sort my mail - anyone at MS
listening :-)


[snip]
"Russell Y. Webb" wrote:
>
> b = 10
> c = 1
> function test(x)
> global c;
>     b = "this does not change the global";
>     c = "but this does";
>     print(10); -- this uses the read access to globals
> end
>
>> ET: I thought about this myself.  But it complicates the parser a lot.
(Especially: a,b,c,d=foo4()  where a and d are new locals, b an old
local and c a global).  And you may get some strange semantics.
Are your patches anywhere to have a look at?

I for one am prepared to accept that you must use the local keyword in Lua
to stop complicated and diverging code bases. I also use Python and it works
as above - that's just the way it is.

[snip]
> As a smaller language, it seems that
> the risk of people taking it in different directions is more acute than
> with larger bodies of code.  This would be very unfortunate, since it
> would prevent consolidation and growth of a common core and user base.
>> ET: Sol is meant as an experimental platform.  I mainly made it to show
how some ideas could be implemented and the effects of them.  And I
hope that some of the ideas will flow back to Lua.  (If not, I at least
have a neat little language to play with *g*)

I think this is a good idea. You can be far more experimental with this
prototype version. You can also develop it in the open style that you would
like. Support for threads could get in a bit of a state unless work examples
are tinkered with before being implemented in proper in Lua. Python has
various versions being experimented with in parallel (eg. stackless) and
these changes could well make it into the core development at some point. I
think its worth saying that Lua probably wouldnt be the language that it is
if the developers hadnt been so focussed and a bit stubborn.

N






Reply | Threaded
Open this post in threaded view
|

Re: language experimentation [was: co-routines]

Luiz Henrique de Figueiredo
In reply to this post by Tom Wrensch
>Personally, I think the discussion forum would be improved by moving it to a
>newsgroup (look at comp.lang.python as an example - you have a community
>there). Different threads could be discussed more coherently. I know this
>has been mentioned before, maybe there are enough people now?

There are currently 292 people in lua-l plus 114 at eGroups (or shold I say
Yahoo now?). That's more than 400 people. (And yet this does not really reflect
on our list of known uses of Lua! Please, everyone, add your project!)

Last time we discussed about this, the general feeling was that it's a lot of
work to create a newsgroup in comp.* plus many people prefered a mailing list
instead of a newsgroup.

>- Or maybe I
>just need to get a decent mail browser that can sort my mail

Don't Netscape Mail readeer or Eudora sort mail into threads?

>I think its worth saying that Lua probably wouldnt be the language that it is
>if the developers hadnt been so focussed and a bit stubborn.

I'm glad I didn't have to say that myself :-)

About our plans for Lua and how we value input from the community, see my
message [hidden email] in the lua-l archives.
This message is almost an year old, but its main points are still valid.
--lhf

Reply | Threaded
Open this post in threaded view
|

Re: co-routines [was Re: Unicode]

Roberto Ierusalimschy
In reply to this post by Edgar Toernig
> Could you be a little bit more specific?  Just some functions to save/set
> L->stack & co and a registration for lock-callbacks on table-set/get?
> Or something more sophisticated?  What about the GC?

We broke the lua_State in two structs; the lua_State itself has the 
thread-specific data, and it points to the other structure with the 
"global" data. There is only one new function (let us call it lua_newstack, 
we don't know the final name): the call lua_newstack(L, stacksize) returns 
a new lua_State sharing the global state with L. Actually, lua_open is now 
a macro lua_newstack(NULL, stacksize).

All stacks sharing the same global part are in a linked list, so the GC
traverse them all.

lua_close destroys a stack. If it is the last stack of the global state,
it also closes the global state.

About the locks: we considered the whole core of Lua as one single critical 
region (it is too complicated and too expensive to lock every single 
internal operation that can have impact in other threads). So all API 
functions have a pair of macros LUA_LOCK/LUA_UNLOCK around them. On the 
other side, whenever Lua calls C, in unlocks its core. So, unless you are 
in a closed loop completely inside Lua, you have plenty of opportunites to 
switch treads. (Notice that the core never blocks; all blocking functions 
are C functions outside). If you really want it badly, you can register an 
empty line-hook (because line-hooks are outside the core, everytime Lua 
calls the line-hook it will unlock and lock again). 

The following code shows the critical parts for implementing multi-threads
(with pthreads) in Lua 4.1. An implementation with co-routines could
follow the same structure, but without the macros for lock/unlock.


file p.h (included in lua.h)

  #include <pthread.h>

  #define LUA_LOCK	pthread_mutex_lock(&mutex)
  #define LUA_UNLOCK	pthread_mutex_unlock(&mutex)

  /* mutex for the Lua core */
  extern pthread_mutex_t mutex;

  ...


file p.c

  #include <pthread.h>

  #include "../lua.h"
  #include "../lauxlib.h"

  pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;

  static void *init_thread (void *arg) {
    lua_State *L = (lua_State *)arg;
    int n = lua_getn(L, 1);  /* number of arguments */
    int i;
    for (i=1; i<=n; i++)  /* push arguments */
      lua_rawgeti(L, 1, i);
    lua_remove(L, 1);  /* table is garbage now */
    pthread_detach(pthread_self());
    lua_call(L, n, 0);
    return NULL;
  }


  static int newthread (lua_State *L) {
    lua_State *NL;
    int ref;
    pthread_t thread;
    luaL_checktype(L, 1, LUA_TFUNCTION);
    luaL_checktype(L, 2, LUA_TTABLE);
    NL = lua_newstack(L, luaL_opt_int(L, 3, 0));
    if (NL == NULL)
      lua_error(L, "cannot create new stack");
    lua_settop(L, 2);  /* remove eventual stacksize */
    /* move table and function from L to NL */
    ref = lua_ref(L, 1);  /* maybe we need a function for that? */
    lua_getref(NL, ref);
    lua_unref(L, ref);  
    ref = lua_ref(L, 1);
    lua_getref(NL, ref);
    lua_unref(L, ref);  
    if (pthread_create(&thread, NULL, init_thread, NL) != 0)
      lua_error(L, "cannot create new thread");
    return 0;
  }

  ...

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: language experimentation [was: co-routines]

Nick Trout-2
In reply to this post by Luiz Henrique de Figueiredo
>> LHF: There are currently 292 people in lua-l plus 114 at eGroups (or
shold I say
Yahoo now?). That's more than 400 people. (And yet this does not really
reflect
on our list of known uses of Lua! Please, everyone, add your project!)
Last time we discussed about this, the general feeling was that it's a lot
of
work to create a newsgroup in comp.* plus many people prefered a mailing
list
instead of a newsgroup.

I think newsgroups are not quite as intimidating as mailing lists and you'll
probably get more traffic generated. You would probably get a broader range
of users creating input asking more varied questions? This would be be good
all round?

>- Or maybe I
>just need to get a decent mail browser that can sort my mail
>> Don't Netscape Mail readeer or Eudora sort mail into threads?

Will have a look, thanks - I use Outlook Express which is good at newsgroups
and most things but cant sort mail headers in threads :-(

Can these sort threads properly, like newsgroups? If you get a hierarchy of
messages how would these be sorted from the headers as the body is at
liberty to change? You could group headers but you wouldn't have proper
threads - which helps develop a discussion fully.

>I think its worth saying that Lua probably wouldnt be the language that it
is
>if the developers hadnt been so focussed and a bit stubborn.
>> I'm glad I didn't have to say that myself :-)

Well it'd just have gone the way of Perl otherwise  :-s




Reply | Threaded
Open this post in threaded view
|

Re: language experimentation [was: co-routines]

Alex Sandro Queiroz e Silva
In reply to this post by Luiz Henrique de Figueiredo
On Thu, 1 Feb 2001, Luiz Henrique de Figueiredo wrote:

> >- Or maybe I
> >just need to get a decent mail browser that can sort my mail
>
> Don't Netscape Mail readeer or Eudora sort mail into threads?

Hallo,
	Certainly Pine and Mutt do. :)

  --Alex	[hidden email]		Lab. de Computação Gráfica/UFC
+----------------------------------------------------------------------------+
|"Minha força vem da solidão. Não tenho medo das chuvas tempestuosas nem das |
| grandes ventanias soltas, pois eu também sou o escuro da noite."	     |
|	- Clarice Lispector						     |
+----------------------------------------------------------------------------+


123