Bookworm Adventures Postmortem

classic Classic list List threaded Threaded
50 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Bookworm Adventures Postmortem

Joshua Jensen
The PopCap Games guys put up a postmortem on their newest game, Bookworm Adventures. It has many references to how they used Lua, modified it to accomplish their needs, and the technology they built around Lua.

http://www.igda.org/casual/quarterly/2_2/index.php?id=2

Josh

Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Luiz Henrique de Figueiredo
> The PopCap Games guys put up a postmortem on their newest game, Bookworm 
> Adventures.  It has many references to how they used Lua, modified it to 
> accomplish their needs, and the technology they built around Lua.
> 
> http://www.igda.org/casual/quarterly/2_2/index.php?id=2

"We heavily modified the Lua interpreter to support many advanced
features. The first thing we did was to change the syntax to be almost
identical to C/C++ since the BASIC-like coding style was irksome, and to
add in sorely missed operators such as +=, *=, ++, etc. "

If they wanted Javascript they knew where to find it, right? :-(

"Furthermore, we made a series of incredible performance enhancements to
the execution speed and memory management of the interpreter. We greatly
decreased table access times and improved the performance of the garbage
collector."

Now, it would be interesting to know details about this.

"Lua was a constant source of extreme anger and frustration."

:-(

"The Lua debugger was highly experimental and didn't become stable until
several months before launch."

So, why blame it on Lua??
--lhf

Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Alex Queiroz
Hallo,

On 3/14/07, Luiz Henrique de Figueiredo <[hidden email]> wrote:

"We heavily modified the Lua interpreter to support many advanced
features. The first thing we did was to change the syntax to be almost
identical to C/C++ since the BASIC-like coding style was irksome, and to
add in sorely missed operators such as +=, *=, ++, etc. "

If they wanted Javascript they knew where to find it, right? :-(


    If those guys think that C/C++ syntax is advanced technology, I
don't think I'm buying their game anytime soon... But the most telling
part is how they think Lua syntax is BASIC-like.

--
-alex
http://www.ventonegro.org/

Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Miles Bader-2
"Alex Queiroz" <[hidden email]> writes:
>     If those guys think that C/C++ syntax is advanced technology, I
> don't think I'm buying their game anytime soon... But the most telling
> part is how they think Lua syntax is BASIC-like.

Yeah, it's a pretty stupid comment.

[I _like_ C/C++ syntax -- but Lua's syntax is elegant and classic.]

-Miles
-- 
People who are more than casually interested in computers should have at
least some idea of what the underlying hardware is like.  Otherwise the
programs they write will be pretty weird.  -- Donald Knuth


Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Stefan Sandberg
In reply to this post by Luiz Henrique de Figueiredo
Ha, way to dig their own hole..

"Furthermore, since the syntax of the language changed as we added/modified features, the game's coding style frequently mixed original Lua syntax with C++ syntax. But we weren't about to throw out tens of thousands of lines of source code just to beautify things."



Luiz Henrique de Figueiredo wrote:
The PopCap Games guys put up a postmortem on their newest game, Bookworm Adventures. It has many references to how they used Lua, modified it to accomplish their needs, and the technology they built around Lua.

http://www.igda.org/casual/quarterly/2_2/index.php?id=2

"We heavily modified the Lua interpreter to support many advanced
features. The first thing we did was to change the syntax to be almost
identical to C/C++ since the BASIC-like coding style was irksome, and to
add in sorely missed operators such as +=, *=, ++, etc. "

If they wanted Javascript they knew where to find it, right? :-(

"Furthermore, we made a series of incredible performance enhancements to
the execution speed and memory management of the interpreter. We greatly
decreased table access times and improved the performance of the garbage
collector."

Now, it would be interesting to know details about this.

"Lua was a constant source of extreme anger and frustration."

:-(

"The Lua debugger was highly experimental and didn't become stable until
several months before launch."

So, why blame it on Lua??
--lhf



Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Eike Decker-2
In reply to this post by Luiz Henrique de Figueiredo
The game is playing well - they made a good job on the gameplay I think. Though
I wonder what the 150k lines of code went to... Would be nice to know what
parts are written in Lua and how much code is used for the engine stuff. I
guess most of it is C++ code - the luacode in the game is using 186kb discspace
in compiled form. So I would guess that it would be something between 20k lines
of code... maybe less. But maybe there's some code hidden in the pak file. 

Anyhow, the thing that I am interested the most is:

"And to aid in rapidly designing code we added a complete state synchronizer
that allowed you to modify any Lua code and reload it while the game was
running."

How can this work? And how stable can this work? Any idea? 
It sounds highly useful, but I can't imagine that this can work stable - what if
certain variables are removed or parts are changed - what is happening then to
the previously assigned values?
When I required a piece of code to be replaced for debugging purposes, I put it
in a file and allowed reloading the whole file, though that always requires
some more lines of lua code handling the reloading stuff. But I would prefer an
automed solution...

-eike

Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Joshua Jensen
In reply to this post by Luiz Henrique de Figueiredo
Luiz Henrique de Figueiredo wrote:
If they wanted Javascript they knew where to find it, right? :-(
I would suppose.  Or grab something like SmallC.

I once modified the syntax heavily like this long ago (Lua 3, as I recall). I wasted a lot of time doing that, and in the end, it didn't help much... it only hindered.

Having operators like += and *= (beaten to death here) would be very nice convenience features, so I can't disagree with that.
"Furthermore, we made a series of incredible performance enhancements to
the execution speed and memory management of the interpreter. We greatly
decreased table access times and improved the performance of the garbage
collector."

Now, it would be interesting to know details about this.
When working on Amped 1 under tight memory situations, the main performance enhancements we did (and previously mentioned on this list) were avoiding Lua constructs which allocated memory and routing memory allocation into pooled heaps of memory. It's not clear which version of Lua PopCap used, so it's hard to speculate.
"Lua was a constant source of extreme anger and frustration."

:-(
Having shipped Lua (through LuaPlus) in Amped 1, 2, and 3, Links 2004, Top Spin 1 and 2, NFL Fever 2004, NHL Rivals 2004, and more, I can say Lua was a tremendous help. Lua was also tremendously helpful in my latest game, Pathstorm (http://cavebug.com/), where I got to use coroutines for the first time in a game.

In any case, Lua is what you make of it. If all you can think about is debugging in your blessed IDE, you're going to have constant frustration. In Amped, we had a remote debugger application (ships in LuaPlus), but all I ever heard about was griping because it wasn't built into Visual Studio and had bugs. We had to task switch between the Remote Lua Debugger and Visual Studio when we stepped in and out of code. That was mildly frustrating, but it was better than having nothing at all.

When you realize Lua isn't C or C++ and is intended for a different purpose and is a higher level language, everything works out just fine.

IMHO.

Josh

Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Glenn Maynard
In reply to this post by Luiz Henrique de Figueiredo
On Wed, Mar 14, 2007 at 08:59:43PM -0300, Luiz Henrique de Figueiredo wrote:
> "We heavily modified the Lua interpreter to support many advanced
> features. The first thing we did was to change the syntax to be almost
> identical to C/C++ since the BASIC-like coding style was irksome, and to
> add in sorely missed operators such as +=, *=, ++, etc. "
> 
> If they wanted Javascript they knew where to find it, right? :-(

I've considered adding those operators a couple times, since
"IndexVariable = IndexVariable + 1" is just tiring.  I'm not quick
to suggest adding operators, but these ones really are complete
no-brainers.  (I havn't yet added them to my tree, due to the cost
of maintaining more forked code and the learning curve imposed by
adding bits to Lua that aren't the Lua that everyone else knows.)

Cosmetically changing the syntax to be more C-like is just gratuitous,
though.

> "Furthermore, we made a series of incredible performance enhancements to
> the execution speed and memory management of the interpreter. We greatly
> decreased table access times and improved the performance of the garbage
> collector."
> 
> Now, it would be interesting to know details about this.

It's always frustrating to me when people say "we took this open source
code and made it better", without any mention of contributing it back.

(Not a suggestion of copyleft or anything like that--my frustration is
how it seems to never even cross their mind.  Copyleft bends arms, but
doesn't create motivation.)

Sure would be nice to see that Lua debugger under the MIT license, too.
It's easy enough to have simple Lua debugging facilities *inside* a game
engine, but it's not so obvious how to do it in a real debugger, stepping
through code inside an embedded Lua state.

> "Lua was a constant source of extreme anger and frustration."
> 
> :-(
> 
> "The Lua debugger was highly experimental and didn't become stable until
> several months before launch."
> 
> So, why blame it on Lua??

"Lua was a constant target of misdirected anger and frustration."

-- 
Glenn Maynard

Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Joshua Jensen
Glenn Maynard wrote:
It's always frustrating to me when people say "we took this open source
code and made it better", without any mention of contributing it back.
In fairness, see starting at the 6th post down: http://developer.popcap.com/viewtopic.php?t=1096

Josh

Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Glenn Maynard
On Wed, Mar 14, 2007 at 11:00:00PM -0600, Joshua Jensen wrote:
> Glenn Maynard wrote:
> >It's always frustrating to me when people say "we took this open source
> >code and made it better", without any mention of contributing it back.
> >  
> In fairness, see starting at the 6th post down: 
> http://developer.popcap.com/viewtopic.php?t=1096

Thanks.  I've probably become a bit oversensitive to it, having seen it
too many times.

-- 
Glenn Maynard

Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Niklas Frykholm-2
In reply to this post by Eike Decker-2

Anyhow, the thing that I am interested the most is:

"And to aid in rapidly designing code we added a complete state synchronizer
that allowed you to modify any Lua code and reload it while the game was
running."

How can this work? And how stable can this work? Any idea? It sounds highly useful, but I can't imagine that this can work stable - what if
certain variables are removed or parts are changed - what is happening then to
the previously assigned values?
In my Lua/C++ mixed projects I always have a "Reload" option which allows you to reload all Lua files. I do this simply by clearing the require-cache and running dofile on "main.lua" thus causing all lua-files to be required again.

I do the "Reload" when I'm on the C++ side, so there is no Lua code running when it is being reloaded. Since the "*.lua" files mostly define functions and methods, reloading is pretty safe. I take care to write the code so that global variables are not obliterated when it is reloaded.

Instead of this:

   Account = class()
   function Account:init()
      ...

I write:

   Account = Account or class()
   function Account:init()
      ...

Now a reload does not recreate the class, it just redefines the methods.

I have error handling set up for lua_pcall(...) so that if an error occurs in the Lua code, the C++ part does not exit. Instead it prints the error message, waits for me to fix the error and reload the lua scripts and then continues running.

I can highly recommend this setup. It is a great speed boost to be able to modify the Lua scripts without having to restart the C++ program.

// Niklas

Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Eike Decker-2
I am using a quite similiar approach:

if Agent then Agent.cleanup() end -- Agent must be nil

Agent = {} 
...

I need to do some cleanups at the end, since the visual objects in Luxinia are
not directly Garbage collected.

Anyhow, this is not always wanted. For instance, my Agent might have a function
called think, that handles the way it behaves. Now the agent approaches a
certain complex situation that resulted randomly. If I reload, the agent code
is reloaded - and thus the agent is deleted. 
Of course, I could always place the thinking method in an extra file and let the
agent reload then thinking function from that file in case it changed...
however, wouldn't it be more functional, if I could just edit the same file and
the changed stuff replaces the running code?
The current approach requires the author to have this pattern in mind. Even
worse, it requires global variables in case of a reload. This bugs me most,
since I often don't want the variables to be global... but this seems to be the
only way to recognize, if a file was reloaded - and accessing the previously
used value.

-eike


> 
> > Anyhow, the thing that I am interested the most is:
> >
> > "And to aid in rapidly designing code we added a complete state
> synchronizer
> > that allowed you to modify any Lua code and reload it while the game was
> > running."
> >
> > How can this work? And how stable can this work? Any idea? 
> > It sounds highly useful, but I can't imagine that this can work stable -
> what if
> > certain variables are removed or parts are changed - what is happening then
> to
> > the previously assigned values?
> >   
> In my Lua/C++ mixed projects I always have a "Reload" option which 
> allows you to reload all Lua files. I do this simply by clearing the 
> require-cache and running dofile on "main.lua" thus causing all 
> lua-files to be required again.
> 
> I do the "Reload" when I'm on the C++ side, so there is no Lua code 
> running when it is being reloaded. Since the "*.lua" files mostly define 
> functions and methods, reloading is pretty safe. I take care to write 
> the code so that global variables are not obliterated when it is reloaded.
> 
> Instead of this:
> 
>     Account = class()
>     function Account:init()
>        ...
> 
> I write:
> 
>     Account = Account or class()
>     function Account:init()
>        ...
> 
> Now a reload does not recreate the class, it just redefines the methods.
> 
> I have error handling set up for lua_pcall(...) so that if an error 
> occurs in the Lua code, the C++ part does not exit. Instead it prints 
> the error message, waits for me to fix the error and reload the lua 
> scripts and then continues running.
> 
> I can highly recommend this setup. It is a great speed boost to be able 
> to modify the Lua scripts without having to restart the C++ program.
> 
> // Niklas
> 




Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Philippe Lhoste
In reply to this post by Luiz Henrique de Figueiredo
Luiz Henrique de Figueiredo wrote:
"We heavily modified the Lua interpreter to support many advanced
features. The first thing we did was to change the syntax to be almost
identical to C/C++ since the BASIC-like coding style was irksome, and to
add in sorely missed operators such as +=, *=, ++, etc. "

If they wanted Javascript they knew where to find it, right? :-(

Indeed. I never had a problem with the syntax of Lua, even if sometime I wish to have ++ -- += and -= (the most used specific operators), because I like long variable names.
But well, I can easily live without them.

"Furthermore, we made a series of incredible performance enhancements to
the execution speed and memory management of the interpreter. We greatly
decreased table access times and improved the performance of the garbage
collector."

Now, it would be interesting to know details about this.

Sure. I saw <http://developer.popcap.com/> that their framework is open source, I don't know if their custom Lua is in there. I have yet to read all this.

--
Philippe Lhoste
--  (near) Paris -- France
--  http://Phi.Lho.free.fr
--  --  --  --  --  --  --  --  --  --  --  --  --  --


Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Philippe Lhoste
In reply to this post by Joshua Jensen
Joshua Jensen wrote:
Having shipped Lua (through LuaPlus) in Amped 1, 2, and 3, Links 2004, Top Spin 1 and 2, NFL Fever 2004, NHL Rivals 2004, and more,

An interesting list, it goes much beyond the usual games mentioned with Lua...

I can say Lua was a tremendous help. Lua was also tremendously helpful in my latest game, Pathstorm (http://cavebug.com/), where I got to use coroutines for the first time in a game.

Looks nice.
You should make a good looking "Buy now" page, even if just to say it isn't ready yet: "The requested URL /pathstorm/order.php was not found on this server." isn't precisely user friendly. :-)

--
Philippe Lhoste
--  (near) Paris -- France
--  http://Phi.Lho.free.fr
--  --  --  --  --  --  --  --  --  --  --  --  --  --


Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Joshua Jensen
Philippe Lhoste wrote:
Joshua Jensen wrote:
Having shipped Lua (through LuaPlus) in Amped 1, 2, and 3, Links 2004, Top Spin 1 and 2, NFL Fever 2004, NHL Rivals 2004, and more,
An interesting list, it goes much beyond the usual games mentioned with Lua...
I come from the games industry, and I am aware of so many more of these titles, too. For instance, any game that uses Anark GameFace would have made use of Lua. Any game that uses PlayFirst's Playground SDK (hit sellers like Diner Dash 2) use Lua. I know of games and non-games companies using Lua through LuaPlus, too (although I'd have to dig through emails to find which ones). Lua is a very popular language, and it wouldn't surprise me if the existing list grew much larger with shipping applications, if only it were known which ones used it.

I can say Lua was a tremendous help. Lua was also tremendously helpful in my latest game, Pathstorm (http://cavebug.com/), where I got to use coroutines for the first time in a game.
Looks nice.
You should make a good looking "Buy now" page, even if just to say it isn't ready yet: "The requested URL /pathstorm/order.php was not found on this server." isn't precisely user friendly. :-)
What a complete and utter embarrassment. <sigh> Pathstorm is shipping and can be downloaded and ordered NOW. It seems the HTML authoring package I use remapped these and removed the ../ I had. They work off the Home page but not one level deep. It is fixed now. Argh.

Josh


Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Niklas Frykholm-2
In reply to this post by Eike Decker-2
Eike Decker wrote:
I am using a quite similiar approach:

if Agent then Agent.cleanup() end -- Agent must be nil

Agent = {} ...

I need to do some cleanups at the end, since the visual objects in Luxinia are
not directly Garbage collected.

Anyhow, this is not always wanted. For instance, my Agent might have a function
called think, that handles the way it behaves. Now the agent approaches a
certain complex situation that resulted randomly. If I reload, the agent code
is reloaded - and thus the agent is deleted.
I don't see why you would need to do cleanup and delete the agent when you reload the lua file. Reloading the file should just change the function and method definitions and give you new behavior. There should be no need to release resources, etc.

// Niklas

Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Zachary P. Landau-4
In reply to this post by Luiz Henrique de Figueiredo
"Lua was a constant source of extreme anger and frustration."

:-(

I may be wrong, but I think they meant the anger and frustration came
from dealing with the issues that cropped up from their modifications.
I don't think they were blaming Lua proper.

--
Zachary P. Landau <[hidden email]>

Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Bret Victor
In reply to this post by Luiz Henrique de Figueiredo
Luiz Henrique de Figueiredo <lhf <at> tecgraf.puc-rio.br> writes:

> "Furthermore, we made a series of incredible performance enhancements to
> the execution speed and memory management of the interpreter. We greatly
> decreased table access times and improved the performance of the garbage
> collector."

I would GUESS that their enhancements involved semantic restrictions or
app-specific tunings.  E.g., limiting table keys to strings, getting rid
of metatables, memory pooling for a common object size...

I'm just guessing, of course, but I wouldn't get my hopes up about
their changes feeding back into Official Lua.





Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Roberto Ierusalimschy
In reply to this post by Joshua Jensen
> The PopCap Games guys put up a postmortem on their newest game, Bookworm 
> Adventures. [...]

Sorry to ask, but why did did they call it a "postmortem"? The first
time I saw it I thought they gave up the project.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Bookworm Adventures Postmortem

Joshua Jensen
Roberto Ierusalimschy wrote:
Sorry to ask, but why did did they call it a "postmortem"? The first
time I saw it I thought they gave up the project.
Heh.  Everyone is dead at the end of a game project.  :)

I'm not sure why, to be honest. That's just what regrouping at the end of a project to discuss things that went right and wrong has always been called, I guess.

Josh

123