Re: Prime Mover v0.1

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

Robert McLay
I have looked at primemover.  I saw that primemover has its own mailing
list but I have more general questions that I think may be of interest
in the Lua community.  

It would be nice to know how long you have been using it.  Where you
think primemover is headed?  

The main feature that looks cool is the object caching.  What other 
advantages do you think pm has over or will have over other choices such
as Make, Scon, Bjam, ...  

One thing that pm would have over others is the ability to run
concurrently.   Some places with big projects find concurrent builds
a big time saving.

Thanks,
R.



Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

Asko Kauppi

What's Prime Mover -- please introduce, or you make this look like spam?

SCons is written like that. :)

As far as I know, make and SCons do concurrent builds. Please, your point...?


Robert McLay kirjoitti 13.10.2006 kello 22.30:

I have looked at primemover. I saw that primemover has its own mailing
list but I have more general questions that I think may be of interest
in the Lua community.

It would be nice to know how long you have been using it.  Where you
think primemover is headed?

The main feature that looks cool is the object caching.  What other
advantages do you think pm has over or will have over other choices such
as Make, Scon, Bjam, ...

One thing that pm would have over others is the ability to run
concurrently.   Some places with big projects find concurrent builds
a big time saving.

Thanks,
R.




Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

David Given
In reply to this post by Robert McLay
Robert McLay wrote:
[...]
> It would be nice to know how long you have been using it.  Where you
> think primemover is headed?  

[Note: I seem to have lectured. Feel free to ignore me if you want...]

I wrote pm because I'd adopted a compiler suite, the Amsterdam Compiler Kit
("tack" on sourceforge, if you're interested), and its build system was a
horrible nightmare. Really. At one point I was trying to figure out how
something got built, and I spent two hours groping through a collection of
weird mutually-recursive makefiles that called shell scripts that called
makefiles, and then I realised something had to be done.

Prime Mover is fundamentally designed to fulfil my needs. I spent some time
looking at the alternatives, and they were all flawed. make loses as soon as
you start doing anything the slightest bit complicated --- recursive
makefiles, ew. Plus, it doesn't understand C dependencies. And the way it
tries (and fails) to guess what you want to do is awful. Scons loses because
it seems to me to be overly complex and requires the end user to install
Python. Ant loses because it's written in Java. All the other build systems I
looked at suffered from the same problems.

Prime Mover is supposed to be is a *simple* build tool that also offers power.
The language is designed to be accessible to people who don't know Lua, but is
supposed to be very simple and expressive:

include "c.pm"

default = cprogram {
	cfile {
		file "hello.c"
	}
}

...is a complete, working pmfile that compiles a source file into an object
into an executable. (cfile, of course, looks inside the C file and
automatically adds the header files to the dependency list.) The behaviour can
be modified by setting properties:

cfile {
	OPTIMISATION = "-Os",
	file "hello.c"
}

...sets the optimisation level for that source file. Or, you could put the
OPTIMISATION clause in the cprogram{} rule to apply it to the whole program...
or at the top level, to apply it to everything.

Rules are object oriented and subclassing a rule to make a new rule is easy.

debug_cfile = cfile {
	class = "debug_cfile",
	CDEFINES = {"DEBUG"}
}

...defines a new rule, debug_cfile, that automatically sets DEBUG for you when
it compiles something.

That's only part of it. One of the key aspects, IMO, is that it's deployed as
a single self-extracting shell script that automatically and invisibly builds
the Lua interpreter for you. This means that, unlike scons, it doesn't require
the user to install any extra software in order to use it. (The pm executable
is about 100kB.) In order to run, it needs sh, sed, head, gzip, and cc (or
gcc), which pretty much all Posix platforms have. This gives the greatest
flexibility: it'll run on bare-bones systems where traditionally all you had
was make, but with far more power.

(A not-incidental advantage is that pm, being written in Lua, has a Lua
interpreter in it. Which means that any project using pm as its build tool
also gains, automatically, the ability to write helper scripts in Lua --- for
free. This can be more valuable than you think.)

It also stores object files itself; rather than being put in with your source
code, it squirrels them away in its own cache. This means that you can build
the same source file several times and not have to worry about object file
clashes. This allows you to, for example, cross compile your application by
changing the value of CC and everything else happens automatically.

The ACK currently makes about 10000 temporary files when it builds. (Mostly
due to building the ACK's own libc once for every platform it supports.) Prime
Mover does this happily in one single pass.

[...]
> One thing that pm would have over others is the ability to run
> concurrently.   Some places with big projects find concurrent builds
> a big time saving.

Unfortunately, the one thing it doesn't do is parallel builds. Sorry; the way
the dependency system works depends on things being built in order.

-- 
+- David Given --McQ-+ "Gaping from its single obling socket was
|  [hidden email]    | scintillating, many fauceted scarlet emerald..."
| ([hidden email]) | --- Jim Theis, _The Eye of Argon_ (spelling
+- www.cowlark.com --+ original)


Attachment: signature.asc
Description: OpenPGP digital signature

Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

Asko Kauppi
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Thanks, good background.

For the record:  SCons - Python + Lua = Hamster

Requires no tools whatsoever, can be used to build OS X, BSD, Linux, Win32 and PocketPC apps.
Can be expanded to suit any OS/compiler, by adding a module for it.

http://kotisivu.dnainternet.fi/askok/hamster


David Given kirjoitti 14.10.2006 kello 1.01:

Robert McLay wrote:
[...]
It would be nice to know how long you have been using it.  Where you
think primemover is headed?

[Note: I seem to have lectured. Feel free to ignore me if you want...]

I wrote pm because I'd adopted a compiler suite, the Amsterdam Compiler Kit ("tack" on sourceforge, if you're interested), and its build system was a horrible nightmare. Really. At one point I was trying to figure out how something got built, and I spent two hours groping through a collection of weird mutually-recursive makefiles that called shell scripts that called
makefiles, and then I realised something had to be done.

Prime Mover is fundamentally designed to fulfil my needs. I spent some time looking at the alternatives, and they were all flawed. make loses as soon as
you start doing anything the slightest bit complicated --- recursive
makefiles, ew. Plus, it doesn't understand C dependencies. And the way it tries (and fails) to guess what you want to do is awful. Scons loses because it seems to me to be overly complex and requires the end user to install Python. Ant loses because it's written in Java. All the other build systems I
looked at suffered from the same problems.

Prime Mover is supposed to be is a *simple* build tool that also offers power. The language is designed to be accessible to people who don't know Lua, but is
supposed to be very simple and expressive:

include "c.pm"

default = cprogram {
	cfile {
		file "hello.c"
	}
}

...is a complete, working pmfile that compiles a source file into an object
into an executable. (cfile, of course, looks inside the C file and
automatically adds the header files to the dependency list.) The behaviour can
be modified by setting properties:

cfile {
	OPTIMISATION = "-Os",
	file "hello.c"
}

...sets the optimisation level for that source file. Or, you could put the OPTIMISATION clause in the cprogram{} rule to apply it to the whole program...
or at the top level, to apply it to everything.

Rules are object oriented and subclassing a rule to make a new rule is easy.

debug_cfile = cfile {
	class = "debug_cfile",
	CDEFINES = {"DEBUG"}
}

...defines a new rule, debug_cfile, that automatically sets DEBUG for you when
it compiles something.

That's only part of it. One of the key aspects, IMO, is that it's deployed as a single self-extracting shell script that automatically and invisibly builds the Lua interpreter for you. This means that, unlike scons, it doesn't require the user to install any extra software in order to use it. (The pm executable is about 100kB.) In order to run, it needs sh, sed, head, gzip, and cc (or gcc), which pretty much all Posix platforms have. This gives the greatest flexibility: it'll run on bare-bones systems where traditionally all you had
was make, but with far more power.

(A not-incidental advantage is that pm, being written in Lua, has a Lua interpreter in it. Which means that any project using pm as its build tool also gains, automatically, the ability to write helper scripts in Lua --- for
free. This can be more valuable than you think.)

It also stores object files itself; rather than being put in with your source code, it squirrels them away in its own cache. This means that you can build the same source file several times and not have to worry about object file clashes. This allows you to, for example, cross compile your application by
changing the value of CC and everything else happens automatically.

The ACK currently makes about 10000 temporary files when it builds. (Mostly due to building the ACK's own libc once for every platform it supports.) Prime
Mover does this happily in one single pass.

[...]
One thing that pm would have over others is the ability to run
concurrently.   Some places with big projects find concurrent builds
a big time saving.

Unfortunately, the one thing it doesn't do is parallel builds. Sorry; the way
the dependency system works depends on things being built in order.

--
+- David Given --McQ-+ "Gaping from its single obling socket was
| [hidden email] | scintillating, many fauceted scarlet emerald..."
| ([hidden email]) | --- Jim Theis, _The Eye of Argon_ (spelling
+- www.cowlark.com --+ original)



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFFMBMNGJtHlJZfjQoRAp/HAJ9zWYGXxGBPnCyDGqhECpCCvcyUnwCcDZU3
cy7Zizm9pF80UYzziNp1N6U=
=N9Nx
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

Wesley Smith
In reply to this post by David Given
make loses as soon as
you start doing anything the slightest bit complicated --- recursive
makefiles, ew. Plus, it doesn't understand C dependencies. And the way it
tries (and fails) to guess what you want to do is awful. Scons loses because
it seems to me to be overly complex and requires the end user to install
Python. Ant loses because it's written in Java. All the other build systems I
looked at suffered from the same problems.


I can't tell you how much I agree you here.  Everytime I've tried to
use make to build a project of any size that uses subdirectories, it
has been a nightmare of looking at GNU's make documentation and
wondering who on earth made such an esoteric beast.  I wonder if the
people making this kind of software have any kind of human qualities
about them, especially common sense with respect to how people
interact with software.  The terminology and syntax is so difficult to
wrap your mind around and it doesn't have to be this way.  I'm looking
forward to trying PM out.

wes

Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

Zed A. Shaw
In reply to this post by Asko Kauppi
On Sat, 14 Oct 2006 01:28:28 +0300
Asko Kauppi <[hidden email]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> Thanks, good background.
> 
> For the record:  SCons - Python + Lua = Hamster
> 
> Requires no tools whatsoever, can be used to build OS X, BSD, Linux,  
> Win32 and PocketPC apps.
> Can be expanded to suit any OS/compiler, by adding a module for it.
> 
> http://kotisivu.dnainternet.fi/askok/hamster
> 

Uh, last time I used Hamster (last month) it was a thin translator from a Scons-like-lua to the Scons-actual-Python, which is entirely not what it's advertised as.  From the project description (and what you said above) it seems like an entirely standalone Lua SCons replacement.  In actuality it's an SCons Python generator for people addicted to Lua.

Basically, what's the point of a Lua build tool if all it does is have the exact same API as SCons and still requires SCons and Python?  I'd rather just use SCons.

Hopefully I got this all wrong and there actually is a Hamster which doesn't need SCons or Python (since, the one I have still needs SCons and then still needs Python).

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

Asko Kauppi

I'm bad at marketing - I know.. :)

You can use Hamster to:
	- generate SCons scripts on the fly
	- generate Makefiles on the fly
	- just Do it All (no dependencies)

The mishap is mostly due to versions, I will need to check that the latest on LuaForge has the stand-alone features.

I should be using Hamster myself, but I am not. My conception has become, Makefiles with Lua scripts "baked in" can be very, very versatile as such. But had I any larger scale project needing a build system (and a dislike to Python?) I'd give Hamster a go. It is 'ready', has no known severe bugs, and out for grabs.

This is the end of Open Source projects; they become "ready" and still no-one uses them. That's life. :)

-asko

...with a somewhat I-haa kind-of attitude

http://games.disneysblast.com/winnie_the_pooh_games/ eeyore_shadow_catcher.html


Zed A. Shaw kirjoitti 14.10.2006 kello 6.47:

...
Uh, last time I used Hamster (last month) it was a thin translator from a Scons-like-lua to the Scons-actual-Python, which is entirely not what it's advertised as. From the project description (and what you said above) it seems like an entirely standalone Lua SCons replacement. In actuality it's an SCons Python generator for people addicted to Lua.

Basically, what's the point of a Lua build tool if all it does is have the exact same API as SCons and still requires SCons and Python? I'd rather just use SCons.

Hopefully I got this all wrong and there actually is a Hamster which doesn't need SCons or Python (since, the one I have still needs SCons and then still needs Python).

--
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.


Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

David Olofson
On Saturday 14 October 2006 16:04, Asko Kauppi wrote:
[...]
> This is the end of Open Source projects; they become "ready" and  
> still no-one uses them.  That's life. :)

Well, obviously, for any product, whether commercial or free, 
Free/Open Source or whatever, there has to be a demand for such a 
product if it's going to be "successful".

However, that's just a small part of the mechanics behind this. There 
is marketing, documentation, cost of migration, severity of 
disadvantages with competing products, various subjective aspects 
etc.

Sometimes, it seems like it's just dumb luck that makes one product 
more popular than another, though most of the time, that's probably 
just because the real reasons are not obvious.


//David Olofson - Programmer, Composer, Open Source Advocate

.-------  http://olofson.net - Games, SDL examples  -------.
|        http://zeespace.net - 2.5D rendering engine       |
|       http://audiality.org - Music/audio engine          |
|     http://eel.olofson.net - Real time scripting         |
'--  http://www.reologica.se - Rheology instrumentation  --'

Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

Asko Kauppi

Let's not start a too deep philosophic discussion, escaping from Lua, but...


I agree. The OSS market is a market economy. Only, it's weird in the way that if I now _were_ to actively market Hamster, that'd actually make my life _less_ comfortable (although I would be proud, of course), since that would incur maintenance requests, contributions, etc... and not bring in actual benefits to my enjoyment.

Just letting things be, moving to new issues, I live an easier life.

Something fishy in that. :) And something completely different from market society.

-asko


David Olofson kirjoitti 14.10.2006 kello 17.37:

On Saturday 14 October 2006 16:04, Asko Kauppi wrote:
[...]
This is the end of Open Source projects; they become "ready" and
still no-one uses them.  That's life. :)

Well, obviously, for any product, whether commercial or free,
Free/Open Source or whatever, there has to be a demand for such a
product if it's going to be "successful".

However, that's just a small part of the mechanics behind this. There
is marketing, documentation, cost of migration, severity of
disadvantages with competing products, various subjective aspects
etc.

Sometimes, it seems like it's just dumb luck that makes one product
more popular than another, though most of the time, that's probably
just because the real reasons are not obvious.


//David Olofson - Programmer, Composer, Open Source Advocate

.-------  http://olofson.net - Games, SDL examples  -------.
|        http://zeespace.net - 2.5D rendering engine       |
|       http://audiality.org - Music/audio engine          |
|     http://eel.olofson.net - Real time scripting         |
'--  http://www.reologica.se - Rheology instrumentation  --'


Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

David Olofson
On Saturday 14 October 2006 17:24, Asko Kauppi wrote:
> 
> Let's not start a too deep philosophic discussion, escaping from
> Lua, but...
> 
> 
> I agree.  The OSS market is a market economy.  Only, it's weird in  
> the way that if I now _were_ to actively market Hamster, that'd  
> actually make my life _less_ comfortable (although I would be proud,  
> of course), since that would incur maintenance requests,  
> contributions, etc... and not bring in actual benefits to my
> enjoyment.
>
> Just letting things be, moving to new issues, I live an easier life.
>
> Something fishy in that. :)  And something completely different from  
> market society.

Not really. If you're not selling the software, and you're not using 
it, what's in it for you? Most people tend to do work in order to get 
something back. ;-)

One way of thinking about software is as living organisms that need 
users and active contributors to stay alive.

Some software stays alive because someone has constructed some way of 
having the software generate money, so that programmers can be paid 
to work on it.

Free/Open Source software usually stays alive as a result of being 
useful enough that people pick it up, use it and improve it as needed 
to get the job done. Basically, users are paying in bug reports, code 
and stuff like that, instead of money. The software doesn't rely on 
any single person to stay alive, as new people may take over as 
others lose interest - and this goes for maintainers as well 
contributors and users.


//David Olofson - Programmer, Composer, Open Source Advocate

.-------  http://olofson.net - Games, SDL examples  -------.
|        http://zeespace.net - 2.5D rendering engine       |
|       http://audiality.org - Music/audio engine          |
|     http://eel.olofson.net - Real time scripting         |
'--  http://www.reologica.se - Rheology instrumentation  --'

Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

Sam Roberts-2
In reply to this post by Wesley Smith
On Fri, Oct 13, 2006 at 06:44:58PM -0700, Wesley Smith wrote:
> I can't tell you how much I agree you here.  Everytime I've tried to
> use make to build a project of any size that uses subdirectories, it
> has been a nightmare of looking at GNU's make documentation and
> wondering who on earth made such an esoteric beast.  I wonder if the

make doesn't care what directory your files are in, you haven't figured
out how to use the tool yet if that is causing you trouble.

People have trouble with make because they attempt to use it as an
imperative language. Its not, its a rules-based language (akin to logic
languages like Prolog) wherein you state the conditions that must hold
for a particular goal ("target") to be satisfied. I think its the only
language of that type that is commonly used.

The other thing that people don't get about make is its a unix tool. It
does ONE thing well, evaluating dependency rules to determine how to
achieve goals. The rules themselves are usually built statically, in a
language other than make, i.e., you have to mix make with other tools
for a complete build system.  In particular, dependency rules should be
built by a tool that knows as much as possible about C code... like
a C compiler:

	http://make.paulandlesley.org/autodep.html

Cheers,
Sam


Reply | Threaded
Open this post in threaded view
|

Make+Lua = winner (Re: Prime Mover v0.1)

Asko Kauppi

...which brought to my mind, that Hamster does .h dependencies automatically.

I agree with you; having been both a Make what-the-... and, nowadays a huge fan (to the extent that I get mocked at work about that) of it. It does rule, and I'm suggesting it to use outside of the building processes as well. Any dependencies, that's Make.

To the point? I'd like to take part if someone wants to start a "Lua macros for make" project. It'd basically be a set of Makefiles (for inclusion) that define usable features, using Lua. The use would be to come along without sed, awk, grep and other such Unix candies, allowing the Make+Lua+macros approach to run on Windows (ehem!) based machines as well.

I think LuaRocks will need this; and I'm going to be there to help build it. But the scope of usability would be wider.

-asko


Sam Roberts kirjoitti 16.10.2006 kello 21.26:

On Fri, Oct 13, 2006 at 06:44:58PM -0700, Wesley Smith wrote:
I can't tell you how much I agree you here.  Everytime I've tried to
use make to build a project of any size that uses subdirectories, it
has been a nightmare of looking at GNU's make documentation and
wondering who on earth made such an esoteric beast.  I wonder if the

make doesn't care what directory your files are in, you haven't figured
out how to use the tool yet if that is causing you trouble.

People have trouble with make because they attempt to use it as an
imperative language. Its not, its a rules-based language (akin to logic
languages like Prolog) wherein you state the conditions that must hold
for a particular goal ("target") to be satisfied. I think its the only
language of that type that is commonly used.

The other thing that people don't get about make is its a unix tool. It
does ONE thing well, evaluating dependency rules to determine how to
achieve goals. The rules themselves are usually built statically, in a
language other than make, i.e., you have to mix make with other tools
for a complete build system. In particular, dependency rules should be
built by a tool that knows as much as possible about C code... like
a C compiler:

	http://make.paulandlesley.org/autodep.html

Cheers,
Sam



Reply | Threaded
Open this post in threaded view
|

Re: Make+Lua = winner (Re: Prime Mover v0.1)

Ken Smith-2
I recently posted a patch to the GNU make mailing list that hasn't
gotten a lot of attention yet.

 http://lists.gnu.org/archive/html/help-make/2006-10/msg00018.html

It adds $(lua ...) to make so you can execute lua programs similarly
to the way you can execute shell commands with $(shell ...).  I think
this may obviate the need for Guile support which has been on the
radar for several years.

If any of you think this is a great thing for make, please make some
noise at [hidden email] so we can possibly get the maintainers to
accept it into the official distribution for the next release.

I'm including the original attachments as the linked mail archive
munges the filenames when I try to download them.

 Ken Smith

On 10/16/06, Asko Kauppi <[hidden email]> wrote:

...which brought to my mind, that Hamster does .h dependencies
automatically.

I agree with you; having been both a Make what-the-... and, nowadays
a huge fan (to the extent that I get mocked at work about that) of
it.  It does rule, and I'm suggesting it to use outside of the
building processes as well. Any dependencies, that's Make.

To the point?  I'd like to take part if someone wants to start a "Lua
macros for make" project. It'd basically be a set of Makefiles (for
inclusion) that define usable features, using Lua.  The use would be
to come along without sed, awk, grep and other such Unix candies,
allowing the Make+Lua+macros approach to run on Windows (ehem!) based
machines as well.

I think LuaRocks will need this; and I'm going to be there to help
build it. But the scope of usability would be wider.

-asko


Sam Roberts kirjoitti 16.10.2006 kello 21.26:

> On Fri, Oct 13, 2006 at 06:44:58PM -0700, Wesley Smith wrote:
>> I can't tell you how much I agree you here.  Everytime I've tried to
>> use make to build a project of any size that uses subdirectories, it
>> has been a nightmare of looking at GNU's make documentation and
>> wondering who on earth made such an esoteric beast.  I wonder if the
>
> make doesn't care what directory your files are in, you haven't
> figured
> out how to use the tool yet if that is causing you trouble.
>
> People have trouble with make because they attempt to use it as an
> imperative language. Its not, its a rules-based language (akin to
> logic
> languages like Prolog) wherein you state the conditions that must hold
> for a particular goal ("target") to be satisfied. I think its the only
> language of that type that is commonly used.
>
> The other thing that people don't get about make is its a unix
> tool. It
> does ONE thing well, evaluating dependency rules to determine how to
> achieve goals. The rules themselves are usually built statically, in a
> language other than make, i.e., you have to mix make with other tools
> for a complete build system.  In particular, dependency rules
> should be
> built by a tool that knows as much as possible about C code... like
> a C compiler:
>
>       http://make.paulandlesley.org/autodep.html
>
> Cheers,
> Sam
>


Attachment: GNUmakefile
Description: Binary data

Attachment: make-3.81-lua-5.1.1-function.c.patch
Description: Binary data

Attachment: make-3.81-lua-5.1.1-Makefile-Gentoo-Linux-2.6-ppc.patch
Description: Binary data

Attachment: make-3.81-lua-5.1.1-Makefile.patch
Description: Binary data

Reply | Threaded
Open this post in threaded view
|

Re: Make+Lua = winner (Re: Prime Mover v0.1)

Ralph Hempel
Ken Smith wrote:
I recently posted a patch to the GNU make mailing list that hasn't
gotten a lot of attention yet.

 http://lists.gnu.org/archive/html/help-make/2006-10/msg00018.html

It adds $(lua ...) to make so you can execute lua programs similarly
to the way you can execute shell commands with $(shell ...).  I think
this may obviate the need for Guile support which has been on the
radar for several years.

If any of you think this is a great thing for make, please make some
noise at [hidden email] so we can possibly get the maintainers to
accept it into the official distribution for the next release.

Hmmm. I've used make for years for non-trivial cross platform
embedded systems builds. The same code is compiled and linked
for different processors, target boards, and memory maps. It
even supports multiple compiler toolchains...

Yes, it's a bit complicated, but one invocation of make builds
everything.

In some respects, learning to program make is a bit of an
intellectual leap - sort of like moving from procedural to
functional languages. Please note I'm not saying that if you
don't understand make that you're stupid.

I really like the fact that make looks after the "out of date"
part of the build logic, but using make to figure out things
like path names, extensions, library labels etc is too
complicated.

In many cases, you go out to the shell just to do something
that would be just as easy to do with a built-in scripting
language - like lua.

I'll think about this a bit as I'm currently doing some work
on my build system....

Cheers, Ralph

Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

David Given
In reply to this post by Sam Roberts-2
Sam Roberts wrote:
[...]
> make doesn't care what directory your files are in, you haven't figured
> out how to use the tool yet if that is causing you trouble.

All right, try this: I have fifty files in directory A, all with .c
extensions, that need to be built with one particular set of compilation
options. I also have fifty files in directory B, all with .c extensions, that
need to be built with *another* set of compilation options. How do you write a
single makefile that deals with this cleanly?

The answer is, you can't. make automatically determines what rule to apply to
each file based on the file extension. (Actually, a file pattern. It might be
possible to use patterns of the form 'A/%.c' and 'B/%.c', but I'd be hesitant
to use it.) However, it only allows you to specify a *single* rule for each
pattern. In order to solve the above problem, you have two options: firstly,
you can specify manually which files each rule should apply to --- which
rather defeats the purpose of make's automatic rule mechanism --- or else you
write two makefiles, which is much less effort, but which blows a big hole in
the dependency management.

It gets even worse when your two sets of files are all in *one* directory. It
gets an order of magnitude worse when you try to bolt in automatic C
dependency management. It gets *another* order of magnitude worse when you try
to add support for cross-compilation. And if you try to add support for
multi-stage compilation, make curls up in a corner and cries.

Here's a real-world example I had to cope with in my day job: I'm building
fifty different Special Objects. Each Special Object is made up of several
source files (.cc files). Each .cc file needs to be preprocessed via a custom
program before compilation, where the preprocessor flags vary according to the
Special Object --- and the *first* source file of each Special Object, when
preprocessed, produces an extra .o file that needs to be passed directly to
the linker. This all has to handle C dependencies correctly, and it would be
nice if we could build parallel debug and optimisation versions.

(If you're interested, I solved that one by writing a Lua makefile generator,
where the makefile specified absolutely everything explicitly. Prime Mover
actually developed from that program.)

Trust me. I know how to use make.

-- 
+- David Given --McQ-+ "Gaping from its single obling socket was
|  [hidden email]    | scintillating, many fauceted scarlet emerald..."
| ([hidden email]) | --- Jim Theis, _The Eye of Argon_ (spelling
+- www.cowlark.com --+ original)


Attachment: signature.asc
Description: OpenPGP digital signature

Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

Ralph Hempel
David Given wrote:
All right, try this: I have fifty files in directory A, all with .c
extensions, that need to be built with one particular set of compilation
options. I also have fifty files in directory B, all with .c extensions, that
need to be built with *another* set of compilation options. How do you write a
single makefile that deals with this cleanly?

The answer is, you can't. make automatically determines what rule to apply to
each file based on the file extension. (Actually, a file pattern. It might be
possible to use patterns of the form 'A/%.c' and 'B/%.c', but I'd be hesitant
to use it.) However, it only allows you to specify a *single* rule for each
pattern. In order to solve the above problem, you have two options: firstly,
you can specify manually which files each rule should apply to --- which
rather defeats the purpose of make's automatic rule mechanism --- or else you
write two makefiles, which is much less effort, but which blows a big hole in
the dependency management.

First, dump all of the built in rules - they are for simple default
build scenarios, which yours is clearly not :-)

Then, learn to use the 'A/%.c' and 'B/%.c' patterns, and specify
what you want to do to build the .o files, and what compiler options,
like this:

CCFLAGS_A = some flag settings for scenario A
CCFLAGS_B = some flag settings for scenario B

A/%.o : %.c
	$(COMPILER_BIN)/$(CC) $(CFLAGS_A) $<

B/%.o : %.c
	$(COMPILER_BIN)/$(CC) $(CFLAGS_B) $<

I've got specific definitions for COMPILER_BIN and CC because
I use toolchains for multiple processors in my project. Talk
about flags for this set of files vs files for that set!

It gets even worse when your two sets of files are all in *one* directory. It
gets an order of magnitude worse when you try to bolt in automatic C
dependency management. It gets *another* order of magnitude worse when you try
to add support for cross-compilation. And if you try to add support for
multi-stage compilation, make curls up in a corner and cries.

Yes, but if you talk in a quiet voice and lower yourself to its level
it will eventually come out, and then you can hit it over the head
and make it do what you want :-)

Trust me. I know how to use make.

I think you do too, and I think we all would like a better general
purpose scripting ability for or alternative to make. But there's
a lot of inertia behind that tool....

Cheers, Ralph

Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

Sam Roberts-2
In reply to this post by David Given
On Mon, Oct 16, 2006 at 10:38:34PM +0100, David Given wrote:
> Sam Roberts wrote:
> [...]
> > make doesn't care what directory your files are in, you haven't figured
> > out how to use the tool yet if that is causing you trouble.
> 
> All right, try this: I have fifty files in directory A, all with .c
> extensions, that need to be built with one particular set of compilation
> options. I also have fifty files in directory B, all with .c extensions, that
> need to be built with *another* set of compilation options. How do you write a
> single makefile that deals with this cleanly?

> The answer is, you can't. make automatically determines what rule to apply to
> each file based on the file extension. (Actually, a file pattern. It might be

Thats true of POSIX make. I should have been clear that I was talking
about the GNU dialect (I thought you'd mentioned it specifically, but I
confused you with another).

> possible to use patterns of the form 'A/%.c' and 'B/%.c', but I'd be hesitant
> to use it.) However, it only allows you to specify a *single* rule for each
> pattern. In order to solve the above problem, you have two options: firstly,
> you can specify manually which files each rule should apply to --- which
> rather defeats the purpose of make's automatic rule mechanism --- or else you
> write two makefiles, which is much less effort, but which blows a big hole in
> the dependency management.

This is possible in various ways. We do this in our build system,
off the top of my head, something like:

    CFLAGS = ... common flags
	CFLAGS_FOR_A = .... flags for A

	A_SRC = $(wildcard A/*.c)
	A_OBJ = $(A_SRC:.c=.o)
	$(A_OBJ): CFLAGS+=$(CFLAGS_FOR_A)

	B_SRC = $(wildcard B/*.c)
	B_OBJ = $(B_SRC:.c=.o)
	$(B_OBJ): CFLAGS+=$(CFLAGS_FOR_B)

	libstuff.a: $(A_OBJ) $(B_OBJ)
    # etc....

That's only 3 lines per-subdir, but you can also factor the common stuff
into a sub makefile, and then include that sub-makefile multiple times,
each time doing something different:

    CFLAGS = ... common flags
	CFLAGS_FOR_A = .... flags for A

	DIR:=A
	include dir.mk

	DIR:=B
	include dir.mk

	libstuff.a: $(A_OBJ) $(B_OBJ)
    # etc....

dir.mk:

	$(DIR)_SRC := $(wildcard $(DIR)/*.c)
	$(DIR)_OBJ := $($(DIR)_SRC:.c=.o)

	$($(DIR)_OBJ): CFLAGS+=$(CFLAGS_FOR_$(DIR))

Thats arguably no longer a "single makefile", but it is NOT a recursive
makefile.

> Here's a real-world example I had to cope with in my day job: I'm building
> fifty different Special Objects. Each Special Object is made up of several
> source files (.cc files). Each .cc file needs to be preprocessed via a custom
> program before compilation, where the preprocessor flags vary according to the
> Special Object --- and the *first* source file of each Special Object, when
> preprocessed, produces an extra .o file that needs to be passed directly to
> the linker. This all has to handle C dependencies correctly, and it would be
> nice if we could build parallel debug and optimisation versions.

For situations like the above there are some things that might help.
It's possible for the variable expansion on the right to depend on the
target, for example:

  %.o: %.c CFLAGS+=$(*)_CFLAGS

so you can get a.c to have a_CFLAGS appended to CFLAGS when a.o is built.

> (If you're interested, I solved that one by writing a Lua makefile generator,
> where the makefile specified absolutely everything explicitly. Prime Mover
> actually developed from that program.)

Thats a time-honored approach, and I would say its a very good one.  Its
what I meant by using make as a tool only for rule evaluation, not
necessarily for writing the rules.

> Trust me. I know how to use make.

Sorry, David, I didn't mean to offend.

Sam


Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

Dave Dodge
In reply to this post by David Given
On Mon, Oct 16, 2006 at 10:38:34PM +0100, David Given wrote:
> and the *first* source file of each Special Object, when
> preprocessed, produces an extra .o file that needs to be passed directly to
> the linker. This all has to handle C dependencies correctly,

BTW a subtle bit of GNU make behavior is that pattern rules (but not
normal rules) can do proper dependency handling when a single input
file produces multiple output files.  A typical use case would be
bison, which usually produces both .c and .h files from a single .y
file.  Admittedly the patterns might be tricky to get right sometimes.

> (If you're interested, I solved that one by writing a Lua makefile generator,
> where the makefile specified absolutely everything explicitly.

I agree with the other comment that fully explicit makefiles can be a
completely reasonable technique.  I've done that on projects myself.

And I'll concur that recursive makefiles are undesireable.  A current
project builds in several directories using a single Makefile, which
is great because it's able to do full parallel builds based on
dependency information.  We do have a couple of "include" directives
to pull in things like the compiler settings at runtime, but no
recursion.

Prime mover does sound interesting, but the lack of parallel builds
would be painful on the SMP systems I deal with.

                                                  -Dave Dodge

Reply | Threaded
Open this post in threaded view
|

Re: Prime Mover v0.1

David Given
In reply to this post by Sam Roberts-2
Sam Roberts wrote:
[...]
> Sorry, David, I didn't mean to offend.

No, I'm sorry --- I was tired and snappish (and way off topic) and shouldn't
have said that. The Lua list is a nicely flame free place, and I should take
more care not to jeapardise that...

-- 
+- David Given --McQ-+ "There is one thing a man must do // Before his
|  [hidden email]    | life is done; // Write two lines in APL // And make
| ([hidden email]) | the buggers run."
+- www.cowlark.com --+ --- The Devil's DP Dictionary

Attachment: signature.asc
Description: OpenPGP digital signature

Reply | Threaded
Open this post in threaded view
|

Re: Make+Lua = winner (Re: Prime Mover v0.1)

Asko Kauppi
In reply to this post by Ken Smith-2

What is the benefit that $(lua ...) would bring, especially since a $ (shell lua ...) can be defined to a macro?


Ken Smith kirjoitti 17.10.2006 kello 0.09:

I recently posted a patch to the GNU make mailing list that hasn't
gotten a lot of attention yet.

 http://lists.gnu.org/archive/html/help-make/2006-10/msg00018.html

It adds $(lua ...) to make so you can execute lua programs similarly
to the way you can execute shell commands with $(shell ...).  I think
this may obviate the need for Guile support which has been on the
radar for several years.

If any of you think this is a great thing for make, please make some
noise at [hidden email] so we can possibly get the maintainers to
accept it into the official distribution for the next release.

I'm including the original attachments as the linked mail archive
munges the filenames when I try to download them.

 Ken Smith

On 10/16/06, Asko Kauppi <[hidden email]> wrote:

...which brought to my mind, that Hamster does .h dependencies
automatically.

I agree with you; having been both a Make what-the-... and, nowadays
a huge fan (to the extent that I get mocked at work about that) of
it.  It does rule, and I'm suggesting it to use outside of the
building processes as well. Any dependencies, that's Make.

To the point?  I'd like to take part if someone wants to start a "Lua
macros for make" project. It'd basically be a set of Makefiles (for
inclusion) that define usable features, using Lua.  The use would be
to come along without sed, awk, grep and other such Unix candies,
allowing the Make+Lua+macros approach to run on Windows (ehem!) based
machines as well.

I think LuaRocks will need this; and I'm going to be there to help
build it. But the scope of usability would be wider.

-asko


Sam Roberts kirjoitti 16.10.2006 kello 21.26:

> On Fri, Oct 13, 2006 at 06:44:58PM -0700, Wesley Smith wrote:
>> I can't tell you how much I agree you here. Everytime I've tried to >> use make to build a project of any size that uses subdirectories, it
>> has been a nightmare of looking at GNU's make documentation and
>> wondering who on earth made such an esoteric beast. I wonder if the
>
> make doesn't care what directory your files are in, you haven't
> figured
> out how to use the tool yet if that is causing you trouble.
>
> People have trouble with make because they attempt to use it as an
> imperative language. Its not, its a rules-based language (akin to
> logic
> languages like Prolog) wherein you state the conditions that must hold > for a particular goal ("target") to be satisfied. I think its the only
> language of that type that is commonly used.
>
> The other thing that people don't get about make is its a unix
> tool. It
> does ONE thing well, evaluating dependency rules to determine how to > achieve goals. The rules themselves are usually built statically, in a > language other than make, i.e., you have to mix make with other tools
> for a complete build system.  In particular, dependency rules
> should be
> built by a tool that knows as much as possible about C code... like
> a C compiler:
>
>       http://make.paulandlesley.org/autodep.html
>
> Cheers,
> Sam
>


<GNUmakefile>
<make-3.81-lua-5.1.1-function.c.patch>
<make-3.81-lua-5.1.1-Makefile-Gentoo-Linux-2.6-ppc.patch>
<make-3.81-lua-5.1.1-Makefile.patch>


12