Megapatches (Was: Lua Foundation?)

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Megapatches (Was: Lua Foundation?)

Dirk Laurie-2
2017-04-23 0:24 GMT+02:00 Paige DePol <[hidden email]>:

> I have not had much time in the past couple years for hacking Lua due to
> personal and medical reasons, however, things have finally stabilised and I
> have begun work on updating my patch collection to the latest version of
> Lua. I will have an update and a mega-patch to release soon... the patch is
> 4'ish patches in one, however, I do plan to break down all my hacks into
> discreet individual patches as well.

You have my permission (as does anyone else making a mega-patch too,
of course) to include any of my patches.[1] To this end, may I suggest the
named-library patch, which allows you to switch between compatible
libraries on the command line?

   lua -l json=dkjson myapp.lua
   lua -l json=rapidjson myapp.lua

The patch is a very recent, if not the most recent, addition to
[http://lua-users.org/wiki/LuaPowerPatches the PowerPatches page].

Regards
Dirk

[1] Not that my permission is needed, of course, but people tend to
be polite.

Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Paige DePol
Dirk Laurie <[hidden email]> wrote:

> 2017-04-23 0:24 GMT+02:00 Paige DePol <[hidden email]>:
>
>> I have not had much time in the past couple years for hacking Lua due to
>> personal and medical reasons, however, things have finally stabilised and I
>> have begun work on updating my patch collection to the latest version of
>> Lua. I will have an update and a mega-patch to release soon... the patch is
>> 4'ish patches in one, however, I do plan to break down all my hacks into
>> discreet individual patches as well.
>
> You have my permission (as does anyone else making a mega-patch too,
> of course) to include any of my patches.[1] To this end, may I suggest the
> named-library patch, which allows you to switch between compatible
> libraries on the command line?
>
>   lua -l json=dkjson myapp.lua
>   lua -l json=rapidjson myapp.lua
>
> The patch is a very recent, if not the most recent, addition to
> [http://lua-users.org/wiki/LuaPowerPatches the PowerPatches page].
>
> Regards
> Dirk
>
> [1] Not that my permission is needed, of course, but people tend to
> be polite.

I always do my utmost to keep track of places I obtain source code from,
including repository links, or links to places like StackOverflow.

The mega-patch I spoke of consists solely of my own code currently... and,
upon second glance, if I were to break everything out into discreet patches
it would be as follows:

Preprocessor
Token Storage
Constants, Enumerations, Macros, and Inlines (requires both above)
'nil' Assignments (replaces _ idiom)
Index Data Type (requires all the above, expands Instructions to 64-bit)
C Macro Enhancements (for expkind, expdesc, and other areas)
Debugging Enhancements (in ltests.c; malloc tracing, stack dumping, etc)
Test Suite (separate patch file for adding tests for all new code)

There may be a couple other tiny patches in there as well, but those are the
major patch areas... a bit more than the 4'ish I initially estimated! ;)

I will provide full documentation for using the features, and the code
itself is very heavily documented as well. I figured those who may be
interested in hacking Lua could be interested in some of this code, so
I may as well make it easy to understand! :)

~Paige

PS: Thank you for your permission, I may not have "technically" required it,
but it is appreciated nonetheless! :)


Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

steve donovan
On Tue, Apr 25, 2017 at 9:19 AM, Paige DePol <[hidden email]> wrote:
> There may be a couple other tiny patches in there as well, but those are the
> major patch areas... a bit more than the 4'ish I initially estimated! ;)

I'm curious about how (semi) automated patching could become.

The background is this: I did luabuild because I wanted to create a
straightforward toolkit for creating custom versions of Lua, either
adding external modules to the core, or leaving out standard modules.
(It would then leverage soar and srlua to make standalone executables
with programs, but that is another story).  So I was thinking about
how to raise that to the next level, and allow a person to add a few
patches as well.

It's a hard problem in general because patches can trample on each
other so easily. So what's needed is some kind of compatibility matrix
so a machine could deduce what combination of patches is going to work
as expected.

Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Paige DePol
steve donovan <[hidden email]> wrote:

> On Tue, Apr 25, 2017 at 9:19 AM, Paige DePol <[hidden email]> wrote:
>> There may be a couple other tiny patches in there as well, but those are the
>> major patch areas... a bit more than the 4'ish I initially estimated! ;)
>
> I'm curious about how (semi) automated patching could become.
>
> The background is this: I did luabuild because I wanted to create a
> straightforward toolkit for creating custom versions of Lua, either
> adding external modules to the core, or leaving out standard modules.
> (It would then leverage soar and srlua to make standalone executables
> with programs, but that is another story).  So I was thinking about
> how to raise that to the next level, and allow a person to add a few
> patches as well.
>
> It's a hard problem in general because patches can trample on each
> other so easily. So what's needed is some kind of compatibility matrix
> so a machine could deduce what combination of patches is going to work
> as expected.

Actually, one of my goals is to generate just such a system!

I plan to place all my patches into a matrix, and where necessary, generate
additional patches to resolve conflicts between other patches. Then I plan
to create a website with a listing of patches that can dynamically be
enabled or disabled to create custom versions of Lua that can be downloaded,
or perhaps even supplied as binaries for a subset of targets. Also, this
way, prerequisite patches can be tracked easily for subsequent patches.

For external patches, I would probably have to curate them into the system
as I am sure there would be conflicts that could not be fixed automatically
by software... but I don't think a system like this is outside the realm of
possibility. As long as all patches work from a common baseline version of
Lua it shouldn't be hard to generate most of the work automatically, with
some work required for conflicts. Then again, I could wind up mired down in
a sea of patches... we shall see! ;)

I was thinking of naming the custom version of Lua built by such a system
"Modlua"[1], sort of a play on the language "Modula". I was also thinking
that perhaps a version of Lua could be built with a modular plugin design,
so extensions could be created via a plugin architecture... but that is
something I am still working out the details of in my mind.

If anyone has experience building such a patch matrix and has any sage
advice about creating and maintaining such a system please feel free to
share... offlist is fine if such a conversation is considered offtopic for
the list in general!

~Paige

[1] I hereby reserve the name "Modlua" ;)


Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Sergey Zakharchenko
Hello,

2017-04-25 11:24 GMT+03:00 Paige DePol <[hidden email]>:
 As long as all patches work from a common baseline version of
Lua it shouldn't be hard to generate most of the work automatically

You might be in for some surprises, e.g. https://github.com/elua/elua/pull/71

Best regards,

--
DoubleF
Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

steve donovan
In reply to this post by Paige DePol
On Tue, Apr 25, 2017 at 10:24 AM, Paige DePol <[hidden email]> wrote:
> steve donovan <[hidden email]> wrote:
> "Modlua"[1], sort of a play on the language "Modula". I was also thinking
> that perhaps a version of Lua could be built with a modular plugin design,
> so extensions could be created via a plugin architecture... but that is
> something I am still working out the details of in my mind.

A cool ambition, sir! I like the name....

There really are some cool patches out there, but applying them is a
little fiddly.

Extending tight, focused C in a modular way is definitely a
non-trivial problem!  The obvious bastards would be patches that touch
multiple subsystems.

Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Paige DePol
In reply to this post by Sergey Zakharchenko
Sergey Zakharchenko <[hidden email]> wrote:

> Hello,
>
> 2017-04-25 11:24 GMT+03:00 Paige DePol <[hidden email]>:
>  As long as all patches work from a common baseline version of
> Lua it shouldn't be hard to generate most of the work automatically
>
> You might be in for some surprises, e.g. https://github.com/elua/elua/pull/71
>
> Best regards,
>
> --
> DoubleF

Well, by "baseline" I mean that I will use a specific version of Lua, for example 5.3.4, not just allow the patch matrix to work on any arbitrary version of Lua as that would be a nightmare. Though, I am not sure what the GitHub link you provided was meant to demonstrate?

The patch matrix would inherently require all patches be set up so they all apply to the baseline patch without any issues, then I would start applying patches together, in a specific hierarchy, and then generate conflict patches where necessary.

Eventually, I should have a collection of patches, and patches-for-patches, that should allow any combinations of patches in the matrix to be applied with any other appropriate patches.

~Paige


Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Paige DePol
In reply to this post by steve donovan
steve donovan <[hidden email]> wrote:

> On Tue, Apr 25, 2017 at 10:24 AM, Paige DePol <[hidden email]> wrote:
>> steve donovan <[hidden email]> wrote:
>> "Modlua"[1], sort of a play on the language "Modula". I was also thinking
>> that perhaps a version of Lua could be built with a modular plugin design,
>> so extensions could be created via a plugin architecture... but that is
>> something I am still working out the details of in my mind.
>
> A cool ambition, sir! I like the name....
>
> There really are some cool patches out there, but applying them is a
> little fiddly.
>
> Extending tight, focused C in a modular way is definitely a
> non-trivial problem!  The obvious bastards would be patches that touch
> multiple subsystems.

*Looks at my name* Sir? ;)

Yes, patches that touch multiple systems will be a pain... for now I am just
letting my brain percolate on the idea in the background while I play around
with hacking Lua and creating patches.

I am working towards creating my own derivative of Lua, mostly because I
really wanted to learn how the whole system worked, and what better way than
hacking the source code? :)

I am sure a bunch of my patches will be considered unnecessary, especially
given the minimalist design ethos of Lua. However, a design goal of my Lua
version is to provide the ability to preprocess back into vanilla Lua source
code, which will really help with benchmarking my features vs vanilla Lua!

With some of my ideas, like the Index Data Type, I am working towards
allowing faster access to frequently used data structures, providing a
defined object model, providing a defined interface between C and Lua, and
eventually all these definitions should help when I get JIT working as well.

Other things, like my RegEx implementation are because I just like RegEx's
and wanted them in Lua... even if the RegEx engine[1] is as big as Lua! ;)

I am working on the object model code right now, and hope to have something
for release soon.

~Paige

[1] Using PCRE (Perl Compatible Regular Expressions)


Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Sergey Zakharchenko
In reply to this post by Paige DePol


2017-04-25 11:54 GMT+03:00 Paige DePol <[hidden email]>:
 Though, I am not sure what the GitHub link you provided was meant to demonstrate?

That was a fix to a problem where two otherwise "fine" (running without problems) patches collide without any source control conflicts, just semantically.

Eventually, I should have a collection of patches, and patches-for-patches, that should allow any combinations of patches in the matrix to be applied with any other appropriate patches.

That will likely lead to combinatorial explosion...

Best regards,

--
DoubleF
Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Paige DePol
Sergey Zakharchenko <[hidden email]> wrote:

>
>
> 2017-04-25 11:54 GMT+03:00 Paige DePol <[hidden email]>:
>  Though, I am not sure what the GitHub link you provided was meant to demonstrate?
>
> That was a fix to a problem where two otherwise "fine" (running without problems) patches collide without any source control conflicts, just semantically.
>
> Eventually, I should have a collection of patches, and patches-for-patches, that should allow any combinations of patches in the matrix to be applied with any other appropriate patches.
>
> That will likely lead to combinatorial explosion...
>
> Best regards,
>
> --
> DoubleF

Ahh, okay, now I understand what that link was meant to demonstrate, thanks
for the explanation! :)

However, keep in mind, that to start with, all patches in my matrix will be
patches that have been created by me, not just arbitrary patches from
anywhere. Given that I will have written all the code it is more likely my
own patches will play nicer with each other than random patches from other
people. However, that said, I am sure I will run into some issues along the
way... but part of the fun of coding is solving problems! ;)

Once I have all my patches upgraded to the latest Lua I will work on
creating a matrix for them all and will see just how many combinations of
patches there are to test against each other... the answer will probably
scare me! ;)

~Paige


Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Sean Conner
In reply to this post by Paige DePol
It was thus said that the Great Paige DePol once stated:
> steve donovan <[hidden email]> wrote:
>
> > A cool ambition, sir! I like the name....
>
> *Looks at my name* Sir? ;)

  Perhaps like Marcy always calling Peppermint Patty "sir".

  Also, I think in the US military, everybody is called "sir" regardless of
gender.

  -spc

Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Ross Berteig
In reply to this post by steve donovan

On 4/25/2017 12:47 AM, steve donovan wrote:

> On Tue, Apr 25, 2017 at 9:19 AM, Paige DePol <[hidden email]> wrote:
>> There may be a couple other tiny patches in there as well, but those are the
>> major patch areas... a bit more than the 4'ish I initially estimated! ;)
> I'm curious about how (semi) automated patching could become.
>
> The background is this: I did luabuild because I wanted to create a
> straightforward toolkit for creating custom versions of Lua, either
> adding external modules to the core, or leaving out standard modules.
> (It would then leverage soar and srlua to make standalone executables
> with programs, but that is another story).  So I was thinking about
> how to raise that to the next level, and allow a person to add a few
> patches as well.
>
> It's a hard problem in general because patches can trample on each
> other so easily. So what's needed is some kind of compatibility matrix
> so a machine could deduce what combination of patches is going to work
> as expected.

For a working example of the kind of thing you are imagining, check out
the Yocto Project[1], which is a build system for Linux distros, where
the entire distro is built fresh by downloading from upstream sources,
patching, building, installing, and ultimately creating disk images.

It works.

But man is it a complex beast to find your way into.

There's likely lots of lessons to learn from there.

--

Ross Berteig                               [hidden email]
Cheshire Engineering Corp.           http://www.CheshireEng.com/
+1 626 303 1602


Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Paige DePol
In reply to this post by Sean Conner
Sean Conner <[hidden email]> wrote:

> It was thus said that the Great Paige DePol once stated:
>> steve donovan <[hidden email]> wrote:
>>
>>> A cool ambition, sir! I like the name....
>>
>> *Looks at my name* Sir? ;)
>
>  Perhaps like Marcy always calling Peppermint Patty "sir".
>
>  Also, I think in the US military, everybody is called "sir" regardless of
> gender.
>
>  -spc

Well, I am not in the military, and I have no idea why Marcy called
Peppermint Patty "sir" all the time... that was just a bit strange! ;)

I'm not offended or anything, just thought it was funny since it's not like
my name is ambiguous or anything!

~Paige


Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Paige DePol
In reply to this post by Ross Berteig
Ross Berteig <[hidden email]> wrote:

> For a working example of the kind of thing you are imagining, check out the Yocto Project[1], which is a build system for Linux distros, where the entire distro is built fresh by downloading from upstream sources, patching, building, installing, and ultimately creating disk images.
>
> It works.
>
> But man is it a complex beast to find your way into.
>
> There's likely lots of lessons to learn from there.

Thank you very much for pointing me towards the Yocto Project, I will look
into it further... and actually, that is something I can see myself actually
using as I do like to run servers with only what is needed, without bloat!

As you forgot your footnote I am going to guess you mean this URL:
https://www.yoctoproject.org

~Paige


Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Sean Conner
In reply to this post by Paige DePol
It was thus said that the Great Paige DePol once stated:

> Sean Conner <[hidden email]> wrote:
>
> > It was thus said that the Great Paige DePol once stated:
> >> steve donovan <[hidden email]> wrote:
> >>
> >>> A cool ambition, sir! I like the name....
> >>
> >> *Looks at my name* Sir? ;)
> >
> >  Perhaps like Marcy always calling Peppermint Patty "sir".
> >
> >  Also, I think in the US military, everybody is called "sir" regardless of
> > gender.
> >
> >  -spc
>
> Well, I am not in the military, and I have no idea why Marcy called
> Peppermint Patty "sir" all the time... that was just a bit strange! ;)
>
> I'm not offended or anything, just thought it was funny since it's not like
> my name is ambiguous or anything!

  There's a boy named Sue that begs to differ.

  -spc (I had an aunt named Freddie, and I know a woman who's name is
        George ... )

Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

steve donovan
In reply to this post by Paige DePol
On Wed, Apr 26, 2017 at 7:39 AM, Paige DePol <[hidden email]> wrote:
> I'm not offended or anything, just thought it was funny since it's not like
> my name is ambiguous or anything!

Sorry, I was just being old-fashioned English, where 'sir' is just a
piece of politeness between equals (as in "Pray tell me sir, when did
you first hear of this engagement?"). I certainly do not think you are
knighted!  (Although it's hard for any prominent English person to
avoid this fate)

Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Paige DePol
steve donovan <[hidden email]> wrote:

> On Wed, Apr 26, 2017 at 7:39 AM, Paige DePol <[hidden email]> wrote:
>> I'm not offended or anything, just thought it was funny since it's not like
>> my name is ambiguous or anything!
>
> Sorry, I was just being old-fashioned English, where 'sir' is just a
> piece of politeness between equals (as in "Pray tell me sir, when did
> you first hear of this engagement?"). I certainly do not think you are
> knighted!  (Although it's hard for any prominent English person to
> avoid this fate)

No worries at all, really... I was just being humorous about it! :)

I would, however, accept a knighting! ;) Somehow I don't believe I'd qualify
though being as I am not English, though I did live in London for 5 years
and am a bit of an anglophile! :)

~Paige


Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Hisham
In reply to this post by Paige DePol
On 25 April 2017 at 05:24, Paige DePol <[hidden email]> wrote:
>
> [1] I hereby reserve the name "Modlua" ;)

That's a great pun, but unfortunately that name would cause confusion
in spoken conversation:

https://httpd.apache.org/docs/trunk/mod/mod_lua.html

May I suggest Modulua instead?

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Ross Berteig
In reply to this post by Paige DePol


On 4/25/2017 11:28 PM, Paige DePol wrote:
> As you forgot your footnote I am going to guess you mean this URL:
> https://www.yoctoproject.org

Oops. Yes, that is the site.

Part of the rich complexity is that they break the build into many
fine-grain phases such as fetching all the files, unpacking tarballs and
zipballs, applying patches, configure, make, test, and a number of
installation phases. Their system keeps all of the intermediate steps,
which makes it possible to figure out what happened when something goes
wrong, but also means it needs a fair bit of disk space to do a build.

My challenge was to figure out how to set up a build containing a fair
approximation of the tools we'd need, and then add to the structure
"correctly" a tool that was written in my shop for my customer's
project. That was not made easier by the wall of opaque documentation
that sort of assumes you already know what all the players are and do.
But it was helped along substantially by our chosen board vendor
(Gumstix, purveyors of fine and extremely tiny Linux computers on
modules) providing a complete set of recipes for a working Yocto system
that I could use as a starting point.

It is pretty clear from both the general Yocto community and the Gumstix
user to user support channels that people are happily using Yocto's
build system to create deeply embedded distros for routers and various
internet facing appliances.

As long as you are willing to devote lots of disk space and plenty of
run time for a complete build, it is a nifty piece of work. I'm not sure
I'd recommend my approach, which was to run my builds in a virtual
machine booting Ubuntu on my Windows PC. While it worked, it would
likely have been less fuss and bother in the end to have dedicated a PC
to the task. Aside from performance of the VM and a chronically too
small disk due to needing to take disk away from my Windows box for the
purpose, the other source of friction was managing to create a properly
formatted and configured bootable SD card from the finished Yocto image.
My expedient answer was to just scp the files to a real Linux box where
I could plug in a card reader and run the shell script that partitioned
and formatted the card. Today, I suspect there are other answers that
likely would have worked from the Windows side.

--
Ross Berteig                               [hidden email]
Cheshire Engineering Corp.           http://www.CheshireEng.com/
+1 626 303 1602


Reply | Threaded
Open this post in threaded view
|

Re: Megapatches (Was: Lua Foundation?)

Paige DePol
In reply to this post by Hisham
Hisham <[hidden email]> wrote:

> On 25 April 2017 at 05:24, Paige DePol <[hidden email]> wrote:
>> [1] I hereby reserve the name "Modlua" ;)
>
> That's a great pun, but unfortunately that name would cause confusion
> in spoken conversation:
>
> https://httpd.apache.org/docs/trunk/mod/mod_lua.html
>
> May I suggest Modulua instead?
>
> -- Hisham

Well, darn, of course, you are right... mod_lua and Modlua would be very
confusing to discuss in spoken conversation. :(

Modulua works, but that extra 'u' just doesn't roll off the tongue quite as
well as Modlua does. Ah well, I do still like your suggestion and may choose
to use it... I will have to give it some more thought!

Thank you for bringing this naming conflict to my attention, it is
appreciated! :)

~Paige


12