Pooling of strings is good

classic Classic list List threaded Threaded
103 messages Options
1 ... 3456
Reply | Threaded
Open this post in threaded view
|

Re: Pooling of strings is good

William Ahern
On Wed, Aug 27, 2014 at 02:30:05PM -0700, Coroutines wrote:
<snip>
> I'm thinking about making a library to expose userdata to Lua as file
> handles. You'd have userdata exist in one spot in Lua's memory and then
> you'd create the file handle which references that userdata with this
> library. This isn't doing what I'm talking about in this thread, but I
> thought it would be cool. Basically I'd use shm_open() on Linux to get an
> fd, and on Windows it'd be CreateFileMapping() (which returns a HANDLE
> that you can feed to open_osfhandle() to get an fd again). Would be kinda
> cool to operate on userdata as if it were a file or :read() strings from
> them
<snip>
> Windows: CreateFileMapping()
> Mac OS X: shm_open()
> Linux: shm_open()

I don't think shm_open provides the functionality you want because you won't
be able to wrap arbitrary blocks of memory. You'd have to first copy your
data into the shm space. shm_open + fdopen is really no different than
simply calling io.tmpfile.

POSIX has fmemopen, which is much closer to what you want. However, fmemopen
isn't very widely supported yet (so it's not portable in any practical
sense) and might be too limited for what you're trying to do, anyhow.

Even better than fmemopen are two much more flexible interfaces: 1) GNU
fopencookie, available on Linux/glibc; and 2) BSD funopen, available on OS X
and other BSDs. Both of these allow you to instantiate a real FILE handle
with user-specified function pointers for read, write, seek, and close
operations. And all those function pointers will be passed a user-specified
context object, which can be a structure with all the relevant state you
need to wrap arbitrary application objects.

funopen and fopencookie are nearly identical. It's trivial to write a thin
wrapper over both of them, and you'll be much more portable in practice than
using POSIX fmemopen or open_memstream. (OS X doesn't have fmemopen,
anyhow.)


Reply | Threaded
Open this post in threaded view
|

Re: Pooling of strings is good

William Ahern
In reply to this post by Sean Conner
On Wed, Aug 27, 2014 at 05:58:36PM -0400, Sean Conner wrote:

> It was thus said that the Great Coroutines once stated:
> > On Sat, Aug 23, 2014 at 12:47 AM, Coroutines <[hidden email]> wrote:
> >
> > > My bad, I thought I read static :\  Stack overflows are fun... :p  I
> > > was just thinking offhandedly that you could create a mmap, expose it
> > > to Lua as a file to read and write from -- and pass it to the
> > > networking lib as the buffer to recv into.  Hmm.  And it could be
> > > shared depending on the threading setup -- but that's an afterthought.
> >
> > Hmm.  This is another tangent -- mostly talking to Sean here.  
>
>   I'm stil here.
>
> > I was off-base when I said mmap() -- I haven't used mmap() often and for
> > some reason I thought I could take a [page-sized] block of memory and get
> > an fd from it.  WRONG.  I was really thinking of shm_open().
>
> shm_open() doesn't do what you think it does.  
>
> int shm_open(const char *name, int oflag, mode_t mode);
>
>   That "name" there is the name of a file; if the name doesn't exist, a file
> is created with that name (and I'm checking the man pages of three different
> systems here).  
>
>   No, what you want is open_memstream().  It's a POSIX function, so it'll
> exist on Linux and Mac.  I don't know what you would do under Windows, but I
> would double check the documentation for CreateFileMapping() to make sure it
> does what you think it does.

OS X doesn't have open_memstream. fmemopen and open_memstream were added to
POSIX.1-2008, but OS X is only nominally compliant to POSIX.1-2001**, and
supports very few newer interfaces.

See my other post elsethread regarding fopencookie and funopen.


** I say nominally because OS X doesn't actually implement some interfaces,
such as pselect (POSIX.1-2001), according to the specification. The Open
Group's conformance testing is broken.

Reply | Threaded
Open this post in threaded view
|

Re: Pooling of strings is good

Coroutines
On Wed, Aug 27, 2014 at 4:26 PM, William Ahern
<[hidden email]> wrote:

> OS X doesn't have open_memstream. fmemopen and open_memstream were added to
> POSIX.1-2008, but OS X is only nominally compliant to POSIX.1-2001**, and
> supports very few newer interfaces.
>
> See my other post elsethread regarding fopencookie and funopen.
>
>
> ** I say nominally because OS X doesn't actually implement some interfaces,
> such as pselect (POSIX.1-2001), according to the specification. The Open
> Group's conformance testing is broken.

Mmm, thanks for the good information (both you and Sean!)  -- yeah I
think I may be wrong on shm_open().  I was reading through the shm_*()
functions last night in my copy of Advanced Linux Programming from
CodeSourcery (love this book) -- I gotta go through it again and find
the function I mean.  After Sean's post I did go look at
open_memstream() but yeah Mac OS X doesn't support this [yet].  I
really love CreateFileMapping() on Windows because even though you
have to give it a page to share it allows you to enforce the buffer
size within that page (it seems).  Anyway, all a side-project :-)  You
can tell I'm not a fan of strict/static typing since I love to "duck"
through interfaces, haha

1 ... 3456