luuid vs. forks

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

luuid vs. forks

Alexander Gladysh
Hi, list!

I see different behaviour of uuid:new() module on my Ubuntu Server
9.10 x86_64 and OS X Snow Leopard 10.6.2 boxes.

I fork a process, then call uuid:new() both in parent and in child. On
Ubuntu I get different results, on OS X generated UUIDs are the same.

I can't remember how I installed libuuid on OS X — I just remember
that I had some big problems with it. So, my OS X luuid installation
may be broken. Or is it the expected behaviour? Is there a workaround
then?

See details below.

I use April 2009 version of luuid.

Alexander.

The test code:

$ cat uuid-fork.lua
local posix = require 'posix'
local uuid = require 'uuid'

if assert(posix.fork()) ~= 0 then
  io.write("parent 1: ", uuid.new(), "\n")
  io.write("parent 2: ", uuid.new(), "\n")
  io.write("parent 3: ", uuid.new(), "\n")
  io.write("parent 4: ", uuid.new(), "\n")
  io.write("parent 5: ", uuid.new(), "\n")
else
  io.write("child  1: ", uuid.new(), "\n")
  io.write("child  2: ", uuid.new(), "\n")
  io.write("child  3: ", uuid.new(), "\n")
  io.write("child  4: ", uuid.new(), "\n")
  io.write("child  5: ", uuid.new(), "\n")
end

The output on OS X box:

$ lua -lluarocks.require uuid-fork.lua
parent 1: DF9C3E77-1ACB-4B15-81C8-D0E3AB0A7B4B
parent 2: 68062AA2-721D-43FF-87D6-9007F9265C2A
parent 3: 39EE9673-BE8D-4EB0-8C1D-2D22CBE94920
parent 4: B895873C-B589-4944-AE16-39D811E28E5A
parent 5: DE5A1755-096B-4656-8001-F280DBB33358
child  1: DF9C3E77-1ACB-4B15-81C8-D0E3AB0A7B4B
child  2: 68062AA2-721D-43FF-87D6-9007F9265C2A
child  3: 39EE9673-BE8D-4EB0-8C1D-2D22CBE94920
child  4: B895873C-B589-4944-AE16-39D811E28E5A
child  5: DE5A1755-096B-4656-8001-F280DBB33358

The output on Ubuntu box:

$ lua -lluarocks.require uuid-fork.lua
parent 1: 43276d7a-ec33-418d-bd99-27bb27ed4ed8
parent 2: 58553e1e-81da-41fc-8dab-50c3d9bf8dcc
parent 3: 461a9247-876a-41f7-add5-87b4572d7740
parent 4: cbe66c54-43cd-4225-b37c-a801503606c8
child  1: 15c32294-add7-4a47-82ee-89c019667cb4
parent 5: c1ec9f2f-b1ba-4851-a718-e0d39073d2c0
child  2: 3ece9152-49f1-4133-803d-5842390d63e4
child  3: fda13f6c-b7ac-407b-abac-5b913b519c70
child  4: 16450b69-33ed-4402-a6e0-869c86477213
child  5: 7d5a4920-8bf9-4c3e-9522-d09574b4aece
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Sam Roberts
On Mon, Feb 8, 2010 at 12:27 PM, Alexander Gladysh <[hidden email]> wrote:
> Hi, list!
>
> I see different behaviour of uuid:new() module on my Ubuntu Server
> 9.10 x86_64 and OS X Snow Leopard 10.6.2 boxes.
>
> I fork a process, then call uuid:new() both in parent and in child. On
> Ubuntu I get different results, on OS X generated UUIDs are the same.

I'm not seeing what you think the problem is. Every UUID generated should
be different from every other, and they look to be, and the order of
stdout depends
on OS scheduling.

What about the behaviour concerns you?

Cheers,
Sam
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Alexander Gladysh
>> I fork a process, then call uuid:new() both in parent and in child. On
>> Ubuntu I get different results, on OS X generated UUIDs are the same.

> I'm not seeing what you think the problem is. Every UUID generated should
> be different from every other, and they look to be, and the order of
> stdout depends
> on OS scheduling.

> What about the behaviour concerns you?

Sorry for not being clear. On OS X box UUID series in parent and child processes are identical. I'll use fixed width font (sorry for HTML letter) and reorder lines in my example, so it is more apparent:

$ lua -lluarocks.require uuid-fork.lua
parent 1: DF9C3E77-1ACB-4B15-81C8-D0E3AB0A7B4B
child  1: DF9C3E77-1ACB-4B15-81C8-D0E3AB0A7B4B
parent 2: 68062AA2-721D-43FF-87D6-9007F9265C2A
child  2: 68062AA2-721D-43FF-87D6-9007F9265C2A
parent 3: 39EE9673-BE8D-4EB0-8C1D-2D22CBE94920
child  3: 39EE9673-BE8D-4EB0-8C1D-2D22CBE94920
parent 4: B895873C-B589-4944-AE16-39D811E28E5A
child  4: B895873C-B589-4944-AE16-39D811E28E5A
parent 5: DE5A1755-096B-4656-8001-F280DBB33358
child  5: DE5A1755-096B-4656-8001-F280DBB33358

This looks wrong to me. Also it (indirectly) breaks my tests. So I want to ask — is this normal, or my setup is broken?

Alexander.
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Petite Abeille

On Feb 8, 2010, at 9:49 PM, Alexander Gladysh wrote:

> So I want to
> ask — is this normal, or my setup is broken?

not normal, broken setup :)

Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Luiz Henrique de Figueiredo
In reply to this post by Alexander Gladysh
> I fork a process, then call uuid:new() both in parent and in child. On
> Ubuntu I get different results, on OS X generated UUIDs are the same.

Perhaps try using uuid.new("random").
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Alexander Gladysh
On Tue, Feb 9, 2010 at 00:03, Luiz Henrique de Figueiredo
<[hidden email]> wrote:
>> I fork a process, then call uuid:new() both in parent and in child. On
>> Ubuntu I get different results, on OS X generated UUIDs are the same.

> Perhaps try using uuid.new("random").

Unfortunately, that does not help, UUID sequences are still the same.

Thank you,
Alexander.
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Florian Weimer
In reply to this post by Alexander Gladysh
* Alexander Gladysh:

> This looks wrong to me. Also it (indirectly) breaks my tests. So I want to
> ask — is this normal, or my setup is broken?

Your setup is broken, or more likely, the underlying random number
generator.  fork() makes creating a good one really tricky on UNIX.
RAND_bytes() from OpenSSL allegedly gets this right, or you can open
/dev/urandom directly---but you have to make sure you've disabled
buffered I/O in that case.
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Alexander Gladysh
>> This looks wrong to me. Also it (indirectly) breaks my tests. So I want to
>> ask — is this normal, or my setup is broken?

> Your setup is broken, or more likely, the underlying random number
> generator.  fork() makes creating a good one really tricky on UNIX.
> RAND_bytes() from OpenSSL allegedly gets this right, or you can open
> /dev/urandom directly---but you have to make sure you've disabled
> buffered I/O in that case.

Thanks.

I'd like to avoid writing my own libuuid for OS X though...

I figured out that in my setup I'm using Apple's builtin libuuid (that
is why generated UUID are in all-caps for instance).

Have anyone else used luuid Lua module on OS X? How did you built it?

Alexander.
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Florian Weimer
* Alexander Gladysh:

>>> This looks wrong to me. Also it (indirectly) breaks my tests. So I want to
>>> ask — is this normal, or my setup is broken?
>
>> Your setup is broken, or more likely, the underlying random number
>> generator.  fork() makes creating a good one really tricky on UNIX.
>> RAND_bytes() from OpenSSL allegedly gets this right, or you can open
>> /dev/urandom directly---but you have to make sure you've disabled
>> buffered I/O in that case.
>
> Thanks.
>
> I'd like to avoid writing my own libuuid for OS X though...

Ah, I see.  Obviously, that library isn't fork-safe.  I think on Mac
OS X, the focus is on threading and not so much on forking, so you're
better off switching from a fork() to a fork()/execve() combination.
Chances are that other libraries you might use don't survive forking,
either.

fork() is unsafe, really.
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Javier Guerra Giraldez
In reply to this post by Alexander Gladysh
On Mon, Feb 8, 2010 at 3:49 PM, Alexander Gladysh <[hidden email]> wrote:
> This looks wrong to me. Also it (indirectly) breaks my tests. So I want to
> ask — is this normal, or my setup is broken?
> Alexander.

it _is_ very wrong.  what happens if you run the program a second
time? do you get the same sequence? or is it different each time but
the same on both forks()?

if it's the same sequence on each time, you have to seed the
pseudorandom generator (PRG).  it's a common pitfall on most PRGs.
(others auto-seed on first use if you don't do it yourself, usually
with a time-microdependent value)

if it's different each time, but always the same on both forks, you
have to figure where are you seeding the PRG and make sure it happens
_after_ the fork (or that it happens again, with a different seed, of
course).

--
Javier
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Alexander Gladysh
In reply to this post by Florian Weimer
>> I'd like to avoid writing my own libuuid for OS X though...

> Ah, I see.  Obviously, that library isn't fork-safe.  I think on Mac
> OS X, the focus is on threading and not so much on forking, so you're
> better off switching from a fork() to a fork()/execve() combination.

Not a good news.

> Chances are that other libraries you might use don't survive forking,
> either.

I rarely need to use the same library both in child and parent. I do
forking in tests only. The test code is executed in parent process,
child forks are used to run services being tested.

> fork() is unsafe, really.

Well, unless there is some trick to transform fork() to
fork()/execve() without rewriting existing code, replacing it is not
an option.

I'd more likely drop OS X as a supported platform — I'm only using it
for development anyway.

However, I'd like keep OS X support. Perhaps there is fork-safe
libuuid implementation for OS X? MacPorts do have something named
ossp-uuid, perhaps I should try that...

Thank you,
Alexander.
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Alexander Gladysh
In reply to this post by Javier Guerra Giraldez
>> This looks wrong to me. Also it (indirectly) breaks my tests. So I want to
>> ask — is this normal, or my setup is broken?

> it _is_ very wrong.  what happens if you run the program a second
> time? do you get the same sequence? or is it different each time but
> the same on both forks()?

I get different sequences each time I run the program. But they are
the same on both forks.

> if it's the same sequence on each time, you have to seed the
> pseudorandom generator (PRG).  it's a common pitfall on most PRGs.
> (others auto-seed on first use if you don't do it yourself, usually
> with a time-microdependent value)

> if it's different each time, but always the same on both forks, you
> have to figure where are you seeding the PRG and make sure it happens
> _after_ the fork (or that it happens again, with a different seed, of
> course).

Thank you. But, unfortunately, I do not see a way to seed libuuid's
PRG at all. It seems that the library does that by itself.

Alexander.
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Petite Abeille
In reply to this post by Alexander Gladysh

On Feb 8, 2010, at 10:24 PM, Alexander Gladysh wrote:

> Have anyone else used luuid Lua module on OS X? How did you built it?

FWIW, works for me. Using e2fsprogs-1.41.9/lib/uuid + LHF's luuid binding.

% uname -a
Darwin 10.2.0 Darwin Kernel Version 10.2.0: Tue Nov  3 10:37:10 PST 2009; root:xnu-1486.2.11~1/RELEASE_I386 i386

% lua -v
Lua 5.2.0 (work2)  Copyright (C) 1994-2010 Lua.org, PUC-Rio

% lua uuid-fork.lua
parent 1: afd6a9b8-2099-495f-b0e2-ac602f48ef58
parent 2: 028998a1-da70-4a23-b07d-5faae2f5715a
parent 3: acbf15a3-645d-4c89-a86c-8317a8178994
parent 4: 041305ab-95b2-43c2-bec1-f3c3ba7760aa
parent 5: a5d5e5e4-db07-4b81-bc93-524682c93126
child  1: 8c2a6daf-5aeb-4c9b-b3a5-555ce3d3278e
child  2: 2154eb9f-83d5-4c52-89ff-e3ec592f798d
child  3: dca5343f-d91d-49ee-87e2-2c6c6f198e53
child  4: a14c58c9-5c47-4025-8863-02537cd4d910
child  5: eb469afe-106f-4768-a9d4-21b42200f86e




Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Sam Roberts
In reply to this post by Alexander Gladysh
On Mon, Feb 8, 2010 at 1:50 PM, Alexander Gladysh <[hidden email]> wrote:
>>> This looks wrong to me. Also it (indirectly) breaks my tests. So I want to
>>> ask — is this normal, or my setup is broken?
>
>> it _is_ very wrong.  what happens if you run the program a second
>> time? do you get the same sequence? or is it different each time but
>> the same on both forks()?
>
> I get different sequences each time I run the program. But they are
> the same on both forks.

It's possible libuuid is defaulting to the time-based algorithm on OS
X, which IIRC uses an internal count so
consecutive UUIDs are different, but if the system time doesn't
change, which it might not if 5 quick calls can be made within the
timer resolution, the UUIDs can be the same for multiple
almost-simultaneous uses of the generator.

You could try sleeping for a few seconds in the parent, and see if
that gives you different UUIDs. If its defaulting to time-based, using
the random-based algorithm might work better for you.

If you are using roberto's luuid binding, it looks like you can choose
the algorithm:

{
 uuid_t c;
 char s[2*sizeof(c)+4+1];
 const char *t=luaL_optstring(L,1,NULL);
 if (t==NULL) uuid_generate(c); else
 if (*t=='r') uuid_generate_random(c); else
 if (*t=='t') uuid_generate_time(c); else uuid_generate(c);
 uuid_unparse(c,s);
 lua_pushlstring(L,s,sizeof(s)-1);
 return 1;
}

> Thank you. But, unfortunately, I do not see a way to seed libuuid's
> PRG at all. It seems that the library does that by itself.

Since you don't have source for apple's libuuid, its hard to know what
it's doing. Have you checked if it has docs? Or maybe looked at its
header file? There might be a clue, you might at least learn that it
is an opensource lib, of a particular version. If you find the code it
would probably be immediately obvious what is going wrong.

Cheers,
Sam
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Bruno Silvestre-2
On Mon, Feb 8, 2010 at 8:12 PM, Sam Roberts <[hidden email]> wrote:
>
> It's possible libuuid is defaulting to the time-based algorithm on OS
> X, which IIRC uses an internal count so
> consecutive UUIDs are different, but if the system time doesn't
> change, which it might not if 5 quick calls can be made within the
> timer resolution, the UUIDs can be the same for multiple
> almost-simultaneous uses of the generator.

On my Linux box, libuuid (e2fsprogs) starts a daemon (uuidd) that
generates the UUIDs. So, two or more processes do not generate/receive
the same time-based UUID.

cheers
--
bruno
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Alexander Gladysh
In reply to this post by Petite Abeille
Sorry for the late reply.

>> Have anyone else used luuid Lua module on OS X? How did you built it?

> FWIW, works for me. Using e2fsprogs-1.41.9/lib/uuid + LHF's luuid binding.

Do you remember how you installed e2fsprogs?

Alexander.
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Luiz Henrique de Figueiredo
> > FWIW, works for me. Using e2fsprogs-1.41.9/lib/uuid + LHF's luuid binding.
>
> Do you remember how you installed e2fsprogs?

It works for me even without installing e2fsprogs. Just downloadit and do
        cd e2fsprogs-1.41.10
        ./configure
        cd lib/uuid
        make
        ./tst_uuid

Now change the luuid Makefile to use e2fsprogs-1.41.10/lib/uuid/libuuid.a.
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Alexander Gladysh
On Sun, Feb 28, 2010 at 23:28, Luiz Henrique de Figueiredo
<[hidden email]> wrote:
>> > FWIW, works for me. Using e2fsprogs-1.41.9/lib/uuid + LHF's luuid binding.

>> Do you remember how you installed e2fsprogs?

> It works for me even without installing e2fsprogs. Just downloadit and do
>        cd e2fsprogs-1.41.10
>        ./configure
>        cd lib/uuid
>        make
>        ./tst_uuid

> Now change the luuid Makefile to use e2fsprogs-1.41.10/lib/uuid/libuuid.a.

Thanks, that helped.

I wish luuid's rock worked out of the box on OS X, but this workaround
would do for now.

Alexander.
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Luiz Henrique de Figueiredo
> It works for me even without installing e2fsprogs.

However, it seems that uuid.time is not working: it always returns -1.
Reply | Threaded
Open this post in threaded view
|

Re: luuid vs. forks

Luiz Henrique de Figueiredo
> > It works for me even without installing e2fsprogs.
>
> However, it seems that uuid.time is not working: it always returns -1.

Oops, my bad. I was compiling lluuid with /usr/include/uuid/uuid.h instead of
e2fsprogs-1.41.10/lib/uuid/uuidd.h. The system version of uuid.h in Mac OS X
10.5.8 does not define UUID_VARIANT_OTHER and luuid.c was stubbbing some
functions out.

Perhaps I should drop this test in luuid.c.
12