Re: fcntl always fails to delete lock file, and PID is always -6464

From: Garrett Cooper <gcooper_at_FreeBSD.org>
Date: Mon, 4 Oct 2010 20:17:08 -0700
On Mon, Oct 4, 2010 at 5:38 PM, Daichi GOTO <daichi_at_ongs.co.jp> wrote:
> On Mon, 4 Oct 2010 07:19:45 -0700
> Garrett Cooper <gcooper_at_FreeBSD.org> wrote:
>> >> issues that might be occurring with the software, as per my copy of
>> >> SUSv4 (see the ERRORS section of fcntl). I would print out the
>> >> strerror for that case.
>> >>     Providing a backtrace of the application's execution and the
>> >> architecture and what version of FreeBSD you're using would be
>> >> helpful.
>>
>>     I'm not even getting that far. Logs attached from both runs
>> (WITH_DEBUG_CODE and WITHOUT_DEBUG_CODE).
>
> Yeah, it looks like the same situation.
>
>  1) mozc_server was killed
>      lock file remains  (even though it should be removed)
>  2) mozc_server try to boot
>    1. check lock file there
>    2. there is lock file, so cannot get lock file via fcntl
>    3. lock file means there is another mozc_server running,
>       so mozc_server will stop boot and finish

    Ok, weird. fstat on the file didn't yield anything nasty when I
ran the app, and deleting the file in /tmp allowed the server to go a
ways, then die, as opposed to die quickly, like what happened on the
second try.

> The cause of problem is that kernel does not remove lock file
> after mozc_server killed. Mozc developer explained me that
> fcntl will remove lock file after that process killed. But
> it looks like fnctl() does not remove lock file itself. According
> to FreeBSD fcntl(2) manual:
>
>     All locks associated with a file for a given process are removed when the
>     process terminates.
>
> No explanation lock file removing. Does FreeBSD fnctl(2) not remove lock file
> after process killed?  Apparently from Mozc developer, Linux kernel removes
> lock files after process killed.

    On Linux (RHEL 4.8):

Window 1:
$ ls -l /tmp/lockfile
ls: /tmp/lockfile: No such file or directory
$ ./test_fcntl

Window 2:

$ ls -l /tmp/lockfile
--wxr-s--T  1 garrcoop eng 0 Oct  4 19:49 /tmp/lockfile
$ ./test_fcntl
test_fcntl: fcntl: Resource temporarily unavailable

Ok. This (EAGAIN) matches the Linux requirements specified in the
manpage [1] I found, as well as the POSIX manpage [2]. The author is
wrong about fcntl removing the file at exit though:

$ ls -l /tmp/lockfile
--wxr-s--T  1 garrcoop eng 0 Oct  4 19:49 /tmp/lockfile

The file descriptor is closed though, so I can remove it at will:

$ rm /tmp/lockfile
$ ls -l /tmp/lockfile
ls: /tmp/lockfile: No such file or directory

Following through the same process on FreeBSD...

Window 1:
$ ls -l /tmp/lockfile
ls: /tmp/lockfile: No such file or directory
$ ./test_fcntl

Window 2:

$ ls -l /tmp/lockfile
-rwsr-x---  1 garrcoop  wheel  0 Oct  4 20:14 /tmp/lockfile
$ ./test_fcntl
test_fcntl: fcntl: Resource temporarily unavailable

Well, lookie here! It locked as expected :).

$ ls -l /tmp/lockfile
-rwsr-x---  1 garrcoop  wheel  0 Oct  4 20:14 /tmp/lockfile
$ rm /tmp/lockfile
$ ls -l /tmp/lockfile
ls: /tmp/lockfile: No such file or directory

So something else is going on with the application that needs to be
resolved in that area.

With that aside though, after modifying the test app a bit, I'm
confused at the value of l_pid...

Window 1:
$ ./test_fcntl
My pid: 5372

Window 2:
$ ./test_fcntl
My pid: 5373
test_fcntl: fcntl: Resource temporarily unavailable
PID=1 has the lock

    Huh...? init has the file locked...? WTF?!
    So assuming Occam's Razor, I did a bit more reading and it turns
out that l_pid is only populated when you call with F_GETLK:

     negative, l_start means end edge of the region.  >>> The l_pid and l_sysid
     fields are only used with F_GETLK to return the process ID of the process
     holding a blocking lock and the system ID of the system that owns that
     process.  Locks created by the local system will have a system ID of
     zero.  <<< After a successful F_GETLK request, the value of l_whence is
     SEEK_SET.

    Thus, after fixing the test app I'm getting a sensical value:

Window 1:
$ ./test_fcntl
My pid: 5394

Window 2:
$ ./test_fcntl
My pid: 5395
test_fcntl: fcntl[1]: Resource temporarily unavailable
PID=5394 has the lock

Linux operates in the same manner:

Window 1:
$ ./test_fcntl
My pid: 17861

Window 2:
$ ./test_fcntl
My pid: 17963
test_fcntl: fcntl[1]: Resource temporarily unavailable
PID=17861 has the lock

    Which I would expect because I'm not using anything exotic with
fcntl(2) / open(2).
    I suspect mozc isn't properly initializing / calling fcntl(2), or
the author used a non-POSIX extension that is implementation dependent
and doesn't realize it (the Linux manpage has a pretty fat set of
warnings about POSIX compatibility up at the top of the manpage). The
developer might also want to use O_EXCL in the flags passed to open(2)
as well, unless they want to lock specific sections in the file.
    Verified on UFS2 with SUJ. Test app attached.

>> $ uname -a
>> FreeBSD bayonetta.local 9.0-CURRENT FreeBSD 9.0-CURRENT #9 r211309M:
>> Thu Aug 19 22:50:36 PDT 2010
>> root_at_bayonetta.local:/usr/obj/usr/src/sys/BAYONETTA  amd64
>>
>>     I completely blasted past the part of your reply above where you
>> said your home directory is served up via NFS. It might be a problem
>> if you don't have lockd running (/etc/rc.d/lockd onestatus ? It isn't
>> enabled by default, and definitely isn't on on my machine) or the
>> mount isn't setup with lockd on the client side (nolockd will do this
>> on the initial mount, according to the manpage). There might be
>> `dragons' in the nfsd code that fail to do locking properly, but I
>> think that Rick (rmacklem_at_) or someone else on the list might be
>> better at answering whether or not things work from an NFS
>> perspective.
>
> server side:
>  FreeBSD 7.3-PRERELEASE #0: Mon Mar  1 15:10:07 JST 2010 i386
>  rc.conf
>    nfs_server_enable="YES"
>    mountd_enable="YES"
>    nfs_reserved_port_only="YES"
>    rpc_lockd_enable="YES"
>    rpc_statd_enable="YES"
>
> client side:
>  FreeBSD 9.0-CURRENT #6 r213257: Thu Sep 30 10:30:06 JST 2010 amd64
>  rc.conf:
>    nfs_client_enable="YES"
>    nfs_reserved_port_only="YES"
>    rpc_lockd_enable="YES"
>    rpc_statd_enable="YES"
>
>>     I'd definitely divulge which version of NFS you're using as well
>> as what your NFS server and client are running if enabling lockd both
>> client and server side doesn't solve your problems right away.

[...]

> I have tested with ZFS because I was doubting NFS working well,
> but result was the same. (I didn't test with UFS.)
>
> Thanks truss output!

No problem :).

Cheers,
-Garrett

[1] http://linux.die.net/man/2/fcntl
[2] http://www.opengroup.org/onlinepubs/009695399/functions/fcntl.html

Received on Tue Oct 05 2010 - 01:17:10 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:08 UTC