Re: Possible bug in or around posix_fadvise after r292326

On Tue, Jan 05, 2016 at 01:07:40PM +0200, Konstantin Belousov wrote:
> On Mon, Jan 04, 2016 at 10:05:21PM -0800, Benno Rice wrote:
> > Hi Konstantin,
> > 
> > I recently updated my dev box to r292962. After doing this I attempted to set up PostgreSQL 9.4. When I ran initdb the last phase hung. Using procstat -kk I found it appeared to be stuck in a loop inside a posix_fadvise syscall. I could not ^C or ^Z the initdb process. I could kill it but a subsequent attempt to rm -rf the /usr/local/pgsql/data directory also got stuck and was unkillable by any means. Rebooting allowed me to remove the directory but the initdb process still hung when I re-ran it.
> > 
> > I tried PostgreSQL 9.3 with similar results.
> > 
> > Looking at the source code for initdb I found that it calls posix_fadvise like so[1]:
> > 
> >      /*
> >       * We do what pg_flush_data() would do in the backend: prefer to use
> >       * sync_file_range, but fall back to posix_fadvise.  We ignore errors
> >       * because this is only a hint.
> >       */
> >  #if defined(HAVE_SYNC_FILE_RANGE)
> >      (void) sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WRITE);
> >  #elif defined(USE_POSIX_FADVISE) && defined(POSIX_FADV_DONTNEED)
> >      (void) posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED);
> >  #else
> >  #error PG_FLUSH_DATA_WORKS should not have been defined
> >  #endif
> > 
> > Looking for recent commits involving POSIX_FADV_DONTNEED I found r292326:
> > 
> > https://svnweb.freebsd.org/changeset/base/292326 <https://svnweb.freebsd.org/changeset/base/292326>
> > 
> > Backing this revision out allowed the initdb process to complete.
> > 
> > My current theory is that some how we???re getting ENOLCK or EAGAIN from the BUF_TIMELOCK call in bnoreuselist:
> > 
> > https://svnweb.freebsd.org/base/head/sys/kern/vfs_subr.c?view=annotate#l1676 <https://svnweb.freebsd.org/base/head/sys/kern/vfs_subr.c?view=annotate#l1676>
> > 
> > Leading to an infinite loop in vop_stdadvise:
> > 
> > https://svnweb.freebsd.org/base/head/sys/kern/vfs_default.c?annotate=292373#l1083 <https://svnweb.freebsd.org/base/head/sys/kern/vfs_default.c?annotate=292373#l1083>
> > 
> > I haven???t managed to dig any deeper than that yet.
> > 
> > Is there any other information I could give you to help narrow this down?
> 
> I do not see this issue locally.
> 

I do:

(kgdb) f 9
#9  0xffffffff80ac7956 in vop_stdadvise (ap=0xfffffe081dc6d930) at ../../../kern/vfs_default.c:1087
1087                            error = bnoreuselist(&bo->bo_dirty, bo, startn, endn);
(kgdb) l
1082                    endn = ap->a_end / bsize;
1083                    for (;;) {
1084                            error = bnoreuselist(&bo->bo_clean, bo, startn, endn);
1085                            if (error == EAGAIN)
1086                                    continue;
1087                            error = bnoreuselist(&bo->bo_dirty, bo, startn, endn);
1088                            if (error == EAGAIN)
1089                                    continue;
1090                            break;
1091                    }
(kgdb) info loc
vp = (struct vnode *) 0xfffff8008bdaa9c0
bo = (struct bufobj *) 0xfffff8008bdaab28
startn = 0x0
endn = 0xffffffffffff
start = 0x0
end = 0x8000000000000000
bsize = 0x8000
error = 0x0
(kgdb)

https://people.freebsd.org/~pho/stress/log/kostik855.txt

- Peter