Re: cannot alloc 19968 bytes for inoinfo

From: Eric Anderson <anderson_at_centtech.com>
Date: Fri, 03 Jun 2005 06:35:00 -0500
Julian Elischer wrote:
> Bcc'd to some recipients. (you know who you are..)
> 
> Don Lewis wrote:
> 
>> On  2 Jun, Eric Anderson wrote:
>>  
>>
>>> Don Lewis wrote:
>>>   
>>>
>>>> On  1 Jun, Eric Anderson wrote:
>>>>
>>>>     
>>>>
>>>>> Andre Guibert de Bruet wrote:
>>>>>
>>>>>       
>>>>>
>>>>>> On Wed, 1 Jun 2005, Eric Anderson wrote:
>>>>>>
>>>>>>
>>>>>>         
>>>>>>
>>>>>>> Don Lewis wrote:
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> On 31 May, Eric Anderson wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>>
>>>>>>>>> One of my filesystems won't fsck.  I'm not sure how to fix it, 
>>>>>>>>> or what it's really trying to tell me.
>>>>>>>>>
>>>>>>>>> # fsck -y /vol1
>>>>>>>>> ** /dev/da0s1d
>>>>>>>>> ** Last Mounted on /vol1
>>>>>>>>> ** Phase 1 - Check Blocks and Sizes
>>>>>>>>> fsck_ufs: cannot alloc 19968 bytes for inoinfo
>>>>>>>>>
>>>>>>>>> df -i /vol1 output:
>>>>>>>>> Filesystem  1K-blocks        Used    Avail Capacity  iused     
>>>>>>>>> ifree %iused  Mounted on
>>>>>>>>> /dev/da0s1d 1891668564 1684163832 56171248    97% 55109756 
>>>>>>>>> 189360002 23% /vol1
>>>>>>>>>
>>>>>>>>> Any help would be very appreciated!
>>>>>>>>>               
>>>>>>>>
>>>>>>>> You're probably running into the default 512MB data size limit.  
>>>>>>>> Try
>>>>>>>> setting kern.maxdsiz to a larger value in /boot/loader.conf and
>>>>>>>> rebooting.  I've got mine set to 1GB.
>>>>>>>>   kern.maxdsiz="1073741824"
>>>>>>>>             
>>>>>>>
>>>>>>> Hmm - I don't seem to have that sysctl..  What would create it?
>>>>>>>           
>>>>>>
>>>>>> It's a loader tunable, not a sysctl variable. man 5 loader.conf
>>>>>>         
>>>>>
>>>>> Oh.. oops. :)   Ok, then I have it set correctly but it isn't 
>>>>> helping me.  My fsck still dies the same way.  Looks like it's 
>>>>> taking up about 362MB memory (I have 1GB).  Any more ideas?
>>>>>       
>>>>
>>>> What does the shell limit command say about your datasize limit?  Your
>>>> limit might have been cranked down in login.conf.
>>>>     
>>>
>>> I looked too early at the fsck. It appears to actually be going up to 
>>> the 1GB limit now, and then bombing. It's now bombing at a different 
>>> point:
>>>
>>> # fsck -y /vol1
>>> ** /dev/da0s1d
>>> ** Last Mounted on /vol1
>>> ** Phase 1 - Check Blocks and Sizes
>>> fsck_ufs: cannot increase directory list
>>>
>>>
>>> # limits
>>> Resource limits (current):
>>>   cputime          infinity secs
>>>   filesize         infinity kb
>>>   datasize          1048576 kb
>>>   stacksize           65536 kb
>>>   coredumpsize     infinity kb
>>>   memoryuse        infinity kb
>>>   memorylocked     infinity kb
>>>   maxprocesses         7390
>>>   openfiles           14781
>>>   sbsize           infinity bytes
>>>   vmemoryuse       infinity kb
>>>
>>> So I think I just need more RAM.. This is really a major ceiling for 
>>> anyone that wants a somewhat large filesystem, or someone who needs a 
>>> lot of inodes.  Is there maybe a different way to do the fsck that 
>>> might take longer, but run in 'small' memory footprints like 1GB or 
>>> less?  I know little to nothing about coding fsck tools or memory 
>>> management, but I do know that there's always more ways to do 
>>> something.  Just curious if there could be a 'lowmem' option for fsck 
>>> that would utilize memory differently in order to fsck large fs's.
>>>   
>>
>>
>> You can crank up datasize to be larger than RAM assuming that you have
>> sufficient swap configured.  It'll just be slow as fsck starts paging.
>> At some point on 32-bit architectures like the i386 you'll run into the
>> address space limit, probably at 2-3GB.
>>
>> I think julian_at_ has mentioned having a version of fsck that uses
>> external storage.  I would expect a substantial speed impact, and you
>> would need at least one other file system mounted rw.
>>  
>>
> 
> exactly.
> We haven't done this yet, however it's on our development
> roadmap for the next generation of raids.
> In our "back of the envelope" designs and calculations it
> is hard to say conclusively  that it will be a lot slower. You do
> the work in several passes of the disk where you never go 'backwards'..
> instead, you write out "addresses of interest" that are behind
> you to a "read this on the next pass" list that you write
> out to the other storage.  The "next pass read" lists are sorted
> in ram as much as possible before being written out
> and then you do an on-disk "merge sort" on them as needed
> to produce an "in-order" read list.
> 
> you keep doing this, producing some sorted output lists that detail such 
> things as
> block ranges in use etc. and in the end you reconcile all the output files
> to produce the real bitmaps, find collisions, find ureferenced Inodes etc.
> Onc eagain you put the output files out in chunks htat ar epre-sorted in 
> RAM
> and then do merge-sorts as needed on them to produce sorted output lists.
> The output files would be sorted by different fields for different tests.
> for example a list of referenced block ranges, sorted by start block,
> quickly finds multiple inodes referencing the same blocks and quickly gives
> you the correct bitmaps. A list of referenced inodes, sorted by inode 
> number
> gives you link counts and unreferenced inodes.. etc.etc.
> 
> In the current fsck the majority of time is spent waiting for the head to
> rattle backwards and forwards as it follows links so it is hard to say 
> without trying it
> whether we will be slower than that if we only do forward passes..
> 
> (my guess is "yes, we'll be slower, but not by orders of magnitude, and 
> definitly
> a lot faster than a system that can't fsck it at all due to lack of RAM. 
> :-)

This sounds really awesome!  This would mean I could actually put all my 
filesystems together for one large 18TB partition! :)

Let me know when there is something to test - I'd be more than happy to 
give detailed feedback..


Eric




-- 
------------------------------------------------------------------------
Eric Anderson        Sr. Systems Administrator        Centaur Technology
A lost ounce of gold may be found, a lost moment of time never.
------------------------------------------------------------------------
Received on Fri Jun 03 2005 - 09:35:30 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:35 UTC