Re: [ossig] Two obscure questions: directory access times and atomic rename()

[Julian Gomez <julianjose.gomez@xxxxxxxxxxxxx>]

On Tue, Sep 07, 2004 at 09:49:26AM +0800, Christopher DeMarco spoke thusly:

>1.    It is  certainly  true  that the   time taken to   open() a file
>increases with the number of files in the directory  - e.g.  reading 1
>file out of 100 is faster than reading 1  file out of 1000.  Are there
>any guidelines  for how this overhead  scales?  What do MTA authors do
>to maximize seek time within a directory while  conserving inodes?  If

Break up their queue structure.

Script started on Tue 07 Sep 2004 07:04:47 PM MYT
root@jgomez-p-latitude:/var/qmail/queue
root@jgomez-p-latitude queue]# find . -type d
.
./pid
./intd
./todo
./bounce
./mess
./mess/0
./mess/1
./mess/2
./mess/3
./mess/4
./mess/5
./mess/6
./mess/7
./mess/8
./mess/9
./mess/10
./mess/11
./mess/12
./mess/13
./mess/14
./mess/15
./mess/16
./mess/17
./mess/18
./mess/19
./mess/20
./mess/21
./mess/22
./info
./info/0
./info/1
./info/2
./info/3
./info/4
./info/5
./info/6
./info/7
./info/8
./info/9
./info/10
./info/11
./info/12
./info/13
./info/14
./info/15
./info/16
./info/17
./info/18
./info/19
./info/20
./info/21
./info/22
./local
./local/0
./local/1
./local/2
./local/3
./local/4
./local/5
./local/6
./local/7
./local/8
./local/9
./local/10
./local/11
./local/12
./local/13
./local/14
./local/15
./local/16
./local/17
./local/18
./local/19
./local/20
./local/21
./local/22
./remote
./remote/0
./remote/1
./remote/2
./remote/3
./remote/4
./remote/5
./remote/6
./remote/7
./remote/8
./remote/9
./remote/10
./remote/11
./remote/12
./remote/13
./remote/14
./remote/15
./remote/16
./remote/17
./remote/18
./remote/19
./remote/20
./remote/21
./remote/22
./lock
Script done on Tue 07 Sep 2004 07:04:53 PM MYT

I don't have a box with Postfix handy at the moment, but I'm pretty sure
they do something along the same lines. Previously, sendmail used to
lump all the spool files (qX, dX, xX) files under its main spool
directory.

When our old sendmail derivative COTS product used to bugger up [1],
this meant (over time) HUGE amount of spooled files (60K++ :-) We used
to go for cups of tea while "rm" was working its socks off. As Dinesh
noted, its due to the linear lookup table implementation which doesn't
scale particularly well.

Last time I checked sendmail (which was a while ago mind you); they had
an option (enabled in "sendmail.cf" which allowed you to use a hashed
directory structure as above).

That having been said, ext3 has had a directory hashing patch out for a
while which fixes this problem. Dig through the ext3-users mailing list
if you're interested (look for Theo T'so circa late 2002 / early 2003).

XFS is optimized for large files primarily. I think someone from Namesys
has a patch our for reiserfs. Don't know about JFS.

HTH.

[1] Which, unfortunately, was quite often ;)


---------------------------------------------------------
To unsubscribe: send mail to ossig-request@mncc.com.my
with "unsubscribe ossig" in the body of the message