Problem in /tmp directory + undeletable file

Richard G

Richard G

Verified User
Joined
Jul 6, 2008
Messages
13,993
Location
Maastricht
We got a Centos 5.x server with DA and a problem in the /tmp directory.
We have the /tmp secured and mod_ruid2 installed.

After a reboot of the server the /tmp directory was not cleared, session files from june are still present. That is not a big problem.

The problem is that we got this session file in there:
Code: 
?--------- ? ?          ?             ?            ? sess_la5i8frhehjqibdompk59m24g3
Which really looks like displayed here. We can't remove this.

rm sess_la5i8frhehjqibdompk59m24g3
rm: cannot lstat `sess_la5i8frhehjqibdompk59m24g3': Input/output error
Click to expand...

I thought with a reboot the /tmp directory should be cleared, why does that not happen?

And how can I fix this so this undeletable file will get removed?
 
Richard G said:
[..] After a reboot of the server the /tmp directory was not cleared, session files from june are still present. That is not a big problem.
Click to expand...

I just want to comment on that I think it is normal that /tmp is not cleared after reboot. I am running CentOS 6.x servers, and /tmp is never cleared after reboot.
 
I don't think it's the filesystem, then there would be more problems I think.
In the meantime I've read that not all operating systems clear the /tmp folder after a reboot.

I symlinked the /var/tmp to /tmp.
Like this:
dd if=/dev/zero of=/var/tmpMnt bs=1024 count=5000000
/sbin/mke2fs /var/tmpMnt
cd /
cp -a /tmp /tmp_backup
rm -rf /tmp/.??* /tmp/*
mount -o loop,noexec,nosuid,nodev,rw /var/tmpMnt /tmp
chmod 1777 /tmp
cp -a /tmp_backup/.??* /tmp_backup/* /tmp/
rm -rf /tmp_backup
echo "/var/tmpMnt /tmp ext2 loop,noexec,nosuid,nodev,rw 0 0" >> /etc/fstab
cp -a /var/tmp/.??* /var/tmp/* /tmp
rm -rf /var/tmp
ln -s /tmp /var/tmp
cp -a /home/tmp/.??* /home/tmp/* /tmp
rm -rf /home/tmp
ln -s /tmp /home/tmp
Click to expand...

This is my /etc/fstab:
Code: 
/var/tmpMnt             /tmp                    ext2    loop,noexec,nosuid,nodev,rw   0   0
 
Richard G said:
I don't think it's the filesystem, then there would be more problems I think.
Click to expand...
Not necessarily, I had this same problem on one of my systems, and the only way to remove that file was to unmount it and do an fsck. That is source of your "Input/output error" when you try to delete it. That particular inode has a problem, but that doesn't mean the rest of the filesystem is trashed.
 
The symlink is not from /tmp to /var/tmp, the symlink is from /var/tmp to /tmp
Click to expand...
Yes that is what I wrote too.:)

@Toml: The /tmp is nog a real filesystem, it's in fact a file (/var/tmpMnt) which I mounted as filesystem.
I thought the input/output error was because of the file wasn't recognized neither is the owner, as you can see from the questionmarks present. I find it hard to believe that a filesystem could be corrupted when this error only occurs on 1 file.

However, you still might be correct, it can do no harm to check anyway. Just to be sure that I'm not doing anything wrong, what do I have to do exactly? Is this correct?

umount /tmp
fsck /tmp
mount /tmp
Click to expand...
Or am I forgetting something? Please remind that this filesystem is in fact the /var/tmpMnt file. So I don't need to do a check on tmpMnt, correct?
And I presume I have to do this when (if possible) the lease amound of users is making use of the server?

@Chatwizrd: my tmpwatch already includes /var/tmp as far as I could see:
Code: 
flags=-umc
/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \
        -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \
        -X '/tmp/hsperfdata_*' 240 /tmp
/usr/sbin/tmpwatch "$flags" 720 /var/tmp
for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do
    if [ -d "$d" ]; then
        /usr/sbin/tmpwatch "$flags" -f 720 "$d"
    fi
done
Maybe I should change the 720 to something else?

@Toml: Strange. That single file would not be removed by the rm command, then I got the input/output error.
Now I tried to remove that file with MC (Midnight Commander) and I had no problem removing it with MC, so it's gone now. Strange isn't it?
 
Last edited:
Richard G said:
Just to be sure that I'm not doing anything wrong, what do I have to do exactly? Is this correct?

Or am I forgetting something? Please remind that this filesystem is in fact the /var/tmpMnt file. So I don't need to do a check on tmpMnt, correct?
Click to expand...
You should not try an fsck on /tmp; in fact you probably can't.

What you should do is delete your current /var/tmpMnt file and create a new one as a fixed length file. I've created a How-To on my site which may help you with the details.

If you can't delete the file, then comment out the line in /etc/fstab, reboot, then delete the current file and create the new one, and reboot again.
And I presume I have to do this when (if possible) the lease amound of users is making use of the server?
Click to expand...
We recommend in our How-To that you reboot your server; any time you move or recreate a tmp file you risk breaking running processes, but rebooting the server gives you a clean start; be sure to read my How-To (link above).

Jeff
 
Thank you Jeff.
But as you might have seen, I managed to delete the strange file with Midnight Commander.

I've created a How-To on my site which may help you with the details.
Click to expand...
That's a nice howto, but I don't quite see a difference with the one I use and posted in this thread.
 
I should have reread your entire history before responding, but I didn't :( .

Nevertheless, it was easy enough to add the How-To; I've had it internally for years, so it didn't take too long for me to do it.

In the future I'll try to read threads more carefully before I respond.
 
No problem Jeff.
It happens to us all. It's also good that you posted your How-To. I just could not find any difference, otherwise I would have gladly used your how-to in the future. And if there was a difference, then you now could have pointed it out to me, because I also don't always see everything.;)
So I'm still thankfull for the answer.

However I'm still wondering, how can it happen that I can't remove a file with the rm command as root, but it can be removed by MC? That's odd, isn't it?
 
Technically there should be no difference between removing a file with rm or MC, at a lower level they are both using the unlink() api to delete the file. On a higher level, there may be some differences when the shell gets in the way, for example if there are hidden or funky characters in the filename, but in that case you would see "file not found" errors not Input/Output errors.

The only other reason I can think of why it may have worked, was that your message says that lstat() failed, that is an api used to get a file status, normally used to make sure you can do other operations on a file. I am guessing that rm exited on the failure of lstat() and didn't attempt to do the unlink() (not 100% sure, I am not looking at the source code), where MC just did the unlink().

Normally if you are getting an Input/Output error performing an action on a file, there are two possibilities hardware error or filesystem corruption. I am guessing filesystem corruption. To fix any problems with that, you should either recreate the filesystem file or fsck the file (fsck /var/tmpMnt after unmounting).
 
Thank you for the explanation. Only in this case, the problem was with lstat because of the funky characters in the filename.
I checked and luckely there was no problem with the filesystem.
Thank you all.
 
You must log in or register to reply here.
Back
Top