Yesterday my server went down again. This time it was a combination of hardware and software. On software side my Reizerfs filesystem got corrupted in a funny way - I could not delete some files under /var. Those were files modified after a specific time yesterday, but the effect was only local to /var directory and did not effect any files in any other directories on the same partition. No suspicios messages in logs for the whole day. It gives back EPERM = "Permission denied" errors even for root. I "fixed" the problem by "cp -r /var /var.new && mv /var /var.old && mv /var.new /var". All files were fine in the new directory (no corruption) and I could delete the same files there. But /var.old is still not deletable even by "rm -rf" as root. Any ideas?
Of couse I just though that I would reboot now so that all services would use the new /var. And of course the system did not come up because USB chip died. I had to go to the server room and disable USB in BIOS to get it up again. I hate hardware. :P