Linux in read-only mode

Hi,

I'm not very familiar with linux, expecially when it has crashed. We tried to reboot a RedHat Enterprise 4 machine and it wouldn't start up normally. The thing is that the partion (I think thats what its called but correct me if i'm wrong) that holds the /etc/ folder has no free space, so the fstab file is empty. After the reboot, we keep getting sent to a shell promt, except that whenever we try to empty out the /etc folder, it says its a read-only filesystem, but if I try to write to /var, it allows me to. How can I gain the ability to erase on the /etc drive?

Thanks ahead.

Sounds to me that the partition that is mounted on /etc needs to have fsck run against it to make it clean and marked to be mounted writeable.

Does it mention on boot up that the partition for /etc was not shut down cleanly, it should also attempt to run fsck against it.

If you can, boot into single user mode and run fsck against it

Do you know if it is ext2 or ext3 ?

if it is ext3 the command would look like

fsck.ext3 -y /dev/hdxx or
if it is scsi
fsck.ext3 -y /dev/sdxx

the device being what ever device it is for /etc in the fstab on the right, Which in your case you sound like you might have to guess it, since /etc is the location of these files and that is also where you are having problems.

[edit]
Another option is to boot off of a bootable cd such as knoppix or backtrack, then see if you can see the hard drive devices to run fsck against them. Doing it this way the original /etc is not necessary.
[/edit]

I'll try the fsck command. I believe it is in ext2 because it says something uppon bootup about ext2. Also, I have tried knoppix and it was unable to mount the partition. I'm trying the rescue mode on the Linux installation disk, but no luck there either. I think someone else tried the fsck command, but neither he nor I knew how to use it so I shall try your way.

Any more idea's? I'll post an update soon.

Thanks ahead.

[edit]

Ok, I ran the fsck command on the drive and it gave me this.

Quote:

---

The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem, then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>

---

The only thing is, I really don't know how are what to use that command for. I also tried it as an ext3 filesystem just in case but no luck there either.

[/edit]

In knoppix, you don't really want to mount the /etc device, you need to just find it to run fsck against it. It is best to run fsck on devices that are not mounted anyways.

I ran the command through the linux rescue disk and got the above again. I'm loading knoppix and trying out your suggestion.

[edit]
Same thing with knoppix also. I tried running the command it gave me but I got the same errors.

I don't know if this is any help, but I can explore the drive when it boots without a bootable cd.
[/edit]

Just be careful, if it isn't a ext2 filesystem but you rebuild it as if it were and create a new superblock, you may lose it for good.

Something is not right. It could be complaining about the bad superblock because you have not used the correct file sytem type. If I am not mistaken, Red Hat Enterprise defaults to a ext3 file sytem at install.

I know you've stated you tried ext3. I'd step away for a cigarette or a coffee for a few then come back and go at it as if you've just begun, take your time.

[edit]
My bad, it seems that e2fsck works with both ext2 and ext3 try this.

e2fsck -y /dev/hdxx

ext3 is nothing more than a ext2 that supports journalling, which is used for cleaner recoveries.

Last resorts you could always attempt to trun the e2fsck against the mounted drive, it sometimes works, but isn't really suggested

[/edit]

A couple of questions:

How are you trying to start it?

Do you have SELinux enabled?

Ok, an alternative thought is that all I need is the information in the /var folder. I can see it when it boots, and I just need to get that to a portable hard drive. The fsck command gives me the same error when i run it as an ext3 file type, which it turns out we already tried yesterday according to the person i'm working with. Now if I can just get the usb harddrive to mount in the default rescue mode, then I can copy /var over to it. The only thing is I can't get it to mount.

One question. What is SELinux and how do I know if its running?

I am starting linux by just booting it up. Don't know what else you mean by that. Again, I tried the e2fsck command and still got the same error.

Thanks ahead

SELinux = Security Enhanced Linux put out by the NSA. It is distributed in Red Hat Enterprise 4.

I'm pretty sure it is defined or turned on via the kernel config, I'll concede to IKnowNot on that though

Cool, thanks.

Ok, I'm probably going to have to reinstall RedHat, but I finally figured out a way to copy the files I need. I couln't create a directory for the USB harddrive to mount to, so I thought, what the hell, I'll try and put it as the mnt directory itself, and walah, it works. Thanks for your help so much. I'll still keep you posted on if I get it to work or not.

Thanks ahead

Quote:

---

The thing is that the partion (I think thats what its called but correct me if i'm wrong) that holds the /etc/ folder has no free space, so the fstab file is empty

---

This is bad, very bad.
Who set up the partitions?
Why does it appear full?

Quote:

---

After the reboot, we keep getting sent to a shell promt, except that whenever we try to empty out the /etc folder, it says its a read-only filesystem

---

Which is why I am guessing the system has SELinux enabled.

Quote:

---

but if I try to write to /var, it allows me to.

---

Again, under SELinux, you may have read capabilities, but not write.
How were you booting?
Single user mode?

Quote:

---

How can I gain the ability to erase on the /etc drive?

---

Again, how were you attempting to boot to it, and do you have SELinux enabled?
If you do, did you write any policies yourself?

Just bare in mind, I am not bashing you unless it was you who set up the system. It sounds like, and you may indeed, be dealing with a system that was set up by someone who is either no longer there and/or did not know what they were doing.

It just sounds odd to me that /etc would be on a partition by itself, and/or full.
Especially since /var is writable.

Opus00 was correct as I understand it, RH4 uses ext3 by default.

What comes to mind first is that this is a system that is on read only media with certain directories on read/write media.

Second was the SELinux scenario.

So what else is on the /etc partition?

If the /etc partition is full, then either it is on read only media, the person who installed it did not know what they were doing, or the system was compromised and you have a whole lot of sh*t on there that shouldn't be.

If this is a legitimate system, and if you suspect a compromise, contact the authorities.

If you suspect that it is due to a mis configuration or equipment failure, the first thing I would do is pull it off line and replace it. Then "dd" the drive. Work with the copy to find out what happened and to recover data.

There are ways to turn off SELinux if this was the problem, but it sounds like it was not.

If you are using RH4, then Red Hat should be able to help recovering data and accessing the drive.

Another thing that had me dismayed:

Quote:

---

I couln't create a directory for the USB harddrive to mount to, so I thought, what the hell, I'll try and put it as the mnt directory itself, and walah, it works.

---

If this person is working administering linux network boxes, are they aware that USB devices under *nix might show up as scuzzy devices?

THe more I read, the more I shake my head.

IIRC if fstab is empty the system doesn't know what filesystems there are. If this happens the system will boot to single user mode. It'll ask which shell you want to run. The root filesystem / will always be mounted read-only. After a filesystem check a mount -u / will remount root read/write.

Usually /etc is on the root filesystem, so try to find out where it filled up.. And AFAIK if the filesystem fills up it doesn't destroy existing files. So I'm not sure why fstab is empty.. Maybe some admin nuked it? Perhaps it had a crash and that corrupted fstab?

DO NOT clean out /etc! There are lots of config files/startup scripts in there that are needed to properly boot.

Perhaps he just lost journalling on the partition? or perhaps try to mount -o rw /etc?