What You Don’t See On Your Hard Drive

[magnoon]

I have been working on a paper dealing with how data is stored, deleted, and recovered on hard drives for the last few weeks. In the course of my research I found this paper which covers this subject more eloquently than I could ever hope for.

http://rr.sans.org/incident/dont_see.php

Below is the opening paragraph as a preview.

Just because you don't see it doesn't mean it's not there. By having a knowledge of something that exists, but is hidden from your sight, will give you an advantage because you know it's there. In the security field it is very important to keep up to date on the latest information available. If you don't, someone will take advantage of your ignorance. Things are always changing and becoming bigger, better, faster and sometimes sneakier. A few years back in my Information Technology career I made the change from Desktop Support to the Information Security Group. Since then I have learned a tremendous amount about security. I have learned that you have to train yourself to think differently about things, add a little paranoia. This paper will address two security concerns that I found very interesting. They both have to do with things that are not in plain sight. The first security concern covers the issue of retrieving data that has been deleted. So many people have no idea about data that is left behind when you delete files or fdisk and format your hard drive. The second issue deals with hidden access and control of your computer. I will look at what a rootkit is and look at the recent development of rootkits designed for Microsoft Windows operating systems.

[nabylbt]

Great post
Ad once again the sans institute is doing its job!

[gore]

good job man, thi is really an interesting subject i think, how computers store data and how they delete it, most people think when you delete something its not stll there, its gone, but in reality the computer marks it as free space and its still there, we learned this a few weeks ago in my OSs class.

[firestarter5]

A few weeks ago all my mp3's "magically" vanished. How? Not a clue. I checked my D drive, and sure enough, the amount of available space had increased dramatically. I checked all drives, folders, files, everything. No sign of them. I fired up Norton UnErase and it found them. Where? I don't have any idea. But it was wierd to see that the space the mp3's took up on my D drive WAS no longer there. They were on my system, but masked somehow.

[ammo]

Firestarter: from what you're describing, your mp3 files were indeed deleted*: the fact that the total free space increased reflects that. Notice that I said "deleted*" with an *: when you "delete" files, you basically just erease the "index" that says where the file is stored on the disc. The content of the file itself isn't ereased. Norton UnErase uses that fact to restore files that have been "deleted" but not overwritten yet: it just makes a new "index" for the file to be restored, and voila... This is how you were able to recover your mp3s...


Ammo

Hey, quickly reading the sans article, I notice they didn't mention NTFS hidden data streams which can also be used to hide data... Oddly they already add an article describing that though..! : http://rr.sans.org/threats/win_NTFS.php

A few uitilities exist that can find hidden data streams on your disks, Foundstone has one: SFind: http://www.foundstone.com/knowledge/proddesc/forensic-toolkit.html

Ammo

[firestarter5]

A question then: You have a partitioned drive. C drive is 5G and D drive is 10G. You go to Kazaa and download 9.9G of mp3's on your D drive (this is a hypothetical situation). You write down a list of all these mp3's then delete them! Your PC says you have 10G of available space on your D drive again. Back to Kazaa you go and download another 9.9G of mp3's. Does this mean that your 10G D drive is actually holding 19.8G of mp3's? Or is the info from those origianl mp3's now actually overwritten and unable to be retrieved?

[powertoad5000]

Your D drive would be holding all the new mp3s, having wiped the old ones to make space for the new ones. You would be unable to retrieve the old mp3s. The disk is physically unable to hold more than 10Gb of data.

[ammo]

Like powertoad said...
There are utilities that actually wipe "deleted" data (it in fact overwrites free space with zeros...)...

Ammo

[The Old Man]

Darn good post, good info. I wrote a lengthy account of an incident where un-delete did a good job. But it was too boring. I de-leted it! :) :D

[spyrul]

Originally posted here (http://www.AntiOnline.com/showthread.php?threadid=#post) by ammo
(it in fact overwrites free space with zeros...)...


nothing big, but i think what happens when data is wiped from a HDD is that rather than be a zero or a one, the magnetic charges on the platter are just randomly scattered about.

again, nothing even remotely important, but i was just pretty sure that's what happens. i guess the heads would read those as zeros, though.

[ammo]

Well... no... at least not with software utilities. There are "de-magnetisers" machines (degausser) that are used for disabling drives that have contained top-secret data, and these cost a bundle. However, only using software, you can only have the drive read or write data. The drive heads cannot just "scatter magnetic charges"... how would a head that is designed to read and write individual bits (in block), ie precisely, be able to do that?. Besides, if it were the case, the drive would either need a low level format after that or be filled with bad clusters, ie: be just plain dead.

Here's an extract of DoD 5220.22-M shredding guidlines:
(http://www.dss.mil/isec/chapter8.htm)

a. Degauss with a Type I degausser

b. Degauss with a Type II degausser.

c. Overwrite all addressable locations with a single character.

d. Overwrite all addressable locations with a character, its complement, then a random character and verify. THIS METHOD IS NOT APPROVED FOR SANITIZING MEDIA THAT CONTAINS TOP SECRET INFORMATION.


Also, a comparaison of software "shredders" can be found here:
http://www.fortunecity.com/skyscraper/true/882/Comparison_Shredders.htm
(Check the "Overwrite algorithm" row for wiping technique)


Ammo

[noODle]

I did a Low Level Format a while back on my Maxtor drive with the utility found on their website.
Does this really destroy the actual data ?

And a friend of mine told me a low level format under linux is really simple.
I don't know the exact syntax but it had something to do with /dev=NULL.
Does this destroy data or is it stil 'readable' after doing this.

Furthermore a nice informative thread.

[coolnads]

Originally posted here (http://www.AntiOnline.com/showthread.php?threadid=#post) by firestarter5
A question then: You have a partitioned drive. C drive is 5G and D drive is 10G. You go to Kazaa and download 9.9G of mp3's on your D drive (this is a hypothetical situation). You write down a list of all these mp3's then delete them! Your PC says you have 10G of available space on your D drive again. Back to Kazaa you go and download another 9.9G of mp3's. Does this mean that your 10G D drive is actually holding 19.8G of mp3's? Or is the info from those origianl mp3's now actually overwritten and unable to be retrieved?

Long time back i made a presentation on data hiding in diffrent kinds of RAM's (DRAM,SDRAM etc.) .. this was the paper that i used as an intro to this topic but.. as u will see this paper deals mostly with data on magnetic media.....the author at some points strongly hints that data once overwritten can also be retrieved...... !!!!

so firestarter we can actually find ur previous 9.9 GB's ... but this is quite difficult, will require highly advanced equipments (i think so) but is possible ( probably not all the mp3's but still......)

heres the link

http://www.cs.auckland.ac.nz/~pgut001/pubs/secure_del.html

enjoy.

[spyrul]

Originally posted here (http://www.AntiOnline.com/showthread.php?threadid=#post) by coolnads
the author at some points strongly hints that data once overwritten can also be retrieved...... !!!!

so firestarter we can actually find ur previous 9.9 GB's ... but this is quite difficult, will require highly advanced equipments (i think so) but is possible ( probably not all the mp3's but still......)

actually, i'm pretty sure that it's completely infeasible to get overwritten data. think of it this way:

you have a closet full of empty Coke cans. you've documented how many Coke cans you have in there.

so, for now, a Coke can is data, and your little documentation would be your file allocation table.

now, you want to get rid of some Coke cans. initially, you just erase a few numbers off the documentation, and, as far as you're concerned, you've got more room in your closet. let's say you erase the entire documentation, and now you want to fill the closet with beer cans. because there's not enough physical room to store more than a couple of beer cans, you toss a bunch of Coke cans in to the hall, then place the beer cans in there and document that.

now there is no way to look in the closet and find Coke cans. before you say "you can retrieve them by going in the hall and picking them up", keep in mind that the HDD doesn't have a hallway it can toss its empty Coke cans in to :p

[diavikdozer]

Very nice, this newb is starting to learn and thanks you people

[gunder]

Good post, I remember learning about this a long time ago. And I forgot heh, thanks for bringing me back up to speed :)

-gunder

[ammo]

Originally posted here (http://www.AntiOnline.com/showthread.php?threadid=#post) by spyrul


actually, i'm pretty sure that it's completely infeasible to get overwritten data. think of it this way:

you have a closet full of empty Coke cans. you've documented how many Coke cans you have in there.

so, for now, a Coke can is data, and your little documentation would be your file allocation table.

now, you want to get rid of some Coke cans. initially, you just erase a few numbers off the documentation, and, as far as you're concerned, you've got more room in your closet. let's say you erase the entire documentation, and now you want to fill the closet with beer cans. because there's not enough physical room to store more than a couple of beer cans, you toss a bunch of Coke cans in to the hall, then place the beer cans in there and document that.

now there is no way to look in the closet and find Coke cans. before you say "you can retrieve them by going in the hall and picking them up", keep in mind that the HDD doesn't have a hallway it can toss its empty Coke cans in to :p

Well, no, IT IS possible:

Overwriting data on disk isn't like filling a closet, it's like writing over used paper... What happens is that when the disk heads write on the platter, it re-aligns magnetite (or whatever magnetic compound they use) in a diffrent direction. However, a single write doesn't manage to get all magnetite (or whatever) particuls re-alligned. So while the majority of particuls will have change directions, there will be a few residual ones that will still be oriented in the previous direction.

So finding out what data was there before means using a more sensible device that can distinguish or detect variations in the magnetic field or such... (That's why whiping software will make multiple writing passes, sometimes with randomized caracter, in order to try and re-allign all particuls). Of course this is hard to do and pretty expensive, but people with enough money and resources (think FBI, CIA, NSA...) could and do have the means to do it. In fact, I remember reading somewhere that it was rumored the NSA (I think) was able to recover data after 27 passes!

Ammo

[magnoon]

Spyrul,

actually, i'm pretty sure that it's completely infeasible to get overwritten data. think of it this way

You are correct with this statement regarding simple undelete utilities such as Norton Unerase, but there is a high probability of recovering the data if it has only been overwritten one or two times, but it requires specialized equipment.

When data is written to magnetic media it is written to a magnetic domain. This domain consists of a number of magnetic bits (not to be confused with a data bit 0 or 1) which receive the magnetic charge. Not all of the bits in the magnetic domain will change when the head passes over during the write operation and will retain the magnetic properties from a previous operation. Does this mean that a magnetic domain can contain magnetic bits that are set to a zero and magnetic bits that are set to a one? It certainly does, but if the write operation was setting the bit to a one then the majority are set to a one, strongest at the center of the domain weaker at the edges.

Remember in grade school when you put metal filings on a piece of paper and ran a magnet underneath. Most of the filings lined up and pointed in the same direction, but the ones at the edges didn't all point with the others. The same thing is happening when you write to the hard disk.

Is it easy to peel back these layers to determine what was overwritten? With modern hard disks this is a difficult, costly, and time consuming process but portions of overwritten data can be recovered if it was overwritten or wiped with a single pass process. The minimum process that should be involved in a wipe would be a three pass write. A three pass will make one pass writing 00 followed by it's complement which is an FF and a final pass of random data. It is still possible to recover some data after a 3 pass wipe, but whoever does will want that data very badly and have the $$ to attempt the recovery. Generally a 7 pass wipe will make it near impossible to recover the data and I have never heard of any data being recovered after a Guttman 35 pass wipe. (Disclaimer.. Doesn't mean that those agencies with 3 letter names can't do it but they would really want you bad to go to the expense involved)

Here is a link to a document that describes the process required to dispose of unclassifed DoD computer hard drives.

http://www.c3i.osd.mil/org/sio/ia/diap/documents/ASD_HD_Disposition_memo060401.pdf

Edit:

I was too slow, I see that ammo posted a response as I was writing this :) Ammo, do you remember where you saw the information on the 27 pass recovery?

[ammo]

Unfortunately, I really don't remember where I read that, and can't find it either searching on google and others... :/

I do believe it was a either comp security or comp news site site though (not that that really helps!)...

Ammo

[spyrul]

ah. last time i read about this, i was under the impression that it would realign all the magnetic charges, or some data would just be incomplete.

looks like i have to brush up on harddrives :(


heh, i got owned.

[mrcoffee12]

owned by a fellow canadian though so its not quite as embarrassing eh? :)

Anyway, very informative, thanks for the thread. Learned more in this thread than I have at school so far this year :)

[Tuskin]

no0Dle, if you are interested I did do a review on how to undelete in Linux but it's a little complicated. I made the 'review' as basic as possible and it's source is there as well. If you are indeed interested it's at:
http://www.antionline.com/showthread.php?s=&threadid=223895&highlight=linux+undelete
enjoy

[this_1_guy2002]

Hey great find and post. This is some of the more intresting posts Ive read so far.

[interax]

I for one would like to know how many overwrites the likes of OnTrack can find and recover from. I think they are one of the bestin this field and have seen some of their offices.
Mind you they ain't half expensive. I once heard of a guy at a council who trashed a RAID 5 array. 5 drives recovered for £15,000. Thats Pounds Sterling!!!!

[drew]

i know u guys are worried about 3,7,27,35 passes and such but i have a much safer system. it's the "crush and burn". a hammer and fireplace will do the trick. after running the stringest magnet avail ova ur hdd, smash it then burn it... done =P

if u want ur data "protected" then think about it seriousely for a sec. how can they get the info if the hdd is in 100 pieces. if ur data it THAT important that u need to do more then 7passes on it then y not opt for a new hdd and trash ur old one?

just my opinion.

[idealogical]

this is an excellent discussion on recovering deleted files that were once saved on a hard disk- but what if a file was prepared on a pc but saved on an external disk- is there anyway of recovering/detecting the file that was produced on that machine but was never saved on the hard disk?

my thoughts are that if the file uses the paging file which resides on the hard disk then there is going to be remnants of it- but what if it is just stored in RAM and how would you know whether it is being held in RAM or in the paging file?

[Deadsy]

Interesting paper, though I admit I didn't read too much into it - some of the stuff's a little over my poor newbie head. If this has been covered before I apologise...when you delete files from your hard drive, and subsequantly empty your 'trash,' is the data physically erased from your drive? Or is there still a way to retrieve it? Just curious..

[phishphreek]

Is it only possible to recover data from a hard drive after it has been deleted... or can you do it with other drives too?

By that I mean, CDR-W, floppy disk or flash memory.

I presume that a CDR-W and floppy would be able to recover data from, but flash is a bit different?

I just got one of them usb flash drives. Maybe I'll do some testing on it.
Anyone know of any good but free unerase utilities?

[mark_boyle2002]

Hey All,

I am working on a VB Raw disk reader at the moment to help us all with forensics challenges.

I already have a perfect working DOS raw disk reader/editor. If anyone wants a copy drop a mail to mark_boyle2002@yahoo.co.uk and I will send it to you.

MB - Bytes Ahead.

[jxrry59]

Know of a "safe" free tool to overwrite "free" space????

[phishphreek]

jxrry59:

Why don't u give sdelete a try? I haven't used it, but it looks like it'll do the job. You can always test it by using an unerase program and then using this utility and running the unerase program again. Compare your results.

BTW: This site has a bunch of cool freeware tools. Take a look around while you are there.

http://www.sysinternals.com/ntw2k/source/sdelete.shtml

[jxrry59]

Thanks
---I asked for safe because I once downloaded a program that said it would rewrite over free space and it deleted the directory it was downloaded in(program files) I also had downloaded a few to test and compare and couldn't remember which was which.

So slooowwwllllyyyyy I learn

[Kerms]

Thank you very much! good post!





UlleusLowpr0