Ever Get The Feeling You’ve Been Cheated? (Part 2)

I’m back in graduate school these days which is one of the reasons why I’m long overdue on this blog post. Returning to school has provided me with perspective of a student when thinking about the issue of digital forensics degrees. The more I think about it, the less I like the idea of the digital forensics academic programs compared to some alternatives.

The last blog post resulted in plentiful public and private feedback. A common question was what I expected from the graduate of digital forensics programs. I don’t expect someone with a digital forensics degree and no experience to “hit the ground running” where they are immediately cranking out competent digital forensics exams. What I do expect from undergraduate students is that they will be able to perform basic digital forensics exams with about six months of substantial training from my team. I also expect that they will be able to talk intelligently about file system forensics in the initial job interview. If a candidate doesn’t know digital forensics beyond the tools, they were cheated and they’re yet another digital forensics degree victim. I might as well just draw a chalk outline around the chair they sat in for the interview because it’s a crime scene.

If a candidate has a graduate degree in digital forensics, I have the same six month expectation of when they can start to perform acceptable digital forensics exams. Additionally, they had better be able to keep up in an advanced NTFS discussion during the interview. I won't go into the specifics here because I don't want to give away my hiring methods and questions, but I expect a working knowledge of NTFS from the undergraduate degree holders and much more out of the people with a graduate degree.  If you have that shiny new digital forensics graduate degree, you also better have something you are passionate about and skilled at when it comes to the digital forensics world.

So how do you get to the place where you can be successful in a job interview and land that first job? In general, forget about getting a digital forensics degree at an undergraduate level. You’re better off building a firm intellectual foundation for yourself by mastering the fundamentals of computer hardware and software by going through a program such as computer engineering, electrical engineering, or a similarly structured information technology program. Most digital forensics programs are just warmed over mediocre information technology programs with enough poorly taught digital forensics content so that the school can call it a digital forensics degree.

If you want to be excellent at digital forensics, you need a strong understanding of the fundamentals of the technology that you are going to be investigating. The medical profession figured this out a long time ago when it came to training doctors. Medical school is about teaching students about the fundamentals before they move onto their more specialized job roles. Specialties such radiology and pathology are specializations in the medical world that are roughly similar to what we do in the technical world. Both of those jobs require a rigorous general education in medical school before more highly specialized training through residency and fellowship educational processes.

If someone in high school were to come to me today and ask me what the best way to prepare for a digital forensics career, I would tell them to find the best value they can in a degree such as computer or electrical engineering and to supplement that education with some specialized digital forensics training. The specialized training could take the form of a strong digital forensics undergraduate minor, a graduate or undergraduate certificate program, or a full digital forensics graduate program. Some of the best programs in the digital forensics world aren’t actually full digital forensics programs. You do not have to get an degree in digital forensics to prepare for and begin a rewarding career in the field.

Value is important when it comes education which is why I caution students about taking on excessive student loans. Racking up $80,000 in loans for a mediocre digital forensics degree is senseless. I can understand higher student loans if someone is fortunate enough to get into certain top-tier schools such as Cal Tech, MIT, or Stanford, but the math just isn’t likely to work for an expensive degree in digital forensics from Burning Stump Junction University (BSJU). If you are here in the United States, you likely have very fine options that are being offered in your state schools at in-state tuition prices. You will likely be much better off getting that computer engineering degree from the University of Your State at in-state tuition prices than going into massive debt for digital forensics degree at BSJU.

Gary “Doc” Welt (You Can Help Batman)

In the way of warning, this blog post has almost nothing to do with digital forensics and everything to do with something more important. One of the nice things about having my own blog is that I am my own editor and I don’t have to ask permission to write about something that has very little to do with the original purpose of the blog.

I originally set out to write a follow-up to my last post dealing with the deficiencies that I’m seeing in digital forensics education. That blog post generated quite a bit of interest and I’m grateful for all of the responses both in public and private. I’ll get back to that topic in my next blog post and, as an added “bonus”, I’ll even talk about the new CCFP “cyber forensics” certification being offered by ISC2.

But none of that seems all that important to me as I write this on the 4th of July weekend given how many people over the years have sacrificed everything they had to defend the United States of America and the rest of Western Civilization against a whole host of profoundly bad people. Even a cursory glance at world history shows that peace and prosperity is not the natural state of human affairs. Being able to sustain a place like the United States requires an incredible amount of continuous effort by many people with the brunt of the burden falling on the United States military.

By day, I am a mild mannered digital forensics geek who has the honor and privilege to lead a pack of world-class border collies. By night (and sometimes weekends), among other things, I’m a rookie competitive action shooter. I started doing this early this year and it’s been an amazing experience in large part because of the people involved in the sport. They tend to be some of the nicest and most generous people that I've encountered in many years and this generosity reminds me of the digital forensics community in many ways. My primary game is USPSA action shooting and my home club is the Wyoming Antelope Club in Clearwater, Florida.

It’s through the Wyoming Antelope Club that I became aware of a real live superhero by the name of Gary “Doc” Welt. “Doc” Welt spent around thirty years of his life as a United States Navy SEAL. You can read about Gary’s career here and you will also read why I’m writing this. Gary Welt has been diagnosed with Amyotrophic Lateral Sclerosis (ALS) which also known as Lou Gehrig’s disease. ALS is a very tough set of cards to be dealt. Gary provides a very clear explanation of what he’s up against in this YouTube video. The life expectancy of someone diagnosed with it tends to be two to five years. There is a small percentage of people who live beyond that time. This is the same disease that Stephen Hawking has and, as CNN explains...

Most people with ALS survive only two to five years after diagnosis. Hawking, on the other hand, has lived more than 40 years since he learned he had the disease, which is also known as Lou Gehrig's Disease in America and motor neuron disease, or MND, in the United Kingdom.

So if anyone has a shot at beating the odds in the face of ALS, it’s a  superhero like Gary Welt.  What is interesting is that Gary’s military service might be one of the things that increased his risk for getting ALS. The Mayo Clinic reports that:

Recent studies indicate that people who have served in the military are at higher risk of ALS. Exactly what about military service may trigger the development of ALS is uncertain, but it may include exposure to certain metals or chemicals, traumatic injuries, viral infections and intense exertion.

I call Gary Welt a superhero because he is one. Think about it. No one would deny that Batman is a superhero, but he’s a superhero who doesn’t have any intrinsic superpowers. He wasn’t bitten by a radioactive spider or exposed to gamma radiation which provided him special powers. He wasn’t born on Krypton and sent to Earth. Batman is superhero because he's an exceptionally trained, highly intelligent, and supremely well-conditioned human being with a vast equipment budget. That also describes the US Navy SEALS. Most people can’t even get into their training pipeline much less complete it because of the mental and physical toughness that is required. They do incredibly complicated and challenging work with some of the most sophisticated weapons systems in the world. So even if you are mentally and physically tough enough, you aren’t going to become a SEAL if you are a dullard.

What about equipment? We all know that Batman has all sorts fantastic equipment like the Batmobile, Batcopter, Batcycle, Batboat, and all the rest of his goodies. The SEALS have their own stuff that might as well be right out of a comic book. Check out the picture below.

That’s right. The SEALS have their own version of the Batsub. They just call it a SEAL Delivery Vehicle. Put some capes on those guys and give it a bit more of a snappy name and you’ve got a picture right out of a comic book.

Not enough to convince you? Fine. The SEALS have their own version of the BatBuggy. Look at this:

The SEALS just happen to call their BatBuggy a Desert Patrol Vehicle. Not the most creative name, but it can be equipped with a variety of weapons including a 40mm grenade launcher so it doesn’t need one. Good luck with that, Joker.

The only meaningful difference that I can see between a superhero like Batman and a superhero like Gary Welt is that Batman is fictional and “Doc” Welt and the rest of his SEAL brothers are real. “Doc” Welt is a superhero who has devoted his life to fighting bad guys and protecting the rest of his. Now we have an opportunity to try and return the favor by helping him out when he’s in a tough fight. How often do you get to say that you helped a real life superhero?

As the Red Circle Foundation webpage set for for him explains:

We are raising money to help Gary and his wife modify their home for his condition and for wheelchair access. The VA (Veterans Affairs) does a lot of good, but they are a slow moving bureaucracy and time is critical for the Welt family.

The primary way that you can help Gary is donating money via the Red Circle Foundation website. I think the current setup is that any money you give via that portal will result in 90 percent going to Gary and 10 percent to help pay for Red Circle Foundation costs. If you follow Gary’s progress at the HelpGaryWelt Facebook page you’ll see them discussing how that works.

I know the digital forensics community to be a very generous bunch with a culture of sharing and helping one another out. He’s an opportunity for us stand together to help someone who has done so much for others. How often can you say that you got the help Batman? Please consider giving anything you can to help a real live superhero like Gary “Doc” Welt.

Photo Credits and Captions

SDV Photo:

Atlantic Ocean (May 5, 2005) - Members of SEAL Delivery Vehicle Team Two (SDVT-2) prepare to launch one of the team's SEAL Delivery Vehicles (SDV) from the back of the Los Angeles-class attack submarine USS Philadelphia (SSN 690) on a training exercise. The SDVs are used to carry Navy SEALs from a submerged submarine to enemy targets while staying underwater and undetected. SDVT-2 is stationed at Naval Amphibious Base Little Creek, Va., and conducts operations throughout the Atlantic, Southern, and European command areas of responsibility. U.S. Navy photo by Chief Photographer's Mate Andrew McKaskle (RELEASED)

DPV Photo:

Camp Doha, Kuwait (Feb. 13, 2002) - U.S. Navy SEALs (SEa, Air, Land) operate Desert Patrol Vehicles (DPV) while preparing for an upcoming mission. Each Dune Buggy" is outfitted with complex communication and weapon systems designed for the harsh desert terrain. Special Operations units are characterized by the use of small units with unique ability to conduct military actions that are beyond the capability of conventional military forces. SEALs are superbly trained in all environments, and are the masters of maritime Special Operations. SEALs are required to utilize a combination of specialized training, equipment, and tactics in completion of Special Operation missions worldwide. Navy SEALs are currently forward deployed in support of Operation Enduring Freedom (OEF). U.S. Navy photo by Photographer's Mate 1st Class Arlo Abrahamson. (RELEASED)

Ever Get The Feeling You’ve Been Cheated?

The famous John Lydon quote strikes me as an appropriate title for a blog post on the state of digital forensics academic programs in the United States. I have been a hiring manager for high tech investigations teams since about 2007 and was involved in assessing candidates for the teams that I was before I became a leader. During the early years, it was rare to see applicants who had degrees in digital forensics, but I’m finding it increasingly common in recent years. One of the things that I have been struck by is how poorly most of these programs are doing in preparing students to enter the digital forensics fields.

It’s not just undergraduate programs that are failing to produce good candidates. I have encountered legions of people with masters degrees in digital forensics who are “unfit for purpose” for entry level positions much less for positions that require a senior skill level. The problem almost always isn’t with the students. They tend to be bright and eager people who just aren’t being served all that well. One of the core issues that I see with the programs that aren’t turning out prepared students are the people who are teaching them. It’s almost universal that programs who have professors who do not have a digital forensics background are turning out students who don’t understand digital forensics. This seems like an obvious and intuitive statement, but given how many digital forensics programs there are who are being lead and taught by unqualified people, it apparently isn’t obvious enough.

If you want to learn to be a good digital forensics examiner, you have to be taught be people who are good digital forensics examiners. If you are interested in learning digital forensics from an academic program, it is your responsibility to look beyond the promotional material and be an informed and educated consumer of your education. The last thing you want is a massive student loan and a degree that looks good on a resume, but then falls apart during a technical interview for that great entry level job that you had your heart set on. One of the best ways to make sure you don’t get burned is to carefully study the backgrounds of the professors who will actually be teaching your classes. We’re a bit too early in the development of the digital forensics field to see a host of full tenured professors with PhD’s in Digital Forensics, but that doesn’t mean you can’t screen out professors who have no earthly clue what they are teaching. Pay very close attention to the curriculum vitae of the people who are going to be teaching your classes. Does the CV show any actual interest in the field of digital forensics? I have seen many CV’s for people teaching digital forensics who don’t show any research or training in the digital forensics field. What it looks like is that we have quite a few institutions that have decided that the digital forensics field is hot right now and to capitalize on it, they press unqualified professors into teaching digital forensics classes just so they can lure paying students (and their tuition money) into their programs. Avoid these programs. Your future depends on it.

We are in a time where there are many fine academic programs available to aspiring digital forensics people who wish to learn digital forensics and launch successful careers. Unfortunately, there are more bad programs than good ones. It’s vital if you are going to spend the time and money getting an education that you don’t get cheated. It’s your life and your responsibility to look beyond the glossy promotional material and make sure you are trusting the right people to get you where you want to go.

Microsoft Windows File System Tunneling

I’m way overdue on doing a proper blog post so I thought I’d swing back into action by showing everyone some really esoteric Microsoft Windows knowledge that I picked up by accident several years ago working on a research project. Around 2010 or so, I was working on Adobe Flash cookie research with Kristinn Gudjonsson of log2timeline fame. During that research, I discovered an odd occurrence dealing with Windows date and time stamps which led me into learning about file system tunneling.

So for purposes of this blog post, you don’t need to know anything about Adobe Flash cookies. I presented on them during the CEIC 2010 Conference and end of my presentation I went into the file system tunneling aspect of what I had found. This blog post covers that portion of the presentation. The only thing you need to know for purposes of this blog post is that Adobe Flash cookies, at least back when I as doing the research on them many years ago, can be deleted and then quickly replaced with a file of the exact same name and file location as the previous one.

For purposes of this demonstration, we’re using a Adobe Flash cookie file with the name “settings.sol”. I think I was using EnCase Version 6 for making this presentation in case anyone is curious about the tool being used.

Here is our baseline “settings.sol” file before any changes are made to it.

Here is what happens when a change has been made and the original file has been deleted and replaced with a file of the exact same file name and location. I don’t have the path listed in these screenshots, but the paths of the new files of the same name are being placed in the exact same location as the old files of the same name.

Here is the brand new file with the exact same name in the exact same location:

So we are seeing what we would expect for a brand new file that was created after the old one was deleted…except look at the file creation timestamp. It exactly matches the file created timestamp of the old file that was just deleted. That’s not accurate, is it? We know the file was actually created on 04/01/10 12:17:27PM, but it’s showing the created file time of the deleted file.

Here is another way of looking at it.

Here we see the old “settings.sol” file that was deleted and we also see the new “settings.sol” file that was created in the exact same location. As you can see, the new file has last accessed, last written, and entry modified time stamps that show when it was created, but it has the wrong file created time. It kept the file creation time of the old file of the exact same name that was in the exact same location.

What on earth is going on?

This almost made me go completely insane trying to figure this out. Like any good digital forensics person, I could absolutely not let this one go and had to run it down until I had an answer. I lost track of how many digital forensics gurus that I contacted to get help. At first, I was a little reluctant to do so in case it turned out to be something obvious that I missed. Who wants to humiliate themselves with a flagrant display of ignorance in front of some digital forensics icon? I was able to save face because the responses I received were all along the lines of telling me that I had an interesting problem on my hands and they didn’t have an answer for me.  The person who solved the problem in the end was Eoghan Casey. Eoghan listened to me describe what I was seeing and said he thought I might be observing file system tunneling. I did some further research and it turned out that Eoghan was exactly right.

You can find the official Microsoft write up on file system tunneling at their knowledge base 172190 article at http://support.microsoft.com/kb/172190 The relevant text (and the spelling errors are theirs) from that article is:

The Microsoft Windowsproducts listed at the beginning of this artice contain file system tunneling capabilities to enable compatibility with programs that rely on file systems being able to hold onto file meta-info for a short period of time.

When a name is removed from a directory (rename or delete), its short/long name pair and creation time are saved in a cache, keyed by the name that was removed. When a name is added to a directory (rename or create), the cache is searched to see if there is information to restore. The cache is effective per instance of a directory. If a directory is deleted, the cache for it is removed.

I haven’t performed any recent research to update this information, but at the time I was doing this research in 2010, file tunneling impacted a broad range of Microsoft operating systems including 2k, XP (including 64bit), and NT. Microsoft hasn’t updated the article since 2007, but I’d be surprised if it wasn’t an issue for Windows 7 and potentially Windows 8.

The Microsoft 172190 article goes on to explain that file system tunneling is an issue for both FAT and NTFS file systems because:

The idea is to mimic the behavior MS-DOS programs expect when they use the safe save method. They copy the modified data to a temporary file, delete the original and rename the temporary to the original. This should seem to be the original file when complete. Windows performs tunneling on both FAT and NTFS file systems to ensure long/short file names are retained when 16-bit applications perform this safe save operation.

Microsoft explains how to disable file system tunneling in the same 172190 article. All you need to do is to go to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem setting in the registry and add a DWORD called MaximumTunnelEntries which you then set to 0.

So the Microsoft article talks about “file systems being able to hold onto file meta-info for a short period of time”. The short period of time is a default setting of fifteen seconds. However, if you wish to change that time, it’s yet another registry tweak.  You just have to head over to the same key as you would to disable file system tunneling, but this time you just need to add a new DWORD called “MaximumTunnelEntryAgeInSeconds” and then whatever time you would like. This is all explained in the Microsoft 172190 article.

So like any good forensic geek, I wanted to test what I had found because I didn’t want to look foolish in front of a big audience of people at CEIC. I set up an experiment based on making Flash Cookie modifications that would initiate the same deletion and creation process.  Again, all you need to know for purposes of this blog post is that I was making changes that resulted in near instant file deletion and creation of files with the exact same name in the exact same location.

For example, I used a Windows XP SP2 system that had file system tunneling enabled and obtained these results.

I was making changes that causing these “settings.sol” files to be deleted and then replaced with files of the exact same name. Like before, we are seeing the changes that we would expect from new files being created in that their accessed, written, and modified time stamps reflect the accurate time that they were created. However, all of these new files are keeping the same file created stamp as the original file.  This is file system tunneling at work.

How do we know for sure?  What I did next was to follow the registry modification that I explained before to disable file system tunneling and then to run the experiment again.  Again this was a Windows XP SP2 system, but with file system tunneling disabled.

Now we see that all of the file times are matching the time that the files were created including…finally… the file created times.

And because like all good digital forensics people, I’m always suspicious of what I’m seeing, I repeated the experiment a third time, but this time I used the registry to turn file system tunneling back on which resulted in...

…the exact same file system tunneling behavior that I had observed at the start of all of this.