Jimmy Weg is one of the most talented people that I know in the digital forensics world. One of the things that I recommend to digital forensics examiners is that they work hard to establish a network of sharp people who they can call when they get stuck on something. There is just too much for any one person to know and it is important to have a group of people who are smarter and more talented you who you can reach out to for help. This interview with Jimmy is a fine illustration of why he is one of the first people I reach out to when I’m really in a bind and can’t figure something out.
Professional Biography of Jimmy Weg
I’m a graduate of Fairleigh Dickinson University, Rutherford, NJ. I was a senior examiner for the National Association of Securities Dealers in New York City between 1973 and 1977. From 1977 to 1996, I was Chief of Enforcement for the Montana Securities Department, where I headed the agency’s law enforcement efforts against securities fraud. In 1996, I became the first Supervising Agent of the Medicaid Fraud Control Unit, Division of Criminal Investigation (DCI), Montana Department of Justice, and supervised investigations of Medicaid fraud and patient abuse. Since November 2000, I’ve been Agent in Charge of the Computer Crime Unit of DCI. I conduct and supervise computer forensics for state, local, and federal agencies, and I am an IACIS Certified Forensic Computer Examiner (CFCE).
AFoD Blog: How did you get involved in digital forensics?
Jimmy Weg: I always was short on patience. So, back in the days of the Commodore, the prospect getting information quickly, or "right now," fascinated me. I started my career as a white collar crime investigator and headed the securities fraud law enforcement program in Montana, after working on Wall Street in regulation (I’m a native New Yorker, raised in Jersey). I can remember tracing money and documenting financial transactions on columnar pads with a pencil. I’ll bet that there are people on the lists who don’t what a columnar pad is, and they should be grateful for that fact. There was no “Delete” key, and an eraser was your friend, as long as you didn’t wear out the paper.
Anyway, a company named Wang showed up with a program named 20-20 Spreadsheet. It was amazing. (Remember Lotus 1-2-3?) It added, subtracted, and I could “erase” my errors and start anew in a matter of seconds. The next thing we found, was that crooks also could use electronic paraphernalia, and what were we going to do when we got hold of a box of floppy disks? I heard a tale that deleted files on a floppy weren’t really deleted. That concept was the one that really piqued by curiosity. So, in 1995, I was off to the Federal Law Enforcement Training Center for a two-week course of electronic investigations. We learned the basics of FAT and DOS and all about the most powerful forensic tool of the day: Norton Utilities. I still have a copy of Norton DiskEdit around somewhere. Following the FAT and cluster chains was thrilling, I thought. I came back to work and preformed magic tricks by recovering delete files for my coworkers, just for fun. They were shocked, to say the least. It’s funny, but even today, many folks don’t realize what lurks on their computers.
AFoD: I'm not worthy. You started back in the hex editor days of digital forensics. I never cease to be impressed with the folks who were doing digital forensics in the era where there were few specialized digital forensics tools. What was the first actual specialized digital forensics program that you remember using?
WEG: My first “all-in-one” was iLook, which was law-enforcement-only back then. Elliott Spencer was the developer and was aided by some remarkably talented people, like JB (Jim Bob Baker) and others whose names escape me. It also was free, as major funding came from the IRS, if I recall correctly. So, not only did I have a tool that, IMHO, rivaled the top commercial tools, but also was supported through a great online forum. I should add that iLook still is around today in a commercial form and, from the previews that I’ve seen, it’s still quite a tool. Many law enforcement agencies would not have been able to get into forensics, but for iLook. Of course, it’s easy to fire up a debate among our colleagues when it comes to tools.
I also should mention the pioneer of forensic imaging: SafeBack. It ran from a floppy and we added drivers and the like for SCSI support and often imaged to tape. Otherwise, imaging usually meant cloning and booting with a floppy that locked the drives. That reminds me of Digital Intelligence, which offered a suite of very handy DOS tools, like PDBlock, which was my forensic boot disk.
Way back then, I also recognized that I needed variety in my toolbox. I started early with DataSniffer, as it was called at the time. Steve Payne and Randy Becker developed it, and it was a great file carver. It actually was quite advanced for its time, and is around today as DataLifter. I also was one of the early users of NetAnalysis, and swear by it to this day. Craig Wilson, the developer, is obsessed with accuracy, and his passion for his tools always has given me a great deal of comfort.
It’s fun to think back to the old days, but scary to think where we would be today without the advances that we’ve made. Aside from the folks whom I’ve mentioned, there are a number of forensic pioneers. There are lots of great tools out there.
AFoD: I'm getting close to being in the digital forensics field for a decade now. We've seen quite a bit of development when it comes to tools and more than a little bit of competition recently between the various vendors. What are the primary tools that you find yourself using today? I'm curious, for example, what your favorite tools are for file system digital forensics work and well as more specialized work like Windows registry examination.
WEG: Anyone who has seen my posts on this topic knows that I’m a huge fan of X-Ways Forensics (XWF). I can’t say enough good things about it, but I will make only a few comments so that I don’t make this answer as lengthy as War and Peace. XWF does everything that each (FTK and Encase) of the major tools does, but I liken it to FTK 4.x on steroids. Imagine running four instances of EnCase or FTK on one machine, let alone from a thumb drive. Imagine asking a tech support question and receiving an answer on Sunday afternoon from the guy who wrote the program. That “guy” is Stefan Fleischmann, who is on my list of “all-time forensic gods.” I submit that he one of very few expert authorities on file system forensics. I use XWF on every case.
Nevertheless, an examiner needs a toolbox that contains a variety of applications. Since you mentioned the registry, I believe that this is one area that requires special attention. We should be grateful to Harlan Carvey for making examiners more aware of the wealth of information that lies within the registry. I use RegRipper on many cases, and I also use AccessData’s registry Viewer. I think that we need to remember that case work among practitioners varies. As a law enforcement examiner, my assignments differ from yours. We have different needs, insofar as evidence is concerned. In my average image/video cases, XWF’s registry viewer works superbly. However, when I really need to explore the bowels of the registry and focus on a variety of artifacts, I call upon RegRipper. For reporting results to my clients, I like Registry Viewer. I also use Mark Woan’s RegExtract frequently.
I should add that most of my testing scenarios involve the registry. I want to see how real life actions affect the registry. So, I also rely upon the Sysinternals (MS) suite and tools such as RegShot and RegDatXP. I’ve found that it’s easier to test than to research an issue; it’s more reliable, too.
I mentioned NetAnalysis before, and I use it on almost every case. I’m becoming more accustomed to using Jad Software’s Internet Evidence Finder as well. I have a ton of specialty programs at my reach, and I have a Start menu folder named Forensics, which contains them. They may not be “forensic” applications, but they’re essential. Like SamInside.
I also want to give a plug to Mark Woan, of Woanware. His RegExtract and JumpLister are superb tools and are free! I think that Windows 7 jump lists are the most valuable forensic artifacts to arise in recent years. The accuracy of Mark’s tools is spot on, and he has implemented “wish list” requests overnight. He is a great resource.
I use Paul Sanderson’s tools as well. I really like SkypeAlyzer for Skype cases. Paul’s research, alone, is worth the price of all of his tools.
That also brings to mind that we shouldn’t consider tools, alone, as resources. We should be grateful to those folks who publish papers based upon many hours of research. Aside from validation, I never use a tool unless I understand what it does and what the artifact means. So, I do my homework, determine why I should look for a given artifact, and then look for a tool that finds it reliably.
AFoD: You are bringing up a great point. As much as we rely on our tools in this field, it's ultimately the person behind the tools that makes the difference. What do you think makes a good digital forensic examiner? What qualities will someone who wants to be successful in this filed possess?
WEG: A competent examiner possesses many qualities, so I’ll try to limit my list to some that I believe are essential. The questions go hand in hand. It’s understood that a sense of fairness and adherence to ethical standards go with the job. I won’t recite them, but the IACIS standards are a good model for cops and private practitioners alike. This business is not a contest to see who wins, but is a quest for facts, regardless of whom they favor.
Most everything else relates to skills. A good foundation is an understanding of file and operating systems. After all the years that NTFS has been around, we’re just learning now how different artifacts can be valuable in certain cases. Just think about the different time stamps and system files that, a few years ago, were never considered. Back then, none of the major tools presented these artifacts, at least not in a friendly fashion. However, if you know the file/operating system, you’ll find a way explore beyond what’s presented.
A successful examiner understands that no field changes more rapidly than computer forensics. We must set aside time every day to learn and explore. “Remember what the dormouse said: feed your head” (Grace Slick, loosely paraphrasing Lewis Carroll’s Alice in Wonderland). What I mean is that a successful examiner will have a hungry mind. What’s great about our profession is that there are so many sources to satisfy our minds’ appetites. Just look at the forums and lists that you and I have joined!
Speaking of forums and lists, never be hesitant to ask questions. I’ve heard anecdotes about the opposition using our questions as a sign of ignorance, yet I haven’t seen anyone cite an example where the opposition scored points with such an approach. I suggest that, if an opponent had, the examiner or the prosecutor/counsel did not respond as well as he or she could have.
Try to answer questions as well. If a question pops up and you don’t know the answer offhand, do some research and offer your findings, at least if you find the question intriguing. You’d be surprised at how much you can learn and retain by doing just that. In other words, you don’t need to know the answer at the moment to answer the question! A successful examiner is willing to help others because doing so pays dividends to all concerned.
Next, never be satisfied with a tool out-of-the-box. A successful examiner takes nothing for granted. Validate and re-validate. Be thorough. Double check everything before it leaves your shop. Don’t be satisfied with mediocrity, but be persistent so you know that you met the goals of the assignment.
In my view, a successful examiner writes well. I think that our clients and opponents size us up by what we produce and how we present our product. Good reports are critical. I’ve found that my audience enjoys reports that are concise and cater to non-techs. I encourage everyone who puts pen to paper to get a copy of Patricia T. O’Conner’s Woe Is I: The Grammarphobe's Guide to Better English in Plain English. I don’t like 50-page reports that include 40 pages of screen shots, most of which are “bulk.” I rarely put them in my written reports, though I won’t fault those who use them wisely. Unless you’re a one-person shop, let a teammate review and critique your reports. A touch of peer review is a great asset.
A good examiner thinks on his or her feet, yet doesn’t rush to conclusions. Learn to speak well, too. Most folks who enter this field will end up in court one day. A lesson on courtroom testimony is beyond the scope of your questions, but anyone who is comfortable in addressing an audience and is a good listener will have an advantage.
A successful and examiner is fair minded and is not afraid to change his or her opinion. Today’s research may be proven wrong tomorrow. Count on the unexpected. It’s how you deal with it that can make you rise above the average.
Lastly, before I go too far in terms of length, a successful examiner is a planner. He or she approaches an assignment by studying the case background and setting out a plan to cover everything that may be relevant. The examiner also will cover the basics in every case. The job’s not over when your report leaves the shop. Follow up and be sure that your client understands what you submit.
AFoD: It's timely that you mention learning and writing because you are combining both of these in your new JustAskWeg blog. What are your plans for the blog and why did you decide to create it?
WEG: To be honest, it just seemed like a neat thing to do. I've received many questions requests for advice (which I welcome) on creating virtual machines. I already had sent out quite a few copies of some of the information on my first blog post. I thought that videos would provide a great way to explain the process, so a blog seemed like the ideal way to reach the largest audience. I’ve also seen folks struggle with creating VMs when they really don’t need to face any hurdles.
I had started a long time ago with VMs, and did some testing for a fellow named Michael Penhallurick, the developer of Virtual Forensic Computing (VFC), which, IMHO, is the leading tool for automating the process of creating VMs from images. I learned much from Michael, and some of his research may still be out there. Nevertheless, many of my colleagues, particularly in law enforcement, can’t afford a variety of commercial tools, but also had problems with the major free, open source, VM-creating tool. So, why not lend a hand?
I suggest that creating VMs is one of the greatest advancements in forensics in recent years. I use them almost on every case. In many instances, it’s the quickest, fail-safe way to check configuration settings of applications. You can run programs in their native environment and even hook up a network adapter and do some testing. In that respect, readers need not go beyond my first blog post.
As I go forward, I’ll use a generic, Windows 7 VM to examine shadow volumes. It’ll be a VM that includes X-Ways Forensics and a few other tools, and the reader can add anything he or she desires. I call it the “SEAT” workstation: Shadow Examination and Analysis Toolkit . It’s nothing technically special, but affords an easy way to examine shadow volumes, which should be around in Windows 8, too.
My blog plans are short term. Getting through the shadow volumes is going to take a little time, which is a scarce commodity for me. Looking ahead further, I hope to do some tutorials on forensic approaches to issues, principally with X-Ways Forensics. However, I’m no competition for Ted Smith, who’s published some remarkable tutorials on X-Ways Forensics.
At the moment, I have no plans to write a regular forensics journal of a nature like yours and Harlan Carvey’s. You guys (and several others) do an incredible job of putting out a wealth of information. I simply don’t have the time, or at least I don’t manage my time as well as I could. Then again, if I think of useful topics about which I can write a few paragraphs, I may post more often.
AFoD: For the benefit of people reading this who might not be up to speed on the topic, can you explain what shadow volumes are and why they are important?
WEG: I’ll try to answer with a concise, not-too-technical explanation, as there is a lot of stuff out there on shadow volumes and I don’t want to suggest technical completeness and be technically imprecise. So, let’s start with the statement that a shadow volumes (SV) is a creature of Windows Vista that has carried over into Windows 7 and, from what I understand the moment, will appear in Windows 8. Simply put, SVs are time machines that the system creates periodically based on elapsed time or events.
The Volume Shadow Copy Service (VSS) is a process that makes backup copies (snapshots) of files and folders on a volume at specific points in time. It’s used by Windows System Restore, which allows a user to undo changes made to the operating system and recover from system failures. The System Restore feature automatically creates “restore points,” which users can employ to revert to a previous time. Restore points are created at the time of significant system events (certain program installations) as well as periodically or they may be created manually. VSS also supports a feature known as Previous Versions. Certain editions of Vista and Windows 7 allow a user to restore previous versions of specific files with that feature.
VSS allows an individualized restoration by creating block level, differential backups of files. In simple terms, changes to a file are recorded, and the file can be restored to how it existed at a chosen point in time by assembling the original file plus subsequent changes. Using System Restore will recover the objects that System Restore includes. System Restore does not necessarily include every file that was backed up by VSS, so it may not recover a previous version of a certain file.
Although System Restore and Previous Versions derive their contents from the same snapshot data, each feature provides only the recovery of data specific to the feature, and the data differ from one feature to the other. As restore points are created, files are backed up to those points in time. A file that has been deleted irrecoverably today may be available within VSS. VSS is akin to a “time machine,” which allows a user to travel back in time to visit a file, as it existed previously.
A case in point: Joe routinely downloads videos through peer-to-peer (P2P) file sharing programs and deletes them periodically. Moreover, Joe deletes each file with a robust file wiper. So, Joe download files on Friday, and a SV is created on Saturday. Joe wipes the files on Sunday. On the following Thursday, the cops arrive and seize Joe’s machine. Although Joe’s day is ruined, he feels secure in the thought that he’s wiped all of his videos.
On Friday, Mary Examiner observes no videos in the current P2P downloads folder. She tries to carve videos, without success. Mary knows that videos can be difficult to carve, as they’re large files, possibly fragmented, and more likely to be overwritten, at least in part. However, Mary’s aware of the power of SVs. Using her method of choice, Mary accesses the SV that the system created last Saturday. There, she finds all of the videos that Joe wiped on Sunday. The files are intact, with all of their metadata. Of course, Mary also can study link files, the registry, and other artifacts, within or without the SV, to determine whether the videos were viewed.
One more case: After he’s released from prison, Joe buys another system and returns to his former ways. This time, he keeps his laptop in a box buried in the back yard. However, after tripping over the dog one chilly evening while retrieving his stash, Joe decides that there must be a better way. His former roommate at Butner had mentioned a program named TrueCrypt. So, Joe gets the app and creates an encrypted container. This is pretty slick, Joe thinks. “Just let ‘em try to find my videos now,” he exclaims. However, the cops come back the following week. I’ll cut to the chase. Many of the videos that existed on the system before TrueCrypt was used are in the SVs.
The possibilities are endless. But, so is the amount of data now presented to an examiner. A SV is just that; it’s a volume. The number of SVs can be quite large, depending on a few factors, and I’ve seen more that 30 on one system. Do you want to image 30 volumes? One case turns into 30. Fear not. There are a few approaches to manage the load, ad my blog will review some.
AFoD: So you mentioned that Mary should use her method of choice for accessing the system volume. What are the various options available to her? Has anyone created any tools that make life easier on examiners doing system volume work?
WEG: I have my current method posted on my blog. I simply have a base VM and add the target system (virtual disk) to my VM. Then, I’ll use X-Ways Forensics and Dan Mares’ VSS to examine the SVs. You also can boot the target system VM and run tools like X-Ways and VSS from a thumb drive directly in the target VM.
If you don’t have VMware, I understand that there are similar virtualization tools that may work. I haven’t used them, so I can’t comment on their effectiveness. To be honest, I think that every examiner needs a virtualization tool. One always can clone the image to a disk and attach the disk to a Win 7 box. To me, that takes too much time. I also believe that EnCase has as native virtualization add-on or the like that works, but I’m not an EnCase user.
ProDiscover from Technology Pathways does a nice job of mounting SVs from an image added to the case. Chris Brown kindly gave me a temporary license to work with the tool and propose suggestions, and I think it’s coming along nicely. A user can choose to mount any or all SVs on a system and then examine them within ProDiscover.
There are some Linux mounters out there, which I haven’t tried. Of course, the point is to examine SVs after mounting them, and you could do so in the SANS SIFT Workstation. I’m a huge fan of SANS and Rob Lee.
There’s also a tool named Shadow Scanner, which is produced by EKLsoftware. I did a little testing of it, and it works quite well. It’s designed to do an object comparison among SVs, and I think that’s the primary focus of a SV exam. I should reiterate that “focus” is critical in SV exams. We’re (or I’m) not going to image 30 volumes that exist on one system. I also hope to do a blog post using Shadow Scanner, as my approach is a little different from the method that the authors describe on their site. The site has some great videos, and it’s worth a look.
There probably are other tools out there, but the above tools or methods are what come to mind. I also should add that I don’t examine shadow volumes in every case. It’s a matter of considering the nature of your case and what’s important to all concerned. Then you can judge the likelihood of evidence existing in the SVs. Moreover, you should review the timing of SVs. You may find several that were created within a short period. Consider whether you should study all of them.
AFoD: Thank you for being so generous with your time and knowledge. Is there anything else that you'd like to share with the readers as we conclude this interview?
WEG: Actually, I don’t have much to add, which speaks well for the thoroughness of your interview. I’ll re-emphasize that learning and sharing should be part of our everyday routines. There is way too much information out there for anyone to master completely. Fortunately, many practitioners out there know almost everything about a few subjects and share their knowledge. Note the word “almost.” I point out that word because even super-forensicators ask questions now and then. Perhaps that’s part of the reason why they’ve achieved such success.
Nevertheless, I also think that we should recognize that many of us have more to do than ever before, and resources, at least in government, aren’t expanding as quickly as our caseloads. I understand why some examiners simply have to be “consumers” of list and forum posts, and that’s okay. Read what you can, and save what’s important. I use a little app that pops up with every (100+ per day) email/post that I receive and displays the subject and first few lines. If it’s something that interests me, I go to the message right away. I have an extensive library of list posts (in a PST) that go back as far as 2000. I search it often.
I think the future holds great change in the way we approach our jobs. I see a need for more specialization in the systems and devices that come through our doors. Today, a significant number of list posts concern smartphones. What a difference from a couple of years ago! A fellow from the RIAA mentioned that more children own cell phones than books, and that includes the entire world. Didn’t the Encyclopedia Britannica go out of print recently?
When all is said and done, after a long day at work, I look forward to heading home and having a pint of a local microbrew and dinner with my wife, Kelly, who’s quite tolerant of my schedule and work habits. I also treasure my nightly conversation with my daughter, Kristen, who lives in Las Vegas, and checking on my one-year-old granddaughter, Zoey, who is the newest apple of my eye. In closing, I’ll share what I told Kristen years ago. Kindness is one of the greatest human attributes. Use it generously. Thanks for taking the time to consider my thoughts.
