The softaholic blog

Requirements definition is often a finite operation, but its branching depth inevitably overwhelms the human mind.

Let’s face it: one person project is fine, two is ok, three or more is a mess. Say you are a project manager; you have two ways to do you job:

• You can get strongly involved in project’s everyday tasks: requirements, architecture, implementation details… you know the project and answers come easily when hard questions arise. Extra points if you code some parts of the system.
• You can use the outsider’s approach and do objective project management: take tasks, take resources, mix’em up and then monitor the whole thing day after day. Extra points if you don’t even know what the project is about or what the programmers do to get that damn thing working.

Either approach is great in my view, but your will to live can really get sucked out in either of them, too. If you get deeply involved, you get no time to do any project management: time flies when you engage into a hard technical challenge.

If you take it impersonally, make sure your life outside work is just great, otherwise it’ll take you less than four weeks to start considering suicide.

I guess it ends up like everything else in life: drawing a line at the right point.

Now that we have our images in a file format readable by any JPEG-capable software, all we have to do is find an automated (or manual, depending on the amount of pictures we are dealing with) way to convert the 2Mb chunks into more optimized files (Photoshop cut my file lengths down to 600Kb per image).

This is no big deal for anyone who understood the previous parts. Gimp, ImageMagick or Photoshop are just a few examples of applications able to read and write JPEG files in a variety of quality and compression levels. As easy as opening each image separately and saving it again with the same file format. All the unnecessary junk will be left behind and the image will look the same using much less disk space.

This is it! I hope someone found this tutorial useful. Critics and comments are welcome.

Okay, so now we have a raw copy of the memory card in a file. What to do with it? The more obvious thing would be to get a hex editor, look for JPEG headers and figure out a way to extract all the data after those to separate files.

Actually, that is what we are going to do but in a more automatic way. Say we can write code in any language powerful enough to do some basic file input/output operations efficiently, then why not writing a short program which scans our big data dump from top to bottom, and then dumps chunks of a fixed size to sequential files?

Now before you get all excited about how easy and perfect this method is, let me tell you one thing: the JPEG format is a tricky lady. Computing the size (in bytes) of a JPEG file from its header requires a deep knowledge of the structures and algorithms involved in the JPEG image compression/encoding standard. My approach is to just extract the 2 MBs of data following each occurrence of a JPEG header (I know my digital camera never produces files larger than that). Part IV will explain how to optimize those 2MB files once we have recovered them.

I could get really technical here on how my code works and all that, but I think there is no rocket science in a pattern search program like this and therefore the source code is way enough to get an idea, so here’s the source code.

Anyone can build the source using a decent C compiler like GCC or Microsoft C/C++ compiler (both are free by the way). Once you have compiled the source code, you can run it from the command line like this:

./jpegrescue memory-card-dump.bin 2097152 jpg

With this, you are telling the program to scan the file “memory-card-dump.bin” and dump the 2 first MBs after each occurrence of the JPEG header, using “.jpg” as the output file extension. And… that’s basically it! after a few seconds (or minutes, depending on the input data size), we should have a bunch of jpeg files located within the same directory we ran the application from. If your card has been used enough times, you’ll be astonished about the amount (and age) of pictures recovered.

Now let’s get our hands dirty. First step is getting a safe copy of our memory card contents into a file within our hard disk. This will allow us to play around with its contents way faster than we would by just reading the card every single time.

If you haven’t written anymore to your card since the lost files where removed, your chances for a flawless data recovery are almost of a 100%. As many files you (or your camera) have written to the card, less chances of getting back your files.

This happens because most filesystems don’t erase file data when the file is deleted. They just mark that space as “free”, so that other files can use that available space. If no files were written to your card after deleting those files, then you can be sure your data is still there, only the path to it is lost.

Back to the memory card data dump issue, we well need to open the Terminal program in order to pull some UNIX swiss-army-knife tricks. I assume your memory card is inserted in your card slot, or camera’s slot while the camera is hooked to your PC via USB. If you did this correctly (and if you didn’t, stop reading this post), you should be able to access the card’s contents using Finder without much problem. Now switch back to the Terminal and type “mount”, then you should get some stuff like this:

/dev/disk0s2 on / (local, journaled)
devfs on /dev (local)
fdesc on /dev (union)
on /.vol
/dev/disk0s3 on /Volumes/Untitled (local, read-only)
automount -nsl [185] on /Network (automounted)
automount -fstab [189] on /automount/Servers (automounted)
automount -static [189] on /automount/static (automounted)

Find out which line represents your memory card device and note down the device filename for it (the first path before the “on” word), stripping the last two characters (sX), where X is a integer number.

Finally, we’ll use UNIX command “dd” to dump all the raw binary contents of the card to a file in our HD (replace “/dev/disk2″ with the path you got in the previous step):

dd if=/dev/disk2 of=~/memory-card-dump.bin

Dumping the data will take some minutes, depending on the size of the memory card and the transfer rate of it. For instance, dumping my humble 128 memory stick took around five minutes.

After the command finishes, you will have a perfect binary copy of your memory card (containing you valued pictures somewhere), ready to be analyzed and processed as many as we need. Don’t you start loving this thing?.

Problem: a butthead like me screws up while importing digital pictures from its Sony digital camera into iPhoto, total mess, no backups, no trash-canned files, nothing… just lost data.

What to do now? a few Google searches reveal no big deal of MemoryStick data recovery tools out there. Every vendor offers its very own pay-your-ass-for-rescue deal and, what the heck, I’m not made of money. But I decide to not give up so fast and try to do the recovery myself.

Most people think that data recovery is a complex process involving rather sophisticated algorithms and intensive processing duties. I totally agree with them, but it turns out in some cases a relatively simple and expensive way to recover lost data is available. You just have to understand a few bits from here and there (filesystems, image file formats, programming, digital imaging tools…) and have some patience.

This is the first of a blog post series (around 3 of them) about how to perform DIY data recovery on any flash memory card with flawless results.

I guess I’ve been too busy lately to post about the main theme of my social life during past weeks: the Apple gathering held last saturday in my old high school. What can I say? there’s nothing better than a bunch of local geeks sharing know-how, experience and, hehe, files.

The meeting started around 11am, when the first fanboys showed up with their MacBook (Pro) computers. When enough people was hooked up to the free WiFi (courtesy of the high school board of management), a superb workshop about Safari 3 addons was held by luman. He screwed up my working copy of Safari 3 but it was on the name of knowgledge: fair enough.

Geeks need to eat, too. So around 2pm a group of intrepid geeks drove to the closest fat dealer (we call it TelePizza around here) and brought the joy of pepperoni and baked mushrooms to the hungry masterminds.

When everyone was fed up with pizza and feeling sleepy, I had the chance to brag about my lame skills on Mac OS X programming, and publitize infoscape on the way, which is always cool. Slides available here.

What else… oh yes! the iPhone brought in by wincas was just as expected: delightful and accessible to everyone. I had the opportunity to play with it a few minutes (while everyone around laughed at my iUnfriendly fingers) and no doubt I enjoyed it. Maybe I should have bought one while in the US.

Wrapping up, it all was worth the organizational effort. The people was plain great, the food greasy, the downloads fast and the time flown. Do join us next time!

My Mantis bug-tracking site is finally on-line at http://cyberguijarro.com/mantis. It hosts my three major (minor) projects: infoscape for Windows, infoscape for OS X and disaster.

Note: Anyone can log-in as guest and report bugs using a guest account. Please navigate to the site for more details.

I’ve been lately working on a project which employed continuous integration as a day-to-day development tool. In brief, continuous integration means sistematically building (and testing) a project to ensure that common problems such as broken builds or unit test crashes are detected as early as possible.

Of course, this doesn’t make too much sense in personal projects, because one itself is the only contributor to the source code base (and we all now how to behave ourselves in private). But in collaborative projects, where many people checks out and in source code on a regular basis, a centraliced build system raising alarms when someone screws up is of great help.

Do you like the idea? well, here’s my suggestion for a cool pants free continuous integration scheme: Perforce (free up to 2 users) for source code control, CruiseControl (open source) for schedulling builds and gathering all the data, and finally infoscape (free, check it out on this website) to get round-the-clock notifications on build events via RSS.

Today the Apple’s worldwide developers conference has been held in San Francisco, for those who haven’t noted it yet. A lot of people has criticized that nothing new has been brought into the table an so on… well, that’s what they say every year.

All the hype has been focused on the iPhone, but nothing new has been said (is there anything else to be said?) besides that the device does not allow 3rd party native applications to be installed on it: only web applications in a sandboxed environment with access to the underlying platform have the green light. I don’t understand why people is so surprised about this when no historically no other Apple device (besides computers, of course) has allowed such thing; just look at the iTV or the iPod (well, this one allows crappy games but they must be signed by Apple).

And now comes what everybody seems to be ignoring but is a real breakthrough: Apple has ported its web browser, Safari, to Windows. This is the second Apple application to run in the Microsoft Windows operating system in history! it is going to compete in the front line with IE7 and Firefox, which is no doubt a very interesting move from Apple.

I’m not saying that it is going to be a success. I never liked how the Mac OS X interface blends into the Windows operating system: it looks misplaced and the GUI performance is a flop. I guess iTunes catched well onto Windows because is the only application officially supporting all iPod features, and there’s not much people out there willing to buy a +999$ accessory to their iPod called MacBook. IE7 is still the best choice for me during my Windows sessions.

Also, after my first day using Safari 3 Beta (on Mac OS X), I’ve found what I was praying for during last two months: a version of Safari supporting tabs which rescues me from Firefox’ continuous memory abuse.Not much to say about the rest of the keynote: full 64 bit support in Leopard (I wonder how this will fit into existing 32 bit hardware), more eye candy on the desktop, time machine and the coolest application nobody uses: iChat.