In order to open MP4 (and other video files) in VirtualDub you need a plugin called 'Virtualdub FFMpeg Input Plugin'.
Download the plug in and unzip it in VirtualDub's folder. The ZIP contains two folder one for Windows 32 bit and one for Windows 64 bits. You can delete the one that you don't need.
In my case, the plug in installed looks like this:
This plugin is freeware and open source.
And yes, it also works with H264 (H264 MPEG-4 AVC) movies. Enjoy :)
A video editor that has EMBEDDED (yes, no codec needed) support for MP4 is AviDemux.
Avidemux is a free video editor designed for simple cutting, filtering and encoding tasks. It supports many file types, including AVI, DVD compatible MPEG files, MP4 and ASF, using a variety of codecs.
It is not quite as sophisticated as VirtualDub but is definitively much more easier to use. I use it mainly to crop movies, and as I said it works with many popular formats, right out of the box.
Video related tools: