I had that problem where my mp3 would start playing only after 2-4 seconds.
I found out that the problem is that Itunes totally sucks when it's time to decode a mp3 that was encoded with LAME, I even tried different version, I didn't find any way to decode the mp3 faster.
The only way I did find was to convert again to mp3 using the Itunes encoder, then it would be lightning fast to start a mp3 playback.
If someone has a way to speed up the decoding of an mp3 encoded with LAME in Itunes under mac, please tell me, I would appreciate because I'm not sure I want to convert all of my mp3 collection!!