Is it necessary to rebuild your modules when you rebuild the kernel?

Is it necessary to rebuild your modules when you rebuild the kernel?

... if you are using a subset of the original list of modules? I tried this recently and the kernel couldn't boot, and then when I did a full rebuild it apparently worked. But since I only tried this once I can't be absolutely sure that there wasn't some other thing I changed between the first and second rebuilds. I could try it again, I suppose, but it'll take time :P