What does the term "OS rot" refer to?

It's exactly what it sounds like. As you use the OS for a while and add/remove software, things get bloated and feel slower due to things being left around, or just too many things installed in general. Numerous registry entries, files, and even startup entries (which can cause those annoying "not found" messageboxes on startup) are left behind by people who design poor uninstallers for their products. A lot of files (from installed programs and the user's documents) take a while to get indexed as well which can also degrade performance.