Computer turns itself on after any off mode

"This turned out to be two problems.

The problem is now, whenever I switch
the psu on and try to boot, it doesn't
always turn on. It takes a good amount
of flicking the psu switch on and off
before the motherboard lights up.

As quack pointed out, the PSU was a likely candidate. It was clearly dying, so I replaced it. Now the power always turned on. However, the following problem remained:

Whenever I shut down my computer, or
put it in sleep/hybernate, it turns on
after two seconds. It doesn't post, it
just powers on and then idles.

While attempting to solve this, I went about the following:

Cleared the BIOS many, many times
Tried all possible permutations of bios options relating to power and sleep
Systematically removing hardware/peripherals and then booting/sleeping, until I ended up with only motherboard, cpu, psu, one stick of ram, and a power cord connected (I scheduled a task that put the computer to sleep one minute after boot to test)
Investigated the event viewer for problems
ran powercfg -LASTWAKE to get a report on what woke up the computer

This led me to replace the motherboard. Now I have no problems either booting or sleeping. So in conclusion, the replacement of both the PSU and the motherboard solved the problem."

"Try clearing the BIOS by completely removing power and using the CMOS jumper. Then take BIOS default settings and go into ACPI and turn off any S2/S3 sleep or hibernate modes. Look for and turn off any settings about what devices can wake up the machine, particularly network.

If that doesn't work also check try toggling any BIOS settings about legacy versus USB keyboards and mice."