How to replace the video card?

"There are three video boards used in this system. Depending on your model here are the part numbers:

661-5967 - AMD Radeon HD 6770, 512MB661-5968 - AMD Radeon HD 6970, 1GB661-5969 - AMD Radeon HD 6970, 2GB

As to replacing the video board you will need to completely remove the logic board from the system. There is no IFIXIT guide to do this fully, but this guide will get you mostly there: Installing iMac Intel 27"" EMC 2429 Dual Hard Drive Kit.

As to your heat problem:

This iMac was the first to use the internal thermal sensor in the HD. If you replaced the HD without using a special cable or one of the newer HD which now have the compatible sensor you will find the systems fans will ramp up to full power and stay in that state. Some people attempted some software hacks which to manage the fan speed we found this solution was not very reliable as it didn't allow the discreet fans to run so the GPU would over heat burning it out. Which is what likely happened to you.

So what to do here?

You can get this OWC In-line Digital Thermal Sensor for iMac 2011 Hard Drive Upgrade or you can replace the HD with a newer one that is compatible like the Seagate 3.5"" SSHD drive. Make sure the box has an Apple logo on it (the newer version).

Update

Seagate no longer offers the needed firmware on their drives it looks like only the first few had it and then they took it out ;-{ Apple may have told them to.

Before you do anything you might want to see if the sensors and fan units are working correctly. Using a second Mac putting your system in Target mode as the bootable disk install Temperature Gauge Pro. Once you have it installed you'll need to also setup screen sharing so you can see your system. Now restart your system and your second mac then access yours via screen sharing you should then should be able to see what your systems thermal sensors and fans are doing. Take a snapshot and post it here when its running on the hot side so we can see whats happening."