Yeah, I would agree with BS and think that It's a driver problem and not heat. There is known issues with some cards getting stuck in endless loops and causing the symptoms yours displayed.
As for the 'heatsink' lubricant that goes between the actual heatsink and processor: The purpose of it is to fill and tiny crevasses in the metal that air could possibly get trapped inbetween them. Another thing to reduce this is called 'lapping'. It's sanding the surface of the heatsink to a mirror finish. then just apply an extremely thin amount of lubricant across the entirity of the 'sink to fill any remaining on the processor shell. Here's a guide i found: http://overclockersclub.com/guides/howtolapvideocard.php
As for running the machine with the case off, it would be fine as long as you put a room fan blowing on it which I think he said he was doing. You just don't want the air standing still inside of your machine.