Yes. Don't do it. How do you know you will not just reinfect your machine when you move everything back over? Backups are for before you get infected.
Note that doing a factory restore, format, or Windows reinstall are all last resort efforts. You hardware drivers (and updates), all your security applications and other downloaded programs, customizations are all lost, and you end up months or even years behind in security updates. I recommend you follow these
HijackThis Procedures and see if you can't rid your machine of malware first.
If a restore is the only option, then I would find a trusting soul with a very well secured computer to temporarily attach your hard drive too, then use that machine to scan for malware, and backup any data.