The router firewall should be safe to rely on for a short period of testing. In fact, some people rely on their router firewalls entirely, and don't use a software firewall, but personally, I prefer the safety blanket of both. Just test it with a safe website (such as here), and you should be fine, or, as you mentioned, use Windows Firewall temporarily if you are particularly concerned.