I've got my mac set up to use my router's DHCP server, but I can specify an IP address if I want to. I'd like to be able to do that with my windows machine as well, but I haven't found anything that lets me do that. Anyone know how?

What use your Routers DHCP Server or Manually configure the Windows Machines IP?

To manually set your PC's IP first know the IP range of your DHCP server, the range of IPs your DHCP server hands out. Pick something outside of this range while still on the same network.

Next go to the control panel and slide over to Network Connections. Open that up and select the connection that you are using (ethernet, wifi, etc) and open up its properties. Under the General Tab scroll down until you see "Internet Protocol (TCP/IP)" Click on that and click on properties.

Select "Use the following IP Address" and enter in the desired IP, proper subnet, and the default gateway which in this case is the IP address of the router. Put the routers IP address in the DNS field as well if thats what you are using for DNS.

Example:
IP: 192.168.1.2
Subnet: 255.255.255.0
Gateway: 192.168.1.1

DNS server: 192.168.1.1