It's not a big deal if you have a router and/or firewall in place. IP addresses are not secret. If the person wants to do harm, he can start probing your ports to see if you have any exposed services he can exploit. If you don't you'll be fine. Denial of Service attacks are also possible, but rarely targeted at individuals, because there is not much to gain.
