TL;DR using the IP address is worth a try in an emergency, but expect problems.
First of all, with cheaper hosting providers it's common for several sites to share an IP address. The browser tells the server which site it wants, based on the URL. If you type the IP address rather than the hostname, the browser doesn't know which site you want, so it can't tell the server.
You can try it with any wordpress.com blog, for example. My DNS server says the IP address of retroshareteam.wordpress.com is 220.127.116.11 (actually the IP address changes each time I look it up, because wordpress.com is load-balanced across multiple IP addresses - but that's not important right now). But visiting http://18.104.22.168 in the browser doesn't work - I just get a generic wordpress.com error page, because my browser didn't tell the server which site I wanted.
A second problem is that if you try to visit the HTTPS version of a site, the browser will complain that the connection is insecure, because the name on the site's certificate (for example twitter.com) doesn't match what you typed into the browser (for example 22.214.171.124, which is one of the IP addresses for twitter.com). So if you try to visit https://126.96.36.199, your browser will warn you not to continue.
A third problem is that many web pages try to load parts of the page, such as images and scripts, using URLs that are based on the hostname you typed into the browser. If you type the IP address rather than the hostname, you may be able to fetch the main body of the page but other parts won't load, or it may not even be possible to view the main body of the page. It depends on how the site's put together.
You can see this in action by visiting http://188.8.131.52, which is one of the IP addresses for nytimes.com. The masthead loads, but none of the articles appear.
A final problem is links. A link in a web page can either be relative or absolute. If it's relative, the URL that the link points to is based on your current URL. If you're using the IP address instead of the hostname, relative links will mostly work. But if the link is absolute, the URL that it points to is unrelated to your current URL. So if you want to follow the link, you'll need to copy and paste it into the address bar and replace the hostname with the IP address... which gets boring pretty quickly.
Again, you can see this by visiting http://184.108.40.206 - the link to the World section points to http://www.nytimes.com/pages/world/index.html, not to http://220.127.116.11/pages/world/index.html, so it's an absolute link.