If you're a strong player, but not "famous," then it is difficult to go up against an internationally-famous superstar. It's sort of a David vs Goliath situation.
In addition, Djokovic, Nadal, and Federer all work hard on their fitness, mental toughness, on-court training, sharpness, etc. They travel with a group of family, coaches, PT's, etc helping them. If you lack a similar kind of support system, then you are at a disadvantage before the match ever begins with players like these.
Finally guys like Novak, Rafa, and Fed are *expected* to win. And so, when you're fighting against someone who expects to win, again, it's not easy. If you're unseeded, it's even worse. I'm not saying it's impossible, but it can be very, very challenging, to defeat an opponent like this.