By definition, natural selection IS "survival of the fittest," and genetic mutation creates the random variation which serves as natural selection's raw material. Certainly genetic mutation probably happens a lot more often than selection does, however, again, if you're talking about "fitness," than you're automatically talking selection, so THAT would be more important.
However, natural selection is not the only mechanism by which evolution can occur. There are other possuible mechanisms. For example, and keeping on the subject of mutation, it's possible that not every evolved trait has done so via selection, because not all traits are necessarily selected for. Some may have become fixed in the population through genetic drift, followed by random events that just happen to favor one, selectively neutral trait over others. This is the whole basis for the so-called "neutral theory." Note that drift is more likely to influence evolution if it's occuring in a small population. In large populations, the effects of drift tend to get diluted out, and so are less likely to influence allele frequencies for very long. But regardless of how much it actually contributes to evolution, genetic drift is still always going on, because of the random nature of genetic mutations, recombination and chromosomal assortment during formation of gametes, and it's likely a major source for intrinsic genetic variation in populations of organisms.