It'd be an enormous hassle. What minerals would you use? Some could be faked pretty easily. Denominations would be a problem--for instance, how would you buy a candy bar with gold? How would you buy stuff online? How would credit cards work? How would you deal with exchange rates when the prices of precious minerals are always fluctuating?
I worked in machine shops for many years and synthetic rubies were very popular for tooling, for a hard, shiny surface that doesn't wear--like the jewels in watches. You can buy a 1/4" ruby sphere for like $20. And the only way to tell it from genuine natural ruby is that it doesn't have any flaws! Natural rubies are almost impossible to find these days and the price of gem-quality natural rubies is just astronomical! Likewise cubic zirconium--the only way to tell them from real diamonds is that they're so big! (We used man-made diamonds too, of course, but those wouldn't fool anyone).
I think the big mistake is the idea that natural materials have intrinsic value. Gold, just for example, is worth a lot only because people agree it's worth a lot. It has very few actual practical uses, mostly to coat electrical contacts. In a SHTF situation, where nobody was making electronics, gold would be worthless. You can't eat it, or burn it to keep warm!