Does not using margarine or butter (and which do you prefer) make a noticeable difference in overall taste and/or texture of mashed potatoes?

Update: Can't I put a little more water and/or milk instead to compensate?
Update 2: Is it ok to use 1% milk ? I normally get 2% but am well stocked on 1% and can't bring it back to the store ..
11 answers 11