Is it rude to wear a hat indoors?

Ladies hats may be worn indoors, presuming (as you point out) they do not block the view of others. It is customary for a gentleman to remove his headwear indoors in western society, unles it is in a situation where it is appropriate to retain it such as a mosque, synagogue, etc.