Meaning working in the yard or garden. Comes from the fact that farmers when working in their fields used to wear straw hats.
I was wearing the straw hat this weekend. (working on the yard)