You can still wear them, just be careful about where you put them! Look for scarves or tops that have a pattern with one of those colours (maybe mixed in with other colours so it's not too dominant) on a neutral background. Or try an orange or yellow skirt paired with a top in one of your neutrals or a colour that suits you better -- that's what I did this summer. I'm not a winter, but to give you an idea, I paired my yellow A-line skirt with: a light-wash chambray shirt / a charcoal grey t-shirt / a bright blue dressy tee / a red tank. Also, pay attention to the particular shades of orange and yellow, because you might find one that suits you better than others. Have fun experimenting...