Here is an interesting bit of work that shows that our brain's language regions can exert a top down influence on early color processing in the visual cortex. The experiments show that colors from different linguistic categories presented to the right visual field (which projects to the left linguistic cortex) caused much stronger activation of visual areas 2 and 3 than stimuli presented to the left visual field (which projects to the right hemisphere). Here is the abstract:
The effect of language on the categorical perception of color is stronger for stimuli in the right visual field (RVF) than in the left visual field, but the neural correlates of the behavioral RVF advantage are unknown. Here we present brain activation maps revealing how language is differentially engaged in the discrimination of colored stimuli presented in either visual hemifield. In a rapid, event-related functional MRI study, we measured subjects' brain activity while they performed a visual search task. Compared with colors from the same lexical category, discrimination of colors from different linguistic categories provoked stronger and faster responses in the left hemisphere language regions, particularly when the colors were presented in the RVF. In addition, activation of visual areas 2/3, responsible for color perception, was much stronger for RVF stimuli from different linguistic categories than for stimuli from the same linguistic category. Notably, the enhanced activity of visual areas 2/3 coincided with the enhanced activity of the left posterior temporoparietal language region, suggesting that this language region may serve as a top-down control source that modulates the activation of the visual cortex. These findings shed light on the brain mechanisms that underlie the hemifield- dependent effect of language on visual perception.