No, it most likely will not affect your click rates. However, you should always keep in mind that Pinterest is a visual platform.
While branding is not extremely important on Pinterest as most people are not expecting to see the exact same thing on the website, the importance of branding differs between bloggers or product sellers. The latter might want to stick more to brand colors than the first.
In both cases, feel free to A/B test with different tones of background colors to find out what catches your audience best.