Electron affinity of oxygen is 141 kJ/mol and electron affinity of boron is 27 kJ/mol. I suspect it is because boron nucleus is much better shielded than oxygen nucleus (in the p-shell), but I am not sure if this is the only factor since the difference is so large.
Oxygen's nucleus consists of more protons than boron an thus its Effective nuclear charge is stronger than in case of boron. This means that oxygen will be "more electronegative" than boron, because his nucleus has bigger + charge and attracts electrons more than boron.
Boron's nucleus is actually less shielded than oxygen's as we can determine from Slater's rule, but it is simply not as strong as oxygen's nucleus and because of that, boron has smaller electron affinity.