Olive oil is a triglyceride, and the main fatty acid is oleic acid below.
Now the bromine does not add to the double bond between oxygen and carbon; instead the bromine reacts with carbon-carbon double bond and add itself to it. If you use e.g. coconut oil which has minimal unsaturated (meaning having carbon-carbon double bond) fatty acids (and assume you don't shine light onto it, heating it or otherwise forcing the reaction to start) there should theoretically be no reaction.
But why would bromine want to break the bonds?
Simply put, bromine loves electron. The carbon-carbon double bond happens to be rich in electrons, so the bromine binds to it. While the carbon-oxygen double bond also has electrons, oxygen loves electron more than bromine and tends to keep the electron for itself, so bromine cannot add there.
Does this increase the viscosity of olive oil?
The individual molecules that make up olive oil is held together by weak forces, that arise from van der Waals' forces, which is attraction due to small, unevenness in electron distribution (which leads to very small positive and negative charges on the molecules that subsequently attract other molecules).
This is overly simplified concept though so it's not entirely accurate.
Bromine atom is very large (hence bromine, which molecules consists of only two bromine atom, exists as a liquid in room temperature). The more the electron, the more likely there are some unevenness in electron distribution, and hence stronger attraction between molecules. Also, bromine loves electron so it tends to pull electron closer to itself, creating a small but rather permanent positive and negative charge on the molecule. This also strengthen the force between brominated oil molecules, hence the viscosity goes up.
Viscosity goes up because less double bondings makes the fatty acids more “packable”
This is rather about melting point of the fat / oil, as when freezing, the molecules in liquid need to "pack" into a solid structure. In liquid however, molecules tends not wanting to maintain the "packing", or ordered structure. Adding bromine does increase the melting point but that's another story.