In this question from JEE $2015$, a surprising neighbouring group participation occurs, and the product turns out to be $(3)$. A link for the mechanism can be found here.
However, why doesn't the same COOH group participate in the bromination of fumaric acid (Or maleic acid for that matter)? We would then not get a meso product.
Neighboring group participation (NGP) in organic chemistry has been defined by IUPAC as the interaction of a reaction center with a lone pair of electrons in an atom or the electrons present in a $\sigma$-bond or $\pi$-bond contained within the parent molecule but not conjugated with the reaction center. The NGP shown in the given reaction create a highly energetic $\alpha$-lactone, but is feasible (Ref.1). As a matter of fact, Ref. 1 has shown participation of not just Neighboring group, but by active carboxylic acid groups on other carbons such as $\beta$ and $\gamma$ to give larger and stable cycles such as $\beta$- and $\gamma$-lactones:
As shown in the above scheme, $\alpha$-lactone is the most unstable as it take few minutes to alkylate. As a comparison, $\beta$- and $\gamma$-lactones take hours to alkylate (all at the carbon shown by the vertical arrow). The experimental rate constants for the nitrosation are found to be $4.8\pm 0.1 \times 10^{-2}$, $1.34\pm 0.02 \times 10^{-2}$, and $\pu{0.89\pm 0.03 \times 10^{-2} M-1 s-1}$ for $\alpha$-, $\beta$-, and $\gamma$-aminobutyic acids, respectively. That means NGP through $\alpha$-lactone is the preferred one if there is a choice. For instance, From the reaction between glutamic acid and dinitrogen trioxide, $\alpha$-hydroxyglutaric acid as well as a $\gamma$-lactone can be formed. It has been described that $\alpha$-hydroxyglutaric acid is formed from glutamic acid when the $\mathrm{pH}$ is increased to 7 after the reaction, while the $\gamma$-lactone remains the end product if the $\mathrm{pH}$ is not subsequently increased (Ref.2):
Thus, when you look at bromination of maleic acid, you'd see there two possible NPG to happen: 1) through $\alpha$-lactone and 1) through $\beta$-lactone. But as seen in above cases, most probable participation should be through the $\alpha$-lactone. If that happens, the product would be the meso-isomer:
However, careful pioneering study this field has shown that it is just through the bromination intermediate (Ref.3), the work of which leads to the bromonium ion intermediate mechanism. The bromonium ion is so strong that the steric hindrance such as the presence of t-butyl group does not alter the outcome (Ref.4)), anti-trans-addition of bromide or other nucleophile (as noted in Ref.3: the new atom will approach one of the carbon atoms from the side opposite to the $\ce{X}$ atom already present). As indicated in the scheme, you'd see the carboxylic group cannot be in $180^\circ$ (look at the Newman projection of intermediate) to bromonium ion to make the necessary participation. Regardless, bromonium ion is much more stable than alkyl azide to be readily replaced by weaker nucleophile such as carboxylic ion.
Nonetheless, as shown in the scheme, even if it is possible to have NGP in the bromination of maleic acid (or fumaric acid), you may not be able to avoid meso-product formation. As shown, maleic acid gives meso-product and hence, fumaric acid would give racimic mixture, if NGP is considered. That would be the same outcome if $\beta$-lactone formation is considered instead of $\alpha$-lactone formation as shown in the scheme.
Note: The fact that brominations with the maleate ion resulted in cis-addition driven by repulsion between the negatively charged carboxylic acid anions being stronger than halonium ion formation is established by the authors of Ref.3.
References:
