I have tried searching for what type of organic reaction this is in my textbooks and ended up with nothing. Thus, I could not fully understand and refer to the mechanisms that the solution has explained.
So, please tell me the types of organic reactions (i.e. name the reactions involved for me) that are involved in this question. If possible you can describe how the mechanism of this reaction. Thank You!
tl;dr A slightly anthropomorphic approach with some basis in general organic chemistry:
Here, we have three options – we can protonate the nitrogen (deactivated by the resonance with oxygen), the alcohol group ($\mathrm{sp^3}$) or the carbonyl group ($\mathrm{sp^2}$). As you can see this is basically an acid-base reaction and therefore the strongest base (having highest electron density) would react the fastest and this means since the carbonyl has the highest electron density here the carbonyl is protonated.
Well, we began with $\ce{HCl}$. Time for the $\ce{Cl-}$ to make an appearance? Not really. As you can see, we have an alcohol group in the compound which has electrons to donate. The carbon on the (protonated) carbonyl group is highly electropositive (It was positive enough when there was no positive charge on the oxygen and now it just got more electropositive). Now, the oxygen with the lone pair makes an appearance. It attacks the carbonyl carbon and at the same time the π electrons move to the previously protonated oxygen making it neutral.
Now, as you may have learnt, oxygen is not the most electropositive element. If possible, it doesn't stay positive. Nitrogen comes to the rescue. Another acid base reaction takes place where the newly active amine group deprotonates the oxygen forming $\ce{>NH2+}$.
We still need the final isomer and we also need to remove the positive charges on the compound before we end. Therefore, the ring breaks and the electrons move to the nitrogen removing the positive charge on it. This also means a carbocation is formed in the carbon where the alcoholic group is attached which stabilises it using resonance to form a $\ce{C=OH+}$ bond. (try drawing this step).
But we aren't done.
Nobody needs any positive oxygens, do we? Therefore the oxygen is deprotonated. (Note: this is in equilibrium since medium is acidic). All $\ce{H+}$ used has been released which also makes this a catalytic reaction.
There might be one final question in the back of your mind (for the curious).
This is because as @user55119 very well pointed out in a comment to the other answer. My additions in italic.
In the absence of acid, heating amino ester (a) would give the starting material, the hydroxyamide because esters are more labile (easily broken down or displaced) than amides. The reason this reaction can occur is that the amino group is protonated in acid which shifts the equilibrium.
Welcome to the realms of organic chemistry.
I have commented previously that the success of this reaction depends on the protonation of the free amino group of answer (a).
In the absence of acid, heating amino ester (a) would give the starting material, the hydroxyamide because esters are more labile than amides. The reason this reaction can occur is that the amino group is protonated in acid which shifts the equilibrium.
The specific p-nitrophenylbenzamide of this question could not be found in a Chemical Abstracts search but a study from 70 years ago was uncovered that substantiates my previous comment. Fodor and Kiss[1-3] treated hydroxybenzamide 1 with 2.0 equivalents of absolute ethanolic HCl at room temperature producing a mere 9% yield of amine hydrochloride 2 along with substantial recovery of benzamide 1. With an excess of HCl, equal amounts of 1 and 2 were isolated. Titration of salt 2 with aqueous NaOH to a phenophthalein end point formed the free amine of 2 which, within minutes, underwent O $\Rightarrow$ N migration. The structure 3 is the presumed intermediate.The authors also explored the trans-isomer of 1, which gave less than 1% of the trans-salt of 2 with 2.0 equivalents of HCl. The O $\Rightarrow$ N migration was also slower than in the cis-series.
You need to understand something in organic chemistry. Not every question is going to be part of some named reaction that you just have to apply. For example in this question, the basic concepts of organic chemistry is used nothing more.
First step, you look at the molecule and think, hmmm what can $\ce{H+}$ do? There are two possible things:
The nitrogen could take it forming $\ce{NH2+}$ but see that this does not lead anywhere that is you cannot proceed (also the nitrogen is right next to the carbonyl so the carbocation will also not be stable). Note that protonating the OH group also leads to nothing. (a rearrangement is possible but as none of the options have it we can eliminate it)
The lone pair on the oxygen (of carbonyl) takes up the $\ce{H+}$, forming a postive charge on it and this is a no-no as oxygen wants electron. So the double bond shifts onto the oxygen to form the OH group but this forms a positive carbocation and then … we see that an intramolecular attack can take place from the OH on the other side (O has lone pair of elecrons!) forming a five membered ring!
Then as mentioned in your diagram, an intramolecular proton exchange so as to quench oxygen's positive charge (a standard acid base reaction). Now from this part, see that we can regain the lost carbonyl group by opening up the formed ring in a rearrangement (the bond energy of carbonyl group is high). That's it!
