You can follow the key aspects of group theory and symmetry for chemistry without ever taking formal abstract algebra courses.
The key aspects are presented in survey and specialized chemistry texts. Probably all the exercises on different molecules (and IR modes and the like) will actually give you MORE of a feeling for the complexity of physical symmetry than a math first attitude would.
This is not to say it hurts, but you totally don't need math based group theory.
Consider, for instance, Arfken has a remark (in horror) in his advanced math book for physics at the 200+ space groups for crystals. But a crystallographer or solid state chemist finds them interesting. [Note that this is not even an aspect of chemistry being very detailed...millions of compounds. It's straight up from the math itself...could be lattices of twigs for all we care. But the mathematician just hates this kind of complexity. A chemist is used to it since he has to organize families of organic compounds and the like.]
Personally I think if you want to dive deeper, would start with specialized books on group theory for chemistry (will have a little more than the touch in your survey inorganic text). Here there are sort of two schools:
A. Molecular, with emphasis on point groups and vibrational modes (IR). Think basic coordination compounds of metals.
B. Solid state, with emphasis on space groups and crystallography. Think metal oxides or similar.
Of course you can and will have crystalized groups of molecules as well, so both are relevant.
Only after digging deep into some of the chemical based stuff would I maybe consider the need to go back and get into an abstract algebra based development of group theory (probably starting at set theory and involving things like rings and fields that don't even apply to your need).
And by the way, if you get seriously into crystallography, you can spend a lot of time on that. And a group theory mathematician won't have a feeling for how to spot flaws in crystal structures (either those that don't make sense chemically OR those that are little logical flaws of the math, e.g. equivalent structures).
If anything learning more vector calc and tensors (and from an applied, simple perspective...not total math theory killer) would help you more. Even that is basically just needed if you become a crystallographer. An experimental inorganic chemist working on molecules really doesn't need it--all his xtal structure determinations are outsourced. Solid state guys tend to do more of this themselves but even here, a lot of it is playing with programs on the computer and the theory of the x-ray geometry is really not needed. More a careful attitude to look for reasonable chemistry and where solved structures have more uncertainty ("thermal parameters").