Birefrigence is a result of structural anisotropy observed for crystals other than of the cubic Bravais class. Then, interaction of vectorial properties (e.g., light) with said material is anisotropic, too. It is a phenomenon seen both for inorganic, as well as organic materials (examples listed e.g., here). As seen e.g., by cooling films of molten polymers on a heat stage in polarization microscopy, local deviations from isotropy suffice to observe birefrigence, too (example). Thus, there is no «special functional group» which, if present, suddenly would switch on / switch off a sample to be birefrigent. It's observation and measurement by polarization microscopy allows to characterize the microstructure of e.g., minerals, and thus one of the elder tools in geology (still used today, e.g., Stoiber's «Crystal Identification with the Polarizing Microscope» by 1994 lists some in its freely available back matter). It may be used on organic samples, too (e.g., this tutorial).
The models to describe the crystal structure about ammonium sulfate, $\ce{(NH4)HSO4}$, depend on the temperature of the sample. This might increase or decrease birefrigence (I did not look out for a temperature-dependent recording of this about ammonium sulfate). The explored variation of sample temperature in the work below (at ambient pressure) however do not change the unit cell symmetry this much that birefrigence suddenly would be cancelled out for symmetry alone.
At room temperature ($\pu{293 K}$), the best description of this material that there two hydrogensulfate groups per unit cell which are described independently from each other:

(credit 2012JChemPhysChemA223, Fig. 1a / $\pu{293 K}$)
In the illustration, these are marked by O (ordered) and D (disordered). This is to say that the model describes the reality best assuming positions with $\ce{HSO^-4}$ with more than one orientation and local geometry. (Similar to sitting either forward, or backward on a chair.) This is, taking a different perspective on the sample (the representations omit hydrogen atoms):

(modified representation, based on 1972ActaCrystA445)
with ammonium ions, ordered and disordered $\ce{HSO^-4}$, the sites with the disordered would be either one of the following:

(credit 1972ActaCrystA445)
At lower temperatures, e.g. at $\pu{200 K}$ and at $\pu{125 K}$, it is energetically more favourable if both $\ce{NH^+4}$ and $\ce{HSO^-4}$ are in ordered position and orientation only, e.g.:

(credit 2012JChemPhysChemA223, Fig. 1b / $\pu{200 K}$)
This phase transitions and changes of unit cell symmetries (space group $P2_1/c$ (at $\pu{293 K}$) $\rightarrow$ $Pc$ (at $\pu{200 K}$) $\rightarrow$ $P1$ (at $\pu{125 K}$)) however will not cause a cancellation of birefrigence because of symmetry reasons.
You do not state (this much) why knowing the the spatial arrangement of the atoms, and hence a model of the crystal structure is important for you. However, because you mention to possess some command of chemistry and interest art, be aware among the crystallographers there are some equally with a sense for beauty. One of the low-level entries is IUCr's Crystal growing competition for school children (note the their brochure with examples in the later section) considerably newer than their teaching pamphlet.
On the more artistically appealing side, though, selectively dyeing crystals may be of interest for you. Work in this field was much popularized again by e.g. Bart Kahr:

(credit, includes protocol)

(credit 2001ChemRev893)
with protocols for chemicals easier to access (e.g., potassium dihydrogen phosphate [KDP] and amaranth dye, cf. here). Beside beauty, the selective incorporation of dyes provides insight into the growth mechanism of crystals. You may find additional inspiration in other works by him, e.g., in 1996AdvMat941, 2001ChemRev893 including a 20 page long table of examples («reported
during the past 147 years»), or 2002CrystEngComm514 about «painting crystals» by an easy-to-follow protocol.