Is there a mathematical generalization for the number of isomers (structural and configurational) of a given compound?

The original question was:

How do I find the number of structural and configurational isomers of a given compound? is there any formula?

For a specific example, how would I find the number of structural and configurational isomers of a bromo compound $\ce{C5H9Br}$ formed by addition of $\ce{HBr}$ to 2-pentyne

1. Generally speaking, graph-theoretical enumeration aims at counting chemical compounds as graphs (2D structures). In other words, it is concerned with constitutional (or structural) isomers.

2. To accomplish enumeration of 3D structures (concerned with configurational isomers or stereoisomers), in contrast, it is necessary to take chirality/achirality of substituents into explicit consideration.

3. To grasp perspectives of the theoretical foundations of stereochemistry and of the state-of-the-art chemical enumeration, see also the following recent book: S. Fujita, Mathematical Stereochemistry (De Gruyter, Berlin, 2015)

As I said in my comment, the mathematical way to solve this kind of problem relies on Graph Theory. This article (Applications of Graph Theory in Chemistry, preprint) reviews the problem and further more.

If you google 'isomer' and 'graph theory' you will find anything you want about this exciting topic. But for generic question, there is no generic answer. It is about building all possible graphs or find out inherent properties inside them. You cannot answer to this question unless it is well-defined and properly bounded.

If you have such question, please rewords it and post it on Mathematical Exchange. This should give you fine grained answer. If you just wanted to understand how it works, welcome to the fabulous world of Graph Theory, you will have to learn a bit to answer it by yourself.

A useful tool for finding answers to specific questions along these lines is the Online Encyclopedia of Integer Sequences. It catalogues many sequences and provides a search engine, so if you compute by hand the number of isomers for five or six small cases then it may give you some suggestions for continuations and references.

For example, if you're counting non-stereoisomeric monosubstituted alkanes then you might get counts 1, 2, 3, 5, 8, 14; and one of the (at present) 66 search results is "Number of monosubstituted alkanes C(n-1)H(2n-1)-X with n-1 carbon atoms that are not stereoisomers" with a table with 3000 entries and references to papers by Blair and Henze; and Polyá.