Careful measurements were the foundation of chemical theory
Let's take the questions out of order to simplify the answers.
It doesn't matter what yours you use to measure volume as long as you stick to the same units in all your calculations. Cubic widgets is fine as long as you do all your calculations using them.
The key to understanding how chemists came to know about atomic mass is careful measurements of the widths and volumes involved in reactions. Chemists observed that certain reactions always involved the same mass ratios. More interestingly, when gases are reacted they also involve the same volume ratios. When hydrogen and oxygen react to for water you start with two volumes (2 cubic widgets if you want) of hydrogen and one of oxygen react together to give water. How do we know it is just that ratio? Do the experiment with 3 volumes of hydrogen and one of oxygen and observe that there is still one volume of hydrogen left after the reaction is done.
Those experiments showed that the mass ratios were constant but different to the volume ratios if the reactants were gases. Oh, and we know the weight of the gases because we can measure the weight of a given volume of gas using a vessel of known volume containing a gas at a specific pressure using a carefully constructed vessel. A litre of oxygen weighs about 1.5g which is easy to measure using a good balance and good experimental conditions even in the 19th century.
Chemists hypothesised (and later proved) that the volume of gases were proportional to the number of molecules they contained so the ratio of gas volumes told them something about the number of units of substance (or molecules) involved in reactions. Comparing many reactions led to an understanding of more complicated reactions than hydrogen burning in oxygen. Comparing the known results from gas volume ratios to the known weights of substances involved in liquid and solid reactions extended this idea (with an early glitch caused because chemists didn't realise that many gases consist of diatomic molecules).
The actual number of molecules or atoms in a give volume took a lot longer to determine than the basic principles involved in chemistry (that reactions always involved specific ratios of inputs and outputs). Initial methods involved atomic theory applied to gasses or measuring the charge on a single electron. Later methods involved measuring the mass of a very pure crystal and comparing it with density derived from x-ray measurements to give more accurate values. (see this).
The basic experiments about atomic ratios can be repeated with good experimental technique and apparatus. But you need a very good way to measure weights of gases and volume. Repeatable, but not easy. You won't be easily able to measure the number of atoms in anything with household apparatus, though. That requires fairly sophisticated equipment.
