Spoiler alert: the answer is (S).
The complete rules for determining relative priorities of substituents is described fully in P-92 of Nomenclature of Organic Chemistry: IUPAC Recommendations and Preferred Names 2013 (Blue Book).
As stated in P-220.127.116.11, double- and triple-bonded groups are expanded into versions where each multiply-bonded atom is duplicated as many times as is necessary. However,
only the doubly bonded atoms themselves are duplicated, and not the atoms or groups attached to them; the duplicated atoms may thus be considered as carrying three phantom atoms [...] of atomic number zero.
Phantom atoms have the lowest priority, below even hydrogen atoms, since their atomic number is zero. Using this, we can construct the full digraph. Here, (C) represents a duplicated carbon atom, which is a carbon atom carrying three phantom atoms (denoted as zeroes).
The atoms are colour-coded according to the sphere that they are in: essentially, this refers to the number of bonds from the stereogenic unit (which, in this case, is a chiral centre). The blue carbon is directly attached to the chiral centre, and so is in "sphere I". The green carbons, two bonds away from the chiral centre, are in "sphere II"; and the red atoms are in "sphere III".
P-92.1.5 describes the rules for precedence. The wording is quite difficult to understand, and I cannot yet convince myself to directly quote from the rules. However, having studied many of the examples given in the subsequent section, the analysis broadly proceeds as follows (I think):
- Sphere I for both groups simply consists of 'C', and no comparison can be made.
- Sphere II is 'C(C)(C)' for the alkyne, and 'CC(C)' for the alkene. No comparison can be made at this point (there is no rule stipulating that normal carbon atoms have greater priority over duplicated carbon atoms).
- Moving on to Sphere III, for the alkyne, we have the three groups 'CCH > 000 > 000'; and for the alkene, we have 'CHH > HHH > 000'.
- Comparing the first of each group of three, since CCH has a greater ranking than CHH, we can assign the greater priority to the alkyne group.
See also: Assign stereodiscriptor to 4th carbon of 3-ethyl-2,4-dimethly-5-(propan-2-yl)heptane-1,7-diol for a similar situation.