I'm not sure how easy this is to implement algorithmically, but for what it's worth, the official IUPAC recommendations are as follows.
Section P-93.3 in Nomenclature of Organic Chemistry – IUPAC Recommendations and Preferred Names 2013 (Blue Book) describes the guidelines for assigning configuration at non-tetrahedral atoms. The portions of this answer will be quoted from there.
These guidelines, and further details on certain aspects, are also discussed in Nomenclature of Inorganic Chemistry – IUPAC Recommendations 2005 (Red Book) (link), section IR-9.3.
Stereodescriptors for nontetrahedral configurations are composed of three parts.
(1) A symbol indicating overall geometry called the 'polyhedral symbol';
(2) A symbol called the 'configuration index' that describes the configuration of ligands around the central atom;
(3) A symbol indicating the 'absolute configuration' associated with the central atom called the 'chirality symbol'.
For the first example you provided:
which is a trigonal bipyramid, the appropriate 'polyhedral symbol' is TBPY-5 (P-93.3.2, Blue Book). The numbers in red are the priorities according to the Cahn–Ingold–Prelog system, which will soon be required.
The 'configuration index' is
a series of numbers to identify where each atom or group is located. It is based on the CIP priority order of the atoms attached. The atom or atoms with highest priority are given priority number one '1'; the next priority number two '2', and so on. [...] (P-220.127.116.11)
For a trigonal bipyramid, this "consists of the priority numbers of the two apical atoms (lower number first if different), representing the reference axis of the system" (P-18.104.22.168.1). In this case, the two apical atoms have priority numbers 1 and 2, so the configuration index is simply 12.
The 'chirality symbols' available for a trigonal bipyramidal system are 'A' and 'C'.
The atoms in the plane perpendicular to the reference axis are viewed from the side with the atom or group of highest priority on the reference axis. If the direction from the atom in this plane with highest priority to the atom with the next higher priority is clockwise the chirality symbol is 'C', if anticlockwise 'A' (P-22.214.171.124.1).
Looking along the reference axis (which is the axis containing the axial ligands), from the direction of the higher priority atom (oxygen), the equatorial ligands decrease in priority going in a clockwise order. So, the chirality symbol is C. (Essentially, this is analogous to assigning an R-configuration for an ordinary chiral carbon.)
The three parts of the stereodescriptor are enclosed in parentheses and separated from each other by hyphens and cited at the front of the name separated from it by a hyphen. (P-93.3.1)
The full stereodescriptor is therefore (TBPY-5-12-C).