Let's dissect this:
As I understand it, a DFT calculation uses periodic boundary conditions, whenever you want to simulate a bulk material. This is convenient because all information can then be extracted from the fundamental unit cell.
One important point is that you employ periodic boundary conditions (PBC) only when you are treating a periodic system, i.e. something with translational symmetry. And in those cases you use plane waves as a "basis set".
In the ordinary molecular case you use atom centred basis functions, which fulfil the basic requirements of a wave function (in short): orthonormality, continuous, continuous differentiable, vanishing at infinity.
Therefore a quantum chemical calculation electronic energy is always in vacuum at "0 K".
Currently I am faced with the problem of calculating the band structure of a carbon nanotube, which has a cylindrical geometry, i.e. only bulk in only one direction, say z, and circular in the x and y directions. As my unit cell I am using a ring of C-atoms surrounded by vacuum and I then employ periodic in the z-direction. Obviously, for large diameters, this approach involves a rather large unit cell, which makes the calculation slow.
This approach actually only makes sense if you are treating a very, very long nanotube. I'd believe a molecular approach using appropriate symmetry, Dnh would be far superior.
My question is: Is there an alternative way to do this, which exploits the cylindrical symmetry of the system? Specifically what I have in mind is to also apply periodic boundary conditions in the other directions, but where the Schrödinger equation is expressed in cylindrical coordinates with the middle of the nanotube as the origin.
A nanotube is discrete in two directions, i.e. the wave function has to vanish in these directions, egro apart from applying vacuum as PBC, there is no other choice in this framework.
An alternative approach is to treat your nanotube as a discrete molecule and cap it off at the end with hydrogen, then increase the length of the tube until you have sufficient convergence, obviously using Gaussian or Slater type (atom centred) orbitals.
Another, less important, question is also: Do you ever NOT use periodic boundary conditions in DFT? For example, when you do a calculation for an atom, it always seems like you have to add vacuum around it, which is essentially, because you are applying pbc, and do not want it to interact with other unit cells. Is there another way to solve this problem, where you assume the density to vanish far away?
For discrete molecules: always. For periodic systems: only sometimes not (cluster approach).
Prior to the golden 90s, almost every calculation was a molecular calculation. In the early stages of quantum theory nobody ever dared to dream that we would eventually able to treat a huge stone quantum chemically (a priori) with a computer.
A nanotube is typically right in the middle, between a bulk material and a molecule, and you should either rely on the experience of others, maybe you have an advisor, through the literature, or built it yourself.
There are quite a few groups (I suppose) who work with nanotubes, and some of them have surely tried computations. Do a thorough literature review and apply common practices.
I fear that you are getting your knowledge about DFT calculations from working with a calculation suite designed for bulk materials. You should probably brush up on some of the fundamental theory. I recommend Frank Jensen's book for a rough overview, and Attila Szabo & Neil Ostlund's book for some better maths (See resources for learning chemistry for more details).
A fairly comprehensive "review" of DFT can be found here: DFT Functional Selection Criteria and in the links within.
