What software does the PDB use to generate these "cel-shaded" protein models with outlines around foreground atoms?
(Leptin: May 2012 Molecule of the Month by David Goodsell)
These illustrations are by David Goodsell, and carry his famous look. Whilst I was not able to determine what specific software he uses, this page suggests that he probably creates his illustrations as a composite of renders and that the outlines and depth cueing are achieved by running a 'find edges' filter over the z-buffer of the scene, which can be output from many renderers.
As F'x mentioned, QuteMol can do this style as well, however QuteMol has not been actively developed since about 2007 and is not terribly feature-rich, which is a shame as it's one of very few molecule viewers with competent ambient occlusion.
In the first instance, I tried using the GLSL shader Goodsell style in VMD on the PDB file 1RON, a neuropeptide. I was not satisfied with the results, but this would probably look decent on a protein SES as illustrated on the aforementioned VMD site.
Note that I have used 2 VMD representations to separate out the style of the backbone and the sidechains, minus the hydrogens. I'm not going for a 1-to-1 clone of the referred style, just something thematically similar.
I have shrunk this VMD output down because as F'x pointed out in chat, GLSL shaded spheres are not antialiased. This said, if you are very tricky, you might be able to force antialiasing using a shader method like FXAA, which can be enabled in arbitrary programs in the very latest nVidia beta drivers (I know...). Alternatively, you can do what I did and manually oversample :D.
I output the VdW spheres as two separate VRML files, one for the backbone, one for the sidechains. These can be imported into Blender. I'm going to assume proficiency with Blender, however if you are not familiar, Blender is fairly easy to learn, especially versions 2.5 and later.
I've combined the VdW sphere meshes of each component into a backbone and sidechain object, assigned materials for each (just changing the colours of the default diffuse shader) and have applied one level of Catmull-Clarke subdivision to the meshes, to make the spheres more spherical than the default.
I've also removed the light source and turned on environmental lighting, which lights the scene isotropically and will generate some nice ambient occlusion while we're at it.
I now render this at high resolution, obtaining a standard render and the Z-buffer for the scene. I've scaled them down here. Note that if you want the spheres to be a flat colour, you should set the material to 'shadeless'. I like the subtly shaded look better.
Oh yeah, that looks nice. And here's the Z-buffer. Regrettably, the Z-buffer in blender has no antialiasing which is why I've rendered things huge and scaled down in Photoshop.
I now run the Glowing Edges filter on the Z-buffer and threshold to remove faint spurious edges inside the spheres.
Invert and level, set blendmode to multiply and you're done.
Not perfect, but not bad.
Here's the shadeless version, which is closer to Goodsell's ideal.
Poking around with the Toon shaders in Blender, I have found a super easy way to make Goodsell-esque figures!
If you configure your material like so:
You have a shader that will rapidly shade to black at angles far from normal with a lightsource. To make this work, you should disable environment lighting and position a bright point light behind the camera.
You will note that I have adjusted the Toon size to 0.98, which will create a shadow just shy of perpendicular to the light source normal. You will also note that I have set the specularity to zero, which gives an approximately flat finish to the sphere.
Render it up and you will get something like this: