I learned about molecular modeling while working on my
NanoCAD applet. Now we use it in
nanoENGINEER-1.
Atoms are comprised of a small dense positively-charged
nucleus surrounded by a probabilistic cloud of negatively-charged
electrons. The shape and behavior of the
electron cloud is governed by quantum mechanics. The nucleus is heavy enough that you can think of it in
classical terms. The electrons and the nucleus electrically attract each other.
If you want to get really accurate information about molecular mechanics, you get a
cluster of computers and run
software that solves the
quantum mechanical wave equation. Generally this is a hugely compute-intensive undertaking. If you want to see behavior over a series of moments in time, like an
animation, you probably can't afford to do real quantum mechanics, so you've got to cheat.
The way to cheat is to regard the nuclei as point masses connected by
non-linear springs. These non-linear springs take into account the electrostatic forces with the electrons and other nearby nuclei. This formulation gives energies in terms of
geometric properties such as
bond lengths, bond angles, and dihedral angles. Additionally there are grosser
electrostatics to think about (charged ions, and bonds have electrical
dipole moments if the atoms have different
electronegativities) and one more force between unbonded atoms called the
van der Waals force. The
NIH has a
great web page about this stuff.

Each of these things gives a component of
potential energy in terms of the geometry of the molecule. Taking the negative
gradient of that energy in the 3N
space of atom coordinates gives you the forces acting on the atoms. Plug those into
equations of motion, and
integrate, and you're done.
There are still a few things to think about. One is
numerical instabilities: any non-zero time step will give
approximate answers, and you end up with violations of
conservation of energy. Another thing is that you can't model the making and breaking of
chemical bonds this way, you can only model stable structures that aren't
reacting.
There are some things that can help with numerical stability. One is to notice that the quick motion of the hydrogens, which will consume a lot of your computrons, isn't very interesting. So you can play tricks like making the hydrogens heavier, or locking the hydrogens' positions relative to the atom they bond to (just add their masses to its mass), allowing a longer time step. Another is to use an integration method like
Verlet that does better with energy conservation.
I'll probably write more about this topic in the future. It's deep and interesting, and if the Nanorex experience adds some modest qualifications in molecular modeling to my resume, it will have been time very well spent.