I learned about molecular modeling while working on my 
NanoCAD applet. Now we use it in 
nanoENGINEER-1.
Atoms are comprised of a small dense positively-charged 
nucleus surrounded by a probabilistic cloud of negatively-charged 
electrons. The shape and behavior of the 
electron cloud is governed by quantum mechanics. The nucleus is heavy enough that you can think of it in 
classical terms. The electrons and the nucleus electrically attract each other.
If you want to get really accurate information about molecular mechanics, you get a 
cluster of computers and run 
software that solves the 
quantum mechanical wave equation. Generally this is a hugely compute-intensive undertaking. If you want to see behavior over a series of moments in time, like an 
animation, you probably can't afford to do real quantum mechanics, so you've got to cheat.
The way to cheat is to regard the nuclei as point masses connected by 
non-linear springs. These non-linear springs take into account the electrostatic forces with the electrons and other nearby nuclei. This formulation gives energies in terms of 
geometric properties such as 
bond lengths, bond angles, and dihedral angles. Additionally there are grosser 
electrostatics to think about (charged ions, and bonds have electrical 
dipole moments if the atoms have different 
electronegativities) and one more force between unbonded atoms called the 
van der Waals force. The 
NIH has a 
great web page about this stuff.

Each of these things gives a component of 
potential energy in terms of the geometry of the molecule. Taking the negative 
gradient of that energy in the 3N 
space of atom coordinates gives you the forces acting on the atoms. Plug those into 
equations of motion, and 
integrate, and you're done.
There are still a few things to think about. One is 
numerical instabilities: any non-zero time step will give 
approximate answers, and you end up with violations of 
conservation of energy. Another thing is that you can't model the making and breaking of 
chemical bonds this way, you can only model stable structures that aren't 
reacting.
There are some things that can help with numerical stability. One is to notice that the quick motion of the hydrogens, which will consume a lot of your computrons, isn't very interesting. So you can play tricks like making the hydrogens heavier, or locking the hydrogens' positions relative to the atom they bond to (just add their masses to its mass), allowing a longer time step. Another is to use an integration method like 
Verlet that does better with energy conservation.
I'll probably write more about this topic in the future. It's deep and interesting, and if the Nanorex experience adds some modest qualifications in molecular modeling to my resume, it will have been time very well spent.