I learned about molecular modeling while working on my NanoCAD applet. Now we use it in nanoENGINEER-1.
Atoms are comprised of a small dense positively-charged nucleus surrounded by a probabilistic cloud of negatively-charged electrons. The shape and behavior of the electron cloud is governed by quantum mechanics. The nucleus is heavy enough that you can think of it in classical terms. The electrons and the nucleus electrically attract each other.
If you want to get really accurate information about molecular mechanics, you get a cluster of computers and run software that solves the quantum mechanical wave equation. Generally this is a hugely compute-intensive undertaking. If you want to see behavior over a series of moments in time, like an animation, you probably can't afford to do real quantum mechanics, so you've got to cheat.
The way to cheat is to regard the nuclei as point masses connected by non-linear springs. These non-linear springs take into account the electrostatic forces with the electrons and other nearby nuclei. This formulation gives energies in terms of geometric properties such as bond lengths, bond angles, and dihedral angles. Additionally there are grosser electrostatics to think about (charged ions, and bonds have electrical dipole moments if the atoms have different electronegativities) and one more force between unbonded atoms called the van der Waals force. The NIH has a great web page about this stuff.
Each of these things gives a component of potential energy in terms of the geometry of the molecule. Taking the negative gradient of that energy in the 3N space of atom coordinates gives you the forces acting on the atoms. Plug those into equations of motion, and integrate, and you're done.
There are still a few things to think about. One is numerical instabilities: any non-zero time step will give approximate answers, and you end up with violations of conservation of energy. Another thing is that you can't model the making and breaking of chemical bonds this way, you can only model stable structures that aren't reacting.
There are some things that can help with numerical stability. One is to notice that the quick motion of the hydrogens, which will consume a lot of your computrons, isn't very interesting. So you can play tricks like making the hydrogens heavier, or locking the hydrogens' positions relative to the atom they bond to (just add their masses to its mass), allowing a longer time step. Another is to use an integration method like Verlet that does better with energy conservation.
I'll probably write more about this topic in the future. It's deep and interesting, and if the Nanorex experience adds some modest qualifications in molecular modeling to my resume, it will have been time very well spent.