Martin Fowler has some excellent guidelines on designing software layers.

And as for a story of how code is designed in the real world, the story of how Damien Katz rewrote one of the layers of Lotus Notes is a pretty illuminating. Kudos also to the original Notes designers for layering stuff properly in the first place.

To this day I don't know what possessed Wai Ki to offer me the job. It doesn't make sense really. I can't help but wonder if I was thrown into the fire just because they thought I couldn't do it, that I'd try it, fail miserably and go back to the template team. Maybe, or maybe he thought I would really be able to do it, maybe he saw something in me. All I know is that I was terrified. I knew the odds were far against me, and if I failed it would be because I wasn't good enough. I couldn't blame anybody but myself. But still I said yes, I was going to give it my best shot.

Ray Ozzie, lead designer of Notes, chimes in with a bit of history.