The designers at Valve are the supreme rulers of the realm of level design. Virtually every single game that Valve makes boasts superior artistry and craftiness. They truly are masters of their art.
My question is: "How on Earth do they do it?"
Turns out Valve is awesome enough to tell us.
Just check out the Valve Level Design Wiki for Source Mods: http://developer.valvesoftware.com/wiki/Half-Life_2_Level_Creation. It's a priceless resource. Did I mention Valve is awesome!
http://developer.valvesoftware.com/wiki/Single-Player_Mapping_Tips
Single player mapping tips! That's gold!
http://developer.valvesoftware.com/wiki/Bounce_(level_design)
http://developer.valvesoftware.com/wiki/Adding_More_Detail_to_a_map
http://developer.valvesoftware.com/wiki/Loops_(level_design)
http://developer.valvesoftware.com/wiki/Push_Gameplay
All stuff that is definitely worth taking into consideration when creating a map.
