Enemy Design
Usually AAA games don’t have particularly interesting enemies, especially nowadays because of increasing cost of production and mindless ride towards oversimplification. The situation is even worse if we talk about enemy progression, because most of the time only hitpoints/dps are increased and the player is left to battle the same type again and again, only on different scenarios.
If you want a definition, enemies are often just a collection of stats, so you may decide to line-up a series of encounters that basically represent an escalation of numbers related to their attributes; doing that way, chances are that you end up with a pretty dull experience.
My way to make interesting enemies is by mixing them with other game mechanics. Giving them a role or function allows the player to decide how to deal with each one, and offering meaningful choices is always good in any game. If the player has to adjust the way he plays because of an enemy, then we make our game a dynamic experience.
If your core set of mechanics supports several kinds of challenges (puzzles, information-based, skill-based, etc.), then you can come up with situations where you see the mechanics crossing. That’s true in general and in reaching the final point of making interesting enemies.
A start is considering the basic types: snipers, heavies, spawners, skirmishers, etc., but then I elaborate on that integrating multiple kinds of mechanics, or better, I try to create specific boundaries, to test specific combinations of the mechanics involved.
Placement is another important factor, because making them interact with the layout transforms the navigation of the level from a type of challenge to another (from information to skill-based or puzzle).
An example of that is the rolling enemy in side-scrolling games: you want to jump on a platform but you have to take in consideration where the enemy is moving.
In a 3d game you could have a fast-moving but slow-turning enemy that requires the player to zigzag while running, if he wants to elude him; the challenge is meant to test the ability of the player to dodge and not only proceed straight.
If I have a swinging mechanics (think Bionic Commando, Metroid, etc.), I could introduce enemies that deny me to attach to certain spots or are able to cut my rope. If I can swim and breath underwater, I could have enemies that bite me and make me lose part of my reserve, until I drown.
This having pro and con is part of what makes interesting an enemy to battle with, just like “conflict” makes for an interesting character, from a narrative standpoint. (read here)
We want depth in our encounters, but not excessive complexity. It’s important to not mix too different kind of boundaries together and too quickly. Think about the progression in Portal or in Super Mario Bros.
This leads me to another important point: give the player all the information he needs to decide how to react. In one word: feedback.
As far as I enjoy complete immersion in games, I recognize how abstract interfaces enhance the experience and the comprehension of the rules. Think about the proximity ring in MGS or a visible cone of view to detect where an enemy is looking.
If the player never knows about the different functions generated by the enemy, the lack of information makes for a random encounter without base for his actions.
Without resorting to trial&error, we could showcase a certain behavior with the player in a safe place or with enough space to react: in the case of the fast-moving slow-turning enemy, the player could be surprised and chased in a tight corridor with the unique option to escape back; then an opening offers him the chance to sidestep the chaser that continue forward its ride.
(I’ve seen this same scene in the recent Hard Reset, but it was too late in the game to be of any insight for the player)
Usually, confusion arises when an enemy is clearly designed with its appearance in mind first and by its function later. I usually prefer to start from what the enemy actually does, because you can always adapt it later.
An example is the Fallen Shaman in Diablo 2 who resurrect his minions killed by the player: he clearly casts a spell and then a dead minion returns to life.
This approach easily leads to abstract functions, usually incompatible with a real-world setting, so the oversimplification hinted at the start is partly comprehensible.
Fantasy worlds and alien dimensions are the best choices to be free to brainstorm crazy original behaviors, and for me are the best in general, since I think games are more than often enjoyed for the possibility to experience things impossible otherwise (thanks Will Wright for disproving me).