I believe the art of writing a convincing yet engrossing character is to write them with depth. This may sound obvious but it is something that is so easy to gloss over. By depth, I mean layers. Think about real people. We are not either nice or naughty. Human interaction depends on their own personal beliefs and situations. A readers, we naturally empathize with a character who experiment with their own boundaries.
For example: Say your character is a vegetarian. Doesn’t eat meat for animal cruelty reasons. Fine. Standard. Boring. Now say this vegetarian actually changed their ways to gain the approval of a colleague/a lover/ a friend and in fact eats meat on the sly, when they’re feeling down. Having multiple dimensions is more interesting than being devoted to one characteristic.
We also like characters who intrigue us. And perhaps disturb us. As much as we don’t like to admit it, there is something strangely en capturing about a protagonist who delves into taboo. Child abuse. Pedophilia. Torture. Holocaust. Fetishes are popular. It puts the reader into a new frame of mind which questions their own morals and judgement.
Take Lolita by Vladimir Nabokov – a story about a man in love with a child. It’s wrong. We know this, yet being forced to see this perception it makes us intrigued but disturbed all in one. Exploring these characters are a way to explore the world and the reader themselves. Of course, we don’t indulge in sexual fantasies about prepubescents, but is natural to be intrigued and being forced to empathize with this protagonists makes the reader uncomfortable with their own feelings. And this discomfort is the sign of a good writer.
A good book should make the reader question themselves.