So, I am to pull the maximum actors' potential, aren't I? But is it OK, if I ask them to play in a different manner in one particular scene despite their insistence on another way of perfomance (when it seems to me not corresponding to the atmosphere of the scene)?
I think you're exhibiting a concern that many if not most young filmmakers have: how much should I direct, and how much should I let actors act?
Because it can easily be intimidating, especially when you're working with actors who you think are amazing or who might actually be famous in one way or another. And you think to yourself, "am I in a position to tell this person what to do," "am I allowed to tell them how to do something if they aren't getting it right," "should I tell them this that and the other thing to show them where I am going with this scene or this character," or even "am I worthy to direct this individual?"
But in answer to all of these, as ~Tony Ivan had said, Yes, you are indeed in a position to tell this person what to do. But initially you should be cordial and polite about it because a Director should try to be understanding, cooperative, and even accommodating in some circumstances, in order that the actors, and even the rest of the team, don't take too much control away.
Collaboration is one of the back-bones of the business and is
super important. I could not have shot my last film without a multitude of talented people. But if someone isn't gelling with the production and working towards getting it where it's supposed to be be as far as genre, mood, style, or intention, then they might not be the best person to be working with.
So you should be honest, give the actors room to work, and learn to work
with them. Don't think of them as puppets or robots or something like that, because that can be the other extreme.
In the end, if an actor isn't giving you what you need then just tell them, and see how you might be able to work towards a different and more suitable performance. However, if they are becoming difficult to work with, and they aren't giving you and the rest of your crew what you know that you need, then it's your job to make the call to possibly recast their part. Just be sure that your Producer agrees with that decision.
And I agree with ~Trueindie as well, that once in a great while, an actor's gut feeling may be more fitting to the role and the scene than your original intention. Most notably this happened on the set of
The Empire Strikes Back, when Harrison Ford said "I know" instead of "I love you too," just before he was frozen in carbonite. But I think this situation will come up more likely in the sequel to a previous work, where your actor's have really come to understand their characters on a deeper and more personal level. So their instincts regarding that character might be more intuitive than your own understanding, even though you may be the director and even the writer.