Only a sane man could contrive such a thought-out rational plan. Hamlet's madness allows him to express his feelings, develop new plans, and to gain information, all while keeping people from taking his actions too seriously, in order to eventually kill Claudius. Hamlet's act of masquerading madness allows him to speak his mind behind the disguise of insanity, so that no one would become suspicious of him. Hamlet uses his madness as an excuse and a part of his apology, towards Laertes for murdering Polonius. A mad man would not be able to fabricate such thoughts.
On the other hand, Hamlet acts perfectly sane when acting insane is unnecessary. Hamlet had no reason to act insane with Horatio because he was his close friend, who was clever enough to differentiate between false and true madness. It is more convincing that a sane man could play an insane one, than an insane man could play a sane man, and so reason would deem Hamlet