All I can say is-

Rory has been becoming much more irrational and aggresively violent. Also, in the episode “Let’s Kill Hitler” after witnessing River’s regeneration and being exposed to RAW TIME ENERGY for the first time, he begins to complain of a “banging in my head”, which Amy dismisses as Hitler in the closet. Also think back to “The God Complex”. Rory did not have a room. He was the only character they made a point to say did not have one. And when the Doctor looked into his room all he said was “Of course it was you.” and we hear the wailing of the TARDIS distress call in 4 repetitions. The only other time it has made this sound was when The Master stole it in series 3. Also, Rory is the first human that wasn’t surprised that the TARDIS was bigger on the inside.