Of course this is an irrational assumption, and we all know that. But Loki is (1) severely depressed, which leads him to believe that no one really loves him and to seize on any evidence that could be construed to support that belief; and (2) under severe pressure from Thanos to retrieve the Tesseract, which means he has to fight Thor, which means that he’s strongly motivated to find any reason he can to want to fight Thor.