This is not something we’ve seen him do before. Not that he would have had any occasion to in previous movies, but this is kind of in a different category from other things he does with magic: mostly illusions, shapeshifting, and moving things around.
This is some wild-ass, blatantly self-interested speculation, but… what if the power to draw out other people’s memories is a relatively new ability that he gained from his interaction with the Mind Stone? Scarlet Witch got her powers from experimentation with the Mind Stone, and one of the things we’ve seen her do is get into people’s heads and make them experience things based on their worst fears and most painful memories.
This at least suggests that forcing people to re-experience memories is one of the things that the Mind Stone can do, in addition to co-opting their will to make them serve the aims of the wielder. I think everyone I know has dismissed the idea that Thanos was controlling Loki the way Loki controlled Barton and Selvig; that’s probably not something you can do to a magically powerful being like Loki. BUT Scarlet Witch was able to do the fear-exploiting thing with Thor, and Loki simultaneously experienced and forced the Valkyrie to re-experience her worst memory – that is something you can do to more powerful beings.
So… this provides some circumstantial evidence in favor of the theory I explore in my fic The Abyss Gazes Also about what happened between Thanos and Loki: that Thanos exploited Loki’s own fears, resentments, and insecurities and forced him to re-experience his most traumatic memories involving his family (especially Thor and Odin) to manipulate him into invading Earth on Thanos’s behalf and fighting against his adoptive family.
Tagging people who read my fic and might care: @angrymadsygin, @darklittlestories, @fuckyeahrichardiii, @iamhisgloriouspurpose, @ikoliholic, @illwynd, @lunariagold, @nursejoh53, @raven-brings-light, @writernotwaiting