How can J remember K?

Boris killed K in the past to change the present in which, K doesn’t exist and Aliens started to invade the Earth. Technically speaking, if what you did in the past will also have an effect of the future. However, the reason J remembers K is because coming Jeffrey, “you (J) were there”. Who was he referring to? The young J or the present Men in Black Agent J who travelled back in the past?