The phenomenology of agency and perception is probably underpinned by a common cognitive system based on generative models and predictive coding. I defend the hypothesis that this cognitive system explains core aspects of the sense of having a self in agency and perception. In particular, this cognitive model explains the phenomenological notion of a minimal self as well as a notion of the narrative self. The proposal is related to some influential studies of overall brain function, and to psychopathology. These elusive notions of the self are shown to be the natural upshots of general cognitive mechanisms whose fundamental purpose is to enable agents to represent the world and act in it.