Could be relay, assuming your new fuel pump is working fine. Seen faulty relays doing strange things like that. 20 euros from BMW. I assume yours is a post 2001 car, in that case, relay is under the glovebox. Green one close to the body module (Big white module with 3 plugs)
You could actually swap your relay with rear windscreen defrost one which is located in the trunk (Top right green one). Just temporally to see if that is doing any good.