It probably depends on the nature of your driving.

I do a lot of long runs, live out of big towns, etc. so I've always appreciated the extra solidity and refinement of a 5 series saloon over a 3. If you haven't gone overboard on f-off sized wheels and tyres then they cosset and sooth in a way that I find very appealing. Even a good 3 tends to feel a little more flimsy over time and distance than a decent 3. You do sacrifice a little bit of ultimate chuckability and directness in a 5 series in comparison though. The 5's I've had have never been shabby in this way but because of their extra mass they lose out a little. The 3's tend to be a bit more nimble. The 6 cylinder engine makes a big difference and closes the gap between the 3 and 5 though.

On the other hand, if you are mostly around town, limited parking space, etc. then a 3 makes more sense.

For me, it would be 5 series all the way.