.22 LR is going to be by far the cheapest way to go, but it was never much fun to shoot from a rifle IMO. Not much more bang than an air gun. Fun for target shooting with a Ruger MkII pistol though.
.22 magnums are a little more exciting while still being somewhat affordable. Good for target shooting out to 100yds and critter hunting. Any centerfire ammo will eat up your wallet unless you buy bulk surplus/reloads.
But yeah, guns themselves are in the same price ranges as high end paintball guns. $300-1200 for most, save some newer piston AR's and target rifles and such.