Hard to rationalize refinancing for 1/4 percent, unless you are pulling equity (if you have any) and/or moving to a 15 year rate. Cost to make the move is prohibitive. If you load the loan on the backside with the costs (aka) no front end costs, then you will owe more money each time you do that, and that is money you can never get back and just makes your pay-off cost more when you finally do sell or go to pay it off. You need to wait until for 4.875 or better to bring the pay-back period down.