Refinancing does not erase the interest you already paid, and it does not create a separate penalty just because you start a new loan. What it does is replace your existing mortgage with a new amortization schedule.
That means you should compare the new payment, closing costs, new loan balance, expected time in the home, and total interest over the period you actually expect to keep the loan. A lower rate is useful only if the whole transaction improves your position.