In a large saucepan, melt butter over medium heat. Add graham cracker crumbs, brown sugar, milk and sugar. Bring to a boil and boil for 5 minutes, stirring constantly. (Begin timing 5 minutes from when you see the first bubbles.) Remove from heat and spread half of butter mixture evenly over crackers. Place another single layer of crackers evenly over butter mixture, cutting to fit if necessary. Spread remaining butter mixture evenly over crackers. Top evenly with remaining crackers.