Grease a muffin tin, and use muffin papers if desired. Set muffin tin aside.
With a mixer cream the sugar and the butter together. When light and fluffy add lemon extract, vanilla extract, and eggs. Beat together until well blended.
Add lemon zest, sour cream, and poppy seeds. Mix until all ingredients are just blended in.
In a separate bowl combine flour, baking powder, and salt. Stir until the mixture is uniform.
To combine the dry ingredients with the creamed butter, add a third of the flour into the butter, and then add about a third of the milk. Stir gently, then add in the next third of the flour and milk, and stir gently. Scrape the sides of the bowl to make sure all ingredients are well incorporated. Add remaining flour, and then the remaining milk. Stir until just blended.
Fill each muffin tin 2/3 full. Bake muffins for 24 minutes. You may want to start to check the muffins at about 20 minutes by inserting a toothpick to see if they are done. When the toothpick comes out clean, the muffins are done. Remove from the oven.
In a small bowl combine powdered sugar, milk, lemon extract, and lemon zest. Stir until well blended. When the muffins have cooled to room temperature, you can drizzle the glaze over the muffins.