Hi,
it's strange you have so many database queries when your site doesn't have that many visits. Maybe you have many components installed or one component creating many URLs, try to find it in your created URLs and turn JoomSEF off for it. Although there's going to be some less database queries in next release of JoomSEF, I don't know if it will be enough for you.