Fig tree in New Jersey.. help?

We planted a fig tree in South Jersey. We cover it from the end of November until May. From May and throughout the summer a great number of figs START growing but they don't ripen quickly. As a result we only get a handful of figs and the rest rot when it gets cold. Is there any way to speed up the growing time? Other people get figs (in our area) why can't we? We let the bush get HUGE and out of control? Would pruning it during the season help? We usually cut it to nothing before the frost. Thanks for any info.