Controlling humidity in an outdoor 4x5 shed

Howdy please give your thoughts for the best way to control my humidity problem. I run my lights at night and when the lights are on the humidity goes from 40 to 50% up to 75% progressively. I have two AC infinity 4 inch one pulling in one exhausting could I just upgrade to a 6 inch exhaust? is that swing enough to cause bud rot and p.m.? would you set up the exhaust to turn on when humidity rises or have it on all the time? thank you