My boys are older now (15 and 16) but we have always brought them with us on trips. We mostly backpack now, but started with overnights in the backyard as infants, road trips with a teardrop trailer for nap time as toddlers and graduated to short overnight winter backpacking trips in Joshua Tee and Anza Borrego when they were 4 and 5.  I would always let them help pack, choose the trail and ultimately pick the campsite. After 16 years some of my favorite memories were from these early trips.