delays on holidays and weekends in both directions is not unheard of. A lot of people have cabins in Prescott but live in the Valley. The go around in case of a traffic accident adds about 3 hours to the travel. The traffic congestion/delays are hardest cuz the traffic is at a stand still in the summer heat. For me, traveling with horses, I avoid traveling I-17 during the summer for any weekend/holiday outings. On a long holiday weekend it is not uncommon for the traffic back up on I-17 to start slowing in Phoenix. Even the S/B traffic from Sunset Point on I-17 is heavily congested. The exit ramps for Prescott are also congested but they clear quicker than I-17. There's Prescott and then there's Prescott Valley. Which area are you considering?
If you can work your travel hours around less congested times or don't intend to travel out of Prescott very often it's a nice place to live. It's slightly cooler than the Valley.
My biggest caveat is to educate yourself about water rights before buying anything.
AZ is a red state and even though there are those saying that it will turn blue, the red mentality is still very present, but IMO the west side of AZ is better than the east side. Not to offend anyone, but lots of LDS in NE AZ and their ideals don't gel with most D or progressive ideas. Does Paul Gosart ring a bell to you?
I've lived in AZ over 30 years. You can DM me for more info if you want. I have a friend who had a cabin in Prescott for years and knows the town pretty well. the Prescott Rodeo is a big tourist draw.