A couple of the previous responses have made a good point: Evidence of maintenance is equally as important as mileage. If the machine appearance reflects pride in ownership, the odds are in your favor that the owner has sweated the small details. While engine condition is important, keep in mind that there are many things that can or will fail before the engine - clutches, shafts, bearings, chains, etc. These things are not cheap to fix either.
Personally I feel that sitting does a sled no good. Living in a metro area, mine do not get run as often as I would like so I pay special attention to maintenance. Metro area sleds often have lower mileage than sleds in more rural areas but that doesn't necessarily translate into greater reliability. My brother, who lives up north, has more miles on one of his than I have on all of mine combined but I believe his sleds are as reliable as mine.
As Dsnomo said, you should probably consider how many miles you are going to ride it each year. If you ride 3000+ miles, you may not want to buy a sled with 4000 - 5000 miles. If you ride 400 - 1000, then I think a well maintained sled with 4000 miles should serve you well for several years. If you're looking at Polaris, the liberty engines seem to tolerate higher mileage relatively well (at least based on my brother's experience - he as three that are pushing 10,000 miles each).
Good luck! There are a lot of good used sleds out there but if you find something that you feel is special, be prepared to grab it. The really good ones go fast. I put my mint condition Edge X on J.D. a couple of years back and within 24 hours I had six committed buyers queued up. The first party to respond took it.