Ok. Well I'm not sure experience has too much to do with it. I just think that if one tree is being drawn over the top of another one then in certain places it just doesn't look good.
I think strictly speaking, if one tree is overlapping another then it's trunk needs to be lower than the one behind to give the illusion that it is closer to the viewer. It also can't be too close to covering the whole tree, unless the whole forest is like that so it appears that the viewing angle is lower.
The way I have done it so far (and it is by no means perfect yet) is basically your own way of doing it modified to allow a little random tree movement. This is fine on the X-axis using the current method as you can easily find out where the last tree was positioned. It's not too clever with jitter on the Y-axis though as it doesn't know where the tree above is. This is perfectly ok when the trees are spaced a little but the more compact they are the harder it is to make them appear randomly placed as the Y jitter is restrained too much because of bad tree placement. And that's what I was trying to work towards - a really compact forest that follows the rules of overlapping but without too many obvious rows or columns.
So I believe the best way to go from here, which is what I hope you are doing, is rewriting it to populate an array that holds every tree position.
My own next move with the version I have would be to just hold the previous row of trees in an array to see if that can solve the Y jitter problem.
Anyway... here's the source of what I have so far. I managed to get about half an hour on it this morning so I may have broken it a little. I was playing around with allowing Y-jitter when there is a slight overlap but I didn't have time to see it through to completion so it may behave oddly there.
I'm looking forward to seeing your new version. How long do you think it will take you?
I'll make a post with the few new features I've been considering later.