I've just finished re-reading the fascinating bestseller Stumbling Upon Happiness (by Harvard psychologist Daniel Gilbert), a 300 page book which, per the paperback's back cover, examines the "foibles of imagination and illusions of foresight that cause each of us to misconceive our tomorrows and misestimate our satisfactions". Documented by several hundred citations of clinical studies, Gilbert's conclusions can help you "course correct" the inherent biases in thinking that cause the misconceptions and misestimations. Because most of the work that I do with my clients involves, at least to some degree, projections of future happiness that would result from making choice A versus choice B (or C or D), I will review Gilbert's main conclusions here.
First, imagination (projecting a future that has not yet been experienced) fails to account for the fact that humans have a very powerful ability to make themselves feel better by "proving" that they're right. We do this through selective perception (noticing phenomena or facts that "support our case" and ignoring others that contradict it), through surrounding ourselves with people who share and thus confirm our points-of-view, and by rationalizing our errors ("I should never have taken that IT job, but it taught me that I need to work in an environment with lots of person-to-person interaction"). Of course, if someone is prone to depression they can use these same means to confirm the correctness of their bleak outlook, but most people have a wired-in desire to feel better, rather than worse, about themselves (thanks to natural chemicals like dopamine and seratonin).
Second, imagination is heavily influenced by what is going on in the present. Everyone is familiar with the recommendation that upon ending a long-term relationship it is unwise to immediately jump into another. Why? Because the minuses of the just-ended relationship weight so heavily on judgment that it is likely that a new partner will be chosen primarily because he / she has the opposite qualities. Another example: if you've just eaten a large meal, your estimate of how enjoyable a large meal would be if consumed on the next day is likely to be too low. Gilbert goes as far as to claim that "We cannot feel good about an imaginary future when we are feeling bad about an actual present" because it's so hard to imagine that our future self will be able to view things from a different perspective.
Third, evolution has "trained" us to react more negatively to the possibility of loss than positively to the possibility of gain. Humans are inherently risk averse. For example, most people would refuse a bet that had an eighty-five percent chance of doubling our life savings and a fifteen percent chance of losing it, even though the gain was almost six times as likely as the loss. People place greater-than-warranted emphasis on the downside of a prospective job or relationship change than on the upside.
Fourth, our culture strongly influences our imagination. Take the deeply engrained belief (constantly fortified entertainment and advertising) that being rich (or being able to buy lots of things) leads to increased happiness. We imagine that a 25% increase in pay (say from $100,000 to $125,000) would create a large bump in our satisfaction with life, but study after study confirms that, once people achieve a comfortable level of existence (somewhere between $60,000 and $90,000, with all of the necessities and an occasional luxury provided for), an increase in income (even a large one) has a negligible effect on happiness. An even more surprising research finding: having and raising children is strongly correlated with decreased levels of happiness, and that when children finally leave the home of their parents, the parents' happiness levels rise. Of course, our society would eventually disappear if this were a deeply held cultural norm, so the opposite conclusion is promulgated: that children are among the chief joys of life.
How to correct for the biases that come into play when we try to imagine or project the future? An excellent way to "...make predictions about our own emotional futures is to find someone who is having the experience we are contemplating and ask them how they feel". Career testing turns out not to be a substitute for this, for numerous reasons - see article on the numerous flaws of Myers-Briggs:
Interviewing a few people who are currently doing what you are thinking of doing turns out to be a more accurate gauge than might be expected. If that information is added to an awareness of, and compensation for, the biases mentioned above, there will be a much greater chance of your "stumbling upon happiness" and making better choices.