[PMC] The “de-biasing” survey

Quick tophat: I am working to better understand how self-employed devs improve their career. Would you be willing to spare 3m for a survey? It will mean the world to me.

-> https://www.getfeedback.com/r/fNWSDcfj

I’m not selling anything; you have my NO SALES PITCH GUARANTEE.

(Readin’ time: 3m 46s)

I have, uh… opinions about how consultants might invest in their career. 🙂

I’d like to think most of these opinions are actually points of view that can be substantiated, or a posteriori assessments based on modeling what super-successful consultants have done.

But at some level, these opinions are baked into how I see the world.

And therefore, if I plop myself down to write survey questions to attempt to answer the following questions, I have to be careful:

  • “How do self-employed software developers invest in their career?”
  • “How do they make decisions around investing in their career?”

I have to be careful because my opinions, points of view, and worldview around this issue will almost certainly influence how I write my survey questions.

First, I may choose words that are loaded, or have some sort of value judgement embedded in them.

Here’s a cartoonish example of that:

Q: Do you invest time or money in your career?

  • A1: Yes, I am a forward-thinking, wise self-employed person.
  • A2: No, I am short-sighted and choose to shoot myself in the foot by not investing.

Again, that’s a cartoonish example, exaggerated for effect (actually, it reminds me a lot of the bullshit passive-aggressive wording that still seems to be common on popup opt-in forms on websites. Blech!). But in my survey, this is actually a serious potential problem, and I don’t arrogantly assume I’m free from bias.

Second potential problem: I might write questions with multiple choice answers that are biased in some way. The answer options might exclude options that I could learn a lot from but I didn’t even think to include because my worldview filters them out. Or the answer options I do choose to provide are consciously or unconsciously chosen based on my preferences or worldview rather than my audience’s and so I get skewed responses.

There are certainly other biasing mechanisms I’m likely to suffer if I just plop myself down unprepared and start writing survey questions, but the above two seem like the most significant threats to me.

To address this, I’m running what I think of as a “de-biasing” survey first before I write and recruit for the “real” survey.

The de-biasing survey is probably more important than the “real” survey, meaning that if I decided to skip the de-biasing survey I may as well also skip the “real” survey because it would likely be too biased to be helpful.

The key difference between the two surveys is the structure of the questions. The de-biasing survey is much more open-ended in how the questions are phrased. It’s the result of me consciously trying to take as much of my PoV out of the questions as possible.

Here are the questions I used for the de-biasing survey:

  • In the last two years, have you spent time or money on career development?
    • Yes
    • No
  • [If Y] What have you spent time and money for career development? [open ended]
  • In the last two years, have you spent time or money on developing your technical skills?
    • Yes
    • No
  • [If Y] What have you spent time and money for developing your technical skills? [open ended]
  • In the last two years, have you spent time or money on business or self-employment skills?
    • Yes
    • No
  • [If Y] What have you spent time and money for business or self-employment skills? [open ended]
  • Consider your entire career as a self-employed software developer and times you have gotten new opportunities, better projects, or other forms of career improvement. What do you think led to these improvements in your career? [open ended]
  • What is your age?
  • Would you like me to share the 100% anonymous results of this survey with you? If so, please leave your email address here and I will use it only to communicate with you about the results of this survey.

You can see that the critical questions are open-ended, and I use multiple-choice questions only to help respondents skip past seeing open-ended questions where they’ll have nothing to say.

This allows me to learn about my audience’s wording, phrasing, and–to a partial degree–the mental schemata they use when thinking about investing in their career.

This will be critically important as I design the “real” survey, which I’m going to stop calling the real survey and simply refer to as the second survey in this research project. It’s not any more or less real or important than the de-biasing survey.

I’ll also reach out to some of the folks who left me their email address and request a brief interview so I can drill deeper into their thinking.

Again, I am working to better understand how self-employed devs improve their career. Would you be willing to spare 3m for a survey? It will mean the world to me.

-> https://www.getfeedback.com/r/fNWSDcfj

I’m not selling anything; you have my NO SALES PITCH GUARANTEE.

Tomorrow I’ll talk about recruitment,

-P