Poets and Writers - WTF???

I've been involved in conducting and evaluating survey research for more than twenty years, ever since completing graduate work in regulatory economics with an emphasis on survey research design.  I was dismayed that Poets and Writers, in its November/December on-line magazine, published rankings of MFA programs that are based on bogus research methods. The author of the rankings has no credentials as a pollster.  Why would Poets and Writers entrust such an important assignment -- one with a lot at stake for many individuals and writing programs  -- to a blogger?   Here's the letter I sent to the Editor: 

Dear Mr. Larimer:

 

It is astonishing that Poets & Writers would publish Seth Abramson’s MFA program rankings (www.pw.org Nov/Dec 2009) without scrutinizing his research methods, which are utterly bogus. Unlike a valid poll, which would survey a randomly selected representative sample of the total pool of current and potential MFA applicants, Abramson's poll reflects only the responses of self-selected readers of his blog, and there is nothing to prevent individuals from responding more than once from multiple locations.  The people drawn to a single point-of-view blog are already inclined to agree with the blogger's biases, and even a cursory look at Abramson's blog reveals just how many extreme biases he has. Any responsible social science researcher knows that findings based on self-selected survey responses are not just worthless -- they can be harmful and misleading.

 

Sincerely,

Stacey Harwood

Comments

[Laura Orem]

Great letter! And good for you for calling them on it.

[Seth Abramson]

Stacey,

P&W asked me to do this because they know--as thousands of MFA applicants have known for three years now--that while my research is in part unscientific (and in no way attempts to obscure that fact), it's also far and away the most extensive research ever done on MFA programs. It's worth noting that the sort of scientific polling you're demanding has never been done for MFA programs--by anyone. The 1996 rankings were based on a single question in a single questionnaire sent to MFA faculty members; unless you consider MFA faculty members a representative sampling of what impoverished aspiring student-writers half their professors' age believe about what makes for the best MFA experience, that USNWR polling method should have borne the brunt of even more ire--at least this unscientific method attempts to poll the right population, and asks the right questions. It's a step forward. It's worth noting, too, that a) most of the polling was not done on my website, as you suggest it was, but on the MFA Weblog, which as a blog is POV-neutral, and the voting posts on my own blog were all POV-neutral posts individually linked to from the MFA Weblog (i.e., those who use my personal blog for its MFA data rarely report being regular readers of the rest of the blog), and b) you're responding only to one portion of what P&W published--the rankings also represent the first-ever comprehensive listing of full-res MFA programs in the U.S., and offer P&W's readership the first-ever non-subjective rankings of programs' hard-data funding, selectivity, and postgraduate placement records, as well as including a veritable mountain of hard-data information about each program. I think the most important thing is that my research methods are totally transparent--you may consider them insufficient (in the process demanding a level of scientific certainty in rankings that no MFA rankings have ever achieved; by way of another example, the "rankings" published by The Atlantic in 2007 were based on one journo's opinions, but I don't know that you or anyone staged a protest then), but you can't imply they're somehow deceitful. Likewise, the only "bias" you could possibly be referring to (as to MFA programs) is my preference for funded programs--a preference mirrored not only by 100% of the MFA applicant community, but 100% of undergraduate professors who are advising their students on MFA programs. You won't find anyone in the field of MFA research, or MFA programs generally, who doesn't share this [what you call] "bias." Even Columbia, whose lack of funding I've certainly criticized publicly and at some length (and this is clearly what you mean when you urge people to take a "cursory look" at my blog, though again you're actually referring to Tom Kealey's MFA Weblog, not my blog, where I don't discuss Columbia) has had their own faculty admit to media outlets that they wish they had more funding to offer. Are they "biased" against themselves? As President Obama likes to say, the perfect must not be the enemy of the good; these rankings are the best option presently available in the field of MFA research, they're better than what's been done before, and they're continually being perfected to the best of my ability--I wasn't paid to do this research, I've been doing this since 2007 because I enjoy it. Whatever you think of me (and your bias against me is well-documented, as is your personal stake in these rankings), you can't deny that a huge portion of the MFA community--perhaps not many in your social circle, as admittedly the unfunded NYC programs in some instances fared poorly in the poll (though not in others: Brooklyn; Hunter; &c)--has already seen these rankings online and concluded that they look about right. Just today I was speaking with a former MFA director who said the top 25 offered virtually no surprises to him whatsoever. That's an opinion I've heard countless times.

Take care,
Seth

[Stacey]

The length of the reply above correlates directly with the weakness of the position. The crucial point, which the writer wishes to bury amid the verbiage, is that his work is admittedly unscientific, partial, capricious, and undertaken by someone with no real qualifications for the task. Just because a scientific survey hasn't been done, doesn't mean it can't be done. And that's no justification for passing off faux research as "good." Taking an assignment without pay doesn’t justify shoddy work. While no survey is perfect, one based on a self-selected sample is a bad one and can cause harm. The researcher has no control over the sample. The survey is not "better than what's been done before." The research is not transparent. The blogger doesn’t reveal the response rate. Of those who could have responded to the survey, how many did? Of those who didn't, why didn't they? If the blog and website are as popular as claimed, a response of only 500 is pitiably low, further compromising the results. No account is given for those excluded from the survey -- those current and potential MFA applicants who didn't know about it, could not have participated, and whose opinions are not represented. The skills and knowledge to address questions about sampling and response rate are covered in survey research 101. Poets and Writers should know better.

[Laura Orem]

I'm confused by Mr. Abramson's response, not only by the convoluted explanation of the survey's methodology, but especially regarding Stacey's "well-documented" biases against him and regarding MFA programs. First off, Stacey's critique is of Mr. Abramson's methods, which, as a professional statistician, she has legitimate concerns with, not Mr. Abramson himself. Second, I've known Stacey for 15 years. Not once have I heard her express an opinion one way or another toward Mr. Abramson; in fact, I've never heard her mention his name or, prior to today, read a word of hers that refers to him. Finally, what personal stake of hers in MFA programs? As far as I am aware, except for her own MFA degree, which thousands and thousands of us share and which she received years ago, Stacey has no professional connection to any MFA program. She works for the state of NY in a completely different field, and her work here on the blog is completely independent of any MFA program.

[Mark Gibbons]

"-I wasn't paid to do this research, I've been doing this since 2007 because I enjoy it." This is hilarious. I'm interested in biology - I think I'll perform some surgery today because I enjoy it!

[jesusangelgarcia]

It doesn't even matter. As far as I can tell from my own comprehensive research, most MFA programs and (ahem) Maui writing workshops are a scam -- great for administrators who run the show and reap the profits; good for writers who need to teach to earn a living wage; bad for the bank accounts of aspiring writers, most of whom will quit writing soon after graduation.

www.badbadbad.net

[Laura Orem]

Shabby research and faulty results do matter, particularly when they are presented as reliable and "scientific."

If writers quit writing after their MFAs, that's the fault of the writers, not of the program. Being a writer is hard. It's easy to find excuses for not writing, but blaming an unsatisfactory program for your own not-writing is a cop-out. Your degree won't get your published, and it won't write your work for you. Hopefully it will expand your experience and knowledge, and give you time to read other writers and to work on your craft, but the rest is up to you.

[Stacey]

Readers, Since the last comment was posted, we have received several comments from Mr. Abramson, which we cannot post not only because they are far too long but because they are inappropriate and defamatory. We are committed to the proposition that a frank exchange of ideas can take place with civility and without recourse to name-calling, hysteria, and logorrhea, often a thinly disguised form of bullying.
Stacey

[Victor Montero]

Let me see if I have this right. Something purporting to be a survey of MFA programs polled only applicants and potential applicants -- and not graduates, current students, and current and former faculty. Is this wise? Applicants are only as good as their sources of information. How well can you judge the quality of an institution that you've never even been part of?

[Reader]

This is the general tendency on the internet from what I have seen. Anyone who questions the validity of these rankings in anyway is attacked incessantly by their creator. It is no surprise that he is doing the same here.

Mind you, he now has a very real financial stake in the validity of the rankings. Even worse, this has now been extended into a consultation company that helps college students with deep pockets get that extra edge over their competition.

He also used the rankings as a tool to blatantly self-promote his own poetry. The supposedly 'objective' rankings (when they were online) were covered with advertisements for the creator's book.

[Anne Caston]

I suspect if a poll offers "no surprises whatsoever," it's a waste of time and effort.

[Michael]

I have no problem with his survey showing up in Poets and Writers. Sure, the editors could have commissioned (and probably should still commission) a properly scientific survey, in which readers could have more faith. This may well be more reliable . . . in a year or more, when it's finally finished and the people who need to know this stuff right now no longer need to know it.

Certainly, a disclaimer is in order, alerting readers to the non-scientific nature of the survey. That said, the results are still worth considering. After all, I have yet to see proof that his results are wrong.

Finally, I think it seems kind of hinky to omit Mr. Abramson's follow-up comments. I'd prefer that we readers be the ones to assess whether a comment is too inappropriate or bullying. Don't worry. We can take it.

[Phoebe]

After all, I have yet to see proof that his results are wrong.

How would one prove that program rankings are right or wrong?

[Elisa]

"even a cursory look at Abramson's blog reveals just how many extreme biases he has"

Extreme biases? This reminds me of O'Reilly calling liberals "far left." What are these supposed extreme biases?

[Noah Burke]

You were right not to let abramson hijack this space. He would shout down everyone else in the room. And he has a bias against everyone who disagrees with him. NB

[Bobby]

I'm shocked the P&W ran Abramson's rankings without mentioning* that he is also running a consulting firm that tries to get students admitted to the same MFA programs he's ranking. (For more on which, see here.) I'm no expert, but that seems to me a fairly disqualifying conflict of interest.

* At least in the online version of the article, I haven't seen the print version.

[X]

abramson's "ALC" has nothing to do with abramson- or ALC- having any interaction whatsoever with the programs. it's a tutorial service for those who haven't had a chance yet to fully workshop their portfolios. It's the same thing creative writing instructors do when they assist undergrads who pay tuition to take undergrad classes. if you wouldn't protest lyn hejinian compiling and publishing data about mfas, you shouldn't be protesting this. get your facts straight please.

[Bobby]

I have my facts straight. (Though not, apparently, my HTML tags.) Abramson is selling a service that purports to help people get into MFA programs. The more of his clients who get into top programs, the better his service looks and the more new clients he can attract and the more he can charge. He therefore has a direct financial incentive for skewing his rankings toward those programs that he knows will be easier to get people into.

I'm not accusing Abramson of doing this--as Stacey points out, his "research methods" are so ridiculously inadequate that there's no way to know if he is skewing them. It doesn't matter, though: the conflict of interest exists whether he acts on it or not. No newspaper (not even the much-reviled USNWR) would have allowed him to publish under the shadow of that kind of conflict, but I'm simply amazed that P&W let him get away without a disclosure.

[Jason Lester]

"The more of his clients who get into top programs, the better his service looks and the more new clients he can attract and the more he can charge. He therefore has a direct financial incentive for skewing his rankings toward those programs that he knows will be easier to get people into."

This statement reveals an unawareness (quite possibly widespread outside of the "industry") of the highly competitive nature of MFA programs in creative writing. Harvard Law School has an acceptance rate of 13%. I would be astonished if any programs listed in Abrahamson's top 20 has an acceptance rate higher than 5%.

[Bobby]

This statement reveals an unawareness (quite possibly widespread outside of the "industry") of the highly competitive nature of MFA programs in creative writing.

This statement reveals an unawareness (quite possibly widespread inside the "industry") of logic. It doesn't matter if Abramson's top 20 have acceptance rates below 0.5%. As long as all of the schools don't have identical rates, then some schools will be easier to get into than others, and the conflict exists.

But maybe I'm misreading you. Maybe your point is that MFA programs are so competitive that the "service" Abramson is selling can't actually help anyone gain admission to a school they weren't going to get into in the first place, and that therefore a person would do well to squirrel away any money they might have spent on such a service for the day when she realizes just how hard it is to make a living as a writer, so that she won't have to dream up some scheme for fleecing young writers as a way to supplement her non-existent income. If that's the case, then I'd say, well, yes.

[Jason Lester]

Dear me, good thing Abrahamson has been banished from this comment section so we're spared hostility and ad hominem attacks.

You said, "The more of his clients who get into top programs, the better his service looks." I responded that the "top programs" in his ranking all have exceptionally low acceptance rates, implying that there is no program "easier to get people into" that he can cherry pick for his clients and then deviously bump up in his rankings. In fact, the one notable outlier in this respect is Columbia, which has a disproportionately high acceptance rate relative to its ranking. Make of Abrahamson's real or perceived bias against this program what you will.

Now, in response, you say "the conflict exists" as long as "some schools will be easier to get into than others," not just the "top schools." I can only respond that I suppose poets should not write poetry reviews, compile poetry anthologies, teach poetry, judge poetry contests (actually, that last one isn't too bad of an idea), or indeed be involved in anything where they stand to make a profit off of poetry because "the conflict exists." I think if Abrahamson wants to glean a little patronage off of bewildered or affluent writers, that's his prerogative. At the very least, it has to be money better spent than applying to a writing contest judged by Jorie Graham.

[X]

bobby, really none of this changes the fact that you are advancing a conspiracy theory- you haven't laid out any plausible causality to link the money mfa applicants pay to a tutorial service- no different from the money paid to sackett street, any of the services advertised in p&w, or even undergrad workshops or online workshops- to rankings which were 100% compiled publicly as p&w says- yes, all votes and all data were taken from public sources, so how in the world could it get skewed, unless you're saying abramson just hopes no one ever checks the data, which is absurd- so i actually think you've yet to say anything accurate. the service doesn't 'purport to help people get into mfa programs' any more than an undergrad cw workshop does- read the alc website. and the alc operation doesn't advertise where or if writers they work with got in, so how does that mess the theory up. i also agree with the poster above who pointed out that what you've described, the wild conspiracy theory is an impossibility due to the fact that how does one 'manipulate' acceptance rates lower than harvard undergrads?- not to mention that you can look at the public data yourself, p&w actually did a selectivity ranking, and see that the more selective programs are the highest ranked, not the lowest. to say 'there's no way to know' if the data is being skewed makes no sense when all of the data was and still is public. look, it's fine to think the guy is annoying but i think it's another level to put on a tinfoil hat and make claims that show you dont understand the way the rankings were done.

[Stacey]

OK Folks, No new points are coming out of the discussion. Please don't comment unless you can add something new. Some serious concerns have been raised here, chiefly with the quality of the sample on which the poll results are based, the pollsters lack of qualifications, and the perception of the pollster's conflict of interest. If you share these concerns, why not drop P & W a line? And here's an interesting document to guide your letter(20 Questions a Journalist Should Ask About Poll Results): http://www.ncpp.org/node/4/#4 Click on my name below for the link.
Stacey

[Brent Jones]

If a consulting service for MFA applicants (other than his own) published rankings of MFA programs based on half-baked assumptions and self-serving research methods, Abrahamson would be the first to cry conflict of interest.

[Bobby]

In keeping with Stacey's request, and for the sake of my own mental well-being, I'll ignore the responses to my comments above, which seem rather dramatically (and, I have to presume, willfully) to misunderstand the point.

But one question that X raises demands a response. Abramson's selectivity rankings are no more "objective" than the rest of his junk data. As he concedes in a long comment he left beneath the P & W article, he couldn't be bothered to ask writing programs for application data, a lapse he incredibly tries to justify in the name of transparency. In fact, in addition to the helpful tip sheet that Stacey pointed to, I'd recommend a look at SA's comment for superabundant proof why no serious editorial enterprise should have touched his rankings with a ten-foot pole.