Regular readers will know that I've been a long-time critic of the American Arabic language satellite TV station al-Hurra. Beyond my skepticism about its ability to compete in the intensively competitive Arab TV market, doubts about how it has been conceptualized, its relative lack of transparency about its programming, and amazement about its quality levels, I have been particularly exercised by its claims about audience size. Again and again, the BBG has released excited claims about vast market share which don't seem to be replicated by any independent surveys. This "boosterist" approach to reporting audience survey research badly undermined the BBG's credibility in my eyes - I just couldn't take any of it seriously, even if some of it might have been useful.
Well, it wasn't just me. The GAO has just released a tough report on al-Hurra TV and Radio Sawa. The punchline:
"we were unable to determine the accuracy of MBN's reported audience size and program credibility estimates due to weaknesses in MBN's methodology and documentation."
And again:
"The BBG did not take certain steps that could have explained and increased the reliability of its estimates, such as by fully documenting its research and estimation methods, measuring the level of uncertainty surrounding its estimates, disclosing significant limitations, and consistently implementing policies and procedures for verifying data... it has primarily used a methodology that cannot be reliably projected to the broader population... the BBG has not taken steps to explain and increase the reliability of MBN's performance information, such as by maintaining more detailed documentation to support its estimates, reporting significant data limitations, limiting the scope of its projections to areas actually covered by its surveys, and developing BBG policies and procedures for verifying performance data."
The GAO authors note that "in 12 out of 14 cases, the BBG used nonprobability surveys, which cannot be reliably projected to the broader population, to develop its regional estimates for audience size and credibility." The survey of Morocco, for instance, covered only 35% of the general population; in Egypt, the response rate was only 19%; and Jordan's only surveyed some governorates. In each case, according to the GAO results were reported as reflecting national samples despite such data limitations.
Documentation was also lacking, making it difficult for independent analysts to ascertain the credibility of the surveys: "the BBG and its research contractors were unable to provide us with certain documentation commonly required by international broadcasting research standards. CIBAR requires that in all measurement research the sampling methods used and other technical aspects of the survey be open to independent scrutiny... For two cases, the BBG was unable to provide us with any survey documentation, and for all but one case the BBG and its contractors were unable to provide us all the detailed information we requested." The data shortcomings were so severe that "the BBG has not been able to measure sampling errors... the BBG only has a rough idea of what the margins of error might be for its surveys, further limiting confidence in the reliability of its current performance information." The GAO does not pursue the question of what the shoddy documentation might or might not conceal.
The bottom line: the GAO considers the BBG's audience research to be deeply problematic, literally unusable for determining whether al-Hurra has met performance targets. The BBG offers a rebuttal to the GAO report, of course, defending the professionalism and quality of its audience research, comparing it favorably to other surveys (like Pew and Zogby), and attributing the acknowledged shortcomings to the difficulties of working in the field in the Middle East. Some of its complaints seem reasonable, others reek of desperation (how can they claim that nobody expected to have to produce certain documentation when such documentation is the industry standard?). My only disappointment here is that the GAO did not push the question of the BBG's reporting of the audience data.
The shortcomings in audience research are compounded by the fact that the stations have not been subjected to mandated program reviews: "While the BBG calls for its broadcasters to undergo an annual program review, Radio Sawa has only held one such review, and Alhurra has not completed one to date. Furthermore, unlike reviews of other broadcasting services and contrary to BBG guidance, the Radio Sawa program review was less comprehensive in that it relied upon several audience panels but did not include input from in-house analysts or external expert listeners. The lack of annual, comprehensive program reviews hampers BBG's ability to assure its audience, Congress, and the BBG that its services are complying with its journalistic standards and mission."
This gets to another of my longstanding complaints - the lack of transparency about the station's content. If I want to know who appeared on al-Jazeera to discuss Lebanon, or how many times Darfur was mentioned on al-Arabiya, I can use the generally functional search engines on their content-rich websites (which offer full transcripts of their talk shows and programs). I don't have access to anything like that kind of information about al-Hurra, and neither does Congress or any of the oversight boards. The BBG does send out a roundup of guests and topics to an email list, but it isn't systematic or publicly available (I used to get them, but haven't received one since my scathing post about al-Hurra's coverage of the Israel-Lebanon war, for whatever reason.) If they don't even have internal program review boards offering oversight, then there isn't really any at all.
Look, I'm glad that the GAO has weighed in on my side of this debate. It's refreshing. But I don't find it a particularly useful or interesting debate. Al-Hurra's struggles are obvious and well-known, and far too much time has been wasted on pointless bureaucratic arguments. I've said before that there's no reason at this point to kill al-Hurra - the sunk costs are enormous, and now that we've paid for this station we might as well get something out of it. My hope is that an authoritative external voice like the GAO can wake up the right people and start a badly needed revamping process. With Ken Tomlinson potentially on the way out, that may actually finally be possible.
Marc, sadly this was a wate of taxpayers hard earned money. What did this six or seven months of time and effort cost the American taxpayers. Nothing changed! Again, the GAO send inexperienced GS-9 and 11 auditors to review and basically they had not real world understanding of 1) television, and 2) financial management. Read the repeort and you will see.
Posted by: PatrickF | June 14, 2007 at 10:29 PM