Gartner Magic Quadrant Under Fire – Lawsuit Alleges Defamation and more

Gartner Magic Quadrant

A storm is brewing throughout the analyst community as one of the largest and most influential technology analyst firms comes under fire for one of their highest prized research artifacts – The Gartner Magic Quadrant (MQ) – ZL Technologies has filed a lawsuit alleging damages from Gartner’s Email and Archiving MQ and the MQ process as a whole, in which ZL has been positioned as a Niche player since 2005.

From ZL technologies website (here)…

ZL Technologies, a San Jose-based IT company specializing in cutting-edge enterprise software solutions for e-mail and file archiving, is challenging Gartner Group and the legitimacy of Gartner’s “Magic Quadrant.” In a complaint filed on May 29, 2009, ZL claims that Gartner’s use of their proprietary “Magic Quadrant” is misleading and favors large vendors with large sales and marketing budgets over smaller innovators such as ZL that have developed higher performing products.

The complaint alleges: defamation; trade libel; false advertising; unfair competition; and negligent interference with prospective economic advantage.

For those unfamiliar with analysts, Gartner and the Magic Quadrant let me provide a quick overview:

On Gartner: Gartner is able to gain insight into markets, trends, and strategies that most are not – that breadth and depth of knowledge is very difficult to replicate without the support of a large firm.

As the CTO of a enterprise software company we interact pretty regularly with a host of analyst firms, but I want to provide my experience as a Gartner Analyst working in the information security and risk group during the mid-2000’s.

What people may not realize is that an analyst can speak with on average several dozen large enterprise companies, in almost every vertical, in almost every geography around the world every week. Couple this with access to 700 other analysts who do the same thing, add in access to what all the vendors are doing, the fact that analysts spend their time doing nothing but looking at specific problem/solution sets, plus most analysts have extensive experience in the industry to begin with. For example I joined Gartner after spending close to a dozen years with vendors developing security solutions.

In my experience as a former analyst and currently as a CTO of an enterprise software company, the analyst community provides valuable insight into the current and future state of macro and micro technology trends.

On Objectivity: Gartner is a for-profit company and is in business to embrace and enjoy the benefits of capitalism, in this way they function as most publicly traded organizations function. They are beholden to the bottom line and their share holders whilst trying to maintain  customer satisfaction and drive a growing business. There has always been cries that Gartner is a pay-for-play firm, that there are serious conflicts of interest, that the more you spend the more positive the research output is. There are pay-for-play analysts firms, Gartner isn’t one of them.

In my experience Gartner worked very hard to insulate the analysts from the business side of the companies financial efforts. Analysts generally have no idea how much a client spent with the firm, they sometimes are not aware if the vendor is even a client and there is no requirement that a vendor must be a client to be included in research such as MQ’s.

It is not a stretch to imagine that the largest technology providers spend the most money with analysts. CA, Cisco, IBM, Microsoft, Oracle, and Symantec spend a ton with analyst firms, not surprising considering the vast array of products and services they all provide.  They also share another common characteristic, every single one of them has been placed in the niche (lower left) and challenger (upper right) quadrants in multiple areas even though they have spent millions.

On Magic Quadrants: The Gartner MQ, love it or hate it, is is probably the highest read output of any research product that Gartner, or any analyst firm, provides. The quadrant has two dimensions; completeness of vision (horizontal x-axis) and ability to execute (vertical y-axis). Completeness of vision is focused on aspects of the product and marketing/positioning efforts and the ability to execute is focused on sales execution, company size and resources.

For those unfamiliar with the MQ process, here is an overview.

  1. Analyst has a defined market he would like to quadrant, if not already in existence there is a required review process to develop a new MQ
  2. Analysts defines the inclusion and evaluation criteria for each MQ and then adjusts weighting for each area to better address market dynamics over time
  3. Analysts sends out survey questions and spreadsheets for vendors that may be included in an MQ, which requires vendor provided references
  4. Analysts parse through vendor data, perform reference checks, review inquires, conversations, and discuss with other analysts
  5. Analysts plug the data into a spreadsheet template, which automagically calculates positioning
  6. Analysts spend the next several weeks debating “facts” with vendors, most are not happy with either their position or the relative position of their competitors, these couple weeks were without a doubt the worst part of the job.
  7. MQ is published and some use it properly to help guide opinion and others who are too lazy only look at the leaders as a starting point for vendors they will evaluate
  8. Lather, rinse, and repeat annually

What the MQ doesn’t provide

  • Gartner does not maintain a testing facility, nor works with one. The MQ does not provide an evaluation of technology based on an internal evaluation or competitive analysis. In some cases the only interaction an analysts has had with a product is a 30-60 minute web demo
  • Gartner looks at macro trends in markets, not what is important to your specific company. The problem is these macro trends and criteria may or may not meet your needs, what is good for company A may be disastrous for company B – think about your data center automation or CMDB initiatives.
  • Gartner does not dynamically update MQ’s between publication date. The MQ is a point in time snapshot it doesn’t reflect the pace of technology change during the period from one MQ to the next until they are able to manually update the MQ the following year
  • Gartner isn’t based on facts, it is based on opinion – highly tuned and supported opinion, but still opinion – and relies entirely on manual user input. It is prone to error and misinterpretation.

The MQ is only one piece of the evaluation and procurement process and its main purpose is to help guide and validate position, but anyone who uses it as sole source vehicle is most likely going to experience some time in the trough if disillusionment.

In terms of influencing the influencers there is no doubt that the more money spent with an analyst firm provides benefits, such as access to their feelings and opinion on market dynamics – obviously aligning with their thinking provides better positioning. Also the more time you spend with the analysts the more you are able to articulate your benefits, strategy, wins, and company direction – obviously this is beneficial as they become more comfortable with a vendors technology.

On the Analyst “Technology Tax”: Analysts firms are a technology tax. For many it is a cost of doing business in the IT sector. Regardless of what anyone says the more money you spend with analysts the better your benefits. That isn’t necessarily a bad thing, but for many analysts budgets are managed through the marketing budget and AR people are in marketing. Many vendors, regardless of what they say are more interested in positive research mentions and MQ position than in valuable insight. This is unfortunate because the truth about Gartner is they have a massive infrastructure, hundreds of extremely talented analysts, and a global reach to support the level of insight that organizations require to navigate the tactical and strategic IT initiatives that face an increasingly complex, global, and sophisticated IT environment.

As for the ZL Technologies lawsuit, although I am not a lawyer, it seems to be baseless and a case of a frustrated vendor not receiving the position they feel they deserve. For Gartner’s part their defense is quite simple:

Expressions of opinion on matters of public concern are entitled to the protections of the First Amendment and Article I, Section 2 of the California Constitution,” according to Gartner’s reply. “Under the First Amendment, a protected ‘opinion’ is one that does not assert or imply facts capable of being proven true or false. …. As protected speech, expressions of such opinions on matters of public concern are nonactionable.

Curious that more people do not sell product that can neither be proven true or false and when questioned can point to the Constitution as a defense.

Btw – Dave Kellogg CEO of Mark Logic has a great write-up on the lawsuit, including references and links to the court documents – check it out (here)

Advertisements

16 thoughts on “Gartner Magic Quadrant Under Fire – Lawsuit Alleges Defamation and more

  1. This could be a PR strategy. Before this week, how many have actually heard of ZL Technologies? Right, not much. That’s why they are in the loser quadrant. Now most everyone has heard of them. Not sure what their legal fees are, but given what it costs to get companies to write about a company, it could put them on the map.

    Not sure it’s a map I’d personally like to be on, but stranger things have happened.

    • Hey Mike,

      I am positive that the PR aspect of this was part of the discussion at ZL, there is no way marketing and PR were not involved in the discussion. Whether this was purely done for PR as opposed to a knee-jerk, frustration reaction based on years in the “niche” quadrant – not sure – but it has definitely raised their visibility across the industry.

  2. Amrit,

    Thanks for a clear, concise, and well written explanation.

    While I, like you, suspect the suit is without merit – it doesn’t help Gartner fight the ‘pay-for-play’ perception that, IMHO, seems to be growing yearly. I don’t know how they fix it – I just observe it happening.

    Cheers,

    Armorguy

    • Hey Armorguy,

      Thanks for the comments. I think Gartner’s influence goes through periods of scrutiny, but I agree that lately their influence and those who think there are issues with their research methodology is increasing. The MQ, in its current state, is nearing the end of its usefulness and will need to be updated to limit inconsistencies, transparency, and the way the information is used.

  3. I’m using the MQ as a necessary evil in shortlisting vendors for proposals. I know it isn’t fair and Gartner’s opinion doesn’t reflect reality or at least the reality I will have to deal with. However, how objective can you be? If I’m refering to analysts aren’t we doing that because we are looking for an opinion? I think the Forrester Wave is more valuable (and in some cases I use that instead of MQ when available, or a cross section of both). At least they give you the criteria and disclose the detailed results of each criteria (and you can change the weights according to your need). They are more open and hide far less than Gartner does. The law suit might a PR stunt, but I think it’s about time Gartner get’s bad press for their MQ that is just blindly used by too many people that want to come to a decision quickly and say “Don’t blame me, Gartner said this company is a leader, if the leader wasn’t able to deliver how much worse do you think we would have been with some other vendor?” (it’s the “buy cisco and don’t get fired” syndrome).

  4. The good thing about the MQ is that the decision makers love it and as such can assist you in having your choices being executed.
    As for informational value it is on par with the contextual adds google gives for free with it’s answers.

    • Hey Mokum,

      That is the main use of the MQ, those who already made a decision use it to validate their decision. I think you give the MQ too much credit though, Google has a fairly scientific and sophisticated method for serving up contextual ads. I don’t think anyone can argue the MQ has anywhere near the same level of math, but in terms of its value – perhaps 😉

      • Au contraire, the main use of the MQ is for lazy users to not have to do the work to establish the proper short list. Yes, the enlightened few can certainly use the MQ to justify a decision they’ve already made – but those folks are about as common as the Yeti.

      • Hey Mike,

        That has not been my experience. Especially when you are talking about a high-cost technology (either in $ or resources) the majority of the inbound calls that I was involved in around the MQ were generally focused on companies that already had gone pretty far down the path and wanted some validation on who they would move to the next level. It wasn’t just the leaders. There were certainly those that did only look at leaders and those tended to increase in small, nascent markets geenrally due to their being a lack of large vendors putting significant dollars in

  5. The MQ owes its birth to Gideon I. Gartner himself. It was ingenious way of taking theoretical research and turning it into a marketable product that end users could employ to justify their buying decisions and blame Gartner if the installation went south.

    Having run analyst relations and market intelligence for Symantec and worked for a number of analyst firms such as Giga, Sageza, Zona, Frost & Sullivan, etc. – I can attest to the fact that analysts, Gartner’s in particular, are insulated from financial incentives based on vendor spending.

    The game of moving the dot in the MQ is a sport vendors have speculated on for years – especially those as Mike says who are in the Loser’s Quadrant.

  6. Great post Amrit. The key conclusion any reasonable executive of an IT start up must draw from this post is that you must be a Gartner client. If you cannot afford the Gartner fees, don’t bother trying to sell to the Global 2000 market. In addition, Gartner must be at the center of your marketing strategy and tactics.

    Regarding ZL, Mike Rothman called the company a loser. I would rephrase that and say that ZL’s marketing is a loser. Since Mike Rothman is a Chief Marketing Officer, this may be a distinction without a difference.

    • Bill,
      I said ZL was in the loser quadrant. The official term is “niche,” which to most folks in the industry means loser. That’s not to say the company is a loser, there are lots of folks that start in that quadrant and grow to be market leaders – based on changing criteria or better execution (or being bought by a substantial company that provides resources a smaller entity doesn’t have).

      And from my perspective there is no difference between a weak company and weak marketing. With good sales execution folks, a small company gets on Gartner’s radar, regardless of outbound marketing. Though effective outbound marketing can definitely accelerate the process.

      Mike.

  7. Amrit, this is an interesting post. All too many times people trust the magic quadrant to be accurate and helpful in their product decision making process. The problem is the MQ does not treat all products equally or take into account a companies requirements or needs. In fact, in a lot of cases the MQ is completely inaccurate, for example check out the NSS labs’ review of Endpoint security and then compare that to the MQ vendors (it is very clear they are completely wrong). I recently wrote a post that highlights some of these points.

    <http://foregroundsecurity.com/index.php?option=com_content&view=article&id=116:it-is-garners-world-and-we-are-just-lucky-to-live-in-it&catid=42&Itemid=195

  8. Hello,

    On December 4, 2009, ZL Technologies filed an amended complaint against Gartner, Inc. in the United States District Court for the Northern District of California. The Court granted ZL the opportunity to clarify and augment our earlier allegations of defamation and trade libel.

    In the first round of ZL’s legal dispute with Gartner, Gartner argued to the Court that its rankings and other statements in the proprietary “Magic Quadrant Reports” are merely opinions that are not based upon fact, and that they are understood as such by the readers of those reports. However, Gartner’s past statements in marketing materials, white papers, blogs and even the Magic Quadrant Reports themselves, assert that their research and analysts’ opinions are based on a body of facts compiled through what is asserted to be a rigorous process.

    The amended complaint clarifies ZL’s contentions about the inaccuracy of Gartner’s reports, the inherent conflict of interest arising out of Gartner’s voluminous business with the vendors it reviews, and its subsequent bias towards large and established vendors. The amended complaint also adds new detail about Gartner’s repeated claims that its research is based on objective fact—a position exactly opposite to the stance forwarded by Gartner in court.

    While this case is focused on ZL’s dispute with Gartner over the erroneous statements in Gartner’s publications, the issues here also implicate Gartner’s larger business model. Gartner plainly admits that it attempts to leverage value from its largest clients, many of whom are also vendors covered in the company’s research. ZL’s legal filings describe how that business model causes Gartner to favor those large companies at the expense of identifying the best technologies, thus misleading not just the vendors who are inaccurately reviewed by Gartner, but the consumers who base their IT purchasing decisions on Gartner’s biased research.

    ZL is seeking injunctive relief as well as compensatory and punitive damages from Gartner.

    The amended complaint can be found here: http://www.zlti.com/courtdocs/docs/First_Amended_Complaint.pdf

  9. Pingback: Gartner BPMS Magic Quadrant 2010 « Adam Deane

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s