A storm is brewing throughout the analyst community as one of the largest and most influential technology analyst firms comes under fire for one of their highest prized research artifacts – The Gartner Magic Quadrant (MQ) – ZL Technologies has filed a lawsuit alleging damages from Gartner’s Email and Archiving MQ and the MQ process as a whole, in which ZL has been positioned as a Niche player since 2005.
From ZL technologies website (here)…
ZL Technologies, a San Jose-based IT company specializing in cutting-edge enterprise software solutions for e-mail and file archiving, is challenging Gartner Group and the legitimacy of Gartner’s “Magic Quadrant.” In a complaint filed on May 29, 2009, ZL claims that Gartner’s use of their proprietary “Magic Quadrant” is misleading and favors large vendors with large sales and marketing budgets over smaller innovators such as ZL that have developed higher performing products.
The complaint alleges: defamation; trade libel; false advertising; unfair competition; and negligent interference with prospective economic advantage.
For those unfamiliar with analysts, Gartner and the Magic Quadrant let me provide a quick overview:
On Gartner: Gartner is able to gain insight into markets, trends, and strategies that most are not – that breadth and depth of knowledge is very difficult to replicate without the support of a large firm.
As the CTO of a enterprise software company we interact pretty regularly with a host of analyst firms, but I want to provide my experience as a Gartner Analyst working in the information security and risk group during the mid-2000′s.
What people may not realize is that an analyst can speak with on average several dozen large enterprise companies, in almost every vertical, in almost every geography around the world every week. Couple this with access to 700 other analysts who do the same thing, add in access to what all the vendors are doing, the fact that analysts spend their time doing nothing but looking at specific problem/solution sets, plus most analysts have extensive experience in the industry to begin with. For example I joined Gartner after spending close to a dozen years with vendors developing security solutions.
In my experience as a former analyst and currently as a CTO of an enterprise software company, the analyst community provides valuable insight into the current and future state of macro and micro technology trends.
On Objectivity: Gartner is a for-profit company and is in business to embrace and enjoy the benefits of capitalism, in this way they function as most publicly traded organizations function. They are beholden to the bottom line and their share holders whilst trying to maintain customer satisfaction and drive a growing business. There has always been cries that Gartner is a pay-for-play firm, that there are serious conflicts of interest, that the more you spend the more positive the research output is. There are pay-for-play analysts firms, Gartner isn’t one of them.
In my experience Gartner worked very hard to insulate the analysts from the business side of the companies financial efforts. Analysts generally have no idea how much a client spent with the firm, they sometimes are not aware if the vendor is even a client and there is no requirement that a vendor must be a client to be included in research such as MQ’s.
It is not a stretch to imagine that the largest technology providers spend the most money with analysts. CA, Cisco, IBM, Microsoft, Oracle, and Symantec spend a ton with analyst firms, not surprising considering the vast array of products and services they all provide. They also share another common characteristic, every single one of them has been placed in the niche (lower left) and challenger (upper right) quadrants in multiple areas even though they have spent millions.
On Magic Quadrants: The Gartner MQ, love it or hate it, is is probably the highest read output of any research product that Gartner, or any analyst firm, provides. The quadrant has two dimensions; completeness of vision (horizontal x-axis) and ability to execute (vertical y-axis). Completeness of vision is focused on aspects of the product and marketing/positioning efforts and the ability to execute is focused on sales execution, company size and resources.
For those unfamiliar with the MQ process, here is an overview.
- Analyst has a defined market he would like to quadrant, if not already in existence there is a required review process to develop a new MQ
- Analysts defines the inclusion and evaluation criteria for each MQ and then adjusts weighting for each area to better address market dynamics over time
- Analysts sends out survey questions and spreadsheets for vendors that may be included in an MQ, which requires vendor provided references
- Analysts parse through vendor data, perform reference checks, review inquires, conversations, and discuss with other analysts
- Analysts plug the data into a spreadsheet template, which automagically calculates positioning
- Analysts spend the next several weeks debating “facts” with vendors, most are not happy with either their position or the relative position of their competitors, these couple weeks were without a doubt the worst part of the job.
- MQ is published and some use it properly to help guide opinion and others who are too lazy only look at the leaders as a starting point for vendors they will evaluate
- Lather, rinse, and repeat annually
What the MQ doesn’t provide
- Gartner does not maintain a testing facility, nor works with one. The MQ does not provide an evaluation of technology based on an internal evaluation or competitive analysis. In some cases the only interaction an analysts has had with a product is a 30-60 minute web demo
- Gartner looks at macro trends in markets, not what is important to your specific company. The problem is these macro trends and criteria may or may not meet your needs, what is good for company A may be disastrous for company B – think about your data center automation or CMDB initiatives.
- Gartner does not dynamically update MQ’s between publication date. The MQ is a point in time snapshot it doesn’t reflect the pace of technology change during the period from one MQ to the next until they are able to manually update the MQ the following year
- Gartner isn’t based on facts, it is based on opinion – highly tuned and supported opinion, but still opinion – and relies entirely on manual user input. It is prone to error and misinterpretation.
The MQ is only one piece of the evaluation and procurement process and its main purpose is to help guide and validate position, but anyone who uses it as sole source vehicle is most likely going to experience some time in the trough if disillusionment.
In terms of influencing the influencers there is no doubt that the more money spent with an analyst firm provides benefits, such as access to their feelings and opinion on market dynamics – obviously aligning with their thinking provides better positioning. Also the more time you spend with the analysts the more you are able to articulate your benefits, strategy, wins, and company direction – obviously this is beneficial as they become more comfortable with a vendors technology.
On the Analyst “Technology Tax”: Analysts firms are a technology tax. For many it is a cost of doing business in the IT sector. Regardless of what anyone says the more money you spend with analysts the better your benefits. That isn’t necessarily a bad thing, but for many analysts budgets are managed through the marketing budget and AR people are in marketing. Many vendors, regardless of what they say are more interested in positive research mentions and MQ position than in valuable insight. This is unfortunate because the truth about Gartner is they have a massive infrastructure, hundreds of extremely talented analysts, and a global reach to support the level of insight that organizations require to navigate the tactical and strategic IT initiatives that face an increasingly complex, global, and sophisticated IT environment.
As for the ZL Technologies lawsuit, although I am not a lawyer, it seems to be baseless and a case of a frustrated vendor not receiving the position they feel they deserve. For Gartner’s part their defense is quite simple:
Expressions of opinion on matters of public concern are entitled to the protections of the First Amendment and Article I, Section 2 of the California Constitution,” according to Gartner’s reply. “Under the First Amendment, a protected ‘opinion’ is one that does not assert or imply facts capable of being proven true or false. …. As protected speech, expressions of such opinions on matters of public concern are nonactionable.
Curious that more people do not sell product that can neither be proven true or false and when questioned can point to the Constitution as a defense.
Btw – Dave Kellogg CEO of Mark Logic has a great write-up on the lawsuit, including references and links to the court documents – check it out (here)