Inside the Magic Quadrant: What Gartner Gets Right (and Wrong)

If you've ever seen a Gartner Magic Quadrant and wondered what really happens behind the scenes, who decides where vendors land and why, here's what I learned as a former Gartner analyst.

On Gartner

Gartner is able to gain insight into markets, trends, and strategies that most are not, and that breadth and depth of knowledge is very difficult to replicate without the support of a large firm.

As the former CTO of multiple enterprise software companies, I interacted regularly with a host of analyst firms, but I want to share my experience from the other side of the table, as a Gartner analyst working in the information security and risk group during the mid-2000s.

What people may not realize is that an analyst can speak with several dozen large enterprise companies, in almost every vertical, in almost every geography around the world, every week. Couple this with access to 700 other analysts who do the same thing, add in visibility into what all the vendors are doing, the fact that analysts spend their time doing nothing but looking at specific problem and solution sets, and that most analysts arrive with extensive industry experience. I joined Gartner after spending close to a dozen years with vendors developing security solutions.

In my experience as a former analyst and a CTO and founder of enterprise software companies, the analyst community provides valuable insight into the current and future state of macro and micro technology trends.

On Objectivity

Gartner is a for-profit company and is in business to embrace and enjoy the benefits of capitalism. In this way they function as most publicly traded organizations function: beholden to the bottom line and their shareholders while trying to maintain customer satisfaction and drive a growing business.

There have always been cries that Gartner is a pay-for-play firm, that there are serious conflicts of interest, that the more you spend the more positive the research output. There are pay-for-play analyst firms. Gartner isn't one of them.

In my experience, Gartner worked very hard to insulate the analysts from the business side of the company's financial efforts, at least they did when I was there. Analysts generally have no idea how much a client spent with the firm. They sometimes are not aware whether the vendor is even a client, and there is no requirement that a vendor be a client to be included in research such as MQs.

However, it isn't a stretch to imagine that the largest technology providers spend the most money with analysts. Apple, Cisco, IBM, Microsoft, Oracle, Palo Alto Networks, and their peers spend a great deal with analyst firms, which is not surprising considering the vast array of products and services they provide. They also share another common characteristic: every single one of them has been placed in the niche (lower left) and challenger (upper left) quadrants in multiple markets, despite spending millions, and long before some of them became massive publicly traded behemoths.

On Magic Quadrants

The Gartner MQ, love it or hate it, is probably the most read output of any research product that Gartner, or any analyst firm, provides. The quadrant has two dimensions: completeness of vision on the horizontal x-axis and ability to execute on the vertical y-axis. Completeness of vision focuses on aspects of the product and marketing and positioning efforts. Ability to execute focuses on sales execution, company size, and resources.

For those unfamiliar with the MQ process, here is an overview:

1. An analyst has a defined market they would like to quadrant. If an MQ does not already exist, there is a required review process to develop a new one.
2. Analysts define the inclusion and evaluation criteria for each MQ, then adjust the weighting for each area to better address market dynamics over time.
3. Analysts send out survey questions and spreadsheets to vendors that may be included, which requires vendor-provided references.
4. Analysts parse through vendor data, perform reference checks, review inquiries and conversations, and discuss with other analysts.
5. Analysts plug the data into a spreadsheet template, which automagically calculates positioning.
6. Analysts spend the next several weeks debating "facts" with vendors. Most are not happy with either their position or the relative position of their competitors. These couple of weeks were without a doubt the worst part of the job.
7. The MQ is published. Some use it properly, as one input to guide opinion. Others, who are too lazy to do the work, look only at the leaders as the starting list of vendors they will evaluate.

Lather, rinse, and repeat annually.

What the MQ Doesn't Provide

Gartner does not maintain a testing facility, nor does it work with one. The MQ does not provide an evaluation of technology based on internal testing or hands-on competitive analysis. In some cases, the only interaction an analyst has had with a product is a 30 to 60 minute web demo.

Gartner looks at macro trends in markets, not what is important to your specific company. These macro trends and criteria may or may not meet your needs. What is good for company A can be disastrous for company B. Think about your digital transformation, cloud security, or AI initiatives.

Gartner does not dynamically update MQs between publication dates. The MQ is a point-in-time snapshot. It does not reflect the pace of technology change from one MQ to the next until the manual update the following year.

The MQ is not based on facts. It is based on opinion, sometimes highly tuned and well-supported opinion, but still opinion, and it relies entirely on manual input. It is prone to error and misinterpretation.

The MQ is only one piece of the evaluation and procurement process. Its main purpose is to help guide and validate position. Anyone who uses it as a sole-source vehicle is most likely going to experience some time in the trough of disillusionment.

In terms of influencing the influencers, there is no doubt that money spent with an analyst firm provides benefits, such as access to their feelings and opinions on market dynamics. Aligning with their thinking provides better positioning. The more time you spend with analysts, the better you can articulate your benefits, strategy, wins, and company direction, and the more comfortable they become with your technology.

On the Analyst "Technology Tax"

Analyst firms are a technology tax. For many, it is a cost of doing business in the IT sector. Regardless of what anyone says, the more money you spend with analysts, the better your benefits. That isn't necessarily a bad thing, but at many companies the analyst relations budget is managed through marketing, and AR people sit in marketing.

Many vendors, regardless of what they say, are more interested in positive research mentions and MQ position than in valuable insight. This is unfortunate, because the truth about Gartner is that they have a massive infrastructure, hundreds of talented analysts, and the global reach to support the level of insight organizations need to navigate tactical and strategic IT initiatives in an increasingly complex, global, and sophisticated environment.

For Gartner's part, their legal defense of their positions is quite simple:

"Expressions of opinion on matters of public concern are entitled to the protections of the First Amendment and Article I, Section 2 of the California Constitution," according to Gartner's reply in litigation over its ratings. "Under the First Amendment, a protected 'opinion' is one that does not assert or imply facts capable of being proven true or false. As protected speech, expressions of such opinions on matters of public concern are nonactionable."

In the end, the Magic Quadrant isn't magic. It's a snapshot of collective opinions.

Use it as a guide, not gospel.