There are significant public disagreements between analyst firms and reality about the accuracy of their data. Take concerns about Gartner's sizing of the server markets and IDC's analyst goings and comings and the associated conflicts between analysts on market sizes and forecasts. Market research is still very important to investors and with vendors for forecasting & planning, but savvy executives have begun to move away from "betting the farm" on what these researchers say and are relying more on developing their own internal methodologies and resources to do their forecasting.
Basically, the way that market research firms produce their numbers is to ask vendors for data on shipments, revenues, etc. and feed these into their own proprietary market models.
These models, many of which have been in use in their basic forms for decades, are then used to crank out their estimates as to market size, growth, share, etc. The researchers then usually validate these numbers using public information on vendor revenues, etc. and primary research with buyers. In practice though, not all numbers come from the vendors or public and primary sources. Many researchers, especially those who are working with mature, long-lived models will estimate inputs to their models based on their observation of the marketplace and run it by the vendors for validation.
Though expedient, this practice will almost certainly introduce errors over the long run.
Market research is only as good (or bad) as the model used to generate it. These models are the source of frequent debate amongst vendors, researchers and their firms. Vendors want them to incorporate elements that work to their advantage and the researchers and their firms opt for consistency and will vehemently protect "their" model from internal and external attempts to change it. The single most contentious issue is how the raw data is collected from the vendors and then validated by the researchers. In some firms, the same relatively senior researchers collect and analyze the data and then author the report, in others, junior staffers collect, while the authors analyze and report. Using senior researchers to do everything is expensive and is difficult to oversee or police, while using junior, inexperienced staff for data collection increases the chance for mistakes and inconsistencies. Some of the criticisms the Market Researchers face are:
- Not giving vendors enough time to respond to requests for data
- Lack of coverage of emerging markets
- Not doing enough primary research and thereby putting
- Too much stock in numbers given by vendors
- Double counting revenues - Some vendors report the same revenues in multiple different categories and some don't. This leads to bad input and no matter how good the model, the results won't reflect reality.
The practical reality is that every researcher, at every firm has weaknesses in their models. Most of the time, these weaknesses have little or no bearing on the outcome of the report, but sometimes they have a huge impact on the viability and validity of their findings. In general the key to good models is to develop ones that minimizing the guessing that goes into the numbers they produce. To do this, most firms and researchers strive to build models that have the following common characteristics:
- Detailed description of marketplace and its exact definition and segmentation
- Consistency of inputs across multiple disparate vendors.
- Currency of model in an ever-changing marketplace.
- Relevancy in hard to define marketplaces.
- Accuracy in verifying and justifying inputs from multiple vendors.
- Uniformity in handling cases where little or no information is offered by some vendors while others fully comply.
Accountability of numbers used as inputs through full disclosure of sources, assumptions, etc. includes disclosure of which numbers come from audited financials, and which are simply provided by vendors with no back up or validation.
Be aware that vendors play games with market share numbers. It should be no mystery that every technology vendor wants to present their products in the most positive way possible and it also should not be a mystery that sometimes they get a little carried away. Though self-defeating in many ways, vendors play with numbers to make themselves more attractive, viable, stable and competitive than they really are. In the market research world, this overzealous behavior usually results in artificially high or low numbers (depending on which is more advantageous) being reported to the researchers. This is exactly why good market researchers rely on their models and not the vendors' numbers when reporting share, etc., but sometimes some of these "bad" numbers creep through. Market researchers are expected to catch these numbers, but it is impossible to catch them all. More on this topic in coming posts...
