I won’t use this weekend reading to debate the merits of the futures market for terrorist attacks. Although I think the idea is innovative and useful, I understand reasonable people disagree on this at a very fundamental level. I copied several articles exploring the concept of idea markets, such as the one killed by bad press this week, and hope you’ll print them to read at your leisure. Even if you disagree with the concept of a futures market for terrorism, you may learn how the mechanism could help answer difficult questions and resolve competition for scarce resources.
(Brief literary sidebar:
If you are as fascinated by this topic as I am, I encourage you to read The Shockwave Rider by John Brunner.
Published more than 25 years ago, the author predicted the emergence of
the Internet; in his vision, it is essentially a futuristic Delphic
oracle to which people pose questions and receive answers when a
statistically significant number of respondents have responded. Basically, a global Dear Abby.)
The idea of a futures market for terrorism certainly starts from a basic disadvantage: it’s pretty easy for critics to call it “betting on terrorism,” and pretty hard for proponents to explain that’s not what it is. Because, that’s what it is.
A futures market to estimate the likelihood of a terrorist attack would capitalize on a very simple idea: all of us are smarter than any one of us. (Some people call this democracy.) In The Shockwave Rider, John Brunner illustrates this concept by noting that if you ask a random person what year the US Civil War started, he’d probably get the answer wrong. But if you ask 10 people and average their answers, you’ll get close to the correct answer. If you ask 100 people and average their answers, you’ll get even closer; if you average 1,000 responses, you’ll get a very accurate answer. (1861, by the way.)
The political luminaries who attacked this plan willfully ignore the hundreds of millions of people around the world who use or benefit from exactly the same mechanism every day. Futures markets allow investment managers, farmers, orange juice drinkers, wealthy individuals, and union pension funds to hedge their bets (another, briefer literary sidebar: for a fascinating story about how hedge funds work, read about the collapse of Long Term Capital Management in When Genius Failed by Roger Lowenstein) and protect themselves from the risks inherent in any marketplace. I suspect at least a few of the US Congress’ many millionaires are direct investors in hedge funds that use exactly the same instruments they railed against this week to preserve their wealth. Some of them may even wager on horses, another example of this financial instrument that seems to work well enough that few in Congress see a need to ban it.
And yet, primarily to achieve political gains, some of our elected public servants called out an extremely small budget item (about $8 million out of more than a trillion and a half dollars, roughly analogous to the value of Warren Buffet’s pocket lint) and killed it with rhetoric – even though it is a completely legitimate strategy for deriving valuable insights from experts who do not typically gather in any one location for any amount of time.
To make my point even more bluntly: to embarrass other political leaders or advance their own careers, or both, some public servants destroyed a valuable anti-terrorism tool. Even more interesting is that several of the publications that participated in the feeding frenzy that killed the program today published thoughtful commentary on what a great idea it would have been (see articles following my signature).
There are too many valuable lessons to discuss in depth, so let me offer just a few below:
§ Be aware of the motivations of the person you’re listening to.
§ The instant news cycle of our Internet era does not lend itself to thoughtful consideration of complex ideas.
§ We discover the merit in some ideas that sound outrageous only when it’s too late. Haste can, indeed, make waste.
§ Negative media attention is almost impossible to overcome with equivalent positive media attention – you need many times more positive exposure, and the only way to get that is to pay for it (advertising).
§ The best defense is a good offense: if you’re advocating a potentially controversial idea, get in front of the negative attention with respected third parties who will campaign for the idea.
§ Prepare to explain the benefits of your idea in language as simple and compelling as your opponents will use to attack it – even if you don’t think your idea will get attacked.
§ Be wary of anyone who claims that a mechanism to solicit the ideas and opinions of many people is a bad idea. He may be smarter than any one person, but he’s not smarter than every one.
______________________
You may manage
Weekend Readings by visiting www.analects-ink.com,
or clicking the commands below:
p
Subscribe
to Weekend
p
Unsubscribe to Weekend
p
Add a friend or colleague to the list of Weekend Reading recipients
To Learn What People Want,
Trade 'Idea Stocks'
The New York Times
By BARNABY J. FEDER
Except for its unusually
informal setting, the exercise resembled the kind of simulated trading of
stocks or bonds that is routine in the finance curriculum at many business
schools.
But this was a marketing
class conducted by Ely Dahan, a professor at M.I.T.'s Sloan School of Business.
The students were not trading securities but the traits of various goods and
services, like automobiles and ski resorts. The exercise was part of an effort
by Professor Dahan and colleagues at M.I.T. to prove that simulated trading
could answer that most basic of all product development questions: What do
consumers want?
One session here traded “stocks”
representing the Pontiac Aztek, Acura MDX and other rivals in the new market
segment that melds sport utility vehicles, minivans and cars. The students also
traded the characteristics of ski resorts — whether skiers could drive to the
resort or had to fly to it, for example, and whether it had crowded slopes.
Other sessions focused on designs for laptop-computer carrying cases and
variations on hand-held digital assistants.
In as little as 20 minutes,
the students arrived at fairly stable rankings of the relative worth of what
they were trading. More intriguingly, as was driven home by interviews with
skiers nearby at
“It's amazing that you can
get effective results in areas where people don't know that much,” said Jay
Livens, an M.I.T. student, after Professor Dahan showed the outcomes of their
vehicle trading.
The closing prices were
generally consistent both with consumer sentiments culled from more traditional
market research and with previous trading experiments at M.I.T. In just
minutes, for example, the trading echoed the disappointment with the Aztek and
enthusiasm for the MDX demonstrated by car buyers over the last year. “General
Motors (news/quote) might have been able to save itself a lot of pain if it had
run trading like this a couple of years ago,” Professor Dahan said.
To participants, market
research had never seemed so cool.
“I feel like we are at the
genesis of a potentially great idea,” said Kristina Larson, a first-year
student.
In real markets, the value
that buyers place on goods — reflected in prices — is established as a
byproduct of the exchange of goods. In Professor Dahan's markets, buyers'
opinions are the whole point of the exercise. If the operations of regular
markets are like fishing for food, then Professor Dahan's work is like fishing
to gather data about pollution levels in the ocean.
Professor Dahan's research
was inspired by the successes of several Internet-based trading exchanges that
have been called “decision markets” by Robin Hanson, an economics professor at
Another virtual market, the
Hollywood Stock Exchange (www.hsx.com), has become one of the movie industry's
most trusted predictors of how new films will perform at the box office. The
exchange has more than 850,000 registered members. After completing the free
registration form, traders get two million “
If a film is trading based on
expectations that it will gross $80 million in its first four weeks, investors
who think it will reach $100 million buy shares. Those who anticipate weaker
results sell.
The Hollywood Stock Exchange,
owned by a British subsidiary of Cantor Fitzgerald, the Wall Street firm, sells
data about the trading to entertainment companies.
Several companies have
experimented internally with similar trading programs. The most successful
experiment, Professor Hanson said, was at Hewlett-Packard (news/quote) in 1997.
Predictions of Unix workstation sales, derived from trading shares in different
projections by Hewlett-Packard sales managers, repeatedly proved more accurate
than official estimates by the unit's marketing group. The trading apparently
filtered out the tendency of sales executives to be optimistic in their
projections to superiors.
Professor Dahan is trying to
forecast the appeal of competing packages of product features, some of which
may never exist other than as ideas. Such trading cannot be measured against a
real outcome, as in an election. But the trading of product attributes may help
a business pick the most popular combinations of features early in the design
process.
Trading contests may also
help a company find consumers who are especially knowledgeable about its
business — or one in which the company wants to expand. Such consumers, called
lead users, are prized in the product development world.
Professor Dahan's main goal
is to permit product developers to do research that is fast, easily modified
and ultimately much more thorough. The technique now viewed as the best, a
polling practice called conjoint analysis, was developed in the late 1960's.
Marriott used it to design its Courtyard by Marriott chain (the slogan:
designed by business travelers for business travelers).
But conjoint analysis for
more than a few attributes is unwieldy because the possible combinations
mushroom quickly to unmanageable numbers. That is a big shortcoming in a world
where many products have more than 100 distinguishable features that might
influence consumers, Professor Dahan said.
In theory, trading can
establish simultaneous values for scores or even hundreds of product attributes
— just as it does each day for thousands of publicly listed stocks. That
assumes that large groups of people are willing to trade for fun or for modest
rewards.
Professor Dahan, 44, had his
attention drawn to stock trading by two M.I.T. colleagues, Tomaso Poggio, a
neuroscientist, and Andrew Lo, a finance expert. They suggested that he apply
related research in which they had been involved to M.I.T.'s Virtual Customer
Initiative, which seeks ways to use the Internet and other tools to speed
product development. Despite the intriguing results, Professor Dahan remains a
cautious champion.
“I think businesses will be
trading these stocks like crazy in the future, but this is just a complement to
other techniques,” he said. “It would be heresy among marketing academics to
say we no longer need to measure individual preferences. Trading doesn't
reflect them.”
Professor Dahan said research
suggested that in 38 percent of the trades, sellers like the specific products
or features more than the buyer. Presumably, the sellers unload the stock
because they think the price will fall.
How often game traders will
behave as normal financial investors, however, is one of many areas about which
little is known. Other questions involve what rewards are needed to keep enough
traders playing and how to determine winners.
Professor Hanson says he
thinks the biggest hurdles to the spread of research- driven trading in
business may be cultural. Companies continuously make investments based on
assumptions about the future that are, in essence, bets. But stock trading is
so obviously speculative that many of them may be reluctant to be seen using a
form of it as a data source for important decisions.
Slate’s
explainer
What
Weird Futures Can You Buy?
A guide to online prediction markets.
By Brendan
Posted
Article URL: http://slate.msn.com/id/2086316/
The
Pentagon has scrapped its plans to operate the Policy Analysis Market,
which would have allowed online traders to wager on the likelihood of future
terrorist attacks. Aside from commodities like pork
bellies, what sorts of futures can wannabe brokers buy and sell?
A
whole galaxy, thanks to the proliferation of Internet-based prediction markets, also known as decision markets. These
online bazaars allow punters to plunk down money, real or imagined, on the
potential of films, ideas, or the
The
granddad of online prediction markets is the Iowa Electronic Markets,
which was started in 1988 to forecast the fortunes of presidential candidates;
the market now covers the Fed's interest rate decisions as well. IEM
participants can use real money, with starting accounts capped at $500. The
market is regulated by the Commodity Futures Trading Commission. (Those with more
Teutonic tastes may prefer Wahl$treet,
a market for futures on German politics.)
Sports
fans can take advantage of their nightly SportsCenter viewings on the Athletic Stock
Exchange. Not much money to be made here, as the entry fee is $10 and the
athletes trade for just a few pennies. But sports aficionados may enjoy
grabbing a share of Bobby Labonte (Ticker symbol: LBNT) or Hideki Matsui (MATS) and seeing where fortune takes them.
There are also veritable athletic indexes, jock versions of the S&P 500,
like the Quarterback Pool, which aggregates the fortunes of all
National Football League signal callers.
If
news rather than sports is your informational pastime of choice, check out the
current-events section at TradeSports.com,
an Ireland-based betting service that made a name for itself when its market
more or less predicted the date of Saddam Hussein's ouster. A future on a
Saddam arrest this month just went up three points, in light of the capture of one of his chief bodyguards.
Not
all prediction markets require that you risk actual cash. By far the most
famous is the Hollywood Stock
Exchange, which permits trade in both “MovieStocks” (pegged to the
box-office success of upcoming releases) and “StarBonds” (tied to the future
fabulousness of silver screeners), all in fictional money. Today, for example,
the HSX is bearish on the soon-to-be-released S.W.A.T., down $5.67 on
a volume of nearly 7 million shares. Angelina Jolie, whose Tomb Raider
sequel disappointed over the weekend, is also having a rough Tuesday; her
StarBond is down a buck.
The Foresight Exchange Prediction
Market allows traders to bet on the likelihood of a range of events, from
the resignation of Donald Rumsfeld by October to a devastating
earthquake in the western
If
you ever had trouble making sense of the blogosphere, Blogshares may help
separate the wheat from the chaff. No money's exchanged on this market—though
there is a $500 contest taking place right now—but it does give bloggers
bragging rights as to the popularity of their daily thoughts among Web surfers.
Curiously, Slate resident blog “Kausfiles“ doesn't appear
to be listed on the exchange. Explainer eagerly awaits the IPO.
Bonus Explainer: At
first glance, the trading of weather
derivatives on the Chicago Mercantile Exchange may seem like a prediction
market of sorts—after all, weather forecasting seems ripe for such wagering.
But these investors are using the market to hedge their bets, not to get rich
off hunches about upcoming weather. For example, an energy company that craves
a long, hot summer in a certain city—the better for folks to run their air
conditioners incessantly—will manage risk by purchasing futures that predict a
cooler than anticipated summer. That way, they won't get burned so badly if
Mother Nature doesn't comply. Since these investors are interested parties
whose betting strategies are designed solely to manage risk, the weather
derivatives market isn't a classic predictions market.
Brendan I. Koerner is a fellow at the New
A Good Idea With Bad Press
The New York Times,
By HAL R. VARIAN
http://www.nytimes.com/2003/07/31/business/31SCEN.html
THE Pentagon-sponsored
futures market in terrorism indicators was announced and squashed in all of two
days. Too bad. It was a good idea, killed by terrible public relations.
Consider the problem that
intelligence agencies face. They have to consider the likelihood of various
destabilizing events — like assassinations or terrorist attacks. That's a major
part of their job; there's no way around it.
In assessing such likelihoods
they draw on expert opinion, but they also make considerable use of market data
— oil price futures, currency exchange rates and the like — to see how market
traders assess risks of political instability.
Markets do an awfully good
job of forecasting many events and trends. The futures price for oil is about
the best predictor available for this critical commodity, and it is widely used
for forecasting by both political and economic analysts.
The question is whether
speculative markets would work well for other policy-relevant events.
There is lots of evidence
that they do. The Iowa Electronic Markets, www.biz.uiowa .edu/iem/, has been
predicting election results for 12 years using a system very much like the one
that the Defense Advanced Research Projects Agency at the Pentagon proposed.
One of the markets the
As it turns out, these
political stock markets provided somewhat better forecasts than polls right
before the election — and they provide much better (and less volatile)
forecasts several months before the elections. Thus, markets do best exactly
where the public opinion polls and expert opinion polls are weakest.
This is not an isolated
example. Similar markets have been organized to predict shifts in Federal
Reserve monetary policy, the outcome of political conventions and sales of
consumer products. The results are that markets typically perform at least as
well, and generally better, than feasible alternatives, and they are much
cheaper to organize.
An online betting site based
in
There is good reason to
believe that a market set up to forecast the sort of political instability that
leads to terrorism might work well, too. At least, there is enough reason to
warrant an experiment, given the high payoff to having better forecasts of
these events.
This is why the Pentagon
thought it was important to finance research in this area.
The objections raised by
politicians and opinion writers were generally based on misunderstandings of
what was actually proposed.
Take the claim that
terrorists could speculate in their own future activities and reap a windfall.
First, the maximum gain from a trade was restricted to less than $100. This
would not be of much help to terrorists' fund-raising activities.
Second, the market would have
been monitored for evidence of insider trading. If someone managed to
circumvent the trading limits and made a significant amount of money
speculating on a terrorist attack, you could be sure the C.I.A. would be right
behind the I.R.S. in investigating.
If the federal authorities
were alert enough to find enough evidence to indict Martha Stewart, why would
one think that Osama bin Laden would be able to avoid detection?
There were also objections
that the auctions were immoral.
Senator Byron L. Dorgan, a
Democrat of North Dakota, asked what would happen if another country set up a
betting parlor where people wagered on the assassination of an American
political figure.
I'm sure he is right that
there would be public outrage. But let's turn the question around: If such a
market were put in place, should the Secret Service monitor it? If there were
an assassination attempt, should the authorities look for suspicious prior
trading activity? And the most important question of all: If the market
indicated that the probability of an assassination attempt went up, should the
target take extra care? If you were a potential target, wouldn't you want the
best possible forecasts of possible attempts on your life? I would.
I would also prefer that such
forecasts not be quite so public. A private market in the probability of
specific assassination attempts — say within an expert community like the
C.I.A. — could make sense, assuming, of course, that it provided useful
information.
Would Mr. Dorgan object to
such an internal market as a way of aggregating expert opinion? I hope not.
In any event, assassination
futures were not among the planned securities, contrary to most press reports.
The market design allowed traders to propose securities in various events, and
the Policy Analysis Market Web site speculated that some traders might propose
to add securities in assassinations. That off-the-cuff speculative remark had
fatal consequences, alas.
The securities that were an
essential part of the auction design were indexes of political, economic and
military activity. The most useful such market would probably have been a
market for futures in a “political instability index,” a weighted average of
various political indicators, like the number of mass demonstrations,
unemployment levels, arrests — and, yes, assassination attempts. (Type “political
instability index” into Google for some examples.)
If the markets had been
pitched in this way, I doubt that there would have been many objections raised.
But instead politicians,
reporters and editorial writers mistakenly jumped onto “assassination and
terrorist attack futures” as a fundamental part of the market design.
Given the public outcry, it
seems clear that there will not be a public market in assassination futures
anytime soon. But there is no reason not to have a futures market in political,
social and economic indicators, which is what the Pentagon's project was
actually about.
We desperately need better ways
to forecast political instability, and the Policy Analysis Market had
significant promise. It's sad to see poor public relations torpedo a
potentially important tool for intelligence analysis.
The Furor Over 'Terrorism Futures'
The
Thursday,
By Justin Wolfers and Eric Zitzewitz
http://www.washingtonpost.com/wp-dyn/articles/A5696-2003Jul30.html
Betting
on human lives seems ethically questionable. Yet if it helps save lives, surely
the moral questions are mitigated. Not so, according to those in Congress (and
elsewhere) who created such a furor this week over a planned Pentagon program
to project geopolitical risks that the program was quickly shut down. The plan
was to use markets to “price” such risks, and it was quickly dubbed a “terrorism
futures market.” Unfortunately, in hastily ending this program, the government
may be closing the door on an important source of information and a promising
avenue for research.
The
idea was simple: By creating a market in which people can buy and sell
contracts that pay $100 if certain political events occur in the
Financial
markets are incredibly powerful aggregators of information, and are often
better predictors than traditional methods. The examples are numerous. The
futures market in orange juice concentrate is a better predictor of
The
reason markets work so well is that they reflect our collective wisdom. And
your opinion will be reflected only to the extent that you are willing to put
your money where your mouth is.
While
the joke about military intelligence being an oxymoron is an old one, it bears
repeating here. It's no coincidence that we don't have the same doubts about
financial markets. Recent events have underscored the difficulty of aggregating
information from lower levels of the intelligence bureaucracy. Imagine if a “Niger
Uranium sale” contract had been trading in January; our guess is that this
would have been close to valueless, reflecting the hard intelligence available
at the time that such sales never occurred.
Political
critics contend that betting on political disruptions not only is distasteful
to many but that it also provides an incentive to create political strife,
because an assassin might be able to bet that political strife is likely and
then go out and cause the strife himself.
But
those opportunities and incentives to make trouble already exist. The stock
market fell 2.8 percent following President Kennedy's assassination; it fell
4.9 percent following the attacks on the
Terrorists
determined to profit from their actions can easily buy derivatives to cash in
on their actions. Of course they will leave behind a paper trail. But more to
the point, all that these prediction markets will do is make the information
content of such trading more transparent for policymakers.
A
more serious concern raised is that terrorist organizations might trade in the
market to create disinformation. But terrorist organizations already have lots
of channels for disinformation, and the profit motive in these markets provides
incentives for those who know the truth to correct such disinformation. The one
thing that economists actually agree on, is that people do respond to such
incentives.
Legislatures
are not the best institutions for directing research, economic or otherwise.
The banished Pentagon program didn't deserve this sort of treatment.
The
writers are assistant professors of economics at
Wired, 02:00 AM Jul. 30, 2003 PT
http://www.wired.com/news/politics/0,1283,59818,00.html
Critics blasted policy-makers Tuesday for dropping
a controversial plan to create a futures market to help predict terrorist
strikes.
Legislators like Sen. Ron Wyden (D-Ore.) may have found the Pentagon's
Policy Analysis Market, or PAM, “grotesque.” But proponents of “idea markets”
say PAM's quicksilver cancellation will rob the country's intelligence agencies
of a tool with a strong history of accurately predicting future events.
It's a decision that's “pure political,” said Bill
Adkins, with Neoteric Technologies,
which has a Defense Advanced Research Projects Agency, or Darpa, grant to
design markets trading on the future of hybrid electric vehicles and the spread
of SARS.
Senators expressed concerns when they discovered
the PAM project,
an effort to speculate on possible events in the Middle East -- like the
overthrow of Jordan's monarchy or the assassination of Palestinian leader
Yasser Arafat -- as if they were stocks. The higher the price, the theory goes,
the more likely the incident.
“The idea of a federal betting parlor on
atrocities and terrorism is ridiculous and it's grotesque,” fumed Wyden.
The fact that PAM came from Darpa's Information Awareness Office, the group of
minds behind the notoriously invasive Terrorism Information Awareness
database project, made the trading floor effort seem even more distasteful.
A day after Wyden and Sen. Byron Dorgan (D-N.D.)
held a press conference blasting the program, the Pentagon agreed to drop PAM.
But supporters of the project point out that
gathering intelligence is often a messy business, with payoffs to unsavory
characters and the elimination of potential adversaries. The futures market,
ugly as it may sound, doesn't involve any of those moral compromises, said Robin Hanson, one of the earlier promoters of
the concept of trading floors for ideas and a PAM project contributor. It's
just a way of capturing people's collective wisdom.
“Among the many things we do for intelligence,
this is one of the least reprehensible,” Hanson said. “Paying people to tell us
about bad things. That's intrinsic to the intelligence process.”
And a trading floor could be more effective than
paying off a snitch.
Projects similar to PAM, like the Iowa Electronic Markets, which
speculate on election results, have been surprisingly reliable indicators of
what's going to happen next.
The Iowa market hasn't been perfect -- it forecast
a Democrat-controlled Senate in 2002. But over the course of 14 elections, the
Iowa Electronic Markets' stock prices were on average a half of a percentage
point closer to the results of the actual political races than the final polls
were.
The price of orange juice futures has even been
shown to accurately predict the weather, noted David
Pennock, a senior research scientist at Overture Services who has done
extensive surveys on the reliability of such markets.
Traders on the Hollywood
Stock Exchange last year correctly picked 35 of the 40 Oscar nominees in
the eight biggest categories, according to The New Yorker
magazine.
“Market mechanisms are more accurate than asking
people their opinions because they're putting their money or reputation on the
line,” said Ken Killitz of the Foresight
Exchange, which speculates on everything from the future of human cloning
to the possibility that Roman Catholic priests will be allowed to marry. “It
gives people an incentive to reveal what they know.”
Markets can be wrong about events, or companies,
of course. Just look at the muscle-bound stock prices of flimsy firms during
the dot-com boom.
But exchanges “tend to predict events really well
when no one person knows the answer -- when information is distributed among
many people with different knowledge bases,” said Joyce Berg, a University of
Iowa professor who helped organize the political trading floors. “Markets have
been shown to be really good at aggregating that information.”
Markets also bring together people with
information about a particular subject in a way blue-ribbon panels of experts
can't, added Hanson.
“You get people that know things about a subject,
but don't have the credentials to say so,” he said. “You get people who live in
these areas (of the Middle East).”
There's also “less of an ability to spin” in
markets than in policy debates, Hanson noted. “So you get what people actually
think, not what they say.”
The current flap over the justifications for the
Iraq war shows the need for such spin-less interpretations.
Members of the Senate are concerned, however, that
by creating a financial market for Middle East events, a terrorist could
basically bet on himself, and profit from his strikes.
Senate Minority Leader Tom Daschle (D-S.D.)
expressed the feelings of several colleagues when he called PAM “an incentive
actually to commit acts of terrorism.”
But Daschle has it all wrong, according to
Pennock, the market academic.
“The very fact of the terrorist doing that
(investing money in an attack) would reveal his hand,” he said. Prices would
rise as the terrorist invested his cash, and that would tip leaders off to the
potential for a strike.
“The market would know something is going to
happen that people never would have known otherwise,” Pennock added.
There are problems with the PAM system, Pennock
conceded. The market only allows for speculating on preconceived ideas.
Innovative terror plots wouldn't make it to the PAM trading floor before they
were put into action. It seems unlikely, therefore, that PAM traders would have
foreseen al Qaeda's use of airplanes as missiles before the terror attacks of
Sept. 11, 2001.
Allowing anonymous trades on incidents is a bad
idea, Pennock said. There should be audit trails, like in the real stock
markets, in order to catch terrorists who might be foolish enough to speculate
on their own exploits.
These problems are fixable, he was quick to add.
“I don't think the idea should be dismissed out of
hand,” Pennock said.
Too late. By early Tuesday, the Senate already had
reached its conclusion.