Did the stimulus create jobs?
Daniel Wilson of the San Francisco Federal Reserve Bank has just released what might be the first real evidence-based effort to resolve this question. One apparent problem with drawing inferences from the experience of the past couple years is that we have only one experiment to look at. But Wilson points out that in some sense, we have 50 separate experiments because stimulus spending differed substantially across states. You can potentially learn a lot from 50 experiments.
Unfortunately, they’re not controlled experiments, because stimulus funds were not allocated randomly. States with particularly weak economies probably got more Medicaid funds. States with bloated and inefficient bureaucracies might have been slow to complete necessary paperwork and hence slow to receive funds. If those weak economies or shamblng bureaucrats also had an effect on job growth, then the experiments are not clean.
But fortunately there are substantial components of the funds that were distributed according to objective formulas (demographics, number of highway miles, and so forth). Wilson makes competent use of these components, together with standard econometric techniques, to zero in on the subset of stimulus spending that can be considered effectively random. Now that he’s got his fifty more-or-less controlled experiments, he also controls for other confounding variables that could plausibly affect state-by-state economic growth. All of which is the right way to do this.
As Wilson points out, his appears to be the first attempt along these lines. Previous studies fall into two broad categories. First, there are the model-based approaches that forecast employment both with and without the stimulus (so that they’re testing not the observed effects of the stimulus, but its forecasted effects). Second, there are the survey-based approaches that require recipients of stimulus money to report the number of jobs they created or saved. Aside from all the obvious ways in such reports are likely to be inaccurate, they account only for the first-round effects of the stimulus, ignoring any second-round jobs created, ignoring effects on consumer spending, and ignoring God knows what else.
Wilson’s bottom lines:
- The number of jobs created or saved by the spending is about 2.0 million as of March, but drops to near zero as of August.
- The effects varied enormously among sectors. The biggest impact was in construction, where we saw a 23% increase in employment (as of August 2010) relative to what it would have been without the stimulus.
- It mattered a lot how the money was spent. Spending on infrastructure and other general purposes had a large positive impact
- Aid to state government to support Medicaid may have actually reduced state and local government employment. This seems a little surprising. It might be driven by the fact that these funds come with strings attached, requiring state and local governments to meet so many burdens that they’re led to cut spending and employment in non-Medicaid areas.
None of this will give unmitigated aid and comfort to ideologues of any stripe. And none of it is definitive, because no single study is ever definitive. But it’s a welcome start toward figuring out what really happened over the past couple of years.
I will add only that “Did the stimulus create jobs?” is not at all the same question as “Was the stimulus a good idea?”. But it’s an important question in its own right, and I’m glad someone’s finally trying to answer it in a sensible way.
A very interesting post, which nicely shows how an economic model lurks behind any econometric result. Wait, weren’t you and I just debating that on the death penalty thread?
No small part of the genius of a practicing social scientist is the capacity to identify approximations of controlled experiments arising “in the wild.” Cool beans!
Whereas states with emaciated and inefficient bureaucracies would have been faster?
Excellent that this problem can be approached in this way. I don’t quite understand how it is controlled for the counterfactual. The author says:
“To control for the counterfactual – what would have happened without the stimulus – I include in the regression model any variables that (1) are likely to be predictive of subsequent employment growth, (2) could potentially be correlated with the instruments for stimulus spending, and (3)were known at the time of ARRA passage (so arguably exogenous with respect to subsequent
economic outcomes).”
This paper covers only federal spending. Earlier we saw Krugman quite keen to point out that much of the increase in federal spending was matched by a drop in local government spending. I presume this is controlled for in the counterfactual.
Fig 30 and 31 shows this paper following Monaceti et al , then jumping to Ramey. This paper is the only one to show this dramatic downturn. What factors of this paper lead to this drop? This could be where much of interest lies.
Ken B: Have you actually *read* any of the death penalty papers whose content you’re so certain of?
@Steve
Fair question. Did you actually cite any?
I have read stuff in the past; absent a link here I cannot tell if it is one you are referring to. But let’s stipulate not. So what? here is your para from another thread:
“Mike H got this number from my book, The Armchair Economist, where I give it as a more-or-less-consensus estimate of the deterrent effect of executing a murderer. So it does not mean this murderer will kill 8 people; it means that the failure to execute will weaken deterrence sufficiently so that 8 additional murders will be committed (presumably by other murderers).”
A number like that is not a simple measurement. It is an inference. It depends on a theory of what variables are relevant — and hence will be controlled for. All such numbers od unless you can do a controlled experiment you cannot infer causality otherwise.
Ken B: We both agree that any empirical measurement is contingent on certain assumptions. What we disagree about is the *type* of assumptions that went into these particular estimates, and in particular whether there is any assumption about incentive effects. Since I’ve read the papers and you haven’t (you might start with Isaac Ehrlich) I have no idea how you formed your opinion or why you persist in it.
Ken B:
Here’s an example:
Researcher A has a story to tell about how cigarettes cause lung damage and lung damage leads to cancer. He tests this theory by comparing the correlations between cigarettes and lung damage, between lung damage and cancer, and between cigarettes and cancer.
Researcher B notes that in states whenever cigarette taxes go up, cancer rates go down. As best he can, he controls for the possibility of reverse causation, etc. He concludes that cigarettes cause cancer.
Each researcher has made some assumptions — Researcher A made some assumptions about the *mechanism* by which cigarettes cause cancer (we call this “structural modeling”) while Researcher B made some assumptions about the statistical independence of cigarette tax hikes, etc (we call this “instrumenting”).
It is simply NOT TRUE that researcher B made any assumptions about the mechanism by which cigarettes cause cancer. Likewise, a death penalty study need not make any assumptions about the mechanism by which capital punishment leads to fewer murders. In particular, such a study need make no assumptions about incentive effects. And in fact, most of these studies don’t, and your claiming otherwise (from a posture of pure ignorance) can’t change that.
Here’s a general rule I hold: If I need a statistician to tell me if something worked or not, it didn’t work.
In short, we spent roughly $400,000 per job that was (briefly) “created or saved”? That doesn’t sound like a winning move…
(814 billion spent / 2 million jobs = 407k per job)
Let’s review. One poster noted that the figure 8 murders averted per execution was the result of an economic theory. You said, no it’s the result of empirical analysis. I noted that it is ineluctably both. You objected, and then added the provision about “structural” models. Is this a fair recapitulation?
In your example B I would say that smoking is the (presumed) causal link between cigarrette tax hikes and falling cancer rates. The causal link would lead to the prediction that a tax hike should reduce cancer. The study B carries out is really an attempt to test that hypothesis. But it would fall apart if we found that there was no signifigant effect on cig sales would it not? That is a bit of theory lurking in the calculations, which is an example of why these things are never simple empirical facts. But my point is this. B is not just corelating taxes and cancers; he is testing a model.
Ehrlich himself I see from a quick google of abstracts does not claim to be theory free. Some abstracts of his papers:
Abstract:
We argue that the deterrent effects of the certainty and severity of punishments on murder depend on the status of the death penalty. Legalizing the death penalty not only adds capital punishment as a deterrent but also increases the marginal productivity of other deterrence measures in reducing murder rates. From the econometric standpoint, this suggests that the structure of the murder supply function depends on the status of the death penalty, which is in itself endogenous. Consistent estimation of the deterrent effects of capital punishment and other preventive measures requires taking into account the endogenous nature of the status of the death penalty. Using switching regression techniques and a well-known US data set, we test the implications of the model. The empirical results are consistent with the theoretical predictions of the model and robust to alternative specifications. They also lend strong support to the deterrence hypothesis.
Abstract:
Leamer and McManus applied Extreme Bound Analysis (EBA) in an empirical study of the deterrent effects of capital punishment and other penalties. Their analysis has questioned the validity of the deterrence hypothesis. The thrust of our paper is twofold: first, by applying EBA to well-known econometric models of demand, production, and human-capital investment, our analysis exposes and illustrates the inherent flaws of EBA as a method of deriving valid inferences about model specification. Second, since the analysis shows Leamer and McManus’s inferences about deterrence to be based on a flawed methodology, we offer an alternative, theorybased sensitivity analysis of estimated deterrent effects using similar data. Our analysis supports the deterrence hypothesis. More generally, it emphasizes the indispensable role of theory in guiding sensitivity analyses of model specification.
Abstract:
Crime is a subject of intense emotions, conflicting ideologies. However, economists have generally explained it as a reflection of individual choice and equilibrating market forces. Two major themes of the literature are outlined: the evolution of a ‘market model’ to explain the diversity of crime across time and space, and the debate about the usefulness of ‘positive’ versus ‘negative’ incentives. Systematic analyses generally indicate that crime is affected on the margin by both positive and negative incentives; there are serious limitations to the effectiveness of incapacitation and rehabilitation; and optimal enforcement strategies involve trade-offs between narrow efficiency and equity considerations.
I agree this is a “welcome start” but not definitive. One plausible alternative theory that they cannot have controlled for is that the Fed, as the “last mover”, would have done more monetary stimulus if there were no fiscal stimulus.
Ken B:
Let’s review. One poster noted that the figure 8 murders averted per execution was the result of an economic theory. You said, no it’s the result of empirical analysis. I noted that it is ineluctably both. You objected, and then added the provision about “structural” models. Is this a fair recapitulation?
I think Steve said the opposite, that selecting the instrumental variables does not involve any “structural” model.
The study B carries out is really an attempt to test that hypothesis [that cigarettes cause cancer].
That’s exactly right: He’s testing a hypothesis, not a model.
Some quotes from the 1975 Ehrlich paper:
“Section 2 is devoted to the empirical implementation of the model”.
“Assuming the offender behaves as if to maximise expected utility, a necessary and sufficient condition for murder to occur is that o’s expected utility from crime exceeds his expected utility from an alternative action…”
“To illustrate the behaviuoral implications of the model…”
“An immediate implication of the model that is independent of the specific motives leading to an act of murder… but the somewhat more detailed formulation if the model adopted in this paper makes it possible to derive more specific predictions concerning the relative deterent effects of apprehension, conviction and execution…”
“In the empirical investigation an attempt is made to test the main behavioural implications of the theoretical model. The econometric model of crime developed by the author is applied to aggregate crime statistics…”
“Conclusions: The basic strategy I have attempted to follow in formulating an adequate analytic procedure has been to develop a simple economic model of murder and defense against murder, to derive on the basis of this model a set of specific behavioural implications that could be tested against available data, and to test those implications ststistically.”
It seems to me that Ehlrich thought his work was based on a model
He has determined the absolute deterent rate, thus de-coupling the effect of lower conviction rates where the sentence is execution.
Link to the paper:
http://deathpenalty.procon.org/sourcefiles/Issac%20Ehrlich%20The%20Deterrent%20Effect%20of%20Capital%20Punishment%20A%20Question%20of%20Life%20and%20Death.pdf
Harold: It’s forever since I read Ehrlich’s paper, so I could be wrong, but my strong recollection is that:
a) He uses a model to make some predictions about the deterrent effect of capital punishment. This model makes assumptions about the response to incentives.
b) He uses econometric techniques to test whether the predictions are accurate. The econometric techniques confirm the predictions. Those techniques make some assumptions, but those assumptions do NOT include anything about response to incentives.
The analogy is: I have a model of how lungs work. My model predicts that cigarettes cause cancer. I test this model by checking that when cigarette taxes rise, cancer rates tend to fall. That emprical fact *confirms* the model but it does not *depend* on the model. The empirical fact stands on its own.
Jeff Semel: I think you’ve got this exactly right.
I have read one or two books with sections about stimulus spending. If I remember correctly, they argued that if your goal is to stimulate the economy, it doesn’t matter what you (the govt) spend it on. You can spend $1bil on infrastructure, or $1bil on HDTVs, or dump $1bil in Main Square for anyone who happens by to pick up, and it has the same stimulating effect on the economy. The argument for spending it on infrastructure is that it helps your economy later – which is nothing to do with any current need for economic stimulus.
However, your post about the article says that it *did* matter (in the USA) what the money was spent on, and that spending on infrastructure worked better.
Why is this? Is the book I read earlier (or my memory of it) rubbish? Is there something wrong with Wilson’s work? Or is it just because of some coincidence that (say) infrastructure projects have bigger ‘knock-on’ effects on money supply than (say) cash for clunkers?
Mike H: I think you can view this paper as an empirical test of the theories you read in those books.
As I understand it, the only way you can assert each execution prevents 8 others is by reference to a model. The number of murders actually went down as the number of executions went down, so there is no obvious correlation. In your cigarette example, it is as if the number of cancers increased as cigarette taxes rise. There is no way the empirical fact stands on its own. I think it is working like this: the paper uses the empirical data to determine regression coefficients for the various terms in the model. These then reveal how strongly each term affects murder rate.
How about this. Our theorist observes that cancer rates rise as cigarette taxes rise. Thats odd, thinks he, If cigarettes cause cancer, it should be the other way round. I will construct a model of cancer. He puts into the model smoking, and also factor X, a cancer causing agent which is increasing. His model predicts how cancer will vary with both smoking and factor X. He then has an econometric model of how tax rates influence smoking. He inputs the empirical data for factor X and tax on smoking and cancer rates. Lo! He finds that tax rates are negatively correlated with cancer when applying his model, even though cancer rates go up when tax rates go up.
This corresponds to the deterent effect of execution and murder rates. The simple correlation is that executions go down and murder rates go down, i.e. a positive correlation. This is why Sellin concluded there was no deterent effect. When you apply the model with the other factors, then the correlation with execution becomes negative. Thisis in accordance with the behavioural model. But the findings are still only as good as the model.
There are 2 models in each case. In the cancer one there is the model of how the lung works, and then the model on how tax affects smoking. In the murder case there is the behavioural model and the econometric model of the murder supply function.
I am not sure I have this totally correct, but overall it seems you must apply some model in order to turn a positice correlation into a negative one.
@Jeff Semel (& Steve):
Yes exactly, that provision that SL added was about structural models. But I wasd talking about models in general, and never said structural models. SL has now written:
“a) He uses a model to make some predictions about the deterrent effect of capital punishment. This model makes assumptions about the response to incentives. … That emprical fact *confirms* the model but it does not *depend* on the model. The empirical fact stands on its own.” This is probably where the confusion and or disagreement is. The empirical facts are really just the murder rates in various places at various times. Those are the facts. These facts are processed into the result SL quotes by applying OTHER aspects of a model, to wit deciding what regressions, corrections, controls etc to apply. Did the paper factor in the changing demographics? If so then a model of the effect of demographics has had its say in the result has it not? The number 8 is an inference from the (raw) empirical data. It is not like counting toes.
Harold:
I am not sure I have this totally correct, but overall it seems you must apply some model in order to turn a positice correlation into a negative one.
No, you only need to introduce an instrumental variable.
Suppose I notice that as hospital stays go up, so do durations of illness. I am not committed to any structural model and therefore have no idea which way the causality runs.
Then I find a subpopulation who for some reason was *randomly assigned* to spend certain lengths of time in the hospital. I find that among *this* population, as hospital stays go up, the duration of illness goes down. This turns the positive correlation into a negative one and convinces me that the hospital stays *cause* drops in the duration of illness. (The causation can no longer run the other way because the hospital stays were randomly assigned.)
This procedure DOES rely on assumptions —- it assumes that the hospital stays in this subpopulation were truly random, that nobody monkeyed with the asssingments, etc. But that’s
NOT THE SAME as relying on a model of the WAY in which hospital stays affect illness durations. It makes NO ASSUMPTIONS whatsoever about that.
Now when I go to write the paper, I am likely to include a theoretical part, which presents a model in which hospital stays reduce illness FOR SOME REASON that makes sense to me. And I will include the empirical part, which shows that (under my randomness assumptions) hospital stays cause shorter illnesses. But that empirical part stands on its own and does not depend on the reasoning given in the theoretical part.
Yes, you need assumptions. No, you do not need *structural* assumptions.
Ken B: Yes, the number 8 is an inference. The question is whether that inference relies on an assumption about the role of incentives. I claim that it need not (which I’m sure of) and to the best of my recollection it does not (which of course I’m less sure of, not having looked at the paper in a long time).
OK, the first model in the paper, based on incentives, is used to make predictions. Many of the references I quoted earlier refer to this part. These are then tested in the empirical part, the second part.
The bit I don’t get is how that correlation gets turned the other way. If you correlate murders with executions you get a positive correlation. You may conclude (as did Sellin) that executions do not deter murders. Now Ehrlich comes along and finds that the correlation is negative, and executions do deter murders. How did he get from the positive correlation in the raw data to a negative correlation? In your hospital example you found a sub-population that (somewhat implausibly) had been randomly assigned hospital stays of fixed duration, regardless of their state of health. What is the equivalent for Ehlich? Section IIa is the key, where Ehrlich says “In the empirical investigation an attempt is made to test the main behavioural implications of the theoretical model. The econometric model of crime developed by the author [1973a] is applied to aggregate crime statistics…”
So he is applying an econometric model developed earlier to test the behavioural model. Now as far as I can work out, where we seem to differ is the level of assumptions needed to make a model. If we make assumptions about the lack of interference with data etc, then this a model does not make. However, if we make enough assumptions, can we not call the collected assumptions a model? This will be a different model (or set of assumtions) to the first, behavioural model. It seems to me that the “assumptions” made for the empirical part of the paper are a model.
He says “the results change substantively…when the full econometric framework developed in the preceding section is implemented”. Is not this framework what he refered to as “the econometric model”?
Harold:
So he is applying an econometric model developed earlier to test the behavioural model. Now as far as I can work out, where we seem to differ is the level of assumptions needed to make a model.
If we differ on what’s *needed* then I’m certainly right and you’re certainly wrong. If we differ on what Ehrlich actually *did*, you’re the one with the paper in front of you and I’m the one relying on memory, so there’s a good chance you’re right.
@Steve:
I agree with the last to me. The inferred 8 is taken as a test of the claim that there will be an incentive effect. Model predicts rise in the murder rate *beacuse of the postulated effect of the incentive*; empirical analysis confrims the rise *based on other assumptions* and is taken as confirmation of prediction thus lending support to the postulated incentive effect.