In this Economic Letter from the SF Fed, Leila Bengali, Mary Daly, and Rob Valletta argue that this is, in fact, something to worry about. They "find evidence, reinforcing other research, that the recent decline in participation likely has a substantial cyclical component" (i.e. their analysis concludes that exit from the labor market is temporary for a substantial number of people, and they will begin seeking work again when the economy improves):
Will Labor Force Participation Bounce Back?, by Leila Bengali, Mary Daly, and Rob Valletta, Economic Letter, FRBSF: The most recent U.S. recession and recovery have been accompanied by a sharp decline in the labor force participation rate. The largest declines have occurred in states with the largest job losses. This suggests that some of the recent drop in the national labor force participation rate could be cyclical. Past recoveries show evidence of a similar cyclical relationship between changes in employment and participation, which could portend a moderation or reversal of the participation decline as the current recovery continues.
Since the beginning of the recession in 2007, the U.S. labor force participation rate has dropped sharply. Some of this decline reflects long-term demographic trends and other factors that helped push down the participation rate before 2007. But the recent withdrawal of prime-age workers from the labor market is unprecedented and may reflect a cyclical component that could reverse as the labor market recovery solidifies. The return of these workers to the labor force would partially offset the longer-term demographic influences and potentially cause the participation rate to bounce back (Daly et al. 2012, Van Zandweghe 2012). Moreover, the increase in the number of active jobseekers in the labor force associated with higher participation could slow the decline in the unemployment rate.
Assessing the contribution of cyclical factors and the likelihood of a reversal or slower decline in labor force participation is difficult based on aggregate labor market data alone. Such data cannot perfectly distinguish between long-term trends and shorter-term cyclical factors, particularly given the severity of the labor market dislocation during the past recession. To assess the role of cyclical factors in the current recovery, we examine state-level variation in the relationship between changes in the labor force participation rate and changes in employment over several business cycles. ...After a detailed analysis, they conclude that:
The U.S. labor force participation rate has declined sharply since 2007, intensifying a downward trend that has been evident since about 2000. Distinguishing between long-term influences on the participation rate, such as demographics, and short-term cyclical effects is important because it helps us understand and predict the future path of macroeconomic variables such as the unemployment rate. Using state-level evidence on the relationship between changes in employment and labor force participation across recessions and recoveries, we find evidence, reinforcing other research, that the recent decline in participation likely has a substantial cyclical component. States that saw larger declines in employment generally saw larger declines in participation. A similar positive relationship was evident in past recessions and recoveries. In the current recovery, it will probably take a few years before cyclical components put significant upward pressure on the participation rate because payroll employment is still well below its pre-recession peak.Let me add, once again, that the costs of being wrong are not symmetric. If we are going to make a policy mistake, the bias ought to be toward keeping policy in place too long (and perhaps enduring a temporary bout of inflation) rather than putting the brakes on too quick (and ending up with an elevated unemployment rate and all of the short-run and long-run consequences that come with it).