Acta Universitatis Danubius. Œconomica, Vol 14, No 5 (2018)

The Sensitivity of the Effect of on-the-Job Training on Employment Outcomes in Experimental and Non-Experimental Settings

Ana Gorenca¹, Theodhor Kuro²

Abstract: This paper examines the effect of on-the-job training on the probability of finding a job. We reevaluate the existing training program using experimental data from the National Employment Service, 2013 and non-experimental data from the Labor Force Survey, 2013. Moreover, we employ the Propensity Score Matching method to estimate the training effect and to check its sensitivity to a different model specification and to different degrees of randomization. The results show that the average training effect on the treated is smaller and the reduction in the selection bias is higher when a different specification is used. Moreover, the effect is also sensitive to different degrees of randomization settings, i.e., the effect is smaller in a non-experimental setting compared to the quasi-experimental setting. Hence, we conclude that the average training effect on the treated decreases if we increase the randomization of the treated group.

Keywords: on-the-job training employment; propensity score matching; sensitivity; ATT

JEL Classification: J28

Introduction

The employment promotion programmes (EPP), have gained a considerable attention by labor market institutions worldwide. In specific, on-the-job training is considered as a measure that would not only tackle unemployment, but would also contribute in the skills gap reduction among unemployed jobseekers. Cahuc and Zylberberg (2004) argue that training programs raise the overall quality of the workforce. However, in unemployment, investment in training is costlier (Mortensen, 1986). Hence, the financial constraint that training imposes to unemployed people might reduce the incentives to invest in their human capital. Becker (1964) suggests that this problem is solved by government intervention through subsidies. There are several examples from different countries that have implemented such interventions. In this paper we focus on training programmes, specifically those implemented in European countries. Cahuc and Zylberberg (2004) consider the American experience to be unimpressive in terms of the efficiency of the implemented subsidies.

The literature concludes that there is a double effect of training on exiting from unemployment. One one hand, training programs are considered as layers of control by potential employers (Richardson & Van den Berg, 2002). On the other hand, training might increase the job seeker’s reservation wage, and this leads to longer unemployment spells (Fougère, Crépon & Ferraci, 2007). Nevertheless, other authors argue that the effect of training on unemployment duration might depend on timing. For instance, it is likely that in the short run, e.g., several weeks after the end of the training program, training incentivizes jobseekers to exit from unemployment (Fougère, Crépon & Ferraci, 2007; Richardson & Van den Berg, 2002). In addition, McGuinness, O’Connell and Kelly (2014) argue that in the long run, e.g., several months after training, the impact might disappear. Aside from timing, the effect of training on unemployment duration depends also on the nature of training. For instance, Smet (2012) argues that on-job-training raises the probability of employment or reemployment compared to job-search-training. Below, we present a few programmes implemented in Northern European countries.

In Finland, the Työhön experiment (Job Search Programme, 1996-1976) was designed to subsidize the recently unemployed jobseekers to smooth their transition to employment, and prevent their mental health effects caused by the struggle to find a job (Hämäläinen, Uusitalo & Vuori, 2008). The programme aimed to help participants enhance their job search skills through job search training. Hämäläinen et al., 2008 argue that the recruitment (selection) was voluntary and the assignment into treatment (training) was random. That is, the participants chose to enter the program but they were randomly split into treatment and control groups. Using several propensity matching strategies, the authors find that the difference in employment rates between the treated and untreated ranges between negative 3 percent and 30 percent.

Similarly, in Sweeden, Björklund and Regnér (1996) evaluate the effect of a social experiment in the form of a job-search assistance for 410 unemployed jobseekers, randomly assigned into treatment and control groups. Those treated, participated in the training program for 7.5 hours on a weekly basis, and their counterparts received the treatment for only 1.5 hours/week. After 9 months, the rate of employment for the treated group was 13 percentage points higher than the employment rate of the control group.

In Norway, Torp (1994) evaluates the effect of labor market training (LMT) on unemployment duration using non-experimental data of unemployed jobseekers. The treated group is drawn from the LMT 1989 survey and the untreated group (non-participants) is randomly drawn from the stock of unemployed jobseekers in 1989. The authors employ Tobit and Heckman two- step estimation models. Their results indicate that participation in training improves the employability of unemployed jobseekers. However, this applies only to short and long period trainings. The contrary is found for semi-long training courses.

In Albania, the National Employment Service has provided a gamut of programmes targeting unemployed youth and vulnerable groups. In this work, we examine the effect of on-the-job training on exiting unemployment using a Propensity Score Matching approach. First, we reevaluate the existing program in a quasi-experimental setting to check the sensitivity of the average training effect on the treated to a different model specification. Second, we rely on non-experimental methods by Dehija and Wahba (1999) to estimate the same effect and examine whether the effect is sensitive to different degrees of randomization.

This paper is organized as follows. Section (2) provides a brief review of the existing on-the-job training program. Section (3) describes the empirical model and section (4) presents the data and the results of this study. Lastly, section (5) concludes and presents the motivation for future work.

Review of Existing Programmes

This section reviews ILO-EU-IPA (2014) final report: “Employment Promotion Programmes in Albania: An assessment of its quality in the formulation and implementation processes (2008-2014)”. Specifically, we reevaluate the effect of the training program on the probability of becoming employed using a different model specification. The employment promotion programme (EPP) we are interested in is that of on-the-job training, approved by the Council of Ministers, with Decision no.47 (CoM no. 47). The mechanism the program delves into reducing unemployment can be described as follows: the programme provides financial support to employers who offer a traineeship to jobseekers registered in the programme. The duration of on-job-training is approximately 6 months. Applications to the programme were submitted by employers. The unemployed jobseekers are selected by the National Employment Service (NES) and a brief profile of the potential participants into the program is submitted to the companies. In the report, it is mentioned that there is dissatisfaction from the side of employers regarding the low profile of the selected jobseekers. Moreover, there is a mismatch between the skills demand from the employers and the needs of jobseekers. This has an important implication concerning the design of the program. While NES offices selected discouraged jobseekers in order to improve their labor market situation, companies aimed at already skilled participants so that the chances to employ them after the programme would be higher. This is reflected in the reasons why companies applied to the programme: mainly to recruit highly skilled workforce (which goes in line with Cahuc and Zylberberg (2004) expectations) and to improve the quality of their business plans. Additionally, the largest number of recruitments is registered in the small sized companies, mainly those operating in the clothing confection and construction sectors, i.e., companies that would employ low-skilled job-seekers. All considered, the outcome, employment of the participants after the training programme, is biased towards the needs of the NES.

Given the mechanism the program is designed, there are a few issues that require attention. First, ILO-EU-IPA (2014) argue that the treated group is not random. Second, there is self-selection into treatment from the side of the applicant and the employer. That is, the employers tend to select the already skilled jobseekers. Whilst voluntary participation is found to yield biased estimates owning to the participant’s unobserved motivation (Hämäläinen et al., 2008), entirely caseworker's assessment might lead to selection into treatment bias. Third, given that the program design assumes the features of a quasi-experiment rather than a social experiment, the true counterfactual does not exist (ILO-EU-IPA, 2014). However, given the lack of randomization, their control group is drawn from the same survey. To this extent, we consider non-experimental methods to reevaluate the training effect on employment outcomes. More on non-experimental methods is provided in section (4).

Empirical Model

The empirical framework of this work borrows from program evaluation methods which in the context of this study examine the effect of active labor market policies (ALMP) on the labor market position of unemployed jobseekers.³ Specifically, we employ Propensity Score Matching (PSM) non-parametric methods to evaluate the impact of on-job-training on the probability of finding a job.

In essence, PSM is a mechanism that potentially solves the bias of selection into treatment. The latter arises when we want to identify the difference between the participant’s outcomes with and without treatment (Caliendo & Kopeinig, 2005). Since we cannot simultaneously observe both outcomes, the matching idea is to construct a counterpart of the treated group, the control group, with similar pre-treatment characteristics. Thus, the difference in the outcomes of the treated and control group will only be attributed to the programme (treatment).

The empirical model builds on the work of Dehejia and Wahba (1999). Let X, be a vector of observable characteristics. Owing to the curse of dimensionality, we neglect exact matching. Thus, we rely on Roy-Rubin model of balancing scores b(X), where b is a function of X, such that the conditional distribution of X given the balancing scores is independent of assignment into treatment. Let i denote the assignment into treatment, i.e., i is a binary treatment indicator as given by (1).

(1)

The potential outcomes, e.g., employment status, are denoted by Yi( i). Hence, the treatment effect for the ith individual, can be written as:

(2)

Since we can only observe either Yi(1) or Yi(0), we fail in estimating the individual effect of treatment. Therefore, the parameter of interest would be the average treatment effect of the treated (ATT):

, (3)

where the last term denotes the counterfactual outcome which is not observed. Coliendo and Kopeinig (2005) argue that using the mean outcome of untreated individuals, is not the best measure to evaluate the ATT in non-experimental studies. That is, the determinants of assignment into treatment would also determine the outcomes of interest, leading to self-selection bias. Using the mean outcome of the untreated group, we can define the bias as follows:

(4)

The true parameter, ATT, is identified only when the bias vanishes. Moreover, Deheija and Wahba (1999) argue that in social experiments where the treated group is fully randomized the bias is zero. To obtain consistency of the PSM estimator, a few assumptions are needed (see Proposition (1) and Corollary in Deheija & Wahba, 1999). Let p(Xi) be the probability that i is assigned into treatment i, or in other terms, the propensity score:

(5)

Then, the conditional independence assumption tells that given X, unaffected by treatment, the potential outcomes are independent of assignment into treatment:

(6)

and given

(7)

Coliendo and Kopeinig (2005) define the expressions in (6) and (7) as the unconfoundness assumptions given X and the propensity scores, respectively. Additionally, we could assume the stable unit treatment value assumption (SUTVA) which rules out the case(s) when the outcomes of untreated individuals are affected by the treatment of the treated. The last two issues before we proceed to estimation are the common support (overlap condition) and the PSM estimator. The overlap condition rules out perfect predictability of treatment given the covariates in X, i.e., as shown in Eq.(5). Lastly, the PSM estimator, ATT, can be written as:

(8)

To provide more intuition about the mechanism the PSM works, we apply the law of iterated expectations to Eq.(4), assuming that the bias is zero, and rewrite it as follows:

(9)

Deheija and Wahba (1999) argue that the PSM estimator conditions on the propensity scores rather than the covariates of . This intermediate step is possible owing to the unconfoundedness assumption (7), which tells that the distribution of the covariates in X is the same for observations with the same propensity score. Hence, the intermediate step solves the curse of dimensionality.

Data and Estimation Strategy

In this paper, the approach to randomize the treated group borrows from non-experimental methods (Dehejia & Wahba, 1999). Given that the labor force participation survey (LFS) of 2012-2013 in Albania was conducted at the same time with the CoM no.47 survey, we randomly draw a sample of unemployed jobseekers from the LFS with similar characteristics as the initial control group. This would reduce the heterogeneity among groups, i.e., the labor market conditions would the same given the timing the data was collected. However, the limitation in this case is the insufficient number of draws. That is, we drop from the sample all individuals who did not participate in any training program during the year the survey was conducted. The total number of replacements is 146. Nevertheless, since we aim to perform a sensitivity analysis of ATT, we estimate the effect of training on employment in both quasi and non-experimental settings.

Sample Characteristics

The following analysis presents the sample characteristics of the original treated and control group. In addition, we perform mean test comparison to check whether the two groups are statistically indistinguishable (Table 1). The sample contains information on 1149 registered jobseekers from which 932 are treated and 217 are untreated. The majority of jobseekers registered in the programme are women. Regarding gender, the treated group is indistinguishable from the control group. However, there is statistical difference in the age of both groups. The same is concluded regarding their education attainment. While the majority in the treated group have earned a primary or lower secondary education degree, most of the untreated jobseekers are graduates from general high school programs. The mean comparison tests indicate that at most of the matching covariates, the treated and untreated are not statistically indistinguishable. Therefore, matching methods are necessary.

Mean comparison tests

Estimation strategy (matching choice)

Regarding the estimation strategy, we consider the following issues. First, our model choice is logit over linear probability model. The former would violate the overlap condition since the values of p(X) would lie outside of the unit interval. Second, the variable choice should satisfy the conditional independence assumption. To this extent, there are several differences in our variable selection compared to the matching covariates used by ILO-EU-IPA (2014). Our variable selection is based on the statistical significance, as one of the selection criterion suggested in Coliendo and Kopeinig (2005). Third, our matching algorithm is the nearest neighbor (NN) with replacement. The mechanism how the NN works is straightforward. For each treated individual i from 1 to N, we assign a neighbor h(i) to the control group such that the difference in (10) in minimized.

(10)

Whilst the NN with replacement reduces the bias and increases the overall matching quality, it might also increase the variance since an untreated jobseeker is used more than once as a match. This is a two-step estimation. In the first step, we estimate the logistic model of treatment predictability. In the second step, we estimate the non-parametric regression conditional on the propensity scores from the first step. Table (2) presents the logit results of assignment into treatment from the quasi-experimental and non-experimental settings.

First step: logit results of assignment into treatment

a) denotes the specification of the same variable in quasi-experimental matching

b) denotes the specification of the same variable in non-experimental matching

- reference category in parentheses

While males have higher odds of receiving the treatment in the quasi-experimental setting, females are more likely to participate in the training program when non-experimental data are used. In both settings, younger jobseekers (specifically, those aged from 15-34 in the quasi-experimental setting and those aged from 20 to 44 in the other setting) are more likely to receive the treatment. Regarding schooling, the likelihood of participation in the programme is higher for those with low levels of education. Despite the measurement of unemployment duration, i.e., in levels (months) for the quasi-experimental setting and a dummy indicator of short/long term duration in the non-experimental setting, it is evident that those with longer unemployment spells are more likely to receive the treatment. As expected, unemployment benefits do not affect the assignment into treatment.

To assess the quality of matching, Coliendo and Kopeinig (2005) propose the Pseudo R2. In the non-experimental setting, the value of Pseudo R2, shows that the observable characteristics predict 22 percent of the participation probability. In contrast, the predictability indicator for the quasi-experimental setting is only 3 percent. Hence, the matching quality is higher when we use non-experimental data. Alternatively, the common support tests (see Table A1 & A2 in Appendices), associated with the distribution of the treated and untreated after matching (see Figure A1 & A2 in Appendices) confirms the same result. Nevertheless, we fail in rejecting null hypothesis under balanced matching in both cases at the 1 percent level.

Given that the overlap condition is satisfied, it is safe to proceed to the second step: estimation of the average treatment effect on the treated (Table 3). Regarding the quasi-experimental setting, the ATT is 0.39. That is, participation in the training program increases the probability of finding a job after treatment by 39 percent. The estimate reported by ILO-EU-IPA (2014) is 0.55. Hence, we find that within the same setting, the ATT estimate is considerably sensitive to model specification. Moreover, the large effect of 0.55 might be overestimated owing to the unobservable heterogeneity or the considerable skills gap among the treated and control groups. The ATT estimate for the non-experimental setting is 0.33, i.e., on-job-training increases employment chances by 33 percent. The comparison of the quasi-experimental and non-experimental ATT estimates indicates that the effect of training is also sensitive to the degree of randomization of the treated group.

Second step: average treatment effect on the treated

Conclusion

In this paper we aim to reevaluate the effect of on-job-training on exiting unemployment using experimental and non-experimental methods. Moreover, we aim to check the sensitivity of the effect to different model specifications and different degrees of randomization using the same estimation strategy. The reevaluated average training effect on the treated is considerably smaller (23 percentage points lower) when we use a different model specification. In addition, the quality of matching and the reduction of the selection bias (within the same matching algorithm) are higher. The ATT estimate in a non-experimental setting is 6 percentage points lower than the estimate in the quasi-experimental setting, i.e., the estimate is sensitive to different degrees of randomization of the treated group. In this paper we omit the problem of unobserved heterogeneity (e.g., previous labor market experience, motivation to work and proximity to the employer offering the training among other factors) as there is data limitation in the quasi-experimental survey. However, more robust estimations would concern future research.

References

Arendt, J.N.; Rosholm, M. & Jensen, T.P. (2005). The importance of literacy for employment and unemployment duration. Syddansk Universitet.

Becker, G.S. (1964). Human capital: A theoretical and empirical analysis, with special reference to schooling. NY: National Bureau of Economic Research.

Björklund, A. & Regner, H. (1996). Experimental evaluation of European labour market policy. International Handbook of Labour Market Policy and Evaluation. Edward Elgar, Cheltenham.

Borjas, G.J. (2010). Labor Economics. Vol. 3. Boston, MA: McGraw-Hill.

Brännström, L. & Stenberg, S.Å. (2007). Does social assistance recipiency influence unemployment? Macro-level findings from Sweden in a period of turbulence. Acta Sociologica, 50(4), pp. 347-362.

Cahuc, P. & Zylberberg, A. (2004). Labor economics. MIT Press.

Caliendo, M. & Kopeinig, S. (2005) Some Practical Guidance for the implementation of Propensity Score Matching. IZA, Discussion Paper No. 1588. The Institute for the Study of Labor. Bonn.

Collier, W. (2003). The impact of demographic and individual heterogeneity on unemployment Duration: A regional study. No. 03, 02. Department of Economics Discussion Paper, University of Kent.

Dehejia, R.H. & Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American statistical Association, 94(448), pp. 1053-1062.

Fougère, D.; Crépon, B. & Ferracci, M. (2007). Training the Unemployed in France: How Does It Affect Unemployment Duration and Recurrence? Institute for the Study of Labor, France.

Gorenca, A. (2018). Determinants of the school to work transition. Manuscript submitted for publication.

Huibregtse, A. (2010). Evaluation of the Labor Market Programs (ALMP) based on the D.C.M, No. 47/48 and 873.

Hämäläinen, K.; Uusitalo, R. & Vuori, J. (2008). Varying biases in matching estimates: Evidence from two randomised job search training experiments. Labour Economics, 15(4), pp. 604-618.

ILO-EU IPA. (2014). Employment Promotion Programmes in Albania: An assessment of its quality in the formulation and implementation processes (2008-2014).

Kyyrä, T.; Parrotta, P. & Rosholm, M. (2009). The Effect of Receiving Supplementary UI Benefits on Unemployment Duration (No. 3920). Institute for the Study of Labor (IZA).

Landmesser, J. (2010). Assessing the impact of training on unemployment duration using hazard models with instrumental variables. Metody Ilościowe w Badaniach Ekonomicznych, 1(11), pp. 100-109.

McGuinness, S.; O’Connell, P.J. & Kelly, E. (2014). The impact of training programme type and duration on the employment chances of the unemployed in Ireland. The Economic and Social Review, 45(3, Autumn), pp. 425-450.

Mortensen, D.T. (1977). Unemployment insurance and job search decisions. ILR Review, 30(4), pp. 505-517.

Mortensen, D.T. (1986). Job search and labor market analysis. Handbook of labor economics, 2, pp. 849-919.

Richardson, K. & van den Berg, G.J. (2002). The effect of vocational employment training on the individual transition rate from unemployment to work. Swedish Economic Policy Review, Review 8, pp. 175-213.

Smet, M. (2012, January). The impact of vocational training programs on the duration of unemployment spells of school leavers from technical and vocational secondary education in Flanders. In 7^th Annual International Symposium on Economic Theory, Policy and Applications 23-26 July 2012: economic abstracts, p. 67. The Athens Institute For Education And Research.

Torp, H. (1994). The impact of training on employment: Assessing a Norwegian labour market programme. The Scandinavian Journal of Economics, pp. 531-550.

Van Leuvensteijn, M. & Koning, P. (2000). Duration dependence in unemployment insurance and social assistance: Consequences of profiling for the unemployed, Vol. 163. CPB Netherlands Bureau for Economic Policy Analysis.

Appendices

Common support test to check whether matching is balanced for the quasi-experimental setting

Table A1. Common support test

H0: Matching is balanced

Ha: Matching is not balanced

P>chi2 : 0.503 : failed in rejecting H0

Common support test to check whether matching is balanced for the non-experimental setting

Table A2. Common support test

H0: Matching is balanced

Ha: Matching is not balanced

P>chi2 : 0.404 : failed in rejecting H0

Distribution of treated and untreated cases after matching

Figure A1. Quasi-experimental

Figure A2. Quasi-experimental

Refbacks

• There are currently no refbacks.