The Department of Higher Education and Training’s (DHET) recent Report on the Evaluation of the 2019 Universities’ Research Output has led to some excitement with various claims made about the positioning of certain universities in comparison to others.

The DHET’s research outputs policy encourages universities to engage in research. On an annual basis university research outputs are evaluated and generate research outputs subsidies, intended to provide operational funds to support and stimulate research activity.

Depending on the mandate and mission of an institution, we would expect to find different outcomes in terms of research productivity. The slavish, and sometimes perverse pursuit of ‘research productivity’ can lead to skewed outcomes with quantity out playing quality.

‘Brute’ data on publication output units is reflected in Table 1 on page 20. This table reports the quantity of publication output units for the system, based on the DHET’s proportional evaluation system. At face value it shows the ‘top six universities’ to be UKZN, UJ, UP, SU, UCT and WITSin that order. These six institutions collectively produce 60% of the publication output units for the system. The next eight institutions, NWU, UNISA, UFS, UWC, RU, NMU, UL and UFH, in that order, collectively produce 30% and the remaining institutions, including all the Universities of Technology (UoTs), produce the final 10% of the system’s publication units.  

The majority of units (81.80%) are awarded for journal publications. Sorting the ‘brute data’ on journal publication outputs from highest to lowest gives UKZN in first position, followed by UP, UCT, UJ, SU and WITS. Journal articles in accredited journals are important research products and perhaps the best measure of research publication productivity.

The next highest proportion of publication output units are awarded for books (12.15%). Doing a similar sort, the top six are UJ, SU, UFS, UP, WITS, and UCT, in that order. UKZN has dropped to eighth, replaced by UFS. Book publication units increased substantially after a change in policy in 2016. Incentivising scholarly books is important, however recognition of quality outputs needs work.

Finally, the smallest contribution proportionally is for conference publication units (6.05%).  The ‘top 6’ are: UJ, NWU, SU, UCT, UP and UNISA. What is remarkable is that UJ has 23.20% of all publication units of this type, compared to NWU with 9.34%.  The quality of research outputs in conference proceedings and gaming of the system in conference proceedings was highlighted in a DHET commissioned report, The Quality of South Africa’s Research Publications (CREST, 2019). Suspicious activity, including a university claiming outputs for 113 conference papers by a single author in 2017, was identified. The report recommends that a maximum number of conference publication units per author per annum should be implemented to discourage unethical behaviour. 

Table 1 needs to interpreted with caution. If comparisons are being made then other metrics need to be looked at, and in particular normalised data needs to be considered.

Table 12 shows the publication units per capita. This considers the number of permanently employed academics in each institution. The same 6 universities are at the top of the list however the order is changed. The highest number of publication units awarded per academic is recorded for UKZN (1.83), followed by SU (1.72), UJ (1.71), UP (1.68), UCT (1.64) and Wits (1.59). Smaller institutions such as RU and UFH move up the table.

However, this still leaves out an important aspect of a university’s research activity - the production of research Masters and Doctoral graduates, a core responsibility of research focused universities. Table 13 considers the ‘weighted per capita research output’ which considers publication and research graduate output units per permanent academic staff.  Here the ‘top six’ change again. UKZN remains top with 3.62 per capita weighted research output units, followed by UP (3.61), SU (3.39), WITS (3.04), UCT (2.88) and RU (2.82).  This table would be most appropriate when attempting to ‘rank’ institutions in terms research productivity.

In Table 13 the top six, based on the proportion of permanent academics with Doctoral qualifications are UP (69.70%) followed by WITS (66.00%), UCT (62.40%), UKZN (61.50%), UWC (59.90%), and RU (59.40%).  Interestingly SU is replaced by UWC in this list. One would expect a high level of correlation between staff with doctoral degrees and the per capita publication and weighted per capita research output.

Table 14 shows the ratio of doctoral graduates to doctorate staff members by university. This ratio could be interpreted as a measure of efficiency of staff in relation to supervision and doctoral student completion.  UKZN is again on top of the table with a ratio of 0.59, followed by SU (0.53), UP (0.47), UNISA (0.42), RU (0.41) and NWU (0.39). In this list WITS, with a ratio of 0.37 drops to position 8.

The report illustrates how research outputs of the system have improved over time. Certain universities are consistently in the top tier of the various lists. While all universities are to some extent involved in research activity, traditional universities are more research productive on the whole than comprehensive universities or UoTs.

One critique is the report does not reveal any qualitative data about the system. For example, it does not report on impact; it says nothing about whether or not this is research that matters; there is no information on outputs produced through collaborative partnerships; or, what proportion is produced by researchers in conjunction with their Doctorate or Masters students (a form of double dipping).

While the research outputs policy and subsidy system has effectively incentivised institutions to produce research, it has also had some unintended consequences with gaming the system.

A 2018 ASSAf publication, Recognising Individual Contributions to Collaborative Research shows that the proportional system implemented by the DHET fails to recognise the contribution of individuals within large research teams of greater than 100 authors. Changes in the practice of modern science in disciplines including physics, genomics, medicine and statistics is a global phenomenon and have resulted in individuals working within large teams publishing impactful research. The ASSAf report makes some suggestions on how this kind of research, which is highly important, impactful and really matters can be ‘counted’.

In addition, transdisciplinary research directed at societal challenges may need special incentivisation going forward. Such research addresses complex, complicated and intersectional challenges including for example pandemics, unemployment, inequality, poverty and climate change. It not only requires the crossing of disciplinary borders and boundaries but also knowledge co-creation with other stakeholders in society including government, the private sector and civil society. The proportional evaluation system currently implemented is unlikely to recognise this sufficiently.

In conclusion the research outputs report provides interesting data on research productivity, but is a partial story and should not be seen as some kind of university ranking system. The DHET’s commitment to working with the system to develop a research quality framework that can assist in instilling ethical research practices and driving the quality of research as opposed to quantity is welcome. The need to better understand the changing research practices across scientific fields and in particular to find ways of recognising high impact research produced by large research teams will be important. The system must also consider how to factor in high impact research that matters and leads to innovation.