St Marys Hospital
From Mmc
Dear Professor Paice
MTAS 2007
We are writing in response to the experience of both junior and senior medical staff with the MTAS system. These have highlighted major problems, which make the new system unacceptable for the appointment of junior doctors. These concerns relate to the process and its administration, the loss of confidence by senior medical staff, and the requirement for Trust management to have confidence that the doctors appointed to their hospitals are the highest quality and most appropriate trainees for their posts
The application and shortlisting process has given rise to many concerns some of which are listed below:-
The methodology is untested and there would seem to be no evidence base for it. The application form, and the scoring system are flawed – the form consists of a number of questions, the answers to which are limited to 150 words. This word limit and the nature of the questions make it very difficult to differentiate between candidates, and experience at shortlister training events demonstrated that a wide range of marks was awarded for the same answers. This suggests that it is impossible for them to be used as any form of objective marker of suitability. The information contained in answers is also often impossible to validate as being true, and there are now commercial companies who are offering a service filling out application forms according to the criteria for profit. Clearly this is undesirable and inappropriate. The suggestion that all cases of reported plagiarism will be investigated is an inadequate response, which makes no attempt to identify the size of the problem or deal with it effectively.
In addition in Section B the criteria relating to academic qualifications and prizes mean that Distinction in examinations at Finals and 1st Class BSc qualifications count for nothing, whilst unvalidated prizes may score points.
The process of shortlisting was also deeply flawed: The original protocol was that each question was seen by only two shortlisters, and for large specialties these were not due to be the same for all candidates, leading to the possibility of significant inter-observer variation. In the event the situation was much worse than this because many shortlisters were presented with an impossible timescale and ended up asking untrained colleagues to help shortlist large numbers of candidates. It is impossible in these circumstances to guarantee any consistency in scoring at all.
There is also well documented evidence that the Deanery suggested that if forms were not scored on time the candidates would be excluded from the first round of applications. We believe this was potentially unfair on the candidates and was inappropriate
Shortlisting has been done in general either on single questions or small groups of questions. The consequences of shortlisting a single question may be very negative: Nobody has an overview of the application overall, and candidates can score on several questions using the same information. This appears to have happened on a number of occasions, and is completely inappropriate. Information on previous employment, and place of qualification is also excluded from consideration by shortlisters. However we believe this can be very valuable in assessing an application.
We believe that the above factors have given rise to the situation that occurred after the publication of the shortlisting results: namely that outstanding young doctors who have worked hard to gain exceptional qualifications, and who are of very high calibre have not been shortlisted, who would have been if the methodology was robust. This has lead to a loss of confidence in the system by consultants - both those who have been involved in shortlisting, and those who have not.
We have seen your letter of March 2nd to Clinical Tutors in which you acknowledge some of the deficiencies in the system, but state that Round 1 interviews should go ahead without further review. However we believe the interviews will not identify whether the right candidates have been called because there is no opportunity to compare them with those who have not been shortlisted.
We are now aware that a review has been set up by the Department of Health with clear terms of reference as of 1800hrs 6.3.07.
However we submit that, whilst round one may need to be completed for the sake of the trainees, the threshold for appointment must be very high, only appointing clearly outstanding candidates and further that panels must have in front of them the candidate’s full Curriculum Vitae. These thresholds must be agreed by the relevant Specialty Training Committees and interview panels.
In addition when round two starts – it being scheduled for 28.4.07 –the shortlisters should have the benefit of the candidates’ CV to consider, with whatever other material the review deems appropriate.
It is vital that this review is thorough and is not afraid to make major changes to the system, as it cannot function if it does not have the confidence of senior and junior doctors alike.
We look forward to your early response.
Yours sincerely
Mr R Touquet. Chair, Trust Medical Advisory Committee on behalf of all the Consultant Staff at St Mary’s Hospital
Dr David Mitchell. Medical Director
Dr Tim Orchard. Director of Clinical Studies
CC Professor Ian Gilmore PRCP, Royal College of Physicians, St Andrew’s Place, London
Mr Bernard Ribeiro PRCS, Royal College of Surgeons of England, Lincoln’s Inn Fields, London
Mr Keith Smith, MMC Programme Lead
Professor Neil Douglas PRCP, Royal College of Physicians of Edinburgh, and Chair of the Independent Review Panel into MTAS and MMC
