33 Lessons
From practical experience of control self-assessment programmes.

1. Why controls are not performed
Controls are generally not performed for one of three reasons:

  1. The control is hard to perform or even impossible
  2. The control is not seen as important (I don't have the time)
  3. They don't know how to do it (haven't been trained)

In setting up CSA you map the processes and define the controls and difficult or impossible controls are "kicked out". The questionnaire is only aimed at the key controls and therefore all those that are in the questionnaire are by definition important. If the user does not know how to perform the control, they have the back-up of the best practice references and the action plans to help them improve and to help them understand what they have to do. In short, the system helps train them.

2. What gets measured gets improved
By implementing CSA, we are measuring the state of controls within the business. What gets measured gets improved because it is reported on, because it is discussed and because a clear way to improve is provided (by the action plans within the system). "From Worst to First" by Gordon Bethune (CEO Continental Airlines) majors on the idea of measurement as the driver of improvement to the extent that it transformed his company from possibly the worst airline performer to one of the best.

3. Responsibility
One of the major reasons for implementing a CSA programme is to make it clear that the responsibility for controls, the ownership of controls, rests with the user. If the user is simply sent a questionnaire (paper or system), that is not responsibility, that is checking up on them. The system needs to be accessible at any time to the user so they come to use it as they need it, so that it becomes theirs - that is ownership. One very large intranet system was made available once per day to the users and the questionnaire had to be filled in on that day - the programme failed within three months.

4. Common Language
One of the benefits that is always quoted for CSA is the provision of a common language of control. This can be a hard concept to grasp. One way in which this becomes clear is in the naming of the programme. Giving the system an easy, memorable or friendly name will help acceptance of the task. Then users will refer to their CSA results by the name of the system and will be talking a language of control without realising it. At ASDA, the system was called SAS (self-assessment system) and users talked about their SAS scores. At Conforama, the system was called Co-ol (contrôles on-line) which was seen as something in line with the culture of the company.

5. Give and take
The principle of reciprocity is very important. We are asking people to do something for us - to report on their control environment. Naturally, they want to see some benefit in return. By giving them access to procedures in an accessible format (simple on-screen format) linked to the CSA system, then they are getting something in return. If the procedures do not actually exist, the drafting of simple procedures that relate to the control processes being surveyed can be a very big win for the programme.

6. Visual impact
The "look" of the tool is important. We are asking operators to do something and so the tool needs to be easy, simple, intuitive and attractive. If the first thing the user says is "I don't like the look of that" or "it isn't very colourful, is it?", then the battle is lost in advance.

7. Forget paper
We are talking about a system-driven control self-assessment programme. Nevertheless, some companies try to perform a pilot on paper first, to test the questions (and finish off the system in the mean time). This is a bad move unless the paper-based programme is actually a series of screen-shots of the actual system. In general, paper questionnaires get a 30-50% response rate, even after several reminders. Paper pilots fail to test the acceptance of the system itself and get the programme off to a very bad start.

8. Satori in CSA
"Satori" is the Japanese word for a kick in the head or a sudden realisation. Successful programmes deliver something extra that the programme team perhaps never expected. These have included the tool being used directly for training of users, and also for investigations into problems or discrepancies.

9. A dog is not just for Christmas (12-month rule)
80% of CSA programmes fail in the first 12 months. Failure means that users stop responding, stop using the system. If the programme reaches its first anniversary then it is successful and will probably continue to be successful. Learning from other programmes that are less than 12 months old may not be beneficial because it is not yet clear whether or not it is a good model to base learning on.

10. Pilot, not Big Bang
One of the features of the 80% projects is that they have all been rolled out in a "big bang" approach with no piloting and with far too many modules at one go. Piloting enables us to assess the acceptability of the questions, the suitability to the culture and the best way of working for the particular organisation.

11. Ownership
We want the users to own the controls and to own the system. We therefore have to listen to them and take on board their comments in the pilot phase and also be seen to listen to the comments that they make in the monthly or quarterly returns.

12. Culture
It is frequently said that CSA requires an organisation to have a very open culture where people see failure as the precursor to success and where people are not sacked as soon as they do something wrong. On that basis, one very large US consumer markets company (the largest in its subsector) refused to consider CSA. In fact, an open culture is not a prerequisite, the process can be introduced gradually if linked very tightly to provision of procedures and best practices. A reduced number of questions, say six, are asked specifically on the procedures provided. This gets the CSA process started and it can grow from there. A culture of ownership and accountability makes the job easier, but it is not necessarily impossible without it.

13. Too many questions
Systems with hundreds of questions tend to fail or quickly acquire a bad reputation with the users. One large UK organisation had 500 questions issued once a year and they tended to get a response rate of around 20%. Multinationals with similarly sized questionnaires have found that huge questionnaires do not encourage responsiveness or ownership.

14. Auditor suitability
The "client's" internal audit people take on a new role in CSA. They have to sell the tool and the programme to the users, they have to train them, implement the tool, encourage response, help facilitate knowledge sharing, and validate the results on a sample basis. This may be foreign to them and they may not like doing it. If they don't like doing it, they will do it badly and the implementation will be flawed; the users will not take to the system easily. It is important to make sure that the internal auditors involved in the project understand the process and are keen to deliver it.

15. Effective reporting
Reporting is the output of the whole system. The users need to see how they have done immediately they have completed the questionnaire, and they need to be able to compare this to how they did previously. The reporting templates need to show the big picture for the organisation and also be capable of drilling down into the detail. The reports should be meaningful and easy to use; they should show change from quarter to quarter and thus give 100% coverage on the assessment of controls every period. The reporting should also enable comparison between units and between processes.

16. Mapping
We start the programme by mapping the processes and identifying the key controls. After that we produce the content: the detailed questions, the way of answering the question, the action plan. The mapping can reveal control weaknesses at the very start which can be amended; it can identify controls that are redundant or difficult to perform. It can also reveal control gaps and procedural gaps. The opportunity here is to introduce a control or a procedure as part of the implementation of the CSA programme.

17. Weighting
Most systems allow weighting of responses to different questions. That is understandable but there are two problems. The first is that if the questionnaire is only testing key controls, then surely they should have the same weighting. We do not want to be asking for responses to minor control questions. The second point is that users often go back and try to check their overall score. If there are weightings used in the calculation of an overall score, then the calculation will be difficult to perform and the users will start to distrust the system.

18. Include risk
The system should include the risk being faced if the control is not in place, the risk that the control is managing. This lends to the importance of the control and also starts to raise the risk awareness of the users.

19. Link to best practice
The link to best practice is crucial, either through hyperlink or within the system itself in modified form. It gives a strong reason for the users to use the system and it generates dialogue and change for the better in terms of procedures and knowledge sharing.

20. Keep it up-to-date
If the questions become out of date owing to changes in the way things are done, or if the relevant procedures are also out of date, then the system loses credibility and will soon fall out of use. Keeping it up to date, and as the place to go to find out about new procedures and best practice, then it will be successful.

21. Address a need
A CSA programme is useful in itself. The choice of which area to address first is critical since to implement it in other areas, we build on previous success. Acceptance of the system is helped if the area that is being addressed first by the programme is an area of acknowledged need, an area where there is perhaps a problem that is not being successfully addressed at the moment.

22. Action plans
To get improvement, an action plan is required. Normally this is produced by the system for any questions where the response has shown a weakness. The action plans need to be drafted with care and this is frequently more difficult than it may appear. If someone is not performing a control and acknowledges this in the questionnaire, then the action plan is not simply "do it". The action plan needs to state how to improve, say by referring to training or to procedures, and then taking a course of action that directly addresses the weakness.

23. Specific questions and specific answers
For questions to be meaningful, they should be clearly stated and detail specifically how they should be answered. This is so that there is consistency in the answers and so that the results can lead to a specific improvement. Vague policy type questions do not tend to produce results that can be acted on effectively.

24. Bottom-line benefit
Many companies need to see a bottom-line benefit to a CSA programme. This can be difficult to produce unless it is worked out upfront in expectation of delivery of the benefit. There is case history to prove that a bottom-line benefit can be achieved. Improved controls can be shown by the reporting out of the CSA system but sometimes this is not convincing enough for the board and a bottom-line benefits case has to be prepared.

25. Other feedback
A return from various parts of the organisation is an opportunity to get other information, not just the comment sections attached to the control questions but other information as well such as specific operational data or customer satisfaction feedback perhaps on a service provided by head office. This has been done successfully for items of data that are not reported in normal management reporting but which are an issue at any point in time, and also on satisfaction with an outsourced payroll processor.

26. Link to bonus
Some companies like to make the CSA scores a factor in the calculation of management or staff bonuses. This can work and it can also lead to cheating on the scores. As long as cheating is dealt with appropriately in this context, then the linkage to bonus can work beneficially.

27. Sponsoring authors
Writing procedures to go with the programme is not sustainable long-term by the Internal Audit department. The role of Internal Audit with regard to sharing best practice and disseminating new procedures is that of facilitator, a sort of funnel between the various parts of the organisation where procedures may originate (the sponsoring authors) and the users who rely on them. Making sure this facilitation works, is a key success factor for the CSA programme.

28. Leverage
The way to leverage a CSA programme is to build on its infrastructure and expand the CSA process to other areas of operation. Once a CSA implementation is seen to be successful in an organisation and is talked about, other departments tend to queue up to implement their own CSA programme using the structure established in the first implementation.

29. Be flexible
There is no right or wrong method of scoring (1-10, ABC, Y/N, 123), no right or wrong timing of returns (monthly, quarterly, half-yearly), just the one that suits the company best given its culture and given what it is trying to achieve in the programme. Some companies like to diarise the completion of the questionnaire in small chunks, other like to add a procedural news bulletin at the sign-on to the system, funnelling all news about controls down this same channel. There is no one-size-fits-all solution.

30. Are they telling the truth?
Control self-assessment relies on users telling the truth. The results show the status of controls within the organisation and it is important to feel sure that these are accurate. Internal Audit tend to perform validations on the results by re-performing the questionnaire on a sample of returns where that is possible. The sample may be chosen at random or as a result of comparing the return to other known data that may indicate that the result is erroneous.

31. Senior management buy-in
If senior management don't buy in to the project, don't get enthusiastic about it, then you may have problems with the response rate. If senior management are interested, if they ask about it, talk about it, discuss it, then users see it as important. There are cases where the project has been faltering until it was realised that senior management did not actually understand the project; once they did, they were strongly in favour and the project became much more successful.

32. Response rate
We expect a response rate of over 98%. That is normal for a successful implementation. Senior management buy-in, as mentioned above, helps the process. In addition, checking the users have understood it shortly after the training, then reminders to the users toward the completion date, and also on the completion date will help make it happen. Non-responding users are then highlighted and reported to senior management. Usually that is enough to ensure very high response rates for system based control self-assessment programmes.

33. Project plan
If there is no project plan, or there is one but it is not used to control the project, the project can easily fail. There are several examples of this in the history of CSA where well-conceived CSA programmes have failed because the project manager has not stuck to the plan or has not had one.