07-21-11

Heuristics for Evaluating IT Security Management Tools (Paper 7)

Pooya Jaferian, University of British Columbia
Kirstie Hawkey, Dalhousie University
Andreas Sotirakopoulos, University of British Columbia
Maria Velez-Rojas, CA Technologies
Konstantin Beznosov, University of British Columbia

This paper arose from a struggle to evaluate the usability of IT Security Management (ITSM) tools. Recruiting actual IT managers for lab or field studies proved difficult, so the authors chose to use the “discount” usability evaluation technique of asking experts armed with heuristics to evaluate the tools.

For this process to work, you need good heuristics. Building on guidelines from a prior paper as well as HCI activity theory, the authors developed seven heuristics:

• Visibility of activity status
• History of actions and changes on artifacts
• Flexible representation of information
• Rules and constraints
• Planning and dividing work between users
• Capturing, sharing, and discovery of knowledge
• Verification of knowledge

To evaluate the heuristics, the authors set up a between-subjects study in which experts were asked to evaluate one tool using the new ITSM heuristics or with existing, non-domain-specific Nielsen’s heuristics. The authors then evaluated how successfully participants in each condition identified major and minor problems in the target tool.

Major results include:

• More high-severity problems were found using the new ITSM heuristics than with the Nielsen’s heuristics.
• The ITSM heuristics were rated as easy to learn, as easy to apply and as effective as Nielsen’s by the participants, all of whom had used Nielsen’s heuristics before.
• In general, comprehensively evaluating complex ITSM tools may require more evaluators than for simpler interfaces, to ensure full coverage.
• The ITSM and Nielsen’s heuristics are complementary and should be used together for maximum effectiveness.

Read the full paper at http://cups.cs.cmu.edu/soups/2011/proceedings/a7_Jaferian.pdf