|
|
Client Satisfaction Survey
| Final Report
|
FEASIBILITY
REPORT ON
A LABOUR
RELATIONS
BOARD CLIENT
SATISFACTION
SURVEY
Table
of Contents
EXECUTIVE
SUMMARY
FOUNDATIONS
OF THIS REPORT
CLIENT
SATISFACTION
SURVEYS - GENERAL
What
are They?
Purpose
The
Critical Questions
Comments
from the Auditor
General's Office
Comments
from Alberta
Treasury, Statistics
Comments
from Alberta
Labour, Information
Services
Agency
Surveys
Exceptions
Not
Cover The Substance
of Decisions
Not
Cover the Composition
of the Board
Methods
of Conducting
a Survey
Factors
to Consider
How
Often to Do a
Survey
EXPERIENCES
OF OTHERS
Alberta
Ontario
Federal
United
States
A
Sampling
AN
LRB CLIENT SATISFACTION
SURVEY
Purpose
Input
from the Management
Team
Factors
to Consider
Appendix
1 - Written Materials
Reviewed
I
was asked to
explore and report
on "the
possibilities
of ...(a client
satisfaction
survey)... with
some recommendations
about what could
be done ... to
advise (the Chair)
about matters
including possible
outside consultants
who could be
of assistance
(including their
availability
to fast-track
such a project
and the estimated
cost) and the
experience of
other tribunals
... to provide
(the Chair) with
a range of options
to consider."
Client
satisfaction
surveys are one
way of assessing
performance measurements
for an organization.
They can also
assist in building
stronger relationships
and implementing
change mechanisms.
The most effective
and useful surveys
are those that
address specific
versus general
comments and
assess process
oriented areas.
The general concept,
purpose and methodology
for client satisfaction
surveys are outlined
in the report.
A
quasi-judicial
tribunal needs
to consider the
elements of independence
and impartiality
when determining
whether to or
what to survey
clients about.
Such a tribunal,
such as the Labour
Relations Board,
should exclude
any survey questions
touching on the
substance of
the decisions
made by the tribunal
and the persons
appointed to
the tribunal
and comprising
any of its panels.
All
of the professionals
recommend the
Board address
and answer two
questions before
it decides to
engage in a survey.
If it cannot
answer these
two questions,
or decides it
cannot or will
not act on the
information obtained,
it should not
proceed with
a survey:
1.
What information
do we need to
gather from the
clients? (Why
are we doing
this?)
2.
What decisions
will we make
with the information
we gather? (Will
we act on the
results and make
changes?)
Included
in the report
are other items
for consideration,
both generally
and specifically
for the Labour
Relations Board.
Comments from
other tribunals
who have engaged
in the surveys,
as well as from
Alberta Treasury,
Statistics, the
Auditor General's
office, and Alberta
Labour, Information
Services are
set out.
Client
satisfaction
surveys have
become a common
instrument for
government in
Alberta, although
few, if any,
quasi-judicial
tribunals in
Alberta have
engaged in such
an exercise.
Across the country,
Ontario is the
province where
these measures
are next most
popular because
of the Agency
Reform initiatives
occurring there.
There is some
movement occurring
in the federal
sector as well.
The Manitoba
Labour Board
and the Public
Service Staff
Relations Board
are two labour
tribunals who
have undertaken
such a survey.
A sampling of
the initiatives
across the country
is included.
The
surveys which
I reviewed were
extremely varied,
as were the experiences
of the providers.
Cost varies with
the provider
and the methodology
selected. All
providers agreed
that the extent
of the survey,
both content
and number of
clients, would
affect the cost.
Recommendations
My
recommendations
are set out below.
Do
We Do A Survey?
From
all the research
I have done,
I recommend the
Board engage
in a process
oriented client
satisfaction
survey. Not only
is this a popular
instrument for
client interaction
at this time,
but it is one
which is new
to the Board's
clients and may
generate a higher
level of interaction
with us outside
of the application
process. There
is a high level
of enthusiasm
for the initiative
among both the
management team
and all the staff.
Although I was
asked not to
discuss the matter
in the community
or with the Board
members, I am
confident the
Board members
would enthusiastically
support the initiative
as well.
The
management team
is sufficiently
cohesive on the
purpose and content
of the survey,
as well as its
commitment to
act on the information
obtained. The
surveys done
by the Manitoba
Labour Relations
Board, the Ontario
Human Rights
Commission and
the Public Service
Staff Relations
Board cover similar
areas for input
as we would seek
and can be valuable
starting points
for us.
If
we engage in
a survey, we
should seriously
consider including
the Board members
and staff as
two distinct
client groups.
This may require
some modification
to the questionnaire
and a different
methodology for
these two groups,
but the comparison
information (internal
versus external
perspectives)
could be valuable
for a small addition
to the cost.
What
Methodology?
The
methodology will
be driven by
the number of
questions the
Board wishes
to ask, the required
sampling sizes
to obtain representative
samples and to
some extent,
the recommendations
of the chosen
provider to enable
us to accomplish
the goal in the
desired time
and budgetary
restraints. From
all the discussions
I have had to
date, I would
lean towards
a written or
telephone survey,
combined with
some personal
interviews or
focus groups.
As you will see
from the details,
there is a wide
range of options
on methodology
as well as content.
Next
Steps
Now
that this report
is complete,
the next steps
would be:
- By
February
1, 1999,
a decision
that we
will proceed
with a survey.
- Agreement
on the purposes
of the survey
(which will
drive the
content).
- Commitment
as to the
funding
to undertake
such a survey.
- Commitment
from the
management
team that
the Board
will act
upon the
information
obtained
in the survey.
- Draft
and send
out the
RFP to these
suppliers
to that
they can
provide
a more accurate
and detailed
tender.
(They will
need at
least one
week to
reply.)
- Deadline
for RFP's
to be February
16, 1999.
- Selection
of the provider
and signing
of contract.
- Establishment
of the survey
instrument
and methodology.
- Pre-testing
of the questionnaire
and amendment
as necessary.
- Administration
of the survey.
- Analysis
of the results
and preparation
of the reports.
- Presentation
of the final
report to
the management
team by
May 30,
1999.
Go
back to Top
In
preparing for
this report,
I reviewed some
written material
(see attached
list), spoke
to members of
the Department
of Labour, solicited
input from the
national board
of the Council
of Canadian Administrative
Tribunals, spoke
to some members
of Alberta's
tribunal community,
met and spoke
with representatives
of the Auditor
General's Department
and Alberta Treasury,
Statistics Branch,
met with the
Board's management
team, spoke with
potential providers,
spoke with representatives
of Ontario's
Agency Reform
Management Board
Secretariat and
representatives
of some of the
Ontario tribunals,
and spoke with
Chairs of the
Saskatchewan
and Manitoba
Labour Relations
Boards and representatives
of the BC Labour
Relations Board.
Bob Poburn assisted
me by doing some
research and
by attending
a seminar on
the topic presented
by a representative
of Alberta Treasury,
Statistics and
other investigations.
Go
back to Top
An
organization
typically will
assess its performance
measures in three
broad areas:
efficiency (objective
- dollars and
time), effectiveness
(whether the
intended public
impact has occurred
given the public
purposes of the
organization),
and quality (focus
on product -
how well the
service meets
the clients needs
it is designed
to satisfy).
Currently,
client satisfaction
is seen to be
one measure of
the quality of
performance and
can be used to
assess effectiveness
of an organization.
Although used
in private sector
business extensively,
governments have
recently adopted
"client
satisfaction"
as a key indicator
for use in business
planning exercises.
More recently,
this view has
been altered
to reflect the
perspective that
content specific
indications of
client satisfaction
are more useful
to the organization
that general
indications of
pleasure or displeasure.
Surveys
to assess client
satisfaction
range from the
very general
"how do
you like us approach"
to "do you
know about us"
to "how
well did we do
last time"
to specific "what
could be done
to improve the
mediation experience
- mark 1 - 7
that apply".
The nature, design,
focus and delivery
method of the
survey varies
with its sponsor
and its purpose.
Go
back to Top
Although
client satisfaction
surveys can be
used as information
gathering initiatives
for general knowledge,
it is an expensive
and time consuming
way to obtain
"feel good"
information.
In
a nutshell, client
satisfaction
surveys are better
used:
- as
instruments
of accountability
(for public
bodies it
is seen
to be one
method of
keeping
in touch
with the
community
which the
body serves
and of reporting
on the body's
ability
to meet
its established
standards
for service
by "assessing
the effectiveness,
efficiency
and quality
of service
against
stated objectives"
[Ont]),
- to
provide
specific
information
for use
internally
in setting
performance
targets
(standards
for service);
and
- to
provide
specific
information
for use
in developing
action plans
to improve
performance
(best practices)
- to
provide
specific
information
for use
in the allocation
of resources.
Optimumly,
client satisfaction
surveys should
be:
- directly
related
to performance
goals, best
practices
or operational
aspects
of the tribunal
- clear
and concise
- seek
specific
versus "feel
good"
information
- content
oriented
- re-usable
on a regular
basis with
the expectation
of receiving
comparable
information.
Response
rates to surveys
(particularly
mail-outs) will
be affected by:
- the
format of
the questionnaire
- the
timing of
the survey
- the
credibility
attached
to the survey
- the
nature of
the questions
asked (if
topical
and interesting
to client)
- the
link to
the benefits
for the
client
- assurances
of confidentiality.
Go
back to Top
It
has been repeatedly
stressed by the
professionals
who do this work
that an organization
should not undertake
a survey unless
it can answer
two critical
questions:
1.
What information
do we need to
gather from the
clients? (Why
are we doing
this?)
2.
What decisions
will we make
with the information
we gather? (Will
we act on the
results and make
changes?)
Go
back to Top
In
his 1997 Annual
Report, the Auditor
General included
some comments
about client
satisfaction
surveys (page
25). Recommendation
No. 7 in that
report says:
It
is recommended
that ministries
ensure their
client satisfaction
survey methods
produce valid
and reliable
results. It is
also recommended
that standards
be developed
for reporting
survey information.
He
concluded the
section by saying
he would prepare
additional information
on the use of
surveys in reporting
performance.
On the Auditor
General's website,
such a document
is posted, Client
Satisfaction
Surveys.
Contained
in these guidelines
are suggestions
for survey design,
execution and
reporting. Some
highlights of
those suggestions
include:
- survey
design is
critical
to its execution
and results
- surveys
should be
pre-tested
- surveys
should take
no more
than 30
minutes
to complete
- client
samples
must be
randomly
chosen amongst
the various
groups to
be polled
- reliability
of survey
results
should be
shown using
a confidence
level of
95% +/-
5%
- one
can increase
the response
rate by
sending
a letter
in advance
and providing
a contact
person as
well as
the name
of the survey
professional
- certain
information
should be
contained
in the report
to enable
the reader
to understand
how the
information
was collected
and compiled
and how
the results
relate to
the goals
of the organization.
Some
additional comments
by the Auditor
General's Office
were provided
in the form of
questions or
tips for the
Board to consider:
What
are you trying
to come to grips
with about the
relationship
you have to manage?
What
are you going
to do with the
information?
What happens
if you get negative
information?
(Once you get
it, you have
to do something
with it.)
If
a quasi -judicial
body is not able
to or willing
to change the
way it does business
because of independence
or impartiality
issues, it should
not engage in
a survey.
The
key is the intent/goals
of the survey.
Do the people
who interact
with us feel
well served?
It
is not necessary
to ask "Are
you satisfied",
rather you can
get the information
by asking more
specific questions.
To
satisfy the Auditor
General's office,
a tribunal needs
to be able to
disclose the
criteria which
it used to decide
what to survey
and who to survey.
If the sample
is too selective
(but without
criteria), it
is not a representative
sample.
The
Auditor General's
Office has offered
to review the
draft questionnaire
and provide comments
for us.
Go
back to Top
Alberta
Treasury outlines
three steps to
completing a
successful survey:
- plan
- must have
a clearly
defined
purpose
(eg. Be
designed
to improve
service
to the public)
and
designed
to support
all final
analysis
(identify
the information
needs, the
data collection
method,
the questionnaire
content,
the client
listing
and the
sample selection)
- development
- review
and use
existing
research
and surveys
where applicable;
decide to
use interview,
mail survey
or telephone
survey;
determine
the survey
content
which should
be subjective,
confirm
the customer
relationship,
avoid assumptions,
use clear,
concise
and consistent
wording,
continue
minimal
personal
bias, measure
a specific
concept
and allow
for various
degrees
of satisfaction
and input.
The format
of the survey
is important:
image is
critical,
group questions
into groups,
use white
space, balance
compactness
and legibility.
On the sample
selection,
size matters,
simple random
sampling
is best
or a systematic
sample or
a stratified
random sample,
and the
sample must
support
the level
of analysis
for the
final report.
A sample
pretest
of 10 -
12 clients
should be
done to
ask about
the cover
letter and
questionnaire
and to look
for patterns
in responses.
- analysis
- employ
accepted
testing
and analytical
procedures,
avoid unsupported
conclusions,
report with
precision,
report specific
limitations
and do not
exceed the
data limitations.
They
caution that
surveys can create
an unnecessary
expense for the
organization
and burden on
the respondents.
They can produce
inadequate or
stale data. They
also caution
that providers
must be carefully
instructed and
supervised to
avoid abdication
of the role by
the organization.
For the Board
and the consultant,
Treasury recommends
everyone spend
sufficient time
at the front
end thing about
what is desired
and how it will
be used so that
the appropriate
results are obtained
to support the
analysis desired,
which affects
the design of
the survey.
Alberta
Treasury urges
agencies to carefully
seek in the RFP
and examine in
the proposals
the specific
description of
methodologies
used in the analysis
and the types
of analysis to
be provides as
well as the techniques
used.
Go
back to Top
Alberta
Labour reinforced
much of what
was related to
us by other sources.
They were able
to provide assistance
with some specific
questions which
arose from the
general review.
On
the topic of
focus groups,
they confirmed
that results
from these interviews
are not statistically
reliable, however
they do provide
in-depth comments
of the nature
that the management
team is seeking.
It is common
practice for
the consultant
to pay an honorarium
to focus group
participants,
even on government
initiatives.
They suggested
that focus groups
may be a viable
way to supplement
a written or
telephone questionnaire.
It is a useful
tool for surveying
all of the lawyers
for example,
by holding firm
by firm focus
groups. Personal
interviews may
work for the
staff input.
Alberta
Labour suggested
our client size
of 1000 is manageable
enough to enable
us to survey
all 1000 clients
by either written
questionnaire
or by telephone.
They caution
that the telephone
questionnaire
will, by its
nature, be less
in-depth (a view
professed by
one of the providers
as well.)
Finally,
Alberta Labour
provided us some
suggestions and
comments on the
use of Requests
for Proposals
and how to evaluate
the proposals
we receive.
Go
back to Top
Eight
common goals
for tribunals,
as critical to
effective and
efficient performance
and service quality,
were identified
in Everyday
Justice,
the Report of
the Agency Reform
Commission on
Ontario's Regulatory
& Adjudicative
Agencies (April,
1998). At page
18, the commission
describes these
goals as:
1)
Fairness: The
provision of
service and performance
of statutory
functions in
an impartial,
lawful, unbiased
and just manner.
2)
Accessibility:
The ability to
provide information
and services
that are simple
and easy to use.
3)
Timeliness: The
performance of
tasks within
established time
frames based
on reasonable
expectations.
4)
Quality and Consistency:
The production
of accurate,
relevant, dependable,
understandable
and predictable
information and
results, with
no errors in
law or fact.
5)
Transparency:
The use of policies
and procedures
that are clear
and understandable
to everyone involved.
6)
Expertise: The
possession and
use of the skill,
knowledge and
technical competence
required to discharge
all statutory
responsibilities
and maintain
public confidence.
7)
Optimum Cost:
The provision
of services at
a cost that is
based on best
practices and
is cost effective
for everyone
involved.
8)
Courtesy: The
demonstration
of respect to
everyone who
come into contact
with the agency.
These
eight goals comprise
the general areas
within which
a tribunal may
seek client satisfaction
input relative
to its operations,
especially the
core business
functions.
Go
back to Top
Two
important qualifications
must be added
when addressing
client satisfaction
surveys for adjudicative
bodies, such
as the Board.
Those qualifications
are that any
survey:
1.
not cover the
substance of
the decisions
2.
not cover the
composition of
the Board.
Go
back to Top
The
Board is a quasi-judicial
tribunal which
must at all times
consider and
act to preserve
its independence
and impartiality.
Just like the
courts, the Board
cannot be seen
to be seeking
input from the
clients on the
results of our
adjudicative
efforts. The
process is adversarial
by design and,
therefore, we
can naturally
expect that one
party in every
dispute will
be unhappy with
the decision
- usually the
party who lost.
The survey content
and methodology
must not give
the impression
that the Board
will be swayed
by client opinion
when deciding
future cases;
to do so would
raise the spectre
of a reasonable
apprehension
of bias.
This
distinction was
recently recognized
in Everyday
Justice,
where at page
18 the Commission
notes:
The
commission recognizes
that it is important
to separate independence
of decision making
from agency performance
accountability.
Performance accountability
looks at how
the agency performs
its work, not
what decision
is made in any
case or cases.
One
way of assessing
the quality of
decision making
(that reasons
for decision
are high quality
and consistent
within cases
and adjudicators)
is to submit
decisions to
an external review
committee (who
have experience
and knowledge)
to review and
report to the
Chair on areas
for improvement.
This removes
the determinative
substance of
the decision
from scrutiny
and focuses on
the process aspects
which can be
addressed by
training on decision
writing, etc.
It also removes
the presumptive
conclusion (and
the corresponding
bias) that at
any time one
half of the parties
will be dissatisfied
with a particular
decision - because
they lost.
Go
back to Top
Surveying
clients about
their views on
the members a
tribunal could
also create a
reasonable apprehension
of bias, detrimentally
affecting the
independent decision
making authority
of the tribunal.
It could potentially
put the Chair
in a difficult
position when
assigning members
to sit cases
if the community
expressed strong
views about a
particular member.
The
judiciary has
recognized this
dilemma and the
debate in that
forum continues.
However, both
Manitoba and
Nova Scotia have
proceeded with
judicial evaluation
surveys. In Manitoba
the survey focussed
on the process,
not individual
judges. In Nova
Scotia, the reverse
occurred with
the focus on
individual performance,
but the results
were not published.
In
Ontario, the
Performance
Management
report (written
by two tribunal
adjudicators
and which is
currently being
implemented in
Ontario) emphasizes
the need to separate
performance measurement
of the agency
from that of
the individuals
within the agency,
as a result of
the requirement
of the individuals
to work independently
of government.
It
is more appropriate
to establish
and maintain
a separate appointment
process to ensure
that qualified,
competent and
credible decision
makers are appointed,
trained, and
retained. This
view was also
stated in the
Everyday
Justice
report at page
15. The Labour
Relations Board
currently has
an open, transparent
and accessible
process for the
appointment and
reappointment
of Board members.
We should not
undermine that
process by seeking
opinions about
appointed members.
The same can
be said to apply
to staff members.
Go
back to Top
Client
satisfaction
can be determined
by one or more
methods, such
as:
- statistical
review
- written
questionnaires
- telephone
interviews
- in
person interviews
- focus
groups
- any
combination
of the above.
Each
has its advantages
and disadvantages.
All methods depend
on the purpose
for doing the
survey and the
quality of the
questions being
asked to elicit
productive results.
Cost and time
increases as
you move from
statistical reviews
to focus other
methods. In each
case (outside
of statistical
review) the cost
and time associated
with the design
of the questionnaire
and the analysis
and report are
about the same.
The cost varies
in the field
costs of the
survey instrument.
In
each case, the
providers indicate
that the time
leading up to
the production
of the survey
questions can
be time consuming
(which adds to
the time for
the survey).
This time can
be reduced if
the agency is
focussed on the
purpose and content
of the survey.
A
statistical review
could be conducted
using the information
available in
the Board's data
base. It would
reveal time frames
for processing
matters, costs
associated by
case, and other
information such
as the number
of clients using
legal representatives,
the number of
cases of a particular
type processed
etc. It is essentially
what we now do
in the annual
report. Such
a review, however,
reveals little
or no information
about client
reaction to such
things as communication
efforts, officer
assistance, ADR
efforts, etc.
Books
have been written
about how to
design a successful
written questionnaire.
The words of
John K. Norton
are worthwhile
keeping in mind
when considering
this method.
"The time
of busy people
is sometimes
wasted by time-consuming
questionnaires
dealing with
inconsequential
topics, worded
so as to lead
to worthless
replies, and
circulated by
untrained and
inexperienced
individuals,
lacking in facilities
for summarizing
and disseminating
any worthwhile
information which
they may obtain."
In other words,
be focussed,
concise and clear
about what you
want answered.
Telephone
and in-person
interviews take
less time, as
do focus groups,
and elicit more
detailed answers
and immediate
follow-up. Obviously,
they generate
a higher rate
of return because
of the personal
contact. These
three methods
require fewer
completed surveys
to obtain results
that can be viewed
with confidence.
Normally,
persons who attend
focus groups
are given an
honorarium (often
$50.00) for their
time. This would
appear to be
a questionable
practice for
an adjudicative
tribunal however
the rate of participation
may depend on
it. Honorariums
would add to
the cost of such
a method. Alberta
Treasury advises
that focus groups
are normally
done in conjunction
with one of the
other methods,
not alone. In
the case of our
client group
size (1000),
they caution
that it would
be difficult
to sample sufficient
clients through
focus groups
alone to obtain
reliable results.
Go
back to Top
Regardless
of the methodology
selected, the
literature and
my interviews
reveal that all
successful surveys
address the following
criteria:
- freedom
from bias
- this often
translates
into someone
outside
the tribunal
conducting
the survey
to remove
potential
bias from
the questions
asked and
the interpretation
of the results.
- validity
- the ability
of the survey
to measure
what it
purports
to measure.
Essentially,
the survey,
to be meaningful,
must measure
satisfaction
against
established
tribunal
goals.
- reliability
- the survey
must poll
sufficient
clients
with a sufficiently
high rate
of return
that the
results
can be interpreted
to be a
real and
true expression
of client
views. If
less than
the full
client base
is surveyed,
the sample
selected
must be
representative
of those
who actually
use the
service.
The
Auditor General
of Alberta recommends
that a survey
on general satisfaction
receive a 60%
return rate to
enable the author
to draw meaningful
conclusions.
The professionals
who do this work
advised that
to obtain a 95%
reliability with
a + / - 4.8%
error rate (which
is considered
acceptable) within
a population
of 1000 clients
(our potential),
we would need
to obtain at
least 400 completed
surveys. This
number would
increase if we
had distinct
segments within
the audience
that we wished
to poll and obtain
confidence in
the segment results.
Additional
factors to consider
are:
- cost
- ability
to contact
the clients
(more difficult
with in
person interviews
and focus
groups)
- how
many clients
can be contacted
(anything
more than
a mailed
questionnaire
can be limiting)
- whether
clients
need to
be stratified
or grouped
for input
and analysis
- ease
of completion
(it is easiest
to fill
out a form
on your
own time)
- ease
of tabulation
(written,
quantitative
questionnaires
are easiest
to tabulate)
- uniformity
(interviewers
may alter
the question)
- time
needed to
complete
the survey
- how
the survey
results
will be
published
and used.
Go
back to Top
How
often varies,
but the key is
do to one regularly
after the first
survey is complete.
Repeating the
same survey assists
the tribunal
in assessing
whether the standards
it adopted and
the changes it
made were effective.
The Auditor General's
office spoke
of one organization
that is doing
a survey every
2 years using
an outside provider.
This enables
them to obtain
good and consistent
information in
a less costly
way and gives
them the necessary
time in between
surveys to implement
change.
If
the results of
this survey confirm
what the Board
is already able
to extrapolate
from our administrative
reports, we may
find it more
useful to engage
in sampling focus
groups in the
future. This
would enable
the Board to
obtain the detail
comments and
suggestions,
rather than statistical
trends.
Go
back to Top
Eight
departments reported
client satisfaction
survey results
in their 1997-98
annual reports.
The Auditor General
predicted that
12 would do so
this fiscal year.
Few of the administrative
tribunals have
engaged in client
satisfaction
surveys. The
WCB is one exception,
but I understand
that survey did
not extend to
the Appeals Commission.
Both Alberta
Treasury and
the Auditor General's
office have commented
on the surveys
conducted by
the Departments
and have gone
so far as to
cooperatively
prepare and publish
some guidelines
about surveys.
However, neither
of them had been
asked to consider
the different
circumstances
arising from
surveys by or
for quasi-judicial
administrative
tribunals.
Go
back to Top
As
a result of the
Everyday
Justice
report in Ontario,
all Ontario agencies
will be required
to adopt a process
for performance
measurement by
March, 1999 including
the use of annual
client satisfaction
surveys. Janet
Skelton of the
Ontario Agency
Reform Management
Board Secretariat
says the Commission's
recommendations
were accepted
by Cabinet and
are being implemented.
All agencies
have completed
at least the
first draft of
their performance
measures. Many
have included
some type of
client sampling
in those measures,
although the
Secretariat is
leaving it to
the Auditor to
comment on the
quality of the
client surveys
or how they should
be conducted.
Go
back to Top
A
similar approach
is being taken
in the federal
jurisdiction,
as a result of
requirements
that federal
agencies now
report to Parliament
in the spring
(their planning
document) and
in the fall (their
performance reporting)
every year. These
agencies are
to set up performance
indicators for
reporting, including
statistical reports
and client satisfaction
surveys, although
it is not clear
that surveys
are a required
indicator.
Go
back to Top
Although
popular in the
US Federal government
sphere as well,
at least one
paper, Customer
Satisfaction
Measurement Issues
in the Federal
Government
by Tracey R.
Wellens and Elizabeth
A. Martin, has
been written
discussing the
uses and limitations
of this type
of measurement
in a government
setting. The
same concerns
are identified
in the US as
exist here about
the quality of
the survey and
the reliability
of the results.
Go
back to Top
Provincial
agencies in Manitoba
and Alberta have
voluntarily used
client satisfaction
surveys in either
general or specific
ways.
A
sampling of some
of these Canadian
national and
provincial initiatives
include:
- the
Public
Service
Staff Relations
Board
- in 1998
used a written
survey to
seek input
on its entire
operations.
Chair Tarte
advises
his tribunal
used Consulting
and Audit
Canada to
complete
the survey.
It took
about six
months from
start to
finish.
CAC met
with the
Chair to
review the
scope of
the survey,
designed
the questions,
sampled
a group
to test
the validity
of the questions,
conducted
a focus
group on
the questions,
administered
the survey
by mail,
analysed
the results
and prepared
a written
report.
The PSSRB
did not
see the
returned
surveys
to protect
confidentiality.
The Chair
began the
process
himself
by personally
meeting
with the
heads of
all the
unions and
employers
to seek
input into
the Board's
operations.
He indicated
the results
of the survey
were predictable,
but have
also been
helpful
in several
planning
and budgeting
areas. For
example,
as a result
of the conclusion
that there
needed to
be more
consistency
in the Board's
decisions,
he has prioritized
a training
initiative
for board
members.
- the
Ontario
Human Rights
Commission
- in 1997
used a written
questionnaire
given to
each participant
to assess
its mediation
initiative.
The Commission
has an in-house
statistician
who oversaw
the survey
and analysed
the results
which was
all designed
and delivered
in-house.
The Commission
was very
pleased
with the
results
it obtained
and was
able to
use them
to modify
its mediation
system.
It began
an extensive
training
program
for its
staff in
this area
and in the
intake area
which, when
combined
with a new
computerized
intake system,
has resulted
in significant
changes
and productivity
in the organization.
They handle
about 2000
new cases
per year
and currently
have a backlog
of 2500
cases which
they hope
to eliminate
by 2000.
The
Commission has
just contracted
a firm to conduct
a quality assurance
review on all
aspects of the
organization.
It began with
an internal focus
and has produced
a preliminary
report with recommendations
to re-engineer
the operational
side of the organization.
They anticipate
doing a targeted
client satisfaction
survey at some
point in this
process as well.
- the
Ontario
Environmental
Assessment
Board
- is currently
drafting
a client
satisfaction
survey.
- the
Canadian
Transportation
Agency
- employed
an outside
firm, EKOS,
in 1998
to conduct
a survey
relating
to the Act,
including
areas of
client satisfaction.
The first
results
were obtained
December
21, 1998.
- the
Alberta
Environmental
Appeal Board
- in 1997
used a single
question
to solicit
comments
on its Rules
and Procedures.
(This is
not uncommon
to the type
of practice
the LRB
has used
in the past
and is currently
using on
the change
to the witnesses'
oath procedure.)
- the
Manitoba
Injury Compensation
Appeals
Commission
- in 1998
used a written
questionnaire
to seek
client input
on the treatment
of clients,
its communication
efforts,
and the
timing and
processing
of matters
to hearing.
- the
Alberta
Workers'
Compensation
Board
- in 1997
conducted
a client
survey which
included
telephone
surveys,
focus groups,
interviews
and responses
to background
papers.
It is currently
undergoing
a Policy
Consultation
process
which it
calls the
largest
public review
in its history.
- the
Alberta
Labour Employment
Standards
Branch -
in
August,
1998 used
Coopers
& Lybrand
to assist
it in conducting
a review
of the Regulation.
The final
report reveals
that the
Department
set the
overall
scope and
focus of
the project,
identified
the issues
and developed
the specific
questions,
upon which
C &
L then provided
input. The
Department
administered
the survey
and C &
L input
and categorized
the data
and provided
the summary
of results
and a report.
- the
Manitoba
Labour Relations
Board
- conducted
its own
survey in
1995 in
response
to resource
shortages.
The Board
used the
Department's
research
staff to
assist with
the survey
design and
analysis.
It mailed
out questionnaires
to all the
regular
community
- employers,
unions and
counsel,
but did
not include
one time
clients.
They received
a 39% overall
response
with very
satisfactory
responses
to even
very specific
questions.
Go
back to Top
If
it is to engage
in a client satisfaction
survey, the most
important thing
the Board must
determine is
the purpose for
conducting such
a survey. It
appears that
both the Auditor
General's Department
and Alberta Treasury
agree that a
tribunal (or
a department)
should not engage
in such an initiative
unless there
is a process
related purpose
to it. "Feel
good" questions
or general "how
are we doing"
questions are
expensive and
far less productive
and meaningful
for a tribunal
such as the Board,
although they
may be more applicable
in a different
context and for
a different user.
Currently,
the Board has
an extremely
strong and reliable
way of compiling
information which
can assist us
in assessing
performance and
in establishing
best practices
-- the data base
and the reports
generated from
it. From that
information,
we can draw conclusions
about performance
in many of the
eight areas mentioned
in the Everyday
Justice
report. For example,
we can draw conclusions
about fairness
based on the
number of objections
filed and the
number of reconsiderations
and judicial
reviews. We can
assess timeliness
by comparing
actual processing
times to those
standards we
have set. The
cost can be broken
down by case
and compared
to previous years.
What
we cannot conclude
or can at best
guess at are
those things
within the client's
knowledge. For
example, are
the materials
we produce easily
accessible, understandable
and useable?
How and where
do clients access
the materials?
How do the time
frames which
we have set as
standards in
various matters
relate to the
client's involvement
in the matter?
Are they too
lenient or too
ambitious? Have
we maintained
public confidence?
What drives that
view? What costs
does the client
incur in appearing
before the Board?
Are those costs
reasonable? What
can the Board
do to make its
procedures "cost
effective for
everyone?"
Do the clients
believe they
are being treated
with respect
and courtesy?
What can we do
to improve that?
These are the
types of information
a survey could
expose.
The
Board's operation
has two focuses
which could drive
the purpose of
a survey -- the
business plan
and the development
of internal best
practices. Although
the latter may
be reflected
to some degree
in the business
plan, it is more
comprehensive
in both scope
and detail. The
business plan,
if we are to
continue to follow
the model expected
by the government,
is concise and
limited to about
four performance
measures which
cover the general
scope of the
Board's role.
Any
Board sponsored
survey should
have as its purpose
tied to either
or both of these
functions. That
will also assist
in focussing
the areas of
input and the
questions into
process or content
related topics
which will be
more productive
for the Board.
Go
back to Top
After
meeting with
the Board's management
team, it appears
that they see
the purpose of
such a survey
could be to:
- better
utilize
and balance
our resources
- plan
and budget
for needed
resources
- assist
the Board
in determining
performance
measures
within the
business
plan
- assist
the Board
in developing
best practices
- to
assist the
Board in
identifying
training
needs
- to
assist the
Board in
selling
change to
the staff
and caucus.
In
addition to wanting
to know where
the problem/
potential change
areas are, the
management team
would like to
know how extensive
the problem is:
is it a one time
event or a recurring
situation? Is
it experienced
by a one time
client or a regular
client? They
would like to
hear suggestions
for change or
improvement.
The management
team would also
like to be able
to see an analysis
of the results
by category of
client and who
they represent,
by frequency
of client interaction
with the Board,
and by industry
or sector group.
These would affect
the initial design
of the survey.
Go
back to Top
When
considering whether
and how to conduct
a client satisfaction
survey for the
Board, some factors
are:
- the
costs of
any survey.
- the
survey needs
to be underway
(and if
possible,
complete)
by March
31, 1999
- the
results
should be
independently
validatable
or supportable
from other
sources,
such as
the annual
report or
the data
base reports.
- the
areas to
be surveyed
should be
meaningful
and useful
to the Board
for the
purposes
of establishing:
- benchmarks
towards
best
practices,
- priorities
for accounting
and budgeting
- performance
goals in
the business
plan , or
- identifying
areas for
improvement
in the Board
procedures
- identifying
ways to
better utilize
Board resources.
- the
potential
clients
should include
all those
who regularly
appear before
the Board
[available
on the data
base] plus
all those
who had
contact
with the
Board in
the last
six months
to one year
[again available
on the data
base]. Persons
who contact
the Board
because
they are
seeking
Employment
Standards
or the Labourers'
Union should
not be contacted
as they
do not fall
within the
mandated
service
area of
the Board.
It would
also be
impossible
to identify
information
callers
unless they
subsequently
filed an
application
with the
Board.
The
broadest area
of "general
clients"
would be the
members of any
of the unions
or employees
of any of the
employers who
have bargaining
relationships
under the Board's
jurisdiction.
In order to identify
even a representative
sampling of this
group, we would
need the cooperation
of the employers
and the unions.
Some of those
groups may face
FOIP considerations
and be unable
to release the
information to
us. Given the
time and limitations
of cost, this
is probably too
wide a scope
for input.
- the
survey results
should be
published
- probably
on the Board's
website
- and a
summary
provided
to every
client who
participated
in the survey.
- the
survey results
could be
used to
adjust the
Board's
business
plan and
budget objectives,
to develop
best practices
and to adjust
the Board's
procedures
and publications.
If the survey
is to proceed,
this means
that the
management
team has
to make
a commitment
to using
the results
which are
obtained.
- the
survey should
produce
an analysis
of the results,
rather than
just a summary
of them.
From the
discussions
both Bob
Poburn and
I engaged
in with
the various
critics,
it became
clear that
this is
an area
that often
produces
dissatisfaction
with the
final product.
While it
is not difficult
to summarize
the results
obtained
(we did
such a summary
of the evaluation
of the Health
Care Conference),
it is more
difficult
to analyse
and compare
the various
responses
to produce
meaningful
information
and recommendations
for action.
The PSSRB
survey report
appears
to produce
the most
comprehensive
analysis
of any that
I observed.
- The
management
team discussed
the possibility
of including
the Board
members
and the
staff as
two distinct
groups of
"clients"
because
of their
"inside
perspective."
This would
further
enable us
to compare
the internal
perception
with the
external
perception
and address
changes
which need
to be made
there. As
a word of
caution,
it is likely
the survey
will have
an ancillary
impact on
the staff
and Board
members,
if the management
team does
not act
on the results,
as it will
likely decrease
the confidence
of those
persons
in the credibility
of the Board
(and the
management
team) once
the expectation
of action
has been
created.
- An
additional
caution
addresses
the manner
of selection
of the clients
who will
participate
related
to the credibility
of the survey.
There are
some clients
who are
so regularly
before the
Board that
to exclude
them would
be to create
the appearance
of not seeking
a valid
sampling
of results.
This is
particularly
so with
the lawyers.
One option
may be to
survey all
the lawyers
who appear
before the
Board as
it is a
relatively
small yet
easily definable
group.
Go
back to Top
APPENDIX 1 -
WRITTEN MATERIALS
REVIEWED
Everyday
Justice,
Report of
the Reform
Commission
on Ontario's
Regulatory
& Adjudicative
Agencies
April, 1998
Performance
Management
for Ontario's
Adjudicative
and Regulatory
Agencies,
Agency Reform
Management
Board Secretariat,
September,
1998
Clients
Give Agriculture
Staff Top
Marks for
Service
Connexus,
Summer,
1998
Evaluating
Judicial
Evaluation,
Canadian
Lawyer,
March, 1998
Alberta
Employers:
Shaping
WCB Policies,
Profile,
Fall, 1998
Measuring
Customer
Satisfaction
(Survey
design,
use and
statistical
analysis
methods)
2nd edition,
Bob E. Hayes,
(1998, ASQ
Quality
Press)
Questionnaires:
Design and
Use,
Douglas
R. Berdie
and John
F. Anderson
(1974, The
Scarecrow
Press, Inc.)
Administrative
Law: Reigning
in Unruly
Fiefdoms
or Threatening
Independence?
Agency Reform
in Ontario:
Perspectives
on Independence
and Accountability
Brian
Goodman,
prepared
for the
Current
Developments
in Administrative
and Employment
Law Conference
Public
Service
Staff Relations
Board Client
Satisfaction
Survey,
March 1998
Evaluation
of the Environmental
Appeal Board,
1997 Stakeholder
Consultation
1997
Service
Satisfaction
Survey:
Final Report
August,
1997
Manitoba
Automobile
Injury Compensation
Appeal Commission
Survey
Annual
Report of
the Auditor
General
of Alberta
1997-98
Review
of Employment
Standards
Regulation
Alberta
Labour,
August,
1998
Alberta
Labour Relations
Board Business
Plan,
1999 - 2001
Manitoba
Labour Board
Client Survey,
October,
1995
Customer
Satisfaction
Measurement
Issues in
the Federal
Government,Tracey
R. Wellens
and Elizabeth
A. Martin
Increasing
Profits
with Customer
Surveys,
James Sauer
1999, http://expert-market.com/client/seminars/strategicperformance-s.html
Customer
Surveys,
inforsurv,
www.infosurv.com/cusotmers_surveys.htm
Competitive
Benchmarking
Associates,
www.competitiveanalysis.com/contents.htms
Client
Satisfaction
Surveys,
Auditor
General
of Alberta,
October,
1998
Trial
by Survey,
The Canadian
Bar Association
National,
October,
1997
Evaluating
Customer
Comments,
The
Business
Research
Lab, http://strategis.ic.gc.ca
Engineering
Consultant
Selection:
Proposal
Evaluation
Criteria
Reference
Guide, Alberta
Transportation
and Utilities
Go
back to Top
|
