|
|
Client Satisfaction Survey
| Final Report
|
FEASIBILITY
REPORT ON
A LABOUR
RELATIONS
BOARD CLIENT
SATISFACTION
SURVEY
Table
of Contents
EXECUTIVE
SUMMARY
FOUNDATIONS
OF THIS REPORT
CLIENT
SATISFACTION
SURVEYS - GENERAL
What
are They?
Purpose
The
Critical Questions
Comments
from the Auditor
General's Office
Comments
from Alberta
Treasury, Statistics
Comments
from Alberta
Labour, Information
Services
Agency
Surveys
Exceptions
Not
Cover The Substance
of Decisions
Not
Cover the Composition
of the Board
Methods
of Conducting
a Survey
Factors
to Consider
How
Often to Do a
Survey
EXPERIENCES
OF OTHERS
Alberta
Ontario
Federal
United
States
A
Sampling
AN
LRB CLIENT SATISFACTION
SURVEY
Purpose
Input
from the Management
Team
Factors
to Consider
Appendix
1 - Written Materials
Reviewed
I
was asked to
explore and report
on "the
possibilities
of ...(a client
satisfaction
survey)... with
some recommendations
about what could
be done ... to
advise (the Chair)
about matters
including possible
outside consultants
who could be
of assistance
(including their
availability
to fast-track
such a project
and the estimated
cost) and the
experience of
other tribunals
... to provide
(the Chair) with
a range of options
to consider."
Client
satisfaction
surveys are one
way of assessing
performance measurements
for an organization.
They can also
assist in building
stronger relationships
and implementing
change mechanisms.
The most effective
and useful surveys
are those that
address specific
versus general
comments and
assess process
oriented areas.
The general concept,
purpose and methodology
for client satisfaction
surveys are outlined
in the report.
A
quasi-judicial
tribunal needs
to consider the
elements of independence
and impartiality
when determining
whether to or
what to survey
clients about.
Such a tribunal,
such as the Labour
Relations Board,
should exclude
any survey questions
touching on the
substance of
the decisions
made by the tribunal
and the persons
appointed to
the tribunal
and comprising
any of its panels.
All
of the professionals
recommend the
Board address
and answer two
questions before
it decides to
engage in a survey.
If it cannot
answer these
two questions,
or decides it
cannot or will
not act on the
information obtained,
it should not
proceed with
a survey:
1.
What information
do we need to
gather from the
clients? (Why
are we doing
this?)
2.
What decisions
will we make
with the information
we gather? (Will
we act on the
results and make
changes?)
Included
in the report
are other items
for consideration,
both generally
and specifically
for the Labour
Relations Board.
Comments from
other tribunals
who have engaged
in the surveys,
as well as from
Alberta Treasury,
Statistics, the
Auditor General's
office, and Alberta
Labour, Information
Services are
set out.
Client
satisfaction
surveys have
become a common
instrument for
government in
Alberta, although
few, if any,
quasi-judicial
tribunals in
Alberta have
engaged in such
an exercise.
Across the country,
Ontario is the
province where
these measures
are next most
popular because
of the Agency
Reform initiatives
occurring there.
There is some
movement occurring
in the federal
sector as well.
The Manitoba
Labour Board
and the Public
Service Staff
Relations Board
are two labour
tribunals who
have undertaken
such a survey.
A sampling of
the initiatives
across the country
is included.
The
surveys which
I reviewed were
extremely varied,
as were the experiences
of the providers.
Cost varies with
the provider
and the methodology
selected. All
providers agreed
that the extent
of the survey,
both content
and number of
clients, would
affect the cost.
Recommendations
My
recommendations
are set out below.
Do
We Do A Survey?
From
all the research
I have done,
I recommend the
Board engage
in a process
oriented client
satisfaction
survey. Not only
is this a popular
instrument for
client interaction
at this time,
but it is one
which is new
to the Board's
clients and may
generate a higher
level of interaction
with us outside
of the application
process. There
is a high level
of enthusiasm
for the initiative
among both the
management team
and all the staff.
Although I was
asked not to
discuss the matter
in the community
or with the Board
members, I am
confident the
Board members
would enthusiastically
support the initiative
as well.
The
management team
is sufficiently
cohesive on the
purpose and content
of the survey,
as well as its
commitment to
act on the information
obtained. The
surveys done
by the Manitoba
Labour Relations
Board, the Ontario
Human Rights
Commission and
the Public Service
Staff Relations
Board cover similar
areas for input
as we would seek
and can be valuable
starting points
for us.
If
we engage in
a survey, we
should seriously
consider including
the Board members
and staff as
two distinct
client groups.
This may require
some modification
to the questionnaire
and a different
methodology for
these two groups,
but the comparison
information (internal
versus external
perspectives)
could be valuable
for a small addition
to the cost.
What
Methodology?
The
methodology will
be driven by
the number of
questions the
Board wishes
to ask, the required
sampling sizes
to obtain representative
samples and to
some extent,
the recommendations
of the chosen
provider to enable
us to accomplish
the goal in the
desired time
and budgetary
restraints. From
all the discussions
I have had to
date, I would
lean towards
a written or
telephone survey,
combined with
some personal
interviews or
focus groups.
As you will see
from the details,
there is a wide
range of options
on methodology
as well as content.
Next
Steps
Now
that this report
is complete,
the next steps
would be:
- By
February
1, 1999,
a decision
that we
will proceed
with a survey.
- Agreement
on the purposes
of the survey
(which will
drive the
content).
- Commitment
as to the
funding
to undertake
such a survey.
- Commitment
from the
management
team that
the Board
will act
upon the
information
obtained
in the survey.
- Draft
and send
out the
RFP to these
suppliers
to that
they can
provide
a more accurate
and detailed
tender.
(They will
need at
least one
week to
reply.)
- Deadline
for RFP's
to be February
16, 1999.
- Selection
of the provider
and signing
of contract.
- Establishment
of the survey
instrument
and methodology.
- Pre-testing
of the questionnaire
and amendment
as necessary.
- Administration
of the survey.
- Analysis
of the results
and preparation
of the reports.
- Presentation
of the final
report to
the management
team by
May 30,
1999.
Go
back to Top
In
preparing for
this report,
I reviewed some
written material
(see attached
list), spoke
to members of
the Department
of Labour, solicited
input from the
national board
of the Council
of Canadian Administrative
Tribunals, spoke
to some members
of Alberta's
tribunal community,
met and spoke
with representatives
of the Auditor
General's Department
and Alberta Treasury,
Statistics Branch,
met with the
Board's management
team, spoke with
potential providers,
spoke with representatives
of Ontario's
Agency Reform
Management Board
Secretariat and
representatives
of some of the
Ontario tribunals,
and spoke with
Chairs of the
Saskatchewan
and Manitoba
Labour Relations
Boards and representatives
of the BC Labour
Relations Board.
Bob Poburn assisted
me by doing some
research and
by attending
a seminar on
the topic presented
by a representative
of Alberta Treasury,
Statistics and
other investigations.
Go
back to Top
An
organization
typically will
assess its performance
measures in three
broad areas:
efficiency (objective
- dollars and
time), effectiveness
(whether the
intended public
impact has occurred
given the public
purposes of the
organization),
and quality (focus
on product -
how well the
service meets
the clients needs
it is designed
to satisfy).
Currently,
client satisfaction
is seen to be
one measure of
the quality of
performance and
can be used to
assess effectiveness
of an organization.
Although used
in private sector
business extensively,
governments have
recently adopted
"client
satisfaction"
as a key indicator
for use in business
planning exercises.
More recently,
this view has
been altered
to reflect the
perspective that
content specific
indications of
client satisfaction
are more useful
to the organization
that general
indications of
pleasure or displeasure.
Surveys
to assess client
satisfaction
range from the
very general
"how do
you like us approach"
to "do you
know about us"
to "how
well did we do
last time"
to specific "what
could be done
to improve the
mediation experience
- mark 1 - 7
that apply".
The nature, design,
focus and delivery
method of the
survey varies
with its sponsor
and its purpose.
Go
back to Top
Although
client satisfaction
surveys can be
used as information
gathering initiatives
for general knowledge,
it is an expensive
and time consuming
way to obtain
"feel good"
information.
In
a nutshell, client
satisfaction
surveys are better
used:
- as
instruments
of accountability
(for public
bodies it
is seen
to be one
method of
keeping
in touch
with the
community
which the
body serves
and of reporting
on the body's
ability
to meet
its established
standards
for service
by "assessing
the effectiveness,
efficiency
and quality
of service
against
stated objectives"
[Ont]),
- to
provide
specific
information
for use
internally
in setting
performance
targets
(standards
for service);
and
- to
provide
specific
information
for use
in developing
action plans
to improve
performance
(best practices)
- to
provide
specific
information
for use
in the allocation
of resources.
Optimumly,
client satisfaction
surveys should
be:
- directly
related
to performance
goals, best
practices
or operational
aspects
of the tribunal
- clear
and concise
- seek
specific
versus "feel
good"
information
- content
oriented
- re-usable
on a regular
basis with
the expectation
of receiving
comparable
information.
Response
rates to surveys
(particularly
mail-outs) will
be affected by:
- the
format of
the questionnaire
- the
timing of
the survey
- the
credibility
attached
to the survey
- the
nature of
the questions
asked (if
topical
and interesting
to client)
- the
link to
the benefits
for the
client
- assurances
of confidentiality.
Go
back to Top
It
has been repeatedly
stressed by the
professionals
who do this work
that an organization
should not undertake
a survey unless
it can answer
two critical
questions:
1.
What information
do we need to
gather from the
clients? (Why
are we doing
this?)
2.
What decisions
will we make
with the information
we gather? (Will
we act on the
results and make
changes?)
Go
back to Top
In
his 1997 Annual
Report, the Auditor
General included
some comments
about client
satisfaction
surveys (page
25). Recommendation
No. 7 in that
report says:
It
is recommended
that ministries
ensure their
client satisfaction
survey methods
produce valid
and reliable
results. It is
also recommended
that standards
be developed
for reporting
survey information.
He
concluded the
section by saying
he would prepare
additional information
on the use of
surveys in reporting
performance.
On the Auditor
General's website,
such a document
is posted, Client
Satisfaction
Surveys.
Contained
in these guidelines
are suggestions
for survey design,
execution and
reporting. Some
highlights of
those suggestions
include:
- survey
design is
critical
to its execution
and results
- surveys
should be
pre-tested
- surveys
should take
no more
than 30
minutes
to complete
- client
samples
must be
randomly
chosen amongst
the various
groups to
be polled
- reliability
of survey
results
should be
shown using
a confidence
level of
95% +/-
5%
- one
can increase
the response
rate by
sending
a letter
in advance
and providing
a contact
person as
well as
the name
of the survey
professional
- certain
information
should be
contained
in the report
to enable
the reader
to understand
how the
information
was collected
and compiled
and how
the results
relate to
the goals
of the organization.
Some
additional comments
by the Auditor
General's Office
were provided
in the form of
questions or
tips for the
Board to consider:
What
are you trying
to come to grips
with about the
relationship
you have to manage?
What
are you going
to do with the
information?
What happens
if you get negative
information?
(Once you get
it, you have
to do something
with it.)
If
a quasi -judicial
body is not able
to or willing
to change the
way it does business
because of independence
or impartiality
issues, it should
not engage in
a survey.
The
key is the intent/goals
of the survey.
Do the people
who interact
with us feel
well served?
It
is not necessary
to ask "Are
you satisfied",
rather you can
get the information
by asking more
specific questions.
To
satisfy the Auditor
General's office,
a tribunal needs
to be able to
disclose the
criteria which
it used to decide
what to survey
and who to survey.
If the sample
is too selective
(but without
criteria), it
is not a representative
sample.
The
Auditor General's
Office has offered
to review the
draft questionnaire
and provide comments
for us.
Go
back to Top
Alberta
Treasury outlines
three steps to
completing a
successful survey:
- plan
- must have
a clearly
defined
purpose
(eg. Be
designed
to improve
service
to the public)
and
designed
to support
all final
analysis
(identify
the information
needs, the
data collection
method,
the questionnaire
content,
the client
listing
and the
sample selection)
- development
- review
and use
existing
research
and surveys
where applicable;
decide to
use interview,
mail survey
or telephone
survey;
determine
the survey
content
which should
be subjective,
confirm
the customer
relationship,
avoid assumptions,
use clear,
concise
and consistent
wording,
continue
minimal
personal
bias, measure
a specific
concept
and allow
for various
degrees
of satisfaction
and input.
The format
of the survey
is important:
image is
critical,
group questions
into groups,
use white
space, balance
compactness
and legibility.
On the sample
selection,
size matters,
simple random
sampling
is best
or a systematic
sample or
a stratified
random sample,
and the
sample must
support
the level
of analysis
for the
final report.
A sample
pretest
of 10 -
12 clients
should be
done to
ask about
the cover
letter and
questionnaire
and to look
for patterns
in responses.
- analysis
- employ
accepted
testing
and analytical
procedures,
avoid unsupported
conclusions,
report with
precision,
report specific
limitations
and do not
exceed the
data limitations.
They
caution that
surveys can create
an unnecessary
expense for the
organization
and burden on
the respondents.
They can produce
inadequate or
stale data. They
also caution
that providers
must be carefully
instructed and
supervised to
avoid abdication
of the role by
the organization.
For the Board
and the consultant,
Treasury recommends
everyone spend
sufficient time
at the front
end thing about
what is desired
and how it will
be used so that
the appropriate
results are obtained
to support the
analysis desired,
which affects
the design of
the survey.
Alberta
Treasury urges
agencies to carefully
seek in the RFP
and examine in
the proposals
the specific
description of
methodologies
used in the analysis
and the types
of analysis to
be provides as
well as the techniques
used.
Go
back to Top
Alberta
Labour reinforced
much of what
was related to
us by other sources.
They were able
to provide assistance
with some specific
questions which
arose from the
general review.
On
the topic of
focus groups,
they confirmed
that results
from these interviews
are not statistically
reliable, however
they do provide
in-depth comments
of the nature
that the management
team is seeking.
It is common
practice for
the consultant
to pay an honorarium
to focus group
participants,
even on government
initiatives.
They suggested
that focus groups
may be a viable
way to supplement
a written or
telephone questionnaire.
It is a useful
tool for surveying
all of the lawyers
for example,
by holding firm
by firm focus
groups. Personal
interviews may
work for the
staff input.
Alberta
Labour suggested
our client size
of 1000 is manageable
enough to enable
us to survey
all 1000 clients
by either written
questionnaire
or by telephone.
They caution
that the telephone
questionnaire
will, by its
nature, be less
in-depth (a view
professed by
one of the providers
as well.)
Finally,
Alberta Labour
provided us some
suggestions and
comments on the
use of Requests
for Proposals
and how to evaluate
the proposals
we receive.
Go
back to Top
| 
