**QUANTITATIVE METHODS IN ASSESSMENT**

(Sources: Schuh and Upcraft, “Using Quantitative Methods” in Schuh and Upcraft, Assessment in Student Affairs; and Sommer, B. and Sommer, R. A., “Descriptive Statistics” and Inferential Statistics” in Sommer and Sommer, A Practical Guide To Behavioral Research)

**I.
****Definition:
Quantitative methods involve assigning numbers to objects, events, or observations
according to some rule**

**II.
****Step
One: Define the Problem**

**A. ****Example of a theoretical problem: determine
why recruiters have mixed satisfaction with Placement Services**

III. **Step Two:** **Determine the Purposes of the Study**

** A. Determine how recruiters
rated various placement services**

** B. Determine
if recruiter responses varied by gender, ethnicity, size of company, and number
of previous visits to the placement service.**

**IV.
****Step
Three: Determine the Appropriate Assessment Approach**

**
A.
****Factors:
purpose of the study (find out which group of recruiters were not as happy with
the placement services**

1. size of the population (2,500 recruiters)

2. time limitations (needed results fairly quickly to correct potential problems)

3. resource limitations (not a large staff to conduct individual interviews with a large number of recruiters)

**
B.
****Decision:
quantitative assessment, using a representative sample**

**V.
****Step
Four: Define the Population**

**
A.
****Use
of sampling techniques**

1. simple random sampling

a. use of table of random numbers

2. systematic sampling

a. everyone in the population is placed on a list and every fourth person is chosen

b. people are not chosen independently of each other as in random sampling

3. stratified sampling

a. selection ensures that certain subgroups will be represented in the sample

b. helpful when subgroups are important to be studied

4. cluster sampling

a. pick groups, rather than individuals for the sample

b. in this example, draw a random sample of companies and then include all recruiters from that company in the sample

**
B.
****Decision:
Stratified random sampling chosen**

1. need to study subgroups

**
C.
****Sample
size: related to the type of statistical analysis to be done**

1. Consider sample error: possible difference between the sample results and the results if studied the entire population

__Random
Sample Size Sample
Error__

196 7 percent

294 6 percent

384 5 percent

600 4 percent

1,067 3 percent

2,401 2 percent

9,604 1 percent

2. in this example, if 35% of a sample of 1,100 said the service was outstanding, with a sample error of 3%, we could be 95% sure that between 32% & and 38% of all respondents thought the service was outstanding.

3. Subgroup statistical analysis: need at least 30 in each of 12 subgroups (subgroups are based on gender, size of company, and ethnicity)

4. Want a sample error no larger than 5%

5. Estimated return rate: 40%

6. Sample size of 1000 was chosen to get a 5% sample error rate, with at least 30 in 12 subgroups for statistical analysis, with an anticipated return of 400 with a 40% return rate

**VI.
****Step
Five: Determine the Instrument To Be Used**

**
A.
****Instrument
available or self-constructed?**

**
B.
****Self-constructed
instruments **

1. Types of Measurement Scales (See Conducting and Administering Surveys)

a. categorical scales: nominal scale categorizes objects (gender, ethnicity, type of institution)

b. ordinal scale: rank-orders according to how much of a variable they possess- finish of a horse race

c. continuous scales: interval scales (i.e. Likert-type scale with rating along a continuum from strongly agree to strongly disagree

d. ratio scales: can order variables, plus have a meaningful zero

2. Types of scales chosen for this hypothetical study

a. Demographic information: Categorical scales (nominal scales- gender, ethnicity, and company size)

b. Ratings of quality of placement services: continuous scales (interval scales- a Likert-type scale)

c. Number of times recruiter had visited previously: ratio scale

**VII.
****Step
Six: Determine Types of Statistical Analyses: Descriptive and Inferential**

**
A.
****Two
types of statistics**

1. Descriptive statistics

a. measures of central tendency

(1) mean: average of scores

(2) median: middle score in the rank order of scores

(3) mode: most frequently occurring score

b. measures of spread of dispersion:

(1 ) range: highest to lowest numbers

(2) standard deviation: sum of scores’ dispersal around the mean

c. measures of relative position

(1) percentile rank

d. measures of relationships

(1) correlations: values from -1.0 to +1.0.

(2) correlation does not imply cause and effect

(3) Use of the coefficient of determination (square of the correlation coefficient)

(a) if the correlation coefficient between grades and SAT scores is 50%, then 25% (.50 squared) of the variability in SAT scores and college GPA can be predicted or explained by aptitude or achievement

(b) with a correlation coefficient of .30 (.09 squared), 91% of the variability of GPA is unrelated to academic aptitude

(c) in general, correlations under .30 indicate little relationships

** Correlation Coefficient Coefficient Determination Strength of Relationship**

.00 to .20 .00 to .04 Negligible

.21 to .40 .05 to .16 Low

.41 to .60 .17 to .36 Moderate

.61 to .80 .37 to .64 Substantial

.81 to 1.0 .65 to 1.0 High to very high

e. showing results

__What You Want to
Do Kind of Data Use this Method__

**Examine responses**

**Of one group to one
question** Interval, ratio, Frequency distribution,

Or ordinal bar graph/line graph

Nominal Frequency distribution,

Bar graph

**Compare the responses**

**Of two groups to one**

**Question** Any
kind Paired
frequency distributions

Or graphs

**Compare the responses
of**

**One group to two
questions** Interval or
ratio Scattergram

Ordinal or nominal Cross-tabulation table

2. Inferential Statistics: compare the results with chance expectations

a. introduction to inferential statistics

(1) you would like to generalize from the sample to the population

(2) you would like to generalize from the situation studied to situations not studied

(3) you would like to know if your results are a chance finding or are they significant because of your program

(4) If you conducted a similar program again, would you get similar results?

(5) Answers to these questions can be found by using inferential statistics and tests of significance

(6) Two types of inferential statistics

(a) test relationship between two groups (bivariate analysis): chi-squares, t-tests, and one-way analysis of variance

(b) test relationship between multiple groups (multivariate analysis): analysis of variance (ANOVA)

(7) Statistical significance

(a) level of statistical significance: probability that the difference between the means of two groups is due to chance rather than a “real” difference between the two groups

(b) Levels of significance are commonly set at the .05 level or higher

(c) The .05 level of significance means that one would find a difference between two means under investigation only 5 times in a hundred

(d) This is expressed as “p>.05”

(e) “P>.001” means that the difference between the two means would occur only once in a thousand times

(f) Statistical significance doesn’t speak to the strength of the difference between the two means, but to the possibility of the difference occurring by chance

(g) This difference is inferred to be “real” and due to your program because of there being so little probability the difference occurred by chance

(h) Statistical significance is influenced by the size of the sample

(i) a sample size of 2,000 will get a statistically significant difference at the p>.05 level with a very small statistical difference

(ii) in that type of situation, it is better to increase the level of significance to at least .01.

** **

** Using Descriptive Statistics
to Describe Results**

__What is Described?
What Kind of Data?
Use this Statistic__

**The average response**

**To a question** Interval or ratio Mean or median

Ordinal Semi-interquartile range

Nominal Mode

**The spread or
variability**

**Of responses to a** Interval or Ratio Standard Deviation

**Single question**

Ordinal Semi-interquartile range

Nominal Proportion falling outside

Mode

**The degree of
relationship**

**Between responses to**

**Two questions** Interval or Ratio Pearson’s
Product-Moment

Correlation Coefficient

Ordinal Spearman’s rank-order

Correlation Coefficient

Nominal Cramer’s index of contingency

**The degree of
relationship**

**Among responses to
three**

**Or more questions** Interval or ratio Multiple
correlation or

Partial correlation

Ordinal

Correlation

__Using Statistical Analysis to Test
Differences Among Groups__

__What kind of
data? How many
subgroups? Use this analysis__

**Interval or ratio Two t-test**
for two independent means

Three or more One-way analysis of variance

**Ordinal ** Two Mann-Whitney U-test

Three or more Kruskal-Wallis one-way

Analysis of variance

**Nominal
** Two t-test for
two proportions

Two of more Chi-square test of association

** Using Statistical
Analyses To Test Differences in Response**

__What kind of
data? How many responses
to compare? Use this analysis__

**Interval or ratio** Two
t-tests for matched pairs

Three or more One-way analysis of variance

**Ordinal ** Two Sign
test

Three of more Friedman two-way

Analysis of variance

**Nominal ** Two McNemar test for

Significance of change

Three or more Cochran Q-test

**VIII. ****Step Seven: Data Collection Plan**

**A. In the hypothetical case of placement services, a 1000 surveys were
mailed out to recruiters using a stratified sample**

**IX.
****Step
Eight: Record The Data In Usable Form**

**
A.
****Machine
Readable Instrument Forms Are Very Useful With Larger Groups**

**
B.
****These
Forms Were Used in the Recruiter Study**

**X.
****Step
Nine: Conduct the Appropriate Analyses: Statistical Packages for the Social
Services (SPSS) is a commonly used computer software package**

**
A.
****Results
from the Recruiter Survey**

1. Using the chi-square test of goodness of fit, the sample was representative of the population.

2. There were at least 35 usable responses in each of the subgroups

3. Using the Kruskal-Wallis one-one analysis of variance to analyze the rank ordering of reasons for choosing the institution to interview, no differences were found among the subgroups

4. Using the Pearson product-moment correlation to analyze the ratio scale of number of previous visits to the institution, no differences were found.

5. Using the mean responses to each of the services rated, the highest and lowest rated service areas were found

6. Using one-way analysis of variance, differences were found by gender and size of company.

7. Using a one-way analysis of variance, no differences in ratings of services were found in the number of previous visits to the institution.

**XI.
****Step
Eleven: Evaluation, or the Meaning of Analyses for Policy and Practice**

**
A.
****Use
of the Results to Increase** **Satisfaction
with Services**

**XII.
****Step
Twelve: Strategy For Use of Results**

**
A.
****Follow
up with focus groups and interviews**

**
B.
****Implement
Changes In Services**

**
C.
****Re-do
the study in two years**

**XIII. ****Common Statistical Notations**

**
A.
****< less than**

**
B.
****> greater than**

**
C.
****N Number of scores**

**
D.
****df**** degrees of freedom**

**
E.
****p probability
that the results of a statistical test are due to chance**

**
F.
****ns Not significant**

**
G.
****M Mean (Arithmetic average)**

**
H.
****X Individual score**

**
I.
****Y Individual
score (used in a second set of scores)**

**
J.
****SD Standard deviation, a descriptive statistic
that indicates variability or dispersion from the mean**

**
K.
****SS Sum of squares, refers to the sum of
squared deviations from the mean**

**
L.
****MS Mean square: SS divided by df.**

**
M.
**** T t ratio, an inferential statistic used for contrasting two
means**

**
N.
****ANOVA Analysis of Variance**

**
O.
****F F ratio, an inferential statistic for testing significant
differences among two or more means.
Calculated by ANOVA**

**
P.
****X
Squared: Chi-Square, an inferential
statistic used for analyzing categorical scores **

**
Q.
****r: Pearson
product-moment coefficient; a measure of correlation for continuous data**