Empirical Research

http://ahatter.wordpress.com/research-methods/empirical-research-terminology/

Foundational terms in empirical research:

Measurement - the process of quantifying variables.
- Its 2 parts are qualitative and quantitative.
- 2 options are to select:

1. measurements which already exist and have proven reliability and validity.
   - Measures which exist can be direct or indirect.
2. measurements which don’t exist. In this case you have to show reliability and validity.

Variables - That which is manipulated in quantitative reseach. These are described in qualitative research.

Qualitative Research - identify (fact), describe (definition) potential variables, and attempt to prove they exist (& that they have construct validity, reliability, etc.). You actually have to persuade people that this is the case, and so you’re always already engaged in a rhetorical practice, even when you’re doing empirical research.

Quantitative Research - object is to show relationship (quality) between variables–i.e. to persuade people that some [usually causal] relationship exists. This presupposes that the variables exist, which means that you have to 1st qualitatively show this to be the case.

Methods of evaluating composition:
(Critical question is always are these direct or indirect measures? IOW, do these methods measure the student…or the rater…? Answer to this will tell whether they’re direct or indirect measures)

- Holistic Evaluation - Give one score based on overall impression. No factorial breakdown. [low inter-rater reliability]
- Analytic Evaluation - Break down based on categories or variables
- Primary Traits - Put the object into categories based on how well it fits into the description of that category. Will end up with several scores. (e.g. the Eng 103 descriptive grading rubric)

Types of Data:
(Type must match statistical measure to ensure validity.)

• Nominal - Classifications which can be named.

• Ordinal - Rank ordering. Not equidistant between points. [e.g. A, B, C, D, F...Likert scales]

• Interval - Has equal distance between variables. [more powerful statistics are associated with these]
• Ratio - Interval data with an absolute zero. Almost never crops up with human subjects.
  - Z-Score - lets you normalize across a large population. But you have to exclude to normalize. (measure of reliability)

Reliability - measure of agreement or disagreement between raters or instruments (r = -1 - +1). Inter-rater reliability should approach +0.7 to have predictive power that what you’re describing is actually there.
(Out for a nightmarish situation: if your research is predicated on r=0.7, and you don’t achieve this, you can argue that r is a social construct, which is inherently predicated on truth by consensus, and is therefore not really a “scientific” or precise measure.)
3 types of reliability:
- Equivalency - you’re triangulating with multiple instruments and all of the instruments are giving the same results.
- Stability - do the instruments/people change over time? If yes, that’s a reliability issue.
- Internal - Consistency of instruments/people. Granularity and scale.

Validity - measuring what you say you’re measuring. Reliability is a necessary condition for validity, but it’s not sufficient. You can have construct validity, without being able to measure it reliably. Likewise, you can measure something reliably without it having construct validity.

What is the difference between reliability and validity? You can reliably measure something you did not intend to measure in your study, or around which your study does not hinge.