This weblog is created for the purposes of assignment for the subject BYL 7134, Cyberlaw by Faculty of Management, Multimedia University (MMU), Malaysia. The materials posted on this weblog are for the purposes of the assignment as well as study and non-profit research only. Appropriate acknowledgements to the materials that do not belong to the weblog owner have been publicly made. If you are the author or a copyright or a trademark owner of any of the articles or images or graphics or pictures or any material posted in this weblog herein and you object to such posting on any grounds, including copyright or trademark infringement, please contact me and I will take your material down. I state herein that I am relying on the doctrine of fair use.

INTERNAL CONSISTENCY RELIABILITY

Internal consistency reliability defines the consistency of the results delivered in a test, ensuring that the various items measuring the different constructs deliver consistent scores.

For example, an English test is divided into vocabulary, spelling, punctuation and grammar. The internal consistency reliability test provides a measure that each of these particular aptitudes is measured correctly and reliably.
One way of testing this is by using a test–retest method, where the same test is administered some after the initial test and the results compared.
However, this creates some problems and so many researchers prefer to measure internal consistency by including two versions of the same instrument within the same test. Our example of the English test might include two very similar questions about comma use, two about spelling and so on.
The basic principle is that the student should give the same answer to both – if they do not know how to use commas, they will get both questions wrong. A few nifty statistical manipulations will give the internal consistency reliability and allow the researcher to evaluate the reliability of the test.
There are three main techniques for measuring the internal consistency reliability, depending upon the degree, complexity and scope of the test.
They all check that the results and constructs measured by a test are correct, and the exact type used is dictated by subject, size of the data set and resources.

SPLIT-HALVES TEST

The split halves test for internal consistency reliability is the easiest type, and involves dividing a test into two halves.For example, a questionnaire to measure extroversion could be divided into odd and even questions. The results from both halves are statistically analysed, and if there is weak correlation between the two, then there is a reliability problem with the test.

The split halves test gives a measurement of in between zero and one, with one meaning a perfect correlation.

The division of the question into two sets must be random. Split halves testing was a popular way to measure reliability, because of its simplicity and speed.However, in an age where computers can take over the laborious number crunching, scientists tend to use much more powerful tests.

KUDAR-RICHARDSON TEST

The Kudar Richardson test for internal consistency reliability is a more advanced, and slightly more complex, version of the split halves test.In this version, the test works out the average correlation for all the possible split half combinations in a test. The Kudar Richardson test also generates a correlation of between zero and one, with a more accurate result than the split halves test. The weakness of this approach, as with split-halves, is that the answer for each question must be a simple right or wrong answer, zero or one.
For multi-scale responses, sophisticated techniques are needed to measure internal consistency reliability.

CRONBACH'S ALPHA TEST

The Cronbach's Alpha test not only averages the correlation between every possible combination of split halves, but it allows multi-level responses.For example, a series of questions might ask the subjects to rate their response between one and five. Cronbach's Alpha gives a score of between zero and one, with 0.7 generally accepted as a sign of acceptable reliability.
The test also takes into account both the size of the sample and the number of potential responses. A 40-question test with possible ratings of 1 – 5 is seen as having more accuracy than a ten-question test with three possible levels of response.
Of course, even with Cronbach's clever methodology, which makes calculation much simpler than crunching through every possible permutation, this is still a test best left to computers and statistics spreadsheet programmes.

SUMMARY

Internal consistency reliability is a measure of how well a test addresses different constructs and delivers reliable scores. The test–retest method involves administering the same test, after a period of time, and comparing the results.By contrast, measuring the internal consistency reliability involves measuring two different versions of the same item within the same test.
source: http://www.experiment-resources.com/internal-consistency-reliability.html

Interrater reliability (Kappa) Using SPSS

Interrater reliability (Kappa)

            Interrater reliability is a measure used to examine the agreement between two people (raters/observers) on the assignment of categories of a categorical variable. It is an important measure in determining how well an implementation of some coding or measurement system works.

A statistical measure of interrater reliability is Cohen’s Kappa which ranges generally from 0 to 1.0 (although negative numbers are possible) where large numbers mean better reliability, values near or less than zero suggest that agreement is attributable to chance alone.

Example Interrater reliability analysis

            Using an example from Fleiss (1981, p 213), suppose you have 100 subjects whose diagnosis is rated by two raters on a scale that rates the subject’s disorder as being either psychological, neurological, or organic.  The data are given below: (KAPPA.SAV)

		RATER A
		Psychological	Neurological	Organic
RATER B	Psychological	75	1	4
	Neurological	5	4	1
	Organic	0	0	10

The data set KAPPA.SAV contains variables, Rater_A, Rater_B and Count. The figure below shows the data file in count (summarized) form. 

           

To analyze this data follow these steps:

1.     Open the file KAPPA.SAV. Before performing the analysis on this summarized data, you must tell SPSS that the Count variable is a “weighted” variable. Select Data/Weight Cases...and select the “weight cases by” option with Count as the Frequency variable

2.     Select Analyze/Descriptive Statistics/Crosstabs.

3.     Select Rater A as Row, Rater B as Col.

4.     Click on the Statistics button, select Kappa and Continue.

5.     Click OK to display the results for the Kappa test shown here:

            The results of the interrater analysis are Kappa = 0.676 with p < 0.001. This measure of agreement, while statistically significant, is only marginally convincing. As a rule of thumb values of Kappa from 0.40 to 0.59 are considered moderate, 0.60 to 0.79 substantial, and 0.80 outstanding (Landis & Koch, 1977). Most statisticians prefer for Kappa values to be at least 0.6 and most often higher than 0.7 before claiming a good level of agreement. Although not displayed in the output, you can find a 95 % confidence interval using the generic formula for 95% confidence intervals:

Estimate ± 1.96SE

Using this formula and the results in the table an approximate 95% confidence interval on Kappa is (0.504, 0.848).  Some statisticians prefer the use of a weighted Kappa, particularly if the categories are ordered.  The weighted Kappa allows “close” ratings to not simply be counted as “misses.”  However, SPSS does not calculate weighted Kappas.

A more complete list of how Kappa might be interpreted (Landis & Koch, 1977) is given in the following table

Kappa	Interpretation
< 0	Poor agreement
0.0 – 0.20	Slight agreement
0.21 – 0.40	Fair agreement
0.41 – 0.60	Moderate agreement
0.61 – 0.80	Substantial agreement
0.81 – 1.00	Almost perfect agreement

Reporting the results of an interrater reliability analysis

The following illustrate how you might report this interrater analysis in a publication format.

Narrative for the methods section:

“An interrater reliability analysis using the Kappa statistic was performed to determine consistency among raters.”

Narrative for the results section:

“The interrater reliability for the raters was found to be Kappa = 0.68 (p <.0.001), 95% CI (0.504, 0.848). ”

Reference

Landis, J. R., Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33:159-174. 

source:http://www.stattutorials.com/SPSS/TUTORIAL-SPSS-Interrater-Reliability-Kappa.htm

reliability of test

Test reliability (consistency) is an essential requirement for test validity. Test validity is the degree to which a test measures what it is designed to measure.

       Researchers use four methods to check the reliability of a test: the test-retest method, alternate forms, internal consistency, and inter-scorer reliability. Not all of these methods are used for all tests. Each method provides research evidence that the responses are consistent under certain circumstances. There are four distinct types of reliability.

(1.) Test-Retest - a method of estimating test reliability in which a test developer or researcher gives the same test to the same group of research participants on two different occasions. The results from the two tests are then correlated to produce a stability coefficient. Studying the coefficients for a particular test allows the assessor to see how stable the test is over time.

Example: The information obtained for test-retest reliability of the WISC-IV was evaluated with information from 243 children. The WISC-IV was administered two separate times with the test-retest mean interval of 32 days. The average corrected Full Scale IQ stability coefficient was .93.

 

 (2.) Alternate Forms - This type of reliability makes a second form of a test consisting of similar items, but not the same items. Researchers administer this second “parallel” form of a test after having already administered the first form. This allows researchers to determine a reliability coefficient that reflects error due to different times and items and allow to control for test form. By administering form A to one group and form B to another group, and then form B to the first group and form A to the second group for the next administration of the test, researchers are able to find a coefficient of stability and equivalence. This is the correlation between scores on two forms and takes into account error of different times and forms.

Example: The ACT is is an academic test used in the college admission process. There are four academic subtests: English, mathematics, reading, and natural science reading. A standard score scale is used to report scores on the four academic tests. There is also a composite score – the average of standard scores on the four subtests. Scaled scores equivalents are provided for each four of the test by the equipercentile method based on the score distribution of an anchor form of the ACT. New forms of the test are equated to older forms by giving both forms to parallel samples of students and then equating the forms by the equipercentile method (Aiken, 1985).

(3.) Internal Consistency- three ways to measure the consistency of a test with only one form. 

          

(A)  Split-Half Reliability

What is Split-Half Reliability?

A test given and divided into halves and are scored separately, then the score of one half of test are compared to the score of the remaining half to test the reliability (Kaplan & Saccuzzo, 2001).

Why use Split-Half?

Split-Half Reliability is a useful measure when impractical or undesirable to assess reliability with two tests or to have two test administrations (because of limited time or money) (Cohen & Swerdlik, 2001).

How do I use Split-Half?

1st-divide test into halves. The most commonly used way to do this would be to assign odd numbered items to one half of the test and even numbered items to the other, this is called, Odd-Even reliability. 

2nd- Find the correlation of scores between the two halves by using the Pearson r formula.

3rd- Adjust or reevaluate correlation using Spearman-Brown formula which increases the estimate reliability even more. The longer the test the more reliable it is so it is necessary to apply the Spearman-Brown formula to a test that has been shortened, as we do in split-half reliability (Kaplan & Saccuzzo, 2001).

 Spearman-Brown formula

r = 2 r
1+ r

r = estimated correlation between two halves (Pearson r) (Kaplan & Saccuzzo, 2001).

          

(B)  Kuder-Richardson Formula

Another way to internally evaluate a test would be to use the Kuder-Richardson 20. This is only advisable if you have dichotomous item in a test (usually for right or wrong answers).

KR 20= r = N (S 2 alpha pq)
N-1 (S 2)

  KR20 = reliability estimate (r)

N= the number of items on the test

S2 = the variance of the total test score

p = proportion of people getting each item correct (this is found separately for each item)

q = the proportion of people getting each item incorrect. For each item q equals 1-p.

alpha p q = the sum of the products of p times q for each item on the test.

(Kaplan, Saccuzzo.2001)

         (C)  Cronbach Alpha/Coefficient Alpha

The Cronbach Alpha/Coefficient Alpha formula is a general formula for estimating the reliability of a test consisting of items on which different scoring weights may be assigned to different responses

insert equation here

k = the number of items

si2 = the variance of scores on item i

st2 = the variance of total test scores

(Aiken, 2003)

(4.)Inter scorer reliability- measures the degree of agreement between persons scoring a subjective test (like an essay exam) or rating an individual. In regards to the latter, this type of reliability is most often used when scorers have to observe and rate the actions of participants in a study. This research method reveals how well the scorers agreed when rating the same set of things. Other names for this type of reliability are inter-rater reliability or inter observer reliability.

Estimates of Inter-rater Agreement and Reliability

In addition to simple percentages of agreement, Cohen's kappa (KAPPA) was also calculated for the exact match percentages of agreement, and the results are shown in Table 2. The KAPPA coefficients indicate the extent of agreement between the raters, after removing that part of their agreement that is attributable to chance. As can be seen, the values of the KAPPA statistic are much lower than the simple percentages of agreement (Goodwin, 2001).

Simple percentages of agreement and kappa. To estimate inter-rater reliability of observational data, percentages of agreement are often calculated--especially if the number of scale points is small. Percentages of agreement can be calculated in a number of different ways, depending on the definition of agreement.

For Example:

In Table 2, the percentages of agreement between the raters for each occasion (day) are presented two ways: first, for the case in which agreement meant an exact match between raters in their assigned ratings; second, for the case in which agreement was defined more leniently as either exact agreement, or differences between the two raters' scores of not more than one point in either direction. (This latter definition of agreement has been used fairly often in the estimation of interrater agreement of some types of measures, such as parent-infant interaction scales [Goodwin & Sandall, 1988].) As would be expected, percentages of agreement are lower when agreement is defined in the more conservative way (exact match). The results shown in Table 2 demonstrate that the median percentage of agreement for the 6 days, when agreement was defined as exact match, was 20%; the median percentage of agreement for the 6 days, when the more liberal definition of agreement was used, was 80%. (Percentages of agreement were not calculated for the total scores in Table 1 because this approach to reliability estimation is rarely used if the range of scores is large; here, the total scores could range from 6 to 42.) 

source: http://web.sau.edu/WaterStreetMaryA/NEW%20intro%20to%20tests%20&%20measures%20website_files/reliability.htm

Steps in Research Process

1. Formulating the Research Problem: Main function is to decide what you want to find out about. And the way you formulate a problem determines almost every step that follows.Steps in formulation of a research problem : Working through these steps presupposes a reasonable level of knowledge in the broad subject area within which the study is to be undertaken. Without such knowledge it is difficult to clearly and adequately ‘dissect’ a subject area.

Step 1 Identify a broad field or subject area of interest to you.
Step 2 Dissect the broad area into sub areas.
Step 3 Select what is of most interest to you.
Step 4 Raise research questions.
Step 5 Formulate objectives.
Step 6 Assess your objectives.
Step 7 Double check.

2. Extensive Literature Review

 -Essential preliminary task in order to acquaint yourself with the available body of knowledge in your area of interest.
-Literature review is integral part of entire research process and makes valuable contribution to every operational step.
-Reviewing literature can be time-consuming, daunting and frustrating, but is also rewarding. Its functions are:

a. Bring clarity and focus to your research problem;
b. Improve your methodology;
c. Broaden your knowledge;
d. Contextualise your findings.

3. Developing the objectives

-Objectives are the goals you set out to attain in your study.
-They inform a reader what you want to attain through the study.
-It is extremely important to word them clearly and specifically.

Objectives should be listed under two headings:

Main objectives (aims):

The main objective is an overall statement of the thrust of your study. It is also a statement of the main associations and relationships that you seek to discover or establish.

Sub-objectives.

The sub-objectives are the specific aspects of the topic that you want to investigate within the main framework of your study.

-They should be numerically listed.
-Wording should clearly, completely and specifically communicate to your readers your intention.
-Each objective should contain only one aspect of the Study.
-Use action oriented words or verbs when writing objectives.The wording of objectives determines the type of research (descriptive, correlational and experimental) and the type of research design you need to adopt to achieve them.
e.g. 
Descriptive studies:
-To describe the types of incentives provides by Hotel XYZ to employees in Mumbai.
-To find out the opinion of the employees about the medical facilities provided by five star hotels in Mumbai.
Correlatinal studies:
-To ascertain the impact of training on employee retention.
-To compare the effectiveness of different loyalty programmes on repeat clientele.
Hypothesis –testing studies:
-To ascertain if an increase in working hours will increase the incidence of drug/alcohol abuse.
-To demonstrate that the provision of company accommodation to employees in Mumbai hotels will reduce staff turnover.

4. Preparing the Research Design including Sample Design

 Research design is the conceptual structure within which research would be conducted. The function of research design is to provide for the collection of relevant information with minimal expenditure of effort, time and money. The preparation of research design, appropriate for a particular research problem, involves the consideration of the following :

1. Objectives of the research study.
2. Method of Data Collection to be adopted
3. Source of information—Sample Design
4. Tool for Data collection
5. Data Analysis-- qualitative and quantitative

5. Collecting the Data

Having formulated the research problem, developed a study design, constructed a research instrument and selected a sample, you then collect the data from which you will draw inferences and conclusions for your study. Depending upon your plans, you might commence interviews, mail out a questionnaire, conduct experiments and/or make observations.

6. Analysis of Data

Processing and analyzing data involves a number of closely related operations which are performed with the purpose of summarizing the collected data and organizing these in a manner that they answer the research questions (objectives).

The Data Processing operations are:

1. Editing- a process of examining the collected raw data to detect errors and omissions and to correct these when possible.

2. Classification- a process of arranging data in groups or classes on the basis of common characteristics.

3. Tabulation-Tabulation is the process of summarizing raw data and displaying the same in compact form for further analysis. It is an orderly arrangement of data in columns and rows.

7. Generalization and Interpretation

8. Preparation of the Report or Presentation of Results-Formal write ups of conclusions reached

TYPES OF RESEARCH

Research can be classified from three perspectives:

1. Application of research study
From the point of view of application, there are two broad categories of research:
- pure research and
- applied research.
Pure research involves developing and testing theories and hypotheses that are intellectually challenging to the researcher but may or may not have practical application at the present time or in the future.

Applied research is done to solve specific, practical questions; for policy formulation, administration and understanding of a phenomenon. It can be exploratory, but is usually descriptive. It is almost always done on the basis of basic research. Applied research can be carried out by academic or industrial institutions. Often, an academic institution such as a university will have a specific applied research program funded by an industrial partner interested in that program.

2. Objectives in undertaking the research

From the viewpoint of objectives, a research can be classified as:
-descriptive
-correlational
-explanatory
-exploratory
Descriptive research attempts to describe systematically a situation, problem, phenomenon, service or programme, or provides information about, say, living condition of a community, or describes attitudes towards an issue.
Correlational research attempts to discover or establish the existence of a relationship/ interdependence between two or more aspects of a situation.
Explanatory research attempts to clarify why and how there is a relationship between two or more aspects of a situation or phenomenon.
Exploratory research is undertaken to explore an area where little is known or to investigate the possibilities of undertaking a particular research study (feasibility study / pilot study).
In practice most studies are a combination of the first three categories.

3. inquiry mode employed

From the process adopted to find answer to research questions – the two approaches are:
- Structured approach
- Unstructured approach
Structured approach: The structured approach to inquiry is usually classified as quantitative research. Here everything that forms the research process- objectives, design, sample, and the questions that you plan to ask of respondents- is predetermined. It is more appropriate to determine the extent of a problem, issue or phenomenon by quantifying the variation. e.g. how many people have a particular problem? How many people hold a particular attitude?
Unstructured approach: The unstructured approach to inquiry is usually classified as qualitative research. This approach allows flexibility in all aspects of the research process. It is more appropriate to explore the nature of a problem, issue or phenomenon without quantifying it. Main objective is to describe the variation in a phenomenon, situation or attitude. e,g, description of an observed situation, the historical enumeration of events, an account of different opinions different people have about an issue, description of working condition in a particular industry.

In many studies you have to combine both qualitative and quantitative approaches. For example, suppose you have to find the types of cuisine / accommodation available in a city and the extent of their popularity.
Types of cuisine is the qualitative aspect of the study as finding out about them entails description of the culture and cuisine. The extent of their popularity is the quantitative aspect as it involves estimating the number of people who visit restaurant serving such cuisine and calculating the other indicators that reflect the extent of popularity.

What is Research

When you say that you are undertaking a research study to find answers to a question, you are implying that the process;
1. is being undertaken within a framework of a set of philosophies ( approaches);
2. uses procedures, methods and techniques that have been tested for their validity and reliability;
3. is designed to be unbiased and objective .

The difference between research and non-research activity is, in the way we find answers: the process must meet certain requirements to be called research. We can identify these requirements by examining some definitions of research.

Research is a structured enquiry that utilizes acceptable scientific methodology to solve problems and create new knowledge that is generally applicable. Scientific methods consist of systematic observation, classification and interpretation of data.

Research is a process of collecting, analyzing and interpreting information to answer questions. But to qualify as research, the process must have certain characteristics: it must, as far as possible, be controlled, rigorous, systematic, valid and verifiable, empirical and critical.

Source: RESEARCH METHODOLOGY ( For Private Circulation Only)