Booklet 3 - Figures

Figure 1: Choosing Type of Method

What are your evaluation questions?

If you are trying to learn: How many? How much? What percentage? How often? What is the average amount?	Choose quantitative methods
If you are trying to learn: What worked best? What did not work well? What do the numbers mean? How was the project useful? What factors influenced success or failure?	Choose qualitative methods

Figure 2: Evaluation Using Quantitative Methods

Step 1	Design Your Data Collection Methods - Surveys Write your evaluation questions Develop the data collection tool (i.e., questionnaire) Pilot test the questionnaire
Step 2	Collect Your Data - Surveys Decide who will receive the questionnaire Maximize response rate Check for non-response bias Provide motivation and information about risks and participants' rights
Step 3	Summarize and Analyze Your Data Compile descriptive data Calculate measures of central tendency and dispersion Simplify data to explore trends Provide comparisons
Step 4	Assess the Validity of Your Findings Calculate response rate Look for low completion rate of specific sections of surveys Investigate socially desirable responding

Figure 3: Aligning Evaluation and Survey Questions

Evaluation Question: How do participants rate the quality of the training session?

Survey Items	Response Options
How satisfied were you with the information presented during the training session?	Very satisfied Somewhat satisfied Neutral Somewhat dissatisfied Very dissatisfied
Would you recommend this session to others?	Yes No Don't know
Do you think you will use the online resources in the future?	Yes No Don't know

Figure 4: Survey Item Formats

	Example Question	Example Answers	Tips
Example 1: Two Options	Have you used MedlinePlus since the training session?	Yes No Not sure	This yes-no item works well for collecting factual information, such as people's participation in activities, exposure to publicity materials, or experience with specific online resources Other two-option formats are "true/false," "support/oppose," and "agree/disagree." Include a "don't know" or "not sure" option for participants who either cannot remember or are not sure about the information you are requesting. Note: In electronic survey design the convention is to use round "radio buttons" for questions that direct respondents to choose one option.
Example 2: Best Option	The last time you looked for health information on the Internet, who were you getting it for? (Choose one.)	Myself A family member A friend or co-worker A supervisor A client Other (please describe ___)	Best option items are good for collecting information about the respondent's attributes and behaviors. Make sure that choices do not overlap so that each person can easily choose only one response. Provide an "other" response for options that are not included on the list.
Example 3: Multiple Options	Where do you get health information? (Check all that apply.)	From my doctor At the clinic Newspapers Magazines Television Friends or family members The Internet Other (please describe___)	This is a faster version of the "yes/no" format: A check means "yes" and a blank means "no." If your list of options is more than 6 or 7 items, use a "yes-no" format instead. If the list is too long, people may not consider every item. When forced to respond, they are more likely to look at each item. Use "Other" even if you think you have listed all possible responses. People will use this option if they are not sure where their option fits. Note: In electronic survey design, the convention is to use square "check boxes" for questions that direct respondents to choose all applicable options.
Example 4: Rating Scales Version 1	Version 1 Please select the option that indicates your level of agreement with the statement. Because of the training session, I am much more confident about my ability to find information about my health concerns.	Strongly Disagree Somewhat Disagree Uncertain Somewhat Agree Strongly Agree
Example 4: Rating Scales Version 2	Version 2 Please select the option that indicates your level of agreement with the statement. The exercises were helpful.	Not at all helpful 1 __ 2 __ 3 __ 4 __ 5 __ Very helpful	These two formats are good for collecting information from respondents about their attitudes, feelings, beliefs, and opinions. The numbers should match the tone and intensity of the response labels. For example, associate a "1" with the most negative response label and "5" with most positive. A neutral point is usually recommended for participants who do not have strong opinions in either direction about the item. Some survey designers use even numbers of responses. This approach works best for items eliciting opinions or attitudes, particularly if you think people may be reluctant to voice their opinions and will "hide" in the neutral response option. You can provide as many response choices as you want, but most experts believe 5-7 options are adequate. Keep the direction of the responses consistent throughout the questionnaire. For example, always have the most negative response option on the far left and the most positive option on the far right.
Example 5: Rank-Order	Listed below are different health topics that could be included on a consumer health website. Rank the features in terms of how important each topic is to you, with "1" as the most important feature and "7" as the least important.	Specific health conditions Wellness information Alternative medicine Prescription drugs Health insurance, Medicaid, Medicare Clinical trials Health news	This format should be avoided. Ranking items is a difficult task for respondents. Also, you may force respondents to rank two items that are of equal importance to them. When possible, choose a rating scale (Example 4) instead of a rank-order item. Statistical analysis of rank-ordered items is very tricky because responses across individuals are not comparable. Using the item above as an example, two people may rank prescription drugs as the most important feature of a website relative to the other features in the list. However, the first respondent may think everything on the list is important and the second may think nothing is important, so a "1" tells you nothing about the strength of the importance to each respondent. To analyze this type of data, the best you can do is show how many times an item was ranked, for instance, as 1 or 2.
Example 6: Open-Ended	List at least two important things you learned in the training session today:	1. [respondent's content here] 2. [respondent's content here]	This format yields qualitative data, but it is often helpful in interpreting the statistical information you gather on your survey. To analyze open-ended questions, use the methods described beginning with Step Three of the "Qualitative Methods" of this booklet on page 27. Avoid starting a survey with open-ended questions. Open-ended questions can be overwhelming and people may choose to not take the survey. Draw the respondent in with some interesting, easy quantitative questions and save your open-ended questions for later in the survey.

Figure 5: How to Administer Surveys

1. Send a Pre-Survey Letter	Always send a personalized pre-survey letter (print surveys) or email (electronic surveys) to the target audience from someone influential or well-liked by the group. Let them know that a survey will be sent via email within the next week. For pre-notification emails, have your influential endorser's name or email address in the "from" or "subject" field if possible.
2. Send the Survey	Within a week after the pre-survey letter, send the survey with a personalized cover letter (e.g., "Dear Jane Smith") or personalized email with a link to the survey. (Microsoft Word will create mail-merged email messages and put them in Outlook's outbox.)
3. Send a Personalized Reminder	Within a week after sending the survey, send a personalized reminder postcard or email to non-responders.
4. Send a Second Reminder	Within two weeks, send or email another survey to non-responders, again with a personalized cover letter.
5. Keep Track of Undeliverable Surveys	If you mail surveys, be sure to use first-class mail so undeliverable surveys are returned to you. If you send surveys through email, keep track of the returned emails and, if possible, send print surveys to those participants. This mixed-method approach has been shown to increase response rates for electronic surveys.

From: Dillman DA et al. Internet, mail, and mixed-mode surveys [8]

Figure 6: Checklists for Writing Survey Correspondence

1. Checklist for pre-notification letters/emails:

• For mailed surveys, send the letter with the signature that your respondents know and trust. For electronic communication, use an electronic signature or the name and signature line of the person they know and trust. Also, for electronic questionnaires, use that person's email address in the "FROM" field if possible.
• Briefly describe the project and why it is important that they respond. (Emphasize how it is important to them, their families, and their communities as well as how their responses are important to you and your organization.)
• Explain when and how (e.g., mail, email with a link) they can expect the questionnaire and who will send it to them.
• Thank them in advance for their help.

2. Checklist for cover letters/emails

• Briefly describe your project and why you want respondents to reply to your questionnaire.
• Directly ask recipients to participate. Explain how the respondents' participation will benefit them (e.g., by helping the organization improve service; making their opinions heard)
• Appeal to the respondent's altruism by describing how his or her participation will help others.
• State the sponsors of the study.
• Explain how to complete the questionnaire (e.g., "click on the link below" or "complete the enclosed questionnaire and return it in the enclosed, postage-paid envelope.") You may want to have the link near the top of the email message.
• Describe who will see their individual responses and how you will maintain confidentiality. Usually, confidentiality is maintained by removing identifiable information from responses before they are shown to others, and only reporting summary data.
• Describe any risks to the respondent if he or she chooses to respond. (If you don't know of any risks, state that.)
• Indicate the voluntary nature of their participation and their right to withhold answers at any point in the survey.
• Describe incentives. (An incentive may include a promise to send them results of the questionnaire and how you used the results.)
• Include a statement that the participant is free to refuse to participate and can stop responding at any time.
• Request that they respond by a certain date.
• Give contact information if a respondent has further questions.

3. Checklist for reminder letters/emails:

• State that you are sending a reminder for them to complete the questionnaire.
• Thank those who have already responded. (Note: You can send targeted follow-ups to those who have not responded, but include the thank-you statement anyway. It adds an air of politeness and it communicates to the non-responders that some of their peers have responded.)
• Request those who have not responded to do so by a given date.
• For electronic questionnaires, include a link to the questionnaire in every follow-up. For print questionnaires, you send a reminder postcard for the first follow-up and a letter with another copy of the questionnaire on the second follow-up.

Figure 7: Frequency Table

Question: Please indicate your level of agreement with this statement.

I am more confident about finding prescription drug information on the Web because of this training session.
N=100

Response	Strongly Disagree	somewhat Disagree	Uncertain	Somewhat Agree	Strongly Agree	Total	Missing
Response value	(1)	(2)	(3)	(4)	(5)
Frequencies	0	2	5	36	54	97	3
Percent	0.0%	2.0%	5.0%	36.0%	54.0%		3%
Valid Percent	0.0%	2.1%	5.1%	37.1%	55.7%

Definitions

N - Number of people responding to the survey. (Note: 100 people returned a survey, but only 97 responded to this particular question.)
Frequencies - The number of respondents choosing each response.
Percent - The number of those choosing that response divided by the number of people who completed the survey.
Valid Percent - The number of respondents choosing that response divided by the number of respondents who answered the question. In this example, we had 100 people complete the survey, but only 97 actually responded to this particular question.

Figure 8: Measures of Central Tendency and Dispersion

Table A: Nominal-Level (Categorical) Data

Question: Where do you live?
N = 144

	Delaware	New York	New Jersey	Pennsylvania
	16	56	49	23
Measure of central tendency	Mode	56
Measure of dispersion	[none]

Mode: The mode is the most frequent response.

Table B: Ordinal-level (Ranked) Data

Question: Please indicate your level of agreement with this statement.
I am more confident about finding prescription drug information on the Web because of this training session.
N=100

Response	Strongly Disagree	Somewhat Disagree	Neither Agree nor Disagree	Somewhat Agree	Strongly Agree	Total	Missing
Response value	(1)	(2)	(3)	(4)	(5)
Frequencies	0	2	5	36	54	97	3
Percent	0.0%	2.0%	5.0%	36.0%	54.0%
Valid Percent	0.0%	2.1%	5.2%	37.1%	55.7%

Median	5
Range	Lowest Score: 2, Highest Score: 5

Median: The median is the score that is in the middle of the distribution, with half of the scores above and below.
Range: The range shows the highest and lowest scores reported

Table C: Interval/Ratio-Level (Equal Distance) Data

Question:
How many times did you visit your public library in the last 30 days?
N=25

Response	(Responses were open-ended)
Mode	2 visits
Median	1 visit
Range	Lowest=0 Highest=7
Mean	1.8
Standard Deviation	1.9

Mean: The mean is the "average" response in your distribution. It is computed by adding all responses and dividing by the number of respondents who answered the question.
Standard deviation: The standard deviation gives you an idea of how closely scores cluster around the mean. A higher standard deviation means a bigger spread of scores. In a normal (bell-shaped) distribution, about 68% of scores will fall between one standard deviation below and one standard deviation above the mean.

Figure 9: Two Ways to Present Frequencies and Percentages

Table A: Participants' Self-Report of Confidence in Using Databases

N=50

	Strongly Disagree	Somewhat Disagree	Neither Agree nor Disagree	Somewhat Agree	Strongly Agree
The training session helped me develop more confidence in using MedlinePlus.	0 0%	2 4%	9 18%	16 32%	23 46%
The training session helped me develop more confidence in using PubMed.	2 4%	3 6%	13 26%	22 44%	10 20%

Analysis: The majority of respondents agreed or strongly agreed that the training sessions helped them gain confidence in using the NLM online resources. Ratings seemed to be slightly more positive for MedlinePlus. This indicates that we achieved our objective of increasing confidence in use of online resources with the majority of our participants.

Table B: Participants' Self-Report of Confidence in Using Databases

N=50

	Negative (Disagree/ Strongly Disagree)	Neutral (Neither Agree nor Disagree)	Positive (Strongly Agree/ Agree)
The training session helped me develop more confidence in using MedlinePlus.	2 4%	9 18%	39 78%
The training session helped me develop more confidence in using PubMed.	5 10%	13 26%	32 64%

Analysis: This table makes the pattern of positive ratings more obvious for the items introduced in Table 3 It also confirms that ratings were more positive for the MedlinePlus session compared with the PubMed session. One explanation might be that PubMed is more difficult to use and requires a longer training session or more training sessions compared with MedlinePlus.

Figure 10: Comparing Pre-Project and Post-Project Data

Average Number of NLM Resources Used Before and One Month After Training
How many of the following websites have you used in the past month? (Check all that apply of 6 resources.)
N=100

	Average # of Websites Before Training	Average # of Websites One Month After Training	Difference
Mean	1.85	3.37	1.52
Standard Deviation	.95	.71

Analysis: Of the six websites demonstrated in the training session, participants on average had used less than two of them before training. One month after training, they had, on average, visited more than three of the websites. This finding suggests that we chose websites that our participants found to be useful.

Figure 11: Comparing Subgroups within a Distribution

Average Number of NLM Resources Used Before and One Month After Training by Profession
(Total Number of Websites Taught=6)
N=80

	N	Average # of Websites Before Training (Mean)	Average # of Websites Before Training (St Dev)	Average # of Websites One Month After Training (Mean)	Average # of Websites One Month After Training (St Dev)	Increase in Use
Health Sciences Librarians	18	3.7	1.1	4.3	.9	.6
Social Workers	13	1.2	1.7	3.0	1.6	1.7
Nurses	22	2.2	1.2	3.6	.91	1.4
General Public	27	.6	.5	2.6	1.26	2.0

Analysis: We did not seem to increase the variety of websites used by the health science librarians, probably because, on average, they already had used more than half of the websites we demonstrated. Our training seemed to have the greatest impact on the general public, who had used very few of the websites before training. We had moderate success with social workers and nurses. For planning future sessions, we may want to conduct a preliminary survey to find out what websites are popular with health science librarians so we can adjust our training content to cover websites they do not know. We also should get a better idea of the health information needs of social workers and nurses.

Figure 12: Comparing Actual Results to Objectives

Percentage of Participants Who Used Resources After Training Compared with Targets in Objectives
N=100

	Actual	Target	Difference
Number of participants using MedlinePlus after training	62%	50%	+12%
Number of participants using PubMed after training	45%	50%	-5%

Analysis: We exceeded our target for the number of participants who used MedlinePlus after completing our training sessions. However, we were slightly below our target for PubMed usage. Because PubMed is more academic and MedlinePlus is more consumer-oriented, it is possible our users simply had more occasions to use MedlinePlus the month following the session. We may want to explore this explanation in a follow-up interview with a few users who took both sessions.

Figure 13: Evaluation Using Qualitative Methods

Step 1

Design Your Data Collection Methods - Interviews

• Write your evaluation questions
• Develop the data collection tool (i.e., interview guide)
• Pilot test the interview guide

Step 2

Collect Your Data - Interviews

• Decide who will be interviewed
• Provide informed consent information
• Record the interviews
• Build trust and rapport through conversation
• Start the analysis during data collection

Step 3

Summarize and Analyze Your Data

• Prepare the text
• Note themes (or categories)
• Code the text
• Interpret results

Step 4

Assess the Trustworthiness of Your Findings

• Use procedures that check the fairness of your interpretations
• Present findings to reflect multiple points of view

Figure 14: Types of Questions

Type of Question	Information Collected	Example
Experience/Behavior	What did respondents do?	"The last time you needed health information, where did you go to get it?"
Sensory	What did respondents experience through their five senses? (This is a variation on the experience/behavior question but focuses on what they saw, heard, touched, smelled, or tasted.)	"How did your doctor act when you showed her the information you found at MedlinePlus?"
Opinion/Value	What do respondents think or believe to be important?	"What do you like best about MedlinePlus?"
Feeling	What were respondents' emotional reactions?	"How did you feel when you could not find information about your child's health condition?"
Knowledge	What factual information does the respondent know?	"What are the busiest times of day for the computer lab?"
Background/ Demographic	What are the characteristics of your respondent?	"What do you do for a living?"

From: Patton, Qualitative Research and Evaluation Methods [14]

Figure 15: Coding Interview Data

The following section is from a fictional interview with a lay health adviser from a faith-based outreach program. In the original booklet, it has been color coded using the highlighting method described in the text, with the following codes: Uses of MedlinePlus, Outcomes, Barriers, and Suggestions for improving the program. In this HTML version, for accessibility, the color coding has been eschewed in favor of a table layout that indicates the coding. This is page 4 of the interview with Respondent 1.

Interviewer	Respondent 1	Coding
Describe some ways you have used MedlinePlus in your work as a lay health adviser.	This lady from the community came to see me because she was having terrible heartburn almost every day. We looked up heartburn on MedlinePlus.	Uses of MedlinePlus
What did you find?	We found out there are better medicines than what she was taking and she did not have to get a prescription. She talked to the pharmacist because she is on another medication, because MedlinePlus said don't mix these pills with other pills. The pharmacist told her it was okay for her to take the pills but to talk to her doctor about her heartburn. We researched MedlinePlus again and found out that heartburn that occurs frequently can lead to cancer. The woman said the medicine got rid of her heartburn, but she also made an appointment to talk to her doctor about it. She didn't realize how serious heartburn can be.	Uses of MedlinePlus & Outcomes (in bold)
Do you have any other examples?	There was a woman whose sister was diagnosed with breast cancer and she was so worried. We read a little bit about it and found out that "stages" tell you how serious the cancer is. She went back and asked her sister about her breast cancer and found out it was stage 1. That means her sister has a really good chance of surviving cancer. So this lady was so relieved. Also, the lay health advisers wanted to do a healthy eating promotion in March, which is National Nutrition Month. We were able to print out some handouts and create a bulletin board with tips and the latest news related to nutrition. We used a lot of information from MedlinePlus. We even used MedlinePlus to locate a dietitian in the area who came to do a presentation on fighting childhood obesity.	Uses of MedlinePlus & Outcomes (in bold)
Have you had any problems finding information for people?	No, we can always find information on the topics people bring to us. But sometimes people don't want to tell us too much about their problems, especially if it's kind of a sensitive topic. We all know each other around here, so people don't always want you to know things about them.	Barriers
So how do you help them?	We show them MedlinePlus and how to search for a health topic, then give them privacy with the computer. It works okay as long as they know a little bit about using a computer. Most of our clients do use computers.	Uses of MedlinePlus
Do you have suggestions for making MedlinePlus more useful to lay health advisers in your organization?	We have some new lay health advisers starting in a month or so. It would be great if your librarian could do another demonstration of the website for our new staff.	Suggestions for improving the program

¹ Patton, MQ. Practical evaluation. Beverly Hills (CA): Sage; 1982, p.161.

Figure 16: Organizing and Analyzing Coded Data

Data coded as "Uses of MedlinePlus" have been organized onto one page and sub-themes have been identified. A description is also provided for each theme and sub-theme. Note that the interviewee and the interview transcript page are identified so that the coder can go back to read the information in its original context.

Theme: "Uses of MedlinePlus" Theme description: Uses of MedlinePlus by Lay Health Advisers	Sub-theme
Respondent 1: This lady from the community came to see me because she was having terrible heartburn almost every day. We looked up heartburn on MedlinePlus	Learn about health problem
Respondent 1: We found out there are better medicines than what she was taking and she did not have to get a prescription. She talked to the pharmacist because she is on another medication, because MedlinePlus said don't mix these pills with other pills. The pharmacist told her it was okay for her to take the pills but to talk to her doctor about her heartburn. We researched MedlinePlus again and found out that heartburn that occurs frequently can lead to cancer.	Learn about prescription drug
Respondent 1: There was a woman whose sister was diagnosed with breast cancer and she was so worried. We read a little bit about it and found out that "stages" tell you how serious the cancer is. She went back and asked her sister about her breast cancer and found out it was stage 1. That means her sister has a really good chance of surviving cancer.	Learn about a loved one's health problem
Respondent 1: Also, the lay health advisers wanted to do a healthy eating promotion in March, which is National Nutrition Month. We were able to print out some handouts and create a bulletin board with tips and the latest news related to nutrition. We used a lot of information from MedlinePlus. We even used MedlinePlus to locate a dietitian​ in the area who came to do a presentation on fighting childhood obesity.	Get information for presentations Find a health professional
Respondent 1: We show them MedlinePlus and how to search for a health topic, then give them privacy with the computer. It works okay as long as they know a little bit about using a computer. Most of our clients do use computers.	Teach others to use MedlinePlus

Notes: One of the projected outcomes of teaching lay health advisers about MedlinePlus was that people in the community would have better access to useful health information. Our interview with Respondent 1 gave us an idea of how the lay health advisers use MedlinePlus. Respondent 1 used it to help individual community members find information about health conditions and about drugs. She also helped someone research a family member's health condition. This is an important use of MedlinePlus because this person was quite worried about her sister but, because she is not her sister's caretaker, she could not talk directly to her sister's doctor. The lay health advisers also used the materials for a promotional effort and to locate a health professional in the area. The lay health advisers have also figured out ways to help members of the community use MedlinePlus even when the community members do not want to disclose information about their health conditions.

Figure 17: Describing Your Themes

Upper-level category:	Use of MedlinePlus
Lower-level category:	Learn about a loved one's health problem
Definition:	Lay health workers gave examples of helping community members who were worried about loved ones' health conditions. Sometimes the community members were caretakers. They often used MedlinePlus to find information that would support their ability to take care of their loved ones. Other times, community members were concerned but not closely involved in the care of the loved ones and did not have access to the loved one's health care provider. They used MedlinePlus to learn more about their loved ones' condition.
Quote:	"There was a woman whose sister was diagnosed with breast cancer and she was so worried. We read a little bit about it and found out that 'stages' tell you how serious the cancer is. She went back and asked her sister about her breast cancer and found out it was stage 1. That means her sister has a really good chance of surviving cancer." [Respondent 1, page 4, Line 209] Note: Most word processing software has a line numbering function that can help you keep track of the location of quotes.

Back to Booklet 3