Comparison of Two Different HIV Prevention Program Evaluation

Comparison of Two Different HIV Prevention Program Evaluation: Bavinton (2013) and Pedrana et al (2012)

1.     Intervention and Evaluation Objectives

The HIV prevention through peer education workshops (PEWs) are intended to increase sexual health knowledge and capacity among gay men in Sydney. The paper by Bavinton (2013) is not necessarily about evaluating the program, instead, it explored the challenges in evaluating PEWs programs in community settings.  However, it is implicitly mentioned that the objective of the evaluation of the PEWs program was to assess if there is an increase in sexual health knowledge and capacity.

This evaluation can be considered an impact evaluation as it was done right after the program ended. The purpose of the evaluation is not clearly mentioned in the paper, but it was most likely done for accountability towards stakeholder and organisational learning. It might be for scientific enquiry as well, even though there are many weaknesses in the process.

On the other hand, the DDU campaign consisted of 3 phases, with the first phase in 2008-2009 aimed to improve health-seeking behaviour and STI testing as well as raise HIV/STI knowledge among gay men in Melbourne (Pedrana et al, 2012). Therefore, the evaluation objectives were to evaluate the campaign awareness, HIV/STI knowledge, health-seeking behaviour, and HIV/STI testing.

This evaluation can sit on impact evaluation as it was done in a short period (less than a year) after the campaign ended. Similar to Bavinton (2013), the aim of doing an evaluation is not clearly mentioned. However, it was most likely done as a report to stakeholders, organisational learning, and scientific enquiry. With the success of the first phase campaign, it was also possible to gather funding for future phases.

2.     Development of Program Components

A formative evaluation was done and used in both interventions. In the PEWs program, formative evaluation found that behavioural change intervention was proven to be effective in reducing unprotected anal intercourse (UAI) and enhancing condom use, in which the impacts last for 12 months. It also found that there are 4 components of the most effective behavioural HIV interventions, including:

a.     theoretical model of behavioural change,

b.     interpersonal skills training,

c.     more than 1 educational session with 4 or more hours of exposure,

d.     4 or more methods of educational delivery such as counselling, group discussion, demonstration, and role play.

With the knowledge from formative evaluation, the PEWs program tried to implement those 4 components:

a.     Interpersonal skills training was most possibly conducted towards peer educators and participants during the workshops.

b.     Conducted 4-6 workshop sessions, with 2.5 hours for each session, once per week.

c.     Used more than 4 methods of educational delivery, including group discussion, hypothetical scenarios, role play, quizzes, didactic presentations, video scenarios, and practical skill-building activities.

d.     The paper does not clearly mention which behavioural change theory was used in the intervention. However, it is clear that the intervention applied it, as the workshops were done to improve knowledge and skills that will enhance the self-efficacy of participants and hopefully lead to behaviour change (Glanz, 2022).

It was quite hard to conduct a formative evaluation as most PEW programs in Australia are not well documented or evaluated which resulted in difficulties of doing replication. Therefore, it is essential to conduct a pilot program towards a smaller group of the target population. Unfortunately, it was never done. If they did, they would know whether the program will work or not, the potential challenges and obstacles that need to be anticipated, as well as simulate how the evaluation will be conducted. Therefore, they will be more prepared to run the program.

The Drama Downunder (DDU) social marketing campaign was also informed by formative evaluation. Formative evaluation found that although gay men were already well informed about HIV prevention, generally they still lack knowledge about STIs. It also found that delivering campaigns through traditional gay media, such as gay press and posters in gay venues, was considered as old. All this information suggested that a new way of delivering campaigns was needed for future campaigns.

The DDU campaign then developed a campaign with new ways of delivery. It still used the traditional gay media, including print and radio advertisements, printed resources, and outdoor advertisements in public events. However, a novel approach was conducted by advertising the campaign on gay dating sites. Other innovative methods were also conducted by releasing the campaign’s messages on daily stuff such as fridge magnets, drink holders, and even underwear. By doing it, the campaign message could come closer to the target audience. A campaign event (the “Drama Down Underwear” show) was also conducted to more engagement towards the campaign. Moreover, the message of the campaign was also created to be light-hearted so it could easily catch the audience’s attention.

A pilot program was not conducted in this campaign as it did not seem to be necessary in this case. However, a logic model was clearly needed to guide the campaign implementation. Unfortunately, there is not enough information about whether they used it or not.

3.     Assessment of Program Delivery

Both papers do not explain much about program delivery evaluation. Program delivery of the PEWs program was evaluated by measuring the participants’ attendance rate. However, the paper does not clearly mention how they analysed the data and what the result was. It also just briefly mentioned that the program was delivered towards 388 gay and bisexual men, but it does not mention whether the number was on target. Information about the number of conducted sessions is available but there is no information on whether it was aligned with their plan.

In this kind of program, it is essential to evaluate participants’ satisfaction towards the program. It will be important to understand how they feel about the program or what expectation is not fulfilled, so the program can be improved in the future, or in the remaining time, and it can be recorded as a lesson learnt. Unfortunately, this evaluation was never done.

Similar to the PEWs program, program delivery of DDU did not seem to be evaluated while many aspects can be evaluated. Campaign awareness can be done both during and after the campaign to see if it actually reaches the target audience. Instead, it was conducted only after the campaign to see whether the target audiences were able to recall the campaign either with prompts (aided) or without prompts (unaided). Evaluating campaign awareness during the campaign period will be useful in deciding whether the dose is sufficient or needs to be enhanced. Considering that the campaign was also run on dating sites, it is also important to evaluate the engagement and reach to decide if they need to spend more money to advertise it. Besides, it is also important to evaluate the sale or distribution of the campaign materials (fridge magnet, drink holder, and underwear) and the number of attendees in the campaign events. There is also no information about whether an assessment of the audience’s feelings or opinions towards the campaign was conducted or not.

4.     Assessment of Program Impacts

PEWs program impacts were assessed by evaluating sexual health knowledge and capacity. It seems that the impacts evaluation was not conducted well and more likely to be unprepared. It was conducted by using an anonymous questionnaire in which participants gave their demographic information and answered questions about whether they had ever done HIV testing, Participants also answered questions from the Sexual Health Capacity Scale (SHCS) in the form of the Likert scale to give information about their perceived knowledge and capacity about sexual health.

Evaluators collected data by using SHCS 3 times to understand the participants’ perception of sexual health capacity before the program (BB SHCS), perception of sexual health capacity before the program that was collected after the program (BA SHCS), and perception of sexual health capacity after the program (AA SHCS), which is quite confusing. Moreover, the SHCS is only measuring the perceived capacity, not the actual capacity. Participants might perceive that they know about sexual health while actually don’t.

The result shows that participants believe that they have more knowledge and capacity after the program. However, the evaluator did not do prior research to identify the factors that might impact capability. Therefore, the analysis could not be conducted thoroughly to identify whether the perceived capability improvement happened because of the program itself as there was not enough data.

DDU campaign impacts were evaluated better. It seems much more well-prepared. The evaluation was conducted through an online survey of gay men at 3 different times. Participants were recruited from gay community venues or events, and through the snowball method, which is actually prone to bias but easy to be conducted in this case. Instead of an anonymous survey, it used a unique code, so it could be matched over time to enable both cross-sectional and longitudinal analysis.

Evaluation of the DDU campaign was also conducted by analysing both primary and secondary data. HIV/STI knowledge and health-seeking behaviour were evaluated through the online survey, while HIV testing was evaluated by combining primary data from the online survey and secondary data from the Victoria Primary Care Networked for Sentinel Surveillance on Bloor-Borne Viruses and STIs (VPCNSS). These multiple data sources are essential to allow a broader context of evaluation.

5.     Evaluation Method Appropriateness According to the Evaluation Objectives

Evaluation of the PEWs program would be more suitable to be conducted by using a cohort study with the exposed group and unexposed group as a control. The control group could be recruited from a group of gay and bisexual men in NSW who potentially had similar knowledge levels. Therefore, it can be followed over time to see whether the intervention group will have better knowledge and capacity compared to the control group. It is also important to note that instead of evaluating perceived knowledge and capacity, which are prone to causing bias, evaluators should measure the real knowledge and capacity.  

Pre-test and post-test study design should become the last option when it is too hard to find a suitable control group and it needs to be executed better. As they needed to pair the data form prior to and after the program, they should have used a questionnaire with a unique code, as being used in the DDU campaign evaluation. Unless it will be impossible to pair the data and a paired sample t-test could not be conducted. With a unique code, participants can still feel secure and be honest as they do not need to put their identity.

Collecting data in retrospective way (BA SHCS) is also prone to bias. BA SHCS, which was paired to AA SHCS, should not need to be collected as BB SHCS and AA SHCS can be directly paired if only they placed a unique code on the questionnaire. In a few months after the program, evaluators can collect another SHCS data to see the impact over a longer period.

On the other hand, the impact evaluation of the DDU campaign was conducted appropriately. Cross-sectional analysis was conducted to see the difference in STI knowledge, health-seeking behaviour, and STI testing among those who were aware and unaware of the campaign. Multivariable logistic regression was used to evaluate associations between campaign awareness and STI testing while controlling covariates. Linear regression was used to evaluate the STI testing trends, while Poisson regression was also conducted to see the change in STI testing rate before, during, and after the campaign. These analyses indicate that even though STI testing rates had already increased before the campaign, Poisson regression shows that the increase in testing accelerated during the campaign. This is an important finding that emphasizes the impact of the campaign towards the STI testing rate. Unfortunately, the absence of a control group makes it hard to directly link the campaign and its impacts.  

6.     Public Health Lesson Learnt

From the PEWs evaluation, it is important to ensure that evaluators have the skills to conduct evaluation, including analysing the data. If it is unsure whether the program staff have the skill or not, better to involve a partnership, such as researchers, as they are potentially more skilful in conducting evaluations, getting used to handling data and performing statistical analysis. Partnership needs to be involved during the project development stage to guide the things that will be evaluated, the needed data, the way evaluation will be done, and the statistical analysis will be performed.

From the DDU campaign, it is essential to collect data from multiple sources to strengthen the evaluation as it can provide a broader context. In this case, the multivariable analysis explains that campaign awareness was associated with STI testing, which is supported by ecological evidence from clinical data by VPCNSS. Choosing the right statistical analysis is also important. Accelerated rates will remain unknown if Poisson regression was not done. So, it is important for evaluators to understand it or to involve stakeholders who understand it well.

Hopefully, in the future, public health program evaluation can be conducted with the presence of a control group as it is still missing on both evaluation papers.

References:

Bavinton, B. R., Gray, J., & Prestage, G. (2013). Assessing the effectiveness of HIV prevention peer education workshops for gay men in community settings. Australian and New Zealand Journal of Public Health, 37(4), 305–310. https://doi.org/10.1111/1753-6405.12076

Glanz, K. (2022). Social and Behavioral Theories. https://obssr.od.nih.gov/sites/obssr/files/Social-and-Behavioral-Theories.pdf

Pedrana, A., Hellard, M., Guy, R., El-Hayek, C., Gouillou, M., Asselin, J., Batrouney, C., Nguyen, P., & Stoovè, M. (2012). Stop the Drama Downunder. Sexually Transmitted Diseases, 39(8), 651–658. https://doi.org/10.1097/olq.0b013e318255df06