Global Fund Advocacy Evaluation

The Gates Foundation commissioned ITAD to evaluate the effectiveness of its Global Fund Advocacy Portfolio (GFAP) — a network of NGO grantees working to influence governments to increase contributions to the Global Fund for AIDS, TB and Malaria ahead of the 7th replenishment conference.

CASM Technology provided the quantitative social media analysis module. The brief was to map what grantees were actually saying online, how their messaging varied by market and by organisation, and how advocacy tactics (inside track vs outside track) were distributed across the portfolio.

My analysis fed into ITAD's broader mixed-methods evaluation alongside country surveys, interviews and case studies — providing the systematic, large-scale view of advocacy content that qualitative methods alone could not produce.

Technical Lead. I was responsible for the full pipeline design — from data architecture and collection through classification, analysis and synthesis.

I designed the three-tier relevancy framework (Direct / Indirect / Irrelevant), built and validated the 18-theme thematic framework from scratch, and produced all narrative and tactics analysis at both market and grantee level.

I also designed and ran the model evaluation — benchmarking classification performance against 300 manually annotated samples to validate the pipeline before it was used to analyse the full corpus.

• Posts analysed: 43,282 across Twitter, Facebook and Instagram
• Grantees: 16 NGO grantees monitored across all markets
• Markets: UK, US, EU/EC, France, International
• Languages: English and French
• Time period: January 2020 – October 2022, covering the COVID-19 pandemic period and the Seventh Replenishment Conference advocacy cycle
• Media types: Social media (Twitter, Facebook, Instagram) and mainstream media (websites, news outlets, blogs)
• Topics discovered: 380 (English corpus), 162 (French corpus) via unsupervised topic modeling
• Themes identified: 14 (English), 15 (French) covering narratives and advocacy tactics
• Evaluation: 300-sample human annotation benchmark for relevancy and theme classification

Phase 1 — Data Collection: posts and mainstream media articles collected across grantee accounts in five markets; online articles segmented into coherent units for consistent processing across source types.

Phase 2 — Topic Modeling: text embedded using all-mpnet-base-v2 for English and paraphrase-multilingual-mpnet-base-v2 for French and other languages (sentence-transformers); embeddings reduced via UMAP and clustered using density-based clustering (HDBSCAN) to surface latent themes in the corpus without supervised labels.

Phase 3 — Cluster Analysis: clusters interpreted across two analytical layers — a relevancy layer (Directly Relevant / Indirectly Relevant / Irrelevant) and a thematic layer (Themes and Subthemes representing narratives and advocacy tactics).

Phase 4 — Zero-Shot Refinement: zero-shot classification (facebook/bart-large-mnli, batched via PyTorch DataLoader) applied to both the relevancy and thematic layers to clean noisy cluster boundaries and improve accuracy — validated against a 300-sample human annotation benchmark.

Phase 5 — Analysis & Reporting: per-market narrative breakdowns, grantee-level thematic fingerprints, network analysis, and cross-market synthesis delivered across English and French corpora.

A key analytical layer was mapping the GFAN (Gates Global Fund Advocacy Network) — identifying how accounts clustered thematically, who shared messaging with whom, and which organisations drove narrative cohesion across the network.

Network graphs were built for mentions, hashtag co-use and link-sharing using NetworkX, with node size weighted by degree centrality. This produced structural portraits of how advocacy coordinated (or failed to coordinate) across grantees.

Per-market analysis tracked volume over time, theme distribution, grantee engagement and advocacy tactics — producing both strategic overviews for ITAD and granular, grantee-specific breakdowns for country case leads.

The French-language corpus was analysed in parallel with an equivalent pipeline, with separate evaluation benchmarks confirming comparable performance on French content.

43,282 posts analysed across 5 markets and 2 languages. Full thematic and relevancy classification applied to English and French corpora.

Country-level narrative synthesis delivered for the UK, US, EU/EC, France and International markets — each combining quantitative theme distributions with qualitative narrative analysis. Grantee-level advocacy portfolios built for each NGO in the network.

Cross-market synthesis identified divergences in how different markets framed the replenishment ask — with pledging language dominant in the US and UK, and health systems framing stronger in the EU.

Analysis was integrated into ITAD's published evaluation for the Gates Foundation and presented to stakeholders at the November 2022 advocacy workshop.