A practical guide for designing, evaluating, and reporting on professional learning communities in AMR genomics and pathogen surveillance
Based on: SAGESA Community of Practice Evaluation
- Overview and Purpose
- What is the Value Creation Framework?
- Why the VCF Suits Genomics Networks
- Phase 1 — Pre-Programme Design
- Phase 2 — During Programme Implementation
- Phase 3 — Post-Programme Evaluation
- Adapting VCF Indicators for AMR Genomics
- Data Collection Templates by Cycle
- Reporting and Communication
- Lessons from SAGESA
- Quick Reference Checklist
Appendix A. Glossary of VCF Terms
This guideline supports teams who are planning, implementing, or evaluating a Community of Practice (CoP) for professional development in genomics and antimicrobial resistance (AMR). It provides step-by-step guidance on how to apply the Value Creation Framework (VCF) prospectively — that is, integrated into programme design from the outset rather than applied retrospectively.
The guideline draws on the experience of the SAGESA Community of Practice (Sub-Saharan African Genomics and Epidemiology Surveillance of AMR), a multi-country professional learning network established over 18 months (July 2021–December 2022). SAGESA engaged scientists from 46 countries and used the VCF to design and evaluate community-level value creation across five cycles. The lessons documented here reflect what worked, what proved challenging, and what future implementers should anticipate.
Note
Who this guide is for: Programme managers, trainers, and evaluation specialists designing professional learning initiatives in genomics, AMR surveillance, pathogen genomics, or related bioinformatics fields. The principles apply broadly to any CoP for technical professional development.
The guide is structured in three phases — Pre-Programme, During Implementation, and Post-Evaluation — mirroring the lifecycle of a CoP. Sections 4–6 provide detailed guidance for each phase. Sections 7–9 provide practical tools: indicator frameworks, data collection templates, and reporting guidance. Sections 10–11 summarise lessons from SAGESA and a quick-reference checklist.
Implementers new to the VCF should read Sections 2 and 3 first. Those already familiar with the framework can proceed directly to Section 4.
| Term | Definition |
|---|---|
| Community of Practice (CoP) | A group of practitioners who share a domain, engage in community interaction, and develop shared practice — unified by common problems, not institutional roles. |
| Value Creation Framework (VCF) | A five-cycle evaluation framework developed by Wenger-Trayner et al. (2011) for understanding value creation in social learning systems over time. |
| VCF Cycle | One of five levels of value — Immediate, Potential, Applied, Realised, or Transformative — through which value creation is assessed. |
| Prospective evaluation | Integrating evaluation design into programme planning from the outset, rather than designing evaluation after the programme ends. |
| Programme logic model | A structured diagram linking inputs → activities → outputs → outcomes, used to make explicit the theory of change underlying the CoP. |
The Value Creation Framework (VCF) was developed by Etienne and Beverly Wenger-Trayner to address a fundamental challenge in evaluating professional learning communities: traditional outcome-based evaluation methods poorly capture the emergent, relational, and longitudinal nature of value generated in social learning systems.
The VCF organises value creation across five cycles, each representing a different type and timescale of impact. Unlike linear programme logic models, the VCF recognises that value in learning communities is not only instrumental (achieving predefined outcomes) but also relational and transformative — it emerges from participation, identity, and sustained engagement over time.
Value created in the moment of participation — engagement, energy, and perceived relevance during activities.
| Dimension | Description | Example indicators |
|---|---|---|
| Engagement | Active participation and perceived relevance of activities | Attendance rates; percentage reporting expectations met or exceeded |
| Productive interactions | Quality of dialogue and exchange | Participant-reported learning during sessions; facilitator observations |
| Meaning and energy | Sense of belonging, motivation | Post-activity feedback: 'gained new perspectives'; 'felt connected to peers' |
Knowledge, skills, ideas, and relationships produced — the learning assets that participants carry away.
| Dimension | Description | Example indicators |
|---|---|---|
| New knowledge / skills | Learning gained through activities | % agreeing new knowledge acquired; % agreeing learning objectives met |
| New relationships / networks | Connections formed across institutions | New professional contacts; networking utility ratings |
| New ideas / resources | Tools, datasets, protocols shared | Resources downloaded; tools introduced; concepts encountered |
Changes in practice — how participants use what they gained in their professional work.
| Dimension | Description | Example indicators |
|---|---|---|
| Direct application | Specific tools or skills applied in work | % reporting applying skills; specific tools used (e.g. genome assembly, phylogenetics) |
| Changed approaches | New methods or workflows adopted | Qualitative descriptions of practice changes; workflow integration |
| Knowledge sharing | Sharing learning with colleagues | % sharing with team; informal dissemination reported |
Documented outcomes — performance improvements, research outputs, or organisational benefits that can be attributed to participation.
| Dimension | Description | Example indicators |
|---|---|---|
| Performance outcomes | Measurable improvements in technical or professional performance | Improved data quality; new analytical pipelines implemented; case detection improved |
| Research outputs | Publications, reports, datasets produced | Manuscripts submitted; datasets shared; policy briefs produced |
| Organisational impact | Changes to team capacity, systems, or workflows | New bioinformatics functions established; AMR genomics integrated into routines |
Systemic or identity-level changes — shifts in professional identity, community sustainability, and broader system-level influence.
| Dimension | Description | Example indicators |
|---|---|---|
| Identity shift | Changes in how participants see themselves as professionals | Self-described role evolution; new professional leadership taken on |
| System influence | Contribution to field, policy, or institutional change | Cited in policy documents; invited to advisory roles; influencing curricula |
| Community sustainability | CoP continuing beyond formal programme period | Member-led activities; sustained use of platforms; new members recruited |
Evaluating professional development in genomics presents specific challenges that generic outcome-based frameworks handle poorly. The VCF is particularly well suited to this context for five reasons.
Competency in AMR genomics workflows — from sequence quality control through to phylogenetic interpretation and decision-making — is not acquired in a single training event. It develops iteratively, through repeated practice, troubleshooting, and peer consultation. The VCF's temporal structure (Immediate → Transformative) captures this developmental trajectory in a way that a pre/post survey cannot.
Much of what a genomics CoP produces is relational: awareness of peers doing similar work, informal mentoring, shared protocols, and a sense of professional community. These are captured by the VCF's Potential and Transformative cycles but are invisible to purely output-focused evaluation.
Participants in a genomics CoP are simultaneously embedded in institutional, national, and regional systems. Attributing improvements in AMR surveillance to any single training initiative is rarely defensible. The VCF explicitly accounts for this by acknowledging 'potential-to-applied' gaps and treating attribution with appropriate nuance across cycles.
The VCF was designed specifically for social learning systems. It does not impose an instrumental logic onto a relational process. For a CoP in which the community itself is the intervention (not merely the delivery vehicle), this alignment is essential.
When applied prospectively — with evaluation instruments designed before programme launch — the VCF supports real-time learning. Formative data from Cycle 1 (immediate engagement) can be used to adapt subsequent activities. This quality-improvement function is only available when evaluation is integrated from the outset.
Tip
SAGESA experience: In SAGESA, prospective VCF integration meant that the baseline needs assessment, webinar feedback forms, course evaluations, and follow-up survey were all designed to map onto specific VCF cycles. This made the final synthesis substantially more coherent than a retrospective analysis would have produced.
The most important decisions in a CoP are made before launch. This phase covers six design steps that should be completed before any community activity begins.
A CoP requires a clearly articulated domain — the shared problem or practice area that unites members. For AMR genomics, the domain should be defined by shared technical practice rather than by institutional role, sector, or geography. This inclusivity is a deliberate design choice: it expands the community's breadth and increases the likelihood of cross-sector knowledge transfer.
| Decision | Guidance | SAGESA approach |
|---|---|---|
| What is the domain? | Define in terms of shared technical practice (e.g. 'bioinformatics analysis for AMR genomics') not sector or disease area | AMR genomics and bioinformatics, open to all sectors |
| Who can join? | Define membership criteria broadly — open membership strengthens diversity and sustainability | Open to any practitioner using genomics/bioinformatics for AMR-related work |
| Geographic scope | Consider time zones for live events; asynchronous platforms extend reach | Pan-African, with global membership; activities designed for asynchronous participation |
| Expected community size | Estimate participation levels for platform planning | Anticipated 200–500; actual: 471 across 46 countries |
A structured needs assessment serves two purposes. It gathers evidence to inform programme design, and it establishes a baseline for evaluation. It should cover current capacity levels, learning priorities, preferred formats, existing resources, and professional development gaps.
- Design the needs assessment to map onto VCF Cycle 1 (what do members currently value and prioritise?) and Cycle 2 (what learning assets do they seek?)
- Include both closed (quantitative) and open (qualitative) items to capture diversity of need
- Analyse and share results with the community before finalising the programme plan. This builds trust and demonstrates responsiveness
Tip
Key fields for a genomics needs assessment: Current sequencing/bioinformatics tools used | Confidence levels by skill area (beginner/intermediate/advanced) | Priority topics for webinars | Preferred format (live/recorded/hands-on) | Current barriers to applying genomics in practice | Existing professional networks and affiliations
A programme logic model makes explicit the pathway from inputs and activities to outcomes. For a CoP, it should show how community activities (webinars, training, peer exchange) generate intermediate outcomes (skills, networks, confidence) that in turn produce longer-term outcomes (practice change, research productivity, institutional capacity).
The logic model serves three purposes: it guides design decisions, specifies what will be measured, and makes the CoP's value proposition explicit to funders and stakeholders.
| Component | Definition | Example (genomics CoP) |
|---|---|---|
| Inputs | Resources invested in the CoP | Staff time, facilitators, platform subscriptions, course materials |
| Activities | What the CoP delivers | Monthly webinars, bioinformatics training course, Slack community, newsletter |
| Outputs | Direct products of activities | Number of webinars delivered, participants trained, resources shared |
| Short-term outcomes | Immediate and potential value (VCF Cycles 1–2) | New knowledge, new peer connections, increased confidence |
| Medium-term outcomes | Applied and realised value (VCF Cycles 3–4) | Skills applied in practice, new pipelines implemented, research outputs |
| Long-term outcomes | Transformative value (VCF Cycle 5) | Sustained community, systemic influence, professional identity development |
Once the logic model is in place, design the evaluation instruments before the programme begins. Each instrument should map explicitly onto one or more VCF cycles. Table 4.1 shows the recommended instrument suite and timing for a CoP evaluation.
| Instrument | Timing | VCF cycles covered | Key questions |
|---|---|---|---|
| Baseline needs assessment | Before programme launch | 1, 2 | What do members currently know? What do they need? |
| Post-activity feedback forms | After each webinar/training | 1, 2 | Were expectations met? What new knowledge was gained? |
| Platform analytics | Ongoing (monthly) | 1, 2 | Active members, content engagement, platform growth |
| Facilitator observation notes | During activities | 1, 3 | Quality of dialogue, observable skill application |
| Follow-up survey | ~2 months post-intervention | 2, 3, 4, 5 | Skills applied in practice, outcomes realised, identity, sustainability |
| Longitudinal check-in | 6–12 months post-programme | 4, 5 | Long-term practice change, community sustainability |
Platform selection significantly affects engagement, accessibility, and sustainability. For a global genomics CoP, a layered multi-platform approach is recommended: one platform for synchronous engagement (live webinars), one for asynchronous community interaction (Slack, WhatsApp, or equivalent), and one for content hosting and archiving.
| Platform type | Purpose | Recommended tools | Key considerations |
|---|---|---|---|
| Synchronous (webinar) | Live learning events, expert presentations, Q&A | Zoom, Teams, WebEx | Record all sessions; enable captioning; manage time zones |
| Asynchronous community | Daily peer interaction, Q&A, resource sharing | Slack (preferred), WhatsApp, Discourse | Moderation plan essential; set community norms early |
| Content hosting | Archive presentations, recordings, resources | GitHub, OSF, institutional repository | Choose open/persistent platform for long-term access |
| Training delivery | Hands-on bioinformatics skills courses | Galaxy, MyBinder, Jupyter Hub | Low-barrier access; no local installation required |
A functional CoP requires explicit governance — including who is responsible for moderation, content curation, member onboarding, and community facilitation. Underestimating this requirement is a common implementation failure point.
- Appoint a dedicated community manager
- Establish a moderation policy covering expected conduct, response times, and content standards
- Identify community champions — active early members who can model engagement and support newer members
- Document succession planning: what happens to the community if key staff leave or funding ends?
Once the CoP is launched, evaluation data collection should run in parallel with programme delivery. This section covers how to capture VCF-aligned evidence during the programme lifecycle.
Every live event is an evaluation opportunity. A standardised post-event feedback form, completed immediately after each activity, generates the Cycle 1 and Cycle 2 evidence base. Keep forms short (5–8 items) to maximise completion rates.
| Item type | Example item | VCF cycle | Data type |
|---|---|---|---|
| Expectations met | Overall, this webinar met my expectations: (1–5 scale or Yes/Exceeded/Partially/No) | 1 | Quantitative |
| New knowledge | I gained new knowledge or skills from this webinar: (Agree/Disagree scale) | 2 | Quantitative |
| Most valuable aspect | What was the most valuable aspect of today's session? | 1–2 | Qualitative |
| Suggested improvement | What would have made this session more useful for you? | 1 | Qualitative |
| Application intent | Do you intend to apply anything from today's session in your work? If yes, what? | 3 | Mixed |
| Net Promoter Score | How likely are you to recommend this CoP to a colleague? (0–10) | 2 | Quantitative |
Platform analytics provide passive, ongoing Cycle 1 evidence: community size over time, message frequency, content downloads, and active member counts. Collect these monthly and visualise trends to identify engagement peaks and troughs linked to programme activities.
- Track: total members, active members (posting/reacting in past 30 days), messages sent per channel, resources downloaded/viewed
- Segment by region or country where possible to monitor geographic equity of participation
- Note qualitative platform events: spontaneous peer questions, resource-sharing by members, community-initiated discussions
- Flag low-engagement periods for follow-up — consider whether a programme activity or prompt could re-engage the community
Structured facilitation notes capture evidence of engagement quality and early applied value (Cycle 3) that quantitative instruments miss. After each activity, the facilitating team should document observations against a brief structured template.
| Observation category | What to record |
|---|---|
| Participation quality | Active discussion, depth of questions, peer-to-peer exchange (not just presenter-to-audience) |
| Knowledge application signals | Examples of participants describing how they will use content; references to current work challenges |
| Unexpected value | Topics raised spontaneously; connections made between session content and participants' own contexts |
| Inclusion and equity | Whether voices from different countries/career stages/sectors were present in discussion |
| Technical access issues | Platform problems, connectivity issues, accessibility barriers encountered |
The follow-up survey is the most important single evaluation instrument for capturing Cycles 3, 4, and 5 evidence. It should be administered approximately 8–12 weeks after the programme's core activities conclude. This timing is critical: too soon, and applied value cannot yet be observed; too late, and recall fades.
Key design principles for the follow-up survey:
- Limit to 20–30 items (approximately 15 minutes to complete)
- Include a mix of closed (scale/multiple choice) and open (free text) items
- Open items for Cycles 3–5 typically yield the richest evidence
- Pilot with 3–5 members before distribution to check clarity and completion time
- Offer the survey in multiple languages if the community is multilingual
Tip
Response rate strategies: In SAGESA, follow-up survey response rates were modest. Strategies to improve response rates include: personal email invitations from a known community contact rather than automated systems; sending reminders at 7 and 14 days; offering a brief summary of findings as an incentive; and making completion as frictionless as possible (direct link, mobile-compatible, no login required).
Alongside formal evaluation instruments, implementers should maintain a structured implementation journal throughout the programme. This captures contextual knowledge that no survey can: decisions made and why, unexpected challenges, adaptation points, and observed community dynamics.
At minimum, record: (a) major programme adaptations and their rationale; (b) community incidents (positive or negative) that affected engagement; (c) external factors affecting participation (e.g. major conferences, institutional events, global disruptions); and (d) informal feedback received through conversations and communications.
Post-programme evaluation involves synthesising evidence from all data sources into a coherent VCF-structured account. The goal is not to prove impact, but to construct a credible and transparent account of value created across cycles — acknowledging where evidence is stronger and where gaps remain.
| VCF Cycle | Primary data sources | Analysis approach | Common gaps |
|---|---|---|---|
| 1 — Immediate | Post-event forms, platform analytics, observation notes | Descriptive statistics; thematic analysis of open items | Low completion rates for short-feedback forms |
| 2 — Potential | Post-event forms, needs assessment comparison, follow-up survey | Descriptive statistics; before/after comparison where applicable | Difficulty attributing knowledge gains to CoP vs. other sources |
| 3 — Applied | Follow-up survey (open items), facilitator observations | Thematic analysis; count of reported applications | Short follow-up window; small response n |
| 4 — Realised | Follow-up survey, longer-term check-in, institutional records | Descriptive; case examples | Attribution difficulty; requires longer time horizon |
| 5 — Transformative | Longitudinal check-in, community observation, qualitative interviews | Narrative analysis; identity change indicators | Requires 12–24 months; rarely captured in funded evaluation periods |
A persistent challenge in CoP evaluation is attributing observed changes to the CoP rather than to other concurrent factors (other training, institutional changes, the passage of time). The VCF approach does not claim strict causal attribution; instead, it documents plausible contributions using transparent evidence.
- Use 'contribution' language rather than 'impact' language: 'the CoP contributed to...' rather than 'the CoP caused...'
- Triangulate evidence: if multiple independent data sources (survey, observation, platform analytics) converge on the same finding, confidence in the contribution claim increases
- Acknowledge competing explanations and contextual factors that may have contributed to observed changes
- For Cycles 4 and 5, case examples and narrative accounts are often more credible than aggregated statistics, given typical sample sizes
Most evaluation designs — including SAGESA — capture Cycles 1–3 well but have limited evidence for Cycles 4 and 5. This is partly a funding and time horizon problem (programme funding ends before the most significant value is observable), and partly a design problem (longitudinal instruments are not built into the evaluation plan).
Implementers should plan from the outset for at least one longitudinal contact point at 6–12 months post-programme. This can be a brief survey (5–10 items) or semi-structured interviews with a purposive sample. Even a modest longitudinal dataset substantially strengthens the transformative value account.
Note
SAGESA note: Post-evaluation observations indicated that the SAGESA community continued to operate beyond the formal project period — members remained active on platforms, new connections were maintained, and community infrastructure persisted. This provides preliminary evidence of transformative sustainability value, but was not captured through a formal longitudinal instrument. Future CoP evaluations should plan for this explicitly.
Generic VCF indicators require adaptation to be meaningful in an AMR genomics context. Table 7.1 provides a comprehensive indicator framework for genomics network CoPs, mapped to each VCF cycle.
| VCF Cycle | Indicator domain | Specific indicator | Measurement approach |
|---|---|---|---|
| 1 — Immediate | Participation | % webinar/training attendees reporting expectations met or exceeded | Post-event form (scale item) |
| 1 — Immediate | Knowledge gain | % agreeing: 'I gained new knowledge or skills today' | Post-event form (agree/disagree) |
| 1 — Immediate | Platform engagement | Monthly active members / total members (%) | Platform analytics |
| 2 — Potential | Technical skills | % agreeing: 'I have new knowledge of [tool/technique] relevant to my work' | Post-event form; follow-up survey |
| 2 — Potential | Learning objectives | % of training participants agreeing learning outcomes were fulfilled | Post-training evaluation |
| 2 — Potential | Networking utility | % of follow-up respondents finding CoP useful for professional networking | Follow-up survey |
| 2 — Potential | Resources shared | Number of tools, protocols, datasets or publications shared via community channels | Platform content audit |
| 3 — Applied | Skill application | % of follow-up respondents reporting applying specific tools in professional work | Follow-up survey (open item) |
| 3 — Applied | Tool-specific application | Named tools/pipelines reported in use (e.g. assembly, variant calling, phylogenetics) | Follow-up survey (open text) |
| 3 — Applied | Knowledge sharing | % who have shared CoP knowledge with team/institution members | Follow-up survey |
| 3 — Applied | Workflow change | Narrative descriptions of changed analytical approaches or new protocols adopted | Follow-up open items |
| 4 — Realised | Research outputs | Publications, datasets, or reports produced attributable to CoP-acquired skills | Longitudinal check-in |
| 4 — Realised | Surveillance improvement | Documented improvements in AMR genomic surveillance capacity or outputs | Longitudinal check-in; institutional contact |
| 4 — Realised | Institutional benefit | New bioinformatics capacity established; team skills uplifted | Follow-up / longitudinal |
| 5 — Transformative | Community sustainability | Active community beyond programme period; member-led activities | Post-programme observation; longitudinal |
| 5 — Transformative | Professional identity | Self-described role as AMR genomics practitioner; increased leadership | Longitudinal qualitative |
| 5 — Transformative | Field influence | Invited to advisory roles, teaching, policy consultation related to AMR genomics | Longitudinal tracking |
Administer immediately after each webinar or training session. Target completion time: 5 minutes.
| # | Item | Response format |
|---|---|---|
| 1 | Overall, this session met my expectations | Exceeded / Met / Partially met / Did not meet |
| 2 | I gained new knowledge or skills from this session | Strongly agree / Agree / Neutral / Disagree / Strongly disagree |
| 3 | The content was relevant to my current work | Strongly agree / Agree / Neutral / Disagree / Strongly disagree |
| 4 | I intend to apply something from this session in my work | Yes (please describe) / Possibly / No |
| 5 | The most valuable aspect of this session was: | Free text |
| 6 | One thing that would have improved this session: | Free text |
| 7 | How likely are you to recommend this CoP to a colleague? | 0–10 (0=not at all likely, 10=extremely likely) |
Administer at end of a structured training course. Target completion time: 10 minutes.
| # | Item | Response format |
|---|---|---|
| 1 | The learning objectives of this course were fulfilled | Strongly agree – Strongly disagree (5-point) |
| 2 | The practical exercises were at the right level for my current skills | Too advanced / About right / Too basic |
| 3 | I am confident I could now apply [specific tool] independently | Repeat for each tool covered; Yes/Partially/No |
| 4 | The course pace was: | Too fast / About right / Too slow |
| 5 | The most useful session or activity was: | Free text |
| 6 | I would have preferred more time on: | Free text |
| 7 | Any other comments about the training: | Free text |
Send approximately 8–12 weeks after core programme activities end. Target completion time: 15 minutes.
| # | Item | VCF cycle | Response format |
|---|---|---|---|
| 1 | Since participating in [CoP name], I have applied specific tools or skills in my professional work | 3 | Yes / No / Partially |
| 2 | If yes to Q1: please describe what you have applied and in what context | 3 | Free text |
| 3 | Please list any specific bioinformatics tools or pipelines you have used as a result of CoP activities | 3 | Free text |
| 4 | I have shared knowledge or resources from the CoP with colleagues at my institution | 3 | Yes / No |
| 5 | Participation in the CoP has improved my confidence to work with genomic data | 2–3 | Strongly agree – Strongly disagree |
| 6 | The CoP has been useful for professional networking | 2 | Strongly agree – Strongly disagree |
| 7 | I can identify one or more professional outcomes I attribute partly to CoP participation (e.g. new collaboration, research output, capacity in my team) | 4 | Yes (describe) / No |
| 8 | Are you still active in the CoP community (e.g. on Slack, attending events)? | 5 | Yes, regularly / Occasionally / No longer |
| 9 | What has been the most significant value of the CoP for your professional development? | 2–5 | Free text |
| 10 | What would you change or improve about the CoP? | — | Free text |
Brief 5–8 item survey or semi-structured interview guide for longitudinal tracking.
| # | Item / Question | VCF cycle |
|---|---|---|
| 1 | Since completing [CoP], have you produced any research outputs (publications, reports, datasets, presentations) in which CoP-acquired skills played a role? | 4 |
| 2 | Has your institutional role changed in ways related to genomics or AMR work? (e.g. new responsibilities, new team functions) | 4–5 |
| 3 | Are you still connected to other CoP members? In what ways? | 5 |
| 4 | Has the community continued to produce value for you beyond the formal programme period? Please describe. | 5 |
| 5 | Have you contributed to any training, mentoring, or knowledge sharing in AMR genomics that you attribute partly to your CoP experience? | 5 |
| 6 | If you could describe the most significant long-term impact of the CoP for you personally, what would it be? | 5 |
An evaluation report for a VCF-based CoP should follow the same five-cycle structure as the framework, with a clear methods section explaining how evidence was collected and synthesised. The report should be transparent about sample sizes, response rates, and the limitations of the evidence at each cycle.
| Report section | Content | Approx. length |
|---|---|---|
| Executive summary | Key findings across all five cycles, 3–5 recommendations | 1–2 pages |
| Methods | Programme description, evaluation design, instruments, analytical approach | 3–4 pages |
| Findings: Cycle 1 | Engagement data, feedback form results, platform analytics summary | 2–3 pages |
| Findings: Cycle 2 | Knowledge and skills gained, networking value, resources produced | 2–3 pages |
| Findings: Cycles 3–4 | Applied practice changes, reported outcomes, case examples | 3–4 pages |
| Findings: Cycle 5 | Community sustainability, identity, systemic influence | 2 pages |
| Lessons learned | Implementation reflections, what worked, what to change | 2–3 pages |
| Recommendations | Specific, actionable recommendations for future CoPs | 1–2 pages |
VCF findings need to be translated for different stakeholder audiences, each of whom values different aspects of the evidence:
- Funders and programme officers: focus on participation numbers, satisfaction rates, and evidence of practice change (Cycles 1–4). Use clear visuals — bar charts for quantitative items, brief quotes for qualitative insight.
- Community members: share findings back with the community promptly after the follow-up survey. This validates their participation and builds trust. A one-page infographic or brief newsletter is more engaging than a full report.
- Academic audiences: focus on methodological contribution of prospective VCF use, triangulated evidence across cycles, and honest acknowledgement of limitations. Follow-up peer review publication strengthens the evidence base for CoP evaluation approaches.
- Policy and institutional stakeholders: translate Cycle 4–5 findings into institutional and sector-level terms: workforce capacity strengthened, surveillance systems improved, regional knowledge networks established.
Where possible, evaluation data and reports should be publicly archived to support the broader evidence base for CoP evaluation in genomics and AMR. This includes:
- Pre-registering the evaluation design (OSF, AsPredicted)
- Archiving anonymised datasets from surveys on a persistent open repository (Zenodo, Dryad, Figshare)
- Publishing the evaluation report alongside the manuscript as a supplementary document with a persistent DOI
- Following FAIR principles: Findable, Accessible, Interoperable, Reusable
Note
SAGESA approach: The SAGESA evaluation report and supplementary data are archived via a publicly accessible GitHub repository (https://wcscourses.github.io/SAGESA-AMR-Genomics-Network/), with the full repository archived on Zenodo to mint a persistent DOI. This enables the evaluation to be cited, reused, and built upon by future CoP implementers.
The SAGESA CoP (2021–2022) provides a rich implementation case for prospective VCF use in an AMR genomics network. The following lessons are drawn from the evaluation and implementation experience and are offered to inform future CoP designers.
| Lesson | What we found | Recommendation for future CoPs |
|---|---|---|
| 1. Prospective VCF alignment pays off | Mapping instruments to cycles from the outset made synthesis substantially more coherent and reduced post-hoc analytical effort. | Invest design time upfront. Map every evaluation instrument to specific VCF cycles before programme launch. |
| 2. Needs-led design drives engagement | The baseline needs assessment shaped the programme's content priorities and demonstrated responsiveness to the community. High satisfaction rates (100% expectations met/exceeded at webinars) were attributable partly to this alignment. | Always conduct and act on a structured baseline needs assessment. Communicate back to the community how their needs shaped the design. |
| 3. Multi-platform infrastructure extends reach | The combination of Zoom (synchronous), Slack (asynchronous), and a content repository meant the CoP was accessible across time zones and worked for members without live event access. | Design for asynchronous participation from the outset. No single platform serves all engagement styles and contexts. |
| 4. Administration is a substantive function | Community management — moderating platforms, scheduling events, onboarding members, managing communications — consumed significantly more time than anticipated. It cannot be treated as incidental. | Budget explicitly for a community manager (≥0.5 FTE for >200 members). Include administration as a named input in the logic model. |
| 5. Follow-up survey response rates are challenging | The follow-up survey achieved n=27 from a community of ~200 active members. This is sufficient for qualitative insight but limits statistical generalisability. | Plan response rate strategies from the outset: personal invitations, reminders, short forms, incentives. Consider supplementing surveys with brief semi-structured interviews for richer Cycle 3–5 data. |
| 6. Cycles 4–5 require longer time horizons | Evidence for realised and transformative value was limited by the 18-month programme scope. Post-evaluation observations suggested sustained community activity, but this was not captured through a formal longitudinal instrument. | Build a longitudinal follow-up (6–12 months post-programme) into the evaluation plan and budget. Even a brief survey at this point substantially strengthens the evidence base. |
| 7. Peer community is the intervention | The training activities (webinars, bioinformatics course) drove engagement, but it was the CoP infrastructure — the ongoing peer connections, Slack community, and shared professional identity — that generated sustained value beyond individual events. | Design activities as vehicles for community building, not as ends in themselves. The most durable value of a CoP is the community it creates. |
| 8. Plan for sustainability from day one | The CoP continued to operate beyond the formal project period, but sustainability planning was not proactively built into the original design. Member-led sustainability emerged organically rather than being intentionally cultivated. | Include a sustainability plan in the design phase. Identify potential community leads, consider membership models, and establish clear plans for platform maintenance before programme funding ends. |
Use this checklist to confirm readiness at each programme phase.
- Domain and membership criteria defined and documented
- Baseline needs assessment designed, piloted, and distributed
- Needs assessment results analysed and shared with the community
- Programme logic model drafted (inputs → activities → outputs → outcomes)
- Evaluation instruments designed and mapped to VCF cycles
- Community platforms selected, configured, and tested
- Moderation and governance structures in place
- Community manager role resourced (≥0.5 FTE for >200 members)
- Sustainability planning initiated
- Ethics and data governance reviewed (anonymisation, storage, consent)
- Post-event feedback form distributed after every webinar and training
- Platform analytics collected monthly and stored
- Facilitator observation notes completed after each activity
- Evaluation data reviewed periodically for formative learning
- Community engagement monitored; low-engagement periods addressed
- Implementation journal maintained throughout
- Follow-up survey designed and piloted (ready for distribution at programme end)
- Follow-up survey distributed at 8–12 weeks post-programme
- Response rates tracked and reminder strategy implemented
- Evidence synthesised across all five VCF cycles
- Limitations of evidence acknowledged transparently
- Findings shared back with community
- Evaluation report completed and archived with persistent DOI
- Longitudinal check-in scheduled for 6–12 months
- Lessons documented and shared for future implementers
| Term | Definition |
|---|---|
| Applied value (Cycle 3) | Changes in practice — how participants use learning in their professional work. |
| Attribution | The process of crediting an observed change to a specific intervention. In CoP evaluation, 'contribution' language is preferred over strict causal attribution. |
| Community of Practice (CoP) | A group united by shared domain, community interaction, and shared practice — distinct from a network, team, or project group. |
| Domain | The shared subject area or problem space that defines what the CoP is about. |
| Formative evaluation | Evaluation conducted during programme delivery to inform real-time adaptation. |
| Immediate value (Cycle 1) | Value experienced in the moment of participation — engagement, energy, and perceived relevance. |
| Logic model | A structured representation of how a programme is expected to work: inputs → activities → outputs → outcomes. |
| Potential value (Cycle 2) | Learning assets produced — knowledge, skills, relationships, and resources created through participation. |
| Prospective evaluation | Evaluation designed and integrated into programme planning before launch (vs. retrospective evaluation designed after the programme ends). |
| Realised value (Cycle 4) | Documented improvements in performance, research outputs, or institutional capacity attributable to CoP participation. |
| Summative evaluation | Evaluation conducted after programme delivery to assess overall effectiveness and outcomes. |
| Transformative value (Cycle 5) | Systemic and identity-level changes — sustained community, professional identity shift, and field-level influence. |
| Value Creation Framework (VCF) | A five-cycle evaluation framework developed by Wenger-Trayner et al. for understanding how value is created in social learning systems. |