[DMP 2026]: IDinsight Experiment Analytics & Bandits Engine Enhancements

[markbotterill]

Ticket Contents

Description

Evidential is an open-source experimentation platform designed for nonprofits to run rigorous, low-cost experiments (A/B tests, Bayesian methods, and bandits) on digital and hybrid programs. This ticket focuses on building and improving the experimentation analytics and adaptive allocation engine.

Goals & Mid-Point Milestone

Goals

• Continue to build out the multi-armed bandit engine (e.g. Thompson Sampling / UCB)
• Add contextual bandits support using user-level features
• Improve experiment results unified dashboard
• Improve existing integrations with platforms like Turn.io

Setup/Installation

Quickstart guide is here

Expected Outcome

No response

Acceptance Criteria

No response

Implementation Details

Backend:

• Python (FastAPI)
• SQLAlchemy / Postgres
• Frontend in React

Experimentation:

• Frequentist methods:
  • t-tests / proportion tests
• Bayesian methods:
  • Beta-Bernoulli models
• Bandits:
  • Thompson Sampling
  • UCB
  • Contextual bandits (Bayesian linear regression)

Data:

• Event data + warehouse integration
• Metric computation layer

APIs:

• Experiment creation
• Result retrieval
• Allocation updates

Mockups/Wireframes

No response

Product Name

Evidential

Organisation Name

IDinsight

Domain

Open Source Library

Tech Skills Needed

Python, SQL, Docker, CI/CD, React

Mentor(s)

@markbotterill @poornimaramesh

Category

Data Science

[Somsubhra-Nandi]

Hi @markbotterill @poornimaramesh,

I'm very interested in tackling Issue #766 for DMP 2026. I have strong experience building out complex Python backends (FastAPI) and automated data pipelines, and I'd love to help build out the experimentation analytics engine.

I noticed the quickstart guide link and located the main codebase over at agency-fund/evidential-be and have started reviewing the current FastAPI routes and SQLAlchemy models.

Before I draft a formal implementation plan for the Multi-Armed Bandit (Thompson Sampling / UCB) logic, I had a quick architectural question: Are we planning to calculate the metric computation layer purely on the fly within the FastAPI backend, or should the logic rely heavily on the Postgres warehouse integration for pre-aggregating the event data?

I will drop by the Discord soon to discuss further, but I'm excited to start working on this!