Fundraising tech/Contribution tracking

Current tracking flow for a one-time card donation

Queue-mediated flow for one-time card donation

Contribution tracking, as of early 2020, is the last piece of our payments infrastructure that ties the payments-wiki frontend to the database server in the backend. We want to be able to take the backend db server down for maintenance without needing to shut down the payments frontend. To do this, we will stop creating drupal.contribution_tracking rows directly from the wikis and move to a queue-based insert system.

Sequence Generator

The first step is to stop autoincrementing ids in the database so that updates can happen asynchronously from the front end. To that end, we are using a sequence generator to create non-duplicate unique identifiers.

The sequence generator is provided by a configurable class that implements an interface, all defined in SmashPig under the Core\SequenceGenerators namespace.

• Implementations for Redis and PDO backends are provided.
  • We will use the Redis one in production
  • We use the PDO one with SQLite for tests.
• There is a convenience method Factory::getSequenceGenerator($sequenceName) that instantiates the generator defined in SmashPig's main.yaml under key sequence-generators/$seqenceName. Currently only one is defined, named contribution-tracking.

The transition

In DonationInterface, the new global

$wgDonationInterfaceEnableContributionTrackingQueue

needs to be set to true to enable pulling new IDs from the sequence generator and sending c_t message to the queue. In the CRM codebase, instead of an enabling variable we will deploy a patch that fully switches c_t table writes over to using the c_t sequence generator and queue.

CRM Implementation

The ContributionTrackingQueueConsumer class consumes messages from the c_t queue and inserts or updates messages in the contribution_tracking table. If messages are coming in with ct_ids that are already in the table and have more attributes than just a contribution ID, that could mean the sequence generator is malfunctioning.

Source for diagrams

The diagrams were created with sequencediagram.org

title Current Contribution Tracking (one-time card)

participant DB
participant CRM
participant Queue
participant Payments


group DonationData::normalize
Payments->DB:Insert row with non-normalized data
DB->Payments:Return autoincremented ID for use in creating order ID
Payments->DB:Update row with normalized data
end
group GatewayAdapter::postProcessDonation
Payments->Queue:Donation message with ct_id
end
group DonationQueueConsumer
Queue->CRM:Pop donation message
group wmf_civicrm_contribution_message_import
CRM->DB:Update c_t row with contribution_id
end
end

title Queue Contribution Tracking (one-time card)

participant DB
participant CRM
participant Queue
participant SequenceGenerator
participant Payments

group DonationData::normalize
Payments->SequenceGenerator:get next ID
Payments->Queue:C_T message with normalized data
end
group ContributionTrackingQueueConsumer
Queue->CRM:Pop C_T message
CRM->DB:Write new row
end 
group GatewayAdapter::postProcessDonation
Payments->Queue:Donation message with ct_id
end
group DonationQueueConsumer
Queue->CRM:Pop donation message
group wmf_civicrm_contribution_message_import
CRM->Queue:C_T message with contribution ID
end
end
group ContributionTrackingQueueConsumer
Queue->CRM:Pop C_T message with contrib ID
CRM->DB:Update row
end