Translation Sync

At it’s core, Pontoon is a user interface for editing translations that are stored in a version control system. Because Pontoon does not directly edit the VCS files whenever a user submits a translation, it has to maintain a database of what it thinks the translated strings are. And, periodically, it has to sync with version control to pull newly-submitted strings and translations committed directly, as well as to write its own changes back.

This document describes that sync process in detail.

Triggering a Sync

Pontoon is assumed to run a sync once an hour, although this is configurable. When a sync is triggered, Pontoon finds all projects that are not marked as disabled within the admin interface and schedules a sync task for each one. Sync tasks are executed in parallel, using Celery to manage the worker queue.

Syncing a Project

Syncing an individual project consists of roughly the following steps:

  • Pull the latest commit from version control.
  • Check for changes in VCS or in Pontoon, and if there are none, skip syncing completely.
  • Find all the possible entities across all the resources for this project, searching both the Pontoon database and VCS.
  • For each entity found, compare the VCS version to the Pontoon version (also known as the database version) and decide how to sync it. These changes are collected in a “Changeset” object.
  • Once all the entities are compared, execute the changes in the Changeset.
  • Commit any changes made in the filesystem back to the VCS. If there are no changes, no commit is made.
  • Clean up leftover information in the database.

Comparing Entities

The heart of the syncing process is comparing an entity stored in Pontoon’s database with its matching entity in the resource file in VCS and modifying both the database and the VCS file so that the two are in sync. It’s this process that determines when to update VCS with a submitted translation from Pontoon vs. when to update Pontoon with a translation from VCS, as well as other possible actions.

The comparison takes into account:

  • Whether an entity exists in the Pontoon database or VCS. VCS may be missing an entity due to a developer removing the source string, or Pontoon may be missing an entity due to a new string being added to VCS.
  • Whether a specific locale in VCS has an entity that Pontoon can update.
  • Whether an entity has changed in the Pontoon database since the last sync. This tracks if translations for an entity have been updated or deleted since the last time Pontoon synced.

The actual comparison logic goes something like this:

digraph sync_decision_tree { compare_start[label="Comparing VCS/Pontoon entities"] is_vcs_missing[label="VCS entity\nmissing?" shape=diamond]; vcs_missing[label="Mark as\nobsolete"]; is_db_missing[label="Pontoon entity\nmissing?" shape=diamond]; db_missing[label="Add entity to\nPontoon"]; for_each_locale[label="For each locale\navailable in the project" shape=rectangle]; is_vcs_translation_missing[label="VCS entity missing\nin locale?" shape=diamond] vcs_translation_missing[label="Skip entity,\ncannot update"]; has_db_changed[label="Has Pontoon entity\nchanged since\nlast sync?" shape=diamond]; db_changed[label="Update VCS with\nPontoon translation"]; db_unchanged[label="Update Pontoon with\nVCS translation"]; compare_start -> is_vcs_missing is_vcs_missing -> vcs_missing[label="Yes"] is_vcs_missing -> is_db_missing[label="No"] is_db_missing -> db_missing[label="Yes"] is_db_missing -> for_each_locale[label="No"] for_each_locale -> is_vcs_translation_missing is_vcs_translation_missing -> vcs_translation_missing[label="Yes"] is_vcs_translation_missing -> has_db_changed[label="No"] has_db_changed -> db_changed[label="Yes"] has_db_changed -> db_unchanged[label="No"] }

Executing Changes

Entity comparison produces a Changeset, which is used to make the necessary changes to the database and resource files.

Changesets can perform 4 different operations on an entity:

Update Pontoon from VCS
Add a translation from VCS to Pontoon if necessary. Existing translations that match the VCS translation are re-used, and all non-matching translations are marked as unapproved.
Update VCS from Pontoon
Add a translation from Pontoon to VCS, overwriting the existing translation if it exists.
Create New Entity in Pontoon
Create a new entity in the Pontoon database, including the VCS translation if it is present.
Obsolete Pontoon Entity
Mark an entity in the database as obsolete, due to it not existing in VCS. The entity will no longer appear on the website.

When possible, Changesets perform database operations in bulk in order to speed up the syncing process.