Improving the Team Synchronize support

This document outlines how other plug-ins can participate in the Synchronize View and how the Synchronize View is being refactored for Eclipse 3.0. Interested parties should review this document and verify that their use-cases are addressed and that the solution satisfies their needs.

The work on the Sychronize View is being done to support the following 3.0 plan items:

• 37705 Improve team API
  
  In addition to supporting a generic headless Team API numerous plug-ins are also interested in using the Synchronize technology. Currently the CVS plugin uses this support but we haven't officialy made it available for others. We feel that this is a complicated piece that could be re-used by many other Team plug-ins to provide rich integration of their tool into Eclipse.
  
  
• 36957 Support concurrent activities
  
  The Synchronize operation is an excellent candidate for running in the background. Some changes are required to the sync model to accomodate concurrent calculation of resource synchronization.

This document doesn't cover JSR-147 or CVS concurrent operations.

Last Modified: February 24th, 2004

Motivations

As stated above, there are two motivating reasons for the 3.0 work relating to the Team Synchronize APIs: (1) to enhance workflows, (2) to provide an API for other plug-ins to use. Note that synchronization support doesn't exclusively imply that the Synchronize View is used to display and manipulate the synchronization states. Instead, it is our intention to provide a framework for creating, managing, and then finally displaying this synchronization state. The presentation of the synchronize states can be flexible, and could be then shown in either the Synchronize View, or in other more appropriate places (e.g. editors, dialogs, wizards).

This is a key for Team plug-ins for rich integration into Eclipse. The motivations for the proposed workflow changes changes are described below.

Dynamic syncrhonization model support

One of the most important changes we are proposing is to migrate from a static sync model to a dynamic one. Previously, a user would synchronize a set of resources with their remotes and subsequent changes made in the workspace would only be reflected in the synchronization state shown in the view if the user explicitly re-synchronized. This also meant that you could not browse local changes without running a potentialy long operation. This static model also caused problems if we wanted to refresh the remote state in the background and incrementally update the sync model as remote changes were found.

Synchronize View re-use

The view should support showing multiple types of synchronization targets. The old sync view was re-used by clients but the synchronization state of the previous type was lost. For example, synchronizing with a Dav server then again with CVS. We are currently evaluating ways of providing a view that shows different contexts. Some of the options are to have a dropdown to select the different subscribers or to have a tabbed view. I like the drop down because it takes less screen real estate and you don't have the tab problem of truncating the titles.

Flexible Presentation of Synchronization state

Background event handling

Background fetching of remote state

Currently when a project is synchronized the complete remote tree is fetched and a progress dialog is shown. The user can't perform any other operation while the remote tree is fetched. There is no reason why this couldn't be fetched in the backgroup without blocking the UI and would allow an awareness of what other people in the team are working on and help reduce the time needed to find out what others are doing.

Combine Synchronize/Merge/Compare

The terminology and look and feel of the CVS merge editor, compare editors, and sync view is different. Basically a merge is the same as synchronizing except for the fact that you can't commit changes. Merging is already a tricky operation and having a conflicting usage model between the sync and merge makes it even harder. (35577)

Compare criterias

The comparison criteria that is used to calculate the sync state is hidden from the user and it isn't obvious for them when the criteria should be changed and what advantage it will have. This is also seen when a user synchronizes a project against an existing project on the server and notices that most files are conflicting. There is no easy steps for calculating the 'real' changes. This may be solved by individual Team plug-ins, but it is worth noting that it is a problem encountered in the Synchronize View. (24887, 21612, 28143)

Splitting the Synchronize View from the Compare components

We would like to split the Synchronize view from the compare components and have a dedicated view that shows synchronization state in different forms and then allow the user to browse the changes using the compare editors. Here is a list of motivating reasons for the split:

• Would like to use real editors for merging instead of the neutered ones currently used. For example, the java editor shown in the sync view looks real (e.g. syntax coloring) but is a scaled down java editor. This causes integration and context problems because it is very useful to try and write code in these editors without resorting to opening the file with the real java editor. Then going back to the sync view only to find that the file hasn't refreshed.
• We wanted more flexibility in showing resource synchronization in different ways: table, tree, tabletree, split view with statistics.... Currently the compare DiffTreeViewer subclasses must be a tree and this limits our implementation flexibility.
• The edit/save workflow in a view has always been strange. You couldn't edit, look at another file, then return to your edit without being forced to save the change first. This is not intuitive behavior, and nowhere else in the platform are your forced to save changes just to see another file.
• It made sense to use a perspective to accomplish what the old sync view did by showing all related views (structure diff, structure compare, merge viewers) together.

Show helpful change counts

It would be helpful for the user to understand the scope of changes that are remote (incoming) and local (outgoing) regardless of the filters being applied to the changes. This would help cases were users forget changes because they have been filtered out of the view.

Working Sets

Because the Synchronize view can show the changes for all resources in the workspace that are configured with the current synchronization target the user can use working sets to scale the list of projects to synchronize. When a working set is active the change counts change to show both the counts in the current working set and those in the entire workspace. This can help decide when to disable a working set.

The CVS commands that used to prime the Synchronize View now use a dynamically created working set to automatically reduce the list of projects in the view. For example, when sharing a project for the first time or synchronizing via the Team > Synchronize with Remote menu, a working set is created.

Conflicts are propagated to parent folders

To ensure that conflicts are never ignored, the conflict icon is propagated to parent folders. This will also allow users to quickly find the conflicts.

General usability enhancements

The structure compare part of the 2.1 Synchronize View has many usability problems that we would like to improve.

• We propose that synchronization state be displayed in different formats (table, compressed tree) so that things you are interested in (the leaves of the tree) are not constantly separated by a great distance. This would make it easier to see at a glance what has changed. It also makes multi-selection operations less painful, for example comitting a set of related files that should have the same comment. (12453, 16249)
• There are several also several expansion/selection usability problems that we would like to fix. (35187, 33041, 4931, 5431, 20847, 27237, 16249)
• Synchronize view modes (filtering) and conflict workflows must be improved. This is somewhat related to showing helpful change counts. (10556, 9451, 139, 19712)

The Synchronize API

org.eclipse.team.core.synchronize

This package specifies the API for managing the synchronization state between the local workspace resources and a corresponding variants of those resources. The classes in this package can be used by Subscribers (see the org.eclipse.team.core.subscribers package) or others. The classes are roughly divided into three categories:

• describing the synchronization state of a one or more resources,
• notifying interested parties of changes in the synchronization state.
• filtering a set of resource based on a sync state criteria

org.eclipse.team.core.subscribers

This package specifies the API for Team subscribers. A Subscriber provides access to the synchronization state between the local workspace resources and a set of variants of those resources, whether it be a code repository or some other type of server (e.g. FTP). A subscriber is typically associated with only a subset of the resources in the local workspace, referred to as the set of resources the subscriber supervises. The supervised local resources have a corresponding variant state which describes the state of the variant resources that correspond to the local resources.

A Subscriber provides:

• a set of root resources that define the subset of resources in the workspace that the subscriber supervises (some children of the roots may not be supervised, as indicated by the isSupervised method).
• access to the synchronization state (using SyncInfo) between the resources it supervises and their corresponding variant resources.
• the ability to refresh the the remote state. This implies that the Subscriber would be responsible for caching variants.
• change notification to registered listeners (of type ISubscriberChangeListener) when the variant state changes or when roots are added or removed.

org.eclipse.team.ui.synchronize

The packaage specifies a set of classes and interfaces to support participating in the Synchronize View. Synchronize participants are declared by extending the synchronizeParticipants extension point. There are two classes of synchronize participants: static participants will be created when the synchronize view is created, and dynamic participants that are created by user code at some other time. A synchronize manager (ISynchronizeManager) manages all active synchronize participants, and provides notification of participants which are added and removed. Participants are displayed in a page book view. Each participant implementation is reponsible for creating its page (IPageBookView), which provides freedom of presentation to the synchronize view implementation. A single participant may be displayed simultaneously in multiple synchronize views, and in different workbench windows.

org.eclipse.team.ui.synchronize.viewers

This package contains classes that support displaying synchronization information described by SyncInfo. If you consider the synchronization model described by the classes in org.eclipse.team.core.synchronize as the physical representation of resource variants then the classes in this package allow you to control how the model is presented to the user. One key feature of these APIs is that they support managing dynamic models.

There is no specific support for the Synchronize View in this package. Instead, the classes here can be used to build a page that can be shown in the view, but they can also be used to show synchronization information in dialogs, wizards, and editors.

org.eclipse.team.ui.synchronize.subscribers

This package contains a classes that provides an implementation of a synchronize participant that enables synchronization for a Subscriber. For providers that implement a Subscriber, this is the easiest method of integrating into the Synchronize View. The TeamSubscriberParticipant provides a view of changes (incoming, outgoing, conflicting), modes for showing only a subset of the changes, decorations for identifying the changes, and working sets.

How-To Guide

This section describes how to make use of the Synchronize API including how to integrate into the Team Synchronize view. It is divided into two parts. The first describes how to implement a Subscriber, either from scratch or by building on top of an existing local workspace synchronization infrastructure. The second part describes how to make use of the UI components of the Synchronize API for showing Subscriber state in the Team Synchronize view and other views, editors and dialogs.

Implementing a Subscriber

This section presents a few different perspectives on implementing a subscriber. First, the general anatomy of a subscriber is discussed, followed by a few examples that build on API provided in the org.eclipse.team.core plugin.

The Anatomy of a Subscriber

A subscriber provides access to the synchronization state between the local workspace and another location that mirrors the local workspace resources. This other location is usually, but not necessarily, on a different machine such as a server. In order to describe what a subscriber is, we must first present some of the terminoligy we are going to use:

Resource Variant: A local resource that is mapped to a resource that exists at another location can be referred to as a variant of that resource. That is, the resources are usually very similar but may differ slightly (either due to modificatons to the local resource or changes made the remote copy made by other users). We take a local workspace centric view of this, referring to the local copy as the resource and any remote copy as resource variants.

Two-way vs. Three-way Synchronization: There are two basic types of synchronization state determination: two-way and three-way. A two-way comparison only considers the local resource and a single resource variant, referred to as the remote resource variant. This type of comparison can only show the differences between the two resources but cannot offer hints as to how the changes interelate. Most code repository systems support a three-way comparison for synchronization state determination. This type of comparision involves the local resource, a remote resource variant and a base resource variant. The base resource variant represents a common ancestor for the local and remote resources. This allows for more sophisticated synchronization states that indicate the direction of the change.

A subscriber can then be described as providing access to the synchronization state between the resources in the local workspace and a set of resource variants for these resources using either a two-way or three-way comparison, depending on the nature of the subscriber. A subscriber provides the following capabilities:

local workspace traversal: a subscriber supports the traversal of the local workspace resources that are supervised by the subscriber. As such, the subscriber has a set of root resources that define the workspace subtrees under the subscriber's control, as well as a members method that returns the supervised members of a workspace resource.

resource synchronization state determination: For supervised resources, the subscriber provides access to the synchronization state of the resource, including access to the variants of the resource. For each supervised resource, the subscriber provides a SyncInfo object that contains the synchronization state and the variants used to determine the state. The subscriber also provides an IResourceVariantComparator which determines whether two-way or three-way comparison is to be used and provides the logic used by the SyncInfo to comparing resource variants when determining the synchronization state.

refresh of synchronization state and change notification: Clients can react to changes that happen to local resources by listening to the Core resource deltas. When a local resource is changed, the synchronization state of the resource can then be re-obtained from the subscriber. However, clients must explicilty query the server to know if there are changes to the resource variants. For subscribers, this process is broken up into two parts. A client can explicitly refresh a subscriber. In response the subscriber will obtain the latest state of the resource variants from the remote location and fire synchronization state change events for any resource variants that have changed. The change notification is separate from the refresh since there may be other operations that contact the remote location and obtain the latest remote state.

As stated above, the purpose of the Subscriber API is to provide access to the synchronization state between the local workspace and a remote location. Obviously, underneath the hood, their must be an infrastructure for managing that state. In general, the synchronization state consists of the following basic states.

• local resource is synced with the remote: when a resource is loaded from or written to a remote location, it is consider in-sync with the remote variant. At this point, the local timestamp and remote timestamp are recorded to support future synchronization state checks.
• local resource is modified: If the recorded in-sync local modification timestamp differs from the actual timestamp, the resource is locally modified.
• remote resource is modified: If the timestamp of the remote resource variant differs from the recorded remote timestamp, then the resource has been remotely modified (most likely by a 3rd party).

The infrastructure that maintains the states must persist the timestamps obtained when a resource is synced.

Implementing a Subscriber From Scratch

When implementing a subscriber from scratch, you can make use of some helpful API provided in the org.eclipse.team.core plugin. The org.eclipse.team.core.variants package contains two subclasses of Subscriber which can be used to simplify implementation. The first is ResourceVariantTreeSubscriber which will be discussed in the second example below. The second is a subclass of the first: ThreeWaySubscriber. This subscriber implementation provides several helpful classes for managing the synchronization state of a local workspace. If you do not have any existing infrastructure, this is a good place to start.

Implementing a subscriber from scratch will be illustrated using our file system example available in the org.eclipse.team.examples.filesystem plugin. The code in the following description is kept to a minimum since it is available from the dev.eclipse.org repository. Although not technicaly a three-way subscriber, the file system example can still make good use of this infrastructure. The FTP and WebDav plugins also are built using this infrastructure.

ThreeWaySubscriber

For the file system example, we already had an implementation of a RepositoryProvider that associated a local project with a file system location where the local contents were mirrored. FileSystemSubscriber was created as a subclass of ThreeWaySubscriber in order to make use of a ThreeWaySynchronizer to manage workpace synchronization state. Subsclasses of this class must do the following:

• create a ThreeWaySynchronizer instance to manage the local workspace synchronization state.
• create an instance of a ThreeWayRemoteTree subclass to provide remote tree refresh.
• The class FileSystemRemoteTree was defined for this purpose
• implement a method to create the resource variant handles used to calculate the synchronization state.
  • The class FileSystemResourceVariant (a subclass of CachedResourceVariant) was defined for this
• implement the roots method.
  • The roots for the subscriber are all the projects mapped to the FileSystemProvider. Callbacks were added to FileSystemProvider in order to allow the FileSystemSubscriber to generate change events when projects are mapped and unmapped.

In addition to the subscriber implementation, the get and put operations for the file system provider were modified to update the synchronization state in the ThreeWaySynchronizer. The operations are implemented in the class org.eclipse.team.examples.filesystem.FileSystemOperations.

ThreeWaySynchronizer

ThreeWaySynchronizer manages the synchronization state between the local workspace and a remote location. It caches and persists the local, base and remote timestamps in order to support the efficient calculation of the synchronization state of a resource. It also fires change notifications to any registered listeners. The ThreeWaySubscriber translates these change events into the proper format to be sent to listeners registered with the subscriber.

The ThreeWaySynchronizer makes use of Core scheduling rules and locks to ensure thread safety and provide change notification batching.

ThreeWayRemoteTree

A ThreeWayRemoteTree is a subclass of ResourceVariantTree that is tailored to the ThreeWaySubscriber. It must be overriden by clients to provide the mechanism for fetching the remote state from the server. ResourceVariantTree is discussed in more detail in the next example.

CachedResourceVariant

A CachedResourceVariant is a partial implementation of IResourceVariant that caches any fetched contents for a period of time (currently 1 hour). This is helpful since the contents may be accessed several times in a short period of time (for example, to determine the synchronization state and display the contents in a compare editor).

Building on Top of Existing Workspace Synchronization

Many repository providers may already have a mechanism for managing their synchronization state (e.g. if they have exisitng plugins). The ResourceVariantTreeSubscriber and its related classes provide the ability to build on top of an existing synchronization infrastrucutre. This is the superclass of all of the CVS subscribers, for instance.

ResourceVariantTreeSubscriber

subclasses must provide

base and remote trees

resource comparator

roots

ResourceVariantTree

base and remote tree

creation of resource variant handles

Traversal of a resource variant tree

caches resource variant bytes in a ResourceVariantByteStore

can use Persistant or Session stores or can provide a custom implementation that access the bytes from some other store. For example the ThreeWayRemoteTree makes use of a byte store implementation that stores the remote bytes in the ThreeWaySynchronizer.

subclasses must provide code that fetches the current state of the tree from the server and that creates the resource variant handles

IResourceVariantComparator

SyncInfo

How Does IRemoteSyncElement Relate to Subsciber

Example: synchronizing with a remote file-system

1. Create a subscriber to manage the sync states for the files on the workspace. for now we will use the timestamp at the time the file was copied into the workspace from the remote location
2. decide how to refresh the variants
   
3. create a participant
4. create a set of actions that work on diff nodes