VCS Connector for Bitbucket

Overview

The CA Agile Central VCS Connector for Bitbucket Server and Bitbucket Cloud posts information about Bitbucket repository commits to CA Agile Central, relating those commits to CA Agile Central changesets and artifacts if sufficient information is contained in the VCS commit messages to identify the relevant artifacts. The connector will also update the state of relevant CA Agile Central artifacts, specifically defect state or user story schedule state when commit messages contain appropriate syntax, for example US123 In-Progress. The CA Agile Central VCS Connector for Bitbucket is classified as a one-way and one-time mechanism. Information in Bitbucket is never altered; information is only written in Agile Central and no duplication of data is attempted or allowed. The CA Agile Central VCS Connector for Bitbucket consists of software that you run on your platform according to your desired schedule. The configuration of the connector is policy-based, meaning that you do not need to provide a separate configuration file for each Bitbucket repository for which you want the connector to operate. The policy-based nature allows you to scope to all or a subset of repositories accessible to a Bitbucket user or organization.

The Agile Central VCS Connector as of version 2.0.3 includes functionality that detects the creation of Bitbucket pull request items and posts corresponding pull request items in Agile Central when the configuration option for this is present in the target configuration file. Currently, the pull request functionality is only available on User Stories. To show pull requests in Agile Central as Third Party Context, add the PullRequests value set to True. The AC artifact formatted ID (ex. US1234) can be mentioned in either the pull request title or in of one of the commit messages that is associated to the pull request in order to be processed by the connector. The connector will include open as well as a merged pull requests.

This connector uses Bitbucket REST API, and will operate with both Bitbucket Server and Bitbucket Cloud. Depending on if you are using Bitbucket Server or Bitbucket Cloud, there will be minor variations in the configuration file items.

This guide includes:

Software Requirements

Connector Download

To download the connector, follow the steps here.

Installation

  1. Note: You must have Python 3.5.x or 3.6.x available in your environment.
  2. Run these commands:
    pip3 install requests==2.18.4
    pip3 install pyral==1.4.0
    pip3 install PyYAML==3.12
    pip3 install cryptography==1.7.1
  3. Unpack vcsconn-2.0.3.zip. Change your working directory (cd) to a directory where you want to install the connector. Unzip vcsconn-2.0.3.zip (or use a suitable program that can unzip a .zip file):
        cd vcsconn-2.0.3
        ls -laR # observe the unpacked contents, or use dir on Windows

         vcsconn                       # vcsconn module root directory
         vcs2ac                        # connector initiation script, this is what you will run
         config                        # holds any configuration files used with this connector
            sample_bitbucket_server.yml# a sample configuration to use as a base reference for Bitbucket Server
            sample_bitbucket_cloud.yml # a sample configuration to use as a base reference for Bitbucket Cloud
            sample_user_map.yml        # a sample text file for user mapping
         log                           # hold log file and time file per configuration
         README_BITBUCKET.txt          # this file
         User-Mapping-Strategies.txt   # details Bitbucket user to AgileCentral username mapping strategies
  4. **Please navigate to the bottom of this document (Downloads section) for a sample file of a config with all the options available for this connector. The file is called: "a sample file with explanatory notations.txt"

Setup

General recommendations:

  1. Start simple.

  2. Locate the configuration subdirectory.

  3. Copy the sample_bitbucket_(server or cloud).yml file to a file named suitably for your environment for example, cp sample_bitbucket_server.yml to product_x.yml (or some other suitably named file that has a .yml suffix).

  4. Edit your product_x.yml file. For example, change the sample values for credentials, workspace, project, token, and so on to values that are relevant and valid for your environment.

  5. Decide on the appropriate SecurityLevel.

  6. Decide if you want to update state of CA Agile Central artifacts based on commit messages.

  7. Decide if you want to map a Bitbucket committer to a valid Agile Central user and choose one of the three mapping methods: Passthrough, FileBasedUserNameLookup, or UserNameDomainAugmentLookup.

  8. Decide on whether the repositories you want processed are repositories you own (use UserSlug) or are repositories owned by a team of which you are a member (use TeamSlug).

 

       *See Appendix A on configuration file syntax.

Operation

Manual

Using a terminal window or console: cd to the installation root directory, for example /opt/local/sw/vcsconn-1.2.4. Then run python3 vcs2ac product_x.yml. This software requires that the configuration file reside in the configuration subdirectory. You specify the name of the file on the command line. Do not specify the subdirectory in the command line argument.

Scheduled

Use either cron, launchctl, or Windows Task Scheduler. Make sure the environment in effect when running this software has an appropriate environment set so that you can run: python3 $VCS/vcs2ac your_config_file_name.yml where $VCS is the reference to an environment variable containing the fully qualified path to the directory where the software is installed. For example, if you unzipped the package in /opt/local/sw, then your VCS would be set like this: export VCS=/opt/local/sw/vcsconn-1.2.4.

Time File

In normal operation, the connector writes a time file (in the log directory) whose name is based on the configuration file name. Example: If the configuration file name is product_x.yml, then the associated time file name would be product_x_time.file. The content of the time file is a single line containing a human readable date/time stamp value in the format YYYY-MM-DD hh:mm:ss Z. The value represents the timestamp of the last commit reflected in Agile Central. When the connector is run a subsequent time, it consults the time file to determine which jobs need to be considered for the current run by only processing the commits whose start time is greater than or equal than the time file value. It is possible to set the time file value artificially (using ISO format, such as 2017-05-12 11:37:55 Z), so that you can go back in time and pick up commits from some arbitrary point in the past. The connector does not duplicate commits so you do not have to worry about duplicated information getting posted to Agile Central. In the absence of a time file, the connector pulls commits starting at a point three days prior to the current time.

Troubleshooting

The connector always writes a log file named based on the configuration file name. The log file is written into the log subdirectory under the base installation directory. You can control the extent of logging by specifying the level in the configuration file. Within the configuration file, you can set the LogLevel which determines the amount of logging information written to the log file. Under the Service section, modify the LogLevel setting (valid values are ERROR, WARN, INFO, DEBUG). When you set the LogLevel to DEBUG, you will get the full range of logging messages that can be helpful in pinpointing where a problem occurred and what information was being processed at the time of the anomaly.

It can be very helpful to run the connector in Preview mode when setting things up for the first time. This allows you to get the connections to Agile Central and Bitbucket to initialize and validate correctly without posting any changesets.

*Issue: Pull Requests are not transferring to Agile Central.
Answer: Currently, pull request functionality is only available for User Stories in Agile Central. If you are not seeing Pull Requests in User Stories, please verify the following line is in the service section config file:

     PullRequests : True # Default is False, when True VSTS PullRequests are detected and reflected in Agile Central


*Issue: Error seen when using * in the include statement to pull all repos.

FATAL: VCSConnectorRunner.run - VCSConnectorRunner.getConfiguration vcs_connector_runner.py(288) - Unable to parse config/vsts-cloud.yml successfully, while scanning an alias
  in "<unicode string>", line 25, column 26:
                Include   :  *
                             ^
expected alphabetic or numeric character, but found '\n'
  in "<unicode string>", line 25, column 27:
                Include   :  *
                              ^
Answer: Escape the * in the include statement with quotes.
        Repository:
            Include   : "*"

Security Level

When credentials, including ProxyUsername and ProxyPassword, in the configuration file are no longer in clear text, SecurityLevel settings can only upgraded, but not downgraded. For example, a configuration file with encrypted credentials cannot be used with SecurityLevel reset from Encrypted to Encoded. If users want to downgrade security level, they have to start with a configuration file where credentials are in clear text.

Appendix A - Configuration file editing

The Agile Central VCS connector for Bitbucket uses a text file in the YAML format. For complete information, consult the web page at or any other website on the subject of YAML. Please see the Download Samples section at the bottom of this page for a sample configuration file with explanatory notes.

For brevity, this document mentions several of the most relevant syntax items and covers the three sections of a valid YAML configuration that can be used with the connector.

  • Use a text editor (not MS Word or Google Doc) to edit the file.
  • Never use a tab character in the file. YAML does not allow or recognize tab characters.
  • Save the file in UTF-8 format.
  • Use a monospace font.
  • Be consistent with the number of spaces you use to indent.
  • On a line, the first occurrence of a non-quoted # character indicates a comment, the # character and all characters to the right are ignored in processing.
  • Keep the sections in the same order as is present in the sample.yml file.
  • Be aware that the colon character : is significant, it separates a key from the value.
  • Be aware that the dash character - is significant, it denotes the start of a list which may have one or more key value pairs that constitute the list item.
  • You usually do not have to quote values if they have spaces in them; you will have to quote the value if it contains an embedded # character.

Skeleton of the template_config.yml file
 Bitbucket:
         AgileCentral:
             ...  # several key value pairs are relevant for this section
         Bitbucket:
             ...  # several key value pairs are relevant for this section
         Service:
             ...  # a few key value pairs relevant for the overall operation
                  # of the connector appear in this section
 

The AgileCentral section specifies values to use to obtain a connection with Agile Central. The Bitbucket section specifies values to use to obtain a connection with Bitbucket and specify the policies governing what repositories in Bitbucket get processed to result in posting of changesets to Agile Central. The Service section controls some aspects of the connector behavior on an overall basis.

*The required Project line in the CA Agile Central section is used for for connection purposes; commits will still show in CA Agile Central based on the workspace scope.

Revision History

  • 2.0.3-Master --- 9-Feb-2018
    • Fixed name of StateExtractor class in sample config file
    • Fixed assignment of Changeset URI for Bitbucket Server
  • 2.0.0-Master --- 21-Dec-2017
    • Unified all newer policy-based VCS connectors into one distribution package
  • 1.3.2-Master --- 7-Dec-2017
    • Added support for pull requests
  • 1.3.1-Master --- 4-Dec-2017
    • Transition from author to committer for Bitbucket Server
    • Added support for appending committer name to commit message
  • 1.2.4-Master --- 17-Nov-2017
    • Connector rewritten in Python
    • Addition of Policy-based feature
    • Support added for Bitbucket Cloud

Feedback

Need more help? The CA Agile Central Community is your one-stop shop for self-service and support. To submit feedback or cases to CA Agile Central Support, find answers, and collaborate with others, please join us in the CA Agile Central Community.