CA Agile Central VCS Connector for GitHub and GitHub Enterprise

Overview

The CA Agile Central VCS Connector for GitHub and GitHub Enterprise (GHE) posts information about GitHub repository commits to Agile Central (AC), and relates those commits to AC changesets and artifacts if sufficient information is contained in the VCS commit messages to identify the relevant artifacts.

Example:

"Fixed DE1987 by changing preamble paragraph 3"

If Agile Central defect DE1987 had a state of Open prior to the commit and run of the connector, then subsequent to the operation of the connector processing this particular changeset, the state of Agile Central defect DE1987 would display as Fixed. Please note this message is case-sensitive, fixed is not the same as Fixed. The .yml file will need UpdateArtifactState value set to True to process a state change.

The commit message may contain references to more than one artifact. For example, this is known to work:

"Test commit msg with multiple artifacts Fixed DE9 Closed DE8"

Pull requests are supported by the connector. To show pull requests in Agile Central as Third Party Context, add the PullRequests value set to True. The AC artifact formatted ID (ex. US1234) can be mentioned in either the pull request title or in one of the commit messages that is associated to the pull request in order to be processed by the connector. The connector will include open as well as a merged pull requests.

The Agile Central VCS Connector for GitHub is classified as a one-way and one-time mechanism. Information in GitHub is never altered; information is only written in Agile Central and no duplication of data is attempted or allowed. The Agile Central VCS Connector for GitHub consists of software that you run on your platform according to your desired schedule. The configuration of the connector is policy-based, meaning that you do not need to provide a separate configuration file for each GitHub repository for which you want the connector to operate. The policy-based nature allows you to scope to all or a subset of repositories accessible to a GitHub user or organization.

This guide includes:

Software Requirements

Connector Download

To download the connector, follow the steps here.

Installation

  1. Note: You must have Python 3.5.x or 3.6.x available in your environment.
  2. Unpack the ZIP file:
    cd MyInstallDir # change your working directory to where you want to install the connector
    unzip vcsconn-2.0.3.zip  #  or use a suitable program to unzip the file
    cd vcsconn-2.0.3
    ls -laR # observe the unpacked contents, or use dir on Windows
            #       ghconn                  # ghconn module root directory
            #       vcs2ac                  # connector initiation script, this is what you will run
            #       configs                 # holds any config files used with this connector
            #       sample.yml              # a sample config to use as a base reference
            #       sample_user_map.txt     # a sample text file for user mapping
            #       logs                    # hold log file and time file per configuration
            #       README.txt              # installation and users guide
  3. Run these commands:
    pip3 install requests==2.18.4
    pip3 install pyral==1.4.0
    pip3 install PyYAML==3.12
    pip3 install cryptography==1.7.1
  4. **Please navigate to the bottom of this document (Downloads section) for a sample file of a config with all the options available for this connector. The file is called: "Github sample file with explanatory notations.txt"

Setup

  1. Set up CA Agile Central:
    • Verify that your target WorkspaceConfiguration object has BuildandChangsetEnabled set to true.
    • If not, your CA Agile Central workspace administrator needs to enable this for your target workspace. If it is not enabled, the connector will work, but you will not be able to see any of the changeset or change information that is associated with your Agile Central work items (story, defect, task).
    • To enable this optiion, the workspace administrator must edit the workspace and do the following:
      • Click the Setup icon .
      • Click the Workspaces & Projects tab.
      • On the Workspaces & Projects summary page, select your workspace.
      • From the Actions drop-down, select Edit.
      • On the workspace editor, select the checkbox labeled Enable Build and changeset.
      • Click Save & Close.
  2. Setup a configuration file:
    • Locate the "config" subdirectory.
    • Copy the sample.yml file to a file named suitably for your environment (for example: "cp sample.yml to product_x.yml" --- or some other suitably named file that has a .yml suffix).
    • Edit your product_x.yml file.
    • Change the sample values for credentials, Workspace, Project, Job, View, Folder to values that are relevant and valid for your environment.
    • See Appendix A on configuration file syntax.

Configuration

Configuration of the github_connector is done through a YAML file which should be in the configuration subdirectory. There is a sample configuration file named sample.yml in the configuration subdirectory that you can use as a template. Copy it to a file with a name reflecting your intended usage or environment, and edit the file substituting with values relevant to your situation.

General recommendations:

  1. Start simple.

  2. Locate the configuration subdirectory.

  3. Copy the sample.yml file to a file named suitably for your environment, such as copy sample.yml to product_x.yml.

  4. Edit your product_x.yml file. For example, change the sample values for credentials, workspace, project, token, and so on to values that are relevant and valid for your environment.

  5. Decide on the appropriate SecurityLevel.

  6. Decide whether you want to update state of Agile Central artifacts based on commit messages.

  7. Decide whether you want to map a GitHub committer to a valid Agile Central user and choose one of the three mapping methods: Passthrough, FileBasedUserNameLookup, or UserNameDomainAugmentLookup.

 

       *See Appendix A on configuration file syntax.

Operation

Manual

Using a terminal window or console: cd to the installation root directory, for example /opt/local/sw/vcsconn-2.0.3. Then run python3 vcs2ac product_x.yml. This software requires that the configuration file reside in the configuration subdirectory. You specify the name of the file on the command line. Do not specify the subdirectory in the command line argument.

Scheduled

Use either cron, launchctl, or Windows Task Scheduler. Make sure the environment in effect when running this software has an appropriate environment set so that you can run: python3 $VCS/github_connector your_config_file_name.yml where $VCS is the reference to an environment variable containing the fully qualified path to the directory where the software is installed. For example, if you unzipped the package in /opt/local/sw, then your VCS would be set like this: export VCS=/opt/local/sw/vcsconn-2.0.3.

Time File

In normal operation, the connector writes a time file (in the log directory) whose name is based on the configuration file name. Example: If the configuration file name is product_x.yml, then the associated time file name would be product_x_time.file. The content of the time file is a single line containing a human readable date/time stamp value in the format YYYY-MM-DD hh:mm:ss Z. The value represents the timestamp of the last commit reflected in Agile Central. When the connector is run a subsequent time, it consults the time file to determine which jobs need to be considered for the current run by only processing the commits whose start time is greater than or equal than the time file value. It is possible to set the time file value artificially, (using ISO format, such as 2017-05-12 11:37:55 Z) so that you can go back in time and pick up commits from some arbitrary point in the past. The connector does not duplicate commits so you do not have to worry about duplicated information getting posted to Agile Central. In the absence of a time file, the connector pulls commits starting at a point three days prior to the current time.

Extensions

To support another SaaS VCS, you will need to develop a connection spoke for your build system that can support a call to getOrganizations, and getRecentRepositoryActivity(ref_timestamp). That method needs to return a list of commit items, where a commit item has the following attributes:

  • SHA
  • timestamp
  • author
  • message
  • changes

Troubleshooting

The connector always writes a log file named based on the configuration file name. The log file is written into the logs subdirectory under the base installation directory. You can control the extent of logging by specifying the level in the configuration file. Within the configuration file, you can set the LogLevel which determines the amount of logging information written to the log file. Under the Service section, modify the LogLevel setting (valid values are ERROR, WARN, INFO, DEBUG). When you set the LogLevel to DEBUG, you will get the full range of logging messages that can be helpful in pinpointing where a problem occurred and what information was being processed at the time of the anomaly.

It can be very helpful to run the connector in Preview mode when setting things up for the first time. This allows you to get the connections to Agile Central and GitHub to initialize and validate correctly without posting any changesets.

*Issue: Pull Requests are not transferring to Agile Central.
Answer: Verify the following line is in the config file:

     PullRequests : True # Default is False, when True VSTS PullRequests are detected and reflected in Agile Central


*Issue: Error seen when using * in the include statement to pull all repos.

FATAL: VCSConnectorRunner.run - VCSConnectorRunner.getConfiguration vcs_connector_runner.py(288) - Unable to parse config/vsts-cloud.yml successfully, while scanning an alias
  in "<unicode string>", line 25, column 26:
                Include   :  *
                             ^
expected alphabetic or numeric character, but found '\n'
  in "<unicode string>", line 25, column 27:
                Include   :  *
                              ^
Answer: Escape the * in the include statement with quotes.
        Repository:
            Include   : "*"

Security Level

When credentials in the configuration file are no longer in clear text, SecurityLevel settings can only upgraded, but not downgraded. For example, a configuration file with encrypted credentials cannot be used with SecurityLevel reset from Encrypted to Encoded. If users want to downgrade security level, they have to start with a configuration file where credentials are in clear text.

Appendix A - Configuration file editing

The Agile Central github_connector for GitHub uses a text file in the YAML format. For complete information, consult the web page at or any other website on the subject of YAML. Please see the Download Samples section at the bottom of this page for a sample configuration file with explanatory notes

For brevity, this document mentions several of the most relevant syntax items and covers the three sections of a valid YAML configuration that can be used with the connector.

  • Use a text editor (not MS Word or Google Doc) to edit the file.
  • Never use a tab character in the file. YAML does not allow or recognize tab characters.
  • Save the file in UTF-8 format.
  • Use a monospace font.
  • Be consistent with the number of spaces you use to indent.
  • On a line, the first occurrence of a non-quoted # character indicates a comment, the # char and all chars to the right are ignored in processing.
  • Keep the sections in the same order as is present in the sample.yml file.
  • Be aware that the colon char : is significant, it separates a key from the value.
  • Be aware that the dash char - is significant, it denotes the start of a list which may have one or more key value pairs that constitute the list item.
  • You usually do not have to quote values if they have spaces in them; you will have to quote the value if it contains an embedded # character.

Skeleton of the template_config.yml file
 GitHub:
         AgileCentral:
             ...  # several key value pairs are relevant for this section
         GitHub:
             ...  # several key value pairs are relevant for this section
         Service:
             ...  # a few key value pairs relevant for the overall operation of the connector appear in this section
 

The AgileCentral section specifies values to use to obtain a connection with Agile Central. The GitHub section specifies values to use to obtain a connection with GitHub and specify the policies governing what repositories in GitHub get processed to result in posting of changesets to Agile Central. You must specify the server for your GitHub Enterprise in the GitHub section of the configuration file. The Service section controls some aspects of the connector behavior on an overall basis. For example, For example, Pull Requests can be added to this section as True.

Revision History

  • 2.0.3-Master --- 9-Feb-2018
    • Fixed name of StateExtractor class in sample config file
    • Fixed assignment of Changeset URI for Bitbucket Server
  • 2.0.0-Master --- 21-Dec-2017
    • Unified all newer policy based VCS connectors into one distribution package
  • 1.3.2-Master --- 7-Dec-2017
    • Support for appending committer name to commit message
  • 1.3.0-Master --- 28-Nov-2017
    • Added support for pull requests
  • 1.2.2-Master --- 16-Oct-2017
    • Included pull request title to be parsed for formatted id
  • 1.1.1-Master --- 12-Sept-2017
    • Added support for GitHub Enterprise (GHE) as of version 2.10.0
  • 1.0.0-master --- 30-May-2017
    • Initial release

Feedback

Need more help? The CA Agile Central Community is your one-stop shop for self-service and support. To submit feedback or cases to CA Agile Central Support, find answers, and collaborate with others, please join us in the CA Agile Central Community.