CA Agile Central Build Connector for Jenkins Installation and User Guide

Untitled Document

Overview

The CA Agile Central build connector for Jenkins posts information about Jenkins job builds to CA Agile Central, and relates those build items to AC changesets and artifacts if sufficient information is contained in the VCS commit messages to identify the relevant artifacts. This connector supports artifact types of Defect, User Story and Tasks.

The CA Agile Central Build Connector is classified as a one-way and one-time mechanism. Information in Jenkins is never altered. The AC Build Connector consists of software that you run on your platform according to your desired schedule. The configuration of the connector is policy-based, meaning that you do not need to provide configuration information in Jenkins for each job for which you want the connector to operate. While you can configure specific jobs with this connector, you can also configure by views and folders with the option to use shell regex syntax to include jobs or exclude jobs. The policy based nature allows you to add Jenkins jobs and not have to alter the configuration in order to get the builds for those jobs to be posted to Agile Central. At this time Multi Branch Pipeline jobs are not supported.

This guide includes:

Software Requirements

  • Linux, Windows and Mac Platforms are supported.
  • Linux and Mac Platforms:
    1. Jenkins 2.2 or higher (While we do not certify support for Cloudbees, customers have reported success using the build connector with Cloudbees)
    2. Python 3.5 or newer. You can retrieve the installer package from www.python.org.
  • Windows Platform:
    1. Jenkins 2.2 or higher (While we do not certify support for Cloudbees, customers have reported success using the build connector with Cloudbees)
    2. Python 3.5 or newer - we recommend using the 64-bit version. You can retrieve the installer package from www.python.org.
    3. win32com module - available from SourceForge here. If you are using the 64-bit version Python3.5.x or newer, pick the appropriate file from the SourceForge list of Python for Windows Extensions.

Connector download

To download the connector, follow the steps here.

Installation

Once you have Jenkins and Python installed - and for Windows users only, the win32com module installed (information for the previously mentioned can be found in the Software Requirements section above), install the following Python packages:

  1. pip3.5 install requests==2.8.1
  2. pip3.5 install pyral==1.2.3
  3. pip3.5 install PyYAML==3.12
  4. change your working directory (cd) to a directory where you want to install the connector
  5. unzip bldeif-1.0.0.zip (or use a suitable program that can unzip a .zip file)
  6. cd bldeif-1.0.0
  7. ls -laR # observe the unpacked contents, or use dir on Windows

Sample contents:

            bldeif            # bldeif module root directory
            bldeif_connector  # connector initiation script, this is what you will run
            config            # holds any config files used with this connector
            sample.yml        # a sample config to use as a base reference
            README.txt        # this file

Setup

Set up CA Agile Central

Verify that your target WorkspaceConfiguration object has BuildandChangsetEnabled set to true.

Your CA Agile Central workspace administrator needs to enable this for your target workspace. If it is not enabled, the connector will work, but you will not be able to see any of the changeset or change information that is associated with your Rally work items (story, defect, task).

The workspace administrator must edit the workspace and do the following:

  1. Click the Setup icon .
  2. Click the Workspaces & Projects tab.
  3. On the Workspaces & Projects summary page, select your workspace.
  4. From the Actions drop-down, select Edit.
  5. On the workspace editor, select the checkbox labeled Enable Build and changeset.
  6. Click Save & Close.

Setup:

  1. Locate the config subdirectory.
  2. Copy the sample.yml file to a file named suitably for your environment eg, cp sample.yml to product_x.yml (or some other suitably named file that has a .yml suffix).
  3. Edit your product_x.yml file.
  4. Change the sample values for credentials, Workspace, Project, Job, View, Folder to values that are relevant and valid for your environment.

*See Appendix A on config file syntax.

Set up Jenkins

The userid for Jenkins will need access to the "Manage Jenkins" page (Admin permissions)

Operation

Manual

Using a terminal window or console: cd to the installation root directory eg. /opt/local/sw/bldeif-1.0.0 and run the following command replacing product_x.yml with the name of your .yml file: python3.5 bldeif_connector product_x.yml (on Windows, this command may be: python bldeif_connector your_configfile.yml) This software requires that the configuration file reside in the config subdirectory. You specify the name of the file on the command line (don't specify the subdirectory in the command line argument).

Scheduled

Use either cron or launchctl or Windows Task Scheduler. Make sure the environment in effect when running this software has an appropriate environment set so that you can run: python3.5 $BLDEIF/bldeif_connector your_config_file_name.yml where $BLDEIF is the reference to an environment variable containing the fully qualified path to the directory where the software is installed. As an example: If you unzipped the package in /opt/local/sw, then your BLDEIF would be set like this: export BLDEIF=/opt/local/sw/bldeif-1.0.0.

Time File

In normal operation, the connector writes a time file in the log directory whose name is based on the configuration file name.

Example: If the configuration file name is product_x.yml then the associated time file name would be product_x_time.file. The content of the time file is a single line containing a human readable date and time stamp value in the format YYYY-MM-DD hh:mm:ss Z. The value represents the timestamp of the last Jenkins jobs considered (after some negative value adjustment to insure no Jenkins jobs are ever overlooked). When the connector is run a subsequent time, it consults the time file to determine which jobs need to be considered for the current run by only processing the builds whose start time is greater than or equal than the time file value. It is possible to set the time file value artificially (but in the correct format) so that you can revert to builds from some arbitrary point in the past. The connector does not duplicate build records so you don't have to worry about duplicated information getting posted to Agile Central.

Sample Scenario Involving the Connector

  1. Make a commit to your repository (the same one that has been configured in a Jenkins Job). This has also worked when using GitHub with the Rally service to push changesets to Agile Central. (Note: In this scenario, GitHub will need to push information to Jenkins as well). To use with a Bitbucket repository, the repository will need to be cloned as a Git repository (currently, the connector is not able read data from Bitbucket). The commit should reference a Rally artifact in its message, i.e.: git commit -m "test for DE82 fixes"
  2. Run your VCS connector (i.e. Git or SVN) against the Git repository in Step 1 (not needed if using GitHub).
  3. In Agile Central, verify the changeset was added to the artifact mentioned in the commit message in step 1 (DE82 in the example).
  4. Run the Jenkins Job connected to the Git (or GitHub) repository in Step 1.
  5. Run the Build Connector for Jenkins with the following command from the installation root directory: python3.5 bldeif_connector your_configfile.yml (on Windows, this command may be: python bldeif_connector your_configfile.yml)
  6. In Agile Central, verify the build information was added to the changeset in Step 3. Build information can also be viewed via one of the Build apps on the Agile Central dashboard (like "Build Traceability" or "Build Health").

Troubleshooting

The connector always writes a log file name based on the configuration file name. The log file is written into the log subdirectory under the base installation directory. Within the configuration file, you can set the LogLevel which determines the amount of logging information written to the log file. When you set the LogLevel to DEBUG, you will get the full range of logging messages that can be very helpful in pinpointing where a problem occurred and what information was being processed at the time of the anomaly.

It can be very helpful to run the connector in Preview mode when setting things up for the first time. This allows you to get the connections to Agile Central and Jenkins to initialized and validate correctly without posting any build information. This mode also can show you what Jenkins jobs would actually be considered without actually posting any build information to Agile Central.

*Issue: The following error is seen when the win32com module is not installed:

C:\Connectors\bldeif-1.1.3>python ac_build_connector sample_jenkins.yml
ERROR: ac_build_connector encountered an ERROR condition.
  File "ac_build_connector", line 31, in main
    connector_runner.run()
  File "C:\Connectors\bldeif-1.1.3\bldeif\bld_connector_runner.py", line 137, in run
    self.proclaim_existence(build_system_name)
  File "C:\Connectors\bldeif-1.1.3\bldeif\bld_connector_runner.py", line 103, in proclaim_existence
    proc = ProcTable.targetProcess(os.getpid())
  File "C:\Connectors\bldeif-1.1.3\bldeif\utils\proctbl.py", line 96, in targetProcess
    aps = ProcTable.allProcesses()
  File "C:\Connectors\bldeif-1.1.3\bldeif\utils\proctbl.py", line 29, in allProcesses
    all_procs = ProcTable.posixProcesses()
  File "C:\Connectors\bldeif-1.1.3\bldeif\utils\proctbl.py", line 44, in posixProcesses
    ps_output = subprocess.Popen(command_vector, stdout=subprocess.PIPE).communicate()[0]
  File "C:\Users\user01\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "C:\Users\user01\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 997, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
Answer: Install the win32com module. See "Software Requirements" section in this document for installation instructions.

*Issue: The following error is seen on screen when running the connector before the logfile is generated:

C:\Users\tester\bldeif-1.1.3>python ac_build_connector test1.yml
ERROR: ac_build_connector encountered an ERROR condition.
File "ac_build_connector", line 31, in main
connector_runner.run()
File "C:\Users\tester\bldeif-1.1.3\bldeif\bld_connector_runner.py", line 137, in run
self.proclaim_existence(build_system_name)
File "C:\Users\tester\bldeif-1.1.3\bldeif\bld_connector_runner.py", line 103, in proclaim_existence
proc = ProcTable.targetProcess(os.getpid())
File "C:\Users\tester\bldeif-1.1.3\bldeif\utils\proctbl.py", line 96, in targetProcess
aps = ProcTable.allProcesses()
File "C:\Users\tester\bldeif-1.1.3\bldeif\utils\proctbl.py", line 29, in allProcesses
all_procs = ProcTable.posixProcesses()
File "C:\Users\tester\bldeif-1.1.3\bldeif\utils\proctbl.py", line 47, in posixProcesses
cmd_column_ix = hdr_line.index('CMD')
ValueError: substring not found
Answer: The win32com module version did not match the python version (ex: The win32com module for python 3.5 was installed on a system running python 3.6). See "Software Requirements" section in this document for installation instructions.

Known Limitations

BuildDefinition names and URLs

Within AgileCentral there is an entity called a BuildDefinition which for the purposes of the Build Connector for Jenkins contains the name of the Jenkins job. The Jenkins job as recorded in a BuildDefinition item is the full URL of the Jenkins job, including any folder and view elements. A BuildDefinition Name is limited to 256 characters. In the event that a Jenkins job has a URL whose length exceeds 256 characters, the behavior of the connector is to delete off-leading characters on the the job URL such that the result is no longer than 256 characters and this resulting string value is used as the BuildDefinition name. Be aware that in some edge cases you might have a Jenkins installation in which this type of modification of the full Job URL for various Jenkins Job URLs could result with the same abbreviated value. We do not anticipate that this will be a common occurrence. If you are in the situation where your Jenkins installation contains folder and view names that are fairly long (> 40 characters per folder or view name) with a high degree of nesting (> 6 levels) then you may need to alter your connector config files such that jobs whose URL results in a string of > 256 characters are not specified in your config file.

Example Jenkins Job URL:

    http://bigjenkins.stegasaurus.ancient:8080/job/ReallyLongScientificFolderName/.../job/FernCoveredLowlands/job/MickyDinosaur

Result

    .../job/FernCoveredLowlands/job/MickDinosaur

How the connector handles duplicately named Jobs or Folders or Views

See Appendix B.

VCS Support

  • Currently the connector will process changesets related to builds as long as the related job is using a Git repository.
  • Some configurations with Subversion have been successful, but there are combinations of Jenkins version, Subversion version and Jenkins Subversion plugin version that do not work with our connector due to variances in the json data returned for the build information.
  • We recommend using a designated VCS connector for Subversion along with Jenkins connector to capture commit/changeSet and build information. In those cases set ShowVCSData property in the Jenkins config file to False.

Appendix A - Configuration file editing (notes)

The CA Agile Central bldeif connector for Jenkins uses a text file in the YAML format. For complete information, consult the web page at www.yaml.org/start.html or any of the many other fine websites on the subject of YAML.

For brevity, this document mentions several of the most relevant syntax items and covers the 3 sections of a valid YAML config that can be used with the connector.

  • Use a text editor (not MS Word or Google Doc) to edit the file.
  • Never use a tab character in the file. YAML does not allow or recognize tab characters.
  • Save the file in UTF-8 format.
  • Use a monospace font.
  • Be consistent with the number of spaces you use to indent.
  • On a line, the first occurrence of a non-quoted # character indicates a comment, the # char and all chars to the right are ignored in processing.
  • Keep the sections in the same order as is present in the sample.yml file.
  • Be aware that the colon char : is significant, it separates a key from the value.
  • Be aware that the dash char - is significant, it denotes the start of a list which may have 1 or more key value pairs that constitute the list item.
  • You usually do not have to quote values if they have spaces in them; you will have to quote the value if it contains an embedded # character.


Skeleton of the Sample_config.yml
JenkinsBuildConnector:
        AgileCentral:
                ...  # several key value pairs are relevant for this section
        Jenkins:
                ...  # several key value pairs are relevant for this section
        Service:
                ...  # a few key value pairs relevant for the overall operation of the connector appear in this section

The Agile Central section specifies values to use to obtain a connection with Agile Central. The Jenkins section specifies values to use to obtain a connection with Jenkins and specify the policies governing what jobs in Jenkins get processed to result in posting of build information to Agile Central. The Service section controls some aspects of the connector behavior on an overall basis.

A sample file with some explanatory notations:

 ---------------------------------------------
 
JenkinsBuildConnector:
     AgileCentral:   # all of the possible key value pairs, not all must be used, see the right-hand side
                     # comments for designation as either 'R' required or 'O' optional
         Server:       : rally1.rallydev.com      # R
         Username      : henry5@hauslancaster.uk  # R   if an APIKey entry is used, then this isn't needed
         Password      : 2MuchAngst1415           # R   if an APIKey entry is used, then this isn't needed
         APIKey       : _zzzyyyy234twqwtqwet89y4t38g38y0  # O can use this instead of a Username and Password
         ProxyServer   : wu-tank.smooth.org       # O
         ProxyUser     : jessonbrone              # O
         ProxyPassword : S-e-c*r*ET321!789       # 0
         Workspace     : My Onliest Workspace     # R  name of the Agile Central Workspace where
                                                  # SCMRepository, Changeset and Build records are written

     Jenkins:        # all of the possible key value pairs, not all must be used, see the right-hand side
                     # comments for designation as either 'R' required or 'O' optional
         Protocol    : http                       # R
         Server      : jenkado.mydomain.com       # R
         Port        : 8080                       # R
         Prefix      :                            # O  for custom Jenkins installation in a non-standard directory
         Username    : validuser                  # R  provide a value when Jenkins requires credentials for access
         Password    : somepasswd                 # R  provide a value when Jenkins requires credentials for access
         API_Token   : 320ca9ae9408d099183aa052ff3199c2  # O can use in place of Password when credentials required
         MaxDepth    : 3                          # O  how many folder levels will be considered, default is three
                                                  #    this will accommodate a scenario like AlphaFolder // BetaFolder // CherryFolder and the jobs in that folder
         AgileCentral_DefaultProject : an Agile Central Project name  # R

         Folders:
             - Folder: a folder name
               include: toaster,microwave,stove  # O use adequate non-ambiguous patterns of job names to include
                                                 #   only those specified. this example would include the jobs
                                                 #   stove-hot, stove-warm, stove-burning
             - Folder: another folder name
               exclude: beta-,post-prod      # O use adequate non-ambiguous patterns of job names to exclude,
                                             # you do not have to specify the full job name
               AgileCentral_Project: Divison X // National // Engineering

         Views:
             - View  : view name
               AgileCentral_Project: Divison Y // Provincial // Engineering
             - View  : another name
               include: prod-
               exclude: pre-prod

         Jobs:
             - Job   : job name
               AgileCentral_Project: Beta Test for Northeast
             - Job   : another job name     # this job with a unique name could live in some view or folder not at the top level

     Service:
         Preview      : True   # When set to True, the connector shows the items that would be processed
                               #  but doesn't actually post any build information to Agile Central
                               # When you have completed the setup and the connections with AgileCentral and Jenkins
                               # are made successfully, you can change this value to True
         LogLevel     : INFO   # This is the default value, can also be DEBUG, WARN, ERROR. DEBUG is very verbose
         MaxBuilds    : 100    # Use a non-negative integer value. This "limit" pertains to builds for a particular job.

 ------  end of the file ---------------------------------------

Note: In order to run this connector, you must use a Jenkins administrator username and password in the configuration field of the Jenkins section.

Appendix B - Handling duplicate names in Jenkins

The connector will process jobs under named folders, there is not currently the facility to specify an upper-level folder and process all jobs directly in that folder and in any contained folders in any level of nesting. To get jobs in folders to be processed you must specify the folder name in the config file.

You may have a nested folder structure as illustrated below:
      - upper folder
        -- job 1
        -- lower folder
           -- job 2

To insure that both job 1 and job 2 are picked up by the connector the Folders section of the config file must look as follows:
         Folders:
           - Folder : upper folder
           - Folder : lower folder

If you have multiple folders in various locations in your Jenkins Job organization that have the same name,
      you must specify each folder using a fully qualified path with // as a separator between each folder/view level.
        Example config file snippet...
            Jenkins:
                ...
                MaxDepth : 5
                FullFolderPath : True
                ...
                Folders:
                    - Folder: Area 51 // Intermediate Stuff // Good Stuff
                    - Folder: Level1 //  Level 2 // Level 3 // Good Stuff


Appendix C: AgileCentral Project specification


    In AgileCentral, project names do not have to be unique. The connector provides a mechanism of distinguishing projects with the same name.
    For example, if a fragment of a project tree looks like this:

    |__ Jenkins
        |__ Salamandra
        |__ Corral
            |__ Salamandra

   we recommend that in order for the build definitions to be assigned to the intended projects in AC the following syntax should be used:


     Example config file snippet...

       Jenkins:
           ...
           AgileCentral_DefaultBuildProject :  Jenkins

           Jobs:
               - Job: monitor sub-tropical habitat
                 AgileCentral_Project: Jenkins // Salamandra
               - Job: fill the water bottles
                 AgileCentral_Project: Jenkins // Corral // Salamandra

Supported versions

Version End of Support
Jenkins 2.2 or higher N/A

Revision History

  • 1.1.3-Master --- 20-Oct-2017
    • Fixes
      • Updated timestamp function issue
  • 1.1.2-Master --- Mar-2017
    • Initial release

Feedback

Need more help? The CA Agile Central Community is your one-stop shop for self-service and support. To submit feedback or cases to CA Agile Central Support, find answers, and collaborate with others, please join us in the CA Agile Central Community.