CSC216 Basics of Jenkins Guide

Target Audience: This tutorial is intended for CSC 216 students at NC State University who are working with Jenkins.

Jenkins is a continuous integration program that will automatically compile and test your programs (both with your tests and the teaching staff tests) and provide feedback on test results, test coverage, potential errors, and programming style. Jenkins will automatically evaluate all of the code that you upload to your CSC 216 git repositories.

This guide introduces you to the feedback produced by Jenkins. You should use that feedback to help improve your source code, test cases, and the comments and style of your code.

If you disagree with a CheckStyle or PMD comment, send an email to the PTFs or post a private message on Piazza with a screen shot of the comment and why you disagree with it. At a minimum, you must conform to the Departmental Style Guidelines. For style, Jenkins will evaluate the syntax and check for bugs. The teaching staff will evaluate your style for content. However, our review of your style will be deeper because we can understand what names and comments mean, while CheckStyle can only check that they conform to a particular syntax and structure.

You have full control over the tests that you write; there is no reason not to get 100% with testing! (We will be checking for good tests. Using assertTrue(true) just to get a passing test method is just lazy programming.)

Note: The screenshots in this guide were made from an administrative account, so you will not have the same menu or navigation options shown here. Additionally, the screenshots were made with a project set up just for this tutorial, so the name of the project, packages, and classes will not match those for your projects.

Jenkins Home Page

When you first login to Jenkins, the All tab will list all the Jenkins Jobs to which you have access. The “status” of your project (the colorful circle beneath the “S” column) can help you understand your progress in completing an assignment:

• Red: serious errors indicating that your code could not compile, that the project could not be found in your repository, etc.
• Yellow: your code compiles and builds, but you have Checkstyle/PMD/FindBugs warnings, failing tests, and/or < 80% statement coverage.
• Green: your code compiles and builds with no Checkstyle/PMD/FindBugs warnings, no failing tests, and > 80% statement coverage.

Click the project name to view the Project Summary page.

Project Summary Page

The Project Summary page displays information about your project, including latest test results and trend charts for Checkstyle warnings, FindBugs warnings, and PMD warnings. The page also displays the build history. You can click any of the previous builds to view the build summary page.

Build Summary Page

The build summary page provides details for a given build of your project. The build summary page includes:

• Build Number & Timestamp
• Changes: descriptions of all recent git commits/pushes since the last build
• Started By: indicates whether a build was automatically started by you pushing code to your git repository, or whether the build was started manually (the person’s name will be listed)
• Revision: the git revision identifier
• Checkstyle: the number of Checkstyle warnings for the current build
• FindBugs: the number of FindBugs warnings for the current build
• PMD: the number of PMD warnings for the current build
• Test Results: The number of test failures, and the +/- change in the number of failing tests from the previous build

Note: the coverage report is not summarized on the build summary page. You must manually click the Coverage Report link to ensure you have satisfied the coverage requirements.

Test Results

Click Test Results (on the left navigation bar) to view the summary page of test results. The summary page shows the number of failing tests out of the total tests executed along with a list of each individual failed test case.

IMPORTANT: Any test case that begins with “TS” is one of the teaching staff test cases. The teaching staff tests evaluate how well your work meets the design of the program.

Clicking a failed “TS” test case opens a separate page showing hints of why the test may have failed. Each hint identifies the method under test, but the underlying cause of the failure may be due to another method that is called to set up the test (for example, adding a Student to a Course before removing the Student from the Course) or that is called by the method under test itself. Ask for clarifications on the hints that you don’t understand on Piazza. The teaching staff can investigate the failure in more detail.

Coverage Report

Click Coverage Report to view details of your code coverage from executing your test cases. The Coverage Report summary page presents the following information:

• A line chart of your coverage trends from your build history.
• An overall report of instruction, branch, complexity, line, method, and class coverage.

IMPORTANT: For this course, you are required to maintain >= 80% statement or line coverage for each non-GUI/non-UI class. In the figure below, the line coverage is the metric of interest. The cell with the line coverage is highlighted in orange because at least one class non-GUI/non-UI class has < 80% line coverage.

Click a package name to view coverage details for classes within the package.

Click a class name to view additional details about coverage per method in the class.

FindBugs Report

Click FindBugs Warnings to view results from running FindBugs on your code. The FindBugs summary page includes the following information:

• All Warnings: the total number of warnings for this build
• New This Build: the total number of new warnings that appear in this build compared to the previous build
• Fixed Warnings: the total number of warnings that were fixed in this build compared to the previous build
• Priority: the total number of high, normal, and low priority warnings
• Details: a breakdown of the warnings for the current build
  • Files tab: warnings per class
  • Categories tab: warnings per bug category (Bad Practice, Performance, etc.)
  • Types tab: warnings by type of warning (Unread Field, Format String Uses New Line, etc.)
  • Warnings tab: a detailed view showing file, line, priority, rank, type, and category for each warning
  • Details tab: verbose descriptions of each warning.

Note: The figures for this section are from an earlier version of Jenkins.

Click a file name to see more details about FindBugs warnings in a class.

You can find the exact locations of the notifications by running FindBugs in your local Eclipse instance.

CheckStyle Report

Click Checkstyle Warnings to view a summary of Checkstyle warnings. The summary page includes the following information:

• All Warnings: the total number of Checkstyle warnings in this build
• New Warnings: the total number of new warnings compared to the previous build
• Fixed Warnings: the total number of fixed warnings since the previous build
• Priority: the number of high, normal, and low priority warnings
• Details: descriptions for each of the Checkstyle warnings in the current build

You can find the exact locations of the notifications by running CheckStyle in your local Eclipse instance.

PMD Report

Click PMD Warnings to view details of the PMD warnings for your project. The summary page includes:

• Priority: the number of high, normal, and low priority PMD warnings for the current build.
• Details: Descriptions for each PMD warning in the current build.

Click on the class file name to view details about warnings in the class.

Console Output

The console output is a text output of everything that is printed to standard out and standard error while your Jenkins build is executing. The console output is useful for identifying problems like compiler errors or classes with insufficient output. You can find the link to Console Output in the left bar of a build.

The image below shows the compiler errors that are generated in a failing build. You can use the compiler errors to help you fix any compile issues with your project or where you project may not be meeting the teaching staff design.

Grade Estimation Example

Suppose the maximum number of points you can receive for a programming assignment is 150, with the point breakdown as follows:

• Process: 25 points
  • Process Points 1 Deadline: As described in the project description - 10 points
  • Process Points 2 Deadline: As described in the project description - 10 points
  • High quality commit messages throughout project - 5 points
• Correctness / Testing Grade: 65 points (all from Jenkins evaluation)
  • Student Unit Tests - 20 points
  • Student Test Coverage - 20 points
    NOTE: You are expected to have 80% or better coverage of each model class.
  • Teaching Staff Tests - 25 points
• Static Analysis, Comments, Style - 15 points (all from Jenkins evaluation)
• Acceptance Tests by Teaching Staff - 25 points
• System Tests - 10 points
• Javadoc - 10 points (plus additional Late Credit adjustments or penalties if the teaching staff cannot import your project as an Eclipse project)

IMPORTANT: The number of points possible for each of these categories may vary for actual assignments.

Your grade will calculated as (the number of points)/150 * 100%.

Your Jenkins results determine the total number of points for Correctness/Testing and Static Analysis, Comments, and Style. You can use these to estimate your overall grade. However, there may be adjustements to the Static Analysis, Comments, and Style grades when manually reviewing your work.

Determining your Correctness/Testing points

Your Correctness/Testing points come directly from the Jenkins evaluation.

Determining your Static Analysis, Comments, Style points

Your SpotBugs, Comments, Style points ome directly from the Jenkins evaluation.

Estimating other points PLUS the final assignment grade

The remaining points for an assignment are not based on Jenkins evaluations. Again, keep in mind that these are example estimates.