Jenkins GSoC

It has been a while since the last blogpost about Google Summer of Code in Jenkins. GSoC 2018 has officially finished on August 23, and we had a Jenkins Online Meetup where we had final presentations of the GSoC projects. It is never late to provide more context, so I would like to summarize the results and provide updates of what was happening in Jenkins GSoC Special Interest Group over last 2 months. In this blogpost you can find project status overviews and updates from the Jenkins GSoC SIG.

But first of all, I would like to thank all our students, their mentors and to all other contributors who proposed project ideas, participated in student selection, community bonding and further reviews. Google Summer of Code is a major effort which would not be possible without active participation of the Jenkins community

Summary

This year we started preparing for Google Summer of Code in early December. 14 project ideas and 12 potential mentors we published on our website, and we got dozens of students reaching out to us during the application period. After processing applications, we have selected 4 applications for GSoC. Unfortunately one project got cancelled due to student eligibility issues.

So, we had the following projects: Code Coverage API plugin, Remoting over Apache Kafka, and Simple Pull-Request Job Plugin (also known as Pipeline as YAML). All these projects have a significant value to the Jenkins community. They were focused on areas which have been discussed in the community for a long time, but which had no progress so far. Google Summer of Code allowed us to kick-start these projects, and to make significant progress there. All projects have been released and made available in the Jenkins community (common or experimental update centers).

In total there were 9 blogposts about GSoC projects on jenkins.io, and also 2 Jenkins Online Meetups. GSoC results have been also presented at DevOps World - Jenkins World conference and the contributor summit.

Code Coverage API Plugin

There are many code coverage plugins in Jenkins: Cobertura, JaCoCo, Emma, etc., etc. The problem with these plugins is that each of them implements all code coverage features on their own. So you get different feature sets, UIs, CLI commands and REST APIs. The idea of this project was to unify the existing functionality and offer a new API plugin which other plugins could extend. It would help to simplify existing plugin and to create new plugins for coverage tools.

The project has started really well, and we had the first demo after a week of coding. Then Shenyu continued extending the plugin’s functionality over coding periods. Here is the list of the key features offered by the plugin:

  • Flexible data structure for defining and storing coverage metrics within Jenkins

  • Coverage charts and trends

  • Source code navigation

  • REST API for retrieving coverage stats and trends

  • Report aggregation for parallel steps

  • Extension points which allow integrating other plugins

In addition to the Code Coverage API Plugin, Shenyu added integration to the Cobertura Plugin and also created a new llvm-cov plugin which is expected to be released soon.

After GSoC Shenyu continued contributing to the Jenkins project. He works on the Code Coverage API plugin and also participates in the Chinese Localization SIG.

Simple Pull-Request Job Plugin

This project focused on introducing a way to easily define pull-request build job definitions in YAML. This project has been shaped a lot during the application period and community bonding, so that the project fit the existing Jenkins Ecosystem better. Finally it was decided to build the new plugin on the top of Pipeline: Multi-Branch Plugin. There was also an idea to offer extra syntax sugar, templating and automatic resolution for common flows, so that users need less time to define Pipelines for common use-cases.

The plugin allows defining Pipeline jobs as YAML being stored in SCM. Original design presumed a new job type, but during community bonding and Phase 1 prototyping it was decided to build the plugin on the top of the existing Pipeline ecosystem and extension points. Currently the plugin generates Declarative Pipeline code from YAML so that it gets a lot of Pipeline features out-of-the box. In addition to that, Simple Pull Request Job Plugin uses a an engine provided by the Configuration as Code plugin to convert YAML snippets to Pipeline step definitions.

The plugin has been well described by Abhishek in his Pipeline as YAML blogpost in August. Currently it is available in the Experimental Update Center as an alpha version. Pham Vu Tuan, one of our GSoC students, have also joined the plugin team. At the DevOps World - Jenkins World hackfest we had discussions with the Jenkins Pipeline team, and we have a plan towards making this plugin available as an Incubated Pipeline project. The final implementation may change, but in any case the project gave us a working prototype and a lot of information about obstacles we need to resolve.

Remoting over Apache Kafka

Last but not least, Remoting over Kafka is another challenging project we had. To implement communication between its controllers and agents, Jenkins widely uses home-grown protocol implementations based on TCP (JNLP 1..4 protocols). There are some performance and stability implementations, and there have been discussions about using an industry-standard message bus or queue. Pham Vu Tuan proposed to use Apache Kafka for it, and after some experiments during community bonding and first coding phase we agreed to go forward with this implementation.

During his project Vu Tuan extended Jenkins Core and Remoting to allow implementing an agent communication channel in a plugin. Then he has created a new Remoting over Kafka plugin which is now available in the main Jenkins Update cente. Once the plugin is installed, it is possible to connect to agents over Apache Kafka and execute all types of Jenkins jobs there. There are also official jenkins/remoting-kafka-agent images available on DockerHub.

Vu Tuan continued contributing to the Jenkins project after GSoC, currently he maintains the Remoting over Kafka plugin. He visited the DevOps World - Jenkins World US conference in September, presented his GSoC project at the Jenkins Contributor Summit. You can find his slides here. After the conference he also participated in the hackfest where he helped to migrate Jenkins' DNS services to Microsoft Azure.

What could we do better?

After the end of GSoC we had a Retrospective with GSoC students and mentors. We discussed the issues we encountered during the projects, and ways to improve the student and mentor experience.

Main takeaways for us:

  • GSoC projects should be aligned with Jenkins Special Interest Groups (SIGs) or subprojects in order to get a wider list of stakeholders Projects should be aligned with SIG priorities when possible

  • In addition to GSoC SIG meetings and Jenkins Online Meetups during student evaluation, we should also run regular status updates within SIGs so that there more contributors involved in projects

  • We should invest more time into forming mentor teams before the application period starts. This year there were changes in mentor teams after the community bonding started, and it complicated the work

  • We should pay more attention to student eligibility. This year we started from 4 projects, but unfortunately one project (EDA plugins for Jenkins) got cancelled due to the visa limitations the student had.

  • We should do regular office hours for mentors/students so that it is possible to exchange information between GSoC projects within the organization. This year we cancelled them at the end of phase and relied only on regular project meetings and mailing lists, but this is not enough.

For me personally the main takeaway is also to reduce direct involvement into the project as a mentor and technical advisor. Doing org administration, logistics and mentorship is not good from a bus factor PoV, and I believe I was pushing my vision too hard in few cases. Will do my best to prevent it next year.

If you want to share your feedback and ideas, please reach out to us using the GSoC mailing list.

What’s next?

In order to improve GSoC organization in Jenkins, we have have created a GSoC Special Interest Group which will be running non-stop as other SIGs in Jenkins. The objective of the SIG is to organize GSoC, work with potential students/mentors, and to help students stay involved in the community after GSoC ends. In this SIG we will have monthly meetings to sync-up on GSoC. If you are interested to contribute, please join the SIG.

According to the Retrospective, next year we plan to invest more into communication with mentors. We will also try to tie new project proposals to Jenkins Special Interest Groups so that the students become a part of ongoing coordinated efforts. This weekend Martin d’Anjou, Jeff Pearce and me are participating in the GSoC Mentor summit to share experiences and to study from other GSoC organizations. On October 17 we will have a GSoC SIG meeting to discuss our experience and to discuss next steps.

In addition to that, Jenkins Google Summer of Code will be presented at DevOps World - Jenkins World Nice and at the contributor summit. If you plan to visit the conference and you are interested to participate in Google Summer of Code and other community activities, please join us at the contributor summit or stop by at the community booth.

And, elephant in the room…​ GSoC 2019. Of course we are going to apply, stay tuned for new announcements. We have already started collecting project ideas for the next year. If you are interested to participate as a student or mentor, please reach out to us using the GSoC SIG mailing list.

About the Author
Oleg Nenashev

Jenkins core maintainer and board member. Oleg started using Hudson for Hardware/Embedded projects in 2008 and became an active Jenkins contributor in 2012. Nowadays he leads several Jenkins SIGs, outreach programs (Google Summer of Code, Hacktoberfest) and Jenkins meetups in Switzerland and Russia. Oleg works for CloudBees and focuses on key projects in the community.