Automated Testing of Pelican Blog Using Travis-CI

Background

I use Pelican to maintain this blog website. The source of the site content (ReStructured text, not html) is hosted at this GitHub repository: https://github.com/jhshi/blog_source. The generated site is hosted at here: https://github.com/jhshi/jhshi.github.com.

My previous work flow was:

1. Write content locally
2. Do a bit testing using local HTTP server (mostly looking for format glitches)
3. Commit and push to the jhshi/blog_source repo.
4. Do a fab github to generate the site and push to the jhshi/jhshi.github.com repo.

As you can see, the process (especially step 2 - 4) is a bit tedious, and the site is not throughly tested. In particular, I do not always run linkchecker before I push. And finally, I need to manually publish the site every time.

So the goal is to shift some of the heavy-lifting, especially testing and publishing, to a continuous integration tool. Travis seems the natural choice given its excellent integration with Github.

Before you get started, I suppose you have already connected Travis with your Github account. Refer to the documentation from Travis for more details: https://docs.travis-ci.com/user/getting-started/.

Build Environment

As a first step, we first tell Travis how to build our project. Put these content in a file named ".travis.yml" in your projects root directory.

1
2
3
4
5

language: python
install:
    NO_SUDO=1 source setup.sh
script:
    make html

There are two things worth noticing:

1. The setup.sh script, which can be found here: https://github.com/jhshi/blog_source/blob/master/setup.sh, is responsible to set the pelican environment, including cloning proper plugins and themes repository. The NO_SUDO flag tells the script to not use sudo and also use https URLs for repository instead of ssh.
2. If you use ga_page_view plugin, the build will fail since the private key file ga.pem will not exist in the freshly cloned repository: it shouldn't. I'll talk about how to deal with this later. But for now, we can tell pelican to silently ignore this error. Change the ga_page_view configuration in pelicanconf.py slightly like this:

1
2
3
4
5
6
7
8

if os.path.isfile('ga.pem'):
    GOOGLE_SERVICE_ACCOUNT = 'xxx'
    GOOGLE_KEY_FILE = os.path.join(PROJECT_ROOT, 'ga.pem')
    GA_START_DATE = '2005-01-01'
    GA_END_DATE = 'today'
    GA_METRIC = 'ga:pageviews'
else:
    print "[WARN] No key found for Google Analytics"

Commit and push to the master branch of your source repository, and check travis logs to make sure it can successfully build the site.

Private Files

There are two private files in my case: the key for Google Analytics API and the deploy key for the website repository.

Let's first create the deploy key if you don't have it already. Note that this must NOT your primary ssh key. So I suggest you create a new pair of SSH keys just for the website repo.

1

$ ssh-keygen -f ./deploy_key

Copy the content of deploy_key.pub to your project's deploy keys settings, then move it to somewhere else or delete it: we don't really need it anymore.

Next, install travis CLI tool, which is used to encrypt the private files. travis depends on ruby greater than 2.0. So I recommend to use the Brightbox ppa for ruby.

1
2
3
4
5
6
7
8

$ sudo add-apt-repository ppa:brightbox/ruby-ng
$ sudo apt-get update
$ sudo apt-get install ruby2.2-dev ruby-2.2
$ sudo gem install travis
$ travis -v
1.8.2 # <--- your mileage may vary
$ travis login
# <--- enter your GITHUB username and password

Next, use travis to encrypt the file. Note that you can only encrypt one file in total. Here we have two files to encrypt: the key for Google API and the key for deploy the website. So we need to tar it first:

1
2

$ tar cvzf secrets.tgz ga.pem deploy_key
$ travis encrypt-file secrets.tgz

Finally, unpack those secrets by add these lines to the .travis.yml.

1
2
3

before_install:
- openssl aes-256-cbc -K $encrypted_XXXXXX_key -iv $encrypted_XXXXXX_iv -in secrets.tgz.enc -out secrets.tgz -d
- tar xvf secrets.tgz

Replace XXXXXX with the magic number you got from travis encrypt-file.

Now, delete secrets.tgz, and add deploy_key and ga.pem to your .gitignore, commit all changes, and push to master branch. Check travis logs to make sure the sites gets built. In particular, you should not see the No key warning for ga_page_view plugin.

Automatic Deployment

Finally, let's tell travis to deploy the website if the test passes. Add this rule to your Makefile:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

check: clean publish
    cd $(OUTPUTDIR) && $(PY) -m pelican.server && cd - &
    sleep 3
    linkchecker http://localhost:8000
    pgrep -f "^python -m pelican.server" | xargs kill -9


travis: clean check publish
    chmod 600 deploy_key &&\
        eval `ssh-agent -s` &&\
        ssh-add deploy_key &&\
        cd $(OUTPUTDIR) &&\
        git init . &&\
        git config user.email "robot@travis-ci.org" && \
        git config user.name "Travis" && \
        git remote add origin $(GITHUB_PAGES_REPO) &&\
        git add --all --force . &&\
        git commit -am "Site updated on `date -R`" &&\
        git push origin master --force &&\
        cd -

Note that we also check URL links using linkchecker before deploying. Change the make target in .travis.yml to make travis, commit and push. And now if the site is good, it should be automatically deployed.