Tag Archives: code

How to package the TeamCity build agent for Debian/Ubuntu

This post brought to you by Thumbtack Engineering and the letters C and I.

How to package the TeamCity build agent for Debian/Ubuntu

We’ll use fpm to package the TeamCity 9 build agent for Debian 8 (“Jessie”) or Ubuntu Linux, including a systemd service file to start it automatically.

First, read the TeamCity 9 docs for “Setting up and Running Additional Build Agents”.

Get the TeamCity build agent

Download TeamCity 9 from JetBrains. I’m going to be using TeamCity 9.0.3. You can also get a version of the agent matched to your TeamCity install by going to the Agents tab and clicking “Install Build Agents” in the upper right.

Install fpm if you don’t have it already.

Unzip TeamCity-9.0.3.tar.gz and find the buildAgent directory. This is where all of the agent’s code and config files live. Since TeamCity wasn’t written with the FHS in mind, we’ll put it in /opt/TeamCity/buildAgent.

Make the agent scripts executable

Run chmod +x buildAgent/bin/*.sh. You won’t need to run them on the system you’re using for packaging, but fpm preserves the executable flag when it creates a package.

Create a systemd service file

Create a file named teamcity-build-agent.service and put this in it.

[Unit]
Description=TeamCity build agent

[Service]
User=teamcity
ExecStart=/opt/TeamCity/buildAgent/bin/agent.sh run

[Install]
WantedBy=multi-user.target

The run argument keeps the startup script from forking so that systemd can manage the process itself. The User section should be changed to reflect whatever user you plan on running TeamCity as. (See the systemd.exec man page for more process control options.)

This file will eventually be copied to /lib/systemd/system/teamcity-build-agent.service, because systemd is too cool for /etc.

Build a package with fpm

fpm has some usage examples on its wiki. We’re going to use the -s dir input type to take a directory and the -t deb output type to generate a Debian .deb package.

fpm
-s dir
-t deb
--name teamcity-build-agent
--version 9.0.3
--architecture all
--depends java6-runtime-headless
buildAgent=/opt/TeamCity/
teamcity-build-agent.service=/lib/systemd/system/teamcity-build-agent.service

This fpm invocation tells it thje package name and version, that the agent doesn’t care what architecture you’re using but requires Java 6, and that everything should go under /opt/TeamCity, except for the service file, which goes where systemd is expecting it.

You should now have a file named teamcity-build-agent_9.0.3_all.deb.

Create a user

Before installing the package on your target machine, run sudo adduser teamcity --system --group --home /opt/TeamCity, or add something equivalent to your configuration manager if that’s where you manage users. (I’ll be using Puppet to manage users on production machines that will have this package). The --group option creates a teamcity group along with the teamcity user, so that you can then add other users to the teamcity group if they need to look at TeamCity builds or configuration.

For extra credit, you could create and destroy the user in preinstall/postremove scripts.

Install the package

Copy the file to your target machine and run sudo dpkg -i teamcity-build-agent_9.0.3_all.deb.

Configure the package

Copy /opt/TeamCity/buildAgent/conf/buildAgent.dist.properties to /opt/TeamCity/buildAgent/conf/buildAgent.properties. Edit serverUrl to point it at your TeamCity server. You may also want to set name to something recognizable, or the agent will default to your hostname. If you’re using Puppet, this is a good file to templatize.

Create writable directories

You’ll need to create the directories named in bin/agent.sh and conf/buildAgent.properties, including at least logs, work, temp, and system, as well as update and backup directories that I’ve only seen in log output. They should be writable by the teamcity user.

sudo install -d -o teamcity -g teamcity -m ug=rwx /opt/TeamCity/buildAgent/logs
sudo install -d -o teamcity -g teamcity -m ug=rwx /opt/TeamCity/buildAgent/work
sudo install -d -o teamcity -g teamcity -m ug=rwx /opt/TeamCity/buildAgent/temp
sudo install -d -o teamcity -g teamcity -m ug=rwx /opt/TeamCity/buildAgent/system
sudo install -d -o teamcity -g teamcity -m ug=rwx /opt/TeamCity/buildAgent/update
sudo install -d -o teamcity -g teamcity -m ug=rwx /opt/TeamCity/buildAgent/backup

Alternatively, you can map them to more FHS-friendly locations in /tmp and /var. logs is used by agent.sh for both logging and a PID file, but that can be changed by setting the LOG_DIR and PID_FILE environment variables in the service file.

More extra credit: there’s no easy way to specify file ownership in an fpm-generated .deb package. This is a known issue. This is not a limitation of the .deb format itself, which, broadly speaking, is a pair of tarballs. Solve this problem and your package can create those writable directories itself.

Start the agent

Run sudo systemctl start teamcity-build-agent. You can also run sudo systemctl status teamcity-build-agent to see if it started successfully.

Talking in both directions

Communication between a TeamCity server and a TeamCity agent is two-way. Agent→server connections can and should use HTTPS, but server→agent connections don’t yet support encryption.

You’ll need to make sure the agent can reach the server (outbound on on whatever port you used in the serverUrl section of the agent config file), and the server can likewise reach the agent (inbound on the port specified in ownPort and the IP address specified in ownAddress, if you’ve bound your agent to a specific IP). The inbound port is 9090 by default, and again, it’s not encrypted, so you’ll need to use a VPN, SSH tunnel, or equivalent between the server and agent.

Tagged , , , , , ,

find hidden files with lsflags, show them with chflags

I want to go fishing in my work Mac’s ~/Library folder for some app settings files, but Apple has decided to hide it from me. I could use -shift-G, or open ~/Library, but that’s not the kind of guy I am. Why is it hidden? The filename doesn’t start with a dot, so the Finder’s going on something other than Unix convention there. Were this 1998, I’d launch ResEdit and clear the folder’s HFS hidden flag. Maybe that’s still around in some form? I’ll look at the file’s extended attributes, because I’ve seen other HFS stuff like resource forks in there.

Arachnoscope:Desktop steelpangolin$ xattr -l ~/Library
com.apple.FinderInfo:
00000000  00 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00  |........@.......|
00000010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000020

Well, I was hoping for something more like com.apple.Hidden, but this is a start. Except it’s completely opaque. Googling… aha. A hidden flag is mentioned in chflags. But how do I confirm that this flag is actually why the Library’s omitted from the Finder’s listings? Is there any way to read flags?

Arachnoscope:Desktop steelpangolin$ ls /usr/bin | grep flag
chflags

Nope. Back to the man pages; I’ll have to write one myself: lsflags.c.

Now I’ll give it a go:

Arachnoscope:Desktop steelpangolin$ gcc -std=c99 -Wall lsflags.c -o lsflags
Arachnoscope:Desktop steelpangolin$ ./lsflags ~/Library
/Users/steelpangolin/Library: hidden
Arachnoscope:Desktop steelpangolin$ chflags nohidden ~/Library
Arachnoscope:Desktop steelpangolin$ ./lsflags ~/Library
/Users/steelpangolin/Library: 
Arachnoscope:Desktop steelpangolin$ xattr -l ~/Library

Hooray. The flag shows up, and it is indeed stored in the com.apple.FinderInfo xattr, because both disappear when I remove it. But what’s this, at the end of the first man page I found?

You can use “ls -lO” to see the flags of existing files.

Note to self: read all the way to the end next time.

Tagged ,

hello world, meet gpt-surgeon

gpt-surgeon has a new home on Launchpad! Look for it at https://launchpad.net/gpt-surgeon. Downloads compliant with the Launchpad release scheme (GPG-signed tarball) available here: https://launchpad.net/gpt-surgeon/+download

I’ve decided to go with GPLv2 as the licensing scheme, no copyright assignment required, to keep this simple utility open and available to the public with a minimum of hassle and a maximum of fresh code.

gpt-surgeon will still be available here on Bat Country for a time, but fresh development and new versions will be on the Launchpad site. This version will be frozen as soon as the Launchpad version is available.

Tagged , , , , , , ,

Better TurboGears LDAP authentication

TurboGears doesn’t have built-in LDAP authentication. This is annoying. I couldn’t get soldapprovider.py as provided from the TurboGears identity recipes page to work out of the box for my project (validate_password() was never getting called), and I wanted it to use my LDAP directory for more than just password validation, so I wrote my own soldapprovider.py, starting from the TurboGears soprovider.py. Unlike the original soldapprovider.py, mine only uses SQLObjects for storing user-visit associations. Everything else, including user object attributes, group memberships, and permissions, is pulled from the LDAP directory.

In the case of this project, the users and groups are set up as in RFC 2307. Users have a structural object class of inetOrgPerson and some extra attributes from a custom auxiliary class, which, if present, will be set as attributes on the user object. Attributes, group memberships, and permissions are read once, when the user object is instantiated inside load_identity() or validate_identity(). The key identity.soldapprovider.user_safe_attrs selects which attributes are used, and how they’re transformed. As is, the key’s default value is a mapping of names to functions, so this might not be ready for external configuration through TurboGears .conf files, which look sort of like Python code but aren’t.

Permissions are implemented as groupOfNames objects and named with cn attributes. My directory only uses two permissions, can_modify_account for users that are trusted to change their own passwords and edit their own contact into, and admin_powers, which is exactly what it sounds like. These two were set up so I could use them in ACLs in slapd.conf, which doesn’t support more complicated user-to-group mappings. I’m not entirely sure how permissions differ from groups in TurboGears other than in name, but this usage works for my app.

***

update:

This was written for TurboGears 1.0 and is no longer relevant, but I fixed a broken link to my soldapprovider.py anyway. All new development should be using TurboGears 2.0, which uses repoze for authentication and supports LDAP through repoze.

Tagged , , ,

ViewVC on DreamHost

I should start actually buying servers instead of recycling whatever boxes I have that would only be useful as footrests otherwise: bat-country.us was hosted on an old Power Mac G4 back at Caltech, but predictably, right after I left campus for Christmas break, it fell off the Internet, just like Turbine did all those years ago. Rather than harass some poor soul in Dabney into fixing it for me, I decided to move all my stuff into The Cloud, which seems to be the thing these days. Now the main site’s on DreamHost and the blog is this WordPress account which I forgot I had until finding an old bookmarks file the other day.

ViewVC has improved quite a bit since I was using it on Turbine, so now I’m using it again. Bernie Zimmerman has a handy page on getting it to work properly on DreamHost. The one bit he didn’t cover was mapping URLs to something more friendly.

Normally I’d just use Apache’s ScriptAlias directive to turn /viewvc.cgi/path/to/file into /code/path/to/file, but DreamHost doesn’t let you use ScriptAlias. They do allow mod_rewrite directives in .htaccess files, which covers the incoming URLs. My .htaccess looks like this:

RewriteEngine on
RewriteRule ^code/(.*)$ viewvc.cgi/$1
RewriteRule ^code$ viewvc.cgi

However, viewvc.cgi (the ViewVC entry point CGI script) needs to be told that it’s appearing to the user’s browser at the URL /code instead of the URL /viewvc.cgi, because otherwise all the links on pages generated by ViewVC point to the wrong place. Normally, ScriptAlias would take care of that by setting the SCRIPT_NAME CGI environment variable, but since I can’t use ScriptAlias, and ViewVC doesn’t seem to support any sort of config option that alters SCRIPT_NAME, I added the following line to the end of viewvc.cgi just before the main ViewVC code is invoked:

### complement to mod_rewrite
os.environ['SCRIPT_NAME'] = "/code"

# go do the work
import sapi
import viewvc

server = sapi.CgiServer()
cfg = viewvc.load_config(CONF_PATHNAME, server)
viewvc.main(server, cfg)

Problem solved.

Syntax highlighting in the beta version of ViewVC 1.1.0 that I’m using uses Pygments, and of course DreamHost’s Python installation doesn’t include Pygments. If you use Python on DreamHost, I highly recommend setting up a virtualenv wrapper for DreamHost’s Python, as described in the DreamHost wiki page on Python. First, I had to tell bash to look for executables in my own bin directory first:

export PATH="${HOME}/bin:${PATH}"
echo "export PATH=\"\${HOME}/bin:\${PATH}\"" >> ~/.bash_profile

Then I set up virtualenv wrappers for Python 2.5 and 2.4, in that order, which leaves the default python command pointing to 2.5 (a personal preference because it gives me the ternary conditional operator):

wget http://pypi.python.org/packages/2.4/v/virtualenv/virtualenv-1.3.2-py2.4.egg
unzip virtualenv-1.3.2-py2.4.egg
/usr/bin/python2.5 virtualenv-1.3.2-py2.4/virtualenv.py $HOME
/usr/bin/python2.4 virtualenv-1.3.2-py2.4/virtualenv.py $HOME

DreamHost’s Python 2.5 installation doesn’t have the Python bindings for Subversion, so I used 2.4, which does, for ViewVC purposes. (Don’t bother trying to build them for 2.5 unless you want to start from the very bottom: compiling stuff on DreamHost is a crapshoot, and my attempt crapped out while trying to link in some system library.) virtualenv comes with setuptools, so getting Pygments was a breeze:

easy_install-2.4 Pygments

The last step was changing the shebang/interpreter line at the top of viewvc.cgi from

#!/usr/bin/python

to

#!/home/my_username/bin/python2.4

to make CGI execution use the virtualenv version of Python 2.4 instead of the system’s Python, thus getting Pygments support and syntax coloring in ViewVC.

I only have one Subversion repository, so I wanted to suppress all the links to listing repository roots, to reduce visual clutter. This involved editing viewvc.conf to set default_root to my repository, disable root_as_url_component (which ignores default_root), and removing the roots view from allowed_views.

default_root = BatCountrySVN
root_as_url_component = 0
allowed_views = markup, annotate, co

I also had to tweak the newvc template I was using a bit, to remove the repository list references from the path navigation bar. Edited newvc/include/header.ext below vc_current_path to accomplish that, and done.

addition: ViewVC has an option to make Apache serve static content such as images and stylesheets, rather than serving them through the viewvc.cgi script, which is slower. I set

docroot = /viewvc_static

in viewvc.conf and then set up a mapping from http://bat-country.us/viewvc_static to /home/my_username/share/viewvc/templates/docroot using the DreamHost control panel (Domains ➔ Remap Sub-Dir).

Tagged , , , , , , ,