Category Archives: blog

Init scripts are a pain in the ass

Ideally you’d ship no init script and let downstream packaging figure it out, since there are even more init systems than there are package formats: various versions of SysV init; upstart for old versions of Ubuntu; systemd for new versions of Ubuntu, Debian, and Red Hat; OS X launchd for all those Homebrew users; and if you’re generous, or a masochist, the several variants of BSD rc.d; Solaris SMF; or even Windows Services.

If you don’t have the luxury of being popular enough that distros package for you, or you’re working on in-house stuff, then you might end up writing a SysV init script. It won’t be portable. Even if you’re using daemonize to handle the hard stuff like daemonizing, PID/lock files, and running as a non-root user, there are still little matters like the helper functions you need even with daemonize to kill the process or get its status. Are they in /etc/rc.d/init.d/functions? Are they in /lib/lsb/init-functions? Does your target system actually have either, and if so, what’s in them? Do you need to run Red Hat chkconfig after installing the script? How about Debian update-rc.d? Do you need an INIT INFO block? Does your target system claim to use LSB-compliant init scripts? Which LSB version? Do you really need all those entry points? Are you going to test it on systemd or upstart systems to see if their SysV backwards compatibility actually works?

OK, for modern systems, you can probably get away with just systemd and launchd, but no wonder a lot of new kids think “background service” means “run it in tmux“.

Advertisements

Fractalspotting: AVISynth

I mentioned denoising as an application of fractal image processing techniques in my Review of Fractal Image Compression and Related Algorithms, and recently found out that there has been at least one implementation in the wild, called FrFun7, for the AVISynth frameserver. Sadly, it’s closed source and the author appears to have vanished.

Tagged ,

How to package the TeamCity build agent for Debian/Ubuntu

This post brought to you by Thumbtack Engineering and the letters C and I.

How to package the TeamCity build agent for Debian/Ubuntu

We’ll use fpm to package the TeamCity 9 build agent for Debian 8 (“Jessie”) or Ubuntu Linux, including a systemd service file to start it automatically.

First, read the TeamCity 9 docs for “Setting up and Running Additional Build Agents”.

Get the TeamCity build agent

Download TeamCity 9 from JetBrains. I’m going to be using TeamCity 9.0.3. You can also get a version of the agent matched to your TeamCity install by going to the Agents tab and clicking “Install Build Agents” in the upper right.

Install fpm if you don’t have it already.

Unzip TeamCity-9.0.3.tar.gz and find the buildAgent directory. This is where all of the agent’s code and config files live. Since TeamCity wasn’t written with the FHS in mind, we’ll put it in /opt/TeamCity/buildAgent.

Make the agent scripts executable

Run chmod +x buildAgent/bin/*.sh. You won’t need to run them on the system you’re using for packaging, but fpm preserves the executable flag when it creates a package.

Create a systemd service file

Create a file named teamcity-build-agent.service and put this in it.

[Unit]
Description=TeamCity build agent

[Service]
User=teamcity
ExecStart=/opt/TeamCity/buildAgent/bin/agent.sh run

[Install]
WantedBy=multi-user.target

The run argument keeps the startup script from forking so that systemd can manage the process itself. The User section should be changed to reflect whatever user you plan on running TeamCity as. (See the systemd.exec man page for more process control options.)

This file will eventually be copied to /lib/systemd/system/teamcity-build-agent.service, because systemd is too cool for /etc.

Build a package with fpm

fpm has some usage examples on its wiki. We’re going to use the -s dir input type to take a directory and the -t deb output type to generate a Debian .deb package.

fpm
-s dir
-t deb
--name teamcity-build-agent
--version 9.0.3
--architecture all
--depends java6-runtime-headless
buildAgent=/opt/TeamCity/
teamcity-build-agent.service=/lib/systemd/system/teamcity-build-agent.service

This fpm invocation tells it thje package name and version, that the agent doesn’t care what architecture you’re using but requires Java 6, and that everything should go under /opt/TeamCity, except for the service file, which goes where systemd is expecting it.

You should now have a file named teamcity-build-agent_9.0.3_all.deb.

Create a user

Before installing the package on your target machine, run sudo adduser teamcity --system --group --home /opt/TeamCity, or add something equivalent to your configuration manager if that’s where you manage users. (I’ll be using Puppet to manage users on production machines that will have this package). The --group option creates a teamcity group along with the teamcity user, so that you can then add other users to the teamcity group if they need to look at TeamCity builds or configuration.

For extra credit, you could create and destroy the user in preinstall/postremove scripts.

Install the package

Copy the file to your target machine and run sudo dpkg -i teamcity-build-agent_9.0.3_all.deb.

Configure the package

Copy /opt/TeamCity/buildAgent/conf/buildAgent.dist.properties to /opt/TeamCity/buildAgent/conf/buildAgent.properties. Edit serverUrl to point it at your TeamCity server. You may also want to set name to something recognizable, or the agent will default to your hostname. If you’re using Puppet, this is a good file to templatize.

Create writable directories

You’ll need to create the directories named in bin/agent.sh and conf/buildAgent.properties, including at least logs, work, temp, and system, as well as update and backup directories that I’ve only seen in log output. They should be writable by the teamcity user.

sudo install -d -o teamcity -g teamcity -m ug=rwx /opt/TeamCity/buildAgent/logs
sudo install -d -o teamcity -g teamcity -m ug=rwx /opt/TeamCity/buildAgent/work
sudo install -d -o teamcity -g teamcity -m ug=rwx /opt/TeamCity/buildAgent/temp
sudo install -d -o teamcity -g teamcity -m ug=rwx /opt/TeamCity/buildAgent/system
sudo install -d -o teamcity -g teamcity -m ug=rwx /opt/TeamCity/buildAgent/update
sudo install -d -o teamcity -g teamcity -m ug=rwx /opt/TeamCity/buildAgent/backup

Alternatively, you can map them to more FHS-friendly locations in /tmp and /var. logs is used by agent.sh for both logging and a PID file, but that can be changed by setting the LOG_DIR and PID_FILE environment variables in the service file.

More extra credit: there’s no easy way to specify file ownership in an fpm-generated .deb package. This is a known issue. This is not a limitation of the .deb format itself, which, broadly speaking, is a pair of tarballs. Solve this problem and your package can create those writable directories itself.

Start the agent

Run sudo systemctl start teamcity-build-agent. You can also run sudo systemctl status teamcity-build-agent to see if it started successfully.

Talking in both directions

Communication between a TeamCity server and a TeamCity agent is two-way. Agent→server connections can and should use HTTPS, but server→agent connections don’t yet support encryption.

You’ll need to make sure the agent can reach the server (outbound on on whatever port you used in the serverUrl section of the agent config file), and the server can likewise reach the agent (inbound on the port specified in ownPort and the IP address specified in ownAddress, if you’ve bound your agent to a specific IP). The inbound port is 9090 by default, and again, it’s not encrypted, so you’ll need to use a VPN, SSH tunnel, or equivalent between the server and agent.

Tagged , , , , , ,

what servers am I running?

nmap localhost will tell you what ports are open:

Nmap scan report for localhost (127.0.0.1)
Host is up (0.00035s latency).
Not shown: 993 closed ports
PORT      STATE SERVICE
22/tcp    open  ssh
445/tcp   open  microsoft-ds
548/tcp   open  afp
631/tcp   open  ipp
3689/tcp  open  rendezvous
9001/tcp  open  tor-orport
10001/tcp open  scp-config

Nmap done: 1 IP address (1 host up) scanned in 3.74 seconds

but sudo lsof -i -n -P | grep LISTEN will tell you what's behind them. -i selects IPv4 and IPv6 sockets, -n turns off hostname translation, and -P turns off port name lookup, because the day /etc/services is correct or relevant will probably be the day the Sun burns out.

launchd       1           root   10u  IPv6 0x83483e3bed83a175      0t0    TCP *:548 (LISTEN)
launchd       1           root   11u  IPv4 0x83483e3bed84015d      0t0    TCP *:548 (LISTEN)
launchd       1           root   27u  IPv6 0x83483e3bed8399b5      0t0    TCP *:445 (LISTEN)
launchd       1           root   28u  IPv4 0x83483e3bed83f2ed      0t0    TCP *:445 (LISTEN)
launchd       1           root   33u  IPv6 0x83483e3bed8395d5      0t0    TCP [::1]:631 (LISTEN)
launchd       1           root   34u  IPv4 0x83483e3bed83ebb5      0t0    TCP 127.0.0.1:631 (LISTEN)
launchd       1           root   36u  IPv6 0x83483e3bed853175      0t0    TCP *:22 (LISTEN)
launchd       1           root   37u  IPv4 0x83483e3bed83e47d      0t0    TCP *:22 (LISTEN)
kdc          50           root    6u  IPv6 0x83483e3bed852d95      0t0    TCP *:88 (LISTEN)
SpotifyWe   627      jehrhardt    6u  IPv4 0x83483e3befe90bb5      0t0    TCP 127.0.0.1:4370 (LISTEN)
SpotifyWe   627      jehrhardt    7u  IPv4 0x83483e3befe9047d      0t0    TCP 127.0.0.1:4380 (LISTEN)
Spotify     637      jehrhardt   11u  IPv4 0x83483e3bed83d60d      0t0    TCP 127.0.0.1:4381 (LISTEN)
Spotify     637      jehrhardt   12u  IPv4 0x83483e3befe912ed      0t0    TCP 127.0.0.1:4371 (LISTEN)
Spotify     637      jehrhardt   26u  IPv4 0x83483e3c00bd1ed5      0t0    TCP *:32740 (LISTEN)
Spotify     637      jehrhardt   30u  IPv4 0x83483e3bf35eea25      0t0    TCP *:57621 (LISTEN)
Dropbox     639      jehrhardt   19u  IPv4 0x83483e3befe8eed5      0t0    TCP *:17500 (LISTEN)
Dropbox     639      jehrhardt   25u  IPv4 0x83483e3befe8e065      0t0    TCP 127.0.0.1:26164 (LISTEN)
ssh        2408      jehrhardt    6u  IPv6 0x83483e3bedb83175      0t0    TCP [::1]:54334 (LISTEN)
ssh        2408      jehrhardt    7u  IPv4 0x83483e3c07168a25      0t0    TCP 127.0.0.1:54334 (LISTEN)
python    11368      jehrhardt    5u  IPv4 0x83483e3c0182ca25      0t0    TCP 127.0.0.1:51637 (LISTEN)
AptanaStu 11507      jehrhardt  151u  IPv6 0x83483e3c00dbc9b5      0t0    TCP 127.0.0.1:8020 (LISTEN)
AptanaStu 11507      jehrhardt  162u  IPv6 0x83483e3bef15fd95      0t0    TCP *:9980 (LISTEN)
AptanaStu 11507      jehrhardt  200u  IPv6 0x83483e3c00dbcd95      0t0    TCP *:10001 (LISTEN)
AptanaStu 11507      jehrhardt  201u  IPv6 0x83483e3c00dbc5d5      0t0    TCP *:9001 (LISTEN)
AptanaStu 11507      jehrhardt  329u  IPv6 0x83483e3bed30b9b5      0t0    TCP *:51636 (LISTEN)
iTunes    21480      jehrhardt   39u  IPv4 0x83483e3c01b30ed5      0t0    TCP *:3689 (LISTEN)
iTunes    21480      jehrhardt   40u  IPv6 0x83483e3bed8529b5      0t0    TCP *:3689 (LISTEN)
Skype     37217      jehrhardt   50u  IPv4 0x83483e3c0053ed45      0t0    TCP *:62671 (LISTEN)
memcached 85144      jehrhardt   18u  IPv6 0x83483e3bed30c175      0t0    TCP [::1]:11211 (LISTEN)
memcached 85144      jehrhardt   19u  IPv4 0x83483e3c07156bb5      0t0    TCP 127.0.0.1:11211 (LISTEN)
memcached 85144      jehrhardt   20u  IPv6 0x83483e3c00dbd175      0t0    TCP [fe80:1::1]:11211 (LISTEN)

animated GIFs from Sea of Dreams

# trim and scale down in QuickTime Player
# FFMPEG: r = 10 FPS
ffmpeg -i the-colors.mov -r 10 -pix_fmt rgb24 the-colors.%04d.png
# ImageMagick: delay = 10/100 of a second between frames
convert -delay 10 the-colors.*.png the-colors.gif

Doesn’t take advantage of any of the intra-frame GIF optimizations, or a global color table, so the output is huge (~10 MiBs for 4 seconds), but the animated GIF is the haiku of the 20th century: a format defined by its limitations.

WordPress rewrites image tags to a scaled version which fits the theme width, so click to see the originals if these don’t animate.


colors-1


colors-2

my voting guide for propositions in California’s 2012 election

http://voterguide.sos.ca.gov/propositions/

30. YES to increase sales and income tax (supposedly progressively) for educational funding. California’s budget is so crippled by years of stupid propositions that I will vote up almost anything that increases taxes.

31. NO to this confused mess of state budget rule changes. The proposal to stop “gut and amend” by requiring that the full text of a proposed bill that has been approved by subcommittees be frozen to major changes is interesting, but the rest is a grab bag of land mines.

32. NO to reducing the paid political speech abilities of unions while leaving plenty of loopholes for corporations. Prop 32 is brought to you by everyone’s favorite shadowy billionaire douchebags, the Koch brothers.

33. NO on this insurance-company-backed proposition, which would permit rate hikes for people without a history of car insurance coverage. For example: people who didn’t have cars for a long period.

34. YES to stop the death penalty. Has some budget consequences I don’t understand – can anyone clarify?

35. NO to attaching the nebulous “human trafficking” label to a wide variety of crimes for no reason. The continued expansion of what is considered a “sex offense” is bad enough. Would also cost millions of dollars allocated to “human trafficking” which would probably come out of other law enforcement budget allocations.

36. YES to revise the Three Strikes law to focus on violent crimes only. Would let a lot of nonviolent offenders out of prison. Projected savings in the high tens of millions.

37. YES to require GMO products to be labeled. Some cost, not funded, but since Big Agritech is virulently opposed to 37, I’m all for it.

38. YES to a second income tax increase to fund education.

39. YES on a large tax increase on multistate corporations doing business in California. This would force them to pay their fair share instead of using a loophole specific to multistate businesses that lets them evade most CA corporate taxes.

40. YES to keep the Citizens Redistricting Commission-delineated districts as they apply to the state senate. Court action mandated that anyway, so the Commission lines are safe, but vote YES just to give an extra F U to the California Republican Party’s gerrymandering attempt.

failure of imagination

Mirror, mirror: The New York Times wants to serve you info as you’re brushing your teeth

It’s another smart house scheme that lives in your bathroom mirror and tracks your prescriptions by RFID. I got almost this exact pitch when I toured Microsoft Research’s demo smart house as a Microsoft intern in 2008. So what’s new? It’s an ad platform:

And it’s also a retail opportunity. So, say, based on my use of this product, that a local retailer wants to promote something to me.

“Mirror: show coupon.”
“Mirror: show coupon.”

I can call up a coupon here, and then save it to my phone, and then go to a physical location to redeem it. So, again, it’s a conversation that happens in front of the mirror, but then it can drive behavior elsewhere, out in the world.

Hey. Creative Technologist for R&D Brian House. This is 2011. In the event that I do purchase a magic mirror that shows me ads, and I see something I like, I’m not gonna bother asking it for a coupon, OBEXing or mailing or whatevering it to my phone, and then going to some brick and mortar store with it. Technology is supposed to save time! I’m gonna order the thing from Amazon and get it in the mail.

status update: single

Microsoft and I have parted ways.

Side effect: I will temporarily have more time to post here.

preparing to hibernate

This is more or less my non-work technical blog. Problem is, when you do coding for a living, you start to find coding for fun less fun. The major attraction to this blog is gpt_surgeon.py, and I’ll make sure that has a place somewhere so other people can work on it (Launchpad or SourceForge or CodePlex or something), but I don’t use a dual-boot configuration any more and I don’t have the time or the inclination to give gpt_surgeon.py the upgrades it needs. I’ll file a bug report at work with all the information I’ve been given by you, my readers, and hopefully (no promises) the underlying problem will get fixed somewhere down the line. If not, well, I hope someone picks up gpt_surgeon.py.

I plan to do something with my free time that doesn’t involve digging around in system guts for a change.

best-targeted spam ever

Subj: Necronomicon in PDF

WHY YES I DO WANT A COPY OF THAT. Sadly, the payload wasn’t even a trojan, just a Chinese-hosted pillz site. The default blocking of images from domains you don’t trust here is good; most clients have it now, and it was clearly developed as a response to web bugs but it also protects one from basilisks. It’s still trivial in the current email protocol to forge a sender, so trust no one – not even yourself.

Tagged , , , ,