Category Archives: tips & tricks

How to find the name of an Elastic Beanstalk environment from inside one of its instances

At Thumbtack, we’ve started using Amazon Elastic Beanstalk (EB) to deploy web services. We already use Papertrail as a log aggregator for our existing systems, and have installed their syslog forwarding agent on our EB apps using .ebextensions. However, I didn’t have a way to group log output from multiple EB-managed EC2 instances, or distinguish EB environments by name, because the default EC2 hostnames are just prefixed IP addresses. So I decided to use the EB environment name, which is unique across all EC2 regions, and tell Papertrail’s agent to use that instead of a hostname.

EB sets the environment ID and name as the elasticbeanstalk:environment-id and elasticbeanstalk:environment-name EC2 tags on all of the parts of an EB app environment: load balancer, EC2 instances, security groups, etc. Surprisingly, EC2 tags aren’t available to instances through the instance metadata interface, but they are available through the normal AWS API’s DescribeTags call. EB app container instances are based on Amazon Linux and have Python 2.6 and Boto preinstalled, so rather than go through the shell gyrations suggested by StackOverflow, I wrote a Python script (get-eb-env-name.py) that uses Boto to fetch the instance ID and region, then uses those two together with the instance’s IAM role to fetch its own tags, and prints the environment name.

You’ll need to make sure the IAM role used by your EB app environment has permissions to describe tags, as well as describing instances and possibly instance reservations as well. I’ve included an IAM policy file (EC2-Describe-IAM-policy.json) that you can apply to your IAM role to grant it the permission to describe any EC2 resource.

See my gist:

[gist 08880dc2c74b9c26cb5b /]

There’s an annoying wrinkle around getting the region for an EC2 instance through the instance metadata: you can’t, at least not through a directly documented method. You can get the availability zone (AZ) from http://169.254.169.254/latest/meta-data/placement/availability-zone, which will be something like us-west-2a, and then drop the last letter, giving you a region name like us-west-2. However, Amazon has never documented the relationship between AZ names and region names, so it’s possible this could break in the future.

You can also fetch an instance identity document in JSON format from http://169.254.169.254/latest/dynamic/instance-identity/document. Its contents aren’t documented either, but on all the EC2 instances I’ve examined, it looks like this, and contains the region as well as the AZ:

{
  "instanceId" : "i-12abcdef",
  "billingProducts" : null,
  "architecture" : "x86_64",
  "imageId" : "ami-dc123456",
  "pendingTime" : "2014-12-18T01:20:42Z",
  "instanceType" : "m3.2xlarge",
  "accountId" : "1234567890",
  "kernelId" : null,
  "ramdiskId" : null,
  "region" : "us-west-2",
  "version" : "2010-08-31",
  "privateIp" : "172.16.1.1",
  "availabilityZone" : "us-west-2a",
  "devpayProductCodes" : null
}

Since the instance identity document actually has a key named region, I decided to go with that method to find the region for an EC2 instance from inside that instance.

Previously, we’d set an environment variable or customize a config file to tell Papertrail what “hostname” to use. Finding the EB environment name at runtime from EC2 tags means that we don’t have to customize EB app bundles for each target environment, and thus can use the same app bundle across multiple environments; for example, from a CI build server’s test environment to a staging environment, and then to production.

Tagged , , , , , ,

Modifying SCons environment variables, globally

SCons, the Python-based build system, tries to isolate itself from the user’s environment as much as possible, so that one developer’s weird environment variables don’t lead to an irreproducible build. This is great until you want to use tools from places that SCons doesn’t think are standard, in which case you can make use of its site_scons extension mechanism to make them standard.

In this case, I’m using Homebrew. Homebrew normally installs into /usr/local. I think this default is completely mental, because /usr/local is the Balkans of Mac software packaging: routinely invaded and full of land mines. I’ve installed Homebrew into /opt/homebrew, where it’s safe from having parts overwritten at random by unmanaged package installers. For Make builds, I’ve included these lines at the end of my .zshrc to make Homebrew components available:

export PATH="/opt/homebrew/bin:$PATH"
export CFLAGS='-I/opt/homebrew/include
export LDFLAGS='-L/opt/homebrew/lib'

For SCons, I’ve created the file $HOME/.scons/site_scons/site_init.py to modify the Environment object used by SCons subprocesses:

Specifically, my site_init.py decorates the Environment initializer to always add Homebrew to the PATH, and now I have binaries like sdl-config available. (SCons has support for parsing the flags emitted by those config programs.) This should extend to CFLAGS and LDFLAGS, except that they’re most likely strings instead of lists. I’ve since refined the GIST to add CFLAGS, CXXFLAGS, and LDFLAGS to the SCons environment.

The end result is that I can assume that any SCons build on my system will have access to libraries and tools installed through Homebrew.

Tagged ,

ViewVC on DreamHost, revisited

While getting ready to install MoinMoin on bat-country.us to replace my current static pages, I noticed my ViewVC install was two years out of date. Readers seem to have found my writeup of the last time I installed ViewVC on DreamHost useful, so here’s an update.

DreamHost‘s standard environment now includes Python 2.5.2, virtualenv, and Python bindings for Subversion 1.5. This makes setting up ViewVC a good deal simpler than it used to be. This time around, I’ll be using FastCGI rather than classic CGI as the mechanism for getting Apache to talk to Python. I’d use WSGI, the preferred method of running Python webapps, but WSGI is apparently a little flaky on DreamHost.

Make sure $HOME/bin is in your $PATH environment variable before /usr/bin. My instructions below assume your working directory is $HOME.

Set up a Python virtualenv in your home directory, if you don’t have one already:
$ virtualenv $HOME

ViewVC uses Flup for FastCGI support, and Pygments for syntax highlighting. Install Flup and Pygments into the virtualenv:
$ easy_install Flup
$ easy_install Pygments

Download and unpack the latest stable release of ViewVC (currently 1.1.8):
$ wget http://viewvc.tigris.org/files/documents/3330/48659/viewvc-1.1.8.tar.gz
$ tar -xzf viewvc-1.1.8.tar.gz
$ rm viewvc-1.1.8.tar.gz

Run ViewVC’s installer script:
$ viewvc-1.1.8/viewvc-install --prefix=$HOME/viewvc

Delete the installer directory:
$ rm -rf viewvc-1.1.8

Configure viewvc.conf and the ViewVC templates to suit yourself, as per the ViewVC install instructions and my previous post on ViewVC setup. Remember to set up static content serving with the docroot option in viewvc.conf and the appropriate directory mapping.

Set up an .htaccess file in your domain folder to use Apache’s mod_rewrite for pretty URLs:

RewriteEngine on
RewriteRule ^code/(.*)$ viewvc.fcgi/$1
RewriteRule ^code$ viewvc.fcgi/

Copy viewvc.fcgi, the FastCGI entry point to ViewVC, to your domain folder:
$ cp viewvc/bin/wsgi/viewvc.fcgi example.org/viewvc.fcgi

ViewVC’s installer script will automatically create a viewvc.fcgi that points to the Python interpreter in your virtualenv (the one that has Flup and Pygments installed), so you don’t need to edit the #! line. You will still need to add a line to viewvc.fcgi to set the SCRIPT_NAME environment variable to a URL that matches your .htaccess rewrite rules. Instead of changing os.environ, change environ inside the application function that is the Flup entry point:

def application(environ, start_response):
  environ['SCRIPT_NAME'] = '/code'
  server = sapi.WsgiServer(environ, start_response)
  cfg = viewvc.load_config(CONF_PATHNAME, server)
  viewvc.main(server, cfg)
  return []

And with that, you’re in business. Took me less than half an hour, and that’s counting the time it took to write this.

Tagged , , , , , , ,

Digsby: useful, once you turn off the ads

Adium is the alpha and omega of chat clients. It does everything right. Unfortunately, it only runs on Macs. I just can’t seem to find a good multiprotocol IM client for Windows.

Way back in the dark ages, when Pidgin was GAIM 1.5 and the UI team hadn’t adopted their “our way or the highway” policy, I used that. But the Gtk+ interface always bugged me and GAIM 2.0 disabled a bunch of UI tweaks that I liked. There was a fork called Carrier or FunPidgin that re-enabled them, but like most annoyance-related forks, there wasn’t a lot of backing and it’s now long dead. So I kicked it to the curb and used Miranda for a while. Unfortunately, maintaining a Miranda install means dealing with at least a dozen plugins to provide features that should have been built in, like grouping a person’s multiple chat handles together (“metacontacts”), the ability to change smileys, a decent chat-history viewer, or a UI that doesn’t look like a throwback to Windows 95. Too much hassle.

Then I heard about Digsby somewhere. It’s not open source. It’s actually kind of unclear what their business model is. However, in terms of feature set, it’s the closest thing to Adium you can get on Windows, with the notable exceptions of OTR encryption and IRC support. It supports metacontacts, it works with Facebook messaging (always surprised by how many people use that), it has a decent message display, and it can be configured to be quite unobtrusive, a major selling point for me. Part of that configuration is turning off all the crap they ship with it. Honestly, I hate ad-supported free stuff, and I hate installers that carry a crapware payload even more. If it’s good, just make it payware. I’ll buy it. If it’s not good, I’m not going to put up with ads because it’s free – there are plenty of other freeware and OSS products out there that are usable.

To the credit of Digsby’s developers, you can turn off the various user-exploitation features! Here’s how.

There’s a constantly varying crapware payload in the Digsby installer: last time I installed Digsby, it was the Ask.com toolbar (link goes to uninstall directions). Real useful. Ask.com’s installer screen has the checkboxes for “Yes, please install this useless piece of data-harvesting crap” pre-checked, so don’t blindly click through.

Once you’ve got Digsby installed, open Digsby’s Preferences window from the systray icon or the Tools menu (shown):

Location of Digsby's Preferences window

The newest publicly available version of Digsby added ads under every conversation window; that’s what pissed me off enough to go looking for some way to kill them. In the Conversations pane, uncheck the box next to “Support Digsby development by showing an ad in the IM window“.

Location of checkbox to disable Digsby's conversation window ads

I want to smack anyone who sends emails ending with “sent from my iPhone” (or BlackBerry, or Hotmail account, or whatever). I prefer my messages unbranded, thanks. Digsby does some similar self-promotion in your AIM profile. In the General & Profile pane, uncheck the box next to “Promote Digsby in my AIM profile“.

And then there’s the most insidious piece: Digsby uses Plura’s platform to appropriate your CPU cycles for commercial purposes. This is one step away from botnet membership, and it uses the same verbiage as legitimate volunteer grid computing like BOINC (the platform that powers SETI@home). In the Research Module pane, uncheck the box next to “Allow Digsby to use CPU time to conduct research“. Maybe some day, commercial grid computing will advance to the point where I can make more money by running it than it costs to power the machines it’s running on, but on that day, the money better be going to me, not whoever landed a parasite program on my computer first.

With this junk disabled, Digsby is a nice little multiprotocol messenger program.

Tagged ,