Tag Archives: python

How to find the name of an Elastic Beanstalk environment from inside one of its instances

At Thumbtack, we’ve started using Amazon Elastic Beanstalk (EB) to deploy web services. We already use Papertrail as a log aggregator for our existing systems, and have installed their syslog forwarding agent on our EB apps using .ebextensions. However, I didn’t have a way to group log output from multiple EB-managed EC2 instances, or distinguish EB environments by name, because the default EC2 hostnames are just prefixed IP addresses. So I decided to use the EB environment name, which is unique across all EC2 regions, and tell Papertrail’s agent to use that instead of a hostname.

EB sets the environment ID and name as the elasticbeanstalk:environment-id and elasticbeanstalk:environment-name EC2 tags on all of the parts of an EB app environment: load balancer, EC2 instances, security groups, etc. Surprisingly, EC2 tags aren’t available to instances through the instance metadata interface, but they are available through the normal AWS API’s DescribeTags call. EB app container instances are based on Amazon Linux and have Python 2.6 and Boto preinstalled, so rather than go through the shell gyrations suggested by StackOverflow, I wrote a Python script (get-eb-env-name.py) that uses Boto to fetch the instance ID and region, then uses those two together with the instance’s IAM role to fetch its own tags, and prints the environment name.

You’ll need to make sure the IAM role used by your EB app environment has permissions to describe tags, as well as describing instances and possibly instance reservations as well. I’ve included an IAM policy file (EC2-Describe-IAM-policy.json) that you can apply to your IAM role to grant it the permission to describe any EC2 resource.

See my gist:

[gist 08880dc2c74b9c26cb5b /]

There’s an annoying wrinkle around getting the region for an EC2 instance through the instance metadata: you can’t, at least not through a directly documented method. You can get the availability zone (AZ) from http://169.254.169.254/latest/meta-data/placement/availability-zone, which will be something like us-west-2a, and then drop the last letter, giving you a region name like us-west-2. However, Amazon has never documented the relationship between AZ names and region names, so it’s possible this could break in the future.

You can also fetch an instance identity document in JSON format from http://169.254.169.254/latest/dynamic/instance-identity/document. Its contents aren’t documented either, but on all the EC2 instances I’ve examined, it looks like this, and contains the region as well as the AZ:

{
  "instanceId" : "i-12abcdef",
  "billingProducts" : null,
  "architecture" : "x86_64",
  "imageId" : "ami-dc123456",
  "pendingTime" : "2014-12-18T01:20:42Z",
  "instanceType" : "m3.2xlarge",
  "accountId" : "1234567890",
  "kernelId" : null,
  "ramdiskId" : null,
  "region" : "us-west-2",
  "version" : "2010-08-31",
  "privateIp" : "172.16.1.1",
  "availabilityZone" : "us-west-2a",
  "devpayProductCodes" : null
}

Since the instance identity document actually has a key named region, I decided to go with that method to find the region for an EC2 instance from inside that instance.

Previously, we’d set an environment variable or customize a config file to tell Papertrail what “hostname” to use. Finding the EB environment name at runtime from EC2 tags means that we don’t have to customize EB app bundles for each target environment, and thus can use the same app bundle across multiple environments; for example, from a CI build server’s test environment to a staging environment, and then to production.

Tagged , , , , , ,

ViewVC on DreamHost, revisited

While getting ready to install MoinMoin on bat-country.us to replace my current static pages, I noticed my ViewVC install was two years out of date. Readers seem to have found my writeup of the last time I installed ViewVC on DreamHost useful, so here’s an update.

DreamHost‘s standard environment now includes Python 2.5.2, virtualenv, and Python bindings for Subversion 1.5. This makes setting up ViewVC a good deal simpler than it used to be. This time around, I’ll be using FastCGI rather than classic CGI as the mechanism for getting Apache to talk to Python. I’d use WSGI, the preferred method of running Python webapps, but WSGI is apparently a little flaky on DreamHost.

Make sure $HOME/bin is in your $PATH environment variable before /usr/bin. My instructions below assume your working directory is $HOME.

Set up a Python virtualenv in your home directory, if you don’t have one already:
$ virtualenv $HOME

ViewVC uses Flup for FastCGI support, and Pygments for syntax highlighting. Install Flup and Pygments into the virtualenv:
$ easy_install Flup
$ easy_install Pygments

Download and unpack the latest stable release of ViewVC (currently 1.1.8):
$ wget http://viewvc.tigris.org/files/documents/3330/48659/viewvc-1.1.8.tar.gz
$ tar -xzf viewvc-1.1.8.tar.gz
$ rm viewvc-1.1.8.tar.gz

Run ViewVC’s installer script:
$ viewvc-1.1.8/viewvc-install --prefix=$HOME/viewvc

Delete the installer directory:
$ rm -rf viewvc-1.1.8

Configure viewvc.conf and the ViewVC templates to suit yourself, as per the ViewVC install instructions and my previous post on ViewVC setup. Remember to set up static content serving with the docroot option in viewvc.conf and the appropriate directory mapping.

Set up an .htaccess file in your domain folder to use Apache’s mod_rewrite for pretty URLs:

RewriteEngine on
RewriteRule ^code/(.*)$ viewvc.fcgi/$1
RewriteRule ^code$ viewvc.fcgi/

Copy viewvc.fcgi, the FastCGI entry point to ViewVC, to your domain folder:
$ cp viewvc/bin/wsgi/viewvc.fcgi example.org/viewvc.fcgi

ViewVC’s installer script will automatically create a viewvc.fcgi that points to the Python interpreter in your virtualenv (the one that has Flup and Pygments installed), so you don’t need to edit the #! line. You will still need to add a line to viewvc.fcgi to set the SCRIPT_NAME environment variable to a URL that matches your .htaccess rewrite rules. Instead of changing os.environ, change environ inside the application function that is the Flup entry point:

def application(environ, start_response):
  environ['SCRIPT_NAME'] = '/code'
  server = sapi.WsgiServer(environ, start_response)
  cfg = viewvc.load_config(CONF_PATHNAME, server)
  viewvc.main(server, cfg)
  return []

And with that, you’re in business. Took me less than half an hour, and that’s counting the time it took to write this.

Tagged , , , , , , ,

hello world, meet gpt-surgeon

gpt-surgeon has a new home on Launchpad! Look for it at https://launchpad.net/gpt-surgeon. Downloads compliant with the Launchpad release scheme (GPG-signed tarball) available here: https://launchpad.net/gpt-surgeon/+download

I’ve decided to go with GPLv2 as the licensing scheme, no copyright assignment required, to keep this simple utility open and available to the public with a minimum of hassle and a maximum of fresh code.

gpt-surgeon will still be available here on Bat Country for a time, but fresh development and new versions will be on the Launchpad site. This version will be frozen as soon as the Launchpad version is available.

Tagged , , , , , , ,

Convert Adium smileys to Colloquy smileys

Like almost every chat client, Colloquy uses its own format for smileys. There are many more smileys available for Adium than there are in Colloquy format, so I wrote a utility to convert Adium emoticon packages to Colloquy emoticon packages. It’s a Python script, and requires the plistlib module, which means you’ll need Python 2.5 if you’re running it on a Mac or Python 2.6 if you’re not.

Note: plistlib only handles XML-format plists, so if you get parse errors, your Adium smiley pack probably has a binary-format Emoticons.plist (the file inside the pack that describes the provided smileys.) To fix this, use OS X’s plutil tool to rewrite Emoticons.plist:

plutil -convert xml1 smileys.AdiumEmoticonSet/Emoticons.plist
Tagged , ,

Better TurboGears LDAP authentication

TurboGears doesn’t have built-in LDAP authentication. This is annoying. I couldn’t get soldapprovider.py as provided from the TurboGears identity recipes page to work out of the box for my project (validate_password() was never getting called), and I wanted it to use my LDAP directory for more than just password validation, so I wrote my own soldapprovider.py, starting from the TurboGears soprovider.py. Unlike the original soldapprovider.py, mine only uses SQLObjects for storing user-visit associations. Everything else, including user object attributes, group memberships, and permissions, is pulled from the LDAP directory.

In the case of this project, the users and groups are set up as in RFC 2307. Users have a structural object class of inetOrgPerson and some extra attributes from a custom auxiliary class, which, if present, will be set as attributes on the user object. Attributes, group memberships, and permissions are read once, when the user object is instantiated inside load_identity() or validate_identity(). The key identity.soldapprovider.user_safe_attrs selects which attributes are used, and how they’re transformed. As is, the key’s default value is a mapping of names to functions, so this might not be ready for external configuration through TurboGears .conf files, which look sort of like Python code but aren’t.

Permissions are implemented as groupOfNames objects and named with cn attributes. My directory only uses two permissions, can_modify_account for users that are trusted to change their own passwords and edit their own contact into, and admin_powers, which is exactly what it sounds like. These two were set up so I could use them in ACLs in slapd.conf, which doesn’t support more complicated user-to-group mappings. I’m not entirely sure how permissions differ from groups in TurboGears other than in name, but this usage works for my app.

***

update:

This was written for TurboGears 1.0 and is no longer relevant, but I fixed a broken link to my soldapprovider.py anyway. All new development should be using TurboGears 2.0, which uses repoze for authentication and supports LDAP through repoze.

Tagged , , ,

ViewVC on DreamHost

I should start actually buying servers instead of recycling whatever boxes I have that would only be useful as footrests otherwise: bat-country.us was hosted on an old Power Mac G4 back at Caltech, but predictably, right after I left campus for Christmas break, it fell off the Internet, just like Turbine did all those years ago. Rather than harass some poor soul in Dabney into fixing it for me, I decided to move all my stuff into The Cloud, which seems to be the thing these days. Now the main site’s on DreamHost and the blog is this WordPress account which I forgot I had until finding an old bookmarks file the other day.

ViewVC has improved quite a bit since I was using it on Turbine, so now I’m using it again. Bernie Zimmerman has a handy page on getting it to work properly on DreamHost. The one bit he didn’t cover was mapping URLs to something more friendly.

Normally I’d just use Apache’s ScriptAlias directive to turn /viewvc.cgi/path/to/file into /code/path/to/file, but DreamHost doesn’t let you use ScriptAlias. They do allow mod_rewrite directives in .htaccess files, which covers the incoming URLs. My .htaccess looks like this:

RewriteEngine on
RewriteRule ^code/(.*)$ viewvc.cgi/$1
RewriteRule ^code$ viewvc.cgi

However, viewvc.cgi (the ViewVC entry point CGI script) needs to be told that it’s appearing to the user’s browser at the URL /code instead of the URL /viewvc.cgi, because otherwise all the links on pages generated by ViewVC point to the wrong place. Normally, ScriptAlias would take care of that by setting the SCRIPT_NAME CGI environment variable, but since I can’t use ScriptAlias, and ViewVC doesn’t seem to support any sort of config option that alters SCRIPT_NAME, I added the following line to the end of viewvc.cgi just before the main ViewVC code is invoked:

### complement to mod_rewrite
os.environ['SCRIPT_NAME'] = "/code"

# go do the work
import sapi
import viewvc

server = sapi.CgiServer()
cfg = viewvc.load_config(CONF_PATHNAME, server)
viewvc.main(server, cfg)

Problem solved.

Syntax highlighting in the beta version of ViewVC 1.1.0 that I’m using uses Pygments, and of course DreamHost’s Python installation doesn’t include Pygments. If you use Python on DreamHost, I highly recommend setting up a virtualenv wrapper for DreamHost’s Python, as described in the DreamHost wiki page on Python. First, I had to tell bash to look for executables in my own bin directory first:

export PATH="${HOME}/bin:${PATH}"
echo "export PATH=\"\${HOME}/bin:\${PATH}\"" >> ~/.bash_profile

Then I set up virtualenv wrappers for Python 2.5 and 2.4, in that order, which leaves the default python command pointing to 2.5 (a personal preference because it gives me the ternary conditional operator):

wget http://pypi.python.org/packages/2.4/v/virtualenv/virtualenv-1.3.2-py2.4.egg
unzip virtualenv-1.3.2-py2.4.egg
/usr/bin/python2.5 virtualenv-1.3.2-py2.4/virtualenv.py $HOME
/usr/bin/python2.4 virtualenv-1.3.2-py2.4/virtualenv.py $HOME

DreamHost’s Python 2.5 installation doesn’t have the Python bindings for Subversion, so I used 2.4, which does, for ViewVC purposes. (Don’t bother trying to build them for 2.5 unless you want to start from the very bottom: compiling stuff on DreamHost is a crapshoot, and my attempt crapped out while trying to link in some system library.) virtualenv comes with setuptools, so getting Pygments was a breeze:

easy_install-2.4 Pygments

The last step was changing the shebang/interpreter line at the top of viewvc.cgi from

#!/usr/bin/python

to

#!/home/my_username/bin/python2.4

to make CGI execution use the virtualenv version of Python 2.4 instead of the system’s Python, thus getting Pygments support and syntax coloring in ViewVC.

I only have one Subversion repository, so I wanted to suppress all the links to listing repository roots, to reduce visual clutter. This involved editing viewvc.conf to set default_root to my repository, disable root_as_url_component (which ignores default_root), and removing the roots view from allowed_views.

default_root = BatCountrySVN
root_as_url_component = 0
allowed_views = markup, annotate, co

I also had to tweak the newvc template I was using a bit, to remove the repository list references from the path navigation bar. Edited newvc/include/header.ext below vc_current_path to accomplish that, and done.

addition: ViewVC has an option to make Apache serve static content such as images and stylesheets, rather than serving them through the viewvc.cgi script, which is slower. I set

docroot = /viewvc_static

in viewvc.conf and then set up a mapping from http://bat-country.us/viewvc_static to /home/my_username/share/viewvc/templates/docroot using the DreamHost control panel (Domains ➔ Remap Sub-Dir).

Tagged , , , , , , ,