A Jinja macro for generating an html select box with US states

More on Jinja macros

{% macro states_select(name, value='', class='', id='') -%}
 {% set states = ["AL", "AK", "AZ", "AR", "CA", "CO", "CT", "DE", "FL", "GA", "HI", "ID", "IL", "IN", "IA", "KS", "KY", "LA",
"ME", "MD", "MA", "MI", "MN", "MS", "MO", "MT", "NE", "NV", "NH", "NJ", "NM", "NY", "NC", "ND", "OH",
"OK", "OR", "PA", "RI", "SC", "SD", "TN", "TX", "UT", "VT", "VA", "WA", "WV", "WI", "WY"] %}
 <select name="{{name}}" class="{{class}}" id="{{id}}">
 {% for state in states %}
 <option value="{{state}}" {{'selected' if value==state else ''}}>{{state}}</option>
 {% endfor %}
 </select>
{%- endmacro %}

Capitalizing on the Value of the Elderly in Modern Society

As modern societies develop, become richer, and life expectancies increase, the ratio between the elderly, non-working population and younger, industrious population shifts towards unsustainability. In the United States in 2010, for every 100 working-age adults there are 22 non-working elderly. By 2030 there will be 37 non-working for every 100 working age adults. (from census) The increased dependency ratio on the 20-65 age group will slow economic growth and add burden to already strained government support programs.

What seems so backwards is that instead of providing value and filling important roles in society, much of the 65+ age group is filed away in retirement homes. As we age we often become less effective at the kind of social roles that modern society has chosen to put value on like manual labor and creative ingenuity.

What I suggest is that there are in fact important social roles in which the elderly populations can be very effective, but are not currently taking up in a systematic way.

Many grandparents are already playing a small role in the upbringing of grandchildren, but cultural, physical, and economic factors are preventing the full potential for that role to be reached. For every 100 working age adults there are 45 children who must be supported. Why are there so many child care facilities while there are so many able-minded elderly with nothing to do? One burdensome population can actually lessen the burden of the other.

Jared Diamond just gave an eye-opening TED talk comparing the value of elderly populations between traditional, tribal societies and modern westernized societies. Elderly were valued for their wisdom, leadership, craftsmanship, and child rearing abilities, freeing younger populations to hunt and fill other physically intensive roles. Elderly often lived among all ranges of age groups and could provide value wherever able. Responsibilities were shared among a community and not just a single immediately family. Though we live in a very different world now, we would be smart to learn something from historical social structures and see where we may institute programs to make better use of the human resource we collectively share.

Lawrence Lessig’s TED Talk on Reclaiming the Republic

Lessig talks about how the people have lost control of our government to those who have money. He tells stories of politicians and everyone connected to the political and legislative process evolving their thinking to appease their financiers. Deregulation is politically counter-productive because those involved get steady cash from lobbyists to manipulate those regulations. Total reform is nearly impossible because keeping the status quo means keeping your financial backing.

Lessig offers an emotional argument to draw attention to this issue of corruption, a word he uses many times. He reminds us that regardless of how impossible the task of forcing change in the engine of political finance may seem, we must do it because we love this country as we love our child. Lessig gives no path of action or inspirational light at the end of his talk, making it all the easier to push the problem out of our minds. I would at least resolve to keep the issue in the back of my mind in hope of having some small idea to affect change.

Beauty Can Make the World Safer

It’s amazing how appearances affect our behavior. Edi Rama, an artist and the former mayor of the capital of Albania gave an inspiring TED talk about transforming his city by adding color to old buildings and improving public spaces. Streets that used to be crime ridden became safer because they looked nicer. People are less likely to litter on clean, well groomed streets than streets already covered in dirty, covered in rubble, and broken windows. Shop owners and residents regain their pride and are protective of their neighborhood.

It’s amazing how this concept extends to so many areas.  Rama tells a story of combating corruption, again, using the tool of appearances. By improving a government building from a shack with a dark hole to a well-lit building with face to face interactions with officials it reduced the frequency of bribes. And a little closer to personal experience – when my office’s kitchen sink has a few dirty dishes laying about everyone feels like they can add to the pile, but as long as the sink is clean people tend to wash their own.

Comparison of Python Hashing Functions

At PubNub we deal with some pretty huge traffic and efficiency is always top of mind. Recently I needed a hashing function and decided to do some benchmarks. I found Robert Zaremba’s post which did a great job comparing various frameworks, but it seemed incomplete and didn’t include python’s built-in hashing functions. The pyhash library includes most algorithms so I could simplify the benchmark by quite a lot by not having to install those packages individually.

I had a feeling python’s built-in hash function would be faster, but I had no idea how much (700x faster than md5!!). (Update: oops! not really. The built-in hash function caches results so hashing the same string many times is not a fair comparison. Thanks to this comment on hacker news for the tip)

built-in hash: 0.00078284740448
md5: 0.574619698524
sha1: 0.666962075233
fnv1a_64: 0.38213365078
fnv1_64: 0.30924808979
murmur3_32: 0.117427706718
murmur2_x86_64b: 0.0808323383331
lookup3 0.153313565254
super_fast_hash 0.167356562614


"""
Adapted from Robert Zaremba

http://rz.scale-it.pl/2012/06/04/hash_function_benchmark_in_python.html

"""

import random
import string
import timeit
from time import time
from hashlib import md5, sha1
import pyhash # pip install pyhash
# http://code.google.com/p/pyfasthash/

hasher1 = pyhash.fnv1a_64()
hasher2 = pyhash.fnv1_64()
hasher3 = pyhash.murmur3_32()
hasher4 = pyhash.murmur2_x86_64b()
hasher5 = pyhash.lookup3()
hasher6 = pyhash.super_fast_hash()

#generate a 20000 character random string
data = ”.join(random.choice(string.ascii_letters + string.digits) for x in range(20000))
num = 10000
repeat_n=10
setup_c=”””
import random
import string
from hashlib import md5
import pyhash
data = ”.join(random.choice(string.ascii_letters + string.digits) for x in range(20000))
hasher1 = pyhash.fnv1a_64()
hasher2 = pyhash.fnv1_64()
hasher3 = pyhash.murmur3_32()
hasher4 = pyhash.murmur2_x86_64b()
hasher5 = pyhash.lookup3()
hasher6 = pyhash.super_fast_hash()
“””

print “*”*40
print “built-in hash:”, sum(timeit.repeat(stmt=”hash(data)”, setup=setup_c, repeat=repeat_n, number=num))/repeat_n
print “md5:”, sum(timeit.repeat(stmt=”md5(data).digest()”, setup=setup_c, repeat=repeat_n, number=num))/repeat_n
print “sha1:”, sum(timeit.repeat(stmt=”sha1(data).digest()”, setup=setup_c, repeat=repeat_n, number=num))/repeat_n
print “fnv1a_64:”, sum(timeit.repeat(stmt=”hasher1(data)”, setup=setup_c, repeat=repeat_n, number=num))/repeat_n
print “fnv1_64:”, sum(timeit.repeat(stmt=”hasher2(data)”, setup=setup_c, repeat=repeat_n, number=num))/repeat_n
print “murmur3_32:”, sum(timeit.repeat(stmt=”hasher3(data)”, setup=setup_c, repeat=repeat_n, number=num))/repeat_n
print “murmur2_x86_64b:”, sum(timeit.repeat(stmt=”hasher4(data)”, setup=setup_c, repeat=repeat_n, number=num))/repeat_n
print “lookup3″, sum(timeit.repeat(stmt=”hasher5(data)”, setup=setup_c, repeat=repeat_n, number=num))/repeat_n
print “super_fast_hash”, sum(timeit.repeat(stmt=”hasher6(data)”, setup=setup_c, repeat=repeat_n, number=num))/repeat_n

Arguments Against Owning a Home: Specialization of Labor

I recently read this article on TechCrunch by James Altucher entitled “Why Entrepreneurs Should NOT Buy Homes“. In discussing the topic with some friends, I was turned on to this article in the Wall Street Journal by economist Robert Bridges showing statistically why home ownership is almost always a poor investment. The idea rings true to me. Altucher focuses on Entrepreneurs, but I believe his argument could be extended to almost anyone and it has to do with specialization of skill.

You could summarize Altucher’s article in two core points:

  1. Entrepreneurs should not own homes because the investment is illiquid meaning you can’t use the money for other things when you need it. In the same sense, when you might need to pick up and move to a new city for a job or your level of income changes it can be difficult and expensive to sell your house.
  2. Owning a home takes time, new skills, and adds a lot of stress that distracts your from things that are most important: being really good at what you do and enjoying your life.

Altucher’s argument is similar to the idea of specialization of skill and division of labor. Economic prosperity and development really took off when people stopped doing everything for themselves and started to focus more on what they’re good at. If you’re good at making furniture it’s more efficient for you to spend your time doing that and investing money in better tools rather than spending half your day farming and investing money in farming equipment. Leave the food to someone who does that for a living and sell him some furniture in exchange. Likewise, leave the property management and investment to someone who does that for a living and focus your time and money on being better at whatever it is you do – be it startups or photography or teaching economics.

If, on the other hand, you really ENJOY fixing up houses and aren’t worried about moving soon, it could make sense to buy a house for personal reasons, but less as an economic investment. Also, this argument ignores the psychological aspects of lack of disciplin for investing controlled by forced mortgage payments, but a similar effect can maxing out your 401k and other investment strategies.

How-to Set Up Ubuntu w/ MongoDB Replica Sets on Amazon EC2

This tutorial is intended for beginners who aren’t familiar with EC2 yet, but are generally familiar with mongoDB. EC2 is actually pretty easy, but a lot of the basic info you need to get started is interspersed across numerous websites and articles. This post hopefully puts all the necessary details in one place.

The first thing to understand is that every EC2 instance runs an AMI (Amazon Machine Image) which is basically a bundle of one or more EBS (Elastic Block Storage) snapshots. The physical machine that your instance is hosted on has build in hard drive space, but it isn’t persistent. When you shut down or reboot the server whatever is on that disk will be wiped. Amazon already has a database of community AMI’s including basic Ubuntu installs. We can use one of these, then install the necessary packages, update configs, etc. and save the configured snapshot as our own AMI. Problem is, when you search the community AMI’s for ‘ubuntu’ you get some 500 results, so which one do we pick? http://alestic.com is a good resource for things related to EC2 and Ubuntu and they have a list of ‘official’ AMIs from Canonical. I’m basing my EC2 instance in amazon’s us-east1 data center so the AMI identifier for Ubuntu 11.04 EBS 64bit is ami-1aad5273. If your EC2 instances are located somewhere else, you’ll need the corresponding AMI identifier for that data center, which can be found on alestic.com

To start off, you can follow the EC2 getting started guide, except instead of the Basic Linux AMI you can use the Ubuntu AMI that I mentioned above. There’s also no need to terminate the instance at the end since we’ll just roll right into customizing this instance for MongoDB.

I like to start but getting any system updates that have come out since the AMI was created:

sudo apt-get update
sudo apt-get upgrade

I also like to install the linux tools dstat and htop to monitor system performance.

After following Amazon’s Getting Started Guide you should have a blank Ubuntu box and be SSH’ed into it. The linux root partition is usually an EBS volume and I like to make a second EBS volume that I can mount for just the mongodb database directory. This way I can detach the database volume and move it to another running instance. So go into the AWS Management Console and click on Volumes on the left. Create a new volume that has ample space for your database. You can’t resize these things so leave room to grow. After you create the EBS volume you need to attach it to your EC2 instance and define a mount point. I usually use /dev/sde.

Next, let’s log into the EC2 instance by ssh. We need to format the new volume, mount it, and add it to /etc/fstab so it auto-mounts when we restart. (note: on Ubuntu Natty 11.04 the drive ends up appearing as /dev/xvde, but on older systems and other flavors of linux it might still be /dev/sde)

sudo mkfs -t ext4 /dev/xvde

I’m going to mount my new volume at /db

sudo mkdir /db
sudo vim /etc/fstab

add the following line to the bottom of your /etc/fstab

/dev/xvde        /db     auto    noatime,noexec,nodiratime 0 0

We can either restart to auto-mount it or we can manually mount it now using

sudo mount /dev/xvde /db

Now lets install mongodb. Here are the official docs.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
sudo vim /etc/apt/sources.list
deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen
sudo apt-get update
sudo apt-get install mongodb-10gen

sudo mkdir /db/mongodb
sudo chown mongodb:mongodb /db/mongodb

Now lets edit /etc/mongodb.conf and change the location of the database. Near the top change dbpath so it looks like this:

dbpath=/db/mongodb

I also like to change my oplogSize to something larger than the default so if a secondary instance is down I have longer to bring it back up before it becomes too stale to re-sync. I also recommend turning on journaling to prevent data corruption.

oplogSize = 10000
replSet = myReplicaSet
journal = true

If you’re using a hostname in the replica set configuration instead of the IP address, you need to configure that in /etc/hostname and /etc/hosts

/etc/hostname:

db1

/etc/hosts:

127.0.0.1     db1   db1.mydomain.com    localhost.localdomain    localhost
xxx.xxx.xxx.xxx    db1   db1.mydomain.com

(where xxx.xxx.xxx.xxx is this machine’s IP address that you use in the replica set config. Usually the elastic IP.)

After changing hostname information you’ll need to restart the instance for it to take affect.

You need to add a hole in the EC2 firewall for the other replica nodes. Do this by going to the Security Groups section of the EC2 dashboard. Click on the security group you’re using and add a custom line TCP from port 27017, with /32 as the IP address for each node. (where xxx.xxx.xxx.xxx is the instances IP address). Each node of the replica set needs to be able to access every other node of the replica set. Best way to do this is use the same security group for all of them and add all IP addresses to the allowed list.

When you have the instance basically set, go back into the AWS control panel, right click the instance and choose Create Image. You can start up any number of these for the replica set, but you need to change the /etc/hostname and /etc/hosts file to reflect the individual IP address and hostname of the bot (db1, db2, db3, etc.)

From here on the instructions in MongoDB Replica Set Configuration docs are valid. You don’t need to specify the replSet name on the command line since we already set it in the config file. mongoDB should be already running, but you can restart it with /etc/init.d/mongodb restart if you change any configuration parameters.

Running cherrypy on multiple ports (example)

As a continuation of my previous post on how to run cherrypy as an SSL server as HTTPS (port 443), this tutorial show how to run a single cherrypy instance on multiple ports for both HTTP (port 80) and HTTPS (port 443)

We need to do a few things differently than in most examples out there like how to set configs when not using the quickstart() function and creating multiple Server() objects. But after reading through the source code a little it becomes clear.

import cherrypy

class RootServer:
    @cherrypy.expose
    def index(self, **keywords):
        return "it works!"

if __name__ == '__main__':
    site_config = {
        '/static': {
            'tools.staticdir.on': True,
            'tools.staticdir.dir': "/home/ubuntu/my_website/static"
        },
        '/support': {
            'tools.staticfile.on': True,
            'tools.staticfile.filename': "/home/ubuntu/my_website/templates/support.html"
        }
    }
    cherrypy.tree.mount(RootServer())

    cherrypy.server.unsubscribe()

    server1 = cherrypy._cpserver.Server()
    server1.socket_port=443
    server1._socket_host='0.0.0.0'
    server1.thread_pool=30
    server1.ssl_module = 'pyopenssl'
    server1.ssl_certificate = '/home/ubuntu/my_cert.crt'
    server1.ssl_private_key = '/home/ubuntu/my_cert.key'
    server1.ssl_certificate_chain = '/home/ubuntu/gd_bundle.crt'
    server1.subscribe()

    server2 = cherrypy._cpserver.Server()
    server2.socket_port=80
    server2._socket_host="0.0.0.0"
    server2.thread_pool=30
    server2.subscribe()

    cherrypy.engine.start()
    cherrypy.engine.block()

Using SSL HTTPS with cherrypy 3.2.0 Example

It took me more time than it should have to piece together the right bits of current information for using SSL with cherrypy. Here’s a fully working example of cherrypy 3.2.0 serving up HTTPS requests.

Quick notes – if you haven’t tried cherrypy, do it. It’s awesome in its simplicity. Also, I got my SSL cert from godaddy, which was the cheapest I found. This particular cert uses a certificate chain, so when all is said and done we have my_cert.crt, my_cert.key, and gd_bundle.crt.

ssl_server.py:

import cherrypy

class RootServer:
    @cherrypy.expose
    def index(self, **keywords):
        return "it works!"

if __name__ == '__main__':
    server_config={
        'server.socket_host': '0.0.0.0',
        'server.socket_port':443,

        'server.ssl_module':'pyopenssl',
        'server.ssl_certificate':'/home/ubuntu/my_cert.crt',
        'server.ssl_private_key':'/home/ubuntu/my_cert.key',
        'server.ssl_certificate_chain':'/home/ubuntu/gd_bundle.crt'
    }

    cherrypy.config.update(server_config)
    cherrypy.quickstart(RootServer())

Launch the server like:

sudo python ssl_server.py

You need to use sudo because it runs on port 443. You should be asked to “Enter PEM pass phrase” that you set when generating your key.

Update: In a follow-up post I show how you run an HTTPS server (port 443) and an HTTP server (port 80) at the same time.