simon.net.nz » code

python-nexus – a generic nexus (.nex, .trees) reader for python

Simon — Thu, 17 Dec 2009 01:12:17 +0000

python-nexus - a generic nexus (.nex, .trees) reader for python

© Simon J. Greenhill, 2009

Contact me if you have any problems.

Download:

You can download python-nexus from the bitbucket repository.

Usage:

Reading a Nexus

PLAIN TEXT
PYTHON:
>>> from nexus import NexusReader
>>> n = NexusReader()
>>> n.read_file('example.nex')

# or
>>> n = NexusReader('example.nex')
...

# display blocks found in data file
>>> n.blocks
{'data': 2 characters from 4 taxa>}

#`data` blocks
# get the number of characters
>>> n.data.nchar
2

# get the number of taxa
>>> n.data.ntaxa
4

# get the format string
>>> n.data.format
{'datatype': 'standard', 'symbols': '01', 'gap': '-'}

# access the data matrix
>>> n.data.matrix
{'Simon': ['01'], 'Louise': ['11'], 'Betty': ['10'], 'Harry': ['00']}

# access by taxon:
>>> n.data.matrix['Simon']
['01']

# get list of taxa
>>> n.data.taxa
['Betty', 'Harry', 'Louise', 'Simon']

# loop over taxa and data
>>> for taxon, characters in n.data:
...

# `tree` blocks

>>> n = NexusReader('example.trees')
# get the number of trees
>>> n.trees.ntrees
3

# get a tree
>>> n.trees.trees[0]
'tree tree.0.1065.603220 = (((((((Chris:0.0668822155,Bruce:0.0173144449):0.0062091603,Tom:0.0523825242):0.0206190840,(Henry:0.0482653647,Timothy:0.0744964092):0.0183093750):0.0401805957,(Mark:0.0066961591,Simon:0.0755275882):0.0264078188):0.0536464636,((Fred:0.0428499135,Kevin:0.0734738565):0.0937536292,Roger:0.0538708492):0.0438297939):0.0453008384,(Michael:0.0953237112,Andrew:0.0654710419):0.0803079594):0.0630363263,David:0.0855948485);'

# loop over trees
>>> for tree in n.trees:
...

Writing a Nexus File

PLAIN TEXT

PYTHON:

>>> from nexus import NexusWriter
>>> n = NexusWriter()
# Add a comment to appear in the header of the file
>>> n.add_comment("I am a comment")
#data are added by using the "add" function -
#which takes 3 arguments, a taxon, a character name, and a value
>>> n.add('taxon1', 'Character1', 'A')
>>> n.data
{'Character1': {'taxon1': 'A'}}
>>> n.add('taxon2', 'Character1', 'C')
>>> n.add('taxon3', 'Character1', 'A')
#Characters and values can be strings or integers
>>> n.add('taxon1', 2, 1)
>>> n.add('taxon2', 2, 2)
>>> n.add('taxon3', 2, 3)
#NexusWriter will interpolate missing entries (i.e. taxon2 in this case)
>>> n.add('taxon1', "Char3", '4')
>>> n.add('taxon3', "Char3", '4')
# ... when you're ready, you can generate the nexus using `make_nexus` or `write_to_file`:
>>> n.make_nexus(interleave=True, charblock=True)
>>> n.write_to_file(filename="output.nex", interleave=True, charblock=True)

r8spermute

Simon — Wed, 14 Oct 2009 02:09:35 +0000

r8spermute - Runs all permutations of r8s constraints.

© Simon J. Greenhill, 2009

This is a small program to assess how robust divergence times are based on the calibrations given to the program r8s.

The program r8s by Mike Sanderson (http://loco.biosci.arizona.edu/r8s/) [1, 2] performs a number of phylogenetic dating methods on a set of phylogenetic trees. In essence, you plug in a number of calibration points from known historical evidence and the program r8s takes your estimated trees, and "smoothes" the rates of change observed on those trees, using the calibrations to convert the branches into time.

However, it is often useful to explore the effect that certain calibration has on the various date estimates. For example, in the paper [3], I wanted to estimate the age at which a large language family originated. We had about 10 different calibration points based on historical evidence. Some of these calibrations were, however, more controversial than the others. So, I wrote this program to take a tree (or set of trees) and analyse them under all the combinations of these calibrations. This allowed us to assess the relative effects on the date estimates of the different calibrations.

If you find this program useful, please cite reference [3].

Contact me if you have any problems.

Download:

You can download r8spermute from the bitbucket repository here: download r8spermute

Usage:

You will need two things: 1) A set of trees, 2) A r8s command block. Examples of both of these are provided in the "examples" directory.

Your r8s block file will look something like this:

PLAIN TEXT

CODE:

begin rates;
blformat lengths=persite nsites=1000 ultrametric=no round=yes;
collapse;
mrca node1 Daisy Fiona;
constrain taxon=node1 min_age=1.8 max_age=2.5;
mrca node2 Robert Tamara;
constrain taxon=node2 min_age=1.1 max_age=1.3;
unfixage taxon=Simon;
constrain taxon=Simon min_age=0.7 max_age=1.2;
set num_restarts=5;
set smoothing=10;
divtime method=pl algorithm=tn;
showage shownamed = yes;
profile taxon=node1 parameter=age;
profile taxon=node2 parameter=age;
describe plot=chrono_description;
end;

... thus, we have 3 calibrations - node1, node2, and a constraint on the terminal taxon "Simon".

We can then run r8spermute like this:

PLAIN TEXT

CODE:

r8spermute.py example/example.trees example/example.r8s

This will start r8s running and you should then see some output like this:

PLAIN TEXT

CODE:

Constraints found: 3
Constraints ignored: 0
Total Constraint Combinations: 7
Running analysis 1 of 7 (0.14%)
Staging in: /var/folders/9Z/9Z3O2T2o2RmgRU+F75TSx++++TQ/-Tmp-/tmpXn1lw9
Constraints:
+ constrain taxon=node1 min_age=1.8 max_age=2.5;
Logging in: example/example_100.log
r8s version 1.71 (compiled May 16 2006)
[...reading file /var/folders/9Z/9Z3O2T2o2RmgRU+F75TSx++++TQ/-Tmp-/tmpXn1lw9]
Elapsed time: 4.28s (0.07 minutes)
Average run-time so far: 4.28s (0.07 minutes)
Estimated run-time left: 25.68s (0.43 minutes)

...etc.

This tells us that we've got one constraint turned on (+ constrain taxon=node1 min_age=1.8 max_age=2.5;)' in this sub-analysis, as well as various other run-time statistics. ....and now we wait for this to finish. It could take some time..

When the analysis has finished, we use the program r8spermute_results.py to parse the results:

PLAIN TEXT

CODE:

python r8spermute_results.py example/

This will loop over the log files and extract the relevant information. You probably want to output this information to a file, so run it like this:

PLAIN TEXT

CODE:

python r8spermute_results.py example/> results.txt

Now, we can look at the results, which will look something like this:

PLAIN TEXT

CODE:

node1 node2 Simon node1 node2
0 0 1 15.22 2.84
0 1 0 54.60 1.30
0 1 1 49.45 1.30
1 0 0 2.50 4.38
1 0 1 2.50 4.18
1 1 0 2.50 1.30
1 1 1 2.50 1.30

It's probably easiest to load this into a spreadsheet. Each row is an analysis. The first 3 columns (containing 1's and 0's) are our constraints and whether they're turned on (=1) or off (=0) in the analysis. The last two columns "node1" and "node2" are our estimated ages of those nodes, under the calibrations.

So, the first line shows that node1 is estimated to be 15.22 years old and node2 is 2.84 years old when only the calibration called "Simon" is used.

The second line shows that when only node2 is calibrated, the estimate for node1 and node2 is 54.60 and 1.30 respectively.

In contrast, the very last line shows that when all calibrations are turned on, then the age estimates are 2.50 and 1.30 respectively.

References:

[1] Sanderson, M. J. 1997. A nonparametric approach to estimating divergence
times in the absence of rate constancy. Mol. Biol. Evol. 14:1218-1231.

[2] Sanderson, M. J. 2002. Estimating absolute rates of molecular evolution and
divergence times: a penalized likelihood approach. Mol. Biol. Evol.
19:101-109.

[3] Gray, R.D., Drummond, A.J., & Greenhill, S.J. (2009) Language Phylogenies
Reveal Expansion Pulses and Pauses in Pacific Settlement. Science, 323:
479-483.

genbank-download

Simon — Thu, 08 Oct 2009 00:42:42 +0000

genbank-download is a small script to download nucleotide sequences from genbank using an accession number.

Usage:

python genbankdownload.py [options] ACCESSION-NUMBER
Example Usage
python genbankdownload.py J01415
python genbankdownload.py J01415 > mysequence.xml
python genbankdownload.py -m fasta J01415 > mysequence.fasta
The program source-code is available here or you can run:
easy_install genbank-download

Simple webserver file alteration monitoring using integrit

Simon — Wed, 06 Jun 2007 06:06:32 +0000

Intrusion detection?

Over on Mezzoblue, Dave Shea found out that his website had been compromised subtly. The attacker had exploited some (as yet unknown) security hole and quietly modified his website to link to the standard spam sites.

Within a few hours there were tens of posts from people who'd checked their websites and found similar modifications that had been sitting their un-noticed, with people pointing the finger either at old WordPress installations or a guesses that their hosting service had been compromised.

This shows us once again, that any software you run on your website needs to be kept up-to-date immediately, but what shocked me was that so many people out there running websites and are not watching them for file changes. They had no idea that their sites had been hacked until they went and looked for it.

So - in bold: Anyone running a website or webserver of any type needs to watch out for unexpected access and changes.

The easiest way to do this is to use some intrusion detection software (IDS). This sounds complex, but it's actually quite easy to do. All these programs do is to monitor your files and warn you when they change. This would have immediately spotted this type of attack.

Because of this, I've decided to write up an easy guide to simple file alteration monitoring - here it is.

Choose your weapon:

There are plenty of intrusion detection/file modification apps out there - some of the better known ones include AIDE, Samhain and Tripwire. These are all very cool, and highly powerful, but are also quite complex and hard to install, especially on cheap shared hosting.

Therefore, my weapon of choice, for the last few months has been a lightweight and fast application called integrit, so I'm going to tell you how to install it here.

Before we start: Do make sure that you're not compromised right now, there's no point running an IDS if you're already hacked. While you're at it, make sure everything's upgraded too.

Step 1: Make a place to store integrit:

Since you're on shared hosting, you can't install integrit properly into /usr, but you need to to put it somewhere anyway.

I decided to install it to a directory called "integrit" inside my home dir, so:

PLAIN TEXT

CODE:

mkdir ~/integrit

In the following commands, do remember to replace any mention of ~/integrit with the directory you used.

Step 2: Download and install integrit:

The integrit webpage is at: http://integrit.sourceforge.net/, so go there and get the latest version (currently 4.1), or you could cut'n' paste this:

PLAIN TEXT

CODE:

cd ~/integrit
wget http://optusnet.dl.sourceforge.net/sourceforge/integrit/integrit-4.1.tar.gz

Once the integrit archive file is in your ~/integrit directory, we need to decompress it and install it:

PLAIN TEXT

CODE:

cd ~/integrit
tar -zxvf integrit-4.1.tar.gz
cd integrit-4.1
./configure
make

Next we need to put the integrit binary somewhere where we can get it, here I've just dumped it into the ~/integrit
directory, but you could put it in ~/bin or something nicer if you want:

PLAIN TEXT

CODE:

cp integrit ~/integrit

Step 3: Set up integrit:

Look in the integrit-4.1/examples directory and make a config file from the example.

You need three things at least:

The known file database - this is where the integrit database is stored.
The current file database - this is where integrit stores the modified info.
A root directory to monitor - this is the full path to the directory we want to watch.

We can also tell integrit to ignore directories, by listing with an exclamation mark at the start of the line. We want to ignore the ~/integrit directory, and on dreamhost, we'll need to ignore the webserver log directory (because it changes a lot, and parts of it our user can't access which will cause errors).

All in all, it'll look something like this:

PLAIN TEXT

CODE:

# database locations (FULL PATHS!)
known=/home/simon/integrit/src_known.cdb
current=/home/simon/integrit/src_current.cdb
# What do we want to check (no trailing slash!)
root=/home/simon
# ignore the integrit dir:
!/home/simon/integrit
# ignore the webserver logs dir:
!/home/simon/logs
# oh, and the bash logfile
!/home/simon/.bash_history

Note: You'll need to change "simon" to your user name, and "integrit" to where you installed integrit to in step 2.

Step 4: Create integrit database

We need to get integrit to store a list of the current files, and their vital statistics, so run this command:

PLAIN TEXT

CODE:

~/integrit/integrit -C home.conf -u

...where "home.conf" is the integrit configuration file that you generated in step 3.

If all goes well, you'll see something like this:

PLAIN TEXT

CODE:

integrit: ---- integrit, version 4.1 -----------------
integrit: output : human-readable
integrit: conf file : home.conf
integrit: known db : /home/simon/integrit/src_known.cdb
integrit: current db : /home/simon/integrit/src_current.cdb
integrit: root : /home/simon
integrit: do check : no
integrit: do update : yes
integrit: current-state db RMD160 --------------
integrit: 3d6b135343a5031d357b5bb2d7d7dc39c7ab5646 /home/simon8/integrit/src_current.cdb

Once that's done, copy the newly created database to the known database location:

PLAIN TEXT

CODE:

cp ~/integrit/src_current.cdb ~/integrit/src_known.cdb

Step 5: Test that integrit's working:

So lets make sure that integrit's working properly. To do this, we can add an empty file somewhere and see if integrit spots it.

PLAIN TEXT

CODE:

cd ~
touch foo

Now we can run integrit:

PLAIN TEXT

CODE:

~/integrit/integrit -C ~/integrit/home.conf -c

If all goes well, you'll see something like this:

PLAIN TEXT

CODE:

integrit: ---- integrit, version 4.1 -----------------
integrit: output : human-readable
integrit: conf file : /home/simon8/integrit/home.conf
integrit: known db : /home/simon8/integrit/src_known.cdb
integrit: current db : /home/simon8/integrit/src_current.cdb
integrit: root : /home/simon8
integrit: do check : yes
integrit: do update : no
new: /home/simon8/foo p(664) t(100000) u(767504) g(203016) z(0) m(20070605-162156)
new: /home/simon8/foo s(9c1185a5c5e9fc54612808977ee8f548b2258d31)
integrit: not doing update, so no check for missing files

Notice how integrit's spotted the foo file that's not in the database? If one of the files has changed, you'll get much the same output, with "changed:" instead of "new:". So - remove the dummy file:

PLAIN TEXT

CODE:

rm ~/foo

Step 7: Get integrit to run daily:

Now, we want to set up a cron job, so that integrit is run automatically for us. First of all, we should make a quick little shell script to run integrit and email the results to us:

PLAIN TEXT

CODE:

#!/bin/bash
DATE=`/bin/date "+%F"`
/home/simon/integrit/integrit -C /home/simon/integrit/home.conf -cu | /usr/bin/mutt -s "integrit - $DATE" email@example.com

Paste the above into a file called "run_integrit.sh" (a good place to put it would be in your ~/integrit directory), edit the paths to match your setup, and change the email address. Finally, make this file executable:

PLAIN TEXT

CODE:

chmod +x ~/integrit/run_integrit.sh

Now all we have to do is to add that to our crontab.

PLAIN TEXT

CODE:

crontab -e

...and add a line that looks something like this:

PLAIN TEXT

CODE:

59 21 * * * /home/simon/integrit/run_integrit.sh

This will run integrit at 21.59 every day, if you don't know what that means, then have a google for "crontab tutorial".

Save the file, and you're off.

Updating the database after valid changes:

When you've changed or added a file yourself, then you'll need to update your known database with these changes. To do this, just generate a current database, and copy it over the old one. The script I've got above will automatically generate a current one, so you can just use that version, or repeat Step 4.

Final considerations:

Note: If you can, you should run integrit (that is BOTH the database files and the binary files) off a "safe" partition, that's not writable. Unfortunately, most of us on shared hosting don't have that privilege, so just be aware that if a really clever attacker does get you, then they're likely to disable or modify the IDS if they can.

A good way of dealing with this is to copy your known file database off the webserver and make sure that the one on the server matches this one every so often.

--Simon

Query PubMed for citation information using a DOI and Python

Simon — Mon, 29 Jan 2007 14:01:44 +0000

Here's a simple little script to query PubMed for a Digitial Object Identifier (a DOI)

Usage is quite simple, find a DOI somewhere, e.g. 10.1038/nature02029 (for this groundbreaking paper), and run this:

PLAIN TEXT

CODE:

lurch:~ python pythonquery.py 10.1038/nature02029

... and via the magic of webservices and XML, and with a bit of luck, you'll get something like this back:

PLAIN TEXT

CODE:

Language-tree divergence times support the Anatolian theory of Indo-European origin.
Gray, RD, Atkinson, QD
Nature 2003, 426 (6965):435-9
Languages, like genes, provide vital clues about human history. The origin of
the Indo-European language family is "the most intensively studied, yet still
most recalcitrant, problem of historical linguistics". Numerous genetic studies
of Indo-European origins have also produced inconclusive results. Here we
analyse linguistic data using computational methods derived from evolutionary
biology. We test two theories of Indo-European origin: the 'Kurgan expansion'
and the 'Anatolian farming' hypotheses. The Kurgan theory centres on possible
archaeological evidence for an expansion into Europe and the Near East by
Kurgan horsemen beginning in the sixth millennium BP. In contrast, the Anatolian
theory claims that Indo-European languages expanded with the spread of
agriculture from Anatolia around 8,000-9,500 years bp. In striking agreement
with the Anatolian hypothesis, our analysis of a matrix of 87 languages with
2,449 lexical items produced an estimated age range for the initial Indo-European
divergence of between 7,800 and 9,800 years bp. These results were robust to
changes in coding procedures, calibration points, rooting of the trees and priors
in the bayesian analysis.

The Code:

PLAIN TEXT

PYTHON:

#!/usr/bin/env python# Simple script to query pubmed for a DOI
# (c) Simon Greenhill, 2007
# http://simon.net.nz/
import urllib
from xml.dom import minidom
def get_citation_from_doi(query, email='YOUR EMAIL GOES HERE', tool='SimonsPythonQuery', database='pubmed'):
params = {
'db':database,
'tool':tool,
'email':email,
'term':query,
'usehistory':'y',
'retmax':1
}
# try to resolve the PubMed ID of the DOI
url = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?' + urllib.urlencode(params)
data = urllib.urlopen(url).read()
# parse XML output from PubMed...
xmldoc = minidom.parseString(data)
ids = xmldoc.getElementsByTagName('Id')
# nothing found, exit
if len(ids) == 0:
raise Exception("DoiNotFound")
# get ID
id = ids[0].childNodes[0].data
# remove unwanted parameters
params.pop('term')
params.pop('usehistory')
params.pop('retmax')
# and add new ones...
params['id'] = id
params['retmode'] = 'xml'
# get citation info:
url = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?' + urllib.urlencode(params)
data = urllib.urlopen(url).read()
return data
def text_output(xml):
"""Makes a simple text output from the XML returned from efetch"""
xmldoc = minidom.parseString(xml)
title = xmldoc.getElementsByTagName('ArticleTitle')[0]
title = title.childNodes[0].data
abstract = xmldoc.getElementsByTagName('AbstractText')[0]
abstract = abstract.childNodes[0].data
authors = xmldoc.getElementsByTagName('AuthorList')[0]
authors = authors.getElementsByTagName('Author')
authorlist = []
for author in authors:
LastName = author.getElementsByTagName('LastName')[0].childNodes[0].data
Initials = author.getElementsByTagName('Initials')[0].childNodes[0].data
author = '%s, %s' % (LastName, Initials)
authorlist.append(author)
journalinfo = xmldoc.getElementsByTagName('Journal')[0]
journal = journalinfo.getElementsByTagName('Title')[0].childNodes[0].data
journalinfo = journalinfo.getElementsByTagName('JournalIssue')[0]
volume = journalinfo.getElementsByTagName('Volume')[0].childNodes[0].data
issue = journalinfo.getElementsByTagName('Issue')[0].childNodes[0].data
year = journalinfo.getElementsByTagName('Year')[0].childNodes[0].data
# this is a bit odd?
pages = xmldoc.getElementsByTagName('MedlinePgn')[0].childNodes[0].data
output = []
output.append(title)
output.append('') #empty line
output.append(', '.join(authorlist))
output.append( '%s %s, %s (%s):%s' % (journal, year, volume, issue, pages) )
output.append('') #empty line
output.append(abstract)
return output
if __name__ == '__main__':
from sys import argv, exit
if len(argv) == 1:
print 'Usage: %s ' % argv[0]
print ' e.g. %s 10.1038/ng1946' % argv[0]
exit()
citation = get_citation_from_doi(argv[1])
for line in text_output(citation):
print line

--Simon

Simple FAQ application for Django

Simon — Sat, 30 Dec 2006 22:32:28 +0000

This is an exceedingly simple FAQ application for Django, which I wrote for Henry.

Do what you want with it, & let me know of any bugs etc.

The Model

To store the FAQs!

PLAIN TEXT

PYTHON:

from django.db import models
from django.conf import settings
# Create your models here.
class Faq( models.Model ):
"""Stores FAQS"""
question = models.CharField(maxlength=255) # The Q
answer = models.TextField() # the A
post_date = models.DateTimeField(auto_now_add=True, db_index=True) # when we answered it
def __str__( self ):
return self.question
class Admin:
search_fields = ( 'question', 'answer' )
date_hierarchy = 'post_date'
list_display = ( 'question', 'post_date' )
fields = ((None, { 'fields': ( 'question', 'answer' )}),)

The View:

To get the FAQs...

PLAIN TEXT

PYTHON:

from django.shortcuts import render_to_response
from django.template import RequestContext
# Change "myproject" to your project name
from myproject.about.models import *
def about(request):
"""Displays all the FAQs"""
faqs = Faq.objects.all()
return render_to_response( 'about.html', { 'faqs':faqs, },
context_instance=RequestContext(request) )

The Template:

...and finally, the template to bring it all together:

PLAIN TEXT

HTML:

{% extends "base.html" %}
{% load markup %}
{% block content %}
Frequently Asked Questions:
{% comment %}
build a "table of contents first..."
{% endcomment %}
href="http://simon.net.nz/wp-admin/#%7B%7B%20faq.id%20%7D%7D">{{ faq.question|escape }}

{% endfor %}

{% comment %}

now show the FAQs

{% endcomment %}

{% for faq in faqs %}

 title="{{ faq.id }}" name="{{ faq.id }}">

 class="q"> {{ faq.question|escape }}

 class="faq"> {{ faq.answer|textile }}

    {% endfor %}

{% endblock %}

The End:

You couldn't get any simpler than that without using a textfile..

Note: this requires textile - but you can easily change the filter in the template to whatever you want. You will need to have django.contrib.markup installed though.

Download the Django FAQ Application

--Simon

Django Database Export, v.00001

Simon — Sat, 04 Nov 2006 10:11:26 +0000

Here's something I wrote to export database contents from a django application into the Django database API format.

It's certainly not going to do all the work for you - in particular you'll need to reorder things to make sure things that Django wants to be objects (i.e. Foreign Keys) are instantiated before the related object gets made.

Hope this helps someone somewhere.

PLAIN TEXT

PYTHON:

#!/usr/bin/env python
"""
Script to export data from a Django database and generate
the appropriate django DB-API commands
Make sure you run it from the same directory as your settings.py
NOTE: it does NOT handle relationships yet, so, you'll need to take the output
and hand edit it to make sure that anything that Django needs as an object,
gets instantiated first
(c) Simon Greenhill / dev@simon.net.nz / http://simon.net.nz
Version: 0.00001
"""
debug = False # just a debug flag to spit out what we're doing.
import sys
from django.core.management import setup_environ
try:
import settings
except ImportError:
print "You don't appear to have a settings file in this directory!"
print "Please run this from inside a project directory"
sys.exit()
setup_environ(settings)
from django.db import models
# do model imports
print """
###############################################################
# #
# Model imports #
# #
###############################################################
"""
for app in models.get_apps():
for model in app.models.get_models():
print "from %s import %s" % ( app.__name__, model._meta.object_name )
# what apps we've done
seen = []
for app in models.get_apps():
for model in app.models.get_models():
if model not in seen:
print
print "# %s.%s " % (app.__name__, model._meta.object_name)
print
if debug:
print "#\t", model
for row in model.objects.iterator():
if debug:
print "#\t\t",row
attrs = []
for col in row._meta.fields:
key = col.name
value = getattr(row, key)
if col.get_internal_type() == 'ForeignKey':
###print key, 'bork bork bork'
pass
# if it's numeric, we don't need to quote it.
if isinstance(value,int) or isinstance(value,float):
pass
# if it's a string...
elif isinstance(value,str):
# ... and it's got a quote mark or new line in it, then triple-quote it.
if value.find('"')> 0 or value.find("\n")> 0:
value = '"""%s"""' % value
# ... otherwise single quote it.
else:
value = '"%s"' % value
# don't know what it is, but single quote it
else:
value = '"%s"' % value
attrs.append( '%s=%s' % ( key, value ) )
if debug:
print "#\t\t\t", key, getattr(row, key)
print """%s(%s).save()""" % ( model._meta.object_name, ', '.join(attrs) )
seen.append(model)

Update 19/01/2007: limodou has written a far superior export solution here.

--Simon

mnemosyne – a personal information wiki

Simon — Thu, 19 Oct 2006 07:12:35 +0000

I've just released my personal wiki system mnemosyne to the public. It runs under Django, written in Python and I've released it under the BSD licence, check it out at mnemosyne.simon.net.nz

Features:

internal and external linking
page versioning
edit history tracking
backlinks
users (sort of - in progress),
page tagging
bibtex handling
a pretentious name
'sensible' file attachment handling
'sensible' data storage
full search capabilities
swappable wiki parsers (textile, reStructured Text, plain text and
should be easy to add others)
macros for extra functionality

This is the first major app. I've tackled in both python and Django, and I would love to hear comments / criticisms / suggestions. Be brutal!

--Simon

benchmarking .htaccess performance

Simon — Sun, 27 Aug 2006 11:37:07 +0000

Introduction:

One of the common tips to increasing Apache performance is to turn off the per-directory configuration files (aka .htaccess files) and merge them all into your main Apache server configuration file (httpd.conf).

Jeremy raised an interesting question about when the performance loss caused by using many htaccess files is offset by the ease of maintenance. He's arguing - and I agree - that it makes sense to keep the configuration locally inside .htaccess files, despite the performance loss as these are easier to maintain.

It's fairly logical that the multiple .htaccess file route will be slower - for every node in the request URI, the webserver has to look for an .htaccess file and merge the rules found in every one. So, we're going to have to have a filesystem seek'n'read for every subdirectory.

However, is this a major issue? How much of a performance hit is there? Let's find out...

Set-up:

Ok. Let's make two docroots each with the same structure and files.

1) htdocs_access - the .htaccess version. This has one .htaccess file in the leaf directory.

2) htdocs_config - the httpd.conf version. This has the same rule as the above, but the rule is in the server-wide httpd.conf file and htaccess support is turned OFF (AllowOverride None).

Next, we need to get the .htaccess/httpd.conf files to do something ( mainly so we can see if Apache's merged them in ). So, we'll make a number of files in the last random directory (the leaf node), and give half of them the extension .foo, and the other half .bar. We'll then tell Apache to process the .bar's with PHP, and the .foo's as text. All files will have the same content:

PLAIN TEXT

PHP:

Here's the (python) code I used to generate this structure:

PLAIN TEXT

PYTHON:

#!/usr/bin/env python
import os
# where we'll place the generated structure
staging = '/Users/simon/server'
htdocs_access = os.path.join(staging, 'htdocs_access')
htdocs_config = os.path.join(staging, 'htdocs_config')
# how deep to go!
dir_depth = 10
# how many files in the leaf node of the dir.
num_files = 50
# what content to put in the files
content = ""
# the actual htaccess file
htaccess = """
AddHandler application/x-httpd-php .bar
"""
# make directory structure
dir = ''
for dirnum in range( 0, dir_depth ):
dir = os.path.join( dir, str( dirnum ) )
hta = os.path.join( htdocs_access, dir )
htc = os.path.join( htdocs_config, dir )
os.makedirs( hta )
os.makedirs( htc )
# make the files...
for filenum in range( 0, num_files ):
# assign the file types - half .foo, and half .bar
if filenum % 2 == 0:
filename = '%d.foo' % filenum
else:
filename = '%d.bar' % filenum
f = open( os.path.join( hta, filename ), 'w+' )
f.write( content )
f.close()
f = open( os.path.join( htc, filename ), 'w+' )
f.write( content )
f.close()
# now, add the .htaccess file inside the lead htdocs_access dir
f = open( os.path.join( hta, '.htaccess' ), 'w+' )
f.write( htaccess )
f.close()
# and we'll place it in the root of the htdocs_config dir as
# httpd.conf to remind ourselves to add it to the httpd.conf file
f = open( os.path.join( htdocs_config, 'httpd.conf' ), 'w+' )
f.write( htaccess )
f.close()

Here's what we end up with:

PLAIN TEXT

CODE:

0/
1/
2/
3/
4/
5/
6/
7/
8/
9/
0.foo
1.bar
10.foo
11.bar
(...etc...)
6.foo
7.bar
8.foo
9.bar

Where htdocs_access has a .htaccess file in 9/ and htdocs_config doesn't.

Server Configuration:

Here are the two httpd.conf files for the configurations:

htdocs_config httpd.conf:

PLAIN TEXT

CODE:

### Section 1: Global Environment
ServerRoot "/usr/local/apache2"
PidFile logs/httpd.pid
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
DirectoryIndex index.html
AccessFileName .htaccess
HostnameLookups Off
# fixes crashes on OSX??
AcceptMutex fcntl
StartServers 5
MinSpareServers 5
MaxSpareServers 5
MaxClients 100
MaxRequestsPerChild 10
### Section 2: 'Main' server configuration
User nobody
Group #-1
DocumentRoot "/Users/simon/server/htdocs_config"
LoadModule php5_module modules/libphp5.so
Listen 8111
Options Indexes FollowSymLinks
AllowOverride None
AddHandler application/x-httpd-php .bar

htdocs_access httpd.conf:

PLAIN TEXT

CODE:

### Section 1: Global Environment
ServerRoot "/usr/local/apache2"
PidFile logs/httpd.pid
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
DirectoryIndex index.html
AccessFileName .htaccess
HostnameLookups Off
# fixes crashes on OSX??
AcceptMutex fcntl
StartServers 5
MinSpareServers 5
MaxSpareServers 5
MaxClients 100
MaxRequestsPerChild 10
### Section 2: 'Main' server configuration
User nobody
Group #-1
DocumentRoot "/Users/simon/server/htdocs_access"
LoadModule php5_module modules/libphp5.so
Listen 8111
Options Indexes FollowSymLinks
AllowOverride All

Results:

Benchmarking was done with "ab" the Apache Benchmark program, which was set to access one page 1,000 times with 10 concurrencies. Each configuration was benchmarked five times in random order (to minimise the effect of any running background processes etc).

	htdocs_config		htdocs_access
Test:	Time Taken (s):	Requests per Second:	Time Taken (s):	Requests per Second:
1	12.683213	78.84	13.21618	75.66
2	12.854491	77.79	13.574916	73.67
3	11.777676	84.91	13.163296	75.97
4	13.668398	73.16	12.26475	81.53
5	13.76753	76.47	13.264527	75.39
AVERAGE:	12.9	78.23	13.1	76.4

So - we're looking at a difference of around 2.3% extra requests per second when htaccess files are disabled. This is really quite trivial, and should only be worried about when you're really loaded.

Issues:

There are a number of areas where this could be improved:

Try different directory depths i.e. the more nested the directory is, the slower it should be under the .htaccess scenario. In contrast, if there's only 2 or 3 levels then it should be faster.
Have multiple .htaccess files in the intermediate nodes to see how Apache handles the merging of these files. Here we've just used one .htaccess file, and we should probably see further slowdowns if Apache has to merge some complicated rule sets.
Access different files - I just requested one file repeatedly, so we might be getting a lot of interference from any caching systems (harddrive, ram, php caches etc) that I forgot about. Additionally, requesting multiple URI's is a more realistic test case for a webserver.

Processing mod_security audit logs with logwatch

Simon — Sat, 26 Aug 2006 07:40:56 +0000

This is a quick and dirty LogWatch script to process mod_security audit logs. I had a google, and couldn't find any released scripts, so here's my one.

PLAIN TEXT

PERL:

#!/usr/bin/perl -w
my $Detail = $ENV{'LOGWATCH_DETAIL_LEVEL'} || 0;
while (defined($ThisLine = )) {
chomp($ThisLine);
if ( $ThisLine =~ m/^Request:/ )
{
print $ThisLine."\n";
}
if( $ThisLine =~ m/^mod_security-message/ )
{
print $ThisLine."\n";
print "\n";
}
}
exit(0);

On Debian, you should just be able to drop the above script into /usr/share/logwatch/scripts/services (chmod +x too!), and then add this to /etc/logwatch/conf/logfiles/mod_security.conf:

PLAIN TEXT

CODE:

########################################################
# Define log file group for mod_security
########################################################
# Where the log files are - you will need to match this to your configuration
# Note that a relative path like this will be expanded automatically to the
# standard log path (e.g. /var/log/... )
LogFile = apache2/audit.log.1
LogFile = apache2/audit.log
# If the archives are searched, here is one or more line
# (optionally containing wildcards) that tell where they are...
# Note: if these are gzipped, you need to end with a .gz even if you use wildcards...
Archive = apache2/*audit.log.*.gz

--Simon

simon.net.nz » code

python-nexus – a generic nexus (.nex, .trees) reader for python

python-nexus - a generic nexus (.nex, .trees) reader for python

© Simon J. Greenhill, 2009

Download:

Usage:

Reading a Nexus

Writing a Nexus File

r8spermute

r8spermute - Runs all permutations of r8s constraints.

© Simon J. Greenhill, 2009

Download:

Usage:

References:

genbank-download

Simple webserver file alteration monitoring using integrit

Intrusion detection?

Choose your weapon:

Step 1: Make a place to store integrit:

Step 2: Download and install integrit:

Step 3: Set up integrit:

Step 4: Create integrit database

Step 5: Test that integrit's working:

Step 7: Get integrit to run daily:

Updating the database after valid changes:

Final considerations:

Query PubMed for citation information using a DOI and Python

The Code:

Simple FAQ application for Django

The Model

The View:

The Template:

Frequently Asked Questions:

The End:

Django Database Export, v.00001

mnemosyne – a personal information wiki

Features:

benchmarking .htaccess performance

Introduction:

Set-up:

Server Configuration:

htdocs_config httpd.conf:

htdocs_access httpd.conf:

Results:

Issues:

Processing mod_security audit logs with logwatch