simon.net.nz » python http://simon.net.nz Dr. Simon J. Greenhill's website Tue, 29 Nov 2011 09:49:47 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 r8spermutehttp://simon.net.nz/articles/r8spermute/ http://simon.net.nz/articles/r8spermute/#comments Wed, 14 Oct 2009 02:09:35 +0000 Simon http://simon.net.nz/?p=106 r8spermute - Runs all permutations of r8s constraints.

© Simon J. Greenhill, 2009

This is a small program to assess how robust divergence times are based on the calibrations given to the program r8s.

The program r8s by Mike Sanderson (http://loco.biosci.arizona.edu/r8s/) [1, 2] performs a number of phylogenetic dating methods on a set of phylogenetic trees. In essence, you plug in a number of calibration points from known historical evidence and the program r8s takes your estimated trees, and "smoothes" the rates of change observed on those trees, using the calibrations to convert the branches into time.

However, it is often useful to explore the effect that certain calibration has on the various date estimates. For example, in the paper [3], I wanted to estimate the age at which a large language family originated. We had about 10 different calibration points based on historical evidence. Some of these calibrations were, however, more controversial than the others. So, I wrote this program to take a tree (or set of trees) and analyse them under all the combinations of these calibrations. This allowed us to assess the relative effects on the date estimates of the different calibrations.

If you find this program useful, please cite reference [3].

Contact me if you have any problems.

Download:

You can download r8spermute from the bitbucket repository here: download r8spermute

Usage:

You will need two things: 1) A set of trees, 2) A r8s command block. Examples of both of these are provided in the "examples" directory.

Your r8s block file will look something like this:

PLAIN TEXT
CODE:
  1. begin rates;
  2. blformat lengths=persite nsites=1000 ultrametric=no round=yes;
  3. collapse;
  4.  
  5. mrca node1 Daisy Fiona;
  6. constrain taxon=node1 min_age=1.8 max_age=2.5;
  7.  
  8. mrca node2 Robert Tamara;
  9. constrain taxon=node2 min_age=1.1 max_age=1.3;
  10.  
  11. unfixage taxon=Simon;
  12. constrain taxon=Simon min_age=0.7 max_age=1.2;
  13.  
  14. set num_restarts=5;
  15. set smoothing=10;
  16.  
  17. divtime method=pl algorithm=tn;
  18. showage shownamed = yes;
  19.  
  20. profile taxon=node1 parameter=age;
  21. profile taxon=node2 parameter=age;
  22. describe plot=chrono_description;
  23. end;

... thus, we have 3 calibrations - node1, node2, and a constraint on the terminal taxon "Simon".

We can then run r8spermute like this:

PLAIN TEXT
CODE:
  1. r8spermute.py example/example.trees example/example.r8s

This will start r8s running and you should then see some output like this:

PLAIN TEXT
CODE:
  1. Constraints found:  3
  2. Constraints ignored:  0
  3. Total Constraint Combinations:  7
  4.  
  5. Running analysis 1 of 7 (0.14%)
  6. Staging in:  /var/folders/9Z/9Z3O2T2o2RmgRU+F75TSx++++TQ/-Tmp-/tmpXn1lw9
  7. Constraints:
  8. +  constrain taxon=node1 min_age=1.8 max_age=2.5;
  9. Logging in: example/example_100.log
  10. r8s version 1.71 (compiled May 16 2006)
  11. [...reading file /var/folders/9Z/9Z3O2T2o2RmgRU+F75TSx++++TQ/-Tmp-/tmpXn1lw9]
  12. Elapsed time: 4.28s (0.07 minutes)
  13. Average run-time so far: 4.28s (0.07 minutes)
  14. Estimated run-time left: 25.68s (0.43 minutes)

...etc.

This tells us that we've got one constraint turned on (+ constrain taxon=node1 min_age=1.8 max_age=2.5;)' in this sub-analysis, as well as various other run-time statistics. ....and now we wait for this to finish. It could take some time..

When the analysis has finished, we use the program r8spermute_results.py to parse the results:

PLAIN TEXT
CODE:
  1. python r8spermute_results.py example/

This will loop over the log files and extract the relevant information. You probably want to output this information to a file, so run it like this:

PLAIN TEXT
CODE:
  1. python r8spermute_results.py example/> results.txt

Now, we can look at the results, which will look something like this:

PLAIN TEXT
CODE:
  1. node1 node2 Simon node1 node2
  2. 0 0 1 15.22 2.84
  3. 0 1 0 54.60 1.30
  4. 0 1 1 49.45 1.30
  5. 1 0 0 2.50 4.38
  6. 1 0 1 2.50 4.18
  7. 1 1 0 2.50 1.30
  8. 1 1 1 2.50 1.30

It's probably easiest to load this into a spreadsheet. Each row is an analysis. The first 3 columns (containing 1's and 0's) are our constraints and whether they're turned on (=1) or off (=0) in the analysis. The last two columns "node1" and "node2" are our estimated ages of those nodes, under the calibrations.

So, the first line shows that node1 is estimated to be 15.22 years old and node2 is 2.84 years old when only the calibration called "Simon" is used.

The second line shows that when only node2 is calibrated, the estimate for node1 and node2 is 54.60 and 1.30 respectively.

In contrast, the very last line shows that when all calibrations are turned on, then the age estimates are 2.50 and 1.30 respectively.

References:

[1] Sanderson, M. J. 1997. A nonparametric approach to estimating divergence
times in the absence of rate constancy. Mol. Biol. Evol. 14:1218-1231.

[2] Sanderson, M. J. 2002. Estimating absolute rates of molecular evolution and
divergence times: a penalized likelihood approach. Mol. Biol. Evol.
19:101-109.

[3] Gray, R.D., Drummond, A.J., & Greenhill, S.J. (2009) Language Phylogenies
Reveal Expansion Pulses and Pauses in Pacific Settlement. Science, 323:
479-483.

]]> http://simon.net.nz/articles/r8spermute/feed/ 0 Query PubMed for citation information using a DOI and Pythonhttp://simon.net.nz/articles/query-pubmed-for-citation-information-using-a-doi-and-python/ http://simon.net.nz/articles/query-pubmed-for-citation-information-using-a-doi-and-python/#comments Mon, 29 Jan 2007 14:01:44 +0000 Simon http://simon.net.nz/articles/query-pubmed-for-citation-information-using-a-doi-and-python/ Here's a simple little script to query PubMed for a Digitial Object Identifier (a DOI)

Usage is quite simple, find a DOI somewhere, e.g. 10.1038/nature02029 (for this groundbreaking paper), and run this:

PLAIN TEXT
CODE:
  1. lurch:~ python pythonquery.py 10.1038/nature02029

... and via the magic of webservices and XML, and with a bit of luck, you'll get something like this back:

PLAIN TEXT
CODE:
  1. Language-tree divergence times support the Anatolian theory of Indo-European origin.
  2.  
  3. Gray, RD, Atkinson, QD
  4. Nature 2003, 426 (6965):435-9
  5.  
  6. Languages, like genes, provide vital clues about human history. The origin of
  7. the Indo-European language family is "the most intensively studied, yet still
  8. most recalcitrant, problem of historical linguistics". Numerous genetic studies
  9. of Indo-European origins have also produced inconclusive results. Here we
  10. analyse linguistic data using computational methods derived from evolutionary
  11. biology. We test two theories of Indo-European origin: the 'Kurgan expansion'
  12. and the 'Anatolian farming' hypotheses. The Kurgan theory centres on possible
  13. archaeological evidence for an expansion into Europe and the Near East by
  14. Kurgan horsemen beginning in the sixth millennium BP. In contrast, the Anatolian
  15. theory claims that Indo-European languages expanded with the spread of
  16. agriculture from Anatolia around 8,000-9,500 years bp. In striking agreement
  17. with the Anatolian hypothesis, our analysis of a matrix of 87 languages with
  18. 2,449 lexical items produced an estimated age range for the initial Indo-European
  19. divergence of between 7,800 and 9,800 years bp. These results were robust to
  20. changes in coding procedures, calibration points, rooting of the trees and priors
  21. in the bayesian analysis.

The Code:

PLAIN TEXT
PYTHON:
  1. #!/usr/bin/env python#   Simple script to query pubmed for a DOI
  2. #   (c) Simon Greenhill, 2007
  3. #   http://simon.net.nz/
  4.  
  5. import urllib
  6. from xml.dom import minidom
  7.  
  8. def get_citation_from_doi(query, email='YOUR EMAIL GOES HERE', tool='SimonsPythonQuery', database='pubmed'):
  9.     params = {
  10.         'db':database,
  11.         'tool':tool,
  12.         'email':email,
  13.         'term':query,
  14.         'usehistory':'y',
  15.         'retmax':1
  16.     }
  17.    
  18.     # try to resolve the PubMed ID of the DOI
  19.     url = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?' + urllib.urlencode(params)
  20.     data = urllib.urlopen(url).read()
  21.    
  22.     # parse XML output from PubMed...
  23.     xmldoc = minidom.parseString(data)
  24.     ids = xmldoc.getElementsByTagName('Id')
  25.    
  26.     # nothing found, exit
  27.     if len(ids) == 0:
  28.         raise Exception("DoiNotFound")
  29.  
  30.     # get ID
  31.     id = ids[0].childNodes[0].data
  32.  
  33.     # remove unwanted parameters
  34.     params.pop('term')
  35.     params.pop('usehistory')
  36.     params.pop('retmax')
  37.     # and add new ones...
  38.     params['id'] = id
  39.  
  40.     params['retmode'] = 'xml'
  41.  
  42.     # get citation info:
  43.     url = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?' + urllib.urlencode(params)
  44.     data = urllib.urlopen(url).read()
  45.  
  46.     return data
  47.  
  48.     def text_output(xml):
  49.         """Makes a simple text output from the XML returned from efetch"""
  50.  
  51.         xmldoc = minidom.parseString(xml)
  52.  
  53.         title = xmldoc.getElementsByTagName('ArticleTitle')[0]
  54.         title = title.childNodes[0].data
  55.  
  56.         abstract = xmldoc.getElementsByTagName('AbstractText')[0]
  57.         abstract = abstract.childNodes[0].data
  58.  
  59.         authors = xmldoc.getElementsByTagName('AuthorList')[0]
  60.         authors = authors.getElementsByTagName('Author')
  61.         authorlist = []
  62.         for author in authors:
  63.             LastName = author.getElementsByTagName('LastName')[0].childNodes[0].data
  64.             Initials = author.getElementsByTagName('Initials')[0].childNodes[0].data
  65.             author = '%s, %s' % (LastName, Initials)
  66.             authorlist.append(author)
  67.            
  68.         journalinfo = xmldoc.getElementsByTagName('Journal')[0]
  69.         journal = journalinfo.getElementsByTagName('Title')[0].childNodes[0].data
  70.         journalinfo = journalinfo.getElementsByTagName('JournalIssue')[0]
  71.         volume = journalinfo.getElementsByTagName('Volume')[0].childNodes[0].data
  72.         issue = journalinfo.getElementsByTagName('Issue')[0].childNodes[0].data
  73.         year = journalinfo.getElementsByTagName('Year')[0].childNodes[0].data
  74.  
  75.         # this is a bit odd?
  76.         pages = xmldoc.getElementsByTagName('MedlinePgn')[0].childNodes[0].data
  77.  
  78.         output = []
  79.         output.append(title)
  80.         output.append('') #empty line
  81.         output.append(', '.join(authorlist))
  82.         output.append( '%s %s, %s (%s):%s' % (journal, year, volume, issue, pages) )
  83.         output.append('') #empty line
  84.         output.append(abstract)
  85.         return output
  86.  
  87. if __name__ == '__main__':
  88.     from sys import argv, exit
  89.     if len(argv) == 1:
  90.         print 'Usage: %s <query>' % argv[0]
  91.         print ' e.g. %s 10.1038/ng1946' % argv[0]
  92.         exit()
  93.        
  94.         citation = get_citation_from_doi(argv[1])
  95.         for line in text_output(citation):
  96.             print line

--Simon

]]> http://simon.net.nz/articles/query-pubmed-for-citation-information-using-a-doi-and-python/feed/ 2 Simple FAQ application for Djangohttp://simon.net.nz/articles/simple-faq-application-for-django/ http://simon.net.nz/articles/simple-faq-application-for-django/#comments Sat, 30 Dec 2006 22:32:28 +0000 Simon http://simon.net.nz/articles/simple-faq-application-for-django/ This is an exceedingly simple FAQ application for Django, which I wrote for Henry.

Do what you want with it, & let me know of any bugs etc.

The Model

To store the FAQs!

PLAIN TEXT
PYTHON:
  1. from django.db import models
  2. from django.conf import settings
  3.  
  4. # Create your models here.
  5. class Faq( models.Model ):
  6. """Stores FAQS"""
  7. question = models.CharField(maxlength=255) # The Q
  8. answer = models.TextField() # the A
  9. post_date = models.DateTimeField(auto_now_add=True, db_index=True) # when we answered it
  10.  
  11. def __str__( self ):
  12. return self.question
  13.  
  14. class Admin:
  15. search_fields = ( 'question', 'answer' )
  16. date_hierarchy = 'post_date'
  17. list_display = ( 'question', 'post_date' )
  18. fields = ((None, { 'fields': ( 'question', 'answer' )}),)

The View:

To get the FAQs...

PLAIN TEXT
PYTHON:
  1. from django.shortcuts import render_to_response
  2. from django.template import RequestContext
  3.  
  4. # Change "myproject" to your project name
  5. from myproject.about.models import *
  6.  
  7. def about(request):
  8. """Displays all the FAQs"""
  9. faqs = Faq.objects.all()
  10. return render_to_response( 'about.html', {  'faqs':faqs, },
  11. context_instance=RequestContext(request) )

The Template:

...and finally, the template to bring it all together:

PLAIN TEXT
HTML:
  1. {% extends "base.html" %}
  2. {% load markup %}
  3.  
  4. {% block content %}
  5. <h2>Frequently Asked Questions:</h2>
  6. {% comment %}
  7. build a "table of contents first..."
  8. {% endcomment %}
  9. <ul>         {% for faq in faqs %}
  10.     <li><a href="http://simon.net.nz/wp-admin/#%7B%7B%20faq.id%20%7D%7D">{{ faq.question|escape }}</a></li>
  11. {% endfor %}</ul>
  12. {% comment %}
  13. now show the FAQs
  14. {% endcomment %}
  15.  
  16. {% for faq in faqs %}
  17. <a title="{{ faq.id }}" name="{{ faq.id }}"></a>
  18. <h3 class="q"> {{ faq.question|escape }}</h3>
  19. <p class="faq"> {{ faq.answer|textile }}</p>
  20.     {% endfor %}
  21.  
  22. {% endblock %}

The End:

You couldn't get any simpler than that without using a textfile..

Note: this requires textile - but you can easily change the filter in the template to whatever you want. You will need to have django.contrib.markup installed though.

Download the Django FAQ Application

--Simon

]]> http://simon.net.nz/articles/simple-faq-application-for-django/feed/ 1 Django Database Export, v.00001http://simon.net.nz/articles/django-database-export-v00001/ http://simon.net.nz/articles/django-database-export-v00001/#comments Sat, 04 Nov 2006 10:11:26 +0000 Simon http://simon.net.nz/articles/django-database-export-v00001/ Here's something I wrote to export database contents from a django application into the Django database API format.

It's certainly not going to do all the work for you - in particular you'll need to reorder things to make sure things that Django wants to be objects (i.e. Foreign Keys) are instantiated before the related object gets made.

Hope this helps someone somewhere.

PLAIN TEXT
PYTHON:
  1. #!/usr/bin/env python
  2. """
  3. Script to export data from a Django database and generate
  4. the appropriate django DB-API commands
  6. Make sure you run it from the same directory as your settings.py
  8. NOTE: it does NOT handle relationships yet, so, you'll need to take the output
  9. and hand edit it to make sure that anything that Django needs as an object,
  10. gets instantiated first
  12. (c) Simon Greenhill / dev@simon.net.nz / http://simon.net.nz
  14. Version: 0.00001
  15. """
  16.  
  17. debug = False # just a debug flag to spit out what we're doing.
  18.  
  19. import sys
  20.  
  21. from django.core.management import setup_environ
  22.  
  23. try:
  24. import settings
  25. except ImportError:
  26. print "You don't appear to have a settings file in this directory!"
  27. print "Please run this from inside a project directory"
  28. sys.exit()
  29.  
  30. setup_environ(settings)
  31.  
  32. from django.db import models
  33.  
  34. # do model imports
  35. print """
  36. ###############################################################
  37. #                                                             #
  38. #   Model imports                                             #
  39. #                                                             #
  40. ###############################################################
  41. """
  42.  
  43. for app in models.get_apps():
  44. for model in app.models.get_models():
  45. print "from %s import %s" % ( app.__name__, model._meta.object_name )
  46.  
  47. # what apps we've done
  48. seen = []
  49. for app in models.get_apps():
  50. for model in app.models.get_models():
  51. if model not in seen:
  52. print
  53. print "# %s.%s " % (app.__name__, model._meta.object_name)
  54. print
  55.  
  56. if debug:
  57. print "#\t", model
  58.  
  59. for row in model.objects.iterator():
  60.  
  61. if debug:
  62. print "#\t\t",row
  63.  
  64. attrs = []
  65. for col in row._meta.fields:
  66. key = col.name
  67. value = getattr(row, key)
  68. if col.get_internal_type() == 'ForeignKey':
  69. ###print key, 'bork bork bork'
  70. pass
  71.  
  72. # if it's numeric, we don't need to quote it.
  73. if isinstance(value,int) or isinstance(value,float):
  74. pass
  75. # if it's a string...
  76. elif isinstance(value,str):
  77. # ... and it's got a quote mark or new line in it, then triple-quote it.
  78. if value.find('"')&gt; 0 or value.find("\n")&gt; 0:
  79. value = '"""%s"""' % value
  80. # ... otherwise single quote it.
  81. else:
  82. value = '"%s"' % value
  83. # don't know what it is, but single quote it
  84. else:
  85. value = '"%s"' % value
  86.  
  87. attrs.append( '%s=%s' % ( key, value ) )
  88.  
  89. if debug:
  90. print "#\t\t\t", key, getattr(row, key)
  91.  
  92. print """%s(%s).save()""" % ( model._meta.object_name, ', '.join(attrs) )
  93.  
  94. seen.append(model)

Update 19/01/2007: limodou has written a far superior export solution here.

--Simon

]]> http://simon.net.nz/articles/django-database-export-v00001/feed/ 0 mnemosyne – a personal information wikihttp://simon.net.nz/articles/mnemosyne/ http://simon.net.nz/articles/mnemosyne/#comments Thu, 19 Oct 2006 07:12:35 +0000 Simon http://simon.net.nz/articles/mnemosyne/ I've just released my personal wiki system mnemosyne to the public. It runs under Django, written in Python and I've released it under the BSD licence, check it out at mnemosyne.simon.net.nz

Features:

  • internal and external linking
  • page versioning
  • edit history tracking
  • backlinks
  • users (sort of - in progress),
  • page tagging
  • bibtex handling
  • a pretentious name
  • 'sensible' file attachment handling
  • 'sensible' data storage
  • full search capabilities
  • swappable wiki parsers (textile, reStructured Text, plain text and
    should be easy to add others)
  • macros for extra functionality

This is the first major app. I've tackled in both python and Django, and I would love to hear comments / criticisms / suggestions. Be brutal!

--Simon

]]> http://simon.net.nz/articles/mnemosyne/feed/ 0