Johan's Dev Blog: 2009

Referential Integrity in xml

In this post I will try to explain a very useful feature of xsd; the ability to detect referential integrity constraint violations. Let's start of with some simple xml :

<root>

  <type name="A"/>

  <type name="B"/>

  <type name="C"/>

  <item type="C"/>

  <item type="A"/>

</root>

And the xsd defining this xml :

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">

  <xsd:complexType name="type">

   <xsd:attribute name="name" type="xsd:string"/>

  </xsd:complexType>

  <xsd:complexType name="item">

   <xsd:attribute name="type" type="xsd:string"/>

  </xsd:complexType>

  <xsd:element name="root">

   <xsd:complexType>

   <xsd:sequence>

   <xsd:element name="type" minOccurs="0" maxOccurs="unbounded" type="type"/>

   <xsd:element name="item" minOccurs="0" maxOccurs="unbounded" type="item"/>

   </xsd:sequence>

   </xsd:complexType>

  </xsd:element>

</xsd:schema>

Again nothing new. At this stage, we have a definition in xsd format which can be used to test xml for conformance to our schema. However, any validation would be strictly structural. If we wanted to validate things like whether the items have type attributes which have been defined as type elements, or that there is only one type of element for each name, then we need something more than just the above xsd. In short, we need a definition which would encapsulate the fact that the following xml should be invalid (since the type D is not defined, and since type C is defined twice):

<root>

  <type name="A"/>

  <type name="B"/>

  <type name="C"/>

  <type name="C"/>

  <item type="D"/>

</root>

This can very simply be done by using the unique,key and keyref xsd elements. Here is the xsd after the required modifications:

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">

  <xsd:complexType name="type">

   <xsd:attribute name="name" type="xsd:string"/>

  </xsd:complexType>

  <xsd:complexType name="item">

   <xsd:attribute name="type" type="xsd:string"/>

  </xsd:complexType>

  <xsd:element name="root">

   <xsd:complexType>

   <xsd:sequence>

   <xsd:element name="type" minOccurs="0" maxOccurs="unbounded" type="type"/>

   <xsd:element name="item" minOccurs="0" maxOccurs="unbounded" type="item"/>

   </xsd:sequence>

   </xsd:complexType>

   <xsd:unique name="unique_type_constraint">

   <xsd:selector xpath="type"/>

   <xsd:field xpath="@name"/>

   </xsd:unique>

   <xsd:key name="typeKey">

   <xsd:selector xpath="type"/>

   <xsd:field xpath="@name"/>

   </xsd:key>

   <xsd:keyref name="type_constraint" refer="typeKey">

   <xsd:selector xpath="item"/>

   <xsd:field xpath="@type"/>

   </xsd:keyref>

  </xsd:element>

</xsd:schema>

What we are doing is defining a 'lookup' using the key element. The key is defined by selecting the name attribute from the type elements under the root element. This 'lookup' is then linked to the type attributes of the items under root via the keyref element. Thus, restricting the values of the type attributes of the item element to the values found int the name attributes of the type elements.

The unique element is used to apply a uniquenesses constraint over the values of name attributes of the type elements.

I have written a simple Python script to test the above concepts. This script can be found here. Please note that you will have to install lxml in order to run the script.

Writing a Python xinetd Server

Yesterday I discovered the xinetd; an internet services daemon. I immediately liked the idea of writing simple services that work on stdin and stdout but that can be accessed over the internet via sockets. So, I set out to write a simple Python server that can be integrated with xinetd. Here is the server:

#!/usr/bin/python
import sys

request = ''
while True:
  data = sys.stdin.readline().strip()
  request = request + data + '<br>'
  if data == "":
   print 'HTTP/1.0 200 OK'
   print ''
   print '<html><body><p>'+request+'</p></body></html>'
   sys.stdout.flush()
   break;

I am assuming that a web browser will connect to my server, the server will then 'echo' the request back to the browser allowing the browser to display the request. As you can see the input is received via stdin and output is returned via stdout.

If xinetd is not already installed then you will obviously have to install it first. Since I am doing this on Ubuntu the following works for me:

sudo apt-get install xinetd

After installing xinetd you need to create a config file for the service. I called my service http_echo and my config file (located in /etc/xinetd.d) is named similarly; http_echo. My configuration file looks like this:

service http_echo
{
   protocol = tcp
   disable = no
   port = 9991
   flags = REUSE
   socket_type = stream
   wait = no
   user = johan
   server = /home/johan/code/http_echo/http_echo
   log_on_failure += USERID
}

Most of this file is quite self explanatory. Please refer to the xinetd documentation for more information.The port property should make the service run on the specified port without having to add an entry in the services file (/etc/services) . I have had to add an entry in my services file to make this setup work:

http_echo 9991/tcp

Then simply restart the xinetd service:

sudo /etc/init.d/xinetd restart

Pointing a browser to the server on the specified port (9991), will yield the pleasing results below:

GET / HTTP/1.1
Host: localhost:9991
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.195.33 Safari/532.0
Cache-Control: max-age=0
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

And that is how simple it is to write a service in Python that runs under xinetd.

Simple Input Box Style

Whenever I browse the web I usually keep my eyes open for interesting styles or style components. Today I found a very nice looking input box; here. Here is my reproduction:

Please type here :

Here is the style for the above text box:

.simple
{
              border-width: 0 0 1px;
              border-style:none none solid;
              background:transparent;
              border-color:black;
}

Anyway, I like the look of this simple input box and will be using it more.

Convenient Python Command Runner

Here is a little piece of code which I tend to reuse quite often. It basically runs a command and return True if the command executed successfully or False if it did not. The second item in the return tuple is the stdout output of the command if any or the error if there was an error executing the command.

import subprocess

def runcmd(cmd):
try:
    f = subprocess.Popen(cmd,shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    stdout = f.stdout.read();
    stderr = f.stderr.read();

    if f.stdout.close() == None and f.stderr.close() == None:
      return True,stdout,stderr
    else:
      return False,stdout,stderr
except OSError,x:
    return False,x

As usual the source code can be found here.

Easily Center an Image in a Div

Who would have thought that this would be such a headache? After lots of reading on the web about how some browsers don't support this property and other browsers doesn't support that property I came up with a very simple solution that works perfectly for me in IE8, Firefox and Chrome. The solution is to create a table with one tr and one td inside the div. Then make the table the same size as the div by setting the width and height of the table to 100%. Then simply set the text-align:center style on the td.

<div>
<table width="100%" height="100%">
<tr><td style="text-align:center"><img src="apple.jpg"></img></td></tr>
</table>
</div>

Enjoy. Let me know if there is any issues with this solution.

Python SMTP Mail Sink

I've been using this little bit of python code quite extensively. It is basically a SMTP server built by extending the SMTPServer from the smtpd package (all standard with python). I then added a HTTP server frontend to this. This allows me to quickly start up a new SMTP server with a HTTP front end, which allows me to test mail functionality without having to log into a mail client etc. Initially when I wrote it it was because I was disconnected from the web and could not be bothered to install a full blown SMTP server on my laptop. This little application has probably seen more use than what I intended for it initially.

Have a look here for the script. Enjoy and have fun.

Django Subversion Repository Structure

The last couple of days I have been racking my brain to come up with an appropriate Subversion repository structure for my django development. My requirements are simple really:

A single repository must contain all of my django projects. This is required since I am hosting my development on google code, and they only provide you with one repository..
Each django application should be "versioned" independently. This allows me to release new versions of applications independently from django projects. It also allows different django projects to use various versions of the same application.
Each django project should be "versioned" independently.
The structure should support both branches and tags. Tags are releases of either applications or projects, while branches are bigger developments that could span a couple of tags or releases.
It must be possible to patch tags or releases easily.
Django projects should be able to reference specific versions of the django applications.

These requirements have led me to a structure which looks like this:

/
/django-app
/app_1
      /trunk/app_1
      /tags
/1.0/app_1
        /2.0/app_1
      /branches
        /awsome_new_feature/app_1
/django-projects
    /project_1
      /trunk
        /project_1
          /apps
      /tags
        /1.0
          /project_1
        /1.1/project_1      /branches
/client_color_test/app_1

It is quite easy to see that requirements 1 to 4 are easily satisfied by this structure. Requirement 5 is also met; as long as all patches on a tag or release are done in another tag or release that is related to the original tag, by way of the version number. So if a patch needs to be done to version 1.0 of an application the procedure would be to create a tag from the version 1.0 tag and name it something like version 1.1. Then apply any changes to the version 1.1 tag and release a new package from the version 1.1 tag. This way the original version 1.0 tag or release stays untouched.

Requirement 6 is met but not in the structure itself, but by employing a very nifty feature in Subversion called, externals. (By the way, the repeated project and application names (project1/trunk/application1 and project_1/tags/1.0/project_1) in the repository tree was introduced to make the use of externals more transparent, as can be see below.) The external property gets added to the project directory under the /trunk of the project. In my case I add an external that looks a bit like the following (it assumes that you are in the django-projects/project_1/trunk directory):

svn propget svn:externals project_1 ^/django-apps/app_1/trunk apps

svn propget svn:externals project_1 ^/django-apps/app_1/tags/1.0 apps

The first example add the externals property to my project_1 directory linking my project_1 to the trunk of app_1. When a checkout is done the app_1/trunk contents will be checked out into the project_1/apps directory. The second example will link the release version 1.0 of the same application rather than the trunk version of the application to project_1.

One of the problems with this scheme is that if you reference the trunk of an application and create a release, then the release will also reference the trunk. However, this is easily managed by always developing against release versions of applications. Obviously this is not always possible and in these cases vigilant management of the projects will be required to ensure that a project release doesn't accidentally reference incorrect versions of applications. I can see the possibility of writing some code to validate these rules.

Thats it for now. I am quite excited to start using this structure.

Adding CKEditor to Django Admin

This article will show how to enhance the django admin site's textareas by replacing them with the CKEditor control.

Install CKEditor

Download the CKEditor here. Place the CKEditor source in such a spot that the static web server can serve it. In my case I have created a media directory under the document root of the web static web server and then placed the CKEditor files in a directory ckeditor. Thus I can access the ckeditor.js like this http://localhost/media/ckeditor/ckeditor.js.

Update the MEDIA_URL = 'http://localhost/media/' in the settings.py of your django project.

My sample model is very simple and looks like this:

from django.db import models

class SampleModel(models.Model):
title = models.CharField(max_length=50)
text = models.TextField()

def __unicode__(self):
return self.title

The magic happens in the custom model admin form. The for for my sample model looks like this:

from sampleapp.models import SampleModel
from django.contrib import admin
from django import forms
from django.db import models

class SampleModelAdmin(admin.ModelAdmin):
formfield_overrides = { models.TextField: {'widget': forms.Textarea(attrs={'class':'ckeditor'})}, }

class Media:
js = ('ckeditor/ckeditor.js',) # The , at the end of this list IS important.

admin.site.register(SampleModel,SampleModelAdmin)

The important things to notice is that I add class attribute to the textarea widget. When ckeditor then satrt up it knows to replace any textareas with the ckeditor code.

The last thing to notice is the the media location is relative to the MEDIA_URL since there is no preceding /.

That's it.

Her is the sample application: django-ckeditor.zip. Or you can get is directly from the subversion repository:

svn checkout http://johanscode.googlecode.com/svn/devblog/django/ckeditor johanscode-read-only

Johan's Dev Blog Template

The template system at blogger.com (where this blog is being hosted) is quite nice and it allowed me to modify the look and feel of my blog within a couple of minutes to be close to what I wanted. I have uploaded the template that I am using here (look for devblog.css) if anybody is interested.

Easy LAMP Setup

This is the easiest way to install a LAMP stack on an Ubuntu machine:

sudo apt-get install apache2 php5-mysql libapache2-mod-php5 mysql-server

After the installation has finished remember to restart the Apache server.

sudo /etc/init.d/apache2 restart

Connect and enjoy.

Johan's Dev Blog