GeoDjango

GeoDjango is an included contrib module to make easy GIS (Geographic Information System) web apps with location-based services. GeoDjango provides a toolbox of utilities for building GIS web applications and also bindings to popular spatial libraries such as GEOS, GDAL, and GeoIP, which can be used separately without Django in any Python application or interactively in the shell.

Let’s get started by installing Python 3.

$ python3 --version

Now let’s install GeoDjango dependencies by running the following commands.

$ sudo aptitude install gdal-bin libgdal-dev 
$ sudo aptitude install python3-gdal

Now, we have to create spatial databases. PostGIS 2 ships with PostgreSQL as an extension for spatial functionality. Geodjango also supports Sqlite (via Spatialite), MySql and Oracle backends.

$ createdb  <db name> 
$ psql <db name>
> CREATE EXTENSION postgis;

Let’s now get started with our project. Let’s call our project geodjango by running:

$ django-admin startproject geodjango

This will initialize the project. Now, we can create a django project within our project.

$ cd geodjango 
$ python manage.py startapp world

Let’s edit the database connection in geodjango/settings.py to match our setup.

DATABASES = {
'default': {
'ENGINE': 'django.contrib.gis.db.backends.postgis',
'NAME': 'geodjango',
'USER': 'geo',
},
}

In addition, modify the INSTALLED_APPS setting to include django.contrib.admin, django.contrib.gis, and world (your newly created application):

INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.gis',
'world',
]

Geographic Data

World Borders

We can avail the data of the world borders from Here. Let’s create a data directory in the world application, download the world borders data, and unzip. Use the following commands:

$ mkdir world/data
$ cd world/data
$ wget https://thematicmapping.org/downloads/TM_WORLD_BORDERS-0.3.zip
$ unzip TM_WORLD_BORDERS-0.3.zip
$ cd ../..

The world borders ZIP file contains a set of data files which is one of the most popular geospatial data formats known as an ESRI Shapefile. When unzipped, the world borders dataset includes files with the following extensions:

  • .shp: Holds the vector data for the world borders geometries.
  • .shx: Spatial index file for geometries stored in the .shp.
  • .dbf: Database file for holding non-geometric attribute data (e.g., integer and character fields).
  • .prj: Contains the spatial reference information for the geographic data stored in the shapefile.

The GDAL ogrinfo utility allows examining the metadata of shapefiles or other vector data sources:

$ ogrinfo world/data/TM_WORLD_BORDERS-0.3.shp 
INFO: Open of `world/data/TM_WORLD_BORDERS-0.3.shp'
using driver `ESRI Shapefile' successful.
1: TM_WORLD_BORDERS-0.3 (Polygon)

ogrinfo tells us that the shapefile has one layer and that this layer contains polygon data. Let’s specify the layer name and use the -so option to get the important summary information:

$ ogrinfo -so world/data/TM_WORLD_BORDERS-0.3.shp TM_WORLD_BORDERS-0.3
INFO: Open of `world/data/TM_WORLD_BORDERS-0.3.shp'
      using driver `ESRI Shapefile' successful.

Layer name: TM_WORLD_BORDERS-0.3
Geometry: Polygon
Feature Count: 246
Extent: (-180.000000, -90.000000) - (180.000000, 83.623596)
Layer SRS WKT:
GEOGCS["GCS_WGS_1984",
    DATUM["WGS_1984",
        SPHEROID["WGS_1984",6378137.0,298.257223563]],
    PRIMEM["Greenwich",0.0],
    UNIT["Degree",0.0174532925199433]]
FIPS: String (2.0)
ISO2: String (2.0)
ISO3: String (3.0)
UN: Integer (3.0)
NAME: String (50.0)
AREA: Integer (7.0)
POP2005: Integer (10.0)
REGION: Integer (3.0)
SUBREGION: Integer (3.0)
LON: Real (8.3)
LAT: Real (7.3)

This detailed summary information tells us the number of features in the layer, the geographic bounds of the data, the spatial reference system (“SRS WKT”), as well as type information for each attribute field. For example, FIPS: String (2.0) indicates that the FIPS character field has a maximum length of 2. Similarly, LON: Real (8.3) is a floating-point field that holds a maximum of 8 digits, up to three decimal places.

Geographic Models

Defining a Geographic Model

Let’s create a GeoDjango model to represent the data:

from django.contrib.gis.db import models

class WorldBorder(models.Model):
    # Regular Django fields corresponding to the attributes in the
    # world borders shapefile.
    name = models.CharField(max_length=50)
    area = models.IntegerField()
    pop2005 = models.IntegerField('Population 2005')
    fips = models.CharField('FIPS Code', max_length=2)
    iso2 = models.CharField('2 Digit ISO', max_length=2)
    iso3 = models.CharField('3 Digit ISO', max_length=3)
    un = models.IntegerField('United Nations Code')
    region = models.IntegerField('Region Code')
    subregion = models.IntegerField('Sub-Region Code')
    lon = models.FloatField()
    lat = models.FloatField()

    # GeoDjango-specific: a geometry field (MultiPolygonField)
    mpoly = models.MultiPolygonField()

    # Returns the string representation of the model.
    def __str__(self):
        return self.name

The default spatial reference system for geometry fields is WGS84 . In other words, the field coordinates are in longitude, latitude pairs in units of degrees. To use a different coordinate system, let’s set the SRID of the geometry field with the srid argument. Use an integer representing the coordinate system’s EPSG code.

Run migrate

After defining your model, we need to sync it with the database. First, we need to create a database migration. Run the following command:

$ python manage.py makemigrations
Migrations for 'world':
  world/migrations/0001_initial.py:
    - Create model WorldBorder

You may inspect the raw code generated by the above migration:

$ python manage.py sqlmigrate world 0001

The output:

BEGIN;
--
-- Create model WorldBorder
--
CREATE TABLE "world_worldborder" (
    "id" serial NOT NULL PRIMARY KEY,
    "name" varchar(50) NOT NULL,
    "area" integer NOT NULL,
    "pop2005" integer NOT NULL,
    "fips" varchar(2) NOT NULL,
    "iso2" varchar(2) NOT NULL,
    "iso3" varchar(3) NOT NULL,
    "un" integer NOT NULL,
    "region" integer NOT NULL,
    "subregion" integer NOT NULL,
    "lon" double precision NOT NULL,
    "lat" double precision NOT NULL
    "mpoly" geometry(MULTIPOLYGON,4326) NOT NULL
)
;
CREATE INDEX "world_worldborder_mpoly_id" ON "world_worldborder" USING GIST ( "mpoly" );
COMMIT;

Now, let’s run migrate to execute the sql:

$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions, world
Running migrations:
  ...
  Applying world.0001_initial... OK

GDAL Interface

GeoDjango also includes a Pythonic interface to GDAL’s powerful OGR library that can work with all the vector data sources that OGR supports.

First, let’s invoke the Django shell:

$ python manage.py shell

We can determine the path of World Border data using Python’s built-in os module:

>>> import os
>>> import world
>>> world_shp = os.path.abspath(os.path.join(os.path.dirname(world.__file__),
...                             'data', 'TM_WORLD_BORDERS-0.3.shp'))

Now, open the world borders shapefile using GeoDjango’s DataSource interface:

>>> from django.contrib.gis.gdal import DataSource
>>> ds = DataSource(world_shp)
>>> print(ds)
/ ... /geodjango/world/data/TM_WORLD_BORDERS-0.3.shp (ESRI Shapefile)

Data source objects can have different layers of geospatial features; however, shapefiles are only allowed to have one layer:

>>> print(len(ds))
1
>>> lyr = ds[0]
>>> print(lyr)
TM_WORLD_BORDERS-0.3

We can see the layer’s geometry type the features it contains:

>>> print(lyr.geom_type)
Polygon
>>> print(len(lyr))
246

The Layer may also have a spatial reference system associated with it. If it does, the srs attribute will return a SpatialReference object:

>>> srs = lyr.srs
>>> print(srs)
GEOGCS["GCS_WGS_1984",
    DATUM["WGS_1984",
        SPHEROID["WGS_1984",6378137.0,298.257223563]],
    PRIMEM["Greenwich",0.0],
    UNIT["Degree",0.0174532925199433]]
>>> srs.proj4 # PROJ.4 representation
'+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs '

This shapefile is in the popular WGS84 spatial reference system.

In addition, shapefiles also support attribute fields that may contain additional data. Here are the fields on the World Borders layer:

>>> print(lyr.fields)
['FIPS', 'ISO2', 'ISO3', 'UN', 'NAME', 'AREA', 'POP2005', 'REGION', 'SUBREGION', 'LON', 'LAT']

The following code will let you examine the OGR types (e.g. integer or string) associated with each of the fields:

>>> [fld.__name__ for fld in lyr.field_types]
['OFTString', 'OFTString', 'OFTString', 'OFTInteger', 'OFTString', 'OFTInteger', 'OFTInteger', 'OFTInteger', 'OFTInteger', 'OFTReal', 'OFTReal']

We can iterate over each feature in the layer and extract information from both the feature’s geometry as well as the feature’s attribute fields (whose values are accessed via get() method):

>>> for feat in lyr:
...    print(feat.get('NAME'), feat.geom.num_points)
...
Guernsey 18
Jersey 26
South Georgia South Sandwich Islands 338
Taiwan 363

Layer objects may be sliced:

>>> lyr[0:2]
[<django.contrib.gis.gdal.feature.Feature object at 0x2f47690>, <django.contrib.gis.gdal.feature.Feature object at 0x2f47650>]

And individual features may be retrieved by their feature ID:

>>> feat = lyr[234]
>>> print(feat.get('NAME'))
San Marino

Boundary geometries may be exported as WKT and GeoJSON. Both are plaintext representations of shapes which are originally encoded in binary.

>>> geom = feat.geom
>>> print(geom.wkt)
POLYGON ((12.415798 43.957954,12.450554 ...
>>> print(geom.json)
{ "type": "Polygon", "coordinates": [ [ [ 12.415798, 43.957954 ], [ 12.450554, 43.979721 ], ...

LayerMapping

To import the data, use a LayerMapping in a Python script. Create a file called load.py inside the world application, with the following code:

import os
from django.contrib.gis.utils import LayerMapping
from .models import WorldBorder

world_mapping = {
    'fips' : 'FIPS',
    'iso2' : 'ISO2',
    'iso3' : 'ISO3',
    'un' : 'UN',
    'name' : 'NAME',
    'area' : 'AREA',
    'pop2005' : 'POP2005',
    'region' : 'REGION',
    'subregion' : 'SUBREGION',
    'lon' : 'LON',
    'lat' : 'LAT',
    'mpoly' : 'MULTIPOLYGON',
}

world_shp = os.path.abspath(
    os.path.join(os.path.dirname(__file__), 'data', 'TM_WORLD_BORDERS-0.3.shp'),
)

def run(verbose=True):
    lm = LayerMapping(WorldBorder, world_shp, world_mapping, transform=False)
    lm.save(strict=True, verbose=verbose)

Let’s invoke the Django shell from the geodjango project directory:

$ python manage.py shell

Next, let’s import the load module, call the run routine:

>>> from world import load
>>> load.run()

grinspect

The ogrinspect command introspects a GDAL-supported vector data source (e.g., a shapefile) and generates a model definition and LayerMapping dictionary automatically.

The general usage of the command goes as follows:

$ python manage.py ogrinspect [options] <data_source> <model_name> [options]

data_source is the path to the GDAL-supported data source and model_name is the name to use for the model. Command-line options may be used to further define how the model is generated.

For example, the following command nearly reproduces the WorldBorder model and mapping dictionary created above, automatically:

$ python manage.py ogrinspect world/data/TM_WORLD_BORDERS-0.3.shp WorldBorder \
    --srid=4326 --mapping --multi

The command produces the following output, which may be copied directly into the models.py of a GeoDjango application:

# This is an auto-generated Django model module created by ogrinspect.
from django.contrib.gis.db import models

class WorldBorder(models.Model):
    fips = models.CharField(max_length=2)
    iso2 = models.CharField(max_length=2)
    iso3 = models.CharField(max_length=3)
    un = models.IntegerField()
    name = models.CharField(max_length=50)
    area = models.IntegerField()
    pop2005 = models.IntegerField()
    region = models.IntegerField()
    subregion = models.IntegerField()
    lon = models.FloatField()
    lat = models.FloatField()
    geom = models.MultiPolygonField(srid=4326)

# Auto-generated `LayerMapping` dictionary for WorldBorder model
worldborders_mapping = {
    'fips' : 'FIPS',
    'iso2' : 'ISO2',
    'iso3' : 'ISO3',
    'un' : 'UN',
    'name' : 'NAME',
    'area' : 'AREA',
    'pop2005' : 'POP2005',
    'region' : 'REGION',
    'subregion' : 'SUBREGION',
    'lon' : 'LON',
    'lat' : 'LAT',
    'geom' : 'MULTIPOLYGON',
}

Spatial Queries

Spatial Lookups

GeoDjango adds spatial lookups to the Django ORM. We can find the country in the WorldBorder table that a point belongs to. Let’s fire up the management shell:

$ python manage.py shell

Define the point of interest:

>>> pnt_wkt = 'POINT(-95.3385 29.7245)'

The pnt_wkt string represents the point at -95.3385 degrees longitude, 29.7245 degrees latitude. The geometry is in a format known as Well Known Text (WKT), a standard issued by the Open Geospatial Consortium (OGC). [ Import the WorldBorder model, and perform a contains lookup using the pnt_wkt as the parameter:

>>> from world.models import WorldBorder
>>> WorldBorder.objects.filter(mpoly__contains=pnt_wkt)
<QuerySet [<WorldBorder: United States>]>

We can also use a GEOS geometry object. Here, we can combine the intersects spatial lookup with the get method to retrieve only the WorldBorder instance for San Marino instead of a queryset:

>>> from django.contrib.gis.geos import Point
>>> pnt = Point(12.4604, 43.9420)
>>> WorldBorder.objects.get(mpoly__intersects=pnt)
<WorldBorder: San Marino>

Automatic Spatial Transformations

When doing spatial queries, GeoDjango automatically transforms geometries if they’re in a different coordinate system.

>>> from django.contrib.gis.geos import GEOSGeometry, Point
>>> pnt = Point(954158.1, 4215137.1, srid=32140)

pnt may also be constructed with EWKT, an “extended” form of WKT that includes the SRID:

>>> pnt = GEOSGeometry('SRID=32140;POINT(954158.1 4215137.1)')

GeoDjango’s ORM will automatically wrap geometry values in transformation SQL, allowing the developer to work at a higher level of abstraction:

>>> qs = WorldBorder.objects.filter(mpoly__intersects=pnt)
>>> print(qs.query) # Generating the SQL
SELECT "world_worldborder"."id", "world_worldborder"."name", "world_worldborder"."area",
"world_worldborder"."pop2005", "world_worldborder"."fips", "world_worldborder"."iso2",
"world_worldborder"."iso3", "world_worldborder"."un", "world_worldborder"."region",
"world_worldborder"."subregion", "world_worldborder"."lon", "world_worldborder"."lat",
"world_worldborder"."mpoly" FROM "world_worldborder"
WHERE ST_Intersects("world_worldborder"."mpoly", ST_Transform(%s, 4326))
>>> qs # printing evaluates the queryset
<QuerySet [<WorldBorder: United States>]>

Lazy Geometries

GeoDjango loads geometries in a standardized textual representation. When the geometry field is first accessed, GeoDjango creates a GEOSGeometry object, exposing powerful functionality, such as serialization properties for popular geospatial formats:

>>> sm = WorldBorder.objects.get(name='San Marino')
>>> sm.mpoly
<MultiPolygon object at 0x24c6798>
>>> sm.mpoly.wkt # WKT
MULTIPOLYGON (((12.4157980000000006 43.9579540000000009, 12.4505540000000003 43.9797209999999978, ...
>>> sm.mpoly.wkb # WKB (as Python binary buffer)
<read-only buffer for 0x1fe2c70, size -1, offset 0 at 0x2564c40>
>>> sm.mpoly.geojson # GeoJSON
'{ "type": "MultiPolygon", "coordinates": [ [ [ [ 12.415798, 43.957954 ], [ 12.450554, 43.979721 ], ...

This includes access to all of the advanced geometric operations provided by the GEOS library:

>>> pnt = Point(12.4604, 43.9420)
>>> sm.mpoly.contains(pnt)
True
>>> pnt.contains(sm.mpoly)
False

Putting your data on the map

Geographic Admin

GeoDjango extends Django’s admin application with support for editing geometry fields.

Basics

GeoDjango also supplements the Django admin by allowing users to create and modify geometries on a JavaScript slippy map.

Let’s create a file called admin.py inside the world application with the following code:

from django.contrib.gis import admin
from .models import WorldBorder

admin.site.register(WorldBorder, admin.GeoModelAdmin)

Next, edit the urls.py in the geodjango application folder as follows:

from django.contrib.gis import admin
from django.urls import include, path

urlpatterns = [
    path('admin/', admin.site.urls),
]

Create an admin user:

$ python manage.py createsuperuser

Next, start up the Django development server:

$ python manage.py runserver

Browse in to http://localhost:8000/admin/, and log in with the user you just created. Browse to any of the WorldBorder entries. The borders may be edited by clicking on a polygon and dragging the vertices to the desired position.

GeoDjango turns Django into an effective geographic Web framework. GeoDjango eases the effort of making GIS (Geographic information system) web apps with location-based services.

Ruby vs Python

Ruby vs Python: The ultimate battle

A tough question that goes through the mind of all web developers. Ruby vs Python. Which one is better? Both of them are popular languages suited for developing web applications. In short, it’s the clash of the Titans.

The Language

Python is inspired by languages like C/C++, Java, Perl, Lisp, and ICON. Python believes in “only one best way to do a particular thing”, thus being strict on indentation and layout, even the amount of whitespace to use. Therefore Python is said to be easy to learn and extremely readable.

Ruby is influenced by languages like C, Java, Perl, and C++. Ruby believes in “language’s code should always cause as little confusion as possible for the developer.” Ruby makes use of blocks or first-class objects. A block is treated as a unit within the program, resulting in the language to be extremely potent, combining with the use of lambdas and functional programming. Additionally, Ruby takes the OOP concept to a great extent.

The Frameworks

Ruby on Rails is the popular web framework built using the Ruby programming language while the most popular web framework built using the Python programming language is Django.

The Similarity

Being high-level, object-oriented scripting languages, both Python and Ruby have similar performance. They both come with standard libraries, an interactive shell, and persistence support and are excellent choices for web development.

The Differences

 Ruby and Python take a different approach to solve problems.

Ruby is flexible and empowers the programmers. Ruby on Rails allows the developers to perform many little tricks to make an elegant web framework. The flexibility seems to be magical at times but this could even do harm at times. Sometimes code works when you didn’t expect it to and leaves you feeling really impressed. Other times the tracking down bugs could be hard and will cost you hours.

Python follows a more direct approach to programming. The aim is to ensure that everything is clear to the programmer. Python loses the elegance that Ruby gives but makes it easier to learn the code and debug problems.

Ruby showcases the flexibility of the language while Python focuses on the directness and readability of the code. Ruby focusses on giving the programmers the freedom to adopt the approach they want. Python emphasis on the ease of use and learning in a single right way of doing something. As a result, Python developers are conservative in their approach and the developments and new features of Django are added slowly. On the other hand, Ruby developers enjoy the freedom and change they get resulting in the Rails framework to undergo constant changes.

The Community Support

Both Python and Ruby both have huge community support, influencing in the direction of the language, updates, and the way software is built.

Python has a much more diverse community than Ruby as there are many academic use cases in both Math and Science that Python resulting in the support in those areas. Python is also installed on almost every Linux systems making it the perfect language for server usage.

Ruby’s started to gain popularity since Rails came out in 2005. The Ruby community grows quickly around Rails and focused on web development. The community gets more diverse with time, but it does not have the same diversity that Python has.

Major Users

Python has been used by many companies including Google, Pinterest, Instagram, National Geographic, Mozilla Firefox, and the Washington Post.

Ruby on Rails is the popular web framework built using the Ruby programming language while the most popular web framework built using the Python programming language is Django.

Ruby has been used by companies like Apple, Twitter, Airbnb, Shopify, Hulu, Github, and Groupon.

The Verdict

It’s still hard to point out which language is the better one. Each language has preferred areas where they are better, due to the features and support they provide. If you are into developing web applications, go with Ruby on Rails. They have a very strong community who are always on the bleeding edge. If you are interested in developing web applications but would also like to learn the language more generally, focusing on academic and scientific programming, go for Python and Django. They have a diverse community, influenced and supported by the various industries.

PyCon India 2014 – Days of Future Past

Banglore cantonment station was busy as usual, Taxi wala blocking passengers with their offers. People running away not to get caught in their trap. Sreekanth, Tevin, Nidhin with me trying best for a taxi and a good bargain. Even though we were tired of the eight hour journey in an over cooled AC coach. Nuventure had already arranged all the facilities for the python team to attend the conference. Unfortunately VC and Joju had to face unpredicted issues and back out from the journey.

Train was late for 40 minutes. We had to reach Nimhans Conventional Center in 20 minutes fighting with the heavy morning traffic. Nidhin and me got our workshops scheduled to start at 9:30.

We were impatient whenever we got blocked at red light. Our driver completed the ‘Voyage’ in 40 minutes and we are at Nimhans. Me and Nidhin rocketed to the workshop Auditorium, missing breakfast. Quickly finishing up the check in formalities of event. We went to concerned auditoriums for our sessions.

Day 1

Writing security tools in python by Yashin Mehaboobe was the first session I was running for. Passion for computer security was inside me when I first breached my home computer’s bios password, locked by my elder brother, using Kobian motherboard’s keyboard based bios restoring in 2001.

Yashin could explain situations where we need to write our own tools while many third party or ready made are available. Basic security tasks like Port Scanning, Socket programming, Banner Finger printing, Brute forcing passwords, Packet Crafting, Packet spoofing, Packet Sniffing etc were covered in the session. He showed us how powerful libraries like KeyCzar, impacket help hackers do their jobs.

Extracting EXIF data using Pillow library was very easy. He quickly covered some basic data mining/scrapping with BeautifulSoup and lxml libraries, How Py2Exe, Py2Elf libraries help to pack executable files as single binaries. The session was simple even a python beginner could follow the content.

When I came out of the session, I could find Nidhin, discussing with an enthusiastic college boy at the nearest stall, the details of workshop, Scrape Anything, he attended. Sreekanth and Tevin had already moved to the hotel after finishing up the Registration procedure.

We chatted for some time about our sessions and moved to food corner, where PyCon organizers arranged delicious Vegetarian food. We had some dosas and various side dishes. (Trust me I don’t remember food names much).

The conference along with Workshops were zipped in three days. All the workshops were scheduled on 27th. Two auditoriums were occupied with workshops running parallel. Conference and speeches were programmed on 28th and 29th. Finishing the food, we roamed around the stalls. Most of the stalls were getting ready for the Conference, expected to start next day.

I got back to the venue after my Friday Prayer. Nimhans’s lobby getting filled with Pythonista’s all around the globe. Next workshop was scheduled at 4:30, and I got plenty of time to explore the Convention Center. Situated at the heart of Banglore, Nimhans CC has been venue for many open source conferences and events.

Ben Postlethwaite and Alex Vados from Plotly were setting up their kiosk. I had used plotly graphs for some of my projects, Plotly draws cooler interactive graph for the input data provided. Very much helpful for BigData fans. Inquired more about Javascript based graph plotting options and issues I had faced, Ben showed me some of the upcoming features of Plotly on his Vagrant development environment.

My mobile had already drained off and Mac book pro running with 10% power. Wandering along the venue for a power socket, I met Pratyush Who has been working as a Chartered Accountant and spending his life as a pythonista. Hardcore fan of Django, founder of Screener.in, helping many investors for their investment options. He explained me differences between Fundamental investment and Technical investment and how some guys use best of both. Also showed me a nice Firefox plugin Vimperator for making the browser to Vi compatible interface. I gave some instructions, cloud options and async libraries for the scalability of screener project.

I was amazed when two kids around 11 and 9 ages talking to people about pygame and their adventures at the lobby while waiting for the charging to get completed. The alarm @ 16:30 woke me up and I had to attend the Test your webapp with Selenium session.

Anisha Narang’s session was a real workshop. We had a partial code to download and test the webapp, Coding along with many others in the flow. Many volunteers to help out if anyone got blocked in the middle. Even though I had heard the word selenium many times, I haven’t tried it till the conference. Using selenium is very efficient to do Single Page or Multipage web application functionality testing.

I had written many unittests for API’s at Nuventure as part of our test driven development, using python libraries like re, unittests, BeautifulSoup and lxml. But haven’t tested UI functionality via full automated way. Anisha showed us how pyselenium is so handy to test webapps functionality with Firefox, Chrome Browser or Headless one. The workshop showed how effectively we could Emulate Button click or key press and run the web application on intended flow.

I met Baiju Muthukadan who is an active FOSS activist and open source contributor. He explained me how object oriented programming could be used to extend pyselenium’s functionality, and why ‘Go’ language is gaining popularity with its inbuilt concurrency and increased performance. We had a chitchat about how Grunt, Gulp, Jenkins, Ansible affect the productivity and automation. Talked a little about private clouds and architecture of applications.

Sreekanth and Tevin was at a session, Decorators demystified, which me and Nidhin had a glimpse. Anand Chitipothu, maintainer of web.py framework was explaining from beginner to advanced. He visualized simple decorators like memoize help to compute in a faster way. Explained usage of decorator in the architecture of flask framework.

Our room was booked at Madivala, around 3.5 KM away from NHCC. The travel back  on the line bus was tragedy. Took more than 40 minutes to reach hotel.

The 30 minute long bath helped me to recover from the tiredness of the rush day. I sat on the couch, tried to make some notes of the day. The sweet aroma from Air conditioner and Tandoor Chicken from stomach pulled me to the bed.

Day 2

Morning ride, back to the venue was not that bad. A 10 minute flash in the auto. May be the weekend and Dussehra holidays  dragged people away from the city. We reached  at venue around 9.15 and  stunned when we saw the queue of ‘Python aspirants’.

We got the files and a ‘QR coded’ ID. We rushed for the food and felt disappointed with the ‘long queue’, They were providing wide variety of food items for the day.

Fighting with the queue,  at last, we got food. The session by Kushal Das, Cpython developer and Fedora mentor had already started. The hall was full and we had to ascend to balcony for the session.

Kushal was funny and unfolded his journey through the open source sea. Starting from a small village at Bengal to the Python Software Foundation.

We saw Jayaram, our project manager, Tinu and Faris, directors, who flew from Cochin on previous day. We had a small discussion on last day’s sessions.

I went  to auditorium 2 afterwards for a session on BigData analysis using PySpark, a library for Apache Spark in memory Map Reduce system.

Later We attended Django design patterns by Arun Ravindran. He explained about his experiments and briefed why some of the most common design patterns like abstract factory pattern are ‘Unnecessary Complications’ in python’s case.

Anand Chittipothu’s speech on ‘Messing with Govt data’, was his experience with ‘Electronic election campaign organization’. He covered how he could create an effective election campaign using the available Govt data. Gave short intro on how he created beautiful reports with Report lab library.

We had a fantastic list of items for our lunch. Awesome food, met Oommen, friend of Jayaram, who do research on Biotechnology. He hinted how Biopython is used extensively on biotechnology researches.

The tasty lunch was enough for a good sleep. But the session was so interesting that I had to dispose my sleepy mood. Some talented college guys were trying to implement a faster VM for python. ‘Medusa’ which converts python keywords to equivalent Dart code, and ran on Dart VM. Resulting a much faster python execution. Especially in the case of recursive functions.

Dreamworks guys who coded their workflow with python, was next on stage. They were using twisted matrix, a highly customizable asynchronous framework, helping you to write network programs in any layer. I had never met anyone in person who used Twisted Matrix after my project with a Japanese firm, where we built HTTP Sticky Load balancer. Dreamworks guys explained how twisted helps them transfer large files across the sea, via dedicated line connections in between, by using client server architecture.

The documentary about “Aaron Swartz: The internet’s own boy” was inspirational. He fought for freeing knowledge. Indeed his early death was a great loss to the humanity.

There were lightning talks, which people came out with their ideas/talks and presented for 5 minutes. Was a brilliant idea which covered many new topics in a flash.

Some one took a session on ExpEYES, a cheap CRO, with a USB connector and UI software created on Python. It will definitely help young entrepreneur with Internet of Things device ideas.

A 12 year old and 9 year old kid took sessions on PyGame and how they created games in the simple framework. Python is so simple even kids could program 😀

The day went really well. Visited Forum Mall while returning back to hotel. Done a quick shopping and ‘roaming’ though the mall with Sreekanth’s friends. We could try Chicken 65 and tandoor roti, Banglore version from the hotel at Madivala and had a peaceful sleep.

Day 3

Freshened up early, We started our journey for the third day. Seems early morning journey on Saturday and Sunday is not much affected by traffic as on week days. We reached the venue and queued for breakfast. Breakfast was delicious and appetizing. The big queue drawn me to session hall early without fighting much.

The Keynote speaker, Michael Foord, who is an active developer of Cpython and Ubuntu Juju. Started his career at IronPython, he later became active developer in Cpython. He gave introduction to Ubuntu Juju, an orchestration management for various cloud service providers and services. While normal cloud management systems like Ansible, Puppet and Chef provides provisioning capabilities, Juju understands the relationship between those services.

The next session was by  Aditya Manthramurthy @Plivo. Covered how Asynchronous IO systems like Gevent help to scale servers to handle heavy amount of traffic. While traditional codes to fetch data from db, calling thirdparty apis from servers etc block the operation of ‘request to response’, async operations help to run these operations in parallel and speed up response generation time and  handle more requests per second. He also explained how WSGI server handles a request and generate the response. If number of WSGI servers are increased, with the expense of more RAM, handle more number of requests per second.

Faster data processing with python by Anand S, was a mind blowing session. Explained how he could handle the ‘slow’ python for a faster data processing. Done a walk through, pros and cons of data serialization options like csv, json, pickle. His approach to speedup code via eliminate redundant operation, optimize time taking operations via reducing number of hits and reducing number of time per hit was awesome. Described how line profiling is useful for optimizations. He showed how caching some of the recurring functions gives  speedup to the code execution. Briefed Cython, numba with static typing to speedup execution if we need much faster calculations. He showed how some of the functions at numpy and pandas are faster than built in python equivalents.

There was a panel discussion after Anand’s session. Django, Flask or web.py which is better framework for web development. While Arun Narayanan represented Django, Kiran Jonnalagadda of hasgeek represented Flask, and Anand Chitipothu represented web.py.

The discussion included community response, maintenance, optimizations, performance and future of these frameworks. One thing that was new to me, at PyCon they introduced a miscall voting system to vote for the frameworks, and Django was winner in voting, as most of the people use Django for their development.

Next session I went, Development to Production by Anubhav Sinha. He briefed about Vagrant and Docker and explained why docker containers help to ‘Write somewhere and run anywhere’ applications. Docker helps to run applications and environments work across operating systems.

The last session I attended was Narahari Allamraju’s ‘Which messaging layer should you use if you want to build a loosely coupled distributed Python app?’. How to identify the patterns to make application loosely coupled. Briefed about advantages, disadvantages of Zeromq, RabbitMQ, Redis and some other MQ systems.

Most of the people who used iPython Notebook in their presentation. It is an easy web interface for iPython to try codes and save them like a notebook.

We went back to hotel and freshened up, packed things for our return journey. Our train was scheduled @9.30 evening from Banaswadi station, Had to travel 2 hours, roam around a little to find a restaurant near railway station. After 12 hours of train journey we reached Ernakulam Town Monday morning.

And we still miss the PyCon Days 🙁

* Kushal’s Session Original photo link

Vim Bootstrap – generate .vimrc for your favorite programming language

Programmers in the current era like to work using IDEs because it is more user friendly. But there are reasonable number of programmers who still use command line for programming. Their favourite editor might be Vim. Vim is easy to use, light-weight and comes with almost every Unix distribution. Unlike IDEs, it does not hang easily. The only problem with Vim editor is that it does not have inbuilt syntax highlighters. Checkout Vim Bootsrap , problem solved. Using the site you can generate .vimrc file for your favourite programming language. Copy the .vimrc file in your home directory. Now, you are ready to start programming using your favourite language. Vim Bootstrap comes with support of c, python, php, javascript, perl, go and lot more.

Pelican with github pages

Pelican is a static site generator written in python which allows you to write your content directly using reStructuredText or Markdown. In this blog post, we will discuss how to create and host a static site using pelican and github pages.

Installation

Lets wrap all our installations inside a separate virtual environment using python virtual environment wrapper.

Install python virtualenvwrapper

After installing virtual environment wrapper, create a new environment for our project:

$ mkvirtualenv pelican-blog
$ workon pelican-blog

Run pelican installation:

$ pip install pelican

Checkout more installation options for pelican

Run markdown installation for writing the site contents:

$ pip install Markdown

You can also use reStructuredText for writing your content instead of markdown.

Quickstart your blog

Now we have finished all our installations and let’s create a skeleton for our site using the pelican-quickstart command. This will ask you few questions and will pre-populate the settings data as per your answers. These settings can be changed at any point of time.

$ pelican-quickstart

Please see how I have answered the quickstart questions below:

Welcome to pelican-quickstart v3.4.0.

This script will help you create a new Pelican-based website.

Please answer the following questions so this script can generate the files needed by Pelican.

> Where do you want to create your new web site? [.]
> What will be the title of this web site? Sreekanth's Blog
> Who will be the author of this web site? Sreekanth
> What will be the default language of this web site? [en]
> Do you want to specify a URL prefix? e.g., http://example.com (Y/n)
> What is your URL prefix? (see above example; no trailing slash) http://sreekanthkaralmanna.github.io
> Do you want to enable article pagination? (Y/n) How many articles per page do you want? [10]
> Do you want to generate a Fabfile/Makefile to automate generation and publishing? (Y/n)
> Do you want an auto-reload & simpleHTTP script to assist with theme and site development? (Y/n)
> Do you want to upload your website using FTP? (y/N) Do you want to upload your website using SSH? (y/N)
> Do you want to upload your website using Dropbox? (y/N)
> Do you want to upload your website using S3? (y/N)
> Do you want to upload your website using Rackspace Cloud Files? (y/N)
> Do you want to upload your website using GitHub Pages? (y/N)

Done. Your new project is available at /home/sreekanth/Documents

Once you finish with all those questions, you will be having a directory structure as follows:

yourproject/
 ├── content
 │ └── (pages)
 ├── output
 ├── develop_server.sh
 ├── fabfile.py
 ├── Makefile
 ├── pelicanconf.py # Main settings file
 └── publishconf.py # Settings to use when ready to publish

Customise your theme

Pelican comes with a default theme for every site generated. But you can use a variety of other themes to beautify your site.

In order to include your desired theme, clone your preferred theme and save the theme to a theme directory inside your project folder. Now tell pelican that it has to use this particular theme to generate html for your site. In pelicanconfig.py, add the following:

THEME = "themes/theme_name"

Also make sure to change your pelican settings as per the settings given in the theme documentations(if any)

Writing your blog content

We are done with configuring our pelican site, now let’s start writing our blog contents  inside the content folder. Below given is a sample markdown file.

Title: Pelican Test Blog
Date: 2014-05-01 10:00
Category: Python
Tags: python
Author: Sreekanth
Summary: Pelican blog for testing.

A Pelican blog for testing

Generate html files and run your site

We are all done to test our site locally.

The following command will create the output folder for your site and serves your site locally at http://localhost:8000.

$ make devserver

Run the following command to stop the localhost server:

$ ./devserver stop

Publish your blog using Github pages

For hosting your blog using github pages, create a repository named username.github.io and this will be the remote repository for your site. Contents of your output directory has to be stored in this repository.

After creating the github repository, push your output files to the remote repository. Follow the steps given below for doing the same.

$ cd output
$ git init
$ git remote add origin https://github.com/username/username.github.io.git
$ git add --all
$ git commit -m "initial commit"
$ git push origin master

We are all done with hosting our pelican site using github pages. Checkout username.github.com to see your site on live.

Also checkout this slideshow explaining hosting pelican sites using github pages

Happy Blogging 🙂

Executing bash commands via python

In python, executing bash commands can be done using subprocess module. It’s pretty easy to use and it’s a powerful module. For simple commands,  we can use subprocess.call

Usage:

import subprocess

subprocess.call("command1")

subprocess.call(["command1", "arg1", "arg2"])

Eg:

import subprocess

subprocess.call(["ls", "-l"])
total 4

-rw-rw-r-- 1 tevin tevin 15 Sep 3 15:29 test.txt

You can also use subprocess.check_call  and subprocess.check_output

Popen:

For more flexibility, you can use Popen . Using this you can store the output of command as well as any error occurred during command execution.

Eg:

process = subprocess.Popen(["ls", "-l"])
(output, err) = process.communicate()

communicate  method interacts with process and waits for the process to complete. It returns a tuple consisting of stdout and stderr.

Executing commands in background:

subprocess.Popen() only runs a process in the background if nothing in the python script depends on the output of the command being run. For example, the following code won’t be executed in background.

import subprocess

process = subprocess.Popen(["ls", "-l"], stdout=subprocess.PIPE)

Thinking about what is subprocess.PIPE?

subprocess.PIPE: Special value that can be used as the stdin, stdout or stderr argument to Popen and indicates that a pipe to the standard stream should be opened.

Changing directory(cd command):

You might be thinking that why ‘cd’? Why it can’t be executed using subprocess?

If you use subprocess.call(“cd ..”), it will throw an error ‘No such file or directory’. It is because cd is a shell internal. So you can only call it as

subprocess.call('cd ..', shell=True)

But it is pointless to do so.  As no process can change another process’s working directory (again, at least on a UNIX-like OS, but as well on Windows), this call will have the subshell change its dir and exit immediately. Don’t worry you can change directory using os.chdir(path).

Concept of Mutable & Immutable objects in Python

An Immutable object, in the lime light of object-oriented and functional programming, is an object whose state cannot be modified after it is created.

Objects of built-in types like (intfloatboolstrtupleunicode) are immutable,

Mutable object, can be mutated or state can be modified after it is created. A mutable object will have at least a single method able to mutate the object.

Objects of built-in types like (listsetdict) are mutable, Custom classes are generally mutable. To simulate immutability in a class, one should override attribute setting and deletion to raise exceptions:

Please refer “names concept” in python, if you don’t have a clear concept onobjects,identifier, `variables.

A practical example to findout the mutablity of object types

x = 10
x = y

We are creating an object of type int. identifiers x and y points to the same object.

id(x) == id(y)
id(y) == id(10)

if we do a simple operation.

x = x + 1

Now

id(x) != id(y)
id(x) != id(10)

The object in which x was tagged is changed. object 10 was never modified. Immutable objects doesn’t allow modification after creation

In the case of mutable objects

m = list([1, 2, 3])
n = n

We are creating an object of type list. identifiers m and m tagged to the same list object, which is a collection of 3 immutable int objects.

id(m) == id(n)

Now poping an item from list object does change the object,

m.pop()

object id will not be changed

id(m) == id(n)

m and n will be pointing to the same list object after the modification. the list object will now contain [1, 2]

Unexpected results can be expected if you use mutable objects in

  • Function’s default arguments
  • Class inheritance