Use Python to post to twitter

Here is an example of using python to post to twitter.

def twit(login, password, msg):

payload= {'status' : msg, 'source' : 'TinyBlurb'}
payload= urllib.urlencode(payload)

base64string = base64.encodestring('%s:%s' % (login, password))[:-1]
headers = {'Authorization': "Basic %s" % base64string}
url = "http://twitter.com/statuses/update.xml"

result = urlfetch.fetch(url, payload=payload, method=urlfetch.POST, headers=headers)

return result.content

Boost Python - Abstract Classes

In the previous article I covered setting up Boost Python to work in Visual Studio.

In this article were going to get into a bit more detail of describing the bindings of classes. Specifically were going to deal with abstract classes and polymorphism.

Consider the following:

struct AbstractClass {
    virtual std::string speak() const = 0;
};

struct DerivedClassA : AbstractClass {
    virtual std::string speak() const {return "Derived Class A";}
};

struct DerivedClassB : AbstractClass {
    virtual std::string speak() const {return "Derived Class B";}
};

We have an Abstract Class and two derived class that implement the speak method. How do we model this in Boost Python such that in Python we can get a pointer to one of these objects and call speak on the proper derived class.

To do this we have to build a wrapper class:

struct WrapperClass : AbstractClass , boost::python::wrapper<AbstractClass> { 
    virtual std::string speak() const {
        return this->get_override("speak")();
    }
};

This wrapper class extends the AbstractClass, and extends a special boost::python::wrapper. For each virtual method you wrap it and call it with this->get_override().

Lastly you need to define the bindings for Python, this is similar to the Hello World example in article one:

BOOST_PYTHON_MODULE(PyHelloWorld)
{
    using namespace boost::python;
    def("getDerivedClassA",&getDerivedClassA,return_value_policy< manage_new_object >());
    def("getDerivedClassB",&getDerivedClassB,return_value_policy< manage_new_object >());
    class_<WrapperClass, boost::noncopyable > ("DerivedClass") 
        .def("speak",boost::python::pure_virtual(&AbstractClass::speak))
        ;
    
}

Here we added two test methods called getDerivedClassA and getDerivedClassB these will be in the package PyHelloWorld as dictated by 'BOOST_PYTHON_MODULE(PyHelloWorld)'

Lastly we want to be able to execute python in the same process as our executable. If you followed the instructions in part 1 then you simply need to add this method:

void start( ) {        
    using namespace boost::python;            
    Py_Initialize();    
    // Explicitly call the method that the BOOST_PYTHON_MODULE macro created.
    // This sets up our DLL to be used by python. 
    initPyHelloWorld();
    std::string pythonCommand;
    try {
        PyRun_SimpleString("import PyHelloWorld");        
        while(1) {
            std::cout << ">>> "; 
            getline(std::cin,pythonCommand);                           
            if (pythonCommand == "quit") {
                break; 
            }else {
                std::cout << PyRun_SimpleString(pythonCommand.c_str());
            }
        }
    } catch ( error_already_set ) {
        PyErr_Print();
    }
    Py_Finalize();                    
}

So now lets try it:

>>> import PyHelloWorld
0>>> a = PyHelloWorld.getDerivedClassA()
0>>> b = PyHelloWorld.getDerivedClassB()
0>>> print a.speak()
Derived Class A
0>>> print b.speak()
Derived Class B
0>>>

Boost Python

Boost Python allows seamless interoperability between C++ and Python. In this article we'll setup Boost and build a simple hello world program that will be able to run in Python.

Boost, in case your unfamiliar, is a powerful set of libraries for C++. There website is at www.boost.org.

For this post we will be using Boost 1.36.0

Boost can be tricky to setup. Their documentation recommends using their own build tool called bjam, but I typically like to build it myself. For this article we'll walk through setting up Boost Python for Visual Studio 2005.

If you haven't already, download boost 1.36.0 library to your local machine. I've expanded the library to c:\download\boost\boost-1.36.0 on my machine.

Now open up Visual Studio 2005. New Project, Visual C++ -> Win32 -> Win32 Console Applicaion.

At the bottom of the screen enter the name for your project: PyHelloWorld, and the location, in my case c:\download\PyHelloWord. The solution name should be filled out for you (PyHelloWorld), and the check mark for 'Create directory for solution' should be checked.

Click Ok.

In the wizard click next. Select 'Dll' as application type, and Empty Project from additional options.

Click finish.

The project we just created will hold our client code, and our python bindings code. Before we can do that though we need to build Boost Python. To do this we'll create another project.

Right click on your solution 'PyHelloWorld', Click Add->Add New Project.

Like before we'll create a Win32 Console Application but we'll call it BoostPython this time. I've decided to keep the project located in the same solution folder of PyHelloWorld.

Follow the steps in the wizard again to create an empty, dll project.

In order to build boost python we need to include all the source files. Visual studio should have created a 'Source Files' filter in the BoostPython project. Right click and select Add->Existing Item...

You want to add all the files from

• %PATH_TO_BOOST%\libs\python\src


• %PATH_TO_BOOST%\libs\python\src\converter


• %PATH_TO_BOOST%\libs\python\src\object

You now need to instruct the compiler that these are part of the boost source code. You do this by setting a preprocessor define. Right click on the BoostPython project, select c/c++ from the property pages, select Preprocessor, and append 'BOOST_PYTHON_SOURCE' to the end of the Preprocessor definitions. Its a semi-colon separated list so add a semi-colon then 'BOOST_PYTHON_SOURCE'.

Next set the include directory to find the Boost install. Right click on the BoostPython project, open the c/c++ -> General tab. Add your path to %PATH_TO_BOOST%, in my case that is C:\download\boost\boost_1_36_0

We've now set up the boost portion, we need to setup python by telling our project how to find our local python installation. First you'll need to download Python

The standard install saved Python under c:\Python25 on my machine. The standard python install comes with c++ headers so we'll point our BoostPython project at those headers.

Once again open the property page for BoostPython, C/C++ -> General , and add c:\Python25\include (or your python install) to the include path.

You can now build the BoostPython project, all the files should compile. The link step will fail though because it needs to link to the Python library as well. So one more tweak to the BoostPython project, this time Linker -> Input , Additional Dependencies. Add C:\Python25\libs\Python25.lib.

Build the project again and it should link successfully.

Okay now we have BoostPython ready to go. Lets Use it.

For the PyHelloWorld project we'll need to do the same steps as above to set the Include Path, and the Linker. More explicity you need to set:

• include path to boost
• include path to python
• linker to python lib

If you don't remember how look above to see what we set.


Now create a new cpp file by right clicking on the project, add->New Item. Selecting cpp file. Call the file hello.cpp

Add the following to the file.

#define BOOST_ALL_NO_LIB
#include <boost/python.hpp>

char const* speak()
{
    return "Hello World!";
}


BOOST_PYTHON_MODULE(PyHelloWorld)
{
    using namespace boost::python;
    def("speak", speak);
}

The first define instruct boost that we have already build it. The first function is your standard c++ function, the second is our python bindings.

Before you build you need to tell visual studio that this project is dependent on the BoostPython Project. Right click on PyHelloWorld project and click Project Dependencies. Select BoostPython.

Now Build. Everything should compile and build succesfully.

So what have we accomplished? We've build a dll that exposes method that python can understand. Lets test it and see if it works.

Open a Command shell, and navigate to your output directory. In my case its c:\download\PyHelloWorld\PyHelloWorld\debug. In order to make it easier for Python to find our dll lets rename it to pyd.

move PyHelloWorld.dll PyHelloWorld.pyd

Now lets load up the interpreter and try it out:

c:\download\PyHelloWorld\PyHelloWorld\debug>c:\Python25\python.exe
Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import PyHelloWorld
>>> print PyHelloWorld.speak();
Hello World!
>>>

There you go, python interacting with your c++ file. Now that you have the BoostPython project created it would be easy to do this binding for more complex types. The official docs cover how to use this for classes, virtual functions, etc. For more info I recommend you start there. http://www.boost.org/doc/libs/1_36_0/libs/python/doc/tutorial/doc/html/index.html

Hope that helps...

Using java as a shell language

I've been trying to find what shell language I like better as the 'glue' needed between real applications. I've used perl a lot in the past, experimenting with python and ruby now. But I don't want an interpreted language. I really want a compile time language.

So today I figured why not use java.

The main problem is that most scripts are easy to modify and don't require compiling. For example you start your python script with

#!/usr/bin/python

It would be nice if I could do the same for Java. Well now you can.

Introducing the 'JavaLoader'

import java.io.*;
class JavaLoader {
  public static void main(String[] args) throws Exception {
    String pwd = System.getProperty("user.dir");
    System.out.println( "Received argument " + args[0]);
    String javaFile = args[0];
    if (javaFile.startsWith("./"))
      javaFile = javaFile.replace("./","");

    // Find class file
    String className = (javaFile.split("\\."))[0];

    // Comment out the shebang line so we can compile it.
    RandomAccessFile file = new RandomAccessFile(javaFile, "rw");
    file.seek(0);
    file.write("//".getBytes());
    file.close();

    // Compile it
    executeCommand(String.format("javac -cp $CLASSPATH:%s %s",pwd,javaFile));

    //Run
    executeCommand(String.format("java -cp $CLASSPATH:%s %s",pwd,className));

    // Put back #! line.
    file = new RandomAccessFile(javaFile, "rw");
    file.seek(0);
    file.write("#!".getBytes());
    file.close();
  }
  public static void executeCommand(String cmd) throws Exception {
    System.out.println("Executing command: [" + cmd + "]");
    Process p = Runtime.getRuntime().exec(cmd);
    printStream(p.getInputStream());
    printStream(p.getErrorStream());
  }
  public static void printStream(InputStream is) throws Exception{
    BufferedReader input = new BufferedReader(new InputStreamReader(is));
    String line;
    while ((line = input.readLine()) != null) {
      System.out.println(line);
    }
    input.close();
  }
}

By using this you can now write a script like so:

#!/usr/bin/java JavaLoader

class my {

  public static void main(String[] args) {
    System.out.println ("my: from here I am in java");
  }
}

To execute you first need to make sure the JavaLoader is compiled and in your classpath. Then execute your script. My script was called my.java so I did:

[root@pioneer local]# ./my.java
Received argument ./my.java
Executing command: [javac -cp $CLASSPATH:/usr/local my.java]
Executing command: [java -cp $CLASSPATH:/usr/local my]
my: from here I am in java

This is great no more settling, I get to use Java everywhere now!

Python and CUDA

CUDA is Nvidia's api for leveraging the power of the GPU for parallel processing. The cuda api is in C and can be daunting to use. The following how to shows how to use PyCuda to access this powerful API from your python code.

First install PyCuda. You can fetch the latest package from http://pypi.python.org/pypi/pycuda.

Before you can use Cuda you must initialize the device the same way as you would in your C program.

import pycuda.driver as pycuda

pycuda.init()
assert cuda.Device.count() >= 1

cudaDev = cuda.Device(0)
cudaCTX = dev.make_context()

For a cuda program the basic methodolgy is to copy from system memory to devices memory, perform processing, then copy data back from the device to the system. PyCuda provides facilities to do this.

First let's create a numpy array of data that we wish to transfer:

import numpy
a = numpy.random.randn(4,4)
a = a.astype(numpy.float32)
a_gpu = cuda.mem_alloc(a.size * a.dtype.itemsize)
pycuda.memcpy_htod(a_gpu, a)

We now have our data on the device, we need to instruct the GPU to execute our Kernel. A Kernel, when talking about CUDA, is the actual code that will be executed on the GPU. PyCuda requires that you write the kernel in C and pass it to the device.

For example here is a kernel that adds one to the value of each element.

mod = cuda.SourceModule("""
  __global__ void addOne(float *a)
  {
    int idx = threadIdx.x + threadIdx.y*4;
    a[idx]+= 1;
  }
  """)

Now tell the device to execute our kernel.

func = mod.get_function("addOne")
func(a_gpu, block=(4,4,1))

Lastly we copy the contents from the device back to system memory and print the results.

a_addOne = numpy.empty_like(a)
pycuda.memcpy_dtoh(a_doubled, a_gpu)
print a_doubled
print a

Other Resources:

• http://tiker.net/doc/pycuda/
• ftp://ftp.graviscom.com/pub/code/python-cuda/
  

Python web service client using soaplib

Here is an example of a python web service client using soaplib. I personally find the SOAPpy implementation better.

from soaplib.wsgi_soap import SimpleWSGISoapApp
from soaplib.service import soapmethod
from soaplib.serializers.primitive import String
from soaplib.client import make_service_client
class YYY(SimpleWSGISoapApp):
    __tns__ = 'http://webservices.daehosting.com/ISBN'
    @soapmethod(String,_returns=String,_outVariableName='IsValidISBN13Result')
    def IsValidISBN13(self,sISBN):
        pass
client = make_service_client('http://webservices.daehosting.com/services/isbnservice.wso?WSDL',YYY())
print client.IsValidISBN13('0000000000000')
print client.IsValidISBN13('9780061374227')

Python read command line options

Here is an example of how to read command line arguments with Python.


import getopt, sys

def main():
try:
opts, args = getopt.getopt(sys.argv[1:], ":i:r:")
except getopt.GetoptError, err:
# print help information and exit:
print str(err) # will print something like "option -a not recognized"
usage()
sys.exit(2)
output = None
verbose = False
for opt in opts:
if opt[0] == "-i":
id = opt[1]
if opt[0] == "-r":
root = opt[1]
print id
print root
if __name__ == "__main__":
main()


Python web service using SOAPPy

Here is an example of how to call a WebService from Python. For more information on Python and Web Services I recommend the following book: Python Cookbook.

This requires the fpconst and SOAPPy package. You can download them from http://research.warnes.net/projects/RStatServer/fpconst/ and http://prdownloads.sourceforge.net/pywebsvcs/SOAPpy-0.11.6.tar.gz?download. Unzip the packages then run python setup.py install on each package.

The following code will then connect to the web service and execute the remote method.

import sys

from SOAPpy import WSDL

url = 'http://webservices.daehosting.com/services/isbnservice.wso?WSDL'
server = WSDL.Proxy(url);
print server.IsValidISBN13(sISBN ='0000000000000')


Note there is another web service package called ZSI, I found SOAPPy to be easier to use, but from what I understand both packages are going to merge together.

Python database access

Here is an example of how to connect to a MysqlDB using python.

#!/usr/bin/python

import MySQLdb
import sys

class Table:
    def __init__(self, db, name):
        self.db = db
        self.name = name
        self.dbc = self.db.cursor()

    def execute(self):
        query = "select 1,2 from you_table "
        self.dbc.execute(query)
        return self.dbc.fetchall()

    def next(self):
        return self.dbc.fetchone()

db = MySQLdb.connect(host="<your host>",user="<your username>", db="<your db>", passwd=<your passwd>)
event_table = Table(db, "event_table")

records = event_table.execute()
for r in records:
    print str(r[0])+","+str(r[1])

Python mysql access

Here is how you can access mysql using python:

#!/usr/bin/python

import MySQLdb
import sys
import time

db = MySQLdb.connect(host="reportsdb",user="root", db="tool_metrics")
dbc = db.cursor()
str = "select '1','2','3'";
dbc.execute(str)
records = dbc.fetchall()

for r in records:
    print r[0]+","+r[1]+","+r[2]

For more information on Python and databases I recommend the following book for futher reading: Python Pocket Reference (Pocket Reference (O'Reilly))