Month: July 2012

Log Parser

There are some amazing utilities available which never seem to get much recognition. One of these in my opinion is Microsoft’s Log Parser; you can download v2.2 from here and find an introduction here. It gives you the ability to use SQL-like queries on common log files. While it is presented as a command line program most of the work is done by the accompanying DLL so would it be possible to access it from Python.

Looking in the Log Parser help file it documents a COM interface which gives us our way in. Before you can use the COM interface you must register the DLL with the following command

regsvr32 LogParser.dll

All the examples are in VBScript unsurprisingly but we can use them with a bit of trial and error. The first item of interest is the name of the COM object, MSUtil.LogQuery. Now we know what to pass to the dispatch method. If you have used a library that confirms to the Python database API think of this as the connect method.

Next comes the two ways of working, batch or interactive. Interactive seems the most natural place to start so we need to use the Execute method. This takes the SQL-like query and optionally the input format. We’ll leave it to guess the input format for this test.

The execute method returns a LogRecordSet. Here is where the VBScript example helps. This is an interface to a set a records (obviously) and allows us to find out the columns and get the records. At a stretch you can think of it as cursor. Putting all this together we can query the System event log to find out all the times the computer was switched on (Event ID 12) with the following code

import win32com.client
lp = win32com.client.Dispatch("MSUtil.LogQuery")
sql = "SELECT * FROM System WHERE EventID = 12"
lrs = lp.Execute(sql)
for col in range(rs.getColumnCount()):
    print col, rs.getColumnName(col)

Now we have the result of query in the record set we need to get at the values. We can get each record sequentially using the record set interface. This returns a LogRecord object which basically just allows you to get each value by specifying the index you want. The example above gives us two columns we could use, I’ll use the TimeGenerated one. So lets display them

# continuing on from above
while not lrs.atEnd():
    lr = lrs.getRecord()
    print "Computer started at",lr.getValue(2)
    lrs.moveNext()

With a working COM interface, I wrote the following program to allow the SQL statements to be run from the command line in an interactive environment. The output formatting is poor but it makes a good example program.

Advertisements

Modules

At the risk of simply repeating the document on modules, to run a method from a different python file you can use either of the following code snippets. The first gives you access to all the methods and variables inside of the file (note you don’t need the file extension) from within its own namespace. The second just the method you requested inside of your own namespace.

# Want access to all of the file's methods
import file
file.method_name()

# Just want a single method without anything
from file import method_name
method_name()

Both statements allow an optional as command to change the name. In the first case this changes the namespace (more on this later). The second changes the the name of the reference name for the method or variable. You will see the from … import a lot in code on the Internet although I tend to stay clear of it. There are a few things to be aware of

# Imports everything (with caveat) from file
# overwrites any object with the same name you already had
from file import *

# does not work
from file import method_name,another_method as new_name,another_name

# new_name refers to another_method, method_name is no longer available
from file import method_name,another_method as new_name

However how do you get access to a python file that is not in the same directory as calling python script, or the PYTHONPATH environment variable / registry value? There are two variations.

The paths search are held in a list object called sys.path and can be manipulated at runtime. Just add the your required path to this list. Don’t replace the list or you’ll lose access to all your standard libraries. As an example, the following code allows you to import any python file from either C:\PythonModules or the modules directory off the

import sys,os
sys.path.append(r"C:\PythonModules")
# getcwd gets the current working directory and add modules directory
sys.path.append(os.path.join(os.getcwd(),"modules"))

If the file you wanted to import was in a sub-directory that is already in your search path you can use the package notation. This works no matter how deep inside the directory structure the file is. So you could a import a file from the sub-directory modules \ local \ custom with the following code. Notice as gives you a shortcut rather than typing in the full namespace each time.

import modules.local.custom.file as mymod
mymod.mymethod()

The limitation of this method is that each directory will need a __init__.py file in each directory. In the above example there would have to be a  __init__.py file the modules, local and custom directories. This file can be empty or can contain initialisation code where required but if it does not exist, the directory will not be searched.

Python 3 users also note that importing a module creates a __pycache__ directory in the files location where it stores the compiled .pyc file rather than storing it in the same directory as the file which it what happened previously. So in Python 3 the above would create __pycache__ directories in modules, local and custom.

Proxies

Accessing information via a URL is easily done using urllib but it is not always able to automatically detect proxy settings. Manually setting a proxy is not that difficult, it even handles authentication.

<br /># Python 3 use urllib.request<br />import urllib<br />proxy = { 'http': 'http://proxy.server:port' }<br />conn = urllib.FancyURLopener(proxy)<br />with conn.open("http://www.bbc.co.uk/") as page:<br /><%%KEEPWHITESPACE%%>    print "Header\n",page.info(),"\nPage\n",page.read()<br />

The proxy passed is a dictionary where the key is the protocol and value is the proxy. Different protocols can have different proxies although this is unlikely in practice.

Once you have connection, you can use the open command and use it a file. If the with statement seems unusual, see this post for an explanation.

Python 3 users note that data returned from the various read methods will return binary data just the same as a file read would. You can convert this to a (unicode) string with calling str.decode()

If you are only using HTTP and you need more control over what is being sent / received you can use httplib. The added power creates added complexity but to add a proxy pass it to the connection method as shown

<br />import httplib<br />conn = httplib.HTTPConnection("proxy.server:port") # or "proxy.server",port<br />conn.request("GET","http://www.bbc.co.uk/")<br />resp = conn.getresponse()<br />print resp.status<br />conn.close()<br />

Accessing COM objects

One powerful feature of Windows which can get overlooked is the COM method of accessing other programs. This defines the standard application binary interface so that programs can communicate and use each other almost as you would a library. The interface is language neutral and can be used from Python just as well as any other language.

COM objects use the client-server model. The program being accessed, or library in the above analogy, is the server and the program doing the calling is the client. Most scripts are clients which use various COM servers already installed on the computer (or other computers on the network). To get a reference to the COM server you use the Dispatch method.

For example I can use COM to start Microsoft Word (assuming I have Word installed on the computer) and open a document with the following script

import win32com.client
app = win32com.client.Dispatch("Word.Application")
doc = app.Documents.Open("file.doc")
app.Visible = True

Four lines of code and I have a word processor! The strength of COM is also its weakness, it only defines how the programs talk; it doesn’t define the methods or properties, what is called the application programming interface. That is left up to the server.

So how did I know about the Documents.Open method or the Visible property? In an ideal world the API would be fully documented and available. In reality there is little or no documents or it is not available or it is difficult to make sense of. If the COM server has been registered in a standard way you can use a COM browser to gain information.

A basic browser comes with pywin32. A more powerful browser, oleview, can be downloaded from Microsoft as part of the Windows Server 2003 Resource Kit Tools. If working with Microsoft Office COM objects, my preferred browser is actually the Object Browser that comes with VBA. Just press Alt + F11 from Word and then F2. I find this has the cleanest interface.

The last way of making a start with a COM interface is to simply dump everything that can be found out about the API into a file and use that as a starting point. You can then use this information with the interactive prompt to query the COM object and see what it returns. The program do this is makepy.py which can be found in the win32client\client directory. I’ll hopefully cover using makepy in another article.

Using WITH

It is not obvious what benefit the with statement brings. For example the preferred way of opening a file is by using the with statement rather than than an assignment as shown below.

# preferred method
with open('filename.ext') as myfile1:
  data1 = myfile1.read() # do something

# old method
myfile2 = open('filename.ext')
data2 = myfile2.read() # do something
myfile2.close()

In other languages with is little more than a shortcut to save typing in fully qualified names but that is not happening here. So why is this preferred? Those paying attention will have noticed I did not close myfile1. This is not a typo.

The reason for using the with command is it ensures that a cleanup of the variable is done once it is finished with. If in the old method the read caused an exception myfile2 would have been left open. Using with, the cleanup is done automatically whether the end of the block is reached or an exception occurs. For the open class, cleanup is naturally to close the file.

You can of course achieve the same result using try at the expense of several ugly lines which does nothing for readability. For those looking for a more technical explanation and how to make a class that can be used with the with statement have a look at Fredrik Lundh’s post here.

One final note is for Python 2.5 users who will not be able to run the code fragment above as it stands. Although the with statement was added in 2.5 it resides in __future__ module and needs to be imported. All later versions include the with statement. If you know the version you are programming for you can just import the statement, otherwise you can check at runtime and import it only if necessary with the following code.

import sys
if sys.version_info[0:2] == (2,5):
    # Running on python 2.5 - need to import with statement
    from __future__ import with_statement

Another WMI tool

Having not so long ago wrote the post about WMI Code Creator I find there is another Microsoft tool called Scriptomatic 2 that does a very similar job and lists Python as a supports language. Again the download just extracts to a runnable file so no installer needed. My only issue was that it’s an hta file rather an executable and needs to be run an an administrator.

The screen layout is a little more efficient (click on the thumbnailScriptomatic 2 showing the processor usage on the right to get a full sized image). The Namespace defaults to root\CIMV2 and you can select the class from the drop down list the same as Code Creator.

The languages supported does include Python but uses win32com.client directly and produces a massive amount of code to do what my previous post did in 5 lines. Still you can just copy the WQL query and use it in your code.

What Scriptomatic gives you that Code Creator doesn’t the ability to run the WMI query and export it. Simply select the output format on the right and click Run. You may want to set the language to VBScript if you have problems. Also by changing the Target Computers list at the bottom you can query other computers rather than just the local one.

WMI explored

The biggest hurdle to using WMI I found was knowing where the information you want is stored. There are various browsers available but one quick and simple entry point is Microsoft’s WMI Code Creator. It extracts to an runnable executable so no installer needed. While it is designed for VB and C# users, if you ignore most of the code it creates and just grab WQL statement I found it one of the quickest ways of getting up and running.

What lead to code creator was a need to monitor the process usage. I suspected correctly I could get this information from the WMI but had no idea how.

Starting WMI Code Creator and using the default namespace of root\CIMV2 you can use the drop down list of classes to browse what is available. The first one I tried was Win32_Processor. Selecting the class gives you the properties. Select the properties (I just select every one) and clicking Search for Property Values gives you a list of the values. Needless to say Win32_Processor gives you the processors installed, not quite what I want.

Next class to try is the long winded Win32_PerfFormatteThe WMI Code Creator in usedData_PerfOS_Processor. This looked more promising, as the thumbnail to the right shows (click for larger image). Getting the properties values brought back far more data than I expected and a slight surprise; I had forgot I had a dual core processor.

Because I have more than one core, the query returns 3 sets of data, one for each processor and a total. WMI Code Creator indents the even sets which helps a little but there are better methods.

The code created in the right hand pane is of little use in Python but you can copy and paste the WQL query to save some time. Time to get this query into Python so we can have a play.

import wmi
mywmi = wmi.WMI()
wql = "SELECT * FROM Win32_PerfFormattedData_PerfOS_Processor"
for item in mywmi.query(wql):
    print(item)

Wow, that’s a lot of information. At this point the documentation comes to the rescue (enter the class name into a search engine). It looks like PercentProcessorTime is the property I need. So with a quick update gives me a very crude processor monitor

import wmi, time
mywmi = wmi.WMI()
wql = 'SELECT * FROM Win32_PerfFormattedData_PerfOS_Processor WHERE Name = "_Total"'
while True:
    for item in mywmi.query(wql):
        print(item.PercentProcessorTime)
    time.sleep(1)