Python: encoding/decoding a string starting with \x

If a string starts with \x (e.g., '\xea \xe1\xeb\xe8\xe6\xed\xe5\xec\xf3' for cyrillic) decode it using 'string_escape' :

string=string.decode('string_escape')

Python: cyrillic to lowercase/uppercase

 print unicode("ПРИВЕТ", encoding='utf-8').lower()

open mdb access file using python

import csv
import pyodbc

MDB = 'c:/path/to/my.mdb'
DRV = '{Microsoft Access Driver (*.mdb)}'
PWD = 'mypassword'

conn = pyodbc.connect('DRIVER=%s;DBQ=%s;PWD=%s' % (DRV,MDB,PWD))
curs = conn.cursor()

SQL = 'SELECT * FROM mytable;' # insert your query here
curs.execute(SQL)

rows = curs.fetchall()

curs.close()
conn.close()

# you could change the 'w' to 'a' for subsequent queries
csv_writer = csv.writer(open('mytable.csv', 'w'), lineterminator='\n')

for row in rows:
    csv_writer.writerow(row)

Graphs in python

Using networkx (http://networkx.lanl.gov/)

Loading data from file (format: one\t two\t23\n):

ass_base={}
ass_file=open("avs_weights_utf8.txt",'r')
for line in ass_file.readlines():
parts=line.split('\t')
ass_base[parts[0]+'\t'+parts[1]]=parts[2].replace('\n','')

import networkx as nx
graph=nx.DiGraph()

for key in ass_base.keys():
parts=key.split('\t')
graph.add_weighted_edges_from([(parts[0],parts[1],float(ass_base[key]))])


#average node degree
round( sum([d[1] for d in graph.degree_iter()])/float(len(graph)), 4)

#get all degrees
degrees=sorted(degrees.values(), reverse=True)

Python Regular Expressions

E.g. we'd like to parse such html-code using regexp:
 <tr><td><font color="#bbbbbb">5587  </font></td><td>изумление</td><td>S</td><td>13.98</td><td>20.65</td>
## <td>#N/A</td><td>#N/A</td><td>#N/A</td><td>#N/A</td><td>#N/A</td><td>#N/A</td><td>#N/A</td><td></td></tr>


The code will be:

rows=re.finditer('(\<tr.+?tr\>)',html) ##nejadnyi (v otlichie ot .+ ischet stroki ne maxim dliny)
for row in rows:
cells=re.finditer('(\<td.+?td\>)(\<td.+?td\>)(\<td.+?td\>)(\<td.+?td\>)(\<td.+?td\>)(\<td.+?td\>)(\<td.+?td\>)(\<td.+?td\>)(\<td.+?td\>)(\<td.+?td\>)(\<td.+?td\>)(\<td.+?td\>)',row.groups()[0])


Round brackets '(', ')' means group, you may iterate or name them.
\ - read as it is
.+ - find any string (any symbols), finds string with maximum length and takes a lot of sources
.+? - find any string (any symbols), not maximum length , better one to parse constructions like
<tr>...</tr>..<tr>...</tr>







'(\<tr.+tr\>)', finds  ({<tr>...</tr>..<tr>...</tr>}), only one
'(\<tr.+?tr\>)', finds  ({<tr>...</tr>},{<tr>...</tr>})



Primitive function to remove html tags:
def remove_tags(html): pattern=re.compile('<.*?>')  
result=pattern.sub("",html)  
return result

Find string that doesn't contain symbol (e.g. '{'):
re.finditer('({[^}]+})', str)

MySQL Database scheme

I've decided to work with utf-8 encoding instead of cp1251.
First, set it:

SET names 'utf8';
DROP TABLE IF EXISTS weights;

DROP TABLE IF EXISTS cues;
DROP TABLE IF EXISTS reacts;

 CREATE TABLE cues
       (
         id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
         data VARCHAR(50) collate  utf8_general_ci
       )
CREATE TABLE reacts
       (
         id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
         data VARCHAR(100) collate utf8_general_ci
       )
CREATE TABLE weights
       (
         cue_id INT NOT NULL,
         react_id INT NOT NULL,
     weight DOUBLE NOT NULL,
     FOREIGN KEY (cue_id) REFERENCES cues(id),
     FOREIGN KEY (react_id) REFERENCES reacts(id),
     PRIMARY KEY(cue_id, react_id)
       )

Python:
# -*- coding: utf-8 -*- 
import MySQLdb


##connection:
conn = MySQLdb.connect (host = "localhost",
                        user = "user1",
                         passwd = "pass1",
                         db = "db_name")
cursor = conn.cursor ()
cursor.execute ("SET names 'utf8'")
....
cursor.execute ("SELECT id FROM cues WHERE data="+"\""+cue+"\"")   
    cue_id=cursor.fetchone ()[0]
..
cursor.close()
conn.close()

MySQL + Python

I've decided to work with datatables instead of text files.
Guide for ubuntu (9.10, 10.4):
If you have mysql & python installed, you need:
1. Download MySQLdb files from SourceForge
2. tar -xzvf MySQL-python-1.2.3.tar.gz
3. cd MySQL-python-1.2.3
4. python setup.py build
Here there might be some errors:
 File "setup.py", line 5, in
    from setuptools import setup, Extension
ImportError: No module named setuptools
To fix:  sudo apt-get install python-setuptools python-dev libmysqlclient15-dev
or EnvironmentError: mysql_config not found
To fix: export PATH=$PATH:/usr/local/mysql/bin
5. Again: python setup.py build
6. sudo python setup.py install
7. Check: python
>>> import MySQLdb

Connect to MySQL using bash:
mysql -h localhost -u root -p

Create database: create database my_db;


Script example:
import MySQLdb

conn = MySQLdb.connect (host = "localhost",
                         user = "root",
                         passwd = "my_ps",
                         db = "my_db")
cursor = conn.cursor ()
cursor.execute ("SELECT VERSION()")
row = cursor.fetchone ()
print "server version:", row[0]
cursor.close ()
conn.close ()
Manual: http://www.kitebird.com/articles/pydbapi.html