Django->internationalization

I've decided to put more detailed information about multilingual pages.

So, how they are implemented in Django and how make it work. Let's do step-by-step.
0.settings.py:
add context processors, it should be something like that:
TEMPLATE_CONTEXT_PROCESSORS = (
    "django.contrib.auth.context_processors.auth",
    "django.core.context_processors.i18n",
    "django.core.context_processors.request",
    "django.core.context_processors.media",
    "django.core.context_processors.csrf",
)

and middleware:
MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',   
    'django.middleware.locale.LocaleMiddleware',
)

and languages:
LANGUAGES = (
        ('ru','Russian'),
        ('en','English'),
)
 

1.Add trans or blocktrans to html you want to be transalted:
e.g., {% trans "Home" %}
2. also, add {% load i18n %} to the beginning of html (usually it's "base.html").
3. Generate dictionaries (.po):
3.1 Set PYTHONPATH (export PYTHONPATH=/path/to/your/project/)
3.2 Go to your project/application directory, create there directiry locale and execute: django-admin.py makemessages -l ru . It will generate .po files in locale/ru/LC_MESSAGES
open the file and add translation. It will look like:
#: templates/flatpages/default.html:15
msgid "About"
msgstr "О нас"
3.3 Compile the files:
django-admin.py compilemessages
If you update the files you should use:makemessages -a
4. Add form to switch languges:
html:

        {% csrf_token %}

         <input name="next" type="hidden" value="/" /> 
            <select name="language">
                             {% for lang in LANGUAGES %}
                 <option value="{{ lang.0 }}"<{{ lang.1 }}>
             {% endfor %}
              </select> 
             <input type="submit" value="Go" />
             </form>

Urls.py: (r'^i18n/', include('django.conf.urls.i18n')),

django:makemessages encoding problem

Tried to execute
 django-admin.py makemessages -l ru

Got error:
Error: errors happened while running xgettext on mogrify.py
xgettext: ./build/psycopg2/examples/mogrify.py:2: Unknown encoding "utf8". Proceeding with ASCII instead.
xgettext: Non-ASCII string at ./build/psycopg2/examples/mogrify.py:37.
          Please specify the source encoding through --from-code or through a comment
          as specified in http://www.python.org/peps/pep-0263.html.

Reason and solution:
xgettext doesn't accept 'utf8', it accepts only 'utf-8'. The same with 'latin-1', it should be replaced with 'ISO-8859-1'.

So, the solution: you should check all your files inside your project or application directory if there are 'utf8' (or 'latin-1') and replace them with 'utf-8'('ISO-8859-1').

Cyrillic support in Latex (ubuntu version)

Error in TexMaker: Package babel Error: No Cyrillic encoding definition files were found.

Solution:
install package: texlive-lang-cyrillic

apt-get install texlive-lang-cyrillic

Python: cyrillic to lowercase/uppercase

 print unicode("ПРИВЕТ", encoding='utf-8').lower()

open mdb access file using python

import csv
import pyodbc

MDB = 'c:/path/to/my.mdb'
DRV = '{Microsoft Access Driver (*.mdb)}'
PWD = 'mypassword'

conn = pyodbc.connect('DRIVER=%s;DBQ=%s;PWD=%s' % (DRV,MDB,PWD))
curs = conn.cursor()

SQL = 'SELECT * FROM mytable;' # insert your query here
curs.execute(SQL)

rows = curs.fetchall()

curs.close()
conn.close()

# you could change the 'w' to 'a' for subsequent queries
csv_writer = csv.writer(open('mytable.csv', 'w'), lineterminator='\n')

for row in rows:
    csv_writer.writerow(row)

awk: calculating frequency

Input:
иностранный язык 311
родной язык 226
настоящий друг 215
лаконичный ответ 204
милый друг 197
лучший друг 193
громкий голос 183
трава зеленая 171
упрямый осел 169
снег белый 158
передать привет 158
история учебник 13
история страны 13
история партии 13
истинный патриот 13
истинный друг 13
иностранный агент 13


output:
иностранный язык 311 0.959877
родной язык 226 1
настоящий друг 215 1
лаконичный ответ 204 1
милый друг 197 1
лучший друг 193 1
громкий голос 183 1
трава зеленая 171 1
упрямый осел 169 1
снег белый 158 1
передать привет 158 1
история учебник 13 0.333333
история страны 13 0.333333
история партии 13 0.333333
истинный патриот 13 0.5
истинный друг 13 0.5
иностранный агент 13 0.0401235

awk: awk -F' ' 'NR==FNR{a[$1]+=$3;next}{print($0,$3/a[$1])}' test test

Java: setting file encoding

Input: UTF8 file (cyrillic)

File file = new File("file_utf8");
StringBuffer buffer = new StringBuffer();
try
{
FileInputStream fin=null;
try {
fin = new FileInputStream(file);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
InputStreamReader isr = new InputStreamReader(fin,"UTF8");

BufferedReader in= new BufferedReader(isr);
int ch;
while ((ch = in.read()) > -1) {
buffer.append((char)ch);
}
isr.close();
fin.close();
String[] lines=buffer.toString().split("\n");
for (String line : lines)
System.out.println(line);
}
catch (IOException e) {
e.printStackTrace();
}

OrientDB: console

1. Execute console: /bin/./console.sh
2. Create database:
create database database-url user password storage-type
Example:

create database local:/usr/local/orient/databases/demo/demo admin admin local

3. Connect to database:
connect database-url user-name user-password

connect local:../databases/demo/demo admin admin 

4. Select:
select conditions

select from person 

5. Insert:
insert into class|cluster:cluster (field-name*) values (field-value)

insert into Profile (name, surname) values ('Jay', 'Miner' ) 

6. Delete:
delete from class|cluster:cluster [where conditions]

delete from Profile where nick is null

delete from OGraphVertex where id = 0

7. Entities:
classes

Django: multi language

1. FlatPages

This approach doesn't work any more (Django > 1.2) !

svn checkout http://django-multilingual.googlecode.com/svn/trunk/ django-multilingual-read-only
cd django-multilingual-read-only
sudo python setup.py install
Follow the instructions: 
http://devdoodles.wordpress.com/2009/02/26/static-content-with-django-multilingual-flatpages/
Check that SITE ID is correct!
2. Add dictionaries
1. Add trans or blocktrans to html you want to be transalted:
e.g., {% trans "Home" %}
2. also, add {% load i18n %} to the beginning of html.


3. Generate dictionaries (.po):
3.1 Set PYTHONPATH (export $PYTHONPATH=/path/to/your/project/)
3.2 Execute: django-admin.py makemessages -l ru . It will generate .po files in locale/ru/LC_MESSAGES
open the file and add translation. It will look like:
#: templates/flatpages/default.html:15
msgid "About"
msgstr "О нас"
3.3 Compile the files:
django-admin.py compilemessages
If you update the files you should use:makemessages -a
4. Add form to switch languges:
html:

        <form action="/i18n/setlang/" method="post"> 
         <input name="next" type="hidden" value="/" /> 
            <select name="language">
          {% for lang in LANGUAGES %
                 <option value="{{ lang.0 }}"<{{ lang.1 }}>
             {% endfor %}
              </select> 
             <input type="submit" value="Go" />
             </form>

Urls.py: (r'^i18n/', include('django.conf.urls.i18n')),

Settings.py:
LANGUAGES = (
        ('ru','Russian'),
        ('en','English'),
)

How the show all special chracters in VI

vi file

:set list - turns on
:set nolist - turns off

awk filtering and soring by parameter

file:
aaa \t bbb \t 3
bbb \t ssd \t 4

Exclude rows with $3 (3rd parameter) <=1 and sort it by 3rd column:

awk -F"\t" '$3>1 {print}' test | sort -n -r -t'     ' -k3> test_filter_gt1

Get number of different words in the 2nd column:
awk -F"      " '{print $2}' test_filter_gt1 | sort  | uniq -c -i | wc -l

Change file encoding in linux

1. Get encoding:

file -bi test.txt
text/plain; charset=us-ascii

2. Change it:

iconv -f ascii -t utf8 [filename] > [newfilename]

R: non-linear regression with nls

dir_degree<-read.table("/home/bliss/Study/Bauman/Magistr&Disser/Data/degrees_undir",sep='\t')
dir_degree<-as.data.frame(dir_degree)
degrees<-dir_degree$V2
#calculate frequences
distr<-tabulate(degrees)
# sort by frequency
distr<-sort(distr,decreasing=T)
#calculate probabilities
prob<-distr/sum(distr)

attach(as.data.frame(prob))
x<-c(1:length(prob))

#non-linear regression
model<-nls(prob ~ alfa*x^(-gamma), start=list(alfa=-10, gamma=-1/4), algorithm="port", trace=T)

awk , add column to file

Add column with line number to file:

awk '{print NR"\t"$0}' degrees_dir > degrees_dir_order

After:

head degrees_dir_order
1 669
2 667
3 600
4 596
5 580
6 549
7 490
8 460
9 454
10 447

Graphs in python

Using networkx (http://networkx.lanl.gov/)

Loading data from file (format: one\t two\t23\n):

ass_base={}
ass_file=open("avs_weights_utf8.txt",'r')
for line in ass_file.readlines():
parts=line.split('\t')
ass_base[parts[0]+'\t'+parts[1]]=parts[2].replace('\n','')

import networkx as nx
graph=nx.DiGraph()

for key in ass_base.keys():
parts=key.split('\t')
graph.add_weighted_edges_from([(parts[0],parts[1],float(ass_base[key]))])


#average node degree
round( sum([d[1] for d in graph.degree_iter()])/float(len(graph)), 4)

#get all degrees
degrees=sorted(degrees.values(), reverse=True)