Working with Giza++

1/ Edit file_spec.h to

     struct tm *local;
     time_t t;
     char *user;
     char time_stmp[19];
     char *file_spec = 0;

     t = time(NULL);
     local = localtime(&t);

     sprintf(time_stmp, "%04d-%02d-%02d.%02d%02d%02d.", 1900 +  local->tm_year,
         (local->tm_mon + 1), local->tm_mday, local->tm_hour,
         local->tm_min, local->tm_sec);
     user = getenv("USER");
  file_spec = (char *)malloc(sizeof(char) *
                 (strlen(time_stmp) + strlen(user) + 1));
  file_spec[0] = '\0';
  strcat(file_spec, time_stmp) ;
  strcat(file_spec, user);
  return file_spec;

2/

$ cp Makefile Makefile.orig
$ sed -i 's/ -DBINARY_SEARCH_FOR_TTABLE//;s/mkdir/mkdir -p/g' Makefile
$ make

3/ Run the scripts: 

plain2snt.out sv-text.txt da-text.txt 

GIZA++ -S sv-text.vcb -T da-text.vcb -C sv-text_da-text.snt  

Tutorial on distributional semantics (vector space models)

A nice tutorial by Chris Potts :

http://web.stanford.edu/class/linguist236/materials/ling236-handout-05-09-vsm.pdf

Python: encoding/decoding a string starting with \x

If a string starts with \x (e.g., '\xea \xe1\xeb\xe8\xe6\xed\xe5\xec\xf3' for cyrillic) decode it using 'string_escape' :

string=string.decode('string_escape')

Lowercase strings in file

 Lowercase strings in text file using sed(works for cyrillic):
 sed -e 's/\(.*\)/\L\1/' file

Increase heap size in weka

java -Xmx600M -jar /usr/share/java/weka.jar

Copy files from remote server

scp -P port user@server:/path_to_the_file/file  /path_to_local_dir/
get file size:
ls -sh file

Select from Many-to-Many with GroupBy

Three entities: 
Cue(id, name) 
Reaction(id, name) 
Freqs(id, cue_id FK, reaction_id FK, some_other_data)

Get most frequent reactions and their frequency:
Select name from rdb_reaction 
where id in (SELECT reaction_id as freq
  FROM rdb_formdata group by reaction_id order by count(*) desc limit 20);

Count number of (cue, reaction) pairs like (cue.name, reaction.name, count):

SELECT distinct
  c.name cue_name,
  b.name reaction_name, 

 (SELECT
  Count(*)
  FROM 
  rdb_freqs a1
  where 
  a1.cue_id =  a.cue_id and 
  a1.reaction_id = a.reaction_id 
  ) as count
FROM 
  rdb_freqs a 
  inner JOIN rdb_reaction b ON a.reaction_id = b.id
  inner JOIN rdb_cue c ON a.cue_id = c.id
  order by count desc

How to fix slow 4G/3G internet on ubuntu

sudo ifconfig wlan0 down
sudo rmmod -f ath9k
sudo modprobe ath9k nohwcrypt=1
sudo ifconfig wlan0 up

Django->internationalization

I've decided to put more detailed information about multilingual pages.

So, how they are implemented in Django and how make it work. Let's do step-by-step.
0.settings.py:
add context processors, it should be something like that:
TEMPLATE_CONTEXT_PROCESSORS = (
    "django.contrib.auth.context_processors.auth",
    "django.core.context_processors.i18n",
    "django.core.context_processors.request",
    "django.core.context_processors.media",
    "django.core.context_processors.csrf",
)

and middleware:
MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',   
    'django.middleware.locale.LocaleMiddleware',
)

and languages:
LANGUAGES = (
        ('ru','Russian'),
        ('en','English'),
)
 

1.Add trans or blocktrans to html you want to be transalted:
e.g., {% trans "Home" %}
2. also, add {% load i18n %} to the beginning of html (usually it's "base.html").
3. Generate dictionaries (.po):
3.1 Set PYTHONPATH (export PYTHONPATH=/path/to/your/project/)
3.2 Go to your project/application directory, create there directiry locale and execute: django-admin.py makemessages -l ru . It will generate .po files in locale/ru/LC_MESSAGES
open the file and add translation. It will look like:
#: templates/flatpages/default.html:15
msgid "About"
msgstr "О нас"
3.3 Compile the files:
django-admin.py compilemessages
If you update the files you should use:makemessages -a
4. Add form to switch languges:
html:

        {% csrf_token %}

         <input name="next" type="hidden" value="/" /> 
            <select name="language">
                             {% for lang in LANGUAGES %}
                 <option value="{{ lang.0 }}"<{{ lang.1 }}>
             {% endfor %}
              </select> 
             <input type="submit" value="Go" />
             </form>

Urls.py: (r'^i18n/', include('django.conf.urls.i18n')),

django:makemessages encoding problem

Tried to execute
 django-admin.py makemessages -l ru

Got error:
Error: errors happened while running xgettext on mogrify.py
xgettext: ./build/psycopg2/examples/mogrify.py:2: Unknown encoding "utf8". Proceeding with ASCII instead.
xgettext: Non-ASCII string at ./build/psycopg2/examples/mogrify.py:37.
          Please specify the source encoding through --from-code or through a comment
          as specified in http://www.python.org/peps/pep-0263.html.

Reason and solution:
xgettext doesn't accept 'utf8', it accepts only 'utf-8'. The same with 'latin-1', it should be replaced with 'ISO-8859-1'.

So, the solution: you should check all your files inside your project or application directory if there are 'utf8' (or 'latin-1') and replace them with 'utf-8'('ISO-8859-1').

Cyrillic support in Latex (ubuntu version)

Error in TexMaker: Package babel Error: No Cyrillic encoding definition files were found.

Solution:
install package: texlive-lang-cyrillic

apt-get install texlive-lang-cyrillic

Python: cyrillic to lowercase/uppercase

 print unicode("ПРИВЕТ", encoding='utf-8').lower()

open mdb access file using python

import csv
import pyodbc

MDB = 'c:/path/to/my.mdb'
DRV = '{Microsoft Access Driver (*.mdb)}'
PWD = 'mypassword'

conn = pyodbc.connect('DRIVER=%s;DBQ=%s;PWD=%s' % (DRV,MDB,PWD))
curs = conn.cursor()

SQL = 'SELECT * FROM mytable;' # insert your query here
curs.execute(SQL)

rows = curs.fetchall()

curs.close()
conn.close()

# you could change the 'w' to 'a' for subsequent queries
csv_writer = csv.writer(open('mytable.csv', 'w'), lineterminator='\n')

for row in rows:
    csv_writer.writerow(row)

awk: calculating frequency

Input:
иностранный язык 311
родной язык 226
настоящий друг 215
лаконичный ответ 204
милый друг 197
лучший друг 193
громкий голос 183
трава зеленая 171
упрямый осел 169
снег белый 158
передать привет 158
история учебник 13
история страны 13
история партии 13
истинный патриот 13
истинный друг 13
иностранный агент 13


output:
иностранный язык 311 0.959877
родной язык 226 1
настоящий друг 215 1
лаконичный ответ 204 1
милый друг 197 1
лучший друг 193 1
громкий голос 183 1
трава зеленая 171 1
упрямый осел 169 1
снег белый 158 1
передать привет 158 1
история учебник 13 0.333333
история страны 13 0.333333
история партии 13 0.333333
истинный патриот 13 0.5
истинный друг 13 0.5
иностранный агент 13 0.0401235

awk: awk -F' ' 'NR==FNR{a[$1]+=$3;next}{print($0,$3/a[$1])}' test test

Java: setting file encoding

Input: UTF8 file (cyrillic)

File file = new File("file_utf8");
StringBuffer buffer = new StringBuffer();
try
{
FileInputStream fin=null;
try {
fin = new FileInputStream(file);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
InputStreamReader isr = new InputStreamReader(fin,"UTF8");

BufferedReader in= new BufferedReader(isr);
int ch;
while ((ch = in.read()) > -1) {
buffer.append((char)ch);
}
isr.close();
fin.close();
String[] lines=buffer.toString().split("\n");
for (String line : lines)
System.out.println(line);
}
catch (IOException e) {
e.printStackTrace();
}

OrientDB: console

1. Execute console: /bin/./console.sh
2. Create database:
create database database-url user password storage-type
Example:

create database local:/usr/local/orient/databases/demo/demo admin admin local

3. Connect to database:
connect database-url user-name user-password

connect local:../databases/demo/demo admin admin 

4. Select:
select conditions

select from person 

5. Insert:
insert into class|cluster:cluster (field-name*) values (field-value)

insert into Profile (name, surname) values ('Jay', 'Miner' ) 

6. Delete:
delete from class|cluster:cluster [where conditions]

delete from Profile where nick is null

delete from OGraphVertex where id = 0

7. Entities:
classes

Django: multi language

1. FlatPages

This approach doesn't work any more (Django > 1.2) !

svn checkout http://django-multilingual.googlecode.com/svn/trunk/ django-multilingual-read-only
cd django-multilingual-read-only
sudo python setup.py install
Follow the instructions: 
http://devdoodles.wordpress.com/2009/02/26/static-content-with-django-multilingual-flatpages/
Check that SITE ID is correct!
2. Add dictionaries
1. Add trans or blocktrans to html you want to be transalted:
e.g., {% trans "Home" %}
2. also, add {% load i18n %} to the beginning of html.


3. Generate dictionaries (.po):
3.1 Set PYTHONPATH (export $PYTHONPATH=/path/to/your/project/)
3.2 Execute: django-admin.py makemessages -l ru . It will generate .po files in locale/ru/LC_MESSAGES
open the file and add translation. It will look like:
#: templates/flatpages/default.html:15
msgid "About"
msgstr "О нас"
3.3 Compile the files:
django-admin.py compilemessages
If you update the files you should use:makemessages -a
4. Add form to switch languges:
html:

        <form action="/i18n/setlang/" method="post"> 
         <input name="next" type="hidden" value="/" /> 
            <select name="language">
          {% for lang in LANGUAGES %
                 <option value="{{ lang.0 }}"<{{ lang.1 }}>
             {% endfor %}
              </select> 
             <input type="submit" value="Go" />
             </form>

Urls.py: (r'^i18n/', include('django.conf.urls.i18n')),

Settings.py:
LANGUAGES = (
        ('ru','Russian'),
        ('en','English'),
)

How the show all special chracters in VI

vi file

:set list - turns on
:set nolist - turns off

awk filtering and soring by parameter

file:
aaa \t bbb \t 3
bbb \t ssd \t 4

Exclude rows with $3 (3rd parameter) <=1 and sort it by 3rd column:

awk -F"\t" '$3>1 {print}' test | sort -n -r -t'     ' -k3> test_filter_gt1

Get number of different words in the 2nd column:
awk -F"      " '{print $2}' test_filter_gt1 | sort  | uniq -c -i | wc -l

Change file encoding in linux

1. Get encoding:

file -bi test.txt
text/plain; charset=us-ascii

2. Change it:

iconv -f ascii -t utf8 [filename] > [newfilename]