D'oh!
pabluk's blog... IPv6 ready!
pabluk's blog... IPv6 ready!
Mar 26th
Sometimes can be useful to crawl sites with Scrapy using temporary resources on the cloud, and Amazon EC2 is perfect for this task. You can launch an Ubuntu OS instance and schedule your spiders using the Scrapyd API.
With boto, a python interface to Amazon Web Services, you can launch instances and install the scrapy daemon using the user data feature to run a script on boot.
First, you need an AWS account with your access keys, a EC2 security group accepting TCP connections on port 6800 and a key pair for the selected region. After that you must choose an Ubuntu EC2 image, here you can find a list of Ubuntu AMIs.
We need to create an user data script with the following content
#!/bin/bash # filename: user_data-scrapyd.sh # Add Scrapy repositories and key echo "deb http://archive.scrapy.org/ubuntu precise main" > /etc/apt/sources.list.d/scrapy.list curl -s http://archive.scrapy.org/ubuntu/archive.key | apt-key add - # Install debian package apt-get update && apt-get -y install scrapyd-0.16 # Restart scrapyd service scrapyd restart
finally we can launch an instance using boto on the python console
$ python
>>> with open('user_data-scrapyd.sh') as f:
... user_data = f.read()
...
>>>
>>> import boto.ec2
>>> conn = boto.ec2.connect_to_region("us-east-1",
... aws_access_key_id='my_aws_access_key',
... aws_secret_access_key='my_aws_secret_key')
>>> conn.run_instances('ami-da0d9eb3', user_data=user_data, key_name='main', instance_type='t1.micro', security_groups=['default'])
Reservation:r-89a2aef3
>>> reservations = conn.get_all_instances()
>>> reservations
[Reservation:r-89a2aef3]
>>> instances = reservations[0].instances
>>> instances
[Instance:i-d4a2c1b8]
>>> i = instances[0]
>>> i.state
u'pending'
>>> i.update()
u'running'
>>> u'http://%s:6800/' % i.public_dns_name
u'http://ec2-54-224-86-173.compute-1.amazonaws.com:6800/'
and pointing your browser to this URL you must see the Scrapyd monitoring page.
Now you can schedule a spider run just by using
$ curl http://ec2-54-224-86-173.compute-1.amazonaws.com:6800/schedule.json -d project=myproject -d spider=spider1
{"status": "ok", "jobid": "70d1b1a6d6f111e0be5c001e648c5a52"}
You can find a copy of these scripts on Github ec2-scrapyd.
Jul 8th
The last 3 weeks I was working on developing an application to participate in a contest called Ubuntu App Showdown.
My application is called Ubatar and its main objective is to provide a solution to this idea. Here you can see some videos of the latest version of Ubatar fulfilling its purpose.
During these 3 weeks I learned many things, reading a lot of source code, looking for examples and discussing with other developers.
I can only say that I really enjoyed and was an incredible experience.
Thanks to all the Ubuntu development team and the community at large.
And this is not the end of the project, there are still many things to improve, if you want to help please contact me via this form. Also any questions or suggestions can be sent to Questions for Ubatar in Launchpad.
Jun 6th
To celebrate the world IPv6 day I decided to activate IPv6 on this blog.
Now you can visit my site with native IPv6 connectivity.
If you want to know more about IPv6 you can start reading this overview
Feb 22nd
Recently I’ve been working on a Django project that uses the Haystack search API and the Solr search engine to perform full text searching on indexed data.
This project was initially deployed on a Debian 5.0 (Lenny) server with some issues with Tomcat, but using the latest stable release of Debian (code named Squeeze) install Tomcat 6 and Solr is very easy.
Just run the following
apt-get install tomcat6 tomcat6-admin
download and install the latest version of Solr
wget http://apache.cict.fr//lucene/solr/1.4.1/apache-solr-1.4.1.tgz tar xvfz apache-solr-1.4.1.tgz cd apache-solr-1.4.1 cp dist/apache-solr-1.4.1.war /var/lib/tomcat6/webapps/solr.war cp -fr example/solr /var/lib/tomcat6/ chown -R tomcat6:tomcat6 /var/lib/tomcat6/solr
after install Solr you need to restart the tomcat6 service
/etc/init.d/tomcat6 restart
And voilà, Solr is running. Point your browser to http://localhost:8080/solr/admin this should show the Solr Admin interface.
Jun 16th
The next release will include support for Launchpad translations, save session to file and more keyboard modes (specific vim and emacs key bindings).
Thanks to Ivan Makfinsky for Fedora packaging and thanks to everyone who reported problems and made suggestions for improvement!
May 25th
Si vous n’avez pas encore mis à jour votre système Ubuntu vers la dernière version 10.04 « Lucid Lynx » et si vous utilisez le greffon de téléchargement de sous-titres du lecteur vidéo Totem, vous remarquerez que le greffon ne marche plus. Cela est dû à un changement de l’API XML-RPC du site opensubtitles.org.
Mais bon pour le corriger, sans mettre à jour votre système Ubuntu 9.10, vous pouvez ouvrir un terminal et lancez la commande suivante
cd /usr/lib/totem/plugins/opensubtitles/ sudo sed -i 's/www/api/' opensubtitles.py
Ainsi, vous pouvez continuer à télécharger vos sous-titres sans problème.
Apr 8th
Just to try Java Wireless Toolkit, also know as J2ME, on my Samsung C3050 mobile phone I wrote a silly and simple application called Another Counter, wich as its name says is used to count things, like a tally counter, then you can count people, events, measure traffic, etc.
A version of the toolkit for Linux can be downloaded from here, it includes build tools, utilities, examples, and a device emulator. With some knowledge of Java programming and reading the API is easy to write small applications like this.
To install and run the application on the phone you have to download these files: AnotherCounter.jad and AnotherCounter.jar.
The source code is licensed under GPL and hosted on GitHub.
Mar 10th
Una simple herramienta para configurar los botones de Metacity (el window manager del escritorio GNOME).
Si estas usando Ubuntu Karmic Koala y querés configurar los botones para que se vean como en la nueva versión Lucid Lynx. O si usas Lucid Lynx y querés volver la configuración al estilo de Karmic Koala, lo podés hacer sin tener que usar el editor de configuraciones de GNOME (gconf-editor).
Me parece muy bueno el cambio en los botones de las ventanas para la nueva versión de Ubuntu, para hacerlo diferente, único. Costará un poco adaptarse pero es como todo, por eso yo ya cambié el estilo de los botones en mi Karmic
. De todas formas si a alguien no le gusta lo puede cambiar a su gusto, incluso si lo preferís, podés sacar todos los botones de las ventanas
wget http://launchpad.net/mwbuttons/trunk/v0.2/+download/mwbuttons chmod +x ./mwbuttons
La aplicación esta hosteada en Launchpad y la idea inicial fue solo para probar el módulo gconf de Python y el plugin de Launchpad para Bazaar.
Jan 19th
The real title of this post should have been Pimp your Metacity, but not everyone knows that Metacity is the official window manager of GNOME.
This is just a modified version of the default theme in Ubuntu Karmic Koala. The main difference is that the minimize, maximize and close window bottons has been removed. They are only activated by mouseover.
It is useful for users who already know the position of the window buttons in GNOME and all the time use keyboard shortcuts to manage the windows. And especially useful on small screens like netbooks where there is not enough space avaliable to display redundant things.
The Human Simple theme can be downloaded from GNOME-Look http://gnome-look.org/content/show.php?content=118052
Jul 1st
This is just a simple tip to use ext4 filesystems in Ubuntu Intrepid Ibex 8.10.
In Ubuntu Jaunty ext4 filesystems is fully supported but in Intrepid was in developing.
To use it we must load the module ext4dev
sudo modprobe ext4dev
to check the filesystems supported by the kernel we can do
cat /proc/filesystems
there should be a line with ext4dev.
Now, if we try to mount a formatted partition as ext4, will see that we can not do
sudo dmesg | tail
shows a line that says
EXT4-fs: sda2: not marked OK to use with test code.
to solve it we need to enable test_fs, an extended option for ext4 partitions
sudo tune2fs -E test_fs /dev/sda1
Now, we can mount it without problems
sudo mount -t ext4dev /dev/sda1 /mnt/disk
That’s all folks!
Note 1: replace sda1 by the appropiate device
Note 2: sorry for my english