July News

28 Jul 2009 15:41

Some of you may be more used to me posting more often, than in last time.
Some of you may wonder why I stopped blogging.

Brussels

Last month was full of adventures. It started 1st of July with me going to Brussels meat our friend to talk about wikipedia-like site about art. We're going to help this man build the most complete site about art using Wikidot software!

BTW, this was my first flight in lifetime. Quite a strange feeling, but generally fine.

Forms

I was working on some nice technical and UI improvements to Wikidot, that is crucial for the art site (but really really nice for Wikidot as well, like forms for editing, entering and viewing structured data to wiki pages).

Search issues

That week was also spent on some massive Wikidot.com search engine tweaks. A stupid one-line bug, which was not exporting proper LC_ALL environmental variable in indexing script, caused many sites that used Asian or East-European languages to be not indexed (most notably the great ИСТОРИЈСКА БИБЛИОТЕКА). At first we though that we can re-index the broken sites, but our re-indexing mechanism was way too slow (would last for weeks for all broken sites).

Pieter then challenged me. He said he can index whole Wikidot in 6 hours. I thought it's not even possible, but then I started to work on that and I managed to index the whole Wikidot in less than 2 hours without indexing tags at first. Then with tags, it took 2 hours and 10 minutes or so. That was damn fast!

Inspired by this and an accident of disk full error on /var partition of our webserver (but this is why we keep user-uploaded files and other important things on separate disks), I also rewrote the incremental indexer, to work in similar way to the whole-Wikidot-re-indexer.

search-api reindex

If you care about some technical details:

  • all search operations are issued with use of search-api, a separate program that can:
    • re-index whole Wikidot
    • queue indexing page/thread
    • queue deleting page/thread
    • queue re-indexing site
    • flush queue
  • search-api is written in Python
  • search-api uses PyLucene - a native Java Lucene library binded to CPython objects with PyJCC. Compiled with GNU Java Compiler to native code (like C programs), this binding has improved performance over using Lucene with Sun's Java.
  • before rewriting it to only-Python, search-api was written in BASH and was a wrapper to:
    • java -jar searchApiHelper.jar search "phrase-to-search"
    • php search-api-helper.php flush
  • search-api also takes care of file locking to assure that
    • only one process tries to modify the index
    • items are added to queue one-after-another
    • when doing some big index modification (read: full re-index) queue is not flushed (so that after the re-index all changes are applied to new index)
    • when flushing queue takes more time, and cron tries to run more flushing processes, they simply end (so only one process flushes the queue at a time)

Union of Rock Festival

Just after week spent in Brussels in nice hotel I went to Węgorzewo, Mazury (Poland biggest lakes distinct) to have fun on rock music festival. Unfortunately, the music level was not very impressive, so I mainly enjoyed the atmosphere on the camping area.

The weather was not great. It was wet everywhere, the ground was covered in 20 centimeters of mud and it was hard to walk around without getting dirty. But during the first day of being there, I learned to do that.

Improved workflow at Wikidot

Some of you noticed, that recently we started to work more efficiently, but this is not quite true. In fact we work as efficiently as before, but we are better organized, and have better priorities on tasks. Also we keep track of what we do, so we can then tell what we've done. So for us, this is a little more work of "documenting" our work (so maybe we work even less efficiently than before?), but for the outside world, we make more noise (in a positive meaning) around that. So basically, people know what we do, what we are going to do, when they can expect changes and most importantly, they understand why some feature request is being postponed. This is (and was) because we have more important things to do, but before they couldn't tell it.

SquarkSquark turned into a professional project manager, that manages our time. pieterhpieterh decided to talk to the Community and listen to their complaints (he reads or at least skims every post on Community forums). He tells Łukasz what needs to be done, Łukasz knows when we will have time to do this. This way communication inside Wikidot improved. Also we (michal-frackowiakmichal-frackowiak and me) no longer look on Community forums (some of you may regret), but this allows us to concentrate on our work.

The work continues

As I mentioned before, we want to introduce a great feature to Wikidot, which is forms. But the implementation now concentrates on the open source version of Wikidot software (once it's ready, working and tested we'll copy the feature to the Wikidot.com service).

aptitude install wikidot

As forms is a huge change, I started to prepare a good ground for it and closed most important bugs in Wikidot open source and I'm about to start making Ubuntu packages for it to allow even-simpler installation on Debian-based systems. Now the installation involves only 6 child-easy steps and in fact can be done by copying&pasting a few commands.

Yesterday's party

Yesterday I went to met some old-school-times friends in the heart of the city. It was meant to be a meeting for "a beer or two" but evolved into beer and dancing till morning. That was first time I get a morning bus (not even the first) to my home just after partying.

It was such a great fun and great folks I met.

Summary

I hope with this long blog post (but divided into friendly sections ;) ) I recompensed long period of not-posting anything here.

Comments: 0

Announcing pymalist

23 May 2009 12:24

I would like to announce pymalist project. It is a stupid and simple highly-modular Pythonic mail list server, that uses basic concepts to do the job well.

Read more on pymalist page, browse the source on GitHub project page.

Comments: 0

Working On Mobile Webbrowser

15 Apr 2009 18:42

As I noted before, I'm running Gentoo in chroot of my iPAQ H3870. As next step of fun with this PDA, I'm willing to create a mobile webbrowser for this (and other) Linux-powered mobile devices. Inspired by iPhone's Safari I want the browser to have the following features:

  • fast
  • easy to use

I want to use Qt and Webkit for this purpose. I will use PyQt for prototyping. As the interface will be minimal this should not add big overhead. For final version probably I'll compile C++ code statically (inserting the latest Qt library into the result program).

What features the browser should have and how I will implement them?

  • fast — using Webkit engine — well integrated with Qt 4.4+
  • fast — using fast JavaScript engine — one of newest Qt/Webkit's features using JIT
  • easy to use — full page zooming — using Qt/Webkit zoomFactor property
  • easy to use — kinetic scrolling — feature popular in iPhone GUI (already implemented by some Qt hackers)
  • fast — some hacky-features should be implemented like weight(-and-number-of-connections)-reducing proxy (like in Opera browser) and some AdBlock-like features (probably non-configurable)

I'm planning VERY minimal interface. No long-history, no bookmark management. Only a button to "save" a page to the browser's dashboard. Also I think about some cool internal things to really make the browser usable and to make it as good as the iPhone's browser.

Comments: 1

Wikidot API

22 Jan 2009 18:57

A few days ago I started working on Wikidot API. The API will be a standardized way to access the Wikidot.com service in a programmable way (i.e. not using a browser) to retrieve, create and update information stored on Wikidot, including site browsing, page editing and commenting.

In simple words this will allow people to write applications that connect to Wikidot.com and perform some actions for the user that runs the application.

Technically, the Wikidot.com API is an XML-RPC service exporting methods from a few especially designed classes.

To connect to an XML-RPC service, you must know its endpoint, which is a regular URL (http:// or https://) address. We decided to use HTTPS to secure the channel from the very start.

The operations we are going to support are:

Browse

  • site.categories
  • site.pages
  • page.get

Above ones are already implemented. Using the API calls you get retrieve almost all data you stored on the Wikidot.com sites!

Modify

  • page.save

This will be the basic method to update the content on your site. We plan several other methods, but this is the one that is the most important.

Comments

  • page.comments
  • page.comment

They will be used to get and post comments on a given page. Using reply_to parameter, there is a possibility to reply to a particular comment.

Forum

  • forum.groups
  • forum.categories
  • forum.threads
  • forum.post

This bunch of methods are going to give you full access to the forums you have started on Wikidot.

How to use the API

We haven't yet enabled the API access to the main Wikidot.com server, but testing the API with Python XML-RPC library is as easy as this:

>>> from xmlrpclib import ServerProxy
>>> s = ServerProxy('SOME-URL')
>>> s.system.listMethods()
['system.listMethods', 'system.methodHelp', 'system.methodSignature', 'system.multicall', 'site.pages', 'site.categories', 'page.get']
>>> print s.system.methodHelp('site.pages')
Get pages from a site
 
Argument array keys:
 site: site to get pages from
 category: category to get pages from (optional)
>>> s.site.categories({'site': 'gamemaker'})
['project', 'rpg', '_default', 'action', 'admin', 'badge', 'beginner', 'contests', 'error', 'event', 'example', 'forum', 'gamemaker', 'gml', 'gmlcode', 'gmupload', 'help', 'ide', 'include', 'mamber', 'member', 'nav', 'portal', 'resource', 'search', 'system', 'talk', 'template', 'tutorial', 'video-tutorial', 'wiki', 'challenge', 'howto', 'recent-changes', 'scratch-pad', 'helpdesk', 'helprequest', 'default', 'testimonial', 'testimonials']
>>> [p['name'] for p in s.site.pages({'site': 'gamemaker', 'category': 'badge'})]
['cool', 'flux', 'c-team', 'gml', 'member', 'madman', 'not-a-noob', 'f-madman', 'start', 'break-it']
>>> print s.page.get({'site': 'gamemaker', 'page': 'badge:cool'})['source']
[[table style="width:98%;margin-right:auto;margin-left:auto;margin-bottom:1%;"]][[row]][[cell style="width:360px;"]]
 
[[div class="error-block"]]
Included page "include:cool-badge" does not exist ([/include:cool-badge/edit/true create it now])
[[/div]]
 
The Cool Badge is given to people who make something really cool.
 
[[/cell]]
[[cell style="vertical-align:top;border:1px solid #ddd;padding:1%;"]]
+++ Display Code
 
[[table class="code"]][[row]][[cell]]
@@[[include include:cool-badge member=member name]]@@
[[/cell]][[/row]][[/table]]
 
+++ Tag
 
{{cool-badge}}
[[/cell]][[/row]][[/table]]
 
[[table style="border:1px solid #ddd;padding:1%;margin-right:auto;margin-left:auto;margin-bottom:1%;width:98%;"]][[row]][[cell]]
 
++ Earn It
 
* Program something really cool using gml
* Make a really cool game
 
+++ Tips
 
* Make sure you post your examples and games on the [[[forum:start|forum]]]. Otherwise no one can see it and nominate you for the cool badge.
[[/cell]][[/row]][[/table]]
 
[[table style="border:1px solid #ddd;padding:1%;margin-right:auto;margin-left:auto; width:98%;"]][[row]][[cell]]
++ Members Who Have Earned the Cool Badge
 
[[module ListPages category="member" order="titleAsc" tag="cool-badge" perPage="100" separate="false"]]
* %%linked_title%%
[[/module]]
 
[[/cell]][[/row]][[/table]]
>>>

A few words of explanation:

  • first we import ServerProxy class from XML-RPC library,
  • then we construct the ServerProxy object s supplying the endpoint URL (SOME-URL in this case, as we don't have yet decided what the URL is going to be)
  • we can see a list of methods by calling system.listMethods on the ServerProxy object
  • we get a help message for a method by calling system.methodHelp
  • then we get categories of site gamemaker (yeah, it's a part of the wikicomplete.info)
  • then we call site.pages method (specifying site and category parameters), but instead of displaying the whole list of structures that describe pages, we only display their names
  • calling page.get returns an array with the information about a page, including:
    • wiki source, array key: source
    • generated HTML, array key: html
    • array with various meta-data, array key: meta
  • we call page.get passing as the argument array that specifies site and page name, get the page object, but displays only what's stored under the source array key

As you see playing with this is really easy as is browsing the available methods and using them.

Why XML-RPC

We've chosen this protocol because it is an easy way to develop both server and client in almost any programming language. Also it gives some flexibility in passed arguments and return values.

We use struct XML-RPC type as the argument and return value type, which is mapped to associative array or dictionary in client (and server) libraries. Each API method gets a bunch of required and optional parameters, that are basically values stored in the struct passed to API methods.

For example site.pages gets a struct with the following keys:

  • site (site name to get pages from) — required
  • category (category to get pages from) — optional

This means, you have to create an associative array (when using PHP) or a dictionary (using Python) and pass it as the method argument:

# PHP
$pages = $server->site->pages(array("site" => "my-site", "category" => "my-category"));
# Python
pages = server.site.pages({"site": "my-site", "category": "my-category"})

Using other programming languages, you'll end with something similar. You can almost always create the array/dictionary in-place, so having this convention is not a big deal.

Applications

I'm working on a filesystem based access to Wikidot site (using FUSE and Python).

We plan having a Wikidot application for iPhone.

A save-it-directly-on-wikidot plugin would be a nice thing for various text editors (and probably other applications).

And probably there are billions of other ways to use this API we're not even aware of. If you have any, feel free to leave a comment.

Comments: 2

page 1 of 41234next »
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License