Towards Urdu Corpus: Mining Wikipedia Urdu using Wikiforia Parser

 

Corpus collection is the first step before you even think of Machine Learning and Linguistics. While there are some serious concerted efforts and progress made in different languages to compile and publish Languages Corpora, Urdu Language is no where to be seen in this context. Why? this warrants a separate detailed post, which I will inshaAllah write some time in near future.

In this post, I want to share my first hand experience of collecting a sizable raw Urdu Plain Text from Open Source Wikipedia, so not only I can use it in my research, but will also be able to publish it under Open Source License for others to benefit from it.

Wikipedia publishes complete database backup dumps of

“all Wikimedia wikis, in the form of wikitext source and metadata embedded in XML. A number of raw database tables in SQL form are also available”. “These snapshots are provided at the very least monthly and usually twice a month.”

As I was interested in Wikipedia Urdu, I downloaded following two files from the Wikipedia Urdu Database Backup Dumps page:

• urwiki-20160501-pages-articles-multistream.xml.bz2 85.2 MB
• urwiki-20160501-pages-articles-multistream-index.txt.bz2 3.2 MB

Next task was to extract actual page content from these dump files which follows a specific schema. Wikipedia has a comprehensive list of open source parsers written in different programming languages and published under different types of open source licenses. Because my goal was to collect open source Urdu Plain “Text” and my programming language choice was “Java”, I opt for Wikiforia.

“Wikiforia is a library and a tool for parsing Wikipedia XML dumps and converting them into plain text for other tools to use.”

So, I began by cloning the Wikiforia github repository locally on my laptop and then ran following command on the terminal:

java -jar wikiforia-1.2.1.jar 
     -pages urwiki-20160501-pages-articles-multistream.xml.bz2 
     -output output.xml

The program worked perfectly and uses concurrency (one thread per logical cores) to speed up the processing. It took few minutes to complete the task, however the output was not pure plain text, in fact it was a simplified form of XML and looks like:

At that time I had two options:

1. Run another tool to convert this output file from XML to Plain Text, or
2. Add custom implementation to Wikiforia to make it output pure Plain Text

I opt for second one for two reasons, first I don’t want to waste additional time and processor cycles to process the generated output once again, secondly I thought that there might be others who will benefit from this modification.

So I forked Wikiforia and add a new Sink Implementation  PlainTextWikipediaPageWriter.java. I also had to modify the main program “App.java”, to add CLI support for additional switch “outputformat” with a sensible default set to “xml”, with only two possible values (for now) “xml” and “plain-text”. And once I did that, I also submitted the “Pull Request” on the Wikiforia github repository, in case they decided to merge the patch on to the original repository.

Then, I ran following modified command to extract the “Plain Text” out from the Wikipedia Urdu Database Dumps:

java -jar wikiforia-1.2.1.jar 
     -pages urwiki-20160501-pages-articles-multistream.xml.bz2 
     -output output.txt
     -outputformat plain-text

And here’s how plain text output.txt looks like:

Finally alhamdolillah! I made my first contribution to the Urdu Corpus Community Project.

Pitching my Idea, “SPAM” @ Karachi Startup Weekend in IBA, Karachi University

I pitched my idea “SPAM – Software Project Artifacts Management” System on 21st of October 2011 @ Karachi Startup Weekend in IBA, Karachi University. My idea did not make it to the final list of 12 short listed ideas . However, I was successful in forming a Team during the Networking Phase and will be working with the Team in next 2 days to come up with a comprehensive Proposal Plan for the VCs and Judges

Setting Null value (reflectively) using Apache Common’s PropertyUtils

If you have a requirement to set a Null value (reflectively) to a property of an object using Apache Commons than following method won’t work for you:

BeanUtils.setProperty(Object bean, String propertyName, Object value);

And neither does this one:

PropertyUtils.setProperty(Object bean, String propertyName, Object value);

The reason is that both method uses ConvertUtils to convert values of source type to a destination type. The default set of converters which are registered with BeanUtils have a default value specified and this is why you can’t set a null value using default settings. If you want null values to be set then you need to register converter implementations for those types with a default value of null. So for example you would do something like...

ConvertUtils.register(new IntegerConverter(null), Integer.class);
ConvertUtils.register(new DoubleConverter(null), Double.class);

(Note: the "null" value in the constructors is the default value)

Alternatively, you can also achieve this by following statement:

PropertyUtils.getWriteMethod(PropertyUtils.getPropertyDescriptor(this,propertyName)).invoke(this, new Object[]{null});

Development by Example: Maven & Archiva

Setup Maven:

1. Download and Install latest Maven bundle.
2. If you are accessing Internet via some proxy on your home and/or office networks than make sure you configured these proxy server settings in Maven Settings file.
3. For example I’ve two proxy servers at my office network and one at home, and part of my maven settings file looks like this:
   
4. Notice how I enabled my desired proxy server using <active>true</active>. There were cases when I have direct access to internet with out any proxy servers in between and in such cases all I had to do is to make sure “active” property of all of my proxy server settings are set to ‘false’.

Setup Archiva:

1. Pre-requisite Tomcat 5.5+
2. Download latest Archiva
3. Install it as a WAR, the easiest way.
4. Use embedded derby DB, no fancy stuff.
5. Goto Archiva web administration area http://<server>:<port>/archiva/ and configure your admin user
6. By clicking on “Repositories” button from the lhs menu, you can view following preconfigured repositories:
   1. Two Managed Repositories
      1. Archiva Managed Internal Repository
         
      2. Archiva Managed Snapshots Repository
         
   2. Two Remote Repositories
      1. Central Repository
      2. Java.net Repository for Maven 2
         
7. Although, we can use above preconfigured repositories as it is, but for more clarity, we’ll going to do following changes
   1. Repository name of “Archiva Managed Internal Repository” to “Archiva Managed Internal Release Repository”
   2. Repository directory from “../internal” to “../releases”
   3. Repository URL from
      “http://server:port/archiva/repository/internal/”
      to
      “http://server:port/archiva/repository/releases/”
   4. Repository name of “Archiva Managed Snapshots Repository” to “Archiva Managed Internal Snapshot Repository”
8. While changes to name and directory can be done by simply editing the repository record, the changes to URL can’t. So we will going to delete “Archiva Managed Internal Repository” and create a new record as follows:
   1. Select “Repositories” from the left hand side menu of Archiva web application. Then from the list of repositories click on “Delete” option of “Archiva Managed Internal Repository”
      Click on “Delete Configuration and Contents”.
   2. Again, select “Repositories” from the left hand side menu. Then click on “Add” option on the top right hand side of the page. Create new “Archiva Managed Internal Release Repository as follows
      
9. Because we don’t have to change the URL of “Archiva Managed Snapshots Repository” we can simply edit to make needed changes. Goto “Repositories”, click on “Edit” option from the “Archiva Managed Snapshots Repository” and change name from “Archiva Managed Snapshots Repository” to “Archiva Managed Snapshot Repository”:
   
10. As shown in diagram below, all artifact requests from your maven clients should be directed to Archiva server which, on the bases of artifact type, release or snapshot, try to search in (local) internal repositories. If a matching artifact found than Archiva will serve the request directly, else, it will forward this request to other Remote repositories available over internet for resolution. Once, an artifact is recieved from remote repository, it will be cached first, in (local) internal repositories and then sent to the client as a response to its initial request.
    
11. Next, we’ll going to configure network proxies. Select “Network Proxies” from left hand side menu of Archiva web application. Then, click on “Add” button on the top right hand side of the page. Fill the form with your network’s proxy information and click “Save Network Proxy”.
    
12. Repeat step 11 to create as many proxies as you wish
13. Now, we’ll going to create proxy connectors where each local or internal repository will be configured as a proxy to external or remote repositories.
14. Click on “Proxy Connectors” from the left hand side menu, then select “Add” option from the top right hand side of the page and fill the form as follows:
    
    Do note “White List” entry
15. Repeat the step again with following values
    Do note “White List” entries
16. Repeat the step again with following values
    
17. Repeat the step again with following values
    
18. Now your Proxy Connectors list will look like this
    
    

Configure local internal repositories in your Maven Settings file:

1. Create a new profile to configure newly created archiva managed repositories as follows:
   
   
2. Profiles and their activation is a very handy feature because you can create as many profiles as you like and can activate or deactivate them based on different network environments.
3. For example, there were cases when our Archiva Server hosting both internal Releases and Snapshots repositories was unavailable for some time. I simply deactivated “archiva” profile from my system’s maven settings file. I was then using remote maven repositories directly via internet. No idle time, no delay, no hurdles.
4. Do note that we are using same repositories for plugins as well.

Ok, now you are all set to start using local internet repositories. Go ahead create a new project. Once done, point your browser to archiva web application. Click “Browse” from the left hand side menu, you can view all those groups, sub groups and artifacts which have been cached and served to the maven client up till now.

The story doesn’t ends here. We are still short of deploying our custom artifacts to local internal repositories.

Deploying to local internal repositories:

1. In order to deploy your custom artifacts to local internet repositories, you need network access to machine hosting archiva repositories.
2. Open your maven’s settings file and add server credentials as follows:
   
3. Now, open your project’s POM file add following:
   
4. All set, run maven’s deploy command and then browse your project’s artifact on archiva web application to verify that your it is uploaded successfully.

Project Lombok – Rich Beans without any clutter

Project Lombok modifies your development environment to enable extra java language features. Right now lombok can inject itself into both javac and eclipse and offers both Automatic Resource Management and generating getters, setters, equals, hashCode, and toString, and a couple of other features. A screencast takes you through how it works and even shows you how to install it, and clocks in at less than 4 minutes. It's open source (MIT license).

Prefuse : Interactive Data Visualization Toolkit

Prefuse - http://www.prefuse.org/

JAXX: New way to Swing

A very promising framework for Swing based development. Check out its key features that separates it from the rest. A look at the roadmap might also help to determine its future course of direction.

JSCoverage: Code Coverage Tool for JavaScript

JSCoverage is a tool that measures code coverage for JavaScript programs.

Code coverage statistics show which lines of a program have been executed (and which have been missed). This information is useful for constructing comprehensive test suites (for which it is often called test coverage).

JSCoverage works by instrumenting the JavaScript code used in web pages. Code coverage statistics are collected while the instrumented JavaScript code is executed in a web browser.

JSCoverage works with any modern standards-compliant web browser - including Internet Explorer (IE 6 and IE 7), Firefox (FF 2 and FF 3), Opera, and Safari - on Windows and Linux.

JSCoverage is free software, distributed under the GNU General Public License.

OpenProj: Free & Open Source alternative of MS Project

OpenProj is a free, open source desktop alternative to Microsoft Project. OpenProj has been downloaded over 175,000 times in the first weeks since it was released and has quickly become one of the most popular open source applications. If you are managing a group project then you should go for Project-ON-Demand. If you want a free replacement to commercial desktop software, then OpenProj is perfect. It's available on Linux, Unix, Mac or Windows, and it's free.