Using Spark with MongoDB

I recently started investigating Apache Spark as a framework for data mining. Spark builds upon Apache Hadoop, and allows a multitude of operations more than map-reduce. It also supports streaming data with iterative algorithms.

Since Spark builds upon Hadoop and HDFS, it is compatible with any HDFS data source. Our server uses MongoDB, so we naturally turned to the mongo-hadoop connector, which allows reading and writing directly from a Mongo database.

However, it was far from obvious (at least for a beginner with Spark) how to use and configure mongo-hadoop together with Spark. After a lot of experimentation, frustration, and a few emails to the Spark user mailing list, I got it working in both Java and Scala. I wrote this tutorial to save others the exasperation.

Read below for details. The impatient can just grab the example application code.

Continue reading

Posted in Coding, MongoDB, Spark | Tagged , | 31 Comments

CoffeeScript woes

I recently wrote a piece comparing CoffeeScript, TypeScript and Dart.  For the UI component of Wellmo, we decided upon using CoffeeScript.  While this has been a great leap from JavaScript, we’ve had our share of setbacks.  Here’s a few issues that are good to know about when using CoffeeScript.

Continue reading

Posted in Coding, CoffeeScript | Tagged | 9 Comments

Modifying the pace of audiobooks

I’ve recently started listening to audiobooks.  They’re a convenient way to enjoy books on your way to work or while driving.  After listening to Mika Waltari’s The Egyptian, I took on The Hunger Games, read by Carolyn McCormick.

Like many who have reviewed the audiobook, I had an immediate disliking of the narration.  It was not so much her voice, but her pacing.  She does not give time for the words sink in.  It was a constant, mild irritation  the book could be so much better if the reader took just a little more pauses.  Rather than giving up on the book, I started coding.

I wrote a Ruby script, Audiobook Pacer, that can change the pace of reading of an audiobook.  (I first tried writing a LADSPA plugin, but it seems they cannot modify the length of the audio.)  The script works by adjusting the length of pauses the reader takes between sentences and paragraphs.  All pauses longer than a specified time are lengthened or shortened by a set percentage.  Breaks between words shouldn’t be modified, as this may break the flow of the sentence.

In the case of The Hunger Games, I increased by 25% the length of all pauses longer than 0.6 seconds.  The change is subtle, but it makes all the difference between constant irritation and enjoyment.

Update:  After listening to the Games for a few hours, I started getting irritated by the narration once more.  It turned out I had converted only half of the files.  After modifying the pace of the rest of the files enjoyment prevailed.

Posted in Coding, Ruby | Tagged , , | Leave a comment

Testing visual appearance with Cucumber + Watir

One of the great things about Cucumber and Watir is that it allows you to write functional tests that are decoupled of the UI.  By using page objects, the definition of how the UI works is decoupled from the tests themselves.  If the UI changes, you only need to update the corresponding page object, and all of your tests still run.

Such tests provide an excellent safety harness in which changes can be made with the confidence of not breaking other features.  The only problem is that the tests verify the functionality, but not the visuals of the pages.  We were missing the safety harness for CSS changes.

For this purpose I implemented a set of tests that verify the visual appearance of certain core pages.  This prevents someone from accidentally making a CSS change that affects other pages as well.

Since these tests are very brittle by definition, I do not recommend having a lot of them.  You need to identify a few core pages from your application that rarely change their visual appearance, but which still cover the most important parts of your CSS.

For the impatient, the example code is on GitHub.
Continue reading

Posted in CSS, Cucumber, Testing, Watir-Webdriver, Web | Tagged , , | Leave a comment

setImmediate, MessageChannel, postMessage broken on Internet Explorer 10

The JavaScript setImmedate function has been proposed and promoted as a faster alternative to setTimeout(fn, 0) (and cleaner than postMessage). While the HTML spec clamps setTimeout(fn, 0) to a minimum delay of 4 ms, setImmediate(fn) is defined to run the function immediately after any pending events have been handled.

The new API has seen opposition in both WebKit and Gecko developers. Currently, the only browser to implement the function is Internet Explorer 10, in addition to node.js. Unfortunately, IE10 has a serious flaw that renders the API practically useless on IE10 Mobile and dubious on IE10 Desktop.
Continue reading

Posted in Bugs, IE, JavaScript | Tagged , , , | 2 Comments

Testing time-dependent features in JavaScript

Many applications use the current time in their functionality.  For example, they can show data for a certain period of time or show the current date within the application.  Writing functional tests for such applications can be tedious.  How do you write a repeatable test for functionality that only occurs on Thursdays?

For this purpose I wrote TimeShift.js.  It is a mock implementation of the normal Date constructor which allows you to set the current time and time zone.

Date = TimeShift.Date;
TimeShift.setTimezoneOffset(-300);
TimeShift.setTime(1275393600000);
new Date().toString();
// => "Tue Jun 01 2010 17:00:00 GMT+0500"

This way you can write repeatable test cases that still depend on the current time.

Continue reading

Posted in Coding, Cucumber, JavaScript, Testing | Tagged , , , | 1 Comment

Simple image devignetting

After image

After

Before image

Before

With the prevalence of smart phone cameras today, they are often used instead of scanners as quick digitization methods for documents.  Unfortunately this leads to excessive vignetting (darkened areas at the edges), which makes it hard to print the document legibly.

For simple text documents and line drawings, however, it just takes four simple steps to correct the image.  The following describes the steps in GIMP, but the same should apply to PhotoShop and probably other image manipulation software as well.

Continue reading

Posted in Gimp, Image manipulation | Tagged , | 1 Comment