The Ruby Passion
Reading large csv files in chunk of memory

Problem Statement : Parse huge csv files (in GBs) under a directory and create corresponding new files with an old text replaced with new text.

For very large files, you will receive “NoMemoryError: failed to allocate memory” error. This is the problem if a file it too big to fit into memory, and there are no known delimiters, or if the records between the delimiters are themselves too big to fit in memory. You have got no choice but to read from the file in chunks of a certain number of bytes. This is also the best way to read binary files.

Based on Ruby CookBook (page 204), a simple reader method added to the File class which is used to loop on the file based on the provided chunk size.

ruby large_file_read.rb [dir] [textToBeReplaced] [newText]

One more reference : http://stackoverflow.com/questions/11949714/parse-huge-file-10gb-and-write-content-in-another-one

Currency exchange rates with openexchangerates API

https://openexchangerates.org provides currency conversion JSON API. The site has over real time and historical exchange rates for many currencies worldwide. For maximum consistency and unbiased data,exchange rates for 165 world currencies are tracked from multiple sources and blended algorithmically, with historical data back to 1999. The exchange rates API is delivered in simple and portable JSON format with HTTPS and JSONP support, for fast and easy use in any language.

You just need to get an APP ID to play with this API. (https://openexchangerates.org/signup)

Here is a simple code which parses the json response and gets USD to other listed currencies conversion rates in a csv file with ruby.

Historical currency exchange rates

Dealt recently with getting historical monthly currency exchange rates for a particular currency. “exchange-rates.org” was a good resource and quickly wrote ruby script to get the historical rates (USD to other currency).

Attending Ruby conference India 2013

I am attending Ruby conference in Pune, India 2013. Excited to listen to some of the good talks and also meet many of my friends … :-)
I am attending RubyConf India 2013

Agile Conference 2013 India - A great learning experience !

I attended Agile Conference 2013, Bengaluru (India) for the very first time. 

The conference was full of good talks, workshops, discussions, tool demos and more than that a whole bunch of enthusiastic people! Due to time constraints and also my technical interest, I only attended the 2-day Technical Conference. (1st - 2nd March) 

The technical conference was mainly targeted at developers, Analysts, Architects & Dev-Ops and it covered quite a wide range of topics this time. 

On the first day, Venkat Subramaniam gave a fantastic key note - ‘It Could Be Heaven or It Could Be Hell: Pleasure and Peril of Being a Polyglot Programmer’. One can actually feel a lot of energy and enthusiasm while he talks and I think he eats, drinks and dreams only software languages :-) True language geek!

Neal Ford presented on Emergent Design. I liked the word he used - FED - Framework Enslaved Development. We should try to come out of this pattern to understand how the real software design emerges.

Venkat also presented a couple of other topics - ‘Reactor your language knowledge portfolio: We are what we can code in’,’Thinking and programming in functional language’ and ‘Caring about code quality’. I loved those. During these presentations, he emphasized on learning more than one language. He also told that as a developer, your focus should be on writing real business logic and that is also to the point - focus on essence instead of ceremony. The functional programming presentation was awesome. Learnt a lot of things. He also gave some examples of Java 8 which has more support of functional style coding. The take away from that session was to learn at least one functional language and experience the power of functions. 

In another track, Rebecca Parsons from ThoughtWorks presented on ‘Evolving NoSQL databases’. Her talk was quite informative where she touched upon various NoSQL database types like Graph, Document, key-value pair and column family. She suggested to read this book - ‘Re-factoring databases’ by Scott Ambler.

There were also some other good topics which I was able to attend among the ongoing parallel tracks - Test Driven Decoupling, Enabling Continuous Delivery With Evolutionary Architecture, Cultivating a DevOps Culture, etc.

Overall, the conference experience was certainly very good. It always feels great when you get a chance to hear good people. I was very lucky to hear some of the Agile Thought leaders like Neal Ford, Venkat Subramaniam, etc.

Really great learning experience ! Many thanks to all the organizers/volunteers for organizing such a beautiful conference !

 

Attending Agile Conference 2013 India

I am very keen to attend Agile India 2013 Conference @ Bengaluru. Though I have been practising Agile since last more than 4-5 years, this is my first time to attend any Agile conference. I’m certainly excited to hear from Agile Thought leaders !  

image

Here is the official link where you can find everything about the conference :- http://2013.agileindia.org/

BTW, I am attending only the Technical Conference (1st-2nd March 2013), and not the entire one. Let’s connect, network, and learn … ! :-)

The Unix Programming Environment

I am currently reading ‘The Unix Programming Environment’ book written by Brian W. Kernighan and Rob Pike. A nice read about the UNIX programming philosophy which is based on the relationships between programs. The book explains the concepts about file system, shell, filters, etc. very well. You will find many examples of how to effectively use the command interpreter or shell. I would definitely recommend this book to get a good understanding of Unix system. Listing down its chapters :-

1. UNIX for Beginners

2. The File System

3. Using the Shell

4. Filters

And some advanced chapters :-

5. Shell Programming

6. Programming with Standard I/O

7. UNIX System Calls

8. Program Development

9. Document Preparation

Automate yourself into a job

Tried to implement a thought/practice from ‘The Passionate Programmer’ book by Chad Fowler - “Identify daily repetitive tasks and try to automate those. Automate yourself into a job!”

Created a batch file auto_run.bat which will open the required software programs / tools with a single click :-

e.g.

start /d “C:\cygwin” Cygwin.bat

start /d “C:\Program Files\JetBrains\RubyMine<version>\bin” rubymine.exe

start /d “C:\sqldeveloper<version>” sqldeveloper.exe

start /d “C:\Program Files\Mozilla Firefox” firefox.exe

start /d “C:\Program Files\Internet Explorer” iexplore.exe


You can add as many as you use. This is just an example!

Good Learning sites for Ruby, Rails …

Listing some of the very good sites for learning Ruby, Rails, javascript, testing, deployment and various technological aspects, etc. You can find many more on the web … Google is the only limit ! :-)

http://www.codeschool.com/

http://rubymonk.com/

https://peepcode.com/

http://railscasts.com/

http://asciicasts.com/

http://www.tutorialspoint.com/index.htm

http://rubylearning.com/

http://ruby.railstutorial.org/

http://manwithcode.com/ruby-programming-tutorials/

http_proxy settings to use gem from behind a proxy firewall

If your computer is behind a HTTP proxy firewall, then you might have received an exception while running gem install <gem_name> as below e.g.

$ gem install rails
ERROR:  http://rubygems.org/ does not appear to be a repository
ERROR:  Could not find a valid gem 'rails' (>= 0) in any repository>

One solution that you can try is adding “gem: —http-proxy=my_proxy_server:8080” to ~/.gemrc. You can check these settings by simply running a ‘gem environment’ command. However, there is one more important config setting that you need to do on your machine. You need to set HTTP_PROXY system environment variable as well.

On Windows 7, go to My Computer and select System Properties -> Advanced System Settings. Select the Advanced tab and then click Environment Variables. Click New under System Variables and enter HTTP_PROXY as the name and your IP address and port as the value. e.g.

This works ! :-)