handling very large Files with Java

In one of our projects we had to address the problem of parsing huge files of more than 8 GB in an efficient way.  The first problem that requires attention is how to parse such a file without having to wait for hours, i.e. speeding up parsing to something close to what the operating system and the underlying hardware (disk drive, cpe and such) is willing and able to deliver in terms of transfer rate.

The obvious and typical answers are buffering, threading and NIO (non blocking IO) in that order. Buffering is the obvious choice and will never do harm as long as you pick correct buffer sizes.

References:

Interesting discussion and comparision of buffersizes using different techniques (standard IO, threading and NIO)

http://stackoverflow.com/questions/964332

Good intro:

http://articles.techrepublic.com.com/5100-10878_11-1046714.html#

Advertisements
This entry was posted in Distributed Computing, Enterprise Java. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s