Large File Processing with high IO using Java is creating problem.

Latest response

A Java component which read a large input file and split the file into 'n' number smaller files based on some header records.
java.nio & bytebuffer with direct allocation is used for faster and better file processing.
When the program is ran in using reading the file IO operation to physically attached memory, processing time of program for 1.3GB file is 1 Min.
But,
When I run the Program on Redhat 7 Linux box which has Gluster mount and processing time for 1.3GB is taking an hour and also it slowly grow the heap over the period oof time and forces us to restart the process.
Draft Code :

try 
    {

        List<ByteBuffer> byteBuffers = macroFileReader.copyLineIntoByteBuffers(BYTE_BUFFER_STRING_LIMIT);
        while(byteBuffers.size()>0)
        {   
            if(header row1 ) {  
                //decide on which folder to write 
            } 
            if(header row2) 
            {
               // decide the on which file to write 
            } 

            /* Header record Found !*/
            if(it is data row){
                // write into current files wrote !
            }
            //read the Next Line 
            byteBuffers = macroFileReader.copyLineIntoByteBuffers(BYTE_BUFFER_STRING_LIMIT);
        }
    }
    catch(Exception e)
    {
    try{
        deleteWriterFile();
    }catch(Exception e1){

    }           
    throw e;
    }finally
    {
        try {
            Files.close();
        } catch (Exception t) {
            throw e;
        }
    }
}
  • If I run the component pointing to /tmp/some_floder it runs faster.
  • if ran the same component pointing to /storage/some_gluster_dir which runs very slow.

Could anyone faced the similar problem ? Is there some configuration either in Gluster or at OS level to limit the number of bytes to write to file and causing the huge delay!

Cs

Responses