Copying a large File > 3 GB from Appserver to client - Forum - OpenEdge Development - Progress Community

Copying a large File > 3 GB from Appserver to client

 Forum

Copying a large File > 3 GB from Appserver to client

This question is not answered

IT seems that when i transfer a file from the appserver to a remote client it works fairly well with Standard Progress ABL code and Raw datatype. However for some reason when the file on the appserver exceeds 3.5 GB the client side file just keeps growing uncontrollably. So if the file on the appserver was 4 GB then the client side file will grow until it consumes all the disk space on the client. (Maybe this is a known bug)

I am using RAW datatype to store 30k Chunks of data and then sending it back to a client machine and reconstructing the file . I need to know if this is the best way  or if there is a faster or better way to transfer large files from an appserver to a remote client.

here is the code that pulls the data from the file:


    import stream ghStream unformatted grChunk no-error.

    /* If there is an error return that there are no more chunks */

    if error-status:error = YES then
        assign lMoreChunks = NO
               error-status:error = NO.
    else
        lMoreChunks = YES.

    /* Find the first temp table record */

    create gttRaw.

    /* Set the raw data for the temp table */

    assign gttRaw.RawData = grChunk.

Then a snippet of client side code does this:

/* Get the first temp table record. */

            find first gttRaw.
             
            /* Write the data to a stream out to local file*/

            put stream strAsFile control gttRaw.RawData.


    
  

All Replies
  • So i guess I may have to submit a support ticket and get an obvious answer back that copy-lob is limited to 4GB in winodws 32 bit  11.3.3 of openedge.

    So the code below works up to 4GB:   this is the server code


        define input  parameter pcSourceFile     as character no-undo.
        define input  parameter piMaxfileoffset  as int64 no-undo.
        define input-output parameter FileOffsetValue as int64.
        
        define output parameter pcfilechunk as longchar.
        define output parameter lMoreChunks   as logical no-undo init YES.
        
        define variable iCurrentChunksize as int64 no-undo.
        
        
        &GLOBAL-DEFINE FILECHUNK-SIZE 120000
        
        
        error-status:error = FALSE.

        iCurrentChunksize = {&FILECHUNK-SIZE} .
        
        if piMaxfileoffset - FileOffsetValue < {&FILECHUNK-SIZE} then
            iCurrentChunksize = (piMaxfileoffset - FileOffsetValue) + 1.
        
        /* Copy directly from the file to a long char to be sent back to the client */   
        
        COPY-LOB FROM FILE pcSourceFile STARTING AT FileOffsetValue FOR iCurrentChunksize TO pcfilechunk no-error.
        
        /* Increment the offset.  */
        
        FileOffsetValue = FileOffsetValue + iCurrentChunksize.
        
        log-manager:write-message (string (FileOffsetValue)).
        
        /* Check to see if we are at the end of the file */
        
        if length(pcfilechunk) < {&FILECHUNK-SIZE} or error-status:error = TRUE  then
            lMoreChunks = NO.

    the client code :

    run get-end-of-file-offeset in hGetUpdate (input pcASFileName,
                                                    output iendoffileoffset).
        
        /* While there are more chunks of data available then */

        do while lMoreChunks:

            /* Get a chunk from the persitent procedure.
               We are using a chunk temp table record instead of
               just an output parameter. This is because the system was crashing with just sending back
               a raw field */

            run get-lob-from-largefile in hGetUpdate (input pcASFileName,
                                                      input iendoffileoffset,
                                                      input-output iFileOffset,
                                                      output lcfileChunk,
                                                      output lMoreChunks).
                
            IF lMoreChunks  then
                copy-lob From lcfileChunk to FILE pcClientFileName APPEND no-convert.   
            
        end.
        
        if length(lcfileChunk) > 0 and length (lcfileChunk) < {&FILE-CHUNK-SIZE} then
            copy-lob From lcfileChunk to FILE pcClientFileName APPEND no-convert.

  • fyi: the documented maximum LOB size is 1 Gigabyte.

    > On Nov 11, 2015, at 4:04 PM, dana wrote:

    >

    > Update from Progress Community [https://community.progress.com/]

    >

    > dana [https://community.progress.com/members/dana]

    >

    > So it seems that this is really an issue with any file over 2GB regardless of whether i use RAW or Copy-LOB. I was trying copy-lob but Trying to figure out how to determine when i am getting to the end of a file.

    >

    > define input parameter pcSourceFile as character no-undo.

    >

    > define input-output parameter FileOffsetValue as int64 init 1.

    >

    > define output parameter pcfilechunck as longchar.

    >

    > define output parameter lMoreChunks as logical no-undo init YES.

    >

    > define variable iCurrentChunksize as int64 no-undo.

    >

    > &GLOBAL-DEFINE FILECHUNK-SIZE 120000

    >

    > error-status:error = FALSE.

    >

    > iCurrentChunksize = {&FILECHUNK-SIZE} .

    >

    > /* Copy directly from the file to a long char to be sent back to the client */

    >

    > COPY-LOB FROM FILE pcSourceFile STARTING AT FileOffsetValue FOR iCurrentChunksize TO pcfilechunck no-error.

    >

    > /* Increment the offset. */

    >

    > FileOffsetValue = FileOffsetValue + length(pcfilechunck).

    >

    > /* Check to see if we are at the end of the file */

    >

    > if length(pcfilechunck)

    >

    > lMoreChunks = NO.

    >

    > View online [https://community.progress.com/community_groups/openedge_development/f/19/p/20414/75006#75006]

    >

    > You received this notification because you subscribed to the forum. To unsubscribe from only this thread, go here [https://community.progress.com/community_groups/openedge_development/f/19/t/20414/mute].

    >

    > Flag [https://community.progress.com/community_groups/openedge_development/f/19/p/20414/75006?AbuseContentId=14f1ff56-513c-4bbd-bea6-05aa6237f85f&AbuseContentTypeId=f586769b-0822-468a-b7f3-a94d480ed9b0&AbuseFlag=true] this post as spam/abuse.

  • The 4gb size limit is by windows fat file system...  You should get win ntfs or linux ext3...