Tutorial :Should I use a TMemoryStream as an efficient buffer before writing to a file?



Question:

I am using D6 Professional and need to create a text file in a specific format from lots of small strings already in memory. For performance reasons, I am considering using a TMemoryStream to collate the file data, and then write it out to disk in one go via a TFileStream.

But I have a half forgotten memory (probably from pre-D6 days) of reading somewhere that TMemoryStream is inefficient, especially after it hits its Capacity size. My Delphi (and Windows API) skill is not good enough to check the Classes.pas code for myself.

(OFFTOPIC) especially code like this: (line 5152 of Classes.pas):
NewCapacity := (NewCapacity + (MemoryDelta - 1)) and not (MemoryDelta - 1);
(/OFFTOPIC)

Adding to my worry is that the conclusion of a related question Using MemoryStream to write out to XML
was not to use TMemoryStream, but didn't say why - whether due to TMemoryStream itself, or because there is sufficient buffering in the TFileStream or the I/O device driver, or just the specifics of the code in question.

Thanks for any advice
Regards,
PhilW.


Solution:1

A normal TFileStram also does buffering, and that is sufficient to optimize I/O. Putting a MemoryStream in front only adds overhead.


Solution:2

The TFileStream itself doesn't perform the buffering, that is handled by the OS and is generally sufficient for most purposes.

My suggestion would be to build a method to write your data to a stream, and then pass a TSTream parameter to this method. This way you can test different options easily, without impacting your program.

For example:

Procedure TForm1.StreamMyObjects(aStream : tStream);  begin    aStream.Write( MyString[1], Length( MyString ) * SizeOf( Char ));    aStream.Write( CRLF, Length( CRLF ) * SizeOf( Char ));    aStream.Write( MyOtherString[1], Length( MyOtherString ) * SizeOf( Char ));    aStream.Write( CRLF, Length( CRLF ) * SizeOf( Char ));  end;  

In the JCL, as before mentioned, there is a TJclBufferedStream which you can then test against to see if there is any performance benefit, which will vary based on what your writing, and how much your writing. For example the following will test a TFileStream, and a tJCLBufferedStream to see what the differences are (yes I know I'm missing the TRY/FINALLY):

var    fstm : tFileSTream;    fBufStm : tJCLBufferedStream;    iTicks : Cardinal;    fModes : word; // for SO formatting.  begin    fModes := fmOpenReadWrite or fmCreate or fmShareExclusive;      iTicks := GetTickCount;    fstm := tFilestream.create('test1.txt',fModes);    StreamMyObjects( fStm );    fstm.free;    ShowMessage('TEST1='+IntToSTr(GetTickCount-iTicks));      iTicks := GetTickCount;    fstm := tFilestream.create('test2.txt',fModes);    fBufStm := tJclBufferedStream.create( fStm );    StreamMyObjects( fBufStm );    fBufStm.free;    fstm.free;    ShowMessage('TEST2='+IntToSTr(GetTickCount-iTicks));  end;  

in my test, the following routine:

procedure TForm1.StreamMyObjects(aSTream: tStream);  var    St : string;    ix : integer;  begin    for ix := 0 to 10000 do      begin        St := 'This is a string which is written to a stream. ' + IntToStr(ix);        aStream.Write(st[1], Length(st) * SizeOf(Char) );      end;  end;  

returned 47 for the tFilestream, and 16 for the tJCLBufferedStream. Without the loop, the time is insignificant, which is why you need to test against your data...and how much your writing.


Solution:3

You can use a TStringList, and call SaveToFile method. TStringList has an advantage that memory are not copied when you add a string to it.

Another option is the Jedi JCL class TJclBufferedStream.


Solution:4

As mentioned by the others adding a TMemoryStream in front of the TFileStream probably wont get you to where you need to get to.

Julian Bucknall produced, in a "The Delphi Magazine" article, a set of freeware units and classes to add functionality to TStream objects. I find them extremely useful in most of my dealings with TStream objects (especially TFileStream).

The link below is from google code search and provides access directly to the aaStrms.pas file. You will need the other units (aaIntDeq.pas, aaIntList.pas, aaRegEx.pas and aaStrBld.pas) to use the aaStrms.pas file. These are all non visual classes, so all you need to do is include the unit and instantiate the class.

http://www.google.com/codesearch/p?hl=en#7-hAM65i1Xc/disks/dmag70.zip|alfresco/AAStrms.pas&q=lang:pascal%20aaStrms

I would suggest that the TaaWriteBufferFilter class is the one you want to use.

HTH,

Ryan


Solution:5

I would investigate if it's worth the trouble to build a limited-size buffer of your own. It's slow to write little bits to a TFileStream (especially when delayed-write has been disabled on the storage volume), because the disk driver would have to locate the sector to write to on each little bit. So if you use a buffer that's the size of a few sectors (a safe guess would be 512KB or 1MB), and write the buffer when it's full, it would speed up your work, without having to worry what to do with a gargantuously bloating TMemoryStream.


Note:If u also have question or solution just comment us below or mail us on toontricks1994@gmail.com
Previous
Next Post »