If you're capturing that many frames at once, I recommend reducing the frequency so that it doesn't capture as many frames. That is, if you're working in Animation Mode. If you can use Presentation mode, this will generate fewer frames.
The problem you're experiencing is due to the way DemoWorks manages memory. We hope the new version will have much better memory handling for capturing larger demos. With the current version there is a known issue when exporting a demo of 600+ frames to Flash output.
Regards,
Greg