I would assume this is a Memory issue. As when returning the full string, propably the native code needs to hold the entire String in memory and this is then copied to the Air-Application that then also needs to save it. As full utf needs 2-bytes for each char, you need at least 18MB for that result at least ... times 2 for the Native-Code and the Air Application you start needing a lot of Ram.
You could try to increase the memory reserved to your Air Application.
You could you try to process your data in chunks (3000 char blocks)?
Another solution would be to have your native code to generate the data into a temporary (perhaps even Memory-Mapped) File and to read it with an input stream in Flex. This way you don't have to have too much memory available.