Still large files problems with Besweet b26

[Eye of Horus]

Hi all and of course especially DSPGuru,

I still have problems in besweet with large files.
My original is a 32 bits 6 channel WAV, about 79 minutes .
I use a normal standard Besweet line to get 6 mono WAV's.
4 of them were 394 MB, 2 of them were 788 MB ??

I tried it on two other WAV's that were shorter, with the same effect.... (63 minutes and 68 minutes)

Commandline :
BeSweet.exe -core( -input t001.wav -output t01- -type wav -6chfloat )

I use NTFS on a XP system.......

I believe (from memory) that this bug is reported before.....

Anyone a solution ??

BTW someone else reported that Besweet crashed on his XP system, when the length of the CD was above 74.59 minutes......
I asked the version he uses and am waiting for an answer......

Before I forget : the log shows nothing specail, so I didn't include it.

kind regards.

EoH

[DSPguru]

Dear EoH,
what's the size of the original file ? and which files are at what size ? do you have enough free space on the relevant hard-drive ?
* including the logfile never harms.. ;)

[Eye of Horus]

Originally posted by DSPguru
Dear EoH,
what's the size of the original file ? and which files are at what size ? do you have enough free space on the relevant hard-drive ?
* including the logfile never harms.. ;)

Hi DSPGuru,

I tried to reproduce what happened, but didn't succeed. Actually it's even worse !
The size of the 6 channel 32 bits WAV : 4.61 GB (4,958,558,264 bytes)
The stereo was : 1.53 GB (1,652,854,784 bytes)
(these sizes are correct : 4,958.... /3 = 1652,....)

I ran Besweet and this time it ended with 6 equal mono Wav's with a length of 105 MB each !?

Here's the log, but as said : nothing special except the time !

BeSweet v1.5b26 by DSPguru.
--------------------------

Logging start : 05/04/04 , 20:39:11.

BeSweet.exe -core( -input 01.wav -output 01- -type wav -6chfloat -logfile d:/log.txt )

[00:00:00:000] +------- BeSweet -----
[00:00:00:000] | Input : 01.wav
[00:00:00:000] | Output: FL, FR, SL, SR, C, LFE
[00:00:00:000] | Floating-Point Process: Yes
[00:00:00:000] | Source Sample-Rate: 44.1KHz
[00:00:00:000] +---------------------
[00:10:27:093] Conversion Completed !
[00:02:47:000] <-- Transcoding Duration

Logging ends : 05/04/04 , 20:41:58.

It stopped after 10:27 .


After a restart I redid the complete process with starting with the 16 bit WAV, converting it to 32 bits, running through Plogue's Bidule to get the 6 channel WAV and used Besweet.
Same result !!

Free space on the "work" harddisc is 6,79 GB.
I had nothing running in the background.

I did other tests with sizes from 400-500 MB without any problem !

Well, this is all the information I can give and I hope you can work it out !

kind regards,

EoH

[daphy]

We got this before, I remember in the ambisonic thread or the splitted thread :rolleyes:
Since there was a 2GB barrier I decided to split the source 32/44.1 or 48 WAV (at about 40~50 min) in to two parts.
then run besweet transcoding to DTS/DTSWAV
then merge those two DTS/DTSWAV with DTSTrim
finally use besplitt (inoffical beta -special-0.9b7) to fix the file

-> perfect result, needs really less time as merging the single monofiles before encoding into DTS/AC3

maybe you should try this

[specise_8472]

Just cluthching at straws here.

The 2 gig limit was a hold over from the days of FAT32 (correct me if I am wrong). NTFS can handle just about any limit.

BUT as most of us are using 32bit processors - They can only address 4 gig of memory. Could there be something in the way Besweet is programmed, to be affected by this limit:confused:
(like searching to the end of the file and running out of steam at 4gig)

Just my 2 credits worth

[Eye of Horus]

Originally posted by specise_8472
Just cluthching at straws here.

The 2 gig limit was a hold over from the days of FAT32 (correct me if I am wrong). NTFS can handle just about any limit.

BUT as most of us are using 32bit processors - They can only address 4 gig of memory. Could there be something in the way Besweet is programmed, to be affected by this limit:confused:
(like searching to the end of the file and running out of steam at 4gig)

Just my 2 credits worth

Yep, I think it is somehow like you describe. If the 6ch 32 bits file is less than 4 GB, there are no problems !

EoH

[kempfand]

@DSPguru: I can definitely confirm that there is a problem around the 2 GB (for 16 bit WAV's) or 4 GB (32 bit WAV's) limit also here (XP-only systen, NTFS only, Intel Xeon 2.4 MHz, 1 GB RAM).

If I remember correctly (from memory), this only happens on certain machines, because I think there were 1 or 2 reorts of "things working now" (although no logfile as proof posted).


Eample 1: 16_00.wav (1.91 GB, == 2,054,369,324 bytes) works fine.

BeSweet v1.5b26 by DSPguru.
--------------------------

Logging start : 05/05/04 , 22:53:07.

C:\BeSweet\BeSweet.exe -core( -input i:\16_00.wav -output m:\16_00-New- -6ch -logfile C:\BeSweet\BeSweet.log )

[00:00:00:000] +------- BeSweet -----
[00:00:00:000] | Input : i:\16_00.wav
[00:00:00:000] | Output: FL, FR, SL, SR, C, LFE
[00:00:00:000] | Floating-Point Process: No
[00:00:00:000] | Source Sample-Rate: 44.1KHz
[00:00:00:000] +---------------------
[01:04:39:114] Conversion Completed !
[00:02:59:000] <-- Transcoding Duration

Logging ends : 05/05/04 , 22:56:07.

Eample 2: 16_01.wav (2.02 GB, == 2,179,430,444 bytes) crashes.

BeSweet v1.5b26 by DSPguru.
--------------------------

Logging start : 05/05/04 , 22:57:14.

C:\BeSweet\BeSweet.exe -core( -input m:\16_01.wav -output i:\16_01-New- -6ch -logfile C:\BeSweet\BeSweet.log )

[00:00:00:000] +------- BeSweet -----
[00:00:00:000] | Input : m:\16_01.wav
[00:00:00:000] | Output: FL, FR, SL, SR, C, LFE
[00:00:00:000] | Floating-Point Process: No
[00:00:00:000] | Source Sample-Rate: 44.1KHz
[00:00:00:000] +--------------------- <i.e. end of logfile>

The DOS-Windows 1st hangs with this message: BeSweet v1.5b26 by DSPguru.
--------------------------

[01:08:34:892] transcoding ...
Thereafter, BeSweet crashed "BeSweet.exe has encountered a problem and needs to close": AppName: besweet.exe AppVer: 0.0.0.0 ModName: besweet.exe
ModVer: 0.0.0.0 Offset: 00010c36
Same "picture" of 32 bit WAV's (no logfile posted).

This error is reliably reproducible here.

I think "it" happens at the very end of BeSweet's pass, as the 6 mono-WAV's are written to their correct length [01:08:34:892].

With the growing number of Bidules, I really hope for a fix (although things can be done manually with CoolEdit(*) or SoftEncode).

Kind regards,

Andreas

(*) Off-topic add-on (because I think someone asked):
If opening multichannel wav files with CoolEdit (i.e. 16_01.wav), it will open things as 16_01.wav, 16_01(1).wav, 16_018(2).wav, ... , 16_01(5).wav.
Assuming that ITU 5.1 channel mapping for Bidule's Audio File Recorder was used, the mapping will be as follows:
16_01.wav -> L
16_01(1).wav -> R
...
16_01(5).wav -> SR

[kempfand]

Add-On: Simple BeSplit seems to work:

BeSplit v0.9b7 by DSPguru.
--------------------------

Logging start : 05/05/04 , 23:32:43.

BeSplit.exe -core( -input M:\16_01.wav -output I:\03-new -type wav -demux -logfile M:\besweet.txt )

[00:00:00:000] +------- BeSplit -----
[00:00:00:000] | Input : M:\16_01.wav
[00:00:00:000] | Source Sample-Rate: 44.1KHz
[00:00:00:000] | Channels Count: 6, Bitrate: 4233kbps
[00:00:00:000] | Output : I:\03-new
[00:00:00:000] +---------------------
[01:08:38:349] +---------------------
[01:08:38:349] Operation Completed !
[00:03:32:000] <-- Process Duration
Logging ends : 05/05/04 , 23:36:15.
BTW: Using same hip.dll for BeSweet & BeSplit (Saturday, February 15, 2003; 31,744 bytes)

Regards,

Andreas

[Eye of Horus]

Originally posted by specise_8472
Just cluthching at straws here.

The 2 gig limit was a hold over from the days of FAT32 (correct me if I am wrong). NTFS can handle just about any limit.

BUT as most of us are using 32bit processors - They can only address 4 gig of memory. Could there be something in the way Besweet is programmed, to be affected by this limit:confused:
(like searching to the end of the file and running out of steam at 4gig)

Just my 2 credits worth

Well, why is the simplest solution always so obvious that we don't think about it ???

Alastor, who is also a member here, found the solution and it's so obvious, I still cannot believe we didn't think about that !

Instead of one 6 channel recorder in Bidule, why not feed 6 one channel recorders. No Besweet necessary , it saves a step !

My compliments, Alastor !!

The new Bidule is on my site and soon on Daphy's !

http://www.app.demon.nl/SAD51inBidulemod.rar

kind regards,

EoH

[kempfand]

Instead of one 6 channel recorder in Bidule, why not feed 6 one channel recorders. I remember there is a small "out-of-sync" then doing this. No advisable at his point in time !

Use BeSplit v0.9b7, or channelx.exe from CDP Multi-Channel ToolKit (http://www.bath.ac.uk/~masrwd/mctools.html) with a commandline like channelx.exe -oDeMux.wav t_01.wav 1 2 3 4 5 6

Kind regards,
Andreas

[specise_8472]

The problems I had with doing this was everytime I imported the individual files into Surcode, one of them was always a different length. Plus as stated, slight delays.

Anyway this does not address the fact that Besweet has a bug when handling 16 bit max filelength 2 gig or 32 bit max filelength 4gig.

[Eye of Horus]

Originally posted by kempfand
I remember there is a small "out-of-sync" then doing this. No advisable at his point in time !

Use BeSplit v0.9b7, or channelx.exe from CDP Multi-Channel ToolKit (http://www.bath.ac.uk/~masrwd/mctools.html) with a commandline like channelx.exe -oDeMux.wav t_01.wav 1 2 3 4 5 6

Kind regards,
Andreas

ChannelX didn't work on a 4.6 GB file !
Besplit......... WORKED !!!!

Thanks kempfand !!!

kind regards,

EoH

[kempfand]

Besplit......... WORKED !!!!
Thanks kempfand !!! :o Handing ower the flowers to DSPGuru.

A solution for BeSweet would rock, as it allows going to DTS/ac3 in one beam. High hopes :D and kind regards,

Andreas

[Eye of Horus]

Originally posted by kempfand
:o Handing ower the flowers to DSPGuru.

A solution for BeSweet would rock, as it allows going to DTS/ac3 in one beam. High hopes :D and kind regards,

Andreas

I thought it worked, because I fnally got my 6 788MB WAV's.
BUT.......... the files are not recognized by Surcode DTS CD !!
It says : Invalid Wave file.

Loading the WAV into Soundforge : Unknown Filetype.
Loading as RAW 32 bits float : it loads a file which is exactly half of the time : 39.02 minutes and is only white noise.

So............ Give the flowers back :D :D :D

EoH

[alastor]

Sorry but i d'ont speake fluently english.

The solution d'ont work with small files like mp3 songs, but work very well with big files like films soundtrack.

I test tree different films 2 chapters in a serie 40 minutes each and a regular film. 105 minutes. 2,5 GB input wav file.

In all cases the bidule's layout work.

May be differences between song wav file and film wave file? The frames indicators? I d'ont know,

I work in 16 bytes deep at 48000

Alastor

[DSPguru]

the 2gb limit for 16bit waves is indeed there.
the 4gb limit for 32bit waves shouldn't be there, but i'm afraid i need to clean-up 8gb of hard-disk space to check this one..

what happens if you encode a bigger-than-4gb float into 6ch mono, instead of 6chfloat? are there any problems there ? is so, it means i only need to free-up 6gb of space ;)

[alastor]

Hi Dspguru

First I work undder XP with NTFS.

In my case your program crashes when procesing the end of archives for files about 4GB.

For example a film with a duration 1:48:37:720, crashes at 1:48:37:718.

Generates a windows interruption message.

No other message as notified.

The archives produced are unreadables.

Best Regards.

Alastor

[DSPguru]

thank you, Alastor. may i see the logfile ?


edit: ok,
after installing an extra hard-disk and formatting to NTFS, i am unable to reproduce the bug (:)), but i think i found something that could be related..
please download b27 (http://dspguru.notrace.dk/BeSweetv1.5b27.zip) and report back.


* a note about b27 had been added to q30 in the BeSweet FAQ (http://forum.doom9.org/showthread.php?s=&threadid=15738)

[alastor]

First,use your tools for a long time.

Thank you very much by so I magnify work.

Download and installed b27

Film Source Jedi returns duration 2:09:04 206 MB

First conversion MP2 WAV AIFF with conversion 44,1 48, nedded for dvd audio.

This is the log

BeSweet v1.5b27 by DSPguru.
--------------------------
Using hip.dll v1.19 by Myers Carpenter <myers@users.sf.net>
Using Shibatch.dll v0.24 by Naoki Shibata & DSPguru (shibatch.sourceforge.net).

Logging start : 05/07/04 , 14:47:37.

C:\Programas de video\BeSweet\BeSweet.exe -core( -input d:\Trabajos de Video\El retorno.mp2 -output d:\Trabajos de DVD\El retorno.wav -2ch -logfile C:\Programas de

video\BeSweet\BeSweet.log ) -ssrc( --rate 48000 )

[00:00:00:000] +------- BeSweet -----
[00:00:00:000] | Input : d:\Trabajos de Video\El retorno.mp2
[00:00:00:000] | Output: d:\Trabajos de DVD\El retorno.wav
[00:00:00:000] | Floating-Point Process: No
[00:00:00:000] +------ Shibatch -----
[00:00:00:000] | Source Sample-Rate: 44.1KHz
[00:00:00:000] | Dest. Sample-Rate: 48.0KHz
[00:00:00:000] | Attenuation : 0.0db
[00:00:00:000] +---------------------
[04:17:06:410] Conversion Completed !
[04:17:06:410] Actual Avg. Bitrate : 214kbps
[00:08:06:000] <-- Transcoding Duration

Logging ends : 05/07/04 , 14:55:43.

Observe the work finish a time-counter 4:17 hours, two times the original

timmming,generates a file 1,38 gb

Second convert file a 24 bit deep, this is the maximum alloweed for sonic foundry for encoding dolby digital.

Test a 16 byte deep input file 4,15 GB the program works.

Log

BeSweet v1.5b27 by DSPguru.
--------------------------

Logging start : 05/07/04 , 15:30:37.

C:\Programas de video\BeSweet\BeSweet.exe -core( -input d:\Trabajos de DVD\El retorno.wav

-output d:\Trabajos de DVD\El retorno6M -6ch -logfile C:\Programas de video\BeSweet\BeSweet.log ) -profile( ~~~~~ Default Profile ~~~~~ )

[00:00:00:000] +------- BeSweet -----
[00:00:00:000] | Input : d:\Trabajos de DVD\El retorno.wav
[00:00:00:000] | Output: FL, FR, SL, SR, C, LFE
[00:00:00:000] | Floating-Point Process: No
[00:00:00:000] | Source Sample-Rate: 48.0KHz
[00:00:00:000] +---------------------
[02:09:04:412] Conversion Completed !
[00:09:41:000] <-- Transcoding Duration

Logging ends : 05/07/04 , 15:40:19.

All files as the same length 709 MB but only FL and FR has sound.

Second test 24 byte deep imput file 6,23 GB, the program crashes

All files same length 3,04 KB

Log

BeSweet v1.5b27 by DSPguru.
--------------------------

Logging start : 05/07/04 , 17:53:47.

C:\Programas de video\BeSweet\BeSweet.exe -core( -input d:\Trabajos de DVD\El retorno246.wav

-output d:\Trabajos de DVD\El retorno246M -6ch -logfile C:\Programas de

video\BeSweet\BeSweet.log ) -profile( ~~~~~ Default Profile ~~~~~ )

[00:00:00:000] +------- BeSweet -----
[00:00:00:000] | Input : d:\Trabajos de DVD\El retorno246.wav
[00:00:00:000] | Output: FL, FR, SL, SR, C, LFE
[00:00:00:000] | Floating-Point Process: No
[00:00:00:000] | Source Sample-Rate: 48.0KHz
[00:00:00:000] +---------------------

[DSPguru]

dear alastor,
thank you for your report and kind works.

please, let's try to keep in focus.
the change in b27 is support for 32bit, floating-point, waves with filesize larger than 4gb. can you confirm/refute a bugfix ?

[alastor]

Sorry i mistake.

I only test 16 and 24 int, i d'dont test 32 floating.

Best regards.
Alastor

[kempfand]

DSPGuru: the change in b27 is support for 32bit, floating-point, waves with filesize larger than 4gb. can you confirm/refute a bugfix ?

I tested on 1 example and it works like a charm here :)

Y_6_32_.wav is a 6-channel 32-bit wav, having 4.56 GB (=4,903,317,560 bytes).
BeSweet v1.5b27 by DSPguru.
--------------------------

Logging start : 05/07/04 , 18:29:31.

C:\BeSweet\BeSweet.exe -core( -input m:\Y_6_32_.wav -output i:\Y_6_32_- -6chfloat -logfile C:\BeSweet\BeSweet.log )

[00:00:00:000] +------- BeSweet -----
[00:00:00:000] | Input : m:\Y_6_32_.wav
[00:00:00:000] | Output: FL, FR, SL, SR, C, LFE
[00:00:00:000] | Floating-Point Process: Yes
[00:00:00:000] | Source Sample-Rate: 44.1KHz
[00:00:00:000] +---------------------
[01:17:13:283] Conversion Completed !
[00:03:38:000] <-- Transcoding Duration

Logging ends : 05/07/04 , 18:33:09.
The resulting mono-wav's are all OK over the full length (01:17) and are packed perfectly thereafter with SurCode.



What is not working (but also was not the goal of the current bugfix) is the output to 16-bit.

Y_6_16_.wav is a 6-channel 16-bit wav, having 2.28 GB (=2,451,714,092 bytes).
BeSweet v1.5b27 by DSPguru.
--------------------------

Logging start : 05/07/04 , 18:34:06.

C:\BeSweet\BeSweet.exe -core( -input m:\Y_6_16_.wav -output i:\Y_6_16_- -6ch -logfile C:\BeSweet\BeSweet.log )

[00:00:00:000] +------- BeSweet -----
[00:00:00:000] | Input : m:\Y_6_16_.wav
[00:00:00:000] | Output: FL, FR, SL, SR, C, LFE
[00:00:00:000] | Floating-Point Process: No
[00:00:00:000] | Source Sample-Rate: 44.1KHz
[00:00:00:000] +---------------------<log-file stops here>

The 6 resultant mono-wav's seem to have the correct size on disc (389 MB; 408,616,960 bytes), but opening them with CoolEdit indicate wrongly a duration of 06:45 (instead of corret 1:17). Also, there is only sound up to about 01:17, the rest up to 06:45 is filled with data (but no audio).

Demuxing of the same 6-channel wav works by opening it in CoolEdit, and saving the de-muxed files with the corresponding channel-labelling, showing that it works in principle.

Not really a big deal, as the 32-bit route is now working.

Thanks for your quick help on this DSPGuru, I really appreciate.
Kind regards and a well-earned "Cheers",

Andreas

[DSPguru]

thank you for your reports :D !


@kempfand
i wonder if you use BeSweet's "-g peak" and the integrated surcode support ? and if you don't, i'm curious, why don't you ?

[alastor]

Excuse me, i mistake.

I d'ont test 32 bytes float, only 16 and 24.
I suppose that modification is for all deep.

Sorry

[kempfand]

DSPGuru: I indeed use the integrated SurCode support on a standard basis, but didn't do so for the recent demux-test with BeSweet v1.5b27, since I wanted to test the bugfix-item only.

However, I currently do not use "-g peak" for the following reasons:

a) For nearly 2 months now, I use the Farina filters to create B-Format (i.e. not B-Pan or Panorama), and I use custom-made signed filters for the decoding to 5 speakers (i.e. 15 parallel convolutions using Prisinte Space, that is 3 (for W,X,Y) times 5 (for speakers). For both steps, it is key not to use any auto-gain feature of the convolver, which has made me a bit "paranoid" in terms of "hands off" from anything which has auto/-gain associated.

b) More factual, for about 4 months now, I use a budget multichannel soundcard with 6 Logitech speakers at the DAW. Whenever I run or create a bidule, I test-listen to things before I run the bidule in offline mode to write the full production. Doing so, I basically gain-adjust in bidule already to give a workable multichannel wav.

c) Lastly, I remember some remarks that the peak-chunk data was not written correcly in one of the earlier betas of the Bidule SW. Maybe it's working now, but I frankly didn't check.

Open for any suggestions on my side ...
Kind regards,

Andreas

[DSPguru]

ok, i c.
i wish this peak issue had been tested, and that the guide been updated to use 32bit multichannel waves as input, 32bit mono waves as output (mentioning the option to transcode to dts directly), and the usage of -g peak instead of -g max. (btw, if BeSweet doesn't find a peak chunk, it will apply -g max instead. however, if the peak chunk is curroupted, i'm not sure what happens..)

[kempfand]

I agree the guide needs update, for sure for the 32 bit in/out and direct transcode to dts. I'll talk to EoH once he is back from is long vacation.

Kind regards,
Andreas

[tebasuna51]

I hope to clarify (despite my english) any questions in this old post.

In the canonical header of a wav there are a four bytes field to put the length of data, then any wav > 4 GB have a incorrect header (4 GB = 2^32 bytes, 2^32 is the maximum unsigned integer in four bytes).

When a soft is writing a wav and arrive to 4 GB can abort (overflow in the counter of bytes) or can continue writing to the finish and fill this field with the maximum ( 4 GB, Aacdrop) or with the excess over 4 GB (faad, foobar, ...).

(Really there are two fields, one for the length of data sound and other for the length of file)

When a soft try to open a wav > 4 GB can refuse them like invalid (SoftEncode, WaveWizard, ...), can inform of a incorrect length (only the excess over 4 GB) or can ignore this fields and assume like length the real length of the file (Tranzcode, BeSplit, BeSweet only for 32 bits-float, ...).

Maybe BeSweet, with 16 bits-integer, use this field like signed integer and can manage only 2^31 (2 GB)?

BeSplit can demux a wav6 > 4 GB in six mono wav, but have a bug in other header field (BlockAlign) and make unusable mono wav. For more info:
http://forum.doom9.org/showthread.php?t=24690&page=2&pp=20

Is interesting to know this limits in function of the time of the wav6. For a wav 16 bits, 48 KHz:
62m -> 1.99 GB
63m -> 2.03 GB
124m -> 3.99 GB
125m -> 4.02 GB

(Limits for 44.1 KHz: 67/68 m and 135/136 m.)
(For 32 bits half time)

[johnman]

Just to be complete on this: The size is stored in the frames so that there can be tags,info and other data placed behind the sampledata. A couple of programs put a tag behind the wav like "This file has been made with program XYS by user ABC". If you ignore the size this text wil be considered as sampledata, which will cause a loud tick. If there is no data behind the samples you can safely use the full file.

If the size is greater then 4 GB (or 2GB in ww v2b) wavewizard sets the size to 0, but it will continue to function normaly. In a couple of weeks there will be a new version which can ignore the size stored in the wav, and use the complete file. I already implemented this, but im working on some extra functionaliity which takes a while to finish.

[tebasuna51]

@johnman
Yes, I know this extrachunks at the end of the file. Is a problem, but a minor problem (Not problem with Foobar, BeSweet, Aacdrop, Faad, ...).

In the future version of WaveWizard, is supported wav Faad generated? (WAVE_FORMAT_EXTENSIBLE with AudioFormat 0xFFFE).
Now it report "Unknown".

[johnman]

@johnman
Yes, I know this extrachunks at the end of the file. Is a problem, but a minor problem (Not problem with Foobar, BeSweet, Aacdrop, Faad, ...).

I know its not a big problem. The only program i have seen doing this is goldwave (or something like that).


In the future version of WaveWizard, is supported wav Faad generated? (WAVE_FORMAT_EXTENSIBLE with AudioFormat 0xFFFE).
Now it report "Unknown".

WAVE_FORMAT_EXTENSIBLE will be supported in a future version, maybe even the next. This format is already in the preferences. But i didnt had any good files to test it with at the time, so i removed WAVE_FORMAT_EXTENSIBLE support again ,and i disabled it as an output format in the preferences.