Zeroing out sectors will improve compression? - TiVo Community Forum Archive 1

READ ONLY ARCHIVES

Welcome to the TiVo Community Forum Archive
This archive covers threads on TiVo Community Forum that have not been posted to from the start until June 30, 2004. Any thread that has a post made to it between 7/1/04 and 12/31/05, that had not been posted to, will be found in Archive 2.
This is a READ ONLY site.

Search | ARCHIVE 2 | MAIN SITE

TiVo Community Forum Archive 1 : Powered by vBulletin version 2.2.8

TiVo Community Forum Archive 1 > Underground Playground > TiVo Underground

>>> Zeroing out sectors will improve compression? <<<

Forum Jump:

Search this Thread:

Last Thread Next Thread

Author

Thread ---> Show Printable Version | Email this Page | Subscribe to this thread

07-24-2000 01:51 PM

kren2000
Guest

Registered: Not Yet
Location:
Posts: N/A

Zeroing out sectors will improve compression?

Hi -

I'm about to launch into expanding my 14hr as well. Thanks to everyone who has posted such great insights and code!

I'm realizing that the backup process involves a direct copy of the drive, empty sectors and all. Since my TiVo is no longer virgin, these sectors used to contain image date, which is highly non-compressible.

Wouldn't it be possible to write a small utility that zeros out all non-utlized sectors? That should improve the compression rate greatly.

Any volunteers? I don't have the appropriate linux setup to make this happen, but would love for someone to give it a try.

Karen

POST #1 | Report this post to a moderator | IP: Logged

07-24-2000 03:03 PM

MikeLaw
Senior Member

Registered: Jun 2000
Location: Richmond, VA, USA
Posts: 29

quote:
Originally posted by kren2000:
Hi -

I'm about to launch into expanding my 14hr as well. Thanks to everyone who has posted such great insights and code!

I'm realizing that the backup process involves a direct copy of the drive, empty sectors and all. Since my TiVo is no longer virgin, these sectors used to contain image date, which is highly non-compressible.

Wouldn't it be possible to write a small utility that zeros out all non-utlized sectors? That should improve the compression rate greatly.

Any volunteers? I don't have the appropriate linux setup to make this happen, but would love for someone to give it a try.

Karen

Boy do we need a FAQ! There are a number of posts on this issue, but the bottom line is that it is more complicated that writing a bunch of zeros and no one has a solution at this time. The best solution at the moment is to make a complete sector by sector backup of your drive to another one of similar capacity. Most people are buying an indentical Quantum and running on the new drive, preserving the original drive in a static-proof bag.

------------------
....mike

POST #2 | Report this post to a moderator | IP: Logged

07-24-2000 07:45 PM

tom_h
TiVo Forum Special Member

Registered: Jul 2000
Location:
Posts: 7

Silly idea. If you take a virgin hard disk from the store, do a file by file copy onto it, and then take a backup of that, would the file by file copy not have taken the leftover image data from the old disk since there was nothing in the file system pointing to it? the resulting image would be small when compressed.

Of course i say this not being completely sure of the ability to mount and copy the tivo file systems and also being unaware of whether a file by file copy vs image copy could produce something in the way of a usable backup. In the event none of this is worthwhile, perhaps someone has a little tool that would open a file in the image partition and write it with zeros until the disk was full and then delete it?

POST #3 | Report this post to a moderator | IP: Logged

07-24-2000 07:56 PM

Otto

Registered: Mar 2000
Location:
Posts: 1031

quote:
Originally posted by tom_h:
Silly idea. If you take a virgin hard disk from the store, do a file by file copy onto it, and then take a backup of that, would the file by file copy not have taken the leftover image data from the old disk since there was nothing in the file system pointing to it? the resulting image would be small when compressed.

True, except that you can't mount most of the partitions on the Tivo. And nobody has figured out what the MFS partition looks like yet. The backup usually involves the "dd" command and is a byte for byte copy.

quote:
In the event none of this is worthwhile, perhaps someone has a little tool that would open a file in the image partition and write it with zeros until the disk was full and then delete it?

Not that simple.
a) Nobody know what the MFS partition (hdx10 and hdx11) is up to yet.. Some weird formatting there.
b) There's more in there than just the MPEG files, and zeroing it out would probably kill your Tivo. Things like sounds and graphics for the menus are in there too.

However, someone mentioned an idea I thought was good. Record a black screen (or blue.. or white, whatever) for all the time you can on your Tivo. Fill it up. Use a channel that has long shows, like the TV Guide channel, and just plug your VCR that outputs a blue screen or some such into the Tivo inputs. Then record it until it is full. Then do a backup and see how well that compresses. Worth a shot anyway. It may not be zeros, but it's gonna all be the same.

------------------
Moderator - AVS Forum - Tivo Underground
"If once you start down the dark path, forever will it dominate your destiny. Consume you it will!" -- Yoda

[This message has been edited by Otto (edited July 24, 2000).]

POST #4 | Report this post to a moderator | IP: Logged

07-24-2000 07:57 PM

sorphin
Advanced Member

Registered: Jun 2000
Location: None Of Your Business
Posts: 7

quote:
Originally posted by Otto:
Not that simple.
a) Nobody know what the MFS partition (hdx10 and hdx11) is up to yet.. Some weird formatting there.
b) There's more in there than just the MPEG files, and zeroing it out would probably kill your Tivo. Things like sounds and graphics for the menus are in there too.

a) correct, but we're working on it, i DO know that it's inode/fsid based.

b) yup, all the animations, et al are on the media partition.

------------------
-ds
Member of the #TiVo Enhancement Project
(Not to be confused with the SourceForge Group)

POST #5 | Report this post to a moderator | IP: Logged

07-24-2000 08:08 PM

tom_h
TiVo Forum Special Member

Registered: Jul 2000
Location:
Posts: 7

Hmm, but since tivo doesnt use VBR wouldnt the mpeg compression of anything, even a blue screen, take up the same 'space' as an mpeg file and since mpeg isnt very compressible, it would take the same space??

Tough luck on the file system. if someone can get the FS mounted then the zeroing idea would work...or if we can at least figure out whats in use and what isnt the remainder can be wiped.

Wish i could help, but my last systems programming involved VAX/VMS and the only unix experience i have was in writing the device drivers for Ultrix v1...

POST #6 | Report this post to a moderator | IP: Logged

07-24-2000 09:23 PM

MikeLaw
Senior Member

Registered: Jun 2000
Location: Richmond, VA, USA
Posts: 29

quote:
Originally posted by tom_h:
Hmm, but since tivo doesnt use VBR wouldnt the mpeg compression of anything, even a blue screen, take up the same 'space' as an mpeg file and since mpeg isnt very compressible, it would take the same space??

I should start with a warning that I don't know anything about what I am talking about, but I don't intend to let that stop me. It is my understanding that the advantage of VBR would be that the perfectly still input signal where each region matched its neighbor (the "blue screen") you could manage to compress 100 hours (or whatever) in the space that a TiVo uses for 12 hours of "Best" video. But since the TiVo uses the same bitrate no matter what, there should be some wasted space when it is fed a perfectly compressable image. How much wasted space and how well it compresses with gzip are an unknown.

At a bare minimum, you are ensuring that each "I" Frame (the static frames used as a baseline) are the same. At the least they have to pack OK. And there should be basically no DCT coefficients since each frame is the same looked at by region, by luminance / chrominance or by motion. My understanding is that encoder is responsible for padding the bit stream to ensure that the bit rate is constant and I assume it packs it with zeros or a constant. The only good way to find out is to test!

------------------
....mike

POST #7 | Report this post to a moderator | IP: Logged

07-26-2000 08:29 PM

bonehead
TiVo Forum Special Member

Registered: Apr 2000
Location: Des Moines, IA
Posts: 424

I ran a quick little test today, and was rather surprised by the results. Not at all what I expected.

What I did was:
1. Use VirtualDub and the FrameTweaker plugin to create a 5 minute MJPEG compressed AVI file of solid black frames.
2. Use the Ligos compressor to compress it into an MPEG2 video stream. (no audio)
3. Compress the MPEG stream using both gzip and bzip2, at the default compression levels.

Here's what I got:
test.m2v 59,995 KB
test.m2v.gz 234 KB
test.m2v.bz2 42 KB

Keep in mind that the mpeg stream was created using exceptionally clean source material (computer generated all-black frames). On the TiVo, you're going to be recording from an analog input source, which will likely introduce some noise into the frames. Even a small amount of noise in the signal could greatly reduce the compressability of the recorded mpeg stream.

[This message has been edited by bonehead (edited July 26, 2000).]

POST #8 | Report this post to a moderator | IP: Logged

07-26-2000 08:48 PM

Otto

Registered: Mar 2000
Location:
Posts: 1031

quote:
Originally posted by bonehead:
I ran a quick little test today, and was rather surprised by the results. Not at all what I expected.
Here's what I got:
test.m2v 59,995 KB
test.m2v.gz 234 KB
test.m2v.bz2 42 KB

That's pretty much what I would have expected. Consider that you were using a Fixed Bit Rate, and any bits not used in the frames were padded out. VBR would be a lot different, as your source file would have probably been ~300 KB instead of 60 MB.

------------------
Moderator - AVS Forum - Tivo Underground
"If once you start down the dark path, forever will it dominate your destiny. Consume you it will!" -- Yoda

POST #9 | Report this post to a moderator | IP: Logged

07-26-2000 08:54 PM

sorphin
Advanced Member

Registered: Jun 2000
Location: None Of Your Business
Posts: 7

quote:
Originally posted by tom_h:
if someone can get the FS mounted then the zeroing idea would work..

you can't mount MFS, it not a 'normal FS' persay...

------------------
-ds
Member of the #TiVo Enhancement Project
(Not to be confused with the SourceForge Group)

POST #10 | Report this post to a moderator | IP: Logged

07-26-2000 08:57 PM

sorphin
Advanced Member

Registered: Jun 2000
Location: None Of Your Business
Posts: 7

i misunderstood what you said you had done in #tivo... (you made it sound like you had done this ON the tivo)... and notice how much better bzip2 compresses? (and luckily, the tivo has bz2 on it if we ever need it)..

quote:
Originally posted by bonehead:
I ran a quick little test today, and was rather surprised by the results. Not at all what I expected.

What I did was:
1. Use VirtualDub and the FrameTweaker plugin to create a 5 minute MJPEG compressed AVI file of solid black frames.
2. Use the Ligos compressor to compress it into an MPEG2 video stream. (no audio)
3. Compress the MPEG stream using both gzip and bzip2, at the default compression levels.

Here's what I got:
test.m2v 59,995 KB
test.m2v.gz 234 KB
test.m2v.bz2 42 KB

Keep in mind that the mpeg stream was created using exceptionally clean source material (computer generated all-black frames). On the TiVo, you're going to be recording from an analog input source, which will likely introduce some noise into the frames. Even a small amount of noise in the signal could greatly reduce the compressability of the recorded mpeg stream.

[This message has been edited by bonehead (edited July 26, 2000).]

------------------
-ds
Member of the #TiVo Enhancement Project
(Not to be confused with the SourceForge Group)

POST #11 | Report this post to a moderator | IP: Logged

All times are GMT. The time now is 11:21 PM.

Last Thread Next Thread

>>> Zeroing out sectors will improve compression? <<<

TiVo Community Forum Archive 1 > Underground Playground > TiVo Underground

Search The Internet

Show Printable Version |

Email this Page |

Subscribe to this thread

Forum Jump:

Search this Thread:

Forum Rules:

You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

HTML code is OFF
vB code is ON
Smilies are ON
[IMG] code is ON

< Contact Us - TiVo Community Forum Archive 1 >

Powered by: vBulletin Version 2.2.8
Copyright ©2000, 2001, Jelsoft Enterprises Limited.
(C)opyright - All Rights Reserved. No information may be posted elsewhere without written permission.
TiVo� is a registered trademark of TiVo Inc. This site is not affiliated with TiVo Inc.
Page generated in 0.03052998 seconds (84.94% PHP - 15.06% MySQL) with 25 queries.

Spider History Index