TiVo Community Forum Archive 1
READ ONLY ARCHIVES

Welcome to the TiVo Community Forum Archive
This archive covers threads on TiVo Community Forum that have not been posted to from the start until June 30, 2004.  Any thread that has a post made to it between 7/1/04 and 12/31/05, that had not been posted to, will be found in Archive 2.
This is a READ ONLY site.

  Search | ARCHIVE 2 | MAIN SITE

TiVo Community Forum Archive 1 : Powered by vBulletin version 2.2.8 TiVo Community Forum Archive 1 > Underground Playground > TiVo Underground
>>> cachecard - its alive <<<

Pages (14): [1] 2 3 4 Next » ... Last »  
Forum Jump:
Search this Thread:
Last Thread   Next Thread
Author
Thread ---> Show Printable Version | Email this Page | Subscribe to this thread Post New Thread    Post A Reply
Gultig is offline Old Post 05-22-2003 01:08 AM
Click Here to See the Profile for Gultig Find more posts by Gultig Add Gultig to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
Gultig
TiVo Forum Special Member

Registered: Jul 2000
Location: a state of unconsciousness
Posts: 29

This is one of a couple of posts from yesterday that I thought were important.

gultig

quote:
Originally posted by jafa
cachecard - its alive
Hi all,

Its alive!

Tivo kernel logs:

sh-2.02# /sbin/insmod -f cachecard.o
cachecard: debug level 3 enabled
cachecard: found 4 chip groups
cachecard: found 2 banks
cachecard: found 2048 rows
cachecard: found 512 cols
cachecard: memory found: 32 MB
cachecard: driver loaded successfully

adding to cache (write): 000fffb3:000ffff4
adding to cache (write): 000ffff5:000ffffd
adding to cache (write): 000ffffe:000ffffe
adding to cache (write): 000fffff:000fffff
adding to cache (read): 000e72ad:000e72ae
found in cache: 000e72ad:000e72ae
adding to cache (read): 000e8b9d:000e8ba4
adding to cache (read): 000ea305:000ea30a
found in cache: 000e72ad:000e72ae
found in cache: 000e8b9d:000e8ba4
found in cache: 000ea305:000ea30a
found in cache: 000e72ad:000e72ae
found in cache: 000e8b9d:000e8ba4
found in cache: 000ea305:000ea30a
adding to cache (write): 000eb81d:000eb824
adding to cache (write): 000eb9ed:000eb9f2
found in cache: 000e72ad:000e72ae
...

This is running with ay 32MB DIMM mapped over the 512MB database.

It is currently doing a full self-verify which means that when it detects a cache hit it reads the harddrive anyway and verifies the integrity of the cache.

Nick





quote:
Originally posted by jnk27
Yahoo!!!
Can we move the delivery dates up from mid-June? (Partially) kidding.





quote:
Originally posted by bsnelson
So, the big question: Will a 512MB DIMM effectively cache the entire database?

OK, maybe it's *A* big question; I'm sure there are bigger ones!

Brad


__________________
(5) Philips DSR6000R (188, 165, 146, 106 and 67 hours),
(1) Hughes HDVR2 (35 hours, for now),
(2) Philips HDR112 (196 and 14 hours)





quote:
Originally posted by jafa

quote:
--------------------------------------------------------------------------------
Originally posted by bsnelson
So, the big question: Will a 512MB DIMM effectively cache the entire database?
--------------------------------------------------------------------------------


Hi Brad,

Sure, there is probably a good case to pre-load the entire database into the cache once at startup although this will add about 3 minutes to the boot time.

I will add some more internal counters so that we can measure things like how often it would help if we also cached the swap partition, var partition, or root partition.

Nick




quote:
Originally posted by joker81
When will the cache update. After its daily call? or some random time during the night?. Will it just dump the current cache and reload the whole DB or just the parts edited?


__________________
Series 2 80 Hour Unmodified(Girlfriend won't let me)
Series 1 HDR312 waiting for it and turbonet to come in gonna install 120GB HD.





quote:
Originally posted by jafa
Hi,

The current code does write-through caching - all writes go to the harddrive. The tivo tends to do many more reads than writes so it will make a good difference.

I plan to update this so that the write accesses are immediatly written to the cache and scheduled to be written to the harddrive as if it would otherwise have normally be scheduled anyway, with the exception that the tivo can still make forward progress rather than having to wait for the write to complete.

Nick




POST #1 | Report this post to a moderator | IP: Logged

Lord Nimon is offline Old Post 05-22-2003 02:11 AM
Click Here to See the Profile for Lord Nimon Find more posts by Lord Nimon Add Lord Nimon to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
Lord Nimon
Delicious!

Registered: Apr 2001
Location:
Posts: 342

quote:
This is running with ay 32MB DIMM mapped over the 512MB database.


Is the database always 512MB? If not, how can I tell how big it is?

If the database is exactly 512MB, then you'd probably want a 1GB DIMM since you'll need space for stuff other than the database.

Besides, I would assume that playing video will fill up the cache very quickly, forcing the database to be flushed. Unless, of course, the video is not cached.

POST #2 | Report this post to a moderator | IP: Logged

bsnelson is offline Old Post 05-22-2003 03:00 AM
Click Here to See the Profile for bsnelson Visit bsnelson's homepage! Find more posts by bsnelson Add bsnelson to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
bsnelson
TiVo Forum Special Member

Registered: Oct 1999
Location: Allen, TX, USA
Posts: 4810

jafa has publically stated that it caches the database only. It wouldn't make musch sense to cache the video, anyway.

Brad

__________________
(3) Philips DSR6000R (188, 146 and 106 hours, in hibernation),
(2) Hughes HDVR2 (221 and 35 hours),
(1) Philips DSR7000/17 (144 hours),
(1) Samsung SIR4040R (35 hours)

POST #3 | Report this post to a moderator | IP: Logged

jafa is offline Old Post 05-22-2003 03:00 AM
Click Here to See the Profile for jafa Visit jafa's homepage! Find more posts by jafa Add jafa to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
jafa
TiVo Forum Special Member

Registered: Jan 2002
Location:
Posts: 2223

Hi,

Right, there is not a lot of point caching the video data.

The system is structured so that the driver has control over what gets cached and you can specify which partitions/sector ranges to cache.

The main two non-video partitions that are using during normal operation are the database and var. The root and swap tend to be used less.

I hope to be able to move the setup from my test tivo to my tv-watching tivo so that I can see how it looks under real usage.

Nick

__________________
Silicondust - Tivo CacheCARD/Turbonet/Airnet
Roomba Robotics - Roomba Community

POST #4 | Report this post to a moderator | IP: Logged

bsnelson is offline Old Post 05-22-2003 03:09 AM
Click Here to See the Profile for bsnelson Visit bsnelson's homepage! Find more posts by bsnelson Add bsnelson to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
bsnelson
TiVo Forum Special Member

Registered: Oct 1999
Location: Allen, TX, USA
Posts: 4810

Thanks for the update, Nick. We're anxious to hear some numbers!

Brad

__________________
(3) Philips DSR6000R (188, 146 and 106 hours, in hibernation),
(2) Hughes HDVR2 (221 and 35 hours),
(1) Philips DSR7000/17 (144 hours),
(1) Samsung SIR4040R (35 hours)

POST #5 | Report this post to a moderator | IP: Logged

Ken Raeburn is offline Old Post 05-22-2003 03:28 PM
Click Here to See the Profile for Ken Raeburn Find more posts by Ken Raeburn Add Ken Raeburn to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
Ken Raeburn
Just some guy

Registered: Jan 2002
Location: Somerville, MA
Posts: 22

quote:
I plan to update this so that the write accesses are immediatly written to the cache and scheduled to be written to the harddrive as if it would otherwise have normally be scheduled anyway, with the exception that the tivo can still make forward progress rather than having to wait for the write to complete.


At what level is optimization done on the ordering of writes to the disk? Is any of it at the IDE controller or disk level? If so, wouldn't this change (causing the kernel to no longer wait for a block to be written before going on with other work, like scheduling certain other blocks to be written) potentially cause database consistency problems if the power is removed at just the wrong time? Strict ordering of certain write operations is one way of helping keep consistency....

POST #6 | Report this post to a moderator | IP: Logged

jafa is offline Old Post 05-22-2003 11:26 PM
Click Here to See the Profile for jafa Visit jafa's homepage! Find more posts by jafa Add jafa to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
jafa
TiVo Forum Special Member

Registered: Jan 2002
Location:
Posts: 2223

Hi,

Right, it is importiant that all real hdd writes are in order or that the dependancies are understood.

With both approaches above, all disk writes remain in order, even if the cachecard driver allows the tivo to continue and only schedules the write.

Think of it this way, by allowing the tivo to think it was written, all it can do is read the disk/cache (harmless) and write the disk/cache (which will still be in order even if the first write hasn't gone to disk yet).

The trick is making sure that you don't loose the cache entry until the write has actually happened.

Nick

__________________
Silicondust - Tivo CacheCARD/Turbonet/Airnet
Roomba Robotics - Roomba Community

POST #7 | Report this post to a moderator | IP: Logged

NuthinBetter is offline Old Post 05-23-2003 12:18 AM
Click Here to See the Profile for NuthinBetter Find more posts by NuthinBetter Add NuthinBetter to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
NuthinBetter
Member

Registered: Dec 2000
Location: Westfield, NJ
Posts: 52

I don't know much about TiVo's DB, but work with enterprise DBs every day...

I want to echo the power-off sentiment, above...

Won't the DB be hosed if you unplug the box before the cached data hits the physical disk? A lot of cache and RAID controllers throw-in a battery option to keep the cache in case of power fail (of course you have to somehow flush the cache on re-start before TiVo knows it's missing).

I guess it's pretty rare, so maybe the better question is: How easy is it to rebuild the db and not lose too much info?

__________________
--AJ

The UNIX Guru's View of Sex:#
unzip ; strip ; touch ; finger ; mount ; fsck ; more ; yes ; umount ; sleep

POST #8 | Report this post to a moderator | IP: Logged

jafa is offline Old Post 05-23-2003 12:25 AM
Click Here to See the Profile for jafa Visit jafa's homepage! Find more posts by jafa Add jafa to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
jafa
TiVo Forum Special Member

Registered: Jan 2002
Location:
Posts: 2223

Hi,

The database is designed to be ok even if the power gets pulled at any time.

The cachecard doesn't delay the writes to the disk so nothing has changed with respect to the power being pulled.

For PCs - most filesystems can't handle the power being pulled (FAT, NTFS, and ext2 for example). That is the motivation for the ext3 filesystem (which I would recommend now that it is stable).

Nick

__________________
Silicondust - Tivo CacheCARD/Turbonet/Airnet
Roomba Robotics - Roomba Community

POST #9 | Report this post to a moderator | IP: Logged

rogo is offline Old Post 05-23-2003 06:42 AM
Click Here to See the Profile for rogo Find more posts by rogo Add rogo to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
rogo
Advanced Member

Registered: Dec 1999
Location:
Posts: 748

NTFS can't handle the power being pulled... Hmmm... Intriguing, I thought MS was finally able to do that.

Any sense of what the card will offer in terms of performance yet?

POST #10 | Report this post to a moderator | IP: Logged

jafa is offline Old Post 05-23-2003 08:53 AM
Click Here to See the Profile for jafa Visit jafa's homepage! Find more posts by jafa Add jafa to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
jafa
TiVo Forum Special Member

Registered: Jan 2002
Location:
Posts: 2223

Wow, ok, I didn't know that NTFS could handle being spontaneously rebooted. Certinally my FAT32 mahine has errors every time it spontaneoulsy reboots, maybe I will upgrade it to NTFS.

Just to be sure lets run a test... pull the power lead to your windows server and see what happens

I will hopefully have some performance data this weekend (I am currently running with full debugging and integrity checking).

Nick

__________________
Silicondust - Tivo CacheCARD/Turbonet/Airnet
Roomba Robotics - Roomba Community

POST #11 | Report this post to a moderator | IP: Logged

Worf is offline Old Post 05-23-2003 11:24 AM
Click Here to See the Profile for Worf Visit Worf's homepage! Find more posts by Worf Add Worf to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
Worf
Senior Member

Registered: Sep 2000
Location:
Posts: 422

Heh. NTFS was designed to be able to handle power down events gracefully (it is journalled). But it appears that Microsoft, in their typical wisdom, amanged to screw that up too (first was using the MMU incompetently, then screwing up the device driver models, and filesystems).

So while NTFS is designed to handle power cycles gracefully, it is highly not recommended to do that.

POST #12 | Report this post to a moderator | IP: Logged

Ken Raeburn is offline Old Post 05-23-2003 12:55 PM
Click Here to See the Profile for Ken Raeburn Find more posts by Ken Raeburn Add Ken Raeburn to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
Ken Raeburn
Just some guy

Registered: Jan 2002
Location: Somerville, MA
Posts: 22

quote:
Originally posted by jafa
With both approaches above, all disk writes remain in order, even if the cachecard driver allows the tivo to continue and only schedules the write.

Think of it this way, by allowing the tivo to think it was written, all it can do is read the disk/cache (harmless) and write the disk/cache (which will still be in order even if the first write hasn't gone to disk yet).



So, IDE controllers and disks aren't allowed to re-order the writes (say, to reduce seek overhead)? Only the OS device driver?

(Just trying to get very clear on this...)

POST #13 | Report this post to a moderator | IP: Logged

Lord Nimon is offline Old Post 05-23-2003 08:31 PM
Click Here to See the Profile for Lord Nimon Find more posts by Lord Nimon Add Lord Nimon to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
Lord Nimon
Delicious!

Registered: Apr 2001
Location:
Posts: 342

Write re-ordering is common in SCSI controllers. I think the consensus is that if power is commonly lost during a write, the solution is to fix the file system and the operating system, not the hardware.

POST #14 | Report this post to a moderator | IP: Logged

TivoTechie is offline Old Post 05-23-2003 09:46 PM
Click Here to See the Profile for TivoTechie Find more posts by TivoTechie Add TivoTechie to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
TivoTechie
Member

Registered: Apr 2000
Location:
Posts: 1

Post Write Caching & Journaling

quote:

see: http://www.linuxjournal.com/article.php?sid=4466

Write Caching

For performance benchmarks, some of the new drives have write-back caching by default. This means the drive reports a write is completed before it is actually on the media. The block is still in the drive's cache, where the writes can be reordered. If this happens, metadata changes might be written before the log commit blocks, leading to corruption if the machine loses power. It is very important to disable write-back caching on both IDE and SCSI drives.

Some hardware RAID controllers provide a battery-backed write-back cache that preserves the cache contents if the system loses power. These should be safe to use, but the cache battery should be checked often. A dramatic performance increase can be seen with these write caches, especially for log intensive applications like mail servers.




Basicly the problem with buffering writes on a journaled filesystem w/o battery backup of the buffer/cache is you could still get into an inconsistent state.

The simplified premise is as follows:
write.journal(attempting_write[to_location])
write.data(data_to_write[to_location])
write.journal(success)

if power fails while writing, when the system comes up it sees a non completed success on a transaction and rolls the change back.

the problem occurs if you say "hey I wrote the journal entry" (cachecard) but then don't actually write it, depending on what region of the disk you are caching for, you might end up writing out video (or worse data/metadata [like the stream/disk space usage map) and put mfs in an iinconsitent state.. this example is "proabbly" fixable by fsfix/mfscheck but would proabbly cause a temporary GSoD

My reccomendation would be to write-through if you are going to be doing any caching of data writen to the disks. And only allow cache hits to occur after you are sure the disk has returned from a successful write.

Assuming you don't reorder the writes, I belive this would end up "ok" but I have to think through a couple more scenarios until I'm convinced it's completly safe.

reordering would definetly be more problematic, I can think of a few scenarios where if it wasn't properly handeled you'd attempt a write, have the cachecard buffer and say it completed, get a cache hit to the area that you thought you wrote to, go to perform another write to the same region, have that cached also, if the two got reordered, you could end up with the wrong checksum finally written as meta data, thus making the FS inconsistent.

Obviously proper caching code would handel the above (ie flush cached data for pending writes, and don't reorder writes to the same block)

It may still be possible if you are performing a write across multiple blocks, if you cache and reorder, you might end up in a weird state where you have pending writes that have been reorders, but need to perform a checksum before all the writes have occured, this checksum would be different then the expected one, thus flagging the FS as inconsistent. (Again, proper cache code would proabbly deny cache hits, or read through for pending writes)

Just my random thoughts.

-TT

POST #15 | Report this post to a moderator | IP: Logged

jafa is offline Old Post 05-24-2003 12:24 AM
Click Here to See the Profile for jafa Visit jafa's homepage! Find more posts by jafa Add jafa to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
jafa
TiVo Forum Special Member

Registered: Jan 2002
Location:
Posts: 2223

Hi,

You are right, you have to be very careful about reordering and this sould only be done in the driver/OS where it understands what can and cannot be reordered.

The cachecard does not do reordering or fully optimized write-back caching for this reason.

Think about it this way...

1) The only thing that matters is writes to the disk. Reads are harmless as they do not store state that would be persistant if power is lost.

2) The cachecard does not defer the writes or reorder the writes. The write request to the hdd is fed to the ide driver in the exact same order whether it was snooped by the cache or not.

3) The cachecard can safely lie and say a write is complete as all that can happen is that a read can come along (which is harmless as no state is stored if the power is lost), or a write can come alone (which will still be in order so power loss makes no difference).

Even if it is writing a cached journal entry for video that isn't cached then the writes are still in order.

There are only two corner cases:

1) The tivo writes a sector, then requests to read the sector before the previous write is really commited to disk. The requirement is that the data is returned from the cache (which will happen anyway because that is the purpose of the cache).

2) A cache entry relating to a write must remain valid until the write is actually committed to disk in order to satify the above requirement.

The first version of the driver will be write-through only.

The above optimization makes sense if you have enough RAM to hold the entire database thus allowing the cache tag to be locked.

Nick

__________________
Silicondust - Tivo CacheCARD/Turbonet/Airnet
Roomba Robotics - Roomba Community

Last edited by jafa on 05-24-2003 at 12:30 AM

POST #16 | Report this post to a moderator | IP: Logged

rogo is offline Old Post 05-24-2003 06:15 PM
Click Here to See the Profile for rogo Find more posts by rogo Add rogo to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
rogo
Advanced Member

Registered: Dec 1999
Location:
Posts: 748

<<Wow, ok, I didn't know that NTFS could handle being spontaneously rebooted. Certinally my FAT32 mahine has errors every time it spontaneoulsy reboots, maybe I will upgrade it to NTFS.>>

Nick, the fact is that NTFS is, oh, at least 100x more robust on this than FAT32. I really don't miss the perpetual Scandisk-ing on bad restarts from the FAT does and the "dangling clusters" they'd always provide.

I recommend NTFS for a million reasons. There is almost zero downside (you can't boot older non-NTFS-compabible OSes on those partitions) and it greatly increases reliability.

Perfect? I doubt it. Better? Immeasurably.

Mark

------

With Microsoft, the third version's the charm.

POST #17 | Report this post to a moderator | IP: Logged

TheD0CTOR is offline Old Post 05-24-2003 07:10 PM
Click Here to See the Profile for TheD0CTOR Find more posts by TheD0CTOR Add TheD0CTOR to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
TheD0CTOR
Member

Registered: Nov 2002
Location:
Posts: 40

quote:
Originally posted by Worf
Heh. NTFS was designed to be able to handle power down events gracefully (it is journalled). But it appears that Microsoft, in their typical wisdom, amanged to screw that up too (first was using the MMU incompetently, then screwing up the device driver models, and filesystems).

So while NTFS is designed to handle power cycles gracefully, it is highly not recommended to do that.



Much like the wonderfully comical, "Now installing Operating System. If computer should stop working, please turn power off and back on". Man them boys at MS sure have a great sense of humor!

-Doc



What do you mean it's not a joke?

POST #18 | Report this post to a moderator | IP: Logged

TK-421 is offline Old Post 05-28-2003 07:40 PM
Click Here to See the Profile for TK-421 Visit TK-421's homepage! Find more posts by TK-421 Add TK-421 to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
TK-421
Stormtrooper

Registered: Aug 2000
Location: Death Star
Posts: 39

Just a question.. would there be any performance boost switching your memory interface from SDRAM to DDR SDRAM? Or would the increased cost outweigh the possible benefit?

POST #19 | Report this post to a moderator | IP: Logged

jafa is offline Old Post 05-28-2003 08:53 PM
Click Here to See the Profile for jafa Visit jafa's homepage! Find more posts by jafa Add jafa to your buddy list Show Printable Version Edit/Delete Message Reply w/Quote
jafa
TiVo Forum Special Member

Registered: Jan 2002
Location:
Posts: 2223

Hi,

Things a progressing well - it is working 100% in a SA and I am working on the DTivo support (unkile the SA, DTivos uses UDMA).

My SA test system pre-loads the entire database into a 512MB DIMM.... the menus are noticibly faster and it works reliably. I need to finish DTivo support to test the speed up of re-arranging season passes (I don't have cable tv).

TK-421 - Sorry but the cpu is 54MHz (SA) and the IDE interface is slower again. DDR memory is incompatable with PC133 memory including a different connector so it is not possible for a board to support both.

I will hopefully be making some more beta boards very soon for wider evaluation.

Nick

__________________
Silicondust - Tivo CacheCARD/Turbonet/Airnet
Roomba Robotics - Roomba Community

POST #20 | Report this post to a moderator | IP: Logged

All times are GMT. The time now is 01:12 PM. Post New Thread    Post A Reply
Pages (14): [1] 2 3 4 Next » ... Last »   Last Thread   Next Thread
>>> cachecard - its alive <<<

TiVo Community Forum Archive 1 : Powered by vBulletin version 2.2.8 TiVo Community Forum Archive 1 > Underground Playground > TiVo Underground
Search The Internet
 
Show Printable Version | Email this Page | Subscribe to this thread

Forum Jump:
 
Search this Thread:

Forum Rules:
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is OFF
vB code is ON
Smilies are ON
[IMG] code is ON
 

< Contact Us - TiVo Community Forum Archive 1 >

Powered by: vBulletin Version 2.2.8
Copyright ©2000, 2001, Jelsoft Enterprises Limited.
(C)opyright - All Rights Reserved. No information may be posted elsewhere without written permission.
TiVoŽ is a registered trademark of TiVo Inc. This site is not affiliated with TiVo Inc.
Page generated in 0.07034898 seconds (85.04% PHP - 14.96% MySQL) with 22 queries.


Spider History Index