iTunes Backup Feature

iTunes has a great feature: iTunes Backup. You can select a playlist or the whole library and can back it up to multiple DVDs. The problem is with the restoration. I’ve had this happen twice. In one case, I had the discs and a Mac and one disc would keep failing. I dumped the contents to a hard drive and copied them into iTunes that way. The other situation was with a Windows machine. I’m wondering if in this situation it was a filename-length issue. The files were created on Mac-formatted drive and he was restoring to FAT32. I overcame that with the hard-drive method. iTunes won’t tell you which file failed, either, so a whole disc is suspect when it fails.

The files themselves aren’t faulty. When you insert an iTunes Backup disc, it will appear in iTunes and you’ll be asked if you want to restore. If you ignore this, you’ll see that it’s a standard disc in the Finder or Windows Explorer. iTunes adds a little magic somehow to identify it as an iTunes Backup disc. To do the hard-drive trick mentioned above, I simply copied to a hard drive, then copied from the hard drive to iTunes. This will definitely bypass the disk-format issue if you use the same drive for both the copy and iTunes. If a file fails, your OS should give you some sort of explicit error for a specific file that you can correct, like “filename too long”.

The other annoying thing about this feature is that it mixes up all the content, so you may have the tracks of a single album strewn over more than one disc. My only theory as to why is that iTunes is trying to maximise how much it can get on a single disc and sticking to a strictly sequential order may leave unused space. This doesn’t matter if you intend to restore the lot and the feature works!

So Apple has a very good feature for the initial part of the operation, but the critical part, restoration, is sadly lacking. Use with care.

LaCie CurrenKey

LaCie CurrenKey 8Gb (Click to enlarge)

Hardware review isn’t really my thing, but I wanted to sing the praises of this USB drive that I only just became aware of. I bought it immediately. I just love funky USB drives. I’m also going to buy a couple of the beautiful new key-themed drives announced today. I bought the 8Gb because I want enough space to store a DVD. You get about 7.5Gb after formattting.

The image above is an icon for use in Mac OS X and Windows. I downloaded a large image from the website, then knocked out the background, added my own shadow, resized to 512 x 512 (largest Mac OS X icon size) and exported as both ICNS (Mac) and ICO (Windows) icon formats. You can download the Mac version here and the Windows version here.

To use on the Mac, drag into the excellent CandyBar, select it, then drag the drive onto the Quick Drop pane. If you don’t have CandyBar, open the ICNS file in Preview, select all, copy, then get info on the drive in the Finder, click the icon in the top left of the dialog and paste. It should look like this on the desktop, with a lovely transparency:

LaCie CurrenKey 8Gb Icon on Mac OS X Desktop

I give up on trying to get this to work on Windows. In my experience, you usually have to install something which provides an icon when you plug in the specific drive. If you plug it into a different PC, it looks generic. I’ve given you the 256-pixel Windows ICO file. If that’s no good, you can download the above large PNG image and make something from that. If you’re successful, I’d love to know how you did it.

This icon trickery is some form of “media management”, so that’s my excuse for being a little off topic!

Your Music in Microcosm

Do you have multiple iPods? Apart from my great love of them, driving me to collect them (I have a policy of always having the latest complete set of iPod models, broken only this year because there was no replacement for my 160Gb Classic and I got the iPhone instead of the Touch), my media library got so large that different types of iPods became a good idea.

My 160Gb Classic has every last music track on it, plus every music video (over 500). I play this at work. The beauty, of course, is the essence of the earliest iPod concept–carry all your music with you. It can be hard to select something to play. I use playlists, shuffling the whole contents and Cover Flow to aid me in this. I just wish I could get the Genius feature without having to downgrade my storage.

My Nano (8Gb) and iPhone (16Gb) necessarily cannot store much. Because of this, I have chosen to keep only the most recently added music on them. I have a playlist that keeps track of this and which forms the basis of what is to be synced. Here are the criteria:

(Click to enlarge)

This takes all the guesswork out of what to put on the iPod/iPhone. Seeing as the music is new, I probably haven’t heard it yet or it’s my latest groove-thang, so it will be at the forefront of my attention. This is another reason why I chose to add only the most recent music.

I find it interesting how much nicer it is to have this small selection of music as it is much easier to comprehend and to choose what to play.

If you have only a small-capacity iPod and you are wondering how to fill it, I recommend the last-added playlist as above.

Music Video: Why Can’t Artists Get it Right?

Time for one of my infrequent rants.

Why is it that artists (or their technical people) constantly make a hash of music videos? Case in point: the independent, Karmacoda. They’ve just released a new video on their website. I downloaded the iPod-ready version and it looked funny. The native aspect ratio of the video, 16:9, had been squashed horizontally into 4:3 and was between QVGA (320 x 240) and the iPod standard VGA (640 x 480). The slightly larger version (“for computer”) looked correct (heads not squashed) but even it was 3:2. Here’s the real kicker in this case: it was shot in HD. That means the trouble and expense of 720 or 1080, which should look amazing, has been squandered on a lousy postage stamp. A prior music video, also shot in HD, was made available on DVD, which I bought. I knew it was going to be NTSC (480 pixels vertical), but was unpleasantly surprised to find that it was 4:3 letterboxed. This just should not be. It’s not hard to make anamorphic video these days. Again, the quality of HD was wasted on this 720 x 400 video (after cropping).

My frustration with Karmacoda stems from the poor responses to emails I sent. The first, after I received the DVD, was extensive and contained a breakdown of the technical issues with the DVD and the free version and offered solutions. I also said that I would like to buy a HD version from their site. The response was a mere acknowledgement that I had sent something. I tried again after getting this new video. Same thing. I’m giving valuable feedback and advice. As they are an independent band, I expected a meaningful two-way conversation with the artists themselves but seemed to have been screened by a manager.

It’s not just independents at fault. Prior to the release of Daft Punk’s Alive 2007, I downloaded a teaser trailer in VGA resolution, H.264 video, AAC audio. Looked quite good. When the album arrived, the enhanced CD included a music video. The specs? QVGA, MPEG 1 muxed. What is EMI thinking? When will record companies wake up and start providing iPod-ready video on enhanced CDs? Nobody is going to want to put the CD in a computer to watch the video.

So why aren’t they done right in the first place? A possible solution is to make the video available on iTunes–after all, they’re giving it away for free on their site, and we know that free often doesn’t mean quality. If they package it for sale, the consumer can be assured of a certain level of quality.

Music video is being denigrated by ignorance or unwillingness to produce a good product. In the past, it’s been used solely as promotional material, where marginal quality would be a non-issue. It’s only been since the iTunes Store started selling them as a product in their own right that quality should be something that artists are aware of.

Am I alone in thinking this?

iTunes Optimisation Follow-Up

Had a scare with iTunes last night. If you’ll remember my earlier optimisation efforts, I put the iTunes database and media files on an external drive. I love the Western Digital My Book, but it tends to go to sleep a little too readily.

I had iTunes open and the system lost the My Book. Not sure why. The drive didn’t seem to want to wake up when I pushed the button on the front. iTunes informed me that it couldn’t save the database. I force-quit, then launched again. This time iTunes informed me that it was unable to sync 100 items to my iPod because it couldn’t find the files. I force-quit again, ejected my iPods and copied the database to another drive. I then opened the database copy and had to point it at the media files. Then it started to remap all its paths to those on the My Book. This took a while, then it crashed on trying to process sound check adjustments.

I gave up, restarted and restored last night’s backup of the database, put it back on the My Book and opened it. All back to normal.

There are some things I’ve learnt from this experience:

1. Daily backup of the database saved a lot of trouble.
2. It’s a good idea to have your database (iTunes folder, excepting iTunes Music) separate from the media folder (iTunes Music). If you move the database, the link to the media is not broken. If the media folder is within the database folder, as is the default, then a relative link is established from database to media folder, thus breaking when they are separated.
3. Put your database on a drive that will always be accessible to the system to prevent database corruption. You can put it on the main system drive which is closest to the motherboard and will probably never become unavailable or you can put it on an external drive but make sure that drive never sleeps.

Currently I’ve set the hard drives to never sleep and the My Book obeys but gets noisy from time to time when it gets warm, so I’m going to consider putting the database on one of my internal drives and turning the hard-drive sleep option back on. The internal drives don’t tend to sleep as they are being used constantly.

Improving Overall iTunes Performance

I’ve been conducting an experiment over the last month in a bid to radically improve the performance of iTunes. The specific issue I was having was incredibly slow response times when doing simple actions such as clicking a playlist, sorting and changing from one view to another. Ripping CDs and syncing were not an issue.

The Configuration:

Computer: PowerMac G5, dual 1.8GHz
RAM: 1Gb
Hard Drive: 2 x 320Gb, RAID-striped on SATA bus (1.5gbps)
iTunes Folder: Approx 450Gb

The Theory:

Before I begin, I’d like to mention that I’m not an expert by any means. I put together this theory based on my general knowledge.

For a G5, 1Gb is not a lot of RAM, especially as this machine is a real workhorse, often ripping a DVD, converting recorded TV and running iTunes all at the same time. The hard drive is also a bottleneck, because the iTunes database, the iTunes media files and the system were all on the same volume. If all three had to be accessed at the same time, the requests would compete. Compound the insufficient RAM which generates a very active virtual RAM file, and performance is severely compromised.

Phase One: Isolate the iTunes Database

My first experiment was to put the iTunes database (contents of ~me/Music/iTunes/ minus the iTunes Music folder) on a USB drive. My reasoning was that isolation of the database itself, which has to be written out in full every time you change the slightest tag, would improve performance because it would no longer compete for access on the main hard drive volume. There were no other files on this drive. The iTunes media files were left on the main volume.

Result: The only performance benefit was slightly faster database load time on launching iTunes. On closing iTunes, sometimes the saving of the database seemed to take longer. The 480mbps bus may have been a factor, as it is more than three times slower than the onboard SATA.

Phase Two: Isolate the iTunes Media Files

I bought an external Western Digital MyBook Pro 1Tb drive. This drive consists of two 500Gb drives, RAID-striped using hardware RAID built into the box. Hardware RAID is always better than software RAID, and if the box does it instead of the computer, even better. The drive is connected via FireWire 800 (800mbps). Only the media files are located on this drive. The database is still on the USB drive.

Result: Still no change in database performance, however, overall system performance seems slightly better.

Phase Three: Move the Database to the External Drive

Frustrated with no real performance boost from the dedicated USB drive, I moved the database to the external FireWire 800 drive to test.

Result: Launch times, save on close and general interface improved, better than in both the original configuration and Phase One.

Phase Four: Increase RAM

I upgraded my RAM to the maximum that the system can take: 8Gb. I didn’t change the location of the database and media files.

Result: Performance about twice as good as that of Phase Three.

Conclusion:

It really does matter where you put your database and your media files. I didn’t expect this much improvement from FireWire 800 as it’s still a lot slower than SATA. However, the computer seems to really like having those files on an external drive. If I had more bays inside the machine I would have used them instead of an external drive.

I was really disappointed that the database-on-USB experiment didn’t yield results. I thought that it would work better because the file isn’t that big (38Mb) and the bus should be fast enough for that one operation.

RAM helps a lot, as expected. I’m not sure how to check, but it seems obvious to me that the virtual RAM file would be smaller and/or less-often accessed.

I am pleased with the performance improvement. I’ve noticed multiple operations that used to choke iTunes seem to be much better handled. An example of this is syncing Apple TV while downloading a podcast while ripping a CD and deleting a song. However, the initial issue that I wanted to eliminate as much as possible–pausing when I click anything (e.g. changing from Music to TV Shows)–is still present, although the pausing is not as long as it used to be. I have concluded that this is due to 2 factors: the age of the machine (4 years old) and the database design. I think the database design is the key factor. I’ve posted earlier that it needs to change.

If you want to improve your own iTunes performance and it still seems a little pokey, as I’ve discovered, please tell Apple about it. If enough of us report this, they will make changes to improve it.

The Importance of Feedback

If there’s a bug or something you don’t like about iTunes, the iPod or Apple TV, you should report it to Apple. If enough of you give feedback about a particular issue, it will be changed. Here are the links to the feedback pages. Report every time you think of something:

• iTunes Application: http://www.apple.com/feedback/itunesapp.html
• iPod: http://www.apple.com/feedback/ipod.html
• Apple TV: http://www.apple.com/feedback/appletv.html

How to Split Your iTunes Library

I am aware that I have a configuration optimised for my own particular computer system, which sometimes makes me shortsighted, and that others’ systems vary widely, but sometimes viewing another’s system really brings home the point.

I have a friend who has three 300Gb drives, but they aren’t RAIDed. He has a space problem that prevents him from moving all that data temporarily while he RAIDs the drives, which was the original plan. I gave him some video for iTunes that he couldn’t fit on his current iTunes drive. He has a huge iTunes library but it’s mostly music, with little video in it.

I have to credit him with the idea that enabled him to split his library. Instead of putting all iTunes content on the one drive, he realised that he could turn off iTunes’ “Copy files to iTunes Music folder when adding to library” option. This means that he can put the videos on a different volume, then add them without copying. iTunes simply writes the path in its database. In summary, here is his set up:

• iTunes library folder (consists of database, support folders and files): drive 1
• Music (in iTunes folder): drive 1
• Video: drive 2

What can be a little confusing is the location of your support files and the media files themselves, as they can be different. A library means a folder containing your database and any support folders/files, such as Artwork, iPod Games, etc. Your media is by default also in this folder but you can scatter any media to your heart’s content, and with the copy option turned off (as covered above), iTunes will track it all. It is advisable to keep everything together, but not essential.

This technique has ramifications for laptop users. Laptops have smaller drives than desktops. With the default preferences, CDs you rip to your library will go into your iTunes Music folder on your laptop. To add video, which makes sense to keep off your laptop due to file sizes, put it somewhere, then add it to the library and iTunes will write pointers to those files. When you want to play them, attach the external drive or mount the network volume and go.

There is a caveat, which is a disadvantage to this system. The volume where the media is has to be mounted to work. It is best to mount it before you launch iTunes, or at least before you interact with any offline files. If you are using network volumes, it is wise to mount them on startup so you don’t have to think about it. On the Mac, you can add any currently mounted volumes to your startup items in System Preferences > Accounts. They will then mount automatically every time you log in.

This is a good solution for those with limited space.

iTunes Needs to Become a Server

What does this mean? iTunes has not been designed to be a multi-user system. It has a proprietary database that one user can access at a time. This is terribly inefficient. I have almost 15,000 items in all my libraries (music, movies, etc.). The total size of the files is almost 440Gb. The database file is 38.7Mb. This file is the heart of iTunes. It’s where iTunes maps the location of all your files and stores your playlists and audit data, CD tags, etc.

Whenever I make a change in iTunes (change one character in any tag, add album art, sync audit data, delete an item, etc.), the whole database is written every time. If two or three actions are being done at the same time, every action is slowed down dramatically. My machine is getting old and really could do with more memory, but I think when it takes a second or two just to register a mouse click, there is a software performance issue, not simply a hardware one.

To make the database better, iTunes needs to be able to read and write records individually, not as a whole file each time. The performance boost just by this one change would be enormous. Another feature that ties into this is multi-user. If a database can be written to on a record-by-record basis, then more than one user could use it at a time. Users could share a single library, with multiple CDs being ripped at the same time, etc. Multi-user makes it more complicated, but I’m sure Apple could figure it out. Maybe individual profiles on machines that stored different audit data and iTunes Store keys. Luckily, I’m not in that boat but the performance issue is key in any case.

I really hope Apple makes this change. I’ve left them feedback to the effect. If you want a better iTunes database experience, I suggest you do too.

Basic DVD-Ripping Steps

May 3 2007: Note: The video specs are now out of date, but the outline of the procedure is still the same.

I should get on with some video-ripping specs. This is the broad outline. I’ll get specific later.

Ripping

Start with Handbrake or MediaFork. Both are essentially the same application. MediaFork is slightly more configurable. Rip using the following specs:

Video

• H.264 (main): 2500 kbps
• 2-pass encoding
• If a grey-scale movie, check Gray Scale
• 4:3 content: 640 x 480
• 16:9 content: 720 x 400

Audio

• Stereo option, if available, otherwise 5.1 will be down-mixed to stereo
• 192 kbps

Tagging

Next step is tagging. We use Lostify for this. Once done, drag into iTunes.

Cover Scan

Scan the cover of the DVD and add this to each track, if from a movie or music video DVD, or the image pane of the first episode in a season if a TV show

TV Show “Album Art”

For TV shows, we use a frame that best describes the episode for individual “album art” on each. Using QuickTime Player, open each video file and copy a frame. Save these in JPEG format.

In iTunes, open each episode one by one and replace the automatic album art with the appropriate frame. For the first episode of any season you will get two images for that episode–the DVD cover and a frame.

There’s a lot of detail to each step but I wanted to blurt out the overall procedure before getting technical.