Repairing Amazon S3 downloads for IE

I use Amazon S3 for cloud storage of big digital-cinema files (up to 3GB) for distribution. It works fine most of the time, but I kept getting the odd complaint: “I can’t download on my PC, I get an error”. Everytime I asked what browser they were using, it was Internet Explorer. I am a Google Chrome man, and I almost never do anything with IE, but still, customer is king, let’s see what could be wrong. So I tested it myself with IE and yes, most files can be downloaded, but some couldn’t. Sometimes one would get an empty page, sometimes the following: “XML 5619: Incorrect document syntax

So I fire up  Fiddler2 – an invaluable tool to see what’s going on under the hood of the communication between your web browser and the web server. I look at the client and server HTTP headers and see something interesting:

1) Download via Chrome

CLIENT:

User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch

SERVER:

Content-Type: binary/octet-stream
Content-Length: 26176425
Server: AmazonS3

2) Download via IE for a file that can be downloaded:

CLIENT:

User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)
Accept-Encoding: gzip, deflate

SERVER:

Content-Type: binary/octet-stream
Content-Length: 26176425
Server: AmazonS3

3) Download via IE for a file that can NOT be downloaded:

CLIENT:

User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)
Accept-Encoding: gzip, deflate

SERVER:

Content-Type: application/x-zip-compressed
Content-Length: 687411306
Server: AmazonS3

It was a consistent pattern: every time the Content-Type of a file was x-zip-compressed, I couldn’t download . It might have something to do with MS KB 841120: the server that recompresses .zip files with gzip, and the browser mis-interpreting.

Anyway, I used CloudBerry S3 Explorer to go and explicitly change  every file’s HTTP headers and now I can download all files with IE. If I ever forget about this IE quirk, now I’ve written down the solution!

Review: Synology DS410 8TB NAS

Storage vendors should come to me for heavy duty testing, I have way too much hard disks break down on me. Last year my 4TB Lacie drive died. It’s a good thing I’m paranoid about data storage and I had 2 copies of my photo archive (now about 1.4 TB) elsewhere. Although my Lacie drive had ‘protected’ storage, after repair it came back reformatted. I decided to never buy Lacie anymore – I have had a 1TB, 2TB and a 4TB drive and they have all broken down at some point. My next storage solution would be a stand-alone NAS with 4 disks!

After reading some reviews on QNAP and Synology, I decided on the Synology DS410. I ordered it at Memoryshop for a decent price and some days later it was shipped to me together with 4 Samsung 2TB drives. Installation is swift and uneventful. I configured it as one big 6TB RAID-5 volume and started copying all my pictures, music and movies. The device comes with the shares /music, /video and /pictures preconfigured, and copying to these folders makes sense, because then the music appears in the handy iTunes server, and all media shows up in the DLNA Media Server.

The advantage of a Linux-powered NAS is that it comes with a number of easy-to-install applications (Torrent Client, MySQL, LAMP stack web server) and you can even install, through ipkg, lots of standard Linux packages. On the QNAP server at the office, I have file sync tasks running at regular intervals and it works flawlessly.

If you’re serious about your storage (because you need it for your work), don’t be content with just an external USB drive. Invest a bit more to have a NAS you can trust. And also: never trust it 100%. I now have +- 16TB of storage at home so that I have multiple copies of everything and I also use Mozy cloud storage for my exported pictures (‘only’ 12 GB for the moment).

Review: Panasonic TX-L42E30E LCD TV

I bought and/or used quite some new gear the last couple of months and I’ve been meaning to write about my experiences and never got around to actually start. Because I like reading other people’s reviews before I buy anything, I’ll start writing my own now! Let’s start with the biggest one:

Why I ‘needed’ a new TV

When digital TVs came out, the price for full HD (1080 lines instead of 720) was very high. I remember seeing all those 2000€+ beasts and it just didn’t make sense to switch yet, also there was almost no source of Full HD video. Blueray was launched around 2008 and seemed more like a ploy to make you buy all your old films again in a slightly better format. However, the prices of Full HD has dropped a lot, and I had more and more devices at home that had HDMI outputs and 1080p output that I could not show in full glory. So I started shopping around beginning of this year.

I knew I wanted an A-brand, which boiled down to Sony, Samsung, Panasonic, Philips, LG or Toshiba. Shopping for digital TV is exhausting. Each brand  seems to have 150 different models which are sometimes hard to tell apart. I know I was thinking during the shopping: I wish Apple made TV screens: they would have a 32, a 44 and a 56″ model: just pick one. I wish I was able to say that I made a huge spreadsheet with all models, features and prices, created a N-variable price model and chose based on that, but honestly, I just picked one that looked nice in the shop (Vandenborre) and seemed like a good bargain. So I now have the Panasonic 42″ Viera TV. Vandenborre offered to come bring and install it, but I opted for picking it up in the shop.

Continue reading Review: Panasonic TX-L42E30E LCD TV

Idea: Experimenting with movie hashing/fingerprinting

MEDIA FINGERPRINTING
Neal Krawetz wrote an interesting article on image fingerprinting, or how to search for images that are similar. He proposes an algorithm to do image fingerprinting and reproduce the functionality of TinEye, a service that allows to give one image and get back all the web pages where that picture, or a slightly modified version of it, is included. By resizing the image to 8×8 pixels, creating a B/W version, and then a binary (only black or white pixels, no grey levels), he reduces a picture to a hash that is 8 x 8 = 64 bits. This can then be compared to a database of hashes of millions of other pictures found on the web (by calculating the “Hamming distance” – read the article for details).
On the other hand, something similar can be done on segments of audio. Youtube has been doing it for years (using technology from Audible Magic) and recently the Echo Next has released Echo Print, a music fingerprint and identification service that does the same thing for free.

VIDEO FINGERPRINTING
Since I work (and play) a lot with video, I was thinking about how to extend the ideas Neal proposes to video. Video material consists of audio (which I will ignore for now) and a sequence of images (typically between 24 and 30 per second). I’ve taken a video clip from Youtube as inspiration, but I won’t tell you yet which one, let’s discover it while I create the fingerprint of 1 of the frames, using the method described by Neal:

first we reduce the frame to a small format, e.g. 16×8 pixels
then we desaturate: make the picture grayscale
then we normalize it: we maximize the contrast, the darkest pixel become the new black, the lightest becomes the new white
we calculate the average darkness and any pixel that is darker we make black, and the others become white -this is what is proposed in the article. Total size of fingerprint: 16 pix * 8 pix * 1 bit = 96 bits = 16 bytes
I found the details in the above fingerprint too coarse, so I used 4 color levels instead of 2. Total size of fingerprint = 16 pix * 8 pix * 2 bits = 32 bytes

Continue reading Idea: Experimenting with movie hashing/fingerprinting

Track your (Synology) NAS when it’s stolen

When a friend of mine recently got his MacBook stolen, I quickly verified if I had installed Prey Project on each laptop/desktop PC I have. For those who do not know Prey:

Prey lets you keep track of your phone or laptop at all times, and will help you find it if it ever gets lost or stolen. It’s lightweight, open source software, and free for anyone to use. And it just works.

Yes, I had Prey running on each PC. And then I looked at my Synology NAS (DS410, 4 disks, 8TB raw storage). It could be stolen too. And it’s basically a Linux box. And Prey is available for Linux …

So I figured out how to install Prey on a Synology box:

  1. login via ssh as root
  2. install the ipkg/’Bootstrap’ module on your NAS server – (from forum.synology.com) and this is a list of the right bootstraps for the right Synology model.
  3. install bash shell – “ipkg install bash” (from forum.synology.com)
  4. install textutils – “ipkg install textutils” (from forum.synology.com)
  5. goto /usr/share and download the latest Linux version of Prey (wget http://preyproject.com/releases/...linux.zip ) and unzip it
  6. create an account on Prey and get your API key from your Account profile.
  7. create a new device (e.g. ‘NAS8TB (Syn410)’), indicate OS as Debian (it’s close enough) and get the device key.
  8. edit the /usr/share/prey/config file and fill in the API and device key
    # you can get both of these from Prey's web service
    api_key='yyyyyyyyyy'
    device_key='xxxxxx'
  9. now run the “bash /usr/share/prey/prey.sh” a first time – you should get a “-- Got status code 200! -- Nothing to worry about. :) -- Cleaning up!” response.
  10. now edit /etc/crontab and add a line
    5-55/20 * * * * root /opt/bin/bash /usr/share/prey/prey.sh >  /usr/share/prey/lastrun.log
  11. Now restart crontab in the following (non-standard-Linux) way (from forum.synology.com):
    /usr/syno/etc.defaults/rc.d/S04crond.sh stop
    /usr/syno/etc.defaults/rc.d/S04crond.sh start
  12. And it’s running! When your Synology is stolen, you set its status in your Prey account to ‘Missing’ and you will start getting email reports every 20 minutes. Because it’s a NAS, there is no webcam and no screenshots can be taken, but the external IP address will let you see where the device turns up.
    Remote IP: 78.29.245.xxx
    Private IP: 192.168.0.108
    Gateway IP: 192.168.0.1
    MAC Address: xx:xx:xx:xx:xx:xx

Would this work on a QNAP server? I’m guessing, yes.

TV Show scoring: the decline of True Blood

I got the DVDs of the 3 True Blood seasons for Christmas and when I started watching , I really got into it. Sookie is adorable, Jason is charming with a low IQ (and strongly resembles George W), there’s the interesting parts of how vampires and humans live together, with some comments on racism and homophobia. Season 1 was just great, and I was telling everyone around me to start watching it too.

Then, during Season 2, the quality seemed to drop. The dialogues seemed ridiculous at times, the script didn’t go anywhere, and then at the end, there were two really silly episodes. I’m currently watching Season 3, and still I am frequently disappointed by the acting, the lines, the plot. So I thought: wouldn’t it be nice that if you start watching a show that has been around for some time, several seasons for instance, that you could get some idea about what quality to expect. Is there data for this? Yes, my favourite TV show portal TV.com has collected scores for each episode from its users. All I needed was some way to summarize and visualize it. So that’s what I made.

http://tools.forret.com/tv.com/

TV Show scores for True Blood

Continue reading TV Show scoring: the decline of True Blood

Shooting portraits of strangers

Shooting Portraits of Strangers
This was quite a discovery: Danny Santos has been shooting portraits of complete strangers on Orchard Road, Singapore. The story was featured on JPG Magazine with some beautiful examples and that post pointed to his Facebook album, which pointed to his blog post about it. This phrase he wrote made me think: “Suddenly, strangers were no longer unwitting victims subjects, they were now willing participants … and that gave different life to the photograph, and a new awareness and dimension to my idea of shooting in the streets.” Touches a weak point of mine: asking strangers for permission to photograph them.

What I like a lot about the photographs is their shallow depth-of-field. He talks about his material: a Nikon D300 with a 85mm f/1.4 lens. That’s a $999 lens, so one can expect some good results.

Let’s see what that looks like in my depth-of-field calculator: there is one picture on his blog that shows him taking a picture of someone at +- 2m away, so let’s use that as distance. If we fill in the right values (1.5 crop factor for a Nikon D300, f/1.4 aperture, 85mm focal length, 2m distance, 3:2 aspect ratio) we get this:

Depth of field calculation
0.019 mm Circle of confusion
1.99 m Near limit (anything closer will be too blurred)
2.01 m Far limit (anything further will be too blurred)
2.8 cm Total (1.4% of the subject distance)

portraits of strangers: depth-of-field

Continue reading Shooting portraits of strangers

How to delete old data in Google Calendar

I use Google Calendar as a vital piece of milonga.be: me and some 20 other editors keep an up-to-date calendar of tango events in Belgium. We’ve been doing that for the last 3 years, so there was a lot of old, no-longer-relevant data in the agenda. The way I use the calendar on the site is that I download all the appointments as a .ICS (iCal/gCal) file and then format/display it with another program. But with all the old data still present, that ICS file had grown to more than 1MB, and this size slowed down the updates (I download the whole thing every 30 min). So I decided to delete all old data (2007 – 2009). Not that easy.

Google Calendar’s web interface doesn’t really allow you to bulk delete. There is no way to select several dozens of appointments and delete them in one go. But I found a way that works (suggested here):

  • Install Mozilla Thunderbird (desktop email client)
  • Install Mozilla Lightning (calendar plugin for Thunderbird)
  • Install Provider for Google Calendar (Gcal plugin for Lightning)
  • Look up the Google Calendar Private iCal URL of your calendar (something like http://www.google.com/calendar/ical/...%40group.calendar.google.com/private-.../basic.ics)
  • Add it to Thunderbird with FILE/NEW/CALENDAR/NETWORK/GOOGLE CALENDAR
  • You now have a read/write connection to your Google Calendar!

Select the appointments you want to delete, hit the ‘Del’ button and see them disappear one by one.

DROA: now with a Belgian lawyer

It seems my contacts with the people from Domain Registry of America (DROA) have gone one legal step up. Last week I have received an official letter from a Brussels law office, representing the Domain Registry of America – DROA (Buffalo, NY, USA), Domain Registry of Canada – DROC (Markham, Ontario, Canada), Domain Renewal Group (London UK) and Mr Alan Benlolo. Apparently my claims about the “domain renewal scam”, as most legit domain registrars and thousands of fooled users call it, are “completely unfunded” and constitute “defamation and libel” (Article 443 – translated from Dutch by me).

Furthermore they claim that

  • I supposedly have published the home address of Mr Benlolo on my blog. I have checked and in none of my blog posts about the DROA (DROA scam, Govern yourselves accordingly and DROA: the saga continues) have I published any address belonging to a person: all addresses are PO Boxes – so not physical addresses – belonging to companies, not persons. These addresses I have found either on the letters what were sent by the DROA or on their web site.
    The letter I received actually shows Mr Benlolo’s address, and I can confirm it is nowhere on my blog. Other unhappy customers might have done that on other blogs/forums, but not me.
  • I am supposed to remove the three blog posts mentioned above ASAP. I have reread all three and in each I just cite known convictions of persons and companies involved, as well as reactions from institutions like the  FTC and Better Business Bureaus who all disapprove of the type of business that the DROA is active in.
  • Interestingly I’m also supposed to remove a fourth blog post “Amy Cross is spamming Technorati” which I did not know was related. Maybe it’s just because there’s a link to the DROA article.

I have gotten seven days to comply, and that period has just expired. I have tried to contact the lawyers but no one has bothered to call me back. I don’t agree with their claims, and I still stand by the blog posts I have written.

Fix for error -15000 Remote speakers on iTunes / Windows 7

So, just to be clear: this is not about iTunes on MacOSX, it’s only about iTunes 10 on Windows 7. It might work on Windows Vista, but I have no PC with Vista to test. It might help on versions earlier than iTunes 10 (I read the version 8 already had the problem), but I haven’t tested that.

The problem is the following: while it used to be possible (in older versions of iTunes), your iTunes  might not stream the music to an Airport Express / Apple TV (the feature they call ‘Airtunes‘), and every time you try, iTunes will come back with an error ‘An error occurred connecting to the remote speaker (…). An unknown error occurred (-15000).‘. What is happening is that the Windows Firewall is blocking the UDP connections for Airtunes (ports 6001-6003 if you’re asking). For some reason the old firewall rules for iTunes don’t work with the new version.

How to fix this: only if you’re not afraid of diving into your PC’s configuration. If you’re a novice or feel unsure, ask someone more knowledgeable to do it for you. Check if the person knows what a firewall is used for. “Security” is a too general answer.

Continue reading Fix for error -15000 Remote speakers on iTunes / Windows 7