Tango, photography and whatever's bleeding edge
Ine mentions that Dreamhost has become a more reliable hoster. I am actually tracking Dreamhost performance, and I can only agree.
This is the current response time of a WordPress blog on Dreamhost:
Wordpress is a database-powered PHP application, so this response time includes the MySQL queries and PHP overhead.
This is the very similar performance of a second blog on another Dreamhost server:
Continue reading ‘Dreamhost has better performance now’
Just how many pictures does Flickr receive every day? I found a way to estimate the # of images that they add to their database, and another way to get average (original) file sizes for those images. The result? Their storage growth, i.e. their upload bandwidth, and the growth rate of their storage system (how many days to reach a terabyte?)
You see here that weekends, Sundays specifically, are the most busy days for uploads. You can see peaks of almost 68.000 pictures an hour (almost 20 pictures a second). Peaks are around 22h CET (or 1 PM PDT – in California). The lowest rates (still around 20.000 photos/hour) are 12h apart: 10h CET (1 AM PDT).
The average inflow of pictures is: 38.400 photos/hour. That is around 10 photos/second, 920.000 photos/day.
And how big are those pictures? I have found a way to estimate average filesize (and maximum, while I’m at it). It’s not perfect, but quite accurate. How? That’s classified. I could tell you, but then I’d have to … Anyway: these are the numbers:
On average, a picture uploaded to Flickr is 555.2 KB big. They receive files up to 7.3 MB (what number of megapixels would that be?) and quite a lot of 3MB images. My Canon 350D makes 8 megapixel images (3456 x 2304 pixels) that are between 2 and 4 MB large. But the ones I send to Flickr (after Picasa processing) are typically smaller: 1200 x 800 (300 – 600KB) or 1024 x 683 (200 – 400KB).
What happens if we multiply both numbers?
38.400 pictures/hour x 555,2 KB/pic = 21,3 GB/hour = 5,9MB/sec or 47,3 Mbps. Storagewise, this is 15,3 terabyte/month of new pictures. Thank God storage prices are dropping.
Five years ago, a server with a few hundred gigabytes of storage – one of many needed to handle uploads of member photographs – would have cost Flickr about $250,000. Today, Mr Butterfield says, “you can get a terabyte of storage for about $5,000”. (via ft.com)
Peak bandwidth usage: let’s take 60.000 pictures/hour x 3MB/pic: 180GB/hour = 50 MB/sec or 400 Mbps. This is probably still peanuts compared to their outgoing bandwidth.
After a conversation with Ine on HD formats (1080i vs. 1080p), I researched the topic a bit further. Let me resume some of the things I have learned up till now:
HD or ‘high definition’ as defined for screens, projectors and TV, defines 2 resolutions. The smaller one has 720 lines of each 1280 pixels, the bigger one 1080 lines of each 1920 pixels. They can be used with different frame rates: refreshed at 24 fps (a common movie standard) up to 50/60fps (often used for TV). To limit the necessary bandwidth in some cases ‘interlaced scanning’ is used: 1 frame contains all the odd lines, the next only the even lines. This effectively halves the throughput, at the cost of image quality (rapid moving lines appear jagged).
The two most common formats are:
What kind of resolution do we have now? Regular digital TV (SD or ‘Standard Definition’) consists of 480 lines of 720 pixels each. DVD, for instance, allows for 480i and 480p. So, HD delivers at least 3x that resolution.
“HD Ready“, a label that a lot of TVs/screens carry now, just indicates that:
- The minimum native resolution of the display (e.g. LCD, PDP) or display engine (e.g. DLP) is 720 physical lines in wide aspect ratio.
- The display device accepts HD input via Analogue YPbPr1, DVI or HDMI
- HD capable inputs accept the following HD video formats: 1280×720 @ 50 and 60Hz progressive (“720p”), and 1920×1080 @ 50 and 60Hz interlaced (“1080i”)
- The DVI or HDMI input supports content protection (HDCP)
from eicta.org (PDF)
Even if the display can only show 720p, and so must ‘downsample’ an incoming 1080i signal to that lower resolution, it can be called “HD Ready”.
Continue reading ‘HD – 720p, 1080i and 1080p’
I’ve been following the Database War Stories of O’Reilly Radar: how companies use text-based alternatives to classic relational database systems in order to cope with huge volumes. Check out the stories of Findory/Amazon, Google File System, Flickr and Second Life. Anyway, this seemed like a good moment to share some of my database war stories. Let me take you back to the early nineties.
1993 @ Ukkel
I arrive at Sopres, one of the larger direct marketing / database management companies in Belgium. Fresh from university (and 1 year of military service), I expect to see RDBMS everywhere and dive into SQL. Imagine my surprise when I see that, yes, there are a lot of Sybase SQLServer databases around, but the bulk of the work is done with something they call ‘square files’ (see below). They have built a whole set of tools to work with those and by using them myself, I learn to appreciate the advantanges of the system (speed, mainly) and grow a fairly accurate intuition for things like queries, indexes and outer joins.
Continue reading ‘Database war stories: DB vs ‘square’ files’
Recent Comments