From the NYT:
A new digital library of Europe’s cultural heritage crashed just hours after it went online and will be out of operation for several weeks, the European Commission said Friday, attributing the embarrassing failure to overwhelming public interest.
Europeana, a Web site of two million documents, images, video and audio clips, opened on Thursday with international publicity and acclaim from researchers. But by Friday, those trying to log on were greeted with a message telling them that the service may not be running again until mid-December, while computer capacity is upgraded.
Hot damn. After all this time, Adobe released a 64 bit Flash plugin for Linux about a week ago, and while it's still alpha, it seems to actually work. So far, YouTube videos play fine and the Daily Kos Electoral Scoreboard works better under the 64 bit plugin than it did under the 32 bit plugin running under nspluginwrapper. Time may show that it's still unstable, but at least npviewer.bin isn't sitting there sucking up CPU and memory now.
After I took the plunge and upgraded my Linux desktop from hardy to intrepid, I noticed that (among other things gone horribly wrong) I no longer had virtual consoles. All I got was a blank screen with a blinking cursor if I tried to get to any of them by using Ctrl-Alt-F<num> from X. At first I thought maybe my luck had run out from upgrading across so many versions (this computer hadn't had a full reinstall since "edgy eft", possibly since "dapper drake"), so I did a full reinstall. This didn't fix the problem though. Finally, after much tooth gnashing and head banging, I found out what was going on.
It seems that for some reason, the interaction between the nVidia drivers and the splash screen makes it so the ttys don't actually display anything, despite getty running, and magic SysRq keys working as they should. I eventually found a solution in this Ubuntuforums post. My solution wasn't exactly the same as his, though. Here's what I did:
Enabled "vesafb" by commenting it out in /etc/modprobe.d/blacklist-framebuffer
Rebuilt initramfs with update-initramfs -u -k all
In /boot/grub/menu.lst, I changed splash to nosplash in the kernel boot options and added "vga=769". There's a table of values for vga=xxx that you can reference there. I picked "769" because I was trying to get the virtual consoles to look normal. Finally, this is entirely esthetic, but I took out quiet from the boot options too.
While I was getting the ttys working again, I also found that running dpkg-reconfigure console-setup will let you set the console font back to the VGA font, so it looks normal again. I may be in the minority on this, but I never particularly cared for fancy looking consoles.
This definitely works for "intrepid ibex". I can't test previous versions, but these steps should fix the problem if they have it.
If you're running ipfilter and ipnat on a fairly busy network, you may have to tune ipfilter to accomodate more than the default 30,000 NAT routings. These settings in your rc.conf should allow you to operate comfortably on a GigE network.
ipfilter_flags=”-D -T ipf_nattable_sz=34487, ipf_nattable_max=600011, fr_tcpidletimeout=7200, fr_tcphalfclosed=300, fr_statemax=21529, fr_statesize=30757 -E” Keep in mind: Some of the tunings need to be prime numbers or you’ll get a lot of bad NATs (viewed using the command ipnat -s).
ImageMagick's convert utility is a great way to process images on *nix machines. You can handle images in batches to, say, resize and convert an entire folder down to web-friendly jpgs, or do some more complex things like add drop shadows to images, combine multiple images into one, and so forth and so on.
The configuring and installing process for ImageMagick is fairly simple and well-documented. However, you can't just configure, make, install your way into drop shadow-ing images on the fly. The main thing you have to be aware of is that ImageMagick, unlike some other multi-media utilities like ffmpeg, doesn't include all the necessary libraries you need to do what you want.
Using "large pages" (aka HugeTLB or "huge pages") for MySQL under Linux is pretty sweet, since it lets you use larger memory pages. Unfortunately, if you're like me, you can set up huge pages according to the documentation, but still get errors like this when you start up MySQL:
080324 01:38:04 mysqld started
InnoDB: HugeTLB: Warning: Failed to allocate 12884918272 bytes. errno 22
InnoDB HugeTLB: Warning: Using conventional memory pool
080324 1:38:04 InnoDB: Error: cannot allocate 12884918272 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 153659216 bytes. Operating system errno: 12
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: Note that in most 32-bit computers the process
InnoDB: memory space is limited to 2 GB or 4 GB.
InnoDB: We keep retrying the allocation for 60 seconds...
InnoDB: Fatal error: cannot allocate the memory for the buffer pool
Warning: Failed to allocate 220200960 bytes from HugeTLB memory. errno 22
The fix, it turns, out, is simple. While it's not actually spelled out anywhere in the documentation, you need to adjust the kernel.shmall and kernel.shmmax syctls (also found at /proc/sys/kernel/shmall and /proc/sys/kernel/shmmax) to allow enough memory to be allocated to those big pages. What you set those values to will depend on how much RAM you have, how much RAM you've allocated to InnoDB, and how many large pages you've reserved. These are the values I used on the DKos db servers:
vm.nr_hugepages = 6350
kernel.shmmax = 17179869184
kernel.shmall = 4194304
OK, while not everything's absolutely totally ready, I'll throw this up before I start throwing up some actual content. Basically I'm starting this blog up to post about stuff that I come across while working the backend of Daily Kos that might prove helpful for other people. Hopefully it succeeds in that. To start things off, I've reposted some things I had posted over at Daily Kos. Now I'll start going through some of the random stuff I've encountered over the years along with their fixes as I remember them.
A while back, I discussed the caching used on DailyKos with a hacked up mod_magnet and a lua script to serve up pages for anonymous users out of memcached, so it would avoid hitting the much larger mod_perl apaches on the backend running Scoop. It worked very well, but I never liked having all the extra overhead of mod_magnet when I was only using a small amount of its functionality.
Scaling a website or blog is a constant battle. One of the first things people do is set up a two-tiered system on their webservers, with a lighter proxy httpd server sitting in front of the heavy duty backend application httpd. This way, the lighter webserver program can handle the common lighter requests, reserving the backend for the harder dynamic tasks. Serving up things like images, javascript files, CSS files, and other static content with a light weight process without the overhead of a full mod_perl/mod_python/etc. installation can increase performance dramatically.
The two-tiered setup with webservers is common knowledge. As time goes by and your site's traffic starts increasing, however, performance will begin to suffer again and you'll have to find different ways to scale.
A common way to do that is to cache dynamic content for anonymous users. A good rule of thumb is that anonymous browsers make up 90% of your website's traffic, so if you can avoid regenerating those pages constantly for anonymous visitors you can theoretically reduce your website's load to a tenth of what it was. Writing pages out to disk is a good way to cache pages, but it isn't perfect. If you're going to cache more than just the index page and other pages found at a consistent URL, you'll have to find a way to dump the pages out to disk and use some rewrite trickery to properly serve up the cached pages. If you have more than one webserver in a pool, then the cache isn't shared between the machines. Finally, and this was becoming a problem on Daily Kos, you can have performance issues caused by dumping the files to disk by itself when you have multiple webservers all updating their caches simulateously.
I solved this problem by moving from an Apache 1.3 proxy to a hacked up lighttpd proxy using mod_magnet and memcached to cache anonymous pages, sharing the work of caching them between all the webservers, and greatly expanding how much content I could cache. Below the fold is a description of how it works, how to set it up for yourself, caveats, and ideas for future development.
Note: This works for me, but like all things computer, it may not work for you for no clear reason whatsoever. Always make backups and have a disaster recovery plan in place.
Or, a look at what I've been working on for the site.
It's been three months or so in the making, but today I've finally finished moving the site to its new hardware. Many of the major and minor things that could go wrong, went wrong, but at last everything's squared away and we're up and running on the new machines. Now that that's finished, I can even start rolling out some new features that I've been waiting on until this was all finished.
WARNING: There will be honest, frank discussion of computers and technology issues in this diary. Please, for the love of God, do not post a comment saying that you don't understand what I'm talking about. It will only antagonize the violent beast within.
Let's get started.