Speeding up your Updates with Ubuntu and APT Cacher NG

If you are in a building with more than one computer running Ubuntu, you may have wondered why everyone has to download updates separately. The answer is, they don't. Here's how to accomplish it with a tool called apt-cacher-ng.

apt-cacher-ng is a fork of a project called apt-cacher, which in turn is an alternative to apt-proxy which is poorly maintained and upon my testing, was unreliable.

Setting up the APT Cacher Server

Begin by installing apt-cacher-ng from Synaptic Package Manager, or from the Terminal with the following command:

sudo apt-get install apt-cacher-ng

By default, the version of apt-cacher included with Ubuntu does not currently include security updates. We can easily add the security updates in, however, by following these steps:

As root, create a new file called /etc/apt-cacher-ng/ubuntu_security

sudo nano /etc/apt-cacher-ng/ubuntu-security

This file will be a list of mirrors from which the updates may be downloaded. We only want to insert a single line, the location of Ubuntu's official security updates server:

http://security.ubuntu.com/ubuntu

Save and close the file.

Next, edit the file /etc/apt-cacher-ng/acng.conf

sudo nano /etc/apt-cacher-ng/acng.conf

While we're here, you will see a couple of lines like this:

# Set to 9999 to emulate apt-proxy

Port:3142

I recommend that you follow the instructions and set the port to 9999 to emulate apt-proxy. Not only will this make your server compatible with systems expecting an apt-proxy server, but it will also make the port number easier for you to remember.

Next, look for this section:

# Repository remapping. See manual for details.

# In this example, backends file is generated during package installation.

Remap-debrep: file:deb_mirror*.gz /debian ; file:backends_debian

Remap-uburep: file:ubuntu_mirrors /ubuntu ; file:backends_ubuntu

Remap-debvol: file:debvol_mirror*.gz /debian-volatile ; file:backends_debvol

We're going to add one more line at the end of this section to describe the Ubuntu Security repository that we are adding (make sure the following all goes on one line):

Remap-ubusec: file:ubuntu_security /ubuntu-security ; http://security.ubuntu.com/ubuntu

Now, save changes to this file and restart the apt-cacher-ng service.

sudo service apt-cacher-ng restart

If all went well, the server is now working. No you may proceed to setting up the clients. I strongly recommend making the server into a client of its own apt-cacher, as there is no reason for that system to download the updates from the Internet twice.

Setting up an APT Cacher Client

The basic idea of how to set up a client is to change all the lines in /etc/apt/sources.list to point at the local apt-cacher-ng server instead of at the internet servers. I recommend taking an additional step first so that you can easily flex between different servers. We will create a hostname alias called "apt-cacher" that points at your apt-cacher-ng server so you can simply re-point the hostname whenever you want to switch servers.

sudo nano /etc/hosts

We will add our own entry just after these two lines:

127.0.0.1 localhost

127.0.1.1 your-computer-name

If the apt-cacher-ng server is running on the same computer which you are setting up as a client, its IP address will be 127.0.0.1, otherwise, you need to find (or set) the static LAN IP address for your server. I will assume it is 192.168.1.10 in this example because that's what it is in our building here at CCC. The line you will add will look like this:

192.168.1.10 apt-cacher

Save and close the hosts file.

Now we will replace the entries in the sources.list to point at the cacher:

sudo nano /etc/apt/sources.list

The easiest way to handle this will be a search and replace.

First, replace every instance of "us.archive.ubuntu.com" with "apt-cacher:9999" (no quotes on either) If you are using nano, this is accomplished by pressing Ctrl+W, Ctrl+R, then entering the search string and pressing enter, then entering the string to replace it with, and pressing enter again, then when prompted, press "A" for all.

Next, replace every instance of "security.ubuntu.com/ubuntu" with "apt-cacher:9999/ubuntu-security" (no quotes on these, either.) If you are using nano, use the same steps as given above.

(In this example, we won't be handling the partner repository or any third party repositories which you might have installed.)

Save and close your sources.list. Your final sources.list if you are running Ubuntu 10.04 Lucid Lynx will look something like this:

# deb cdrom:[Ubuntu 10.04 LTS _Lucid Lynx_ - Release i386 (20100429)]/ lucid main restricted

# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to

# newer versions of the distribution.

deb http://apt-cacher:9999/ubuntu/ lucid main restricted

deb-src http://apt-cacher:9999/ubuntu/ lucid main restricted

## Major bug fix updates produced after the final release of the

## distribution.

deb http://apt-cacher:9999/ubuntu/ lucid-updates main restricted

deb-src http://apt-cacher:9999/ubuntu/ lucid-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu

## team. Also, please note that software in universe WILL NOT receive any

## review or updates from the Ubuntu security team.

deb http://apt-cacher:9999/ubuntu/ lucid universe

deb-src http://apt-cacher:9999/ubuntu/ lucid universe

deb http://apt-cacher:9999/ubuntu/ lucid-updates universe

deb-src http://apt-cacher:9999/ubuntu/ lucid-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu

## team, and may not be under a free licence. Please satisfy yourself as to

## your rights to use the software. Also, please note that software in

## multiverse WILL NOT receive any review or updates from the Ubuntu

## security team.

deb http://apt-cacher:9999/ubuntu/ lucid multiverse

deb-src http://apt-cacher:9999/ubuntu/ lucid multiverse

deb http://apt-cacher:9999/ubuntu/ lucid-updates multiverse

deb-src http://apt-cacher:9999/ubuntu/ lucid-updates multiverse

## Uncomment the following two lines to add software from Canonical's

## 'partner' repository.

## This software is not part of Ubuntu, but is offered by Canonical and the

## respective vendors as a service to Ubuntu users.

# deb http://archive.canonical.com/ubuntu lucid partner

# deb-src http://archive.canonical.com/ubuntu lucid partner

deb http://apt-cacher:9999/ubuntu-security lucid-security main restricted

deb-src http://apt-cacher:9999/ubuntu-security lucid-security main restricted

deb http://apt-cacher:9999/ubuntu-security lucid-security universe

deb-src http://apt-cacher:9999/ubuntu-security lucid-security universe

deb http://apt-cacher:9999/ubuntu-security lucid-security multiverse

deb-src http://apt-cacher:9999/ubuntu-security lucid-security multiverse

After you have have saved your sources.list, run the following command:

sudo apt-get update

This will download the package index from the apt-cacher server. You should see lots of "Get" "Hit" and "Ign" lines coming from http://apt-cacher (and other lines for partner and 3rd party servers) if everything is working right. Remember that the update command will still be running at the normal speed because it has to fetch the indexes from the Internet every time to determine if the cache needs to download any new files. Also, the first time a given package is downloaded will still be at normal speed, as well.

To try it out, run sudo apt-get upgrade to download any available updated packages, or just install a new package of your choice. The apt-cacher should be functioning from the command line tools, from Update Manager, and from Synaptic Package Manager. Just remember not to adjust your repository checkboxes under the "Ubunto Software" tab in Synaptic's Settings:Repositories menu, because it no longer knows which repositories we've enabled (they are visible, however, on the "Other Software" tab, and you may adjust them there.)

How To Switch Locations

If you are using a laptop or netbook, you may appreciate the ability to quickly switch from one loication to another, and even to be able to download updates when you are not near your regular apt-cacher-ng server at all.

I have come up with a strategy to accomplish this. All you need to do (and you must do this while your current apt-cacher-ng server is accessible), is repeat the above steps for "Setting up the APT Cacher Server" on your local machine, then you can simply edit your /etc/hosts file and change the entry for apt-cacher to point at 127.0.0.1 when you're on the run, or back to your server's IP when you are at home or work. Making that one line change will allow you to continue using all of the APT tools smoothly.

There is one disadvantage, especially for netbooks or devices with smaller hard drives, and that is that you will be storing two different caches on your hard drive for files which were downloaded while "on the go", one in the apt-cacher folder, and the other in the system's regular apt system cache under /var/cache/apt

Because of this, you may want to occasionally issue the command:

sudo apt-get clean

This will clean the system cache of installed packages out of /var/cache/apt, saving some disk space. This is generally a good idea to do on any Ubuntu system with disk space limitations, whether or not you are running apt-cacher-ng.

Also, if you ever need to delete the files under /var/cache/apt-cacher-ng, you may safely do so, and apt-cacher will download them again the next time they are needed.

Congratulations on successfully setting up apt-cacher-ng -- I hope! It has saved hours and hours of downloading here in our classroom.

Apt-Cacher-NG Client side perl script

First off a short explanation of Apt-Cacher-NG (acng) is in order. Based on a similar project simply called Apt-Cacher, acng is a download proxy for software packages, primarily for Debian/Ubuntu clients. (http://freshmeat.net/projects/acng/)
The way it works is the server at Community Computer Center, goes out to the official package and release servers and downloads the latest updates, that way when I, or anyone else wants to download updates, our computers simply have to download from the CCC server over the lan, which makes the download about ten times faster.

(Editorial Note: If you haven't set up apt-cacher-ng yet, follow this post first.)

The only downside is whenever I want to download the updates and I'm not at CCC, I have to go and edit the hosts file in /etc, so I wrote a perl script to cut down on the steps.

Copy the code into a text file and save it as a .pl, then to use, type sudo perl yourfile.pl.
While this is not the most elegant solution, it was my first attempt at using perl, enjoy.

#!usr/bin/perl -w
#script by @robots_unix

{
print "Are you at CCC?";
$name = ;
chomp ($name);
if ($name eq "yes"){ #When I'm at CCC
open(INFILE, '>/etc/hosts');
print INFILE "127.0.0.1 localhost
127.0.1.1 uname
apt-cacher_ip apt-cacher
#127.0.0.1 apt-cacher
apt-cacher_ip server name
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts\n";
close(INFILE);
}else{ #When I'm not at CCC
open(INFILE, '>/etc/hosts');
print INFILE "127.0.0.1 localhost
127.0.1.1 uname
#apt-cacher_ip apt-cacher
127.0.0.1 apt-cacher
apt-cacher_ip servername
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts\n";
close(INFILE);
}
}

Mutt and gmail

I think gmail is cool but it could be quicker to use. Right now it is slow and if you use the html version it lacks shortcuts. I think the cool way to go is to install mutt and use it to read your mail from the terminal. This article talks about some of the reasons why it is nicer and some methods to use to make mutt notify you about new mail. And how you might be able to use gmail contacts inside mutt. I think it would be cool to use googlecl to grab a contact while in mutt.

install mutt with gmail through IMAP

http://crunchbanglinux.org/wiki/howto/howto_setup_mutt_with_gmail_imap

I like the fact you can make your own shortcuts in mutt that go to certain folders that you have set up like gmail has.

the following will make it so you go to the inbox when typeing gi and others.

bind editor noop

macro index gi "=INBOX" "Go to inbox"

macro index ga "=[Gmail]/All Mail" "Go to all mail"

macro index gs "=[Gmail]/Sent Mail" "Go to Sent Mail"

macro index gd "=[Gmail]/Drafts" "Go to drafts"

all the rest of the options are listed in the link above. I like the fact that you can change the editor for mutt too.

and the wiki might be helpful

http://wiki.mutt.org/index.cgi?MuttGuide

if anyone tries to install gmail or another client with mutt then post here and let us know how it went.

Too many messages in mutt

I have learned something useful for anyone that uses mutt to mail from the terminal. I can be used to get messages from cron that runs periodically. Cron can run as much as once a minute. My cron has been sending me the message "don't update" for 6 months. It has caused about 40MB or 29000 messages to pile up in mutt. Now I have fixed the problem so it won't echo out for success. It should only tell me about failures not successes since they don't happen very often.

Usually you can delete hundreds of messages by holding down the delete button but since I had so many messages I had to learn how to delete bult amounts of mutt mail all at once. Here is how to do it:

type D

This will tell mutt to delete using a certain pattern. I tried using "update" but it didn't work so I figured out what will tag all of the messages at once.

type .

Thats right, a period.

Then type $ to have mutt ask if you want to purge 29000 messages. It takes a second and its done.

Using Exclamation marks in Google Command Line

As I found out in my previous post, you can't just add an ! into your post without getting a bash error. You can sort of escape them with a /, but then they both show up (like this: /!), which is silly. You can get around this by first, making sure that you use single quotes around your post and then adding a $ right before the quote that begins your post:

google blogger --title 'this is a blog post' $'this is an exciting blog!'

Testing out the new Google CLI

Wow, Google keeps surprising me with new toys. I know I could just log into the GUI, but this is just too cool(bang)

Ubuntu 10.04 Lucid Lynx

We've had the chance to install Ubuntu 10.04 "Lucid Lynx" on several machines now. Except for one particular model of Dell, the experience has been wonderful. The boot time is great--less than 10 seconds on reasonably new hardware. The new default visual theme is refreshing and beautiful. It "just works," which is more than I can say for the aging Windows XP at this time, even when used with older hardware that it should "know about."

In fact, this release has been so good, that I've now reinstalled my personal notebook computer with a fresh copy of Lucid.

There are some downsides, such as the lack of customization in the login screen (a problem that has carried over from the previous release, Karmic Koala) but the trade off for speed and stability is well worth it, and I'm sure the missing options will be reintroduced in time.

For fun, I also tried installing Debian Squeeze, and was not as happy with the outcome. The Ubuntu team really does do a lot of work to make Linux more user-friendly. I also tried Linux Mint 9 Isadora, which is based off of Ubuntu Lucid Lynx, and it seems quite formidable in its own right.

Of note--I particularly like the inclusion of the new Thunderbird 3 email client. It has some amazing features, but its attempt to autodetect the settings for my mailbox was a failure, and I had to manually set the options the correct way--a task I don't think a normal user would have been able to do, since it involved turning off features that were turned on that in past versions of Thunderbird never were turned on. On the positive side, I love the new "Smart Folders" and the improved search capabilities.