Planet Netfilter

Jeremy Kerr: debian on a qs22 cell blade

2008-06-30T04:25:28+00:00

Seeing as the QS22 blades are out, here's a short guide to getting debian installed.

[jk@qs22 ~]$ grep -m1 ^cpu /proc/cpuinfo
cpu             : Cell Broadband Engine, altivec supported
[jk@qs22 ~]$ lsb_release -d
Description:    Debian GNU/Linux unstable (sid)

kernel

You'll need a kernel that has support for the IBM Cell blades. If you configure your kernel with the 'cell_defconfig' target, you should have all the necessary options:

[jk@pingu linux-2.6.25]$ make cell_defconfig

Specifically, you need:

CONFIG_PPC_IBM_CELL_BLADE;
CONFIG_SERIAL_OF_PLATFORM;
CONFIG_FUSION_SAS;
CONFIG_ROOT_NFS;
CONFIG_IP_PNP_DHCP and
CONFIG_SPU_FS.

root filesystem

The QS22s have no internal disk (they're compute nodes, right?), so you'll have to either:

use a remote root filesystem, like NFS; or
add a LSI SAS adaptor to the blade, and use an external SAS disk for the root filesystem.

The installation process will be different depending on which you choose, so just skip to the appropriate section here.

NFS root

For the first option, there's a number of NFS-root howtos around. First up, we need to build the actual debian filesystem, using debootstrap. For example:

[jk@pingu ~]$ sudo debootstrap --arch=powerpc --foreign sid /srv/nfs/qs22/

This will create an entire debian filesystem in /srv/nfs/qs22. We need to make a few modifications though:

add the following line to /etc/inittab:

T0:23:respawn:/sbin/getty -L ttyS0 19200 vt100

and couple of extra device nodes:

[jk@pingu ~]$ cd /srv/nfs/qs22/dev
[jk@pingu dev]$ sudo mknod console c 5 1
[jk@pingu dev]$ sudo mknod ttyS0 c 4 64

Once this is done, we need to complete the bootstrap on the QS22. Set up your NFS server, and export the appropriate directory. Boot the QS22 with the nfs root kernel options, plus "rw init=/bin/sh" (eg root=/dev/nfs nfsroot=server_ip:/srv/nfs/qs22 ip=dhcp rw init=/bin/sh). Then, once the machine has booted:

sh-3.2# PATH=/:/bin:/usr/bin:/sbin:/usr/sbin /debootstrap/debootstrap --second-stage

This should finish the bootstrap. After it has completed (it should finish with "I: Base system installed successfully"), reboot the machine with the same kernel command line, minus the rw init=/bin/sh arguments. Once it boots, you should have the debian login prompt. Login as root (there will be no password, but don't forget to set one) and away you go.

SAS disk

If you're using SAS, the install is much more straightforward, as you can just use the standard debian installer. However, you may need to use a custom kernel which supports the QS22s. This is a matter of building your own kernel, using the powerpc64 debian installer image as the initramfs:

[jk@pingu linux-2.6.25]$ wget http://ftp.us.debian.org/debian/dists/testing/main/installer-powerpc/current/images/powerpc64/netboot/initrd.gz
[jk@pingu linux-2.6.25]$ gunzip -c < initrd.gz > initrd
[jk@pingu linux-2.6.25]$ make cell_defconfig
[jk@pingu linux-2.6.25]$ sed -ie 's,^CONFIG_INITRAMFS_SOURCE=".*",CONFIG_INITRAMFS_SOURCE="'$PWD'/initrd",' .config
[jk@pingu linux-2.6.25]$ make

Then, just boot the kernel in arch/powerpc/boot/zImage.pseries. The debian installer should start, and guide you through the rest of the installation. Since you're netbooting, you can ignore any messages about not having a bootstrap partition, or not being able to install a kernel or yaboot

software

Entirely optional, but you'll probably get the most out of your QS22 with a few extra packages:

[jk@qs22 ~]$ sudo apt-get install openssh-server libspe2-dev spu-gcc build-essential

Patrick McHardy: Bored kids

2008-06-30T00:00:00+00:00

These two kids were looking for something fun to do on Saturday evening. They first tried to climb the wall of the Freiburg theater (visible in the back of the picture), but the left one only made it half way up. Very disappointed, he stated he felt emasculated and both of them left. He tried to regain his manliness a few minutes later when they returned with a stolen table from the bar next door and used it to slide down the stairs.

Getting ready to launch ...

They made it down the stairs. The left one is really having fun, the right one is also beginning to feel like a man again.

End of the ride.

Also don't miss out on the movie of the waitress slapping them around and their flight.

Thanks to Elena for the pictures and redacting the face of an innocent bystander :)

Rusty Russell: Linux Foundation's Device Driver Statement

2008-06-27T15:02:00+00:00

Someone noted that I didn't sign the LF "proprietary modules are bad" statement. This is entirely due to my slackness and not any lack of support.

As kernel module maintainer I feel obliged to maintain the status quo with proprietary modules, but I have noticed many colleagues becoming more annoyed about them.

David Miller: Think I've got it...

2008-06-27T07:39:00+00:00

I've redone everything about how I'll approach the multiqueue TX issues. I'll try to detail this a bit tomorrow.

A big chess match, Sydney I posted a small teaser on netdev

David Miller: Network device TX lock.

2008-06-26T04:45:00+00:00

If you want to turn the world upside down and rework an entire core interface, that's fine, but you'd better completely understand how things currently work before proceeding.

That's the situation I'm in right now.

If you're following netdev then you know that I'm trying to get the network device transmit path all sorted wrt. multiqueue devices. The current situation is that the entire path is serialized so really it's not possible to take advantage of sophisticated devices on the send side.

It also turns out that the modelling for device throttling is overly simplistic. Whether a packet can currently be accepted by the device depends upon a lot of things, some of which are determined by aspects of the specific packet that is next to be sent.

So how are things locked now and why?

Network device driver transmit locking? Yee haw!

Go far back in Linux history and we come to a time when SMP support was it's infancy or simply non-existent. The transmit path into the driver was guarded by a simple bit lock on a word stored in the struct net_device. It was named "tbusy".

You can identify old time experienced Linux networking hackers by their being able to explain to you what "tbusy" is and how it worked.

The initial SMP support in Linux was all about essentially holding a global lock across all kernel execution. The basic invariant preserved by this was lack of re-entrancy.

At this time drivers did things like disable interrupts to prevent concurrent access to the transmit datastructures. Very simple and very stupid. And we had to find some way to preserve the semantics these simple devices wanted while initially SMP threading the entire network stack.

Add into this the complexity of our packet scheduler. We have all of these sophisticated semantics you can tie to the packet flow going out a particular device. These sophisticated algorithms have a lot of shared data structures which guide the packet sending decisions. So these need protection too.

So we have two state machines on the path a packet takes out of the system. It's very nearly a two coupled loop system, you know the kind Van Jacobson talked about in his LCA06 presentation. The one's that tend to be more unstable than a single coupled loop.

On the one hand we have the device which can queue up a fixed number of packets at a time. And on the other we have the packet scheduler which wants to know when the device receives the packet, and when the device's queue fills up, so that it can make rate calculations and decide to (in some cases) even drop packets when the backlog gets beyond a certain point.

The solution to the re-entrancy problem for the simplistic drivers was to wrap all calls to ->hard_start_xmit() with a spinlock stored in the generic network device structure. It guaranteed that there would only be one thread of control executing in the driver's transmit function for a particular device instance.

The packet scheduler state is managed with it's own locks. And we have this complex interaction between the two sets of activity because one cannot make progress without knowing the state of the other.

So now, some 8 or 9 years later, we have on the order of 400 networking drivers. This means if changes to how this works will occur, we better think long and hard about it, because we'll have to make any such change 400 times and make sure we get it right. The main goal is to get it so that the driver can control the locking (only it knows how to handle multiple queues, etc.) and also manage the queue backlog (which also might need to be multiqueue).

Currently I'm the one with this task of figuring out how to make this work even with all of this legacy stuff sitting around.

As a first step I drew up some patches for the NIU and TG3 drivers, which move the TX backlog handling into the drivers themselves. This is sort of a baby step towards what needs to actually happen. In the end we somehow have to integrate the qdisc dequeueing logic into these places.

Outta here you swine!

Next, for every driver that does not request lockless transmit handling, I added a per-driver spinlock that protects the entirety of the ->hard_start_xmit() handler. Things get real exciting here because there are code paths that believe that holding the TX lock means there is nothing executing inside of this routine, and that if you do something like:

	netif_tx_lock(dev);
	netif_stop_queue(dev);
	netif_tx_unlock(dev);

then you can be sure afterwards that nothing will call into the ->hard_start_xmit() handler.

Preserving that semantic is non-trivial now that the lock is grabbed inside of the handler. I'm currently supposing that a near equivalent can be obtained using something like:

	spin_lock(&tp->tx_lock);
	netif_stop_queue(tp->dev);
	spin_unlock(&tp->tx_lock);
	synchronize_net();

But this brings us to another fun topic. Currently the maintainence of the TX queue throttling is made inside of this generic TX lock. This is part of why I decided that the TX backlog had to be taken care of inside of the driver. The only way to protect the queue state properly, turning it off and re-enabling it later when TX space becomes free again, is now going to be with the driver private TX lock. So now the backlog is also the driver's business and problem. The state maintainence follows the locking responsibility.

The biggest headache is how to tie this into the packet scheduler state management.

I want to see this hit 2.6.27 if possible. Very optimistic, and my espresso machine will be working overtime.

Patrick McHardy: Companies to avoid

2008-06-23T00:00:00+00:00

Warning: long rant. Note to any company mentioned below: this is *my* opinion and my opinion only.

Just added Lenovo to the list of companies I won't buy from anymore. I got a Z61p about 9 months ago and had nothing but trouble since. Air circulation seems to be broken by design, from the beginning the graphic card overheated to over 100° celsius, then starting makeing squeaky sounds and showing flickering moving lines across the entire screen, before shutting down completely (not the notebook, only the card). There are plenty of reports on the internet of people having similar problems. The processor also often heats up until it reaches the shutdown temperature when doing CPU intensive work. A technician tried to fix it by replacing some parts, but without any success. Additionally on my travel to LinuxTag, the fan broke and it would refuse to boot. This cured itself a few days later, but now it sounds like a rusty lawn mower. Since last weekend, it doesn't detect the battery anymore, even though a voltmeter shows its working perfectly fine. Sad, back when IBM was still producing ThinkPads I never had trouble.

The other two companies on this list of pride are (there are more, but most of them are irrelevant since they are either almost broke or small enough to avoid easily):

- Deutsche Telekom and all their subsidiaries. This is the most ridiculous company I've ever seen, the highlights of their doings include:

When installing a DSL connection in my appartment, the technician (also known as Telekomiker, roughly translated as Telekom commedean, but more funny in German) wasn't able to locate the wires in the switch cabinet. So since there was no way to test what we was doing, he apparently decided not to do anything at all and just left the wires in the socket unconnected, after which he told me "all done" and left. I managed to get him on the phone personally by calling a hotline, telling them my story and being forwarded like eight times. He showed up again two hours later and did it properly :)
Delay and displace contract changes and cancellations, not sending confirmations and so on. The most recent incident was billing us for months for a canceled telephone line, which numbers were already ported to a different company. On every call to their support, they promised to look into it, only to send a new invoice a few days later. Most ridiculous, when calling their business support late at night, we reached a call center, with a friendly telephonist who told me she was unable to even take a message, the only thing she was there for was for telling us we were calling outside business hours. At least the amusement made up for it a bit :)
Sending incorrect invoices over months for some equipment returned to them within the deadline for returning it. As usual, their phonebots proved to be completely incompetent and weren't able to correct the mistake. It went the usual way to their law firm, Seiler & Kollegen, who did correct the mistake after we sent them the a copy of the receipt. So what Telekom did after that was "correct" their invoice, remove the incorrect position, but left all the reminder charges (for an *incorrect* position to begin with) on it. They continued sending invoices with increasing reminder charges (1 euro per invoice) for over two years. Not sure if they're still sending them, I stopped carring.

I've decided not to ever do business not only with Telekom and any of their subsidaries, but also with any company reselling their products or being closely affiliated with them. You just live better this way.

- HP, for not fulfilling their service obligations for fixing my notebook. They first sent some clown from Deutsche Telekom to fix it, who broke it even worse. His second attempt was also unsuccessful, after which HP simply closed the request. Every time I reopened it, it was closed again without further comment. They even had the impudence to ask for my satisfaction with their service - which was pretty obvious from looking at the request. Also sad, because I liked the notebook and there are not many alternatives if you want a big display, but such behaviour is inacceptable.

Jeremy Kerr: linux.conf.au hackfest: the solution, part three

2008-06-22T23:20:13+00:00

In part two of this series, we had just ported a fractal renderer to the SPEs on a Cell Broadband Engine machine. Now we're going to start the optimisation...

Our baseline performance is 40.7 seconds to generate the sample fractal (using the sample fractal parameters).

The initial SPE-based fractal renderer used only one SPE. However, we have more available:

Number of SPEs in currently-available in CBEA machines
Machine	SPEs available
Sony Playstation 3	6
IBM QS21 / QS22 blades.	16 (8 per CPU)

So, we should be able to get better performance by distributing the render work between the SPEs. We can do this by dividing the fractal into a set of n strips, where n is the number of SPEs available. Then, each SPE renders its own strip of the fractal.

The following image shows the standard fractal, as would be rendered by 8 SPEs, with shading to show the division of the work amongst the SPEs.

In order to split up the work, we first need to tell each SPE which part of the fractal it is to render. We'll add two variables to the spe_args structure (which is passed to the SPE during program setup), to provide the total number of threads (n_threads) and the index of this SPE thread (thread_idx).

struct spe_args {
        struct fractal_params fractal;
        int n_threads;
        int thread_idx;
};

spe changes

On the SPE side, we use these parameters to alter the invocations of render_fractal. We set up another couple of convenience variables:

        rows_per_spe = args.fractal.rows / args.n_threads;
	start_row = rows_per_spe * args.thread_idx;

And just alter our for-loop to only render rows_per_spe rows, rather than the entire fractal:

        for (row = 0; row < rows_per_spe; row += rows_per_dma) {

                render_fractal(&args.fractal, start_row + row,
                                rows_per_dma);

                mfc_put(buf, ppe_buf + (start_row + row) * bytes_per_row,
                                bytes_per_row * rows_per_dma,
                                0, 0, 0);

                /* Wait for the DMA to complete */
                mfc_write_tag_mask(1 << 0);
                mfc_read_tag_status_all();
        }

ppe changes

The changes to the PPE code are fairly simple - instead of just creating one thread, create n threads.

First though, let's add a '-n' argument to the program to specify the number of threads to start:

        while ((opt = getopt(argc, argv, "p:o:n:")) != -1) {
                switch (opt) {

                /* other options omitted */

                case 'n':
                        n_threads = atoi(optarg);
                        break;

Rather than just creating one SPE thread, we create n_threads. Also, we have to set the thread-specific arguments for each thread:

        for (i = 0; i < n_threads; i++) {
                /* copy the fractal data into this thread's args */
                memcpy(&threads[i].args.fractal, fractal, sizeof(*fractal));

                /* set thread-specific arguments */
                threads[i].args.n_threads = n_threads;
                threads[i].args.thread_idx = i;

                threads[i].ctx = spe_context_create(0, NULL);

                spe_program_load(threads[i].ctx, &spe_fractal);

                pthread_create(&threads[i].pthread, NULL,
                                spethread_fn, &threads[i]);
        }

Now, the SPEs should be running, and generating their own slice of the fractal. We just have to wait for them all to finish:

        /* wait for the threads to finish */
        for (i = 0; i < n_threads; i++)
                pthread_join(threads[i].pthread, NULL);

If you're after the source code for the multi-threaded SPE fractal renderer, it's available in fractal.3.tar.gz.

That's it! Now we have a multi-threaded SPE application to do the fractal rendering. In theory, an n threaded program will take 1/n of the time of a single-threaded implementation, let's see how that fares with reality...

performance

Let's compare invocations of our multi-threaded fractal renderer, with different values for the n_threads parameter.

Performance of multi-threaded SPE fractal renderer
SPE threads	Running time (sec)
1	40.72
2	30.14
4	18.84
6	12.72
8	10.89

Not too bad, but we're definitely not seeing linear scalability here; we could expect the 8 SPE case to take around 5 seconds, rather than 11.

what's slowing us down?

A little investigation into the fractal generator will show that some SPE threads are finishing long before others. This is due to the variability in calculation time between pixels. In order to see if a point (ie, pixel) in the fractal does not converge towards infinity (and gets coloured blue), we need to do the full i_max tests in render_fractal (i_max is 10,000 in our sample fractal). Other pixels may converge much more quickly (possibly in under 10 iterations), and so are orders of mangitude faster to calculate.

We end up with slices that are very quick to calculate, and others that take longer. Of course, we have to wait for the longest-running SPE thread to complete, so our overall runtime will be that of the slowest thread.

So, let's take another aproach to distributing the workload. Rather than dividing the fractal into contiguous slices, we can interleave the SPE work. For example, if we were to use 2 SPE threads, then thread 0 would render all the even chunks, and thread 1 would render all the odd chunks (where a "chunk" is a set of rows that fit into a single DMA). This should even-out the work between SPE threads.

This is just a matter of changing the SPE for-loop a little. Rather than the current code, which divides the work into n_threads contiguous chunks:

        for (row = 0; row < rows_per_spe; row += rows_per_dma) {

                render_fractal(&args.fractal, start_row + row,
                                rows_per_dma);

                mfc_put(buf, ppe_buf + (start_row + row) * bytes_per_row,
                                bytes_per_row * rows_per_dma,
                                0, 0, 0);

                /* Wait for the DMA to complete */
                mfc_write_tag_mask(1 << 0);
                mfc_read_tag_status_all();
        }

We change this to render every n_thread^th chunk, starting from thread_idx, which gives us the the interleaved pattern:

        for (row = rows_per_dma * args.thread_idx;
                        row < args.fractal.rows;
                        row += rows_per_dma * args.n_threads) {

                render_fractal(&args.fractal, row,
                                rows_per_dma);

                mfc_put(buf, ppe_buf + row * bytes_per_row,
                                bytes_per_row * rows_per_dma,
                                0, 0, 0);

                /* Wait for the DMA to complete */
                mfc_write_tag_mask(1 << 0);
                mfc_read_tag_status_all();
        }

An updated renderer is available in fractal.4.tar.gz.

Making this small change gives some better performance figures:

Performance of multi-threaded, interleaved SPE fractal renderer
SPE threads	Running time (sec)
1	40.72
2	20.75
4	10.78
6	7.44
8	5.81

We're doing much better now, but we're still nowhere near the theoretical maximum performance of the SPEs. More optimisations in the next article...

Patrick McHardy: Too busy to blog

2008-06-21T11:00:03+00:00

Since I've been slacking with updating this blog lately, and probably will continue to do so for the next week, here's another drawing from Elena from a couple of weeks ago.

Jeremy Kerr: qs22 released

2008-06-17T23:11:07+00:00

The next revision of IBM Cell Broadband Engine machines has just been released - the QS22 blade. The QS22 has five-times the double-precision floating point performance of the previous Cell blade (the QS21), but is instruction-set compatible. This is also the first Cell/B.E. machine to use DDR2 memory, and can hold up to 32GB.

In other powerpc news, Terra Soft Solutions have announced the PowerStation - a deskside development machine, based on 2 dual-core PowerPC 970MP CPUs. It comes with Yellow Dog Linux installed, including the Cell/B.E. SDK. Hugh (amongst many others) has been doing a lot of great work getting this machine out the door (he has a number of posts on his blog if you're keen to see the "making of" feature). Nice work Hugh!

Harald Welte: DVB-T transmit in pure PC software

2008-06-17T02:00:00+00:00

I recently discovered this paper about Soft-DVB, a full PC-software DVB-T transmitter, it apparently is now possible on a 1.8GHz Celeron M based system to do a full software encode/modulation of a MPEG2 transport stream onto a DVB-T compliant carrier that can be received by off-the-shelf consumer DVB-T receivers. And all this on Linux, using gnuradio and the USRP.

This is really great news, and an incredible achievement by the authors of the software, particularly Vincenzo Pellegrini.

There is one (at this time still) moot point, though: The code has not been released yet. It has been demoed at SDR related conferences, so it really exists. Vincenzo has announced on the gnuradio-discuss mailinglist that eventually it will be public - without stating some kind of date, though.

I suppose he probably has to wait until his master thesis has been finalized and approved. That should be in the order of months, not years...

Patrick McHardy: iptables 1.4.1.1 released

2008-06-17T00:00:00+00:00

Just released iptables 1.4.1.1, a pure bugfix release for regressions reported against 1.4.1. Besides this, I'm mainly in bugfix mode for 2.6.26 currently, which is keeping me pretty busy.

Harald Welte: Nokia, FOSS, SIM Locks, DRM and the universe + Motorola's failure

2008-06-14T02:00:00+00:00

As Bruce Perens points out at this blog entry, it is very much possible to design a product, particularly an embedded Linux device such as a mobile handset with all the usual bits and pieces (DRM for mobile media content, SIM locks, etc.) while preserving the freedom of Free Software.

I'm really pissed off by the kind of FUD that big vendors try to spread about it. There are so many claims that the user has to be locked down, that he cannot be allowed to modify/replace the Linux kernel or other bits of the software stack, etc.

I can only agree full-heartedly with Bruce's article. Such claims are all bullshit. I've worked for a long enough time with Free Software, the Licenses involved, the legal framework of those licenses (Copyright Law), the Hardware Industry, lately even a mobile handset manufacturer. I've seen the software and hardware architecture of a number of phones myself by reverse engineering. Never have I found any reason why the bright-line philosophy (see Bruce's article) should not result in a perfectly working, all-interests-satisfied solution.

Let me use this opportunity to point out my disappointment at the failure of Motorola to solve this problem properly. Instead of designing their MotoMAGX family of handsets in a way that preserves the freedom of the Free Software [community, users] and protects their valid business interests, they chose to go the easy shortcut of walking borderline on what they think the GPL permits them: They use cryptographically signed kernel images, a bootloader that only accepts binaries signed by them, plus a kernel that only accepts signed modules, plus a SELinux locked-down userspace that is very restrictive on what userspace programs can still do.

This would all be nice and good _if_ they were to provide the user with a way to either sign his own kernel images with their key, or (better) to store his own signature in the bootloader. So the hardware would accept Motorola-signed kernels and kernels signed by the user (actual owner!) of the device.

The further proprietary bits of the software stack required for DRM protection can simply refuse to operate if not run under a Motorola-signed kernel. Especially with TPM's and similar technologies becoming more widespread in the mobile world, there is a very straight-forward solution to this problem. The bootloader can store the hash of the kernel image in some TPM protected register, and the proprietary DRM system can refuse to operate if the hash is not the original one.

With regard to SIM-Lock, Operator-Lock and all the other locks: As Bruce points out, those are restrictions of the GSM/3G modem. All implemented in the firmware of this device. It doesn't matter if you run Windows Mobile, Symbian, Motorola's own locked-down Linux kernel or a custom user-built Linux kernel on the application processor. The various GSM/3G related locks are never implemented on that processor, but on the baseband side.

I hereby challenge the mobile industry to come up with hard, technical fact about what particular problem they have in designing open, FOSS-compatible devices, where every user can modify and/or replace the FOSS programs, while ensuring the integrity of their DRM, IPR, SIM lock and other business model related technologies. I will sit down and look at any such issue brought forward and I'm extremely confident that for all of such problems there's a straight-forward technical solution (bright-line in Bruce's terminology) which will not require the proprietary or FOSS side to make any sort of moot compromise.

If not only for the reason of legal safety and security, such solutions should always preferred to going borderline with FOSS licenses or against the FOSS developers and users community!

Rusty Russell: stop_machine latency

2008-06-12T11:29:00+00:00

Kathy Staples and I wrote a little program to measure the latency on every CPU on a machine. It sets CPU affinity and high priority (SCHED_FIFO, prio 50) for each thread, then spins doing gettimeofday() for a given duration. The maximum gap in gettimeofday() is reported for each CPU.

I tested this on an old 18-way Power4 box sitting around the lab: CPU 0 is used for the parent process, and the latency is measured on the other CPUS. This was run 100 times. Then a variant which did an insmod system call on CPU 0 was used (this calls stop_machine, which is what we were trying to measure).

The results are interesting and a little surprising. Normal max latency is around 35 usec, the stop_machine increasing it to the 100 range. There's obviously something running periodically on CPU 2: for both runs I had to remove one horrific 150ms latency result (1000 times average!) but there's still a noticeable spike there. I suspect CPU1 is low because CPU0 is mainly idle (same core).

But more concerning is that latency seems to go up with higher CPU numbers, whereas I expected it to be worst on lower CPUs. We launch stop_machine threads in cpu order, so I expected the lower CPUs to wait the longest.

We're running modprobe on cpu 0, which means the stop_machine control thread runs there, too. It loops through creating 17 other threads: as CPU 0 is busy, it gets scheduled on a different idle CPU. The first thing the thread does is try to move itself to its proper CPU.

I suspect what is happening is that we're creating the 17 threads fast enough that they all end up queued on the migration queue for CPU 0 at once: this queueing uses "list_add" not "list_add_tail", so they are in fact deployed by the migration thread in reverse-CPU order.

My simplified version of stop_machine is more intelligent: it moves all the threads to their correct CPUs before waking them all up. This should solve this problem as well as reducing overall latency.

Patrick McHardy: iptables 1.4.1 release

2008-06-10T00:00:00+00:00

Finally released iptables 1.4.1 this morning. I had the impression the -rc phase worked pretty well this time and hoped we had shaken out all the bugs. Unfortunately this hope wasn't fulfilled, the first regression report came in only 5 hours later. Its nothing terribly important, just a cosmetic problem when printing IPv6 masks, but I guess I'll release a 1.4.1.1 bugfix release in a few days.

Harald Welte: Persistent Google recruiters suck

2008-06-09T02:00:00+00:00

I think I've read this before by one or the other Linux/FOSS developers blog: Googles persistent recruitment sucks. At least I've spoken with a number of well-known developers in the community, and they all have been contacted before.

What makes the situation even more difficult is that there are apparently different recruitment teams, so sometimes they want to hire you in Australia, sometimes somewhere else. I've heard rumors that they now have a company-wide blacklist, and I've asked a number of times to not receive further recruitment mail, so I should be on that list by now.

The worst message arrived today. The particular recruiter actually _knew_ that the same department had last contacted me six months ago, and that I was completely not interested. But she was hoping that by now my mind or my job situation had changed, and that she would want to talk to me about employment options at Google.

I'm now really running out of options. I've tried to state it politely a number of times over many years that I am not interested and do not want to receive further emails. As if this wouldn't occur to me automatically, given their omnipresence in the Internet world, and their numerous previous recruitment mails, even in the case I actually was seeking employment now.

I guess I will have to try to be rude now, maybe then they think my personality wouldn't fit the company spirit. I don't know.

Just let me say that this kind of aggressive recruiting is in itself alone reason enough for me to not want to work for this company :(

Patrick McHardy: Release delays

2008-06-05T00:00:00+00:00

The iptables 1.4.1 release got delayed a bit by my notebook breaking 5 minutes after getting on the train to LinuxTag, so I couldn't do any real work the entire last week. I hoped we could test the header fixes last week and release on Monday, but they really need some wider testing, so I'll release another -rc today and hopefully the final release in about a week.

On the kernel side, I'm working on getting the things I would like to merge in 2.6.27 into shape. The netfilter things in my queue so far are mostly minor cleanups and feature additions, with the exception of ebtables IPv6 support from Kuo-lang Tseng.

The non-netfilter things are:

A GARP implementation with a GVRP application on top. Maybe I'll also add GMRP support.
Some VLAN patches to fix inconsistencies related to hardware tagging/stripping/filtering and packet sockets.
A DRR packet scheduler.
Some patches for dynamic packet scheduler class hash sizing for better scalability that I've been carrying for at least a year.

Patrick McHardy: iptables release status

2008-05-28T11:00:03+00:00

I managed to push out an iptables release candidate last week and another one this week. There are some issues with endian-annotated types in the netfilter headers when using ancient linux/types.h versions that need to be fixed before a final release, but we'll hopefully have a final 1.4.1 release by next monday.

Patrick McHardy: Leaving for LinuxTag

2008-05-28T11:00:03+00:00

I'll be leaving for my train to Berlin now. Amazingly it took me more time to decide what to put on my notebook than what to pack in my suitcase :) Hope to see you there ...

Patrick McHardy: Flying outside

2008-05-23T11:00:05+00:00

did not work to well ...

Patrick McHardy: Geek toys

2008-05-23T00:00:00+00:00

Received a bunch of things to play with this morning.

A customer wants me to develop a multipath tunneling protocol. The basic idea is to encapsulate packets, distribute them over N paths, decapsulate them on the other side and restore ordering using sequence numbers. This will allow to use both the combined upstream and downstream bandwidth for single connections. I wrote a prototype some time ago, but its very basic at this point and missing all the fancy features, like microflow seperation, delay aware distribution, dead path detection, etc.

I used to have two internet connections for a couple of months, but canceled the second one in march. Since there's nothing like real-life testing, I ordered a second cable connection again, which was installed this morning. So now I have two 32/2.5mbit connections, but only one of them is used currently. I probably won't be able to resist the urge to work on this for long :)

Additionally I received some VoIP testing equipment. Innovaphone kindly provided me with a H.323/SIP test setup consisting of 2 * 3 different telephone models, two PBXs and some test scenarios. We already support a lot of scenarios, my goal is to extend this as far as possible within reason.

The tricky part are things like call transfers crossing two NATs:

  Phone1-\                                              /------Phone3
          -[Registrar1]-[NAT1]-{  }-[NAT2]-[Registrar2]-
  Phone2-/                                              \------Phone4

When Phone1 calls Phone3 and the call is transfered to Phone2, we currently fail completely (for so far unknown reasons). The ideal outcome would be that NAT1 detects that the transfered call originated in the local network of Phone1 and the RTP streams are set up between Phone1 and Phone2 directly. This not only reduces latency, it avoids having internal calls go over the Internet. For normal calls between Phone1 and Phone2 using an external registrar this already works, provided that the registrar doesn't decide to proxy the calls.

Speaking of geek toys, ThinkGeek has an incredibly fun toy..

The two helicopters are controlled using IR remote controls. They can only go up and down by controlling main rotor speed and spin around the rotor-axis using the rear rotor, but have some small constant forward movement, which allows to fly them quite precisely. The most important feature however is that they can shoot at each other using IR. On the first hit it spins a bit, on the second one it looses power for a short period of time, on the third hit it completely looses power and goes down. This is accompanied by shooting sounds. Unfortunately they break pretty fast, I wrecked four of them within only a few days - well actually three of them, the fourth one was last seen flying over my neighbours garden :)

Being hooked on the fun, I got a a different model , which is controllable on all axis and supposed to be more robust.

Unfortunately it also has a lot more power, so I managed to wreck another one within hours: one gear broke, the tail bent to a 45° angle and then broke, the flight bar also took some damage. To be fair, it really is more robust (and luckily you can also order replacement parts), I'm just not used to the two two-dimensional controls. Instead of powering down when getting to close to the walls, I tried directing it away by pushing all sticks in the desired direction, causing it to increase power and crash in the wall.

I have some replacement parts and a second helicopter, but I'll try to resist flying inside again until I can get some training in a less dangerous environment. Regardless of these problems, they are really great fun :)

Patrick McHardy: LinuxTag

2008-05-22T00:00:00+00:00

I'll be visiting LinuxTag in Berlin next week, probably the entire day of Thursday and Friday until sometime in the afternoon. Until a couple of years ago, I used to visit LinuxTag annually, but then the quality of the presentation declined, with a lot of the topics being along the lines of "We're company XYZ and we're using Linux, yay". This year the talks look more interesting again, and I'm looking forward to meet with Harald and DaveM.

Anyone interested in meeting and discussing some netfilter or networking related topics, drop me an email, there should be plenty of time.

Harald Welte: Last minute: Presenting at LinuxTag

2008-05-21T02:00:00+00:00

As apparently there was a last-minute drop-out in the Security track of LinuxTag 2008, I have been invited to present. It is great that I could convince them to do a talk about my current favorite subject: Enabling more security research in communications protocols outside the TCP/IP/Ethernet based Internet.

I don't want to spoil it by providing too much information upfront. I'm sure there will be recordings available afterwards. For now, you can get the main points from the abstract

David Miller: Grub on Sparc...

2008-05-20T02:22:00+00:00

Just for fun I started trying to get Grub working on Sparc, there is some code there for ieee1275 firmware stuff but mostly there hasn't been much work done. Current CVS of grub2 doesn't even build on sparc.

A long time ago I contacted the GRUB author, and wanted to contribute to making sparc work, but he told me at the time something crazy. He said that since I worked on our existing loader SILO, I was "contaminated" and therefore couldn't write code for GRUB. I was pretty stunned, as SILO is GPL'd :-) But, whatever.

I figured in 5 years they'd have something working by now, but that is not the case. So I'm going to make this work and if the GRUB folks decide to still be unreasonable and not take my code we'll just have a fork in every distribution I guess. They can happily be without working sparc support for another 5 years, in that case.

The first task is to get the first stage boot block going. There are two choices on how to do this. When you type "boot" for a block device, the firmware loads 7.5K starting at the second 512-byte block. If this block device is the third partition (the "all disk" partition in the Sun disk label) this bootblock starts after the disk label.

Under Linux we really can only use 512 bytes of that boot block area, because it's possible for the filesystem superblock to show up as early as the very next 512 byte block.

The firmware lets you put in the block some sparc executable image (tried by the firmware as 64-bit ELF, then 32-bit ELF, and finally as A.OUT) or tokenized forth. Since 1) we only have 512 bytes and 2) there are no fully GPL'd forth tokenizer implementations (the openbios folks use something that is BSD and MIT licensed) we'll need to go the sparc image route.

The task of this 512 byte sequence of code is to load the next stage of the bootloader. For GRUB I've choosen a multi-tiered scheme similar to how the x86 stuff works. The first stage bootloader loads a single block from the disk and jumps to it. This block we load is actually a header block of the main boot loader binary, a second stage loader, which loads the rest of the image.

This first stage loader therefore needs a few parameters. It needs to know the OF device path where the second stage header block resides. It needs to know the block to read, and finally it needs to know where to load that block and thus where to jump to it for execution.

We'd also like to print some status messages to the screen while this happens and have at least some minimal error handling. Not a small feat in 512 bytes.

We put everything in the text section, and the first thing we do is jump over our embedded data bits:

	.text
	.align	4
	.globl	_start
_start:
	/* OF CIF entry point arrives in %o4 */
pic_base:
	call	boot_continue
	 mov	%o4, CIF_REG

The "CIF" is the client interface to openfirmware. Calls are made by initializing an array of cells (64-bits on sparc64) in memory which describe the call to be made, the input arguments, and the return values (if any). This value provided in %o4 is the OF entry point we jump to when making calls. The only register argument goes in %o0 and is the base of the aforementioned array of cells.

The offsets into the bits coming up are defined in a GRUB boot.h header file so that tools can patch in values during bootblock installation.

	. = _start + GRUB_BOOT_MACHINE_VER_MAJ
boot_version:		.byte	GRUB_BOOT_VERSION_MAJOR, GRUB_BOOT_VERSION_MINOR
boot_path:
	. = _start + GRUB_BOOT_MACHINE_KERNEL_ADDRESS
kernel_sector:		.xword 2
kernel_address:		.word  GRUB_BOOT_MACHINE_KERNEL_ADDR

The boot_version is just a version blob that various tools and sub-bootloaders could validate for compatibility if they wanted to. It is unused currently.

The boot_path will be filled in by the boot block installation tools with the boot device OF path. kernel_sector and kernel_address tell where to load the image from the device into memory. Next, we have string constants we'll need to make OF calls and put messages onto the console:

prom_finddev_name:	.asciz "finddevice"
prom_chosen_path:	.asciz "/chosen"
prom_getprop_name:	.asciz "getprop"
prom_stdout_name:	.asciz "stdout"
prom_write_name:	.asciz "write"
prom_bootpath_name:	.asciz "bootpath"
prom_open_name:		.asciz "open"
prom_seek_name:		.asciz "seek"
prom_read_name:		.asciz "read"
prom_exit_name:		.asciz "exit"
grub_name:		.asciz "GRUB "

To simplify things we'll write all of our code as position independent. There are other macros in the boot.h header which describe these register name macros such as CIF_REG, PIC_REG, etc. most are local registers which are not volatils across OF calls, so we can keep stable values in them. It also defines macros to compute absolute addresses of a symbol using this PIC_REG, into a register.

The next chunk of code are helpers for doing OF calls with various sets of input and output arguments.

prom_open_error:
	GET_ABS(prom_open_name, %o2)
	call	console_write
	 mov	4, %o3
	/* fallthru */

prom_error:
	GET_ABS(prom_exit_name, %o0)
	/* fallthru */

	/* %o0: OF call name
	 * %o1: input arg 1
	 */
prom_call_1_1:
	mov	1, %g1
	ba	prom_call
	 mov	1, %o5

	/* %o2: message string
	 * %o3: message length
	 */
console_write:
	GET_ABS(prom_write_name, %o0)
	mov	STDOUT_NODE_REG, %o1
	/* fallthru */

	/* %o0: OF call name
	 * %o1: input arg 1
	 * %o2: input arg 2
	 * %o3: input arg 3
	 */
prom_call_3_1:
	mov	3, %g1
	mov	1, %o5
	/* fallthru */
	
	/* %o0: OF call name
	 * %g1: num inputs
	 * %o5: num outputs
	 * %o1-%o4: inputs
	 */
prom_call:
	stx	%o0, [%l1 + 0x00]
	stx	%g1, [%l1 + 0x08]
	stx	%o5, [%l1 + 0x10]
	stx	%o1, [%l1 + 0x18]
	stx	%o2, [%l1 + 0x20]
	stx	%o3, [%l1 + 0x28]
	stx	%o4, [%l1 + 0x30]
	jmpl	CIF_REG, %g0
	 mov	%l1, %o0

All of those routines are "call" invoked, and we do the CIF OF call using "jmpl" with no return register write in order to make the CIF OF call return to the stub caller, ie. a tail call. This way we don't need to allocate a register window here or anything complicated like that.

Now for the code we jumped to at the _start header. Our first task is to save away the PIC_REG (%o7 was set by the initial "call" instruction, and will be equal to _start, we use it to reference our data blobs PC relative). Also we setup register %l1 which holds the base of the OF cell argument data block used above. SCRATCH_PAD is defined to 0x10000, which is guarenteed to be mapped by OF and outside of where we will be executing (which is at 0x4000).

boot_continue:
	mov	%o7, PIC_REG		/* PIC base */
	sethi	%hi(SCRATCH_PAD), %l1	/* OF argument slots */

We need the console stdout handle to write console messages, this is done by finding the "/chosen" directory in the OF device tree, and fetching from there the "stdout" property which is a 32-bit handle.

	GET_ABS(prom_finddev_name, %o0)
	GET_ABS(prom_chosen_path, %o1)
	call	prom_call_1_1
	 clr	%o2
	ldx	[%l1 + 0x20], CHOSEN_NODE_REG
	brz	CHOSEN_NODE_REG, prom_error
	 GET_ABS(prom_getprop_name, %o0)
	mov	4, %g1
	mov	1, %o5
	mov	CHOSEN_NODE_REG, %o1
	GET_ABS(prom_stdout_name, %o2)
	add	%l1, 256, %o3
	mov	1024, %o4
	call	prom_call
	 stx	%g1, [%l1 + 256]
	lduw	[%l1 + 256], STDOUT_NODE_REG
	brz,pn	STDOUT_NODE_REG, prom_error

Since we have very little space, the error handling has to be small. The "clr %o2" in the prom_call_1_1 invocation causes the return value cell slot (at %l1 + 0x20) to be cleared by the prom_call code. Zero is not a prom handle we'll get back, so if the slot stays zero we took an error. In the getprop call to get the 'stdout' handle, we rely upon OF providing us with zero'd memory outside of the boot block image.

Now that we have the console output cookie, we can print out a message:

	 GET_ABS(grub_name, %o2)
	call	console_write
	 mov	5, %o3

Next, we open up the boot device:

	GET_ABS(prom_open_name, %o0)
	GET_ABS(boot_path, %o1)
	call	prom_call_1_1
	 clr	%o2

	ldx	[%l1 + 0x20], BOOTDEV_REG
	brz,pn	BOOTDEV_REG, prom_open_error

We now seek to the appropriate block:

	 GET_ABS(prom_seek_name, %o0)
	mov	BOOTDEV_REG, %o1
	clr	%o2
	LDX_ABS(kernel_sector, 0x00, %o3)
	call	prom_call_3_1
	 sllx	%o3, 9, %o3

and finally read the block into memory:

	GET_ABS(prom_read_name, %o0)
	mov	BOOTDEV_REG, %o1
	LDUW_ABS(kernel_address, 0x00, %o2)
	call	prom_call_3_1
	 mov	512, %o3

This image will be an A.OUT image as well, so we jump into it right past the A.OUT header:

	LDUW_ABS(kernel_address, 0x00, %o2)
	jmpl	%o2 + GRUB_BOOT_AOUT_HEADER_SIZE, %o7
	 nop
1:	ba,a	1b

Just for fun we put an endless loop after the jump in case it does return for some reason. We could, alternatively, call prom_error instead. Now skip to 4 bytes right before the end of the 512-byte block and write a signature cookie.

	. = _start + GRUB_BOOT_MACHINE_CODE_END

/* the last 4 bytes in the sector 0 contain the signature */
	.word	GRUB_BOOT_MACHINE_SIGNATURE

And that's the whole boot block implementation. This file is compiled normally into first an ELF image using GCC. Then we strip out all the symbols and other junk, and finally use objcopy to output an A.OUT object. It is exactly 512 bytes in size and the bootblock installer verifies this.

The second stage header block code now can take advantage of all of the values the first stage has computed, such as the stdout handle, the handle for the boot device, etc. In the second stage image, it begins with assembler just like the first stage does above. But, it implements a block + length tuple list which grows down from the end of the block. This tells us where to read in the rest of the GRUB kernel from the actual file on the disk.

The GRUB installer program links in with modules that understand how various filesystems work. It has a block traverser that can be run on arbitrary files. This is the mechanism used to build the block list which is patched into this second stage loader block list.

I've tested both of these using a Sun LDOM guest, which is the fastest way to test low-level stuff like this since resetting (which you need to do every test boot attempt) is nearly instantaneous.

The current task is to flesh out the installer programs and make sure the rest of the GRUB ieee1275 code is going to work properly on sparc. I already know some bits that will need some tweaking. For example, partitions on OF paths are indicated (aparently) by adding ":N" where N is a partition number. On sparc this "N" is instead a letter.

More to come...

Harald Welte: Bought another motorbike: Yamaha FZ6 Fazer

2008-05-20T02:00:00+00:00

During the last week or so, I spent a lot of time test riding a number of various motorbikes. Both real sports / supersports bikes, as well as 'sportive touring bikes'. I wasn't really sure if I should go for a true/real sports bike like the Suzuki GSX-R (750/1000) or start with something less 'extreme' first. One thing I learned, though, is if I went for a sports/supersports bike, I'd definitely have to keep my BMW F650ST around. Those racing bikes are just not useful for casual riding in city traffic. But I want both, fun at the motorway, as well as a useful bike for local travel inside Berlin.

Then I got a really irresistible offer for a two-year-old FZ6 Fazer (with ABS), and I had to buy it. So for now, it is this. It's probably reasonable to first go from the familiar 48bhp to 98bhp before reaching to the 160bhp range of the Suzuki GSX-R. So in the end, I can even claim that I'm being rational and reasonable here, going "only" to an (already-ridiculous) amount of power, than a beyond-ridiculous amount ;)

And please don't worry too much. I'm not suicidal, and I've been riding quite safely for more than 11 years now ;) This is not going to change!

Harald Welte: Chaosradio on Software Defined Radio

2008-05-17T22:32:00+00:00

I've had the pleasure of being invited to Chaosradio Express maker Tim Pritlove to talk about Software Defined Radio in general, and gnuradio plus USRP specifically. You can listen to the resulting 2+ hours of podcast (in German).

It's been a great experience, and I have a good feeling that it was possible for us to explain this fairly detailed subject to our already at least moderately technical audience.

SDR is really hard since it combines aspects of traditional radio, i.e. physics of electric waves, electrical engineering both analog and digital, digital signal processing and software. The biggest part is really advanced mathematics, and at least from all the subjects that I've seen, it's probably the most direct and close-to-theory incarnation of applied math.

Luckily, a fairly high-level understanding of the algorithms and principles involved are already sufficient to do a lot, since most of the deep-down mathematical details of many algorithms have already been implemented as building blocks for gnuradio. Still, I assume the number of developers who are actually able to use gnuradio is far too low. If you're looking for an interesting field of software right now, I suggest going for digital signal processing. It's in every area of communications, ranging from analog modems over ISDN, DSL, WiFi, USB2, Bluetooth, GSM, UMTS, DECT, ZigBee, Ethernet, VoIP and probably any other communication technology that we use today.

Rusty Russell: Tuning VirtIO and virtio_net: part I

2008-05-16T11:02:00+00:00

One premise of virtio is that we should be as fast as reasonably possible. While there's nothing which should make us slow, that's not the same as actually being fast. So this week, I've been doing some simple benchmarks on my patch queue, which includes major changes to accelerate the tap device and allow async packet sends.

I've been using lguest rather than kvm because it's far more hackable, and my test has been a 1GB (1024x1024x1024 byte) TCP send using netcat. And host->guest results were awful: instead of the current 12 seconds it was taking 70 seconds to receive 1GB. So I started breaking that down.

The first things that I found was that simply allocating large receive buffers (of which only 1500 bytes is used) is expensive. Just this change alone takes the time from 12 seconds to 29, and there are two reasons for this so far.

The first is because each 1500 byte packet takes two descriptors (we have a header containing metadata), whereas a fully populated paged skb takes 2 + 65536/PAGE_SIZE + 2 == 20 descriptors. That means we only fit 6 large packets in lguest's 128-descriptor ring, vs 64 for the small packet case. Increasing lguest's rings to 1024 drops the time from 29 to 25: not as much as you'd expect. Increasing it further has marginal effect (logically, we should see equivalence at 1280 descriptors, but it has to be a power of 2).

The second reason is that alloc_page is quite slow. A simple cache of allocated pages drops the time from 25 to 19 seconds.

But we're still 50% slower than allocating 1500-byte receive buffers, and today's task is to figure out why. It seems unlikely that the increased overhead of skb_to_sgvec, get_buf and add_buf would account for it. Cache effects also seem unlikely: 1024 descriptors are still only 8k. It's unfortunate that oprofile doesn't work inside lguest guests, so this will be old school.

If the overhead really is inherent in large descriptors, we have several options. The obvious one is to add a separate "large buffer" queue, or allow mixing buffer sizes and expect the other end to try to forage for the minimal sized one. Both require a change to the server side. We can add a feature bit for backwards-compat, but that's always a last resort. Another option is to try for multi-page allocations for our skbs: as they're physically contiguous they'll use fewer descriptors.

Patrick McHardy: Netfilter move to git almost completed

2008-05-16T00:00:00+00:00

I've completed the move of (most of) the netfilter repositories to git today. I still need to change the email notification script to make the commit emails more readable. They don't look very nice by default and I made it even worse. For today my limit of the amount of shell scripts I can look at is reached though.

These were the last SVN repositories I was using. I'm tempted to leave a long rant about SVN, but its probably better to simply forget about it as quickly as possible :)

Next I'll try if I can manage to roll a release candidate for iptables. We're currently releasing too infrequently. Since we're usually merging at least one new extension or revision per kernel release, there also should be one iptables release per kernel version, so users can actually use new things. The ideal time for this would be shortly before kernel releases, since that allows us to merge userspace extensions for things targeted at the next kernel release early enough so they can be used for testing. So thats what I'll try to do in the future. Luckily we didn't merge anything requiring new userspace extensions during the last merge window, so we won't need a new release for 2.6.26.

Harald Welte: Motorbike troubles again

2008-05-15T02:00:00+00:00

It seems like I lost all my luck. Only a three weeks ago, the Yamaha TW-225 in Taipei had problems after my arrival. Now that I'm back to Berlin, my BMW F-650 had some serious trouble, too.

Starting the engine turned out to be really hard (started only on something like the 10th attempt, even though usually the first one is sufficient). Furthermore, pulling the gas handle only the tiniest little bit kills off the engine completely, independent of how far the choke is asserted.

So today I spent some five hours in disassembling almost the entire bike, removing the twin-carburetor, disassembling and cleaning it and putting the entire bike back together again. The engine is running fine again. I just wonder why I have this kind of carburetor problem already the second time in the last couple of years.

There's almost no visible dirt inside the carburetor, and all the fittings are fine, no signs of any leakage, no signs of any significant wear of any of the involved parts. Still, cleaning and re-assembling it clearly removes the problem.

Eric Leblond: Netfilter Workshop 2008 in Paris

Administrator — 2008-05-14T12:16:20+00:00

INL is proud to announce that we are the official organisers of the Netfilter Workshop issue this year.
The Netfilter workshop will take place in Paris, from Sep 29th to Oct 3rd 2008.
The workshop is planned to be structured in two parts :

One User day, which will be a public event. Every Netfilter user is invited to contribute a paper to speak about their use of Netfilter. Original and advanced uses of Netfilter will be appreciated.
Four Developer days, including the Netfilter core team, and some invited guests.

The Netfilter Workshop website 2008 is available.

Patrick McHardy: Illustrations

2008-05-14T11:00:06+00:00

Apparently my blog is too boring to read, so Elena kindly offered to illustrate it. Since this spares me from writing some actual content, I gladly accepted.

What you're seing below is me resting in a deck chair, enjoying a Rothaus beer, exhaling some unidentified fumes and apparently being haunted by thoughts of ip_route_me_harder() :)

Harald Welte: Back from WGT

2008-05-14T02:00:00+00:00

There are two fixed dates every year that I never miss: The annual Chaos Communication Congress in Berlin between Christmas and new years eve, and the Wave Gotik Treffen music festival in Leipzig.

This year I was camping at the event campsite again, following two lazy years in a hotel. I enjoyed it a lot, especially since the weather was perfect. Only sunshine, not a single drop of rain for the entire four days.

The festival itself was like always. Great. :) I think my personal favorites this year was the industrial (probably better: rhythmic noise) act NULLVEKTOR as well as INADE.

David Miller: gdb bugs...

2008-05-12T08:11:00+00:00

So I spent the weekend fishing out some gdb bugs on sparc. Every time I think I understand and know how all of this stuff works I get thrown some new surprises. This time was no exception.

The kernel has all of these neat features, via ptrace()'s PTRACE_SETOPTIONS, that allows a debugger to get notifications when a process forks, vforks, does an exec, etc.

When these events occur in the inferior process, it does a ptrace_notify() which sends a SIGTRAP to the process with the event encoded into the siginfo exit code.

As a result, when the process tries to return back into userspace, it'll do signal processing. As part of that, it will invoke ptrace_stop() which sleeps the task and wakes up the debugger parent so it can examine the event and process state.

The debugger has several choices of what to do. For many cases it will do a ptrace() PTRACE_CONT with a code indicating how the process should continue. Another thing the debugger can do is decide that it's no longer interested in tracing this task, and therefore it does a ptrace() PTRACE_DETACH. This is part of the first bug.

A long time ago we picked the values for PTRACE_this and PTRACE_that on sparc. Some of them mirrored the SunOS values. One of those was PTRACE_DETACH. We unconditionally recognized the SunOS PTRACE_DETACH value, even for Linux processes. Unfortunately, this is the value that also ended up in the sys/ptrace.h Linux header file. So that's the value every Linux application ended up using too.

I yanked out the SunOS PTRACE_foo call support long ago, and it's amazing how much works without a properly functioning PTRACE_DETACH. Putting in a compat translation from the SunOS to the intended Linux value for PTRACE_DETACH in the Sparc ptrace code solved this bug.

Which brings us to the next bug...

What brought me down this path in the first place was examining why running gdb under itself didn't work properly. This kind of game is always fun:

bash$ ./gdb ./gdb
(top-gdb) set args test
(top-gdb) run
(gdb) run
Hello World!
...

This would hang when the inner-gdb tries to run the test program, and I had to figure out why. After lots of tracing I found that the inner-gdb was hanging in a sigpause() call.

GDB uses sigpause() to wait for SIGCHLD events when it is simply waiting for running inferiors to ptrace_stop() or take some other kind of signal.

In comes the issue of system call restart. Handling this wrt. debuggers is not easy. One nice feature of debuggers like GDB is that you can ask them at any point in time to call some function in the program they are running.

(gdb) p printf("hi\n")
hi
$1 = 2
(gdb)

and after calling the function it completely restores the process state to what it was before the call. How does it implement this?

First it saves the process state, mostly this comprises the registers. Next it allocates some stack space for the call, pushes the arguments for the function call onto the stack and in registers. Next, it makes the function return address point to a breakpoint it can uniquely recognize. Finally, it sets the program counter to point at the function to call.

When the function returns it hits the breakpoint, and the debugger restores all of the saved state it stored away before the special call.

Now, back to system call restart. When a signal interrupts a system call, it can return immediately to process the signal. Internally to the kernel system calls return special error codes to let the signal dispatch know what to do with the system call that was in progress. It can say that -EINTR should be returned and the system call completed. But it can also say that the program counter should be rewound to the system call trap, the argument registers re-setup with their original values, and the system call thus replayed.

In the example above, imagine what happens if the debugger calls an inferior function at the time that the signal is dispatched. Somehow, in order to restore the state properly after the call, this "we're in a syscall and should do syscall restarting" state has to get saved and restored too.

Long ago I had this clever idea wherein I tried to solve this problem entirely inside of the kernel to shield debuggers from having to deal with it. The idea was that we'd modify the process state and perform the system call restart operations before we stopped the program to let the debugger see the state.

Although great in theory, in practice it's an unworkable solution. We don't know what the debugger is going to do with the process. As I stated earlier it can pass along the signal to the process, or it can cancel the signal delivery altogether. This decision influences whether we should do system call restart or not, but we already pre-commited that state and let the debugger see it already. We can't know what the debugger is going to do ahead of time, therefore it is impossible to do the right thing.

This is what was causing the inner debugger to hang. The inner gdb is receiving a SIGCHLD because the 'test' program it is debugging has hit ptrace_stop(). The top-level gdb looks at this and says "ok, let's just let the inner gdb see the signal, PTRACE_CONT." But my funny in-kernel ptrace syscall restart code already setup for a syscall restart of sigpause(), but what should have happened was a return of -EINTR.

The inner gdb is now wedged forever, it missed the SIGCHLD and the debugged process is sleeping waiting to be woke up by the inner gdb.

So I ripped out all of this silly code, and ended up doing what powerpc, x86, and other platforms do. I added a piece of software binary state (an unused bit in one of the processor state registers), that gdb can control. It is the "we're in a system call" state. When the debugger does an inferior call, it clears this bit when changing the program counter register. This forces the kernel to not do system call restart processing when the task wakes out of ptrace_stop().

But later, when gdb restores all of the register state after the call, that special bit will be restored, and we'll do the right thing as we deliver the original signal and subsequently do syscall restart processing as needed.

It's always fun to find land mines like these ones.

Harald Welte: Victory: Skype withdraws appeals case, judgement from lower court accepted

2008-05-08T02:00:00+00:00

The court hearing in the "Welte vs. Skype Technologies SA" case went pretty well. Initially the court again suggested that the two parties might reach some form of amicable agreement. We indicated that this has been discussed before and we're not interested in settling for anything less than full GPL compliance.

The various arguments by Skype supporting their claim that the GPL is violating German anti-trust legislation as well as further claims aiming at the GPL being invalid or incompatible with German legislation were not further analyzed by the court. The court stated that there was not enough arguments and material brought forward by Skype to support such a claim. And even if there was some truth to that, then Skype would not be able to still claim usage rights under that very same license.

The lawyer representing Skype still continued to argue for a bit into that direction, which resulted one of the judges making up an interesting analogy of something like: "If a publisher wants to publish a book of an author that wants his book only to be published in a green envelope, then that might seem odd to you, but still you will have to do it as long as you want to publish the book and have no other agreement in place".

In the end, the court hinted twice that if it was to judge about the case, Skype would not have very high chances. After a short break, Skype decided to revoke their appeals case and accept the previous judgement of the lower court (Landgericht Muenchen I, the decision was in my favor) as the final judgement. This means that the previous court decision is legally binding to Skype, and we have successfully won what has probably been the most lengthy and time consuming case so far.

Harald Welte: Back from the trip to Taiwan

2008-05-07T02:00:00+00:00

It's been some time since my last blog post, mainly because I've been quite busy in Taiwan. First there was the conference, then there were a number of meetings with various companies to educate them about GPL licensing and how to interoperate with the FOSS community for better hardware/driver support.

The other part was actual spare time. I spent many months in Taipei during my work for OpenMoko, but I never really had much time to explore the city, or even other parts of the country.

This time I explored quite a bit of the Taipei nightlife, visiting places like Luxy, Lava, Room18, Barcode, ageha, and even the so-called "meat market" of Carnegies and Tavern.

I've also had time to try one of the many hot spa's of Taipei in Beitou, as well as a really great motorbike trip to the national forest in the Wulai mountain region.

Unfortunately the weather wasn't that great, so I had to postpone my plans to visit the northeastern and the eastern coast to some future trip.

And the most interesting part is: I actually made contact to Taiwanese people who are not at all in any way related to work :)

Further Taipei exploration brought me to the Wufenpu fashion wholesale area, as well as Ximending. Most impressive is also the "Taipei underworld", i.e. the various underground shopping malls near Taipei Main Station, such as the Taipei City Mall, Station Front Mall and ZhongShen Mall I and II. You can literally walk for many kilometers underground...

Now I am one day in Frankfurt, and tomorrow one day in Munich, Friday one half day at home, and then there will be four days of music festival at WGT 2008.

Harald Welte: Tomorrow: Court hearing in Welte vs. Skype GPL case

2008-05-07T02:00:00+00:00

Tomorrow at 10:30am at the Oberlandesgericht Muenchen (higher regional court of Munich) there will be an oral hearing in the "Welte vs. Skype Technologies SA" case. The hearing is to be held in room E.06.

This case is about a GPL violation of Skype, related to their sales of Wifi Skype phones based on the Linux operating system kernel.

I'm fighting as part of the gpl-violations.org project in enforcing the GPL against Skype since February 2007. Initially Skype didn't respond, we then applied for a preliminary injunction. That injunction was granted by the court in June 2007, but Skype chose to file an appeals case against it.

The court hearing tomorrow is exactly to debate about this appeal.

Interestingly, Skype is arguing against the validity of the GPL as a whole, asserting that it is violating anti-trust regulation and similarly strange claims.

Patrick McHardy: Summer office

2008-05-07T00:00:00+00:00

The weather has been great the past days, so I set up my summer workplace :)

Working outside is really pleasant after a month of almost constantly grey sky. Below the balcony there's a small stream, and hundreds of birds sit in the trees and sing, which makes an amazing scenery.

I sent out a first batch of HIFN fixes today to avoid causing too much conflicts in the series in case something turns up during review. Caught a good time during which both Evgeniy and Herbert were responsive and it only took about an hour to get all patches reviewed, fix a minor bug and get them merged. The remaining ones are hopefully in shape by tommorrow, the descriptor accounting still needs a bit more work. Herbert also merged some patches from Loc Ho today for async hashing support, which is cool because I already started adding hashing support to the HIFN driver until I noticed the CrytoAPI doesn't support it asynchronously yet :)

Also sent out a few netfilter patches and fixed a slightly embarrasing bug in the macvlan driver. It would crash the kernel on module unload because cleanup was performed incorrectly, causing the kernel to jump to a NULL function pointer when receiving the next packet on the underlying device. I wonder why I've never noticed this.

Patrick McHardy: Fighting the HIFN driver

2008-05-06T11:00:03+00:00

What I hoped initially to be just a simple fix for a few arithmetic errors in the driver for the HIFN 795x crypto accelerator cards turned into a week long struggle, accompanied by at least a hundred crashes and reboots.

The initial bug manifested itself by going into an endless loop when the CryptoAPI issued a request for less data than the full scatterlist, caused by an integer underflow while calculating the remaining amount of data to be processed. The fix was straight-forward: only use the minimum of the scatterlist size and the crypto request size. While at it, I also fixed some endian bugs, missing error propagation for errors that shouldn't happen, but did because of the underflow, and some overly strict data alignment checks.

Testing looked good, no more crashes, but surprisingly the testcases of the tcrypt module using algorithms provided by HIFN randomly failed. This turned out to be caused by an incorrect return value indicating synchronous processing to the CryptoAPI, while the request was in fact processed asynchronously. So when the result was not already available when returning from the driver, testcases failed.

After fixing the tcrypt failures, next was some real-life testing using IPsec. The first attempt resulted in an immediate crash in crypto_authenc_genivc(). This one was fixed fairly quickly, the asynchronous completion handler interpreted a pointer as an incorrect structure.

The second attempt looked more promising, no crashes, packets went through and looked like IPsec. The remote side failed to parse them however, closer looking revealed that they were incorrectly constructed and had 16 bytes of garbage at the end. From my last attempt to fix the driver I remembered that this was most likely caused by missing initialization vector size initialisation of the CBC modes. Naively, I changed the driver to properly initialize the ivsize. To my surprise, attempting to add SAs using cbc(aes) now failed with -ENOENT.

Figuring out the reason took me almost an entire day. When the ivsize is already initialized, the CryptoAPI attempts to spawn a new instance of the algorithm. Algorithms are identified by name, possibly combined with modes, like cbc(aes). When spawning new algorithms, the driver name is used for the lookup however, which in the case of HIFN was "hifn-aes" for all AES modes, causing the lookup to return the ofb(aes) algorithm instead of cbc(aes). Using unique driver names for the different algorithm modes fixed this problem.

While chasing this bug, I noticed some DMA memory corruption issues in the HIFN driver. When a request contains more than a single scatterlist element, the driver programmed the hardware to perform one crypt operation per scatterlist element, but for the full request size, corrupting the memory after its tail. The fix for this was a bit more involved since using the correct length also requires to perform only a single operation for all scatterlist elements since the source and destination descriptors don't necessarily have identical lengths. This complicates keeping track of free descriptor entries. Previously, each operation needed exactly one command, source, destination and result descriptor. With only a single operation, it needs one command and result descriptor and a varying amount of source and destination descriptors. On the upside, this reduces the number of interrupts per request to exactly one instead of one per scatterlist element and gets rid of some atomic operations. Additionally tcrypt can now detect destination buffer corruption for cipher tests.

Continuing testing with IPsec, things now looked better, packets were properly sized and the receive side worked properly. Outgoing packets were still dropped by the receiver however. Looking more closely at the packets showed that they contained what looked like a block of unencrypted data at the end. Additionally there still were some rare random crashes in the CryptoAPI. The crashes were caused by a missing check for end-of-scatterlist in one of the CryptoAPI scatterlist helpers, the unencrypted block of data by an off-by-one in the eseqiv sequence number generator. Both problems were fixed by Herbert Xu. The first victory - IPsec now worked properly using ping. TCP connections stalled after a short period however.

Half a day later, I also figured out the reason for the stalls. The HIFN driver needs to keep some context for each request since it processes them asynchronously. The driver used the global per-transform storage for this context instead of the per-request storage, corrupting existing contexts when more than one request was outstanding. Even in flood mode, ping exhibits ping-pong behaviour, waiting for a reply before sending the next request, which is why it wasn't affected by this problem. With this also fixed, IPsec seemed to be working properly, at least on the HIFN side. There still appears to be some corruption of the XFRM CB with asynchronous processing, causing outgoing tunnel mode packets to be sent without IP_DF, but that should be easily fixed.

Next was testing with dm-crypt, for which I actually purchased the card. Testing worked fine while debugging was enabled, without debugging it reproduceably crashed in the device mapper code. This was fairly nasty to debug since enabling debugging stopped the bug from happening. After following lots of dead ends and some suggestions from Evgeniy, I found the cause: when no descriptors are currently available, the request is queued and processed once enough descriptors are available again. The queue length is limited (in the case of HIFN to 1), when the limit is reached the behaviour depends on the flags specified by the caller. When using CRYPTO_TFM_REQ_MAY_SLEEP, the caller goes to sleep and waits for notification from the driver when its ready to accept more requests. When dequeuing the crypto queue, asynchronous crypto drivers need to check for backlogged clients and wake them before continuing processing. This was missing from the HIFN driver, causing it to call the dm-crypt completion handler for a request that wasn't fully initialized.

With this bug also fixed, dm-crypt survived a 24 hour stress test. I'm a bit reluctant at this point to use it for real data though, all those bugs didn't exactly instill confidence. The patches are in an almost upstream-submittable state, just the descriptor accounting needs some minor cleanup. I hope to get this done today or tommorrow and then attend to the huge backlog in my inbox that has grown over the past week.

On the netfilter front, nothing too exciting has happened during the last two weeks. 2.6.25 appears to have gone pretty well, netfilter-wise, except for one nasty hashing regression on ARM, fixed by Philip Craig. The amount of patches merged during the 2.6.26 merge window was smaller than usual, the highlights are:

A large amount of SIP helper fixes and improvements
DCCP conntrack/NAT
UDP-Lite NAT
SCTP NAT
Completion of network namespace support for {ip,ip6,arp}_tables

I'm particulary happy about finally managing to merge the SIP helper patches, which I had queued for almost 9 month. If you've tried using it and it didn't work, now is a good time to try again and submit bug reports :)

Patrick McHardy: Overcoming laziness

2008-05-06T11:00:03+00:00

I decided to give blogging another try. My last attempt failed after just one or two entries because of me being too lazy to actually write something, but since I enjoy reading other people's blogs, I hope I can keep the motivation up a bit longer this time :)

Jeremy Kerr: linux.conf.au hackfest: the solution, part two

2008-05-05T11:57:57+00:00

In the last article we finished with a SPE-based fractal renderer, but with a limited maximum fractal size of 64 × 64 pixels:

We'd like to generate full-size fractals, but the DMAs (which we use to transfer the fractal image out of the SPE) have a maximum size of 64kB. The solution is to perform multiple DMAs each containing a subset of the image's rows.

Each invocation of render_fractal() should render a DMA-able chunk of fractal data, then we perform the DMA. We do this until the SPE has processed the entire image:

We just need to modify the spe-fractal code (spe-fractal.c) a little. At present, we just render the whole fractal in one pass and DMA the data in the main() function:

        render_fractal(&args.fractal);

        mfc_put(args.fractal.imgbuf, ppe_buf,
                args.fractal.rows * args.fractal.cols * sizeof(struct pixel),
                0, 0, 0);

        /* Wait for the DMA to complete */
        mfc_write_tag_mask(1 << 0);
        mfc_read_tag_status_all();

First, we need to modify our render_fractal() fuction to take a starting row, and a number of rows to render. This is the new prototype of render_fractal():

static void render_fractal(struct fractal_params *params,
                int start_row, int n_rows)

In the SPE program's main(), we just need to set up some convenience variables:

        bytes_per_row = sizeof(*buf) * args.fractal.cols;
        rows_per_dma = sizeof(buf) / bytes_per_row;

And do the rendering and DMAs in a loop:

        for (row = 0; row < args.fractal.rows; row += rows_per_dma) {

                render_fractal(&args.fractal, row, rows_per_dma);

                mfc_put(buf, ppe_buf + row * bytes_per_dma,
                                rows_per_dma * bytes_per_row,
                                0, 0, 0);

                /* Wait for the DMA to complete */
                mfc_write_tag_mask(1 << 0);
                mfc_read_tag_status_all();
        }

This loop will render as many image rows as will fit into a single DMA, then DMA the rendered data back to main memory.

Now, we're able to render fractals larger than 64 × 64 pixels:

The source for the updated fractal renderer is available in fractal.2.tar.gz.

performance

Now that we can generate full-size fractals, we can compare the running times with the PPE-based fractal renderer. The following table shows running times with a standard fractal (using these fractal parameters).

Running times of fractal renderer
Implementation	Time (sec)
PPE	55.7
1 SPE	40.7

So, we get a 27% speedup by moving the fractal generation code to run on a SPE. We're still a way behind the optimal performance though, and benchmarking on other systems gives better times (for example, generating the same fractal on an Intel Core 2 Duo @ 2.4GHz takes 13.8 seconds).

We can improve the Cell performance by a large amount - stay tuned for the next article to see how.

David Miller: Effective GIT bisecting...

2008-04-30T06:08:00+00:00

I've had to do a lot of this lately, and the most efficient attempts take on a certain pattern.

So you have a bug, and you can readily reproduce it. Also, the bug appeared in the last pull you made from Linus's tree. Perfect.

At this point you know that 'master' has the bug and that 'ORIG_HEAD' lacks the bug. You could just blindly bisect the whole thing, but you can save yourself some time (and also learn a bit about the nature of the bug) by using some clues and some quick tests to narrow things down a lot from the beginning.

This determination can be easy. Your goal is to first find a spot which you think works. You'd like it to be something a bit further than ORIG_HEAD, as you're trying to narrow things down.

The easy case is some driver breaks or similar, or you see some error message and it's clear what subsystem that came from. Take that information and use it to scan over the changesets you got from your pull:

	gitk ORIG_HEAD..

Note that when you select a changeset in gitk, the SHA ID of that commit becomes the current X selection. You can use that to do things more quickly below.

So you're found a sequence of commits that look suspicious. Pick the changeset before the first commit in the suspicious set, and check out a test tree with it as the tip into a test branch:

	git checkout -b test $(SHA_ID)

Build that kernel, and make sure the bug doesn't happen. Let's assume that this kernel passes your test. You have a few options on how to proceed.

The easiest thing to do is to just bisect using the information you now have:

	git bisect start
	git bisect bad master
	git bisect good test

and so on. Build, test boot, and if it shows the bug:

	git bisect bad

else if it succeeds:

	git bisect good

and repeat the process until GIT shows you the guilty commit.

The other option is to try and figure out an approximate more optimal end point for bisection. Take the set of "suspicious" changeset your determined above, and take the one after the last and go:

	git checkout -b test2 $(SHA_ID)

If this shows the bug, you're in business:

	git bisect start
	git bisect bad test2
	git bisect good test

and continue as detailed above.

When you're done with all of this:

	git bisect reset

and report your results to the mailing list.

Harald Welte: First ASUS day of OpenTechSummit Taipei

2008-04-26T02:00:00+00:00

As I might have indicated before, I have the pleasure of being invited to the OpenTechSummit 2008 in Taiwan. Two days ago, I was at the opening dinner. The problem of that dinner was the lack of attendees. There were loads of delicious (free, sponsored) food, but not even remotely enough people to eat it.

Today I had a bit of a problem finding the ASUS venue, since it was said to be at "exit 2" of the MRT station. Unfortunately it had two exits of that name, one on each side of the station :)

One presentation there I found particularly embarrassing was the one about the eePC SDK. First of all, I will ignore my thoughts about why you actually need such an SDK if it really is nothing more than a customized Debian Linux with Eclipse. But even then, why fly in a foreing speaker to do a click-by-click walk-thhrough on how to create a 'hell world' Qt program using eclipse?

My favourite of the day was definitely the presentation on the OpenPattern router board.

Harald Welte: Back to Taipei

2008-04-24T02:00:00+00:00

After a break of almost six months, I'm back to Taipei. Obviously I now see everything from a quite different angle: I no longer work for OpenMoko, Inc., thus I actually have spare time here and can explore both the capital city as well as the country much better than before with that ever-growing OpenMoko workload.

However, the first day wasn't quite as relaxing as it should have been. First, the apartment key that was supposed to be with the guard of the apartment building accidentally was mixed up with some other key and got sent to the landlord.

A couple of hours later I discover that my Yamaha TW225 motorbike doesn't work anymore. First diagnosis: Battery is empty (not surprisingly). I try for like 15minutes to kickstart it, to no avail. Not even a single explosion in the engine. Then I tried to push it, and got it to a couple of explosions after which it died again. Further push-starting was prevented by the way-too-smooth floor of the parking garage, where the wheel just slides as soon as you release the clutch :(

Some disassembly revealed where the battery is (I don't know this bike at all, much opposed to my F650ST in Germany). The battery was severely short of acid/fluid, maybe somebody pushed the bike over and it leaked. Obtaining battery additive and refilling results in only 800mA charge current. I think it's dead. Now I'm in the process of ordering a new battery.

Let's hope the next couple of days are better than the start of this trip...

Harald Welte: Review of DORS/CLUC 2008 in Zagreb, Croatia

2008-04-21T02:00:00+00:00

I've spent the last five days in beautiful Croatia - most of the time in its capital Zagreb. The local conference DORS/CLUC has been around for a couple of years, and in fact I've been at a previous incarnation three years ago.

It's a nice, small but great event. And in fact, for the invited speakers as myself it feels more like an all-inclusive holiday than a conference. The organizers went out of their way to make us feel at home, including a trip to the waterfalls of Plitvice national park (photos will be available shortly at my public photo album.

It was also great to spend some time with Alan Cox again, who to my surprise was also attending the event with two lectures. Hope his luggage didn't get lost again on his way home...

Harald Welte: Further studying of Abis protocols, moving towards implementation

2008-04-12T02:00:00+00:00

The first quarter of 2008 is already gone, and I still haven't found all the time that I wanted to find to play with my BS11 base station[s].

However, I've spent quite a bit of time over the last couple of days further studying the GSM/3GPP 08.5x documents, as well as a thorough read through the mISDN source code.

GSM/3GPP 08.5x describe the layer1, 2 and 3 protocols of the Abis link between BSC (Base Station Controller) and BTS (Base Transceiver Station) in a GSM network. It's modelled on top of a E1 link in PCM30C configuration, i.e. TS0 is for CRC4 and synchronization, TS16 is used for the layer2+layer3 protocols, whereas the other time slots are used for transfer of the actual voice call data.

After looking at the various different driver options on Linux, I have determined that mISDN is the most promising and flexible architecture available. mISDN also has a layer0 + layer1 driver for the NT mode of the HFC-E1 card that I'm using. mISDN is great in a way that every layer is strictly separated from the other layer, and that at any layer parts of the stack can be implemented in userspace using library API.

Thus, I've started to write some mISDNuser based code to attach to the kernel-side hardware and lower-layer drivers. I'm not yet sure if the Q.921 (ISDN Layer2, also called LAPD) of the mISDN kernel side can be reused for Abis or not. The differences between standard Q.921 used on European ISDN and the Abis Layer2 are fairly small, so I hope to get it working with the existing LAPD code.

Unfortunately, I have paid work to take care of, so I will once again be distracted from this most interesting of my toy projects.