netfilter project logo Planet Netfilter

September 13, 2016

Eric Leblond: Updated status in vim and bash

Powerline

Powerline is a status extension software changing the prompt or status line for shell, tmux and vim. The result is nice looking and useful for bash: Powerline bash prompt and for gvim: Powerline in gvim Only point is that even if documentation is good, installation is not straightforward. So here’s what I’ve done.

Installation on Debian

System
sudo aptitude install fonts-powerline powerline python-powerline

On Ubuntu 16.04 you may have to install python3-powerline instead of python-powerline.

Install configuration

mkdir ~/.config/powerline
cp /usr/share/powerline/config_files/config.json .config/powerline/
Then edit the file to change default theme to default_leftonly that bring git status:
--- /usr/share/powerline/config_files/config.json	2016-07-13 23:43:25.000000000 +0200
+++ .config/powerline/config.json	2016-09-14 00:05:04.368654864 +0200
@@ -18,7 +18,7 @@
 		},
 		"shell": {
 			"colorscheme": "default",
-			"theme": "default",
+			"theme": "default_leftonly",
 			"local_themes": {
 				"continuation": "continuation",
 				"select": "select"
Now, you need to refresh the fonts or restart X.
Bash
Then edit ~/.bashrc and add at then end
. /usr/share/powerline/bindings/bash/powerline.sh
Vim
Easiest way is to have vim addon installed:
sudo aptitude install vim-addon-manager
Then you can simply do:
vim-addons install powerline
Then add to your ~/.vimrc:
set laststatus=2
Vim status with powerline
gvim
Installation is a bit more complex as you need to install a patched font from Powerline modified fonts. In my case:
mkdir ~/.fonts
cd ~/.fonts
wget 'https://github.com/powerline/fonts/raw/master/DejaVuSansMono/DejaVu%20Sans%20Mono%20for%20Powerline.ttf'
fc-cache -vf ~/.fonts/
Then edit .vimrc:
set guifont=DejaVu\ Sans\ Mono\ for\ Powerline\ 10

August 16, 2016

Harald Welte: (East) European motorbike tour on 20y old BMW F650ST

For many years I've always been wanting to do some motrobike riding accross the Alps, but somehow never managed to do so. It seems when in Germany I've always been too busy - contrary to the many motorbike tours around and accross Taiwan which I did during my frequent holidays there.

This year I finally took the opportunity to combine visiting some friends in Hungary and Bavaria with a nice tour starting from Berlin over Prague and Brno (CZ), Bratislava (SK) to Tata and Budapeest (HU), further along lake Balaton (HU) towards Maribor (SI) and finally accross the Grossglockner High Alpine Road (AT) to Salzburg and Bavaria before heading back to Berlin.

It was eight fun (but sometimes long) days riding. For some strange turn of luck, not a single drop of rain was encountered during all that time, travelling accross six countries.

The most interesting parts of the tour were:

  • Along the Elbe river from Pirna (DE) to Lovosice (CZ). Beautiful scenery along the river valey, most parts of the road immediately on either side of the river. Quite touristy on the German side, much more pleaant and quiet on the Czech side.
  • From Mosonmagyarovar via Gyor to Tata (all HU). Very little traffic alongside road '1'. Beatutil scenery with lots of agriculture and forests left and right.
  • The Nothern coast of Lake Balaton, particularly from Tinany to Keszthely (HU). Way too many tourists and traffic for my taste, but still very impressive to realize how large/long that lake really is.
  • From Maribor to Dravograd (SI) alongside the Drau/Drav river valley.
  • Finally, of course, the Grossglockner High Alpine Road, which reminded me in many ways of the high mountain tours I did in Taiwan. Not a big surprise, given that both lead you up to about 2500 meters above sea level.

Finally, I have to say I've been very happy with the performancee of my 1996 model BMW F 650ST bike, who has coincidentially just celebrated its 20ieth anniversary. I know it's an odd bike design (650cc single-cylinder with two spark plugs, ignition coils and two carburetors) but consider it an acquired taste ;)

I've also published a map with a track log of the trip

In one month from now, I should be reporting from motorbike tours in Taiwan on the equally trusted small Yamaha TW-225 - which of course plays in a totally different league ;)

July 23, 2016

Harald Welte: python-inema: Python module implementing Deutsche Post 1C4A Internetmarke API

At sysmocom we maintain a webshop with various smaller items and accessories interesting to the Osmocom community as well as the wider community of people experimenting (aka 'playing') with cellular communications infrastructure. As this is primarily a service to the community and not our main business, I'm always interested in ways to reduce the amount of time our team has to use in order to operate the webshop.

In order to make the shipping process more efficient, I discovered that Deutsche Post is offering a Web API based on SOAP+WSDL which can be used to generate franking for the (registered) letters that we ship around the world with our products.

The most interesting part of this is that you can generate combined address + franking labels. As address labels need to be printed anyway, there is little impact on the shipping process beyond having to use this API to generate the right franking for the particular shipment.

Given the general usefulness of such an online franking process, I would have assumed that virtually anyone operating some kind of shop that regularly mails letters/products would use it and hence at least one of those users would have already written some free / open source software code fro it. To my big surprise, I could not find any FOSS implementation of this API.

If you know me, I'm the last person to know anything about web technology beyond HTML 4 which was the latest upcoming new thing when I last did anything web related ;)

Nevertheless, using the python-zeep module, it was fairly easy to interface the web service. The weirdest part is the custom signature algorithm that they use to generate some custom soap headers. I'm sure they have their reasons ;)

Today I hence present the python-inema project, a python module for accessing this Internetmarke API.

Please note while I'm fluent in Pascal, Perl, C and Erlang, programming in Python doesn't yet come natural to me. So if you have any comments/feedback/improvements, they're most welcome by e-mail, including any patches.

Harald Welte: Going to attend Electromagnetic Field 2016

Based on some encouragement from friends as well as my desire to find more time again to hang out at community events, I decided to attend Electromagnetic Field 2016 held in Guildford, UK from August 5th through 7th.

As I typically don't like just attending an event without contributing to it in some form, I submitted a couple of talks / workshops, all of which were accepted:

  • An overview talk about the Osmocom project
  • A Workshop on running your own cellular network using OpenBSC and related Osmocom software
  • A Workshop on tracing (U)SIM card communication using Osmocom SIMtrace

I believe the detailed schedule is still in the works, as I haven't yet been able to find any on the event website.

Looking forward to having a great time at EMF 2016. After attending Dutch and German hacker camps for almost 20 years, let's see how the Brits go about it!

Harald Welte: EC-GSM-IoT: Enhanced Coverage GSM for IoT

In private conversation, Holger mentioned EC-GSM-IoT to me, and I had to dig a bit into it. It was introduced in Release 13, but if you do a web search for it, you find surprisingly little information beyond press releases with absolutely zero information content and no "further reading".

The primary reason for this seems to be that the feature was called EC-EGPRS until the very late stages, when it was renamed for - believe it or not - marketing reasons.

So when searching for the right term, you actually find specification references and change requests in the 3GPP document archives.

I tried to get a very brief overview, and from what I could find, it is centered around GERAN extension in the following ways:

  • EC-EGPRS goal: Improve coverage by 20dB
    • New single-burst coding schemes
    • Blind Physical Layer Repetitions where bursts are repeated up to 28 times without feedback from remote end
      • transmitter maintains phase coherency
      • receiver uses processing gain (like incremental redundancy?)
    • New logical channel types (EC-BCCH, EC-PCH, EC-AGC, EC-RACH, ...)
    • New RLC/MAC layer messages for the EC-PDCH communication
  • Power Efficient Operation (PEO)
    • Introduction of eDRX (extended DRX) to allow for PCH listening intervals from minutes up to a hour
    • Relaxed Idle Mode: Important to camp on a cell, not best cell. Reduces neighbor cell monitoring requirements

In terms of required modifications to an existing GSM/EDGE implementation, there will be (at least):

  • changes to the PHY layer regarding new coding schemes, logical channels and burst scheduling / re-transmissions
  • changes to the RLC/MAC layer in the PCU to implement the new EC specific message types and procedures
  • changes to the BTS and BSC in terms of paging in eDRX

In case you're interested in more pointers on technical details, check out the links provided at https://osmocom.org/issues/1780

It remains to be seen how widely this will be adopted. Rolling this cange out on moderm base station hardware seems technicalyl simple - but it remains to be seen how many equipment makers implement it, and at what cost to the operators. But I think the key issue is whether or not the baseband chipset makers (Intel, Qualcomm, Mediatek, ...) will implement it anytime soon on the device side.

There are no plans on implementing any of this in the Osmocom stack as of now,but in case anyone was interested in working on this, feel free to contact us on the osmocom-net-gprs@lists.osmocom.org mailing list.

July 16, 2016

Harald Welte: Deeper ventures into Ericsson (Packet) Abis

Some topics keep coming back, even a number of years after first having worked on them. And then you start to search online using your favorite search engine - and find your old posts on that subject are the most comprehensive publicly available information on the subject ;)

Back in 2011, I was working on some very basic support for Ericsson RBS2xxx GSM BTSs in OpenBSC. The major part of this was to find out the weird dynamic detection of the signalling timeslot, as well as the fully non-standard OM2000 protocol for OML. Once it reached the state of a 'proof-of-concept', work at this ceased and remained in a state where still lots of manual steps were involved in BTS bring-up.

I've recently picked this topic up again, resulting in some work-in-progress code in http://git.osmocom.org/openbsc/log/?h=laforge/om2000-fsm

Beyond classic E1 based A-bis support, I've also been looking (again) at Ericsson Packet Abis. Packet Abis is their understanding of Abis over IP. However, it is - again - much further from the 3GPP specifications than what we're used to in the Osmocom universe. Abis/IP as we know consists of:

  • RSL and OML over TCP (inside an IPA multiplex)
  • RTP streams for the user plane (voice)
  • Gb over IP (NS over UDP/IP), as te PCU is in the BTS.

In the Ericsson world, they decided to taka a much lower-layer approach and decided to

  • start with L2TP over IP (not the L2TP over UDP that many people know from VPNs)
  • use the IETF-standardized Pseudowire type for HDLC but use a frame format in violation of the IETF RFCs
  • Talk LAPD over L2TP for RSL and OML
  • Invent a new frame format for voice codec frames called TFP and feed that over L2TP
  • Invent a new frame format for the PCU-CCU communication called P-GSL and feed that over L2TP

I'm not yet sure if we want to fully support that protocol stack from OpenBSC and related projects, but in any case I've extende wireshark to decode such protocol traces properly by

  • Extending the L2TP dissector with Ericsson specific AVPs
  • Improving my earlier pakcet-ehdlc.c with better understanding of the protocol
  • Implementing a new TFP dissector from scratch
  • Implementing a new P-GSL dissector from scratch

The resulting work can be found at http://git.osmocom.org/wireshark/log/?h=laforge/ericsson-packet-abis in case anyone is interested. I've mostly been working with protocol traces from RBS2409 so far, and they are decoded quite nicely for RSL, OML, Voice and Packet data. As far as I know, the format of the STN / SIU of other BTS models is identical.

Is anyone out there in possession of Ericsson RBS2xxx RBSs interested in collboration on either a Packet Abis implementation, or an inteface of the E1 or packet based CCU-PCU interface to OsmoPCU?

June 15, 2016

Rusty Russell: Minor update on transaction fees: users still don’t care.

I ran some quick numbers on the last retargeting period (blocks 415296 through 416346 inclusive) which is roughly a week’s worth.

Blocks were full: median 998k mean 818k (some miners blind mining on top of unknown blocks). Yet of the 1,618,170 non-coinbase transactions, 48% were still paying dumb, round fees (like 5000 satoshis). Another 5% were paying dumbround-numbered per-byte fees (like 80 satoshi per byte).

The mean fee was 24051 satoshi (~16c), the mean fee rate 60 satoshi per byte. But if we look at the amount you needed to pay to get into a block (using the second cheapest tx which got in), the mean was 16.81 satoshis per byte, or about 5c.

tl;dr: It’s like a tollbridge charging vehicles 7c per ton, but half the drivers are just throwing a quarter as they drive past and hoping it’s enough. It really shows fees aren’t high enough to notice, and transactions don’t get stuck often enough to notice. That’s surprising; at what level will they notice? What wallets or services are they using?

June 06, 2016

Harald Welte: Recent public allegations against Jacob Appelbaum

In recent days, various public allegations have been brought forward against Jacob Appelbaum. The allegations rank from plagiarism to sexual assault and rape.

I find it deeply disturbing that the alleged victims are putting up the effort of a quite slick online campaign to defame Jakes's name, using a domain name consisting of only his name and virtually any picture you can find online of him from the last decade, and - to a large extent - hide in anonymity.

I'm upset about this not because I happen to know Jake personally for many years, but because I think it is fundamentally wrong to bring up those accusations in such a form.

I have no clue what is the truth or what is not the truth. Nor does anyone else who has not experienced or witnessed the alleged events first hand. I'd hope more people would think about that before commenting on this topic one way or another on Twitter, in their blogs, on mailing lists, etc. It doesn't matter what we believe, hypothesize or project based on a personal like or dislike of either the person accused or of the accusers.

We don't live in the middle ages, and we have given up on the pillory for a long time (and the pillory was used after a judgement, not before). If there was illegal/criminal behavior, then our societies have a well-established and respected procedure to deal with such: It is based on laws, legal procedure and courts.

So if somebody has a claim, they can and should seek legal support and bring those claims forward to the competent authorities, rather than starting what very easily looks like a smear campaign (whether it is one or not).

Please don't get me wrong: I have the deepest respect and sympathies for victims of sexual assault or abuse - but I also have a deep respect for the legal foundation our societies have built over hundreds of years, and it's principles including the human right "presumption of innocence".

No matter who has committed which type of crime, everyone deserve to receive a fair trial, and they are innocent until proven guilty.

I believe nobody deserves such a public defamation campaign, nor does anyone have the authority to sentence such a verdict, not even a court of law. The Pillory was abandoned for good reasons.

June 01, 2016

Harald Welte: Nuand abusing the term "Open Source" for non-free Software

Back in late April, the well-known high-quality SDR hardware company Nuand published a blog post about an Open Source Release of a VHDL ADS-B receiver.

I was quite happy at that time about this, and bookmarked it for further investigation at some later point.

Today I actually looked at the source code, and more by coincidence noticed that the LICENSE file contains a license that is anything but Open Source: The license is a "free for evaluation only" license, and it is only valid if you run the code on an actual Nuand board.

Both of the above are clearly not compatible with any of the well-known and respected definitions of Open Source, particularly not the official Open Source Definition of the Open Source Initiative.

I cannot even start how much this makes me upset. This is once again openwashing, where something that clearly is not Free or Open Source Software is labelled and marketed as such.

I don't mind if an author chooses to license his work under a proprietary license. It is his choice to do so under the law, and it generally makes such software utterly unattractive to me. If others still want to use it, it is their decision. However, if somebody produces or releases non-free or proprietary software, then they should make that very clear and not mis-represent it as something that it clearly isn't!

Open-washing only confuses everyone, and it tries to market the respective company or product in a light that it doesn't deserve. I believe the proper English proverb is to adorn oneself with borrowed plumes.

I strongly believe the community must stand up against such practise and clearly voice that this is not something generally acceptable or tolerated within the Free and Open Source software world. It's sad that this is happening more frequently, like recently with OpenAirInterface (see related blog post).

I will definitely write an e-mail to Nuand management requesting to correct this mis-representation. If you agree with my posting, I'd appreciate if you would contact them, too.

May 27, 2016

Harald Welte: Keynote at Black Duck Korea Open Source Conference

I've been giving a keynote at the Black Duck Korea Open Source Conference yesterday, and I'd like to share some thoughts about it.

In terms of the content, I spoke about the fact that the ultimate goal/wish/intent of free software projects is to receive contributions and for all of the individual and organizational users to join the collaborative development process. However, that's just the intent, and it's not legally required.

Due to GPL enforcement work, a lot of attention has been created over the past ten years in the corporate legal departments on how to comply with FOSS license terms, particularly copyleft-style licenses like GPLv2 and GPLv3. However,

License compliance ensures the absolute bare legal minimum on engaging with the Free Software community. While that is legally sufficient, the community actually wants to have all developers join the collaborative development process, where the resources for development are contributed and shared among all developers.

So I think if we had more contribution and a more fair distribution of the work in developing and maintaining the related software, we would not have to worry so much about legal enforcement of licenses.

However, in the absence of companies being good open source citizens, pulling out the legal baton is all we can do to at least require them to share their modifications at the time they ship their products. That code might not be mergeable, or it might be outdated, so it's value might be less than we would hope for, but it is a beginning.

Now some people might be critical of me speaking at a Black Duck Korea event, where Black Duck is a company selling (expensive!) licenses to proprietary tools for license compliance. Thereby, speaking at such an event might be seen as an endorsement of Black Duck and/or proprietary software in general.

Honestly, I don't think so. If you've ever seen a Black Duck Korea event, then you will notice there is no marketing or sales booth, and that there is no sales pitch on the conference agenda. Rather, you have speakers with hands-on experience in license compliance either from a community point of view, or from a corporate point of view, i.e. how companies are managing license compliance processes internally.

Thus, the event is not a sales show for proprietary software, but an event that brings together various people genuinely interested in license compliance matters. The organizers very clearly understand that they have to keep that kind of separation. So it's actually more like a community event, sponsored by a commercial entity - and that in turn is true for most technology conferences.

So I have no ethical problems with speaking at their event. People who know me, know that I don't like proprietary software at all for ethical reasons, and avoid it personally as far as possible. I certainly don't promote Black Ducks products. I promote license compliance.

Let's look at it like this: If companies building products based on Free Software think they need software tools to help them with license compliance, and they don't want to develop such tools together in a collaborative Free Software project themselves, then that's their decision to take. To state using words of Rosa Luxemburg:

Freedom is always the freedom of those who think different

I may not like that others want to use proprietary software, but if they think it's good for them, it's their decision to take.

May 26, 2016

Harald Welte: Osmocom.org GTP-U kernel implementation merged mainline

Have you ever used mobile data on your phone or using Tethering?

In packet-switched cellular networks (aka mobile data) from GPRS to EDGE, from UMTS to HSPA and all the way into modern LTE networks, there is a tunneling protocol called GTP (GPRS Tunneling Protocol).

This was the first cellular protocol that involved transport over TCP/IP, as opposed to all the ISDN/E1/T1/FrameRelay world with their weird protocol stacks. So it should have been something super easy to implement on and in Linux, and nobody should have had a reason to run a proprietary GGSN, ever.

However, the cellular telecom world lives in a different universe, and to this day you can be safe to assume that all production GGSNs are proprietary hardware and/or software :(

In 2002, Jens Jakobsen at Mondru AB released the initial version of OpenGGSN, a userspace implementation of this tunneling protocol and the GGSN network element. Development however ceased in 2005, and we at the Osmocom project thus adopted OpenGGSN maintenance in 2016.

Having a userspace implementation of any tunneling protocol of course only works for relatively low bandwidth, due to the scheduling and memory-copying overhead between kernel, userspace, and kernel again.

So OpenGGSN might have been useful for early GPRS networks where the maximum data rate per subscriber is in the hundreds of kilobits, but it certainly is not possible for any real operator, particularly not at today's data rates.

That's why for decades, all commonly used IP tunneling protocols have been implemented inside the Linux kernel, which has some tunneling infrastructure used with tunnels like IP-IP, SIT, GRE, PPTP, L2TP and others.

But then again, the cellular world lives in a universe where Free and Open Source Software didn't exit until OpenBTS and OpenBSC changed all o that from 2008 onwards. So nobody ever bothered to add GTP support to the in-kernel tunneling framework.

In 2012, I started an in-kernel implementation of GTP-U (the user plane with actual user IP data) as part of my work at sysmocom. My former netfilter colleague and current netfilter core team leader Pablo Neira was contracted to bring it further along, but unfortunately the customer project funding the effort was discontinued, and we didn't have time to complete it.

Luckily, in 2015 Andreas Schultz of Travelping came around and has forward-ported the old code to a more modern kernel, fixed the numerous bugs and started to test and use it. He also kept pushing Pablo and me for review and submission, thanks for that!

Finally, in May 2016, the code was merged into the mainline kernel, and now every upcoming version of the Linux kernel will have a fast and efficient in-kernel implementation of GTP-U. It is configured via netlink from userspace, where you are expected to run a corresponding daemon for the control plane, such as either OpenGGSN, or the new GGSN + PDN-GW implementation in Erlang called erGW.

You can find the kernel code at drivers/net/gtp.c, and the userspace netlink library code (libgtpnl) at git.osmocom.org.

I haven't done actual benchmarking of the performance that you can get on modern x86 hardware with this, but I would expect it to be the same of what you can also get from other similar in-kernel tunneling implementations.

Now that the cellular industry has failed for decades to realize how easy and little effort would have been needed to have a fast and inexpensive GGSN around, let's see if now that other people did it for them, there will be some adoption.

If you're interested in testing or running a GGSN or PDN-GW and become an early adopter, feel free to reach out to Andreas, Pablo and/or me. The osmocom-net-gprs mailing list might be a good way to discuss further development and/or testing.

May 21, 2016

Harald Welte: Slovenian student sentenced for detecting TETRA flaws using OsmocomTETRA

According to some news report, including this report at softpedia, a 26 year old student at the Faculty of Criminal Justice and Security in Maribor, Slovenia has received a suspended prison sentence for finding flaws in Slovenian police and army TETRA network using OsmocomTETRA

As the Osmocom project leader and main author of OsmocomTETRA, this is highly disturbing news to me. OsmocomTETRA was precisely developed to enable people to perform research and analysis in TETRA networks, and to audit their safe and secure configuration.

If a TETRA network (like any other network) is configured with broken security, then the people responsible for configuring and operating that network are to be blamed, and not the researcher who invests his personal time and effort into demonstrating that police radio communications safety is broken. On the outside, the court sentence really sounds like "shoot the messenger". They should instead have jailed the people responsible for deploying such an insecure network in the first place, as well as those responsible for not doing the most basic air-interface interception tests before putting such a network into production.

According to all reports, the student had shared the results of his research with the authorities and there are public detailed reports from 2015, like the report (in Slovenian) at https://podcrto.si/vdor-v-komunikacijo-policije-razkril-hude-varnostne-ranljivosti-sistema-tetra/.

The statement that he should have asked the authorities for permission before starting his research is moot. I've seen many such cases and you would normally never get permission to do this, or you would most likely get no response from the (in)competent authorities in the first place.

From my point of view, they should give the student a medal of honor, instead of sentencing him. He has provided a significant service to the security of the public sector communications in his country.

To be fair, the news report also indicates that there were other charges involved, like impersonating a police officer. I can of course not comment on those.

Please note that I do not know the student or his research first-hand, nor did I know any of his actions or was involved in them. OsmocomTETRA is a Free / Open Source Software project available to anyone in source code form. It is a vital tool in demonstrating the lack of security in many TETRA networks, whether networks for public safety or private networks.

May 01, 2016

Harald Welte: Developers wanted for Osmocom GSM related work

Right now I'm feeling sad. I really shouldn't, but I still do.

Many years ago I started OpenBSC and Osmocom in order to bring Free Software into an area where it barely existed before: Cellular Infrastructure. For the first few years, it was "just for fun", without any professional users. A FOSS project by enthusiasts. Then we got some commercial / professional users, and with them funding, paying for e.g. Holger and my freelance work. Still, implementing all protocol stacks, interfaces and functional elements of GSM and GPRS from the radio network to the core network is something that large corporations typically spend hundreds of man-years on. So funding for Osmocom GSM implementations was always short, and we always tried to make the best out of it.

After Holger and I started sysmocom in 2011, we had a chance to use funds from BTS sales to hire more developers, and we were growing our team of developers. We finally could pay some developers other than ourselves from working on Free Software cellular network infrastructure.

In 2014 and 2015, sysmocom got side-tracked with some projects where Osmocom and the cellular network was only one small part of a much larger scope. In Q4/2015 and in 2016, we are back on track with focussing 100% at Osmocom projects, which you can probably see by a lot more associated commits to the respective project repositories.

By now, we are in the lucky situation that the work we've done in the Osmocom project on providing Free Software implementations of cellular technologies like GSM, GPRS, EDGE and now also UMTS is receiving a lot of attention. This attention translates into companies approaching us (particularly at sysmocom) regarding funding for implementing new features, fixing existing bugs and short-comings, etc. As part of that, we can even work on much needed infrastructural changes in the software.

So now we are in the opposite situation: There's a lot of interest in funding Osmocom work, but there are few people in the Osmocom community interested and/or capable to follow-up to that. Some of the early contributors have moved into other areas, and are now working on proprietary cellular stacks at large multi-national corporations. Some others think of GSM as a fun hobby and want to keep it that way.

At sysmocom, we are trying hard to do what we can to keep up with the demand. We've been looking to add people to our staff, but right now we are struggling only to compensate for the regular fluctuation of employees (i.e. keep the team size as is), let alone actually adding new members to our team to help to move free software cellular networks ahead.

I am struggling to understand why that is. I think Free Software in cellular communications is one of the most interesting and challenging frontiers for Free Software to work on. And there are many FOSS developers who love nothing more than to conquer new areas of technology.

At sysmocom, we can now offer what would have been my personal dream job for many years:

  • paid work on Free Software that is available to the general public, rather than something only of value to the employer
  • interesting technical challenges in an area of technology where you will not find the answer to all your problems on stackoverflow or the like
  • work in a small company consisting almost entirely only of die-hard engineers, without corporate managers, marketing departments, etc.
  • work in an environment free of Microsoft and Apple software or cloud services; use exclusively Free Software to get your work done

I would hope that more developers would appreciate such an environment. If you're interested in helping FOSS cellular networks ahead, feel free to have a look at http://sysmocom.de/jobs or contact us at jobs@sysmocom.de. Together, we can try to move Free Software for mobile communications to the next level!

April 08, 2016

Rusty Russell: Bitcoin Generic Address Format Proposal

I’ve been implementing segregated witness support for c-lightning; it’s interesting that there’s no address format for the new form of addresses.  There’s a segregated-witness-inside-p2sh which uses the existing p2sh format, but if you want raw segregated witness (which is simply a “0” followed by a 20-byte or 32-byte hash), the only proposal is BIP142 which has been deferred.

If we’re going to have a new address format, I’d like to make the case for shifting away from bitcoin’s base58 (eg. 1At1BvBMSEYstWetqTFn5Au4m4GFg7xJaNVN2):

  1. base58 is not trivial to parse.  I used the bignum library to do it, though you can open-code it as bitcoin-core does.
  2. base58 addresses are variable-length.  That makes webforms and software mildly harder, but also eliminates a simple sanity check.
  3. base58 addresses are hard to read over the phone.  Greg Maxwell points out that the upper and lower case mix is particularly annoying.
  4. The 4-byte SHA check does not guarantee to catch the most common form of errors; transposed or single incorrect letters, though it’s pretty good (1 in 4 billion chance of random errors passing).
  5. At around 34 letters, it’s fairly compact (36 for the BIP141 P2WPKH).

This is my proposal for a generic replacement (thanks to CodeShark for generalizing my previous proposal) which covers all possible future address types (as well as being usable for current ones):

  1. Prefix for type, followed by colon.  Currently “btc:” or “testnet:“.
  2. The full scriptPubkey using base 32 encoding as per http://philzimmermann.com/docs/human-oriented-base-32-encoding.txt.
  3. At least 30 bits for crc64-ecma, up to a multiple of 5 to reach a letter boundary.  This covers the prefix (as ascii), plus the scriptPubKey.
  4. The final letter is the Damm algorithm check digit of the entire previous string, using this 32-way quasigroup. This protects against single-letter errors as well as single transpositions.

These addresses look like btc:ybndrfg8ejkmcpqxot1uwisza345h769ybndrrfg (41 digits for a P2WPKH) or btc:yybndrfg8ejkmcpqxot1uwisza345h769ybndrfg8ejkmcpqxot1uwisza34 (60 digits for a P2WSH) (note: neither of these has the correct CRC or check letter, I just made them up).  A classic P2PKH would be 45 digits, like btc:ybndrfg8ejkmcpqxot1uwisza345h769wiszybndrrfg, and a P2SH would be 42 digits.

While manually copying addresses is something which should be avoided, it does happen, and the cost of making them robust against common typographic errors is small.  The CRC is a good idea even for machine-based systems: it will let through less than 1 in a billion mistakes.  Distinguishing which blockchain is a nice catchall for mistakes, too.

We can, of course, bikeshed this forever, but I wanted to anchor the discussion with something I consider fairly sane.

April 01, 2016

Rusty Russell: BIP9: versionbits In a Nutshell

Hi, I was one of the authors/bikeshedders of BIP9, which Pieter Wuille recently refined (and implemented) into its final form.  The bitcoin core plan is to use BIP9 for activations from now on, so let’s look at how it works!

Some background:

  • Blocks have a 32-bit “version” field.  If the top three bits are “001”, the other 29 bits represent possible soft forks.
  • BIP9 uses the same 2016-block periods (roughly 2 weeks) as the difficulty adjustment does.

So, let’s look at BIP68 & 112 (Sequence locks and OP_CHECKSEQUENCEVERIFY) which are being activated together:

  • Every soft fork chooses an unused bit: these are using bit 1 (not bit 0), so expect to see blocks with version 536870914.
  • Every soft fork chooses an start date: these use May 1st, 2016, and time out a year later if it fails.
  • Every period, we look back to see if 95% have a bit set (75% for testnet).
    • If so, and that bit is for a known soft fork, and we’re within its start time that soft fork is locked-in: it will activate after another 2016 blocks, giving the stragglers time to upgrade.

There are also two alerts in the bitcoin core implementation:

  • If at any stage 50 of the last 100 blocks have unexpected bits set, you get Warning: Unknown block versions being mined! It’s possible unknown rules are in effect.
  • If we see an unknown softfork bit activate: you get Warning: unknown new rules activated (versionbit X).

Now, when could the OP_CSV soft forks activate? bitcoin-core will only start setting the bit in the first period after the start date, so somewhere between 1st and 15th of May[1], then will take another period to lock-in (even if 95% of miners are already upgraded), then another period to activate.  So early June would be the earliest possible date, but we’ll get two weeks notice for sure.

The Old Algorithm

For historical purposes, I’ll describe how the old soft-fork code worked.  It used version as a simple counter, eg. 3 or above meant BIP66, 4 or above meant BIP65 support.  Every block, it examined the last 1000 blocks to see if more than 75% had the new version.  If so, then the new softfork rules were enforced on new version blocks: old version blocks would still be accepted, and use the old rules.  If more than 95% had the new version, old version blocks would be rejected outright.

I remember Gregory Maxwell and other core devs stayed up late several nights because BIP66 was almost activated, but not quite.  And as a miner there was no guarantee on how long before you had to upgrade: one smaller miner kept producing invalid blocks for weeks after the BIP66 soft fork.  Now you get two weeks’ notice (probably more if you’re watching the network).

Finally, this change allows for miners to reject a particular soft fork without rejecting them all.  If we’re going to see more contentious or competing proposals in the future, this kind of plumbing allows it.

Hope that answers all your questions!


 

[1] It would be legal for an implementation to start setting it on the very first block past the start date, though it’s easier to only think about version bits once every two weeks as bitcoin-core does.

March 27, 2016

Harald Welte: You can now install a GSM network using apt-get

This is great news: You can now install a GSM network using apt-get!

Thanks to the efforts of Debian developer Ruben Undheim, there's now an OpenBSC (with all its flavors like OsmoBSC, OsmoNITB, OsmoSGSN, ...) package in the official Debian repository.

Here is the link to the e-mail indicating acceptance into Debian: https://tracker.debian.org/news/755641

I think for the past many years into the OpenBSC (and wider Osmocom) projects I always assumed that distribution packaging is not really something all that important, as all the people using OpenBSC surely would be technical enough to build it from the source. And in fact, I believe that building from source brings you one step closer to actually modifying the code, and thus contribution.

Nevertheless, the project has matured to a point where it is not used only by developers anymore, and particularly also (god beware) by people with limited experience with Linux in general. That such people still exist is surprisingly hard to realize for somebody like myself who has spent more than 20 years in Linux land by now.

So all in all, today I think that having packages in a Distribution like Debian actually is important for the further adoption of the project - pretty much like I believe that more and better public documentation is.

Looking forward to seeing the first bug reports reported through bugs.debian.org rather than https://projects.osmocom.org/ . Once that happens, we know that people are actually using the official Debian packages.

As an unrelated side note, the Osmocom project now also has nightly builds available for Debian 7.0, Debian 8.0 and Ubunut 14.04 on both i586 and x86_64 architecture from https://build.opensuse.org/project/show/network:osmocom:nightly. The nightly builds are for people who want to stay on the bleeding edge of the code, but who don't want to go through building everything from scratch. See Holgers post on the openbsc mailing list for more information.

March 14, 2016

Harald Welte: Open Source mobile communications, security research and contributions

While preparing my presentation for the Troopers 2016 TelcoSecDay I was thinking once again about the importance of having FOSS implementations of cellular protocol stacks, interfaces and network elements in order to enable security researches (aka Hackers) to work on improving security in mobile communications.

From the very beginning, this was the motivation of creating OpenBSC and OsmocomBB: To enable more research in this area, to make it at least in some ways easier to work in this field. To close a little bit of the massive gap on how easy it is to do applied security research (aka hacking) in the TCP/IP/Internet world vs. the cellular world.

We have definitely succeeded in that. Many people have successfully the various Osmocom projects in order to do cellular security research, and I'm very happy about that.

However, there is a back-side to that, which I'm less happy about. In those past eight years, we have not managed to attract significant amount of contributions to the Osmocom projects from those people that benefit most from it: Neither from those very security researchers that use it in the first place, nor from the Telecom industry as a whole.

I can understand that the large telecom equipment suppliers may think that FOSS implementations are somewhat a competition and thus might not be particularly enthusiastic about contributing. However, the story for the cellular operators and the IT security crowd is definitely quite different. They should have no good reason not to contribute.

So as a result of that, we still have a relatively small amount of people contributing to Osmocom projects, which is a pity. They can currently be divided into two groups:

  • the enthusiasts: People contributing because they are enthusiastic about cellular protocols and technologies.
  • the commercial users, who operate 2G/2.5G networks based on the Osmocom protocol stack and who either contribute directly or fund development work at sysmocom. They typically operate small/private networks, so if they want data, they simply use Wifi. There's thus not a big interest or need in 3G or 4G technologies.

On the other hand, the security folks would love to have 3G and 4G implementations that they could use to talk to either mobile devices over a radio interface, or towards the wired infrastructure components in the radio access and core networks. But we don't see significant contributions from that sphere, and I wonder why that is.

At least that part of the IT security industry that I know typically works with very comfortable budgets and profit rates, and investing in better infrastructure/tools is not charity anyway, but an actual investment into working more efficiently and/or extending the possible scope of related pen-testing or audits.

So it seems we might want to think what we could do in order to motivate such interested potential users of FOSS 3G/4G to contribute to it by either writing code or funding associated developments...

If you have any thoughts on that, feel free to share them with me by e-mail to laforge@gnumonks.org.

Harald Welte: TelcoSecDay 2016: Open Source Network Elements for Security Analysis of Mobile Networks

Today I had the pleasure of presenting about Open Source Network Elements for Security Analysis of Mobile Networks at the Troopers 2016 TelcoSecDay.

The main topics addressed by this presentation are:

  • Importance of Free and Open Source Software implementations of cellular network protocol stacks / interfaces / network elements for applied telecom security research
  • The progress we've made at Osmocom over the last eight years.
  • An overview about our current efforts to implement at 3G Network similar to the existing 2G/2.5G/2.75G implementations.

There are no audio or video recordings of this session.

Slides are available at http://git.gnumonks.org/index.html/laforge-slides/plain/2016/telcosecday/foss-gsm.html

March 08, 2016

Harald Welte: Linaro Connect BKK16 Keynote on GPL Compliance

Today I had the pleasure of co-presenting with Shane Coughlan the Linaro Connect BKK16 Keynote on GPL compliance about GPL compliance.

The main topics addressed by this presentation are:

  • Brief history about GPL enforcement and how it has impacted the industry
  • Ultimate Goal of GPL enforcement is compliance
  • The license is not an end in itself, but rather to facilitate collaborative development
  • GPL compliance should be more engineering and business driven, not so much legal (compliance) driven.

The video recording is available at https://www.youtube.com/watch?v=b4Bli8h0V-Q

Slides are available at http://git.gnumonks.org/index.html/laforge-slides/plain/2016/linaroconnect/compliance.html

The video of a corresponding interview is available from https://www.youtube.com/watch?v=I6IgjCyO-iQ

February 24, 2016

Harald Welte: Report from the VMware GPL court hearing

Today, I took some time off to attend the court hearing in the GPL violation/infringement case that Christoph Hellwig has brought against VMware.

I am not in any way legally involved in the lawsuit. However, as a fellow (former) Linux kernel developer myself, and a long-term Free Software community member who strongly believes in the copyleft model, I of course am very interested in this case - and of course in an outcome in favor of the plaintiff. Nevertheless, the below report tries to provide an un-biased account of what happened at the hearing today, and does not contain my own opinions on the matter. I can always write another blog post about that :)

I blogged about this case before briefly, and there is a lot of information publicly discussed about the case, including the information published by the Software Freedom Conservancy (see the link above, the announcement and the associated FAQ.

Still, let's quickly summarize the facts:

  • VMware is using parts of the Linux kernel in their proprietary ESXi product, including the entire SCSI mid-layer, USB support, radix tree and many, many device drivers.
  • as is generally known, Linux is licensed under GNU GPLv2, a copyleft-style license.
  • VMware has modified all the code they took from the Linux kernel and integrated them into something they call vmklinux.
  • VMware has modified their proprietary virtualization OS kernel vmkernel with specific API/symbol to interact with vmklinux
  • at least in earlier versions of ESXi, virtually any block device access has to go through vmklinux and thus the portions of Linux they took
  • vmklinux and vmkernel are dynamically linked object files that are linked together at run-time
  • the Linux code they took runs in the same execution context (address space, stack, control flow) like the vmkernel.

Ok, now enter the court hearing of today.

Christoph Hellwig was represented by his two German Lawyers, Dr. Till Jaeger and Dr. Miriam Ballhausen. VMware was represented by three German lawyers lead by Matthias Koch, as well as a US attorney, Michael Jacobs (by means of two simultaneous interpreters). There were also several members of the in-house US legal team of VMware present, but not formally representing the defendant in court.

As is unusual for copyright disputes, there was quite some audience following the court. Next to the VMware entourage, there were also a couple of fellow Linux kernel developers as well as some German IT press representatives following the hearing.

General Introduction of the presiding judge

After some formalities (like the question whether or not a ',' is missing after the "Inc." in the way it is phrased in the lawsuit), the presiding judge started with some general remarks

  • the court is well aware of the public (and even international public) interest in this case
  • the court understands there are novel fundamental legal questions raised that no court - at least no German court - had so far to decide upon.
  • the court also is well aware that the judges on the panel are not technical experts and thus not well-versed in software development or computer science. Rather, they are a court specialized on all sorts of copyright matters, not particularly related to software.
  • the court further understands that Linux is a collaborative, community-developed operating system, and that the development process is incremental and involves many authors.
  • the court understands there is a lot of discussion about interfaces between different programs or parts of a program, and that there are a variety of different definitions and many interpretations of what interfaces are

Presentation about the courts understanding of the subject matter

The presiding judge continued to explain what was their understanding of the subject matter. They understood VMware ESXi serves to virtualize a computer hardware in order to run multiple copies of the same or of different versions of operating systems on it. They also understand that vmkernel is at the core of that virtualization system, and that it contains something called vmkapi which is an interface towards Linux device drivers.

However, they misunderstood that this case was somehow an interface between a Linux guest OS being virtualized on top of vmkernel. It took both defendant and plaintiff some time to illustrate that in fact this is not the subject of the lawsuit, and that you can still have portions of Linux running linked into vmkernel while exclusively only virtualizing Windows guests on top of vmkernel.

The court went on to share their understanding of the GPLv2 and its underlying copyleft principle, that it is not about abandoning the authors' rights but to the contrary exercising copyright. They understood the license has implications on derivative works and demonstrated that they had been working with both the German translation a well as the English language original text of GPLv2. At least I was sort-of impressed by the way they grasped it - much better than some of the other courts that I had to deal with in the various cases I was bringing forward during my gpl-violations.org work before.

They also illustrated that they understood that Christoph Hellwig has been developing parts of the Linux kernel, and that modified parts of Linux were now being used in some form in VMware ESXi.

After this general introduction, there was the question of whether or not both parties would still want to settle before going further. The court already expected that this would be very unlikely, as it understood that the dispute serves to resolve fundamental legal question, and there is hardly any compromise in the middle between using or not using the Linux code, or between licensing vmkernel under a GPL compatible license or not. And as expected, there was no indication from either side that they could see an out-of-court settlement of the dispute at this point.

Right to sue / sufficient copyrighted works of the plaintiff

There was quite some debate about the question whether or not the plaintiff has shown that he actually holds a sufficient amount of copyrighted materials.

The question here is not, whether Christoph has sufficient copyrightable contributions on Linux as a whole, but for the matter of this legal case it is relevant which of his copyrighted works end up in the disputed product VMware ESXi.

Due to the nature of the development process where lots of developers make intermittent and incremental changes, it is not as straight-forward to demonstrate this, as one would hope. You cannot simply print an entire C file from the source code and mark large portions as being written by Christoph himself. Rather, lines have been edited again and again, were shifted, re-structured, re-factored. For a non-developer like the judges, it is therefore not obvious to decide on this question.

This situation is used by the VMware defense in claiming that overall, they could only find very few functions that could be attributed to Christoph, and that this may altogether be only 1% of the Linux code they use in VMware ESXi.

The court recognized this as difficult, as in German copyright law there is the concept of fading. If the original work by one author has been edited to an extent that it is barely recognizable, his original work has faded and so have his rights. The court did not state whether it believed that this has happened. To the contrary, the indicated that it may very well be that only very few lines of code can actually make a significant impact on the work as a whole. However, it is problematic for them to decide, as they don't understand source code and software development.

So if (after further briefs from both sides and deliberation of the court) this is still an open question, it might very well be the case that the court would request a techncial expert report to clarify this to the court.

Are vmklinux + vmkernel one program/work or multiple programs/works?

Finally, there was some deliberation about the very key question of whether or not vmkernel and vmklinux were separate programs / works or one program / work in the sense of copyright law. Unfortunately only the very surface of this topic could be touched in the hearing, and the actual technical and legal arguments of both sides could not be heard.

The court clarified that if vmkernel and vmklinux would be considered as one program, then indeed their use outside of the terms of the GPL would be an intrusion into the rights of the plaintiff.

The difficulty is how to actually venture into the legal implications of certain technical software architecture, when the people involved have no technical knowledge on operating system theory, system-level software development and compilers/linkers/toolchains.

A lot is thus left to how good and 'believable' the parties can present their case. It was very clear from the VMware side that they wanted to down-play the role and proportion of vmkernel and its Linux heritage. At times their lawyers made statements like linux is this small yellow box in the left bottom corner (of our diagram). So of course already the diagrams are drawn in a way to twist the facts according to their view on reality.

Summary

  • The court seems very much interested in the case and wants to understand the details
  • The court recognizes the general importance of the case and the public interest in it
  • There were some fundamental misunderstandings on the technical architecture of the software under dispute that could be clarified
  • There are actually not that many facts that are disputed between both sides, except the (key, and difficult) questions on
    • does Christoph hold sufficient rights on the code to bring forward the legal case?
    • are vmkernel and vmklinux one work or two separate works?

The remainder of this dispute will thus be centered on the latter two questions - whether in this court or in any higher courts that may have to re-visit this subject after either of the parties takes this further, if the outcome is not in their favor.

In terms of next steps,

  • both parties have until April 15, 2016 to file further briefs to follow-up the discussions in the hearing today
  • the court scheduled May 19, 2016 as date of promulgation. However, this would of course only hold true if the court would reach a clear decision based on the briefs by then. If there is a need for an expert, or any witnesses need to be called, then it is likely there will be further hearings and no verdict will be reached by then.

February 23, 2016

Harald Welte: Software under OSA Public License is neither Open Source nor Free Software

It seems my recent concerns on the OpenAirInterface re-licensing were not unjustified.

I contacted various legal experts on Free Software legal community about this, and the response was unanimous: In all feedback I received, the general opinion was that software under the OSA Public License V1.0 is neither Free Software nor Open Source Software.

The rational is, that it does not fulfill the criteria of

  • the FSF Free Software definition, as the license does not fulfill freedom 0: The freedom to run the program as you wish, for any purpose (which obviously includes commercial use)
  • the Open Source Initiatives Open Source Definition, as the license must not discriminate against fields of endeavor, such as commercial use.
  • the Debian Free Software Guidelines, as the DFSG also require no discrimination against fields of endeavor, such as commercial use.

I think we as the community need to be very clear about this. We should not easily tolerate that people put software under restrictive licenses but still call that software open source. This creates a bad impression to those not familiar with the culture and spirit of both Free Software and Open Source. It creates the impression that people can call something Open Source but then still ask royalties for it, if used commercially.

It is a shame that entities like Eurecom and the OpenAirInterface Software Association are open-washing their software by calling it Open Source when in fact it isn't. This attitude frankly makes me sick.

That's just like green-washing when companies like BP are claiming they're now an environmental friendly company just because they put some solar panels on the roof of some building.

February 20, 2016

Harald Welte: Osmocom.org migrating to redmine

In 2008, we started bs11-abis, which was shortly after renamed to OpenBSC. At the time it seemed like a good idea to use trac as the project management system, to have a wiki and an issue tracker.

When further Osmocom projects like OsmocomBB, OsmocomTETRA etc. came around, we simply replicated that infrastructure: Another trac instance with the same theme, and a shared password file.

The problem with this (and possibly the way we used it) is:

  • it doesn't scale, as creating projects is manual, requires a sysadmin and is time-consuming. This meant e.g. SIMtrace was just a wiki page in the OsmocomBB trac installation + associated http redirect, causing some confusion.
  • issues can not easily be moved from one project to another, or have cross-project relationships (like, depend on an issue in another project)
  • we had to use an external planet in order to aggregate the blog of each of the trac instances
  • user account management the way we did it required shell access to the machine, meaning user account applications got dropped due to the effort involved. My apologies for that.

Especially the lack of being able to move pages and tickets between trac's has resulted in a suboptimal use of the tools. If we first write code as part of OpenBSC and then move it to libosmocore, the associated issues + wiki pages should be moved to a new project.

At the same time, for the last 5 years we've been successfully using redmine inside sysmocom to keep track of many dozens of internal projects.

So now, finally, we (zecke, tnt, myself) have taken up the task to migrate the osmocom.org projects into redmine. You can see the current status at http://projects.osmocom.org/. We could create a more comprehensive project hierarchy, and give libosmocore, SIMtrace, OsmoSGSN and many others their own project.

Thanks to zecke for taking care of the installation/sysadmin part and the initial conversion!

Unfortunately the conversion from trac to redmine wiki syntax (and structure) was not as automatic and straight-forward as one would have hoped. But after spending one entire day going through the most important wiki pages, things are looking much better now. As a side effect, I have had a more comprehensive look into the history of all of our projects than ever before :)

Still, a lot of clean-up and improvement is needed until I'm happy, particularly splitting the OpenBSC wiki into separate OsmoBSC, OsmoNITB, OsmoBTS, OsmoPCU and OsmoSGSN wiki's is probably still going to take some time.

If you would like to help out, feel free to register an account on projects.osmocom.org (if you don't already have one from the old trac projects) and mail me for write access to the project(s) of your choice.

Possible tasks include

  • putting pages into a more hierarchic structure (there's a parent/child relationship in redmine wikis)
  • fixing broken links due to page renames / wiki page moves
  • creating a new redmine 'Project' for your favorite tool that has a git repo on http://git.osmocom.org/ and writing some (at least initial) documentation about it.

You don't need to be a software developer for that!

February 19, 2016

Harald Welte: Some update on recent OsmoBTS changes

After quite some time of gradual bug fixing and improvement, there have been quite some significant changes being made in OsmoBTS over the last months.

Just a quick reminder: In Fall 2015 we finally merged the long-pending L1SAP changes originally developed by Jolly, introducing a new intermediate common interface between the generic part of OsmoBTS, and the hardware/PHY specific part. This enabled a clean structure between osmo-bts-sysmo (what we use on the sysmoBTS) and osmo-bts-trx (what people with general-purpose SDR hardware use).

The L1SAP changes had some fall-out that needed to be fixed, not a big surprise with any change that big.

More recently however, three larger changes were introduced:

proper Multi-TRX support

Based on the above phy_link/phy_instance infrastructure, one can map each phy_instance to one TRX by means of the VTY / configuration file.

The core of OsmoBTS now supports any number of TRXs, leading to flexible Multi-TRX support.

OCTPHY support

A Canadian company called Octasic has been developing a custom GSM PHY for their custom multi-core DSP architecture (OCTDSP). Rather than re-inventing the wheel for everything on top of the PHY, they chose to integrate OsmoBTS on top of it. I've been working at sysmocom on integrating their initial code into OsmoBTS, rendering a new osmo-bts-octphy backend.

This back-end has also recently been ported to the phy_link/phy_instance API and is Multi-TRX ready. You can both run multiple TRX in one DSP, as well as have multiple DSPs in one BTS, paving the road for scalability.

osmo-bts-octphy is now part of OsmoBTS master.

Corresponding changes to OsmoPCU (for full GPRS support on OCTPHY) are currently been worked on by Max at sysmocom.

Litecell 1.5 PHY support

Another Canadian company (Nutaq/Nuran) has been building a new BTS called Litecell 1.5. They also implemented OsmoBTS support, based on the osmo-bts-sysmo code. We've been able to integrate that code with the above-mentioned phy_link/phy_interface in order to support the MultiTRX capability of this hardware.

Litecell 1.5 MultiTRX capability has also been integrated with OsmoPCU.

osmo-bts-litecell15 is now part of OsmoBTS master.

Summary

  • 2016 starts as the OsmoBTS year of MultiTRX.
  • 2016 also starts as a year of many more hardware choices for OsmoBTS
  • we see more commercial adoption of OsmoBTS outside of the traditional options of sysmocom and Fairwaves

February 14, 2016

Harald Welte: Back from netdevconf 1.1 in Seville

I've had the pleasure of being invited to netdevconf 1.1 in Seville, spain.

After about a decade of absence in the Linux kernel networking community, it was great to meet lots of former colleagues again, as well as to see what kind of topics are currently being worked on and under discussion.

The conference had a really nice spirit to it. I like the fact that it is run by the community itself. Organized by respected members of the community. It feels like Linux-Kongress or OLS or UKUUG or many others felt in the past. There's just something that got lost when the Linux Foundation took over (or pushed aside) virtually any other Linux kernel related event on the planet in the past :/ So thanks to Jamal for starting netdevconf, and thanks to Pablo and his team for running this particular instance of it.

I never really wanted to leave netfilter and the Linux kernel network stack behind - but then my problem appears to be that there are simply way too many things of interest to me, and I had to venture first into RFID (OpenPCD, OpenPICC), then into smartphone hardware and software (Openmoko) and finally embark on a journey of applied telecoms archeology by starting OpenBSC, OsmocomBB and various other Osmocom projects.

Staying in Linux kernel networking land was simply not an option with a scope that can only be defined as wide as wanting to implement any possible protocol on any possible interface of any possible generation of cellular network.

At times like attending netdevconf I wonder if I made the right choice back then. Linux kernel networking is a lot of fun and hard challenges, too - and it is definitely an area that's much more used by many more organizations and individuals: The code I wrote on netfilter/iptables is probably running on billions of devices by now. Compare that to the Osmocom code, which is probably running on a few thousands of devices, if at all. Working on Open Source telecom protocols is sometimes a lonely fight. Not that I wouldn't value the entire team of developers involved in it. to the contrary. But lonely in the context that 99.999% of that world is a proprietary world, and FOSS cellular infrastructure is just the 0.001% at the margin of all of that.

One the Linux kernel side, you have virtually every IT company putting in their weight these days, and properly funded development is not that hard to come by. In cellular, reasonable funding for anything (compared to the scope and complexity of the tasks) is rather the exception than the norm.

But no, I don't have any regrets. It has been an interesting journey and I probably had the chance to learn many more things than if I had stayed in TCP/IP-land.

If only each day had 48 hours and I could work both on Osmocom and on the Linux kernel...

February 10, 2016

Harald Welte: netdevconf 1.1: Osmocom kernel-level GTP implementation

Today I had the pleasure of co-presenting with Andreas Schultz at netdevconf 1.1 about the Osmocom kernel-level GTP implementation.

The video recording is available from https://www.youtube.com/watch?v=puCMipd8fck

Slides are available at http://git.gnumonks.org/index.html/laforge-slides/plain/2016/netdevconf-gtp/netdev-gtp.html

February 09, 2016

Harald Welte: netdevconf 1.1: Running cellular infrastructure on Linux

Today I had the pleasure of presenting at netdevconf 1.1 a tutorial about Running cellular infrastructure on Linux. The tutorial is intended to guide you through the process of setting up + configuring yur own minimal private GSM+GPRS network.

The video recording is available from https://www.youtube.com/watch?v=I4i2Gy4JhDo

Slides are available at http://git.gnumonks.org/index.html/laforge-slides/plain/2016/netdevconf-osmocom/running-foss-gsm.html

January 31, 2016

Harald Welte: On the OpenAirInterface re-licensing

In the recent FOSDEM 2016 SDR Devroom, the Q&A session following a presentation on OpenAirInterface touched the topic of its controversial licensing. As I happen to be involved deeply with Free Software licensing and Free Software telecom topics, I thought I might have some things to say about this topic. Unfortunately the Q&A session was short, hence this blog post.

As a side note, the presentation was actually certainly the least technical presentation in all of the FOSDEM SDR track, and that with a deeply technical audience. And probably the only presentation at all at FOSDEM talking a lot about "Strategic Industry Partners".

Let me also state that I actually have respect for what OAI/OSA has been and still is doing. I just don't think it is attractive to the Free Software community - and it might actually not be Free Software at all.

OpenAirInterface / History

Within EURECOM, a group around Prof. Raymond Knopp has been working on a Free Software implementation of all layers of the LTE (4G) system known as OpenAirInterface. It includes the physical layer and goes through to the core network.

The OpenAirInterface code was for many years under GPL license (GPLv2, other parts GPLv3). Initially the SVN repositories were not public (despite the license), but after some friendly mails one (at least I) could get access.

I've read through the code at several points in the past, it often seemed much more like a (quick and dirty?) proof of concept implementation to me, than anything more general-purpose. But then, that might have been a wrong impression on my behalf, or it might be that this was simply sufficient for the kind of research they wanted to do. After all, scientific research and FOSS often have a complicated relationship. Researchers naturally have their papers as primary output of their work, and software implementations often are more like a necessary evil than the actual goal. But then, I digress.

Now at some point in 2014, a new organization the OpenAirInterface Software Association (OSA) was established. The idea apparently was to get involved with the tier-1 telecom suppliers (like Alcatel, Huawei, Ericsson, ...) and work together on an implementation of Free Software for future mobile data, so-called 5G technologies.

Telecom Industry and Patents

In case you don't know, the classic telecom industry loves patents. Pretty much anything and everything is patented, and the patents are heavily enforced. And not just between Samsung and Apple, or more recently also Nokia and Samsung - but basically all the time.

One of the big reasons why even the most simple UMTS/3G capable phones are so much more expensive than GSM/2G is the extensive (and expensive) list of patents Qualcomm requires every device maker to license. In the past, this was not even a fixed per-unit royalty, but the license depended on the actual overall price of the phone itself.

So wanting to work on a Free Software implementation of future telecom standards with active support and involvement of the telecom industry obviously means contention in terms of patents.

Re-Licensing

The existing GPLv2/GPLv3 license of the OpenAirInterface code of course would have meant that contributions from the patent-holding telecom industry would have to come with appropriate royalty-free patent licenses. After all, of what use is it if the software is free in terms of copyright licensing, but then you still have the patents that make it non-free.

Now the big industry of course wouldn't want to do that, so the OSA decided to re-license the code-base under a new license.

As we apparently don't yet have sufficient existing Free Software licenses, they decided to create a new license. That new license (the OSA Public License V1.0 not only does away with copyleft, but also does away with a normal patent grant.

This is very sad in several ways:

  • license proliferation is always bad. Major experts and basically all major entities in the Free Software world (FSF, FSFE, OSI, ...) are opposed to it and see it as a problem. Even companies like Intel and Google have publicly raised concern about license Proliferation.
  • abandoning copyleft. Many people particularly from a GNU/Linux background would agree that copyleft is a fair deal. It ensures that everyone modifying the software will have to share such modifications with other users in a fair way. Nobody can create proprietary derivatives.
  • taking away the patent grant. Even the non-copyleft Apache 2.0 License the OSA used as template has a broad patent grant, even for commercial applications. The OSA Public License has only a patent grant for use in research context

In addition to this license change, the OSA also requires a copyright assignment from all contributors.

Consequences

What kind of effect does this have in case I want to contribute?

  • I have to sign away my copyright. The OSA can at any given point in time grant anyone whatever license they want to this code.
  • I have to agree to a permissive license without copyleft, i.e. everyone else can create proprietary derivatives of my work
  • I do not even get a patent grant from the other contributors (like the large Telecom companies).

So basically, I have to sign away my copyright, and I get nothing in return. No copyleft that ensures other people's modifications will be available under the same license, no patent grant, and I don't even keep my own copyright to be able to veto any future license changes.

My personal opinion (and apparently those of other FOSDEM attendees) is thus that the OAI / OSA invitation to contributions from the community is not a very attractive one. It might all be well and fine for large industry and research institutes. But I don't think the Free Software community has much to gain in all of this.

Now OSA will claim that the above is not true, and that all contributors (including the Telecom vendors) have agreed to license their patents under FRAND conditions to all other contributors. It even seemed to me that the speaker at FOSDEM believed this was something positive in any way. I can only laugh at that ;)

FRAND

FRAND (Fair, Reasonable and Non-Discriminatory) is a frequently invoked buzzword for patent licensing schemes. It isn't actually defined anywhere, and is most likely just meant to sound nice to people who don't understand what it really means. Like, let's say, political decision makers.

In practise, it is a disaster for individuals and small/medium sized companies. I can tell you first hand from having tried to obtain patent licenses from FRAND schemes before. While they might have reasonable per-unit royalties and they might offer those royalties to everyone, they typically come with ridiculous minimum annual fees.

For example let's say they state in their FRAND license conditions you have to pay 1 USD per device, but a minimum of USD 100,000 per year. Or a similarly large one-time fee at the time of signing the contract.

That's of course very fair to the large corporations, but it makes it impossible for a small company who sells maybe 10 to 100 devices per year, as the 100,000 / 10 then equals to USD 10k per device in terms of royalties. Does that sound fair and Non-Discriminatory to you?

Summary

OAI/OSA are trying to get a non-commercial / research-oriented foot into the design and specification process of future mobile telecom network standardization. That's a big and difficult challenge.

However, the decisions they have taken in terms of licensing show that they are primarily interested in aligning with the large corporate telecom industry, and have thus created something that isn't really Free Software (missing non-research patent grant) and might in the end only help the large telecom vendors to uni-directionally consume contributions from academic research, small/medium sized companies and individual hackers.

January 03, 2016

Harald Welte: Conferences I look forward to in 2016

While I was still active in the Linux kernel development / network security field, I was regularly attending 10 to 15 conferences per year.

Doing so is relatively easy if you earn a decent freelancer salary and are working all by yourself. Running a company funded out of your own pockets, with many issues requiring (or at least benefiting) from personal physical presence in the office changes that.

Nevertheless, after some years of being less of a conference speaker, I'm happy to see that the tide is somewhat changing in 2016.

After my talk at 32C3, I'm looking forward to attending (and sometimes speaking) at events in the first quarter of 2016. Not sure if I can keep up that pace in the following quarters...

FOSDEM

FOSDEM (http://fosdem.org/2016) a classic, and I don't even remember for how many years I've been attending it. I would say it is fair to state it is the single largest event specifically by and for community-oriented free software developers. Feels like home every time.

netdevconf 1.1

netdevconf (http://www.netdevconf.org/1.1/) is actually something I'm really looking forward to. A relatively new grass-roots conference. Deeply technical, and only oriented towards Linux networking hackers. The part of the kernel community that I've known and loved during my old netfilter days.

I'm very happy to attend the event, both for its technical content, and of course to meet old friends like Jozsef, Pablo, etc. I also read that Kuninhiro Ishiguro will be there. I always adored his initial work on Zebra (whose vty code we coincidentally use in almost all osmocom projects as part of libosmovty).

It's great to again see an event that is not driven by commercial / professional conference organizers, high registration fees, and corporate interests. Reminds me of the good old days where Linux was still the underdog and not mainstream... Think of Linuxtag in its early days?

Linaro Connect

I'll be attending Linaro Connect for the first time in many years. It's a pity that one cannot run various open source telecom protocol stack / network element projects and a company and at the same time still be involved deeply in Embedded Linux kernel/system development. So I'll use the opportunity to get some view into that field again - and of course meet old friends.

OsmoDevCon

OsmoDevCon is our annual invitation-only developer meeting of the Osmocom developers. It's very low-profile, basically a no-frills family meeting of the Osmocom community. But really great to meet with all of the team and hearing about their respective experiences / special interest topics.

TelcoSecDay

This (https://www.troopers.de/events/troopers16/580_telcosecday_2016_invitation_only/) is another invitation-only event, organized by the makers of the TROOPERS conference. The idea is to make folks from the classic Telco industry meet with people in IT Security who are looking at Telco related topics. I've been there some years ago, and will finally be able to make it again this year to talk about how the current introduction of 3G/3.5G into the Osmocom network side elements can be used for security research.

Rusty Russell: Bitcoin And Stuck Transactions?

One problem of filling blocks is that transactions with too-low fees will get “stuck”; I’ve read about such things happening on Reddit.  Then one of my coworkers told me that those he looked at were simply never broadcast properly, and broadcasting them manually fixed it.  Which lead both of us to wonder how often it’s really happening…

My approach is to look at the last 2 years of block data, and make a simple model:

  1. I assume the tx is not a priority tx (some miners reserve space for these; default 50k).
  2. I judge the “minimum feerate to get into a block” as the smallest feerate for any transaction after the first 50k beyond the coinbase (this is an artifact of how bitcoin core builds transactions; priority area first).
  3. I assume the tx won’t be included in “empty” blocks with only a coinbase or a single non-coinbase tx (SPV mining); their feerate is “infinite”.

Now, what feerate do we assume?  The default “dumb wallet” fee is 10000 satoshi per kilobyte: bitcoin-core doesn’t do this pro-rata, so a median 300-byte transaction still pays 10000 satoshi by default (fee-per-byte 33.33).  The worse case is a transaction of exactly 1000 bytes (or, a wallet which does pro-rata fees), which would have a fee-per-byte of 10.

So let’s consider the last two years (since block 277918).  How many blocks in a row we see with a fee-per-byte > 33.33, and how many we see with a feerate of > 10:

Conclusion

In the last two years you would never have experienced a delay of more than 10 blocks for a median-size transaction with a 10,000 satoshi fee.

For a 1000-byte transaction paying the same fee, you would have experienced a 10 block delay 0.7% of the time, with a 20+ block delay on eight occasions: the worse being a 26 block delay at block 382918 (just under 5 hours).  But note that this fee is insufficient to be included in 40% of blocks during the last two years, too; if your wallet is generating such things without warning you, it’s time to switch wallets!

Stuck low-fee transactions are not a real user problem yet.  It’s good to see adoption of smarter wallets, though, because it’s expected that they will be in the near future…

December 30, 2015

Harald Welte: 32C3 is over, GSM and GPRS was running fine, osmo-iuh progress

The 32C3 GSM Network

32C3 was great from the Osmocom perspective: We could again run our own cellular network at the event in order to perform load testing with real users. We had 7 BTSs running, each with a single TRX. What was new compared to previous years:

  • OsmoPCU is significantly more robust and stable due to the efforts of Jacob Erlbeck at sysmocom. This means that GPRS is now actually still usable in severe overload situations, like 1000 subscribers sharing only very few kilobits. Of course it will be slow, but at least data still passes through as much as that's possible.
  • We were using half-rate traffic channels from day 2 onwards, in order to enhance capacity. Phones supporting AMR-HR would use that, but then there are lots of old phones that only do classic HR (v1). OsmoNITB with internal MNCC handler supports TCH/H with HR and AMR for at least five years, but the particular combination of OsmoBTS + OsmoNITB + lcr (all master branches) was not yet deployed at previous CCC event networks so far.

Being forced to provide classic HR codec actually revealed several bugs in the existing code:

  • OsmoBTS (at least with the sysmoBTS hardware) is using bit ordering that is not compliant to what the spec says on how GSM-HR frames should be put into RTP frames. We didn't realize this so far, as handing frames from one sysmoBTS to another sysmoBTS of course works, as both use the same (wrong) bit ordering.
  • The ETSI reference implementation of the HR codec has lots of global/static variables, and thus doesn't really support running multiple transcoders in parallel. This is however what lcr was trying (and needing) to do, and it of course failed as state from one transcoder instance was leaking into another. The problem is simple, but the solution not so simple. If you want to avoid re-structuring the entire code in very intrusive ways or running one thread per transcoder instance, then the only solution was to basically memcpy() the entire data section of the transcoding library every time you switch the state from one transcoder instance to the other. It's surprisingly difficult to learn the start + size of that data section at runtime in a portable way, though.

Thanks to our resident voice codec expert Sylvain for debugging and fixing the above two problems.

Thanks also to Daniel and Ulli for taking care of the actual logistics of bringing + installing (+ later unmounting) all associated equipment.

Thanks furthermore to Kevin who has been patiently handling the 'Level 2 Support' cases of people with various problems ending up in the GSM room.

It's great that there is a team taking care of those real-world test networks. We learn a lot more about our software under heavy load situations this way.

osmo-iuh progress + talk

I've been focussing basically full day (and night) over the week ahead of Christmas and during Christmas to bring the osmo-iuh code into a state where we could do a end-to-end demo with a regular phone + hNodeB + osmo-hnbgw + osmo-sgsn + openggsn. Unfortunately I only got it up to the point where we do the PDP CONTEXT ACTIVATION on the signalling plane, with no actual user data going back and forth. And then, for strange reasons, I couldn't even demo that at the end of the talk. Well, in either case, the code has made much progress.

The video of the talk can be found at https://media.ccc.de/v/32c3-7412-running_your_own_3g_3_5g_network#video

meeting friends

The annual CCC congress is always an event where you meet old friends and colleagues. It was great talking to Stefan, Dimitri, Kevin, Nico, Sylvain, Jochen, Sec, Schneider, bunnie and many other hackers. After the event is over, I wish I could continue working together with all those folks the rest of the year, too :/

Some people have been missed dearly. Absence from the CCC congress is not acceptable. You know who you are, if you're reading this ;)

December 27, 2015

Eric Leblond: Out of [name]space issue

Introduction

I’m running Debian sid on my main laptop and if most of the time if works well there is from time to time some issues. Most of them fixes after a few days so most of the time I don’t try to fix them manually if there is no impact on my activity. Since a few weeks, the postinst script of avahi daemon was failing and as it was not fixing by itself during upgrade I’ve decided to have a look at it.

The usual ranting

As Debian sid is using systemd it is super easy to find a decent troll subject. Here it was the usual thing, systemctl was not managing to start correctly the daemon and giving me some commands if I wanted to know more:
So after a little prayer to Linux copy paste god resulting in a call to journalctl I had the message:
-- Unit avahi-daemon.service has begun starting up.
Dec 26 07:35:25 ice-age2 avahi-daemon[3466]: Found user 'avahi' (UID 105) and group 'avahi' (GID 108).
Dec 26 07:35:25 ice-age2 avahi-daemon[3466]: Successfully dropped root privileges.
Dec 26 07:35:25 ice-age2 avahi-daemon[3466]: chroot.c: fork() failed: Resource temporarily unavailable
Dec 26 07:35:25 ice-age2 avahi-daemon[3466]: failed to start chroot() helper daemon.
Dec 26 07:35:25 ice-age2 systemd[1]: avahi-daemon.service: Main process exited, code=exited, status=255/n/a
Dec 26 07:35:25 ice-age2 systemd[1]: Failed to start Avahi mDNS/DNS-SD Stack.
-- Subject: Unit avahi-daemon.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit avahi-daemon.service has failed.
-- 
-- The result is failed.
So a daemon was not able to fork on a rather quiet system.

Understanding the issue

A little googling lead me to this not a bug explaining the avahi configuration includes ulimit settings. So I checked my configuration and found out that Debian default configuration file as a hardcoded value of 5. My next command was to check the process running as avahi:
ps auxw|grep avahi
avahi    19159  1.0  3.0 6939804 504648 ?      Ssl  11:31   3:39 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.1.1.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.network.bind_host=0.0.0.0
So an Elasticsearch daemon was using the avahi user. This could seems strange if you did not know I’m running some docker containers (see https://github.com/StamusNetworks/Amsterdam). In fact in the container I have:
$ docker exec exebox_elasticsearch_1 ps auxw
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
elastic+     1  1.0  3.0 6940024 506648 ?      Ssl  10:31   3:46 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.1.1.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.network.bind_host=0.0.0.0
So the issue comes from the fact we have:
$ docker exec exebox_elasticsearch_1 id elasticsearch
uid=105(elasticsearch) gid=108(elasticsearch) groups=108(elasticsearch)
$ id avahi
uid=105(avahi) gid=108(avahi) groups=108(avahi)
That’s a real problem, the space of user ID in the container and in the host are identical and this can result in some really weird side effect.

Fixing it

At the time of the writing I did not found something to setup the user id mapping that is causing this conflict. An experimental feature using the recent user namespace in Linux will permit to avoid this conflict in a near future but it is not currently mainstream. A super bad workaround was to stop the docker container before doing the upgrade. It did do the job but I’m not sure I will have something working at reboot. I really hope this new feature in docker will soon reach mainstream to avoid similar issue to other people.

December 26, 2015

Harald Welte: 32C3: Running your own 3G/3.5G cellular network

Today I had the pleasure of presenting at 32C3 about Running your own 3G/3.5G cellular network. The tutorial covers the ongoing effort of creating a HNB-GW and Iuh/IuCS/IuPS support as part of the Osmocom project.

The video recording is available from https://media.ccc.de/v/32c3-7412-running_your_own_3g_3_5g_network

Slides are available at http://git.gnumonks.org/index.html/laforge-slides/plain/2015/osmo_iuh/osmo_iuh.pdf

December 22, 2015

Rusty Russell: Bitcoin: Mixed Signs of A Fee Market

Six months ago in a previous post I showed that 45% of transactions have an output of less that $1, and estimated that they would get squeezed out first as blocks filled.  It’s time to review that prediction, and also to see several things:

  1. Are fees rising?
  2. Are fees detached from magic (default) numbers of satoshi?
  3. Are low value transactions getting squeezed out?
  4. Are transactions starting to shrink in response to fee pressure?

Here are some scenarios: low-value transactions might be vanishing even if nothing else changes, because people’s expectations (“free global microtransactions!” are changing).  Fees might be rising but still on magic numbers, because miners and nodes increased their relayfee due to spam attacks (most commonly, the rate was increased from 1000 satoshi per kb to 5000 satoshi per kb).  Finally, we’d eventually expect wallets which produce large transactions (eg. using uncompressed signatures) to lose popularity, and wallets to get smarter about transaction generation (particularly once Segregated Witness makes it fairly easy).

Fees For The Last 2 Years

The full 4 year graph is very noisy, so I only plotted the mean txfee/kb for each day for the last two years, in Satoshi and USD (thanks to the Coindesk BPI data for the conversion):

 

Conclusion: Too noisy to be conclusive: they seem to be rising recently, but some of that reflects the exchange rate changes.

Are Fees on Magic Boundaries?

Wallets should be estimating fees: in a real fee market they’d need to.

Dumb wallets pay a fixed fee per kb: eg. the bitcoin-core wallet pays 1,000 (now 5,000) satoshi per kb by default; even if the transaction is 300 bytes, it will pay 5,000 satoshi.  Some wallets use (slightly more sensible) scaling-by-size, so they’d pay 1,500 satoshi.  So if a transaction fee ends in “000”, or the scaled transaction fee does (+/- 2) we can categorize them as “fixed fee”.  We assume others are using a variable fee (about 0.6% will be erroneously marked as fixed):

This graph is a bit dense, so we thin it by grouping into weeks:

 

Conclusion: Wallets are starting to adapt to fee pressure, though the majority are still using a fixed fee.

Low Value Transactions For Last 4 Years

We categorize 4 distinct types of transactions: ones which have an output below 25c, ones which have an output between 25c and $1, ones which have an output between $1 and $5, and ones which have no output below $5, and graph the trends for each for the last four years:

Conclusion: 25c transactions are flat (ignoring those spam attack spikes).  < $1 and <$5 are growing, but most growth is coming from transactions >= $5.

Transaction Size For Last 4 Years

Here are the transaction sizes for the last 4 years:

Conclusion: There seems to be a slight decline in transaction sizes, but it’s not clear the cause, and it might be just noise.

Conclusion

There are signs of a nascent fee market, but it’s still very early. I’d expect something conclusive in the next 6 months.

The majority of fees should be variable, and they’re not: wallets remain poor, but users will migrate as blocks fill and more transactions get stuck.

A fee rate of over 10c per kb (2.5c per median transaction) hasn’t suppressed 25c transactions: perhaps it’s not high enough yet, or perhaps wallets aren’t making the relative fees clear enough (eg. my Trezor gives fees in BTC, as well as  only offering fixed fee rates).

The slight dip in mean transaction sizes and lack of growth in 25c transactions to may point to early market pressure, however.

Six months ago I showed that 45% of transactions were less than a dollar.  In the last six months that has declined to 38%.  I previously estimated that we would want larger blocks within two years, and need them within three.  That still seems a reasonable estimate.

Data

I used bitcoin-iterate and a really crappy Makefile to generate CSVs with the data.  You can see the result on github or go straight to downloading the Gnumeric spreadsheet with the graphs.

Disclaimer: I Work For Blockstream

On lightning.  Not on drawing pretty graphs.  But I wanted to see the data…

 

December 05, 2015

Harald Welte: Volunteer for Openmoko.org USB Product ID maintenance

Back when Openmoko took the fall, we donated the Openmoko, Inc. USB Vendor ID to the community and started the registry of free Product ID allocations at http://wiki.openmoko.org/wiki/USB_Product_IDs

Given my many other involvements and constant overload, I've been doing a poor job at maintaining it, i.e. handling incoming requests.

So I'm looking for somebody who can reliably take care of it, including

  • reviewing if the project fulfills the criteria (hardware or software already released under FOSS license)
  • entering new allocations to the wiki
  • informing applicants of their allocation

The amount of work is actually not that much (like one mail per week), but it needs somebody to reliably respond to the requests in a shorter time frame than I can currently do.

Please let me know if you'd like to volunteer.

Harald Welte: Anyone interested in supporting SMPP interworking at 32C3?

Sylvain brought this up yesterday: Wouldn't it be nice to have some degree of SMS interfacing from OpenBSC/OsmoNITB to the real world at 32C3? It is something that we've never tried so far, and thus definitely worthy of testing.

Of course, full interworking is not possible without assigning public MSISDN to all internal subscribers / 'extensions' how we call them.

But what would most certainly work is to have at least outbound SMS working by means of an external SMPP interface.

The OsmoNITB-internal SMSC speaks SMPP already (in the SMSC role), so we would need to implement some small amount of glue logic that behaves as ESME (external SMS entity) towards both OsmoNITB as well as some public SMS operator/reseller that speaks SMPP again.

Now of course, sending SMS to public operators doesn't come for free. So in case anyone reading this has access to SMPP at public operators, resellers, SMS hubs, it would be interesting to see if there is a chance for some funding/sponsoring of that experiment.

Feel free to contact me if you see a way to make this happen.

December 04, 2015

Harald Welte: python-libsmpp works great with OsmoNITB

Since 2012 we have support for SMPP in OsmoNITB (the network-in-the-box version of OpenBSC). So far I've only used it from C and Erlang code.

Yesterday I gave python-smpplib from https://github.com/podshumok/python-smpplib a try and it worked like a charm. Of course one has to get the details right (like numbering plan indication).

In case anyone is interested in interfacing OsmoNITB SMPP from python, I've put a working example to send SMS at http://cgit.osmocom.org/mncc-python/tree/smpp_test.py

December 01, 2015

Harald Welte: Python tool to talk to OsmoNITB MNCC interface

I've been working on a small python tool that can be used to attach to the MNCC interface of OsmoNITB. It implements the 04.08 CC state machine with our MNCC primitives, including support for RTP bridge mode of the voice streams.

The immediate first use case for this was to be able to generate MT calls to a set of known MSISDNs and load all 14 TCH/H channels of a single-TRX BTS. It will connect the MT calls in pairs, so you end up with 7 MS-to-MS calls.

The first working version of the tool is available from

The code is pretty hacky in some places. That's partially due to the fact that I'm much more familiar in the C, Perl and Erlang world than in python. Still I thought it's a good idea to do it in python to enable more people to use/edit/contribute to it.

I'm happy for review / cleanup suggestion by people with more Python-foo than I have.

Architecturally, I decided to do things a bit erlang-like, where we have finite state machines in an actor models, and message passing between the actors. This is what happens with the GsmCallFsm()'s, which are created by the GsmCallConnector() representing both legs of a call and the MnccActor() that wraps the MNCC socket towards OsmoNITB.

The actual encoding/decoding of MNCC messages is auto-generated from the mncc header file #defines, enums and c-structures by means of ctypes code generation.

mncc_test.py currently drops you into a python shell where you can e.g. start more / new calls by calling functions like connect_call("7839", "3802") from that shell. Exiting the shell by quit() or Ctrl+C will terminate all call FSMs and terminate.

November 15, 2015

Harald Welte: GSM test network at 32C3, after all

Contrary to my blog post yesterday, it looks like we will have a private GSM network at the CCC congress again, after all.

It appears that Vodafone Germany (who was awarded the former DECT guard band in the 2015 spectrum auctions) is not yet using it in December, and they agreed that we can use it at the 32C3.

With this approval from Vodafone Germany we can now go to the regulator (BNetzA) and obtain the usual test license. Given that we used to get the license in the past, and that Vodafone has agreed, this should be a mere formality.

For the German language readers who appreciate the language of the administration, it will be a Frequenzzuteilung für Versuchszwecke im nichtöffentlichen mobilen Landfunk.

So thanks to Vodafone Germany, who enabled us at least this time to run a network again. By end of 2016 you can be sure they will have put their new spectrum to use, so I'm not that optimistic that this would be possible again.

November 14, 2015

Harald Welte: No GSM test network at 32C3

I currently don't assume that there will be a GSM network at the 32C3.

Ever since OpenBSC was created in 2008, the annual CCC congress was a great opportunity to test OpenBSC and related software with thousands of willing participants. In order to do so, we obtained a test licence from the German regulatory authority. This was never any problem, as there was a chunk of spectrum in the 1800 MHz GSM band that was not allocated to any commercial operator, the so-called DECT guard band. It's called that way as it was kept free in order to ensure there is no interference between 1800 MHz GSM and the neighboring DECT cordless telephones.

Over the decades, it was determined on a EU level that this guard band might not be necessary, or at least not if certain considerations are taken for BTSs deployed in that band.

When the German regulatory authority re-auctioned the GSM spectrum earlier this year, they decided to also auction the frequencies of the former DECT guard band. The DECT guard band was awarded to Vodafone.

This is a pity, as this means that people involved with cellular research or development of cellular technology now have it significantly harder to actually test their systems.

In some other EU member states it is easier, like in the Netherlands or the UK, where the DECT guard band was not treated like any other chunk of the GSM bands, but put under special rules. Not so in Germany.

To make a long story short: Without the explicit permission of any of the commercial mobile operators, it is not possible to run a test/experimental network like we used to ran at the annual CCC congress.

Given that

  • the event is held in the city center (where frequencies are typically used and re-used quite densely), and
  • an operator has nothing to gain from permitting us to test our open source GSM/GPRS implementations,

I think there is little chance that this will become a reality.

If anyone has really good contacts to the radio network planning team of a German mobile operator and wants to prove me wrong: Feel free to contact me by e-mail.

Thanks to everyone involved with the GSM team at the CCC events, particularly Holger Freyther, Daniel Willmann, Stefan Schmidt, Jan Luebbe, Peter Stuge, Sylvain Munaut, Kevin Redon, Andreas Eversberg, Ulli (and everyone else whom I may have forgot, my apologies). It's been a pleasure!

Thanks also to our friends at the POC (Phone Operation Center) who have provided interfacing to the DECT, ISDN, analog and VoIP network at the events. Thanks to roh for helping with our special patch requests. Thanks also to those entities and people who borrowed equipment (like BTSs) in the pre-sysmocom years.

So long, and thanks for all the fish!

November 07, 2015

Harald Welte: Progress on the Linux kernel GTP code

It is always sad if you start to develop some project and then never get around finishing it, as there are too many things to take care in parallel. But then, days only have 24 hours...

Back in 2012 I started to write some generic Linux kernel GTP tunneling code. GTP is the GPRS Tunneling Protocol, a protocol between core network elements in GPRS networks, later extended to be used in UMTS and even LTE networks.

GTP is split in a control plane for management and the user plane carrying the actual user IP traffic of a mobile subscriber. So if you're reading this blog via a cellular interent connection, your data is carried in GTP-U within the cellular core network.

To me as a former Linux kernel networking developer, the user plane of GTP (GTP-U) had always belonged into kernel space. It is a tunneling protocol not too different from many other tunneling protocols that already exist (GRE, IPIP, L2TP, PPP, ...) and for the user plane, all it does is basically add a header in one direction and remove the header in the other direction. User data, particularly in networks with many subscribers and/or high bandwidth use.

Also, unlike many other telecom / cellular protocols, GTP is an IP-only protocol with no E1, Frame Relay or ATM legacy. It also has nothing to do with SS7, nor does it use ASN.1 syntax and/or some exotic encoding rules. In summary, it is nothing like any other GSM/3GPP protocol, and looks much more of what you're used from the IETF/Internet world.

Unfortunately I didn't get very far with my code back in 2012, but luckily Pablo Neira (one of my colleagues from netfilter/iptables days) picked it up and brought it along. However, for some time it has been stalled until recently it was thankfully picked up by Andreas Schultz and now receives some attention and discussion, with the clear intention to finish + submit it for mainline inclusion.

The code is now kept in a git repository at http://git.osmocom.org/osmo-gtp-kernel/

Thanks to Pablo and Andreas for picking this up, let's hope this is the last coding sprint before it goes mainline and gets actually used in production.

Harald Welte: Osmocom Berlin meetings

Back in 2012, I started the idea of having a regular, bi-weekly meeting of people interested in mobile communications technology, not only strictly related to the Osmocom projects and software. This was initially called the Osmocom User Group Berlin. The meetings were held twice per month in the rooms of the Chaos Computer Club Berlin.

There are plenty of people that were or still are involved with Osmocom one way or another in Berlin. Think of zecke, alphaone, 2b-as, kevin, nion, max, prom, dexter, myself - just to name a few.

Over the years, I got "too busy" and was no longer able to attend regularly. Some people kept it alive (thanks to dexter!), but eventually they were discontinued in 2013.

Starting in October 2015, I started a revival of the meetings, two have been held already, the third is coming up next week on November 11.

I'm happy that I had the idea of re-starting the meeting. It's good to meet old friends and new people alike. Both times there actually were some new faces around, most of which even had a classic professional telecom background.

In order to emphasize the focus is strictly not on Osmocom alone ( particularly not about its users only), I decided to rename the event to the Osmocom Meeting Berlin.

If you're in Berlin and are interested in mobile communications technology on the protocol and radio side of things, feel free to join us next Wednesday.

November 02, 2015

Harald Welte: Germany's excessive additional requirements for VAT-free intra-EU shipments

Background

At my company sysmocom we are operating a small web-shop providing small tools and accessories for people interested in mobile research. This includes programmable SIM cards, SIM card protocol tracers, adapter cables, duplexers for cellular systems, GPS disciplined clock units, and other things we consider useful to people in and around the various Osmocom projects.

We of course ship domestic, inside the EU and world-wide. And that's where the trouble starts, at least since 2014.

What are VAT-free intra-EU shipments?

As many readers of this blog (at least the European ones) know, inside the EU there is a system by which intra-EU sales between businesses in EU member countries are performed without charging VAT.

This is the result of different countries having different amount of VAT, and the fact that a business can always deduct the VAT it spends on its purchases from the VAT it has to charge on its sales. In order to avoid having to file VAT return statements in each of the countries of your suppliers, the suppliers simply ship their goods without charging VAT in the first place.

In order to have checks and balances, both the supplier and the recipient have to file declarations to their tax authorities, indicating the sales volume and the EU VAT ID of the respective business partners.

So far so good. This concept was reasonably simple to implement and it makes the life easier for all involved businesses, so everyone participates in this scheme.

Of course there always have been some obstacles, particularly here in Germany. For example, you are legally required to confirm the EU-VAT-ID of the buyer before issuing a VAT-free invoice. This confirmation request can be done online

However, the Germany tax authorities invented something unbelievable: A Web-API for confirmation of EU-VAT-IDs that has opening hours. Despite this having rightfully been at the center of ridicule by the German internet community for many years, it still remains in place. So there are certain times of the day where you cannot verify EU-VAT-IDs, and thus cannot sell products VAT-free ;)

But even with that one has gotten used to live.

Gelangensbescheinigung

Now in recent years (since January 1st, 2014) , the German authorities came up with the concept of the Gelangensbescheinigung. To the German reader, this newly invented word already sounds ugly enough. Literal translation is difficult, as it sounds really clumsy. Think of something like a reaching-its-destination-certificate

So now it is no longer sufficient to simply verify the EU-VAT-ID of the buyer, issue the invoice and ship the goods, but you also have to produce such a Gelangensbescheinigung for each and every VAT-free intra-EU shipment. This document needs to include

  • the name and address of the recipient
  • the quantity and designation of the goods sold
  • the place and month when the goods were received
  • the date of when the document was signed
  • the signature of the recipient (not required in case of an e-mail where the e-mail headers show that the messages was transmitted from a server under control of the recipient)

How can you produce such a statement? Well, in the ideal / legal / formal case, you provide a form to your buyer, which he then signs and certifies that he has received the goods in the destination country.

First of all, I find if offensive that I have to ask my customers to make such declarations in the first place. And then even if I accept this and go ahead with it, it is my legal responsibility to ensure that he actually fills this in.

What if the customer doesn't want to fill it in or forgets about it?

Then I as the seller am liable to pay 19% VAT on the purchase he made, despite me never having charged those 19%.

So not only do I have to generate such forms and send them with my goods, but I also need a business process of checking for their return, reminding the customers that their form has not yet been returned, and in the end they can simply not return it and I loose money. Great.

Track+Trace / Courier Services

Now there are some alternate ways in which a Gelangensbescheinigung can be generated. For example by a track+trace protocol of the delivery company. However, the requirements to this track+trace protocol are so high, that at least when I checked in late 2013, the track and trace protocol of UPS did not fulfill the requirements. For example, a track+trace protocol usually doesn't show the quantity and designation of goods. Why would it? UPS just moves a package from A to B, and there is no customs involved that would require to know what's in the package.

Postal Packages

Now let's say you'd like to send your goods by postal service. For low-priced non-urgent goods, that's actually what you generally want to do, as everything else is simply way too expensive compared to the value of the goods.

However, this is only permitted, if the postal service you use produces you with a receipt of having accepted your package, containing the following mandatory information:

  • name and address of the entity issuing the receipt
  • name and address of the sender
  • name and address of the recipient
  • quantity and type of goods
  • date of having receive the goods

Now I don't know how this works in other countries, but in Germany you will not be able to get such a receipt form the postal office.

In fact I inquired several times with the legal department of Deutsche Post, up to the point of sending a registered letter (by Deutsche Post) to Deutsche Post. They have never responded to any of those letters!

So we have the German tax authorities claiming yes, of course you can still do intra-EU shipments to other countries by postal services, you just need to provide a receipt, but then at the same time they ask for a receipt indicating details that no postal receipt would ever show.

Particularly a postal receipt would never confirm what kind of goods you are sending. How would the postal service know? You hand them a package, and they transfer it. It is - rightfully - none of their business what its content may be. So how can you ask them to confirm that certain goods were received for transport ?!?

Summary

So in summary:

Since January 1st, 2014, we now have German tax regulations in force that make VAT free intra-EU shipments extremely difficult to impossible

  • The type of receipt they require from postal services is not provided by Deutsche Post, thereby making it impossible to use Deutsche Post for VAT free intra-EU shipments
  • The type of track+trace protocol issued by UPS does not fulfill the requirements, making it impossible to use them for VAT-free intra-EU shipments
  • The only other option is to get an actual receipt from the customer. If that customer doesn't want to provide this, the German seller is liable to pay the 19% German VAT, despite never having charged that to his customer

Conclusion

To me, the conclusion of all of this can only be one:

German tax authorities do not want German sellers to sell VAT-free goods to businesses in other EU countries. They are actively trying to undermine the VAT principles of the EU. And nobody seem to complain about it or even realize there is a problem.

What a brave new world we live in.

October 31, 2015

Harald Welte: small tools: rtl8168-eeprom

Some time ago I wrote a small Linux command line utility that can be used to (re)program the Ethernet (MAC) address stored in the EEPROM attached to an RTL8168 Ethernet chip.

This is for example useful if you are a system integrator that has its own IEEE OUI range and you would like to put your own MAC address in devices that contain the said Realtek etherent chips (already pre-programmed with some other MAC address).

The source code can be obtaned from: http://git.sysmocom.de/rtl8168-eeprom/

Harald Welte: small tools: gpsdate

In 2013 I wrote a small Linux program that can be usded to set the system clock based on the clock received from a GPS receiver (via gpsd), particularly when a system is first booted. It is similar in purpose to ntpdate, but of course obtains time not from ntp but from the GPS receiver.

This is particularly useful for RTC-less systems without network connectivity, which come up with a completely wrong system clock that needs to be properly set as soon as th GPS receiver finally has acquired a signal.

I asked the ntp hackers if they were interested in merging it into the official code base, and their response was (summarized) that with a then-future release of ntpd this would no longer be needed. So the gpsdate program remains an external utility.

So in case anyone else might find the tool interesting: The source code can be obtained from http://git.sysmocom.de/gpsdate/

October 29, 2015

Harald Welte: Deutsche Bank / unstable interfaces

Deutsche Bank is a large, international bank. They offer services world-wide and are undoubtedly proud of their massive corporate IT department.

Yet, at the same time, they fail to get the most fundamental principles of user/customer-visible interfaces wrong: Don't change them. If you need to change them, manage the change carefully.

In many software projects, keeping the API or other interface stable is paramount. Think of the Linux kernel, where breaking a userspace-visible interface is not permitted. The reasons are simple: If you break that interface, _everyone_ using that interface will need to change their implementation, and will have to synchronize that with the change on the other side of the interface.

The internet online banking system of Deutsche Bank in Germany permits the upload of transactions by their customers in a CSV file format.

And guess what? They change the file format from one day to the other.

  • without informing their users in advance, giving them time to adopt their implementations of that interface
  • without documenting the exact nature of the change
  • adding new fields to the CSV in the middle of the line, rather than at the end of the line, to make sure things break even more

Now if you're running a business and depend on automatizing your payments using the interface provided by Deutsche Bank, this means that you fail to pay your suppliers in time, you hastily drop/delay other (paid!) work that you have to do in order to try to figure out what exactly Deutsche Bank decided to change completely unannounced, from one day to the other.

If at all, I would have expected this from a hobbyist kind of project. But seriously, from one of the worlds' leading banks? An interface that is probably used by thousands and thousands of users? WTF?!?

October 28, 2015

Harald Welte: The VMware GPL case

My absence from blogging meant that I didn't really publicly comment on the continued GPL violations by VMware, and the 2015 legal case that well-known kernel developer Christoph Hellwig has brought forward against VMware.

The most recent update by the Software Freedom Conservancy on the VMware GPL case can be found at https://sfconservancy.org/news/2015/oct/28/vmware-update/

In case anyone ever doubted: I of course join the ranks of the long list of Linux developers and other stakeholders that consider VMware's behavior completely unacceptable, if not outrageous.

For many years they have been linking modified Linux kernel device drivers and entire kernel subsystems into their proprietary vmkernel software (part of ESXi). As an excuse, they have added a thin shim layer under GPLv2 which they call vmklinux. And to make all of this work, they had to add lots of vmklinux specific API to the proprietary vmkernel. All the code runs as one program, in one address space, in the same thread of execution. So basically, it is at the level of the closest possible form of integration between two pieces of code: Function calls within the same thread/process.

In order to make all this work, they had to modify their vmkernel, implement vmklinux and also heavily modify the code they took from Linux in the first place. So the drivers are not usable with mainline linux anymore, and vmklinux is not usable without vmkernel either.

If all the above is not a clear indication that multiple pieces of code form one work/program (and subsequently must be licensed under GNU GPLv2), what should ever be considered that?

To me, it is probably one of the strongest cases one can find about the question of derivative works and the GPL(v2). Of course, all my ramblings have no significance in a court, and the judge may rule based on reports of questionable technical experts. But I'm convinced if the court was well-informed and understood the actual situation here, it would have to rule in favor of Christoph Hellwig and the GPL.

What I really don't get is why VMware puts up the strongest possible defense one can imagine. Not only did they not back down in lengthy out-of-court negotiations with the Software Freedom Conservancy, but also do they defend themselves strongly against the claims in court.

In my many years of doing GPL enforcement, I've rarely seen such a dedication and strong opposition. This shows the true nature of VMware as a malicious, unfair entity that gives a damn sh*t about other peoples' copyright, the Free Software community and its code of conduct as a whole, and the Linux kernel developers in particular.

So let's hope they waste a lot of money in their legal defense, get a sufficient amount of negative PR out of this to the point of tainting their image, and finally obtain a ruling upholding the GPL.

All the best to Christoph and the Conservancy in fighting this fight. For those readers that want to help their cause, I believe they are looking for more supporter donations.

October 27, 2015

Harald Welte: What I've been busy with

Those who don't know me personally and/or stay in touch more closely might be wondering what on earth happened to Harald in the last >= 1 year?

The answer would be long, but I can summarize it to I disappeared into sysmocom. You know, the company that Holger and I founded four years ago, in order to commercially support OpenBSC and related projects, and to build products around it.

In recent years, the team has been growing to the point where in 2015 we had suddenly 9 employees and a handful of freelancers working for us.

But then, that's still a small company, and based on the projects we're involved, that team has to cover a variety of topics (next to the actual GSM/GPRS related work), including

  • mechanical engineering (enclosure design)
  • all types of electrical engineering
    • AC/electrical wiring/fusing on DIN rails
    • AC/DC and isolated DC/DC power supplies (based on modules)
    • digital design
    • analog design
    • RF design
  • prototype manufacturing and testing
  • software development
    • bare-iron bootloader/os/application on Cortex-M0
    • NuttX on Cortex-M3
    • OpenAT applications on Sierra Wireless
    • custom flavors of Linux on several different ARM architectures (TI DaVinci, TI Sitara)
    • drivers for various peripherals including Ethernet Switches, PoE PSE controller
    • lots of system-level software for management, maintenance, control

I've been involved in literally all of those topics, with most of my time spent on the electronics side than on the software side. And if software, the more on the bootloader/RTOS side, than on applications.

So what did we actually build? It's unfortunately still not possible to disclose fully at this point, but it was all related to marine communications technology. GSM being one part of it, but only one of many in the overall picture.

Given the quite challenging breadth/width of the tasks at hand and problem to solve, I'm actually surprised how much we could achieve with such a small team in a limited amount of time. But then, there's virtually no time left, which meant no gpl-violations.org work, no blogging, no progress on the various Osmocom Erlang projects for core network protocols, and last but not least no Taiwan holidays this year.

ately I see light at the end of the tunnel, and there is again a bit ore time to get back to old habits, and thus I

  • resurrected this blog from the dead
  • resurrected various project homepages that have disappeared
  • started some more work on actual telecom stuff (osmo-iuh, for example)
  • restarted the Osmocom Berlin Meeting

October 26, 2015

Harald Welte: Weblog + homepage online again

On October 31st, 2014, I had reeboote my main server for a kernel upgrade, and could not mount the LUKS crypto volume ever again. While the techincal cause for this remains a mystery until today (it has spawned some conspiracy theories), I finally took some time to recover some bits and pieces from elsewhere. I didn't want this situation to drag on for more than a year...

Rather than bringing online the old content using sub-optimal and clumsy tools to generate static content (web sites generated by docbook-xml, blog by blosxom), I decided to give it a fresh start and try nikola, a more modern and actively maintained tool to generate static web pages and blogs.

The blog is now available at http://laforge.gnumonks.org/blog/ (a redirect from the old /weblog is in place, for those who keep broken links for more than 12 months). The RSS feed URLs are different from before, but there are again per-category feeds so people (and planets) can subscribe to the respective category they're interested in.

And yes, I do plan to blog again more regularly, to make this place not just an archive of a decade of blogging, but a place that is alive and thrives with new content.

My personal web site is available at http://laforge.gnumonks.org/ while my (similarly re-vamped) freelancing business web site is also available again at http://hmw-consulting.de/.

I still need to decide what to do about the old http://gnumonks.org/ site. It still has its old manual web 1.0 structure from the late 1990ies.

I've also re-surrected http://openezx.org/ and http://ftp.gpl-devices.org/ as well as http://ftp.gnumonks.org/ (old content). Next in line is gpl-violations.org, which I also intend to convert to nikola for maintenance reasons.

October 20, 2015

Rusty Russell: ccan/mem’s memeqzero iteration

On Thursday I was writing some code, and I wanted to test if an array was all zero.  First I checked if ccan/mem had anything, in case I missed it, then jumped on IRC to ask the author (and overall CCAN co-maintainer) David Gibson about it.

We bikeshedded around names: memallzero? memiszero? memeqz? memeqzero() won by analogy with the already-extant memeq and memeqstr. Then I asked:

rusty: dwg: now, how much time do I waste optimizing?
dwg: rusty, in the first commit, none

Exactly five minutes later I had it implemented and tested.

The Naive Approach: Times: 1/7/310/37064 Bytes: 50

bool memeqzero(const void *data, size_t length)
{
    const unsigned char *p = data;

    while (length) {
        if (*p)
            return false;
        p++;
        length--;
    }
    return true;
}

As a summary, I’ve give the nanoseconds for searching through 1,8,512 and 65536 bytes only.

Another 20 minutes, and I had written that benchmark, and an optimized version.

128-byte Static Buffer: Times: 6/8/48/5872 Bytes: 108

Here’s my first attempt at optimization; using a static array of 128 bytes of zeroes and assuming memcmp is well-optimized for fixed-length comparisons.  Worse for small sizes, much better for big.

 const unsigned char *p = data;
 static unsigned long zeroes[16];

 while (length > sizeof(zeroes)) {
     if (memcmp(zeroes, p, sizeof(zeroes)))
         return false;
     p += sizeof(zeroes);
     length -= sizeof(zeroes);
 }
 return memcmp(zeroes, p, length) == 0;

Using a 64-bit Constant: Times: 12/12/84/6418 Bytes: 169

dwg: but blowing a cacheline (more or less) on zeroes for comparison, which isn’t necessarily a win

Using a single zero uint64_t for comparison is pretty messy:

bool memeqzero(const void *data, size_t length)
{
    const unsigned char *p = data;
    const unsigned long zero = 0;
    size_t pre;
    pre = (size_t)p % sizeof(unsigned long);
    if (pre) {
        size_t n = sizeof(unsigned long) - pre;
        if (n > length)
            n = length;
        if (memcmp(p, &zero, n) != 0)
            return false;
        p += n;
        length -= n;
    }
    while (length > sizeof(zero)) {
        if (*(unsigned long *)p != zero)
            return false;
        p += sizeof(zero);
        length -= sizeof(zero);
    }
    return memcmp(&zero, p, length) == 0;
}

And, worse in every way!

Using a 64-bit Constant With Open-coded Ends: Times: 4/9/68/6444 Bytes: 165

dwg: rusty, what colour is the bikeshed if you have an explicit char * loop for the pre and post?

That’s slightly better, but memcmp still wins over large distances, perhaps due to prefetching or other tricks.

Epiphany #1: We Already Have Zeroes: Times 3/5/92/5801 Bytes: 422

Then I realized that we don’t need a static buffer: we know everything we’ve already tested is zero!  So I open coded the first 16 byte compare, then memcmp()ed against the previous bytes, doubling each time.  Then a final memcmp for the tail.  Clever huh?

But it no faster than the static buffer case on the high end, and much bigger.

dwg: rusty, that is brilliant. but being brilliant isn’t enough to make things work, necessarily :p

Epiphany #2: memcmp can overlap: Times 3/5/37/2823 Bytes: 307

My doubling logic above was because my brain wasn’t completely in phase: unlike memcpy, memcmp arguments can happily overlap!  It’s still worth doing an open-coded loop to start (gcc unrolls it here with -O3), but after 16 it’s worth memcmping with the previous 16 bytes.  This is as fast as naive with as little as 2 bytes, and the fastest solution by far with larger numbers:

 const unsigned char *p = data;
 size_t len;

 /* Check first 16 bytes manually */
 for (len = 0; len < 16; len++) {
     if (!length)
         return true;
     if (*p)
         return false;
     p++;
     length--;
 }

 /* Now we know that's zero, memcmp with self. */
 return memcmp(data, p, length) == 0;

You can find the final code in CCAN (or on Github) including the benchmark code.

Finally, after about 4 hours of random yak shaving, it turns out lightning doesn’t even want to use memeqzero() any more!  Hopefully someone else will benefit.

October 01, 2015

Harald Welte: hardwear.io 2015 keynote: Running cellular infrastructure on Linux

Eric Leblond: My “Kernel packet capture technologies” talk at KR2015

I’ve just finished my talk on Linux kernel packet capture technologies at Kernel Recipes 2015. I would like to thanks the organizer for their great work. I also thank Frank Tizzoni for the drawing

regit

In that talk, I’ve tried to do an overview of the history of packet capture technologies in the Linux kernel. All that seen from userspace and from a Suricata developer perspective.

You can download the slides here: 2015_kernel_recipes_capture.

August 15, 2015

Rusty Russell: Broadband Speeds, New Data

Thanks to edmundedgar on reddit I have some more accurate data to update my previous bandwidth growth estimation post: OFCOM UK, who released their November 2014 report on average broadband speeds.  Whereas Akamai numbers could be lowered by the increase in mobile connections, this directly measures actual broadband speeds.

Extracting the figures gives:

  1. Average download speed in November 2008 was 3.6Mbit
  2. Average download speed in November 2014 was 22.8Mbit
  3. Average upload speed in November 2014 was 2.9Mbit
  4. Average upload speed in November 2008 to April 2009 was 0.43Mbit/s

So in 6 years, downloads went up by 6.333 times, and uploads went up by 6.75 times.  That’s an annual increase of 36% for downloads and 37% for uploads; that’s good, as it implies we can use download speed factor increases as a proxy for upload speed increases (as upload speed is just as important for a peer-to-peer network).

This compares with my previous post’s Akamai’s UK numbers of 3.526Mbit in Q4 2008 and 10.874Mbit in Q4 2014: only a factor of 3.08 (26% per annum).  Given how close Akamai’s numbers were to OFCOM’s in November 2008 (a year after the iPhone UK release, but probably too early for mobile to have significant effect), it’s reasonable to assume that mobile plays a large part of this difference.

If we assume Akamai’s numbers reflected real broadband rates prior to November 2008, we can also use it to extend the OFCOM data back a year: this is important since there was almost no bandwidth growth according to Akamai from Q4 2007 to Q7 2008: ignoring that period gives a rosier picture than my last post, and smells of cherrypicking data.

So, let’s say the UK went from 3.265Mbit in Q4 2007 (Akamai numbers) to 22.8Mbit in Q4 2014 (OFCOM numbers).  That’s a factor of 6.98, or 32% increase per annum for the UK. If we assume that the US Akamai data is under-representing Q4 2014 speeds by the same factor (6.333 / 3.08 = 2.056) as the UK data, that implies the US went from 3.644Mbit in Q4 2007 to 11.061 * 2.056 = 22.74Mbit in Q4 2014, giving a factor of 6.24, or 30% increase per annum for the US.

As stated previously, China is now where the US and UK were 7 years ago, suggesting they’re a reasonable model for future growth for that region.  Thus I revise my bandwidth estimates; instead of 17% per annum this suggests 30% per annum as a reasonable growth rate.

August 04, 2015

Rusty Russell: The Bitcoin Blocksize: A Summary

There’s a significant debate going on at the moment in the Bitcoin world; there’s a great deal of information and misinformation, and it’s hard to find a cogent summary in one place.  This post is my attempt, though I already know that it will cause me even more trouble than that time I foolishly entitled a post “If you didn’t run code written by assholes, your machine wouldn’t boot”.

The Technical Background: 1MB Block Limit

The bitcoin protocol is powered by miners, who gather transactions into blocks, producing a block every 10 minutes (but it varies a lot).  They get a 25 bitcoin subsidy for this, plus whatever fees are paid by those transactions.  This subsidy halves every 4 years: in about 12 months it will drop to 12.5.

Full nodes on the network check transactions and blocks, and relay them to others.  There are also lightweight nodes which simply listen for transactions which affect them, and trust that blocks from miners are generally OK.

A normal transaction is 250 bytes, and there’s a hard-coded 1 megabyte limit on the block size.  This limit was introduced years ago as a quick way of avoiding a miner flooding the young network, though the original code could only produce 200kb blocks, and the default reference code still defaults to a 750kb limit.

In the last few months there have been increasing runs of full blocks, causing backlogs for a few hours.  More recently, someone deliberately flooded the network with normal-fee transactions for several days; any transactions paying less fees than those had to wait for hours to be processed.

There are 5 people who have commit access to the bitcoin reference implementation (aka. “bitcoin-core”), and they vary significantly in their concerns on the issue.

The Bitcoin Users’ Perspective

From the bitcoin users perspective, blocks should be infinite, and fees zero or minimal.  This is the basic position of respected (but non-bitcoin-core) developer Mike Hearn, and has support from bitcoin-core ex-lead Gavin Andresen.  They work on the wallet and end-user side of bitcoin, and they see the issue as the most urgent.  In an excellent post arguing why growth is so important, Mike raises the following points, which I’ve paraphrased:

  1. Currencies have network effects. A currency that has few users is simply not competitive with currencies that have many.
  2. A decentralised currency that the vast majority can’t use doesn’t change the amount of centralisation in the world. Most people will still end up using banks, with all the normal problems.
  3. Growth is a part of the social contract. It always has been.
  4. Businesses will only continue to invest in bitcoin and build infrastructure if they are assured that the market will grow significantly.
  5. Bitcoin needs users, lots of them, for its political survival. There are many people out there who would like to see digital cash disappear, or be regulated out of existence.

At this point, it’s worth mentioning another bitcoin-core developer: Jeff Garzik.  He believes that the bitcoin userbase has been promised that transactions will continue to be almost free.  When a request to change the default mining limit from 750kb to 1M was closed by the bitcoin lead developer Wladimir van der Laan as unimportant, Jeff saw this as a symbolic moment:

What Happens If We Don’t Increase Soon?

Mike Hearn has a fairly apocalyptic view of what would happen if blocks fill.  That was certainly looking likely when the post was written, but due to episodes where the blocks were full for days, wallet designers are (finally) starting to estimate fees for timely processing (miners process larger fee transactions first).  Some wallets and services didn’t even have a way to change the setting, leaving users stranded during high-volume events.

It now seems that the bursts of full blocks will arrive with increasing frequency; proposals are fairly mature now to allow users to post-increase fees if required, which (if all goes well) could make for a fairly smooth transition from the current “fees are tiny and optional” mode of operation to a “there will be a small fee”.

But even if this rosy scenario is true, this begsavoids the bigger question of how high fees can become before bitcoin becomes useless.  1c?  5c?  20c? $1?

So What Are The Problems With Increasing The Blocksize?

In a word, the problem is miners.  As mining has transitioned from a geek pastime, semi-hobbyist, then to large operations with cheap access to power, it has become more concentrated.

The only difference between bitcoin and previous cryptocurrencies is that instead of a centralized “broker” to ensure honesty, bitcoin uses an open competition of miners. Given bitcoin’s endurance, it’s fair to count this a vital property of bitcoin.  Mining centralization is the long-term concern of another bitcoin-core developer (and my coworker at Blockstream), Gregory Maxwell.

Control over half the block-producing power and you control who can use bitcoin and cheat anyone not using a full node themselves.  Control over 2/3, and you can force a rule change on the rest of the network by stalling it until enough people give in.  Central control is also a single point to shut the network down; that lets others apply legal or extra-legal pressure to restrict the network.

What Drives Centralization?

Bitcoin mining is more efficient at scale. That was to be expected[7]. However, the concentration has come much faster than expected because of the invention of mining pools.  These pools tell miners what to mine, in return for a small (or in some cases, zero) share of profits.  It saves setup costs, they’re easy to use, and miners get more regular payouts.  This has caused bitcoin to reel from one centralization crisis to another over the last few years; the decline in full nodes has been precipitous by some measures[5] and continues to decline[6].

Consider the plight of a miner whose network is further away from most other miners.  They find out about new blocks later, and their blocks get built on later.  Both these effects cause them to create blocks which the network ignores, called orphans.  Some orphans are the inevitable consequence of miners racing for the same prize, but the orphan problem is not symmetrical.  Being well connected to the other miners helps, but there’s a second effect: if you discover the previous block, you’ve a head-start on the next one.  This means a pool which has 20% of the hashing power doesn’t have to worry about delays at all 20% of the time.

If the orphan rate is very low (say, 0.1%), the effect can be ignored.  But as it climbs, the pressure to join a pool (the largest pool) becomes economically irresistible, until only one pool remains.

Larger Blocks Are Driving Up Orphan Rates

Large blocks take longer to propagate, increasing the rate of orphans.  This has been happening as blocks increase.  Blocks with no transactions at all are smallest, and so propagate fastest: they still get a 25 bitcoin subsidy, though they don’t help bitcoin users much.

Many people assumed that miners wouldn’t overly centralize, lest they cause a clear decentralization failure and drive the bitcoin price into the ground.  That assumption has proven weak in the face of climbing orphan rates.

And miners have been behaving very badly.  Mining pools orchestrate attacks on each other with surprising regularity; DDOS and block withholding attacks are both well documented[1][2].  A large mining pool used their power to double spend and steal thousands of bitcoin from a gambling service[3].  When it was noticed, they blamed a rogue employee.  No money was returned, nor any legal action taken.  It was hoped that miners would leave for another pool as they approached majority share, but that didn’t happen.

If large blocks can be used as a weapon by larger miners against small ones[8], it’s expected that they will be.

More recently (and quite by accident) it was discovered that over half the mining power aren’t verifying transactions in blocks they build upon[4].  They did this in order to reduce orphans, and one large pool is still doing so.  This is a problem because lightweight bitcoin clients work by assuming anything in the longest chain of blocks is good; this was how the original bitcoin paper anticipated that most users would interact with the system.

The Third Side Of The Debate: Long Term Network Funding

Before I summarize, it’s worth mentioning the debate beyond the current debate: long term network support.  The minting of new coins decreases with time; the plan of record (as suggested in the original paper) is that total transaction fees will rise to replace the current mining subsidy.  The schedule of this is unknown and generally this transition has not happened: free transactions still work.

The block subsidy as I write this is about $7000.  If nothing else changes, miners would want $3500 in fees in 12 months when the block subsidy halves, or about $2 per transaction.  That won’t happen; miners will simply lose half their income.  (Perhaps eventually they form a cartel to enforce a minimum fee, causing another centralization crisis? I don’t know.)

It’s natural for users to try to defer the transition as long as possible, and the practice in bitcoin-core has been to aggressively reduce the default fees as the bitcoin price rises.  Core developers Gregory Maxwell and Pieter Wuille feel that signal was a mistake; that fees will have to rise eventually and users should not be lulled into thinking otherwise.

Mike Hearn in particular has been holding out the promise that it may not be necessary.  On this he is not widely supported: that some users would offer to pay more so other users can continue to pay less.

It’s worth noting that some bitcoin businesses rely on the current very low fees and don’t want to change; I suspect this adds bitterness and vitriol to many online debates.

Summary

The bitcoin-core developers who deal with users most feel that bitcoin needs to expand quickly or die, that letting fees emerge now will kill expansion, and that the infrastructure will improve over time if it has to.

Other bitcoin-core developers feel that bitcoin’s infrastructure is dangerously creaking, that fees need to emerge anyway, and that if there is a real emergency a blocksize change could be rolled out within a few weeks.

At least until this is resolved, don’t count on future bitcoin fees being insignificant, nor promise others that bitcoin has “free transactions”.


[1] http://www.coindesk.com/bitcoin-mining-pools-ddos-attacks/

“Bitcoin Mining Pools Targeted in Wave of DDOS Attacks” Coinbase 2015

[2] http://blog.bettercrypto.com/?p=1131 “Block Withholding Attacks – Recent Research” N T Courtois 2014

[3] https://bitcointalk.org/index.php?topic=327767.0 “GHash.IO and double-spending against BetCoin Dice” mmtech et. al 2013

[4] https://www.reddit.com/r/Bitcoin/comments/3c8tq2/questions_about_the_july_4th_bip66_fork/cstgajp “Questions about the July 4th BIP66 fork”

[5] https://twitter.com/petertoddbtc/status/608475414449778688 “350,000 full nodes to 6,000 in two years…” P Todd 2015

[6] https://getaddr.bitnodes.io/dashboard/?days=365 “Reachable nodes during the last 365 days.” Bitnodes.io

[7] https://bitcointalk.org/index.php?topic=532.msg6306#msg6306 “Re: Scalability and transaction rate” Satoshi 2010

[8] http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg08161.html “[Bitcoin-development] Mining centralization pressure from non-uniform propagation speed” Pieter Wuille 2015

July 08, 2015

Rusty Russell: The Megatransaction: Why Does It Take 25 Seconds?

Last night f2pool mined a 1MB block containing a single 1MB transaction.  This scooped up some of the spam which has been going to various weakly-passworded “brainwallets”, gaining them 0.5569 bitcoins (on top of the normal 25 BTC subsidy).  You can see the megatransaction on blockchain.info.

It was widely reported to take about 25 seconds for bitcoin core to process this block: this is far worse than my “2 seconds per MB” result in my last post, which was considered a pretty bad case.  Let’s look at why.

How Signatures Are Verified

The algorithm to check a transaction input (of this form) looks like this:

  1. Strip the other inputs from the transaction.
  2. Replace the input script we’re checking with the script of the output it’s trying to spend.
  3. Hash the resulting transaction with SHA256, then hash the result with SHA256 again.
  4. Check the signature correctly signed that hash result.

Now, for a transaction with 5570 inputs, we have to do this 5570 times.  And the bitcoin core code does this by making a copy of the transaction each time, and using the marshalling code to hash that; it’s not a huge surprise that we end up spending 20 seconds on it.

How Fast Could Bitcoin Core Be If Optimized?

Once we strip the inputs, the result is only about 6k long; hashing 6k 5570 times takes about 265 milliseconds (on my modern i3 laptop).  We have to do some work to change the transaction each time, but we should end up under half a second without any major backflips.

Problem solved?  Not quite….

This Block Isn’t The Worst Case (For An Optimized Implementation)

As I said above, the amount we have to hash is about 6k; if a transaction has larger outputs, that number changes.  We can fit in fewer inputs though.  A simple simulation shows the worst case for 1MB transaction has 3300 inputs, and 406000 byte output(s): simply doing the hashing for input signatures takes about 10.9 seconds.  That’s only about two or three times faster than the bitcoind naive implementation.

This problem is far worse if blocks were 8MB: an 8MB transaction with 22,500 inputs and 3.95MB of outputs takes over 11 minutes to hash.  If you can mine one of those, you can keep competitors off your heels forever, and own the bitcoin network… Well, probably not.  But there’d be a lot of emergency patching, forking and screaming…

Short Term Steps

An optimized implementation in bitcoind is a good idea anyway, and there are three obvious paths:

  1. Optimize the signature hash path to avoid the copy, and hash in place as much as possible.
  2. Use the Intel and ARM optimized SHA256 routines, which increase SHA256 speed by about 80%.
  3. Parallelize the input checking for large numbers of inputs.

Longer Term Steps

A soft fork could introduce an OP_CHECKSIG2, which hashes the transaction in a different order.  In particular, it should hash the input script replacement at the end, so the “midstate” of the hash can be trivially reused.  This doesn’t entirely eliminate the problem, since the sighash flags can require other permutations of the transaction; these would have to be carefully explored (or only allowed with OP_CHECKSIG).

This soft fork could also place limits on how big an OP_CHECKSIG-using transaction could be.

Such a change will take a while: there are other things which would be nice to change for OP_CHECKSIG2, such as new sighash flags for the Lightning Network, and removing the silly DER encoding of signatures.

July 06, 2015

Rusty Russell: Bitcoin Core CPU Usage With Larger Blocks

Since I was creating large blocks (41662 transactions), I added a little code to time how long they take once received (on my laptop, which is only an i3).

The obvious place to look is CheckBlock: a simple 1MB block takes a consistent 10 milliseconds to validate, and an 8MB block took 79 to 80 milliseconds, which is nice and linear.  (A 17MB block took 171 milliseconds).

Weirdly, that’s not the slow part: promoting the block to the best block (ActivateBestChain) takes 1.9-2.0 seconds for a 1MB block, and 15.3-15.7 seconds for an 8MB block.  At least it’s scaling linearly, but it’s just slow.

So, 16 Seconds Per 8MB Block?

I did some digging.  Just invalidating and revalidating the 8MB block only took 1 second, so something about receiving a fresh block makes it worse. I spent a day or so wrestling with benchmarking[1]…

Indeed, ConnectTip does the actual script evaluation: CheckBlock() only does a cursory examination of each transaction.  I’m guessing bitcoin core is not smart enough to parallelize a chain of transactions like mine, hence the 2 seconds per MB.  On normal transaction patterns even my laptop should be about 4 times faster than that (but I haven’t actually tested it yet!).

So, 4 Seconds Per 8MB Block?

But things are going to get better: I hacked in the currently-disabled libsecp256k1, and the time for the 8MB ConnectTip dropped from 18.6 seconds to 6.5 seconds.

So, 1.6 Seconds Per 8MB Block?

I re-enabled optimization after my benchmarking, and the result was 4.4 seconds; that’s libsecp256k1, and an 8MB block.

Let’s Say 1.1 Seconds for an 8MB Block

This is with some assumptions about parallelism; and remember this is on my laptop which has a fairly low-end CPU.  While you may not be able to run a competitive mining operation on a Raspberry Pi, you can pretty much ignore normal verification times in the blocksize debate.


 

[1] I turned on -debug=bench, which produced impenetrable and seemingly useless results in the log.

So I added a print with a sleep, so I could run perf.  Then I disabled optimization, so I’d get understandable backtraces with perf.  Then I rebuilt perf because Ubuntu’s perf doesn’t demangle C++ symbols, which is part of the kernel source package. (Are we having fun yet?).  I even hacked up a small program to help run perf on just that part of bitcoind.   Finally, after perf failed me (it doesn’t show 100% CPU, no idea why; I’d expect to see main in there somewhere…) I added stderr prints and ran strace on the thing to get timings.

July 03, 2015

Rusty Russell: Wrapper for running perf on part of a program.

Linux’s perf competes with early git for title of least-friendly Linux tool.  Because it’s tied to kernel versions, and the interfaces changes fairly randomly, you can never figure out how to use the version you need to use (hint: always use -g).

But when it works, it’s very useful.  Recently I wanted to figure out where bitcoind was spending its time processing a block; because I’m a cool kid, I didn’t use gprof, I used perf.  The problem is that I only want information on that part of bitcoind.  To start with, I put a sleep(30) and a big printf in the source, but that got old fast.

Thus, I wrote “perfme.c“.  Compile it (requires some trivial CCAN headers) and link perfme-start and perfme-stop to the binary.  By default it runs/stops perf record on its parent, but an optional pid arg can be used for other things (eg. if your program is calling it via system(), the shell will be the parent).

June 25, 2015

Rusty Russell: Hashing Speed: SHA256 vs Murmur3

So I did some IBLT research (as posted to bitcoin-dev ) and I lazily used SHA256 to create both the temporary 48-bit txids, and from them to create a 16-bit index offset.  Each node has to produce these for every bitcoin transaction ID it knows about (ie. its entire mempool), which is normally less than 10,000 transactions, but we’d better plan for 1M given the coming blopockalypse.

For txid48, we hash an 8 byte seed with the 32-byte txid; I ignored the 8 byte seed for the moment, and measured various implementations of SHA256 hashing 32 bytes on on my Intel Core i3-5010U CPU @ 2.10GHz laptop (though note we’d be hashing 8 extra bytes for IBLT): (implementation in CCAN)

  1. Bitcoin’s SHA256: 527.7+/-0.9 nsec
  2. Optimizing the block ending on bitcoin’s SHA256: 500.4+/-0.66 nsec
  3. Intel’s asm rorx: 314.1+/-0.3 nsec
  4. Intel’s asm SSE4 337.5+/-0.5 nsec
  5. Intel’s asm RORx-x8ms 458.6+/-2.2 nsec
  6. Intel’s asm AVX 336.1+/-0.3 nsec

So, if you have 1M transactions in your mempool, expect it to take about 0.62 seconds of hashing to calculate the IBLT.  This is too slow (though it’s fairly trivially parallelizable).  However, we just need a universal hash, not a cryptographic one, so I benchmarked murmur3_x64_128:

  1. Murmur3-128: 23 nsec

That’s more like 0.046 seconds of hashing, which seems like enough of a win to add a new hash to the mix.

June 19, 2015

Rusty Russell: Mining on a Home DSL connection: latency for 1MB and 8MB blocks

I like data.  So when Patrick Strateman handed me a hacky patch for a new testnet with a 100MB block limit, I went to get some.  I added 7 digital ocean nodes, another hacky patch to prevent sendrawtransaction from broadcasting, and a quick utility to create massive chains of transactions/

My home DSL connection is 11Mbit down, and 1Mbit up; that’s the fastest I can get here.  I was CPU mining on my laptop for this test, while running tcpdump to capture network traffic for analysis.  I didn’t measure the time taken to process the blocks on the receiving nodes, just the first propagation step.

1 Megabyte Block

Naively, it should take about 10 seconds to send a 1MB block up my DSL line from first packet to last.  Here’s what actually happens, in seconds for each node:

  1. 66.8
  2. 70.4
  3. 71.8
  4. 71.9
  5. 73.8
  6. 75.1
  7. 75.9
  8. 76.4

The packet dump shows they’re all pretty much sprayed out simultaneously (bitcoind may do the writes in order, but the network stack interleaves them pretty well).  That’s why it’s 67 seconds at best before the first node receives my block (a bit longer, since that’s when the packet left my laptop).

8 Megabyte Block

I increased my block size, and one node dropped out, so this isn’t quite the same, but the times to send to each node are about 8 times worse, as expected:

  1. 501.7
  2. 524.1
  3. 536.9
  4. 537.6
  5. 538.6
  6. 544.4
  7. 546.7

Conclusion

Using the rough formula of 1-exp(-t/600), I would expect orphan rates of 10.5% generating 1MB blocks, and 56.6% with 8MB blocks; that’s a huge cut in expected profits.

Workarounds

  • Get a faster DSL connection.  Though even an uplink 10 times faster would mean 1.1% orphan rate with 1MB blocks, or 8% with 8MB blocks.
  • Only connect to a single well-connected peer (-maxconnections=1), and hope they propagate your block.
  • Refuse to mine any transactions, and just collect the block reward.  Doesn’t help the bitcoin network at all though.
  • Join a large pool.  This is what happens in practice, but raises a significant centralization problem.

Fixes

  • We need bitcoind to be smarter about ratelimiting in these situations, and stream serially.  Done correctly (which is hard), it could also help bufferbloat which makes running a full node at home so painful when it propagates blocks.
  • Some kind of block compression, along the lines of Gavin’s IBLT idea. I’ve done some preliminary work on this, and it’s promising, but far from trivial.

 

June 03, 2015

Rusty Russell: What Transactions Get Crowded Out If Blocks Fill?

What happens if bitcoin blocks fill?  Miners choose transactions with the highest fees, so low fee transactions get left behind.  Let’s look at what makes up blocks today, to try to figure out which transactions will get “crowded out” at various thresholds.

Some assumptions need to be made here: we can’t automatically tell the difference between me taking a $1000 output and paying you 1c, and me paying you $999.99 and sending myself the 1c change.  So my first attempt was very conservative: only look at transactions with two or more outputs which were under the given thresholds (I used a nice round $200 / BTC price throughout, for simplicity).

(Note: I used bitcoin-iterate to pull out transaction data, and rebuild blocks without certain transactions; you can reproduce the csv files in the blocksize-stats directory if you want).

Paying More Than 1 Person Under $1 (< 500000 Satoshi)

Here’s the result (against the current blocksize):

Sending 2 Or More Sub-$1 Outputs

Let’s zoom in to the interesting part, first, since there’s very little difference before 220,000 (February 2013).  You can see that only about 18% of transactions are sending less than $1 and getting less than $1 in change:

Since March 2013…

Paying Anyone Under 1c, 10c, $1

The above graph doesn’t capture the case where I have $100 and send you 1c.   If we eliminate any transaction which has any output less than various thresholds, we’ll catch that. The downside is that we capture the “sending myself tiny change” case, but I’d expect that to be rarer:

Blocksizes Without Small Output Transactions

This eliminates far more transactions.  We can see only 2.5% of the block size is taken by transactions with 1c outputs (the dark red line following the block “current blocks” line), but the green line shows about 20% of the block used for 10c transactions.  And about 45% of the block is transactions moving $1 or less.

Interpretation: Hard Landing Unlikely, But Microtransactions Lose

If the block size doesn’t increase (or doesn’t increase in time): we’ll see transactions get slower, and fees become the significant factor in whether your transaction gets processed quickly.  People will change behaviour: I’m not going to spend 20c to send you 50c!

Because block finding is highly variable and many miners are capping blocks at 750k, we see backlogs at times already; these bursts will happen with increasing frequency from now on.  This will put pressure on Satoshdice and similar services, who will be highly incentivized to use StrawPay or roll their own channel mechanism for off-blockchain microtransactions.

I’d like to know what timescale this happens on, but the graph shows that we grow (and occasionally shrink) in bursts.  A logarithmic graph prepared by Peter R of bitcointalk.org suggests that we hit 1M mid-2016 or so; expect fee pressure to bend that graph downwards soon.

The bad news is that even if fees hit (say) 25c and that prevents all the sub-$1 transactions, we only double our capacity, giving us perhaps another 18 months. (At that point miners are earning $1000 from transaction fees as well as $5000 (@ $200/BTC) from block reward, which is nice for them I guess.)

My Best Guess: Larger Blocks Desirable Within 2 Years, Needed by 3

Personally I think 5c is a reasonable transaction fee, but I’d prefer not to see it until we have decentralized off-chain alternatives.  I’d be pretty uncomfortable with a 25c fee unless the Lightning Network was so ubiquitous that I only needed to pay it twice a year.  Higher than that would have me reaching for my credit card to charge my Lightning Network account :)

Disclaimer: I Work For BlockStream, on Lightning Networks

Lightning Networks are a marathon, not a sprint.  The development timeframes in my head are even vaguer than the guesses above.  I hope it’s part of the eventual answer, but it’s not the bandaid we’re looking for.  I wish it were different, but we’re going to need other things in the mean time.

I hope this provided useful facts, whatever your opinions.

Rusty Russell: Current Blocksize, by graphs.

I used bitcoin-iterate and gnumeric to render the current bitcoin blocksizes, and here are the results.

My First Graph: A Moment of Panic

This is block sizes up to yesterday; I’ve asked gnumeric to derive an exponential trend line from the data (in black; the red one is linear)

Woah! We hit 1M blocks in a month! PAAAANIC!

That trend line hits 1000000 at block 363845.5, which we’d expect in about 32 days time!  This is what is freaking out so many denizens of the Bitcoin Subreddit. I also just saw a similar inaccurate [correction: misleading] graph reshared by Mike Hearn on G+ :(

But Wait A Minute

That trend line says we’re on 800k blocks today, and we’re clearly not.  Let’s add a 6 hour moving average:

Oh, we’re only halfway there….

In fact, if we cluster into 36 blocks (ie. 6 hours worth), we can see how misleading the terrible exponential fit is:

What! We’re already over 1M blocks?? Maths, you lied to me!

Clearer Graphs: 1 week Moving Average

Actual Weekly Running Average Blocksize

So, not time to panic just yet, though we’re clearly growing, and in unpredictable bursts.

Copyright (C) 2001-2010 by the respective authors.