A while ago I said Koen from Emweb made an interesting proposal at FOSDEM about emerge, the KDE Windows build tool.

Yesterday, Jarosław Staniek and I reaffirmed our commitment to ’emerge’. Today, I’d like to go a bit further: let’s bring more developers to emerge by opening it up to other projects. Keep reading!

What is emerge, why is it important and what was Koen’s proposal?

Fact: Microsoft Windows is very different to Unix in regards to development.

On Unix platforms -that includes Linux and Mac OS X-, software is usually installed to /usr: applications in /usr/bin and /usr/sbin, libraries in /usr/lib, headers in /usr/include, common resources in /usr/share, etc. Also, dependency management is usually something you can count on: when you install kdelibs5-dev in Ubuntu, it will automatically install libqt4-dev, kdelibs5-data, libfreetype (runtime), etc That makes setting up a development environment a very easy task: look for shared libraries, header files, etc in the common places and you will probably find them.

On Windows there is nothing like that. When you want to compile an application, you need to provide (build and install) all its dependencies, and you need to tell Visual Studio where to find everything. Even CMake usually needs some help in the form of a hint for CMAKE_PREFIX_PATH. As you may imagine, building KDE, which has more than 200 third party dependencies and tens of modules (and with the move + split to git, many more) becomes an almost insurmountable task.

‘Emerge’ to the rescue: inspired by Gentoo‘s emerge, Ralf Habacker, Christian Ehrlicher, Patrick Spendrin and others (yours faithfully included) developed a tool which downloads the source, configures, builds, installs and packages KDE and its dependencies. It makes a world of difference when building KDE. Actually, it makes building KDE on Windows possible. Once more: thank you very much guys, impressive tool.

There are two well-differentiated parts in emerge, the ‘engine’ and the ‘recipes’.

The ‘engine‘ takes care of downloading the sources (tarballs, checkouts from Subversion and git clones, etc) , configuring and building for the usual build systems (QMake, CMake, NMake makefiles, etc), installing, bundling packages together and much more. All that for four compilers (MSVC2008, MSVC2010, MinGW32 and MinGW-W64). It’s the equivalent to Gentoo‘s emerge, Debian‘s dpkg + debhelper/cdbs or Homebrew and MacPorts for Macintosh.

The ‘recipes’ are the package-specific “instructions” a particular piece of software needs to build: what’s the name of the tarball? URL to download from? What versions are available? What build system to use to configure and build? Does it need any Windows-specific patch? They are written in Python using some helper modules ’emerge’ provides (the debhelper-like part I mentioned above).

Currently, there are about 70 recipes in emerge, organized in our ‘portage tree’ (bear in mind the names are taken from Gentoo but the internals of the tool are completely different). With those 70 recipes, we are able to build most KDE modules. Problem is to provide all the optional features, we are missing probably 200 to 250 more recipes. Given that KDE on Windows is quite short on developers, we have to decide: either we fix bugs and improve the integration of KDE on Windows, or we keep track of the dependencies. I won’t lie: for KDE on Windows, I’d rather focus on development than on packaging.

Given that writing and maintaining recipes does not strictly require programming skills (although they help 🙂 ), during my talk at FOSDEM I kept asking people to join the KDE on Windows team as packagers, even if you know little to nothing about software development. If you think about it, that’s what we have in Linux distributions: there are hundreds (thousands?) of packagers in Debian, Fedora, openSuse, etc which take care of the ‘recipes’ to build software developed by someone else (‘upstream’, in the parlance). What we have in KDE on Windows, where we are the KDE packagers, the KDE developers (upstream) and the emerge developers, is actually an anomaly.

Then Koen took the microphone and said: why don’t you open emerge to other projects?

And I there I realized he was damn right

We have developed an awesome tool to download, configure, build, package and install third party software on Windows, to manage dependencies, to update, etc. For four compilers, no less. The ‘engine’ of emerge is not really tied to KDE. We already have a lot of recipes for software which is not KDE specific: openssl, Qt, bzip2, libpng, mysql, etc.

I see no actual reason to disallow Gnome, GStreamer, OpenSceneGraph, LibreOffice and many others in (assuming someone wants to take care of them, of course – no, do not look at me! I don’t have time!). Currently, the only ‘barrier’ preventing those recipes in is ’emerge’ is developed in the KDE repository and you need a KDE developer account to commit your recipes. That’s about it.

There is also a ‘perception’ problem: “if emerge is developed in KDE, it’s because it’s a KDE-thingy, right?”. Well, no. One way to avoid that would be to graduate emerge from KDE and make it an independent project, one in which the current developers have accounts, and new developers get an “emerge project” account instead of a KDE account to commit recipes. Please note “graduating” does not mean “expelling”, “firing” or anything peyorative. It does not mean that ’emerge developers’ (engine and recipes) are worse than KDE developers, or less KDE developers than our god David Faure: in KDE, we consider Debian, Fedora, openSuse, etc KDE packagers as equals, they are also KDE developers and many of them are in the KDE eV (funny thing is I am not).

My wish for today: think of one application or library you’d like to see in Windows and become its maintainer in emerge for Windows.

For now, let’s do this in the kde-windows mailing list and if this idea succeeds, then we’ll talk about graduating. You could also try to add support for a new compiler (OpenWatcom, Intel C++, etc) but that’s a lot more more and it’s not a priority now.

PS: Yes, I know about CoApp, Microsoft‘s similar effort, with some magic involved. I’ve been watching and trying it since day 1. I’m really interested. It’s just not there yet and I do not have spare time to help Garrett (hey Microsoft, hire me and I’ll have all the time in the world! 🙂 )

8 Thoughts on “A wish a day 7: make emerge a generic package manager for Windows

  1. Always when I see this kind of “porting stuff to Windows” I think… WTF did they finish the Linux part?

  2. Keith Rusler on Thursday 31st March 2011 at 21:11:05 said:

    While it would be nice to have plasma, kdevelop, etc on Windows. One of the main reasons I quit bothering with it, was simply because of the kde emerge tool.

    It is so hard to work with. It’s hard to actually get it to run good with 64bit Windows due to C:\Program Files (x86)\ (since that is where Visual Studios installs it’s crap >.<.

    Also, it's even more painful due to it's installing binaries, when I only care about compiling. If you really want to make it like Gentoo's portage, please do whatever it takes to actually port Gentoo's portage or use the same like commands they use and have a make.conf that uses the use flags etc.

    One last thing, it was still using really outdated packages like Boost-1.34.0 when there was 1.42.0. A lot of the packages (3rd party) are quite outdated. Be nice to have the unmasking features, etc for people who want the latest packages.

  3. jospoortvliet on Friday 01st April 2011 at 02:47:45 said:

    Now I’m an openSUSE dude so maybe I’m biassed, but wouldn’t using the Open Build Service be far more convenient than some bunch of inflexible scripts? OBS is build for exactly this, has experimental windows support already – and is far easier. Building packages in a clean VM, controlled from a convenient web interface or a powerful command line tool – seems a better solution to me. And ANYONE can maintain an app that way, or make a fork, fix things and do a merge request back etcetera – OBS has all the needed collaboration tools for that. build.opensuse.org already has almost 30.000 users building 120.000 packages for 21 distro’s on 6 architectures – once windows support is up, it’s just one among many and building also for windows won’t be much harder than selecting it as build target… That is, if your app compiles properly 😉

    Now this tool has a perception issue too, being made by openSUSE, but the fact that projects like VLC and MeeGo use it signals that it’s already slightly further spread than ’emerge for windows’. Yes, VLC uses it to build windows and afaik Mac OS X packages, too, and so can KDE. MeeGo builds for ARM, yes, indeed, and x86 as well. Oh and the code is on gitorious and you can download a fully ready-to-run OS image with OBS : http://susegallery.com/a/psNYmd/obs-215

    I’m not saying this will be ready to build all of KDE in a few hours. But some KDE apps are already build with this, several projects use it to build Windows apps too – and it’s capable of building whole linux distro’s so it surely can build all of KDE and its dependencies. As I said, windows support is experimental, help, testing and patches more than welcome. Imho a better long-term solution than obscure commandline Emerge like tools – it is far easier to use and much easier to get contributors for the building-KDE-for-windows efforts! Anyone with a web browser could contribute – instead of having to install it on his/her own machine – so yes, even those ONLY using windows can contribute, AND those ONLY using MAC OS X, or MeeGo, or Debian, or Haiku for that matter 😀

    Enjoy 😉

  4. This is not (only) about compiling/building, this is also about package management on Windows, and I doubt that the OpenSUSE build service can help there.

    And I, for one, find the emerge tool pretty easy to handle. I don’t know when Keith tried it but it certainly does have a recent version of the boost library (1.44 as of today). I can imagine that it was back when emerge was pretty new and the devs wanted to get it up and running with versions of the libraries that they knew they worked with the specific KDE stuff they were trying to compile at this time.

    While it’s true that the build service alone could be handled pretty well with other build services, that is only the smallest part of the problem. Fixing the packages for Win32 bugs, Microsoft compilers, and writing CMake scripts (because a stock Windows doesn’t have any POSIX shell to run autoconf for example) are the most important things. Actually building the packages is easy as soon as you made these preparations.

    -Darkstar

  5. I looked at he linked emerge web page and again and again and again, KDE folks do the same mistake: you have a nice subversion web frontend but all you give is a svn:// URL. WHY?
    Why is there no link to http://websvn.kde.org/trunk/kdesupport/emerge/ on that webpage?

    Many of the information on this site belongs into an INSTALL file in the sources. BTW: giving the README, etc. a .txt ending helps a LOT on Windows!

  6. Mike Lothian on Friday 01st April 2011 at 08:42:14 said:

    I was just wondering if you considered using the real Gentoo’s portage and ebuilds (modified of course) using the prefix options – I know people use this on OSX

    It might be more worthwhile adding windows support to prefix than having to maintain your own “portage tree”

  7. Keith Rusler on Friday 01st April 2011 at 09:19:09 said:

    It was about 6 months or so ago when I last tried it, then gave up when I couldn’t really get it working with 64bit Windows. But I will give it a try again. But I do blame most of this on Microsoft’s part. Because they do not make the system Programmer friendly.

  8. While waiting for my flight at Schiphol Airport today I started writing an emerge-recipe for libsdl, complete with CMakeLists.txt file (although that one is currently hardcoded to Windows+DirectX). Not really hard (it was actually my first fully self-created CMake recipe and my first ever python code, and I must say, it’s rather straightforward ;-))

    As I don’t have my KDE svn keys on this laptop I will not be able to commit until tomorrow or so, but at least I now know it’s really easy to develop for emerge

    -Darkstar

Leave a Reply

Your email address will not be published. Required fields are marked *

 

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Post Navigation