Libraries and link interfaces

by ⋅ Leave a Comment

Today I received an e-mail from Vincent about some strange warnings dpkg-shlibdeps was showing when building from source the Wt packages on Debian:

dpkg-shlibdeps: warning: symbol pthread_join used by debian/witty/usr/lib/libwthttp.so.2.2.4 found in none of the libraries.
dpkg-shlibdeps: warning: symbol pthread_key_delete used by debian/witty/usr/lib/libwthttp.so.2.2.4 found in none of the libraries.
dpkg-shlibdeps: warning: symbol pthread_mutexattr_init used by debian/witty/usr/lib/libwthttp.so.2.2.4 found in none of the libraries.
dpkg-shlibdeps: warning: symbol pthread_setspecific used by debian/witty/usr/lib/libwthttp.so.2.2.4 found in none of the libraries.
dpkg-shlibdeps: warning: symbol pthread_mutexattr_settype used by debian/witty/usr/lib/libwthttp.so.2.2.4 found in none of the libraries.
dpkg-shlibdeps: warning: symbol pthread_detach used by debian/witty/usr/lib/libwthttp.so.2.2.4 found in none of the libraries.
dpkg-shlibdeps: warning: symbol pthread_mutexattr_destroy used by debian/witty/usr/lib/libwthttp.so.2.2.4 found in none of the libraries.
dpkg-shlibdeps: warning: symbol pthread_getspecific used by debian/witty/usr/lib/libwthttp.so.2.2.4 found in none of the libraries.
dpkg-shlibdeps: warning: symbol pthread_key_create used by debian/witty/usr/lib/libwthttp.so.2.2.4 found in none of the libraries.
dpkg-shlibdeps: warning: symbol pthread_sigmask used by debian/witty/usr/lib/libwthttp.so.2.2.4 found in none of the libraries.
dpkg-shlibdeps: warning: symbol pthread_mutexattr_settype used by debian/witty/usr/lib/libwt.so.2.2.4 found in none of the libraries.
dpkg-shlibdeps: warning: symbol pthread_mutexattr_init used by debian/witty/usr/lib/libwt.so.2.2.4 found in none of the libraries.
dpkg-shlibdeps: warning: symbol pthread_mutexattr_destroy used by debian/witty/usr/lib/libwt.so.2.2.4 found in none of the libraries.

He asked if I knew what was that (a dpkg-shlibdeps bug?) and what did it mean, it case it was correct.

That error means pthread_join, pthread_mutexattr_destroy, etc are used by libwt.so and libwthttp.so but those libraries are not linked to libpthread.so by Wt’s buildsystem. I. e. the linker command line when putting libwt.so and libwthttp.so together does not have a “-lpthread“.

Is dpkg-shlibdeps right about that? Yes, it is:

  • libwt.so uses libpthread.so in the XML library (Wt bundles the sources ot MiniXML and compiles them as part of libwt.so)
  • libwthttp.so uses libpthread.so in WServer.C

So, if the buildsystem is not linking libwt.so and libwthttp.so to libpthread.so, why doesn’t linking fail with “unresolved reference” errors? It’s because of the link interface in Boost is wrong.

Link interface? What’s that?

If you are reading me via Planet KDE, you probably know what I’m talking about because Alex has written about this. If you don’t know what I’m talking about, keep reading.

Say you have libA.so, libB.so and libC.so.

  • libA.so does not link to any external library, save for glibc
  • libB.so links only to libA.so and glibc
  • libC.so links only libB.so and glibc

When you run ldd libC.so, what will you get? This?

$ ldd libC.so

linux-gate.so.1 => (0xb7f66000)
libB.so => /usr/lib/libB.so
libc.so.6 => /lib/tls/i686/cmov/libc.so.6
/lib/ld-linux.so.2 (0xb7f67000)

or this?

$ ldd libC.so

linux-gate.so.1 => (0xb7f66000)
libB.so => /usr/lib/libB.so
libA.so => /usr/lib/libA.so < ======= libc.so.6 => /lib/tls/i686/cmov/libc.so.6
/lib/ld-linux.so.2 (0xb7f67000)

The answer is you’ll see the second output: libC.so is linking to libA.so, although when you linked it you only did gcc -o libC.so libC.c -lB (no -lA, so no explicit linking to libA.so):

libC -> libB -> libA

Why is that? Why is libA.so being dragged to libC.so? It’s because of what we call “link interface”

As libC.so links to libB.so, by default, libC.so‘s ELF header will have every library libB.so links to as NEEDED: libB.so‘s dependencies have been transitively passed into libC.so! Usually that’s not what you want, therefore it is possible to specify a “reduced link interface”: given that libA.so is only used internally in libB.so (users of libB.so do not need to use any type or function defined in libA), there is no need to link libC.so to libA.so.

Therefore:

  • As libwt.so and libwthttp.so link to libboost_thread-mt.so from Boost
  • In Debian, Boost is compiled with threads (i. e. it links to libpthread.so)
  • Boost does not publish a reduced link interface but the full link interface (i. e. every dependency of Boost is transitively passed into applications/libraries using Boost)

… even though libwt.so and libwthttp.so were not linking to libpthread.so explicitly, they were picking libpthread.so from libboost_thread-mt.so and no linkage error happened. If libboost_thread-mt.so would not export libpthread.so (it should not!), linkage of libwt.so and libwthttp.so would have failed and the authors of Wt would have noticed the bug in their build system.

While this is not a critical issue, it makes your application/library load slower, because it needs to resolve and load the NEEDED libraries.

Please note this discussion is valid for any operating system, including Windows and Mac OS X.

If you use CMake as your build system and you want to adopt a reduced link interface, take a look at the CMake docs for TARGET_LINK_LIBRARIES, particularly the LINK_INTERFACE_LIBRARIES section:

Library dependencies are transitive by default. When this target is linked into another target then the libraries linked to this target will appear on the link line for the other target too. See the LINK_INTERFACE_LIBRARIES target property to override the set of transitive link dependencies for a target.

target_link_libraries( LINK_INTERFACE_LIBRARIES
[[debug|optimized|general] ] …)

The LINK_INTERFACE_LIBRARIES mode appends the libraries to the LINK_INTERFACE_LIBRARIES and its per-configuration equivalent target properties instead of using them for linking. Libraries specified as “debug” are appended to the the LINK_INTERFACE_LIBRARIES_DEBUG property (or to the properties corresponding to configurations listed in the DEBUG_CONFIGURATIONS global property if it is set). Libraries specified as “optimized” are appended to the the LINK_INTERFACE_LIBRARIES property. Libraries specified as “general” (or without any keyword) are treated as if specified for both “debug” and “optimized”.

NB: The comments in this blog do not work due to a hosting issue

ACE in Debian

by ⋅ Leave a Comment

Are you a developer and use ACE, TAO, CIAO? Do you use and/or deploy to Debian, Ubuntu or any other Debian-based distribution?

If you answered affirmatively to those two questions, you have probably noticed Debian still ships ACE 5.6.3, which is 18 months old. The reason is twofold:

  • ACE is a complex beast. The source tarball generates 59 binary packages for 5.6.3, and that’s only to increase in the latest version (5.7.4).
  • The only Debian developer working on ACE, Thomas Girard, is too busy at the moment. He did a great job maintaing ACE for years but now it’s time for others to help him.

Therefore, if you use ACE and Debian directly or indirectly, please step in and help me get the latest ACE in Debian. I’m half done but I’m having trouble with autotools (the autotools build system in ACE seems to need some love, I’m probably moving to the traditional build system) and I do not know where to put some of the new libraries (DAnCE, etc).

NB: The comments in this blog do not work due to a hosting issue, please contact me directly by e-mail: or subscribe to the pkg-ace-devel mailing list.

Conferences are useful: the WtDesktop case

by ⋅ 1 Comment

You probably know I love Wt, the Qt-like C++ library for developing webapplications. I gave up on Ruby on Rails and PHP when I started using Wt about three years ago.

Koen and Wim, the authors of Wt, happen to be from Belgium and they attended aKademy 2008. I “arranged” a metting between Koen, Wim, Richard Dale (the KDEbindings man) and me. We talked about how great Wt is, how much the API ressembles Qt and how nice bindings to other languages would be. Guess what? A few months later, Richard had adapted SMOKE to Wt and had developed Wt::Ruby bindings. Impressive.

I attended last year’s FOSDEM and there we go again: over dinner we came with a new idea. Google Gears makes it possible to run webapplications like they were desktop applications. But it’s Python, it requires a Python interpreter an a load of libraries. Can’t we do the same thing with Wt and generate a single executable by statically linking everything? Can’t we? Of course we can! In theory, it would be very easy: start Wt with its own embedded server (wthttp) in a QThread, then load that in a QtWebKit-powered browser. Koen put that in writing as a “>GSoC idea but EmWeb (the company Koen and Wim founded to support Wt) was not accepted as a mentor in GSoC 2009.

It took me a long time to get started with WtDesktop because I was busy with other stuff but two weeks ago I finally started to cut the meat. And it turned out to be surprisingly easy! Yes, WtDesktop works! It’s not in Wt, it’s not finished yet, but I hope I will be able to work on it and have something very nice in a few weeks time (get WtDesktop Technology Preview 2 here; TP3 is in the works, clone this repository)

What will WtDesktop have in the future?

  • Port autodiscovery (for now it’s hardcoded to run on port 9000, I’m having some trouble mixing Qt and Boost signals and slots to make it choose a new port automatically). This is almost done in TP3, which will be available soon.
  • Call-outs: let the webapp go out of the web and add menus, toolbars, etc to the application when it’s linked against libwtdesktop.so. Yes, you read it correctly: if you link against libwthttp.so you get a webapp with its own embedded webserver, if you link against libwtfcgi.so you get a FastCGI module you can serve from Apache, IIS or whatever, and if you link against libwtdesktop.so you get a desktop appliation. No changes to your code at all! I’d like to have some declarative language like QML for this but that’ll take some time.
  • Printing support. There is basic printing support (i. e. printing the whole page) already but I’d like to be able to connect buttons, frames, etc in the webapp to be able to print only some parts. This would require the above call-outs.
  • Session-sharing and session-stealing. Wt has all the DOM tree for each session in memory, therefore it is able to send the same HTML, CSS, DOM tree, etc to some other address. Session-sharing would be like Remote Assistance on Windows: you can see what the other client is doing but you cannot touch. Session-stealing would be like Remote Desktop: you can click, type, etc and the other client would see a “this application is being used by XXX (IP address: X.Y.Z.T)”.

So yes, attending conferences is definitely fruitful. By the way, I’m attending FOSDEM 2010. Let’s see if I come with some other great idea!

UPDATE: Added git repository

libQtGit

by ⋅ Leave a Comment

A long time ago I started hacking on a library I named libQtGit. As you’ll deduce from the name, it provides a Qt-like API to git, the stupid content tracker.

I had a vision for libQtGit: it was not an end in itself but a piece of a more ambitious framework/library tentatively named QVersioned or libQtVersioned.

QVersioned would provide QVersionedDir and QVersionedFile classes (amongst others, but those two are the most important). Those classes would essentially have the same API QDir and QFile have. QVersioned would open ZIP files and store a .git directory inside. There are cases, like ODF (which itself is a ZIP file) where nothing special is needed. For other cases (for instance, a .txt file), there would be a .txtv (meaning “.txt versioned”) which would be a ZIP file containing the .txt + .git directory (why ZIP? because it’s supported natively by Windows XP and newer and Mac OS).

Now, how did I intend to implement ODF versioning, which was going to be the “this thing works” case?:

  • The .git folder would store the uncompressed contents of the ODF file, i. e. XML, images, etc. This is needed to avoid duplication, allow diffs, etc
  • There would be also a full copy of the latest version (XML, images, etc) in the ZIP file, just like any ODF-capable application not supporting QVersioned would expect (these applications would just ignore the .git directory)
  • When a QVersioned-capable application opens an ODF file, it compares the XML, images, etc to the latest version in git:
    • If the diff is empty, last save was by a QVersioned-capable application, no action needed at moment
    • If the diff is not empty, last save was by a QVersioned-incapable application. A new “git commit -a” is performed. Yes, we probably have lost “versions” of the document in between this commit and the former one but something’s better than nothing.

By using libQtGit and QVersioned, it would also be possible to add collaboration features such as “send me an update” (i. e. send me a diff which transforms my outdated document into your latest version), collaborative editing (send diffs back and forth) and more things I cannot think of right now.

In case you are interested in the libQtGit API (remember QVersioned will offer a higher-level API), this is the signature of the method you would call:

Git::Repository::Commit* Git::Repository::commit ( const
QString& message = QString(),
const Commit* c = 0,
const QString& author = QString(),
const CommitFlags cf = DefaultCommitFlags,
const CleanUpMode cum = DefaultCleanUpBehavior );

That’s equivalent to “git commit -C commit_id -m “message” “.

CommitFlags is a QFlag and CleanUpMode a QEnum:

enum CommitFlag {
DefaultCommitFlags = 0x0 /*< git commit */, OverrideIndexCommit = 0x1 /*< git commit -a */, SignOffCommit = 0x2 /*< git commit -s */, AmendCommit = 0x4 /*< git commit --amend */, AllowEmptyCommit = 0x8 /*< git commit --allow-empty */, NoVerifyCommit = 0x16 /*< git commit -n */ }; Q_DECLARE_FLAGS ( CommitFlags, CommitFlag ) enum CleanUpMode { DefaultCleanUpBehavior = 0x0 /*< git commit */, VerbatimClean = 0x1 /*< git commit --cleanup=verbatim */, WhiteSpaceClean = 0x2 /*< git commit --cleanup=whitespace*/, StripClean = 0x4 /*< git commit --cleanup=strip */, DefaultClean = 0x8 /*< git commit --cleanup=default */ }; Q_ENUMS ( CleanUpMode )

For the "git commit -a -m "Save latest unversioned version on ODF document opening" ", we would use:

// Assuming 'repo' is a valid Git::Repository object

repo->commit ( "Save latest unversioned version on ODF document opening",
0,
"(The application would probably take the
author's name from the product registration)",
Git::Repository::OverrideIndexCommit );

So, how is libQtGit doing? Well, the API is there for git add, commit, init, mv, rm, checkout, clone, branch, revert, reset, clean, gc, status, merge, push, fetch, pull, rebase, config, update-server-info and (partially) symbolic-ref.

When I say “the API is there” I mean “all the QFlags, QEnums, methods, classes and its translation to git parameters is done”. It’s just a matter of implementing the QProcess part, parsing output, etc. Boring and
time-consuming but easy.

Given that I’m busy with other stuff, and some people have been asking for the code for a long time (sorry Riccardo!), I have just published what I have now. It’s unfinished, barely tested and yet to implement in many places. But it’s a starting point. It’s available now from Gitorious: http://gitorious.org/libqtgit/libqtgit

As for QVersioned, nothing is done yet.

PPA: broken on Hardy!

by ⋅ Leave a Comment

A couple of weeks ago, I said I would not be backporting the latest versions of Glib2, Gtk+, GStreamer, etc to Hardy.

Today I decided I needed a newer GStreamer on my Hardy machine (where I build and use KDE from trunk almost daily) because I wanted to use libQtGstreamer.

As I expected, this requires major surgery to Hardy. Two prominent components of Hardy, ffmpeg and PulseAudio, require relatively recent versions. I don’t even know if it’s possible to build them due to limitations in Hardy’s kernel V4L2 support. In addition to that, I’m leaving on holidays to London and I won’t be able to work on this until next Wednesday.

Summary: IF YOU USE MY PPA AT THIS MOMENT, IT WILL BREAK YOUR HARDY INSTALLATION! BE VERY CAREFUL IF YOU RUN APTITUDE FULL-UPGRADE! I don’t like having the PPA broken for so long but I have to wake up in 4 hours and I really need to get some sleep.