A wish a day 14: Wine-based Application Virtualization

by ⋅ 2 Comments

Application virtualization is an umbrella term that describes software technologies that improve portability, manageability and compatibility of applications by encapsulating them from the underlying operating system on which they are executed.

A fully virtualized application is not installed in the traditional sense, although it is still executed as if it were. The application is fooled at runtime into believing that it is directly interfacing with the original operating system and all the resources managed by it, when in reality it is not.

In this context, the term “virtualization” refers to the artifact being encapsulated (application), which is quite different to its meaning in hardware virtualization, where it refers to the artifact being abstracted (physical hardware).

So Wikipedia says.

In layman terms, application virtualization is “I want to run a third party application without it requiring installation”. Probably you know it as “portable application“.

The trivial case

If we talk about Windows applications, for simple applications, application virtualization amounts to copying the .exe and any required DLL to some folder, then running the application from that folder.

Easy, huh?

Well, no.

Registry keys

Many applications add keys to the registry and will not run without those keys. Some even require other files, which are created in AppData, CommonAppData, etc.

To solve this kind of dependencies, application virtualization software packages such as VMware ThinApp, Microsoft App-V and Citrix XenApp monitor the registry and the filesystem for changes. To do so, they require an “application virtualization preparation” procedure. It’s like you have a clean virtual machine, take a snapshot, then install the application, take another snapshot, and compare them.

You could achieve more or less the same result for free by using FileMon + RegMon + RegShot.

Of course I am simplifying more than a bit here 🙂

Common files

Then there is the problem of file redirection.

The whole point of application virtualization is to be able to run the application without installation. But if this application’s setup installed files in CommonAppData (or the application creates them), how do we force the application to look for them in our “virtualized application folder”?

The technique usually involves generating a small .exe launcher that:

  • Sets a special PATH that looks where we want first
  • Uses hooking, DLL injection and some other low level tricks to force system calls to look into the places we want before (or even instead of) they look into the standard places
  • Registry emulation (in combination with hooking and DLL injection) to make sure those registry keys are taken from the registry in the sandbox instead of the actual registry
  • And more

In a way, we are cracking the application.

Cross-platform

What I have described above are the required steps to virtualize an application if the target platform (where the application is going to be run) is the same as the host platform (where the application virtualization was performed). For instance, if we are virtualizing the application on Windows XP SP3 32-bit and the application will run on Windows XP SP3 32-bit.

Even if the steps described above look complex to you (they are), that’s the simple case of application virtualization.

If the target platform is a different version (for instance, the app virtualization takes place on Windows XP 32-bit, while the target platform is supporting also Windows 7 64-bit), then it gets more complex: probably many Windows DLLs will need to be redistributed. Maybe even ntoskrnl.exe from the original system is required.

In addition to the serious problem redistribution poses in terms of licensing to anybody but Microsoft App-V, there is the technical side: how do you run Windows XP SP3 32-bit ntoskrnl.exe on Windows 7 SP1 64-bit? You cannot, which makes some applications (very few) impossible to virtualize.

Device drivers

The more general case for “impossible to virtualize applications” (we are always talking about application virtualization here, not full virtualization or paravirtualization) is device drivers.

None of the application virtualization solutions that exist so far can cope with drivers, and for a good reason: installing a driver requires administration permissions, verifying certificates, loading at the proper moment, etc. All of that needs to be done at kernel level, which requires administration permissions, thus defeating the point of application virtualization. Then there are the technical complications (the complexity of the implementation), and the fact that drivers assume they run with administration permissions, therefore they may perform actions that require administration permissions. If that driver is also app-virtualized, it will not run with administration permissions.

One possible remedy to the “driver virtualization problem” would be to implement a “driver loader process” that requires installation and will run with administration permissions. Think of it as the “runtime requirement” of our application virtualization solution.

I know, I know, it sounds contradictory: the whole point of my app virtualization software solution is to avoid application installation, yet my app virtualization solution requires a runtime installation? We are talking about a one-time installation, which would be perfectly acceptable in a business environment.

But why all this!?

You may wonder why this braindump.

The reason is the job that pays my bills requires virtualization. Given that we need driver support, we resorted to VirtualBox but we run a lot of instances of the applications we need to virtualize, which means we need a lot of Windows licenses for those virtual machines. Worse: we cannot know how many Windows licenses we need because the client may start any number of them and should not notice they run in virtual machines! Even worse: Microsoft have a very stupid attitude in the license negotiations and keep trying to force us to use Windows XP Mode and/or Hyper-V, which are not acceptable to us due to their technical limitations.

So it turned on the “think different switch” in my brain and I came with a solution: let’s port Wine to Windows and use it to virtualize applications.

The case for a Wine-based Application Virtualization solution

Crazy? I don’t think so. In fact, there would be a number of advantages:

  • No need for Windows licenses
  • No need for VirtualBox licenses
  • Wine already sandboxes everything into WINEPREFIX, and you can have as many of them as you want
  • You can tell wine to behave like Windows 2000 SP3, Windows XP, Windows XP SP3, Windows 7, etc. A simple configuration setting and you can run your application the environment it has been certified for. Compare that to configuring and deploying a full virtual machine (and thankfully this has been improved a lot in VBox 4; we used to have our own import/export in VBox 3)
  • No need to redistribute or depend on Microsoft non-freely-redistributable libraries or executables: Wine implements them
  • Wine has some limited support for drivers. Namely, it already supports USB drivers, which is all we need at this moment.

Some details with importance…

There are of course problems too:

  • Wine only compiles partially on Windows (as a curiosity, this is Wine 1.3.27 compiled for Windows using MinGW + MSYS)
  • The most important parts of wine are not available for Windows so far: wineserver, kernel, etc
  • Some very important parts of wine itself, no matter what platform it runs, are still missing or need improvement: MSI, support for installing the .NET framework (Mono is not enough, it does not support C++/CLI), more and more complete Windows “core” DLLs, LdrInitializeThunk and others in NTDLL, etc
  • Given that Windows and the Linux kernel are very different, some important architectural changes are required to get wine running on Windows

Is it possible to get this done?

I’d say a team of 3-4 people could get something working in 1 year. This could be sold as a product that would directly compete with ThinApp, XenApp, App-V, etc.

We even flirted with the idea of developing this ourselves, but we are under too much pressure on the project which needs this solution, we cannot commit 3-4 developers during 1-2 years to something that is not the main focus of development.

Pau, I’m scared!

Yes, this is probably the most challenging “A wish a day” so far. Just take a look at the answer to “Am I qualified?” in Codeweavers’ internships page:

Am I Qualified?

Probably not. Wine’s probably too hard for you. That’s no insult; it’s too hard for about 99% of C programmers out there. It is, frankly, the hardest programming gig on the planet, bar none. But if you think you’re up for a real challenge, read on…

Let’s say, just for the sake of argument, that you’re foolish enough that you think you might like to take a shot at hacking Wine. […]

The most stupid C bug ever

by ⋅ 98 Comments

I have been programming for a number of years already. I have seen others introduce bugs, and I have also introduced (and solved!) many bugs while coding. Off-by-one, buffer-overflow, treating pointers as pointees, different behaviors or the same function (this is specially true for cross-platform applications), race conditions, deadlocks, threading issues. I think I have seen quite a few of the typical issues.

Yet recently I lost a lot of time to what I would call the most stupid C bug in my career so far, and probably ever.

I am porting a Unix-only application which uses tmpfile() to create temporary files:

            else if (code == 200) {     // Downloading whole file
                /* Write new file (plus allow reading once we finish) */
                g = fname ? fopen(fname, "w+") : tmpfile();
            }

For some awkward reason, Microsoft decided their implementation of tmpfile() would create temporary files in C:\, which is totally broken for normal (non-Administrator) users, and even for Administrator users under Windows 7.

In addition to that, the application did not work due to another same-function-diffent-behavior problem.

To make sure I did not forget about the tmpfile() issue, I dutifully added a FIXME in the code:

            else if (code == 200) {     // Downloading whole file
                /* Write new file (plus allow reading once we finish) */
                // FIXME Win32 native version fails here because Microsoft's version of tmpfile() creates the file in C:\
                g = fname ? fopen(fname, "w+") : tmpfile();
            }

Read More →

A wish a day 11: Perl Compatible Regular Expressions in CMake

by ⋅ 6 Comments

CMake is a unified, cross-platform, open-source build system that enables developers to build, test and package software by specifying build parameters in simple, portable text files. It works in a compiler-independent manner and the build process works in conjunction with native build environments, such as Make, Apple‘s Xcode and Microsoft Visual Studio. It also has minimal dependencies, C++ only. CMake is open source software and is developed by Kitware. (taken from Wikipedia)

But you already know that because CMake is the build system we use in KDE 🙂

When I started to use CMake at work to port a large project from Windows to Linux, I needed to write a lot of modules (“finders”). Those modules had to work on Windows and Linux, and sometimes with different versions of some of the third-party dependencies. Parsing header files, Windows registry entries, etc was often required.

If you have ever parsed text, you know you powerful regular expressions are. If you use old software, you know how great Perl Compatible Regular Expressions (PCRE) are because you do not have them available :-).

Read More →

A wish a day 10: KDE Dolphin “Previous versions” support

by ⋅ 8 Comments

Microsoft Windows has supported a feature called Shadow Copy since Windows Server 2003, using Windows 2000 or newer as the client.

Shadow copies, better known as “Previous versions”, by many users are kind of snapshots of a filesystem (“drive”, for Windows users) taken every so often. It is not a replacement for backups or RAID, just a convenient method to access old versions of your files. Thin of it as a revision control system for any file, save for “revisions” are taken at time intervals instead of evey time there are changes, which means not all the changesets are “captured”.

Samba implements the server part of shadow copy since version 3.0.3, released in April 2004, and makes them available to Windows clients.

On the Unix world, you can get more or less the same features, albeit only locally (not for NFS or Samba shared folders), by using filesystems such as ZFS, HAMMER (apparently, only available on DragonFly BSD), Btrfs, Zumastor and many others. The main problem being adoption of those solutions is marginal, and not available through the network (“shared folders”).

Sun (now Oracle) implemented a “Previous versions”-like feature for ZFS snapshots in Nautilus in October 2008:

KDE once had something like that but for the ext3cow file system. It was an application called Time Traveler File Manager (user manual, interface design and details) by Sandeep Ranade. Website is down and ext3cow is effectively dead now.

By this time you probably have guessed my wish for today: implement “Previous versions” support for Samba, ZFS, etc in Dolphin, the KDE file manager, or even better: in KIO, so that it is available to all KDE applications.

A wish a day 9: QML WPF

by ⋅ 11 Comments

When Stephen Elop announced Nokia was adopting Windows Phone 7 as its main development platform for the future, many of us felt a mix of fear for our jobs, incredulousness (are they committing suicide?), anger for betraying open source, then curiosity. What’s this WP7 about and what’s up in regards to development?

The canonical development platform for WP7 consists of Visual Studio 2010 and managed applications developed in .NET 4 using C# as the programming language and a WPF for the user interface. If you are not into .NET, you may be a bit lost now.

Windows Phone 7 is an embedded operating system by Microsoft. It is the successor to Windows Mobile 6.5 but it is incompatible with it and it is targeted at mobile phones.

Visual Studio is Microsoft’s development environment, including IDE, compiler, debugger, profiler and much more. In my opinion, it is one of the finest applications Microsoft has developed. It is many years ahead of QtCreator, KDevelop, Eclipse or anything else.

.NET is a framework Microsoft started more than a decade ago. It includes libraries, a language independence infrastructure, memory management, localization and many more features. Applications can be compiled as native code (for x86 or x64), or as managed code (bytecode). .NET also includes a bytecode JIT compiler. You can target the .NET platform in many languages, including but not limited to, Visual Basic .NET, C#, F#, C++/CLI (it’s NOT the C++ you know!), etc

Interestingly, you can also compile normal C++ to .NET but it will have unmanaged code (i. e. code where memory allocations/deallocations are performed by you instead of the runtime, and pointers). WP7 requires that all code that runs on it be managed. This means that Qt cannot run on WP7, even though it may be compiled targetting the .NET platform and it would run on desktop operating systems.

C# is a multi-paradigm programming language developed by the father of Delphi. It is very powerful and looks a lot like C++2011. When I was learning C#, I couldn’t but think what the actual implementation of C# features were behind the scenes, in C and C++.

WPF is Windows Presentation Foundation (formerly known as Avalon), the technology (library) used for the GUI. In plain terms, it’s a library which provides accelerated everything for Windows applications: widgets, 2D, 3D, scalable graphics, etc

When developing UIs in WPF, you can either use C# code:

// Create a Button with a string as its content.
Button stringContent = new Button();
stringContent.Content = "This is string content of a Button";

// Create a Button with a DateTime object as its content.
Button objectContent = new Button();
DateTime dateTime1 = new DateTime(2004, 3, 4, 13, 6, 55);

objectContent.Content = dateTime1;

// Create a Button with a single UIElement as its content.
Button uiElementContent = new Button();

Rectangle rect1 = new Rectangle();
rect1.Width = 40;
rect1.Height = 40;
rect1.Fill = Brushes.Blue;
uiElementContent.Content = rect1;

// Create a Button with a panel that contains multiple objects 
// as its content.
Button panelContent = new Button();
StackPanel stackPanel1 = new StackPanel();
Ellipse ellipse1 = new Ellipse();
TextBlock textBlock1 = new TextBlock();

ellipse1.Width = 40;
ellipse1.Height = 40;
ellipse1.Fill = Brushes.Blue;

textBlock1.TextAlignment = TextAlignment.Center;
textBlock1.Text = "Button";

stackPanel1.Children.Add(ellipse1);
stackPanel1.Children.Add(textBlock1);

panelContent.Content = stackPanel1;

or use XAML, which is an XML-based language for WPF:

<!--Create a Button with a string as its content.-->
<Button>This is string content of a Button</Button>

<!--Create a Button with a DateTime object as its content.-->
<Button xmlns:sys="clr-namespace:System;assembly=mscorlib">
  <sys:DateTime>2004/3/4 13:6:55</sys:DateTime>
</Button>

<!--Create a Button with a single UIElement as its content.-->
<Button>
  <Rectangle Height="40" Width="40" Fill="Blue"/>
</Button>

<!--Create a Button with a panel that contains multiple objects 
as its content.-->
<Button>
  <StackPanel>
    <Ellipse Height="40" Width="40" Fill="Blue"/>
    <TextBlock TextAlignment="Center">Button</TextBlock>
  </StackPanel>
</Button>

 

More or less, WPF is to .NET what QML is to Qt.

Ouch. This WP7 is completely different to what we are used to. Maybe we can make them less different?

People tend to believe WPF and XAML are the same thing. They are not. XAML is one of the languages (which happens to be used by 99.99% of the people) to write applications in WPF but it is not the only one. You can write WPF applications in C# code (it’s not unusual to mix XAML and C#, what we call “code behind” in WPF parlance), or you could create your own XAML-equivalent language.

Or you could make my wish for today true: create a compatibility layer for QML to be able to develop WPF applications in QML, effectively replacing XAML. It is technically possible. This will also require a glue-layer for C#, similar to what we have now to bridge QML and C++. This would make porting applications from C++ and QML to C#, QML and WPF a relatively easy task, and may provide realistic path to convert Qt applications to WP7 withouth a full rewrite.

Update Fixed XAML vs C# code, thanks ‘aa’. Also, please note using C++ with QML WPF would be straightforward

A wish a day 7: make emerge a generic package manager for Windows

by ⋅ 8 Comments

A while ago I said Koen from Emweb made an interesting proposal at FOSDEM about emerge, the KDE Windows build tool.

Yesterday, Jarosław Staniek and I reaffirmed our commitment to ’emerge’. Today, I’d like to go a bit further: let’s bring more developers to emerge by opening it up to other projects. Keep reading!

What is emerge, why is it important and what was Koen’s proposal?

Fact: Microsoft Windows is very different to Unix in regards to development.

On Unix platforms -that includes Linux and Mac OS X-, software is usually installed to /usr: applications in /usr/bin and /usr/sbin, libraries in /usr/lib, headers in /usr/include, common resources in /usr/share, etc. Also, dependency management is usually something you can count on: when you install kdelibs5-dev in Ubuntu, it will automatically install libqt4-dev, kdelibs5-data, libfreetype (runtime), etc That makes setting up a development environment a very easy task: look for shared libraries, header files, etc in the common places and you will probably find them.

On Windows there is nothing like that. When you want to compile an application, you need to provide (build and install) all its dependencies, and you need to tell Visual Studio where to find everything. Even CMake usually needs some help in the form of a hint for CMAKE_PREFIX_PATH. As you may imagine, building KDE, which has more than 200 third party dependencies and tens of modules (and with the move + split to git, many more) becomes an almost insurmountable task.

‘Emerge’ to the rescue: inspired by Gentoo‘s emerge, Ralf Habacker, Christian Ehrlicher, Patrick Spendrin and others (yours faithfully included) developed a tool which downloads the source, configures, builds, installs and packages KDE and its dependencies. It makes a world of difference when building KDE. Actually, it makes building KDE on Windows possible. Once more: thank you very much guys, impressive tool.

There are two well-differentiated parts in emerge, the ‘engine’ and the ‘recipes’.

Read More →

A wish a day 6: AppStore-like installer for KDE on Windows

by ⋅ 8 Comments

This is another idea that has been in my mind for months, and I’m so thrilled that I added it to the KDE Google Summer of Code 2011 KDE Ideas wiki: a new installer for KDE on Windows.

Currently, KDE on Windows can be installed using either emerge or the end-user installer. The installer is Cygwin-like, which means it is very different from the normal InstallShield, NSIS, MSI, etc installers Windows users are used to:

Have you seen Apple’s Mac App Store?

Read More →

A wish a day 5: KDE UI testing

by ⋅ 10 Comments

Recently I have been appointed “Verification Leader, Core Foundation Team” whatever that means. Essentially it translates into taking care of functional testing. Ah, testing, well regarded, often ignored, always controversial.

“Testing” usually means “unit testing” to developers. Code is what we understand (I can’t speak Russian but I can speak C++ 🙂 ) “I’ve kept my API stable but I’ve optimized the internal implementation. What did I broke?” With proper unit testing, you can go for a coffee and when you are back answer to the question is on screen: “oh, after optimization I forgot about corner case X, I’ll fix it”.

So the code is right. As Knuth would say “Beware of bugs in the above code; I have only proved it correct, not tried it”.

Try. That’s the key. The user does not care whether the application is implemented in C++2011 using lambdas, concepts, variadic templates and throws exceptions across shared libraries, or in plain old Visual Basic 6. Does it work? Does it load the file when I click File -> Open? Will it show a computer when I click and drag the computer icon from the palette to the canvas? That’s what matters.

In KDE we have unit tests in kdelibs (in fact it’s a requirement to get code in) and there are a few UI tests for the KDE User Interface module, which contains essentially widgets for use with your KDE-based application.

What about applications? What about Amarok, Gwenview, Kexi, Okular, etc? How do we know they work fine? How do we know they actions do what they should do and the application renders as it should? Correct me if I’m wrong but I think currently we rely on the input from our users: when they report something’s misbehaving, we oblige.

Can we do better? Yes, we can. UI testing has been there for decades and even the venerable expect can help with that.

There are many tools for automated GUI testing. Some are open source, like Xnee or White, others are closed-source, like Froglogic Squish and SmartBear TestComplete, which also happen to have the best support for Qt-based applications.

Are those tools useful? Why, yes, they are. For instance, there is a well-known bug in KDE Windows which makes Digikam on Windows show empty context menus when the Qt style is set to CDE (the default; if you use the Oxygen style everything looks fine). This kind of bugs can be easily detected by simulating user actions, then OCR‘ing the text of the popup menu. Or even just telling the automatic UI test to take a screenshot of the popup, then comparing pictures. Same for PDFs that render oddly in Okular on some Linux distribution due to fonts, FreeType configuration, an old version of Poppler, etc

The tool I am most familiar with is TestComplete, which suffers a bit with alien widgets (you need to build Qt without accessibility support). I have been told Squish is much better but given that there is no freely downloadable version and it does not support WPF (a requirement for the project I’m working on now), I can’t try it.

Here is a wish for KDE users who want to join development but have little or no programming skills: go to SmartBear’s website, download TestComplete and record/write a few UI tests. Let us know (in the forums, in the mailing lists) how far you got. Maybe we can make UI testing -a must in the corporate software development- part of the KDE development and release process.

Update: apaku says froglogic provides free licenses for KDE, see the comments

Linux drivers running on Windows

by ⋅ 4 Comments

I’m surprised this has gone mostly unnoticed: two Japanese researchers wanted to port the Kernel-based Virtual Machine from Linux to Windows. In order to do that, they created a compatibility layer that allows you to run Linux drivers on Windows (in kernel, no less!). They gave a talk titled WinKVM: Windows Kernel-based Virtual Machine at last year’s KVM Forum, the KVM Developers Conference.

Source code is available here.

Given that Linux never removes support for any hardware from the kernel unless the driver breaks the kernel badly, the first use I can think of is obsolete hardware whose drivers no longer work on modern versions of Windows. That’s a often-encountered case in military and industrial environments: LAN Emulation cards, industrial devices using proprietary buses and proprietary cards to communicate to PCs, medical devices, etc. In those cases, the hardware generally outlives the computer that runs the software by decades (the software is no problem thanks to virtualization).

A second target “market” could be devices which do not have a Windows driver. Believe it or not, they exist in niche markets (and not so niche: the free Kinect driver was initially only available for Linux).

My talk at FOSDEM

by ⋅ Leave a Comment

First talk in the CrossDesktop room on Sunday: KDE on Windows. I expected no more than 10 people to wake up early in the morning and attend a Windows talk, yet about 40 showed. Lots of problems with the projector ðŸ™

I gave the usual talk (slides) and showed KDE running on Windows 7 and Windows 2008. I had some problems while trying to show KDE applications running through Terminal Services and published with 2X but hey, it works (what you want to know: make sure you do not use graphicssystem raster!)

I insisted a lot on our need for more developers, especially packagers for third-party dependencies. Packaging a 3rd-party library for Windows using emerge is rather easy, it can be done by people with relatively few programming skills and, in any case, it is a good way to get started with development on Windows.

In the Q & A session there were the usual questions and Koen from Emweb made an interesting proposal. I think it deserves its own blog entry.