Friday, October 17, 2014

Compiling and Installing the Linux Kernel

The first few times I tried to boot from a custom kernel, I faced a lot of problems and found documentation to be a spread out in different places and also somewhat incomplete depending on the specific distribution. This guide will document the steps to be followed to get a custom kernel running, focusing on Ubuntu and Debian derivatives. The steps followed however, are almost completely, distribution neutral, and hence should be applicable for whichever distro you may be using.


Pre-requisites

I will be using the kernel tree from Kernel.org for this post. If you have a specific version, then great! Otherwise, you can clone the kernel tree using:
$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
(This requires git, but you have that installed, don't you?)

Compilation 

For the compilation process, we first need to create the configuration file. This file specifies the various properties of the kernel, including which modules to be compiled, which to be included in the image, and which to be loaded dynamically. You can, depending on your configuration, go for a modular or a monolithic kernel from here. You can either create one using the kernel utilities, or simply use the one that comes with your distro and which the current kernel is using.
$ ls /boot/*config*`uname -r` -a
Copy the config file to your kernel root directory and rename it to .config
$ cp /boot/*config*`uname -r` ./.config
Now, run make to verify the configuration file for the current distribution. You will see a number of new options that have been added into the new kernel. I would suggest simply going with the defaults on most of them, unless you know what you are doing.
$ make oldconfig
We are ready to start compiling. You simply need to run make. To make things go faster, we can make make spawn multiple jobs. Usually, 2 per core is a good number to go by. I also like to time how long it takes, hence my command is similar to:
$ time make -j$((`nproc`*2)) > ./.build-feedback
The output redirection will save all the build feedback into a file that you come back and read when compilation is over. Oh yes, it's gonna be awhile.

Installation

Assuming there weren't any errors during compile time, you are good to install your flashy new kernel. The installation process is simple and involves installing modules, copying files to the correct directories and updating your bootloader.

Installing modules is a one liner:
% make modules_install
Note at the end of the install process, you will see a version number. Copy it, or use the following command:
$ KVER=`ls /lib/modules -ht | head -n 1`
Now, we need to copy the kernel image to the /boot directory. You will find it under /arch corresponding to your machine architecture. For seeing your machine architecture, run:
$ uname -m
Or we can copy the correct image with this one liner:
% cp ./arch/`uname -m`/boot/bzImage /boot/vmlinuz-$KVER
Next, we need to copy the system map, which stores information regarding the built modules and their relative addresses for dynamic loading.
% cp System.map /boot/System.map-$KVER
Next, comes the config file.
% cp .config /boot/config-$KVER
If your distro is configured to use an initramfs, which can easily be confirmed by the presence (or absence) of initrd files in /boot, then you need to generate it using:
% update-initramfs -c -k $KVER
The last step is updating GRUB, so that the new kernel can be added to the boot list.
% update-grub2

Verification

Let's boot into the new kernel now. As the new kernel should be at the top of the GRUB boot list, you should automatically be booted into it. Once you have a usable desktop, which may or may not happen, open up a terminal, or switch to a virtual terminal, and verify your install version (should match with KVER).
$ uname -r 
Congratulations! You are now using your compiled kernel.


This process can take upto 2.5 hours the first time you attempt it, with successive attempts taking upto an hour depending on your workstation. Nonetheless, it is an exciting process and your first step transitioning towards kernel development.

Saturday, October 11, 2014

Raspbmc Controller using PiTFT



Installed PiTFT in LumenEd devices
While working at LumenEd, I got the chance to work on with a lot of different peripherals and chips that go with the Raspberry Pi. One of those items was the PiTFT touchscreen.

Our goal was to develop a control application, similar to a remote, that would handle all user inputs to RaspBMC, our OS of choice. After a few weeks of tinkering, design and development, we were able to put together a simple, yet sufficient, interface, that has been deployed on our devices. This post explores the bare minimum that was needed to get things working together. To see the device in action, take a look at an
early demo video.

The software application for PiTFT is available on Github. It has been developed in Python, due to the extensive support that it has on the Pi, as well as the reduced development time that it provides (which was a priority for us during the development phase). We have used Pygame as our GUI library, for similar reasons. Also, it depends on TSLIB, for touchscreen input, and fbcon, for output to the touchscreen frame buffer.


Moving to the touchscreen itself; getting it working turned out to be a pit of a puzzle at first. Over the period of 4 days however, we were able to figure out all that was needed for it to perform reasonably well in RaspBMC. Firstly, it is necessary to calibrate the touchscreen for optimal touch performance. This is easy to do once the necessary libraries have been installed. Next, we need to load the calibration profile from within the application. This requires the use of environment variables, specifically the first four as shown below. These variables specify the input driver to use, the configuration of the hardware, and the calibration constants. For Pygame to interpret and process the input correctly, we also need to specify the SDL parameters. The parameter names are self explanatory.

os.environ["TSLIB_TSDEVICE"] = "/dev/input/event0"
os.environ["TSLIB_TSEVENTTYPE"] = "INPUT"
os.environ["TSLIB_CONFFILE"] = "/etc/ts.conf"
os.environ["TSLIB_CALIBFILE"] = "/etc/pointercal"
os.environ["SDL_FBDEV"] = "/dev/fb1"
os.environ["SDL_MOUSEDEV"] = os.environ["TSLIB_TSDEVICE"]
os.environ["SDL_MOUSEDRV"] = "TSLIB"
os.environ["SDL_VIDEODRIVER"] = "fbcon"
os.environ["SDL_AUDIODRIVER"] = "alsa"

Another aspect of the GUI design, was the need for elements such as buttons. Pygame, surprisingly, lacks this basic GUI element. Hence, we wrote a custom class for buttons, and certain functions for some other elements as well.
Navigation Menu
Playback Control











At this point, we started to look at the performance of the application. We saw extremely high CPU usage which given the size and simplicity of the display elements and rendering, seemed counter intuitive. Nonetheless, two optimizations were incorporated and this resulted in a drop in the CPU usages to almost a eighth of the original values. One optimization was to reduce the refresh rate to the minimum possible which can be around 4-5 fps. This does not interfere with input responsiveness as the event generation is independent of the screen updation. The other optimization was to reduce the update area to only those sections that were actually changing on a per second basis. This in itself, resulted in a significant performance improvement.

 The last part of the application was forwarding the inputs to XBMC and querying specific parameters to update the touchscreen display. XBMC features an extensive JSON API accessible via the web server. With the help of a wrapper function, the API calls became short and simple. The latency for these calls was also reasonable short on the local server, making the application responsive.

def nav_down():
method = 'Input.Down'
getJsonRemote(method)

Overall, the application was interesting to develop and we are very happy with its performance till date. If you would like to experiment with this application, then download version 1.0. Later versions have been extended and integrated with device-specific hardware, and hence will not be usable externally. The current versions support a camera interface, and a battery display as well.

 Lastly, if you happen to be starting with the Raspberry Pi, I would certainly encourage you to get a hold of the PiTFT and start hacking. It will make for an interesting learning experience about embedded Linux architecture. Happy hacking!

Friday, March 14, 2014

Microsmooth : Signal Smoothing Library for Arduino

During the past few months we have started work on our quadcopter platform, unimaginatively called MICAV. The quadcopter is a micro aerial vehicle with a span of less than 45 cm. The MICAV is currently being controlled remotely using the Avionic RCBX7 transmitter receiver. The device is fairly complex allowing for 7 channel control, and contains a lot of inbuilt functionality. The signals are sent to the Arduino Uno from the receiver. These are then processed, and the motor control signals are set.

As we were testing the controls, we noticed a rather odd pattern. The RPM was changing quite audibly on the motors with a high frequency. Our initial guess was that the wiring must be loose somewhere and we went over the entire circuit. Next, we tested out the motors using a control signal being generated via values sent over serial. The motors performed correctly. Nothing wrong with the motors. Phew!

This left only when avenue to explore. The RX-TX itself. We then hooked up it up with the Arduino and observed the input values being received. The results explained the problem. It was noise all along!



Noise was interfering with the control signals far too much. This necessitated the need for some signal processing to be done before sending it further. To resolve, we started exploring libraries in Arduino-verse that deal with signal processing. We found no relevant ones. The ones we did find that could work for us, were too expensive to use in terms of time and space. We then moved to studying the algorithms ourselves and implemented them. Over the course of two weeks, we programmed and tested about 5-6 algorithms with many variations of parameters.

The algorithms that we implemented are:
  • Simple Moving Average
  • Cumulative Moving Average
  • Exponential Moving Average
  • Savitzky Golay Filter
  • Ramer Douglas Peucker Filter
  • Kolmogorov Zurbenko Filter
Some performed surprisingly well, given the time and size constraints that were imposed.

The green signal is the result of the SMA filter applied to the input. This was one of the mediocre performing cases.

Now, given that there were no existing libraries that were suited to this task when we started to look, we felt it was best to release our code in the form of library, to help out others who may travel down this path. So, then Microsmooth came into existence!

Microsmooth hosted on Github, maintained by me and Pranav, is intended to be lightweight and fast for use in time critical Arduino-based applications. Given that none of the code is Arduino specific, it can even be used on other microcontrollers and platforms, but we make no promises. Time optimizations will eventually make us utilise microcontroller specific features, which may make the library tied down to a specific platform.

To use the library, paste the files in the source directory, and add the following include:
    #include "microsmooth.h"    

The library is still in development and we aiming to improve the overall library in many areas.
  • Reduced memory utilization
  • Improved algorithm performance
  • Addition of new filters and greater control over existing parameters
  • Switches to result in smaller footprint of the code

The library will become more stable and flexible in the months to come.

Thursday, March 6, 2014

The Real Essential Questions of Every Beta

The blog post Seven Essential Meta Questions of Every Beta gets linked to prominently from every beta's meta. However, this blog post is somewhat inaccurate and actually misdirects new users. In almost every beta, for example, the site design question gets asked and undergoes a fair bit of discussion, even though a probable graduation is too far into the future for this discussion to be useful. Moreover, some of the questions are best asked after the site goes public, such as the moderator nominations, and site promotions. There are certain other issues, that can actually be of immediate use to the community, such as tag synonyms, redundant tags, chat room name, etc. but don't get sufficient attention.

Therefore, I went ahead and tried to develop a list of questions that are relevant today for betas. The most important questions, vary from site to site, and from private to public status. The answers to these will have a lasting effect on how your site operates for a very long time. The questions are divided by site status.

Private Beta

1. Are questions about [subject] on or off topic?

You should actively watch the earliest questions with an eye for quality and purpose. Ask yourself: “Is this the type of question we want on this site? Is it pushing the boundaries of on- and off-topic questions? Are we opening a can of worms?” Talk about these issues in meta, early and often. They are the key to establishing the boundaries around your site.
Remember, this is a class of questions, and not just a single question. The more specific topics that get discussed, the more meaningful the discussions will be.
For example, on Chocolate.SE, discussing:
Should dark chocolates be on-topic?
Will most likely result in flame wars, and reach no useful conclusion. However, discussing:
Should the additives used in the process of making dark chocolates be on-topic?
Will lead to more constructive discussion.

2. How should we tag questions about {subject}?

Tagging questions is an ad hoc way of organizing content. It is mostly improvised by users asking the questions but only to a point. Tag auto-completion and community editing will influence the proper use of tags for a very long time.
The type of things you should look out for: how to handle acronyms common to your subject, brand versus product-specific tags, common terminology, and the use of semantic tags to categorize specific types of questions unique to your community. Every site will have their own unique set of tag-related issues.
The best way to identify tagging problems is to watch new posts closely, and try to build tag wiki excerpts that explain what the tags are for. When tags become ambiguous, too specific (or not specific enough), or just somehow off, raise those issues in meta, and quickly. Proper tagging is very much a lead-by-example activity. The sooner you get the “community standards” for tagging ironed out, the less chance you’ll have to face the drudgery of cleaning them up later.

3. Should tag A be a synonym of tag B?

Usually, in the early stages a fair bit of redundancy gets developed as many users may suggest different variations of the same tag, or some may call the same thing by a different name. These can result in disorganization and confusion in using the tags. It is important to look for and discuss such redundancies that could become problems later on.

4. What’s the “elevator pitch” for our site?

Imagine you’ve just gotten on an elevator with a friendly stranger. You have precisely one floor to describe your community to them. What would you say? The elevator pitch is a brief sentence that describes what your site is about.
This helps in visualising what your community is about and what it is going to be. If you have trouble describing your community in one-line, then its a sign that the community needs to take a relook at its scope and purpose.

5. Whats an interesting name for the chat room?

Every site comes with its official site chat room. This is created automatically and can be reached from the Site Switcher on the top left. The chat room is the third place of your community, after main and meta. It is like the lounge where everybody can get together discuss, complain and have fun. Personalizing this space to reflect your community and what you are, is one of the small steps to take towards building an active an engaged community.
Try to get a few ideas up, discuss and vote on them. The name could be something esoteric in your field, or something that every person may understand. Whichever it may be, go ahead and start the discussion!

Public Beta

1. Who should the moderators be?

Moderators in Stack Exchange perform dual roles. Beside the normal activities of a Moderator, part of their function is to act as liaison — a role which links the Stack Exchange team with the individual communities. Discussing the criteria of a great moderator is important and picking out potential candidates is a great way to introduce outstanding contributors to your community. And we are completely open to appointing temporary Moderators when someone’s contribution makes them a standout choice for your community’s human exception handler. If your meta site does not have a post to nominate Moderators, start one now! Pro Tem appointments will begin about two weeks after the site goes public. The more guidance we receive, the more informed our choice.

2. How do we promote our site?

By now, your site should have 100+ questions, mostly excellent with expert level answers. Next comes the issue of getting the word out i.e. how to promote your site and how to reach out to the experts and peers in your industry. Meta is the perfect venue reach out and ask around about who knows whom. Ask your friends to ask their friends. The people needed to make your site a huge success are already within your reach.
Keep in mind, that this is again not one discussion, but a set of discussions that will collectively come under the tag of site-promotion. For any degree of success, it takes a bit more coordination and discipline than asking a one-shot question and expecting a final solution to simply emerge. It takes individual members to rise up and ask methodical, step-wise questions with an end-goal in mind — and follow through.
Some of the avenues to consider are:
  • Encourage your community to share links to outstanding questions and answers. And if you feel you’re not producing outstanding questions and answers worthy of sharing with the world, endeavor to fix that first.
  • Reach the right kind of publications and bloggers. Make sure that the key experts in every field know about the site; not just the “Martha Stewart” big names; we want to talk to the people who go to these conferences.
  • Make sure evangelists from each community have the opportunity to speak before groups of experts in their fields.
  • Encourage and participate in grassroots conversations in existing discussion groups, where appropriate.
You should be thinking in terms of “what is it really going to take to make this a world-class site?” Don't wait for anybody. You have to get the ball rolling. Being a voice in a crowd works for certain types of collaboration and brainstorming, but sometimes you have to take on the role of the organizational czar.

3. What should our documentation contain?

Most of the content present in the Help Center is same across the entire Network. There are some parts however, that need to be discussed and developed within the community. The most important page is the /help/on-topic page that lists broadly the scope of the site. This page acts as a guide for new members of your community and should be carefully considered and put together to accurately represent the site scope.

4. Do we really need tag [subject]?

As tags start to get better organized, there will be many problematic and extremely specific tags that may have been created initially, but do not actually serve any purpose. Such tags may be meta tags, may be redundant, or may even simply be spelling errors. Whatever the case, it is important to bring these up and discuss their need on the site. Many of these will eventually be removed, and the earlier this happens the cleaner the tags on your site stay.

Useful Links


Originally posted on Meta Stack Overflow. Many parts taken as-is from the official blog posts.