By Antoine, on April 27th, 2009
This is another project I have fun on while I get a bit of spare time from having no life. It has actually been in the works for quite a while but I have just recently started the actual programming (in C++). It is called the Elements Framework. Despite being just another tool for myself, I have had a few friends who expressed interest into knowing a bit of details about it. Since I cannot precisely really formulate what it is myself, I though it would be a worthy exercise to write a short description of the framework. Now you might be asking yourself, how the hell can you not know what you have been working on for the past year? Well the answer is: when I think, I do not necessarily use words, I use ideas, I use examples, I use chunks of code, I use drugs, but since I do not work with verbal constructs to begin with, explaining what I do is somewhat tricky because I have to compile a bunch of concepts into correctly formed sentences. Something I can normally do just fine with everyday objects and concepts (I am not speech impaired), but when it concerns software architecture its philosophy, it gets hard, especially if your interlocutor has no clue as to what HTTP really is.
The idea.
The Elements Framework has one simple goal: apply the philosophy of the web to traditional software. Sounds far fetched? Let me explain. The web is to me one if not the greatest of systems created by men. It has not failed once since its inception, it has seen worldwide acceptance within only a few years, it’s easy to use and it’s actually shaping the future of mankind at a rate never seen before. In fact, it is so ubiquitous and important that it is called “The World Wide Web”. Do you know any other system that got popular enough to get the “World Wide” prefix? I don’t. But above all, the workings of the Web are blindingly simple; it is built upon a principle called REST and relies on only two (RESTful) protocols to function: HTTP and DNS (Maybe WAKA soon…)
Before we move on, let me define one crucial thing: the Web and Internet, even if the two terms are often interchanged and refer to the same entity in popular speech, are not the same systems. The Internet is TCP, UDP, IP, ARP, MAC and a bunch of other protocols. In other words, the Internet is everything below layer 5 of the TCP/IP protocol suite. The Web is just HTTP and DNS. Although HTTP is most of the time tunnelled through the Internet, it does not require it to function. HTTP only assumes an error-free transmission, and TCP is the provider of this service in the case of the Internet.
Now, what the Elements Framework will try to achieve is making software systems that run on it behave just like the Web as well as being part of it. Let me develop on this:
- Out of the box, you will be able to “browse” your software system just like you browse a website. In fact, that software system will look exactly like a website. This means that every component will provide its own HTML (of any other markup language) page for users to check and interact with. Through this, it will be possible to build rich user interfaces through the use of javascript, CSS and whatever other markup and styling language. This also means that every web browser will be able to display those interfaces, be it on a *nix, Windows, Cell-phone, etc. True to the REST philosophy, the server provides a representation of the resource and the browser decides how it displays it. Humans will be able to inspect systems and get a sense of how they work just like they can do with websites since the architecture of the system will be an exact match of its apparent organization as seen from the web.
- An Elements system and its components will be naturally distributable, just like the Web. So now, the keyboard, the microphone, the screen, the XML parser and the neural network can be anywhere there is connectivity. If one becomes unavailable, ask Google for another…
- Interaction will solely happen through HTTP. Since the interface will be standard, it will be much easier to connect heterogenic components together, just like Mashups. Moreover, every component of an Elements system will be able to expose its interface to the rest of the world if it wants to.
- The Elements framework wants the web to be a “Web of things” where every single piece of electronic equipment can be connected to it if needed. To achieve that, it will be made as lean as possible in order to function on the smallest microcontrollers. Imagine every electrical switch and outlet in your home being accessible through your browser? Every home would have its own web, shielded from the WWW of course.
How it works.
The basic construction block of an Elements framework is, as you have guessed it, an Element. The active part of an Element is called a Resource. Simply put, a Resource is a lean HTTP server. It takes HTTP messages, checks the URL if the message was for itself, or forwards it to its children if it is directed to them. If the message was its, it processes it. Take for example three Resources, res1, res2 and res3 with res2 and res3 being children of res1. If you want to direct a message to res2, the message’s address would be /res1/res2/ . res1, just by receiving the message, would know it was the intended recipient (we trust DNS and TCP to get stuff where it is suppose to go), so it would look for the url of the message for the next recipient: res2. Turns out res1 has a children named res2, so res1 would then happily pass the message to res2 who, upon analysis, would figure out it is the final recipient because there is nobody after the last “/”. res2 would then process the content of the message and take appropriate action, like sending a response to the originator if the message was a request or turn on a light fixture in your home.
The other elemental building block of an Elements framework is the Authority. Authorities implement concurrency in the Elements framework and are responsible for a whole lot of other things (hence their authoritative nature). They also have everything a Resource has because they inherit from it so for this short description, there is no need to explain more about authorities.
As stated earlier, an Elements system is just like a website. By assembling a bunch of Resources together and by providing them with the amount of specialization required for them to have a bit of intelligence, we can actually build more complex systems that behave like websites. For example, let’s look at what the organisation of a calculator would look like. Mind you this is a rather simplistic because programs as simple as calculators rarely get componentized.
- Calc/
- Parser.Calc/
- Add.Calc/
- Mul.Calc/
When the user directs his browser to http://Calc/ (with an HTTP GET request), he is presented with a webpage (the representation of http://Calc/) containing a simple form: a text box with on its right a button that reads “=” and some instruction text (or images, publicity, whatever you could put on the WWW). Through reading the instructions, the user promptly figures out how this calculator works and proceeds with doing some math. He types a mathematical expression (5 + 7, 125 * 8, etc.) in the text box and then presses “=” to expecting the result to appear on the right of the button. Upon clicking on the “=”, quite a lot will happen in the Elements framework.
First, the user’s browser will send an HTTP POST request to http://Calc/ . When the root Element (whose name is Calc) receives the request, he analyses the header and deducts that he is the intended recipient and that request is actually a user that needs to do some intense computations. Calc is somewhat lazy; his job is just to display the calculator, not to do the calculations. He then proceeds to build another HTTP POST request and sends the expression he received to http://Parser.Calc/.
Parser receives the request, figures it is for him and then proceeds to decompose the expression. “3 + 6″ is two values and one operand, so Parser now packages two HTTP POST requests. He sends one containing 3 to http://Add.Calc/Left and 6 to http://Add.Calc/Right, waits for the responses and then sends a HTTP GET to http://Add.Calc/Result/ who then returns 9 as a response. Parser then packages the 9 in an HTTP response and sends it to Calc who then re-renders a representation of itself, but this time with the result of the computation.
Neat, it took a massive amount of HTTP message producing, passing and parsing for something that could have been done in a single assembly instruction, but it was just an example. However, the Resource Calc could have been instructed to use add.computations.co.nz (In New-Zealand) as an adder, it would have made absolutely no difference. Also, imagine that Parser is for some reason unavailable and the user of the calculator does not know where to find a working one. Other Parsers on the web, being web pages, can be indexed by search engines; all the user needs to do now is search the web for a working parser, instruct calc to use it and he is up and running again. This, of course, could have all been automated in order to get one hell of a resilient calculator.
Should the user becomes curious about the inner workings of the calculator, he is always free to go check out the Resources that compose it with its browser, they are all HTTP servers after all so they have to. An HTTP GET on http://Add.Calc/ could for example return a page that explains a bit about what this adding Resource does, display an overview of its state (does it need an oil change anytime soon?) and give links to Right, Left and Result. Or it could be that the calculator is in fact configurable, and at http://Mul.Calc you can find options to return different format of numbers.
That concludes the example. If you browse a bit, you will realize that this kind of thing is already possible with current technology, it is in fact done every day through the use of Web APIs. Using the web in such a way is what the Web 2.0 is all about and is only a natural consequence of RESTful architectures. If you want something more like this framework, the closest thing I have found is Java Servlets. I have actually never used this technology myself but I imagine that Servlets will not run on a microcontroller with 1kB of RAM; at least the java virtual machine wont. What I am trying to achieve is to make the leanest possible HTTP server so it can be used on any platform; the smallest computers could only have one simple Element while a modern day computer could host tens of thousands. After that, developing more complex systems is just a matter of assembling Elements together. The URLs topology you use to browse an Elements system is an exact representation of the architecture of the system. If the components of a system entertain a relationship that is somewhat similar to tree, where some are responsible for others then I think that system can be made to use the Elements framework.
So if you want a real-life running example of an “almost” Elements system, the Tree Framework (If you develop programs you are cool, but if you develop frameworks, you have attained a higher degree of coolness) on which this website runs is one.
The Elements Framework is still in the forge. When I get a working version, I will make it available as open source software. There are many other projects in the pipeline that will use the Elements framework when it is done. Below are a few examples, maybe they will give more down-to-earth examples of what you can do if you apply REST to classic software making:
- A home automation system, where every outlet, switch and thermostat is accessible from the web and controlled by an Elements system sitting on a server in my house.
- A web-based oscilloscope and logic analyser.
- A web content management system based on the Elements Framework.
- An alarm system.
You can follow developments right here, as children posts of this project post entry. Until then, go create something!
By Antoine, on April 22nd, 2009
I have been using Tree v0.1a, the initial release, for two good months now and I must say I am overly satisfied with the quality of the product. It has not suffered a single failure and is performing up to my expectations.
I know I am not in position to give constructive comments about my own creation, but I think I am well past the “This thing is the next Windows” point. Even if the framework is seriously alpha, I am not the only person who can use it (to a certain extent) and it has not failed miserably whenever I wanted to show it to someone.
Finally some software of my own making that did not get the boot halfway through development and that everyone can use(everyone visiting my site that is). Yes, defects and enhancements are piling up in Trac, but thats a side effect of improvement.
By Antoine, on April 19th, 2009
I am a strong advocate of judgment being the best anti-virus, anti-trojan, anti-worm and anti-etc, but when it comes to protection against intrusion well, judgement cannot be of any help, so I put my trust in firewalls. However, I have lately seen many of my 8-thoushandish (8000, 8001, 8002) ports that I use for development http servers being taken by unknown programs.
In order to identify the culprits, I portscanned all my interfaces (you can do that with nmap) in order to find the associated protocols in the hope that this would give me hints on what processes are to blame. Turns out portscans just give you the name of the protocol that is registed with that port through IANA, which gives you no guaranty the process bound to this port is using that particular protocol (I momentarily had forgotten that TCP and UDP do not care for what they transport). For instance, 8000 is reserved for irdmi, which seems to be lost technology as no ones has clue on what the hell its for.
The commands that are actually needed to find what process owns what port under *nix OSes (beside netstat, whose output I find painful to read) is lsof (list open files):
sudo lsof -i | grep “number of port”
or
sudo lsof -i | grep “protocol name”
If the port has an associated protocol, lsof will use the name of the protocol instead of the number. The output you get, when not piping to grep of course, is a list of every file (ports, sockets, ttys,, RS-232, etc are considered files by *nix OSes) open on your system with the number and name of the process that owns it.
So in the end, 8000 and 8002 belonged to eclipse while 8001 was Camino’s.
By Antoine, on April 15th, 2009
Look at this, its about time someone made good use of the hot-swappable physique of the SATA connector. I can finally take advantage of my many spare SATA drives without having to buy a external enclosure for each of them. I just take the drive out of its anti-static wrapping, mate it with the BlacX, and watch OSX’s Time Machine do its magic…
No seriously, I really needed to improve my backup system. As it stood, someone robbing my house could have easily taken all of my digital life with them, but now, I would still have that hidden drive. The next step is protection against fire and other natural disasters, but for that, I just need another loose drive that I could keep at a friend’s place. Aaaahhh, backups, sometimes, I wish I could still think like the profane, that computers contain a certain type of smoke, and that once the smoke is released, it stops working. But no, I have witnessed too many case of data loss (A few years ago, one guy showed at our service desk with his masters on a fried hard drive) and I still pride myself in saying that I have not been a victim of technology yet.
Philosophy aside, when I received the unit, it had a loose part inside of it. I then began to ponder whether I should return it or just void the warranty, but as it turned out, the unit did not have any sticker (I probably would have done it anyway), as if Thermaltake wanted you to have a look. Moments later, I had found the culprit: a screw. But before putting it back in place, I took a good glance at the insides of the unit and was surprised at how simple and well built it was. Sturdy plastics, thick PCB and simple mechanics that are not going to break any time soon and even if they do, it would be easily fixed. I want to see more products made this way.
If you are planning on getting one, consider the BlacX to be ISO me.
By Antoine, on April 14th, 2009
I have had many requests to enable comments on my blog, but the reality is not so simple… In fact, it would not be enable comments but rather implement comments. During the development of the Tree Framework (on which my blog runs), I had a primitive commenting system working, but for the version that I fielded, I decided not to use it because it was insecure. I do realize a blog without comments is not really a blog, but the Web is a wild place, and leaving a website on its own with unprotected HTML forms is asking for trouble. Spiders can generate canned spam comments faster than I can delete them.
But rest assured, comments are on the top of my priority list, and I have identified a very neat way of securing the whole thing: ReCaptcha. For those too lazy to click the link, ReCaptcha is a system that aims at validating whether the user submitting a form is human or not by having them solve a captcha; a test computers are notoriously bad at. We have all encountered such a thing when registering on a website, but reCaptcha adds to the concept by using character blocks from books that are undergoing digitization. When the computer is unsure of the result of an OCR (optical character recognition), it submits the ambiguous text to the reCaptcha system so a few humans can confirm the answer. What an ingenious way to harvest brain processing power.
By Antoine, on April 9th, 2009
The person that sits behind my cubicle just asked a newcomer whose voice I do not know how she was. She replied positively and returned the question just to receive the answer she was probably expecting: ” Fine, fine, the long week-end is coming.”
I have never heard anyone saying their day was “pretty shitty as usual” or “was fine until I met you” and probably never will, at least not in this environment. If you are not willing to respond with a negative, it makes the question invalid, since the same answer will implicitely ensue. It turns it into a statement, a statement of politeness.
When I approach someone, I try to avoid this idiom and come up with another way of being polite. Asking questions whose answer you do not care about because you already know is quite rude in my opinion, yet, I hear this one going through every single hour…
By Antoine, on April 9th, 2009
I doubt very much my descendants will ever look into what I did with my free time during the course of my existence. Chances are it will not interest them, because few people enjoy listening to elders talk, especially about programming. But for myself, for my own posterity, I though it would be great to have a paper copy of all my projects. Not only to add a fair amount of resiliency to my backup system but also for history’s sake. Some will build murals with photos of the many vacations they went on. I printed hundreds of pages of uncommented code, organized them in nice binders with a cd-rom copy and stored them on shelves. Now anyone can browse through, and see for themselves that it was true I wrote 35 pages on a communication protocol that will never see the light of day just for the fun of it.
Most parents do keep some tracks of their children’s evolution, but the bookkeeping comes to an end once they quit the family’s nest. After which, it becomes financial institutions who remain keen at maintaining an history of one’s existence, a financial history. Although most of us do fancy looking at the many drawings we made when we were in pre-school, we have to admit that they all looked alike and were of relatively poor historical value (unless you grew up to be a graphic artist). What is the most interesting is the intellectual work that comes later-on, those painful essays in high-school, when producing a 150 work text was a chore. If you actually abstract the constant boredom while doing them, you can easily go back trough time and get a glimpse of how you thought back then, how different you were.
By printing my projects, I will be able to repeat the exercise a few years down the line, albeit with a much deeper analysis, as I never pretented to be a writer, but I do pretend to be a progammer and looking at how I programmed in the past, can surely help me improve even more in the future. Even now, I do go back to my previous creations, mostly to dig out a solution on a pattern I remember dealing with in the past. My work is evolutive and most of the ideas that I am implementing as I write these lines were though of a few years ago during the course of another project. Sadly, those ideas very often came with the realization that the work I did up to that point was inherently flawed, and ended up causing the abortion of that current project in most cases; I suppose it was a necessary step. After all, the relativity theory came from Newton’s celestial mechanics, which in turn was built upon the work of many the many obscure astonomers of those ancient times. The human race thinks upon its own intellect and creates upon its previous creations. We are long past the times where seing a boulder going down a slope gave someone the spark of genius that was the wheel. Things are much too complicated nowadays and history helps us avoid repeating the same mistakes, but it also helps us avoid reinventing the wheel every time.
By Antoine, on March 30th, 2009
A year or two ago, I took it upon myself to reverse the trend of computer proliferation in my life. I offloaded the free support that I used to provide to my friends to other knowledgeable friends, I got rid of many machines that were just sitting there for the looks (they were running Folding@Home) and finally centralized all my activities on one laptop instead of having both a desktop for power and a laptop for mobility.
All was great, and the return in time was direct and fast. The amount of effort invested every month maintaining hardware and software went down dramatically. I once read that you need one technician per 50 machines, which makes sense if the people using computers are not so savvy. But you sometimes still need a few machines, be it for testing, running platform dependent software, or just fooling around with another OS. For that problem I found virtualization to be the solution. While it will spawn a host of new difficulties, it does cut down on the hardware woes and once you tame it, it shows lots of potential; so much that in the process of making physical machine virtual, I had the idea of applying the same philosophy to that grey and blue Linksys box that breaks from time to time and requires its fair share of maintenance and configuration.
So there I am with a router that does not exist physically. Friends come to my place and look for the device to hook their laptop to (no wireless for now), but they get directed to my lab machine. They then give me an “are you sure?” look, but hook the cable anyway, and then proceed to using internet, charmed by the “magic black box” phenomena.
Making a router virtual, while not so trivial, does have a few advantages:
- One less computer if you use a machine that is on anyway (lab computer in my case).
- You get to use better routers like Smoothwall, m0n0wall or Fresco without a standalone computer.
- Less power consumption.
- You can backup your router.
- Essentially free
- It is a fun project.
The setup is actually quite simple, I will not go through the whole process command by command (I’ll keep that for later) but here is an overview of what you will need.
- Time
- Patience
- A computer with enough network interfaces.
- An OS (I use Ubuntu 8.04).
- A router distro (Smoothwall, m0n0wall, Fresco, etc.)
- VirtualBox (Or any other emulation/virtualisation software you migh prefer).
- Bridge Utils or any other bridging software.
And here is what you will do:
- Bridge all your network interfaces but leave one outside the bridge, it will be the outside interface. Make sure other programs cannot use it !
- Install VirtualBox.
- Map you bridge and outside network interface with virtual interfaces on VirtualBox.
- Virtualize the router distro on VirtualBox
- Configure the router like is was a real one.
- Unplug your old router, put it back in its box and keep it just in case…
- Admire your work.
As for wireless support, I do not yet possess a wireless adapter that is compatible with madwifi and can be used in master mode; it is on the way. However, I am unsure whether I will be able to bridge to it or not as I have come across many forums posts of people trying without success. If it is a no go, then maybe making the wireless network a different subnet and routing traffic will do the trick…
By Antoine, on March 26th, 2009
I recently found a nifty solution for the crippling problem of cable management. It might not be suitable for super computers but it does work for my desk.
It was built by screwing Ikea Antonius coat hangers to the underside of the desk using long enough wood screws. To give it a bit of stability, I used a hollow tube (like piping cut to length) for the hangers to sit on.
Here are a few pictures:
Simple and inexpensive…
Due to popular demand, here is a diagram I cooked up in paint that better explains the mounting mechanism. I have no particular suggestion for the length of the pipe except that it should be long enough for the hanger to clear the table (so you can slide the cables in). If my memory is correct, mine are 8cm. As for the screw, just make sure it is long enough so it can drive a good 2cm in the table.
Sorry for the lack of information, next time, I will be more thorough in my description and will take more pictures.
By Antoine, on March 18th, 2009
I once owned a monitor with a permanently printed windows 3.1 desktop on it. It was still usable, but when turned off, you could see the damage well enough to read the windows menus on the normally dark screen. This monitor was a 13 inch black and white CRT that was probably manufactured at the very beginning of the 1990s.
CRT are just electron guns aiming at a phosphorous surface. When an electron hits that surface, an electron belonging to a phosphor atom gets excited and when it drops back to its normal energy level, it emits a photon of a certain frequency. Repeat the same process many times per second and in an ordered fashion and you will be able to display graphics. When the image on a CRT is left standing still for a long duration, the repetitive hit of electrons on the inside phosphorous layer degrade it, making it more transparent where the image is the most bright. Screen savers were invented to prevent this phenomenon; to save screens that were left on.
With LCDs, there is no such phenomenon; using a screen saver is dumb and only wastes power. Instead, set your computer to put it in standby after a few minutes of inactivity, or discipline youself and turn it off using the power button, which is better because the screen uses less standby power that way.
|
|
