Posted on

The Elements framework

This is another project I have fun on while I get a bit of spare time from having no life. It has actually been in the works for quite a while but I have just recently started the actual programming (in C++). It is called the Elements Framework. Despite being just another tool for myself, I have had a few friends who expressed interest into knowing a bit of details about it. Since I cannot precisely really formulate what it is myself, I though it would be a worthy exercise to write a short description of the framework. Now you might be asking yourself, how the hell can you not know what you have been working on for the past year? Well the answer is: when I think, I do not necessarily use words, I use ideas, I use examples, I use chunks of code, I use drugs, but since I do not work with verbal constructs to begin with, explaining what I do is somewhat tricky because I have to compile a bunch of concepts into correctly formed sentences. Something I can normally do just fine with everyday objects and concepts (I am not speech impaired), but when it concerns software architecture its philosophy, it gets hard, especially if your interlocutor has no clue as to what HTTP really is.

The idea.

The Elements Framework has one simple goal: apply the philosophy of the web to traditional software. Sounds far fetched? Let me explain. The web is to me one if not the greatest of systems created by men. It has not failed once since its inception, it has seen worldwide acceptance within only a few years, it’s easy to use and it’s actually shaping the future of mankind at a rate never seen before. In fact, it is so ubiquitous and important that it is called “The World Wide Web”. Do you know any other system that got popular enough to get the “World Wide” prefix? I don’t. But above all, the workings of the Web are blindingly simple; it is built upon a principle called REST and relies on only two (RESTful) protocols to function: HTTP and DNS (Maybe WAKA soon…)

Before we move on, let me define one crucial thing: the Web and Internet, even if the two terms are often interchanged and refer to the same entity in popular speech, are not the same systems. The Internet is TCP, UDP, IP, ARP, MAC and a bunch of other protocols. In other words, the Internet is everything below layer 5 of the TCP/IP protocol suite. The Web is just HTTP and DNS. Although HTTP is most of the time tunnelled through the Internet, it does not require it to function. HTTP only assumes an error-free transmission, and TCP is the provider of this service in the case of the Internet.

Now, what the Elements Framework will try to achieve is making software systems that run on it behave just like the Web as well as being part of it. Let me develop on this:

  • Out of the box, you will be able to “browse” your software system just like you browse a website. In fact, that software system will look exactly like a website. This means that every component will provide its own HTML (of any other markup language) page for users to check and interact with. Through this, it will be possible to build rich user interfaces through the use of javascript, CSS and whatever other markup and styling language. This also means that every web browser will be able to display those interfaces, be it on a *nix, Windows, Cell-phone, etc. True to the REST philosophy, the server provides a representation of the resource and the browser decides how it displays it. Humans will be able to inspect systems and get a sense of how they work just like they can do with websites since the architecture of the system will be an exact match of its apparent organization as seen from the web.
  • An Elements system and its components will be naturally distributable, just like the Web. So now, the keyboard, the microphone, the screen, the XML parser and the neural network can be anywhere there is connectivity. If one becomes unavailable, ask Google for another…
  • Interaction will solely happen through HTTP. Since the interface will be standard, it will be much easier to connect heterogenic components together, just like Mashups. Moreover, every component of an Elements system will be able to expose its interface to the rest of the world if it wants to.
  • The Elements framework wants the web to be a “Web of things” where every single piece of electronic equipment can be connected to it if needed. To achieve that, it will be made as lean as possible in order to function on the smallest microcontrollers. Imagine every electrical switch and outlet in your home being accessible through your browser? Every home would have its own web, shielded from the WWW of course.

How it works.

The basic construction block of an Elements framework is, as you have guessed it, an Element. The active part of an Element is called a Resource. Simply put, a Resource is a lean HTTP server. It takes HTTP messages, checks the URL if the message was for itself, or forwards it to its children if it is directed to them. If the message was its, it processes it. Take for example three Resources, res1, res2 and res3 with res2 and res3 being children of res1. If you want to direct a message to res2, the message’s address would be /res1/res2/ . res1, just by receiving the message, would know it was the intended recipient (we trust DNS and TCP to get stuff where it is suppose to go), so it would look for the url of the message for the next recipient: res2. Turns out res1 has a children named res2, so res1 would then happily pass the message to res2 who, upon analysis, would figure out it is the final recipient because there is nobody after the last “/”. res2 would then process the content of the message and take appropriate action, like sending a response to the originator if the message was a request or turn on a light fixture in your home.

The other elemental building block of an Elements framework is the Authority. Authorities implement concurrency in the Elements framework and are responsible for a whole lot of other things (hence their authoritative nature). They also have everything a Resource has because they inherit from it so for this short description, there is no need to explain more about authorities.

As stated earlier, an Elements system is just like a website. By assembling a bunch of Resources together and by providing them with the amount of specialization required for them to have a bit of intelligence, we can actually build more complex systems that behave like websites. For example, let’s look at what the organisation of a calculator would look like. Mind you this is a rather simplistic because programs as simple as calculators rarely get componentized.

  • Calc/
    • Parser.Calc/
    • Add.Calc/
      • /Left
      • /Right
      • /Result
    • Mul.Calc/
      • /Left
      • /Right
      • /Result

When the user directs his browser to http://Calc/ (with an HTTP GET request), he is presented with a webpage (the representation of http://Calc/) containing a simple form: a text box with on its right a button that reads “=” and some instruction text (or images, publicity, whatever you could put on the WWW). Through reading the instructions, the user promptly figures out how this calculator works and proceeds with doing some math. He types a mathematical expression (5 + 7, 125 * 8, etc.) in the text box and then presses “=” to expecting the result to appear on the right of the button. Upon clicking on the “=”, quite a lot will happen in the Elements framework.

First, the user’s browser will send an HTTP POST request to http://Calc/ . When the root Element (whose name is Calc) receives the request, he analyses the header and deducts that he is the intended recipient and that request is actually a user that needs to do some intense computations. Calc is somewhat lazy; his job is just to display the calculator, not to do the calculations. He then proceeds to build another HTTP POST request and sends the expression he received to http://Parser.Calc/.

Parser receives the request, figures it is for him and then proceeds to decompose the expression. “3 + 6” is two values and one operand, so Parser now packages two HTTP POST requests. He sends one containing 3 to http://Add.Calc/Left and 6 to http://Add.Calc/Right, waits for the responses and then sends a HTTP GET to http://Add.Calc/Result/ who then returns 9 as a response. Parser then packages the 9 in an HTTP response and sends it to Calc who then re-renders a representation of itself, but this time with the result of the computation.

Neat, it took a massive amount of HTTP message producing, passing and parsing for something that could have been done in a single assembly instruction, but it was just an example. However, the Resource Calc could have been instructed to use add.computations.co.nz (In New-Zealand) as an adder, it would have made absolutely no difference. Also, imagine that Parser is for some reason unavailable and the user of the calculator does not know where to find a working one. Other Parsers on the web, being web pages, can be indexed by search engines; all the user needs to do now is search the web for a working parser, instruct calc to use it and he is up and running again. This, of course, could have all been automated in order to get one hell of a resilient calculator.

Should the user becomes curious about the inner workings of the calculator, he is always free to go check out the Resources that compose it with its browser, they are all HTTP servers after all so they have to. An HTTP GET on http://Add.Calc/ could for example return a page that explains a bit about what this adding Resource does, display an overview of its state (does it need an oil change anytime soon?) and give links to Right, Left and Result. Or it could be that the calculator is in fact configurable, and at http://Mul.Calc you can find options to return different format of numbers.

That concludes the example. If you browse a bit, you will realize that this kind of thing is already possible with current technology, it is in fact done every day through the use of Web APIs. Using the web in such a way is what the Web 2.0 is all about and is only a natural consequence of RESTful architectures. If you want something more like this framework, the closest thing I have found is Java Servlets. I have actually never used this technology myself but I imagine that Servlets will not run on a microcontroller with 1kB of RAM; at least the java virtual machine wont. What I am trying to achieve is to make the leanest possible HTTP server so it can be used on any platform; the smallest computers could only have one simple Element while a modern day computer could host tens of thousands. After that, developing more complex systems is just a matter of assembling Elements together. The URLs topology you use to browse an Elements system is an exact representation of the architecture of the system. If the components of a system entertain a relationship that is somewhat similar to tree, where some are responsible for others then I think that system can be made to use the Elements framework.

So if you want a real-life running example of an “almost” Elements system, the Tree Framework (If you develop programs you are cool, but if you develop frameworks, you have attained a higher degree of coolness) on which this website runs is one.

The Elements Framework is still in the forge. When I get a working version, I will make it available as open source software. There are many other projects in the pipeline that will use the Elements framework when it is done. Below are a few examples, maybe they will give more down-to-earth examples of what you can do if you apply REST to classic software making:

  • A home automation system, where every outlet, switch and thermostat is accessible from the web and controlled by an Elements system sitting on a server in my house.
  • A web-based oscilloscope and logic analyser.
  • A web content management system based on the Elements Framework.
  • An alarm system.

You can follow developments right here, as children posts of this project post entry. Until then, go create something!

Posted on

Impressions of v0.1a

I have been using Tree v0.1a, the initial release, for two good months now and I must say I am overly satisfied with the quality of the product. It has not suffered a single failure and is performing up to my expectations.

I know I am not in position to give constructive comments about my own creation, but I think I am well past the “This thing is the next Windows” point. Even if the framework is seriously alpha, I am not the only person who can use it (to a certain extent) and it has not failed miserably whenever I wanted to show it to someone.

Finally some software of my own making that did not get the boot halfway through development and that everyone can use(everyone visiting my site that is). Yes, defects and enhancements are piling up in Trac, but thats a side effect of improvement.

Posted on

Giving a meaning to port scans

I am a strong advocate of judgment being the best anti-virus, anti-trojan, anti-worm and anti-etc, but when it comes to protection against intrusion well, judgement cannot be of any help, so I put my trust in firewalls. However, I have lately seen many of my 8-thoushandish (8000, 8001, 8002) ports that I use for development http servers being taken by unknown programs.

In order to identify the culprits, I portscanned all my interfaces (you can do that with nmap) in order to find the associated protocols in the hope that this would give me hints on what processes are to blame. Turns out portscans just give you the name of the protocol that is registed with that port through IANA, which gives you no guaranty the process bound to this port is using that particular protocol (I momentarily had forgotten that TCP and UDP do not care for what they transport). For instance, 8000 is reserved for irdmi, which seems to be lost technology as no ones has clue on what the hell its for.

The commands that are actually needed to find what process owns what port under *nix OSes (beside netstat, whose output I find painful to read) is lsof (list open files):

sudo lsof -i | grep “number of port”

or

sudo lsof -i | grep “protocol name”

If the port has an associated protocol, lsof will use the name of the protocol instead of the number. The output you get, when not piping to grep of course, is a list of every file (ports, sockets, ttys,, RS-232, etc are considered files by *nix OSes) open on your system with the number and name of the process that owns it.

So in the end, 8000 and 8002 belonged to eclipse while 8001 was Camino’s.

Posted on

The BlacX

Look at this, its about time someone made good use of the hot-swappable physique of the SATA connector. I can finally take advantage of my many spare SATA drives without having to buy a external enclosure for each of them. I just take the drive out of its anti-static wrapping, mate it with the BlacX, and watch OSX’s Time Machine do its magic…

No seriously, I really needed to improve my backup system. As it stood, someone robbing my house could have easily taken all of my digital life with them, but now, I would still have that hidden drive. The next step is protection against fire and other natural disasters, but for that, I just need another loose drive that I could keep at a friend’s place. Aaaahhh, backups, sometimes, I wish I could still think like the profane, that computers contain a certain type of smoke, and that once the smoke is released, it stops working. But no, I have witnessed too many case of data loss (A few years ago, one guy showed at our service desk with his masters on a fried hard drive) and I still pride myself in saying that I have not been a victim of technology yet.

Philosophy aside, when I received the unit, it had a loose part inside of it. I then began to ponder whether I should return it or just void the warranty, but as it turned out, the unit did not have any sticker (I probably would have done it anyway), as if Thermaltake wanted you to have a look. Moments later, I had found the culprit: a screw. But before putting it back in place, I took a good glance at the insides of the unit and was surprised at how simple and well built it was. Sturdy plastics, thick PCB and simple mechanics that are not going to break any time soon and even if they do, it would be easily fixed. I want to see more products made this way.

If you are planning on getting one, consider the BlacX to be ISO me.

Posted on

No comments

I have had many requests to enable comments on my blog, but the reality is not so simple… In fact, it would not be enable comments but rather implement comments. During the development of the Tree Framework (on which my blog runs), I had a primitive commenting system working, but for the version that I fielded, I decided not to use it because it was insecure. I do realize a blog without comments is not really a blog, but the Web is a wild place, and leaving a website on its own with unprotected HTML forms is asking for trouble. Spiders can generate canned spam comments faster than I can delete them.

But rest assured, comments are on the top of my priority list, and I have identified a very neat way of securing the whole thing: ReCaptcha. For those too lazy to click the link, ReCaptcha is a system that aims at validating whether the user submitting a form is human or not by having them solve a captcha; a test computers are notoriously bad at. We have all encountered such a thing when registering on a website, but ReCaptcha adds to the concept by using character blocks from books that are undergoing digitization. When the computer is unsure of the result of an OCR (optical character recognition), it submits the ambiguous text to the ReCaptcha system so a few humans can confirm the answer. What an ingenious way to harvest brain processing power.