Looking for well defined HTTP specs for web server develop..

[N1LAF]

.. development

I have RFC2616, but having difficulty making sense of it when it comes to practical matters. I have developed a simple web server in LabVIEW, but limited to connect, read, respond, and close. I have not gotten the persistent connection squared away yet. I would like the abbreviated IRS (Interface Requirements Spec) that is geared to the fast learning curve.

Anyone have low level HTTP experience? I'll share my web server...

So far, the basics are done, I can load pages, files (of reasonable size), and have duplicated some of the CGI functions, such as FORM handling. This is really neat. I have my custom back doors set up for monitoring and change settings via FORMs.

I have three main parallel processes.

1) Connection loop. I created a TCP wait for connection on port (80), and when a connection request comes in, I enqueue the connection ID to a queue, and wait for another connection request. This will allow me to have multiple simultaneous connections.

2) Connection servicing loop. Right now it just reads TCP that was enqueued, I enqueue the parsed packet to process(function), perform function on that packet, and write response to that connection.

3) Function loop. The process request is dequeued, I perform the function requested, then I send it back to the connection servicing loop.

With a web browser, I enter the address: 127.0.0.1 with my web server running, I see the message come in:
GET<spc>/<spc>HTTP<spc>1.1<spc> followed by the rest of the header. The function (as I call it) will be between the GET and HTTP keywords.

I have logging functions built in for troubleshooting, and now I can review the logs by entering the following address: 127.0.0.1/@READLOG
The log pops up on the web browser. This is really cool.

Uses for a custom server will be ham radio interfaces and home monitoring pages. Since I can interface with signals, and the custom server can process and embed the results into a web page. Can you imagine turning on and off the outside lights by cell phone? Today was a breakthrough day.

Any interests? Any help?

[WØTKX]

I have never done that, and am very interested in what you are doing. I would love to be able to switch stuff on and off remotely. I also have a new radio that I can put on the net and some excellent recycled security cameras.

I have only done some back end stuff for web servers, and have never done much website work. Spent more time as an Exchange Server geek, and smacking the heck out of Outlook. About 12 years ago.

Still make really stupid mistakes like putting a new router on the home network and not nailing down the security right. Got a little worm action for that boo boo.

[N1LAF]

I am definitely learning the hard way, looking at packets using Wireshark. I am not sure how to handle large files, and still need to learn to handle persistent communications. It is a pretty neat project. The next part will be streaming audio from computer audio input in real time. Once I have that, I can use the web browser as a remote terminal for my radio.

[W9PDS]

Can't find a simple doc, but perhaps taking a look at the sourcecode for previously done open source projects & reverse engineering from there?

This one comes to mind:
http://perlwebserver.sourceforge.net/

Server.PM might be the module file to look at there, and it looks to be fairly well documented.

[N1LAF]

Right now, it is by trial and error. I did learn something about differences between MS Internet Explorer and Firefox, When a page request from IE is made, they include slashes, while Firefox uses hex codes. I need to detect '%' and decode to proper hex value. I also have to invert the MS IE slashes, from '\' to '/' (or the other way around).

I also notice a flaw with IE, if you put a specific sequence of text, even between <CODE> and </CODE>, MS IE chops off the page up to and including that segment. Interesting.

[n2ize]

Its been a long time since I did any programming with sockets and stuff. It's not really all that hard to figure out how to write the code. The hard part is designing an efficient algorithm that will handle all the expected funtions efficiently, i.e. critical points, connection queues, and avoid lock ups, race conditiions, etc. This is not to say you cannot do it yourself. But it can be a royal pain especially if/when unexpected problems creep in later

As a suggestion why not use a pre-existing server engine built and engineered on a solid algorthm and overall design that has the majority of these things pre-implenented and the critical parts already ironed out. Then simply modify and tweak your code around it. Yes, it is true you will be using someone else's code wrapped in a layer of abstraction but that is not always a bad thing. It can simplify things and can sometimes prevent problems later on.

[kc2orw]

Programmers should keep this old adage in mind
"Why reinvent the wheel"

If your looking for lightweight or embedded start here and work your way out.
http://en.wikipedia.org/wiki/Comparison ... eb_servers (http://en.wikipedia.org/wiki/Comparison_of_lightweight_web_servers)

[N1LAF]

Because I can customize its operations more exactly than a third party, especially developing embedded systems. Now I can 'plug-in' web page interactions with my LabVIEW applications. I can display a configuration page, give the user options to change parameters (using the FORM tags), and saving the configuration file. Once I develop the routine, I can reuse it on other LabVIEW projects. It is really cool.

[n2ize]

Because I can customize its operations more exactly than a third party, especially developing embedded systems. Now I can 'plug-in' web page interactions with my LabVIEW applications. I can display a configuration page, give the user options to change parameters (using the FORM tags), and saving the configuration file. Once I develop the routine, I can reuse it on other LabVIEW projects. It is really cool.

It sounds like something you can do with a pre-existing engine. If you are embedding this whole thing onto a dedicated device there are plenty of lightweight server apps that should do the job. I still fail to see why you need to scratch build the entire thing from the ground up. It doesn't appear as if you are doing anything unique enough to warrant an entire build from the ground up.

[kc2orw]

Because I can customize its operations more exactly than a third party, especially developing embedded systems. Now I can 'plug-in' web page interactions with my LabVIEW applications. I can display a configuration page, give the user options to change parameters (using the FORM tags), and saving the configuration file. Once I develop the routine, I can reuse it on other LabVIEW projects. It is really cool.

It sounds like something you can do with a pre-existing engine. If you are embedding this whole thing onto a dedicated device there are plenty of lightweight server apps that should do the job. I still fail to see why you need to scratch build the entire thing from the ground up. It doesn't appear as if you are doing anything unique enough to warrant an entire build from the ground up.

Exactly my point start with a functional engine and then add or subtract functionality. Hey if you or someone else desired to do it as a "mental exercise" that is just fine. But one doesn't have to reinvent the wheel simply take an existing engine and customize it to what you want it to be.

[W9PDS]

Perl + CGI + LabPerl = win!

http://perldoc.perl.org/CGI.html
http://jeffreytravis.com/lost/labperl.html

(there's a reason why they call perl Duct-tape)


I wouldn't suggest it if I had not already done it myself. I've built web-controllers for my radios before using perl & LibHam, and even wrote a web-based tool to fire off a relay that hits the button on my auto-tuner in perl:

http://w9pds.org/remotetune.html


(but yea, there is something to be said about building it all yourself. That can be fun, provided you have the time to mess with it)

[kc2orw]

Seems a lot of this is already done a cursory search on labview cgi returns results such as these.

http://www.ni.com/support/labview/toolk ... et/faq.htm (http://www.ni.com/support/labview/toolkits/internet/faq.htm)

http://zone.ni.com/devzone/cda/epd/p/id/1940

controlling devices via the web well there are also the x10 devices to act as an interface. But those wouldn't be related to labview though they do seem to work, quite popular.

http://en.wikipedia.org/wiki/X10_(industry_standard)

Perl as PDS referenced is still a fabulous glue language, does most everything, and as referenced has a library for interfacing with labview

[N1LAF]

Because I can customize its operations more exactly than a third party, especially developing embedded systems. Now I can 'plug-in' web page interactions with my LabVIEW applications. I can display a configuration page, give the user options to change parameters (using the FORM tags), and saving the configuration file. Once I develop the routine, I can reuse it on other LabVIEW projects. It is really cool.

It sounds like something you can do with a pre-existing engine. If you are embedding this whole thing onto a dedicated device there are plenty of lightweight server apps that should do the job. I still fail to see why you need to scratch build the entire thing from the ground up. It doesn't appear as if you are doing anything unique enough to warrant an entire build from the ground up.

Exactly my point start with a functional engine and then add or subtract functionality. Hey if you or someone else desired to do it as a "mental exercise" that is just fine. But one doesn't have to reinvent the wheel simply take an existing engine and customize it to what you want it to be.

Re-inventing the wheel is a valuable learning process, having the knowledge of what the input and outputs are. Here is the first process, create a listener on port 80, similar to the LabVIEW example: date server.

[N1LAF]

As you can see, a listener on port 80 is created. Inside the while loop, the wait on listener waits indefinitely until a connection attempt is made. The connection ID is then Enqueued to be processed.

[N1LAF]

Here is the simple Process Loop. If there is a Connection ID on the Queue, it is dequeued, otherwise it waits 20 milliseconds and statuses the Queue again.

When a Connection ID is dequeued, The request is read, processed, and then a response is written, and connection is closed.

The process request function is being done now. When I learn more about persistent connections, the Process While loop will change accordingly.

[N1LAF]

As you can see, the LabVIEW implementation is easy.

The toolkit costs $532.00 - no thanks.

[M0GLO]

If you are building web interfaces to control things in your home DO NOT forget login security.
Default HTTP login is in no way secure enough to trust your appliances or electric bill to.
Neither is any CGI you are likely to write yourself.

If you are going to do this kind of thing it is imperative that you use a product like SiteMinder or Sun Access Manager.
I personally recommend the Sun product as it is free for home use and quite powerful.

Very cool project, but don't let the black hats p4wn your house or radios!

[kd6nig]

If you are building web interfaces to control things in your home DO NOT forget login security.
Default HTTP login is in no way secure enough to trust your appliances or electric bill to.
Neither is any CGI you are likely to write yourself.

If you are going to do this kind of thing it is imperative that you use a product like SiteMinder or Sun Access Manager.
I personally recommend the Sun product as it is free for home use and quite powerful.

Very cool project, but don't let the black hats p4wn your house or radios!

You took the words out of my mouth. Its not something you'd want people getting into. I'd run the host page on a very odd port or something as well. Sure, it can be scanned, but leaving it on port 80 will really subject it to probes.

Sometimes I think some of the stuff accessible on the internet shouldn't be. Honestly, I'd put something like this on a phone line or something with some kind of control that way-and just monitor whats happening from a website, but thats just me. Some of the things they have integrated into the internet are scary if the security isn't right and constantly monitored :)

I get bad visions, like the scene in I, Robot where he queries the robot that tells him that the house is set for destruction at 8am the next morning, and then right after he walks away it changes to 8pm that day, and its 7:59.....

[N1LAF]

...

[n2ize]

If you are building web interfaces to control things in your home DO NOT forget login security.
Default HTTP login is in no way secure enough to trust your appliances or electric bill to.
Neither is any CGI you are likely to write yourself.

If you are going to do this kind of thing it is imperative that you use a product like SiteMinder or Sun Access Manager.
I personally recommend the Sun product as it is free for home use and quite powerful.

Very cool project, but don't let the black hats p4wn your house or radios!

I doubt he'll have any problem. When I first learned how to create sockets I went a little nuts writing all sorts of insane apps so I could connect to them from remote locations and get them to do every nutty thing in the book, from spitting out information, to making my system do all sorts of things, to serving random numbers, to solving all sorts of crazy equations and just about every idea I could cook up. half the time I was scared shitless, telling myself... "damned, sooner or later you're gonna get yourself pwned". But I managed to keep it pretty darned stealth and never had a problem. matter of fact the only apps that I had that got attacked were commercial apps on windows. None of the Linux stuff or the crazy stuff I wrote ever got touched.

I should cook something together again soon. I've been teaching myself a Lisp variant and I know I am gonna be pretty hyped up and anxious to write something. of course I will have to make sure that whatever it is I can hit it from anywhere.

[n2ize]

...

You've got it locked down good. Nobody's gonna mess with it.