Thread: I, MutantJohn, was... Wrong!!!! (A Node.js Journey)

  1. #1
    Registered User MutantJohn's Avatar
    Join Date
    Feb 2013
    Posts
    2,665

    I, MutantJohn, was... Wrong!!!! (A Node.js Journey)

    So, I'm not sure if anyone recalls but the last time I tried to use Node.js, it left a bad taste in my mouth and I vented some of that frustration here.

    Well, I'm here to say that I was being foolish.

    Last time, I tried to use Node.js to replace the combination of PHP and Apache for generating directory indexes. This is actually pretty straightforward using Apache as Apache will check each directory for an index.html, index.php or custom user-script for each directory request sent via HTTP. PHP is pretty easy to use in this regard and the depth of customization is pretty great too.

    With Node, I looked and looked and couldn't really find a straight-forward example.

    You guys, I did it! I finally figured it out! It turns out, you can use the Express module to handle GET requests with a regular expression. Ultimately, I wound up with something like this :

    index.js (this code is NOT cleaned up and likely has bugs [development was an arduous process and I had to get used to the idiosyncrasies of every module I was using])
    Code:
    var express = require( "express" );
    var fs      = require( "fs" );
    var jade    = require( "jade" );
    var pathModule    = require( "path" );
    
    
    // create the express object
    var app = express();
    
    
    // use Jade as our HTML templating engine
    app.set( "views", "./views" );
    app.set( "view engine", "jade" );
    
    
    // handle generic GET requests for anywhere
    // from the site root and below
    app.get( "/*", function( req, res )
    {
        var path = __dirname + req.url;
    
    
        fs.stat( path, function( err, stats )
        {
            if ( err )
            {
                // console.log( err );
                res.status( 404 ).send( 'Sorry, we cannot find that!' );
            }
    
    
            else if ( stats.isDirectory( path ) )
            {
                // res.render( "index", { title : "Directory",
                //                        message : "Valid directory path : " + req.url } );
    
    
                fs.readdir( path, function( err, files )
                {
                    if ( err )
                    {
                        res.send( "Error reading directory contents" );
                        return;
                    }
    
    
                    var title = pathModule.basename( req.url );
                    var hrefs = [];
    
    
                    for ( var i = 0; i < files.length; ++i )
                    {
                        // hrefs.push( req.url + "/" + files[ i ] );
                        hrefs.push( title + "/" + files[ i ] );
    
    
                        // remove slashes from front of href
                        while ( hrefs[ i ][ 0 ] === "/" )
                        {
                            hrefs[ i ] = hrefs[ i ].substring( 1 );
                        }
                    }
    
    
                    var fileData = [];
    
    
                    for ( var i = 0; i < files.length; ++i )
                    {
                        fileData.push( { "fileName" : files[ i ],
                                         "href" : hrefs[ i ] } );
                    }
    
    
                    res.render( "index", { "title" : title,
                                           "fileData" : fileData } );
                } );
            }
    
    
            else if ( stats.isFile() )
            {
                res.send( "Will eventually add file support sometime in the future" );
            }
        } );
    } );
    
    
    var server = app.listen( 1337, function()
    {
        var host = server.address().address;
        var port = server.address().port;
        console.log( "Example server running at http://%s:%s", host, port );    
    });
    views/index.jade
    Code:
    html
        head
            title!= title
        body
            ol
                each data, i in fileData
                    li  #[a(href=data.href) #{data.fileName}]
    I was silly before and now I "get it". I had to wind up learning how to use Jade and views and stuff like that but so far, it seems to work pretty well.

    There are still some thing I absolutely do not like about Node and that's its lack of an "official" standard library. I'm sure there is one, yes, but it's very small and not very robust. Everything I wind up using is from a third-party and I hate that.

    PHP is significantly better in this one regard that it's official standard includes many of the features Node leaves up to the world to implement.

    That being said, Express and Jade are still pretty decent. Making HTML in Jade is more user-friendly than PHP's DOMDocument or just using echo statements even though you're left up to the Jade developers' mercy.

    All in all, Node's not bad and I'm sorry I judged it so harshly in the beginning. The learning is a little steep and there's literally not one good tutorial I could find about creating a directory indexer in Node so I hope this one can be improved and I can make a blog post about it or something.
    Last edited by MutantJohn; 09-20-2015 at 01:20 PM.

  2. #2
    Registered User Alpo's Avatar
    Join Date
    Apr 2014
    Posts
    877
    Quote Originally Posted by MutantJohn
    you can use the Express module to handle GET requests with a regular expression.
    Finally! I tried showing you how that worked and you just said it looked complicated lol!

    I'm currently doing my first real paid development work, and NodeJS has come through in everything I've wanted from it so far: routing, templating, form parsing, websocket based chat rooms, and even sending email. There are still many aspects that I've never looked into, like how node handles databases etc, but for everything that I've needed, it's been able to do.

    I've got a few gripes with it too about the lack of good documentation, especially for the 3rd party modules.

    That said though, I actually learn a lot faster by poking around inside the modules in interactive mode than I would reading documentation. For instance I mapped out all of the socket.io modules by making many calls like this (interactive mode):

    Code:
    > var http = require("http");
    undefined
    > var sockio = require("socket.io");
    undefined
    > var server = http.createServer(function () {});
    undefined
    > var io = sockio.listen(server);
    undefined
    > console.dir(io);
    { nsps: 
       { '/': 
          { name: '/',
            server: [Circular],
    
    /* --- EDITED DOWN --- */
    
         maxHttpBufferSize: 100000000,
         transports: [ 'polling', 'websocket' ],
         allowUpgrades: true,
         allowRequest: [Function],
         cookie: 'io',
         ws: 
          { domain: null,
            _events: {},
            _maxListeners: 10,
            options: [Object],
            path: null,
            clients: [] },
         _events: { connection: [Function] } } }
    undefined
    >
    So I would recommend doing that if you can't find reliable documentation for some module. It's one of the greatest strengths of NodeJS.

    I haven't used Jade myself, but I understand it has methods that allow it to be used as middleware by Express more easily than other template engines like Handlebars (that's the one I'm researching). From what I remember of Jade, people complain about it's steep learning curve, but credit it with having a lot of features.

    Express works quite well though. I don't really enjoy the "middleware" architecture, but it's simple enough to make work.

    Just make sure to make a dependency file that specifies the current versions of which modules you are using, because NodeJS module developers break backwards compatibility all the time, so you can't rely on correct behaviour if you always install the newest version.


    I thought you might be interested, that I'm writing a very similar thing to make a client side picture slide show with a requested number of random pictures for the site. I had only worked on it for about an hour before getting on here, but it's nearly done. The one difference is that I know where the folder is, and I'm not templating here, but just making a JSON array of image locations:

    Code:
    // Just the request handler, it's wrapped in a closure.
    
    return function (req, res) {
    
            // Parse the number of requested pictures, from the query string part of the url
            // Default to 10 pictures if the query is invalid
    
            var query = url.parse(req.url).query,
                parsed = typeof query === "string" ? qs.parse(query.toLowerCase()) : null,
                numberRequested = (parsed.number && parseInt(parsed.number, 10)) || 10;
    
    
            console.log("Got image request: " + req.url);
            console.log((parsed.number ? "The query was a valid number: " + parsed.number + "." : "The query was invalid defaulting to : " + numberRequested));
    
    
            // Read the directory with the images, create an array of random images either:
            // 1. The number of images requested
            // 2. The number of images in the directory (if requested number was greater)
    
            fs.readdir(__dirname + publicDir + relativeDir, function (err, list) {
                var maxImages = 0,
                    candidate = null,
                    result = [];
                if (!err) {
                    console.log("Available images: " + list + "\n");
                    try {
                        maxImages = Math.min(numberRequested, list.length);
    
                        // Loop through list and collect requested number of unique images
    
                        while (result.length < maxImages) {
                            candidate = getRandomElement(list);
                            if (result.indexOf(candidate) === -1) {
                                result.push(candidate);
                            }
                        }
    
                        // Convert the resulting image paths to a JSON string, 
                        // prepend the relative directory, 
                        // and inform the client of type and length, and send
    
                        result = JSON.stringify(result.map(prependRelativeDir));
                        console.log("Sending images: " + result);
                        res.writeHead(200, {"Content-Type": "application/json", "Content-Length": (new Buffer(result)).length});
                        res.write(result);
                        res.end();
                    } catch (e) {
                        res.writeHead(500);
                        res.write("Server: Error: Error in sending list of images");
                        res.end();
                        throw e;
                    }
                } else {
                    // Filled in later
                    throw err;
                }
            });
        };
    Remember to use the "writeHead" of the http response to specify the status code of responses. Also I couldn't see if you are using it, but you will probably be interested to know that the HTTP response object available inside Express methods has a "sendFile" method that will just send a file for you, which is way easier than the equivalent bare bones NodeJS approach.

    Quote Originally Posted by MutantJohn
    Code:
    app.listen( 1337,
    Code:
    throw new MemeError("We got a bad-ass over here dot PNG");

    PS: I've had an idea for a module that would make using the bare HTTP response object easier, I haven't worked on it yet, but the basic idea is like this:

    Code:
        function respond (res) {
            return (new function (res, done) {
                this.withJSON = function (obj) {
                    if (!done) {
                        var json = JSON.stringify(obj);
                        res.writeHead(200, {"Content-Length": (new Buffer(json)).length, "Content-Type": "application/json"});
                        res.write(json);
                        res.end();
                        done = true;
                    }
                    return this;
                };
                this.withError = function (error) {
                    if (!done) {
                        res.writeHead(500);
                        res.write(error || "Server Error.");
                        res.end();
                        done = true;
                    }
                    return this;
                };
            }(res, false));
        }
    Last edited by Alpo; 09-20-2015 at 06:40 PM.
    WndProc = (2[b] || !(2[b])) ? SufferNobly : TakeArms;

  3. #3
    Registered User MutantJohn's Avatar
    Join Date
    Feb 2013
    Posts
    2,665
    Quote Originally Posted by Alpo View Post
    Finally! I tried showing you how that worked and you just said it looked complicated lol!
    I have to remember you're not my wife. When I panic, I kind of shut down. It's lame, I know. You may not have been around for it but I was pretty stubborn about learning C++ in the beginning too. After all, why would I ever want to leave C behind? XD

    I'm currently doing my first real paid development work, and NodeJS has come through in everything I've wanted from it so far: routing, templating, form parsing, websocket based chat rooms, and even sending email. There are still many aspects that I've never looked into, like how node handles databases etc, but for everything that I've needed, it's been able to do.
    Hey, that's great! I remember you really wanting to break into the professional development world too.

    I've got a few gripes with it too about the lack of good documentation, especially for the 3rd party modules.
    An adequately documented standardization would solve this quite elegantly.

    That said though, I actually learn a lot faster by poking around inside the modules in interactive mode than I would reading documentation. For instance I mapped out all of the socket.io modules by making many calls like this (interactive mode):

    Code:
    > var http = require("http");
    undefined
    > var sockio = require("socket.io");
    undefined
    > var server = http.createServer(function () {});
    undefined
    > var io = sockio.listen(server);
    undefined
    > console.dir(io);
    { nsps: 
       { '/': 
          { name: '/',
            server: [Circular],
    
    /* --- EDITED DOWN --- */
    
         maxHttpBufferSize: 100000000,
         transports: [ 'polling', 'websocket' ],
         allowUpgrades: true,
         allowRequest: [Function],
         cookie: 'io',
         ws: 
          { domain: null,
            _events: {},
            _maxListeners: 10,
            options: [Object],
            path: null,
            clients: [] },
         _events: { connection: [Function] } } }
    undefined
    >
    So I would recommend doing that if you can't find reliable documentation for some module. It's one of the greatest strengths of NodeJS.
    This is actually really great advice! Thanks! I'll use it the next time I want to learn about a module. Express, Jade, FS and Path all have great documentation though.

    I haven't used Jade myself, but I understand it has methods that allow it to be used as middleware by Express more easily than other template engines like Handlebars (that's the one I'm researching). From what I remember of Jade, people complain about it's steep learning curve, but credit it with having a lot of features.
    This is good to hear. I've heard of Handebars but didn't know what it was. Jade is pretty sweet though. If you need a multidimension array passed to it, it's easier to just an array of JSON structures instead of a JSON structure of arrays.

    Express works quite well though. I don't really enjoy the "middleware" architecture, but it's simple enough to make work.
    Express does its job pretty well, I gotta say. I just wish it was part of some sort of an "official" standard.

    Just make sure to make a dependency file that specifies the current versions of which modules you are using, because NodeJS module developers break backwards compatibility all the time, so you can't rely on correct behaviour if you always install the newest version.
    Ah yes, the curse of the bleeding edge.

    Remember to use the "writeHead" of the http response to specify the status code of responses. Also I couldn't see if you are using it, but you will probably be interested to know that the HTTP response object available inside Express methods has a "sendFile" method that will just send a file for you, which is way easier than the equivalent bare bones NodeJS approach.
    Yeah, there's a lot of really awesome stuff in there that I was going to look through later. Thanks for the tip about writeHead though.

    Code:
    throw new MemeError("We got a bad-ass over here dot PNG");
    That's just how I roll


    PS: I've had an idea for a module that would make using the bare HTTP response object easier, I haven't worked on it yet, but the basic idea is like this:

    Code:
        function respond (res) {
            return (new function (res, done) {
                this.withJSON = function (obj) {
                    if (!done) {
                        var json = JSON.stringify(obj);
                        res.writeHead(200, {"Content-Length": (new Buffer(json)).length, "Content-Type": "application/json"});
                        res.write(json);
                        res.end();
                        done = true;
                    }
                    return this;
                };
                this.withError = function (error) {
                    if (!done) {
                        res.writeHead(500);
                        res.write(error || "Server Error.");
                        res.end();
                        done = true;
                    }
                    return this;
                };
            }(res, false));
        }
    Interesting. Be sure to post it when it's more fleshed out!

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Help MutantJohn understand threads!
    By MutantJohn in forum Tech Board
    Replies: 20
    Last Post: 02-21-2015, 01:20 AM
  2. Let's help MutantJohn modify a terminal emulator...
    By MutantJohn in forum Tech Board
    Replies: 7
    Last Post: 06-24-2014, 02:59 PM
  3. A journey to nowhere
    By VirtualAce in forum A Brief History of Cprogramming.com
    Replies: 7
    Last Post: 03-31-2007, 11:55 AM
  4. Journey time prog 1 minute wrong
    By mike_g in forum C Programming
    Replies: 4
    Last Post: 10-12-2006, 03:41 AM
  5. Magneto's journey to becoming a C++ expert.
    By Magneto in forum C++ Programming
    Replies: 21
    Last Post: 07-17-2005, 08:16 PM