"Blob" server?

Printable View

11-23-2009
cyberfish

"Blob" server?

I have this strange idea about making a generic and very simple database server that will store nothing more than key/value pairs. Keys will be strings, and values can be anything serializable (the server won't interpret the values). It's enough for many applications - synchronizing RSS feeds "read" status, share bookmarks, short notes, application settings, etc. In general, applications that require synchronization by sharing small "blobs" of data.

The key is to be extremely simple (for clients). Connections can be done over HTTP (with the server written in PHP), since HTTP libraries exist for just about all languages under the sun, and should be stateless (efficiency is not a big concern here). Authentication could be done using challenge-and-response, but I don't see how it can be made stateless. Maybe not even that (just hashed password).

There would just be a store(key, value) function, and a retrieve(key) function.

And of course, each server can only support 1 user, but a server is no more than a directory on a HTTP server, so it's easy to run 1 "server" for each user.

This is a very simple idea, and I'm guessing I'm not the first one to think of it, but I can't seem to find anything like that on Google.

Many applications don't need the complexity of relational databases, and this can make life easier for many programmers.

Thoughts?
11-23-2009
MK27

Quote:

Originally Posted by cyberfish

This is a very simple idea, and I'm guessing I'm not the first one to think of it, but I can't seem to find anything like that on Google.

I kind of agree SQL style relational databases are over rated and over used. I might guess that the reason you haven't heard of anything like this is because it is so simple -- by the time most web programmers realize they don't have to use SQL, they probably also realize most (or a lot) of stuff can be done just as well only easier with a flat file. I just wrote a site search engine for someone, extracted all the text into a single file -- 9 gb, which is a lot of text -- but you can still perform regexs on that in no time at all. Here's your key value:

Code:

filepath**filedata

Which regular expressions are a lot more powerful (but slower) tool than SQL IMO (but I'm a lil' ignorant, too), and if you are going to code it in PHP, you might include that option.

Also: there are some extemely popular perl modules (YAML, Storable*) around that will serialize hashes, which is what you are talking about. AFAIK you have to load the whole thing into memory tho, which is no good, but doubtless there are other modules that build on this to overcome that limitation and if you looked around on CPAN you'd find them. Those will be in widespread use, but they probably do not have splashy homesites, etc, and would be tricky to detect via google.

So it's probably a good idea in the sense that there will still be a lot of people around who could make use of it if they were aware of the option. Niche niche niche!

* I believe these are some of the successors to dbm that laserlight refers to, dbm modules being available in most scripting languages I think
11-24-2009
laserlight

Quote:

Originally Posted by cyberfish

This is a very simple idea, and I'm guessing I'm not the first one to think of it, but I can't seem to find anything like that on Google.

It sounds like you want to re-invent dbm and its successors.
11-24-2009
MK27

One issue with PHP I read about recently is that it does not work well with mpm_worker, which is the (faster) threaded setup of apache (vs. the forking setup)...;)
11-24-2009
laserlight

Quote:

Originally Posted by MK27

One issue with PHP I read about recently is that it does not work well with mpm_worker, which is the (faster) threaded setup of apache (vs. the forking setup)...

Yes, the PHP manual's installation instructions clearly warn about this. It is in the installation FAQ.
11-24-2009
brewbuck

Quote:

Originally Posted by MK27

One issue with PHP I read about recently is that it does not work well with mpm_worker, which is the (faster) threaded setup of apache (vs. the forking setup)...;)

I wouldn't run a web server in threaded mode... I'm not sure I trust the code enough (especially PHP or any other apache modules). If you compromise the server somehow and it's threaded, you get instant access to all the clients running on that process. At least with separate processes the damage will be confined to a single client.
11-24-2009
cyberfish

Quote:

Originally Posted by MK27

Which regular expressions are a lot more powerful (but slower) tool than SQL IMO (but I'm a lil' ignorant, too), and if you are going to code it in PHP, you might include that option.

Regex's are always fun :). I have tried to learn it a few times already. I didn't get stuck or anything... I just... forget about it after a while.

Quote:

Originally Posted by laserlight

It sounds like you want to re-invent dbm and its successors.

Indeed! Thanks for pointing that out.

So I guess now I just need to write a simple server wrapper thing... or maybe one exists already.
11-24-2009
MK27

Quote:

Originally Posted by brewbuck

I wouldn't run a web server in threaded mode... I'm not sure I trust the code enough (especially PHP or any other apache modules). If you compromise the server somehow and it's threaded, you get instant access to all the clients running on that process. At least with separate processes the damage will be confined to a single client.

Interesting.

I doubt that is a significant issue w/ 90% of web server activity tho. What are you going to do, read a bunch of identical requests for the same pages you have seen already? Spoof some material? Also, by default apache does not maintain persistent connections, so those threads are probably already dead and dying. Of course, you could "answer the phone" maybe :p

But that will be almost the same issue with the forking model -- each process handles bunches of clients.

In a setting where, eg, people are submitting their credit cards or something, I guess there are many things like this you may want to account for.

In fact I think the thread vs. fork performance issue is only significant 1) on large dedicated multi-core servers, 2) on small shared servers, where the memory footprint per client will be much smaller. I just like to diss PHP occasionally.
12-01-2009
rags_to_riches

How about CouchDB?
12-02-2009
CornedBee

There's also memcache, which is what you want, just memory-only. But if you write a server, you might want to be interface-compatible.
12-03-2009
zacs7

Quote:

Originally Posted by MK27

I kind of agree SQL style relational databases are over rated and over used. I might guess that the reason you haven't heard of anything like this is because it is so simple -- by the time most web programmers realize they don't have to use SQL, they probably also realize most (or a lot) of stuff can be done just as well only easier with a flat file.

Sure, and how do you deal with deadlocks, atomic operations, rollbacks and various other features most DBMSs provide? Plus not being relational, or having much structure at all puts a lot of onus on the application rather than the DBMS. Don't forget which group of programmers (read: web designers) use PHP.

Not to mention changing your file structure could spell the end of the world if you have hundreds of gigs of files.

Quote:

Originally Posted by MK27

I just like to diss PHP occasionally.

We ought to start a club for that :)