Hacking Thought Blog

I have been wanting since the release of AppEngine...

2010-04-08T14:32:03.301-04:00

I have been wanting since the release of AppEngine to do an AppEngine IDE as an app running on, obviously, AppEngine. I've already run some tests and actually deploying from one app to another should work, even if the deployer needs modified a bit to run on AppEngine.

I think this notion is most powerfully combined with a strong web-based forking and merging. Have a bug in your favorite webapp? Fork it and fix it in a minute, without even leaving your browser.

I would gladly help out if I can. Not sure if I wi...

2010-01-05T19:16:37.326-05:00

I would gladly help out if I can. Not sure if I will be in a position to help but I will contact you shortly.

Regards,

Brume

@Brume The documents can have foreign keys to othe...

2010-01-04T20:35:48.790-05:00

@Brume
The documents can have foreign keys to other documents. So you don't have to have copies of the data. For example if I was to have a list of a list of friends then I would just have an array of the document ids that are the users friends. One of the reasons I like Riak so much is it support a concept of a link in a document of which you can traverse links which is really cool. In my applications that I wrote using CouchDB I just used a string that pointed to the document id of which I would use a lot in lists and dictionaries (associative arrays).

All I was trying to point out is the documents can have nested data in them which is very difficult or ugly to do in a relational database. In relational databases we end up creating lots of data that should just be nested and use queries to rebuild the nesting in our ORM. When you have a document and it has a small list of things like groups, friends, settings ect most of the time you don't need to create a document for each "thing".

First I have to port Magnum to OS X. Which I have ...

2010-01-04T20:25:30.758-05:00

First I have to port Magnum to OS X. Which I have started to write a generic nonblocking I/O system that uses kqueue, epoll or poll depending on what is supported (probably support libevent pretty easy to). I have the code stubbed out but I need to finish testing and implementing it. I have a little Android app I am writing this week, then a little web app and hopeful I will find a small consulting project and then I can go back and finish the implementation. Once I get the port it shouldn't take more than 2 days to port all the features Frisky has into Magnum. Would love some help on the implementation if anyone can?

What frisky features are you are looking to add to...

2010-01-04T18:04:59.281-05:00

What frisky features are you are looking to add to magnum?.

Have you had a chance to take magnum for a spin?

I am curios to see how well magnum performs.

Brume

Ah!. So each document will contain all data it n...

2010-01-04T17:58:10.840-05:00

Ah!.

So each document will contain all data it needs with fewer queries required to pull data of interest.

One obvious trade off being increased disk space requirements seeing as data is duplicated accross documents in order to obviate joins. As well updates of duplicated data will require more writes to different documents to accomplish.

Interesting.....

Now I see the forest for the trees :)

Duh!. It is hard to think with a non-relational hat on when all the apps you have written are for relational data models.

Brume

@slow disk Compared to CPU speed individual disk s...

2010-01-03T22:28:24.540-05:00

@slow disk
Compared to CPU speed individual disk speeds have stood still for the last 20 years. If the performance had increased a lot of the RDBMS performance issues would not exist.

@Brume
As far as querying data model in a document based NoSQL in the applications that I have been working on the data tends to be less partitioned into documents. For example if you had a system you would have one document to store user information (name, password, authentication type), settings (colors, background, links), contact information (phone numbers, emails, address) and friends (list of other users). Well in a relational model that was normalized instead of having one table it would have maybe 7 (login, setting, phone, email, address, friend) or more potentially. Since all this is stored in a single document then there is no need to traverse other documents. So in this case we have a single object that would represent the document and instead of having 7 ORM mapped object to 7 tables we would just have one. If the use case is to find all the users who have setting X and email them for the next release we can just filter the documents by setting and we are done. In an ORM we can filter by setting then look up the email address, or we could build a join so that we don't have two round trips to the database... But that goes back to my point about performance.

When I was speaking of the map/reduce time savings I was talking development time savings. Specifically I found it much faster to write Javascript conditional to filter documents in CouchDB than write SQL.

As far as getting your hands dirty the issue I had was it took a couple little applications before I got my brain thinking document instead of relation. The first couple scripts I wrote using CouchDB looked like relational data model. I hate to say it but it is hard to "think outside the box".

Hey Lateef, I am trying to follow your train of t...

2009-12-31T13:17:08.508-05:00

Hey Lateef,

I am trying to follow your train of thought....but not yet familiar enough with how document databases work to fully grasp what you mean....

"I have found that the document model allows the data to be organized so that there is a lot less quiring. I also find writing map/reduce to be faster and easier than SQL once you get use to it with some exceptions where it is more difficult but this might just be getting use to thinking in a document model. In general I find that either I didn't have to write the query or the savings in time that map/reduce creates is worth the extra effort on the queries that are more difficult.
"

Could you please elaborate further on this?. If the data you require is spread over so many documents in a document store wouldnt you have to manually get and filter each document in some sort of custom fashion (for each query) - in a sense building queries on the fly using primitive predicates provided by raw javascript code (for example). Or am I missing something fundamental here?.

When you say "the savings in time map/reduce creates" are you referring to time saved in returning the results from a map/reduce call vs the time it takes for the RDBMS equivalent for a complex (non-trivial) query?.

The last poster is also re-echoing your sentiments about higher programmer productivity.

Perhaps I need to get my hands dirty and/or do some actual coding with a document database in order to fully appreciate all this.

Brume

i don't think the trend has anything to do with di...

2009-12-31T11:27:31.867-05:00

i don't think the trend has anything to do with disk drives. Rather, changes in CPUs drive the trend: in the future we can have many slow cores on many machines cheaply, but one fast core becomes impossible.

horizontal scaling got the ball rolling on the NoSQL trend, but developers are now seeing too that programmer productivity is higher and administration lower, in particular with the document oriented databases

The interesting thing with a lot of the consistenc...

2009-12-31T11:24:32.908-05:00

The interesting thing with a lot of the consistency discussions is that we are often comparing a single server RDBMS to a multi server NoSQL cluster. If we were to use NoSQL solutions on a single server too, we would then get instant consistency with many products just like with a SQL database.

Also product by product it varies - HBase IIRC favors consistency over availability.

Often with RDBMS, when one goes multiserver, one uses asynchronous replication which is eventually consistent.

dwight/MongoDB

SQLAlchemy really does take a lot of the pain out ...

2009-12-31T10:26:05.430-05:00

SQLAlchemy really does take a lot of the pain out of SQL. If you have to get down and dirty with stored procedures or what not you still at the bottom of a cliff to learn and debug the oddities of working in the database environment.

"However dont NoSQL databases leave the developer to do most of the heavy lifting required to query the data in intelligent fashion?"
This is a really good question. The first major reason I believe NoSQL increases productivity is data model design. I have found that the document model allows the data to be organized so that there is a lot less quiring. I also find writing map/reduce to be faster and easier than SQL once you get use to it with some exceptions where it is more difficult but this might just be getting use to thinking in a document model. In general I find that either I didn't have to write the query or the savings in time that map/reduce creates is worth the extra effort on the queries that are more difficult.

"riak only allows writing the data management map/reduce logic in erlang - which same requires a steep learning curve in order to become proficient"
Yes this is true. It is however open source and if I knew enough erlang I would try to see if I could hook Python in to do the map/reduce. I don't think this would be to difficult and I expect if Riak becomes popular it will quickly get map/reduce support in javascript and php also. The SQL learning curve at the beginning might seem easy but it gets steep quickly as the data gets normalized and query are non trivial.

A Python map/reduce that could be used from python console would be an awesome tool for Riak and I think go a long way in accelerating the adoption of Riak.

Interesting..... I have recently been researching ...

2009-12-30T22:16:08.987-05:00

Interesting..... I have recently been researching the pros and cons of NoSQL databases and comparing same with the traditional RDBMS which currently hold sway.

As you already mentioned, there are other language bindings for writing Postgresql stored procedures. Unless there are performance issues with writing a stored procedures in a higher level language like say python I would certainly prefer python stored procedures over the dated terse syntax created ages ago.

My experience has been that ORMS like SQLAlchemy help with the back and forth mismatch between the application objects and the data tables as the data model of neccessity evolves while the application likewise is evolving. So it hasnt been that bad (for me at least).

So much has been said about how NoSql databases are more appropriate than the RDBMS for web applications (that need to scale at will in geometric fashion) where a simple key-value data store will suffice, etc. Indeed, NoSQL databases in some quarters are deemed to hold a lot of promise for the future of reliable, scalable distributed/concurrent database backends with more predictable performance characteristics than the RDBMS.

This is all fine and dandy. However dont NoSQL databases leave the developer to do most of the heavy lifting required to query the data in intelligent fashion? i.e Do you expect to see an immediate increase in your productivity w.r.t development related to data management of NOSQL based DBMS's.

I think I saw you mentioned somewhere that riak only allows writing the data management map/reduce logic in erlang - which same requires a steep learning curve in order to become proficient. Writing map/reduce logic using a functional high level language like javascript on the other hand perhaps would be a lot more fun/productive.

I would like to see general intuitive tools/libraries/apis developed to ease the development of application specific data access logic (regardless of the degree of their complexity) to ease the transition from using a traditional RDBMS to using NoSQL databases in cases where it is more appropriate to do so.

Brume

Thanks Brume, I feel like there is this transitio...

2009-12-30T17:16:13.963-05:00

Thanks Brume,

I feel like there is this transition to a nonblocking IO web backend combined with a NoSQL document store (including mobile clients like Android). However there is a lot of dust to settle. I am not an ubercoder so I have to actually write lots of code to evaluate these new technology. It is hard to find the time especially if I need to write something like frisky to experiment with a new technique. I just hope I can share my learning by writing it down.

Thanks for keeping me thinking!
Lateef

Hey Lateef, Thanks for taking the time to clarify...

2009-12-30T16:58:47.437-05:00

Hey Lateef,

Thanks for taking the time to clarify all that. I always find your posts quite informative and I am curios to see how your choice of technology stack evolves - i.e frisky/magnum, riak, etc.

Thanks,

Brume

Brume, Sorry for not being clear I, was planning o...

2009-12-30T16:43:44.079-05:00

Brume,
Sorry for not being clear I, was planning on writing a follow up post however have not had time yet.
1. I was referring to map/reduce. There is a common complaint about map/reduce (not sure if this is just RDBMS fanboy) and I wanted to point out that I really like map/reduce. I have spent a good bit of time writing map/reduce in couchDB so I could speak from that experience. I am less excited with Riak erlang map/reduce but that is just because I have to learn erlang. CouchDB javascript map/reduce I thought was wonderful compared to writing the same queries in SQL.

2. Yes I was referring to CouchDB. I plan to write a post about this once I get enough application development experience with Riak that compares the data stores that I have extensive experience writing web applications with (Postgresql, MySQL, CouchDB and Riak). Off the top of my head the issues are:
* CouchDB size grew much faster than I expected during my testing a medium size web application with it. "Compacting" I believe is what it is called reduce this however I found this to leave a nasty taste in my mouth. Kinda like PostgreSQL before autovacuum was implemented. Sure it is great if you need it but autovacuum would have worked fine for all the web apps I have built using PostgreSQL in the last 9 years or so.
* Replication to me is disappointing. CouchDB replication forces you to do a lot of the thinking up front (some of which should be done any way) and it seemed like I would have to write a lot of replication code in my application . Compared to RDBMS systems it seem better but I was hoping for more.

Mainly I am impressed with the _design_ of Riak. The feature set like: link, scaling up and down, map/reduce and data constraints fits the niche of web applications that I am writing. From my perspective I develop on my laptop so I need the datastore to be usable and testable on a single node but also deploy to many nodes. I also need flexibility on how the data is replicated for performance and high-availability depends on the subset of application data.

I hope this helps.
Lateef.

Maybe its just me but the subject of a couple of t...

2009-12-30T13:22:19.027-05:00

Maybe its just me but the subject of a couple of the statements in this post is not clear to me. So I filled in the blanks based on what (in my opinion) appears to be mostly in support of the title.

1. "After I spent some time writing map/reduce javascript functions with CouchDB I find myself in love with it." (In love with Riak?)

2. "As much as it is a great database for doing specific types of document storage systems it had some side effects that I really didn't like." (Now you are referring to couchdb again - right i.e. couchdb had some side effects).

What by the way are those side effects you encountered?

Again it may just be me, but I thought it would help if you clarified those statements accordingly.

Thanks,

Brume.

More numbers can be a usability issue. I haven't u...

2008-12-30T00:50:00.000-05:00

More numbers can be a usability issue. I haven't used in production, but there is an interesting recipe over at activestate that can generate random IDs that are pronounceable: http://code.activestate.com/recipes/526619/

That was interesting project as well. C library ma...

2008-12-23T16:40:00.000-05:00

That was interesting project as well. C library makes extra hassle, so using frisky could straighten some ways. I saw POST and GET methods handled there, but in which part of the server / web application file uploads (coming from forms) are normally handled?

Last year i made one project with Pylons, but recently used web2py and found it more practical for my projects. As a hobby, train and curiosity im thinking, if i could do little hacks with backend servers.

I found you using FAPWS2 with pylons and it seemed rather simple to implement, and that made me to ask opinions from you, if that simplicity would apply to web2py too.

Thank you for you comments.

Pretty easy I would guess. The main thing is you p...

2008-12-23T16:02:00.000-05:00

Pretty easy I would guess. The main thing is you probably don't want to write a C module. The performance is probably not so much worth the time and effort. You could use pyevent (http://code.google.com/p/pyevent/) which is python library that uses libevent. If you just want to integrate web2py with FAPWS then that would be pretty easy.

There is another option :) I wanted to create a WSGI server that could use asyncore or other (libevent libev, ect) system. So I create this: http://www.bitbucket.org/lateefj/frisky/. It has some basic pylons support but if you proded me I would probably clean it up and add support for web2py. I was getting like 3K req/sec (raw) so it is very fast and does not require 3rd party C code. Even though it runs half the speed FAPWS2 it is still mind blowingly faster than most webservers.

What do you think, how much it would take to use l...

2008-12-23T15:45:00.000-05:00

What do you think, how much it would take to use libevent with web2py framework? I'd like to experiment with it, but don't know where to start. At the moment web2py uses cherrypy wsgiserver: http://code.google.com/p/web2py/source/browse/trunk/gluon/wsgiserver.py

hi, Sorry for disturbance. you should have your an...

2008-11-03T09:18:00.000-05:00

hi,

Sorry for disturbance.
you should have your answers here: http://william-os4y.livejournal.com/5726.html

If you drop me an email I can send you the pylons ...

2008-10-24T09:19:00.000-04:00

If you drop me an email I can send you the pylons project however let me give you as much as I can here.

First I added a run.py in the same directory as the config.ini. This code is below.

import os
from paste.deploy import loadapp
config_path = os.path.abspath(os.path.dirname(__file__))
path = '%s/%s' % (config_path, 'development.ini')
application = loadapp('config:%s' % path)

import _evhttp as evhttp
from fapws2 import base
import sys
sys.setcheckinterval=100000 # since we don't use threads, internal checks are no more required

def start():
evhttp.start("0.0.0.0", 8080)

evhttp.set_base_module(base)

evhttp.gen_http_cb(application)

evhttp.event_dispatch()

if __name__=="__main__":
start()

I have been having issues with it loading index.html. But I have been able to load other static files like images and stuff so not sure what that is all about. I was able to add things to routes and that seems to work. Again if you sling me an email I can email hello world pylons I did.
I can no longer get ahold of William the maintainer of FAWPS his email doesn't work for me anymore. So I don't know how much I can help my code checkout of FAPWS2 is a couple months old. However I did contrib a bit of code so I know the project code pretty well and could probably do some amount of support.

Whao, this is the most mind-blowingly fast reply I...

2008-10-23T22:21:00.000-04:00

Whao, this is the most mind-blowingly fast reply I've ever had! :)

Jerry,This was back in the days of FAWPS1 days. Ho...

2008-10-23T21:50:00.000-04:00

Jerry,
This was back in the days of FAWPS1 days. However I will find the config see if I can get it working with FAWPS2. Give me till end of day on Oct 24.

Hi Jackson,I've been pulling my hair to run my Pyl...

2008-10-23T21:43:00.000-04:00

Hi Jackson,

I've been pulling my hair to run my Pylons application under FAWPS2, could you please share your FAWPS(2) configuration for Pylons?

Many thanks!

Jerry