Hacking Thought Blog

I use Rackspace for this because they have the bes...

2011-09-23T09:53:15.575-04:00

I use Rackspace for this because they have the best pricing for the minimum setup.
Yes I have mongo on the same stack but it would be trivial to push it on its own servers.

Great post! Couple of questions: 1. Which cloud p...

2011-09-23T02:33:22.652-04:00

Great post! Couple of questions:

1. Which cloud provider did you end up using for this? Rackspace?

2. Have you looked at Mongo with the same stack?

nice....!

2011-08-12T02:30:01.908-04:00

nice....!

Udhyakumar take a look at cell tables they are muc...

2011-07-16T12:06:07.075-04:00

Udhyakumar take a look at cell tables they are much better! http://code.google.com/webtoolkit/doc/latest/DevGuideUiCellWidgets.html they are much easier to use and work with mobile devices.

hi friend, i need a help how do i change a cell co...

2011-07-16T01:33:56.484-04:00

hi friend,
i need a help
how do i change a cell color while i clicking on a perticular cell
and also for listbox give me code sample too please
thanks alot

very nice blog

2011-06-20T03:56:57.178-04:00

very nice blog

Today reddit had an entry to: http://www.gnu.org/s...

2010-11-04T15:47:22.707-04:00

Today reddit had an entry to: http://www.gnu.org/software/pythonwebkit/ which seems to suggest we may have an good alternative to Java in GWT soon. This would be very welcome. Last night I played with it a bit and I will need to it hooked into a WSGI app so I can just refresh a page and it will render the output via the WSGI servers instead of writing it to disk. Then I can start using it as an alternative.

@chrismahan I think your right if the server is fa...

2010-11-04T15:41:51.029-04:00

@chrismahan I think your right if the server is faster than your desktop and bandwidth is massive you are all set. Since Ubuntu uses debian package manager I think they are probably similar. My local stack is faster than the remote one I am running in general so I will probably keep running it locally and thus want to run linux on my laptop/desktop.

I went debian stable on vps and putty/ssh into the...

2010-11-03T01:50:15.660-04:00

I went debian stable on vps and putty/ssh into the dev machine. I use whatever on the client, since it doesn't matter. I output html, and do all the heavy lifting on the server. yes, more money monthly on the server, but cheaper machines to buy every 3 years. Also, on debian, you can run a bunch or tools together in different languages...

I just realized that one of the libraries you ment...

2010-09-01T13:05:31.444-04:00

I just realized that one of the libraries you mentioned, Eventlet does use coroutines.

I am a fan of coroutines I just haven't had mu...

2010-09-01T12:32:54.691-04:00

I am a fan of coroutines I just haven't had much time to experiment with them in Python. PEP 342 would be very useful and would be my prefered method for concurrency. The little code I have written in go (golang) I found coroutines very user friendly.

Thanks for the great presentation. Tornado does lo...

2010-09-01T11:52:46.872-04:00

Thanks for the great presentation. Tornado does look like a very nice, easy to use library for asynchronous communication. I'll be comparing it with Twisted soon as I need to learn Twisted.

BTW, have you looked into using coroutines for asynchronous communication as an alternative or augmentation to an event-based approach? I haven't really tried it myself, but I have read about server frameworks in at least Smalltalk and Lisp that use coroutines or the lower-level building blocks continuations. It seems that PEP 342 might be useful for asynchronous communication, but I don't how much people are doing with it yet.

I have been wanting since the release of AppEngine...

2010-04-08T14:32:03.301-04:00

I have been wanting since the release of AppEngine to do an AppEngine IDE as an app running on, obviously, AppEngine. I've already run some tests and actually deploying from one app to another should work, even if the deployer needs modified a bit to run on AppEngine.

I think this notion is most powerfully combined with a strong web-based forking and merging. Have a bug in your favorite webapp? Fork it and fix it in a minute, without even leaving your browser.

I would gladly help out if I can. Not sure if I wi...

2010-01-05T19:16:37.326-05:00

I would gladly help out if I can. Not sure if I will be in a position to help but I will contact you shortly.

Regards,

Brume

@Brume The documents can have foreign keys to othe...

2010-01-04T20:35:48.790-05:00

@Brume
The documents can have foreign keys to other documents. So you don't have to have copies of the data. For example if I was to have a list of a list of friends then I would just have an array of the document ids that are the users friends. One of the reasons I like Riak so much is it support a concept of a link in a document of which you can traverse links which is really cool. In my applications that I wrote using CouchDB I just used a string that pointed to the document id of which I would use a lot in lists and dictionaries (associative arrays).

All I was trying to point out is the documents can have nested data in them which is very difficult or ugly to do in a relational database. In relational databases we end up creating lots of data that should just be nested and use queries to rebuild the nesting in our ORM. When you have a document and it has a small list of things like groups, friends, settings ect most of the time you don't need to create a document for each "thing".

First I have to port Magnum to OS X. Which I have ...

2010-01-04T20:25:30.758-05:00

First I have to port Magnum to OS X. Which I have started to write a generic nonblocking I/O system that uses kqueue, epoll or poll depending on what is supported (probably support libevent pretty easy to). I have the code stubbed out but I need to finish testing and implementing it. I have a little Android app I am writing this week, then a little web app and hopeful I will find a small consulting project and then I can go back and finish the implementation. Once I get the port it shouldn't take more than 2 days to port all the features Frisky has into Magnum. Would love some help on the implementation if anyone can?

What frisky features are you are looking to add to...

2010-01-04T18:04:59.281-05:00

What frisky features are you are looking to add to magnum?.

Have you had a chance to take magnum for a spin?

I am curios to see how well magnum performs.

Brume

Ah!. So each document will contain all data it n...

2010-01-04T17:58:10.840-05:00

Ah!.

So each document will contain all data it needs with fewer queries required to pull data of interest.

One obvious trade off being increased disk space requirements seeing as data is duplicated accross documents in order to obviate joins. As well updates of duplicated data will require more writes to different documents to accomplish.

Interesting.....

Now I see the forest for the trees :)

Duh!. It is hard to think with a non-relational hat on when all the apps you have written are for relational data models.

Brume

@slow disk Compared to CPU speed individual disk s...

2010-01-03T22:28:24.540-05:00

@slow disk
Compared to CPU speed individual disk speeds have stood still for the last 20 years. If the performance had increased a lot of the RDBMS performance issues would not exist.

@Brume
As far as querying data model in a document based NoSQL in the applications that I have been working on the data tends to be less partitioned into documents. For example if you had a system you would have one document to store user information (name, password, authentication type), settings (colors, background, links), contact information (phone numbers, emails, address) and friends (list of other users). Well in a relational model that was normalized instead of having one table it would have maybe 7 (login, setting, phone, email, address, friend) or more potentially. Since all this is stored in a single document then there is no need to traverse other documents. So in this case we have a single object that would represent the document and instead of having 7 ORM mapped object to 7 tables we would just have one. If the use case is to find all the users who have setting X and email them for the next release we can just filter the documents by setting and we are done. In an ORM we can filter by setting then look up the email address, or we could build a join so that we don't have two round trips to the database... But that goes back to my point about performance.

When I was speaking of the map/reduce time savings I was talking development time savings. Specifically I found it much faster to write Javascript conditional to filter documents in CouchDB than write SQL.

As far as getting your hands dirty the issue I had was it took a couple little applications before I got my brain thinking document instead of relation. The first couple scripts I wrote using CouchDB looked like relational data model. I hate to say it but it is hard to "think outside the box".

Hey Lateef, I am trying to follow your train of t...

2009-12-31T13:17:08.508-05:00

Hey Lateef,

I am trying to follow your train of thought....but not yet familiar enough with how document databases work to fully grasp what you mean....

"I have found that the document model allows the data to be organized so that there is a lot less quiring. I also find writing map/reduce to be faster and easier than SQL once you get use to it with some exceptions where it is more difficult but this might just be getting use to thinking in a document model. In general I find that either I didn't have to write the query or the savings in time that map/reduce creates is worth the extra effort on the queries that are more difficult.
"

Could you please elaborate further on this?. If the data you require is spread over so many documents in a document store wouldnt you have to manually get and filter each document in some sort of custom fashion (for each query) - in a sense building queries on the fly using primitive predicates provided by raw javascript code (for example). Or am I missing something fundamental here?.

When you say "the savings in time map/reduce creates" are you referring to time saved in returning the results from a map/reduce call vs the time it takes for the RDBMS equivalent for a complex (non-trivial) query?.

The last poster is also re-echoing your sentiments about higher programmer productivity.

Perhaps I need to get my hands dirty and/or do some actual coding with a document database in order to fully appreciate all this.

Brume

i don't think the trend has anything to do wit...

2009-12-31T11:27:31.867-05:00

i don't think the trend has anything to do with disk drives. Rather, changes in CPUs drive the trend: in the future we can have many slow cores on many machines cheaply, but one fast core becomes impossible.

horizontal scaling got the ball rolling on the NoSQL trend, but developers are now seeing too that programmer productivity is higher and administration lower, in particular with the document oriented databases

The interesting thing with a lot of the consistenc...

2009-12-31T11:24:32.908-05:00

The interesting thing with a lot of the consistency discussions is that we are often comparing a single server RDBMS to a multi server NoSQL cluster. If we were to use NoSQL solutions on a single server too, we would then get instant consistency with many products just like with a SQL database.

Also product by product it varies - HBase IIRC favors consistency over availability.

Often with RDBMS, when one goes multiserver, one uses asynchronous replication which is eventually consistent.

dwight/MongoDB

SQLAlchemy really does take a lot of the pain out ...

2009-12-31T10:26:05.430-05:00

SQLAlchemy really does take a lot of the pain out of SQL. If you have to get down and dirty with stored procedures or what not you still at the bottom of a cliff to learn and debug the oddities of working in the database environment.

"However dont NoSQL databases leave the developer to do most of the heavy lifting required to query the data in intelligent fashion?"
This is a really good question. The first major reason I believe NoSQL increases productivity is data model design. I have found that the document model allows the data to be organized so that there is a lot less quiring. I also find writing map/reduce to be faster and easier than SQL once you get use to it with some exceptions where it is more difficult but this might just be getting use to thinking in a document model. In general I find that either I didn't have to write the query or the savings in time that map/reduce creates is worth the extra effort on the queries that are more difficult.

"riak only allows writing the data management map/reduce logic in erlang - which same requires a steep learning curve in order to become proficient"
Yes this is true. It is however open source and if I knew enough erlang I would try to see if I could hook Python in to do the map/reduce. I don't think this would be to difficult and I expect if Riak becomes popular it will quickly get map/reduce support in javascript and php also. The SQL learning curve at the beginning might seem easy but it gets steep quickly as the data gets normalized and query are non trivial.

A Python map/reduce that could be used from python console would be an awesome tool for Riak and I think go a long way in accelerating the adoption of Riak.

Interesting..... I have recently been researching ...

2009-12-30T22:16:08.987-05:00

Interesting..... I have recently been researching the pros and cons of NoSQL databases and comparing same with the traditional RDBMS which currently hold sway.

As you already mentioned, there are other language bindings for writing Postgresql stored procedures. Unless there are performance issues with writing a stored procedures in a higher level language like say python I would certainly prefer python stored procedures over the dated terse syntax created ages ago.

My experience has been that ORMS like SQLAlchemy help with the back and forth mismatch between the application objects and the data tables as the data model of neccessity evolves while the application likewise is evolving. So it hasnt been that bad (for me at least).

So much has been said about how NoSql databases are more appropriate than the RDBMS for web applications (that need to scale at will in geometric fashion) where a simple key-value data store will suffice, etc. Indeed, NoSQL databases in some quarters are deemed to hold a lot of promise for the future of reliable, scalable distributed/concurrent database backends with more predictable performance characteristics than the RDBMS.

This is all fine and dandy. However dont NoSQL databases leave the developer to do most of the heavy lifting required to query the data in intelligent fashion? i.e Do you expect to see an immediate increase in your productivity w.r.t development related to data management of NOSQL based DBMS's.

I think I saw you mentioned somewhere that riak only allows writing the data management map/reduce logic in erlang - which same requires a steep learning curve in order to become proficient. Writing map/reduce logic using a functional high level language like javascript on the other hand perhaps would be a lot more fun/productive.

I would like to see general intuitive tools/libraries/apis developed to ease the development of application specific data access logic (regardless of the degree of their complexity) to ease the transition from using a traditional RDBMS to using NoSQL databases in cases where it is more appropriate to do so.

Brume

Thanks Brume, I feel like there is this transitio...

2009-12-30T17:16:13.963-05:00

Thanks Brume,

I feel like there is this transition to a nonblocking IO web backend combined with a NoSQL document store (including mobile clients like Android). However there is a lot of dust to settle. I am not an ubercoder so I have to actually write lots of code to evaluate these new technology. It is hard to find the time especially if I need to write something like frisky to experiment with a new technique. I just hope I can share my learning by writing it down.

Thanks for keeping me thinking!
Lateef