Sunday, October 9. 2011Rebooting the baconbird projectTrackbacks
Trackback specific URI for this entry
No Trackbacks
Comments
Display comments as
(Linear | Threaded)
Perl threads can be used - for ex. Padre uses them. Also there are several modules on CPAN for parallel execution. Or just simple fork().
Every Perl guru in my proximity advised me not to use threads.
Which (specific) modules are you talking about? Do they use any other technique than forking? (which, btw, is not an option here)
For ex. Coro, it foes not use fork. Mojo::UserAgent. All other modules that I see, are event-based, but in paragraph "The overall structure is simple" you describe event model .
P.S. Why forking is not an option?
Sorry, but I won't bother to discuss with you if you don't understand the specific problems I'm having. And no, the paragraph re: the overall structure does not describe an event-based model.
Did you publish the baconbird code anywhere? I'd love to have a poke around in it.
That'd definitely be convertible to non-blocking communication, but since you don't care about event-based code you'd likely not be very interested. Props on the clean code though.
A question: In gockel you have three goroutines, one of which handles the twitter communication. When a certain twitter request takes a long time to complete, will all other requests have to wait until that one is done, before they get sent?
"A question: In gockel you have three goroutines, one of which handles the twitter communication. When a certain twitter request takes a long time to complete, will all other requests have to wait until that one is done, before they get sent?"
No, new tweets are only sent to the "model" goroutine when they are actually received, while the "model" goroutine does a select on that channel among others. Bad performance, long waits or even interrupted connections or bad requests have no influence on the other goroutines.
I wasn't thinking about interactions between goroutines, just what happens within that one goroutine.
Example: I trigger the loading of the stream of a user and for whatever reason that takes a long time. While that's busy i decide to go and write a tweet complaining about it. Will the complaint tweet be sent immediately and alongside the previous timeline load, or will it only be sent after the timeline is loaded? In other words: Do you have any concurrency within the model goroutine?
The continuous loading from the userstream API runs in a different goroutine than the other API calls (like posting a tweet, retweeting, favoriting a tweet, ...). These other API call can run concurrently to the userstream API, but are currently queued in a buffered channel. I could fire off those as separate goroutines, though, since each API calls is an independent HTTP request.
Alright, so you're opting for "good enough" in the implementation, which makes sense given what it is.
At its core, the problem you're trying to solve is IO-bound, which means an event-based system the correct solution. (Threads are a tool to be used for CPU-bound problems.)
Right now you still have a few edge cases, like the "request getting stuck" thing i mentioned that might become issues. Additional goroutines might be a bandaid for this, but the docs seem to imply that they'd create more threads, which means you run the risk of exhausting system resources. The only perfect solution for this would be an event system, but i have to admit, the likelihood of these things occurring is small enough to not bother them. Which means the simple three-threaded solution you have right now is "good enough" to not have to worry about any issues in real world application.
Err, the go runtime is the event-based system, it abstracts away all the details (see e.g. http://swtch.com/libtask/ for preceding work by one of the Go developers). goroutines are not threads. The number of OS-level processes that is created is always limited by GOMAXPROCS.
"the go runtime is the event-based system, it abstracts away all the details"
That is a perspective on the matter i had not considered before. You might be right on that, especially since the task of writing event-based code in itself is like writing parts of an OS yourself.
Noob can't do multithreading in perl in his noob project, cries loud. More at 11.
In all seriousness: If you're suggesting to use threading in Perl then that says more about your level of expertise than anything else.
I've used Perl ithreads in anger successfully. The result is the first generation of Padre's Task API for background tasks. That being said, ithreads are more like a portable variant of fork'ed processes (with additional headaches for XS modules) than what most people expect when you use the word "thread".
I totally admit it, I'm dumb, and so I enjoy using programming languages that make things reasonably easy for me.
BTW: I never cried out loud, I merely mentioned it (actually I posted this blog entry to point at my shiny new project, not the problems I had with Perl), others put the focus on my Perl criticism.
yeah, that was me at http://blogs.perl.org/users/philip_durbin/2011/10/perl-dropped-and-go-adopted-due-to-concurrency-issues-in-baconbird.html . i hope you don't mind! i've learned a lot from this discussion! and i do plan to give gockel a try, maybe even package it up!
It may not be CPAN, but there is the Go Dashboard.
http://godashboard.appspot.com/package |
Calendar
QuicksearchShow tagged entries22c3 23c3 amsterdam announcement apache argentina army austria beer berlin book borland bsd c c++ camera censorship christianity cms complaint concert cooking egypt electronic music email fail feedreader fefe food fun gas mask gcc german germany git gnu golang google google earth hacking history html http i18n imap internet israel job kaminer lecture linux linz mobile movie music network newsbeuter noos panorama pearl jam performance photo photography photos pictures polaroid police politics problem programming python quiz rant recommendation release rss ruby screencast seagull security series server ska skabucks stfl terrorism travelling tv unix usa video videos vienna war weird wikipedia windows work wplotd youtube
Blog AdministrationLinksBlogroll• xkcd.com
• Planet Debian • MY POV ([expect the unexpected]) • C skills • Planet Erlang / Published News • armstrong on software • Photos from akrennmair • Das Metalab informiert • dive into mark • /usr/local/bin • F!XMBR • heise online News (full feed) • JLog • SecuriTeam Blogs • .:Computer Defense:. • Riot Porn • Chaosradio • Radiomultikulti vom RBB: Russendisko unplugged • AK's weblog • The Recurity Lablog • milw0rm.com • seclog.de • ilja's blag • udo.kernecker.at - mein leben als prinzregent... ;-) • grabnerandi.at diary feed • Hilli's WebLog • accidents waiting to happen • Venzi's Weblog • TaoSecurity • Irrlicht3d.org • murphee's Rant • waiterrant.net • grml development blog • mutt Changelog • nion's blog • Wannabe Everything • blog@bytelabs • Knowledge Brings Fear • Die wunderbare Welt von Isotopp • Fefes Blog • law blog • mikas blog • BILDblog • GoogleWatchBlog • Krone - Blog • The Lunatic Fringe • mp's blog • Su-Shee 2.0 • Sex, Drugs & Compiler Construction • Qbi's Weblog • gedankensplitter • Ohns Gehirnschleimschmiede • fh • Clifford Wolf's Blog • AK's moblog • Telepolis News • Slashdot • Newssystem von bundesheer.at • Riding Rails - home • Serendipity • O'Reilly Ruby • CCC Events Weblog • del.icio.us/dubrider • del.icio.us/timpritlove • del.icio.us/ak • del.icio.us/mika • AK's Soup • Friends of ak • Astronomy Picture of the Day • german-bash.org - Die neuesten Zitate • QDB • WeirdWeirdWorld Latest Feed • I CAN HAS CHEEZBURGER? • The Trailer Mash • Cruel.Com • fun.drno.de • Peter Pilz grüner Sicherheitssprecher Österreich Wien • NPD-BLOG.INFO • INSM Watchblog • Hitler-Blog • Everybody loves Eric Raymond • Dilbert |