tag:blogger.com,1999:blog-19725519.post3004683555275506856..comments2024-02-01T05:22:59.677-05:00Comments on Erik Engbrecht's Blog: Adventures in Widefinding: ComplexityErik Engbrechthttp://www.blogger.com/profile/11174963559600768092noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-19725519.post-26232709261265377862007-12-05T22:58:00.000-05:002007-12-05T22:58:00.000-05:00Actually the only serial part is the reading data ...Actually the only serial part is the reading data from disk into a direct buffer. The rest is all parallel. The basic Scala for loop is just syntactic sugar for a call to foreach.<BR/><BR/>Here's the main behind-the-scenes code:<BR/>http://mysite.verizon.net/erik.engbrecht/pio.scala.html<BR/><BR/>Go to the foreach method in ParallelLineReader. It spawns off a coordinator actor, which then spawns 1 more workers than you have processors. Each worker reads a chunk of the file into a direct buffer and the passes the file channel on to the next worker. The workers form a circular list, so I/O is always done sequentially. Also, memory consumption is limited to the number of workers * the size of the buffers. So if the processing runs behind the input, then it will stop reading data in until it catches up rather than eating up all your memory.<BR/><BR/>The hard part is the line boundaries, because the beginning and the end of any given read most likely is the middle of a line, not an end.Erik Engbrechthttps://www.blogger.com/profile/11174963559600768092noreply@blogger.comtag:blogger.com,1999:blog-19725519.post-12245296982149784652007-12-05T15:36:00.000-05:002007-12-05T15:36:00.000-05:00That's interesting, apparently you have paralleliz...That's interesting, apparently you have parallelized the line reading and splitting, but the counting itself is linear, isn't it?<BR/><BR/>So this is the opposite of my attempt, where splitting is sequential and counting (+ matching) is done in parallel. We should try to combine this :-)Anonymousnoreply@blogger.com