multithreading - Workload balancing between akka actors -

-

i have 2 akka actors used crawling links, i.e. find links in page x, find links in pages linked x, etc...

i want them progress more or less @ same pace, more not 1 of them becomes starved , 1 consumes resources.

i've tried following approaches (simplified). single page crawling done following actor:

class crawler extends actor {   def receive = {     case crawl(url, kind) =>       // download url       // extract links       sender ! parsed(url, links, kind)   } }

approach 1:

class coordinator extends actor {   val linksa = ...   val linksb = ...   def receive = {     case parsed(url, links, kind) =>       val store = if (kind == kinda) linksa else linksb       val newlinks = links -- store       store ++= links       newlinks.foreach { link =>         val crawler = context.actorof(props[crawler])         crawler ! crawl(link, kind)       }   } }

approach 2:

class coordinator extends actor {   val linksa = ...   val linksb = ...   val rrprops = props[crawler].withrouter(roundrobinrouter(nrofinstances = 10)   val crawlera = context.actorof(rrprops)   val crawlerb = context.actorof(rrprops)   def receive = {     case parsed(url, links, kind) =>       val store = if (kind == kinda) linksa else linksb       val newlinks = links -- store       store ++= links       newlinks.foreach { link =>         if (kind == kinda) crawlera ! crawl(link, kind)         else crawlerb ! crawl(link, kind)       }   } }

second approach made things better, didn't fix whole.

is there way make crawlers of both kinds progress more or less @ same pace? should send messages between them unblocking each other in turn?

i'm working on similar program workers have non-uniform resource cost (in case task performing database queries , dumping results in database, crawling different websites have different costs different queries have different costs). 2 ways of dealing i've employed:

  1. replace roundrobinrouter smallestmailboxrouter
  2. don't have coordinator send out of messages @ once - instead send them out in batches, in case have ten workers sending out forty messages should keep them busy initially. whenever worker completes task sends message coordinator, @ point coordinator sends out message go worker completed task. (you can in batches, i.e. after receiving n "task complete" messages coordinator sends out n messages, don't make n high or else workers extremely short tasks may idle.)

a third option cheat , share concurrentlinkedqueue between actors: after filling queue coordinator sends "start" message workers, , workers poll queue til it's empty.


Comments

Post a Comment

Popular posts from this blog

python - TypeError: start must be a integer -

-
i trying draw 5 turtles, getting typeerror on line 25. here code: import turtle wn = turtle.screen() redrose = turtle.turtle() color = input("what background color be?") fillcolor_f = input("what color of rose be?") redrose.hideturtle() redrose.speed(30) redrose.penup() redrose.left(180) redrose.forward(175) redrose.right(90) redrose.forward(30) redrose.right(90) redrose.pendown() def drawrose(red): redrose.color("pink") redrose.fillcolor(fillcolor_f) redrose.fill(true) in range(red): redrose.forward(i) redrose.right(49) in range(5): drawrose(redrose) redrose.penup() redrose.forward(350) redrose.right(144) redrose.pendown() redrose.fill(false) drawrose(50) wn.bgcolor(color) i trying draw 5 roses, produces errors. doing in interactivepython.org. you recursively calling drawrose wrong parameter. on line 23 ( for in range(red): ) expect red integer...
Read more

c# - DevExpress RepositoryItemComboBox BackColor property ignored -

-
i have following code almost works properly; private void cbstatus_drawitem(object sender, listboxdrawitemeventargs e) { string item = e.item string; if (item != null) { switch (item) { case "1": e.appearance.forecolor = color.green; e.appearance.backcolor = color.green; break; case "2": e.appearance.forecolor = color.orange; e.appearance.backcolor = color.orange; break; case "3": e.appearance.forecolor = color.red; e.appearance.backcolor = color.red; break; } } } when dropdown shown, items forecolor correct, backcolor remains whatever backcolor of theme is; i.e. if i've got set dark theme, backcolor dark, cells in gridview, rather being green/orange/red. i've tried setting e.appearance.options.usebackcolor trying set e....
Read more

django - Creating multiple model instances in DRF3 -

-
1) create multiple model instances 1 api call, asked here: how create multiple model instances django rest framework? tried solution named in link, no success. i trying upload multiple files in 1 api call. result: files uploaded (only once overwrote perform_create) 1 instance created (if send 2 files, latter created instance). my code: class fileuploadserializer(serializers.modelserializer): def __init__(self, *args, **kwargs): many = kwargs.pop('many', true) super(fileuploadserializer, self).__init__(many=many, *args, **kwargs) class meta: model = fileupload read_only_fields = ('created', 'datafile', 'owner') class fileuploadviewset(viewsets.modelviewset): queryset = fileupload.objects.all() serializer_class = fileuploadserializer parser_classes = (multipartparser, formparser, ) def perform_create(self, serializer): file_list = self.request.data.getlist('file') ...
Read more