Subscribe: perl.ipc.dirqueue
http://www.nntp.perl.org/rss/perl.ipc.dirqueue.rdf
Added By: Feedage Forager Feedage Grade B rated
Language: English
Tags:
dirqueue  file mode}  file  hash  ipc dirqueue  ipc  job  jobs  patch  queue  queued job  queued  user sys  visit jobs 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: perl.ipc.dirqueue

perl.ipc.dirqueue



...



Published: Tue, 12 Dec 2017 08:45:30 +0000

Copyright: Copyright 1998-2017 perl.org
 



Serial processing by rk mag

Sun, 03 Mar 2013 07:15:43 +0000

hi guys, Good day..

i want to ask if IPC dir queue can be set to run the same queue list
serially?? just like a playing a song from a playlist... I tried it but it
did not wait for the preceding job to finish and immediately went to play
the next song..in short I want it to play one song at a time..

regards,
Richmon



released version 1.0 of IPC::DirQueue by jm

Fri, 18 Apr 2008 08:45:05 +0000

Hi --

Version 1.0 of IPC::DirQueue is now released, and should be showing
up on CPAN rsn. Changes since 0.09:

- worker_still_working() is broken; it always claimed that the worker was
no longer running. Fix thanks to Brian Duggan org>

- use buffered print instead of unbuffered syswrite() when creating data
files using enqueue_sub()

- fix code to match POD documentation regarding the range of chars
permitted in metadata during enqueueing, thanks to Brian Grossman for
report

- update POD docs: visit_all_jobs() does not return a value. thanks to
Simon on AnnoCPAN:
http://annocpan.org/%7eJMASON/IPC-DirQueue-0.09/lib/IPC/DirQueue.pm#note_1570


[the 1.0 version bump is just because -- no good reason. ;)
this isn't really more stable than any other version!]

--j.



released version 0.09 of IPC::DirQueue by jm

Fri, 27 Apr 2007 13:40:48 +0000

Hi --

Version 0.09 of IPC::DirQueue is now released, and should be showing
up on CPAN rsn. Changes since 0.08:

------------------------------------------------------------------------
r8418 | jmason | 2006-12-21 19:00:03 +0000 (Thu, 21 Dec 2006) | 1 line

add IPC::DirQueue::Job::get_data() API, to access queued job data directly
as a string instead of as a filename, thanks to Collin Winter gmail.com>

------------------------------------------------------------------------
r8379 | jmason | 2006-12-11 20:32:13 +0000 (Mon, 11 Dec 2006) | 1 line

fix documentation; it was unclear for beginners as to how job data got
into and out of IPC::DirQueue, and that reading/writing files is required

------------------------------------------------------------------------
r8103 | jmason | 2006-11-24 14:27:37 +0000 (Fri, 24 Nov 2006) | 1 line

fix platform-specific tests: must use 'use constant' in the conditional
variables, otherwise the tests appear to be failing on win32 platforms
when in fact we're just trying to skip them (see
http://www.nntp.perl.org/group/perl.cpan.testers/377405 for example)


(by the way, IPC::DirQueue users might be interested in this account
of how qpsmtpd and IPC::DirQueue powers the traps.SpamAssassin.org
spamtraps: http://taint.org/2007/04/17/132339a.html )

--j.



Re: cancelling queued items by jm

Thu, 22 Mar 2007 12:32:29 +0000


Brian Duggan writes:
> On Wednesday, March 21, Justin Mason wrote:
> > It sounds pretty good -- just one thing -- how is a caller supposed to get
> > the datapath? Presumably they don't construct it themselves...
>
> >>From the $job object (obtainable by sending a callback
> to visit_all_jobs()). Though actually, it looks like using
> $job->{pathqueue} instead of the datapath might be easier,
> so that pickup_queued_job() doesn't have to read the
> queuefile (to find out the datapath).
>
> > that's ok, that suggestion is a one-liner. Having said that -- a test case
> > "t" script would rock though :)
>
> Okay, I've attached one.

Cool. That makes the idea pretty clear, too. I had to move a little
stuff around to make it more efficient, but it works fine; I also copied
the test script for fanout and ordered queue modes, too. Checked in:

: jm 516...; svn commit -m "added support to pickup a queued item by its data path, thanks to Brian Duggan "
Sending MANIFEST
Sending lib/IPC/DirQueue.pm
Adding t/60pickup_path_basic.t
Adding t/60pickup_path_fanout.t
Adding t/60pickup_path_ordered.t
Transmitting file data .....
Committed revision 9297.

> thanks!
> Brian
>
> ps On a separate note, I noticed that read_control_file() is always
> called with two arguments, but the second argument is ignored. Is
> this perhaps an alternative to saying 'local *IN' in read_control_file()?

oops! bug ;) Thanks for spotting this. fixed:

: jm 745...; svn commit -m "read_control_file() was always opening the control file for read twice, redundantly; fixed" lib/IPC/DirQueue.pm
Sending lib/IPC/DirQueue.pm
Transmitting file data .
Committed revision 9296.

--j.



Re: cancelling queued items by Brian Duggan

Wed, 21 Mar 2007 17:35:12 +0000

On Wednesday, March 21, Justin Mason wrote:
> It sounds pretty good -- just one thing -- how is a caller supposed to get
> the datapath? Presumably they don't construct it themselves...

From the $job object (obtainable by sending a callback
to visit_all_jobs()). Though actually, it looks like using
$job->{pathqueue} instead of the datapath might be easier,
so that pickup_queued_job() doesn't have to read the
queuefile (to find out the datapath).

> that's ok, that suggestion is a one-liner. Having said that -- a test case
> "t" script would rock though :)

Okay, I've attached one.

thanks!
Brian

ps On a separate note, I noticed that read_control_file() is always
called with two arguments, but the second argument is ignored. Is
this perhaps an alternative to saying 'local *IN' in read_control_file()?




Re: cancelling queued items by jm

Wed, 21 Mar 2007 08:50:52 +0000

Brian Duggan writes:
> Hello,
>
> I'm using IPC::DirQueue -- very nice module, thanks.
>
> I wanted the ability to cancel not-yet-active items from
> anywhere in the queue (a la lprm), and was trying
> to think of a safe way to do it. It occurred to me
> that one easy modification to IPC::DirQueue that would
> allow this would be to have pickup_queued_job() take
> an optional parameter which is the name of the data path,
> and then just add something like
>
> next if exists($args{datapath}) &&
> $nextfile ne $args{datapath};
>
> to the while loop.
>
> Any chance this could be incorporated? Or is there
> already another safe way to cancel an item?

It sounds pretty good -- just one thing -- how is a caller supposed to get
the datapath? Presumably they don't construct it themselves...

> thanks
> Brian
>
> (p.s. happy to send a patch if you prefer)

that's ok, that suggestion is a one-liner. Having said that -- a test case
"t" script would rock though :)

--j.



Re: usage question by Thomas J Pinkl

Mon, 05 Mar 2007 09:06:57 +0000

On Mon, Mar 05, 2007 at 11:28:03AM -0500, Thomas J Pinkl wrote:
> Justin, here's a patch which adds an optional "readonly" parameter
> to visit_all_jobs(). If the parameter isn't passed, then it defaults
> to true, thus maintaining backward compatibility. The patch
> includes changes to the POD as well. It's against the current SVN
> source.

Perhaps I whipped this up too quickly. The read-only flag works, as
far as it goes, but jobs are not considered "active" when read-only
is false.

If I understand the code correctly, then I'll also have to do
something to make the job "active" from within visit_all_jobs().

Is that correct?

--
Thomas J. Pinkl | T: 215-442-9300
Senior Systems Architect | 800-444-1427
Health Business Systems, Inc | F: 215-442-7555
An SXC Company |
738 Louis Drive | http://www.hbsrx.com/
Warminster, PA 18974 | http://www.sxc.com/



Re: usage question by Thomas J Pinkl

Mon, 05 Mar 2007 08:28:23 +0000

On Mon, Mar 05, 2007 at 11:24:07AM +0000, Justin Mason wrote:
> Alternatively, I'd be happy to apply a patch that implements
> a version of visit_all_jobs() which allows writes somehow, or
> a way to access a job returned by visit_all_jobs() and render
> it writable.

Justin, here's a patch which adds an optional "readonly" parameter
to visit_all_jobs(). If the parameter isn't passed, then it defaults
to true, thus maintaining backward compatibility. The patch
includes changes to the POD as well. It's against the current SVN
source.

--
Thomas J. Pinkl | T: 215-442-9300
Senior Systems Architect | 800-444-1427
Health Business Systems, Inc | F: 215-442-7555
An SXC Company |
738 Louis Drive | http://www.hbsrx.com/
Warminster, PA 18974 | http://www.sxc.com/



Re: usage question by jm

Mon, 05 Mar 2007 03:25:14 +0000


hi Thomas --

an interesting use-case I hadn't considered! ;) The problem is that
when the queue is accessed in ordered mode, it's simply a lexical
sort by filename; and when accessed in unordered mode, it's simply
the OS-based ordering rather than in any way "random".

What about re-queueing the deferred jobs? ie.

my $dq = IPC::DirQueue->new( \%dq_opts );
while (1) {
my $job = $dq->pickup_queued_job();
last if (! $job);

my $rc = &process_job( $job );
if (!$rc) {
$job->enqueue_file($job->get_data_path());
}
$job->finish();
}

in other words, if a job is to be deferred, take it off the queue
and re-queue it under a new name.

Alternatively, I'd be happy to apply a patch that implements
a version of visit_all_jobs() which allows writes somehow, or
a way to access a job returned by visit_all_jobs() and render
it writable.

--j.

Thomas J Pinkl writes:
> I'm attempting to use IPC::DirQueue 0.08 for a project where messages
> are queued for later processing. However, when dequeuing these
> messages, it will be possible that a message cannot be processed
> immediately, but could be processed later.
>
> I thought I'd be able to use IPC::DirQueue like this:
>
> my $dq = IPC::DirQueue->new( \%dq_opts );
> while (1) {
> my $job = $dq->pickup_queued_job();
> last if (! $job);
>
> my $rc = &process_job( $job );
> if ($rc) {
> $job->finish();
> } else {
> $job->return_to_queue();
> }
> }
>
> But if, say, the first job is returned to the queue, then a subsequent
> call to pickup_queued_job() returns that same job. Turning the loop
> above into an infinite loop (absent an external event which suddenly
> causes process_job() to return true).
>
> I've read through the IPC::DirQueue and IPC::DirQueue::Job pod pages,
> but I'm not seeing any way to iterate through the queue the way that I
> need. The closest method I can see is visit_all_jobs(), but that only
> provides read-only Job objects. I suppose I could play with the
> read-only flag within the callback function, but that doesn't seem
> right.
>
> Does anyone have any suggestions?
>
> --
> Thomas J. Pinkl | T: 215-442-9300
> Senior Systems Architect | 800-444-1427
> Health Business Systems, Inc | F: 215-442-7555
> An SXC Company |
> 738 Louis Drive | http://www.hbsrx.com/
> Warminster, PA 18974 | http://www.sxc.com/



usage question by Thomas J Pinkl

Sat, 03 Mar 2007 08:37:09 +0000

I'm attempting to use IPC::DirQueue 0.08 for a project where messages
are queued for later processing. However, when dequeuing these
messages, it will be possible that a message cannot be processed
immediately, but could be processed later.

I thought I'd be able to use IPC::DirQueue like this:

my $dq = IPC::DirQueue->new( \%dq_opts );
while (1) {
my $job = $dq->pickup_queued_job();
last if (! $job);

my $rc = &process_job( $job );
if ($rc) {
$job->finish();
} else {
$job->return_to_queue();
}
}

But if, say, the first job is returned to the queue, then a subsequent
call to pickup_queued_job() returns that same job. Turning the loop
above into an infinite loop (absent an external event which suddenly
causes process_job() to return true).

I've read through the IPC::DirQueue and IPC::DirQueue::Job pod pages,
but I'm not seeing any way to iterate through the queue the way that I
need. The closest method I can see is visit_all_jobs(), but that only
provides read-only Job objects. I suppose I could play with the
read-only flag within the callback function, but that doesn't seem
right.

Does anyone have any suggestions?

--
Thomas J. Pinkl | T: 215-442-9300
Senior Systems Architect | 800-444-1427
Health Business Systems, Inc | F: 215-442-7555
An SXC Company |
738 Louis Drive | http://www.hbsrx.com/
Warminster, PA 18974 | http://www.sxc.com/



Re: getting: Illegal octal digit '8' by Thomas J Pinkl

Tue, 16 Jan 2007 11:23:32 +0000

On Tue, Jan 16, 2007 at 07:04:44PM +0000, Justin Mason wrote:
> Are you providing values for data_file_mode or queue_file_mode in
> the constructor?

Yes, both are being set.

> It's worth noting that they should be formatted as octal-formatted
> *strings*, not numbers. e.g. '0666', not 0666. Could that be it?

Ah. I'll try that. Thank you.

--
Thomas J. Pinkl | T: 215-442-9300
Senior Systems Architect | 800-444-1427
Health Business Systems, Inc | F: 215-442-7555
An SXC Company |
738 Louis Drive | http://www.hbsrx.com/
Warminster, PA 18974 | http://www.sxc.com/



Re: getting: Illegal octal digit '8' by jm

Tue, 16 Jan 2007 11:05:15 +0000


Thomas J Pinkl writes:
> I've just started using IPC::DirQueue 0.08. Thanks to the author(s)
> for a module that appears to deliver exactly what I need.

You're welcome! ;)

> I received these warnings:
>
> Illegal octal digit '8' ignored at \
> /usr/lib/perl5/site_perl/5.6.1/IPC/DirQueue.pm line 146.
> Illegal octal digit '8' ignored at \
> /usr/lib/perl5/site_perl/5.6.1/IPC/DirQueue.pm line 148.
>
> which corresponds to this section of code:
>
> $self->{data_file_mode} ||= '0666';
> $self->{data_file_mode} = oct ($self->{data_file_mode});
> $self->{queue_file_mode} ||= '0666';
> $self->{queue_file_mode} = oct ($self->{queue_file_mode});
>
> in sub new(). Lines 146 and 148 are the ones that use 'oct'.

Are you providing values for data_file_mode or queue_file_mode in
the constructor?

It's worth noting that they should be formatted as octal-formatted
*strings*, not numbers. e.g. '0666', not 0666. Could that be it?

--j.

> The fix would appear to be:
>
> 146: $self->{data_file_mode} = oct sprintf "0%o",$self->{data_file_mode};
>
> 148: $self->{queue_file_mode} = oct sprintf "0%o",$self->{queue_file_mode};
>
> Can anyone else confirm this?
>
> --
> Thomas J. Pinkl | T: 215-442-9300
> Senior Systems Architect | 800-444-1427
> Health Business Systems, Inc | F: 215-442-7555
> An SXC Company |
> 738 Louis Drive | http://www.hbsrx.com/
> Warminster, PA 18974 | http://www.sxc.com/



getting: Illegal octal digit '8' by Thomas J Pinkl

Tue, 16 Jan 2007 10:41:08 +0000

I've just started using IPC::DirQueue 0.08. Thanks to the author(s)
for a module that appears to deliver exactly what I need.

I received these warnings:

Illegal octal digit '8' ignored at \
/usr/lib/perl5/site_perl/5.6.1/IPC/DirQueue.pm line 146.
Illegal octal digit '8' ignored at \
/usr/lib/perl5/site_perl/5.6.1/IPC/DirQueue.pm line 148.

which corresponds to this section of code:

$self->{data_file_mode} ||= '0666';
$self->{data_file_mode} = oct ($self->{data_file_mode});
$self->{queue_file_mode} ||= '0666';
$self->{queue_file_mode} = oct ($self->{queue_file_mode});

in sub new(). Lines 146 and 148 are the ones that use 'oct'.

The fix would appear to be:

146: $self->{data_file_mode} = oct sprintf "0%o",$self->{data_file_mode};

148: $self->{queue_file_mode} = oct sprintf "0%o",$self->{queue_file_mode};

Can anyone else confirm this?

--
Thomas J. Pinkl | T: 215-442-9300
Senior Systems Architect | 800-444-1427
Health Business Systems, Inc | F: 215-442-7555
An SXC Company |
738 Louis Drive | http://www.hbsrx.com/
Warminster, PA 18974 | http://www.sxc.com/



Re: [PATCH] Advanced tagging and filtering by jm

Thu, 04 Jan 2007 05:34:52 +0000

hey, btw -- something's up with the latest version, at least; whenI apply the patches and make test, it hangs in t/10enq_string as soonas pickup_queued_job() is called. SVN trunk works fine, as doesthe first patch, but as soon as the second patch is applied thishappens.Here's the current rev of the combined patch against SVN:http://taint.org/x/2007/filter.patchThat contains your two patches, my documentation changes, and a "warn" inthe test script to illustrate this hang.--j.Malte S. Stretz writes:> On Wednesday 03 January 2007 16:38 CET Justin Mason wrote:> > backwards compatibility is not a problem here, so go ahead and> > add the tag in addition to the hash.> >> > I think the best location is just before the hash, e.g.> >> > 50.20040909232529941258.TAG.HASH[.PID.RAND]> >> > (vs the simple> >> > 50.20040909232529941258.HASH[.PID.RAND]> >> > when tags are not in use.)> >> > > Actually, when people start to filter the files based on their name,> > > the format must not change anymore in future. Maybe the filter should> > > apply to the tag only...> >> > Yes, I think that's a good idea... people shouldn't have to worry about> > the rest of the filename changing.> > Ok, attached is a patch on top of the last one which changes the code as you > described it above. Additionally there's a revamped test.> > While I was hunting another totally stupid bug, I found out that I actually > forgot to add the $filter in the second call to pickup_job in wait_for_job. > Gave some nasty surprises. And visit_all_jobs now supports $filter, too.> > The syntax for $filter has changed a bit: If it is a string, it must be > equal the tag, if it is a RE created with qr// it is matched as a RE.> > Can you update the QUEUE DIRECTORY STRUCTURE section for me? Your English > is a lot better than mine :)> > Cheers,> Malte> part 3 text/x-diff 7472> --- DirQueue.pm.patch1 2006-12-30 01:27:31.000000000 +0100> +++ DirQueue.pm 2007-01-03 21:08:17.000000000 +0100> @@ -173,8 +173,10 @@> $self->{ordered} = 1;> }> > - $self->{tag} ||= hash_string_to_filename($self->gethostname().$$);> - $self->{tag_sub} ||= sub { return $self->{tag}; };> + $self->{hash} ||= hash_string_to_filename($self->gethostname().$$);> + if (defined $self->{tag}) {> + $self->{tag_sub} ||= sub { return $self->{tag}; };> + }> $self->{tag_max_length} ||= 128;> if (!defined $self->{tag_warn}) {> $self->{tag_warn} = 1;> @@ -456,9 +458,10 @@> > Pick up the next job in the queue, so that it can be processed.> > -The parameter C<$filter> can be used to specify a regular expression which> -is matched against the queued filename. All files which don't match will be > -skipped.> +The parameter C<$filter> can be used to specify either a string or a regular > +expression (with qr//) which is compared (the the first case) or matched > +(in the latter case) against the tag part of the queued filename. All files > +which don't match will be skipped.> > If no job is available for processing, either because the queue is> empty or because other worker processes are already working on> @@ -709,7 +712,7 @@> while (time == $qdirlaststat) {> Time::HiRes::usleep ($pollintvl);> dbg "wait_for_queued_job: spinning until time != stat $qdirlaststat";> - my $job = $self->pickup_queued_job();> + [...]



Re: [PATCH] Advanced tagging and filtering by Malte S. Stretz

Wed, 03 Jan 2007 12:16:38 +0000

On Wednesday 03 January 2007 16:38 CET Justin Mason wrote:
> backwards compatibility is not a problem here, so go ahead and
> add the tag in addition to the hash.
>
> I think the best location is just before the hash, e.g.
>
> 50.20040909232529941258.TAG.HASH[.PID.RAND]
>
> (vs the simple
>
> 50.20040909232529941258.HASH[.PID.RAND]
>
> when tags are not in use.)
>
> > Actually, when people start to filter the files based on their name,
> > the format must not change anymore in future. Maybe the filter should
> > apply to the tag only...
>
> Yes, I think that's a good idea... people shouldn't have to worry about
> the rest of the filename changing.

Ok, attached is a patch on top of the last one which changes the code as you
described it above. Additionally there's a revamped test.

While I was hunting another totally stupid bug, I found out that I actually
forgot to add the $filter in the second call to pickup_job in wait_for_job.
Gave some nasty surprises. And visit_all_jobs now supports $filter, too.

The syntax for $filter has changed a bit: If it is a string, it must be
equal the tag, if it is a RE created with qr// it is matched as a RE.

Can you update the QUEUE DIRECTORY STRUCTURE section for me? Your English
is a lot better than mine :)

Cheers,
Malte



Re: [PATCH] Advanced tagging and filtering by jm

Wed, 03 Jan 2007 07:39:31 +0000


Malte S. Stretz writes:
> On Wednesday 03 January 2007 13:02 CET Justin Mason wrote:
> > hi Malte!
> >
> > Thanks for this, it looks very useful indeed. One tweak, though:
> > > b) Makes the TAG part of the filename configurable. Ah, yeah, TAG was
> > > called HASH before :) I added the parameters tag (static tag), tag_sub
> > > (callback to generate a dynamic tag) and two more (tag_max_length and
> > > tag_warn, cf. inline doc).
> >
> > I would prefer to *add* the TAG part *as well as* the HASH stuff.
> > The hash is useful, since it avoids collisions -- TAG doesn't make
> > that guarantee (nor should it have to).
> >
> > That should be OK, right?
>
> Yeah, that was my original plan. But I wanted to keep the filename format
> backwards compatible. Not sure if thats really needed, I also wasn't sure
> if I should add the tag in the front, the end or in between. In the end I
> went for compatibility and assumed that in most cases the tags will still
> be quite unique -- at least in the producer/consumer case where you've got
> most probably some kind of pid in there they will be at least as unique as
> the current hostname/pid based ones. And if not, we've got the timestamp
> plus the additional really random part at the end :)

backwards compatibility is not a problem here, so go ahead and
add the tag in addition to the hash.

I think the best location is just before the hash, e.g.

50.20040909232529941258.TAG.HASH[.PID.RAND]

(vs the simple

50.20040909232529941258.HASH[.PID.RAND]

when tags are not in use.)

> Actually, when people start to filter the files based on their name, the
> format must not change anymore in future. Maybe the filter should apply to
> the tag only...

Yes, I think that's a good idea... people shouldn't have to worry about
the rest of the filename changing.

--j.



Re: [PATCH] Advanced tagging and filtering by Malte S. Stretz

Wed, 03 Jan 2007 07:28:22 +0000

On Wednesday 03 January 2007 13:02 CET Justin Mason wrote:
> hi Malte!
>
> Thanks for this, it looks very useful indeed. One tweak, though:
> > b) Makes the TAG part of the filename configurable. Ah, yeah, TAG was
> > called HASH before :) I added the parameters tag (static tag), tag_sub
> > (callback to generate a dynamic tag) and two more (tag_max_length and
> > tag_warn, cf. inline doc).
>
> I would prefer to *add* the TAG part *as well as* the HASH stuff.
> The hash is useful, since it avoids collisions -- TAG doesn't make
> that guarantee (nor should it have to).
>
> That should be OK, right?

Yeah, that was my original plan. But I wanted to keep the filename format
backwards compatible. Not sure if thats really needed, I also wasn't sure
if I should add the tag in the front, the end or in between. In the end I
went for compatibility and assumed that in most cases the tags will still
be quite unique -- at least in the producer/consumer case where you've got
most probably some kind of pid in there they will be at least as unique as
the current hostname/pid based ones. And if not, we've got the timestamp
plus the additional really random part at the end :)

Actually, when people start to filter the files based on their name, the
format must not change anymore in future. Maybe the filter should apply to
the tag only...

Cheers,
Malte



Re: [PATCH] Advanced tagging and filtering by jm

Wed, 03 Jan 2007 06:13:42 +0000


hi Malte!

Thanks for this, it looks very useful indeed. One tweak, though:

> b) Makes the TAG part of the filename configurable. Ah, yeah, TAG was
> called HASH before :) I added the parameters tag (static tag), tag_sub
> (callback to generate a dynamic tag) and two more (tag_max_length and
> tag_warn, cf. inline doc).

I would prefer to *add* the TAG part *as well as* the HASH stuff.
The hash is useful, since it avoids collisions -- TAG doesn't make
that guarantee (nor should it have to).

That should be OK, right?

otherwise, all looks good...

--j.



[PATCH] Advanced tagging and filtering by Malte S. Stretz

Fri, 29 Dec 2006 12:51:56 +0000

Hi Justin,I was looking for a nice IPC module on CPAN and what did I find? Yet another Perl Module created by you :)Unfortunately was IPC::DirQueue really slow in one of the cases I tried to use it for. Thats why I wrote the attached patch. But first: What do I need?I have a server which has a tcpdump running, listening for UDP packets, and where you can connect via TCP to request notifications about some kind of packets. The notifications are based on ids which are inside the UDP packets. (This is actually used to do some UDP hole punching as a University project.)So we've got two parallel queues, one for the incoming requests and one for the outgoing replies. On the one side we can have n processes pushing requests into the incoming queue and waiting for replies on the outgoing one. On the other side is just a single consumer/producer, which parses the tcpdump output, looks wether there's a request for the id in the incoming queue and if so, puts a reply into the outgoing queue which is then sent via TCP to the user.That worked pretty well but at some point I got an almost-deadlock: If one of the clients pushes loads of requests into the incoming queue, it can overwhelm the consumer which can happen to look at all those new packets first before it sees the request it was actually looking for. Additionally is there quite some overhead because each time the metadata has to be read first before I can identify the owner.Actually the following simple code can have the effect I described above: use IPC::DirQueue; my $dq = IPC::DirQueue->new({ dir => 'dq' }); $dq->enqueue_string("foo!", {id => 'foo'}); $dq->enqueue_string("bar!", {id => 'bar'}); while (1) { my $job = $dq->wait_for_queued_job(0, 0.1); my $id = $job->{metadata}->{id}; #print $id; if ($id eq 'foo') { my $data = $job->get_data(); $job->finish(); print $data; } else { $job->return_to_queue(); } }If started often enough, at some point we will have loads of bars which block the foos. Just uncomment the print $id and the app will scream bar all the time but never foo. Hmmm... I guess there's actually another bug hidden because sometimes I see no foo at all; maybe the dir iterator is reset or something.Whatever, I actually went to increase the speed for applications like this. The attached patch does the following:a) Adds a parameter file_mode to the constructor; doesn't really increase speed but I guess in most cases one wants to change both modes and this way its less code :)b) Makes the TAG part of the filename configurable. Ah, yeah, TAG was called HASH before :) I added the parameters tag (static tag), tag_sub (callback to generate a dynamic tag) and two more (tag_max_length and tag_warn, cf. inline doc).c) Adds $filter parameter to wait_for_queued_job() and pickup_queued_job(). These accept a regular expression which is matched against the queue file name. I don't really like the position of the $filter parameter in the wait method, I'd prefer to have it as the first parameter and experimented with a 'ref $_[1] eq Regexp' to keep it backwards compatible but then just went the simple way and put it to the end :)d) The actual filtering stuff is done in the iterators. I guess I did quite some refactoring/rewriting there. The most important thing is the $iter->{filter} code, that I made the iterator more or less independent from the $self object and thus could use it to rewrite the fanout stuf[...]



Re: released version 0.08 of IPC::DirQueue by jm

Tue, 21 Nov 2006 06:21:10 +0000


ha -- that's version 0.08, not "x.xx" ;)

--j.

Justin Mason writes:
> Hi --
>
> Version 0.08 of IPC::DirQueue is now released, and should be showing
> up on CPAN rsn. Changes since 0.07:
>
> ------------------------------------------------------------------------
> r8093 | jmason | 2006-11-21 13:04:53 +0000 (Tue, 21 Nov 2006) | 1 line
>
> add t/12_no_dir.t, to test for bug 21312 (no queue directory causes die())
> ------------------------------------------------------------------------
> r6840 | jmason | 2006-09-04 11:18:38 +0100 (Mon, 04 Sep 2006) | 1 line
>
> calling visit_all_jobs() with a nonexistent queue directory triggered some broken err
> or-handling code, resulting in a die(); fixed. thanks to Keith Amling, http://rt.cpa
> n.org/Public/Bug/Display.html?id=21312
> ------------------------------------------------------------------------
> r6839 | jmason | 2006-09-04 11:16:10 +0100 (Mon, 04 Sep 2006) | 1 line
>
> documentation clarification on queue structure; some of the grammar was poor
> ------------------------------------------------------------------------
> r6694 | jmason | 2006-07-24 14:10:27 +0100 (Mon, 24 Jul 2006) | 1 line
>
> adding CHANGES file to svn



released version x.xx of IPC::DirQueue by jm

Tue, 21 Nov 2006 05:51:32 +0000

Hi --

Version 0.08 of IPC::DirQueue is now released, and should be showing
up on CPAN rsn. Changes since 0.07:

------------------------------------------------------------------------
r8093 | jmason | 2006-11-21 13:04:53 +0000 (Tue, 21 Nov 2006) | 1 line

add t/12_no_dir.t, to test for bug 21312 (no queue directory causes die())
------------------------------------------------------------------------
r6840 | jmason | 2006-09-04 11:18:38 +0100 (Mon, 04 Sep 2006) | 1 line

calling visit_all_jobs() with a nonexistent queue directory triggered some broken err
or-handling code, resulting in a die(); fixed. thanks to Keith Amling, http://rt.cpa
n.org/Public/Bug/Display.html?id=21312
------------------------------------------------------------------------
r6839 | jmason | 2006-09-04 11:16:10 +0100 (Mon, 04 Sep 2006) | 1 line

documentation clarification on queue structure; some of the grammar was poor
------------------------------------------------------------------------
r6694 | jmason | 2006-07-24 14:10:27 +0100 (Mon, 24 Jul 2006) | 1 line

adding CHANGES file to svn




Re: Changing metadata inside of a job runner by jm

Wed, 10 May 2006 14:46:50 +0000


Duncan Hill writes:
> I'm looking at your IPC::DirQueue module as a solution to my current
> home-grown queue runner, and have a few questions that a brief check of the
> module doesn't answer.
>
> 1) Can the metadata be written to from a job runner? There's no function for
> this, but there's nothing obvious saying 'don't do that'.
> 2) If not, would you consider this feature?
>
> Essentially, the code I'm fitting dq into talks to an SQL database a lot. If
> a log file encounters a failure being parsed, I'd like to be able to tag the
> metadata with a 'don't touch, leper' flag so that other queue runners don't
> try to handle it when it gets returned to the queue. Without the ability to
> change the metadata, all I can do is crank the priority down, and have a
> specialised runner that only picks up low priority items and moves them out
> of the main queue (which would work I suppose).

hi --

Unfortunately not :( Right now, the metadata is written to only
once, at enqueue time, and only read thereafter.

It'd make a good patch, though ;)

--j.



CPAN Upload: J/JM/JMASON/IPC-DirQueue-0.07.tar.gz (fwd) by Justin Mason

Mon, 10 Apr 2006 13:34:21 +0000


changes:

- support running the tests with a perl that isn't /usr/bin/perl, e.g.
after running '/usr/local/perl561/bin/perl Makefile.PL; make; make test'

- The utime() idiom being used to touch files and directories was only
added in perl 5.7.2. Add backwards-compat code to deal with the 5.6.x
callers

- oops, indexd t scripts required POE, so the tests failed if that module
wasn't installed. make them optional and not run if POE isn't installed

--j.

------- Forwarded Message

Date: Mon, 10 Apr 2006 22:30:27 +0200
From: PAUSE
To: "Justin Mason"
Subject: CPAN Upload: J/JM/JMASON/IPC-DirQueue-0.07.tar.gz

The uploaded file

IPC-DirQueue-0.07.tar.gz

has entered CPAN as

file: $CPAN/authors/id/J/JM/JMASON/IPC-DirQueue-0.07.tar.gz
size: 25658 bytes
md5: 2b8ffecc3ab15c4610c459fd3c038434




IPC::DirQueue 0.06 released by jm

Mon, 09 Jan 2006 18:47:11 +0000

-----BEGIN PGP SIGNED MESSAGE-----Hash: SHA1hey all --I've just uploaded v0.06 of IPC::DirQueue. This has a few interestingchanges; lemme quote CHANGES on the big ones: - API CHANGE: invalid metadata in an enqueued job will now causes die() to be called, instead of silent ignore. - add dq-indexd, an optional, experimental central server to maintain DQ indexes, instead of using readdir() filesystem APIs. It requires POE - fix race condition in wait_for_queued_job() that can result in missing jobs enqueued inside the same 1-second window. This would only manifest if no further files were enqueued to the same dir. - make the 600-second active-lock timeout configurable; document it better; and reconcile code behaviour with what the documentation says it does re behaviour when a stale lock occurs on a task supposedly active on a remote system. (The code was changed to match the documentation.) - fixed a long-running bug; left-over control files in 'active'. Turns out these were a side effect of qproc A completing a task before qproc B even started creating a lockfile; in that case, qproc B would get a lock on the now-complete task, then find that the queue control file was nonexistent and give up, attempting to remove the 'active' file, but instead unlinking the now-already-unlinked temporary filename, and leaving the real 'active' file behind. Totally harmless, fixed anyway. - support Reiserfs and XFS queue dirs, which do not update directory mtimes when a file is creating within them. - patch from Anton Berezin : return reference of sorted files list internally, and use built-in sort instead of a { $a cmp $b } callback. Both are good for speed of ordered dequeuingIt's filtering through CPAN now. I wouldn't recommend using dq-indexdfor anything serious just yet -- it's still experimental! -- but therest of those features are solid, and the fix for the "leftover controlfiles" bug is welcome. ;)for what it's worth, here's some dq-indexd benchmarks from a mixedethernet/wireless (home) network, in case you all are curious ;) --LOCAL TIMINGS: 85changing_indexd.t real 0m53.525s user 0m5.228s sys 0m0.446s real 0m53.452s user 0m5.113s sys 0m0.536s real 0m53.437s user 0m5.411s sys 0m0.454s 85changing_ordered.t real 0m52.359s user 0m0.350s sys 0m0.121s real 0m52.432s user 0m0.352s sys 0m0.104s real 0m51.588s user 0m0.256s sys 0m0.077s 85changing_basic.t real 0m52.314s user 0m0.143s sys 0m0.043s real 0m51.337s user 0m0.346s sys 0m0.106s real 0m52.480s user 0m0.345s sys 0m0.122sREMOTE TIMINGS: 85changing_indexd.t real 3m07.718s user 0m3.191s sys 0m1.065s real 3m06.452s user 0m3.881s sys 0m1.324s real 3m09.563s user 0m3.532s sys 0m1.244s 85changing_ordered.t real 3m21.705s user 0m0.646s sys 0m0.836s real 3m31.031s user 0m0.630s sys 0m1.002s real 3m26.674s user 0m0.645s sys 0m1.027s 85changing_basic.t real 3m46.062s user 0m0.585s sys 0m0.764s real 3m40.202s user 0m0.465s sys 0m0.609s real 3m49.065s user 0m0.558s sys 0m0.854scheers,- --j.-----BEGIN PGP SIGNATURE-----Version: GnuPG v1.4.1 (GNU/Linux)Comment: Exmh CVSiD8DBQFDwx+rMJF5cimLx9ARAkphAJ4xSceOTjc/6q55vlrgvXCt0[...]



Re: [PATCH] speed up ordered dequeueing in IPC-DirQueue 0.05 by jm

Thu, 05 Jan 2006 18:59:39 +0000

-----BEGIN PGP SIGNED MESSAGE-----Hash: SHA1(cc'd mailing list)Anton Berezin writes:> Justin,> > On Thu, Dec 29, 2005 at 02:18:06PM +0100, Anton Berezin wrote:> > > I don't think it is possibly to do much with the algorithm as such,> > since the sorting needs to be done, and the results cannot really be> > cached if we want the ordered queue to be really ordered, but I found a> > definite thing to improve.> > On the further thought, even the existing ordered method does not> provide us with an absolute guarantee that the things are going to be> processed in order, since there is a race between readdir and newer> enqueue operations. So it is possible to somewhat relax it even further> and achieve a really good speed up for some, but not all, cases.> > The idea is to keep a last_enqueued file in the queue dir. We are only> interested in its mtime. Then, on a dequeue operation, we cache the> result of readdir and only re-read the file list if the cached file list> is empty or if the mtime of last_enqueued file has changed.> > This obviously won't help much if there are a lot of concurrent enqueue> operations, but for the case "first enqueue a bunch of jobs, then launch> several processors, then repeat after some time" it helps absolutely> tremendously.Hi Anton --many thanks for this work! The patches are welcome.I've applied the ordered-dequeueing speedup patch from the previous mail- -- with a few changes to match latest SVN (see below). It's in as r2353.I never thought the interpreter-level code path would have such a largeeffect on execution speed; I was assuming it'd all be system calls, which*should* be a lot slower. Looks like I was wrong. ;)However, regarding the patch from *this* mail, it's important to note thatthe underlying code changed in SVN at the start of December; r2255 added a"last-enqueue" flag file, initially at least. I then realised that thequeue directory *itself* can be used for this purpose, in abackwards-compatible way, so now it just adds similar semantics to thedirectory inode itself.I found that this change was necessitated to support Reiserfs andXFS-hosted queue directories, since those filesystems didn't modify themtime of a directory when changes were made within it, as ext3 did. (Iwas assuming that this was a POSIX requirement; again, looks like I waswrong. ;)There were then several more changes since then to SVN trunk, made to fiximportant bugs (including another race condition) and add new,experimental functionality.This patch looks very useful -- I'd be happy to add it, as a patch againstcurrent SVN, if that'd be possible. Also, it may be worthwhile making itthe ordered default, assuming I can't think of a good reason why not ;)Right now though, it doesn't apply cleanly to SVN.by the way there is a mailing list; tosubscribe. it might be worth sending future patches there so other users cancomment on them too.- --j.> I did not think it was a good idea to modify the default "ordered"> behavior, so my patch makes "ordered" options to be a tri-value, with> "2" denoting this "approximately ordered" way of doing things.> > The result of the same tests as yesterday, with ordered => 2 is> > t[...]



CPAN Upload: J/JM/JMASON/IPC-DirQueue-0.05.tar.gz (fwd) by jm

Fri, 22 Apr 2005 13:16:16 +0000

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


IPC::DirQueue version 0.05 is released! Here's the key new features:

- new API: added enqueue_sub() API, to enqueue data from a closure.

- bug: more paranoid link_into_dir_no_retry implementation, hopefully
will fix a bug that leaves tmp active files lying around.

- bug: skip fork()-requiring tests on win32.

should be showing up on
http://www.cpan.org/modules/by-authors/id/J/JM/JMASON/ rsn...

- --j.


- ------- Forwarded Message
> Date: Fri, 22 Apr 2005 22:05:22 +0200
> From: PAUSE
> To: "Justin Mason"
> Subject: CPAN Upload: J/JM/JMASON/IPC-DirQueue-0.05.tar.gz
>
> The uploaded file
>
> IPC-DirQueue-0.05.tar.gz
>
> has entered CPAN as
>
> file: $CPAN/authors/id/J/JM/JMASON/IPC-DirQueue-0.05.tar.gz
> size: 16861 bytes
> md5: 4ceb6ae1882c5412c3daed7e30f04f3a
>
> No action is required on your part
> Request entered by: JMASON (Justin Mason)
> Request entered on: Fri, 22 Apr 2005 20:03:17 GMT
> Request completed: Fri, 22 Apr 2005 20:05:22 GMT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFCaVstMJF5cimLx9ARAu4hAKCVooT93bvn4JGmQW3qO93OaGTRhQCcCcjs
+gvXBYoxsCcTHVaJdWNtvfI=
=+2G+
-----END PGP SIGNATURE-----




qpsmtpd queue plugin? by David Sparks

Thu, 21 Apr 2005 00:21:55 +0000

Greetings,

I've been investigating using this module with qpsmtpd. Does such a
plugin already exist?

I hacked together a quick proof of concept to test performance of the
methods and found roughly:

$dq->enqueue_file: 2600 msg/min
$dq->enqueue_string: 2900 msg/min

Note these are qpsmtpd #s and the queue dir lives on a NFS mounted
volume so it isn't showing IPC::DirQueue performance.

It also seems that when using the enqueue_file method the original file
still exists and has to be rm'd separately? Is this expected? My
assumption (love em) was that the original file would be re-linked into
the queue.

Cheers,

ds



Re: how to "kill" a job by jm

Wed, 02 Feb 2005 18:59:14 +0000

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Bruce Robertson writes:
> Is there such an API point?
> If not, do you have something under development?
> Else, any pointers on how to do it?

There isn't yet such a thing, no, and I haven't planned it. (yet)

Kill an as-yet unprocessed job would be easy -- simply an unlink.
But if the job is active, it'd be quite a bit harder; it's easy in the
LPR-style of queueing, because that has (a) a central server and (b) each
job is a separate process, so a kill(3) of the process can be used to kill
the job, but in DQ, there's no separate process per job -- one process
can deal with many jobs sequentially.

I would say the cleanest way to do it would be to arrange that dq servers
can provide a new constructor switch indicating a signal that should
be delivered if a job kill is to be requested; if that switch isn't
present, killing an active job isn't supported.

The DQ server code would then write that to a flag file in the queue
directory, so that DQ clients (ie. a new "dq-kill" script) can be aware
that the server(s) support this.

The server code should then implement a signal handler for that signal;
that handler will check to see if it is currently processing a killed job
(ie. if the data file has been unlinked before it's complete), and give up
the remainder of the job's processing if it is.

(alternatively, the server could just ignore the signal, if it's not an
issue for that server's design, e.g. if the jobs are very lightweight, and
completing processing of a killed job will not cause problems.)

If that constructor switch was not specified, I think it would be unsafe
to assume that DQ kill is supported by that queue; we can't go unlinking
active jobs without some kind of "this is supported" negotiation, as a
disappearing job file may cause a die() in server code that wasn't aware
this was likely to happen.

- --j.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFCAZN6MJF5cimLx9ARAr5NAJ0XaiEfTu55tve/g2gmuylLfwrK7wCdF2yF
dLOlbsvNhHl50TfJYtzwJfA=
=jxvb
-----END PGP SIGNATURE-----




IPC::DirQueue 0.04 released by jm

Tue, 18 Jan 2005 17:22:32 +0000

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi Jeffrey --

as the (so far) only known dq user, thought you might like to
know about a few DQ bits and pieces. I've cc'd the all-new mailing
list, for any other users out there ;)

First off, there's a new release -- changes:

- bug: dqserver should run forever when --njobs is 0, or unset; instead,
it was exiting immediately. Patch from Jeffrey Wescott binaryfeed . org>.
- bug: tracked down a mysterious condition that resulted in unused files
being left in the active dir.
- added debugging (so far only enableable by setting a package global
var)
- updated documentation on the directory structure used
- added note to POD about mailing list

Download from CPAN, once the mirrors update.

Next, the SVN repository lives here:
http://svn.perl.org/modules/ipc-dirqueue/

and there's a mailing list -- mail ipc-dirqueue-subscribe .at. perl.org to
subscribe.

(thanks to Ask and Robert @perl.org who are now providing free SVN
repos and list hosting for CPAN modules!)

- --j.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFB7bZAMJF5cimLx9ARAvKiAKCdnfTuBhqyjXcaalJ/arsRykMGeACgoyZu
Cu+G9JasWZC6KzF9TrN9SVo=
=pUoq
-----END PGP SIGNATURE-----




Re: bug in IPC-DirQueue 0.03? by jm

Thu, 13 Jan 2005 02:55:59 +0000

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi Jeffrey --

thanks for that patch -- it's applied to svn now. I should really
do a new release RSN...

- --j.

Jeffrey Wescott writes:
> A patch for the bug described below ... I may send you another patch
> for a -q option (quiet).
>
> -------------------------------->8 snip, snip
> Index: dq-server
> ==================================================================RCS file: /usr/local/cvsroot/IPC-DirQueue/dq-server,v
> retrieving revision 1.1.1.1
> diff -u -r1.1.1.1 dq-server
> --- dq-server 11 Jan 2005 21:55:01 -0000 1.1.1.1
> +++ dq-server 11 Jan 2005 23:56:21 -0000
> @@ -56,10 +56,11 @@
> ) or usage();
> $dir or usage();
>
> +my $jobsleft = $njobs;
> my $dq = IPC::DirQueue->new({ dir => $dir });
>
> while (1) {
> - last if ($njobs-- <= 0);
> + $njobs && last if ($jobsleft-- <= 0);
>
> my $job = $dq->wait_for_queued_job();
> if (!$job) { next; }
> -------------------------------->8 snip, snip
>
> ++jeffrey
>
> On Jan 11, 2005, at 15:48, Jeffrey Wescott wrote:
>
> > The dq-server command claims in the documentation that not specifying
> > --njobs or specifying --njobs 0 should mean that the server will run
> > forever (i.e. -- process an unlimited number of jobs). However, if
> > you don't specify --njobs or specify --njobs 0, it just quits.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFB5dxiMJF5cimLx9ARAskvAKCcDqmjF5fv4H7CccGoYk9zMUKK9QCeLDOt
k3L2s9sElVjlOKyrO08l6Go=
=gjXV
-----END PGP SIGNATURE-----