Background tasks precision issue

    • 75 posts
    March 27, 2018 2:22 PM EDT

    It looks like the background tasks, with their cUrl default setting, are very imprecise:

     

    2018-03-23T20:23:04+00:00 DEBUG (7): Task Execution Check [140953545] : Cache Prefetch
    2018-03-23T20:30:28+00:00 DEBUG (7): Task Execution Check [471303029] : Cache Prefetch
    2018-03-23T20:35:41+00:00 DEBUG (7): Task Execution Check [160094155] : Cache Prefetch
    2018-03-23T20:41:29+00:00 DEBUG (7): Task Execution Check [1659734103] : Cache Prefetch
    Seconds in between: 444, 313, 348 - default is 300 ?

    2018-03-23T20:35:41+00:00 DEBUG (7): Task Execution Check [160094155] : Background Mailer
    2018-03-23T20:38:29+00:00 DEBUG (7): Task Execution Check [282509907] : Background Mailer
    2018-03-23T20:40:11+00:00 DEBUG (7): Task Execution Check [1414744580] : Background Mailer
    2018-03-23T20:41:29+00:00 DEBUG (7): Task Execution Check [1659734103] : Background Mailer
    Seconds in between: 168, 102, 78 - default is 15 ?

    2018-03-23T20:32:37+00:00 DEBUG (7): Task Execution Check [1979191693] : Member Data Maintenance
    2018-03-23T20:38:29+00:00 DEBUG (7): Task Execution Check [282509907] : Member Data Maintenance
    2018-03-23T20:40:11+00:00 DEBUG (7): Task Execution Check [1414744580] : Member Data Maintenance
    Seconds in between: 352, 102 - default is 60 ?

     

    Is there anybody who can help us with that? We saw the official docs but they are too generic, for instance: cron is more precise - wonderful we can set a crontab but how are we supposed to set it??? 

    Keep in mind we will need several background tasks,some running very frequently (the most frequent is every 3 seconds)

    Please advise

    • Moderator
    • 6923 posts
    March 28, 2018 5:04 AM EDT

    We were working on a release yesterday and I couldn't ask a core dev for your ticket. Let me ask him when he's available today (if I can pull him out of the coding cave). It's hard lately to do that as we are working on the 4.10 bug stuffs and something pretty awesome. Roadmap post soon!

    • Moderator
    • 6923 posts
    March 28, 2018 7:41 AM EDT

    "In regards to the Curl cron job, it can’t be precise because it requires a user on the site to execute it"

    That should help explain the task schedule rates.

    • 75 posts
    March 28, 2018 7:55 AM EDT
    Donna said:

    "In regards to the Curl cron job, it can’t be precise because it requires a user on the site to execute it"

    That should help explain the task schedule rates.

    To a point - I understood (from reverse engineering) that the cUrl method is connected to the HTTP requests but we have an average of about 15 requests a second, just on our API webservice page, so in theory there would be enough requests to keep it running quite smoothly.

    The docs say the alternative is to use crontab, which is said to be more precise. 

    I don't mind to use crontab, but I need to know what script I am supposed to run (crontab isn't but a unix sort fo timer that runs scripts at given intervals) and with what parameters.

    Is it one crontab for each task? is it a crontab running a controll scrpt that runs the tasks?

    The only obvious thing is that the tasks are not written to be run independently: some other object / script is calling their methods - so what am I supposed to call fromt eh crontab?

    And what when the interval is below one minute, which is the shortest interval that "normally" crontab can handle? There are some well known work arounds... do I need to take care of such aspects or the system si already taking care of them

    ?


    This post was edited by David at March 28, 2018 8:05 AM EDT
    • Moderator
    • 6923 posts
    March 28, 2018 8:11 AM EDT

    ** she blinks as she runs to get Yoda as he'll know ** <-- Yoda is Ray

    Yeah. Kryptonite. I will try to get an answer. 


    This post was edited by socialenginestaff at March 28, 2018 8:12 AM EDT
    • Moderator
    • 6923 posts
    March 28, 2018 8:20 AM EDT

    From Yoda:

    ran a search in our script and it seems we run something like

    echo $(wget -O - 'http://localhost/utility/tasks?key=[Trigger_Access_Key]¬rigger=1') 2>&1 >> '/var/www/temporary/log/tasks.log'


    so we still run the HTTP method, but cron is the one running it instead.
    What do they want to accomplish though? Is cron not running for them or they just want it to run at a specific time?

    • Moderator
    • 6923 posts
    March 28, 2018 8:26 AM EDT

    Also:

    "oh the site does have info. You select “cron” from the drop down and then save

    it gives you a guide what to do"

    • 75 posts
    March 28, 2018 11:45 AM EDT

    I really needed to marry you when I could, Donna :-P

    Please tell Yoda that his answers are finally helping - we are nearly there!

    1)

    What we need to achieve is precision of the intervals. If you show him the log extract that I put in my original post you will see that the tasks are having an abnourmous level of variability despite the fact that there are always and constantly thousands and thousands of HTTP requests every hour.

    One same task for instance has been run every 444, 313, 348 secs (should run every 300 secs), another has been run every 168, 102, 78 etc etc

    We are developing a live cams plugin that is based on per minute monetization. Our tasks need to run quite precisely every N given seconds

    2) 

    I searched through our installation and didn't find exactly the expression he showed to you (the strange ¬ char I suppose is a transcodification copy&paste error)

    Strange character apart, I'd assume that the URL to be set in cron actually is: 

    http://localhost/utility/tasks?key=[Trigger_Access_Key]&notrigger=1

     

    However, I could locate modules/Core/Model/DbTable/Tasks.php and modules/Core/controllers/UtilityController.php which match the URL Yoda quoted.

    Inside Tasks.php I found:

    protected function _triggerMethodCron() { return false; }

     

    So I don't think that presently we are using cronTab at all.

    I'm now going to try and set cron to see if the instructions will popup for me.

     

    In that case:

    a) the trigger key would have to be set manually I suppose - might it get changed by the system without us realizing?

    b) I often see also a &pid= argument in the calls in Tasks.php but that one I don't see how we could possibly pass it via cron

    • 75 posts
    March 28, 2018 2:57 PM EDT

    OK relevant update:

    1) I managed to get the background tasks run via crontab

    2) The intervals between each execution have gotten very precise

    BUT

    3) the intervals don't seem to match the settings, in case of short intervals

     

    EXAMPLE:

    Cache Prefetch has a timeout setting of 300 and it is now being executed exactly every 300 seconds (previously it was 444, 313, 348 etc)

    Background Mailer is now being executed exactly every 60 seconds but it is set up for 15 seconds

    Member Data Maintenance is now being executed exactly every 120 seconds but its default timeout setting is 60 seconds

    Why is that? Is the system deeming those unnecessary and not running them? Or has it to do with the crontab 60 seconds minimum interval?

     

    NOW:

    Our new tasks (we are going to create 5-6 new tasks) will have to run as frequently as once every 3 seconds, every 15 seconds, every 30 secs, etc

    Will they run at the proper frequency / interval? Or will they all end up running once every minute?

    If so, what do you advise?

     

    There are well known hacks for running crontabs more frequently than every 1 minute, just instnatiating the whole web application again every 3 seconds (most frequent seems quite heavy on resources)

    What about creating a task with a never ending while loop and a little sleep inside for the most frequent tasks? That would seem a better solution resources wise but I'm not sure how the system would handle the timeout or the process number restrains as abrupt interruption of the tasks would cause data loss

     

    • Moderator
    • 6923 posts
    March 29, 2018 4:59 AM EDT

    Glad you got some progress. I can't ask today. Please remind me tomorrow and I'll see if I can pull him from the code cave.

    • 75 posts
    March 29, 2018 9:31 AM EDT

    Will do!

    • 75 posts
    March 30, 2018 6:55 AM EDT

    Update (and reminder to you! )

    I spent yesterday experimenting and rev. engineering the tasks management code - I also found, I believe, a bug in SE code (see below).

    It looks like the code simply performs a loop and executes all the tasks in the order they have been stored in the core_tasks table in a very syncronous manner - simply one after the other - unless the testing on core_tasks and core_processes suggests the necessity to skip one round, either because the given taks is still in execution by another process or because the timeout hasn't expired yet (not all details of the interactions between core_process and core_tasks are clear to me yet)

     

    This being the case, tests have been run both with CRON trigger and cURL trigger, under these conditions:

    HTTP requests: about 15 per second

    CPU running between 10-20%

    Tasks settings set as:

    interval trigger = 1

    Max curr processes = 64

    Max task per req = 64 (all this to make sure that there wouldn't be any cap or bottleneck forced on the system for performance reasons)

    Number of listed background tasks: 13 (the core standard ones plus 5 new tasks of our creation which are still empty: they simply write a unix timestamp in the log)

     

    From the results it seems that both the CRON and cURL method have problems.

    CRON:

    It's very precise, of course, but because of how the tasks loop works, it's impossible to run the same task more than once inside the same cycle, asyncronously to the other tasks - so the fast tasks that are supposed to run more than once per minute will not work this way.

    CURL:

    With the amount of CPU power and processes allowed and with the number of HTTP requests that we have, this should have worked for very fast running tasks, but it didn't - I assume because of the overhead? From my last log:

    Task supposed to run every 60 seconds, ran every: 6667, 5399, 61 secs! Task Task supposed to run every 3 secs, ran every 6, 5, 8. Tasks supposed to run every 20 secs, ran every: 5399, 47, 20.

     

    POSSIBLE SOLUTIONS:

    I see two possible solutions, but I'd love to hear your input and advice as well.

    1)

    Create a do-executeCode-sleep-while loop inside the fastest tasks, while at the same time setting their timeout to 0, so that for instance a tasks that is supposed to run every 3 seconds becomes  a task of one minute with a loop inside that is executed 20 times and that sleeps for 3 seconds after each execution. Having timeout == 0 would allow that task to executed again immediately after its conclusion.

    On paper this solution might work with cURL and with a large number of processes and requests. However, given the wild variations that I'm seeing in the logs I'm afraid I'd be just wasting devoloping time.

    2)

    I create specific routes inside our module. I set the crontab to run a little bash script of our creation that calls these new routes with ?notrigger=1 at specific intervals of seconds (so precisely every 3 seconds, every 15 seconds, every 20 secs and so on). With this new routes and with the cron I basically create a high performance / low overhead system of tasks that runs parallel to the official one.

    I suspect that this approach has more chances of actually working, although I'd rather always use standard tools whenever possible.

    What says you?

     

    As to the bug:

    This is code from Core/Model/DbTables/Tasks.php method _executeTaskCheck()

    // Task is not ready to be executed again yet if( $task->timeout > 0 ) { if( time() < $task->started_last + $task->timeout ) { return false; } if( time() < $task->completed_last + $task->timeout ) { return false; } }

    I believe what you actually wanted was more like:

     

    // Task is not ready to be executed again yet if( $task->timeout > 0 ) { if( time() < $task->started_last + $task->timeout ) { return false; } if( time() < $task->completed_last ) { return false; } }

    Otherwise, since always completed_last >= started_last (unless the latest execution has been interrupted) the second IF makes the first one always redundant, and establishes a strictly syncronous behaviour that in my experience is not good with background tasks. More sophisticated approaches than the one I suggested are also possible, but definitely the timeout should always be calculated from the execution start, at least in normal conditions, not from the end.


    This post was edited by David at March 30, 2018 8:01 AM EDT
    • Moderator
    • 6923 posts
    March 30, 2018 8:18 AM EDT

    Copied and posted to a core dev. Actually I mean this link as I've no idea.Thanks for posting. We do have improvements coming though.

    • 75 posts
    March 30, 2018 9:15 AM EDT

    Anything related with the background tasks? I wouldn't like to build all this parallel system and then you guys fix it the following day... :-P

    • Moderator
    • 6923 posts
    April 2, 2018 4:51 AM EDT

    We are improving many things for next major release. We will post a roadmap. It won't be within the next few days for a release.

    • 75 posts
    April 4, 2018 2:57 PM EDT

    it's OK. In the end we decided to add our own trigger system to run our own tasks.

    • Moderator
    • 6923 posts
    April 4, 2018 3:19 PM EDT

    Ok as I did try to pull him from the coding cave. However, he had turned into dracucoder and the light burned him... (basically he's in a super mega critical point in development and can't get away to come answer atm)