Skip to content

External scripts being killed #41

Closed

Description

@it25fg

Steps to reproduce

  1. activate 'workflow external scripts' app
  2. create a rule that calls an external script
  3. trigger the event that will call the external script
  4. wait until the next cron.php schedule

Expected behaviour

Scripts should be started upon next cron.php schedule, and should be allowed to finish

Actual behaviour

External script gets killed by SIGTERM some milliseconds after start

Workflow Script app

Workflow Script app version: (see apps admin page: /index.php/settings/apps)
1.2.0

Server configuration

Operating system: Ubuntu/RedHat/...

Debian 8

Web server: Apache/Nginx
Apache 2.4,

Database: MySQL/Maria/SQLite/PostgreSQL
Postgresql 9.5

PHP version:
7.3.15 FPM (deb.sury.org)

Nextcloud Version: (see admin page)
17.0.3

Where did you install Nextcloud from:
tarball, updated since long ago

Signing status:
ok

List of activated apps:

Enabled:
  - accessibility: 1.3.0
  - activity: 2.10.1
  - admin_audit: 1.7.0
  - bruteforcesettings: 1.5.0
  - checksum: 0.4.4
  - circles: 0.17.10
  - cloud_federation_api: 1.0.0
  - comments: 1.7.0
  - dav: 1.13.0
  - deck: 0.8.0
  - dicomviewer: 1.2.1
  - end_to_end_encryption: 1.3.0
  - federatedfilesharing: 1.7.0
  - federation: 1.7.0
  - files: 1.12.0
  - files_accesscontrol: 1.7.0
  - files_automatedtagging: 1.7.0
  - files_downloadactivity: 1.6.0
  - files_fulltextsearch: 1.3.6
  - files_pdfviewer: 1.6.0
  - files_sharing: 1.9.0
  - files_trashbin: 1.7.0
  - files_versions: 1.10.0
  - files_videoplayer: 1.6.0
  - firstrunwizard: 2.6.0
  - fulltextsearch: 1.3.8
  - fulltextsearch_elasticsearch: 1.4.1
  - gallery: 18.4.0
  - groupfolders: 5.0.5
  - logreader: 2.2.0
  - lookup_server_connector: 1.5.0
  - metadata: 0.11.1
  - music: 0.13.0
  - nextcloud_announcements: 1.6.0
  - notifications: 2.5.0
  - oauth2: 1.5.0
  - password_policy: 1.7.0
  - previewgenerator: 2.2.0
  - privacy: 1.1.0
  - provisioning_api: 1.7.0
  - radio: 0.6.6
  - richdocuments: 3.5.2
  - serverinfo: 1.7.0
  - sharebymail: 1.7.0
  - sharelisting: 0.2.0
  - spreed: 7.0.2
  - support: 1.0.1
  - systemtags: 1.7.0
  - text: 1.1.1
  - theming: 1.8.0
  - theming_customcss: 1.5.0
  - twofactor_backupcodes: 1.6.0
  - twofactor_totp: 4.1.2
  - updatenotification: 1.7.0
  - viewer: 1.2.0
  - workflow_script: 1.2.0
  - workflowengine: 1.7.0

Nextcloud configuration:

Are you using external storage, if yes which one: local/smb/sftp/...
no

Are you using encryption: yes/no
no

Server log (data/nextcloud.log)

no errors

Remarks

I've written a shell script that only logs the signals it receives (only those that can be trapped at all). Some milliseconds after start the script receives a SIGTERM. I suspect systemd's cron implementation is sending this signal when the main PHP process (of cron.php) exits. This will be very quick given the fact that external scripts are called asynchronously, see lib/BackgroundJobs/Launcher.php#71. 90s later, systemd sends SIGKILL to all processes in that process group (and this is logged in systemd's journal).

With a similar configuration but with a native cron this doesn't occur: external scripts survive the termination of cron.php, which is the expected behaviour.

Please advise what to do in this situation.

Activity

  1. blizzz commented on Mar 2, 2020

    @blizzz
    Member

    We don't set a timeout, and, by heart, running the script should not finish the php process, but it waits for the execution first. Maybe I remember wrongly, as I wouldn't know how or why it would be killed.

    If it is native cron vs systemd, then the fix has to happen somewhere on the systemd layer.

  2. it25fg commented on Mar 26, 2020

    @it25fg
    Author

    I have tested further. Contributing factors could be:-

    1. sysdemd-cron instead of cron
    2. php-fpm
    3. Script in Perl. -> NO: a shell script behaves the same way.
    4. dash instead of sh -> NO: using dash (explicitly, not as 'sh'), or bash did not change anything.

    The behaviour I have seen is that cron.php lasts for no more than 2 secs, starts the script, and my script gets killed (no output, no log message) exactly when cron.php finishes. The only way to guard my script is to include a BEGIN block that sets $sig{CHLD,PIPE,TERM,INT} to 'IGNORE'. Then, it's at least immune against these signals, and behaves this way:-

    1. cron.php running
    2. cron.php running AND script running.
    3. cron.php vanished and script continues
    4. +90secs: script vanishes (sigkill can't be trapped)

    As soon as I remove the '&' from Launcher.php#71 the sequence changes to what you described:

    1. cron.php running
    2. cron.php running AND script running.
      --- continues until script has finished ---
      -> script prints its last log message, script and cron.php vanish.

    This would be my temporary solution if I encounter this on any Nextcloud installation. So you can close this issue: I have a workaround. With more and more systemd coverage I'd expect that others run into such problem too.

  3. ldidry commented on Jul 16, 2020

    @ldidry

    I confirm that using cron instead of systemd timer allows the external script to run until it’s normally finished.

    Thanks @it25fg for the hint, I was becoming crazy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions