故障现象

  • 在线重启更新旧版本NASTool时，重启后失联，无法访问WEB页面。

  • 亦或者无法自动目录同步/只能部分目录同步。

  • 故障版本约为3.3.13版本开始出现

  • 查看日志，发现启动时报告错误日志：Exception: [Errno 28] inotify watch limit reached

  • 错误日志

    Exception: [Errno 28] inotify watch limit reached
Callstack:
Traceback (most recent call last):
  File "/nas-tools/app/sync.py", line 348, in run_service
    observer.start()
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/api.py", line 261, in start
    emitter.start()
  File "/usr/local/lib/python3.10/site-packages/watchdog/utils/__init__.py", line 92, in start
    self.on_thread_start()
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/inotify.py", line 119, in on_thread_start
    self._inotify = InotifyBuffer(path, self.watch.is_recursive)
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/inotify_buffer.py", line 37, in __init__
    self._inotify = Inotify(path, recursive)
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/inotify_c.py", line 179, in __init__
    self._add_dir_watch(path, recursive, event_mask)
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/inotify_c.py", line 395, in _add_dir_watch
    self._add_watch(path, mask)
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/inotify_c.py", line 416, in _add_watch
    Inotify._raise_error()
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/inotify_c.py", line 428, in _raise_error
    raise OSError(errno.ENOSPC, "inotify watch limit reached")
OSError: [Errno 28] inotify watch limit reached


Pending jobs:
    SiteUserInfo.refresh_site_data_now (trigger: interval[6:00:00], next run at: 2024-05-20 11:54:21 CST)
    Rss.rssdownload (trigger: interval[0:30:00], pending)
    Subscribe.subscribe_search_all (trigger: interval[6:00:00], pending)
    MediaServer.sync_mediaserver (trigger: interval[12:00:00], pending)
    MetaHelper.save_meta_data (trigger: interval[0:10:00], pending)
    Sync.transfer_mon_files (trigger: interval[0:01:00], pending)
    Subscribe.subscribe_search (trigger: interval[0:05:00], pending)
    Subscribe.refresh_rss_metainfo (trigger: interval[6:00:00], pending)
    MetaHelper.delete_unknown_meta (trigger: interval[12:00:00], pending)
    get_login_wallpaper (trigger: interval[1:00:00], next run at: 2024-05-20 11:53:21 CST)
Pending jobs:
    AutoSignIn.sign_in (trigger: date[2024-05-20 11:53:25 CST], pending)
    AutoSignIn.check_missed_signs (trigger: date[2024-05-20 12:09:46 CST], pending)
    SchedulerUtils.start_job.<locals>.start_random_job (trigger: cron[hour='1', minute='0'], pending)
Pending jobs:
    DoubanSync.sync (trigger: interval[6:00:00], pending)
Traceback (most recent call last):
  File "/nas-tools/run.py", line 114, in <module>
    start_service()
  File "/nas-tools/run.py", line 107, in start_service
    start_config_monitor()
  File "/nas-tools/initializer.py", line 418, in start_config_monitor
    _observer.start()
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/api.py", line 261, in start
    emitter.start()
  File "/usr/local/lib/python3.10/site-packages/watchdog/utils/__init__.py", line 92, in start
    self.on_thread_start()
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/inotify.py", line 119, in on_thread_start
    self._inotify = InotifyBuffer(path, self.watch.is_recursive)
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/inotify_buffer.py", line 37, in __init__
    self._inotify = Inotify(path, recursive)
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/inotify_c.py", line 179, in __init__
    self._add_dir_watch(path, recursive, event_mask)
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/inotify_c.py", line 395, in _add_dir_watch
    self._add_watch(path, mask)
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/inotify_c.py", line 416, in _add_watch
    Inotify._raise_error()
  File "/usr/local/lib/python3.10/site-packages/watchdog/observers/inotify_c.py", line 428, in _raise_error
    raise OSError(errno.ENOSPC, "inotify watch limit reached")
OSError: [Errno 28] inotify watch limit reached
Error submitting job "AutoSignIn.sign_in (trigger: date[2024-05-20 11:53:25 CST], next run at: 2024-05-20 11:53:25 CST)" to executor "default"
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/apscheduler/schedulers/base.py", line 988, in _process_jobs
    executor.submit_job(job, run_times)
  File "/usr/local/lib/python3.10/site-packages/apscheduler/executors/base.py", line 71, in submit_job
    self._do_submit_job(job, run_times)
  File "/usr/local/lib/python3.10/site-packages/apscheduler/executors/pool.py", line 28, in _do_submit_job
    f = self._pool.submit(run_job, job, job._jobstore_alias, run_times, self._logger.name)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
Error submitting job "SiteUserInfo.refresh_site_data_now (trigger: interval[6:00:00], next run at: 2024-05-20 11:54:21 CST)" to executor "default"
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/apscheduler/schedulers/base.py", line 988, in _process_jobs
    executor.submit_job(job, run_times)
  File "/usr/local/lib/python3.10/site-packages/apscheduler/executors/base.py", line 71, in submit_job
    self._do_submit_job(job, run_times)
  File "/usr/local/lib/python3.10/site-packages/apscheduler/executors/pool.py", line 28, in _do_submit_job
    f = self._pool.submit(run_job, job, job._jobstore_alias, run_times, self._logger.name)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Error submitting job "Sync.transfer_mon_files (trigger: interval[0:01:00], next run at: 2024-05-20 11:54:21 CST)" to executor "default"
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/apscheduler/schedulers/base.py", line 988, in _process_jobs
    executor.submit_job(job, run_times)
  File "/usr/local/lib/python3.10/site-packages/apscheduler/executors/base.py", line 71, in submit_job
    self._do_submit_job(job, run_times)
  File "/usr/local/lib/python3.10/site-packages/apscheduler/executors/pool.py", line 28, in _do_submit_job
    f = self._pool.submit(run_job, job, job._jobstore_alias, run_times, self._logger.name)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
root@nas:~#

解决办法

  1. 演示在群晖系统上进行操作

  2. 打开群晖的SSH功能，使用终端软件进行连接，并切换到root状态下。

  3. 复制下面的三条命令，依次执行。