Cofacts chatbot error analysis 2023/01/21 ~
Operation / code changes
- 2023/1/21: Update API server
- https://github.com/cofacts/rumors-api/releases/tag/release%2F20230121
- 2023/2/3 2 a.m.: Update LINE bot, API and site server
- Changes log format so that each conversation emits one line of log instead of multiple lines
- Revives group chat
- Deploys API and website changes tested in https://g0v.hackmd.io/@mrorz/cofacts-meeting-notes/%2FiGqCnbuoQfKlZLTNEyM3dw
- 2023/2/7 2 a.m.: Try to rollback API change on 1/21 by reverting to an image 8 weeks ago (should be https://github.com/cofacts/rumors-api/releases/tag/release%2F20221213 )
- Error rate did not drop, so it’s not due to 1/21 code release
- Error rate did not drop, so it’s not due to 1/21 code release
- Error went away on 2/11
Linode server load <> LINE chatbot errors
- Upper: linode CPU load (%)
- Lower: chatbot errors (request timeout)
- No significant surge in CPU, but error happens every day starting from 1/21
Overall CPU load in 30 days
Out-going network spike on 1/21
- Reason: unknown
Nginx request count <> LINE chatbot errors
- Upper: nginx (includes HTTPs requests to API, chatbot, website,… etc)
- Lower: chatbot errors (request timeout)
- No significant surge in nginx, but error happens every day starting from 1/21