[loggly] Learning notes: Comments from a 30-day free trial user for machine learning usage scenarios

This is a reference for the machine learning developers that want to choose loggly to 
monitor logs such as :
1. Training records (iter, epoch, accuracy, precision, recall ...)
2. CPU, Memory, GPU usage
3. Machines status (training on multi-machine)

My Requirements:
1.  Can figure out training logs sources (logs are sent by which machines, gpu, or task)
2.  Can view raw logs when run time training
3.  Can set alerts by specific conditions (ex: CPU, Mem, GPU alerts)
4.  Can integrate alerts with Slack or Hipchat
5.  Can show dashboard and query dashboard by customed time interval (ex: 1hr, 2hr, 7days)
6.  Lower price

Loggly vs Requirements:
1. Yes
Loggly have powerful searching ability and can manage logs from any source.
I use fluentd to collect logs, and then send logs to loggly by specific tags.
( loggly integrate fluentd )

<match your_match>
    type loggly
    loggly_url https://logs-01.loggly.com/inputs/TOKEN/tag/fluentd
</match>

TOKEN:  your customer token from the source setup page
fluentd: can change tag name here!! example fluentd configuration to monitor training logs and sys logs by task:

<match training_logs_match>
    type loggly
    loggly_url https://logs-01.loggly.com/inputs/TOKEN/tag/TrainingLogs-${Task}
</match>

<match sys_logs_match>
    type loggly
    loggly_url https://logs-01.loggly.com/inputs/TOKEN/tag/SysLogs-${Task}
</match>

Then, we can view specific-tag logs in summary dashboard
2. Yes
Beside, loggly can custom time interval in search.

3. Yes.
The operating logic in loggly is
(1) search by specific condition
(2) save your custom search query
(3) set alerts based on custom search query

The operations on "How to search" and "How to set alerts" are not friendly for the beginning user.
I hope Loggly team can use more video to introduce above operations.


4. Yes
Check following links:
https://www.loggly.com/docs/send-loggly-alerts-hipchat/
https://www.loggly.com/docs/slack-alerts/

5. No
I think this is the main drawback in loggly now.
In most usage, I may want to check the number of events before 1hr, 2hr,.. 7days.
It is very weird that can not custom time interval.
I hope Loggly team can add this feature in the future.


6.  Lower price ?
The price is a little expensive for my use case.
It depends on your case.
If you think loggly pro is affordable for you, why not ? :)
If you want to save money, maybe EFK(Elasticsearch, Fluentd, Kibana) is another a good solution.


Without pro, can not custom dashboards, set alerts and integrate alerts with hipchat.