Logs, in essence, are files that record specific events, when they occurred, where they occurred, and other information related to the incident, depending on the parameters configured. . These are very important when it comes to security and troubleshooting – whether we look at logs to determine the root cause of problems or see if processes are working properly, logs notify us of events occurring in the system and play an important role in providing information to us.
Log format
Logs can be in many different formats. If your logs come from Windows Event Viewer, you can use the .csv, .XML, .evtx and .txt extensions when saving them.
On the other hand, if you are using a Linux system, the log will usually have the .log extension.
Where can I find the diary?
In a Windows environment, the event log will usually be stored in the folder C:Windowssystem32config. Note that to modify, view or edit log files, you will need elevated privileges.
For Access Log in IIS, by default you can find at the path: %SystemDrive%inetpublogsLogFiles
In a Linux environment, the log will usually be stored in the path /var/log. Depending on the log file, you will need the appropriate permissions to read or modify it.
For the Access Log in Apache, Nginx or other Webservers you can find it in the service’s config.
When to log?
In an enterprise environment, almost all devices in the network can be logged. From workstations, endpoints, routers, switches, firewalls and proxies to IoT, logging can be a very effective tool to understand exactly what is happening in these systems. However, it is important to keep a quality log.
Logging events unrelated to your goals can cause informational noise, which often has more disadvantages than benefits. You certainly don’t want to go through lots of unrelated information files just to find something useful to you. It also consumes a lot of storage space.
The goal should always be to log critical events that indicate whether processes are working as expected, unusual traffic, etc.
How much disk space is used when logging?
Depending on the organization and the devices used, the number of EPS (Events Per Second) may vary. For example, on average, a Windows Workstation might have an EPS of 1.0, representing one event per second, while a firewall might have an EPS of 10.0–50.0, representing 10–50 events every second.
However, you also need to consider peak traffic times, as these are often times when a lot of unusual things happen. To calculate how much space is needed on a given day, you can use the following equations:
EPS (per device) * 86,400 (seconds in a day) * 600 (size of an average raw event, in bytes) = size of the uncompressed log file in bytes for a given day, and a single device considers those events stored in raw form.
EPS (per device) * 86,400 (seconds in a day) * 1500 (size of a single event normalized on average, in bytes) = size of the uncompressed log file in bytes for a single event certain date and for a single device considering those events are stored in its normalized form.
So what is the difference between raw event and normalized event?
In essence, a normalized event is a modified version of the raw event. The content of the raw event is rearranged and additional information is added to make the event easier to read and understand for security administrators. This is a very important step to ensure that the logs are providing the information needed to properly secure the system. However, the normalized events are heavier than the raw ones.
Compression
Storing these log files in an uncompressed form can take up a lot of storage space. Therefore, we must use compression to reduce storage space. An 8:1 ratio can be a good starting point, but it is important to take into account variables such as impact on the SIEM system, amount of normalized data versus raw data, etc.
Conclusion
Logs are an important element of Cybersecurity. When implemented properly, logging can provide us with a clear view of what is happening or has happened in our system, allowing us to attest to events with great accuracy. higher accuracy and make adjustments as needed.
Understanding where and what to record is important if we want to extract valuable data from this process. Logging can be costly in storage space, so it’s important to consider various factors (EPS, NE (Normal Event), PE (Peak Event), normalized events and raw, compression speed, number of devices, potential free space, etc.) when allocating storage space resources to write files.