Townsend Security Data Privacy Blog

"This article was originally posted on CorreLog's blog. CorreLog is a high performance correlation, search and log management company and Townsend Security partner."

That’s the thing about myths: they’re only partly true.

Yes, File Integrity Monitoring (FIM) has been part of the distributed computing landscape for a few years now. And yes, real-time enterprise security monitoring is harder to accomplish in a mainframe environment. But as attacks become more sophisticated, FIM needs to be a key component of the entire network, including your mainframe.

There’s a well-known software vendor that has an antivirus “sandbox” that is used to explode viruses in much like a police bomb squad would do with a suspicious package at a crime scene. After said software suspicious package is exploded, the software vendor adds the footprint to its database and the next time that package comes through the network, if it was clean the first time, it gets let through; if not, it’s blocked.

The tricky part of this story is that hackers are now smart enough to detect when they are about to be put into one of these sandboxes. When the A/V program starts to sandbox, the suspected virus, the virus goes into a cloaking mode to evade the sandbox. The A/V tool gives the executable a passing grade and there you have it; virus enters network.

Chances are you won’t have to worry about mainframe viruses anytime soon (though anything is possible these days). The point of the story here is the sophistication at which hackers are attempting to compromise corporate and government IP. For it is a much faster path to market to steal technology than it is to develop it. The same could be said for nation-state attackers who lack the subject matter expertise to develop their own IP, be it leading-edge technologies or schematics for nuclear reactors.

Having a Security Information & Even Management (SIEM) system with Data Loss Prevention (DLP), supported with antivirus detection and Identity/Access Management Systems (IAMS), gives you a fighting chance. Having a means for File Integrity Monitoring, especially on z/OS where the most strategic global banking and government data resides, further fortifies your security strategy and arms your information security team with another data point to determine the level of risk to your data.

FIM protocols are well established on the distributed side, but if you were to ask a mainframe sysprog (system programmer) if they have some type of FIM protocol in place for z/OS, they would look at you like you were speaking a foreign language. FIM on mainframe, or MFIM, must be addressed, at a minimum, to facilitate the Payment Card Industry Data Security Standard (PCI DSS) requirements 10.5.5; 10.6.1; 11.5; and 12.10.5. The standard and corresponding requirements can be found here, and we should note that this blog is not a review of the requirements. The takeaway in today’s blog is to understand that FIM is important to the Payment Card Industry Security Standards Council (and HIPAA, and FISMA, and IRS Pub. 1075, and others). PCI DSS lists FIM in four different requirements and it does not say “do this just on your Windows and UNIX systems.” The requirement says do this for all systems in your datacenter.

The key to MFIM is to look at the mainframe counterparts to the Microsoft Windows install folder. One of these, SYS1.PARMLIB, or the PARMLIB concatenation, is the most important set of datasets on z/OS, listing system parameter values used by nearly every component of z/OS. You can’t just take a checksum "snapshot" of these files, as a SIEM would do with distributed systems, because they’re simply too big for that to be a practical approach.

The details of tracking mainframe event messages are far too many to get into here. Essentially, you need a way of connecting the mainframe and your distributed SIEM system for notifications in real time, and you need a software tool that will convert mainframe events to a distributed event log format — RFC 3264 syslog protocol — so your SIEM system can interpret the data as actionable information.

You can learn more about MFIM and its relevance to PCI DSS from CorreLog’s complementary whitepaper titled “InfoSec Myths Debunked: FIM is only for Windows/UNIX.”

As we discovered in the blog Signs Your IBM i May Have Been Hacked, the combination of secure system logging on the IBM i and log monitoring with a SIEM will help you secure sensitive data and minimize the impact of security breaches.  Hopefully you were able to watch the webinar resource provided (if not, you can request it HERE).  After the webinar, we had a number of questions asked by attendees and answered by industry experts from Townsend Security and Integrity.  Here is a recap of that Q&A session:

Q: Do compliance regulations require system logging?

A: Most regulatory compliance standards such as PCI-DSS, FISMA, GLBA, and HIPAA/HITECH require organizations to monitor their network in real-time and provide audit reports. For the Payment Card Industry Data Security Standard (PCI-DSS), there are numerous logging requirements to be PCI compliant. Auditors want to look at how the logs are generated, whether it’s systematic or whether an operator can access/edit them, go in and pull them off and move them somewhere else. They want to look at if there’s mirrored events, where they go off the system through an automated process without any potential human intervention. It also details if people have the right privileges. Logs will show user events as well as what individuals are accessing libraries, files, or other areas outside of their designations. Logging is not only an industry best practice, it is a critical control to understanding access to a system.

Q: We have some custom applications that run our core business. Can a SIEM solution analyze the log files that come from these applications?

A: Dave Nelson from Integrity answers “Some SIEM applications are able to analyze log files from custom applications, others are not. Integrity’s SIEM can create a custom parser that can take just about any log that you can provide. Integrity can analyze that, we’ll work with your internal application development staff to identify what different error codes or security event log codes or whatever it is that you’re creating to identify a specific event. We can map that then into the parser then we can map those to either standard alerts or we can create new custom alerts, we can customize thresholds and a lot of different things. That’s one of the reasons that our customers choose us most frequently is because they have those internal applications that are custom that a lot of the other SIEM tools can’t handle, but we can handle and we can give them a lot of information about something that’s very unique to their business.”

Q: You mentioned File Integrity Monitoring (FIM), can you further explain how an organization would use it?

A:  It’s not every field that you’re going to want to alert and log and monitor on, but there might be ones with credit card numbers or store order authorization codes that you want to monitor and make sure the data hasn’t been altered or accessed without consent. The point to stress with logging and file integrity monitoring is ultimately it helps the individual system operator. You can have mirror alerts go to multiple people in the company, security officers as well as system operators. With FIM you take responsibility off of any one person having to follow up and do it all and you can create more of a collective team that analyzes this data to help the business.

Q: How can we distinguish a false alarm from a successful attack?

A: Sometimes it can be very difficult to determine a false alarm from a successful attack until you have done an entire investigation.  People that do this day in and day out and can begin to identify the patterns and trends of what makes an attack successful or not.  In our experience, the easiest way to do it is to look for key data points or key events that should have happened. One of the things you can do is jump right to the end if you know that a specific attack is successful, and work your way back through the system to determine the file name and creation date.  This really only comes with experience and practice of identifying the missing pieces.

Please post any additional questions you may have here on the blog!

For a much deeper and more detailed discussion on secure system logging and monitoring as essential controls to detect and mitigate the risk of a data breach, please request a download of the entire webinar:

(Based on a recent webinar with Townsend Security and Dave Nelson, President of Integrity)

Your IBM i may have been hacked and you don’t even know it yet!

Industry experts from Townsend Security and Integrity discuss how the combination of secure system logging on the IBM i and log monitoring with a SIEM will help you secure sensitive data and minimize the impact of security breaches. Topics cover (and go beyond) how log files and log data are the digital evidence (artifacts) that actually take us to a point of action within a system. They look at what the false alarms are within the plethora of data and how to screen those out. Then they also talk about the next steps: What are the red flags to watch for, and what to do with those red flags.

“As we look at the millions of data points that are created each day, every login or logout, every time a user is created, every time a user accesses a resource or adds a new resource or saves a file…. amidst all that data, hacking events happen. What we have to try and do is understand the ways that we can sift through that data and reduce the background noise and address the successful attacks.” (Dave Nelson)

Things to look for in log files as we’re trying to identify what’s real data, false alarms, or red flags:

New users and user accounts - Look for things like random names (like BSX or BS4XOR) and be able to identify new users. Always be able to trace these new user accounts back to a user account request and be able to identify which of those accounts have an approved resource and which ones have not.

New files and directories - Identify new directories, look for batches of files that show up between things that are normally next to each other. One of the things hackers love to do is hide files on any sort of Windows mountable or UNIX mountable directories within your i Series because a lot of times the IBM i doesn’t have an antivirus check or an antivirus application on it.

Date and time stamps - There are some (system) files that you know shouldn’t change. If you start to notice that those file modification dates or the save dates on those files and libraries have changed, that should start to be a red flag.

Significant increase or decrease in the size of a file or a library - Hackers will inject data into the back end of an existing file so that the file itself doesn’t change and it can still be executed. So watch for files that used to be a few kilobytes and are now a few megabytes or even gigabytes.

New processes or services that are running - Anytime you have a batch job that’s running and you’re not familiar with it, that should be something that you look at right away. Look for unusual interactive jobs working between LPAR’s or between systems. Do you normally have data leaving your IBM i and going to another platform? or a direct connection from a Windows server directly into your IBM i?

Cryptic or unusual file names - Create some sort of naming convention within your organization so that you know if something is outside of that standard.

It is suggested that we think of log files as the forensic evidence for the IBM i system and think about monitoring almost as a crime scene investigation. The relationship between the logging agent and the collector of those logs is very important because unexplained system value configuration changes, application changes, changes to privileges and privileged user profiles are indicators of potential malicious activity that you can record. These logging tools are strengths for an organization to really get to know what the system is doing as part of daily business activity, and then how to alert and monitor for data protection.

With all the different types of data that you can look for, the sheer volume of information that’s out there, there’s absolutely no way that an individual system administrator and application developer, even a full time security professional is going to be able to sift through that amount of information. Partnerships between the SIEM (Security Information and Event Management) collector and the logging agent are now industry standard defense and depth controls. Automation and email notifications about potential malicious activity can immediately give you the chain of custody to provide the digital evidence you require to go investigate further. You want to be able to drill down to specific threats, events, and user specific events as part of any good governance risk & compliance program and risk management approach. Essential for a total enterprise solution is the partnership (and strong encryption) between LogAgent and a SIEM.  

As a SIEM solution that partners with Townsend Security’s logging solution*, what Integrity’s done differently is provide a managed SIEM service. Dave tells us “We’ve got clients running this on the i Series platform using Alliance LogAgent to monitor, interfacing with our SIEM, and  they have said ‘Wow, we didn’t have any idea that we could get this much information and that it could be this easy to access and that we can share it’.  Clients want to be able to share that with their network administrators and say ‘See, this is what we’re seeing, we’re seeing this traffic and we don’t know why it’s coming in, can you please stop it and block it’.  One of the best things about Integrity’s SIEM solution from a cost perspective is that there’s no capital investment. You don’t have to spend $100,000 for the software, $50,000 for hardware and then go out and hire a full time person to review these logs and to set up the system and manage another system and application within the environment. It’s all provided for you for a low monthly cost. You get this in a matter of days and weeks instead of a matter of months. So you’re getting immediate return on your investment. In these economic times we all know how important that is to be able to show ‘Hey, we’re getting some real value for this expenditure that we’re making, we’re seeing a lot of things happening’. One of the other benefits is that you’re not going to see just security information from this. The amount of information that you’re going to get, you’re going to see operational things that you hadn’t seen in the past. You’re going to see things that you look at and say ‘Wow, we had no idea the system was operating that way, or those processes were running, or those jobs were running or taking so long to run’. The feedback that we get from our clients is that the value they get from the operational side of the SIEM is almost, if not as much, as what they get from the security side of the SIEM. So just being able to see deeper into the environment and seeing what’s happening, what’s going on has been great for a lot of our clients as well.”

*Townsend Security’s Alliance LogAgent is a comprehensive platform specific solution for IBM i which helps cut through the noise and deliver granular valuable data, providing file integrity monitoring right down to field level changes. Key steps you want and need for compliance purposes as well as data security.  

For a much deeper and more detailed discussion on secure system logging and monitoring as essential controls to detect and mitigate the risk of a data breach, please request a download of the entire webinar:

If these technologies are not in place, do you really know you haven't been hacked?

As we discussed in the blog Understanding System Logging on the IBM i – Part 1, secure system logging on the IBM i (AS/400) can not only help you meet compliance requirements, it can help you stop a data breach before it happens!  Hopefully you were able to watch the webinar resource provided (if not, you can request it HERE).  After the webinar, we had a number of questions asked by attendees and answered by security expert Patrick Townsend.

Here is a recap of that Q&A session:

Q: Can you pick which group of users to audit?

Patrick: In our current version of Alliance LogAgent, our IBM i system logging solution, you can define a list of “excluded users.”  So you can actually tailor events on the IBM i platform and exclude particular user profiles.  We also provide some basic filtering capability allowing you to filter based on IBM event type and on User event type.  So yes, there is a fair amount of filtering capability in the product.  From a security posture, it would be a mistake to filter out significant security events, however the solution allows you to easily choose and select the events you want.  We have an option that maps straight to the security system values so you can easily set LogAgent to process those or you can establish and select any and all of the event types in the product that you want to monitor.

Q: With all the different IBM system events, is it possible to set up filtering of just the IBM security event types?

Patrick: Yes, In our solution, collection of the system security events is set as the default “out of the box” setting.  You have total control over the events you collect, and with just a few keystrokes, you can re-map your collection and filter in additional events, or leave it set to what IBM identifies as the security type events being collected in QAUDJRN.

Q: Can File Integrity Monitoring (FIM) also monitor read access to a database file? What about a user that comes in from another server doing just a select in SQL?

Patrick: The File Integrity Monitoring component in LogAgent is based on a database journal. A record-level read event is not collected in a database journal; but a file open and closed event is collected.  So if a user does access a file through read and there’s no update, insert, or add type of event, it may not appear in the journal. We capture all the information that the system makes available to us in the file integrity monitoring component on significant changes or access to any protected file.

Q: You mentioned File Integrity Monitoring. Can you further explain how an organization would use it?

Patrick:  File Integrity Monitoring is designed to monitor configuration changes and highlight them for early detection.  Generally, when an attack happens on an application it often involves changes to configuration files in an attempt to escalate privileges and give the attacker more authority or access than they should have.  For example if a new user is suddenly granted authority in the HR application to print checks, that is an important indicator of a potential problem!  Another common use for File Integrity Monitoring is to monitor the use of stored sensitive information (credit card information or social security numbers) on your IBM i platform, you would want to use FIM on those databases or applications in order to strengthen your security stance and identify those potential attacks early on.

Q: How do you keep QAUDJRN entries from flooding the network?

Patrick: With the large volume of events that can be collected by QAUDJRN, you can really help minimize network impacts by filtering out events that are not security related.  You can still collect those events, but exclude them from being transmitted to your SEIM console.  Because LogAgent works in real time it helps reduce the impact on your network because you are not transmitting millions of events all at once. We also provide metering capabilities so you can reduce the number of events per second that are being transmitted.

Q: How do these logs get stored?

Patrick: When it comes to log storage, you want them off of your IBM i platform as quickly as possible to avoid potential tampering or corruption.  With LogAgent, we do not make a copy of the events, we take the filtered events out of QAUDJRN in real time, format and transfer them to your log collection server or SEIM.  Since some compliance regulations require the events are stored for a defined period of time, SEIM consoles compress, store, and protect those log events so you have the ability to do forensics, queries, and reporting on them.

To learn more, view our webinar "Understanding Log Management on the IBM i" which examines the security principles, compliance requirements, and technical challenges for log collection and forwarding on the IBM i platform with the following learning objectives:

• Security principles of log collection and monitoring
• Compliance requirements of PCI DSS, HIPAA/HITECH, SOX, GLBA/FFIEC
• Communicating with log collection and SIEM servers
• File Integrity Monitoring and log collection
• IBM i log collection challenges

1. Increased security of sensitive data

The number one advantage of File Integrity Monitoring (FIM) is increased security in your database(s). When you look at how data breaches happen, we often see a very similar chain of events. First, the data breach is discovered by someone inside the company, or a third party investigator. Second, the breach was discovered to have happened weeks, if not months ago. Third, the security holes in the IT infrastructure take several more weeks to plug. And finally, the database administrators discover that the breach could have been completely avoided using tools, such as file integrity monitoring. I won’t even go into the subsequent steps which also include data breach notification and paying hefty fines (an average data breach costs $5.5 million, by the way).

FIM allows you to see potentially harmful changes made in your database in real time. FIM helps you to detect early events by monitoring for changes to access controls, configurations, and all sensitive data at both database and application levels. For example, if you are storing social security numbers, credit card numbers, or other personally identifiable information (PII) on your IBM i, you can subject those fields to file integrity monitoring to catch any changes to that data immediately when it happens.

2. Comply with Industry regulations to pass your next audit

You should always know which data security regulations your organization must comply with. PCI DSS directly requires File Integrity Monitoring controls to prevent unauthorized access or changes to sensitive data (section 11.5). File Integrity Monitoring is also a critical component of the Sarbanes-Oxley (SOX) act for publicly traded companies. The Federal Information Security Management Act (FISMA) as well as the National Institute of Standards and Technology (NIST) also mention File Integrity Monitoring as a recommended security control.

3. Not a Matter of If, but When

There’s a really, really good reason why governments and industries are imposing more and more stringent data security regulations on both public and private organizations: the number of data breaches occurring every year is not slowing down. It’s speeding up! A common sentiment these days is that a data breach within your company isn’t a matter of “if”, but “when”. Think about it this way: How many times have you received a call from your bank informing you that your credit card has been compromised and they are issuing a new number? Once? Twice? Three times? More? The unfortunate reality is that even though data breaches run rampant like wildfire, many businesses are doing too little or nothing at all to protect their data. When the fire hits your business, I bet you won’t be thinking, “good thing I didn’t waste my time on fire alarms and home owner’s insurance!”

For more information on file integrity monitoring and meeting data security compliance regulations, check out our podcast, “File Integrity Monitoring on the IBM i”, featuring Patrick Townsend, founder and CEO of Townsend Security.

As an IBM i security administrator, don’t you sometimes wish that your IBM i would just talk to you? Like a good friend, just tell you stuff that you need to know? The IBM security journal, QAUDJRN, can collect millions of events every day. But there is no way you can really sort through all of that. And, there isn’t even any information that tells you about your sensitive application file changes. It would be great if your IBM i would just inform you of bad stuff that is happening.

Perhaps it could say something like:

“Uh oh. Bill in the shipping department just changed the payroll file and gave himself a really big raise.  He’s now making major Wall Street coin. You might want to check on that.”

Or

“Whoa, big red flag here! Sally in accounting just transferred the credit card history file to an FTP server at 3 AM. There’s no way that should be happening!”

Yes, you DO want to know about these things before it hits the front page of the New York Times.

Actually, this kind of information is now easily within reach. Security applications that perform File Integrity Monitoring (FIM) do just this kind of thing. They let you define a sensitive file, perhaps a configuration file or a file containing credit card information, and then let you set up monitoring rules. You should be able to know in real time when an unauthorized user or application changes a file, and define upper and lower limits for alerts.

That’s exactly what our new Alliance LogAgent Suite with Database Monitoring does. OK, it doesn’t actually talk to you. But it speaks the language of your log monitoring solution. Here’s how it sends the alert about Bill:

<118>Mar 9 15:25:14 S10125BA LogAgentDB:[LGADB@0 column_name="SALARY"

column_text="Annual salary" SECURITY_ALERT_upper_limit="yes"

data_type="P" action="Update" data_image="After" value_option="Clear"

previous_value="35000" value="2800000" file_name="HRMASTER"

file_library="HRPROD" file_member="HRMASTER"

timestamp="20120309152514783008" job_name="QPADEV000K" job_user="BILL"

job_number="648169" jrn_seq="81327" jrn_sys_seq="0" user_profile="BILL"

program_name="QDZTD00001" program_library="*OMITTED"

In this case a whitelist of users was associated with the SALARY field in the HR master file. Bill was not on that list, and Alliance LogAgent raised the security alert message. Everything is in the message that you need to work on this potential problem. You know the date and time that Bill made the change. You can see that the program was the IBM DFU program, a file utility that is never used for real work on HR data. And you can see the previous and new values for the salary. And all of this happens in real-time giving you the chance to head-off a possible big problem.

Alliance LogAgent Suite with Database Monitoring is an affordable and easy-to-deploy solution. It will help you implement a FIM solution to meet PCI DSS and other compliance regulations. Listen to our podcast "File Integrity Monitoring on the IBM i" to learn more about selectively monitoring data access and change activity at the column or field level.

Patrick