How to Read From Specific Location in Splunk
Leveraging Windows Issue Log Filtering and Design Techniques in Splunk
When working with Windows event logs in your Splunk environment it's typical to come across two scenarios: How practice I go rid of specific events that aren't necessary for my utilise case? How do I trim off the duplicated text at the bottom of events that'southward chewing up my license?
The answer to both of these questions is by leveraging the avant-garde filtering techniques at the input level and event routing at the indexing level. In this tutorial, I'll explain how you can do both of these things so you just bring in the information you need. Before we get started, y'all should consider a strategy for how you ingest your Windows event logs. You tin default to permit all with explicit denies, default to deny all with explicit allows, or a hybrid of explicit allows/denies.
Information technology's important to sympathize that by default all event codes volition exist indexed if you exercise non specify a whitelist. If you add asingle whitelist statement, Splunk will only index events which match your whitelist for that particular input stanza and ignore the rest of the events. You should also note that Splunk processes whitelists showtime, then blacklists. This means you lot can combine whitelists/blacklists together to reach a certain effect (I.Due east, default allow all in X eventcode, just deny specific strings in X eventcode).
The primary benefit of whitelists/blacklists for Windows Event Logs is that we get to do the filter at the ingestion pipeline instead of at the typing pipeline, which is how filtering is traditionally handled in Splunk. This means you lot can filter out data before information technology's always sent over the wire and save yourself from wasting precious bandwidth and compute cycles on your indexers. The caveat being the disability to modify event text, so if you want to do that y'all still demand to practice this on the indexers (which I will become over too).
Lastly, I will cover how y'all tin can construction your inputs deployment in a layered approach. This is going to give you more control over what data yous're bringing in and allow you to more easily manage what hosts send what data every bit your environment grows.
Combining these strategies volition go y'all the most bang for your cadet by optimizing your Windows event log data ingestion.
Deployment Strategies
Give Me All the Events, Except Certain Ones
This is the most mutual approach to working with Windows Event Logs, and it's typically the easiest style to get your desired result. With this method you are never declaring a whitelist. So, the default behavior is to catch all outcome codes under that Event Log Channel. To filter down yous and then configure blacklists to drop specific event codes that you lot do not need. Once you take your standard result code blacklist, you can hone in on specific events which aren't useful and use avant-garde filtering techniques to driblet those.
Only Give Me Specific Events
This is another mutual approach often used when you have a limited amount of license capacity to piece of work with. This method is going to crave that you explicitly whitelist exactly what yous want under each of your eventlog stanzas, and you are non going to be using whatever blacklists. When you use this method, you want to be conscientious you lot're not missing precious logs that yous forgot to whitelist.
It'due south Complicated
This is less of an arroyo and more of a reactionary configuration. Unremarkably happens when you didn't follow one of the other ii approaches. The other possibility is y'all accept some complex requirements that make it necessary to juggle whitelists/blacklists to get what you want. As in the previous school of thought, you need to be careful not to miss important result codes.
Input Layering
The to a higher place image shows how you should visualize this approach. The idea is that y'all're going to split up your configuration into multiple apps in gild to apply more granular configuration to the correct prepare of hosts. This is something nosotros consider to be a best do and will exist useful for more than than just your Windows Event logs.
Basic Filtering
At present let's talk about how nosotros become started with filtering. For many environments, you can get away with using just basic filtering. By "bones filtering" I'm referring to straightforward whitelists/blacklists that only filter on upshot codes.BEWARE: You tin can NOT mix bones and advanced formats under the same wineventlog stanza, this will break your ingestion for that log source. For this reason, I would recommend doing your basic filters in the advanced filtering form and then that you can easily expand in the future if needed (example in lawmaking block beneath).
The first affair you need to practise is find the inputs.conf file that is specifying your Windows Result Log stanzas–a lot of people put this in the local folder of the "Splunk_TA_Windows" and deploy the app to all of their Windows hosts. I would caution against this because every bit your environment grows you will likely need to start creating dissimilar apps that plow on specific inputs stanzas with specific event codes for sets of hosts. You should think about this as separating DC'south, App Servers, Exchange Servers, etc.
Later in this web log mail, I volition comprehend how you can do this if you aren't already. Regardless of your setup the approach here is however the same, though you may need to change your app context.
Nether the stanza (for example, [WinEventLog://Security]), yous can specify either whitelist or blacklist, followed past a number if you accept multiple whitelists/blacklists. Each stanza can support upwardly to 10 whitelists and upward to ten blacklists (I.Eastward, whitelist & whitelist1-9). Yous should notation that by default the Windows Add together-on comes with blacklist1 and blacklist2 already in utilise. Once you define your list, yous but specify which event codes apply to that listing, using commas and dashes to intermission it up into groups. Commas volition signify yous're ending that "group", and dashes are used to specify a range for the "grouping".
Here's an example of what your stanza might expect like this:
To break this down:
- No whitelist means nosotros will ingest ALL Security event logs.
- Blacklist3 will be read after our implicit whitelist, which ways we will Not receive anything listed in the blacklist. In this instance we won't get the result codes listed in the range/single keys.
- Note: The reason we're using blacklist3 is considering the Windows add-on comes with blacklist1 and blacklist2 already defined nether this stanza in default/inputs.conf.
Advanced Filtering
What happens when the basic filtering doesn't get the job done? Are yous getting spammed with a particular cord only you demand the events in that event code? Or what if a unmarried service account is generating big amounts of logs between two DC'south?
This is where y'all can utilise advanced filtering techniques to become really granular and selective on the incoming data.
When creating a whitelist or blacklist, you are really operating on a set of key-value pairs. Find I say "set" because you can actually use multiple key-value pairs together (i.e. an event lawmaking AND role of a message). These key value pairs use regular expressions to match on your event code text. Information technology'southward important to know that you can only specify a key once, and if you lot specify it multiple times it will only use the last duplicated primal in the list.
You can observe a list of possible keys in the Splunk Docs Create advanced filters with 'whitelist' and 'blacklist' department.
For this post, we're going to focus on ComputerName, Message, and User as those are the most commonly used keys from my feel. ComputerName and User are pretty cocky explanatory, you lot can apply these in conjunction with an consequence code in order to whitelist/blacklist certain events.
Let's interruption down these examples beneath:
blacklist = EventCode="4663″ ComputerName="(US-EXC-01|European union-EXC-01)\.COMPANY\.com"
- EventCode – Only apply this blacklist to Security Event Logs where the effect code is 4663.
- ComputerName – Only apply this blacklist to Security Effect logs where the Computer Proper noun is "US-EXC-01.COMPANY.COM" or "EU-EXC-01.COMPANY.COM".
blacklist3 = EventCode="4624″ User="HealthMailbox"
- EventCode – Only apply this blacklist to Security Upshot Logs where the event code is 4624.
- User – Only apply this blacklist to Security Event Logs where the User is "HealthMailbox".
In these two blacklist examples we were facing a common issue.
For the kickoff i, a couple of servers were generating a very large corporeality of "An attempt was made to access an object" logs, which were non helpful to our use case and also taking up a lot of deejay/license for no reason. In the second blacklist, nosotros had a service account that was too generating a big volume of "An business relationship was successfully logged on" events that were expected and not useful to our employ case.
The last primal nosotros'll talk about is the Bulletin central, which is likely going to be the most important i. Often what you'll find is that y'all have some events where you merely care well-nigh specific values in the event which don't accept a key divers.
One example would be if you wanted to ingest logs that let y'all to detect Kerberoasting. You lot would want to make certain y'all are whitelisting Event ID 4768 + 4769 with Ticket Encryption types 0x1, 0x3, 0x11, 0x12, 0x17, and 0x18. The outset thing y'all'd notice is that "Ticket Encryption Type" is Non a key value listed in the docs. This ways you will have to apply Regex to match on the Message central instead.
Here'southward what that config ends up looking like:
- EventCode – Only apply this blacklist to Security Issue Logs where the consequence code is 4768 or 4769.
- Message – Only employ this blacklist to Security Effect Logs where the Message field contains the Ticket Encryption Types of 0x1, 0x3, 0x11, 0x12, 0x17, or 0x18.
When dealing with the Message field, it's important to remember that these are multi-line events. What appears to work in your Regex tester may not translate into Splunk'south Regex engine where it will try to decide between using the default way or multi-line mode. To be safety, y'all will want to manually specify the multi-line flag "(?m)" to force Splunk to utilise that mode. Multi-line mode causes "^" and "$" to lucifer the brainstorm/end of each line (not but brainstorm/end of string).
In this case, if we were to leave off the multi-line flag we would NOT end upwards with the expected behavior, this would event in logs non beingness ingested. Splunk would take parsed the entire result as a string and therefore interpret our regex with the "$" indicating the very cease of the event. Instead, what we needed was for Splunk to lucifer on the stop of the ticket encryption type line, and then nosotros did non accidentally match types that started with a 1 or a three (such as 0x13, 0x14, or 0x32). Our regex grouping makes sure it will only capture the values we intendance about for Kerberoasting.
If we accept this advanced example a bit farther, what would happen if nosotros wanted all of that plus only whitelist where the service name was "krbtgt"?
This is where some other flag modifier will come up in handy — "(?ms)". The "southward" flag allows your dots to match newline characters. This lets us practise what you see in "whitelist2" instead of what is done in "whitelist3" (as you lot can encounter, it's a huge difference). These two flags are going to be critical to sympathise in lodge for your whitelists and blacklists to work.
Saving License
In this section I wanted to encompass a best exercise that we have for our customers, which is saving them from unnecessary license usage. If you lot've spent any fourth dimension with Windows upshot logs, you lot'll quickly see a design where many event codes contain a wall of text at the bottom of each event which provide no value.
If y'all take a look at this result, you can see that y'all are actually using up more deejay space/license on the ending event description than the actual event text. There is no do good to keeping this text in the event. The way you would get rid of this text is by using a props/transforms set that discards that text and keeps the rest of the consequence in tact.
Remember, this configuration needs to continue the first SplunkEnterprise system where your Windows Event Logs are being forwarded (hopefully straight to your indexers).
Nosotros can do this easily with the configuration instance below:
Let's dive into what exactly is happening with this transforms call.
Offset we're setting a large lookahead value that allows u.s.a. to grab all the actual event text we care most. Next we're using regex to capture the entirety of the outcome text and shop it in a capture group, leaving off the wall of text that nosotros don't care well-nigh which starts with "This result is generated".
Ultimately this entire operation is going to discard everything except what is within the capture group when it writes the result to a saucepan.
Now, if nosotros have a peek at the props stanza, you'll notice I take both a source and sourcetype stanza for the same thing. I felt this was worth noting because in the older version of the Windows Addition all of the configuration was done by sourcetype. In the newer version of the Windows Add-on all of the configuration is washed by source instead of sourcetype. If yous're not certain which one to use, just become with the source phone call because that should work for the new and sometime version alike.
It's also important to remember that you are responsible for understanding how this willpermanently modify your own production information in Splunk, so please be careful!
How Do I Apply This?
At present that you have all the data you need to make these changes, y'all might be wondering "Where on Globe does this configuration get?"
I'm going to assume you're following Splunk best practices, significant you lot have a Deployment Server setup with your Splunk Universal Forwarders configured as clients and a standalone indexer (or index cluster) where those universal forwarders send information to. If your environment is setup differently, you may accept to adjust this process and put the configuration somewhere else.
Whitelists, Blacklists, and Input Layering
TL;DR: Get your inputs.conf (optionally containing whitelists/blacklists) to your UF's using a Deployment Server.
If you accept administrative experience with Splunk, yous're probably used to putting configuration like to this on an indexer or heavy forwarder since it'south altering data you index. Winevent Log whitelists and blacklists are a special exception because these operate at the input level, direct on the UF (they have a special pipeline/processor ready). This means nosotros need to put this configuration on the Deployment Server within $SPLUNK_HOME/etc/deployment-apps.
If yous think back to earlier, you'll remember that I mentioned most people will put this configuration direct in Splunk_TA_windows/local/inputs.conf. However, since you're reading this, I hope to steer you in a direction that volition allow you to make this configuration more than modular every bit your environment scales.
Leave "Splunk_TA_windows" alone, don't modify it at all. Instead, create a set of apps post-obit a naming scheme. You lot want to remember about how you lot tin apply this with a layered strategy to create "base" layer then add whatsoever custom layers on top which may exist applied to a specific server or gear up of servers.
Here's an instance of what this list might look like when yous're done:
The uf_winevent_base_inputs would exist deployed to all of your Windows systems, and you lot would deploy other apps as needed depending on the part of each server.
For example, your Domain Controllers would end up with both the uf_winevent_base_inputs and uf_winevent_ad_inputs apps in this example. Remember that if you lot have overlapping whitelist/blacklist numbers in two apps, lexicographical order is going to determine which whitelist/blacklist wins (meaning yous might need to adjust an app name then that it wins).
One time yous take your naming scheme and apps created yous're going to define the stanzas yous want in inputs.conf with the whitelists/blacklists configured under each relevant stanza. Your end result might wait something like this:
Now that you have your strategy in identify, you need to create serverclasses which follow the same design of 1 serverclass per "layer".
For simplicity sake, I similar to follow the same naming convention and phone call the serverclass something similar to the relevant app name. Lastly, make sure when yous create your server classes to watch the scope of your customer list so that you lot're not missing whatever hosts or applying configuration to the wrong hosts.
Somewhen, what you lot end up with may look similar this on your deployment server:
Props and Transforms
TL;DR: Go your props.conf & transforms.conf to the Splunk Enterprise system that your UF's forwards winevent logs to.
At present what nigh applying that configuration that got rid of those long walls of text on the Windows Events? In a Splunk deployment matching current all-time practices for receiving data from Universal Forwarders, this configuration is ultimately going to demand to go along your indexer(s). If you take a standalone indexer, it should exist configured equally a customer on your deployment server. In this setup, you would put the configuration in an app on your deployment server in $SPLUNK_HOME/etc/deployment-apps/ and add the app to the serverclass for your indexer. If you're using an indexer cluster, then you want this configuration to go in $SPLUNK_HOME/etc/chief-apps/, then apply your cluster package.
Regardless of how you're setup, your configuration should look something like this (nosotros'll use the app 'baseline_windows_props'):
About Hurricane Labs
Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the U.s.. With a defended, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to assist brand our customers' lives easier.
For more than information, visit world wide web.hurricanelabs.com and follow us on Twitter @hurricanelabs.
Source: https://hurricanelabs.com/splunk-tutorials/leveraging-windows-event-log-filtering-and-design-techniques-in-splunk/
Post a Comment for "How to Read From Specific Location in Splunk"