Some indication of the integrity of a particular peer can be gleaned directly from the Domain Name System (DNS), even before SMTP commands are issued. In particular, various DNS blacklists can be consulted to find out if a particular IP address is known to violate or fulfill certain criteria, and a simple pair of forward/reverse (DNS/rDNS) lookups can be used as a vague indicator of the host's general integrity.
Moreover, various data items presented during the SMTP dialogue (such as the name presented in the Hello greeting) can be subjected to DNS validation, once it becomes available. For a discussion on these items, see the section on SMTP checks, below.
A word of caution, though. DNS checks are not always conclusive (e.g. a required DNS server may not be responding), and not always indicative of spam. Moreover, if you have a very busy site, they can be expensive in terms of processing time per message. That said, they can provide useful information for logging purposes, and/or as part of a more holistic integrity check.
DNS blacklists (DNSbl's, formerly called "Real-time Black-hole Lists" after the original blacklist, "mail-abuse.org") make up perhaps the most common tool to perform transaction-time spam blocking. The receiving server performs one or more rDNS lookups of the peer's IP address within various DNSbl zones, such as "dnsbl.sorbs.net", "opm.blitzed.org", "lists.dsbl.org", and so forth. If a matching DNS record is found, a typical action is to reject the mail delivery. 
If in addition to the DNS address ("A" record) you look up the "TXT" record of an entry, you will typically receive a one-line description of the listing, suitable for inclusion in a SMTP reject response. To try this out, you can use the "host" command provided on most Linux and UNIX systems:
host -t txt 18.104.22.168.dnsbl.sorbs.net
There are currently hundreds of these lists available, each with different listing criteria, and with different listing/unlisting policies. Some lists even combine several listing criteria into the same DNSbl, and issue different data in response to the rDNS lookup, depending on which criterion affects the address provided. For instance, a rDNS lookup against sbl-xbl.spamhaus.org returns 127.0.0.2 for IP addresses that are believed by the SpamHaus staff to directly belong to spammers and their providers, 127.0.0.4 response for Zombie Hosts, or a 127.0.0.6 response for Open Proxy servers.
Unfortunately, many of these lists contain large blocks of IP addresses that are not directly responsible for the alleged violations, don't have clear listing / delisting policies, and/or post misleading information about which addresses are listed. The blind trust in such lists often cause a large amount of what is referred to as Collateral Damage (not to be confused with Collateral Spam).
For that reason, rather than rejecting mail deliveries outright based on a single positive response from DNS blacklists, many administrators prefer to use these lists in a more nuanced fashion. They may consult several lists, and assign a "score" to each positive response. If the total score for a given IP address reaches a given threshold, deliveries from that address are rejected. This is how DNS blacklists are used by filtering software such as SpamAssassin (Spam Scanners).
One could also use such lists as one of several triggers for SMTP transaction delays on incoming connections (a.k.a. "teergrubing"). If a host is listed in a DNSbl, your server would delay its response to every SMTP command issued by the peer for, say, 20 seconds. Several other criteria can be used as triggers for such delays; see the section on SMTP transaction delays.
Another way to use DNS is to perform a reverse lookup of the peer's IP address, then a forward lookup of the resulting name. If the original IP address is included in the result, its DNS integrity has been validated. Otherwise, the DNS information for the connecting host is not valid.
Rejecting mails based on this criterion may be an option if you are a militant member of the DNS police, setting up an incoming MX for your own personal domain, and don't mind rejecting legitimate mail as a way to impress upon the sender that they need to ask their own system administrator to clean up their DNS records. For everyone else, the result of a DNS integrity check should probably only be used as one data point in a larger set of heuristics. Alternatively, as above, using SMTP transaction delays for misconfigured hosts may not be a bad idea.
Similar lists exist for different purposes. For instance, "bondedsender.org" is a DNS whitelist (DNSwl), containing "trusted" IP addresses, whose owners have posted a financial bond that will be debited in the event that spam originates from that address. Other lists contain IP addresses in use by specific countries, specific ISPs, etc.
For instance, the outgoing mail exchangers ("smart hosts") of the world's largest Internet Service Provider (ISP), comcast.net, is as of the time of this writing included in the SPEWS Level 1 list. Not wholly undeserved from the viewpoint that Comcast needs to more effectively enforce their own AUP, but this listing does affect 30% of all US internet users, mostly "innocent" subscribers such as myself.
To make matters worse, information published in the SPEWS FAQ states: The majority of the Level 1 list is made up of netblocks owned by the spammers or spam support operations themselves, with few or no other legitimate customers detected. Technically, this information is accurate if (a) you consider Comcast a "spam support operation", and (b) pay attention to the word "other". Word parsing aside, this information is clearly misleading.